Skip to main content
Table of contents

Monitoring apps

App metrics

Cloud Foundry provides time series data known as metrics for each instance of your PaaS app. You can receive, store and view this data in a monitoring system of your choice by deploying either the:

You can also view all metrics in a one-off snapshot by installing the Cloud Foundry CLI log cache plug-in.

Use the PaaS Prometheus exporter app

The PaaS Prometheus exporter collects metrics from your apps and any backing services configured to send metrics to the PaaS.

To use the PaaS Prometheus exporter, deploy it as an app on the GOV.UK PaaS. Refer to the PaaS Prometheus exporter readme documentation for more information on supported metrics.

Prerequisites

Before you set up the PaaS Prometheus exporter app, you’ll need a:

  • Prometheus service to request or ‘scrape’, store and expose metrics from a metrics endpoint
  • live GOV.UK PaaS account assigned to the orgs and spaces you want to receive metrics on

We recommend this GOV.UK PaaS account:

  • uses the SpaceAuditor role as this role has the minimum permissions needed to meet the requirements of the PaaS Prometheus exporter app
  • is separate to your primary GOV.UK PaaS account

Set up the app

  1. Clone the PaaS Prometheus exporter repository.

  2. Push the PaaS Prometheus exporter app to Cloud Foundry without starting the app:

    cf push --no-start prometheus-exporter --hostname prometheus-exporter-ORGNAME
    

    where ORGNAME is the name of your org. For example:

    cf push --no-start prometheus-exporter --hostname prometheus-exporter-exampleorg
    

    Running this command deploys the PaaS Prometheus exporter app to https://prometheus-exporter-exampleorg.cloudapps.digital without starting the app.

    Refer to the app names and domain hostnames documentation for more information on how to avoid duplicating existing app names.

  3. Set the following mandatory environment variables in the PaaS Prometheus exporter app by running cf set-env prometheus-exporter NAME VALUE:

    Name Value
    API_ENDPOINT - https://api.cloud.service.gov.uk for Ireland
    - https://api.london.cloud.service.gov.uk for London
    USERNAME Cloud Foundry User
    PASSWORD Cloud Foundry Password

    You should use the cf set-env command for these mandatory variables as they contain secret information, and this method will keep them secure.

    You can also set environment variables by changing the manifest file. You should do this for optional environment variables that do not contain secret information. Refer to the PaaS Prometheus exporter repository for more information.

  4. Start the PaaS Prometheus exporter app:

    cf start prometheus-exporter
    
  5. Configure your Prometheus service’s metrics endpoint. The metrics endpoint is the deployed PaaS Prometheus exporter app URL with /metrics added on to the end. For this example:

    https://prometheus-exporter-exampleorg.cloudapps.digital/metrics
    

You can now check your Prometheus service to see if you are collecting metrics.

If you do not receiving any metrics, check the PaaS Prometheus exporter app logs.

If you still need help, contact us by emailing gov-uk-paas-support@digital.cabinet-office.gov.uk.

Add authentication

By default, the PaaS Prometheus exporter app and metrics endpoint are publicly accessible to everyone through the internet.

If you want to add authentication to the app and endpoint, refer to the documentation on adding a route service to provide basic HTTP authentication.

Metrics exporter app with StatsD

To use the metrics exporter, you deploy it as an app on PaaS. Refer to the metrics exporter readme documentation for more information on supported metrics.

Before you set up the metrics exporter app, you will need:

  • a monitoring system to store the metrics with an accompanying StatsD endpoint set up
  • a live Cloud Foundry account assigned to the orgs and spaces you want to receive metrics on

We recommend that this Cloud Foundry account:

  • uses the SpaceAuditor role as this role has the minimum permissions needed to meet the requirements of the metrics exporter app
  • is separate to your primary Cloud Foundry account

To set up the metrics exporter app:

  1. Clone the https://github.com/alphagov/paas-metric-exporter repository.
  2. Push the metrics exporter app to Cloud Foundry without starting the app by running cf push --no-start metric-exporter.
  3. Set the following mandatory environment variables in the metrics exporter app by running cf set-env metric-exporter NAME VALUE:

    Name Value
    API_ENDPOINT - https://api.cloud.service.gov.uk for Ireland
    - https://api.london.cloud.service.gov.uk for London
    STATSD_ENDPOINT StatsD endpoint
    USERNAME Cloud Foundry User
    PASSWORD Cloud Foundry Password

    You should use the cf set-env command for these mandatory variables as they contain secret information, and this method will keep them secure.

    You can also set environment variables by amending the manifest file. We recommend that you use this method for optional environment variables that do not contain secret information. Refer to the https://github.com/alphagov/paas-metric-exporter repository for more information.

  4. Start the metrics exporter app by running cf start metric-exporter.

You can now check your monitoring system to see if you are receiving metrics.

If you are not receiving any metrics, check the logs for the metrics exporter app. If you still need help, please contact us by emailing gov-uk-paas-support@digital.cabinet-office.gov.uk.

More about monitoring

For more information about monitoring apps, see Monitoring the status of your service on the Service Manual.

Backing service metrics

If you use the PostgreSQL, MySQL or Redis backing services, you can view metrics for those backing services in the GOV.UK PaaS admin tool.

When you create your backing service instance, you can access the service instance’s metrics with no extra work. Your backing service plan does not affect which metrics you can view. You can view metrics for any date range within the last year.

Backing services instances exist in a space within an org. You must be able to view a space in the GOV.UK PaaS admin tool to view metrics for backing service instances in that space.

View backing service metrics

  1. Sign into the GOV.UK PaaS admin tool for London or Ireland.
  2. Select the Organisation and Space.
  3. Select the Backing services tab and select your backing service.
  4. Select the Metrics tab.

Metrics definitions - PostgreSQL and MySQL

Free disk space

How much hard disk space your database has left. Your database will stop working if it runs out of disk space. Upgrade your service plan to get more disk space.

CPU utilisation

How much computational work your database is doing. If you think your CPU utilisation is too high, you can optimise your database queries or upgrade your service plan.

Open connections

How many open connections there are to your database. If the values are unexpectedly high, you may need to optimise how your app connects to the database, or upgrade your service plan. Unexpectedly low values may indicate the database is unavailable, or your apps cannot connect to the database.

Available memory

How much memory the virtual machine your database is running on has left. If values are persistently low, you may need to optimise your database queries, or upgrade your service plan.

Read and write operations per second

How many read and write operations per second (IOPS) your database is performing per second. Databases have a limit of 3 IOPS per gigabyte of database hard disk space. This limit applies to the sum of the read and write operations.

For example, a 100 gigabyte database has a 300 IOPS limit. If your database is close to its IOPS limit, you can upgrade your service plan.

See the documentation on Amazon RDS database instance storage for more information.

Metrics definitions - Redis

CPU utilisation

How much computational work your Redis service instance is doing. If you think your CPU utilisation is too high, you can optimise how you read data from and write data to Redis, or upgrade your service plan.

Memory used

How much memory your Redis service instance is using to run itself and to store your app data. Redis service instances have a limited amount of memory to use. This memory limit is fixed based on your service plan and you cannot change this limit.

Your Redis service instance will delete or evict keys when the instance reaches its memory limit. This may indicate you need to review the size of the data you’re storing, or you need to upgrade your service plan.

Swap memory used

If your Redis service instance is running low on memory, the instance will start to swap memory onto the hard disk. To reduce memory swapping, you can reduce the amount of memory your Redis service instance uses, or upgrade your service plan.

Key evictions

Your Redis service instance will delete or evict keys when the instance reaches its memory limit. Memory limit is fixed based on your service plan and you cannot change this limit.

Redis instances on the GOV.UK PaaS use the volatile-lru policy. This means Redis instances can only evict keys that have a set expiry time. Your Redis service instance will try to evict less recently used keys first. If your service instance cannot evict any keys, the instance will return errors when executing commands that increase memory use. Upgrade your service plan to reduce key eviction.

Contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk to find out your plan’s memory limit.

Open connections

How many open connections there are to your Redis service instance. If the values are unexpectedly high, you may need to optimise how your app connects to your service instance, or upgrade your service plan. Unexpectedly low values may indicate that your service instance is unavailable.

Cache hits and misses

The number of successful and unsuccessful Redis key lookups. If the number of cache misses is higher than the number of cache hits, you may need to optimise how you implemented your app.

Item count

The number of items in Redis.

Network bytes in and out

The number of bytes Redis has received and sent.

Logs

Cloud Foundry and apps running on Cloud Foundry generate logs using Loggregator [external link] and stream them to your terminal. You should consult your logs if your app is failing to deploy or crashing, and it’s not clear why.

Your app must write to stdout or stderr instead of a log file for its logs to be included in the Loggregator stream.

Run cf logs in the command line to stream all logs from each Cloud Foundry service involved in your app deployment:

cf logs APP_NAME

Run cf logs with the --recent flag to stream only the most recent logs:

cf logs APP_NAME --recent

You can also run cf events to see all recent app events, such as when an app starts, stops, restarts, or crashes (including error codes):

cf events APP_NAME

Set up the Logit log management service

By default, Cloud Foundry streams a limited amount of logs to your terminal for a defined time. You can use a commercial log management service to keep more logging information for longer. This section describes how to set up the Logit log management service.

Prerequisites

Before you set up Logit, you must:

Configure logstash filters

You must set up logstash to process the Cloud Foundry logs into separate Gorouter and app log types.

  1. Go to your Logit dashboard. For the Logit ELK stack you want to use, select Settings.
  2. On the Stack options menu, select Logstash Filters.
  3. Go to the Logstash Filters page, and replace the code there with the following logstash filter code:

    filter {
        grok {
            # attempt to parse syslog lines
            match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog_ver} +(?:%{TIMESTAMP_ISO8601:syslog_timestamp}|-) +(?:%{HOSTNAME:syslog_host}|-) +(?:%{NOTSPACE:syslog_app}|-) +(?:%{NOTSPACE:syslog_proc}|-) +(?:%{WORD:syslog_msgid}|-) +(?:%{SYSLOG5424SD:syslog_sd}|-|) +%{GREEDYDATA:syslog_msg}" }
            # if successful, save original `@timestamp` and `host` fields created by logstash
            add_field => [ "received_at", "%{@timestamp}" ]
            add_field => [ "received_from", "%{host}" ]
            tag_on_failure => ["_syslogparsefailure"]
        }
    
        # parse the syslog pri field into severity/facility
        syslog_pri { syslog_pri_field_name => 'syslog5424_pri' }
    
        # replace @timestamp field with the one from syslog
        date { match => [ "syslog_timestamp", "ISO8601" ] }
    
        # Cloud Foundry passes the app name, space and organisation in the syslog_host
        # Filtering them into separate fields makes it easier to query multiple apps in a single Kibana instance
        dissect {
            mapping => { "syslog_host" => "%{[cf][org]}.%{[cf][space]}.%{[cf][app]}" }
            tag_on_failure => ["_sysloghostdissectfailure"]
        }
    
        # Cloud Foundry gorouter logs
        if [syslog_proc] =~ "RTR" {
            mutate { replace => { "type" => "gorouter" } }
            grok {
                match => { "syslog_msg" => "%{HOSTNAME:[access][host]} - \[%{TIMESTAMP_ISO8601:router_timestamp}\] \"%{WORD:[access][method]} %{NOTSPACE:[access][url]} HTTP/%{NUMBER:[access][http_version]}\" %{NONNEGINT:[access][response_code]:int} %{NONNEGINT:[access][body_received][bytes]:int} %{NONNEGINT:[access][body_sent][bytes]:int} %{QUOTEDSTRING:[access][referrer]} %{QUOTEDSTRING:[access][agent]} \"%{HOSTPORT:[access][remote_ip_and_port]}\" \"%{HOSTPORT:[access][upstream_ip_and_port]}\" %{GREEDYDATA:router_keys}" }
                tag_on_failure => ["_routerparsefailure"]
                add_tag => ["gorouter"]
            }
            # replace @timestamp field with the one from router access log
            date {
                match => [ "router_timestamp", "ISO8601" ]
            }
            kv {
                source => "router_keys"
                target => "router"
                value_split => ":"
                remove_field => "router_keys"
            }
        }
    
        # Application logs
        if [syslog_proc] =~ "APP" {
            json {
                source => "syslog_msg"
                add_tag => ["app"]
            }
        }
    
        # User agent parsing
        if [access][agent] {
            useragent {
                source => "[access][agent]"
                target => "[access][user_agent]"
            }
        }
    
        if !("_syslogparsefailure" in [tags]) {
            # if we successfully parsed syslog, replace the message and source_host fields
            mutate {
                rename => [ "syslog_host", "source_host" ]
                rename => [ "syslog_msg", "message" ]
            }
        }
    }
    
  4. Select Validate.

  5. Select Apply once the code is valid. If this is not possible, check you have copied the code correctly or contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk .

  6. Go back to the Logit dashboard once the following message appears: “Filters have been applied to logstash, logstash will be restarted, this may take up to 2 minutes”.

Configure app

  1. Select Settings for the stack you want to use.
  2. On the Stack options menu, select Logstash Inputs.
  3. Note your Stack Logstash endpoint and TCP-SSL port.
  4. Run the following in the command line to create the log drain service in Cloud Foundry:

    $ cf create-user-provided-service logit-ssl-drain -l syslog-tls://ENDPOINT:PORT
    
  5. Bind the service to your app by running:

    $ cf bind-service APP_NAME logit-ssl-drain
    
  6. Restage your app by running:

    $ cf restage APP_NAME
    
  7. Select Access Kibana on the Stack options menu and check that you can see the logs in Kibana.

Once you confirm that the logs are draining correctly, you have successfully set up Logit.

Contact us by emailing gov-uk-paas-support@digital.cabinet-office.gov.uk if the logs are not draining correctly or if you have any questions.

Enable security for your ELK stack

By default, Logit allows anyone on the internet to send logs to your ELK stack. You can set up Logit to make sure that your ELK stack only receives logs from GOV.UK PaaS.

  1. Contact GOV.UK PaaS support at gov-uk-paas-support@digital.cabinet-office.gov.uk for a list of syslog drain egress IP addresses.
  2. Send these IP addresses to Logit support at https://logit.io/contact-us and ask that your ELK stack only receives log messages from these addresses.

Further information

Refer to the Cloud Foundry documentation for more information on: