Monitoring apps

App metrics

Cloud Foundry provides time series data known as metrics for each instance of your PaaS app. You can receive, store and view this data in a monitoring system of your choice by deploying either the:

• paas-prometheus-exporter app to collect and display metrics in the Prometheus format
• paas-metric-exporter app to push metrics data in the StatsD format

You can also view all metrics in a one-off snapshot by installing the Cloud Foundry CLI log cache plug-in.

Use the PaaS Prometheus exporter app

The PaaS Prometheus exporter collects metrics from your apps and any backing services configured to send metrics to the PaaS.

To use the PaaS Prometheus exporter, deploy it as an app on the GOV.UK PaaS. Refer to the PaaS Prometheus exporter README documentation for more information on supported metrics.

Prerequisites

Before you set up the PaaS Prometheus exporter app, you’ll need a:

• Prometheus service to request or ‘scrape’, store and expose metrics from a metrics endpoint
• live GOV.UK PaaS account assigned to the orgs and spaces you want to receive metrics on

We recommend this GOV.UK PaaS account:

• uses the SpaceAuditor role as this role has the minimum permissions needed to meet the requirements of the PaaS Prometheus exporter app
• is separate to your primary GOV.UK PaaS account

Set up the app

1. Clone the PaaS Prometheus exporter repository.

2. Push the PaaS Prometheus exporter app to Cloud Foundry without starting the app:

   cf push --no-start prometheus-exporter-ORGNAME
   

   where ORGNAME is the name of your org. Note this is an alternative to using the --hostname flag from cf CLI v6, which was removed in cf CLI v7.

   Running this command deploys the PaaS Prometheus exporter app to https://prometheus-exporter-exampleorg.cloudapps.digital without starting the app.

   Refer to the app names and domain hostnames documentation for more information on how to avoid duplicating existing app names.

3. Set the following mandatory environment variables in the PaaS Prometheus exporter app by running cf set-env prometheus-exporter NAME VALUE:

   Name	Value
   API_ENDPOINT	- https://api.cloud.service.gov.uk for Ireland - https://api.london.cloud.service.gov.uk for London
   USERNAME	Cloud Foundry User
   PASSWORD	Cloud Foundry Password

   You should use the cf set-env command for these mandatory variables as they contain secret information, and this method will keep them secure.

   You can also set environment variables by changing the manifest file. You should do this for optional environment variables that do not contain secret information. Refer to the PaaS Prometheus exporter repository for more information.

4. Start the PaaS Prometheus exporter app:

   cf start prometheus-exporter
   

5. Configure your Prometheus service’s metrics endpoint. The metrics endpoint is the deployed PaaS Prometheus exporter app URL with /metrics added on to the end. For this example:

   https://prometheus-exporter-exampleorg.cloudapps.digital/metrics
   

You can now check your Prometheus service to see if you are collecting metrics.

If you do not receiving any metrics, check the PaaS Prometheus exporter app logs.

If you still need help, contact us by emailing gov-uk-paas-support@digital.cabinet-office.gov.uk.

Add authentication

By default, the PaaS Prometheus exporter app and metrics endpoint are publicly accessible to everyone through the internet.

If you want to add authentication to the app and endpoint, refer to the documentation on adding a route service to provide basic HTTP authentication.

Metrics exporter app with StatsD

To use the metrics exporter, you deploy it as an app on PaaS. Refer to the metrics exporter README documentation for more information on supported metrics.

Before you set up the metrics exporter app, you will need:

• a monitoring system to store the metrics with an accompanying StatsD endpoint set up
• a live Cloud Foundry account assigned to the orgs and spaces you want to receive metrics on

We recommend that this Cloud Foundry account:

• uses the SpaceAuditor role as this role has the minimum permissions needed to meet the requirements of the metrics exporter app
• is separate to your primary Cloud Foundry account

To set up the metrics exporter app:

1. Clone the https://github.com/alphagov/paas-metric-exporter repository.
2. Push the metrics exporter app to Cloud Foundry without starting the app by running cf push --no-start metric-exporter.

3. Set the following mandatory environment variables in the metrics exporter app by running cf set-env metric-exporter NAME VALUE:

   Name	Value
   API_ENDPOINT	- https://api.cloud.service.gov.uk for Ireland - https://api.london.cloud.service.gov.uk for London
   STATSD_ENDPOINT	StatsD endpoint
   USERNAME	Cloud Foundry User
   PASSWORD	Cloud Foundry Password

   You should use the cf set-env command for these mandatory variables as they contain secret information, and this method will keep them secure.

   You can also set environment variables by amending the manifest file. We recommend that you use this method for optional environment variables that do not contain secret information. Refer to the https://github.com/alphagov/paas-metric-exporter repository for more information.

4. Start the metrics exporter app by running cf start metric-exporter.

You can now check your monitoring system to see if you are receiving metrics.

If you are not receiving any metrics, check the logs for the metrics exporter app. If you still need help, please contact us by emailing gov-uk-paas-support@digital.cabinet-office.gov.uk.

More about monitoring

For more information about monitoring apps, see Monitoring the status of your service on the Service Manual.

To learn how to monitor your databases, see our documentation on monitoring backing services.

Logs

Cloud Foundry and apps running on Cloud Foundry generate logs using Loggregator [external link] and stream them to your terminal. You should consult your logs if your app is failing to deploy or crashing, and it’s not clear why.

Your app must write to stdout or stderr instead of a log file for its logs to be included in the Loggregator stream.

Run cf logs in the command line to stream all logs from each Cloud Foundry service involved in your app deployment:

cf logs APP_NAME

Run cf logs with the --recent flag to stream only the most recent logs:

cf logs APP_NAME --recent

You can also run cf events to see all recent app events, such as when an app starts, stops, restarts, or crashes (including error codes):

cf events APP_NAME

There are a number of factors that can affect log delivery and retention through cf logs. You should not rely solely on cf logs for your production systems. You should configure a syslog drain to persist the logs in your own log storage system.

Set up the Logit log management service

By default, Cloud Foundry streams a limited amount of logs to your terminal for a defined time. You can use a commercial log management service to keep more logging information for longer. This section describes how to set up the Logit log management service.

Prerequisites

Before you set up Logit, you must:

• deploy your app on Cloud Foundry
• have a Logit account
• set up Cloud Foundry

Configure logstash filters

You must set up Logstash to process the Cloud Foundry logs into separate Gorouter and app log types.

1. Go to your Logit dashboard. For the Logit ELK stack you want to use, select View Stack Settings.
2. On the Stack options menu, select Logstash Pipelines.

3. Go to the Logstash Pipeline Editor, and replace the code there with the following logstash filter code:

   # updated 2020-07-01
   filter {
       grok {
           # attempt to parse syslog lines
           match => { "message" => "(%{NONNEGINT:message_length} )?%{SYSLOG5424PRI}%{NONNEGINT:syslog_ver} (?:%{TIMESTAMP_ISO8601:syslog_timestamp}|-) +%{DATA:syslog_host} +%{UUID:cf_app_guid} +\[%{DATA:syslog_proc}\] +- +(\[tags@%{NONNEGINT} +%{DATA:cf_tags}\])? +%{GREEDYDATA:syslog_msg}" }
           # if successful, save original `@timestamp` and `host` fields created by logstash
           add_field => [ "received_at", "%{@timestamp}" ]
           add_field => [ "received_from", "%{host}" ]
           tag_on_failure => ["_syslogparsefailure"]
       }
   
       if [cf_tags] {
         kv {
           source => "cf_tags"
           target => "cf_tags"
           value_split => "="
         }
       }
   
       # parse the syslog pri field into severity/facility
       syslog_pri { syslog_pri_field_name => 'syslog5424_pri' }
   
       # replace @timestamp field with the one from syslog
       date { match => [ "syslog_timestamp", "ISO8601" ] }
   
       # Cloud Foundry passes the app name, space and organisation in the syslog_host
       # Filtering them into separate fields makes it easier to query multiple apps in a single Kibana instance
       if [syslog_host] {
           dissect {
               mapping => { "syslog_host" => "%{[cf][org]}.%{[cf][space]}.%{[cf][app]}" }
               tag_on_failure => ["_sysloghostdissectfailure"]
           }
       }
   
       # Cloud Foundry gorouter logs
       if [syslog_proc] =~ "RTR" {
           mutate { replace => { "type" => "gorouter" } }
           grok {
               match => { "syslog_msg" => "%{HOSTNAME:[access][host]} - \[%{TIMESTAMP_ISO8601:router_timestamp}\] \"%{WORD:[access][method]} %{NOTSPACE:[access][url]} HTTP/%{NUMBER:[access][http_version]}\" %{NONNEGINT:[access][response_code]:int} %{NONNEGINT:[access][body_received][bytes]:int} %{NONNEGINT:[access][body_sent][bytes]:int} %{QUOTEDSTRING:[access][referrer]} %{QUOTEDSTRING:[access][agent]} \"%{HOSTPORT:[access][remote_ip_and_port]}\" \"%{HOSTPORT:[access][upstream_ip_and_port]}\" %{GREEDYDATA:router_keys}" }
               tag_on_failure => ["_routerparsefailure"]
               add_tag => ["gorouter"]
           }
           # replace @timestamp field with the one from router access log
           date {
               match => [ "router_timestamp", "ISO8601" ]
           }
           kv {
               source => "router_keys"
               target => "router"
               value_split => ":"
               remove_field => "router_keys"
           }
   
           mutate {
             convert => {
               "[router][response_time]" => "float"
               "[router][gorouter_time]" => "float"
               "[router][app_index]" => "integer"
             }
           }
       }
   
       # Application logs
       if [syslog_proc] =~ "APP" {
           json {
               source => "syslog_msg"
               add_tag => ["app"]
           }
       }
   
       # User agent parsing
       if [access][agent] {
           useragent {
               source => "[access][agent]"
               target => "[access][user_agent]"
           }
       }
   
       if !("_syslogparsefailure" in [tags]) {
           # if we successfully parsed syslog, replace the message and source_host fields
           mutate {
               rename => [ "syslog_host", "source_host" ]
               rename => [ "syslog_msg", "message" ]
           }
       }
   }
   

4. Select Validate.

5. Select Apply once the code is valid. If this is not possible, check you have copied the code correctly or contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk .

6. Go back to the Logit dashboard once the following message appears: “Filters have been applied to logstash, logstash will be restarted, this may take up to 2 minutes”.

This configuration will result in application logs with two different levels of severity. Lines output from the app’s stdout will have a syslog_severity of informational and lines output from the app’s stderr will have a syslog_severity of error. For finer-grained severity levels, the logstash filter would need to be customized to extract this information from line contents.

Configure app

1. Select Settings for the stack you want to use.
2. On the Stack options menu, select Logstash Inputs.
3. Note your Stack Logstash endpoint and TCP-SSL port.

4. Run the following in the command line to create the log drain service in Cloud Foundry:

   $ cf create-user-provided-service logit-ssl-drain -l syslog-tls://ENDPOINT:PORT
   

5. Bind the service to your app by running:

   $ cf bind-service APP_NAME logit-ssl-drain
   

6. Restage your app by running, note that this will cause app downtime:

   $ cf restage APP_NAME
   

7. Select Access Kibana on the Stack options menu and check that you can see the logs in Kibana.

Once you confirm that the logs are draining correctly, you have successfully set up Logit.

Contact us by emailing gov-uk-paas-support@digital.cabinet-office.gov.uk if the logs are not draining correctly or if you have any questions.

Enable security for your ELK stack

By default, Logit allows anyone on the internet to send logs to your ELK stack. You can set up Logit to make sure that your ELK stack only receives logs from GOV.UK PaaS.

1. Read Static egress IP addresses for a list of syslog drain IP addresses.
2. Go to your Logit dashboard. For the Logit ELK stack you want to use, select View Stack Settings.
3. On the Stack options menu, select Logstash Firewall.
4. Select Add New Firewall Group and follow the required steps, for further help on configuring your firewall settings visit Logit.io logstash firewall help

Further information

Refer to the Cloud Foundry documentation for more information on:

• streaming application logs to log management services
• configuring other commercial log management services
• the log format
• viewing your application’s logs
• troubleshooting application deployment and health