Skip to main content

Managing apps

Connecting with SSH

When you deploy an app to GOV.UK PaaS, that app runs in a container.

If you use a standard buildpack, you can connect directly to an app’s container using SSH.

Connecting using SSH gives you secure access to change an app’s container.

If you make changes to your app’s container during an SSH session, these changes are not permanent and are specific to that app instance. You must push a new version of your app to make any permanent changes to your app.

Before you connect to an app’s container using SSH, you must:

  1. Run the following to start an SSH session:

    cf ssh APPNAME

    where APPNAME is the name of the app.

    The Cloud Foundry CLI will generate SSH keys for you.

    Warning If you are using Cloud Foundry CLI version 6, run `cf v3-ssh APPNAME

  2. To match your SSH environment to your app’s environment, run the following from within your SSH session shell:

    /tmp/lifecycle/shell

  3. Run exit to end the SSH session.

For more information, refer to the Cloud Foundry documentation on SSH session environments.

Connecting to multiple instances

If you’re running multiple instances of an app, you can connect to a specific app instance using SSH.

Each instance of an app has an index number. Cloud Foundry assigns index numbers to app instances when you create those app instances.

  1. Run cf app APPNAME to see the app index numbers in the output. For example:

    requested state: started
instances: 3/3
usage: 64M x 3 instances
urls: myapp.london.cloudapps.digital
last uploaded: Wed Dec 21 13:56:24 UTC 2016
stack: cflinuxfs3
buildpack: staticfile_buildpack

.   state     since                    cpu    memory        disk         details
0   running   2016-12-21 02:27:11 PM   3.0%   7.1M of 64M   6.8M of 1G
1   running   2016-12-21 02:44:46 PM   1.0%   3.5M of 64M   6.8M of 1G
2   running   2016-12-21 02:44:46 PM   1.0%   3.5M of 64M   6.8M of 1G

  2. Run cf ssh -i APP_INSTANCE_NUMBER APPNAME to connect to a specific app instance.

    For example, cf ssh -i 2 myapp connects to the last app instance in step 1 of the example.

Connecting to a non-publicly available backing service

The PostgreSQL, MySQL and Redis backing services are not publicly available to everyone on the internet. You can only connect to these backing services from inside Cloud Foundry.

You should use the Conduit plugin to connect to PostgreSQL, MySQL or Redis from your local machine.

If you do not want to use Conduit, you can manually create an SSH tunnel to connect to these backing services from your local machine.

The cf ssh command supports local port forwarding, which allows you to create tunnels from your local system to the app instance container.

The following example shows how to connect to a PostgreSQL service bound to your app using an SSH tunnel.

  1. Run cf env APPNAME to show all environment variables for your app. Example output:

    $ cf env myapp
Getting env variables for app myapp in org myoth / space myorg as randomuser...
OK

    System-Provided:
    {
    "VCAP_SERVICES": {
     "postgres": [
     {
     "credentials": {
     "host": "rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com",
     "jdbcUrl": "jdbc:postgresql://rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com:5432/rdsbroker_9f0_97_aa4?user=rdsbroker_9f0_97_aa4_owner\u0026password=xnYXthsgUFwPUOO",
     "name": "rdsbroker_9f0_97_aa4",
     "password": "xnYXthsgUFwPUOO",
     "port": 5432,
     "uri": "postgres://rdsbroker_9f0_97_aa4_owner:xnYXthsgUFwPUOO@rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com:5432/rdsbroker_9f0_97_aa4",
     "username": "rdsbroker_9f0_97_aa4_owner"
    },
    ...

    Make a note of the:

    • remote "host":
    • remote "port"
    • PostgreSQL "username":
    • PostgreSQL "password:
    • database "name":

  2. Create an SSH tunnel between your local port 6666 and your backing service’s remote host and port:

    cf ssh myapp -T -L 6666:HOST:PORT

    where:

    • HOST is the remote "host": from the previous step
    • PORT is the remote "port" from the previous step

    Refer to the Cloud Foundry documentation on configuring an SSH tunnel for more information.

    Warning When you run this command, you’re in a shell in your remote app container. Once you exit the shell, the shell closes the tunnel to the remote host and port.

  3. Open a new local command line interface and connect to the local port in localhost:6666 using a PostgreSQL client:

    psql postgres://USERNAME:PASSWORD@localhost:6666/DATABASE_NAME

    where:

    • USERNAME is the PostgreSQL "username": from step 1
    • PASSWORD is the PostgreSQL "password": from step 1
    • DATABASE_NAME is the database "name": from step 1

You have manually created an SSH tunnel to connect to your backing services from your local machine.

For a PostgreSQL backing service, you can also back up the PostgreSQL database using pg_dump:

pg_dump postgres://USERNAME:PASSWORD@localhost:6666/DATABASE_NAME > db.dump

Handshake failure error

If you get the following error, you must enable SSH for your app:

FAILED
Error opening SSH connection: ssh: handshake failed: ssh:
unable to authenticate, attempted methods [none password],
no supported methods remain

Managing SSH permissions

Before you can connect to an app’s container with SSH, you must enable SSH for both the app and the space the app is in.

All new apps and spaces have SSH enabled by default. You can enable and disable SSH independently for each space and app.

Enabling SSH for an app

  1. Check if SSH is enabled for an app:

    cf ssh-enabled APPNAME

    where APPNAME is the name of the app.

  2. If you get a message stating that SSH support is disabled for the app, enable SSH by running:

    cf enable-ssh APPNAME

    The enable-ssh command affects all running instances of an app.

Enabling SSH for a space

You must be an org manager or a space manager to enable SSH for a space.

  1. Target the correct space and check if SSH is enabled in that space:

    cf space-ssh-allowed SPACENAME

    where SPACENAME is the name of the space.

  2. If you get a message stating that SSH support is disabled for the space, enable SSH by running:

    cf allow-space-ssh SPACENAME

Limiting SSH access

You should disable SSH where it’s not needed. For example, you can disable SSH in the production space where you host your live apps, but leave SSH enabled in your development and testing spaces.

Run cf disable-ssh APPNAME and restart your app to disable SSH for that app.

Run cf disallow-space-ssh SPACENAME to disable SSH for that space.

Further information

Refer to the Cloud Foundry documentation on accessing apps with SSH.

Scaling

You can manually scale your app to meet increasing demand. You can do this by:

  • changing the number of app instances running
  • increasing memory and disk space allocated to your app

Your organisation quotas limit the resources you can use for each app.

Changing the number of app instances

You can change the number of instances of your app running at the same time. Running more than one app instance:

  • allows your app to handle increased traffic and demand as incoming requests are automatically load-balanced across all instances
  • helps maintain high availability and makes it less likely that the failure of a single component will take down your app

For example, run the following command to set the number of app instances to 5:

cf scale APPNAME -i 5

You can also use the manifest to set the number of instances that will start when you push the app:

---
applications:
- name: APP_NAME
   instances: 2

You should always run more than one instance when deploying an app to production.

Increasing memory or disk space

You can scale an app vertically by increasing the memory or disk space available to each instance of the app.

For example, this command increases the available memory for an app to 1 gigabyte:

cf scale APPNAME -m 1G

This command increases the disk space limit for an app to 512 megabytes:

cf scale APPNAME -k 512M

Further information

For more information, refer to the Cloud Foundry documentation on using cf scale to scale an app in the Cloud Foundry docs.

Autoscaling

Apps can be configured to dynamically increase or decrease the number of running instances using autoscaling.

Scaling your app to fewer instances can reduce the cost during periods of low usage, eg at night.

Scaling your app based on throughput or average latency can improve the user experience of your app during periods of high traffic, eg before a submission deadline.

Scaling methods

There are two ways of scaling:

  1. schedules - scale apps based on time of day and day of week or month
  2. scaling_rules - scale apps based on metrics, eg average latency

Schedules and scaling rules can be used at the same time, or on their own.

Limits

There are two limits which you can apply to an autoscaling policy:

  • instance_max_count - maximum number of instances to run
  • instance_min_count - minimum number of instances to run

The maximum value for instance_max_count is determined by your quota.

The minimum value for instance_min_count is 1.

Metrics

You can scale your app based on some provided metrics:

  • memoryused - how many megabytes of memory an app instance is using
  • memoryutil - percentage of memory utilization where 50 is 50%
  • cpu - percentage of CPU utilization where 100 is 100% of a single CPU
  • responsetime - average time in milliseconds an app instance takes to respond to external HTTP requests
  • throughput - number of external HTTP requests processed per second

You can also scale your app based on custom metrics, please refer to the app-autoscaler user guide on GitHub.

Example

  1. Ensure the autoscaling CLI plugin is installed:

    cf install-plugin -r CF-Community app-autoscaler-plugin

  2. Deploy your app, where APPNAME is the name of your application:

    cf push APPNAME

  3. Create an autoscaler service to scale your app:

    cf create-service autoscaler autoscaler-free-plan scale-APPNAME

  4. Bind the autoscaler service to your app:

    cf bind-service APPNAME scale-APPNAME

  5. Create an autoscaling policy, the following policy will scale your app:

    • adding one instance if CPU usage is greater than 50% for 120 seconds
    • removing one instance if CPU usage is less than 50% for 120 seconds

    It will ensure:

    • there are never more than 5 instances of your app
    • there are never fewer than 3 instances of your app
    • scaling does not occur multiple times within a 60 second period

    Create the following policy in a file called policy.json: 

    {
  "instance_min_count": 3,
  "instance_max_count": 5,
  "scaling_rules": [
    {
      "metric_type": "cpu",
      "breach_duration_secs": 120,
      "threshold": 50,
      "operator": "<",
      "cool_down_secs": 60,
      "adjustment": "-1"
    },
    {
      "metric_type": "cpu",
      "breach_duration_secs": 120,
      "threshold": 50,
      "operator": ">=",
      "cool_down_secs": 60,
      "adjustment": "+1"
    }
  ]
}

  6. Attach the autoscaling policy to your app:

    cf attach-autoscaling-policy APPNAME policy.json

  7. Observe the app scaling automatically:

    cf autoscaling-history APPNAME

Example policies

Scale when throughput increases or decreases

Imagine you have an app which can handle approximately 100 requests per second per instance before the user experience becomes slow, and the app unreliable.

{
  "instance_min_count": 3,
  "instance_max_count": 9,
  "scaling_rules": [
    {
      "metric_type": "throughput",
      "breach_duration_secs": 60,
      "threshold": 90,
      "operator": "<",
      "cool_down_secs": 60,
      "adjustment": "-1"
    },
    {
      "metric_type": "throughput",
      "breach_duration_secs": 60,
      "threshold": 90,
      "operator": ">=",
      "cool_down_secs": 60,
      "adjustment": "+1"
    }
  ]
}

This policy will scale your app:

  • adding one instance if the average number of external HTTP requests per second per instance is greater than 90 for 60 seconds
  • removing one instance if the average number of external HTTP requests per second per instance is less than 90 for 60 seconds

This policy will ensure:

  • there are never more than 6 instances of your app
  • there are never fewer than 3 instances of your app

This policy will avoid scaling multiple times within a 60 second period.

Scale during the day

Imagine you have an app which is used frequently during daytime working hours, but does not receive much traffic otherwise.

{
  "instance_min_count": 2,
  "instance_max_count": 4,
  "scaling_rules": [
    {
      "metric_type": "throughput",
      "breach_duration_secs": 60,
      "threshold": 90,
      "operator": "<",
      "cool_down_secs": 60,
      "adjustment": "-1"
    },
    {
      "metric_type": "throughput",
      "breach_duration_secs": 60,
      "threshold": 90,
      "operator": ">=",
      "cool_down_secs": 60,
      "adjustment": "+1"
    }
  ],
  "schedules": {
    "timezone": "Europe/London",
    "recurring_schedule": [
      {
        "start_time": "08:00",
        "end_time": "18:00",
        "days_of_week": [1, 2, 4, 5],
        "instance_min_count": 4,
        "instance_max_count": 8,
        "initial_min_instance_count": 4
      },
      {
        "start_time": "08:00",
        "end_time": "18:00",
        "days_of_week": [3],
        "instance_min_count": 6,
        "instance_max_count": 10,
        "initial_min_instance_count": 6
      }
    ]
  }
}

This policy uses the same scaling rules as the previous example:

  • adding one instance if the average number of external HTTP requests per second per instance is greater than 90 for 60 seconds
  • removing one instance if the average number of external HTTP requests per second per instance is less than 90 for 60 seconds

This policy will ensure:

  • there are between 6 and 10 instances during Wednesday working hours
  • there are between 4 and 8 instances during working hours on other weekdays
  • there are between 2 and 4 instances otherwise

Scale during the month

Imagine you have an app which is used frequently during the first week of the month, but does not receive much traffic otherwise.

{
  "instance_min_count": 2,
  "instance_max_count": 4,
  "scaling_rules": [
    {
      "metric_type": "throughput",
      "breach_duration_secs": 60,
      "threshold": 90,
      "operator": "<",
      "cool_down_secs": 60,
      "adjustment": "-1"
    },
    {
      "metric_type": "throughput",
      "breach_duration_secs": 60,
      "threshold": 90,
      "operator": ">=",
      "cool_down_secs": 60,
      "adjustment": "+1"
    }
  ],
  "schedules": {
    "timezone": "Europe/London",
    "recurring_schedule": [
      {
        "start_time": "06:00",
        "end_time": "22:00",
        "days_of_month": [1, 2, 3, 4, 5, 6, 7],
        "instance_min_count": 6,
        "instance_max_count": 10,
        "initial_min_instance_count": 6
      }
    ]
  }
}

This policy uses the same scaling rules as the previous example:

  • adding one instance if the average number of external HTTP requests per second per instance is greater than 90 for 60 seconds
  • removing one instance if the average number of external HTTP requests per second per instance is less than 90 for 60 seconds

This policy will ensure:

  • there are between 6 and 10 instances during daylight hours of the first week of the month
  • there are between 2 and 4 instances otherwise

Tips for autoscaling policies

  • Ensure that schedules do not overlap
  • Small changes in scale which occur more often are better than large, infrequent changes
  • Keep your autoscaling policies simple
  • If your app should be highly available, ensure instance_min_count is at least 3

Further information

Refer to the app-autoscaler user guide on GitHub.

Quotas

Cloud Foundry capacity is managed by quotas. Quotas provide a reservation of application routes, memory, compute power, and service instances which your organisation cannot exceed. You can set individual application quotas to control how much of your quota each of your applications can use.

Quota allocations

Your organisation will be assigned a quota based on your stated needs. This will cover the app instances you run.

Your quota sets the following:

  • RAM: The amount of RAM available to your applications. The application also has a compute share derived from its memory limit.

  • Services: The number of service instances available to your organisation.

  • Paid services: Whether or not paid services are available. postgres is a paid service.

  • Routes: The number of routes available to your applications (hostname and domain pairs where an application that exposes a listening port can be reached).

Our quotas are:

Name RAM Routes Services
default (trial) 5.12GB 1000 10
small 10.24GB 1000 10
medium 61.44GB 1000 20
large 102.40GB 1000 40
xlarge 204.80GB 1000 80
2xlarge 409.60GB 1000 160
4xlarge 819.20GB 2000 320
8xlarge 1638.40GB 4000 720

To see your organisation quota, run the command:

cf org YOURORG

where YOURORG is your organisations’s name. (If you do not know the name, you can use cf orgs to find out). This will produce an output which includes the name of the org’s current quota:

name:                  YOURORG
domains:               my.domain,
quota:                 medium
spaces:                space1, space2
isolation segments:

To see the details of that quota, run:

cf org-quota QUOTA

where QUOTA is the name of the quota that GOV.UK PaaS has assigned to your org.

Total Memory           60G
Instance Memory        unlimited
Routes                 1000
Services               20
Paid service plans     allowed
App instance limit     unlimited
Reserved Route Ports   0

To see all quotas available on the GOV.UK PaaS, run the command:

cf org-quotas

Quota limits

If a new application push would exceed your organisation’s quota, the request will fail with status code 400 and a message describing the limit that would be exceeded.

Example:

Creating app APPLICATION in org ORG / space SPACE as USER...
FAILED
Server error, status code: 400, error code: 100007, message:
You have exceeded the memory limit for your organization's quota.

In this situation you have three options:

  1. Delete existing resources with cf delete, cf delete-service, cf delete-route or similar commands.
  2. Reconfigure existing application quotas and redeploy.
  3. Request a quota change: contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk.

Application quotas

As a PaaS tenant, you can divide your organisation’s quota capacity amongst your applications as you see fit, by way of application quotas. Application limits are specified in your application manifest or as cf push command line options.

Use the following commands to set application quota options (in each pair below, the first is the version to use in the manifest, and the second is the command line version.)

  • memory: / -m

    Your application’s memory limit. An application’s compute limit is derived from its memory limit (see below).

  • disk_quota: / -k

    The maximum amount of disk space available to your app.

  • instances: / -i

    Sets the number of application instances to launch. Each additional instance receives the same memory and disk reservation. An application with a manifest specifying memory: 256M and instances: 4 would reserve 1GB (256M x 4) total.

    For a production application, you should always launch more than one instance. See Scaling for more information.

Memory share and compute share

Your application’s compute limit is derived from its memory limit. Each application receives a guaranteed compute share equal to its relative share of memory.

Your application will be guaranteed to receive a portion of the vCPU compute power equal to its portion of memory allocation. In other words, it will receive at least this much vCPU time even if there are other applications competing for time.

Your application will be offered 100% of the vCPU compute power. If no other applications are running, the app can use all the computer power available. If there are other applications competing for time, each application’s guaranteed share determines how much time it will receive.

The application cannot access more than the specified amount of memory.

Application quota sizing

  • The environment default of 512MB memory is sufficient for most applications. Static sites and utility applications such as schedulers or loaders may require less. Use cf app APPNAME to check your application’s current memory and compute utilisation.

    requested state: started
instances: 1/1
usage: 128M x 1 instances
urls:
last uploaded: Wed Jul 22 20:09:56 UTC 2015

     state     since                    cpu    memory          disk
#0   running   2015-07-30 05:58:11 PM   0.0%   94.6M of 128M   80.4M of 128M

  • Any application which exceeds its memory quota will be automatically restarted. Use cf events APPNAME to look for ‘out of memory’ crashes.

    ... description
... index: 0, reason: CRASHED, exit_description: out of memory, exit_status: 255

Redirecting all traffic

If a site moves to a different domain name, you can use a simple static site with a custom nginx.conf file to redirect all traffic from the old domain to the new domain. Example nginx.conf site for NEW_DOMAIN_NAME:

worker_processes 1;
daemon off;


error_log <%= ENV["APP_ROOT"] %>/nginx/logs/error.log;
events { worker_connections 1024; }


http {
  server {
    listen <%= ENV["PORT"] %>;
    server_name localhost;
    return 301 $scheme://NEW_DOMAIN_NAME$request_uri;
  }
}

Deploy your application to NEW_DOMAIN_NAME and then cf push a simple static site with that nginx.conf configuration to the old domain name. You can see a full working example here.

You can read more about nginx customization.

App restarts

Your app can restart without you telling it to, due to:

  • Cloud Foundry changes, such as platform upgrades or operating system patches
  • an unexpected issue, such as your app instance running out of memory or disk space

Cloud Foundry changes

If your app processes take more than 10 seconds to finish, those processes will be affected by a Cloud Foundry-driven restart. Cloud Foundry will send a termination signal (SIGTERM) to any apps it wants to restart. To shut down cleanly, apps must finish any requests within 10 seconds of receiving the termination signal.

If the app does not respond in time, Cloud Foundry will send a SIGKILL signal to terminate the app. The app will not shut down cleanly and requests may fail.

If your app has more than one instance, the running instances will not restart at the same time. This makes sure that there are always available instances for your app.

For more information, refer to the Cloud Foundry documentation on how:

404s after commands that restart your app

After you use a command that restarts application instances, such as cf push or cf restart, your app may temporarily return incorrect 404 errors instead of returning a 5XX error. Apart from the brief downtime, this may lead to problems if the 404 is cached, or visiting web crawling bots (as used by search engines) receive a 404.

Commands known to do this are:

  • cf push
  • cf restage
  • cf restart
  • cf scale when changing disk or memory limits or forcing a restart

We are working on a fix to prevent this happening.

Until this fix is live, you should use a blue-green deployment process. This is where you have two versions of an app, one that is ‘live’ and one that is undergoing an update or restart. There are plugins for the Cloud Foundry command line client to help this process. We recommend the cf-blue-green-deploy plugin.