Table of contents

Managing apps

Connecting with SSH

When you deploy an app to GOV.UK PaaS, it runs in a container, which is like a lightweight Linux virtual machine. Each app runs in its own isolated container.

Sometimes, it can be useful to connect directly to the container with SSH. You would usually only do this to get information for troubleshooting purposes, for example, if you can’t work out what is happening with your app using the cf logs and cf events commands described in the Logs section.

If you do run commands which will change the container temporarily, it’s a good idea to restart the app afterwards.

SSH is enabled by default. In most cases, you will find that you can SSH directly to your app’s container.


Warning If you are using v3 of the Cloud Foundry API, you must run cf v3-ssh instead of cf ssh
  1. Run:

    cf ssh APPNAME
    

    where APPNAME is the name of the app.

  2. For some tasks to work, you need to set up the interactive SSH session to match the buildpack environment. To do this, run:

    /tmp/lifecycle/shell
    

    You need to run this every time you start an SSH session.

    For more information, refer to the Cloud Foundry documentation on SSH Session Environment.

  3. When you’re finished using SSH, use exit to end the session.

If you get an error like this:

FAILED
Error opening SSH connection: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

go to the section below on Enabling SSH for an app.

Note that you do not need to generate any SSH keys. The cf CLI tool handles authentication for you.

Connecting with multiple instances

You may be running multiple instances of an app (created using cf scale or instances: in the manifest).

In this situation, each instance has a numerical instance index to distinguish it. You can see this in this example of the output of cf app exampleapp:

requested state: started
instances: 3/3
usage: 64M x 3 instances
urls: exampleapp.london.cloudapps.digital
last uploaded: Wed Dec 21 13:56:24 UTC 2016
stack: cflinuxfs3
buildpack: staticfile_buildpack

.   state     since                    cpu    memory        disk         details
0   running   2016-12-21 02:27:11 PM   3.0%   7.1M of 64M   6.8M of 1G
1   running   2016-12-21 02:44:46 PM   1.0%   3.5M of 64M   6.8M of 1G
2   running   2016-12-21 02:44:46 PM   1.0%   3.5M of 64M   6.8M of 1G

This example assumes that exampleapp is hosted in the London region.

There are 3 instances, with instance indexes from 0 to 2.

If you have multiple instances like this and use cf-ssh, you will be connected to the app with an instance index of 0.

You can connect to a particular instance. For example, if you want to connect to instance 2, you can do this:

cf ssh --app-instance-index 2

Creating TCP tunnels with SSH

You can use the Conduit plugin to connect your local system to your remote backing service instance. If you instead want to manually create a TCP tunnel with SSH, refer to this section.

The cf ssh command supports local port forwarding, which allows you to create tunnels from your local system to the application instance container. This is useful when you want to connect from your local system to a backing service that is only accessible from an app running on GOV.UK PaaS.

Note: Versions 6.24 and 6.25 of the cf command line client have a bug that means it’s not possible to send more than 2GB of data through an SSH tunnel to a Cloud Foundry application. This bug has been fixed in version 6.26 of the client.

To enable local port forwarding, you can use the parameter -L:

cf ssh APPNAME -T -L LOCALPORT:REMOTEHOST:REMOTEPORT

This will forward the LOCALPORT port on the local system to the given REMOTEHOST host and REMOTEPORT port on the application container side.

Whenever a connection is made to this port, the connection is forwarded over the secure SSH channel, and a connection is made to the host and port REMOTEHOST:REMOTEPORT from the remote application container.

You can use the -L parameter multiple times to forward different ports.

The tunnel will be closed once the cf ssh command is stopped.

For example, you can connect directly to the PostgreSQL service bound to an application following these steps:

  1. Find the details of the service using cf env APPNAME. Here is some simplified and shortened example output:

    $ cf env myapp
    Getting env variables for app myapp in org myoth / space myorg as randomuser...
    OK
    
        System-Provided:
        {
        "VCAP_SERVICES": {
         "postgres": [
         {
         "credentials": {
         "host": "rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com",
         "jdbcUrl": "jdbc:postgresql://rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com:5432/rdsbroker_9f0_97_aa4?user=rdsbroker_9f0_97_aa4_owner\u0026password=xnYXthsgUFwPUOO",
         "name": "rdsbroker_9f0_97_aa4",
         "password": "xnYXthsgUFwPUOO",
         "port": 5432,
         "uri": "postgres://rdsbroker_9f0_97_aa4_owner:xnYXthsgUFwPUOO@rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com:5432/rdsbroker_9f0_97_aa4",
         "username": "rdsbroker_9f0_97_aa4_owner"
        },
        ...
    

    You will need to know:

    • the remote host, displayed as "host":
    • the remote port, displayed as "port".
    • the PostgreSQL username, displayed as "username":
    • the PostgreSQL password, displayed as password:
    • the name of the database, displayed as name:
  2. Create a SSH tunnel using the local port 6666:

    cf ssh myapp -T -L 6666:HOST:PORT
    

    where HOST and PORT are the values you found in the previous step.

    This will open a shell in the remote container, and create a local tunnel using port 6666.

    Note: Be aware that this shell is in the remote application container, not the local system. You will need to open a new console if you want to work locally. The new port is open in the local system.

  3. In a different terminal, you can now connect to the local port in localhost:6666 using a postgres client:

    psql postgres://USERNAME:PASSWORD@localhost:6666/DATABASE_NAME
    

    replacing USERNAME, PASSWORD and DATABASE_NAME with the values from step 1

    You can also dump the database with pg_dump:

    pg_dump postgres://USERNAME:PASSWORD@localhost:6666/DATABASE_NAME > db.dump
    

SSH permissions

SSH can be either enabled or disabled independently for each space and app.

SSH must be enabled for both the space and the app before it will work. For example, if you have an app where SSH is enabled, but it is deployed to a space where SSH is disabled, SSH won’t work.

All new apps and spaces start out with SSH enabled.

Enabling SSH for an app

If you unexpectedly find that you can’t SSH to an app, the most likely cause is that SSH access is disabled for that app. This may be the case if your app was deployed before we enabled SSH access for tenants (prior to around 1600 GMT on 1st December 2016).

To check if SSH is enabled for an app, run:

cf ssh-enabled APPNAME

where APPNAME is the name of the app.

If you get a message like:

ssh support is disabled for APPNAME

you need to enable SSH for the app by running:

cf enable-ssh APPNAME

If you are running multiple instances of an app (created with cf scale or with instances: in the manifest), the enable-ssh command will affect all of them.

You do not need a special account permission level to enable SSH for an app. The default SpaceDeveloper role allows you to do this.

If SSH is already enabled, or enabling it still doesn’t make SSH work, go to Enabling SSH for a space below.

Enabling SSH for a space

If enabling SSH for an app doesn’t let you connect, check that SSH is enabled for the space it’s deployed in.

Check what space you’re working in with:

cf target

Then run:

cf space-ssh-allowed SPACENAME

where SPACENAME is the name of the space.

If you get a message like this:

ssh support is disabled in space 'sandbox'

you need to enable SSH for the space using:

cf allow-space-ssh SPACENAME

Your GOV.UK PaaS account needs the OrgManager or SpaceManager role to be able to enable SSH for a space. If the command above fails, ask someone with the correct role (probably a senior member of your team) to run it for you, or contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk.

Limiting SSH access

You should consider disabling SSH where it is not needed. For example, if you host the live versions of your apps in a production space, you may decide to disable SSH access there, but leave it enabled in your development and testing spaces.

More about using SSH

Refer to the Cloud Foundry documentation on Accessing Apps with SSH.

Scaling

You can manually scale your app to meet increasing demand. You can do this by:

  • changing the number of app instances running
  • increasing memory and disk space allocated to your app

Your organisation quotas limit the resources you can use for each app.

Changing the number of app instances

You can change the number of instances of your app running at the same time. Running more than one app instance:

  • allows your app to handle increased traffic and demand as incoming requests are automatically load-balanced across all instances
  • helps maintain high availability and makes it less likely that the failure of a single component will take down your app

For example, run the following command to set the number of app instances to 5:

cf scale APPNAME -i 5

You can also use the manifest to set the number of instances that will start when you push the app:

---
applications:
- name: APP_NAME
   instances: 2

For a production app, you should always run more than one instance.

Increasing memory or disk space

You can scale an app vertically by increasing the memory or disk space available to each instance of the app.

For example, this command increases the available memory for an app to 1 gigabyte:

cf scale APPNAME -m 1G

This command increases the disk space limit for an app to 512 megabytes:

cf scale myApp -k 512M

Further information

For more information, refer to the Cloud Foundry documentation on using cf scale to scale an app in the Cloud Foundry docs.

Quotas

Cloud Foundry capacity is managed by quotas. Quotas provide a reservation of application routes, memory, compute power, and service instances which your organization cannot exceed. You can set individual application quotas to control how much of your quota each of your applications can use.

Quota allocations

Your organisation will be assigned a quota based on your stated needs. This will cover the app instances you run. Backing services do not count towards this quota.

Your quota sets the following:

  • RAM: The amount of RAM available to your applications. The application also has a compute share derived from its memory limit.

  • Service instances: The number of service instances available to your organization.

  • Paid services: Whether or not paid services are available. postgres is a paid service.

  • Routes: The number of routes available to your applications (hostname and domain pairs where an application that exposes a listening port can be reached).

To see your organisation quota, run the command:

cf org YOURORG

where YOURORG is your organisation’s name. (If you don’t know the name, you can use cf orgs to find out).

To see all quotas available on the GOV.UK PaaS, run the command:

cf quotas

Quota limits

If a new application push would exceed your organization’s quota, the request will fail with status code 400 and a message describing the limit that would be exceeded.

Example:

Creating app APPLICATION in org ORG / space SPACE as USER...
FAILED
Server error, status code: 400, error code: 100007, message:
You have exceeded the memory limit for your organization's quota.

In this situation you have three options:

  1. Delete existing resources with cf delete, cf delete-service, cf delete-route or similar commands.
  2. Reconfigure existing application quotas and redeploy.
  3. Request a quota change: contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk.

Application quotas

As a PaaS tenant, you can divide your organization’s quota capacity amongst your applications as you see fit, by way of application quotas. Application limits are specified in your application manifest or as cf push command line options.

Use the following commands to set application quota options (in each pair below, the first is the version to use in the manifest, and the second is the command line version.)

  • memory: / -m

    Your application’s memory limit. An application’s compute limit is derived from its memory limit (see below).

  • disk_quota: / -k

    The maximum amount of disk space available to your app.

  • instances: / -i

    Sets the number of application instances to launch. Each additional instance receives the same memory and disk reservation. An application with a manifest specifying memory: 256M and instances: 4 would reserve 1GB (256M x 4) total.

    For a production application, you should always launch more than one instance. See Scaling for more information.

Memory share and compute share

Your application’s compute limit is derived from its memory limit. Each application receives a guaranteed compute share equal to its relative share of memory.

Your application will be guaranteed to receive a portion of the vCPU compute power equal to its portion of memory allocation. In other words, it will receive at least this much vCPU time even if there are other applications competing for time.

Your application will be offered 100% of the vCPU compute power. If no other applications are running, the app can use all the computer power available. If there are other applications competing for time, each application’s guaranteed share determines how much time it will receive.

The application cannot access more than the specified amount of memory.

Application quota sizing

  • The environment default of 512MB memory is sufficient for most applications. Static sites and utility applications such as schedulers or loaders may require less. Use cf app APPNAME to check your application’s current memory and compute utilization.

    requested state: started
    instances: 1/1
    usage: 128M x 1 instances
    urls:
    last uploaded: Wed Jul 22 20:09:56 UTC 2015
    
         state     since                    cpu    memory          disk
    #0   running   2015-07-30 05:58:11 PM   0.0%   94.6M of 128M   80.4M of 128M
    
  • Any application which exceeds its memory quota will be automatically restarted. Use cf events APPNAME to look for ‘out of memory’ crashes.

    ... description
    ... index: 0, reason: CRASHED, exit_description: out of memory, exit_status: 255
    

Redirecting all traffic

If a site moves to a different domain name, you can use a simple static site with a custom nginx.conf file to redirect all traffic from the old domain to the new domain. Example nginx.conf site for NEW_DOMAIN_NAME:

worker_processes 1;
daemon off;

error_log <%= ENV["APP_ROOT"] %>/nginx/logs/error.log; events { worker_connections 1024; }

http { server { listen <%= ENV["PORT"] %>; server_name localhost; return 301 $scheme://NEW_DOMAIN_NAME$request_uri; } }

Deploy your application to NEW_DOMAIN_NAME and then cf push a simple static site with that nginx.conf configuration to the old domain name. You can see a full working example here.

You can read more about nginx customization.

App restarts

Your app can restart without you telling it to, due to:

  • Cloud Foundry changes, such as platform upgrades or operating system patches
  • an unexpected issue, such as your app instance running out of memory or disk space

Cloud Foundry changes

If your app processes take more than 10 seconds to finish, those processes will be affected by a Cloud Foundry-driven restart. Cloud Foundry will send a termination signal (SIGTERM) to any apps it wants to restart. To shut down cleanly, apps must finish any requests within 10 seconds of receiving the termination signal.

If the app does not respond in time, Cloud Foundry will send a SIGKILL signal to terminate the app. The app will not shut down cleanly and requests may fail.

If your app has more than one instance, the running instances will not restart at the same time. This makes sure that there are always available instances for your app.

For more information, refer to the Cloud Foundry documentation on how:

404s after commands that restart your app

After you use a command that restarts application instances, such as cf push or cf restart, your app may temporarily return incorrect 404 errors instead of returning a 5XX error. Apart from the brief downtime, this may lead to problems if the 404 is cached, or visiting web crawling bots (as used by search engines) receive a 404.

Commands known to do this are:

  • cf push
  • cf restage
  • cf restart
  • cf scale when changing disk or memory limits or forcing a restart

We are working on a fix to prevent this happening.

Until this fix is live, you should use a blue-green deployment process. This is where you have two versions of an app, one that is ‘live’ and one that is undergoing an update or restart. There are plugins for the Cloud Foundry command line client to help this process. We recommend the cf-blue-green-deploy plugin.