Table of contents

Troubleshooting

Connecting with SSH

When you deploy an app to GOV.UK PaaS, it runs in a container, which is like a lightweight Linux virtual machine. Each app runs in its own isolated container.

Sometimes, it can be useful to connect directly to the container with SSH. You would usually only do this to get information for troubleshooting purposes, for example, if you can’t work out what is happening with your app using the cf logs and cf events commands described in the Logs section.

If you do run commands which will change the container temporarily, it’s a good idea to restart the app afterwards.

SSH is enabled by default. In most cases, you will find that you can SSH directly to your app’s container.

  1. Run:

    cf ssh APPNAME
    

    where APPNAME is the name of the app.

  2. For some tasks to work, you need to set up the interactive SSH session to match the buildpack environment. To do this, run:

    /tmp/lifecycle/shell
    

    You need to run this every time you start an SSH session.

    For more information, see the Cloud Foundry documentation on SSH Session Environment [external link].

  3. When you’re finished using SSH, use exit to end the session.

If you get an error like this:

FAILED
Error opening SSH connection: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

go to the section below on Enabling SSH for an app.

Note that you do not need to generate any SSH keys. The cf CLI tool handles authentication for you.

Connecting with multiple instances

You may be running multiple instances of an app (created using cf scale or instances: in the manifest).

In this situation, each instance has a numerical instance index to distinguish it. You can see this in this example of the output of cf app exampleapp:

requested state: started
instances: 3/3
usage: 64M x 3 instances
urls: exampleapp.london.cloudapps.digital
last uploaded: Wed Dec 21 13:56:24 UTC 2016
stack: cflinuxfs2
buildpack: staticfile_buildpack

. state since cpu memory disk details 0 running 2016-12-21 02:27:11 PM 3.0% 7.1M of 64M 6.8M of 1G 1 running 2016-12-21 02:44:46 PM 1.0% 3.5M of 64M 6.8M of 1G 2 running 2016-12-21 02:44:46 PM 1.0% 3.5M of 64M 6.8M of 1G

This example assumes that exampleapp is hosted in the London region.

There are 3 instances, with instance indexes from 0 to 2.

If you have multiple instances like this and use cf-ssh, you will be connected to the app with an instance index of 0.

You can connect to a particular instance. For example, if you want to connect to instance 2, you can do this:

cf ssh --app-instance-index 2

Creating TCP tunnels with SSH

You can use the Conduit plugin to connect your local system to your remote backing service instance. If you instead want to manually create a TCP tunnel with SSH, refer to this section.

The cf ssh command supports local port forwarding, which allows you to create tunnels from your local system to the application instance container. This is useful when you want to connect from your local system to a backing service that is only accessible from an app running on GOV.UK PaaS.

Note: Versions 6.24 and 6.25 of the cf command line client have a bug that means it’s not possible to send more than 2GB of data through an SSH tunnel to a Cloud Foundry application. This bug has been fixed in version 6.26 of the client.

To enable local port forwarding, you can use the parameter -L:

cf ssh APPNAME -L LOCALPORT:REMOTEHOST:REMOTEPORT

This will forward the LOCALPORT port on the local system to the given REMOTEHOST host and REMOTEPORT port on the application container side.

Whenever a connection is made to this port, the connection is forwarded over the secure SSH channel, and a connection is made to the host and port REMOTEHOST:REMOTEPORT from the remote application container.

You can use the -L parameter multiple times to forward different ports.

The tunnel will be closed once the cf ssh command is stopped.

For example, you can connect directly to the PostgreSQL service bound to an application following these steps:

  1. Find the details of the service using cf env APPNAME. Here is some simplified and shortened example output:

    $ cf env myapp
    Getting env variables for app myapp in org myoth / space myorg as randomuser...
    OK
    
        System-Provided:
        {
        "VCAP_SERVICES": {
         "postgres": [
         {
         "credentials": {
         "host": "rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com",
         "jdbcUrl": "jdbc:postgresql://rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com:5432/rdsbroker_9f0_97_aa4?user=rdsbroker_9f0_97_aa4_owner\u0026password=xnYXthsgUFwPUOO",
         "name": "rdsbroker_9f0_97_aa4",
         "password": "xnYXthsgUFwPUOO",
         "port": 5432,
         "uri": "postgres://rdsbroker_9f0_97_aa4_owner:xnYXthsgUFwPUOO@rdsbroker-01-ff-d2.cwm.eu-west-1.rds.amazonaws.com:5432/rdsbroker_9f0_97_aa4",
         "username": "rdsbroker_9f0_97_aa4_owner"
        },
        ...
    

    You will need to know:

    • the remote host, displayed as "host":
    • the remote port, displayed as "port".
    • the PostgreSQL username, displayed as "username":
    • the PostgreSQL password, displayed as password:
    • the name of the database, displayed as name:
  2. Create a SSH tunnel using the local port 6666:

    cf ssh myapp -L 6666:HOST:PORT
    

    where HOST and PORT are the values you found in the previous step.

    This will open a shell in the remote container, and create a local tunnel using port 6666.

    Note: Be aware that this shell is in the remote application container, not the local system. You will need to open a new console if you want to work locally. The new port is open in the local system.

  3. In a different terminal, you can now connect to the local port in localhost:6666 using a postgres client:

    psql postgres://USERNAME:PASSWORD@localhost:6666/DATABASE_NAME
    

    replacing USERNAME, PASSWORD and DATABASE_NAME with the values from step 1

    You can also dump the database with pg_dump:

    pg_dump postgres://USERNAME:PASSWORD@localhost:6666/DATABASE_NAME > db.dump
    

SSH permissions

SSH can be either enabled or disabled independently for each space and app.

SSH must be enabled for both the space and the app before it will work. For example, if you have an app where SSH is enabled, but it is deployed to a space where SSH is disabled, SSH won’t work.

All new apps and spaces start out with SSH enabled.

Enabling SSH for an app

If you unexpectedly find that you can’t SSH to an app, the most likely cause is that SSH access is disabled for that app. This may be the case if your app was deployed before we enabled SSH access for tenants (prior to around 1600 GMT on 1st December 2016).

To check if SSH is enabled for an app, run:

cf ssh-enabled APPNAME

where APPNAME is the name of the app.

If you get a message like:

ssh support is disabled for APPNAME

you need to enable SSH for the app by running:

cf enable-ssh APPNAME

If you are running multiple instances of an app (created with cf scale or with instances: in the manifest), the enable-ssh command will affect all of them.

You do not need a special account permission level to enable SSH for an app. The default SpaceDeveloper role allows you to do this.

If SSH is already enabled, or enabling it still doesn’t make SSH work, go to Enabling SSH for a space below.

Enabling SSH for a space

If enabling SSH for an app doesn’t let you connect, check that SSH is enabled for the space it’s deployed in.

Check what space you’re working in with:

cf target

Then run:

cf space-ssh-allowed SPACENAME

where SPACENAME is the name of the space.

If you get a message like this:

ssh support is disabled in space 'sandbox'

you need to enable SSH for the space using:

cf allow-space-ssh SPACENAME

Your GOV.UK PaaS account needs the OrgManager or SpaceManager role to be able to enable SSH for a space. If the command above fails, ask someone with the correct role (probably a senior member of your team) to run it for you, or contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk.

Limiting SSH access

You should consider disabling SSH where it is not needed. For example, if you host the live versions of your apps in a production space, you may decide to disable SSH access there, but leave it enabled in your development and testing spaces.

More about using SSH

See the Cloud Foundry documentation on Accessing Apps with SSH [external link].

Failed login rate limit

If your Cloud Foundry account makes too many failed login attempts within a short period of time, your account will be locked for security reasons. While the account is locked, even the correct login details will not work.

Wait 5 minutes without trying to log in for the lock to expire.

Make sure that no automated services (such as Jenkins) are trying to log in using your account during this time.

PORT environment variable error

The PORT environment variable is system-provided, and you should not attempt to set it yourself.

If you are trying to set any environment variable and you get an error like this:

FAILED Server error, status code: 400, error code: 100001, message: The app is invalid: environment_variables cannot set PORT

the cause is that the value of PORT has been changed from the system-provided value.

Running these commands should fix the problem:

cf unset-env myapp PORT
cf restage myapp

where myapp is the name of the affected app instance.

Make sure that the app does not attempt to set the PORT variable.

PostgreSQL connection problems

If you are having trouble connecting your app to a PostgreSQL database, follow the steps below. You can also apply these instructions to other backing services. If you require more guidance for other supported backing services, contact us at gov-uk-paas-support@digital.cabinet-office.gov.uk.

  1. Ensure your app is making a TLS connection to the PostgreSQL service.
  2. Use cf service SERVICE_INSTANCE to check that the service is bound to the app.
  3. Use cf env APPNAME to see the app’s environment variables and confirm that there is a VCAP_SERVICES={"postgres": section, indicating the app has correctly received the environment variables.
  4. Make sure your app writes database connection errors to STDOUT or STDERR, then look for recent errors with cf logs APPNAME --recent.
  5. If restarting the app doesn’t make it connect, try restaging the app with cf restage APPNAME.

Pushing apps fails on Windows (line endings issue)

We are aware of some cases where pushing an app can fail if you are pushing from a Windows machine that has saved project files with Windows-style (CRLF) line endings.

This happens because some buildpacks do not interpret the CRLF line endings correctly. This is an example of an error arising from an executable script with CRLF line endings when using the Ruby buildpack:

2016-12-02T06:03:25.95-0800 [APP/PROC/WEB/0]ERR /usr/bin/env: ruby
2016-12-02T06:03:25.95-0800 [APP/PROC/WEB/0]ERR : No such file or directory
2016-12-02T06:03:25.95-0800 [APP/PROC/WEB/0]OUT Exit status 127

Similar problems may happen in other buildpacks. If you find that pushing an app from Windows is failing, try making sure that files in your project are being saved with Unix-style LF line endings, not Windows-style CRLF line endings. Most text editors aimed at developers allow you to change the line ending style.

If you are working with a Git repository, disable Git’s autocrlf setting:

git config --global core.autocrlf false

then clone the repository again.

Tracing HTTP requests

The Cloud Foundry router automatically adds Zipkin-compatible headers to the incoming HTTP requests so you can match those with your application’s responses.

Zipkin is a tracing system that enables app developers to troubleshoot failures or latency issues. Zipkin provides the ability to trace requests and responses across distributed systems. See Zipkin.io [external link] for more information.

To trace app requests and responses in Cloud Foundry, apps must also log Zipkin headers.

After adding Zipkin HTTP headers to app logs, developers can use cf logs myapp to correlate the trace and span ids logged by the Cloud Foundry router with the trace ids logged by their app. To correlate trace IDs for a request through multiple apps, each app must forward appropriate values for the headers with requests to other applications.

Forgotten passwords

If you believe you have forgotten your GOV.UK PaaS credentials, you can request a password reset using the link below:

Reset your password