Amazon S3

Amazon S3 provides data object storage through a web service interface.

To provide any feedback on this feature, contact the GOV.UK PaaS team at gov-uk-paas-support@digital.cabinet-office.gov.uk.

Set up the service

Provision an Amazon Web Service S3 bucket

Creating a service instance provisions an Amazon Web Service (AWS) Simple Storage Service (S3) bucket.

The name of the service overall is aws-s3-bucket. There’s one type of plan available for the service named default. This plan has unlimited data, storage, and resources.

S3 buckets can be either private or public:

private S3 buckets are only accessible using apps that you bind to that bucket
files in public S3 buckets can be downloaded by anyone with the URL for those files

AWS encrypts S3 buckets using Amazon S3-managed encryption keys. You do not have to enable this encryption because it is enabled by default.

Provision a private AWS S3 bucket

Run the following in the command line to provision a private AWS S3 bucket:

cf create-service aws-s3-bucket default SERVICE_NAME

where SERVICE_NAME is a unique descriptive name for this S3 bucket.

Provision a public AWS S3 bucket

Public S3 buckets can be useful for tasks such as:

storing static objects that you do not want to include with your app
hosting 404 error pages that work independently of your app

Run the following in the command line to provision a public AWS S3 bucket:

cf create-service aws-s3-bucket default SERVICE_NAME -c '{"public_bucket":true}'

where SERVICE_NAME is a unique descriptive name for this S3 bucket. The configuration option '{"public_bucket":true}' makes sure that the S3 bucket is public.

Bind an AWS S3 bucket to your app

You must bind the S3 bucket to your app so you can get credentials to use the AWS S3 API to read from and write to the bucket.

You can bind an S3 bucket using the command line or the manifest file. You must use the command line when you want to specify permissions for the bucket.

Run the following to bind the S3 bucket to your app:

cf bind-service APP_NAME SERVICE_NAME -c '{"permissions": "PERMISSION"}'

where:

APP_NAME is the name of a deployed instance of your app, as specified in your app’s manifest or push command
SERVICE_NAME is a unique descriptive name for this S3 bucket
PERMISSION is the permission level and can be either read-write or read-only

You can use the app’s manifest to bind an S3 bucket to the app with default read and write permissions only. It will bind automatically when you next deploy your app. An example manifest:

--
applications:
- name: APP_NAME
  services:
  - SERVICE_NAME

By default, you can only connect to an S3 bucket from your GOV.UK PaaS-hosted app.

To connect to an S3 bucket from outside the GOV.UK PaaS, you must create a service key.

Connect to an S3 bucket from your app

Restart your app to populate your VCAP_SERVICES environment variable with the credentials needed to connect to the S3 bucket.

An example VCAP_SERVICES environment variable:

{
 "VCAP_SERVICES": {
  "aws-s3-bucket": [
   {
    "binding_name": null,
    "credentials": {
     "aws_access_key_id": "XXXXXXXXXXXXXXXXXXXX",
     "aws_region": "eu-west-2",
     "aws_secret_access_key": "XXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
     "bucket_name": "paas-s3-broker-prod-lon-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
     "deploy_env": ""
    },
    "instance_name": "my-bucket",
    "label": "aws-s3-bucket",
    "name": "my-bucket",
    "plan": "default",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [
     "s3"
    ],
    "volume_mounts": []
   }
  ]
 }
}

You must pass credentials from your VCAP_SERVICES environment variable to your AWS software development kit (SDK). Use the appropriate Amazon SDK documentation to do this.

Connect to an S3 bucket from outside of the GOV.UK PaaS

To connect to an S3 bucket from outside of the GOV.UK PaaS, you must create a service key and credentials that allow external access to the bucket.

Create a service key and credentials to allow external access:
```
cf create-service-key SERVICE_NAME SERVICE_KEY -c '{"allow_external_access": true}'
```
where:
- SERVICE_NAME is the unique descriptive name for the S3 bucket
- SERVICE_KEY is a unique descriptive name for the service key

Get the service key credentials:

cf service-key SERVICE_NAME SERVICE_KEY

You will see output similar to the following:

{
 "aws_access_key_id": "AWS_ACCESS_KEY",
 "aws_region": "AWS_REGION",
 "aws_secret_access_key": "AWS_SECRET_ACCESS_KEY",
 "bucket_name": "BUCKET_NAME",
 "deploy_env": ""
}

Use these credentials to connect to the S3 bucket from outside of the GOV.UK PaaS.

Listing and revoking service keys

You can list all service keys for a service by running the following:

cf service-keys SERVICE_NAME

These credentials persist and do not get automatically deleted when you sign out of the GOV.UK PaaS. If you do not need these credentials, you should delete them by running:

cf delete-service-key MY_SERVICE MY_KEY

Use the service

You can take different actions on an S3 bucket using the AWS S3 API. The actions you can take depend on your permission level for that bucket.

Description	Command	Permission
Download an object from S3 bucket	`s3:GetObject`	`read-write`, `read-only`
Upload an object to S3 bucket	`s3:PutObject`	`read-write`
Delete an object from S3 bucket	`s3:DeleteObject`	`read-write`
Get AWS region of S3 bucket	`s3:GetBucketLocation`	`read-write`, `read-only`
List objects in S3 bucket	`s3:ListBucket`	`read-write`, `read-only`
Upload a CORS policy	`s3:PutBucketCORS`	`read-write`
Download a CORS policy	`s3:GetBucketCORS`	`read-write`, `read-only`

Access objects in a public AWS S3 bucket

Once you have uploaded an object to a public S3 bucket, you can find that object at:

https://BUCKET_NAME.s3.amazonaws.com/KEY

where:

BUCKET_NAME is the bucket_name value in the bucket environment variables
KEY is the key you gave to the object when uploading that object to the bucket

Access objects in a private AWS S3 bucket using presigned URLs

You can generate a presigned URL that gives temporary access to an object in a private bucket. Presigned URLs are useful in scenarios where you want to:

grant limited access to a resource in the bucket after validating the user’s authorisation (for example, downloading a signed document)
allow users to upload files directly to the bucket, without passing the file through your application (for example, large file uploads)

You can generate a presigned URL in your application using the AWS SDK package for your application’s programming language. You can also generate them with the AWS CLI tool by using the S3 bucket credentials from a service key.

You need to allow external access to the bucket if you intend to use a presigned URL from outside of the GOV.UK PaaS. For example, if you want to download the generated urls directly from the browser.

Run the following in the command line to bind the S3 bucket to your app and allow external access:

cf bind-service APP_NAME SERVICE_NAME -c '{"allow_external_access": true}'

Remove the service

Unbind an S3 bucket from your app

Run the following in the command line to unbind the S3 bucket from the app:

cf unbind-service APP_NAME SERVICE_NAME

where:

APP_NAME is the name of a deployed instance of your app, as specified in your app’s manifest or push command
SERVICE_NAME is a unique descriptive name for this S3 bucket

If you unbind the S3 bucket from your app but do not delete the bucket, that bucket will persist even after you delete your app. You can re-bind or re-connect to the bucket in future.

Delete an S3 bucket

You must delete all objects from the S3 bucket before you can delete the bucket itself.

You must have read and write permissions for an S3 bucket to delete objects from that bucket.

Use the AWS S3 API to delete objects from the bucket.

Run the following in the command line to delete the S3 bucket:

cf delete-service SERVICE_NAME

where SERVICE_NAME is a unique descriptive name for this S3 bucket.

Migrate objects from a PaaS S3 bucket to a S3 bucket in a tenant’s AWS account

You can migrate objects from S3 buckets in PaaS AWS accounts to S3 buckets in other AWS accounts.

You must have an S3 bucket in your AWS account before you begin the migration process. If you do not have an S3 bucket in your AWS account, create one.

The migration process has two main steps:

Create an Identity and Access Management (IAM) bucket policy that allows you to copy objects from a PaaS S3 bucket to your S3 bucket.
Synchronise the S3 buckets

Create an IAM bucket policy that allows S3 bucket migration

To start this migration, you need to apply an IAM bucket policy to your AWS account. This IAM policy will let objects move from the PaaS S3 bucket to your S3 bucket.

Before creating the bucket policy, you’ll need the Amazon Resource Name (ARN) of the PaaS bucket’s owner. To get this ARN, create a service key and get the service key credentials.

Export the credentials as AWS environment variables or create a profile in ~/.aws/credentials to make the PaaS user’s credentials available to the AWS CLI.

You now need the ARN of the AWS IAM user that created the PaaS S3 bucket. Run the following AWS CLI command:

% aws sts get-caller-identity --profile <NAME_OF_PROFILE_USED_IN_CREDENTIAL_FILE>

If you’ve already set the AWS credentials in the appropriate AWS environment variables, do not use the --profile option.

If the command is successful, you’ll see an output that looks like this:

{
   "UserId": "USER_ID",
   "Account": "ACCOUNT_NAME",
   "Arn": "arn:aws:iam::YYYYYYYYYY:user/paas-s3-broker/paas-s3-broker-ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ"
}

In the IAM bucket policy, replace:

the AWS value with the Arn value from the output of your aws sts get-caller-identity command
the arn:aws:s3::: values with the Arn value of the bucket you created in your AWS account. You must not paste over the /* characters of the first item in the Resource list

Your IAM bucket policy should look like this:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "PaaS permissions",
           "Effect": "Allow",
           "Principal": {
               "AWS": "<USER_ARN>"
           },
           "Action": [
               "s3:ListBucket",
               "s3:GetObject",
               "s3:PutObject"
           ],
           "Resource": [
               "arn:aws:s3:::<NAME_OF_TENANT_BUCKET>/*",
               "arn:aws:s3:::<NAME_OF_TENANT_BUCKET>"
           ]
       }
   ]
}

Synchronise the S3 buckets

Run this command to synchronise the PaaS S3 bucket and the tenant’s S3 bucket:

aws s3 sync s3://<PAAS_BUCKET_NAME> s3://<TENANT_BUCKET_NAME> --source-region <PAAS_BUCKET_REGION> --region <TENANT_BUCKET_REGION> --profile <NAME_OF_PROFILE_USED_IN_CREDENTIAL_FILE>

Replace:

<PAAS_BUCKET_NAME> with the bucket_name value provided in the output of the cf service-key command
<TENANT_BUCKET_NAME> with the bucket_name of the bucket that you created in your AWS account
<PAAS_BUCKET_REGION> with the AWS region of the PaaS bucket
<TENANT_BUCKET_REGION> with the AWS region of the tenant bucket
<NAME_OF_PROFILE_USED_IN_CREDENTIAL_FILE> with the profile name that you created in your ~/.aws/credentials file

If you’re using AWS environment variables instead, do not use the --profile option.

If the command is successful, the output will look similar to this (dependent upon the objects that exist in the PaaS bucket):

copy: s3://<PAAS_BUCKET_NAME>/spring-music to s3://<TENANT_BUCKET_NAME>/spring-music