Managing custom domains using the cdn-route service

Apps on GOV.UK PaaS are given a default domain of APP_NAME.cloudapps.digital (or APP_NAME.london.cloudapps.digital in the London region) by default. This is OK to use within your team and for development. However, a live service will likely need its own custom domain name, such as my-service.service.gov.uk. This guide will help you to configure a custom domain.

Register a domain name

If you do not already have a custom domain name to use then your first step is to obtain one. If you are running a government service you can request a service.gov.uk domain. If you are running another type of government website you can request a gov.uk domain.

GOV.UK PaaS does not provide domain registration or DNS hosting. Your organisation may already have preferred providers.

Using your domain name with GOV.UK PaaS

Once you have a domain name you need to route requests to GOV.UK PaaS. You can do this using a cdn-route service managed by GOV.UK PaaS. Alternatively you can configure an existing CDN provider.

A CDN service will make your app available on your chosen domain through HTTPS. GOV.UK PaaS redirects non-secure HTTP requests to a secure version of your app.

Set up the GOV.UK PaaS cdn-route service

Once you have started setting up a cdn-route service instance, you cannot update or delete the instance until you have completed the setup process. If you encounter problems, email GOV.UK PaaS support at gov-uk-paas-support@digital.cabinet-office.gov.uk.

Before you start setting up your service, you must be able to set TXT and CNAME DNS records for your domain within 24 hours. This can be done through your DNS provider.

Set up a cdn-route service with one or more custom domains

Warning These instructions assume that you are using GOV.UK PaaS to launch a new service and your subdomains are not already delivering a service.

If this is not the case and you are migrating an existing service, your technical operations team should refer to this documentation when the team are designing a custom migration process for your service.

Sign in to Cloud Foundry, and target the space your app is running in:
```
cf target -o ORG_NAME -s SPACE_NAME
```
where:
- ORG_NAME is the name of your org
- SPACE_NAME is the name of your space
Create the required domain(s) in your organisation:
```
cf create-private-domain ORG_NAME APEX_DOMAIN_NAME
```
where APEX_DOMAIN_NAME is the name of the apex domain, for example example.service.gov.uk
Create an instance of the cdn-route service:
```
cf create-service cdn-route cdn-route SERVICE_INSTANCE \
-c '{"domain": "SUBDOMAIN_LIST"}'
```
where:
- SERVICE_INSTANCE is your chosen name for your service instance
- SUBDOMAIN_LIST is a comma-separated list of your subdomain(s), for example www.example.com,www.example.net (do not include a comma after the final subdomain)
This command includes cdn-route twice because cdn-route is the name of the service and the name of the service plan.

Get the DNS information required to configure your subdomain:

cf service SERVICE_INSTANCE

Example output:

status:    Create in progress
message:   Provisioning in progress.

Create the following CNAME records to direct traffic from your domains to your CDN route
www.example.com => d3j6yjt78pkdqf.cloudfront.net
www.example.net => d3j6yjt78pkdqf.cloudfront.net

To validate ownership of the domain, set the following DNS records
For domain www.example.com, set DNS record

    Name:  _83878ed284b0b5fcaa3e99a618432ac8.www.example.com.
    Type:  CNAME
    Value: _5c7e8297b9691c59197e4e10b5bf0a98.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.com: PENDING_VALIDATION

For domain www.example.net, set DNS record

    Name:  _87fa63d9b29059b0649695eec2cd0fbe.www.example.net.
    Type:  CNAME
    Value:  _6504c2f15bb9307ee3a599c0e8193849.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.net: PENDING_VALIDATION

started:   2020-06-05T09:28:34Z
updated:   2020-06-05T09:29:44Z

Create CNAME records using the DNS information. These records validate your subdomain and allow the GOV.UK PaaS team to issue the TLS certificate used to encrypt traffic.

In the example, you would create 2 CNAME records for your subdomains:

- _83878ed284b0b5fcaa3e99a618432ac8.www.example.com. with a value of _5c7e8297b9691c59197e4e10b5bf0a98.tfmgdnztqk.acm-validations.aws
- _87fa63d9b29059b0649695eec2cd0fbe.www.example.net with a value of _6504c2f15bb9307ee3a599c0e8193849.tfmgdnztqk.acm-validations.aws

Create one CNAME record for each subdomain using the DNS information. This routes requests to your service.

You should not create CNAME records until your cdn-route service instance has been created. Run cf service SERVICE_INSTANCE to check the status of your service instance. If the service has been created, the service status will say Create succeeded.

cf service SERVICE_INSTANCE

Example output:

status:    Create succeeded
message:   Service instance provisioned [www.example.com,www.example.net => cloudapps.digital]; CDN domain d3j6yjt78pkdqf.cloudfront.net

Create the following CNAME records to direct traffic from your domains to your CDN route
www.example.com => d3j6yjt78pkdqf.cloudfront.net
www.example.net => d3j6yjt78pkdqf.cloudfront.net

To validate ownership of the domain, set the following DNS records
For domain www.example.com, set DNS record

    Name:  _83878ed284b0b5fcaa3e99a618432ac8.www.example.com.
    Type:  CNAME
    Value: _5c7e8297b9691c59197e4e10b5bf0a98.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.com: SUCCESS

For domain www.example.net, set DNS record

    Name:  _87fa63d9b29059b0649695eec2cd0fbe.www.example.net.
    Type:  CNAME
    Value:  _6504c2f15bb9307ee3a599c0e8193849.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.net: SUCCESS

started:   2020-06-05T09:28:34Z
updated:   2020-06-05T09:29:44Z

In the example, you would create 2 CNAME records in your DNS provider for www.example.com and www.example.net. Each CNAME record would point the associated subdomain to d3nrs0916m1mk2.cloudfront.net.

- www.example.com with a value of d3j6yjt78pkdqf.cloudfront.net
- www.example.net with a value of d3j6yjt78pkdqf.cloudfront.net

You have now set up the custom domain. Next, you should map a subdomain to your app.

It can take up to one hour for domain setup to finish after you have created the required DNS records. If the process is still incomplete after 2 hours, please refer to the troubleshooting section.

If you have set up a custom domain using the cdn-route service and you want to stop using this custom domain with GOV.UK PaaS, you must remove the custom domain from your cdn-route service.

Amend the service

Add a custom domain to your existing cdn-route service

Check which custom domains your existing cdn-route service has:
```
cf service SERVICE_INSTANCE | grep ^message:
```
where SERVICE_INSTANCE is the name of one of your existing cdn-route service instances.

Example output:
```
message: Service instance provisioned [www.example.com,www.example.net =>
         cloudapps.digital]; CDN domain d3j6yjt78pkdqf.cloudfront.net
```
In the example, there are 2 custom domains: www.example.com and www.example.net.
Update your service with the expanded list of custom domains:
```
cf update-service SERVICE_INSTANCE \
-c '{"domain": "EXPANDED_SUBDOMAIN_LIST"}'
```
where EXPANDED_SUBDOMAIN_LIST is the comma-separated list of subdomains.

For example, your cdn-route service is named custom-domains-production and has 2 custom domains, www.example.com and www.example.net. Run the following to add www.example.org to the list of custom domains:
```
cf update-service custom-domains-production \
  -c '{"domain": "www.example.com,www.example.net,www.example.org"}'
```
Warning Whenever you update the “domain” configuration parameter you must update your DNS records, regardless of whether the value has changed.

Get the DNS information required to configure your subdomain:

cf service SERVICE_INSTANCE

Example output:

status:    Update in progress
message:   Update in progress [www.example.com,www.example.net,www.example.org => cloudapps.digital]; CDN domain d3j6yjt78pkdqf.cloudfront.net

Create the following CNAME records to direct traffic from your domains to your CDN route
www.example.com => d3j6yjt78pkdqf.cloudfront.net
www.example.net => d3j6yjt78pkdqf.cloudfront.net
www.example.org => d3j6yjt78pkdqf.cloudfront.net

To validate ownership of the domain, set the following DNS records
For domain www.example.com, set DNS record

    Name:  _83878ed284b0b5fcaa3e99a618432ac8.www.example.com.
    Type:  CNAME
    Value: _5c7e8297b9691c59197e4e10b5bf0a98.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.com: SUCCESS

For domain www.example.net, set DNS record

    Name:  _87fa63d9b29059b0649695eec2cd0fbe.www.example.net.
    Type:  CNAME
    Value:  _6504c2f15bb9307ee3a599c0e8193849.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.net: SUCCESS

For domain www.example.org, set DNS record

    Name:  _87fa63d9b29059b0649695eec2cd0fbe.www.example.org.
    Type:  CNAME
    Value:  _3504d2f46bb9307ea3a529c0e7593f32.tfmgdnztqk.acm-validations.aws.
    TTL:   86400

    Current validation status of www.example.org: PENDING_VALIDATION

started:   2020-06-05T09:28:34Z
updated:   2020-06-05T09:29:44Z

Create CNAME records using the DNS information. These records validate your subdomain and allow the GOV.UK PaaS team to issue the TLS certificate used to encrypt traffic.

In the example, you would create a single CNAME record for your new domain of _87fa63d9b29059b0649695eec2cd0fbe.www.example.org. with a value of _3504d2f46bb9307ea3a529c0e7593f32.tfmgdnztqk.acm-validations.aws.
Create a CNAME record for the new subdomain using the DNS information. This routes requests to your service.

You should not create CNAME records until your cdn-route service instance has finished updating. Run cf service SERVICE_INSTANCE to check the status of your service instance. If the update has finished, the validation status will say Current validation status of www.example.org: SUCCESS.

In the example, you would create a CNAME record in your DNS provider pointing www.example.org to d3j6yjt78pkdqf.cloudfront.net.

You have now added a custom domain to your existing cdn-route service. Next, you should map a subdomain to your app.

It should take one hour for domain setup to finish after you have created the required DNS records. If the process is still incomplete after 2 hours, please refer to the troubleshooting section.

If you have set up a custom domain using the cdn-route service and you want to stop using this custom domain with GOV.UK PaaS, you must remove the custom domain from your cdn-route service.

Map a subdomain route to your app

Mapping a subdomain route to your app makes that app accessible through subdomains.

For each subdomain, map the route to the app:

cf map-route APP_NAME --hostname www DOMAIN_NAME

where DOMAIN_NAME is the name of the domain.

Your app can now receive requests through the subdomain.

Remove a custom domain from your cdn-route service

Warning Removing the domain results in the service provided by the app on that domain becoming unavailable.

Your org manager should check if any apps are using a custom domain before removing that domain:

cf routes --org-level | grep -we DOMAIN_NAME -e^space

Example output:

space   host     domain        port   path   type   apps     service
tools   www      DOMAIN_NAME                        app-1
tools   info     DOMAIN_NAME                        app-2

In the example, your org manager must tell the owners of app-1 and app-2 that you are removing the domain:

Remove the custom domain from Cloud Foundry’s routing database:
```
cf delete-private-domain DOMAIN_NAME
```
This also deletes the custom domain from every app using that domain.
Check how many custom domains your cdn-route service has:
```
cf service SERVICE_INSTANCE | grep ^message:
```
where SERVICE_INSTANCE is the name of one of your existing cdn-route service instances.

Example output:
```
message: Service instance provisioned [www.example.com,www.example.net =>
         cloudapps.digital]; CDN domain d3j6yjt78pkdqf.cloudfront.net
```
In the example, there are 2 custom domains: www.example.com and www.example.net.

If you cannot see this output, please contact GOV.UK PaaS support at gov-uk-paas-support@digital.cabinet-office.gov.uk. before proceeding.

The next step depends on whether your cdn-route service has one or multiple associated custom domains.

One associated domain

If your cdn-route service only has one associated custom domain, you must delete the service and the associated custom domain:

cf delete-service SERVICE_INSTANCE

where SERVICE_INSTANCE is the name of the appropriate cdn-route service instance.

You have now deleted your custom domain from GOV.UK PaaS. You should next change or remove the domain’s DNS records.

Multiple associated domains

If your cdn-route service has multiple associated custom domains, you must update your service with the reduced list of custom domains:

cf update-service SERVICE_INSTANCE \
-c '{"domain": "REDUCED_SUBDOMAIN_LIST"}'

For example, your cdn-route service is named custom-domains-production and has 3 custom domains, www.example.com, www.example.net and www.example.org. Run the following to remove www.example.com:

cf update-service custom-domains-production \
    -c '{"domain": "www.example.net,www.example.org"}'

Changing your custom domain’s DNS records

Once you have removed the custom domain(s) from your cdn-route service, you should delete any associated DNS records. This stops the removed domains’ DNS records from pointing towards GOV.UK PaaS. You should:

delete or amend the subdomain’s CNAME so that it does not point to GOV.UK PaaS
delete the subdomain’s TXT record you created when you configured the subdomain to work with GOV.UK PaaS

Warning Remove a custom domain from your cdn-route service before removing the DNS records pointing at GOV.UK PaaS.

Otherwise all of the domains` associated with that cdn-route service will fail to renew their TLS certificates without warning at some point over the next 90 days. Your users may not be able to use your service.

Configuring your custom domain

Disabling forwarding cookies

By default cookies are forwarded to your app. You can disable this by setting the cookies parameter to false:

cf update-service my-cdn-route \
    -c '{"cookies": false}'

See the How custom domains work section for details.

Forwarding headers

By default, our service broker configures the CDN to only forward the Host header to your app. You can forward additional headers; in this example you can additionally forward the Accept and Authorization headers:

cf update-service my-cdn-route \
    -c '{"headers": ["Accept", "Authorization"]}'

You can supply up to nine headers. If you need to allow more headers you will have to forward all headers:

cf update-service my-cdn-route \
    -c '{"headers": ["*"]}'

Note that forwarding headers has a negative impact on cacheability. See the How custom domains work section for details.

Troubleshooting custom domains

Service creation timescales

Service creation usually takes approximately one hour. Whilst a service is being created, you will see the status “create in progress” reported from commands like cf service. If it has not finished after 2 hours, we recommend that you check your DNS setup to make sure you completed the CNAME record creation correctly (see step seven). If this does not solve the issue, you may need to contact support.

CloudFront timescales for serving your origin from all locations

It may take some time for CloudFront to serve your origin from all locations, because it is a globally distributed network. You can check in advance by sending requests to CloudFront and passing the custom domain in the Host header, using the example below. The CloudFront domain can be found by running cf service my-cdn-route.

curl -H "Host: <Custom domain>" https://<CloudFront domain>/

When this is working consistently, you can flip your DNS record to start serving content from the CDN. You may still have to wait for up to an hour to make sure the CloudFront distribution is updated everywhere.

“An operation for service instance [name] is in progress”

You may get the following error message when you try to update or delete a service instance:

Server error, status code: 409, error code: 60016, message: An operation for service instance [name] is in progress.

This happens because you cannot do anything to a service instance while it’s in a pending state. A CDN service instance stays pending until it detects the CNAME or ALIAS record. If this causes a problem for you, contact support to ask us to manually delete the pending instance.

View the service configuration

The message field may appear blank following operations such as changing the forwarded headers. In that case, you can get information about the configured domains with the Cloud Foundry CLI (version 8 and above) using:

cf service SERVICE_INSTANCE --params

where SERVICE_INSTANCE is the name of one of your existing cdn-route service instances.

Example output:

{
   "cache_ttl": 0,
   "cloudfront_distribution_id": "ENHBRDSSQXJV0",
   "cloudfront_domain": "d3j6yjt78pkdqf.cloudfront.net",
   "dns_records": [
      {
         "validating_domain_name": "www.example.com",
         "challenge_dns_record": "_83878ed284b0b5fcaa3e99a618432ac8.www.example.com",
         "challenges_dns_record_type": "CNAME",
         "challenges_dns_record_value": "_5c7e8297b9691c59197e4e10b5bf0a98.tfmgdnztqk.acm-validations.aws",
         "status": "SUCCESS"
      }
   ],
   "forward_cookies": true,
   "forwarded_headers": [
      "Host"
   ]
}

This also exposes information such as the CloudFront “distribution id” of the service instance, which can be useful when migrating a domain away from GOV.UK PaaS.

How custom domains work

Overview

Custom domains are configured by our cdn-route service which uses a CloudFront distribution to proxy and/or cache requests to your app.

Caching

CloudFront uses your app’s Cache-Control or Expires HTTP headers to determine how long to cache content.

If your app does not provide these headers, CloudFront will use the configured default time-to-live. This can be particularly confusing as different requests might be routed to different CloudFront Edge endpoints.

When you create your CDN distribution, CloudFront sets the default time-to-live for your CDN-route service at:

24 hours if you created your CDN distribution before 2019-08-12
0 seconds if you created your CDN distribution after 2019-08-12

While there is no mechanism for GOV.UK PaaS users to trigger a cache clear, GOV.UK PaaS support can. Cache invalidation is not instantaneous; Amazon recommends expecting a lag time of 10-15 minutes (more if there are many distinct endpoints).

You can configure CloudFront to forward headers to your app, which causes CloudFront to cache multiple versions of an object based on the values in one or more request headers. See CloudFront’s documentation on headers and distributions for more information. This means the more headers you forward the less caching will take place. Forwarding all headers means no caching will happen.

You can reconfigure the default time-to-live for your CDN route service:

cf update-service my-cdn-route -c '{"default_ttl": 15}'

The above example configures the CDN route to have a default time-to-live of 15 seconds.

Authentication

The CDN route service forwards cookie headers your app by default, so cookie-based authentication will work as expected. Other headers, such as HTTP auth, are stripped by default. If you need a different configuration, see the guidance on forwarding headers.

Verifying protocols and cyphers

You can find the TLS version and cyphers used on GOV.UK PaaS by using an SSL testing tool such as SSL Labs. For your apps, use your custom domain as the hostname.

Further information

See the GOV.UK guidance on:

See the CloudFront documentation on: