Status of this memo

This setup was announced in November 2016. By March 2017, the UIS determined it wasn't feasible to deploy Cloudflare on a large scale; we used Cloudflare only for www.cam.ac.uk between March and October 2017.

As such the mechanism described below was never used. This document remains for curiosity only.

Introduction

Cloudflare is a web content delivery network with an emphasis on denial-of-service protection.

The UIS are aiming to deploy Cloudflare in front of the University's most prominent / sensitive web sites; this service might be extended more widely to other web sites, though it is not currently clear if this will be feasible.

Audience of this document

The notes below are aimed at roughly the following people:

• webmasters who need to know about the DNS aspects of our Cloudflare setup;

• those responsible for administering our Cloudflare account who might need a reminder about certain technical details;

• technical staff around the university who have a general interest in our DNS services.

How Cloudflare works

In a basic web server setup, there are:

• the site name, which appears in the URL;
• the DNS resolves the site name to its origin server;
• the origin server, which hosts the site.

The DNS records at the site name can be address records that refer directly to the origin server, or a CNAME record which refers to the name of the origin server.

Cloudflare acts as a proxy cache that intermediates between browsers and origin servers. In a Cloudflare setup, there are:

• the site name, which appears in the URL;
• the DNS resolves the site name to Cloudflare's servers;
• Cloudflare, which proxies requests for the site name to its origin server;
• the origin server, which hosts the site.

Cloudflare's site name to origin server configuration looks like the set of DNS records that you would normally use to send traffic directly to the origin server, plus an orange or grey cloud icon to indicate whether the site is or is not proxied by Cloudflare. The orange / grey cloud changes the DNS to point to Cloudflare or directly to the origin server.

Cloudflare and Cambridge

Usually Cloudflare completely takes over the DNS for a site, but for our deployment at Cambridge we are retaining control over our DNS. (Cloudflare calls this a "CNAME setup".) This means a lot of Cloudflare's introductory documentation doesn't apply to our setup.

The IP Register database continues to be the place where the DNS is configured for cam.ac.uk (apart from a few special cases). We have added a local mechanism which provides the functionality of the DNS tab in Cloudflare's control panel.

Cloudflare's other web-related functionality (all the non-DNS tabs) is configured in the Cloudflare control panel.

Management of our Cloudflare account

The University's Cloudflare account is centrally managed by the UIS. Cloudflare does not currently have any way to delegate control over subdomains, and many of their configuration settings apply to all web sites in a zone. They also have problematic restrictions on TLS.

Because of these limitations, and the early stage of this project, we are not currently capable of providing Cloudflare for non-UIS web sites. Expressions of interest from colleagues outside the UIS should be directed to ip-register@uis.cam.ac.uk in the first instance.

Access to the Cloudflare control panel

Our account for cam.ac.uk is managed by

• John Burnham
• Tony Finch

There is also a hostmaster user which has DNS-only access for API automation.

Privileges in the IP Register database

There is a UIS-CF mzone which owns the domain cf.arpa.cam.ac.uk (arpa.cam.ac.uk is our domain for DNS infrastructure stuff)

John Burnham and the hostmaster team have access to this mzone.

Setting up Cloudflare requires access to both the web site's mzone and the UIS-CF mzone. Currently John can set up sites under admin.cam.ac.uk and the hostmaster team are able to do any others.

Setup process

1. provision TLS certificate
2. redirect DNS

TLS certificates

At the moment Cloudflare allow us to have only one certificate for all our sites, so provisioning a site requires re-issuing the certificate with an additional subjectAltName and re-uploading it to Cloudflare.

Cloudflare also provide us with a wildcard certificate, but that only works for one level of subdomain, so it does not work for names like www.uis.cam.ac.uk which have two levels of subdomain.

Redirect DNS to Cloudflare

In the IP Register database, you create or remove (or you ask the hostmaster team to create or remove) a special CNAME which does the job of Cloudflare's orange cloud / grey cloud button.

(Yes, this is a disgusting hack. Sorry.)

For example, to enable Cloudflare for www.botolph.cam.ac.uk,

• Go to https://jackdaw.cam.ac.uk/ipreg/cname_ops and input

  name: [ www.botolph.cf.arpa.cam.ac.uk ]
  target_name: [ www.botolph.cam.ac.uk ]
  purpose: [ Cloudflare anchor ]
  [CREATE]
  

• The "target_name" must be the original site name.

• The "name" is similar, but with .cf.arpa. inserted before cam.ac.uk.

The site will move to Cloudflare at the next hourly HH:53 DNS update.

The weird CNAME enforces database constraints on the target (i.e. the site name) which prevent it from being deleted without the Cloudflare setup also being deleted.

Caveats

• The site name should be a "virtual host", i.e. it should be separate from the name you use for managing the origin server, so you can continue to log in to the origin server after the DNS is redirected to Cloudflare.

• The site name (i.e. www.botolph.cam.ac.uk in the example) can be any kind of object in the IP Register database (vbox, cname, etc.), but if it is a cname, the target should not also be a Cloudflare site. If this occurs, the hostmaster team can adjust the contents of the database to avoid this problematic situation.

• The DNS build scripts check .cf.arpa. CNAMEs carefully - if you don't follow the instructions, the hostmaster team gets cronspam, and viewers of the site continue to go direct to the origin server instead of via Cloudflare.

Effect on the DNS

Normally the DNS closely reflects the contents of the IP Register database. The DNS for web sites hosted on Cloudflare is modified in one of two possible manners:

1. There are no other DNS records at the web site name.

   In this case the site's DNS records (whether a CNAME record, or one or more address records) are all replaced with a CNAME record like

   www.botolph.cam.ac.uk. CNAME www.botolph.cam.ac.uk.cdn.cloudflare.net.
   

   This CNAME setup is preferred when possible, because it gives Cloudflare the maximum latitude to perform clever tricks.

2. There are other DNS records at the web site name - most commonly an MX record.

   In this case the site's DNS records are replaced with the address records we get from resolving the CNAME target from case 1, plus an HINFO record to indicate that we would like to put a CNAME here but cannot. The result is like,

   www.botolph.cam.ac.uk. HINFO ( "CNAME"
                            "www.botolph.cam.ac.uk.cdn.cloudflare.net." )
   www.botolph.cam.ac.uk. A     ...
   www.botolph.cam.ac.uk. AAAA  ...
   

   Cloudflare refers to this as "CNAME flattening". Our provisioning process re-does the CNAME flattening lookup every hour.

   The HINFO record is a figleaf towards our policy on external references from the cam.ac.uk domain.

Effect on Cloudflare

When a site is enabled on Cloudflare, our DNS build scripts automatically configure Cloudflare's DNS control panel with address records that refer to the site's origin server. This configuration is automatically updated if the address(es) of the origin server change.

There can be a small delay between adding a site to Cloudflare via its API, and Cloudflare's servers being ready to proxy the site. Our provisioning script waits until Cloudflare is ready to serve a site before it changes the DNS for the site to point at Cloudflare. Note that these checks look at the http version of a site but not https, on the basis that http is mandatory for a web site whereas https is optional. This means there is no check that TLS certificates have been properly provisioned.

When taking a site off Cloudflare, the provisioning script removes the Cloudflare redirection from the DNS straight away, but does not immediately remove the configuration from the Cloudflare DNS control panel. There can be DNS propagation delay of an hour or two, during which browsers will still be directed to Cloudflare, so the configuration must remain. Rather than keeping track of the necessary propagation time, the script removes the configuration in the small hours of the following night, which should allow plenty of time.

Failure scenarios

The main purpose of our Cloudflare deployment is to give us better denial-of-service protection, so it's important to pay attention to what problems can happen when we have poor connectivity.

The Cloudflare automation adds some external dependencies to our hourly DNS build process:

• DNS lookups on Cloudflare host names and IP addresses, for CNAME flattening.
• Queries against the Cloudflare API to read its current configuration.
• Requests to the Cloudflare API to make configuration changes.
• Requests to a Cloudflare proxy to check a new site is working.

The process maintains a cache of DNS lookups and Cloudflare configuration state, so that it can continue successfully even if the University has poor network connectivity to Cloudflare. This works for the first two points listed above.

If the process needs to make changes to the Cloudflare configuration, but is unable to do so, it aborts the DNS build and calls for human attention. This can occur in the second two points listed above.