Cache Control

Preliminaries

This note describes how browsers communicate with the server to make optimal use of documents and resources that have previously been received.

Caching is a broad term that describes the storage mechanisms that keep duplicate copies of resources close to the place where they are needed. The key benefits provided by caches are reduced processing effort, and faster response times due to shorter retrieval distances. All parties to the transaction need to know how caches are managed to prevent unwanted side effects.

This note is concerned with only one type of cache: the local cache on the user's device, which is called the browser cache.

When a user navigates to a web page for the first time, the browser issues a series of requests for the document, together with any associated style sheets, scripts, and media files referenced by the document.

In addition to rendering the web page in the window, the browser also captures and stores those documents and resources in a private storage area on the user's device: this is the browser's cache. Subsequently, when the user navigates to another document on the same website, the new document and its resources are requested, but this time the full request-response cycle could potentially be shortened if it turns out that any of the referenced resources are already in the browser's cache.

Browsers use their cached version of files if they are present, and send requests to the server if not.

The working assumption is that servers will update their resources relatively infrequently, and that users will request resources more than once before they are changed. But this assumption needs to be checked. How can the browser be certain that their duplicate copy has not been changed since it was retrieved from the server? The answer lies in the protocol instructions defined by the server in the cache-control response header sent during the original request.

In its simplist form, the cache-control response contains an instruction to the browser telling it how long the resource should be kept and used, without questioning the server again. This is the max-age instruction, which is the number of seconds, after receipt, that the resource may unconditionally be considered valid.

When a request is made for a resource that is in the browser's cache, but the max-age has been exceeded, the browser issues a conditional request to the server for a fresh copy of the resource. This conditional request includes information that the server can use to determine whether the browser's copy is still valid. If it is, the server responds with status code 304, and renewed cache-control instructions, but without the resource in its payload. On the other hand, if the resource has changed on the server, a fresh copy of the resource is returned in the payload, with a status code of 200.

The server can determine whether a browser's cached duplicate is still valid by comparing timestamps during a conditional request. When a request for a resource is originally fulfilled by the server, it sends a last-modified header in the response, which is stored by the browser in its cache together with the resource itself. Subsequently, when a conditional request is made for a resource, the browser sends a if-modified-since header containing that saved value. The server compares the two values and decides whether to respond with 304 or 200.

The use of timestamps for conditional requests is not perfect, because the server and browser may not have their clocks precisely synchronized, or because a file may have its modification date changed even though its binary contents are identical. The preferred solution to this problem is the use of etag headers. An Etag is a hashed value used as an identifier for a resource version. Whenever a resource is changed, its hashed value results in a new Etag.

The protocol for using Etags is similar to the protocol outlined for modification dates: 1) the original server response sends the resource together with an etag header; 2) the browser saves the Etag in its cache together with the resource; 3) conditional requests from the browser are sent with either a if-match or if-none-match header containing the saved Etag value; 4) the server compares the resource's hashed value against the conditional Etag value and responds with 304 and no payload if they match, or 200 and the resource in the payload if they differ.

Understanding how caching works is important for proper server configuration. Proper use of caching protocols will allow users to receive fresh copies of resources when needed in the shortest amount of time.

Methods

Caching is only applicable to HEAD, GET and PUT requests. When used with PUT requests, the special if-match header value '*' indicates that the payload should only be accepted if the file does not already exist.

The server side of the caching protocol is configured using the cache-control and etag modules. Both of these modules should be enabled for most typical setups.

The server's cache-control section comprises a collection of entries, where each entry has a path-pattern and an instructions attribute comprising a comma-separated list of caching instructions.

Refer to the separate note regarding Path Pattern rules.

As a general guide, the two most useful caching instruction idioms are:

• For resources that the browser should request every time use: no-cache, no-store, must-revalidate.
• For resources that the browser should consider to be valid for the next N seconds, use: public, max-age=N.

The devops staff should make its own determination of what a good value for max-age is. Images, audio, and video may be good candidates for lengthy max-age values, while style sheets and javascript might have shorter values, especially during periods of active development. Similarly, archived documents could have lengthy max-age values, while new documents, that are still under revision, should have a max-age close to zero.

Proxy caching

The above description of browser caching applies to proxy caches too, which sit between the client's device and the server. Caching instructions can be sent to the browser only (by including the keyword private) or both the browser and proxy caches (by including the keyword public). Proxy caches also understand the special instructions s-maxage and proxy-revalidate. Refer to RFC 7234 for details.

Etags

The Etag handler does not have any configurable options: it is either on or off. Etag values are computed as an SHA1 hash of the file's modification time and file size. The full value is shortened to just the first six and last six hex digits of the hash to look something like "a1b2c3...8d9e0f".

Placement

The cache-control configuration section may appear in either the server section or a host section: merging is not supported. If values are placed in the host section they will be used in their entirety; if not, the values in the server section, if any, will be used as a fallback.

Information Headers

When a path-pattern is listed in the cache-control section without any caching instructions, a rw-no-cache-control information header is added to the response.

If a PUT request fails due to unmatched Etags, a rw-if-match or rw-if-none-match information header is added to the response and a status code of 412 is returned.

† Legal file system characters vary by platform

†† See RFC 7234 for exact set of allowable instructions

Review

Key points to remember:

• The caching protocols benefit everyone, and should always be used.
• The use of Etags is preferred over the use of conditional timestamps.
• Setting the max-age value is application specific and should be determined in the context of how often a resource changes and what having an out-of-date resource would mean to the user.