Understanding request and response headers related to caching
This note describes how browsers communicate with the server to make optimal use of documents and resources that have previously been received.
Caching is a broad term that describes the storage mechanisms that keep duplicate copies of resources close to the place where they are needed. The key benefits provided by caches are reduced processing effort, and faster response times due to shorter retrieval distances. All parties to the transaction need to know how caches are managed to prevent unwanted side effects.
This note is concerned with only one type of cache: the local cache on the user's device, which is called the browser cache.
When a user navigates to a web page for the first time, the browser issues a series of requests for the document, together with any associated style sheets, scripts, and media files referenced by the document.
In addition to rendering the web page in the window, the browser also captures and stores those documents and resources in a private storage area on the user's device: this is the browser's cache. Subsequently, when the user navigates to another document on the same website, the new document and its resources are requested, but this time the full request-response cycle could potentially be shortened if it turns out that any of the referenced resources are already in the browser's cache.
Browsers use their cached version of files if they are present, and send requests to the server if not.
The working assumption is that servers will update their resources relatively infrequently, and that users will request resources more than once before they are changed. But this assumption needs to be checked. How can the browser be certain that their duplicate copy has not been changed since it was retrieved from the server? The answer lies in the protocol instructions defined by the server in the
cache-control response header sent during the original request.
In its simplist form, the
cache-control response contains an instruction to the browser telling it how long the resource should be kept and used, without questioning the server again. This is the
max-age instruction, which is the number of seconds, after receipt, that the resource may unconditionally be considered valid.
When a request is made for a resource that is in the browser's cache, but the
max-age has been exceeded, the browser issues a conditional request to the server for a fresh copy of the resource. This conditional request includes information that the server can use to determine whether the browser's copy is still valid. If it is, the server responds with status code
304, and renewed
cache-control instructions, but without the resource in its payload. On the other hand, if the resource has changed on the server, a fresh copy of the resource is returned in the payload, with a status code of
The server can determine whether a browser's cached duplicate is still valid by comparing timestamps during a conditional request. When a request for a resource is originally fulfilled by the server, it sends a
last-modified header in the response, which is stored by the browser in its cache together with the resource itself. Subsequently, when a conditional request is made for a resource, the browser sends a
if-modified-since header containing that saved value. The server compares the two values and decides whether to respond with
The use of timestamps for conditional requests is not perfect, because the server and browser may not have their clocks precisely synchronized, or because a file may have its modification date changed even though its binary contents are identical. The preferred solution to this problem is the use of
etag headers. An Etag is a hashed value used as an identifier for a resource version. Whenever a resource is changed, its hashed value results in a new Etag.
The protocol for using Etags is similar to the protocol outlined for modification dates: 1) the original server response sends the resource together with an
etag header; 2) the browser saves the Etag in its cache together with the resource; 3) conditional requests from the browser are sent with either a
if-none-match header containing the saved Etag value; 4) the server compares the resource's hashed value against the conditional Etag value and responds with
304 and no payload if they match, or
200 and the resource in the payload if they differ.
Understanding how caching works is important for proper server configuration. Proper use of caching protocols will allow users to receive fresh copies of resources when needed in the shortest amount of time.
Caching is only applicable to
PUT requests. When used with
PUT requests, the special
if-match header value
'*' indicates that the payload should only be accepted if the file does not already exist.
The server side of the caching protocol is configured using the
etag modules. Both of these modules should be enabled for most typical setups.
cache-control section comprises a collection of entries, where each entry has a path-pattern and an
instructions attribute comprising a comma-separated list of caching instructions.
Refer to the separate note regarding Path Pattern rules.
As a general guide, the two most useful caching instruction idioms are:
- For resources that the browser should request every time use:
no-cache, no-store, must-revalidate.
- For resources that the browser should consider to be valid for the next N seconds, use:
The devops staff should make its own determination of what a good value for
The above description of browser caching applies to proxy caches too, which sit between the client's device and the server. Caching instructions can be sent to the browser only (by including the keyword
private) or both the browser and proxy caches (by including the keyword
public). Proxy caches also understand the special instructions
proxy-revalidate. Refer to RFC 7234 for details.
The Etag handler does not have any configurable options: it is either on or off. Etag values are computed as an SHA1 hash of the file's modification time and file size. The full value is shortened to just the first six and last six hex digits of the hash to look something like
cache-control configuration section may appear in either the
server section or a
host section: merging is not supported. If values are placed in the
host section they will be used in their entirety; if not, the values in the
server section, if any, will be used as a fallback.
When a path-pattern is listed in the
cache-control section without any caching instructions, a
rw-no-cache-control information header is added to the response.
PUT request fails due to unmatched Etags, a
rw-if-none-match information header is added to the response and a status code of
412 is returned.
|file-system-chars||::=||(ALPHA | DIGIT | †)*|
|wildcards||::=||ASTERISK | QUESTION-MARK|
|path-pattern||::=||(SOLIDUS | file-system-chars | wildcards)*|
|delimited-path-pattern||::=||GRAVE-ACCENT path-pattern GRAVE-ACCENT|
|cache-instruction||::=||'no-cache' | 'no-store' | 'no-transform' | 'public' | 'private' | 'proxy-revalidate' | 'max-age' | 's-maxage' ††|
|cache-control-entry||::=||delimited-path-pattern SP ASTERISK 'instructions' EQUALS_SIGN (cache-instruction ',')* CR|
|cache-control-section||::=||'cache-control' SP LEFT-CURLY-BRACKET CR|
† Legal file system characters vary by platform
†† See RFC 7234 for exact set of allowable instructions
Example 1: No caching, no Etags
Example 2: Caching and Etags
`*.html` *instructions='no-cache, no-store, must-revalidate'
`*.css` *instructions='public, max-age=86400'
`*.js` *instructions='public, max-age=86400'
`/favicon.ico` *instructions='public, max-age=7776000'
`*.gif` *instructions='public, max-age=7776000'
`*.png` *instructions='public, max-age=7776000'
`*.jpeg` *instructions='public, max-age=7776000'
Key points to remember:
- The caching protocols benefit everyone, and should always be used.
- The use of Etags is preferred over the use of conditional timestamps.
- Setting the
max-agevalue is application specific and should be determined in the context of how often a resource changes and what having an out-of-date resource would mean to the user.