Saving bandwidth and increasing throughput using compression

Content Encoding

Preliminaries

This note describes how to configure the server to compress files while in transit between the server and browser.

File compression is an easy way to increase performance: fewer bytes, faster response times. File compression is supported by all browsers, and most third-party HTTP utilities. It is safe and should be enabled in most production scenarios.

The server supports two compression algorithms: deflate and gzip. For most files, gzip is the best choice.

Compression is most useful on text media. Binary media, including images, audio, and video, are already compressed using their own algorithms, and attempting to compress them further with gzip or deflate provides very little improvement, and in some cases actually increases file size. Note that SVG images are actually text files, and they do benefit from compression.

Small files do not benefit from compression. This is due to the fact that data transmission through the TCP/IP stack is governed by the Ethernet MTU (Maximum Transmission Unit) size of 1500 bytes, so that responses, where the HTTP headers plus payload is less than the MTU, do not improve network throughput. The typical size of response headers varies, but with HTTP/2 HPACK header compression it can be relatively small, say 100 bytes. If the file being served is smaller than the MTU minus HPACK headers, there is no need to compress. The server uses a fixed value of 1400 bytes as the threshold for compression, and will not gzip or deflate files smaller than that threshold.

Compression is negotiated between the browser and server with the accept-encoding request header. When an incoming request is recieved by the server, it looks for that header and determines whether or not it can fulfill the request using any of the listed algorithms. If it can, and if the configuration allows it, the response payload is compressed and a content-encoding header is added to the response indicating which algorithm was chosen.

Configuration

Server compression is enabled by adding an entry to the modules section with content-encoding set to on. During development and while troublshooting problems, this can be set to off and the rest of the configuration related to compression will be ignored.

Compression is configured on a MIME-type basis. The content-encoding configuration section specifies, by content type, which compression algorithm to use. It comprises a collection of two-part entries: the left hand side is the MIME-type, and the right-hand side is the compression algorithm.

Compression will only occur when all four of these conditions are true:

  1. The content-encoding module is on.
  2. The browser has included an accept-encoding header with a value that specifies gzip or deflate or both.
  3. The MIME-type of the requested file is configured with an algorithm that matches one of the requestor's values.
  4. The file is larger than 1400 bytes.

Placement

The content-encoding configuration section may appear in either the server section or a host section. When values occur in both the server and host sections, they are merged according to the standard rules defined for the merge attribute.

EBNF

SP ::= U+20
CR ::= U+0D
SOLIDUS ::= U+2F
ASTERISK ::= U+2A
LEFT-CURLY-BRACKET ::= U+7B
RIGHT-CURLY-BRACKET ::= U+7D
media-type ::= 'text' | 'application' | 'image'
subtype ::= (ALPHA | DIGIT | )*
MIME-type ::= media-type SOLIDUS subtype
compression-algorithm ::= 'gzip' | 'deflate' | 'none'
content-encoding-entry ::= MIME-type SP compression-algorithm CR
content-encoding-section ::= 'content-encoding' SP LEFT-CURLY-BRACKET CR
content-encoding-entry*
RIGHT-CURLY-BRACKET CR

† See section 4.2 of RFC 6838 for exact rules

Cookbook

Example 1: Content encoding best practice
server {
modules {
content-encoding on
}
response {
content-encoding {
text/css gzip
text/html gzip
text/plain gzip

application/javascript gzip
application/json gzip
application/pdf none
application/xhtml+xml gzip
application/xml gzip
application/gzip none

image/gif none
image/jpeg none
image/png none
image/svg+xml gzip
image/webp none
image/x-icon none

audio/mpeg none
video/mp4 none
audio/webm none
video/webm none

font/otf gzip
font/ttf gzip
font/woff2 none
}
}
}
Example 2: Content encoding turned off for debugging
server {
modules {
content-encoding off
}
}

Review

Key points to remember:

  • Compression is safe and effective. It should always be turned on in production.
  • The content-encoding section associates MIME-types with compression algorithms.
  • Binary files should not be compressed.

Saving bandwidth and increasing throughput using compression