Saving bandwidth and increasing throughput using compression

Content Encoding

Preliminaries

This note describes how to configure the server to compress files while in transit between the server and browser.

File compression is an easy way to increase performance: fewer bytes, faster response times. File compression is supported by all browsers, and most third-party HTTP utilities. It is safe and should be enabled in most production scenarios.

The server supports three compression algorithms: deflate, gzip and brotli. For most text files, brotli is the best choice. For other uncompressed file types, gzip is a good choice.

Compression is most useful on text media. Binary media, including images, audio, and video, are already compressed using their own algorithms, and attempting to compress them further with gzip or deflate provides very little improvement, and in some cases actually increases file size. Note that SVG images are actually text files, and they do benefit from compression.

Small files do not benefit from compression. This is due to the fact that data transmission through the TCP/IP stack is governed by the Ethernet MTU (Maximum Transmission Unit) size of 1500 bytes, so that responses, where the HTTP headers plus payload is less than the MTU, do not improve network throughput. The typical size of response headers varies, but with HTTP/2 HPACK header compression it can be relatively small, say 100 bytes. If the file being served is smaller than the MTU minus HPACK headers, there is no need to compress. The server uses a fixed value of 1400 bytes as the threshold for compression, and will not gzip or deflate files smaller than that threshold.

Compression is negotiated between the browser and server with the accept-encoding request header. When an incoming request is recieved by the server, it looks for that header and determines whether or not it can fulfill the request using any of the listed algorithms. If it can, and if the configuration allows it, the response payload is compressed and a content-encoding header is added to the response indicating which algorithm was chosen.

Configuration

Server compression is enabled by adding an entry to the modules section with content-encoding set to on. During development and while troublshooting problems, this can be set to off and the rest of the configuration related to compression will be ignored.

Compression is configured on a MIME-type basis. The content-encoding configuration section specifies, by content type, which compression algorithm to use. It comprises a collection of two-part entries: the left hand side is the MIME-type, and the right-hand side is a comma-separated list of acceptable compression algorithms, with best choice first.

Compression will only occur when all four of these conditions are true:

  1. The content-encoding module is on.
  2. The browser has included an accept-encoding header with a value that specifies br or gzip or deflate.
  3. The MIME-type of the requested file is configured with an algorithm that matches one of the requestor's values.
  4. The file is larger than 1400 bytes.

Placement

The content-encoding configuration section may appear in either the server section or a host section. When values occur in both the server and host sections, they are merged according to the standard rules defined for the merge attribute.

EBNF

SP ::= U+20
CR ::= U+0D
SOLIDUS ::= U+2F
ASTERISK ::= U+2A
LEFT-CURLY-BRACKET ::= U+7B
RIGHT-CURLY-BRACKET ::= U+7D
media-type ::= 'text' | 'application' | 'image'
subtype ::= (ALPHA | DIGIT | )*
MIME-type ::= media-type SOLIDUS subtype
compression-algorithm ::= 'br' | 'gzip' | 'deflate' | 'none'
content-encoding-entry ::= MIME-type SP compression-algorithm CR
content-encoding-section ::= 'content-encoding' SP LEFT-CURLY-BRACKET CR
content-encoding-entry*
RIGHT-CURLY-BRACKET CR

† See section 4.2 of RFC 6838 for exact rules

Cookbook

Example 1: Content encoding best practice
server {
modules {
content-encoding on
}
response {
content-encoding {
text/css br,gzip
text/html br,gzip
text/plain br,gzip

application/javascript gzip,deflate
application/json gzip,deflate
application/pdf none
application/xhtml+xml gzip,deflate
application/xml gzip,deflate
application/gzip none

image/gif none
image/jpeg none
image/png none
image/svg+xml gzip,deflate
image/webp none
image/x-icon none

audio/mpeg none
video/mp4 none
audio/webm none
video/webm none

font/otf br,gzip
font/ttf br,gzip
font/woff2 none

// any MIME-type not explicitly set to 'none'
default gzip,deflate
}
}
}
Example 2: Content encoding turned off for debugging
server {
modules {
content-encoding off
}
}

Review

Key points to remember:

  • Compression is safe and effective. It should always be turned on in production.
  • The content-encoding section associates MIME-types with compression algorithms.
  • Binary files should not be compressed.
0

Saving bandwidth and increasing throughput using compression

🔗 🔎