Saving bandwidth and increasing throughput using compression
Content Encoding
Preliminaries
This note describes how to configure the server to compress files while in transit between the server and browser.
File compression is an easy way to increase performance: fewer bytes, faster response times. File compression is supported by all browsers, and most third-party HTTP utilities. It is safe and should be enabled in most production scenarios.
The server supports three compression algorithms: deflate, gzip and brotli. For most text files, brotli is the best choice. For other uncompressed file types, gzip is a good choice.
Compression is most useful on text media. Binary media, including images, audio, and video, are already compressed using their own algorithms, and attempting to compress them further with gzip or deflate provides very little improvement, and in some cases actually increases file size. Note that SVG images are actually text files, and they do benefit from compression.
Small files do not benefit from compression. This is due to the fact that data transmission through the TCP/IP stack is governed by the Ethernet MTU (Maximum Transmission Unit) size of 1500 bytes, so that responses, where the HTTP headers plus payload is less than the MTU, do not improve network throughput. The typical size of response headers varies, but with HTTP/2 HPACK header compression it can be relatively small, say 100 bytes. If the file being served is smaller than the MTU minus HPACK headers, there is no need to compress. The server uses a fixed value of 1400 bytes as the threshold for compression, and will not gzip or deflate files smaller than that threshold.
Compression is negotiated between the browser and server with the accept-encoding
request header. When an incoming request is recieved by the server, it looks for that header and determines whether or not it can fulfill the request using any of the listed algorithms. If it can, and if the configuration allows it, the response payload is compressed and a content-encoding
header is added to the response indicating which algorithm was chosen.
Configuration
Server compression is enabled by adding an entry to the modules
section with content-encoding
set to on
. During development and while troublshooting problems, this can be set to off
and the rest of the configuration related to compression will be ignored.
Compression is configured on a MIME-type basis. The content-encoding
configuration section specifies, by content type, which compression algorithm to use. It comprises a collection of two-part entries: the left hand side is the MIME-type, and the right-hand side is a comma-separated list of acceptable compression algorithms, with best choice first.
Compression will only occur when all four of these conditions are true:
- The
content-encoding
module ison
. - The browser has included an
accept-encoding
header with a value that specifiesbr
orgzip
ordeflate
. - The MIME-type of the requested file is configured with an algorithm that matches one of the requestor's values.
- The file is larger than 1400 bytes.
Placement
The content-encoding
configuration section may appear in either the server
section or a host
section. When values occur in both the server
and host
sections, they are merged according to the standard rules defined for the merge
attribute.
EBNF
SP | ::= | U+20 |
CR | ::= | U+0D |
SOLIDUS | ::= | U+2F |
ASTERISK | ::= | U+2A |
LEFT-CURLY-BRACKET | ::= | U+7B |
RIGHT-CURLY-BRACKET | ::= | U+7D |
media-type | ::= | 'text' | 'application' | 'image' |
subtype | ::= | (ALPHA | DIGIT | †)* |
MIME-type | ::= | media-type SOLIDUS subtype |
compression-algorithm | ::= | 'br' | 'gzip' | 'deflate' | 'none' |
content-encoding-entry | ::= | MIME-type SP compression-algorithm CR |
content-encoding-section | ::= | 'content-encoding' SP LEFT-CURLY-BRACKET CR content-encoding-entry* RIGHT-CURLY-BRACKET CR |
† See section 4.2 of RFC 6838 for exact rules
Cookbook
Example 1: Content encoding best practice
server {
modules {
content-encoding on
}
response {
content-encoding {
text/css br,gzip
text/html br,gzip
text/plain br,gzip
application/javascript gzip,deflate
application/json gzip,deflate
application/pdf none
application/xhtml+xml gzip,deflate
application/xml gzip,deflate
application/gzip none
image/gif none
image/jpeg none
image/png none
image/svg+xml gzip,deflate
image/webp none
image/x-icon none
audio/mpeg none
video/mp4 none
audio/webm none
video/webm none
font/otf br,gzip
font/ttf br,gzip
font/woff2 none
// any MIME-type not explicitly set to 'none'
default gzip,deflate
}
}
}
Example 2: Content encoding turned off for debugging
server {
modules {
content-encoding off
}
}
Review
Key points to remember:
- Compression is safe and effective. It should always be turned on in production.
- The
content-encoding
section associates MIME-types with compression algorithms. - Binary files should not be compressed.