Saving bandwidth and increasing throughput using compression

Content Encoding

Preliminaries

This note describes how to configure the server to compress files while in transit between the server and browser.

File compression is an easy way to increase performance: fewer bytes, faster response times. File compression is supported by all browsers, and most third-party HTTP utilities. It is safe and should be enabled in most production scenarios.

The server supports three compression algorithms: deflate, gzip and brotli. For most text files, brotli is the best choice. For other uncompressed file types, gzip is a good choice.

Compression is most useful on text media. Binary media, including images, audio, and video, are already compressed using their own algorithms, and attempting to compress them further with gzip or deflate provides very little improvement, and in some cases actually increases file size. Note that SVG images are actually text files, and they do benefit from compression.

Small files do not benefit from compression. This is due to the fact that data transmission through the TCP/IP stack is governed by the Ethernet MTU (Maximum Transmission Unit) size of 1500 bytes, so that responses, where the HTTP headers plus payload is less than the MTU, do not improve network throughput. The typical size of response headers varies, but with HTTP/2 HPACK header compression it can be relatively small, say 100 bytes. If the file being served is smaller than the MTU minus HPACK headers, there is no need to compress. The server uses a fixed value of 1400 bytes as the threshold for compression, and will not gzip or deflate files smaller than that threshold.

Compression is negotiated between the browser and server with the accept-encoding request header. When an incoming request is recieved by the server, it looks for that header and determines whether or not it can fulfill the request using any of the listed algorithms. If it can, and if the configuration allows it, the response payload is compressed and a content-encoding header is added to the response indicating which algorithm was chosen.

Configuration

Server compression is enabled by adding an entry to the modules section with content-encoding set to on. During development and while troublshooting problems, this can be set to off and the rest of the configuration related to compression will be ignored.

Compression is configured on a MIME-type basis. The content-encoding configuration section specifies, by content type, which compression algorithm to use. It comprises a collection of two-part entries: the left hand side is the MIME-type, and the right-hand side is a comma-separated list of acceptable compression algorithms, with best choice first.

Compression will only occur when all four of these conditions are true:

The content-encoding module is on.
The browser has included an accept-encoding header with a value that specifies br or gzip or deflate.
The MIME-type of the requested file is configured with an algorithm that matches one of the requestor's values.
The file is larger than 1400 bytes.

Placement

The content-encoding configuration section may appear in either the server section or a host section. When values occur in both the server and host sections, they are merged according to the standard rules defined for the merge attribute.

EBNF

SP	::=	U+20
CR	::=	U+0D
SOLIDUS	::=	U+2F
ASTERISK	::=	U+2A
LEFT-CURLY-BRACKET	::=	U+7B
RIGHT-CURLY-BRACKET	::=	U+7D
media-type	::=	'text' \| 'application' \| 'image'
subtype	::=	(ALPHA \| DIGIT \| ^†)*
MIME-type	::=	media-type SOLIDUS subtype
compression-algorithm	::=	'br' \| 'gzip' \| 'deflate' \| 'none'
content-encoding-entry	::=	MIME-type SP compression-algorithm CR
content-encoding-section	::=	'content-encoding' SP LEFT-CURLY-BRACKET CR content-encoding-entry* RIGHT-CURLY-BRACKET CR

† See section 4.2 of RFC 6838 for exact rules

Cookbook

Example 1: Content encoding best practice

server {
    modules {
        content-encoding on
    }
    response {
        content-encoding {
            text/css                br,gzip
            text/html               br,gzip
            text/plain              br,gzip

            application/javascript  gzip,deflate
            application/json        gzip,deflate
            application/pdf         none
            application/xhtml+xml   gzip,deflate
            application/xml         gzip,deflate
            application/gzip        none
            
            image/gif               none
            image/jpeg              none
            image/png               none
            image/svg+xml           gzip,deflate
            image/webp              none
            image/x-icon            none
            
            audio/mpeg              none
            video/mp4               none
            audio/webm              none
            video/webm              none
            
            font/otf                br,gzip
            font/ttf                br,gzip
            font/woff2              none

            // any MIME-type not explicitly set to 'none' 
            default                 gzip,deflate        
        }
    }
}

Example 2: Content encoding turned off for debugging

server {
    modules {
        content-encoding off
    }
}

Review

Key points to remember:

Compression is safe and effective. It should always be turned on in production.
The content-encoding section associates MIME-types with compression algorithms.
Binary files should not be compressed.