[simonchrz]

Problem

Rewritten npm and Composer metadata is sent uncompressed. With npm_full_metadata on, a large packument is 10–16 MB (typescript: 15.8 MB, @types/node: 11.2 MB). Either clients download all of it, or a reverse proxy in front gzips the same document again on every request.

Changes

Gzip copy in the rewrite cache (rewrite_cache.go)

• Each cached rewrite also keeps its output gzip-compressed, made once when the rewrite is cached. The copy is kept only when it's smaller than the output.
• The copy counts against metadata_rewrite_cache_size and typically adds about an eighth to an entry.

Serving

• Clients whose Accept-Encoding allows gzip (gzip, x-gzip or * with q > 0) get the copy with Content-Encoding: gzip.
• The gzip representation has its own ETag, "<sha256>-gzip", so it never matches the uncompressed one. Both revalidate to 304, also when a reverse proxy weakened the ETag.
• Rewritten metadata always sends Vary: Accept-Encoding, 304s included.

Unchanged

• Rewrites made outside the cache (cache disabled, cooldown on) stay uncompressed, because compressing them on every request is better left to a reverse proxy.
• Last-Modified is the same for both representations.

Cost

Go compress/gzip, default level, once per distinct upstream document:

Document	Size	Gzipped	Time
typescript (full)	15.8 MB	2.0 MB (12.7%)	62 ms
@types/node (full)	11.2 MB	1.4 MB (12.7%)	41 ms

Verification

Built the proxy with cache_metadata and npm_full_metadata on:

• curl /npm/typescript: 2.0 MB with Accept-Encoding: gzip, 15.8 MB with identity. Revalidating with the gzip ETag returns 304.
• curl /composer/p2/symfony/http-kernel.json: 61 KB gzip, 542 KB identity.
• npm install typescript left-pad and composer require symfony/http-kernel through the proxy both succeed.
• go test -race ./... and golangci-lint run ./... pass.