[No QA] Serve the web build's Brotli 11 twins from the Cloudflare Worker

.github/workflows/deploy.yml

@@ -872,7 +872,7 @@ jobs:
           aws s3 cp dist "s3://new-expensify-web-assets-$DEPLOY_ENV/" --recursive \
             --endpoint-url "https://$CLOUDFLARE_ACCOUNT_ID.r2.cloudflarestorage.com" \
             --no-progress \
-            --exclude '*.br' --exclude '*.map'
+            --exclude '*.map'
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.CLOUDFLARE_R2_WEB_ASSETS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.CLOUDFLARE_R2_WEB_ASSETS_SECRET_ACCESS_KEY }}

web/cloudflare-worker/.assetsignore

@@ -1,4 +1,2 @@
-# Precompressed twins. Cloudflare compresses Worker responses at the edge.
-*.br
 # Source maps are uploaded to Sentry at build time and must not be publicly served.
 *.map

web/cloudflare-worker/README.md

@@ -14,11 +14,13 @@ Every request runs the Worker first (`run_worker_first: true`), which reads the
 | HTML (`/`, `/index.html`, any SPA deep link) | Each `<script nonce="nonce-random-value">` gets a per-request nonce that matches the CSP. `Cache-Control: no-cache`, with no `ETag` or `Last-Modified`, since a 304 would pair a cached body with a new nonce |
 | Content-hashed files (`main-<16 hex>.bundle.js`, `name.<10 hex>.css`) | `Cache-Control: public, max-age=31536000, immutable` |
 | Files with fixed names (`version.json`, `service-worker.js`, `manifest.json`, ...) | Default `public, max-age=0, must-revalidate`, with an `ETag` |
+| `js`, `css`, `svg`, `wasm` and `ttf` files | The build's Brotli 11 twin (`foo.js.br`) with `Content-Encoding: br`, about 20% smaller than Cloudflare's edge compression. See [Brotli twins](#brotli-twins) |
+| A request with a matching `If-None-Match` | `304`, answered by the Worker for every non-HTML file |
 | A file the current build doesn't have (`/main-<old hash>.bundle.js`) | The file from a previous build, if the environment has a `PREVIOUS_BUILDS` bucket that holds it (see [Files from previous builds](#files-from-previous-builds)). Otherwise `404`, also when the bucket can't be read, so a stale tab gets a `ChunkLoadError` and recovers instead of parsing HTML as JavaScript |
 | Any other path that does not exist (`/r/123`) | The app shell (`index.html`) with `200` |
 | `/.well-known/apple-app-site-association` and `/apple-app-site-association` | The same file with `Content-Type: application/json` |
 
-[`.assetsignore`](.assetsignore) keeps the Brotli twins (`*.br`) and source maps (`*.map`) out of the upload. Wrangler's `build` command (`scripts/prepareDist.sh`) copies it into `dist/` before every `wrangler dev` and `wrangler deploy`, so always run wrangler from this directory.
+[`.assetsignore`](.assetsignore) keeps source maps (`*.map`) out of the upload. Wrangler's `build` command (`scripts/prepareDist.sh`) copies it into `dist/` before every `wrangler dev` and `wrangler deploy`, so always run wrangler from this directory.
 
 The CSP lives in [`src/csp.ts`](src/csp.ts). The `ENVIRONMENT` variable in [`wrangler.jsonc`](wrangler.jsonc) selects the staging or production policy.
 
@@ -125,9 +127,17 @@ npx wrangler deploy --env test-staging
 
 CI also adds Storybook to `dist/docs` before deploying, so it is served at `/docs/index.html`.
 
+### Brotli twins
+
+The web build writes a Brotli 11 twin next to every `js`, `css`, `svg`, `wasm` and `ttf` file. The Worker serves the twin with `Content-Encoding: br` and `encodeBody: "manual"`, so Cloudflare sends its bytes as they are to clients that accept Brotli, instead of compressing the original itself. Cloudflare presents every request to the Worker with `Accept-Encoding: br, gzip`, and decodes and re-encodes the twin for clients that don't accept Brotli. Twins are never served at their own URLs (`/foo.js.br` is a `404`), and responses for these files carry `Vary: Accept-Encoding`.
+
+The asset store gives a twin the same ETag as its original, so the twin gets its own (`"<etag>-br"`), and the Worker answers `If-None-Match` itself, ignoring the `W/` prefix Cloudflare adds to ETags of responses it re-encodes. Static Assets doesn't support `Range` requests, so those get the whole file.
+
+The test harness decodes responses before the tests read them, so the tests identify the twin by its ETag. Check what clients actually receive against a deployed Worker, for example with `curl -H 'Accept-Encoding: gzip'`.
+
 ### Files from previous builds
 
-Each Worker version only contains the files uploaded with it, but a tab still running an older build keeps requesting that build's chunks after a deploy. So every staging and production deploy also uploads its build (without `*.br`, `*.map` and Storybook) to that environment's R2 bucket, `new-expensify-web-assets-staging` or `new-expensify-web-assets-production`. Every file is re-uploaded each time, so a file still used by recent builds never reaches the bucket's 60-day expiry. When the current build doesn't have a file that a script or image requests, the Worker looks it up in its environment's bucket (the `PREVIOUS_BUILDS` binding; `test-staging` uses the staging one) and returns `404` only if it isn't there either. The current build always takes priority, and navigations never read from the bucket.
+Each Worker version only contains the files uploaded with it, but a tab still running an older build keeps requesting that build's chunks after a deploy. So every staging and production deploy also uploads its build, including Brotli twins but without `*.map` and Storybook, to that environment's R2 bucket, `new-expensify-web-assets-staging` or `new-expensify-web-assets-production`. Every file is re-uploaded each time, so a file still used by recent builds never reaches the bucket's 60-day expiry. When the current build doesn't have a file that a script or image requests, the Worker looks it up, its twin first, in its environment's bucket (the `PREVIOUS_BUILDS` binding; `test-staging` uses the staging one) and returns `404` only if it isn't there either. The current build always takes priority, and navigations never read from the bucket.
 
 The buckets and their 60-day lifecycle rule are managed in Terraform. The upload uses `CLOUDFLARE_R2_WEB_ASSETS_ACCESS_KEY_ID` and `CLOUDFLARE_R2_WEB_ASSETS_SECRET_ACCESS_KEY`, an R2 token that can only read and write those buckets. `wrangler dev` uses an empty local bucket.
 

web/cloudflare-worker/src/index.ts

@@ -6,6 +6,19 @@ const HASHED_ASSET_PATH = /^\/[^/]+(?:-[0-9a-f]{16}\.bundle\.js|\.[0-9a-f]{10}\.
 /** Paths that can only be files. A miss on one of these is a 404 rather than the SPA shell. */
 const STATIC_FILE_PATH = /\.(?:js|mjs|css|map|json|wasm|png|jpe?g|gif|webp|avif|svg|ico|woff2?|ttf|otf|mp3|wav|ogg|lottie|pdf|txt|bcmap|br)$/i;
 
+/** Files the web build also emits as a Brotli 11 twin (`foo.js.br`), which is smaller than Cloudflare's edge compression. */
+const PRECOMPRESSED_CONTENT_TYPES: Record<string, string> = {
+    js: 'text/javascript',
+    css: 'text/css',
+    svg: 'image/svg+xml',
+    wasm: 'application/wasm',
+    ttf: 'font/ttf',
+};
+const PRECOMPRESSED_PATH = /\.(?:js|css|svg|wasm|ttf)$/;
+const TWIN_PATH = /\.(?:js|css|svg|wasm|ttf)\.br$/i;
+
+const REVALIDATE_CACHE_CONTROL = 'public, max-age=0, must-revalidate';
+
 const AASA_PATH = '/.well-known/apple-app-site-association';
 const LEGACY_AASA_PATH = '/apple-app-site-association';
 
@@ -19,6 +32,34 @@ function isHTMLResponse(response: Response): boolean {
     return response.headers.get('Content-Type')?.startsWith('text/html') ?? false;
 }
 
+function getOpaqueTag(etag: string): string {
+    return etag.trim().replace(/^W\//, '').replace(/^"|"$/g, '');
+}
+
+/** The twin and the original share the asset's ETag, so the twin gets its own to keep each representation's validator distinct. */
+function getBrotliETag(etag: string): string {
+    return `"${getOpaqueTag(etag)}-br"`;
+}
+
+/** If-None-Match uses weak comparison, and Cloudflare weakens ETags on responses it compresses. */
+function matchesIfNoneMatch(request: Request, etag: string | null): boolean {
+    const ifNoneMatch = request.headers.get('If-None-Match');
+    if (!ifNoneMatch || !etag) {
+        return false;
+    }
+    const opaqueTag = getOpaqueTag(etag);
+    return ifNoneMatch.split(',').some((candidate) => candidate.trim() === '*' || getOpaqueTag(candidate) === opaqueTag);
+}
+
+/** Decoded, because the asset router also maps `/foo.js.%62r` to the twin. */
+function isTwinPath(pathname: string): boolean {
+    try {
+        return TWIN_PATH.test(decodeURIComponent(pathname));
+    } catch {
+        return TWIN_PATH.test(pathname);
+    }
+}
+
 /** The asset router 307-redirects any other encoding (`@` -> `%40`), which would change the deep link the user opened. */
 function toAssetRouterEncoding(pathname: string): string {
     return pathname
@@ -33,13 +74,14 @@ function toAssetRouterEncoding(pathname: string): string {
         .join('/');
 }
 
-function buildAssetRequest(request: Request, url: URL): Request {
+/** Conditional headers are dropped, because the Worker chooses the representation and answers revalidation itself. */
+function buildAssetRequest(request: Request, url: URL, pathSuffix = ''): Request {
     const assetURL = new URL(url);
-    assetURL.pathname = url.pathname === LEGACY_AASA_PATH ? AASA_PATH : toAssetRouterEncoding(url.pathname);
-    if (assetURL.pathname === url.pathname) {
-        return request;
-    }
-    return new Request(assetURL, request);
+    assetURL.pathname = (url.pathname === LEGACY_AASA_PATH ? AASA_PATH : toAssetRouterEncoding(url.pathname)) + pathSuffix;
+    const assetRequest = new Request(assetURL, request);
+    assetRequest.headers.delete('If-None-Match');
+    assetRequest.headers.delete('If-Modified-Since');
+    return assetRequest;
 }
 
 function setSecurityHeaders(headers: Headers, environment: Env['ENVIRONMENT'], nonce: string) {
@@ -52,14 +94,52 @@ function setSecurityHeaders(headers: Headers, environment: Env['ENVIRONMENT'], n
     headers.set('Document-Policy', 'js-profiling');
 }
 
-function getPreviousBuildFileHeaders(object: R2Object): Headers {
-    const headers = new Headers({'Cache-Control': 'public, max-age=0, must-revalidate', ETag: object.httpEtag});
-    object.writeHttpMetadata(headers);
-    return headers;
+function createNotFoundResponse(environment: Env['ENVIRONMENT'], nonce: string): Response {
+    const notFound = new Response('Not Found', {status: 404, headers: {'Content-Type': 'text/plain; charset=utf-8'}});
+    setSecurityHeaders(notFound.headers, environment, nonce);
+    return notFound;
+}
+
+type ServedFile = {
+    response: Response;
+    isBrotli: boolean;
+};
+
+function createBrotliResponse(body: ReadableStream | null, headers: Headers, etag: string | null): Response {
+    headers.set('Content-Encoding', 'br');
+    // A shared fallback tag would let one twin's 304 answer another's revalidation.
+    if (etag) {
+        headers.set('ETag', getBrotliETag(etag));
+    } else {
+        headers.delete('ETag');
+    }
+    headers.delete('Content-Length');
+    return new Response(body, {headers});
+}
+
+/** The asset, or its Brotli twin when the build has one. */
+async function getAsset(env: Env, request: Request, url: URL, mayHaveTwin: boolean): Promise<ServedFile> {
+    const asset = await env.ASSETS.fetch(buildAssetRequest(request, url));
+    if (!mayHaveTwin || asset.status !== 200 || isHTMLResponse(asset)) {
+        return {response: asset, isBrotli: false};
+    }
+
+    const twin = await env.ASSETS.fetch(buildAssetRequest(request, url, '.br'));
+    if (twin.status !== 200 || isHTMLResponse(twin)) {
+        await twin.body?.cancel();
+        return {response: asset, isBrotli: false};
+    }
+
+    await asset.body?.cancel();
+    return {response: createBrotliResponse(twin.body, new Headers(asset.headers), asset.headers.get('ETag') ?? twin.headers.get('ETag')), isBrotli: true};
+}
+
+async function getPreviousBuildObject(bucket: R2Bucket, key: string, method: string): Promise<R2Object | R2ObjectBody | null> {
+    return method === 'HEAD' ? bucket.head(key) : bucket.get(key);
 }
 
 /** A file uploaded by an earlier deploy, for tabs still running that build. Null when no build had it. */
-async function getPreviousBuildFile(env: Env, url: URL, method: string): Promise<Response | null> {
+async function getPreviousBuildFile(env: Env, url: URL, method: string, mayHaveTwin: boolean): Promise<ServedFile | null> {
     if (!env.PREVIOUS_BUILDS) {
         return null;
     }
@@ -72,13 +152,23 @@ async function getPreviousBuildFile(env: Env, url: URL, method: string): Promise
     }
 
     try {
-        if (method === 'HEAD') {
-            const object = await env.PREVIOUS_BUILDS.head(key);
-            return object ? new Response(null, {headers: getPreviousBuildFileHeaders(object)}) : null;
+        if (mayHaveTwin) {
+            const twin = await getPreviousBuildObject(env.PREVIOUS_BUILDS, `${key}.br`, method);
+            if (twin) {
+                const extension = key.slice(key.lastIndexOf('.') + 1);
+                // The twin's stored type describes the `.br` file, not the content it encodes.
+                const headers = new Headers({'Cache-Control': REVALIDATE_CACHE_CONTROL, 'Content-Type': PRECOMPRESSED_CONTENT_TYPES[extension] ?? 'application/octet-stream'});
+                return {response: createBrotliResponse('body' in twin ? twin.body : null, headers, twin.httpEtag), isBrotli: true};
+            }
         }
 
-        const object = await env.PREVIOUS_BUILDS.get(key);
-        return object ? new Response(object.body, {headers: getPreviousBuildFileHeaders(object)}) : null;
+        const object = await getPreviousBuildObject(env.PREVIOUS_BUILDS, key, method);
+        if (!object) {
+            return null;
+        }
+        const headers = new Headers({'Cache-Control': REVALIDATE_CACHE_CONTROL, ETag: object.httpEtag});
+        object.writeHttpMetadata(headers);
+        return {response: new Response('body' in object ? object.body : null, {headers}), isBrotli: false};
     } catch (error) {
         // A 404 lets the tab recover with a reload, as it does for a file no build has.
         console.error(`Failed to read ${key} from the previous builds bucket`, error);
@@ -90,47 +180,68 @@ export default {
     async fetch(request, env): Promise<Response> {
         const url = new URL(request.url);
         const nonce = generateNonce();
+
+        // Twins are only served as an encoding of the original.
+        if (isTwinPath(url.pathname)) {
+            return createNotFoundResponse(env.ENVIRONMENT, nonce);
+        }
+
         const isNavigation = request.headers.get('Sec-Fetch-Mode') === 'navigate';
         const isAASAPath = url.pathname === AASA_PATH || url.pathname === LEGACY_AASA_PATH;
-        let assetResponse = await env.ASSETS.fetch(buildAssetRequest(request, url));
+        const isPrecompressed = PRECOMPRESSED_PATH.test(url.pathname);
+        // Cloudflare presents every request with `Accept-Encoding: br, gzip` and decodes the twin for clients without Brotli.
+        let served = await getAsset(env, request, url, isPrecompressed);
 
         // The SPA fallback answers every miss with index.html. A browser asking for a missing chunk needs the file from
         // an earlier build, or a real 404.
-        if (isHTMLResponse(assetResponse) && (isAASAPath || (!isNavigation && STATIC_FILE_PATH.test(url.pathname)))) {
-            const previousBuildFile = isAASAPath ? null : await getPreviousBuildFile(env, url, request.method);
+        if (isHTMLResponse(served.response) && (isAASAPath || (!isNavigation && STATIC_FILE_PATH.test(url.pathname)))) {
+            const previousBuildFile = isAASAPath ? null : await getPreviousBuildFile(env, url, request.method, isPrecompressed);
             if (!previousBuildFile) {
-                const notFound = new Response('Not Found', {status: 404, headers: {'Content-Type': 'text/plain; charset=utf-8'}});
-                setSecurityHeaders(notFound.headers, env.ENVIRONMENT, nonce);
-                return notFound;
+                await served.response.body?.cancel();
+                return createNotFoundResponse(env.ENVIRONMENT, nonce);
             }
-            assetResponse = previousBuildFile;
+            await served.response.body?.cancel();
+            served = previousBuildFile;
         }
 
-        const response = new Response(assetResponse.body, assetResponse);
-        setSecurityHeaders(response.headers, env.ENVIRONMENT, nonce);
+        const {response: file, isBrotli} = served;
+        const headers = new Headers(file.headers);
+        setSecurityHeaders(headers, env.ENVIRONMENT, nonce);
 
         if (isAASAPath) {
-            response.headers.set('Content-Type', 'application/json');
+            headers.set('Content-Type', 'application/json');
         }
 
         if (HASHED_ASSET_PATH.test(url.pathname)) {
-            response.headers.set('Cache-Control', 'public, max-age=31536000, immutable');
+            headers.set('Cache-Control', 'public, max-age=31536000, immutable');
         }
 
-        if (!isHTMLResponse(response)) {
-            return response;
+        if (!isHTMLResponse(file)) {
+            if (isPrecompressed) {
+                headers.append('Vary', 'Accept-Encoding');
+            }
+
+            if (matchesIfNoneMatch(request, headers.get('ETag'))) {
+                await file.body?.cancel();
+                headers.delete('Content-Length');
+                headers.delete('Content-Encoding');
+                return new Response(null, {status: 304, headers});
+            }
+
+            // The twin is already Brotli-encoded, so Cloudflare must send its bytes as they are.
+            return new Response(file.body, {status: file.status, statusText: file.statusText, headers, encodeBody: isBrotli ? 'manual' : 'automatic'});
         }
 
         // The nonce changes per request, so a 304 would pair a cached body with a CSP that no longer matches it.
-        response.headers.set('Cache-Control', 'no-cache');
-        response.headers.delete('ETag');
-        response.headers.delete('Last-Modified');
+        headers.set('Cache-Control', 'no-cache');
+        headers.delete('ETag');
+        headers.delete('Last-Modified');
         return new HTMLRewriter()
             .on(`script[nonce="${NONCE_PLACEHOLDER}"]`, {
                 element(element) {
                     element.setAttribute('nonce', nonce);
                 },
             })
-            .transform(response);
+            .transform(new Response(file.body, {status: file.status, statusText: file.statusText, headers}));
     },
 } satisfies ExportedHandler<Env>;