The gateway refused the script it was serving - #27
Conversation
A release changed the document's inline theme bootstrap and left gateway.mjs byte identical. The cutover therefore did the right thing and left the gateway running, and the running gateway went on allowing the previous build's script hash while refusing the one it was itself serving: Executing inline script violates the following Content Security Policy directive 'script-src 'self' 'sha256-TLx68/+2SeR3+dZreFBX1fiMo91olqe4aVTfKFd48Ik='' Every page loaded with the theme bootstrap blocked. The only sign was a console error, which is why this is the second defect this week whose whole visible symptom was something nobody was looking at. The cause is a contradiction inside this file. The static root is a fixed path whose contents a release swaps underneath it, and that is deliberate: it is exactly why a frontend change needs no gateway restart, and the release script says so in as many words. The policy was computed once at start-up and did not follow that swap. One half of the file tracked the build behind the path and the other half remembered the build that was there when the process began. It follows the file now, keyed on size and modification time, so a document costs one stat and a hash is computed only when the file actually changes. The test writes two builds behind the same path and asserts the policy names the second and not the first, including the case where both land in the same second, which is a window a release fits inside. Against the previous start-up-pinned version it fails.
The first version of this fix cached the policy against the document's size and modification time. That key is wrong in a way that would have been very hard to find afterwards: two builds of the same length restored to the same instant share it, which is exactly what a tar extraction preserving mtimes can produce, and the resulting failure is a blocked script and nothing else. The test asserted that case and passed for the wrong reason. `utimesSync` takes a Date, so it wrote a millisecond-truncated time while `mtimeMs` carries finer precision, and the keys differed by an accident of rounding rather than by anything the code intended. The document is 3.4 kilobytes and is the same file the response is about to serve. The saving was never worth the class of bug it kept open, so it is read each time. The test now covers both dimensions on purpose: a build that changes the length and the time, and a build of exactly the same length with the modification time put back. Neither may be load bearing on its own. Against a start-up-pinned policy it still fails.
|
Correcting my own first attempt at this, before anyone reviews it. The fix originally cached the policy against the document's size and Worse, the test I wrote for that case passed for the wrong reason. The document is 3.4 kilobytes and is the same file the response is about to The test now covers both dimensions on purpose: a build that changes the length |
Found by the gate, in production, and rolled back
ac2bcfaf5was cut over at 03:42:50Z. The chain page smoke against the publicorigin caught this and the release was rolled back to
1b12b26f3at 03:43:35Z,46 of 46 probes at 200 across the rollback, verified clean afterwards.
Two different hashes: the one the gateway allowed, and the one the document it
was serving actually needed. Every page loaded with the theme bootstrap
blocked. Nothing else showed it. The routes answered, the identities matched
the manifest, the visual matrix was green, and the page rendered.
Root cause, which is a contradiction inside gateway.mjs
ROOTis a fixed path whose contents a release swaps underneath it. That isdeliberate and load bearing: it is why a frontend change needs no gateway
restart, and
release.shsays so out loud every cutover:The document policy was computed once at start-up:
So one half of the file tracked the build behind the path and the other half
remembered the build that was there when the process began. The gateway on
this host had been running since 2026-08-29 10:23:31 and was still enforcing
that morning's hash.
#21 changed
index.mempool.htmland was correct to; it is only visible from here becauseit is the first release in a while to change that file without also changing
gateway.mjs.The fix
The policy follows the file, keyed on its size and modification time, so a
document costs one stat and a hash is computed only when the file behind the
path actually changes.
Restarting the gateway on an index change would also have worked and is worse:
it adds a special case to the release script for a rule the gateway should
enforce itself, and it gives up the seamless frontend cutover in the one case
where nothing about the gateway changed.
The test
Writes two builds behind the same path and asserts the policy names the second
and not the first, including when both land in the same second, which is a
window a release fits inside. Against the previous start-up-pinned version:
With the fix, 15 pass.
Release
ac2bcfaf5needs rebuilding on top of this before it goes back out. Therunning release is
1b12b26f3, which is unaffected: it does not carry theindex change, so the hash the gateway holds is the one its document needs.
🤖 Generated with Claude Code