Freeze the Nauvis render parity to fixtures (#227) - #350
Conversation
| */ | ||
| expectRecordedRows(SECTION, 73); | ||
| afterAll(flushRecording); |
There was a problem hiding this comment.
expectRecordedRows(SECTION, 73) only guards record mode — it populates a map that flushRecording reads, and flushRecording returns immediately unless FMW_FREEZE_TIER3=1 (see test/frozenTable.ts). On a normal run, nothing asserts that all 73 frozen rows are actually consulted. tier3Frozen.ts exports frozenCount for exactly this, but it's never imported here.
All three tier-2 planet specs carry this guard in assert mode, e.g. test/wasmNauvisParity.spec.ts#L922-L928, whose own comment states the purpose: "the table must cover the whole grid, so a case or field dropped from the sweep cannot shrink the frozen surface unseen." The same reasoning applies here — once #227 deletes the TypeScript arm, this frozen table is the only thing grading these renders. If a freeze(...) call site is later deleted or skipped, its row stays in tier3-render-checksums.json forever un-consumed, and the suite still goes green with no signal that coverage shrank.
Consider adding something like if (!RECORDING) expect(frozenCount(SECTION)).toBe(73); (importing frozenCount and RECORDING from ./tier3Frozen), mirroring the tier-2 pattern.
There was a problem hiding this comment.
Confirmed and fixed in f8341ac - thank you, this was a real gap.
I checked the precedent before acting on it: all three tier-2 planet specs do
carry the guard in assert mode, and wasmNauvisParity.spec.ts states the reason
in its own comment. This file was missing it.
I went further than the suggested one-liner, because it does not close the hole
you describe. expect(frozenCount(SECTION)).toBe(73) compares the committed
file against a literal, and when a freeze(...) call site is deleted both
sides stay 73 - the row simply goes un-consulted, which is exactly the case
your second paragraph names. Tier 2's version works because its right-hand side
is FIELD_NAMES.length * CASES.length, derived from the spec's own arrays; mine
is a flat literal spanning seven describe blocks and four inline lever tables,
so it has nothing to shrink.
So frozenTable.ts now tracks the distinct rows each run actually looks up, and
the spec asserts both numbers. They fail on opposite mistakes: the table count
catches a re-record that wrote a different surface, the consulted count catches
a call site that stopped asking.
Planted rather than reasoned. Deleting the freeze("spawn two points", ...)
call site:
× consults every frozen row exactly once
AssertionError: distinct rows this run looked up: expected 72 to be 73
Tests 1 failed | 37 passed (38)
All 37 other tests stayed green, which is the finding rather than the fix -
without this guard that deletion was invisible.
The guard is declared last so it sees every call the run made, and under -t it
is filtered out like any other test rather than failing on a partial run.
d95d865 to
e8ac8dd
Compare
The three render parity specs get their TypeScript arm by calling `runRenderRequest` with the engine argument left off. #227 deletes those branches, after which both arms are the SAME code - a comparison that passes while grading nothing. This freezes the first of the three, so the deletion is a subtraction rather than a hole. `test/tier3Frozen.ts` folds each rendered RGBA buffer to a u64 and checks it against `test/fixtures/tier3-render-checksums.json`, which holds 73 rows under `nauvis:render` - every window, lever setting and routed view the spec already covered. Each was recorded only after the two arms were compared and agreed, so the table cannot be wrong while both renderers exist. The fold runs in JavaScript over bytes both arms already return, so this adds no Rust export and rebuilds no engine.wasm. The table plumbing moves to `test/frozenTable.ts`, shared with tier 2 so the guards live in one place rather than two; `tier2Frozen.ts` becomes a thin wrapper over it with its exports unchanged. All nine tier-2 consumers stay green. The spec also asserts its own COVERAGE, which the three tier-2 planet specs already do and this file was missing. `expectRecordedRows` guards only a record run - it feeds `flushRecording`, which returns immediately unless FMW_FREEZE_TIER3=1 - so nothing checked that the 73 rows are consulted on a normal run. `frozenTable.ts` now tracks the distinct rows each run looks up, and a final test asserts both the table count and the consulted count. The two fail on opposite mistakes: the table count catches a re-record that wrote a different surface, the consulted count catches a call site that stopped asking. Comparing the file against a literal alone would move with neither. Planted rather than predicted, all four RUN: - a corrupted row reddens the wasm arm by name - a deleted row fails "no frozen checksum" rather than skipping quietly - a deleted `freeze` call site leaves all 37 other tests GREEN and is caught only by the coverage guard, at "expected 72 to be 73" - the row-count guard fired for real: a first record run declared 73 and recorded 60, and flushRecording DROPPED the section rather than committing a short table. The 13 missing rows were the four overlay lever loops, which share an identical body. The ABI-cap test is deliberately NOT frozen. Both its arms are the TypeScript renderer, which is its whole claim, so a frozen row would capture an image the engine can never reproduce. It belongs to the `> 8` spawn carve-out, and the deletion removes both together. Also corrects two stale counts that nothing asserted: the tier-2 table holds 1,168 folds, not 942 - nauvis grew to 84x6 and the five primitives sections were never counted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015W4pLtEAk7Evo14mBkQ1HN
e8ac8dd to
f8341ac
Compare
The second of the three render parity specs, using the machinery #350 added, so this is the conversion only. `vulcanus:render` holds 26 rows: 4 terrain windows, the 5 routed views, 4 rocks, 5 resources, 4 composite and 4 cliffs. Each was recorded only after the Rust and TypeScript arms were compared and agreed. The Nauvis section came through the re-record byte-identical, which is the merge behaviour `flushRecording` is meant to have and is worth having seen rather than assumed. The three `renderTiled` helpers are deliberately NOT frozen, checked rather than assumed: every one of their `runRenderRequest` calls passes the engine, so they compare tiled against whole rather than Rust against TypeScript. The deletion leaves them grading exactly what they grade now. The coverage guard is here from the start rather than added after review. Planted: deleting the terrain `freeze` call site leaves **all 16 other tests green** and is caught only by the guard, at "expected 22 to be 26". That is the second spec where a deleted call site was invisible to every pre-existing assertion. Claude-Session: https://claude.ai/code/session_015W4pLtEAk7Evo14mBkQ1HN Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freezes the first of the three render parity specs, which is a prerequisite for
the #227 deletion rather than part of it.
Why this has to land before the deletion
The render parity specs get their TypeScript arm by calling the same function
with the engine argument left off:
#227 deletes the branches that second line reaches. After that it is not a
weaker arm, it is the same arm -
runRenderRequestwould have nothing elseto be - so every byte-identity assertion in the file would pass while grading
nothing.
What this does
test/tier3Frozen.tsfolds each rendered RGBA buffer to a u64 and checks itagainst
test/fixtures/tier3-render-checksums.json.nauvis:renderholds 73rows covering every window, lever setting and routed view the spec already
had. Each was recorded only after the two arms were compared and agreed, so the
table cannot be wrong while both renderers exist.
The existing
expect(wasm).toEqual(ts)assertions stay, so a failure stillgives a pixel-level diff rather than two hex numbers. The deletion PR drops
those lines and the
tslocals; the frozen rows are what survive.The fold runs in JavaScript, not Rust. Both arms already hand back RGBA
bytes, so folding there keeps them symmetric and adds no export - which means
this rebuilds no
engine.wasmand cannot go stale against the committed binary.The fold takes the byte length first, so a truncated buffer cannot collide with
a shorter render that shares a prefix.
Shared plumbing, separate tables
test/frozenTable.tsnow holds the table machinery, and bothtier2Frozen.tsand
tier3Frozen.tsare thin wrappers overmakeFrozenTable. The guards thereare the load-bearing part and each was added after a specific way of writing a
wrong table, so one implementation beats two.
tier2Frozen.tskeeps its exportsand its prose; all nine tier-2 consumers stay green.
Tier 3 keeps its own file because a row means a different thing - one
rendered image, not a field folded over a grid - and because
tier2Coverage.spec.tsanchors tier 2's rows to the module's ownchecksum_*exports, which render rows do not have.
Planted, not predicted
cliffs richness 0: cliffs: wasm 11549297961623709281 != frozen 16045690984833335023cliffs richness 0: cliffs: no frozen checksum- a failure, not a skipThe table was restored byte-identically afterwards.
The row-count guard fired for real. The first record run declared 73 and
recorded 60, so
flushRecordingdropped the section and wrote nothing ratherthan committing a short table. The 13 missing rows were the four overlay lever
loops, which share an identical body and which I had skipped. That is exactly
the failure the guard exists to catch, caught.
One test deliberately NOT frozen
refuses the engine for a spawn list longer than the ABI cap. Both its arms arethe TypeScript renderer - that is its whole claim - so a frozen row would
capture an image the engine can never reproduce, and would fail the moment the
carve-out goes. It belongs to the
> 8spawn carve-out, and the deletionremoves both together. Commented in place.
Drive-by, and why it is here
Two counts had drifted and nothing asserted them:
CLAUDE.mdandPROVENANCE.jsonboth said the tier-2 table holds 942 folds. It holds 1,168primitives:sections were never counted.Corrected rather than left, because this PR adds a sibling entry directly beside
them.
Verification
pnpm run verifygreen in 15m13s, including the committed-wasm check.wasmNauvisRenderParity.spec.tsruns 37 tests green against the frozen tablein a normal (non-recording) run.
Refs #227.
🤖 Generated with Claude Code
https://claude.ai/code/session_015W4pLtEAk7Evo14mBkQ1HN