Mask non-deterministic chart data in visual (Argos) tests - #126
Conversation
Visual regression snapshots run against real NDIF, whose float outputs drift run-to-run and cause pixel noise in heatmap cell colors and plotted line curves. Add a NEXT_PUBLIC_VISUAL_TEST flag that, when set, neutralizes just the noisy values before they reach the nnsightful widgets: - Logit-lens heatmap: blank every cell except the final prediction (last token of the last layer) and flatten tracked/entropy so the remaining cell color and layer skyline are stable. - Activation-patching line plot: flatten the plotted series so the drawn lines are constant; axes, mode bar and token selector still render. The real NDIF request/response path is unchanged; only rendering is masked. The flag is enabled in the e2e workflow before the build. Co-authored-by: Jonathan Bell <jon@jonbell.net>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Argos notifications ↗︎
|
|
🚀 Preview deployed
|
Co-authored-by: Jonathan Bell <jon@jonbell.net>
2 participants
Problem
The visual regression suite (Argos) runs against the real NDIF service and is flaky. Real model runs are only approximately reproducible: floating-point results drift run-to-run across NDIF hardware, so even for a fixed prompt the heatmap cell colors and plotted line curves shift by a few pixels. Argos flags this noise as a diff even though the UI is correct.
We want to keep exercising the real NDIF request/response path, but stop comparing the noisy pixels.
Approach
Add a
NEXT_PUBLIC_VISUAL_TESTflag. When set, we neutralize just the non-deterministic values right before they reach thennsightfulwidgets — the network path, polling, and widget wiring are all still exercised:Lens2Display): blank every cell's top-k except the final prediction — the last token of the last layer (which the widget already treats as the final-prediction cell). The predicted token text there stays real;tracked/entropyare flattened so that cell's color and the per-layer skyline are stable. Fresh workspaces show no trajectory line-plot by default, so only the (masked) heatmap is captured.ActivationPatchingDisplay): flatten the plotted series (lines/ranks/prob_diffs) so the drawn curves are constant and ignored. The chart chrome (axes, mode bar, token selector, layer count, labels) still renders.The masking is centralized in
src/lib/visualTest.tsand gated byisVisualTestMode(), memoized so the widgets don't rebuild every render. Outside visual-test mode there is zero behavior change.Enabling it
NEXT_PUBLIC_VISUAL_TESTis a build-time-inlined public env var, so it's added to.envin the e2e workflow beforenext build.Files
workbench/_web/src/lib/visualTest.ts— flag +maskLogitLensDataForVisualTest/maskActivationPatchingDataForVisualTest.workbench/_web/.../lens2/[chartId]/components/Lens2Display.tsx— apply mask to widget data.workbench/_web/.../activation-patching/[chartId]/components/ActivationPatchingDisplay.tsx— apply mask to widget data..github/workflows/e2e.yml— setNEXT_PUBLIC_VISUAL_TEST=true.Verification
bunx tsc --noEmit: no new errors from these changes (the one remaining error on the AP widgetdataprop is pre-existing — verified against the base commit — and silenced byignoreBuildErrors).bun run lint: no new lint errors in the changed files.Note: baselines will need to be re-approved once in Argos, since the rendered snapshots change (that's the intended, now-deterministic output).