The regression harness has two independent ways of misleading a reader silently. Both were hit on 2026-08-21, in two separate sessions, and both produced output that would have gone into a pull request body as evidence.
Provenance: the PATH hang was found by the layer-overlap (#298) investigation; the cache-propagation mechanism was diagnosed by the Δ' reference-length (#403) investigation (shared-cached-row observation and the UNIQUE schema read) and confirmed by the layer-overlap investigation's forced experiment. Related but distinct from #338 (Manifest drift across refs), which is a third way the same comparison can mislead.
1. A bare julia from PATH can hang forever
The harness resolves julia from PATH in four places: the version probe at regression-harness/src/env.jl:66, and the three GPEC subprocess launches at regression-harness/src/runner.jl:278, :480 and :597.
Where julia on PATH is the juliaup shim (~/.juliaup/bin/julialauncher), the probe hangs indefinitely rather than returning. Two consecutive runs sat for ~18 min and ~4 min producing no output, indistinguishable from a long precompile. The process tree is the only tell — parent at 0% CPU with a stuck child:
3997 julia --project=regression-harness regression-harness/regress.jl ... 0.0% CPU
4024 /bin/sh /Users/.../.juliaup/bin/julialauncher --version 0.0% CPU
subprocess_julia_version() already wraps the probe in try/catch returning "unknown", so graceful degradation was intended. A hang is not an exception, so that guard never fires.
Remedies:
- Resolve the interpreter from the running process rather than
PATH — Base.julia_cmd() or joinpath(Sys.BINDIR, "julia"). The harness is already running under a working Julia, so the correct binary is known and need not be rediscovered. Fixes all four sites.
- Put a timeout on the
--version probe so it errors instead of wedging if ever spawned externally. "unknown" was the right answer and should be reachable.
Workaround: export PATH="$HOME/.julia/juliaup/julia-<version>/bin:$PATH" before invoking. Passing an absolute path to regress.jl is not sufficient — the hang is in the child it spawns.
2. One cached sample per commit becomes that commit's canonical value
runs is declared UNIQUE(commit_hash, case_name) (regression-harness/src/database.jl:24), so exactly one row exists per (SHA, case). Whichever sample is taken first becomes that SHA's value until someone passes --force, and every later comparison against that SHA inherits it.
For a deterministic case that is a pure win. For one with any run-to-run variation it silently propagates a fixed offset into unrelated comparisons. Measured on diiid_slayer_n1, same case, same two refs, cached versus both sides forced:
| quantity |
vs. cached row |
both sides forced |
ratio |
| Q_root |
5.6e-06 |
5.0e-10 |
11,000x |
| omega_Hz |
1.9e-01 |
1.1e-05 |
17,000x |
| gamma_Hz |
7.6e-02 |
1.5e-05 |
5,000x |
All 16 layer-input quantities were bit-identical in both runs, so no source effect was involved. The residual after forcing is four orders below the cached figure.
The cross-session evidence is the clearest part: two different branches, compared by two different people against the same cached 06be78884 row, both reported omega_Hz = 1.9e-01. An identical figure from two unrelated source deltas is the cache, not physics. Both were on course to be quoted in PR bodies as real movement.
Remedies, in increasing order of ambition:
- Print provenance per side in the report — the
runs table already carries a timestamp, so marking each column cached or fresh makes a poisoned comparison visible rather than invisible.
- Imply
--force for the ref that equals the working tree's HEAD when producing a stamp for a pull request, since that is the side a reviewer is being asked to trust.
- Consider allowing more than one sample per (SHA, case) so a repeat run measures the reproducibility floor instead of overwriting it. Today
--refs X,X cannot measure it: the ref list is not deduplicated (regress.jl:196), but the UNIQUE key means both sides read the same surviving row and the report prints exactly 0.0.
The artifact selectively poisons the comparisons that matter most
Measured across four comparisons on the same case, the cached sample's jitter is only visible where the genuine source effect is at or below sample scale:
| comparison |
genuine mechanism |
cache artifact visible? |
| K^(2mu) reference-length conversion (#403) |
42% gamma |
no — reproduces to 2 digits under --force |
| Delta' shift from the shooting tolerance (#413) |
0.02% gamma |
no — reproduces to 4 digits under --force |
| analytic-derivative swap only |
~nil |
yes — omega 1.9e-01, collapses to 1.8e-03 forced |
| layer-width additions (no SLAYER path, #298) |
nil |
yes — omega 1.9e-01, collapses to 1.1e-05 forced |
Large real effects punch straight through the artifact. Null and near-null results are dominated by it. That is the worst possible selectivity, because "did anything change?" is the question the harness exists to answer, and a null result is the answer most often taken at face value.
Why these belong together
Both fail silently in a tool whose entire purpose is producing a numerical claim a reviewer will trust. Neither produces an error, a warning, or a visible anomaly. The first costs time; the second costs correctness, and did so in two sessions on the same day without either noticing until the same number appeared twice.
The regression harness has two independent ways of misleading a reader silently. Both were hit on 2026-08-21, in two separate sessions, and both produced output that would have gone into a pull request body as evidence.
Provenance: the PATH hang was found by the layer-overlap (#298) investigation; the cache-propagation mechanism was diagnosed by the Δ' reference-length (#403) investigation (shared-cached-row observation and the
UNIQUEschema read) and confirmed by the layer-overlap investigation's forced experiment. Related but distinct from #338 (Manifest drift across refs), which is a third way the same comparison can mislead.1. A bare
juliafrom PATH can hang foreverThe harness resolves
juliafromPATHin four places: the version probe atregression-harness/src/env.jl:66, and the three GPEC subprocess launches atregression-harness/src/runner.jl:278,:480and:597.Where
juliaonPATHis the juliaup shim (~/.juliaup/bin/julialauncher), the probe hangs indefinitely rather than returning. Two consecutive runs sat for ~18 min and ~4 min producing no output, indistinguishable from a long precompile. The process tree is the only tell — parent at 0% CPU with a stuck child:subprocess_julia_version()already wraps the probe intry/catchreturning"unknown", so graceful degradation was intended. A hang is not an exception, so that guard never fires.Remedies:
PATH—Base.julia_cmd()orjoinpath(Sys.BINDIR, "julia"). The harness is already running under a working Julia, so the correct binary is known and need not be rediscovered. Fixes all four sites.--versionprobe so it errors instead of wedging if ever spawned externally."unknown"was the right answer and should be reachable.Workaround:
export PATH="$HOME/.julia/juliaup/julia-<version>/bin:$PATH"before invoking. Passing an absolute path toregress.jlis not sufficient — the hang is in the child it spawns.2. One cached sample per commit becomes that commit's canonical value
runsis declaredUNIQUE(commit_hash, case_name)(regression-harness/src/database.jl:24), so exactly one row exists per (SHA, case). Whichever sample is taken first becomes that SHA's value until someone passes--force, and every later comparison against that SHA inherits it.For a deterministic case that is a pure win. For one with any run-to-run variation it silently propagates a fixed offset into unrelated comparisons. Measured on
diiid_slayer_n1, same case, same two refs, cached versus both sides forced:All 16 layer-input quantities were bit-identical in both runs, so no source effect was involved. The residual after forcing is four orders below the cached figure.
The cross-session evidence is the clearest part: two different branches, compared by two different people against the same cached
06be78884row, both reportedomega_Hz = 1.9e-01. An identical figure from two unrelated source deltas is the cache, not physics. Both were on course to be quoted in PR bodies as real movement.Remedies, in increasing order of ambition:
runstable already carries a timestamp, so marking each column cached or fresh makes a poisoned comparison visible rather than invisible.--forcefor the ref that equals the working tree's HEAD when producing a stamp for a pull request, since that is the side a reviewer is being asked to trust.--refs X,Xcannot measure it: the ref list is not deduplicated (regress.jl:196), but theUNIQUEkey means both sides read the same surviving row and the report prints exactly0.0.The artifact selectively poisons the comparisons that matter most
Measured across four comparisons on the same case, the cached sample's jitter is only visible where the genuine source effect is at or below sample scale:
--force--forceLarge real effects punch straight through the artifact. Null and near-null results are dominated by it. That is the worst possible selectivity, because "did anything change?" is the question the harness exists to answer, and a null result is the answer most often taken at face value.
Why these belong together
Both fail silently in a tool whose entire purpose is producing a numerical claim a reviewer will trust. Neither produces an error, a warning, or a visible anomaly. The first costs time; the second costs correctness, and did so in two sessions on the same day without either noticing until the same number appeared twice.