Skip to content

test: retain benchmark invocation evidence and fixed AA calibration - #171

Merged
Iviesever merged 5 commits into
mainfrom
codex/v6-benchmark-attribution
Sep 8, 2026
Merged

Iviesever merged 5 commits into
mainfrom
codex/v6-benchmark-attribution

Conversation

@Iviesever

@Iviesever Iviesever commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Final decision: accept attribution tooling; keep #170 on HOLD

Final source/execution/artifact review and acceptance decision

  • Exact base: d01bb920e030202c33639f59139491ac7f49117c.
  • Reviewed head: 928a5b9e3ce415c1741c441ee7702d4736291ac2.
  • Candidate and native PR test-merge tree: af6b293f04521575a2bdbd2fc31bc156b676d0c8.
  • Native test-merge b5352e1cf12ab43ec43e9e42bdaa93958b0c8cef has exactly those base/head parents.
  • Four benchmark/test/CI files, 617 additions/19 deletions. No product source/header, manifest, scheduler, native executable inventory, CLI/freshness or VERSION changes.
  • VERSION stays5.5.0; no historical tag/asset or formal Release change.

Continue #164's existing route with one attribution topic. #170's runtime source is not modified by this PR. Tooling acceptance does not resolve historical causality or authorize #170 merge.

Delivered behavior

The optional InvocationEvidenceDirectory in benchmark_mqb.ps1 retains started markers, full captured merged TEXT lines, exit/error status and four monotonic invocation boundaries before parsing timing results. It refuses overwritten attempts. Cwd/native-and-capture/pop spans are distinct; native child PID remains unavailable rather than being invented from the host PID. Final source copies are explicitly post-suite snapshots.

With the observer OFF, original native launches and Stopwatch boundary are preserved. The entire original fixture generation, scenario sequence and cache-count assertions are byte-identical. The new observed clock is labeled external_stopwatch_observed, not falsely merged with the old external clock.

The collector archives actual executable bytes/hashes, build/source provenance, exact observer script, allowlisted host environment, plan and each invocation/report. Validation recomputes raw clock arithmetic, original timing/exit correspondence and complete semantic counters. Failures retain the attempted/completed boundary and abort, with no repeated slot or AA subtraction.

Two independent experiments — keep source pairs and clocks separate

Normal PR validation: observer OFF

Full19-row exact-PR-base/head ABBA and adverse samples

Performance Evidence #553/34237296101 built d01 versus928 and measured19x4 AB/BA/AB/BA. All76pairs independently recomputed; all72instrumented counters/breakdowns/hit-miss identities match; four timings-off counter sets remain unavailable. External no-op paired median −0.747ms/−6.43%, MAD1.611ms, max adverse+2.286ms; the original >1ms AND>10% median gate was not crossed. Base55.9592ms versus candidate11.3388ms is a retained favorable extreme, not a speedup claim. Cold, scale-no-op-auto and modules-no-op are slower4/4; all other adverse and favorable results remain.

Fixed attribution investigation: observer ON

FullAA/AB table, actual binaries and localized tails

The advance declaration fixed runtime A=d01, B=held#170 head283b50733601926840fdcd0474d10a06a4f75c7d, observed by928. Eight AA pairs plus eight AB pairs =32 complete reports/672 recorded invocations (64primes),304scenario-pairs. All288instrumented semantic pairs match;16timings-off pairs preserve unavailable counters. All19,872merged output lines, plan/order, timestamps, original JSON/results and exact binaries were independently checked. AA uses the SAME A path and bytes in both roles; contrast order/orientation alternate. No adaptive budget, outlier deletion, retry or AA subtraction.

Observed timings-off no-op: AA+0.1216ms/+1.0806%, AB+0.1088ms/+0.9631%. Neither new median crosses the declared investigation threshold. Yet same-binary AA block3 scale-cold has external+1634.2199ms/internal+1633.615ms, with only+0.0780ms cwd change. AA block2 single-TU has external+1012.9075ms, coinciding with link and cache-save scopes. These are NEW-run localization facts, not proof of the cause of #170's earlier no-op crossing.

Both actual images are1,814,016bytes;19bytes differ in timestamp/decorated-name fields. Raw .text is identical but whole executables are not. Different paths/metadata, observer effects and unrecorded OS effects remain. Cache-save work includes serialization and several file operations, not just disk wait. Nested/parallel work is never added as elapsed. All18 non-no-op-off scenarios still enable internal instrumentation; only timings-disabled-no-op is timings-OFF. No per-child CPU/PID,ETW/syscall trace or byte-exact per-stream capture is claimed.

Completed applicable validation

Native Debug #710/34237296089 actually built and executed all subshards plus freshness/runtime/parameter checks. Native Release #604/34237296111 completed all shards, full self-host and exact runtime-only package validation; publish-release skipped. Documentation #165/34237296136 passed. Attribution #2/34237296093 completed32synthetic helper contracts and the full fixed study. NormalABBA#553 actually measured, not skipped. Cross-Stage/ownership workflows are path-scoped and not triggered by these files; they are not claimed as passed runs.

32-case contract artifact and source/clock compatibility review

Initial0a747 head failed the mistakenly specified legacy bracket-path test after25passing contracts; the fixed study did not start. The correction preserves that legacy limitation as an expected refusal, then tests compatibility on a plain path. Initial Debug#709/Release#603/ABBA#552 were cancelled by the genuine test-source update; ABBA had not reached measurement. The partial artifact/log and initial diff-count typo correction are retained. No same-head rerun or relabeling failed validation as passed.

Original acceptance gates retained

Exact final diff/tree and applicable complete native/release/package/documentation; actual helper tests with valid and rejected cases; unchanged fixture bodies; independent observer-off19x4ABBA with72semantic identities and the unchanged both>1ms and>10% external no-op median gate. Fixed study must complete32reports/672records with288instrumented identities,16offpairs,actual archived binaries,original diagnostics and all adverse samples. Evaluate AA/AB separately using the same investigation threshold. No risk-adjusted subtraction, automatic#170approval or historical reconstruction is implied by collection success.

All applicable tooling gates passed. Accept ordinary expected-head merge under branch protection. Keep #170 Draft/HOLD. Next narrow the cache-save and link/process execution scopes with credible subspan/tool/OS evidence under a declared fixed budget, rather than blindly repeating medians. Historical missing binaries/traces and cold-tail debt remain; no AV/disk/root-cause, performance gain or zero-overhead claim. CLI cancellation, external-writer quiescence and writer leases remain unauthorized.

Verified raw artifacts: initial contracts10060163305,corrected contracts10060400928,normalABBA10060805409,fixed study10061149177. Exact SHA256/CRC checks and runnable read-only recomputation are retained in the review/delivery; Actions retention30days. Title changed only after actual#553/#2 completed; later skipped perf jobs do not replace measured evidence. Final VERSION/release work stays separate.

Copy link
Copy Markdown
Owner Author

First-head Windows contract failure retained; narrow test correction

Head 0a747bac0d867b62a2a2b0ae304c5415e2e90a50 is not accepted. Attribution run 34236726301, job 102096235241, failed during the inactive-observer legacy compatibility test. The legacy helper uses Push-Location $WorkingDirectory (wildcard Path), and the test mistakenly expected it to accept [literal] directory with spaces. The new observed helper's literal-path and output/error persistence tests had already passed. Fixed AA/AB investigation was skipped: zero study reports, not a failed or favorable measurement.

Downloaded original artifact 10060163305, 12,911 bytes, SHA-256 45f0388d38c0838802e10f4cb93b6681c3633f966d5102052097e3946fe1bb28. It retains 25 completed passing checks plus raw empty/nonzero/2400-line/missing-cwd/malformed-timing records. The subsequent unhandled legacy Push-Location exception is preserved in the job log; the partial 25-check JSON is not claimed to be the full suite passing.

Correction is confined to the test script: explicitly assert the legacy bracket-path limitation remains an expected refusal, then run inactive-observer clock/counter compatibility on a plain path with spaces. Do NOT change the historical benchmark launcher to conceal the test mismatch. Observed literal-path coverage stays in place. The fixed study budget, source pair, thresholds, observer implementation and fixtures are unchanged; new head requires its own tests and exact-base/head ABBA. No same-slot retry or same-head rerun.

Metadata correction: initial full diff is 608 additions/19 deletions, not 586 as transcribed in the initial PR body (138+102+244+124 = 608). This is an arithmetic description correction, not a changed scope.

Copy link
Copy Markdown
Owner Author

Corrected-head contract artifact independently verified

Head 928a5b9e3ce415c1741c441ee7702d4736291ac2, base d01bb920e030202c33639f59139491ac7f49117c. Windows contract job 102098194572 in attribution run 34237296093 passed. Artifact 10060400928, 13,038 bytes, downloaded ZIP SHA-256 8d5d92c8b2d17063aa31a5a6243d2abf1a0c3da37fbd071868eba9dbc48cfa07 and CRCs verified.

Read all 32/32 passing contract records under PowerShell7.6.5 and the underlying invocation files: empty output remains an empty array; exit7 retains both original stdout/stderr text lines; all2400 volume-test lines survive; prelaunch cwd failure retains observer_error with exit_code null; malformed timing output is saved before parser rejection. The legacy bracket-path limitation is explicitly checked as refusal, while its ordinary-path compatibility check passes. No synthetic invocation is counted as MSVC benchmark evidence.

Independent static comparison confirms the entire fixture generation, scenario invocation sequence and cache-count assertion body is byte-identical to base. Candidate content Git blobs match reviewed local files: benchmark 3bf6c2d19581a2acf804874925886199d1736b5b; collector ac3021181f430e6c6c716d13fb5d865cf3ab8714; verifier 59266b91ace9267f409a34aa2931d04dd41e76f1; workflow 2a3334828145feb8a88fbad370274cff7eee480e. Corrected full diff is617additions/19deletions.

Initial head's Native Debug#709,Release#603 and ordinary ABBA#552 were cancelled by this genuine test-source correction, not counted as passing validation; Documentation passed. Initial attribution failed before the fixed investigation was admitted. Its partial25-check artifact and log remain. No same-head retry or discarded AA/AB slot.

Cross-Stage is path-scoped to cpp/** and its own script/workflow; it is not triggered by these four benchmark/CI files and is not claimed as a completed run for this PR. Full Native/Release, observer-off independent ABBA, and the fixed study remain the applicable independent gates. #170 HOLD is unchanged.

Copy link
Copy Markdown
Owner Author

Independent PR-base/head ABBA completed with observer OFF — separate from the fixed study

Exact base d01bb920e030202c33639f59139491ac7f49117c, head 928a5b9e3ce415c1741c441ee7702d4736291ac2. Performance Evidence #553 /34237296101 actually checked out both refs, built both Release binaries, verified instrumentation, measured ABBA and uploaded. Downloaded artifact 10060805409,35,246bytes,SHA256 d2743ff36ae5bc1ff1e1b1ec13c539c79a06903e9bffb63a08647aabc4639796; ZIP CRC and all paired statistics independently checked.

76pairs/19scenarios, AB/BA/AB/BA. All 72 instrumented pairs preserve complete counters, breakdowns and compile/link/archive hit-miss. Four timings-off samples have unavailable counters, not zeros. No InvocationEvidenceDirectory is supplied to this original comparator; its external clock remains external_stopwatch, not the new observed clock. This is validation of #171's inactive observer, NOT the A=d01/B=283b #170 investigation.

Scenario Paired median Δms Paired median Δ% MAD ms Paired nearest-rank P95 ms Slower pairs
cold +69.898 +4.57% 39.646 +837.973 4/4
no-op +0.084 +2.26% 0.185 +0.312 2/4
single-tu −8.880 −1.00% 72.529 +686.536 2/4
public-header +9.276 +9.87% 83.447 +158.148 2/4
build-run +0.387 +2.63% 0.796 +1.814 3/4
link-only +8.478 +13.72% 43.884 +306.901 3/4
target-scale-cold +80.777 +2.13% 51.991 +163.401 3/4
target-scale-no-op +0.728 +2.87% 0.373 +1.914 3/4
target-scale-no-op-auto +0.874 +3.59% 0.594 +48.440 4/4
target-scale-no-op-j1 +0.976 +3.15% 2.619 +5.584 3/4
target-scale-common-header-no-op +1.225 +4.26% 3.531 +6.838 2/4
target-scale-single-tu +31.236 +24.92% 18.280 +53.100 3/4
discovery-cold −68.449 −4.19% 32.523 −3.426 0/4
discovery-no-op +0.060 +0.95% 0.251 +0.466 2/4
discovery-header −7.723 −6.19% 13.337 +9.420 2/4
modules-cold −47.705 −2.80% 104.705 +99.310 2/4
modules-no-op +0.311 +6.81% 0.180 +1.840 4/4
timings-enabled-no-op +0.027 +0.91% 0.074 +0.226 2/4
timings-disabled-no-op (external) −0.747 −6.43% 1.611 +2.286 1/4

Only the last row is timings-off external elapsed. Other rows are instrumented internal totals. With four pairs, nearest-rank P95 is the maximum difference, not a well-estimated population tail. No overlapping internal work is added as elapsed.

The original median gate (worse by BOTH1ms and10%) is not crossed for this exact PR pair. No performance improvement/zero-overhead claim. All four original external pairs are retained:

  1. base55.9592ms /candidate11.3388ms, delta−44.6204ms;
  2. base10.9797ms /candidate13.2659ms, delta+2.2862ms;
  3. base11.7141ms /candidate11.1562ms, delta−0.5579ms;
  4. base11.5726ms /candidate10.6360ms, delta−0.9366ms.

The particularly favorable first difference is not discarded or credited as a speedup. Cold, scale-no-op-auto and modules-no-op are slower4/4; the +837.973ms cold tail and +24.92% scale-single-TU median remain. Causes are not isolated. This original workflow artifact contains comparison JSON, not original stdout/stderr or the actual performance binary pair; the separate new study addresses those missing dimensions prospectively. No same-head rerun requested. #170's previous gate crossing and historical cold-tail debt remain unchanged.

Copy link
Copy Markdown
Owner Author

Fixed AA/AB study complete and independently recomputed — prospective attribution, not historical reconstruction

Observer head 928a5b9e3ce415c1741c441ee7702d4736291ac2. Runtime A=d01bb920e030202c33639f59139491ac7f49117c, B=283b50733601926840fdcd0474d10a06a4f75c7d (#170 held source). This is NOT the separate PR-base/head observer-OFF ABBA#553.

Attribution #2 /34237296093 completed the predeclared budget exactly: 32/32 full reports, 672/672 invocation records including64 primes, 304 measured scenario-pairs. Independent audit verified the persisted schedule against actual slot markers, executable paths, original timestamps/arithmetics, all32report identities and original timing JSON/diagnostics. 288/288 instrumented AA/AB pairs preserve complete counters/breakdowns/compile-link-archive hit-miss; 16 timings-off pairs retain unavailable counters. All19,872captured merged text lines are archived. No sample/slot retry, deletion or AA subtraction.

Artifact 10061149177, 8,123,575bytes /10,060entries, ZIP SHA256 7dba9cc42c7a53bf485eb127172c3bd816b360b503d3983db693b039b524dce9 and CRCs independently verified. The artifact contains both actual measured executables (also duplicated in bootstrap), build transcript, source/build hashes, the exact CRLF observer script, plan, all starts/completions, reports and final source snapshots. Host: Windows10.0.26100, .NET10.0.11, PowerShell7.6.5,4CPUs, image20260824.214.3; Stopwatch frequency10,000,000. The environment record is an allowlist of inherited host build variables, not all credentials or the compiler's fully reconstructed vcvars environment.

Actual executable evidence

Both images are 1,814,016bytes, but whole-file hashes differ:

  • A e3f83dc08fb91cb22684ca21a9d072a9119596e8b4f9f4b938c8b7ccabb9effd
  • B ee550f86e29ab4994265a6a6f2e61504e33d4bd7d085fa0cfb708eac12105783

The source input trees for cpp/src,cpp/include,manifest and VERSION match; both build script hashes match. Independent PE parsing/byte comparison found exactly19 differing bytes: COFF/debug-directory timestamp fields and hex suffixes inside two decorated type-name strings. The complete raw .text sections (1,482,240bytes, including section padding) are identical, SHA256 30fa89e330b9e6d7afa3944ec67f45447a30c00b8842b1fa7bf6e3538c9312a9; .pdata,.fptable,.reloc also match. .rdata/.data contain the differences. Offsets/context and the original images are retained. This is not whole-executable identity or proof of identical runtime cost; different paths/file identity and build metadata remain possible influences. In AA, both sides truly invoke the same A path/bytes. PE field interpretation follows the Microsoft PE format specification, not a guessed timestamp search.

All19 observed-external results, kept separately by contrast

All rows use the NEW observed external interval. Only the final row is timings-OFF; the other18 still enable internal MQB timings. Median percentages are per-pair medians, not ratios of marginal medians. With8pairs nearest-rankP95 is again the maximum, not a stable population-tail estimate.

Scenario AA median Δms / Δ% AA max Δms AA slower AB median Δms / Δ% AB max Δms AB slower
cold −11.076 / −0.67% +750.034 3/8 +11.876 / +0.64% +247.629 6/8
no-op +0.016 / +0.14% +1.740 4/8 −0.087 / −0.72% +2.013 3/8
single-tu +15.928 / +14.66% +1012.908 7/8 +31.906 / +28.39% +383.263 7/8
public-header +11.625 / +9.28% +776.264 5/8 +18.549 / +15.37% +165.666 6/8
build-run −0.423 / −1.77% +3.366 3/8 −0.704 / −3.04% +6.757 3/8
link-only +8.400 / +12.04% +356.124 5/8 +21.528 / +30.06% +287.918 6/8
target-scale-cold +53.069 / +1.25% +1634.220 5/8 +17.641 / +0.43% +958.078 5/8
target-scale-no-op +0.877 / +2.26% +4.620 5/8 +2.039 / +5.40% +8.839 6/8
target-scale-no-op-auto +3.255 / +8.97% +10.211 5/8 +4.028 / +10.25% +27.691 6/8
target-scale-no-op-j1 +0.193 / +0.40% +14.307 5/8 +0.043 / +0.09% +13.774 4/8
target-scale-common-header-no-op +2.127 / +5.29% +7.571 5/8 −0.153 / −0.39% +17.670 4/8
target-scale-single-tu +4.445 / +3.40% +76.251 6/8 +8.285 / +5.47% +138.579 4/8
discovery-cold −14.072 / −0.83% +440.515 4/8 +28.596 / +1.72% +290.143 5/8
discovery-no-op +0.607 / +3.97% +5.626 6/8 +1.056 / +6.74% +12.507 7/8
discovery-header −3.780 / −3.16% +91.279 3/8 +0.603 / +0.52% +198.913 5/8
modules-cold −53.483 / −3.10% +122.327 3/8 −6.714 / −0.34% +393.734 4/8
modules-no-op +0.016 / +0.13% +25.147 4/8 +0.330 / +2.68% +1.371 5/8
timings-enabled-no-op +0.062 / +0.50% +0.949 5/8 −0.175 / −1.39% +15.963 4/8
timings-disabled-no-op +0.1216 / +1.0806% +1.2812 5/8 +0.1088 / +0.9631% +6.5661 5/8

No-op MAD: AA0.35365ms, AB1.05970ms. Neither new observed-no-op median crosses the unchanged >1ms AND>10% investigation threshold. This neither subtracts AA from AB nor clears the historical #170 crossing. In particular, AB block8 has external+6.5661ms: native-and-capture+5.3200ms, cwd push/pop+1.2461ms. Timings are off, so no internal/child CPU explanation is invented for that invocation. The favorable AB block3 difference−6.5523ms is retained too.

Concrete localization gained, not an OS root-cause claim

AA block3 target-scale-cold, same A binary/path: external3882.1961→5516.4160ms (+1634.2199ms). Internal total3872.817→5506.432ms (+1633.615ms); cwd contribution changes only +0.0780ms. Reported compile phase2292.227→3716.105ms and toolchain-discovery wall1479.809→1633.235ms localize most of this run's delay inside MQB's existing measured scopes, not the host directory switch. Parallel compile work is not summed as elapsed.

AA block2 single-tu, same A binary/path: external107.1000→1120.0075ms (+1012.9075ms), internal98.671→1112.142ms. The recorded link phase40.711→846.015ms; nested link_execution37.097→842.340ms. The compile phase55.754→263.732ms co-occurs with compile_cache_write work3.107→213.241ms, while compile_execution work43.901→42.895ms. Source review of CompileCacheFile::save confirms its ScopedCacheWrite encloses serialization, directory/temporary-file creation, write/flush/close and replacement; the label is NOT isolated disk wait or proof of an antivirus cause. Nested/overlapping work is not added to wall totals. These exact records, not only selected summary numbers, are retained.

What this proves: substantial new-run tails occur without any A→B code revision, and some are inside identifiable existing compilation/link/cache-save scopes. What it does not prove: the first #170 no-op crossing or earlier cold tails had that same cause, that every observed AB difference is noise, or that execution paths/metadata/OS effects are harmless. No per-child PID/CPU, ETW/syscall trace or raw native stream bytes were collected. Four observer clock reads and out-of-interval record writes can perturb later samples. Fixed8pairs/contrast and one host are not a population stability study.

The code-generated fields historical_cause_resolved=false and authorizes_merge=false are still accurate. #171 tooling acceptance is a separate decision; #170 stays HOLD pending a targeted follow-up on these narrowed scopes, not another blind rerun of whole-suite medians. Historical raw evidence and cold-tail debt remain open.

@Iviesever Iviesever left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final review — accept attribution tooling; #170 remains on HOLD

Implementation/source self-review with separate read-only artifact and statistical recomputation, not an independent human approval.

Exact base: d01bb920e030202c33639f59139491ac7f49117c.
Reviewed head: 928a5b9e3ce415c1741c441ee7702d4736291ac2.
Tree: af6b293f04521575a2bdbd2fc31bc156b676d0c8.
Native test-merge b5352e1cf12ab43ec43e9e42bdaa93958b0c8cef has that identical tree and exactly those base/head parents. Re-read the complete final four-file diff: 617 additions/19 deletions. No product source/header, manifest, scheduler, native executable inventory, CLI/freshness or VERSION change. The original fixture-generation/scenario-order/cache-assertion body is byte-identical. The observer-off native calls and original Stopwatch interval remain intact.

The four files independently match the reviewed Git blob identities: benchmark 3bf6c2d19581a2acf804874925886199d1736b5b; collector ac3021181f430e6c6c716d13fb5d865cf3ab8714; verifier 59266b91ace9267f409a34aa2931d04dd41e76f1; workflow 2a3334828145feb8a88fbad370274cff7eee480e. Local patch replay/whitespace validation used a minimal base-file Git fixture, not a local full MQB build. This Linux environment has no local MSVC/PowerShell execution; actual Windows evidence follows.

All applicable corrected-head gates completed on their first attempt

  • Native C++ #710 /34237296089: real Debug build, ambient/include/runtime/parameter checks, default-library/include-resolution/__has_include freshness, linker side-output repair, all four paired subshard jobs and final Native C++ gate passed. Not classifier-induced skipping.
  • Native Release #604 /34237296111: stage0, every Release shard, full MQB self-host, runtime-only package and exact packaged-artifact validation passed. publish-release skipped.
  • Documentation #165 /34237296136: passed.
  • Observer contracts /attribution #2 /34237296093: all32 synthetic expectations passed; the complete fixed AA/AB study ran and was archived.
  • Independent normal ABBA #553 /34237296101: actual exact PR base/head checkouts, both Release builds, instrumentation verification, 19x4 measurement with observer OFF and upload passed.

Cross-Stage and ownership-matrix workflows are path-scoped and not triggered by these four benchmark/CI changes. They are not claimed as passed runs. The code under those scopes is unchanged; their absence was documented before final review, not substituted with old PR checks.

Separate evidence sets — do not mix the runtime pairs or clocks

Observer-off exact-PR-base/head ABBA, all19 rows and adverse samples: all76pairs recomputed; all72 instrumented counters/breakdowns/compile-link-archive hit-miss match. Four external counters remain unavailable. External no-op paired median −0.747ms/−6.43%, MAD1.611ms, maximum adverse+2.286ms; unchanged >1ms AND>10% median gate not crossed. The base55.9592ms versus candidate11.3388ms favorable outlier is retained, not claimed as a speedup. Cold, scale-no-op-auto and modules-no-op are slower4/4; cold maximum+837.973ms and scale-single-TU median+24.92% remain unexplained.

Fixed study, full AA/AB table, actual executable comparison and concrete scope localization: runtime A=d01 above, B=held #170 head 283b50733601926840fdcd0474d10a06a4f75c7d, observed by this PR's exact928 head. Completed exactly 32 reports/672 invocation records, including64 primes; 304 scenario-pairs, all288 instrumented semantic identities match and all16 timings-off pairs preserve unavailable counters. Verified persisted versus executed plan, all start/completion identities, clock arithmetic, original timings/outcomes, all19,872 merged text lines and executable provenance. No adaptive sample count, repeated slot, removed pair or AA subtraction.

Observed timings-off no-op medians: AA+0.1216ms/+1.0806%, AB+0.1088ms/+0.9631%. Neither crosses the investigation threshold; this does not explain the historical #170 crossing. AA uses the exact same A path and bytes in both roles. Nevertheless AA block3 scale-cold has external +1634.2199ms, internal +1633.615ms, cwd contribution only +0.0780ms. AA block2 single-TU has external +1012.9075ms, with link phase40.711→846.015ms and cache-save work3.107→213.241ms. Source review confirms cache-save encloses serialization and several file operations, not isolated disk wait. Nested/parallel work is not added as elapsed. These records localize NEW tails without an A→B code revision, not the causes of historical events.

The two actual runtime executables are 1,814,016bytes each, with19 differing bytes. Their full raw .text sections match, but the whole images do NOT: timestamp fields and two decorated-name suffixes differ. Actual binary/path/metadata identity and the original bytes are preserved; code-section equality is not a universal runtime-cost guarantee. No per-child PID/CPU, ETW/syscall record or byte-exact per-stream capture exists. The inherited environment allowlist is not a full compiler vcvars snapshot. Observer clock reads and record writes can perturb later samples; AA does not establish zero observer effect.

Verified raw ZIPs and first failure retained

  • Initial contract artifact 10060163305,12911bytes: 45f0388d38c0838802e10f4cb93b6681c3633f966d5102052097e3946fe1bb28.
  • Corrected contracts 10060400928,13038bytes: 8d5d92c8b2d17063aa31a5a6243d2abf1a0c3da37fbd071868eba9dbc48cfa07.
  • Normal ABBA 10060805409,35246bytes: d2743ff36ae5bc1ff1e1b1ec13c539c79a06903e9bffb63a08647aabc4639796.
  • Fixed study 10061149177,8123575bytes/10060entries: 7dba9cc42c7a53bf485eb127172c3bd816b360b503d3983db693b039b524dce9.

All ZIP hashes and CRCs checked. Initial head0a747 failed a wrongly specified legacy bracket-path compatibility test after25 passing contracts; fixed study never started. Corrected test explicitly preserves legacy wildcard-path refusal and exercises ordinary-path compatibility. No launcher or measurement threshold was changed. Initial Debug#709/Release#603/normalABBA#552 were cancelled by the genuine source correction; ABBA cancellation was during candidate build, before measurement. Those runs are not passing evidence or discarded measured samples. The original partial artifact/log and arithmetic-description correction are retained. No same-head rerun was requested.

Decision and next boundary

Accept ordinary merge with expected head 928a5b9e3ce415c1741c441ee7702d4736291ac2, under current branch protection. Acceptance is for the opt-in observer, reproducible fixed study and improved diagnostic evidence. It is NOT a speedup claim, zero-overhead proof, historical root-cause resolution, cancellation integration or lease authorization. The predeclared thresholds are unchanged; all adverse/favorable data remain.

Keep #170 open Draft/HOLD. A lower new median and AA tails do not reconstruct its first +1.6946ms/+13.7988% crossing or explain every AB effect. Next focused investigation should separate cache persistence subspans (serialization, directory/open/write/flush/close/replace) and credible link/process launch/wait/capture observations under a declared fixed budget, rather than repeating whole-suite medians until green. Do not guess antivirus/disk/scheduler causality from a scope name. Historical missing binaries/traces and cold-tail debt remain explicit limitations.

VERSION5.5.0, historical tags/assets and formal release remain untouched. Retitle only after completed #553/#2 evidence; any later skipped perf job is not a replacement. Actions retain artifacts30days, with raw copies and runnable read-only audits additionally preserved in the delivery.

@Iviesever Iviesever changed the title perf: retain benchmark invocation evidence and fixed AA calibration test: retain benchmark invocation evidence and fixed AA calibration Sep 8, 2026
@Iviesever
Iviesever marked this pull request as ready for review September 8, 2026 14:49
@Iviesever
Iviesever merged commit c8c8c01 into main Sep 8, 2026
20 checks passed
@Iviesever
Iviesever deleted the codex/v6-benchmark-attribution branch September 8, 2026 14:49
Iviesever added a commit that referenced this pull request Sep 9, 2026
…ness

Accept the event-backed readiness iteration after exact-tree Native Debug #718, full Release/self-host/package #610, Documentation #171, Cross-Stage #195, 36 synthetic contracts, actual withheld-provider refusal and four complete calibrated MSVC traces, unchanged default56/private42, and independent ABBA #566. Require actual same-session event delivery and single opening calibration before compiler admission; rematch witnesses against authoritative ETL records. Preserve original C2398/old incomplete traces, C1041 controls, managed-service failures and all adverse performance samples. No same-head retry or weakened gates. Keep #170/#172 held; no M1b completion, CLI cancellation, external writer quiescence or writer-lease authorization. VERSION remains5.5.0; no historical tag/asset or formal release change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant