test: retain benchmark invocation evidence and fixed AA calibration - #171
Conversation
First-head Windows contract failure retained; narrow test correctionHead Downloaded original artifact Correction is confined to the test script: explicitly assert the legacy bracket-path limitation remains an expected refusal, then run inactive-observer clock/counter compatibility on a plain path with spaces. Do NOT change the historical benchmark launcher to conceal the test mismatch. Observed literal-path coverage stays in place. The fixed study budget, source pair, thresholds, observer implementation and fixtures are unchanged; new head requires its own tests and exact-base/head ABBA. No same-slot retry or same-head rerun. Metadata correction: initial full diff is 608 additions/19 deletions, not 586 as transcribed in the initial PR body (138+102+244+124 = 608). This is an arithmetic description correction, not a changed scope. |
Corrected-head contract artifact independently verifiedHead Read all 32/32 passing contract records under PowerShell7.6.5 and the underlying invocation files: empty output remains an empty array; exit7 retains both original stdout/stderr text lines; all2400 volume-test lines survive; prelaunch cwd failure retains observer_error with exit_code null; malformed timing output is saved before parser rejection. The legacy bracket-path limitation is explicitly checked as refusal, while its ordinary-path compatibility check passes. No synthetic invocation is counted as MSVC benchmark evidence. Independent static comparison confirms the entire fixture generation, scenario invocation sequence and cache-count assertion body is byte-identical to base. Candidate content Git blobs match reviewed local files: benchmark Initial head's Native Debug#709,Release#603 and ordinary ABBA#552 were cancelled by this genuine test-source correction, not counted as passing validation; Documentation passed. Initial attribution failed before the fixed investigation was admitted. Its partial25-check artifact and log remain. No same-head retry or discarded AA/AB slot. Cross-Stage is path-scoped to cpp/** and its own script/workflow; it is not triggered by these four benchmark/CI files and is not claimed as a completed run for this PR. Full Native/Release, observer-off independent ABBA, and the fixed study remain the applicable independent gates. #170 HOLD is unchanged. |
Independent PR-base/head ABBA completed with observer OFF — separate from the fixed studyExact base 76pairs/19scenarios, AB/BA/AB/BA. All 72 instrumented pairs preserve complete counters, breakdowns and compile/link/archive hit-miss. Four timings-off samples have unavailable counters, not zeros. No InvocationEvidenceDirectory is supplied to this original comparator; its external clock remains
Only the last row is timings-off external elapsed. Other rows are instrumented internal totals. With four pairs, nearest-rank P95 is the maximum difference, not a well-estimated population tail. No overlapping internal work is added as elapsed. The original median gate (worse by BOTH1ms and10%) is not crossed for this exact PR pair. No performance improvement/zero-overhead claim. All four original external pairs are retained:
The particularly favorable first difference is not discarded or credited as a speedup. Cold, scale-no-op-auto and modules-no-op are slower4/4; the +837.973ms cold tail and +24.92% scale-single-TU median remain. Causes are not isolated. This original workflow artifact contains comparison JSON, not original stdout/stderr or the actual performance binary pair; the separate new study addresses those missing dimensions prospectively. No same-head rerun requested. #170's previous gate crossing and historical cold-tail debt remain unchanged. |
Fixed AA/AB study complete and independently recomputed — prospective attribution, not historical reconstructionObserver head Attribution #2 /34237296093 completed the predeclared budget exactly: 32/32 full reports, 672/672 invocation records including64 primes, 304 measured scenario-pairs. Independent audit verified the persisted schedule against actual slot markers, executable paths, original timestamps/arithmetics, all32report identities and original timing JSON/diagnostics. 288/288 instrumented AA/AB pairs preserve complete counters/breakdowns/compile-link-archive hit-miss; 16 timings-off pairs retain unavailable counters. All19,872captured merged text lines are archived. No sample/slot retry, deletion or AA subtraction. Artifact 10061149177, 8,123,575bytes /10,060entries, ZIP SHA256 Actual executable evidenceBoth images are 1,814,016bytes, but whole-file hashes differ:
The source input trees for cpp/src,cpp/include,manifest and VERSION match; both build script hashes match. Independent PE parsing/byte comparison found exactly19 differing bytes: COFF/debug-directory timestamp fields and hex suffixes inside two decorated type-name strings. The complete raw All19 observed-external results, kept separately by contrastAll rows use the NEW observed external interval. Only the final row is timings-OFF; the other18 still enable internal MQB timings. Median percentages are per-pair medians, not ratios of marginal medians. With8pairs nearest-rankP95 is again the maximum, not a stable population-tail estimate.
No-op MAD: AA0.35365ms, AB1.05970ms. Neither new observed-no-op median crosses the unchanged >1ms AND>10% investigation threshold. This neither subtracts AA from AB nor clears the historical #170 crossing. In particular, AB block8 has external+6.5661ms: native-and-capture+5.3200ms, cwd push/pop+1.2461ms. Timings are off, so no internal/child CPU explanation is invented for that invocation. The favorable AB block3 difference−6.5523ms is retained too. Concrete localization gained, not an OS root-cause claimAA block3 target-scale-cold, same A binary/path: external3882.1961→5516.4160ms (+1634.2199ms). Internal total3872.817→5506.432ms (+1633.615ms); cwd contribution changes only +0.0780ms. Reported compile phase2292.227→3716.105ms and toolchain-discovery wall1479.809→1633.235ms localize most of this run's delay inside MQB's existing measured scopes, not the host directory switch. Parallel compile work is not summed as elapsed. AA block2 single-tu, same A binary/path: external107.1000→1120.0075ms (+1012.9075ms), internal98.671→1112.142ms. The recorded link phase40.711→846.015ms; nested link_execution37.097→842.340ms. The compile phase55.754→263.732ms co-occurs with compile_cache_write work3.107→213.241ms, while compile_execution work43.901→42.895ms. Source review of What this proves: substantial new-run tails occur without any A→B code revision, and some are inside identifiable existing compilation/link/cache-save scopes. What it does not prove: the first #170 no-op crossing or earlier cold tails had that same cause, that every observed AB difference is noise, or that execution paths/metadata/OS effects are harmless. No per-child PID/CPU, ETW/syscall trace or raw native stream bytes were collected. Four observer clock reads and out-of-interval record writes can perturb later samples. Fixed8pairs/contrast and one host are not a population stability study. The code-generated fields |
Iviesever
left a comment
There was a problem hiding this comment.
Final review — accept attribution tooling; #170 remains on HOLD
Implementation/source self-review with separate read-only artifact and statistical recomputation, not an independent human approval.
Exact base: d01bb920e030202c33639f59139491ac7f49117c.
Reviewed head: 928a5b9e3ce415c1741c441ee7702d4736291ac2.
Tree: af6b293f04521575a2bdbd2fc31bc156b676d0c8.
Native test-merge b5352e1cf12ab43ec43e9e42bdaa93958b0c8cef has that identical tree and exactly those base/head parents. Re-read the complete final four-file diff: 617 additions/19 deletions. No product source/header, manifest, scheduler, native executable inventory, CLI/freshness or VERSION change. The original fixture-generation/scenario-order/cache-assertion body is byte-identical. The observer-off native calls and original Stopwatch interval remain intact.
The four files independently match the reviewed Git blob identities: benchmark 3bf6c2d19581a2acf804874925886199d1736b5b; collector ac3021181f430e6c6c716d13fb5d865cf3ab8714; verifier 59266b91ace9267f409a34aa2931d04dd41e76f1; workflow 2a3334828145feb8a88fbad370274cff7eee480e. Local patch replay/whitespace validation used a minimal base-file Git fixture, not a local full MQB build. This Linux environment has no local MSVC/PowerShell execution; actual Windows evidence follows.
All applicable corrected-head gates completed on their first attempt
- Native C++ #710 /34237296089: real Debug build, ambient/include/runtime/parameter checks, default-library/include-resolution/__has_include freshness, linker side-output repair, all four paired subshard jobs and final Native C++ gate passed. Not classifier-induced skipping.
- Native Release #604 /34237296111: stage0, every Release shard, full MQB self-host, runtime-only package and exact packaged-artifact validation passed. publish-release skipped.
- Documentation #165 /34237296136: passed.
- Observer contracts /attribution #2 /34237296093: all32 synthetic expectations passed; the complete fixed AA/AB study ran and was archived.
- Independent normal ABBA #553 /34237296101: actual exact PR base/head checkouts, both Release builds, instrumentation verification, 19x4 measurement with observer OFF and upload passed.
Cross-Stage and ownership-matrix workflows are path-scoped and not triggered by these four benchmark/CI changes. They are not claimed as passed runs. The code under those scopes is unchanged; their absence was documented before final review, not substituted with old PR checks.
Separate evidence sets — do not mix the runtime pairs or clocks
Observer-off exact-PR-base/head ABBA, all19 rows and adverse samples: all76pairs recomputed; all72 instrumented counters/breakdowns/compile-link-archive hit-miss match. Four external counters remain unavailable. External no-op paired median −0.747ms/−6.43%, MAD1.611ms, maximum adverse+2.286ms; unchanged >1ms AND>10% median gate not crossed. The base55.9592ms versus candidate11.3388ms favorable outlier is retained, not claimed as a speedup. Cold, scale-no-op-auto and modules-no-op are slower4/4; cold maximum+837.973ms and scale-single-TU median+24.92% remain unexplained.
Fixed study, full AA/AB table, actual executable comparison and concrete scope localization: runtime A=d01 above, B=held #170 head 283b50733601926840fdcd0474d10a06a4f75c7d, observed by this PR's exact928 head. Completed exactly 32 reports/672 invocation records, including64 primes; 304 scenario-pairs, all288 instrumented semantic identities match and all16 timings-off pairs preserve unavailable counters. Verified persisted versus executed plan, all start/completion identities, clock arithmetic, original timings/outcomes, all19,872 merged text lines and executable provenance. No adaptive sample count, repeated slot, removed pair or AA subtraction.
Observed timings-off no-op medians: AA+0.1216ms/+1.0806%, AB+0.1088ms/+0.9631%. Neither crosses the investigation threshold; this does not explain the historical #170 crossing. AA uses the exact same A path and bytes in both roles. Nevertheless AA block3 scale-cold has external +1634.2199ms, internal +1633.615ms, cwd contribution only +0.0780ms. AA block2 single-TU has external +1012.9075ms, with link phase40.711→846.015ms and cache-save work3.107→213.241ms. Source review confirms cache-save encloses serialization and several file operations, not isolated disk wait. Nested/parallel work is not added as elapsed. These records localize NEW tails without an A→B code revision, not the causes of historical events.
The two actual runtime executables are 1,814,016bytes each, with19 differing bytes. Their full raw .text sections match, but the whole images do NOT: timestamp fields and two decorated-name suffixes differ. Actual binary/path/metadata identity and the original bytes are preserved; code-section equality is not a universal runtime-cost guarantee. No per-child PID/CPU, ETW/syscall record or byte-exact per-stream capture exists. The inherited environment allowlist is not a full compiler vcvars snapshot. Observer clock reads and record writes can perturb later samples; AA does not establish zero observer effect.
Verified raw ZIPs and first failure retained
- Initial contract artifact 10060163305,12911bytes:
45f0388d38c0838802e10f4cb93b6681c3633f966d5102052097e3946fe1bb28. - Corrected contracts 10060400928,13038bytes:
8d5d92c8b2d17063aa31a5a6243d2abf1a0c3da37fbd071868eba9dbc48cfa07. - Normal ABBA 10060805409,35246bytes:
d2743ff36ae5bc1ff1e1b1ec13c539c79a06903e9bffb63a08647aabc4639796. - Fixed study 10061149177,8123575bytes/10060entries:
7dba9cc42c7a53bf485eb127172c3bd816b360b503d3983db693b039b524dce9.
All ZIP hashes and CRCs checked. Initial head0a747 failed a wrongly specified legacy bracket-path compatibility test after25 passing contracts; fixed study never started. Corrected test explicitly preserves legacy wildcard-path refusal and exercises ordinary-path compatibility. No launcher or measurement threshold was changed. Initial Debug#709/Release#603/normalABBA#552 were cancelled by the genuine source correction; ABBA cancellation was during candidate build, before measurement. Those runs are not passing evidence or discarded measured samples. The original partial artifact/log and arithmetic-description correction are retained. No same-head rerun was requested.
Decision and next boundary
Accept ordinary merge with expected head 928a5b9e3ce415c1741c441ee7702d4736291ac2, under current branch protection. Acceptance is for the opt-in observer, reproducible fixed study and improved diagnostic evidence. It is NOT a speedup claim, zero-overhead proof, historical root-cause resolution, cancellation integration or lease authorization. The predeclared thresholds are unchanged; all adverse/favorable data remain.
Keep #170 open Draft/HOLD. A lower new median and AA tails do not reconstruct its first +1.6946ms/+13.7988% crossing or explain every AB effect. Next focused investigation should separate cache persistence subspans (serialization, directory/open/write/flush/close/replace) and credible link/process launch/wait/capture observations under a declared fixed budget, rather than repeating whole-suite medians until green. Do not guess antivirus/disk/scheduler causality from a scope name. Historical missing binaries/traces and cold-tail debt remain explicit limitations.
VERSION5.5.0, historical tags/assets and formal release remain untouched. Retitle only after completed #553/#2 evidence; any later skipped perf job is not a replacement. Actions retain artifacts30days, with raw copies and runnable read-only audits additionally preserved in the delivery.
…ness Accept the event-backed readiness iteration after exact-tree Native Debug #718, full Release/self-host/package #610, Documentation #171, Cross-Stage #195, 36 synthetic contracts, actual withheld-provider refusal and four complete calibrated MSVC traces, unchanged default56/private42, and independent ABBA #566. Require actual same-session event delivery and single opening calibration before compiler admission; rematch witnesses against authoritative ETL records. Preserve original C2398/old incomplete traces, C1041 controls, managed-service failures and all adverse performance samples. No same-head retry or weakened gates. Keep #170/#172 held; no M1b completion, CLI cancellation, external writer quiescence or writer-lease authorization. VERSION remains5.5.0; no historical tag/asset or formal release change.
Final decision: accept attribution tooling; keep #170 on HOLD
Final source/execution/artifact review and acceptance decision
d01bb920e030202c33639f59139491ac7f49117c.928a5b9e3ce415c1741c441ee7702d4736291ac2.af6b293f04521575a2bdbd2fc31bc156b676d0c8.b5352e1cf12ab43ec43e9e42bdaa93958b0c8cefhas exactly those base/head parents.Continue #164's existing route with one attribution topic. #170's runtime source is not modified by this PR. Tooling acceptance does not resolve historical causality or authorize #170 merge.
Delivered behavior
The optional
InvocationEvidenceDirectoryinbenchmark_mqb.ps1retains started markers, full captured merged TEXT lines, exit/error status and four monotonic invocation boundaries before parsing timing results. It refuses overwritten attempts. Cwd/native-and-capture/pop spans are distinct; native child PID remains unavailable rather than being invented from the host PID. Final source copies are explicitly post-suite snapshots.With the observer OFF, original native launches and Stopwatch boundary are preserved. The entire original fixture generation, scenario sequence and cache-count assertions are byte-identical. The new observed clock is labeled
external_stopwatch_observed, not falsely merged with the old external clock.The collector archives actual executable bytes/hashes, build/source provenance, exact observer script, allowlisted host environment, plan and each invocation/report. Validation recomputes raw clock arithmetic, original timing/exit correspondence and complete semantic counters. Failures retain the attempted/completed boundary and abort, with no repeated slot or AA subtraction.
Two independent experiments — keep source pairs and clocks separate
Normal PR validation: observer OFF
Full19-row exact-PR-base/head ABBA and adverse samples
Performance Evidence #553/34237296101 built d01 versus928 and measured19x4 AB/BA/AB/BA. All76pairs independently recomputed; all72instrumented counters/breakdowns/hit-miss identities match; four timings-off counter sets remain unavailable. External no-op paired median −0.747ms/−6.43%, MAD1.611ms, max adverse+2.286ms; the original >1ms AND>10% median gate was not crossed. Base55.9592ms versus candidate11.3388ms is a retained favorable extreme, not a speedup claim. Cold, scale-no-op-auto and modules-no-op are slower4/4; all other adverse and favorable results remain.
Fixed attribution investigation: observer ON
FullAA/AB table, actual binaries and localized tails
The advance declaration fixed runtime A=d01, B=held#170 head283b50733601926840fdcd0474d10a06a4f75c7d, observed by928. Eight AA pairs plus eight AB pairs =32 complete reports/672 recorded invocations (64primes),304scenario-pairs. All288instrumented semantic pairs match;16timings-off pairs preserve unavailable counters. All19,872merged output lines, plan/order, timestamps, original JSON/results and exact binaries were independently checked. AA uses the SAME A path and bytes in both roles; contrast order/orientation alternate. No adaptive budget, outlier deletion, retry or AA subtraction.
Observed timings-off no-op: AA+0.1216ms/+1.0806%, AB+0.1088ms/+0.9631%. Neither new median crosses the declared investigation threshold. Yet same-binary AA block3 scale-cold has external+1634.2199ms/internal+1633.615ms, with only+0.0780ms cwd change. AA block2 single-TU has external+1012.9075ms, coinciding with link and cache-save scopes. These are NEW-run localization facts, not proof of the cause of #170's earlier no-op crossing.
Both actual images are1,814,016bytes;19bytes differ in timestamp/decorated-name fields. Raw .text is identical but whole executables are not. Different paths/metadata, observer effects and unrecorded OS effects remain. Cache-save work includes serialization and several file operations, not just disk wait. Nested/parallel work is never added as elapsed. All18 non-no-op-off scenarios still enable internal instrumentation; only timings-disabled-no-op is timings-OFF. No per-child CPU/PID,ETW/syscall trace or byte-exact per-stream capture is claimed.
Completed applicable validation
Native Debug #710/34237296089 actually built and executed all subshards plus freshness/runtime/parameter checks. Native Release #604/34237296111 completed all shards, full self-host and exact runtime-only package validation; publish-release skipped. Documentation #165/34237296136 passed. Attribution #2/34237296093 completed32synthetic helper contracts and the full fixed study. NormalABBA#553 actually measured, not skipped. Cross-Stage/ownership workflows are path-scoped and not triggered by these files; they are not claimed as passed runs.
32-case contract artifact and source/clock compatibility review
Initial0a747 head failed the mistakenly specified legacy bracket-path test after25passing contracts; the fixed study did not start. The correction preserves that legacy limitation as an expected refusal, then tests compatibility on a plain path. Initial Debug#709/Release#603/ABBA#552 were cancelled by the genuine test-source update; ABBA had not reached measurement. The partial artifact/log and initial diff-count typo correction are retained. No same-head rerun or relabeling failed validation as passed.
Original acceptance gates retained
Exact final diff/tree and applicable complete native/release/package/documentation; actual helper tests with valid and rejected cases; unchanged fixture bodies; independent observer-off19x4ABBA with72semantic identities and the unchanged both>1ms and>10% external no-op median gate. Fixed study must complete32reports/672records with288instrumented identities,16offpairs,actual archived binaries,original diagnostics and all adverse samples. Evaluate AA/AB separately using the same investigation threshold. No risk-adjusted subtraction, automatic#170approval or historical reconstruction is implied by collection success.
All applicable tooling gates passed. Accept ordinary expected-head merge under branch protection. Keep #170 Draft/HOLD. Next narrow the cache-save and link/process execution scopes with credible subspan/tool/OS evidence under a declared fixed budget, rather than blindly repeating medians. Historical missing binaries/traces and cold-tail debt remain; no AV/disk/root-cause, performance gain or zero-overhead claim. CLI cancellation, external-writer quiescence and writer leases remain unauthorized.
Verified raw artifacts: initial contracts10060163305,corrected contracts10060400928,normalABBA10060805409,fixed study10061149177. Exact SHA256/CRC checks and runnable read-only recomputation are retained in the review/delivery; Actions retention30days. Title changed only after actual#553/#2 completed; later skipped perf jobs do not replace measured evidence. Final VERSION/release work stays separate.