Skip to content

perf: validate cumulative foreground failure-repair compatibility - #186

Merged
Iviesever merged 2 commits into
mainfrom
codex/v5.6-foreground-failure-recovery
Sep 13, 2026
Merged

Iviesever merged 2 commits into
mainfrom
codex/v5.6-foreground-failure-recovery

Conversation

@Iviesever

Copy link
Copy Markdown
Owner

Test-only slice of the frozen 5.6 cumulative regression

Follows #164 after #185. The perf: prefix runs the existing independent ABBA workflow; no optimization/zero-overhead claim. Draft until substantive acceptance.

Base b7e322cc3aa579d3f67135e42d9c9a7c01ce5027; head 9d39bec18e4a6fb39d9810698e87ecb5294b47b7; tree 848a562f390f924f5adddf0df3c04a0e08e392ac.

Only two added files, 356 lines: tests/native/verify_foreground_recovery.py and .github/workflows/foreground-recovery.yml. Product code, original 79-test inventory, freshness rules, existing benchmark/ABBA scripts, packaging and VERSION=5.5.0 unchanged. No historical tag/asset changes.

Actual gap

Existing ordinary verify_foreground_handoff tests cold/warm launch, compile failure and link failure, then ends. Existing module tests cover warm handoff and provider/IFC invalidation. Add the combined failure -> caller repairs source -> new generation executes -> subsequent no-op sequence for ordinary and named-module pipelines in Debug and Release. This is normal subsequent CLI use, NOT automatic recovery/cancellation/transactions.

Fixed eight phases per mode/configuration: cold, source-first warm, intentional compile failure, compile repaired, repaired warm, intentional link failure, link repaired, repaired warm. Full matrix: 32 MQB invocations. Child verifies stdin, inherited environment, empty/spaced argv and relative event file in a Unicode/spaced cwd. Exact append-only event history, generation-specific stdout/stderr and child exits37/41/43 distinguish stale launches. Failures require exit4/staticassert or exit5/unresolved-symbol diagnostics without old-artifact handoff/event. Repairs compile/link; warm runs neither compile/link nor change object/IFC/executable hashes/mtimes. Mutations explicitly advance source mtimes, not a same-timestamp test.

Evidence contract

Retain source/head/tree/base, actual candidate/seed identities, build transcript, every argv/cwd/input, raw stdout/stderr bytes, source generations, cache/deps bytes, executable generations and object/IFC identities. PDB/live project build trees excluded from upload: no PDB/RPC/OS quiescence or complete-write-set claim. No retry, output overwrite or cache deletion between phases. Unexpected failure stops its dependent sequence and preserves started/completed/error/not-run states. Independent mode/matrix continue; upload runs even on failure. Repeated Actions attempts cannot replace original evidence.

Predeclared acceptance gates

  1. Exact base/head/test-merge identity and only these two test files changed.
  2. Full Windows Debug/Release, all79 tests/eight subshards, original freshness/compatibility gates, pinned-seed Stage0/1/2 self-host and actual runtime-only package/installer lifecycle.
  3. Both new original configuration runs: 16/16 invocations each, four complete sequences, no unexpected failure/error/not-run. Original expected compiler/linker failures retained. Collector self-test has3 positive controls and16 rejecting mutations. Inspect actual artifacts, not only green summaries.
  4. Original independent observer-OFF19x4ABBA on exact base/head; all76pairs,72equal complete instrumented counters/breakdowns/compile-link-archive hit/miss vectors. External counters unavailable, never fabricated zero. Original external no-op paired-median blocking gate remains BOTH >+1ms AND >+10%. Keep every adverse/favorable sample, no AA subtraction, outlier deletion or retry-to-green. This is NOT released v5.5.0 versus the complete candidate cumulative performance test.
  5. Missing/failed required evidence means HOLD.

Local Linux checks: Python compilation/assertion-only selftest/YAML parse passed; native platform guard rejects Linux. A local YAML audit first queried nonexistent key regression (KeyError), then corrected itself to actual recover; workflow unchanged and audit failure retained. No local Windows/MSVC/PowerShell execution claimed.

Boundaries and historical blockers

5.6 feature freeze and #164's6.0 roadmap remain. No default CLI cancellation/write lock/lease transfer, full transaction, automatic recovery, session or residency. #172 HOLD. Historical #172/#176C1041, #170 +1.6946ms/+13.7988%, #180 Performance#599 +2.07175ms/+16.2700%, common129 +5.354s/+6.480s remain release-blocking until evidence-backed disposition. VERSION/release changes require later explicit authorization.

PR creation first returned a connector ReadTimeout. Exact-branch all-state PR lookup and latest-created PR lookup both confirmed no PR existed before this retry; no competing PR is intended.

…bility

Add a fixed eight-phase ordinary/named-module CLI sequence in both Debug and Release. Preserve original diagnostics, cache generations, executable bytes and artifact identities without retries or cache cleanup. No product, native inventory, benchmark, VERSION or release changes.

Copy link
Copy Markdown
Owner Author

Original-head acceptance: HOLD; deterministic fixture-precondition failure retained

Head 9d39bec18e4a6fb39d9810698e87ecb5294b47b7, original Foreground Failure Recovery run 34748925471 attempt1. Downloaded and checked both artifact ZIP hashes:

  • Debug artifact10314419011: a7cfeb1e26dbb5d769ceaaf9851787d196bda8042616b5fed8feefac7a7d5312.
  • Release artifact10315451394: 9f953fbdf845f5699a108a446ca8ebcce7c5254b61b0a1caef8010264b70feb8.

20 actual invocations:16 expected outcomes,4 unexpected warm failures;12 dependent link phases NOT RUN. Every ordinary/module/configuration sequence passed cold, initial warm, intentional compile failure and compile repair, then compile-repaired-warm rebuilt instead of no-op. Do not credit the unexecuted link phases or call the matrix accepted.

Cause grounded in unchanged code and raw timestamps

The newly written fixture used os.utime(old_time + 2 seconds) at every mutation. At compile repair the recorded source was newer than the freshly produced object by Debug ordinary1484.4625ms, Debug module1049.9711ms, Release ordinary1882.8748ms, Release module1624.6630ms. Both raw warm logs explicitly report [source changed]; actual repaired child exit41 and event history remain correct.

CompileCacheValidator::validate intentionally invalidates when the source timestamp is newer than the oldest output. This exact implementation already exists in released v5.5.0 (08cdc20a..., CompileCache.cpp blob 3d5dfc3812476b13a8b33567f19431451fa3cc2e) and existing native tests assert it. This is my fixture's invalid no-op precondition, not evidence that the candidate regressed and not justification to weaken freshness.

Bounded source correction before a new head, not same-head retry

Only repair the fixture: use its real filesystem write time, record a mutation.json, and fail before launching if the write did not advance the source, is future-dated, or is not newer than existing target artifacts. No sleep-until-green, synthetic timestamp, retry, cache cleanup, product change or relaxed warm assertion. Add two positive timestamp controls and three rejecting mutations (total5positive/19rejecting assertion-only controls). Original evidence stays immutable and this original head stays rejected for this test.

Also make git archive explicitly core.autocrlf=false. The original Windows archive used CRLF conversion; read-only LF reconstruction except the explicitly binary install.bat reproduces the exact expected complete tree 848a562f390f924f5adddf0df3c04a0e08e392ac (436files). Raw ZIPs are unchanged. Initial local archive tree-audit mismatch and its line-ending diagnosis are retained.

Do not push over/cancel the still-running original native gates. The original independent ABBA#624 has completed and will be audited in full; any adverse outcome remains evidence. A corrected source head needs its own first Debug/Release, original32-invocation sequence, full native/self-host/package and independent unchanged19x4ABBA. No previous result is relabeled as that new head. All historical #164 release blockers and #172 HOLD remain.

…nce failure

Preserve original run 34748925471:20 invocations,4 failed warm assertions,12 not-run phases. The fixture artificially future-dated sources, conflicting with unchanged v5.5 freshness. Use and record actual write timestamps with stricter pre-invocation checks; keep original warm/diagnostic/event gates. No sleep, retry, cache cleanup or product change. Archive canonical Git source without Windows autocrlf conversion.

Copy link
Copy Markdown
Owner Author

Corrected candidate frozen for first-head validation

New head a876eba9c59e7ddf5abf23bc12fa2d86e2bbc56c, tree 280326bc1ce6bd46b962311d57e02b55f628af44, same base b7e322cc3aa579d3f67135e42d9c9a7c01ce5027. Two added test/workflow files total387lines; no product/native-inventory/benchmark/VERSION changes. Correction details and immutable original failure: comment5652391849 above. Local collector controls now5positive/19rejecting and Python compilation pass; not native evidence.

The original five workflows were allowed to finish before this fast-forward: native Debug#753, Release/self-host/package#632, Documentation#196 and ABBA#624 successful; foreground matrix#1 failed. None was canceled or rerun. ABBA#624's76pairs and72complete equal vectors were recomputed; external paired median -0.26895ms/-2.42079298%, adverse pairs +0.4125ms/+4.4941984% and +0.1032ms/+1.1555516% retained, as are all other18scenarios. Its largest paired delta is modules-cold +97.510ms; no sample deletion or zero-cost claim.

This new source head needs its own first32-invocation matrix, complete Debug/Release/self-host/package and unchanged independent19x4ABBA under exactly the previously declared thresholds. Original head remains unaccepted; the old green workflows cannot accept this head. PR description's original-head section is historical until final decision; use this comment for the current exact candidate. Keep Draft/HOLD while acceptance is incomplete. #172 and all historical #164 release blockers remain.

@Iviesever Iviesever left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

实质验收决定:ACCEPT 此测试增量;不放行 5.6.0 发布

验收对象为当前已提交的 #186,不创建竞争 PR。本轮 Windows 结论来自实际 Actions 执行日志与下载产物的逐项复核;本地 Linux 只运行只读证据审计和 collector self-test,没有冒称本地 Windows/MSVC 测试,也未 rerun/dispatch 原流水线。

1. 精确身份与边界

  • base / 当前 main:b7e322cc3aa579d3f67135e42d9c9a7c01ce5027,tree 4f5594da127e9276e65488ee0e5f0be37c2e1920
  • 接纳 head:a876eba9c59e7ddf5abf23bc12fa2d86e2bbc56c,tree 280326bc1ce6bd46b962311d57e02b55f628af44
  • 已执行 native/release CI test-merge:4c45f7f4d8cbd3bbef3708a8ac7433765f851bb9;Git commit API 核实两 parent 为上述 base/head,tree 与 head 相同。
  • Debug/Release 两份 source.zip 的 436 个文件独立重建 Git tree 均精确等于 head tree;排除两份新增测试文件后完整 tree 精确等于 base tree。只有 .github/workflows/foreground-recovery.ymltests/native/verify_foreground_recovery.py,+387/-0。产品源代码、原生79测试清单、freshness、既有性能脚本/门槛、打包链、VERSION 均未改。
  • 本 PR 正文保留首版 head/356行的历史快照;当前验收冻结身份以此 review 和修正版 comment 5652424919 为准,不把首版与修正版混为一份证据。

2. 首版失败完整保留,修正没有削弱产品规则

首版 9d39bec18e4a6fb39d9810698e87ecb5294b47b7 的 Foreground Failure Recovery #1 / run34748925471 attempt1 仍为 FAILED。已下载并核验 Debug artifact10314419011、Release10315451394:实际20次调用,16次符合各自预期、4次 unexpected repaired-warm rebuild、12个依赖阶段未执行;后者绝不计为通过。

四个失败点的源码比刚生成的 main.obj 还新 1484.4625 / 1049.9711 / 1882.8748 / 1624.6630 ms,原始 stdout 明确 [source changed],与夹具 os.utime(old+2s) 制造未来时间戳一致。修正版改为正常实际写入、读取真实时间戳,并 fail-closed 核对写入确实晚于旧源码/旧产物且不在未来;没有更改 CompileCache freshness、放宽 warm 断言、删除阶段间缓存、重试或用未来时间戳继续制造通过。

首版 ABBA #624 / run34748925470 artifact10314484070 也原样保存,不能替代新 head 的 #625。

3. 新的四条实际完整序列

Foreground Failure Recovery #2 / run34749449057:Debug artifact10315497118、Release10315442214;两配置 attempt1、每配置16次、合计32次实际开始并结束,四条序列全部8阶段完成,无 unexpected failure / not-run。

普通目标与 named-module 两类各覆盖:cold → source-first warm → intentional compile failure → caller source repair → repaired warm → intentional link failure → caller source repair → repaired warm。

独立复核32份 invocation/observation 与原始 stdout.bin/stderr.bin、输入源码 SHA、每次事件追加历史、前后 exe/obj/IFC SHA/大小/mtime、保存的 exe 代际和 cache/deps 字节:

  • 8次预期失败保留完整 static_assert 或 unresolved-symbol/LNK2019/LNK2001 诊断,MQB exit4/5,无 [run]、无旧程序输出/事件追加。
  • 24次成功 handoff 对应原始/编译修复/链接修复三种真实不同 exe,child exit37/41/43;stdin、继承环境、空参数、含空格参数、Unicode/空格 cwd 均核对。
  • 12次 warm 无 compile/link,exe/obj/IFC 哈希、大小、mtime 不变;20次真实源码变更的 freshness 前置条件均核对。
  • Collector self-test 为5 positive /19 rejecting controls,Linux 本地通过,仅作为采集器逻辑检查。

这是调用者修复后的正常下一次 CLI 构建,不是自动恢复。没有证明 PDB/RPC/OS 静止、same-timestamp 编辑安全或 complete write set;没有新增 CLI cancel/write-lock/lease-transfer 权限。

4. 原生、自举、包的实际执行

当前 head 的 Native C++ #754 /34749449113、Native Release #633 /34749449077、Documentation #197 /34749449060 均 completed success。

已逐一读取 Debug 四个和 Release 四个 native 外层 job 的完整日志。每配置的八子分片实际完成数为 8+9+9+9+10+12+11+11 =79/79,没有用总绿灯替代分片执行。既有 prewrite44 / recipe28 / target-admission84、写域39 / mapping33,以及原 CLI、模块、PCH、cache/default-library 等 native 程序仍执行。所有原有警告和诊断保留。

Release自举/打包 job103704125799:pinned seed → Stage0 → Stage1 → clean Stage2,两个阶段实际各83 production TUs;installer install/reinstall/uninstall/PATH ownership 通过。

已另下载 package artifact10315675549,核验7个运行时文件、无多余开发产物、VERSION5.5.0、inner ZIP SHA cfbf10ad6e8c79a9c74c05669fd751cbe9c9ee0fec35f44935b0c69644f80761 与 sidecar一致;包内 mqb.exe SHA ac4a7de6b032f943eaa61456c722a2f2832f72632d9606c06865a1a7003bc3cf 等于 Stage1。Stage2 SHA不同,未声称二进制可重现性。release发布 job skipped,不等于发布授权。

5. 独立 ABBA #625:过原门槛,但不是零开销

run34749449078 / job103703153884 原始日志核实 exact base/head checkout、同一个 pinned v5.0.0-rc.2 seed、分别 clean Release 编译83TUs、原 instrumentation contract、原19×4 ABBA,非 observer 变体。artifact10315362660 SHA 089c65b604739aba79ad8bdbd90b784c2ae2c985d3f0f3b7c0e33aa3595b4a78

重算全部76 pairs /19 scenarios;72组 instrumented pairs 的完整 counters、counter_breakdown、compile/link/archive hit-miss 向量相同。外部4组 stopwatch 无这些计数,保持 unavailable/null/-1,不能说零。

外部 no-op 四个原始 candidate−base 样本:

  1. +63.0149 ms / +392.6039687%
  2. +1.4072 ms / +12.2515432%
  3. +0.1779 ms / +1.5198114%
  4. -0.6240 ms / -5.0731707%

paired median = +0.79255 ms / +6.8856773%,MAD1.0156ms,P95/max+63.0149ms,3/4较慢。预先约定的阻塞条件是中位差同时 >+1ms 与 >+10%,本次未触发;不能把单个超阈值样本删除或解释成已证明的噪声。

还保留 timings-enabled-no-op median+0.247ms(4/4较慢,max+133.053ms)、modules-cold median+17.1025ms(3/4较慢,max+249.271ms)、cold median+51.9515ms(3/4较慢,max+197.710ms)等全部有利/不利场景。未扣 AA、未裁尾、未 rerun-to-green。只接纳 test-only delta,不据此宣称零开销/性能改进;异常样本进入累计候选性能风险记录。

6. 决定及下一门槛

接受当前精确 head 的测试增量,允许正常 merge(必须再次核对 base/head 并使用 expected-head 检查,不绕过保护)。这只完成累计正确性/兼容性中的一片,不宣布整个阶段完成。

下一轮仍在累计回归:先核对 merge 后 main CI;重点核对 PCH creator/consumer 失败→调用者修复→新一代产物→warm 的缺口。当前 mqb_pch_e2e_tests.cpp 明确 --debug,已有 cold/warm、头变更、缺失PCH、static/DLL 路径,不能把 Release 测试宿主自动算成真实 --release PCH目标覆盖;先与全仓现有覆盖去重,再做单主题小PR,只补真实缺口。

#172保持HOLD;#172/#176 C1041、#170 +1.6946ms/+13.7988%、#180 Performance#599 +2.07175ms/+16.2700%、common129 +5.354s/+6.480s 均保持发布阻塞。新绿灯不能抵销它们。仍须 released v5.5.0 对完整候选的独立累计性能及逐项历史风险处置,再文档/打包收口、独立VERSION/release PR及最终授权。VERSION5.5.0不改,历史tag/资产不改,#164 6.0长期路线不扩入当前范围。

@Iviesever
Iviesever marked this pull request as ready for review September 13, 2026 10:16
@Iviesever
Iviesever merged commit 1a7c460 into main Sep 13, 2026
19 checks passed
@Iviesever
Iviesever deleted the codex/v5.6-foreground-failure-recovery branch September 13, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant