Repository navigation
fix(benchmark): run the EdgeBench report adapter and suite on native Windows - #6368
Conversation
…Windows On native Windows `uv run --extra test python -m pytest -q`, the command CONTRIBUTING documents as the full local check, aborts with "Interrupted: 2 errors during collection" and runs none of the 21851 collected tests. Both errors come from `benchmark/edgebench/feedback_hook.py`, which imported POSIX-only `fcntl` at module scope, so the report adapter's suite and the hook's own suite could not be imported at all. The same import failure makes the documented verifier `python -m benchmark.edgebench.export_report --verify <study data>` unusable on that host before it reads any evidence. Delivery keeps its flock-based session lock; guard the import the way `loopx/file_lock.py` guards `fcntl`/`msvcrt`, refuse delivery with a typed error where no supported lock backend exists, and skip the delivery suite itself where POSIX flock is absent. The report adapter has no lock requirement, so it imports and its tests run there again. The published study archive binds sha256 of its own files in `index.json`, so the checkout line-ending translation of the same host made all 13 bindings mismatch and the verifier report the published evidence as corrupted. Pin study archives to LF, as the immutable v1 archive already is. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
The module-level fcntl importorskip skipped the whole delivery suite on hosts without POSIX flock, including the refusal test added for exactly that condition, so the platform this fix targets never ran it. Keep the delivery tests behind a flock marker and leave the refusal case unmarked; it now runs on Windows against a real missing backend. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
…archive Git does not rewrite files already on disk when .gitattributes changes, so an existing checkout keeps reporting digest mismatches until the pinned data directory is restored once. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
…s stale files A plain `git checkout -- <dir>` is a no-op for files that already exist in the worktree: with `text eol=lf` their cleaned content matches the index, so Git considers them unchanged and skips the write. Prescribing it left an existing checkout still failing verification. Delete the stale directory first and restore it from the index; re-ran it here: no CR bytes remain and --verify returns runs=10, files=13, verified=true. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | xhigh
动机
在 Windows 上运行报告导出、验证或本地测试的贡献者。 过去报告模块在缺少 POSIX fcntl 时连导入都失败,现有档案也可能被 Git 转成 CRLF;本改动让只读验证可启动并修好已发布档案,但新导出的 JSON 在入库后仍可能被改写,导致自己的摘要失配。 已发布13文件的验证与无锁读取路径通过;新导出反例在入库前通过,经过当前 Git 属性和重新取出后报 File binding failed: run-rows.json。 本批不提供 Windows 锁后端、启动 benchmark/model、改变评分或把摘要验证当作研究资格。
完整评审基线 806eb21 到 1357ca2 的4文件。读取现有档案的修复有正向价值,但不能用这条成功路径代替新导出报告的可用性。
改动思路
报告读取与需要 flock 的通知交付沿用现有模块,缺后端只阻断交付;档案传输必须保留计算摘要时的原始字节,不能用 Git 的文本规范化替代该不变量。 当前PR应一并修好导入、原档案取出和受该新属性覆盖的新报告往返;无需增加通用锁框架或改动评分。 保留原模块的 guarded import 与失败前无副作用检查是合适的小修复,复用了已有 file_lock 的平台边界思路。最强反对理由不是增加了几行代码,而是新属性同时覆盖后续生成档案,现有 JSON 生产者的换行字节与 Git 存储规则未一起收敛。
具体改动
接受依据 issue #6367,spec_ref=https://github.com/loopx-project/loopx/issues/6367,复现基线 spec_revision=806eb215f8a21e4ae6f51cd942112175ca3839b7;另外读取既有报告 README 的摘要绑定、只读验证及新终态导出约定。
- report_without_flock:实现。实际 CLI 在模块不可用的选择器下,基线导入失败,新 head 读取现有10次运行、13文件通过。
- lock_before_delivery:实现。三个事件类型、两个 session 的6次无后端调用均在创建目录前拒绝;真实 POSIX flock 交付和相同身份回放保持原语义。
- bytes_match_published_bindings:未满足。旧属性、相同 Windows 换行输出与当前新属性做真实 Git 往返,新属性改变两个已绑定 JSON 文件;改成不转换文本的属性,完整往返通过。
关键代码讲解
feedback_hook.fcntl 的模块导入现在允许缺失后端,不把读取报告的可用性当作通知权限。deliver 在第28行入口先检查后端,缺失时 OSError,不写 cursor,也不发出结果。export_report.write 第40行仍使用平台默认文本换行;CSV 的 csv_write 则已经明确 LF,本次没有发现 CSV 换行缺陷。.gitattributes 第6行的新 text eol=lf 会在入库时规范化整个 study/data 子树,因此与 JSON 生产者先计算原始 SHA256 的顺序冲突。
测试按真实 lock/file-mode 条件做 skip 标记,并新增 no-lock 拒绝;README 的删除后重新取出步骤确实能让旧 materialization 更新,单纯 checkout 会跳过仍被认为 clean 的文件。这些改动都已纳入评审,没有要求新建通用锁能力或重写为 TypeScript。
对主干的风险
[P1] 保留新导出报告已经绑定的原始字节 — .gitattributes:6。 在 Windows 默认文本换行语义下,现有 JSON writer 写出 CRLF,index/files 与 settings 先绑定这些原始字节。新规则在 git add 时把 JSON 转成 LF,重新取出后 --verify 报 ValueError: File binding failed: run-rows.json;settings/run.json 的绑定也失配。这里是新导出报告,当前已发布的13文件修复是成功的,不能混为一条路径。
我使用仓库合法终态 fixture、实际报告 CLI、Python TextIOWrapper 的 CRLF 写出选择器及真实 Git,未模拟验证结果。相同输入的三组结果:旧属性入库后退出0;当前属性入库前退出0、重新取出后退出1、两个 JSON 失配;-text 字节保留方案退出0。宿主是 macOS,这不是原生 Windows/ACL 认证。最小修复是对摘要绑定数据使用不改字节的属性,或让所有 JSON 在最初写出/计算摘要前固定 LF;增加导出→验证→Git入库→重新取出→验证的回归,不能靠修改已有摘要或削弱校验掩盖转换。
现有56项 report/hook 测试全通过且无跳过,Ruff/diff hygiene 通过。真实 Git 的 core.autocrlf=true 档案取出对照从13个失配降到0;无 fcntl 的公共读取、无后端拒绝和 POSIX 交付/回放也通过。这解释了为什么作者的现有验证可以全绿,仍遗漏新 JSON 发布路径。
语义与 CI 对齐
此处没有新共享词表或通用控制面义务,继续复用原报告摘要契约;违反的是现有跨生产者/存储/消费者的原始字节不变量。CI按配置未查询;作者原生 Windows 收集数及历史 canary/mypy 报告不冒称本次独立执行,也未用未跑的平台测试替代这个已复现反例。
我的整体评价
REQUEST_CHANGES — 1357ca2。long_horizon 与 user_experience 在新导出→发布→取回路径有 regression:一次成功导出不能保证后续可读,重复手工验证或重算摘要也不应成为正常旅程。现有档案和导入修复方向正确,机制规模合适,但必须在本 PR 修好紧邻的字节传输边界。报告读取与需要 flock 的通知交付沿用现有模块,缺后端只阻断交付;档案传输必须保留计算摘要时的原始字节,不能用 Git 的文本规范化替代该不变量。 当前PR应一并修好导入、原档案取出和受该新属性覆盖的新报告往返;无需增加通用锁框架或改动评分。 相邻未来整理限定为这项可逆的小修复;不要求另造状态、能力或独立项目,也未运行 benchmark/model 或执行合并。
English verdict: REQUEST_CHANGES - 1357ca2: existing archive/import and56tests pass,but text-eol normalization breaks two boundJSON files from a Windows-newline export after realGit roundtrip. Preserve rawbytes or serializeLF beforehashing,and addexport-to-fresh-checkout validation. Windowsnewline was simulated onmacOS;nativeWindows/ACL wasnot independentlyqualified.
export_report.write serialized JSON through the host text default, so a native Windows export wrote CRLF bytes and index.json bound the CRLF digests. The study archive pins text eol=lf, which normalizes the file to LF at git add, so a fresh materialization held different bytes than the ones hashed and --verify failed with File binding failed: run-rows.json. Write JSON with explicit LF bytes so the export is byte-identical on every platform, matching csv_write. State the LF-byte invariant in the archive note and the study README. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
Walk a new export through Git publication with core.autocrlf=true and a fresh materialization, checking the bound digests and the materialized bytes. The export case fails when the writer leaks host newlines, the published-archive case fails when the LF pin stops applying, and a control file proves checkout conversion stayed active. The Windows text layer is modeled by forcing newline="\r\n" on default text writes, so the regression also fails on POSIX test hosts. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
|
Author response - review follow-up at The P1 byte-transport finding is confirmed and fixed on the branch. Reproduced natively on a Windows host at What changed
Evidence on the new head (native Windows host): the writer test and the export roundtrip fail at No runner, scoring or evidence semantics changed; |
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | xhigh
APPROVE — 8e6614ca16b9236d70126411e07e40adac558533。旧 P1 的新报告发布反例已在当前 head 消失;此次重新检查完整六文件,并非继承此前结论。
动机
在 Windows 上运行本地检查、导出报告或验证已发布研究档案的贡献者。过去缺少 POSIX 锁模块会让报告和两个测试模块无法导入,Git 换行转换还会使报告摘要失配;当前读取能够启动,JSON 在计算摘要前固定 LF,重新取出仍可验证。相关模块可执行,无锁读取通过;既有档案返回 10 次运行、13 文件验证成功,新报告经过真实 Git 往返也保持全部摘要绑定。本次不提供 Windows 通知锁后端、不启动 benchmark 或模型任务、不改变评分,也不把文件摘要当作研究结论的资格。
改动思路
固定 LF 写出和档案 Git 属性共同维护原始摘要,缺锁仅阻断通知交付;现有模块已覆盖三个责任,无需新通用框架。 本 PR 完成无锁读取、既有档案和新报告的 Git 往返;Windows 通知锁及 online_judge 仍在范围外。 最强反对方案是只增加 Git LF 属性:生产者仍写 CRLF 时,先绑定的摘要会被入库转换破坏。当前把修复放回已有 JSON writer,既不重新计算公开档案摘要,也不添加平行验证器。loopx.file_lock 的 guarded-import 和已有 immutable archive 属性提供了可复用边界;这里沿用专门 benchmark IO,不另建通用锁或控制面能力。
具体改动
接受依据为 issue #6367,spec_ref=https://github.com/loopx-project/loopx/issues/6367;spec_revision=issue6367-updated-2026-10-13T07:55:33Z。从实际 merge base 806eb215f8a21e4ae6f51cd942112175ca3839b7 到当前 head 共六文件 +131/-2;另单独核对旧评审 head 1357ca22d1699720bb2bdf123323b81984080674 的修复。
- local_check_runs:实现。两个原先中断 collection 的模块在缺 fcntl 条件下实际执行 47 项、跳过 12 项依赖锁的用例;全树在缺 fcntl 条件下实际收集22079项、退出0;收集不等于执行,不把相关模块通过当作全树测试通过。
- verifier_readback:实现。真实已发布档案验证为 10 次运行、13 文件、verified=true;公共 CLI 无锁对照为基线导入失败、当前读取成功。
- archive_byte_bindings:实现。既有档案和新导出都经过真实 Git 重新取出与摘要校验;新导出测试同时证明无关 control.txt 确实发生 CRLF 转换,排除 Git 没有转换的假阳性。
关键代码讲解
export_report.write 第40行在 Path.write_text 明确 newline=LF,先统一原始字节,再由现有 digest/index 绑定;CSV 本来就用 LF,无需添加第二套序列化规则。feedback_hook.deliver 第28行先检查真实锁后端,缺失时在目录、cursor 与输出前抛 OSError;支持 POSIX 的 flock、并发及重复交付路径保持原语义。.gitattributes 第6行规定 study/data 的取出字节;现在其假设与生产者一致。README 说明已有 stale materialization 的一次性恢复,并保留已发布摘要。两个测试模块分别验证缺锁拒绝、平台标记、实际 TextIOWrapper CRLF 默认及旧/新档案 Git 往返,完整 diff 均已纳入。
对主干的风险
没有发现当前阻塞项。最强反例用同一合法终态输入、真实报告 CLI、CRLF 默认文本写出与实际 Git core.autocrlf=true:旧1357生产者在入库前通过,重新取出后退出1且两个 JSON 摘要不匹配;当前8e生产者退出0且全部匹配。这是旧发现被修复的判别证据,不仅是新增测试绿色。实际不可用 fcntl 的三个事件、两种 session 共六次交付均没有 receipts 目录或输出;支持锁的并发、重复交付及篡改摘要反例由完整相关模块覆盖。
当前相关59项全通过、0跳过;实际基线同两模块55项通过。同一 POSIX fixture 的所有导出文件摘要逐项相同,Ruff 四个变更 Python 文件和 diff hygiene 通过。无锁条件47通过/12跳过有具体后端原因,全树 collection 22079项退出0;宿主 macOS 的 CRLF/缺锁选择器不等于原生 Windows 内核、ACL 或整树执行资格。未查询或等待 CI,也未启动研究任务。
语义与 CI 对齐
复用现有 raw SHA256 档案契约,无新共享词表或通用状态义务。生产者、存储与读回的原始字节不变量现在一致。开发 advisory 对 benchmark 目录的覆盖有限,结论来自完整 diff、真实定义/调用搜索与上述往返,不拿空 probe 当语义证明。
我的整体评价
APPROVE 当前 exact head。long_horizon 与 user_experience 均 improved:一次导出以后仍可取回阅读,无锁读者不再被通知模块的依赖阻断;缺后端的通知保持诚实拒绝。规模与原问题相称,未来整理已限定为复用现有 writer 的小修复,不需要通用锁框架或语言迁移。固定 LF 写出和档案 Git 属性共同维护原始摘要,缺锁仅阻断通知交付;现有模块已覆盖三个责任,无需新通用框架。 本 PR 完成无锁读取、既有档案和新报告的 Git 往返;Windows 通知锁及 online_judge 仍在范围外。 这关闭本 issue 的有界修复评审,保留原生 Windows 独立实机验证的残余风险;批准不授予合并、旧 review dismissal 或模型执行权限。
English verdict: APPROVE - 8e6614c: old Windows-newline publication counterexample now passes actual Git roundtrip; full related59 tests, baseline/head POSIX byte parity and6 no-effect lock refusals verified. Native Windows kernel/ACL and whole-suite execution were not independently certified; CI not consulted.
Goal And Delivered Outcome
benchmark/edgebench/feedback_hook.pyimported POSIX-onlyfcntlat module scope, so on native Windows the documented report verifier could not start andpytest -q(the full local check CONTRIBUTING documents) aborted collection before running anything. The published study archive is also not exempt from checkout line-ending translation, so its 13 digest bindings mismatched on acore.autocrlf=truecheckout.pytest -q→Interrupted: 2 errors during collection, 0 tests executed, exit 2 → 22062 tests collected with no collection errors;python -m benchmark.edgebench.export_report --verify benchmark/edgebench/studies/portfolio-ten-arm-20261006/data→ModuleNotFoundError: No module named 'fcntl'→{"runs": 10, "files": 13, "verified": true}; archive bindings → 13 of 13 mismatched → 0 of 13. Rows below.main(806eb215f).Author Declaration
AI disclosure: this change was drafted with AI assistance at my request, under my direction and review.
Implemented against
benchmark/deepswe-gptxhigh-v1/**(commit71cc15692, "Preserve the immutable archive's bytes even with core.autocrlf=true"), and the guarded POSIX import followsloopx/file_lock.py.benchmark/edgebench/feedback_hook.py--verifycommand (validation rows 2, 3)feedback_hook.deliverbenchmark/tests/test_edgebench_feedback_hook.py::test_delivery_refuses_a_host_without_the_posix_session_lockbenchmark/tests/test_edgebench_feedback_hook.pypython -m pytest -v -n 2 -m "not stage2c_e2e"includebenchmark/tests(row 7).gitattributes,benchmark/edgebench/studies/portfolio-ten-arm-20261006/README.mdbenchmark/edgebench/export_report.py,.gitattributesbenchmark/tests/test_edgebench_report.py::test_exported_report_survives_autocrlf_publicationandtest_json_writer_ignores_host_text_newline_translationfcntlexists. I deliberately leftbenchmark/edgebench/online_judge.pyalone (same import pattern, outside the report chain, already guarded by an optional-dependency skip) and did not touch runner, scoring or evidence semantics. Review follow-up: the reviewed P1 byte-transport gap was reproduced natively on Windows - the exporter bound CRLF digests for the JSON files,text eol=lfnormalized them atgit add, and a fresh materialization failed--verifywithFile binding failed: run-rows.json;export_report.writenow writes LF bytes explicitly and the roundtrip has regression coverage.Scope And Continuation
.gitattributesrule, and the exporter writes LF bytes on every platform so published digests keep matching.benchmark/edgebench/online_judge.pykeeps its unconditionalfcntlimport on purpose.Validation
8e6614ca16b9236d70126411e07e40adac558533regression_paritypassed806eb215faborts (Interrupted: 2 errors during collection, 22020 collectible, 0 executed, exit 2); this revision collects 22062 with no errors, exit 0.regression_paritypassedbenchmark/edgebench/studies/portfolio-ten-arm-20261006/data: 13 of 13 mismatch on a fresh base checkout, 0 of 13 on a fresh checkout of this revision. An existing checkout needs one re-materialization, which the study README now documents; re-ran it here: after deleting the stale directory and restoring it, no CR bytes remain and --verify returns runs=10, files=13, verified=true.regression_paritypassedcore.autocrlf=truematerialization failed--verify(File binding failed: run-rows.json); withnewline="\n"the export stays LF and the new export -> git add -> fresh materialization -> verify roundtrip passes for both a new export and the published archive. The writer and both roundtrip tests fail at 1357ca2 and pass at 8e6614c.real_entrypointpassedpython -m benchmark.edgebench.export_report --verify benchmark/edgebench/studies/portfolio-ten-arm-20261006/dataprints{"runs": 10, "files": 13, "verified": true}and exits 0 on a fresh checkout of this revision.unitpassedpython -m pytest -q benchmark/tests/test_edgebench_report.py benchmark/tests/test_edgebench_feedback_hook.py -rs→ 46 passed, 13 skipped (the flock-dependent delivery tests skip; the no-lock refusal case passes on this host).manualpassedfeedback_hook.fcntlisNone;deliverrefuses withOSError: best-only delivery has no supported session-lock backend on this host, and no receipt directory is created.staticpassedpython -m ruff check benchmark/edgebench/feedback_hook.py benchmark/tests/test_edgebench_feedback_hook.py→ all checks passed;git diff --checkclean;scripts/generate_semantic_inventory.py --changed-from <base>reports no new vocabulary carriers;loopx check --scan-path benchmark/clean.integrationnot_runmanualblockedloopx canary premerge --from-git-diffran 8 risk-profile smokes; 5 fail identically at the base revision on this host (pnpm workspace for the DSH plugin phases; Windows path normalization incatalog-run-e2e-smoke.pyandpremerge-validation-gate-smoke.py). They are pre-existing host failures, not from this diff. Canary also reports thebenchmark_sensitivemanual hold, so this PR is left for maintainer review.staticnot_runpython -m mypyon this Windows host fails atloopx/control_plane/effect_runtime.py:437(Module has no attribute "WNOHANG"), identically at the base revision and outside this diff; CI runs mypy on Linux.benchmark/tests, which is why the collection abort was invisible there.Frontend / Visual Evidence
Type of Change
LoopX Area
Technical Direction
Shared-authority RFC fixture impact
Boundary Checklist
.loopx/,.codex/goals/, and liveACTIVE_GOAL_STATE.md).none.Signed-off-bytrailer (git commit -s).