Skip to content

test: publish 0xE9 cross-model microbenchmark results - #100

Draft
zhoubot wants to merge 1 commit into
mainfrom
codex/ops-e9-sidecars-current
Draft

zhoubot wants to merge 1 commit into
mainfrom
codex/ops-e9-sidecars-current

Conversation

@zhoubot

@zhoubot zhoubot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Publish reproducible 0xE9 cross-model microbenchmark artifacts and independent ASL corpus inputs.

  • uniform 8192-byte cross_model_result ABI plus absolute size symbol;
  • verified main and cross_model_stop symbols for direct boot;
  • 372 rebuilt ELF artifacts, all e_machine=0xE9;
  • host-derived golden and pto-asl-elf-sidecar-v1 generators for every active family;
  • integer vector TDIV/TREM divisors normalized away from zero before regeneration.

Corpus

All 341 active microbenchmark/coverage.json cases now have verified corpus rows:

  • scalar: 124
  • memory: 14
  • cube: 11
  • vector: 129
  • fixp: 63

Every row validates ELF/golden SHA-256, exact LLVM/LLD/TileOp/PTO/source identity, writable PT_LOAD result placement, cross_model_result size 8192, absolute cross_model_result_size=8192, and the stop symbol. Every golden is exactly 8192 bytes.

Reproducibility

  • LLVM/LLD: 27c2ff9dce92629ccf9b66140d171a13ce2c0153
  • TileOp: 155f610c838e918e1fc30ae44f631a85c440b76b
  • PTO-SPEC: e51cf7382584a367b0222411966576971c74d264
  • ELF machine: 0xE9

Runtime status

The real scalar add_i32_lat ELF passes the ASL model with result 3905 and an exact 8192-byte golden. The parameterized GTest suite is running with ctest -j8; remaining family results will be added before this draft is marked ready.

Tracks #99.

@zhoubot

zhoubot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Added eacd2ba to align the runnable corpus with PTO v0.58.5 and the corrected ASL runner path:

  • active corpus is now 333 cases; the 8 TPART* binaries are retained as explicit unsupported coverage because those operations are deleted/reserved in PTO v0.58.5
  • CUBE and FIXP sidecars now allow up to 1,000,000 architectural instruction steps, avoiding false PTO_STEP_LIMIT failures
  • fresh compile gates: vector 121/121, CUBE 11/11, FIXP 94/94
  • the two pure expand FP32 cases pass the ASL functional model after the TileOP destination-geometry correction

Dependencies: LLVM PR #78 and TileOP API PR #44. This PR remains unmerged pending the refreshed 333-case ASL run.

@zhoubot
zhoubot force-pushed the codex/ops-e9-sidecars-current branch from eacd2ba to c8577e9 Compare September 3, 2026 07:19
@zhoubot
zhoubot requested review from lvhao7896 and ly-ict September 3, 2026 08:07
@zhoubot

zhoubot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Owner review requested; this PR intentionally remains Draft and must not be merged yet.

Current signed candidate: c8577e91ed1c66099bef3a70a4b17dbd8e7608b2. Independent code review found no remaining producer/verifier defects after the latest fixes:

  • preserves current main benchmark and RES_CHECK behavior; corpus publication is explicit and isolated
  • locks the corpus at 333 active + 29 explicitly unsupported cases
  • validates ELF64/LE/e_machine=0xE9, result ABI, writable segments, sidecar paths/hashes, producer/toolchain/PTO/asl-model/TileOp identities
  • rejects missing, duplicate, mixed-identity, malformed, or hash-mismatched inputs fail-closed
  • commits no generated ELF/sidecar/golden/manifest artifacts

Validation completed: 21 producer/verifier/benchmark-stability tests, generation idempotence, Python compilation, Ruff, and git diff --check.

Not yet release evidence: the real 333-case compiler-to-ASL execution has not run. Please review the code and artifact contract; keep the PR Draft until exact toolchain/model execution produces complete immutable evidence.

ziyang-cheng added a commit to ziyang-cheng/SuperNPUBench that referenced this pull request Sep 10, 2026
- README 状态块新增 2026-09-10 更新: cherry-pick fix/cherrypick-63-100
  (b8669ce+PTO-ISA#63+PTO-ISA#100) 装头后 gfrun 四项全绿(动态编译/执行/非尾轴精度/尾轴回归)
- RECORD 问题28(PTO-ISA#100)/29(PTO-ISA#63) 标记已修复并补验证小节
- RECORD 问题14 注更新 pto-spec#256 裁决 + 本地 patch 过宽核对(dmxq 不受影响)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ziyang-cheng added a commit to ziyang-cheng/SuperNPUBench that referenced this pull request Sep 12, 2026
框架增强(向后兼容单输出算子):
- Case 增加 verify 钩子(自定义比对,支持多输出/专用 golden)与 four_pe 字段
- run_case:prepare 可选、verify 优先于默认 np.allclose、按 four_pe 决定 4-PE

样板 dynamic_mx_quant 8 个 driver(多输出 B 范式):
- 双输出(output+scale_output),verify 调算子自带 dynamic_mx_quant_data_compare.py
  (FP8/FP4 解码 MSE<0.1 + scale 逐字节);golden 由 compile.all 的 gen 脚本生成
- four_pe 按 driver 标记(cublas/ocp 4pe 变体)

注:缺 TileOP PTO-ISA#63/PTO-ISA#100 的发布版工具链上 kernel 编不过->ELF 缺失->如实 SKIP;
工具链缺口修复合入 tag 后自动编过+跑通(条件切换意义)。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wangyuascend-spec pushed a commit that referenced this pull request Sep 12, 2026
* solution: 新增 host-golden 精度入口 res_check_all.py(与 multi_thread 对称)

作为 solution 树的统一精度验证入口,与 test/kernel/multi_thread/res_check_all.py
对称:host-golden 范式(kernel 只做二进制 I/O,host 用 numpy 造 golden + np.allclose),
同一套 CLI(--gfrun/--timeout/可选 --compiler-dir)与 stdout 输出格式。

CASES 留空,由算子侧维护:加算子只需追加一个 Case + prep_ 函数,上层遍历接入零改。
本入口 host-golden 范式替代 kernel 内嵌自判(后者不合规),与 multi_thread 一致。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* solution/res_check_all: 增强 Case 框架 + dynamic_mx_quant 样板接入

框架增强(向后兼容单输出算子):
- Case 增加 verify 钩子(自定义比对,支持多输出/专用 golden)与 four_pe 字段
- run_case:prepare 可选、verify 优先于默认 np.allclose、按 four_pe 决定 4-PE

样板 dynamic_mx_quant 8 个 driver(多输出 B 范式):
- 双输出(output+scale_output),verify 调算子自带 dynamic_mx_quant_data_compare.py
  (FP8/FP4 解码 MSE<0.1 + scale 逐字节);golden 由 compile.all 的 gen 脚本生成
- four_pe 按 driver 标记(cublas/ocp 4pe 变体)

注:缺 TileOP #63/#100 的发布版工具链上 kernel 编不过->ELF 缺失->如实 SKIP;
工具链缺口修复合入 tag 后自动编过+跑通(条件切换意义)。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* solution/res_check_all: 接入 normalization 4 算子 + Case 框架增强

框架增强(向后兼容单输出算子):
- Case 增加 prepare(可选) 与 verify(自定义比对) 钩子、four_pe 字段
- run_case: prepare 可选、verify 优先于默认 np.allclose、按 four_pe 决定 4-PE

样板 B: normalization 4 个(rms_norm/rms_norm_binary/group_norm_grad/group_norm_grad_1d)
- make_prep_gen 调各自 gen 脚本生成 input+golden 到 case_dir
- make_verify_cmpdir 调各自 compare 脚本(--cmp-dir,退出码 0=PASS)
- 多入多出由算子自带 gen/compare 吸收,框架只调度
- matmul_test 暂不接入(verify_matmul_test 一体化自 gfrun,与 run_case 两段式不契合)

注:发布版工具链行归约/TileOP skew 下这些 kernel 编不过->ELF 缺失->如实 SKIP。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant