Add a GB10 latency kit to compare fixed stalls with NAK by aien-spark · Pull Request #254 · aien-dev/CUDApp

aien-spark · 2026-11-01T17:33:35Z

Purpose and scope

Host-built latency kit for fixed stalls on the GB10. Kernels are CUDA++ machine code. The headline chip receipt is evidence/LATMEAS-GB10-20261101-oracle/. Its sm121_vs_NAK.md is the table fill_template.py wrote from that directory's raw.json.

Depends on draft PR #251 to open the device. Both chip receipts are local merges of this branch with #251 6a93cb5. That admission is not copied here.

Headline receipt

fb91560 = 7ec0184 merged with #251, clean worktree. RECEIPT.txt: 2026-11-01T18:47:06Z, Linux 7.0.0-1019-nvidia, gcc 13.3.0, open module 580.173.02, device 000f:01:00.0. raw.json sha256 aa3fe3fee12ffe123f572e5a0c9b26b7e7fc82142635ea59442b6a008033534a matches the receipt. Launch rc 0, error empty, 1408 launches, 11 runs.

case	stall_required	spread
alu_alu	4	0
fma_fma	4	0
mixed	5	0
pred	13	0
fmnmx_alu	4	0
imad_wide_lo	3	0
hmma_reader	20	0

At each of those stalls and above: lanes_matched_min 32, lane_fail_mask 0x00000000, first_mismatch null. One stall below: 0 lanes, mask 0xffffffff, first_mismatch run 0 lane 0 word 2.

lanes_matched_min counts a lane only when words 2..5 all equal oracle (tests/gpu_latmeas_test.c lanes_report, lines 1537-1547). HMMA's oracle is four copies of 1098907648, so 32 at stall 20 means d0..d3 matched on every lane. Every first_mismatch in this file is word 2, because that field keeps the first difference. It does not say whether words 3..5 also differed on the failing stalls. The host self-test plants a word-4-only miss on lane 7 (lines 1648-1650) and requires the reporter to name it. That is a host test, not a chip row. word_mismatch_max is added after this receipt and is not in this raw.json.

RECEIPT.txt says model: Unknown. That is the proc Model line. The doctor in run.log names the same device NVIDIA GB10 (sm_121) from PCI 0x2e12. The committed receipt is the file the script wrote. A later run_spark.sh fills the GB10 name from that PCI id when proc Model is Unknown.

The printed bound is 71 s: 1408 launches times 50 ms, rounded up (print_plan, lines 1269-1272), an allowance for a cold shader invalidate. The line says the window is usually shorter. The operator reported about one second. The log has no elapsed-time field.

Earlier receipts, not the headline: evidence/LATMEAS-GB10-20261101/ (lane 0 only, 03cdcc5) and evidence/LATMEAS-GB10-20261101-allanes/ (c94c2c9, same stalls, raw.json has no oracle or lane fields). UNITS.md in the first directory is the NAK delay comparison.

Command

On the Spark, with #251 applied, from this head (the per-word counts and the PCI model line are not yet on a chip):

sh tools/latmeas/run_spark.sh latmeas-out

Build and make doctor stay outside the lock. Only the launch is inside flock /tmp/aien-gb10.lock, wrapped in timeout 600. The binary prints the bound: 1408 launches, 71 s upper bound.

Evidence on this commit

0c81899 is not the commit that was launched. The launch is fb91560. make test-gpu-latmeas PASS on the host, including the planted word-4 miss and run_spark.sh --receipt-self-test.

make test — NOT RUN here. CI host-tests was still running on 7ec0184 when this was written.
make test-native-artifact — host-gates fresh-checkout passed on 7ec0184.
make zero-cuda-gate — same, passed on 7ec0184. Not re-run locally on 0c81899.
build/hello_gpu --sim — hello-sim passed on 7ec0184.
make doctor — host-gates expects NOT READY on the runner. fresh-checkout passed on 7ec0184.
GB10 — PASS for fb91560, evidence/LATMEAS-GB10-20261101-oracle/. Not a receipt for 0c81899.

CodeQL on 7ec0184 failed: six alerts in tests/gpu_latmeas_test.c (world-writable fopen, and a manifest path and allocation size). 0c81899 opens those outputs with mode 0644 and rejects a manifest id that is not a token, and it caps the read at CODE_CAP. That is not a CodeQL result until the new run finishes. host-tests, x86-host-tests, and native-x86-host were still in progress on 7ec0184. This push starts them over.

What merge still needs

GitHub ruleset main-verified-promotion only blocks deleting main and non-fast-forward pushes. It does not require status checks (docs/CI_ACCEPTANCE.md). docs/review/CODEOWNERS.proposed is inactive until it is copied to .github/CODEOWNERS.

docs/REVIEWING.md section 1 still applies: this PR is a draft; evidence has to be for the exact head that is reviewed; host gates are re-run from a fresh clone of that head; someone other than the author approves; merge with gh pr merge --match-head-commit <sha>; nobody merges their own PR. Host results are not hardware qualification.

#251 is also a draft. Its last CI run was green and mergeStateStatus was CLEAN, with no review. It touches nvrm/, which the proposed owners file would assign to GPU-core reviewers after activation. Until then the same section 1 rules apply. #254 does not open a GB10 without #251.

Dependent-chain kernels, a NAK diff, and a Spark runner that holds flock /tmp/aien-gb10.lock only around the launch (timeout 600). Measured cells stay empty until that run; the host self-test does not invent them. Co-authored-by: aien-spark <aien@aienos.com>

The fixed-latency chains read a destination armed to a sentinel the producer does not read, so a stale value fails the oracle. Stall-1 SASS is checked for that pair with no scoreboard and no yield. Cycle medians at stall 1 are recorded even when every stall matches. GB10 open still depends on the 0xa04 admission; this does not add it. Co-authored-by: aien-spark <aien@aienos.com>

@p0

sm121dis of the overhead kernel showed IADD of the counter with stall 1 (control 0x000fc200) immediately before ISETP, and @!P0 BRA back to the body. The compare then samples the previous counter, so the stored trip count misses the overhead check and every exact oracle fails through stall 15. HMMA has no loop. On the predicate chain, stall 1 leaves the previous true P0 for @p0 BRA, so the poison move exits on the first pass and the clocks cover that exit. The host simulator hides GPR writes for 5 cycles and predicate writes for 14. Overhead must store the trip count, the top of the sweep must match the fresh oracle, stall 1 must match the stale oracle, and putting stall 1 back on the increment must fail the counter check. Co-authored-by: aien-spark <aien@aienos.com>

The launch is cb5bd70, the merge of 03cdcc5 and PR #251 6a93cb5. raw.json, the filled table, and sm121dis --pairs of each threshold and the stall under it are the chip output. UNITS.md compares that stall field with NAK's deps.delay at Mesa 01bab930. Co-authored-by: aien-spark <aien@aienos.com>

Each lane stores the two clocks, four result words, and its lane id. HMMA's four words are FADD of d0..d3 against zero, compared with the host dot of sixteen bf16 ones, not with the stall-36 launch. The host simulator requires the same match on all 32 lanes. This has not been launched. Co-authored-by: aien-spark <aien@aienos.com>

c94c2c9 is the merge of 4f3a4d0 and PR #251. stall_required matches the earlier receipt. raw.json still has one lane-0 word and a null reference, so the file does not itself show the 32-lane or d0..d3 check. Co-authored-by: aien-spark <aien@aienos.com>

raw.json now has the four oracle words, lanes_matched_min, a lane fail mask, and the first mismatching run, lane, and word. run_spark.sh writes RECEIPT.txt and no longer prints a 90 s bound beside the binary's 71 s. Co-authored-by: aien-spark <aien@aienos.com>

fb91560 is 7ec0184 merged with PR #251. Passing stalls have lanes_matched_min 32. Every recorded first_mismatch is word 2, so the file does not by itself show a later word failing. Co-authored-by: aien-spark <aien@aienos.com>

word_mismatch_max counts lanes that miss words 2..5. The receipt uses PCI 10de:2e12 when proc Model is Unknown, which is the doctor's GB10 row. Co-authored-by: aien-spark <aien@aienos.com>

+        }
+        char path[512];
+        snprintf(path, sizeof path, "%s/%s_%u.bin", dir, safe, t.stall);
+        FILE *f = fopen(path, "rb");


aien-spark · 2026-11-01T21:46:40Z

PR desk (2026-11-01): the device-open prerequisite this PR depends on has landed on main through #246 (merge dba5821, GB10 SM 0xa04 admitted with its PCI identity). Rebase onto main to drop the #251 dependency for that part.

github-advanced-security Bot found potential problems Nov 1, 2026

View reviewed changes

Comment thread tests/gpu_latmeas_test.c Fixed

Comment thread tests/gpu_latmeas_test.c Fixed

Comment thread tests/gpu_latmeas_test.c Fixed

Comment thread tests/gpu_latmeas_test.c Fixed

Comment thread tests/gpu_latmeas_test.c Fixed

Comment thread tests/gpu_latmeas_test.c Fixed

cursoragent and others added 8 commits November 1, 2026 17:48

Record the GB10 run that stored the host oracle.

6e3da80

fb91560 is 7ec0184 merged with PR #251. Passing stalls have lanes_matched_min 32. Every recorded first_mismatch is word 2, so the file does not by itself show a later word failing. Co-authored-by: aien-spark <aien@aienos.com>

Record per-word mismatch counts and name the GB10 from PCI.

0c81899

word_mismatch_max counts lanes that miss words 2..5. The receipt uses PCI 10de:2e12 when proc Model is Unknown, which is the doctor's GB10 row. Co-authored-by: aien-spark <aien@aienos.com>

github-advanced-security Bot found potential problems Nov 1, 2026

View reviewed changes

Comment thread tests/gpu_latmeas_test.c

}

char path[512];

snprintf(path, sizeof path, "%s/%s_%u.bin", dir, safe, t.stall);

FILE *f = fopen(path, "rb");

aien-spark mentioned this pull request Nov 1, 2026

Open the GB10: admit SM 0xa04 only with its PCI identity and the Blackwell-B class set #246

Merged

Navigation Menu

Repository navigation

Add a GB10 latency kit to compare fixed stalls with NAK - #254

Add a GB10 latency kit to compare fixed stalls with NAK#254
aien-spark wants to merge 9 commits into
mainfrom
cursor/sm121-nak-latency-da60

aien-spark commented Nov 1, 2026 •

edited by cursor Bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aien-spark commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aien-spark commented Nov 1, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose and scope

Headline receipt

Command

Evidence on this commit

What merge still needs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aien-spark commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aien-spark commented Nov 1, 2026 •

edited by cursor Bot

Loading