Repository navigation
honeykrisp: fold workgroup memory address deltas on G14 (stacked on #6) - #11
Draft
joshuaswarren wants to merge 62 commits into
Draft
joshuaswarren wants to merge 62 commits into
joshuaswarren wants to merge 62 commits into
Conversation
…add32 Add the opcodes and packer (byte-exact against applegpu), a NIR intrinsic bridge, the agx_nir_lower_simdmat lowering, the validator widths, and a gated advertisement of an 8x8x8 fp32 cooperative matrix shape. Everything is behind AGX_SIMDMAT; default Honeykrisp behavior is unchanged.
…le coopmat shaders
VK_KHR_cooperative_matrix on Honeykrisp only advertised 8x8x8 fp32 even
though agx_nir_lower_simdmat.c already tiles a 16x16x16 coopmat into a
2x2x2 grid of simd_matrix_fmadd ops, and the G13 matrix unit takes fp16
operands. Now advertised (subgroup scope, N in {8, 16}, NxNxN):
fp32 A/B, fp32 C/D simd_matrix_fmadd32 with 32-bit pairs
fp16 A/B, fp32 C/D simd_matrix_fmadd32 with 16-bit A/B pairs
fp16 A/B, fp16 C/D simd_matrix_fmadd16
The mixed form is the one Apple's own Metal mul_mm kernel emits
(simd_matrix_fmadd32 r15_r16, r49l_r49h, r46l_r46h, r15_r16); the generic
ALU packer already encodes the per-source size bits, so the lowering
passes A/B at their own width instead of up-converting to fp32, and the
nir intrinsic gains a 16-bit dest form that selects fmadd16.
Bug found while verifying: with fp16 A and fp32 D the register allocator's
early-kill let D (a 32-bit pair) land on top of A (a 16-bit pair inside
that pair). The hardware then produced zeros for that fmadd: 16x16x16
fp16/fp32 failed in every input/layout variant, always sub-block (1,1),
whose last fmadd was the only one with D overlapping A. Apple's compiler
only ever emits D == C, so the RA now keeps A and B live until D is
placed (D may still reuse C).
Verification on Apple M1 (G13G B1), the cmshape test program, one coopMatMulAdd
per run, exact host reference, integer-valued inputs bit-exact and random
floats within tolerance, identity/ones/randint/randf x row/col-major:
stock Mesa 26.1.7: FAIL, extension not exposed
5bb2b28 (before): 8x8x8 fp32 8/8 PASS; 16x16x16 not advertised
this commit, no RA guard: 40/48 (16x16x16 fp16/fp32: 8/8 FAIL)
this commit: 48/48 PASS, plus 48/48 randint over 4 seeds
MLX Release tests (primitive 98, reduce_ops 34, runtime 40,
matmul_family 14 cases) pass identically on the stock ICD and on this
driver with AGX_SIMDMAT=1.
Bug: (word >> s) & mask with a non-constant s returns zero on Honeykrisp
whenever s has any bit set above bit 4 (s >= 32). MLX hit this as
the "dynamic shift-then-mask byte extraction" miscompile family: bool
scatter dropping writes across word lanes, select picking the wrong
operand under broadcast conditions, reduce_general's load_truthy reading
zero, masked_scatter's scan undercounting (s_scan[255] = 32, not 128).
Root cause: agx_nir_fuse_algebraic_late rewrites
iand(ushr(a, b), (1 << bits) - 1) -> ubitfield_extract(a, b, bits)
NIR's ushr uses only the low 5 bits of its count, and by this point the
main nir_opt_algebraic has already relied on that to strip explicit masks
(ushr(a, iand(b, 31)) -> ushr(a, b); ushr(a, ishl(iand(b, 3), 3)) ->
ushr(a, ishl(b, 3))). ubitfield_extract has no such wrap: an offset >= 32
is undefined in NIR and bfeil returns 0 for it on the hardware.
lower_sm5_shift, which masks the count of every surviving shift, runs
after the fusion and never sees the ushr it consumed.
Fix: fuse directly only when the count is a constant below 32; otherwise
fuse with an explicit iand(b, 31) on the offset. Same instruction count as
the unfused shift+and, and the mask CSEs with one the shader already had.
Verification on Apple M1 (G13G B1), standalone Vulkan compute reproducer
(20 variants, 1024 lanes each, host-compared):
stock Mesa 26.1.7: 9 of 16 base variants FAIL (byte_dyn,
byte_dyn_const, byte_dyn_gid, scan_dyn, scan_dyn_const, scan_dyn_rand,
select_bcast, select_dyn, shift_dyn_mask), plus all 4 boundary
variants FAIL (count exactly 32: 1016/1024 wrong; 63: 516/1024;
uniform 40: 1020/1024; divergent 8/40: 509/1024, only the 40 lanes).
this commit: 20 of 20 PASS, 0 mismatches.
MLX suites, stock vs this commit (AGX_SIMDMAT=1), identical:
primitive 98/98 (2699373 assertions), reduce_ops 34/34 (7152),
runtime 40/40 (6307), indexing_ops 53/53 (4093), select_layout 11/11
(1908), scatter_determinism 21/21 (127).
…duction
Three fp32 rounding defects on Honeykrisp (Apple M1 G13G, Mesa 26.1.7)
that MLX patches around in its shaders:
a) GLSL "/" was lowered by nir_opt_algebraic (lower_fdiv) to
a * frcp(b), and frcp was only Newton-refined when the instruction
was `precise`. Even with an exact reciprocal a * RN(1/b) is not
RN(a/b): 80/25 and 10/25 come out one ulp low, 6.55859375/0.030016447
one ulp high. Measured on 1008 pairs: 299 mismatches (298 at 1 ulp,
1 at 2 ulp), 1/b: 204 mismatches; 1M random pairs: 305792 mismatches.
fma() itself is IEEE (0/1000 mismatches), the MLX note
blaming fma contraction was a misdiagnosis.
b) log(x) was flog2(x) * ln2 on the hardware log2, which is only
absolutely accurate: log(3) = 1.0986121892929077 instead of
1.0986123085021973, 2792/4107 mismatches on [0.5, 64] with errors up
to 357 ulp near 1 (log(1.0000001) = 1.65e-7 vs 1.0e-7); log2 had
2905/4107 mismatches.
c) sin/cos reduced the argument as fract(x / 2pi) in fp32, so the
reduced angle lost ulp(x) of precision: sin(3.0) 14 ulp off,
sin(1e5) 1.06e-3 absolute error, sin(1e8) = 0.0, sin(-1e-3) 571 ulp
off, sin(pi_f32) = 0, and the hardware sin_pt_2 core itself is 2 to
3 ulp off (cos(1e-3) 3 ulp).
The lowering moves to a new the AGX compiler pass file agx_nir_lower_math.c,
built only from the exact fp32 fma and marked no-fast-math so
nir_opt_algebraic cannot reassociate it:
- lower_fdiv is dropped from agx_nir_options; fdiv32 becomes Markstein
division (y = refined rcp, q = a y, r = fma(-b, q, a), q + r y) with
the NaN/zero guard keeping q for b = 0, overflow and signed zero.
frcp32 is always refined. fdiv16 keeps a * frcp(b).
- fmul(flog2(x), ln2) as emitted by nir_flog, and any remaining flog2,
become a double-float evaluation of log(m) = 2s + 2s^3 P(s^2) with
s = (m-1)/(m+1) kept as a pair, m folded into [sqrt(1/2), sqrt(2)),
e ln2 split hi/lo and Fast2Sum accumulation (about 40 bits before the
final rounding). Zero/subnormal/negative/inf/NaN are handled by bit
tests, not the hardware log2.
- fsin/fcos: Cody-Waite in radians with pi/2 in three pieces, the k C2
product and its subtraction as TwoProduct/TwoSum so the reduced
argument is a pair accurate to an ulp, for |x| < 2^22; an integer
Payne-Hanek (24-bit mantissa times a 128-bit window of 2/pi, 64
fraction bits kept) for |x| >= 2^22 up to FLT_MAX. The quadrant then
selects a Taylor polynomial in quadrant units (sin degree 13, cos
degree 14, leading terms as pairs, low part of the argument fed to
the derivative). The hardware sin_pt_1/sin_pt_2 core (nir_op_fsin_agx)
is no longer used by the lowering. fp16 sin/cos go through the fp32
path.
Verification (standalone Vulkan compute reproducer):
- division: 1008-pair table incl. 80/25, 10/25, 1000 random: 0
mismatches (stock 299); 1/b 0 (stock 204); 1M random pairs 0
mismatches for both a/b and 1/b; b = 0, inf, NaN, overflow, flushed
subnormal and -0/5 = -0 match host IEEE.
- log: 4107 points on [0.5, 64] plus pinned exact cases: 0 mismatches
(stock 2792), log(3) = 1.0986123085021973; log2 0 (stock 2905);
1M random on [1e-37, 1e38]: log 3 at 1 ulp, log2 14 at 1 ulp,
rest exact.
- sin/cos at 1e5, 5e5, 1e6, 5e6, 1e8: all 0 ulp from the double
reference except cos(5e6) at 1 ulp (stock: 2.8e5 to 1.06e9 ulp);
1310 mixed points max 1 ulp; 1M random over [1e-3, 1e9] and [-10, 10]:
sin 42346 and cos 41184 at 1 ulp, rest exact, max 1 ulp (stock
>= 8 ulp for 627689 / 512816 of them).
- MLX suites on the same Release tree, stock vs fixed:
primitive 98/98, eq_math 7/7, fast_ops 30/30, runtime 40/40,
reduce_ops 34/34 both; complex_ops 30/33 both with the same three
pre-existing cases (failed assertions 31 -> 29). The upstream
ops_tests log(3) pin through libmlx: FAIL on stock, PASS on fixed,
as do 80/25, 10/25 and sin/cos up to the 1e5 limit mlx enforces.
- timing: 16M-element elementwise dispatch is memory bound at ~2.45 ms
on both drivers (sin 2.474 -> 2.463 ms median, within spread). A
32-deep in-register chain shows the ALU cost: sin 6.64 -> 39.6 ms,
log 2.57 -> 34.8 ms, div 2.84 -> 7.24 ms, mul unchanged.
…it late Symptom: any shader converting fp32 to a 64-bit integer aborts in the backend with "Unhandled ALU op fdiv" (agx_compile.c agx_emit_alu); the MLX suite the copy-offset tests crashes on "dtype converting copies cover the integer family and bool". Fragment shaders with perspective-correct varyings hit the same path. Cause: "agx: correctly rounded fdiv/frcp, faithful log/log2, exact sin/cos reduction" dropped lower_fdiv from the compiler options so nir_opt_algebraic no longer splits fdiv into fmul/frcp, and only ran agx_nir_lower_fdiv in agx_preprocess_nir. nir_lower_int64 (f2u64/f2i64: fdiv by 2^32, plus the frem that lower_fmod turns into a second fdiv) and agx_nir_lower_interpolation run after preprocessing, so their fdiv reached the backend, which has no fdiv. Fix: agx_nir_lower_fdiv_late runs in agx_optimize_nir after the int64 cleanup loop and lowers only fdiv, leaving frcp alone so the reciprocals built by the first pass and by the log lowering are not refined twice. Verification: hk/precise-math driver alone reproduced the abort on the 13-inch M1; with this commit the copy-offset tests passes 25/25 .
…0 disables)
Replace the five getenv("AGX_SIMDMAT") checks (agx_preprocess_nir lowering,
the extension table, the feature bit, the supported stages, and the
properties query) with agx_simdmat_enabled(), which reads the variable through
debug_get_bool_option with a default of true. Setting AGX_SIMDMAT=0 restores
the previous behaviour.
The 25/25 MLX battery and every reproducer ran with the gate on
(AGX_SIMDMAT=1), so the default now matches the verified configuration and a
system-installed driver exposes coopmat to unmodified wheels.
… path (G13X)
hk_dispatch_with_usc_launch flushed the full CDM barrier after every launch
unconditionally, ~4.6-5.1 us per dispatch on compute streams of hundreds of
small dependent dispatches. Key the barrier on what the app declared instead:
vkCmdPipelineBarrier2 marks a pending dependency and the next CDM launch
carries the barrier and clears it. Launches the app left unordered have no
Vulkan-defined ordering to enforce, so their drain is skipped. The flag lives
on the command buffer so it survives CS splits and merge_control_streams;
hk_EndCommandBuffer lands an unconsumed barrier on the trailing edge, and
driver-internal ordering (query availability writes, hk_queue_write) keeps its
explicit flush. The compute batch also stays open across a compute-to-compute
vkCmdPipelineBarrier2 when no graphics batch is open.
On G13X the barrier carries the measured designed set {unk_4,5,6,8} + USC
cache invalidate plus bits 0-2 (0x17f). Bit bisection showed every one of the
designed bits is required for bit-exact results: removing any corrupts, and a
USC-invalidate-only barrier is fast but wrong (320/320 greedy token flips).
G13G/G14 keep the previous kitchen-sink barrier.
New perftests: HK_PERFTEST=alwayscdmbarrier (restore the unconditional flush),
nocdmbarrier, usccdmbarrier, designedusccdmbarrier, and HK_CDM_BARRIER_MASK
(exact flag word) for bisecting.
Measured on an M1 Max (G13C), MLX-on-Vulkan Qwen3.5-class 4-bit greedy decode:
the designed-set trim 63.50 -> 68.34 tok/s (10 interleaved paired reps, +7.7%,
identical token digests); with the app declaring only real dependencies
(MLX dependency gating on) the tracked barrier adds +2.9% (81.3 -> 83.7 tok/s),
and 16 interleaved runs of the 10-prompt x 10-pass greedy contract produced
identical token digests in both arms.
Launches inside a meta operation set the pending flag themselves: an image
copy with several regions launches one dispatch per region and the app declares
no barrier between them, so each launch must still order the next one.
A device that created a sparse image or buffer keeps the full CDM barrier word
(bits 0-19). With the reduced word the sparse-image atomic cases of
image.atomic_operations fail and the GPU faults and times out on G13C (141-case
window of the image group: 3 failures and timeouts in 3 of 3 runs; 0 with the
full word in 3 of 3 runs, as on the pre-PR driver).
…_US) HK_SUBMIT_POLL_US=<us> (default 0 = off) makes vk_drm_syncobj_wait_many poll with a zero timeout for up to that long before falling back to the blocking timeline wait with the original deadline. Short submit->signal round trips (one submit + one wait per step of a stepwise decoder loop) pay a host wake-up premium under the sleeping wait; the poll observes the signal the moment the kernel publishes it. After a budget expires without seeing the signal the poll is disabled for budget << misses (capped at 1024x), so long waits (shader cache compile storms, whole-frame jobs) pay at most one budget per backoff period; any poll hit resets the state. Semantics preserved: same ioctl with timeout 0 during the poll budget, original absolute deadline on the fallback, ETIME still maps to VK_TIMEOUT, non-ETIME errors surface immediately, and drivers that do not set sync_wait_poll_us take the identical single-call path. Measured on an M1 Max (G13C): a speech-recognition decoder loop (278 submit+wait round trips) 341 -> 254 ms (-25%) at HK_SUBMIT_POLL_US=2000 (poll budget sweep 0/100/200/500/1000/2000/5000 us: 347/314/292/266/258/248/ 250 ms), transcript unchanged; LLM decode and prefill throughput and token digests unchanged.
The G13 matrix tile spans all 32 lanes of a SIMD group, so the hardware lowering is only correct when every subgroup of the dispatch is fully populated. Any workgroup whose size is not a multiple of 32 took it anyway and returned garbage: local size 16, 8 or 1 got 62 to 64 of 64 elements wrong, and those are valid shaders. Route them to a software lowering instead. Every active invocation keeps a private element array, but only the slots it owns are authoritative: logical element e is owned by invocation e % A, where A is the active invocation count of this subgroup, so partial tail subgroups and runtime-variable workgroup sizes are handled. Tangled operations (load, store, muladd, conversions, arithmetic) run on every lane with uniform bounds and gather non-owned source elements from their owner; untangled ones (insert, extract, copy, bitcast, length) touch local slots only and stay valid on a subset of lanes, per the tangled-instruction list in SPV_KHR_cooperative_matrix rev 10. A valid shader is never rejected. Reported in review. The comments state the default-on switch that this series already has (AGX_SIMDMAT=0 disables it), with the wording of "honeykrisp: restore cooperative matrices on by default".
Loading or storing a matrix through a uint16_t or uint8_t pointer got 56 of 64 elements wrong. The lowering stepped the strided axis on the buffer's own vector type and the contiguous axis on the matrix component type, which only coincides when the two widths match. Count the row in pointer elements, convert to component steps, and index one component-typed deref. The division is by a constant and is exact whenever the row stride is component-aligned, which the cooperative matrix memory alignment rules require; for width-matched buffers it folds back to the previous arithmetic. This also removes AGX_HWMAT_NO_FLATADDR. It selected the two-level address form, which is the form that gets narrow pointers wrong, so it now only names a broken path. Reported in review. The header comment states the default-on switch, as in "honeykrisp: restore cooperative matrices on by default".
The sin zero-return selects the argument when it compares equal to zero. The comparison flushes, so a subnormal argument matches and leaks out unflushed through the select. From flushed sources (in-shader arithmetic) the argument is already zero and nothing changes; from unflushed sources (buffer loads) a shader declaring DenormFlushToZero must see a flushed, sign-preserving zero, which the hardware fmul path it sits behind does not give it. Gate the fix on the DenormFlushToZero fp32 execution mode alone. The CTS float_controls sin cases are all DenormFlushToZero (optionally with SignedZeroInfNanPreserve on other operands); SignedZeroInfNanPreserve alone never exercises sin and preserving the subnormal argument is correct under DenormPreserve and under no mode at all, so those modes keep the existing emission byte-identical. Division is deliberately left unkeyed: the CTS fp32 group exercises OpFDiv only under DenormFlushToZero with in-shader-flushed denorm inputs (expectations V_ZERO/V_UNUSED) and has no DenormPreserve or rounding-mode division cases, and the previously-landed exact sequence flushes the subnormal-result band itself (it is FTZ-by-construction), so no existing sequence satisfies DenormPreserve there and none is required by any current CTS case. Pinned by test-lower-math.cpp: the default-mode fdiv/frcp op histograms are exact pins of the sequences the MLX decode digests were measured against, and the sin case distinguishes the keyed FTZ emission.
…witches
AGX_SIMDMAT, AGX_DECODE_Q4 and AGX_HWMAT_VEC2 change the code that the
compiler emits. hk_physical_device_compiler_flags returned only the
debug flags, so a changed switch reused shaders from the disk cache and
from application pipeline caches that the other setting compiled.
Put the three switches in the flags. The flags feed pipelineCacheUUID,
shaderBinaryUUID and the disk cache key.
From two earlier commits (keep cooperative matrices opt-in and key the compiler switches; drop AGX_HWMAT_NO_FLATADDR from the cache key). AGX_HWMAT_VEC2
uses bit 34 here, because the AGX_HWMAT_NO_FLATADDR switch that had bit
34 is gone ("agx: address cooperative matrix memory in component units").
"honeykrisp: dependency-track the per-launch CDM barrier on the Vulkan path (G13X)" replaced the norobust entry of hk_perf_options with the CDM barrier entries. HK_PERF(dev, NOROBUST) is still read in hk_device.c and hk_shader.c, so HK_PERFTEST=norobust stopped working without an error. Put the entry back. Fixes: the commit "honeykrisp: dependency-track the per-launch CDM barrier on the Vulkan path (G13X)"
The software lowering from "agx: lower cooperative matrices in software outside full subgroups" computes the memory index from the 32-bit SPIR-V stride. A PhysicalStorageBuffer pointer is 64-bit. Thus a cooperative matrix load or store through a buffer_reference, in a workgroup that is not a multiple of 32, failed the index width assertion in nir_build_deref_ptr_as_array. A release build has no assertion, and got an index that is narrower than the pointer. Convert the row, the stride and the column to the pointer width before the address arithmetic, as agx_nir_lower_simdmat.c does for the hardware path. For a 32-bit pointer the conversions do nothing. test-lower-cmat.cpp loads and stores an 8x8 matrix through a 64-bit pointer. Before this change it stops at the assertion.
Follow-up to "agx: keep the mod-32 shift count when fusing ushr+iand into ubfe", addressing a review note.
That commit fixed the miscompile by masking the offset, but kept
ubitfield_extract as the fused form. An out-of-range offset is undefined
for that opcode, so a later NIR pass is entitled to treat the value as
poison and optimise on that basis. The explicit iand makes the current
code correct without making the representation correct: nothing stops a
future pass from reasoning about the undefined case.
ubfe is the SM5 form, defined to take both its offset and its width
modulo 32, which is exactly ushr's behaviour and exactly what this fusion
means. Fusing to it removes the undefined case rather than defending
against it.
The mask stays, because the hardware is the part that does not wrap:
bfeil reads zero for an offset >= 32. It costs nothing, as it CSEs with
the mask the shader had before the fusion.
has_bfe is left unadvertised, so NIR still never forms ubfe on its own;
agx_nir_fuse_algebraic_late remains the only producer, and instruction
selection maps it to bfeil. Unlike ubitfield_extract, a width of zero
reads as zero rather than meaning "all 32 bits", so it gets its own case
rather than sharing the existing one.
Generated code is unchanged. NIR for byte_dyn_const, fused form only:
before: %13 = iand %11, 0x1f
%14 = ubitfield_extract 0x10001, %13, 8
after: %13 = iand %11, 0x1f
%14 = ubfe 0x10001, %13, 8
both: and $r2, r2, 31
bfeil r2, 0, u8, r2, 8
Verification on Apple M1 (G13G B1), this branch built as
Mesa 26.3.0-devel, against the same tree with this commit reverted:
20 of 20 reproducer variants PASS on both, 0 mismatches.
All 20 produce byte-identical disassembly on both, so this cannot
move performance: it is the same machine code.
MLX suites on this commit (AGX_SIMDMAT=1), identical to the
earlier run: primitive 98/98 (2699373 assertions), reduce_ops 34/34
(7152), runtime 40/40 (6307), indexing_ops 53/53 (4093), select_layout
11/11 (1908), scatter_determinism 21/21 (127).
simd_matrix_fmadd16/32 fixed bits 0 through 7, but bit 7 is the first bit of the destination operand, where it carries the register cache hint. Any matrix instruction with a cached destination therefore failed to decode, which trips the compiler's own disassembly selftest on a debug build. Match the seven opcode bits and let the operand own bit 7. Add three cases: both opcodes with a plain destination, and fmadd16 with a cached one.
Token-digest parity of the cooperative matrix path against the reference driver is measured on M1 (G13G) and M1 Max (G13X) only. The extension was advertised on every chip, so an M2 (G14) also got it by default. Advertise VK_KHR_cooperative_matrix by default on G13 only. AGX_SIMDMAT=1 enables it on other chips, for the digest run that qualifies them. AGX_SIMDMAT=0 still disables it everywhere. The compiler lowering and the shader cache key do not change, because a shader can only use the lowering when the extension is enabled.
The dependency-tracked per-launch CDM barrier holds token-digest parity on M1 Max (G13X). No other chip has a measurement, but the barrier had no chip check. On chips other than G13, set HK_PERF_ALWAYSCDMBARRIER by default. HK_PERFTEST=trackcdmbarrier removes that default, for the digest run that qualifies such a chip. HK_PERF_ALWAYSCDMBARRIER now restores all of the old behaviour: vkCmdPipelineBarrier2 ends the compute and graphics batches again and sets no pending barrier. Before, it kept the compute batch open and added a barrier at the end of the command buffer. Thus G14 runs the same command streams as before "honeykrisp: dependency-track the per-launch CDM barrier on the Vulkan path (G13X)".
The "honeykrisp: dependency-track the per-launch CDM barrier on the Vulkan path (G13X)" message and the PR description say that G13G keeps the kitchen-sink CDM barrier. The code also trimmed the G13G barrier to bits 4-8, from an earlier series. That trim has a measurement on M1 (t8103) with a barrier after every launch only. The G13G barrier is now dependency-tracked. On G13X the tracked barrier needs bits 0-2 and the USC cache invalidate, and the G13G set has neither. Remove the G13G trim, so that the code agrees with the description. A G13G token-digest run with the tracked barrier can add a trim again. Only the Vulkan driver emits agx_cdm_barrier, so GL does not change.
AGX_HWMAT_VEC2_OFF=1 disables the paired loads. On the 13-inch M1: LLM prefill is 9.4% faster and bit-exact (the prefill digest is unchanged); decode and speech-recognition throughput are unchanged.
… index Constant deltas between shared-memory addresses are folded into the lload/lstore index immediate, grouped per block. The base stays a real in-range address, so the non-wrapping hardware index add is safe. AGX_LOCAL_FOLD_OFF=1 disables the fold. On the 13-inch M1: prefill is 1.8% faster on top of paired fragment loads, bit-exact; Parakeet and digests are unchanged.
The scheduler serializes the memory round trips of small kernels. AGX_SCHED_COMPUTE=1 restores it. On the 13-inch M1: Qwen decode is 1.4% to 1.8% faster, bit-exact; prefill and Parakeet are unchanged.
The three commits before this one change compute code generation by default: paired fragment loads in the cooperative matrix lowering, the shared-memory offset fold and no register-pressure scheduler for compute shaders. Their token-digest evidence is from M1 (G13G). Paired loads also hold on M1 Max (G13X). Add agx_device_key.g13 and apply the offset fold and the scheduler skip only when it is set. Paired loads need cooperative matrices, which are already on by default on G13 only. Put AGX_HWMAT_VEC2_OFF, AGX_LOCAL_FOLD_OFF and AGX_SCHED_COMPUTE in the shader cache key. AGX_HWMAT_VEC2 has no effect after the paired-load commit, so it leaves the key. Also fix the indentation and the stale opt-in comment of the paired-load commit.
The deferred CDM flush comes from "hk: HK_CDM_DEFER_FLUSH=1 emits the CDM barrier at the end of a compute batch instead of after every launch (default off)" on the M1 lineage of the fork. That commit starts from the per-launch barrier. It records the barrier as pending after each launch and emits it when the compute batch ends: at a Vulkan barrier, query, event or the end of the command buffer. On top of the dependency-tracked barrier, the deferred mode works the same way. vkCmdPipelineBarrier2 ends the compute batch, as before "honeykrisp: dependency-track the per-launch CDM barrier on the Vulkan path (G13X)", and hk_cs_finish_pending_flush emits the barrier there. On G13G the batch-end barrier has bits 4-8, the set of the measured configuration. That configuration is the driver on the M1 test host. It holds the token digests of the per-launch barrier over long streams and repeated runs, and gives +0.9% to +1.4% decode. On M1 Max (G13X) the deferred flush lost digest parity and was slower, so it is on by default on G13G only. HK_CDM_DEFER_FLUSH=0 selects the dependency-tracked barrier, HK_CDM_DEFER_FLUSH=1 enables the deferred mode on other chips, and HK_PERFTEST=alwayscdmbarrier overrides both.
Token digests are bit-exact against the deployed reference driver on an M2 Max (G14X) with cooperative matrices on, in 4 arms x 3 alternating runs. Prefill at 512 tokens goes from 160.1 to 901.9 tok/s and dispatch counts match the reference (1087). G14G has no gate run yet and stays opt-in through AGX_SIMDMAT=1.
…ction step Stress testing on an M2 Pro found a third division failure class the original review missed (2026-09-13): when the exact quotient sits just below the normal boundary, the GPU zeroes the correction step's intermediate value and the Markstein step is silently skipped, giving 1 ulp errors -- 18,382 wrong results over 1M random pairs. The first commit in this series refined a * rcp(b) in place, so the flush landed on an intermediate, not on the result. Divide the normalized mantissas instead and restore the exponent afterwards, so the refinement always runs on operands near one and the flush happens on the exact quotient, before rounding. The rounding decision is integer: the half-ulp point is exact by construction, and the residual of the second Markstein step is exact by the fma contract, so the comparison against it decides the tie without a third refinement. The reciprocal takes the same path, and the log2 series divisor (near two) keeps the unguarded Newton form. This re-lands the division part of the fix first made on 2026-09-13 and pulled with it during the 2026-09-20 A/B isolation windows; it was never re-applied to the omacom/mesa stack. Verified against an exact model over 262,144 pairs covering edge bit patterns and the binade boundaries, and by an independent exact-rational checker; see the receipt in the PR.
nir_builder_opcodes_h.py renders the intrinsic builders from a mako template that references FP_MATH_CTRL, but the index was never imported or passed to render(), so mako resolved the name to UNDEFINED and silently skipped the builder-defaulting block for every FP_MATH_CTRL intrinsic. NoContraction on cooperative-matrix operations was therefore dropped in every driver, not just this one, and the generated header is missing the twelve fp_math_ctrl defaulting blocks. Import the index and hand it to the template. The generated header grows exactly 24 lines, twelve two-line defaulting blocks.
…le-cluster G13 The cooperative-matrix loads reach agx_nir_lower_address as ulea(base, byte_offset, 0). The pass only matched shifts at or above the format shift, so every such load fell back to a materialized 64-bit address with index 0. Index them by the scaled offset instead. Measured per device (bit-identical output on both): dense f16 GEMM +15.1 % on the M1 (T8103, single cluster), -13..17 % on the M1 Max (T6001), so a new device-key bit enables it only for single-cluster, single-die G13.
LowerAddress checks the arithmetic of the fold added in the previous commit: a load addressed by ulea/ilea(base, byte_offset, 0) keeps the base and takes the byte offset shifted down by the format shift as its index (f32 24 -> 6, f16 24 -> 12, signed ilea -8 -> -2); without the fold_subformat_address key bit the 64-bit sum is materialized and the index is 0. Per-device result of the gated fold against its base, three rounds per cell, bit-identical output on both chips (GEMM output hashes and 21 MLX matmul cases; on the M1 Max also the Qwen3.8-2B 4-bit prefill logits and 32 greedy tokens): - M1 (T8103, fold on): dense f16 a @ b 4096^3 1.669-1.695 -> 1.857-1.892 TFLOP/s (+11.5 %), MLX 4096^2 f16 +15.1 %, f32 +10.3 %, bf16 unchanged. - M1 Max (T6001, fold off): shader-db identical to the base, every cell within run-to-run spread.
DivDefaultFastSequencePinned and FrcpDefaultSequencePinned counted the ALU ops of the "hk: enable cooperative matrices by default on G14X" division and reciprocal. Every exact lowering since "agx: flush the exact quotient in precise fp32 division, not the correction step" emits other ops, so agx_tests has failed these three pins on the PR #6 head (found by w7Q, the driver's meson suite: 61/64). The bits of division and reciprocal are pinned on hardware by tests/precise_math.py, ulp_bound.py and div_window.py. What this file can pin is the float-control contract: DenormFlushToZero emits exactly the default lowering. Compare the two op histograms for fdiv and frcp instead of fixed counts.
"hk: order internal full-barrier precomp launches with surrounding CDM work" made every AGX_BARRIER_ALL precomp launch flush the CDM caches before it and again before the next launch. vkCmdCopyBuffer2, vkCmdUpdateBuffer and vkCmdFillBuffer passed AGX_BARRIER_ALL, so each of them paid two flushes even though the application already orders them with vkCmdPipelineBarrier, which sets cdm_barrier_pending. Launch them with AGX_BARRIER_NONE; internal draw-time launches keep the full barrier. Qwen3-4B 4-bit decode on G13C (16-inch M1 Max, 64 tokens): 55,694 -> 46,520 CDM flushes for 60,066 dispatches, 64.99 -> 66.63 tok/s, paired +1.61 tok/s in 5 of 5 rounds. Qwen3.5-2B: 37,376 -> 35,744 flushes, paired +0.41 in 9 of 10 rounds. Greedy digests unchanged at 2B, 4B and 9B.
…another block glslc -O output for llama.cpp's IQ grids declares the table as a function variable with the constant array as initializer and also stores that array into it in a later block. vk_spirv_to_nir lowers the initializer to stores at the start of the function, so the variable has constant stores in two blocks and the pass gave up on it. Such a store changes no byte of the collected data, so accept it; a store in another block that writes different data still disqualifies the variable.
Honeykrisp left nir_opt_large_constants commented out and load_constant unreachable, so nir_lower_vars_to_scratch gave every invocation its own scratch copy of any indirectly indexed constant array of 256 B or more. llama.cpp's IQ1_S/IQ2 grid tables hit this: 4-8 KiB of scratch per invocation, heavy spills, and IQ mat-vec 5-24x slower than with the data read once from memory. For compute shaders, run nir_opt_large_constants (256 B threshold, after nir_lower_var_copies and before agx_preprocess_nir) and read the resulting load_constant from a per-shader constant buffer whose address the root table carries. The load offset is clamped into the variable. HK_LARGE_CONSTANTS=0 restores the old lowering; it is part of the compiler-flags cache key. tests/large_constants.py checks both settings bitwise against a CPU model and checks that the default uses no scratch. G13C (M1 Max): llama-bench Llama-3.1-8B IQ2_M decode 0.95 -> 12.8 tok/s, prompt 512 124 -> 149 tok/s; test-backend-ops MUL_MAT 1684/1684 and MUL_MAT_ID 939/939 correct both ways. iq1_s mat-vec 6141 -> 2849 instructions, 255 -> 95 GPRs, 4096 B -> 0 scratch. MLX decode shaders (Qwen 2B/4B/9B) and greedy outputs are unchanged on G13C, G13G and G14C.
Honeykrisp advertised fp32 and fp16 cooperative-matrix A/B only, so bf16 GEMMs (MLX matmul_coopmat_bf16.comp) had to stage every tile through shared memory, widening bf16 to fp32 with ALU ops, instead of loading the matrices straight from the buffers like the fp16 and fp32 routes. Advertise VK_KHR_shader_bfloat16 (shaderBFloat16Type, and shaderBFloat16CooperativeMatrix with cooperative matrices) and an 8x8x8 and 16x16x16 bf16 A/B, fp32 C shape. bf16 fragments stay 16-bit patterns through load, store, extract, insert and bitcast; MulAdd widens them exactly to fp32 and runs the fp32 A/B simd_matrix_fmadd32 form. Conversions and element-wise ops on bf16 matrices go through fp32 in both the hardware and the software cooperative-matrix lowering. bf2f is a shift; f2bf rounds to nearest even in integer ALU, NaN quieted. G13C (M1 Max) gemm-bench, bf16 4096^3, staged MLX kernel -> direct bf16 cooperative matrices, outputs bit-identical on every element: a@b 2.40 -> 5.65 TFLOP/s; a@b.T 2.90 -> 5.88 (K_UNROLL=4, SWIZZLE_ROWS=8); a.T@b 2.05 -> 3.47. MLX decode greedy outputs unchanged.
…Max/Ultra class) The tracked barrier was default only on G13. On an M2 Max (G14C) with the v3 ICD, tracked vs per-launch barrier, same driver: - 2048-token greedy decode, 2 rounds: digests equal on Qwen3.5-9B, Qwen3-4B and Qwen3.5-2B. Decode tok/s 9B 24.2 -> 26.2, 4B 39.1 -> 41.3, 2B 49.6 -> 55.1. - 9B, 5 paired rounds, 256 tokens: +8.4%, 5 of 5 positive, digests equal. - CTS, 20,683 cases (bf16 + precision + cooperative_matrix list and the compute.pipeline cases): identical statuses in both modes (11,984 Pass, 8,699 NotSupported, 0 Fail). - precise_math and ulp_bound: 0 failures in both modes; div_window over all 2^32 pairs with tracking: 0 mismatches. - Built ICD with this change: default digest equals v3 with HK_PERFTEST=trackcdmbarrier and HK_PERFTEST=alwayscdmbarrier restores the per-launch numbers (4B, 3 rounds). Only gpu_generation == 14 is covered: AGX_CHIP_G14X also maps multi-cluster generation 15 and later parts, which stay on the per-launch barrier until they pass the same gates. Tested on M2 Max only.
… the read
OpImageWrite with MakeTexelAvailable and OpImageRead with MakeTexelVisible
describe the texel of that very access. vtn_handle_image split their
semantics with vtn_split_barrier_semantics, which places MakeAvailable
before the operation and MakeVisible after it. That is right for a release
or acquire atomic, but it put a texel write outside its own availability
operation. nir_lower_memory_model marks only the writes that precede a
MakeAvailable barrier as coherent, so the store stayed non-coherent.
On G13C (M1 Max) a devicecoherent image payload published across
workgroups then failed: dEQP-VK.memory_model.message_passing.ext
u32.coherent.*.{device,queuefamily}.payload_local.image.guard_local.*.comp
(30 cases, deterministic, 30 of 30 in 6 of 6 runs, with and without the
dependency-tracked CDM barrier). The failing and passing shader pairs had
identical barrier and cache-flush sequences; the only difference in the
disassembly was the coherent flag on the image write. Four cache flush
variants did not change the result.
Swap the two sides for ImageRead, ImageSparseRead and ImageWrite. These
carry no Acquire or Release bits, so only the MakeTexel semantics move.
Atomics keep the old placement.
Add opimageread_vis and opimagewrite_avail to the availability and
visibility unit tests. They assert the order, which the existing tests
do not: the available barrier follows the image store and the visible
barrier precedes the image load. Both fail on the parent commit
(barrier position 10 vs load 9, and barrier 14 vs store 15) and pass with
this change; the 27 spirv_tests pass.
Results on G13C (aurora 12.6 prerelease kernel), same driver with and
without the change:
- the 30 cases: Pass in 3 of 3 runs
- memory_model.* and subgroups.basic.* (18,600 cases): 2,283 Pass,
16,317 NotSupported, 0 Fail; 30 Fail -> Pass, 0 Pass -> other
- the 20,683-case bf16, precision, cooperative_matrix and
compute.pipeline list: 0 status changes
Same lists on G13G (M1) and G14C (M2 Max) with and without the change:
the 30 cases pass in 3 of 3 runs, the memory model and subgroup list has
0 status changes (these chips already passed the 30 cases without the
change), and the 20,683-case list has 0 status changes.
… by default for G13 Compute shaders on G13 skip the register-pressure scheduler, so each device load stays right before its first use and its latency is exposed. Add a pass, agx_hoist_loads, that moves each non-coherent device_load up to 64 instructions earlier, to just after the last producer of its sources. It crosses pure ALU (class none, reorderable and eliminable), threadgroup barriers, workgroup memory loads and stores, other non-coherent device loads, and a uniform_store whose 16-halfword window does not overlap any uniform source of the load (an index that is not an immediate is never crossed). It stops at device stores, atomics, memory barriers, coherent and volatile loads, image operations, doorbell, stack_map and similar non-pure instructions, phis, preloads and control flow. The pass is on by default for gpu_generation 13, opt-in on generation 14 with AGX_HOIST_LOADS=1 and never enabled on other generations; AGX_HOIST_LOADS=0 turns it off everywhere. The setting is read for every shader compile in the Vulkan driver and travels in agx_device_key. Unit tests (HoistLoads, 10 cases): a load moves above ALU and a threadgroup barrier and above a uniform_store at a disjoint index or an adjacent one; it does not move above its own source, a device store, a memory barrier, a doorbell, a uniform_store to its own base or a partially overlapping index, and a coherent load stays. Two of the cases fail with a rule that admits every class none opcode, which is the first version of this pass. Measured with the pass on against the pass off (same driver build for each pair), kernel 12.6 prerelease. On the final rule (build "hoist05"): - G13C (M1 Max): decode, 256 tokens, 5 rounds, digests equal: Qwen3-4B +5.2% (5 of 5 rounds), Qwen3.5-4bit +6.2% (5 of 5); 2048 tokens Qwen3.5-4bit +4.6% (2 of 2). - G13G (M1): 2048 tokens, digests equal: Qwen3.5-4bit +2.0% (2 of 2), Qwen3-4B +1.0% (2 of 2). - Bit identity on both chips: the 450-cell dense matmul shape sweep (450 of 450 equal), the batch-row sweep (no row dependence, none wrong), eight MLX test suites (identical counts in both arms). - Registers and spills of a 16-token decode run: no shader gains a register, none spills (max 189 on G13C, 177 on G13G). - CTS, 12,000 cases sampled from synchronization, synchronization2, subgroups and compute shader_object: identical statuses on both chips. - test-backend-ops perf, six interleaved passes, ROPE, CPY, SOFT_MAX: no case slower than -3% on G13C; one of 60 ROPE cases at -5.6% on G13G. On earlier builds of the pass (the same rule before the pure-operation and uniform_store changes; not repeated on the final one): memory_model and subgroups.basic (18,600 cases) and a 20,683-case list with identical statuses on G13C, G13G and G14C; precise_math and ulp_bound with 0 failures; llama.cpp Qwen3-1.7B and Qwen3-8B decode on G13C +7.9% and +1.7%. The setting is part of hk_physical_device_compiler_flags(), so it is in the pipeline cache UUID and the disk cache key: AGX_HOIST_LOADS=0 never gets a shader cached with the pass on, and the reverse. Every on/off comparison above ran with MESA_SHADER_CACHE_DISABLE set in both arms.
The heap was 50% of MemTotal. On a 16 GB M1 that is 7.55 GiB, so a model of about 8.2 GB cannot be loaded while 13 GB of RAM are free. The GPU shares system memory, so the heap is only a cap; the next commit keeps a free-memory reserve, which is what protects the desktop. This change depends on it: with the reserve, a larger heap cannot take the last pages. Apply the two together. heap_memory_percent in ~/.drirc still overrides the default per application. Dedicated inference hosts can raise it; one process is also bounded by the GPU address window of the chip. Measured on a 16 GB M1 (G13G) with the reserve, 256 MiB host-visible chunks filled with random bytes, test processes in a cgroup with no swap, desktop running: - 0.85 (heap 12.84 GiB) and 0.90 (13.60 GiB), one process: refused with VK_ERROR_OUT_OF_DEVICE_MEMORY at 11.00 and 11.25 GiB, MemAvailable 1.25 and 1.05 GiB at the refusal. - two processes at once, same settings: both refused at 5.50 GiB, MemAvailable 1.24 GiB. No OOM kill, same compositor pid, sampled MemAvailable never below 1.05 GiB. - Without the reserve, two processes at the old 50% default took MemAvailable to 0 kB and the kernel OOM killer killed the compositor session. - heapBudget follows free memory: 8.86 GiB idle, 6.15 GiB with 4 GiB held by another process, 3.38 GiB with 8 GiB held (heap at 0.75).
heapBudget is advisory and vkAllocateMemory only checked the static heap size. GPU memory cannot be evicted or swapped, so processes that each fit in the heap can take every free page: on a 16 GB M1 two processes filling their heaps (7.55 GiB each at the old default) drove MemAvailable to 0 kB and the kernel OOM killer killed quickshell and the Hyprland unit. Read MemAvailable before creating the buffer object and return VK_ERROR_OUT_OF_DEVICE_MEMORY when an allocation of 1 MiB or more would leave less than max(1 GiB, 2% of RAM) available. Smaller allocations are not checked: below the reserve every vkAllocateMemory would fail, including the compositor's own. Imported memory is not checked, it allocates nothing. If /proc/meminfo is unreadable only the heap limit applies. Concurrent allocators can still overshoot by one allocation each. Allocations are checked from 1 MiB so that per-tensor allocators (tens of MiB per call) cannot walk past the reserve one small step at a time.
Sparse buffers map every page twice, read-write at X and read-only at X + sparse_ro_offset, and the two halves are told apart by one address bit. The layout code rounded the user window to powers of two and then halved it, twice when the start equalled the size. The kernel reports a 512 GiB window (vm_end 0x7fffff8000) on G13C, G13G and G14C; the heap began at 128 GiB and was 64 GiB long. A single process could not map more than 64 GiB of device memory whatever heap_memory_percent said: at 0.95 on a 94 GiB M2 Max vkAllocateMemory returned VK_ERROR_OUT_OF_DEVICE_MEMORY at 63.75 GiB with 25 GiB of RAM free and the driver printed "Failed to allocate BO VMA". agx_sparse_layout() instead picks the power of two that leaves the largest heap with its read-only shadow still under the kernel window, and the heap now starts right after the shader heap instead of at the next power of two. With the window above: heap at 72 GiB, offset 256 GiB, 152 GiB long. Unit tests cover the layout rule.
Freed device memory is parked in the driver's BO cache and still counts as used in MemAvailable. The cache is trimmed only when another BO is freed (entries older than 2 s) or when creating a BO fails (agx_bo_cache_evict_all in agx_bo_create). The reserve check runs before agx_bo_create, so after an application freed gigabytes the check saw almost no free memory and refused every allocation while the memory sat in the cache: on a 16 GB M1, an MLX process that hit the reserve at 11.75 GiB, freed everything and cleared its own cache could not allocate 1.5 GiB for about 2 s (tiny submissions for 8 s released nothing; the next allocation attempt after one failed bo create succeeded at once and MemAvailable went from 1.05 to 10.75 GiB). When the check would refuse, call agx_bo_cache_evict_all() and read MemAvailable again before returning VK_ERROR_OUT_OF_DEVICE_MEMORY.
… meta work The dependency-tracked CDM barrier flushes only after the app declares a memory dependency with vkCmdPipelineBarrier. The consumers of an internal launch in a render pass (the tessellation and geometry precomps of an indirect draw), or of an image copy, are draws and copies that the app has no barrier for, so nothing flushed their inputs. Once a command buffer begins rendering, starts a meta operation, or launches a precomp that feeds or follows a draw, stop tracking: flush after every launch, and flush a launch left pending by the tracked path first. Command buffers that only dispatch compute (MLX decode) keep the tracking. On G13C, robustness.index_access.draw_indexed_pipeline_robustness_2_vert_tess_frag failed with the tracked barrier and passes with this change. On G13G the draw_indexed_indirect_count tessellation cases of the same group hung the GPU (Job timed out on the DRM scheduler) in a 100-case chunk; with this change the chunk completes three times in a row with the same results as the pre-PR driver.
run_matrix2.sh, precise_math.py, large_constants.py, div_window.py and ulp_bound.py serialized their GPU runs with flock on a lock file name from the machine they were written on. Use /tmp/gpu.lock. No driver file changes.
The reduced G13X barrier word is enough for a compute-only stream. It is not enough after this command buffer records graphics or meta. Query copies and depth suballocation reads need the PBE and texture flush. Those 48 cases pass on pre-PR main and fail on this head, on the 16-inch M1 Max only. The 13-inch M1 uses the full word already and does not fail them. After graphics or meta, hk_cdm_cache_flush writes the full word (bits 0-19). Compute-only streams keep the reduced word.
The full G13X barrier word did not fix the 48 failures. The copy reports occlusion result 0. That value is written by the graphics batch. A CDM cache flush does not wait for it. Close the graphics and compute batches before every copy, not only for timestamp pools.
This reverts commit a5a7723.
This reverts commit 5c8d6b9.
…s the tail stream Control streams are submitted in list order, and each is ordered after every stream listed before it. Since the dependency-tracked barrier, a pipeline barrier with no open graphics batch returns and leaves the compute batch open. If that compute stream is older than a graphics or post-graphics stream recorded after it, the work recorded after the barrier lands in the older stream and runs first. Two shapes hit this on the 16-inch M1 Max. A clear dispatch opens a compute stream, a resumed render pass adds a graphics stream, and the verify dispatch after the barrier joins the older stream (six dynamic_rendering low_resolution_z cases). A multiview timestamp opens a zero-queries stream, EndQuery adds a post-graphics stream, and the query copy after the barrier joins the older stream (42 multiview copy_query_pool_results cases). End the compute batch unless it is the last stream. Compute-only command buffers keep the tracking. Tested: commit 5944609, which has the same source change on the previous head 69bde15. On the 16-inch M1 Max, 48 of 48 of these cases pass with it, 48 of 48 fail on 69bde15, and 48 of 48 pass on pre-PR main built on the same machine. Not run: this exact commit. Not measured: other cases, speed, other chips. Compute launches that join an older stream without a barrier call are not covered.
The pass that moves constant deltas between workgroup memory addresses into the lload and lstore index ran on G13 only. Honeykrisp now runs it on G13 and G14. Other generations and the Gallium driver keep their current behaviour. A new device key field, fold_shared_offsets, replaces the g13 test for this pass. agx_gather_device_key sets it on G13, so every current user keeps its behaviour. Honeykrisp also sets it on G14. AGX_LOCAL_FOLD_OFF still turns the pass off. An empty block in the pass is removed, and the comment on the index unit now names the G14 evidence. The checks below were made on a 14-inch M2 Max (G14C) with a private build of the same compiler source with the g13 test removed, not with this exact commit. The pull request lists what has been run on this commit. The two cooperative matrix kernels of a 4-bit 9B prefill give the same bf16 words as without the pass for five shapes, and 6239 named shared memory and cooperative matrix conformance cases give the same result with the pass on and off (3613 Pass, 2626 NotSupported, no failures).
joshuaswarren
force-pushed
the
agx-fold-g14
branch
from
October 15, 2026 10:09
f5f4c55 to
4484120
Compare
Author
Author
Author
|
Status, 3 Nov 2026 (UTC): still a draft, no change to head Same note as on #6: its base is |
Author
|
Retarget status, 3 Nov 2026 (UTC): this draft is still based on @maralcbr: the question on #6 applies to this PR too. If you want #6 on |
Author
|
Status 2026-11-05 (w7K): still draft, not ready. This stacks on #6. #6 is not on integration/aurora: its head has 67 commits the base lacks, and the branches have diverged (the base is 1979 commits ahead). Porting #6 means a large rebase, which changes the head the frozen comparison at 9323171 covers, so that qualification would have to be redone. Nothing was rewritten and the base of this PR is unchanged. Next: decide the port with the maintainers before any rebase. Scope: status only, no build or device run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOT READY TO MERGE
Head:
44841202b4fd041fc29b6ff77f585d4df6e70dd7.GitHub reports MERGEABLE against
main: no text conflicts. This is not a test pass.This change depends on #6 and stays in draft. The G14 results below keep their stated source and build bindings; they do not qualify a future restack. Finish #6 qualification first. Update this stack only if needed, then review and test that exact candidate before marking it ready.
Earlier review, source and test notes are kept below with their original bindings. They do not change this status.
Status: NOT READY TO MERGE (draft)
The bits check and the broader conformance run on the final library are done on one 14-inch M2 Max (see Results). This change is NOT ready to merge because it is stacked on #6: the pass it enables is added by a commit in #6, so the branch holds the 61 commits of #6 plus one, and #6 is not ready (its full conformance suite has not been run on its current head). Every number below names the build it was measured on. Review only the last commit,
44841202b4fd.What this changes
The pass
agx_fold_shared_offsetsmoves constant differences between workgroup memory addresses into the index field oflloadandlstore. Today it runs on G13 only. This commit runs it on G13 and G14.fold_shared_offsets.agx_gather_device_keysets it on G13 only, so every current user of the compiler, including the Gallium driver, keeps its behaviour.hk_shader.calso sets it on G14.AGX_LOCAL_FOLD_OFFstill turns the pass off.The diff is 4 files, +18 -5.
Why
On a 14-inch M2 Max, the 4-bit matrix multiply kernel of a 9B model prefill (32 by 32 tile, workgroup memory for one operand) compiles from 964 to 922 instructions and from 127 to 107 registers with the pass on (build A, see Results). The matrix multiply-add count (64), the workgroup memory loads and stores and the barriers do not change; only address adds and moves go away.
Results
All G14 numbers below are from one 14-inch M2 Max and from build A: the compiler source of #6 with a one line patch (
key->dev.g13removed from the test), driver library sha25604005ccf00c5a8cf. Build B is the source tree of this commit (treee519ad9805b5; it was built from an earlier commit with the same tree and a different message,f5f4c55e57cb), driver library sha256d8cadfcd45dcfbcb. Build B differs from build A by the device key field, so nothing in the table is a measurement of build B yet. Compile only output of build B is in the next paragraph. The bits check and the conformance run on build B are in the paragraphs after the table. No speed number is claimed for build B.Build B, compile only, same four kernels, pass on and pass off, one 14-inch M2 Max: the final instruction text is identical to build A in all 8 cases (0 differing lines), and with the pass off the two cooperative matrix kernels are identical to the installed driver. The two dense kernels are identical between build A and build B, with the pass on and off. So build B generates the same code as build A for these kernels. This paragraph is a compile only check: it says the code is the same, not that the speed or the results are the same.
Build B, same bits check, same five shapes, one 14-inch M2 Max, boot after a recovery reboot: installed driver (two runs), build B with the pass off, build B with the pass on, four separate processes, every output word equal in all four, and byte equal to the outputs of the build A run; the loaded driver library hash was checked in each process.
Build B, Vulkan CTS, 8619 named cases (the 6239 above, the other 126 compute pipeline cases, and 2254 cases of the families memory_model shared, message_passing, write_after_read and padding, and subgroups basic compute, all of which passed in an earlier run), pass off against pass on, same boot, same ICD and library in both: 8619 of 8619 equal; 5973 Pass in both arms, which is the sum of the Pass counts of an earlier run over 136 groups, and every group reaches its count; the rest are NotSupported in both arms; 0 Fail, 0 Crash, 0 Timeout. 52 names sit in groups with an earlier Pass count of 0 (44 cooperative matrix nv names and 8 other compute pipeline names, NotSupported before and now), so they cannot show the pass is right. The log of every chunk names the build B library and not the installed one.
Not covered by the conformance list: shader_object, task and mesh shaders (no earlier results, or NotSupported only), graphics stages, and any cooperative matrix case beyond the 114 that pass.
Limits