[RyanTheTide]

G17 recycles firmware-visible identities straight away, lowest first: a new logical queue gets the lowest free application context (TTBAT id) and QoS ID, and a new context gets the first matching work-state, descriptor or queue object from the object cache, zeroed. The firmware keys work by (context id, state word), and installed queue records carry the context id and QoS ID. An installed compute queue keeps its record after its owner exits, but the context id goes back to the allocator as soon as the scheduler release is consumed, so the next queue created can take an id that a record still names.

With mesa#184 v1 (eight eager kernel queues per VkDevice) the J616s hit this twice on fleet27:

aurora: G17: recovery MMU fault bank 0: address 0x7061df9800 write Unmapped unit Unknown(217) (0xd9) level 0 vm_slot 15
aurora 28a600000.gpu: RTKit: co-processor has crashed  (EL1t, ESR 0x96000044, FAR 0x0, x0 = work-state object + 0xf40)

The first is a compute-side write through context 15 into the kernel lower arena, with no tracked job. The second is the firmware appending to a work list through a node whose links read zero, which is what a freshly recycled work-state object looks like. Evidence: traces/gpu/20261108-j616s-g17-capacity-j6am and 20261108-j616s-mesa184-bisect-gpux2 (S500).

Change

• Holds. A context and every retained queue record that names it hold its execution identity (mmu::ContextHold) and its QoS ID (Arc<QosLease>). While a hold remains, the identity is handed out only if no other context is free.
• Firmware fence. Once the firmware has seen an identity, the last hold fences it. It is reused only after the firmware has consumed a device-control record published after that point. Control records are consumed in order, so that record proves the firmware finished the release that retired the old owner. g17/channel.rs counts published records across firmware restarts, and every publication and service pass records how far the consumer got.
• Oldest first. Identities are reused oldest first, the same policy the VM slot allocator (slotalloc) already uses for G13 to G16. mmu::pick_returned is the shared helper. Application contexts and QoS IDs both use it.
• Object cache. The cache reuses the object returned longest ago once its fence is met. It reuses a younger one only when the family has no room for a fresh cached mapping, because an uncached mapping is unmapped as soon as its owner drops it.
• No new refusals. When every free context is held or fenced, the oldest is taken and counted. debugfs capacity gains execution_contexts_fenced free_waiting= overruns= and cached_objects_fenced waiting= overruns=.

Cost is a scan of at most 64 contexts, 128 QoS IDs or 64 cache slots at queue or context creation, plus one atomic add per control record. The submit path is unchanged.

Receipts

Test kernel 7.3.0-aurora-gpuk1 is fleet27 95e30d5 plus #1146 and this commit. Its config is identical to fleet27, and its GPU driver source is byte-identical to this PR's head (integration's driver equals fleet27 plus #1146).

J616s (M4 Pro), one-shot on boot d4daba0c, Enforcing, 0 AVC:

run	result
qchurn, stock Mesa, 4 clients x 30 devices x 3, all 8 queues used	12/12 OK, 0 fw crash / recovery / MMU fault / device lost
qchurn, mesa#184 v1 overlay (eager 8, 2 used), 4 x 30 x 3	12/12 OK, 0 incidents
tsem 4-shard with the mesa#184 v1 overlay (the 05:09 crash trigger), x2	2879 and 2880 / 3834 pass, 0 fw crash / recovery / MMU fault / device lost / watchdog. One shard per run ends at a clean vkCreateDevice OOM, as v1 does on fleet27 (GPUX2). 101 s and 88 s (stock fleet27 4-shard: 102 s)
heavy qchurn, 5 clients x 300 devices x 2, stock 8 queues and the v1 overlay	20/20 OK, 0 incidents

Under the heavy run (40 open queues plus 19 idle installed compute queues holding ids, against 58 contexts), contexts overran their hold 444 times and the object cache 448 times, both oldest first. The no-refusal fallback did its job. The debugfs counters show it.

M2 (G14), one-shot: no regression. qchurn 12/12. qual (nosuspend): GPU dmesg 0, AVC 0, warn/err classes 4/1 = baseline 4/1. glmark2 4159/5596/4681/4377 vs fleet27 4612/4549/4387/4696, within post-boot noise. The G13/G14 path does not run the changed code.

Host tests: tools/asahi/run-host-tests.py 195/195. g17-growth-tests stubs the new observe_control call. New KUnit asahi_mmu_execution_context covers the pick order and the fence and hold fallback. Driver compiles without warnings.

Still to do: a same-boot fleet27 baseline of the same qchurn and tsem rows on the J616s, which needs a slot after the morning audible run. The incidents never reproduced on demand (GPUX2: 0 in 9 runs), so the baseline is evidence, not a gate. Possible follow-up: release an idle compute queue's context id under id pressure, so heavy load needs no hold overrun.

Kit: kits/test/20261108-gpuk. Traces: traces/gpu/20261108-j616s-gpuk-fw-id-fence (S500).

🤖 Generated with Claude Code

[RyanTheTide]

Baseline row without this PR: J616s on its fleet27 default (7.3.0-aurora-fleet27, boot b701e6ec, up ~4 h with a GNOME session on seat0, no reboot), Enforcing, the same kit (kits/test/20261108-gpuk), 14:40-14:47.

run	fleet27 (no fix)	gpuk1 (fix)
qchurn stock Mesa, 4 x 30 x 3, all 8 queues used	12/12 OK, 0 incidents	12/12 OK, 0 incidents
qchurn mesa#184 v1 overlay, 4 x 30 x 3	12/12 OK, 0 incidents	12/12 OK, 0 incidents
tsem 4-shard v1 overlay	x4: 2879 / 2880 / 1922 / 2880 of 3834 pass, 102 / 77 / 45 / 50 s; one clean OOM shard per run (two in run 3)	x2: 2879 / 2880, 101 / 88 s
tsem 6-shard v1 overlay	x2: 1923 / 1923, 31 / 55 s, three OOM shards each	not run
heavy qchurn 5 x 300 x 2, stock 8 and v1 overlay	20/20 OK, 0 incidents	20/20 OK, 0 incidents

0 firmware crashes, 0 G17 recoveries, 0 MMU faults, 0 device lost, 0 watchdog, 0 AVC on every fleet27 row. The debugfs capacity counter shows refusals site=LogicalQueues count=26933290, the refused-queue spin that #1146 removed. fleet27 does not have #1146.

So the race did not reproduce on demand without the fix either, which matches GPUX2's 0 in 9. The fix-vs-no-fix rows have no reproducing case to compare. The case for this PR is the code analysis plus the two fleet27 incidents (05:03 MMU fault vm_slot 15, 05:09 RTKit crash), not a reproducer. Timing on the tsem rows is the same within noise. The only behavioural difference the rows show is the fix's debugfs fence/overrun counters under heavy load.

Evidence: traces/gpu/20261108-j616s-gpuk-fw-id-fence/fleet27base (S500): seq/run logs, per-row kernel logs and capacity snapshots, full journalctl -k -b.

[RyanTheTide]

fleet30 rows on the J616s, with and without this PR. fleet30 is the new default and already contains #1146. The fixed kernel is 7.3.0-aurora-gpuk2: fleet30 50d5865 plus this PR's commit cherry-picked cleanly (b08bbf1d), built with the same recipe as gpuk1, config identical to fleet30. It ran as a guarded one-shot.

With the fix (gpuk2, boot 6518a211, 16:33-16:40):

run	result
qchurn stock 8 queues, 4 x 30 x 3	12/12 OK
qchurn mesa#184 v1 overlay, 4 x 30 x 3	12/12 OK
heavy qchurn 5 x 300 x 2, stock 8 and v1 overlay	20/20 OK
tsem 4-shard v1 overlay, x4	2879 / 2879 / 2880 / 2879 of 3834, one clean OOM shard each
tsem 6-shard v1 overlay, x2	1284 / 1284, four OOM shards each

0 firmware crashes, recoveries, MMU faults, device lost, watchdog hits and AVC. The debugfs fence counters after tsem6 read execution_contexts_fenced overruns=656 and cached_objects_fenced overruns=665, so the oldest-first fallback did real work.

Without the fix (fleet30 default, boot bf7b1fd3, right after): tsem 6-shard v1 x2 gave 6 and 1923 pass. In run 1 all six shards hit vkCreateDevice OOM at once. The tsem 4-shard v1 pass that followed crashed the GPU firmware at 16:44:17:

aurora 28a600000.gpu: RTKit: co-processor has crashed
  == Exception taken from EL1t ==  ESR = 0x96000044  FAR = 0x0  x00 = fffffc2003fe4f40

That is the 05:09 signature (EL1t, ESR 0x96000044, FAR 0, x0 at work-state object + 0xf40). All four shards segfaulted, with 4 device-lost and 4 EIO. A plain reboot to fleet30 brought the GPU back.

This is the first on-demand reproduction. The fleet27 baseline above never crashed, and neither did GPUX2's nine runs. The difference is probably #1146: on fleet27, refused queue creation spun (26.9M refusals in my baseline), which throttled the churn. fleet30 refuses without spinning, so ids recycle faster. On fleet30 the same tsem v1 load crashes the firmware without the fix, and with the fix it ran 4x tsem4, 2x tsem6, churn and heavy rows clean on the same machine minutes earlier. One crash is not a rate, but the result points the same way as the code analysis. The 6-shard pass counts are dominated by how many shards OOM at device creation (6 to 1923 without the fix), so I don't read anything into the gpuk2 1284.

The external display had dropped its link 33 s before the crash (dock power save). I don't think that is related, but I'm noting it.

Evidence (S500): traces/gpu/20261108-j616s-gpuk-fw-id-fence/{gpuk2-fleet30,fleet30ctl} (README updated). The J616s is back on fleet30, gpuk2 is removed, Enforcing, 0 AVC.

[RyanTheTide]

Merge-train rows so far (MTR3, 2026-11-08). Test kernel 7.3.0-aurora-f4mtr3g = integration/aurora af6d986 + #1122 + #1123 + #1124 + #1176, built with 0 warnings. Each Mac got one silent one-shot boot. The checks were our FLEET regression suite vs the same Mac's fleet31/fleet30 rows, plus keyboard/trackpad/HID/DART/MSI error counts and gpubench, test boot vs default boot:

• M2 (J413): PASS. No hard regressions. MTP keyboard and multi-touch present, 0 HID/MTP errors.
  • The first glmark read 3890, so I re-checked with 3 runs each: default 4614/4802/4462, test fresh boot 4253/3843/4612, and a plain-reboot control on fleet31 4636/3773/4555. The low reads are fresh-boot noise, not a regression.
  • gpubench light rows are equal.
• Neo (J700): PASS. No hard regressions. glmark x3 2604/2700/2670 (default) vs 2634/2672/2665 (test). gpubench equal on the x3 re-check. AVD bit-exact. Keyboard and trackpad present, 0 HID errors.
• J616s (M4 Pro): PASS. No hard regressions. glmark 6562 vs 6570. dEQP timeline semaphores 3835 pass, the same as fleet30, with 0 firmware crashes, 0 device-lost and 0 GPU faults. Keyboard and trackpad present, 0 HID/DART/PCIe errors.
• M1 Pro (J316s): NOT RUN YET. Other lanes held it, and the swarm is handing over to a new orchestrator now.

Not merged yet: the plan was to merge after the M1 Pro row. The kit is kits/test/20261108-f4mtr3g on the S500, and the row script is work/linux/mtr3-20261108/comborow.sh (M1 Pro, TID=0).

[RyanTheTide]

M1 Pro row done (the last of the four), 7.3.0-aurora-f4mtr3g = integration af6d986 + #1122 + #1123 + #1124 + #1176, one-shot vs the fleet31 default:

• glmark 9861 vs 9848 (+0.1%), kernel errors 0, AVC 0 (enforcing), failed units 0.
• Keyboard/trackpad input devices, hwmon fans and rtkit timeouts/errors: same as the default boot. 0 DART/PCIe/MSI errors.
• Wi-Fi/BT reconnect, camera, 6+6 sleeps, ANE resume: same as the default.
• gpubench A/B within noise (worst row copy.bw buf2img d4 +6.3%, one sample).
• One new boot warning class, kauditd_printk_skb: N callbacks suppressed (audit printk rate limit, AVC 0). It is environmental. None of the four PRs touch audit, and the iommu: apple-dart: keep a T8142 DART that a live coprocessor uses #1122-PCI: apple-msi: program the MSI map of every listed port #1124 paths are gated to T8142, so they don't run on this T6000.

Rows passed on all four Macs (M2, Neo, J616s, M1 Pro). The trial merge onto integration d4d2678 is clean. Merging.