Repository navigation
soc: apple: mailbox: receive T6050 GPU mailboxes by interrupt - #1844
Conversation
… fires The T6050 GPU mailboxes receive by polling, because the recv-not-empty interrupt does not reach the AIC on the 14-inch M5 Max MacBook Pro: a 25 us hrtimer for 1 ms after each send, otherwise a once-per-jiffy poller, and the interrupt is never enabled. On the 16-inch M5 Max MacBook Pro the interrupt works, and there polling costs about 5% of GPU throughput in an inference engine's decode step. Enable the recv-not-empty interrupt on these mailboxes again, and keep both pollers running beside it until it has delivered a message. From then on the fast poller stops and sends no longer arm it: the interrupt receives, as on the other ASC mailboxes. The line is level triggered and the GPU handler is oneshot, so a message that arrives while it runs fires it again. Each mailbox start proves the line again. The jiffy poller keeps running as a watchdog. Where the interrupt delivers, the poller finds the FIFO empty. A message it does find was not delivered by the interrupt (or its thread had not run yet): the poller hands it on and clears the proof, so that the sends after it arm the fast poller again until the interrupt delivers a message again. A line that never fires keeps polling exactly as before. A line that delivers once and then stops, or fires only spuriously, loses no message: the watchdog takes one that it misses on a following pass, normally the next, subject to workqueue scheduling and to its budget of 64 messages per pass. The ASCWrap v8 storm guard now covers these mailboxes too: a line that fires 1000 times in a row with nothing to read is disabled at the AIC and receive falls back to polling. The hardware flag rx_poll_only becomes rx_poll_until_irq. A full send FIFO is still waited out by sleeping polls, without the send-empty line, as before. Tested with KUnit (arm64, QEMU, suite apple_mbox_g17_stall): new cases cover proof by a delivered message, the watchdog taking the proof back when the line is silent with a message queued, the storm guard, and start and stop. They wait on what the pollers receive, and read state after cancelling them or under rx_lock. On an M5 Max MacBook Pro (J716c, whose interrupt fires), an inference engine's decode, two runs each without and with this commit: GPU time of its verification pass 21.93 / 21.93 -> 21.02 / 21.04 ms, single-token decode 62.4 / 63.8 -> 66.4 / 67.2 tok/s, output identical; the GPU mailboxes' recv interrupts count in /proc/interrupts, the watchdog took the proof back 64 times in ~100 s of decode (each time the next message proved the line again), and the storm guard did not trip. Developed with Claude Code. Signed-off-by: Erik Bourget <erik@ebourget.net>
|
Independent review of What was verified at that head:
Notes that still apply after the follow-up commit:
|
ASC mailboxes are interrupt driven; T6050's GPU mailboxes now use the same paths as the other ASCWrap v6 GPU mailboxes. The three T6050 GPU compatibles (apple,t6050-ascwrap-v6, apple,t6050-x1n1-ascwrap-v6 and apple,t6050-j775d-ascwrap-v6) describe the ASC v4 mailbox block behind the ASCWrap v6 wrapper offset, as the T8140, T8142 and T6040 GPU mailboxes do. Their hardware descriptions now differ from those only in the wrapper windows, and they take the same start, stop, receive and send paths: - the recv-not-empty line is the threaded, oneshot GPU receive interrupt (IRQF_ONESHOT | IRQF_NO_AUTOEN | IRQF_NO_SUSPEND), enabled by apple_mbox_start() and disabled by apple_mbox_stop(); - its handler drains the I2A FIFO until it is empty, one receive budget per pass; - a non-atomic sender that finds the A2I FIFO full waits for the send-empty interrupt and its completion. Remove what received them by polling instead: - the rx_poll_until_irq hardware flag (rx_poll_only before); - the 25 us receive hrtimer, its kick from apple_mbox_send() and its deadline; - the jiffy receive poller and its watchdog for these mailboxes, and the rx_irq_proven state; - the storm guard's extension to them; - the sleeping poll of a full send FIFO, which left the send-empty interrupt unused. This undoes, for these mailboxes, the receive and send changes of commits 43c6ca8 ("soc: apple: mailbox: receive T6050 GPU mailboxes by polling"), a23755c ("soc: apple: mailbox: poll T6050 GPU receive quickly while a reply is outstanding") and 41c359d ("soc: apple: mailbox: receive T6050 GPU mailboxes by interrupt once it fires"). The T8152 ASCWrap v8 provider keeps its receive poller, its poll-only default, the ascwrap_v8_rx_irq parameter and its storm guard, unchanged. Drop the KUnit cases that tested the removed receive polling. The apple_mbox_g17_stall suite keeps its receive budget, send stall and receive utilization cases. No exported interface changes. Signed-off-by: Diljit Singh <hey@djdev.me>
|
@erikwb, a change of direction from the project owner on this one. Mailboxes are meant to be driven by interrupts. So the T6050 GPU mailboxes should receive like every other ASC mailbox, with no poll fallback. On top of your commit, I pushed a commit that does that for the three T6050 compatibles, modelled on the original Asahi ASC mailbox driver, with the threaded GPU receive handler that T8140, T8142 and T6040 use:
The net effect is that |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Approving and merging at 788441e0890e.
M5 Pro (J714s) merge gate, slot hw2.
- Arms:
- A = the default kernel (7.3.0-m5-intg6). It predates #1601, so its
mailbox.cis byte-identical to this PR's. - B = 7.3.0-m5-mbox2: integration/aurora
a34d26c2with this branch, the same config, dtb, Stage 2 and command line as the default.
- A = the default kernel (7.3.0-m5-intg6). It predates #1601, so its
- Recv interrupts on both arms (
/proc/interrupts):- A, 165 s after boot:
2302600000.mailbox-recv1,323, spread over all 18 CPUs;2302e00000.mailbox-recv25. - B, 46 s after boot: 811 and 14.
- Both send-empty lines stayed at 0 on both arms.
- A, 165 s after boot:
- Health: no GPU timeouts, recoveries or faults, no oops, and the desktop stayed up throughout. That held over boot, gpubench, glmark2, a timeline-semaphore CTS sample and three idle/load cycles.
- gpubench (the gpubench lane's harness and placement, same Mesa on both arms), A → B, median us per submit:
- compute.light d1 159.4 → 158.1 (gpu_gap 149.7 → 149.3), d16 14.69 → 14.59, d64 14.63 → 14.42;
- render.light tri d16 26.14 → 26.19, d64 26.60 → 24.05;
- sync.signal host 129.5 → 123.9 us;
- compute.heavy bw and copy.bw within 1%;
- two mid-depth rows (compute.light d4, render.light tri d2) +7%.
- The two arms run identical mailbox code, so these differences come from the rest of the integration tree or run-to-run noise, not from this PR.
- glmark2-es2, off-screen 1080p: 4,364 → 4,471.
- Sync CTS:
- On B:
dEQP-VK.synchronization2.op.multi_queue.timeline_semaphore.*, 1,855 cases at 16 jobs, all Pass, 0 GPU or mailbox fault lines. My harness cut the run short (a runner-detection race); the cases that ran all passed. - On the default kernel, i.e. this PR's receive path: the full 4,269-case list gave 4,266 Pass / 0 Fail / 3 NotSupported in earlier runs (mesa#264's gate).
- On B:
CI: all 7 checks green on this head, on "Merge 788441e into 6882c13". integration/aurora has not touched the mailbox files since.
Problem
The T6050 GPU mailboxes (
apple,t6050-ascwrap-v6,apple,t6050-x1n1-ascwrap-v6andapple,t6050-j775d-ascwrap-v6) have received by polling since #1601. After each send, a 25 us hrtimer polls the receive FIFO for 1 ms; otherwise a once-per-jiffy delayed work polls it. A full send FIFO is waited out by sleeping polls.ASC mailboxes are interrupt driven. On the 16-inch M5 Max MacBook Pro (J716c), polling costs about 5% of GPU throughput in an inference engine's decode step.
Change
The T6050 GPU mailboxes take the same start, stop, receive and send paths as the T8140, T8142 and T6040 GPU mailboxes (the
apple,asc-mailbox-v4registers behind the ASCWrap v6 wrapper offset). Receive and send are interrupt driven, as in the original Asahi ASC mailbox driver, with the threaded GPU receive handler those mailboxes already use:IRQF_ONESHOT | IRQF_NO_AUTOEN | IRQF_NO_SUSPEND).apple_mbox_start()enables it andapple_mbox_stop()disables it.Two commits:
soc: apple: mailbox: receive T6050 GPU mailboxes by interrupt once it fires(41c359d). It enabled the receive interrupt beside the pollers.soc: apple: mailbox: receive T6050 GPU mailboxes by interrupt. It removes the polling for these mailboxes:rx_poll_until_irqhardware flag;apple_mbox_send();It also drops the KUnit cases for that machinery. The
apple_mbox_g17_stallsuite keeps its receive budget, send stall and receive utilization cases.Against integration/aurora:
drivers/soc/apple/mailbox.creturns to its state before soc: apple: mailbox: receive T6050 GPU mailboxes by polling #1601;drivers/soc/apple/mailbox-g17-test.his unchanged.The ASCWrap v8 mailboxes (T8152) keep their poller, poll-only default,
ascwrap_v8_rx_irqparameter and storm guard; changing them is a separate change. No exported interface changes.apple_mbox_set_rx_util_min()takes effect on these mailboxes again, since their receive runs in the threaded handler. #1739 (the fast poller's arming flag) no longer applies, because the fast poller is gone.Testing
Hardware, M5 Max MacBook Pro (J716c), erikwb.
2302600000.mailbox-recv962 → 6409,2302e00000.mailbox-recv17 → 245 (0 without it, which does not enable the line).Hardware, M5 Pro MacBook Pro (J714s): the default kernel predates soc: apple: mailbox: receive T6050 GPU mailboxes by polling #1601, so it runs this PR's
mailbox.c. During a dEQP run, the recv interrupts count:2302600000.mailbox-recvwent 1,322 → 1,341 in 5 s, spread across all 18 CPUs;2302e00000.mailbox-recvread 34;A merge-gate run of this branch (gpubench, glmark2 and a synchronization CTS subset, against the default kernel) follows in a comment.
CI: the checks on this head.
🤖 Generated with Claude Code