Skip to content

drm/asahi: g17: M5 Pro: wait for other queues' compute in firmware when it ends visible - #1411

Merged
DjDeveloperr merged 1 commit into
funkzy/t6050-gpu-on-integrationfrom
dj/g17-m5-visible-completion
Nov 10, 2026
Merged

DjDeveloperr merged 1 commit into
funkzy/t6050-gpu-on-integrationfrom
dj/g17-m5-visible-completion

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

On M5 Pro (J714s) a compute command that waits for another queue's compute command waits on the host. The producer's completion interrupt, the event worker and a new publication sit between the two kicks, so a cross-queue timeline-semaphore hop takes 155 µs. macOS on the same firmware (26A428) keeps the wait in firmware and hops in 17.4 µs.

Depends on #1070. This branch is on #1070's head (26d82876dbcf). Until #1070 merges, its commit shows here too. The change itself is the last commit, 93f5fb7b50da.

What macOS does

The kick and descriptor fields come from macOS-HV AGX traces of gpubench-mtl's hop (runs hv9 and hv10). The CDM tails come from WindowServer's compute streams in hv11.

  • Dependencies: the consumer's kick names the producer queue's kick in the completion class (1), plus its own parent. The parent is class 1, or class 2 when published late.
  • Descriptor: each descriptor counts its queue's completion_ordinal, and carries the same count as submit_sequence.
  • Event: the kick waits on its own context's pool event, with no entry gate.
  • Dispatch: the dispatch base is 0x10c08860. The auxiliary control (0x0a731) is 1 only on a kick whose parent is class 2.

Linux's previous M5 Pro form sent ordinal 0, sequence 0, a class-2 parent, fixed event 131, dispatch base 0x14c08a60 and auxiliary control 1. Moving the parent to class 1 without the matching auxiliary control hangs the queue: Hyprland times out at boot, and dEQP loses the device.

Why the wait couldn't simply move to firmware

A firmware wait is satisfied when the producer completes. Honeykrisp ends API dispatches and vk_meta copies, fills and updates with a weak barrier, which reports completion before the writes land.

With macOS's encoding and the waits in firmware, dEQP-VK.synchronization2.op.multi_queue.timeline_semaphore read stale data in 31 of 214 cases ("Memory contents don't match"). Every one had a compute or transfer producer. Ending those streams with macOS's own launch tail (barrier flags 1) still failed 30 of 214. Only the strong, memory-scoped flush (flags 5) is visible at completion.

The change

  • Encoding: M5 Pro compute kicks take macOS's form, as listed above.
  • Firmware waits: M5 Pro explicit waits take the completion class. Render barriers on compute producers do too.
  • UAPI: DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION (compute flag bit 20) lets userspace declare a stream that ends with the strong flush. DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION (bit 34) is reported on M5 Pro only.
  • Gate: on M5 Pro, a submission becomes a firmware source only when its compute command sets the flag. Everything else keeps drm/asahi: g17: Fix M5 Max GPU work running before the work it waits for #1070's host wait, so current Mesa is unaffected.

The Mesa half is https://github.com/aurora-silicon/mesa-aurora/pull/33.

Measured (J714s, pinned to CPU 12; two boots per kernel, measured before any other work)

base this, current Mesa this, Mesa setting the flag
sync.signal queue/timeline hop 155.3 µs 155.9 µs 21.5 µs
compute.light d1 / d4 / d16 / d64 (µs) 155.4 / 40.8 / 19.1 / 19.6 158.1 / 42.3 / 18.8 / 18.7 158.8 / 41.3 / 18.9 / 18.9
render.light tri d16 / d64 (µs) 32.3 / 33.6 32.6 / 33.8 33.0 / 32.1

Light submits are unchanged within run-to-run noise (±5–10% per run).

dEQP on this change, with each Mesa, 0 Fail in every list:

list Pass NotSupported
multi_queue timeline (mq-tl) 214 0
multi_queue v1 (mq-v1) 427 0
multi_queue all (mq-all) 290 106
synchronization2 sample 600 421
synchronization sample 693 389
s0 109 2
compute 223 791
glsl 1209 130
draw 906 74
texture 660 251
query_pool occlusion 440 7

gpubench core-a and feat: 41/41 with each Mesa. No GPU recoveries or timeouts.

How it was tested

The tests used aurora.ko built from this change on the argb integration base and loaded into 7.3.0-m5-integration. That out-of-tree module build sees the integration tree's prebuilt UAPI bindings, so the tested build spelled the two new values out. Separately, bindgen run on this branch's header generates both DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION and drm_asahi_compute_flags_DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION. Apart from those two constants and doc-comment wording, the diff is the tested one, and it applies to #1070's head with no conflicts.

Update (2026-11-10): P-core A/B and the Mesa half

Every pinned run above used CPU 12, which is a P core in argb's CPU order (cpu0-11 M, cpu12-17 P). Re-measured pinned to all P cores (cpu12-17), on fresh boots, measured before anything else, two rounds each:

µs/submit (submit CPU) base this change, Mesa with the flag 7.3.0 stack with this change, current Mesa same stack, Mesa with the flag
compute.light d8 18.8 (17.8) 19.2 (18.0) 15.3 (14.6) 18.0 (17.2)
compute.light d32 19.1 (17.9) 18.9 (17.8) 15.2 (14.2) 17.3 (16.8)
render.light tri d64 29.2 29.1 29.2 28.9

The stack here is argb + #1401 + this change + #1399 + #1405. The last two columns differ by 15% on light compute across separate boots, but that's boot-to-boot variation, not the flag. With the current Mesa, the tail-only Mesa and the flag-setting Mesa interleaved on one boot of that stack (three rounds, P cores), compute.light d8 measures 15.2 / 15.2 / 16.4 µs (rounds 15.4-16.7) and d32 15.2 / 15.0 / 15.1, with submit CPU the same in each (14.3-14.6 µs). So the change and the Mesa flag leave light submits unchanged. The measured submits carry no semaphore waits, so they never take the firmware-source input path anyway.

On that stack the hop is 21.4 µs (timeline) and 24.8 µs (binary) unpinned, from 153. dEQP s0, mq-tl and mq-v1 pass with 0 Fail.

🤖 Generated with Claude Code

@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Full kernel build of this branch at 93f5fb7b50da (j714s integration config, Image modules, LLVM): clean. The generated bindings carry DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION = 17179869184 (1 << 34) and drm_asahi_compute_flags_DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION = 1048576 (1 << 20), the values the tested module used. No compiler warning falls in or next to any changed line; the 24 asahi warnings are all in untouched code. No CI ran on this PR (none is reported on #1070 or #1401 either).

…en it ends visible

On M5 Pro a compute command waiting for another queue's compute command
waits on the host: the producer's completion interrupt, the event worker
and a new publication sit between the two kicks, and a cross-queue timeline
semaphore hop takes 155 us. macOS on the same firmware keeps the wait in
firmware and hops in 17.4 us.

Traces of macOS's hop show how its kicks are encoded: the consumer names
the producer's kick and its own parent in the completion class (1), each
descriptor counts its queue's completion ordinal and carries it as the
submission sequence, the kick waits on its own context's pool event, and
the dispatch base is the qualified 0x10c08860 with the auxiliary control
set only for a kick whose parent is pipelined (class 2). The previous M5 Pro
form (ordinal 0, auxiliary control 1) hangs as soon as a parent takes
class 1. M5 Pro compute kicks now take macOS's form.

A firmware wait is satisfied when the producer completes, which a stream
ending in Honeykrisp's weak barrier reports before its writes are visible:
with the waits in firmware, dEQP-VK.synchronization2.op.multi_queue.
timeline_semaphore read stale data in 31 of 214 cases, every one with a
compute or transfer producer. A stream that ends with the strong
memory-scoped flush is visible at completion. Add
DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION, by which userspace declares such a
stream, and report DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION on M5 Pro;
there only a submission whose compute command sets it is waited for in
firmware, and the others keep the host wait.

Measured on J714s (pinned to CPU 12, two boots of each kernel): the hop is
155.3 us before, 155.9 us with this change and current Mesa, and 21.5 us
with Mesa that sets the flag. Light compute and render submits are
unchanged within run-to-run noise. dEQP s0, compute, glsl, draw, texture,
synchronization, synchronization2, the multi-queue samples and occlusion
queries have no failures with either Mesa.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr
DjDeveloperr force-pushed the dj/g17-m5-visible-completion branch from 93f5fb7 to 8aee10d Compare November 10, 2026 09:10
@DjDeveloperr
DjDeveloperr changed the base branch from funkzy/t6050-gpu-on-integration to j716c/m5-max-host-producer-waits November 10, 2026 09:10
@DjDeveloperr
DjDeveloperr changed the base branch from j716c/m5-max-host-producer-waits to funkzy/t6050-gpu-on-integration November 10, 2026 10:20
@DjDeveloperr
DjDeveloperr merged commit 99d8372 into funkzy/t6050-gpu-on-integration Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant