[erikwb]

The problem

The scheduler's watchdog polls a G17 queue every two seconds and takes a
compute command that a poll finds still running at the head of its queue to
have hung. On the M5 it renews nothing, so the command fails about two
seconds after it got there. A compute job that needs longer (a long prefill
of an inference engine, for example) has to be cut into small command
buffers, and it still fails when other clients share the GPU. On the J716c,
two processes that each submitted a 1.1 s compute job filling the GPU at the
same moment ran side by side: one finished after 1.57 s, the other was killed
2.0 s after its submission with about 0.3 s of its own work left.

Letting one queue's commands run longer has a cost for the other clients: a
command of another client, render or compute, can wait on the GPU until a
long command ends. The waiting job's watchdog defers it behind the other
VM's older work at most eight polls (16 s) and then fails it, and that
client loses its device although it did nothing wrong.

The change

First commit: struct drm_asahi_queue_create gets compute_timeout_ms at
its end, advertised by DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT (bit 36) on G17:
a watchdog allowance for each compute command of the queue.

• For a nonzero request the driver clamps it (at least the default of one
  poll, at most the new compute_timeout_max_ms module parameter, 30000 by
  default) and returns the allowance in effect; a request of 0 keeps the
  default and is returned unchanged.
• At a poll, the watchdog first renews a command that is within its
  allowance.
• pad must be zero. A caller that passes the old struct size gets the
  default. Render commands and queues that do not ask keep the default. One
  case changes under the default: a job that was still in publication
  (waiting for capacity) at one poll now gets one full poll from the first
  poll that finds it published.

It is an allowance, not a deadline:

• The watchdog still polls every two seconds, so a command may run up to one
  poll longer than its allowance.
• The start of a command is known only approximately. The time is counted
  from one poll before the first poll that finds the command at the head of
  the queue. That is no earlier than the command got there and may be later:
  commands submitted to the queue in the meantime postpone that poll.
• Past the allowance, the renewals and deferrals the watchdog already grants
  still apply, in the same order as before: the extra polls of ray and
  acceleration-structure work, the renewals while the GPU retires other work
  (not on the M5), and the deferral of work behind another VM's earlier work.

Second commit: work waiting behind a command that runs inside its allowance
is not failed for that.

• A job's watchdog records when its allowance ends, and the heads the
  cross-VM attribution compares say whether a job is still inside it.
• A job waiting behind another VM's older work whose blocker is live and
  inside its allowance is renewed with a new decision, Excuse, that does
  not use any more of its eight renewals. When the blocker completes, or
  its allowance has passed, the existing rules apply again, with what the
  job has left of its renewals for the blocker's own failure and teardown.
• A blocker that already failed excuses nothing, the oldest unfinished work
  is never excused (it has no blocker), and a blocker under the default
  allowance excuses nothing, so clients that do not ask for an allowance
  see the decisions they saw before.
• Render and compute watchdogs decide through the same attribution and both
  see compute heads as blockers, so render work waiting behind a long
  compute command is covered as well.
• The excuse is bounded on the waiting job's side: at most
  compute_timeout_max_ms in polls (15 with the default 30 s), however many
  blockers take turns. This rule renews a job at most 15 + 8 times by
  default, 46 s of renewals. A job's own allowance and its other deferrals
  are counted separately and come on top, and each renewal re-arms a timer
  rather than an absolute deadline, so the time until a waiting job is
  failed can be longer than that.
• The cross-VM watchdog lines (Compute watchdog ... and Render watchdog ... behind earlier work of another VM) also print excused= and
  blocker_in_allowance=, and are written at the first excuse, at the first
  counted deferral and at the final decision.

Third commit: on the G17 GPUs other than the M5 the watchdog also renews a
published job for up to fourteen polls (thirty seconds) while the GPU keeps
completing other work. Those renewals came after an allowance, so the two
added up: with 30 s requested, a command on a busy G17P could run for about
58 s. A poll that an allowance renews beyond what the default allowance
renews now counts as one of those renewals, so a job is renewed for the
longer of its allowance and the thirty seconds, not for their sum:

• An ordinary compute command that never finishes, with 30 s requested,
  fails one poll after 30 s on a shared GPU or alone, on every G17.
• A shorter allowance on a shared GPU fails at thirty seconds, as a command
  without an allowance does; the one poll every allowance renews after a
  wait in publication is not counted, so asking never leaves a job fewer
  polls than the default.
• The extra polls of ray and acceleration-structure work and the deferrals
  of waiting work still come on top, on every G17 as before. The UAPI
  comment now says the shared renewals run alongside an allowance.

There is still no bound on the sum of the allowances: a queue kept full of
long commands occupies the GPU for as long as it is fed, and the other
clients share the GPU with it as the firmware schedules them. The UAPI
comment says what a waiting client can expect.

Testing

Hardware: M5 Max MacBook Pro (J716c).

• Build: integration/aurora 6970a51 plus this branch, with the J716c
  configuration. No warning in the files the branch touches.

• KUnit, arm64 with 16K pages in QEMU:
  ./tools/testing/kunit/kunit.py run --arch=arm64 --kunitconfig=drivers/soc/apple/sep.kunitconfig:
  Ran 603 tests: passed: 601, skipped: 2. asahi_g17_execution_allowance
  has 31 tests, twelve of them new: the allowance (four), and for the waiting
  client a live blocker inside its allowance past eight renewals (excused),
  the excuse used up (renewals, then spared), a blocker past its allowance
  and a failed blocker (the decisions as before), no blocker and the oldest
  work (own), the bound, blockers taking turns (the counters are the
  waiting job's and are not reset), and when an allowance ends (seven), and
  the allowance and the shared renewals overlapping (one). The existing
  test of the watchdog line also checks when an excuse is reported.

• Host tests, as CI runs them: python3 -I tools/asahi/run-host-tests.py:
  288/293 passed. The five that fail (apple-dcp, g15-pb-growth-tests
  and g15-stats-drain-tests, each of the last two also with --mutations)
  fail the same way on integration/aurora on the build host, which lacks
  the Rust 1.93.1 toolchain the g15 tests ask for.
  g17-ray-query-timeout-tests runs the 31 timeout unit tests,
  queue-priority-tests compiles the queue admission gate as before, and
  the asahi-m4 layout probe compares the frozen queue_create bindings as
  a prefix of the grown struct and asserts the new tail.

• scripts/checkpatch.pl: 0 errors, 0 warnings for each commit.

• On the J716c (first commit, on the integration/aurora of 2026-11-09 with
  the then-open J716c pull requests), with Mesa aurora-wip plus a
  honeykrisp change that passes a requested allowance to the new field
  (driconf hk_compute_timeout_ms), and a Vulkan test program whose compute
  jobs are a single-thread loop of known length:

  case	result
  no request, one 2.13 s job	fails 2.04 s after submission (ETIMEDOUT), as before the change
  30000 requested, jobs of 5 s and 20 s	both complete
  30000 requested, one 40 s job	fails 30.70 s after submission
  600000 requested, one 40 s job	fails 30.69 s after submission (lowered to the maximum)
  30000 requested, three 16 s jobs submitted back to back on one queue	all complete, 16.0, 32.0 and 48.0 s after the first submission
  30000 requested, two processes each with a 1.1 s job filling the GPU	both complete, after 1.93 s and 2.37 s

  The kernel log of the run has the three expected ETIMEDOUT lines, each
  naming the test process, and nothing for any other client.

• Sharing, first commit only, on integration/aurora ea10c4d: one
  client keeps a queue full of compute jobs that fill the GPU (640
  threads), of 1.5 s under the default allowance and of 5 s and 20 s with
  30000 requested; a second client under the default allowance submits a
  short compute job filling the GPU every 100 ms. Most of the second
  client's jobs took about 6 ms (p90 at most 6.5 ms); now and then one
  waited until the first client's running job ended (1.46 s beside a 1.5 s
  job; 3.4 to 8.3 s beside 20 s jobs), and one that waited behind a 20 s
  job was failed after 18.4 s with GPU work of a client failed (ETIMEDOUT, ...) following eight deferrals (decision=Spare). That
  failure is what the second commit removes.

• With both commits, on the J716c (the base above plus this branch):

  • A second client's job of 1024 workgroups submitted 1.1 s into another
    client's 28.9 s compute command (allowance 30 s) waited 26.9 s and
    completed. One Compute watchdog ... excused=0 decision=Excuse ... blocker_in_allowance=1 line; no decision=Spare, no failed client.
  • The two-kind probe (one thread and GPU-filling, every 100 ms) beside a
    20.7 s and a 28.9 s command: 116 and 224 jobs, none failed, longest
    waits 15.0 s and 15.7 s, one or two decision=Excuse lines per run.
  • A compositor with a terminal redrawing beside a 29.7 s command: two
    Render watchdog ... decision=Excuse lines and one Compute watchdog ... decision=Excuse line for the compositor, which kept its device.
  • The limits are unchanged: a 2.13 s job without an allowance is failed at
    2.02 s; with an allowance of 30 s, jobs of 5 s and 20 s complete, a 40 s
    job is failed at 30.7 s, a request of 600 s behaves as 30 s (30.7 s),
    and three queued 16 s jobs complete at 16, 32 and 48 s.

Depends on

Nothing in the kernel. The Vulkan side is a separate pull request against
aurora-silicon/mesa.

Not covered

• Allowances above 30 s (the module parameter was left at its default).
• GPUs other than G17: they do not report the feature, and a queue that sets
  the field there is refused with EINVAL.
• Render commands keep the default allowance; they are covered only as
  waiting work.
• The third commit was not run on a G17 other than the M5 Max (where the
  shared renewals do not apply); it is covered by the unit tests, which
  apply the allowance and the shared renewals in the watchdog's order,
  with and without a wait in publication.

🤖 Generated with Claude Code

[jgoyvaerts]

This advertises DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT on every G17, so G17P (M4 Pro) gets it too, but it was only measured on the J716c. On G17P the shared renewal (renew_shared, 14 polls while other work retires) still runs after the allowance ends, because renews_shared() is false only on M5. So with 30000 requested, a busy G17P can let a command run for about 58 s, and the 40 s case would likely complete rather than fail at 30.7 s.

Two options: advertise the feature only where the allowance is the effective limit (M5) for now, or keep it on all G17 and say in the UAPI comment that on G17P the allowance can be extended by the shared renewals. We can run the PR's timing cases on an M4 Pro (J616s) if that helps.

[erikwb]

Thanks, you're right. I kept the feature on every G17 and fixed it in 8fdaa2b ("drm/asahi: g17: count an allowance against the shared renewals"):

• A poll that an allowance renews beyond what the default allowance renews now counts as one of the job's shared renewals, so the two run from the same start instead of adding up. A job is renewed for the longer of its allowance and the 30 s the shared renewals give any job.
• For an ordinary compute command that never finishes, with 30000 requested: failed one poll after 30 s on a busy G17P too (was about 58 s), as on the M5.
• A shorter request on a shared GPU still gets the 30 s a request-less job gets; the one poll every allowance renews after a wait in publication is not counted, so asking never leaves a job fewer polls than the default.
• The extra polls of ray/AS work and the deferrals of waiting work still come on top, on every G17 as before. The UAPI comment now says the shared renewals run alongside the allowance; mesa#259's copy of the header is updated to match.

Covered by the timeout unit tests (they apply the allowance and the shared renewals in the watchdog's order, with and without a publication wait); KUnit 603/601/2. I could only run hardware on the M5 Max, where the shared renewals don't apply, so yes please on running the timing cases on the J616s: with hk_compute_timeout_ms=30000 a 40 s job should now fail at about 30.7 s while other work keeps the GPU busy.

[DjDeveloperr]

Tested on the M5 Pro (MacBook Pro 14", J714s, T6050, G17S): all five checks pass. Approving.

Setup

• Kernel: our M5 default (integration/aurora eb6b24140dc0) with this PR's first two commits applied (4860c0626b8c, 652679a8a91a).
  • One adjustment: the first commit's UAPI hunk has the USERPTR define as trailing context, and that base lacks it.
  • Only aurora.ko differs from the default, and it imports the same symbols as the default's module. Default Image, DT and loader; one boot.
• Userspace: aurora-silicon/mesa#259 (4ba57b48, hk_compute_timeout_ms). An LD_PRELOAD shim logged each queue_create (size sent, request, value returned) and created extra probe queues.
  • Compute jobs are a dependent ALU loop of calibrated length: one workgroup, or 2048 workgroups of 64 (the GPU saturates at about 256).
  • The render client is an offscreen 1080p draw loop.
• Third commit: 8fdaa2b26cbe arrived during the run and wasn't rebuilt.
  • It only feeds allowance polls into shared_renewals. renew_shared() is the only reader, and renews_shared() is false on M5.
  • The runs below agree: with 30000 a 36 s job fails at 30.2 s, not about 58 s. With no request a 5 s job fails at 2.0 s while the compositor's work keeps retiring.
  • So on the M5 these results hold for the current head.

1. The default is unchanged. Same tests on the unmodified kernel and with the PR, no allowance anywhere:

	without the PR	with the PR
1 s compute, one workgroup, x3	982 / 979 / 979 ms	983 / 979 / 979 ms
~1.1 s GPU-filling compute, x3	all complete	all complete
1080p render loop, 5 s	26,818 frames, p99 0.24 ms, max 5.1 ms	26,668 frames, p99 0.24 ms, max 3.9 ms
gpubench render.light d16/d64, compute.light d1/d16 (us/submit)	26.8 / 27.7 / 164.5 / 14.6	25.7 / 23.9 / 161.5 / 14.0
~5 s compute, no request	lost at 2030 ms	lost at 2003 ms

• dmesg had no GPU watchdog or fault lines during the short jobs and the render.
• Without the option, Mesa sends the old struct size.

2. One ~5 s compute command

• No request: lost at 2003 ms.
• hk_compute_timeout_ms=10000: completes in 4893 ms.
• A ~14 s command with 10000: lost at 10092 ms.
• Each loss logs one GPU work of a client failed (ETIMEDOUT …) line, naming only the test client.

3. The clamp. Bit 36 is reported. Values of compute_timeout_ms returned:

• 1, 500, 1999 and 2000 return 2000.
• 2001 through 30000 return the request.
• 30001, 60000, 600000 and 0xffffffff return 30000.
• pad = 1 returns EINVAL.
• The old 24- and 40-byte structs are accepted.
• A ~36 s job asking for 60000 is lost at 30230 ms.

On the unmodified kernel the same 48-byte requests come back unchanged (60000 returns 60000), because DRM copies the unknown tail back. Mesa#259 checks the feature bit, sends the old size there, and a 5 s job with the option set is lost at 2004 ms as before.

4. Sharing

• Two processes, one ~1.2 s GPU-filling job each, submitted together, two rounds each way:
  • Without the allowance, the job published first is lost at its first poll (2022 ms, 2001 ms). The other logs Compute watchdog … renewals=0 excused=0 decision=Defer … blocker_in_allowance=0 and completes (2240 ms, 2580 ms).
  • With 10000 both complete (2271/2553 ms, 2266/2553 ms), and nothing is logged.
• A ~25 s GPU-filling compute command with 30000. From 1 s in, three clients without an allowance ran beside it: the 1080p render loop, a GPU-filling compute job every 100 ms, and a one-workgroup compute job every 100 ms.
  • The long command completes in 25.9 s.
  • Render: 54,601 frames, longest 152 ms (17 frames over 100 ms).
  • GPU-filling probe: 298 jobs, longest 190 ms. One-workgroup probe: 299 jobs, longest 17 ms.
  • Nothing was lost, there were no watchdog lines, and the compositor was unaffected.
  • On J714s the firmware runs the other clients' work alongside the long command. Nothing waited long enough to reach the 8-renewal deferral or the new decision=Excuse, so this run doesn't cover that path.

5. abi.py, run on the M5 (gcc 16.1.1, rustc 1.98.1): test-uapi.py passes with 307 comparisons and 0 errors for intg6 plus this PR, for this PR's header, and for a header without the field. The three new _Static_asserts compile. The M5's own ld.lld couldn't load its libLLVM, an environment problem there and not this PR, so rustc linked with -fuse-ld=bfd.

CI: host-tests is green on 8fdaa2b2.

Nits (non-blocking)

1. The UAPI comment says the driver "returns the allowance in effect here". A request of 0 comes back as 0, not 2000, because the field is only written for a nonzero request. Either say "for a nonzero request", or write the default back.
2. abi.py: the new drm_asahi_queue_create:size substitution sits inside the if 'struct drm_asahi_cmd_render_with_prior {' not in … block, as the existing client_memory_limit one does.
   • test-uapi.py --mesa against mesa#259's header, which still has that struct, therefore reports drm_asahi_queue_create:size 40 vs 48. It also reports the older drm_asahi_params_global:size 600 vs 616.
   • Both layouts are in fact 48 bytes with the field at offset 40.
   • Fix: apply the two tail substitutions after that block, to any header that has the tail.

[erikwb]

@DjDeveloperr thanks for the thorough run on the J714s. Both nits are fixed and folded into the first commit (the branch is still three commits, the other two are unchanged; head is now 772bb5b):

1. Return value. The UAPI comment now says that for a nonzero request the driver clamps it and returns the allowance in effect, and that a request of 0 keeps the default and is returned unchanged. No behaviour change. mesa#259's header copy is updated to match (e35dcf97d39), and the PR description says the same.
2. abi.py. The two tail substitutions (drm_asahi_params_global:size and drm_asahi_queue_create:size) now run after the historical-extension block, for any header that carries the tail. test-uapi.py --mesa against mesa#259 now passes with 307 comparisons and no errors (before: 40 vs 48 and 600 vs 616). While at it, the queue tail also asserts pad (offset 44, width 4): a header with pad narrowed to __u16 or removed, or with compute_timeout_ms narrowed, now fails, where the narrowed pad used to pass.

checkpatch 0/0, host tests 288/293 (the five failures are the known apple-dcp and g15 toolchain ones).

Fun fact: this fix was written by Qwen3.8-Flash-Next running locally on the J716c itself (M5 Max, Linux, on this PR stack and the open Mesa PRs), driving a coding agent through its OpenAI-compatible server. Two rounds, 43 requests, 14,133 tokens generated at 102.4 tok/s decode (104.4 tok/s on the first round), prefill about 1,320 tok/s, with contexts up to 39K tokens.

[RyanTheTide]

The UAPI matches mesa#259 (offset 40, pad at 44, bit 36) and the default path is unchanged apart from the one extra poll after a wait in publication. The third commit only acts on G17 parts where the shared renewals run, and that is the M4 Pro and A18 Pro, so we will run the timing cases there before merging: 20 s and 40 s jobs with hk_compute_timeout_ms=30000 under a busy compositor, plus a default-path desktop and compute CTS run.

[RyanTheTide]

First results, with the three commits applied to integration e661192 in a one-shot test kernel on the M2 (G14G): desktop boot, glmark2-es2 9767 (production 9822, one short run), no watchdog lines and no GPU faults, 3 s2idle cycles clean. On G13/G14 this only covers the queue_create path with the old struct size. The G17 timing cases on the M4 Pro and A18 Pro are still to run.

[DjDeveloperr]

@erikwb #1610 is merged, and it adds bit 35 at the same place in asahi_drm.h where this branch adds bit 36. So I pushed 215dca833136 to this branch: a merge of integration/aurora (954a6a74) that keeps both defines in bit order (35, 36, 37).

• Your three commits are unchanged, and the PR's +/- lines are identical to 772bb5b1's.
• If you'd rather rebase, go ahead.

Status

• Review: I ran an independent review of 772bb5b1 and found no blocker. File by file, the only changes since the approved 8fdaa2b2 are the UAPI comment and abi.py. No driver .rs file changed, so the M5 Pro results in my review still apply.
• CI:
  • host-tests is green on 772bb5b1, and this push starts a fresh run.
  • The CI-only CI only: erikwb GPU PRs #1610 + #1611 + #1752 composed on integration/aurora (do not merge) #1834 (integration/aurora + drm/asahi: GEM_RESIDENCY, keep an object's memory resident #1610 + drm/asahi: g17: M5: report the performance state, its frequency and busy time #1611 + this PR, with the same header resolution) built the whole kernel with DRM_ASAHI=y and passed KUnit 305/305.
• Before merging: this waits for @RyanTheTide's G17P timing rows (M4 Pro, A18 Pro).

Non-blocking review notes (for a follow-up, not needed here; line numbers at 772bb5b1)

1. A replay doesn't restart the allowance (g17/job.rs:427-441, g17/timeout.rs:100-107).
   • A replay republishes the packet from its start, but Timeout::allowance keeps its first head_since.
   • Scenario: clients A and B both ask for an allowance, and B's command hangs. A waits behind it past its own allowance, is spared and replayed, and B is drained. At A's next poll it is past its allowance, and on G17P count_allowance_poll has already used up its shared renewals. With no blocker left the decision is Own, so A's innocent command fails about one poll after the replay.
   • Fix: restart the allowance on an accepted replay (head_since = None, waited = true, allowance_end back to u64::MAX). MAX_REPLAYS bounds the cost. Otherwise, say in the doc that a replay doesn't restart it.
2. is_published() is read twice (job.rs:432-433). A flip between the two reads counts the placement poll as a shared renewal. Read it once.
3. Late rejection on G13–G16 and M3 (file.rs:1665-1668): a nonzero compute_timeout_ms is rejected only after new_queue() has built the queue. The error path is clean; the work is wasted.
4. No ceiling on compute_timeout_max_ms (asahi_core.rs:185): it has a floor of 2000 ms but no upper bound. It is a load-time parameter, so a sanity cap or a note in its description would do.

mesa#259 conflicts with aurora-wip in two files:

• include/drm-uapi/asahi_drm.h: mesa#264's bit 34 sits at the same place. Keep 34, 36 and 37 in order.
• src/asahi/vulkan/hk_drirc_gen.py: mesa#260's hk_compute_kick_ms sits at the same place. Keep both.

Unless you rebase first, I'll push that resolution to j716c/hk-compute-timeout once this PR lands.