From a2180913e4ffe2744783d0fd3d8b85cbc526103f Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Tue, 10 Nov 2026 21:35:32 -0500 Subject: [PATCH 1/3] drm/asahi: g17: per-queue watchdog allowance for compute commands The scheduler's watchdog polls a G17 queue every two seconds and takes a compute command that a poll finds still running at the head of its queue to have hung: on the M5 it renews nothing, and the command fails about two seconds after it got there. A compute job that needs longer, a long prefill of an inference engine for example, has to be cut into small command buffers with few in flight, and still fails when other clients share the GPU: two processes each submitting a 1.1 s compute job that fills the GPU at the same moment ran side by side, one finished after 1.57 s and the other was killed 2.0 s after its submission, with about 0.3 s of its own work left. drm_asahi_queue_create gets compute_timeout_ms, advertised by DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT (bit 36): a watchdog allowance for each compute command of the queue. The driver clamps it (at least the default of one poll, at most the compute_timeout_max_ms module parameter, 30000 by default) and returns the allowance in effect. At a poll, the watchdog now first renews a command that is within its allowance. Render commands keep the default. It is an allowance, not a deadline: - The watchdog still polls every two seconds, so a command may run up to one poll longer than its allowance. - The start of a command is known only approximately. The scheduler arms the watchdog when the job ahead completes, and again when a later job is pushed, so a job's first expiry comes at least one poll after it reached the head of the queue, later if pushes postponed it; the time is counted from one poll before that expiry. A job still in publication at an expiry (waiting for capacity) has not started: its time counts from the first expiry that finds it published. - Past the allowance, the renewals and deferrals that false_timeout() already grants still apply, in the same order as before: the execution renewals of ray and acceleration-structure work, the shared-work renewals (not on the M5) and the deferral of work behind another VM's earlier work. Under the default allowance the first expiry of a published job is not renewed by it, as before. The one change there: a job that was still in publication at an earlier expiry now gets one poll from the expiry that first finds it published, where before it could fail at that expiry however recently it had been published. There is no bound on the sum of the allowances: a queue kept full of long commands occupies the GPU for as long as it is fed. Nor are the other clients shielded from such a queue: they share the GPU with it as the firmware schedules them, and a command of another client can wait until a long command ends (the numbers are below). The struct grows at its end: a caller passing the old size gets the default, and a kernel without the feature ignores the field. Measured on the J716c (M5 Max MacBook Pro), with compute jobs of a single-thread loop of known length. Before this change: - one job of 2.13 s or 2.41 s: killed at 2.04-2.05 s (ETIMEDOUT, the client sees a lost device); one of 1.86 s: completes. - four jobs of 0.93 s submitted back to back on one queue, with and without timestamp queries: all complete, the last 3.71 s after its submission; two of 1.6 s: the second completes 3.23 s after its submission. Jobs queued behind their own queue do not lose time. - one 1.1 s job on each of two queues of one process: both complete (2.24 s for the second), both queues share one firmware queue. With it, and a Vulkan driver that passes a requested allowance on: - no request: a 2.13 s job is killed at 2.04 s, as before. - 30000 requested: jobs of 5 s and 20 s complete; a 40 s job is killed 30.7 s after its submission, as it is when 600000 is requested. - three 16 s jobs submitted back to back on one queue: all complete, 16.0, 32.0 and 48.0 s after the first submission. - two processes each submitting a 1.1 s job that fills the GPU at the same moment: both complete, after 1.93 s and 2.37 s. Sharing, measured with one client keeping a queue full of compute jobs that fill the GPU (640 threads), of 1.5 s under the default allowance and of 5 s and 20 s with 30000 requested, while a second client submits a short compute job every 100 ms (a single thread of 5 ms, or 5 ms of work filling the GPU) under the default allowance: - On the integration/aurora of 2026-11-09 (about 1000 jobs of each kind beside the long jobs): the second client's jobs took a median of 6 to 9 ms and at most 105 ms, whatever the length of the first client's jobs. The desktop compositor's render jobs, about 1 ms on an idle GPU, took up to 0.7 s beside jobs of any of the three lengths, and 4 to 5 of them completed per second beside the 5 s and 20 s jobs against 9 to 10 beside the 1.5 s jobs and 43 idle; six of its jobs waited between 1.4 s and more than 2 s beside the long jobs, none beside the 1.5 s jobs. - On the integration/aurora this change is based on (about 730 jobs of the kind that fills the GPU): most took 6 ms, but six waited until about the end of the first client's running job: 1.5 s beside a 1.5 s job, and 3.4, 4.5, 5.7, 8.3 and 18.4 s beside 20 s jobs (the 4.5, 8.3 and 18.4 s ones with the first client's queue at low priority, which did not help). The second client's watchdog deferred the ones past two seconds behind the other VM's work; the last one ran out of deferrals after 18 s and failed with ETIMEDOUT, 2.3 s before the first client's job ended. So on the current tree a queue with a long allowance can hold up another client's command for as long as its own command runs, and past 18 s that client's work fails. Under the default allowance the same wait is bounded by the two seconds a command may run. Tests: the timeout tests (g17-ray-query-timeout-tests.sh on the host, 23 passed, and KUnit) cover the requested allowances, the default, a 30 s allowance counted from the head of the queue and ended at the first poll past it, a first expiry delayed by later pushes, and a job that waited in publication. The asahi-m4 layout probe compares the frozen queue_create bindings as a prefix of the grown struct and asserts the new tail. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/asahi_core.rs | 4 + drivers/gpu/drm/asahi/file.rs | 11 ++ drivers/gpu/drm/asahi/g17/job.rs | 17 ++- drivers/gpu/drm/asahi/g17/mod.rs | 3 +- drivers/gpu/drm/asahi/g17/queue/mod.rs | 22 +++- drivers/gpu/drm/asahi/g17/timeout.rs | 101 ++++++++++++++++++ drivers/gpu/drm/asahi/queue/mod.rs | 7 ++ include/uapi/drm/asahi_drm.h | 51 +++++++++ .../selftests/drivers/gpu/asahi-m4/abi.py | 25 ++++- 9 files changed, 235 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/asahi/asahi_core.rs b/drivers/gpu/drm/asahi/asahi_core.rs index 8dcfc3ad8ca40..6b840e524cf04 100644 --- a/drivers/gpu/drm/asahi/asahi_core.rs +++ b/drivers/gpu/drm/asahi/asahi_core.rs @@ -182,6 +182,10 @@ module! { default: 512, description: "G13/G14: GPU event-drain utilization minimum (0..1024)", }, + compute_timeout_max_ms: u32 { + default: 30000, + description: "G17: longest compute watchdog allowance a queue may set, ms (floor 2000)", + }, g15_deadline_ms: u32 { default: 5000, description: "G15: host deadline of a published prefix without progress, ms (min 100)", diff --git a/drivers/gpu/drm/asahi/file.rs b/drivers/gpu/drm/asahi/file.rs index 9fe3de9109731..c1dc902822f8b 100644 --- a/drivers/gpu/drm/asahi/file.rs +++ b/drivers/gpu/drm/asahi/file.rs @@ -102,6 +102,10 @@ pub(crate) const DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION: u64 = /// command may wait for it in firmware. pub(crate) const DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION: u32 = uapi::drm_asahi_compute_flags_DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION as u32; +/// Queues take a watchdog allowance for their compute commands +/// (`drm_asahi_queue_create::compute_timeout_ms`). +pub(crate) const DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT: u64 = + uapi::DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT as u64; /// The tiling half waits only on the render and compute prefixes of the command's stage-prefix /// fields, its fragment half on the whole ones. #[allow(dead_code)] @@ -1584,6 +1588,10 @@ impl File { data.flags, ); + if data.pad != 0 { + cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); + return Err(EINVAL); + } if data.flags != 0 || data.priority > uapi::drm_asahi_priority_DRM_ASAHI_PRIORITY_REALTIME { cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); return Err(EINVAL); @@ -1655,6 +1663,9 @@ impl File { )?; queue.reserve_render_scs(data.render_scs_layout)?; + if data.compute_timeout_ms != 0 { + data.compute_timeout_ms = queue.set_compute_timeout(data.compute_timeout_ms)?; + } data.queue_id = resv.index().try_into()?; resv.fill(Arc::pin_init( diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index 8d2e5f443fe14..8e71c0d25d69d 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -365,6 +365,8 @@ pub(crate) trait Backend: Send + Sync { struct Job { backend: Arc, packet: Arc, + /// The watchdog allowance of the job ([`super::timeout::Allowance`]), in ms. + allowance_ms: u32, } struct Timeout { @@ -372,6 +374,7 @@ struct Timeout { victim_renewals: u8, execution_renewals: u8, shared_renewals: u8, + allowance: super::timeout::Allowance, packet: Arc, } @@ -397,6 +400,7 @@ impl sched::JobImpl for Job { victim_renewals: 0, execution_renewals: 0, shared_renewals: 0, + allowance: super::timeout::Allowance::new(self.allowance_ms), packet: self.packet.clone(), } } @@ -405,6 +409,10 @@ impl sched::JobImpl for Job { if finished || job.packet.completion.take_replay_timeout() { return true; } + // Within the allowance its queue asked for. Past it, the deferrals below still apply. + if job.allowance.renew(job.packet.is_published(), super::now_ns()) { + return true; + } if super::timeout::renew_execution( job.packet.is_published(), job.packet.command.long_job(), &mut job.execution_renewals, ) { @@ -457,6 +465,7 @@ pub(crate) struct Scheduler { entity: sched::Entity>, _scheduler: sched::Scheduler>, backend: Arc, + allowance_ms: u32, } #[derive(Clone)] @@ -486,7 +495,11 @@ impl Fences { } impl Scheduler { - pub(crate) fn new(dev: &AsahiDrmDevice, backend: Arc, engine: Engine) -> Result { + /// `allowance_ms`: the watchdog allowance of each job ([`super::timeout::allowance_ms`]); + /// the watchdog still polls every [`TIMEOUT_MS`]. + pub(crate) fn new(dev: &AsahiDrmDevice, backend: Arc, engine: Engine, allowance_ms: u32) + -> Result + { let (credits, name) = match engine { Engine::Render => ( RENDER_CREDITS.checked_mul(backend.render_pairs()).ok_or(EOVERFLOW)?, @@ -513,6 +526,7 @@ impl Scheduler { entity, _scheduler: scheduler, backend, + allowance_ms, }) } @@ -540,6 +554,7 @@ impl Scheduler { Job { backend: self.backend.clone(), packet: packet.clone(), + allowance_ms: self.allowance_ms, }, )?; if dependencies.len() > MERGED_DEPENDENCIES { diff --git a/drivers/gpu/drm/asahi/g17/mod.rs b/drivers/gpu/drm/asahi/g17/mod.rs index 88610416beab1..7ec5e7808e929 100644 --- a/drivers/gpu/drm/asahi/g17/mod.rs +++ b/drivers/gpu/drm/asahi/g17/mod.rs @@ -2458,7 +2458,8 @@ impl gpu::Gpu for Gpu { u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_PER_COMMAND_INPUT_SYNCS) | u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS) | u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS) - | sync_output::FEATURE_CUMULATIVE_SIGNALS; + | sync_output::FEATURE_CUMULATIVE_SIGNALS + | crate::file::DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT; let variant = self.shared.variant; // The M5 command set has no AS-build flag: its validation refuses one. if !variant.m5_commands { diff --git a/drivers/gpu/drm/asahi/g17/queue/mod.rs b/drivers/gpu/drm/asahi/g17/queue/mod.rs index 47a506a205615..41af18adbc517 100644 --- a/drivers/gpu/drm/asahi/g17/queue/mod.rs +++ b/drivers/gpu/drm/asahi/g17/queue/mod.rs @@ -89,6 +89,9 @@ pub(crate) struct Queue { backend: Arc, render: Option>, compute: Option>, + /// The watchdog allowance of each compute command, set before the first one + /// ([`crate::queue::Queue::set_compute_timeout`]). + compute_allowance_ms: u32, fences: FenceContexts, allocation_failure: Fence, range: Range, @@ -127,7 +130,12 @@ impl Queue { window, variant, } = frontend; - let render = Some(Scheduler::new(dev, backend.clone(), Engine::Render)?); + let render = Some(Scheduler::new( + dev, + backend.clone(), + Engine::Render, + super::timeout::DEFAULT_ALLOWANCE_MS, + )?); Ok(Self { dev: dev.into(), context, @@ -135,6 +143,7 @@ impl Queue { backend, render, compute: None, + compute_allowance_ms: super::timeout::DEFAULT_ALLOWANCE_MS, fences, allocation_failure, range, @@ -158,6 +167,7 @@ impl Queue { &self.dev, self.backend.clone(), Engine::Compute, + self.compute_allowance_ms, )?); } Ok(()) @@ -513,6 +523,16 @@ impl Queue { } impl crate::queue::Queue for Queue { + fn set_compute_timeout(&mut self, requested_ms: u32) -> Result { + // The compute scheduler takes the allowance when the first compute command creates it. + if self.compute.is_some() { + return Err(EBUSY); + } + let maximum = crate::module_parameters::compute_timeout_max_ms.value(); + self.compute_allowance_ms = super::timeout::allowance_ms(requested_ms, maximum); + Ok(self.compute_allowance_ms) + } + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { let pages = [ super::fw::compute::ScratchRequest::new(layouts[0])?.pages(), diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index 3da5e255408b5..d42f4d1e7d856 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -52,6 +52,59 @@ pub(crate) fn as_build(flags: u32) -> Option { if flags & !AS_BUILD != 0 { None } else { Some(flags & AS_BUILD != 0) } } +/// The watchdog allowance of a compute command unless its queue asked for +/// another (`drm_asahi_queue_create::compute_timeout_ms`): one watchdog poll. +pub(crate) const DEFAULT_ALLOWANCE_MS: u32 = POLL_MS; + +/// The allowance a queue gets for its compute commands when it asks for +/// `requested_ms`, given the longest the system allows (`maximum_ms`, the +/// `compute_timeout_max_ms` module parameter): never below the default. +pub(crate) fn allowance_ms(requested_ms: u32, maximum_ms: u32) -> u32 { + requested_ms.min(maximum_ms).max(DEFAULT_ALLOWANCE_MS) +} + +/// A job's watchdog allowance: how long the watchdog renews it for the asking, +/// counted from about when it reached the head of its queue. +/// +/// It is not a deadline. The watchdog only sees a job at its expiries, and +/// does not know when the job started: the scheduler arms a queue's watchdog +/// when a job reaches the head (the job ahead of it completed) and again +/// whenever a later job of the queue is pushed, and the watchdog expires one +/// [`POLL_MS`] later. So a job's first expiry comes at least one poll after it +/// reached the head, later if pushes postponed it, and the time is counted +/// from one poll before that expiry. A job still in publication at an expiry +/// (waiting for capacity) has not started: its time counts from the first +/// expiry that finds it published. Either way a job never loses time to the +/// jobs ahead of it, and it may get up to one poll more than its allowance. +/// Under the default allowance it is never renewed. A job past its allowance +/// is not failed for that alone: the caller's other renewals and deferrals +/// are tried next. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Allowance { + ms: u32, + /// When the job reached the head, once an expiry placed it. + head_since: Option, + /// An expiry found the job still in publication. + waited: bool, +} + +impl Allowance { + pub(crate) const fn new(ms: u32) -> Self { + Self { ms, head_since: None, waited: false } + } + + /// Whether the job is within its allowance at the expiry at `now` (host ns). + pub(crate) fn renew(&mut self, published: bool, now: u64) -> bool { + if !published { + self.waited = true; + return false; + } + let poll = if self.waited { 0 } else { u64::from(POLL_MS) * 1_000_000 }; + let since = *self.head_since.get_or_insert(now.saturating_sub(poll)); + now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 + } +} + pub(crate) fn renew_execution(published: bool, enabled: bool, renewals: &mut u8) -> bool { if !published || !enabled || *renewals >= EXECUTION_RENEWALS { return false; @@ -201,6 +254,54 @@ pub(crate) fn behind_own_head(published: bool, older: Option, progressing: #[cfg_attr(not(test), kernel::macros::kunit_tests(asahi_g17_execution_allowance))] mod tests { use super::*; + const MS: u64 = 1_000_000; + #[test] + fn a_queue_gets_the_allowance_it_asks_for_within_bounds() { + assert_eq!(allowance_ms(0, 30_000), DEFAULT_ALLOWANCE_MS); + assert_eq!(allowance_ms(500, 30_000), DEFAULT_ALLOWANCE_MS); + assert_eq!(allowance_ms(20_000, 30_000), 20_000); + assert_eq!(allowance_ms(u32::MAX, 30_000), 30_000); + // A maximum below the default still leaves the default. + assert_eq!(allowance_ms(20_000, 0), DEFAULT_ALLOWANCE_MS); + } + #[test] + fn the_default_allowance_renews_nothing() { + let mut allowance = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!allowance.renew(true, 100_000 * MS)); + // Not after a wait in publication either. + let mut allowance = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!allowance.renew(false, 100_000 * MS)); + assert!(allowance.renew(true, 102_000 * MS)); + assert!(!allowance.renew(true, 104_000 * MS)); + } + #[test] + fn a_long_allowance_runs_from_the_head_of_the_queue() { + // Reached the head at 10 s (the job ahead completed): first expiry at 12 s. + let start = 10_000 * MS; + let poll = u64::from(POLL_MS) * MS; + let mut allowance = Allowance::new(30_000); + let mut expiry = start + poll; + while allowance.renew(true, expiry) { + expiry += poll; + } + assert_eq!(allowance.head_since, Some(start)); + // Killed at the first poll at or past 30 s after it reached the head. + assert_eq!(expiry - start, 30_000 * MS); + // A first expiry delayed by later pushes only moves the count later. + let mut allowance = Allowance::new(30_000); + assert!(allowance.renew(true, start + poll + 500 * MS)); + assert_eq!(allowance.head_since, Some(start + 500 * MS)); + } + #[test] + fn a_job_waiting_in_publication_starts_at_its_first_published_expiry() { + let mut allowance = Allowance::new(30_000); + assert!(!allowance.renew(false, 10_000 * MS)); + assert_eq!(allowance.head_since, None); + assert!(allowance.renew(true, 12_000 * MS)); + assert_eq!(allowance.head_since, Some(12_000 * MS)); + assert!(allowance.renew(true, 41_999 * MS)); + assert!(!allowance.renew(true, 42_000 * MS)); + } #[test] fn selects_compute_both_render_halves_and_builds() { assert!(!select(false, [false; 2], false)); diff --git a/drivers/gpu/drm/asahi/queue/mod.rs b/drivers/gpu/drm/asahi/queue/mod.rs index 525a2ffffbbfe..a37bb48fa613f 100644 --- a/drivers/gpu/drm/asahi/queue/mod.rs +++ b/drivers/gpu/drm/asahi/queue/mod.rs @@ -70,6 +70,13 @@ fn can_submit_inline(dependencies_ready: bool, previous_status: Option) -> /// Trait implemented by all versioned queues. pub(crate) trait Queue: Send + Sync { + /// Sets the watchdog allowance of the queue's compute commands to `requested_ms` + /// (`drm_asahi_queue_create::compute_timeout_ms`, nonzero) and returns the allowance in + /// effect. Only queues of GPUs that advertise DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT take one. + fn set_compute_timeout(&mut self, _requested_ms: u32) -> Result { + Err(EINVAL) + } + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { if layouts != [0; 2] { return Err(EINVAL); diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index c51b239fb4917..f41bf829e7b7d 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -495,6 +495,13 @@ enum drm_asahi_feature { */ #define DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION ((1ULL) << 34) +/* + * DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT: drm_asahi_queue_create::compute_timeout_ms + * sets the watchdog allowance of the queue's compute commands. A feature bit + * like those of enum drm_asahi_feature, defined here like bit 32. + */ +#define DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT ((1ULL) << 36) + /* * DRM_ASAHI_FEATURE_USERPTR: DRM_IOCTL_ASAHI_GEM_USERPTR is available. A * feature bit like those of enum drm_asahi_feature, defined here like bit 32. @@ -1112,6 +1119,50 @@ struct drm_asahi_queue_create { * Requires DRM_ASAHI_FEATURE_RENDER_RAY_QUERY; zero otherwise. */ __u64 render_scs_layout[2]; + + /** + * @compute_timeout_ms: Watchdog allowance of each compute command of + * this queue, in milliseconds. Zero keeps the driver's default. + * Requires DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT; zero otherwise. + * + * The driver's watchdog polls a queue every two seconds. By default it + * takes a compute command that a poll finds still running at the head + * of the queue (every earlier compute command of the queue has + * completed) to have hung, and fails it, and the queue's VM with it. + * With an allowance, the watchdog lets the command run on at each poll + * until the allowance has passed. For a nonzero request the driver + * raises a shorter allowance to its default of one poll, lowers a + * longer one to the maximum the system allows, and returns the + * allowance in effect here; a request of 0 is returned unchanged. + * + * This is an allowance, not a deadline: + * + * - The watchdog acts at its polls only, so a command may run up to + * one poll longer than its allowance. + * - The start of a command is known only approximately. The time is + * counted from one poll before the first poll that finds the command + * at the head of the queue. That is no earlier than the command got + * there and may be later: commands submitted to the queue in the + * meantime postpone that poll. + * - Once the allowance has passed, the deferrals the watchdog grants + * any command still apply, as they do by default: a command it takes + * to be waiting behind other clients' work, or one of a kind it + * gives extra polls, runs on until those are used up as well. + * + * The allowance is per command, and nothing bounds the sum: a queue + * that is kept full of long commands occupies the GPU for as long as + * it is fed. The driver does not shield the other clients from such a + * queue: they share the GPU with it as the firmware schedules them, + * and a command of another client may have to wait until a long + * command ends, which can be longer than that client's own allowance + * and the deferrals it gets. Userspace should ask for an allowance + * only for queues whose commands need one, and keep those commands as + * short as it can. + */ + __u32 compute_timeout_ms; + + /** @pad: MBZ */ + __u32 pad; }; /** diff --git a/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py b/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py index 7b9bbce635e6e..f9da7cd001c91 100644 --- a/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py +++ b/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py @@ -62,15 +62,13 @@ def run_measurements(command, source, binary): def c_layout(header, bindings, output, tag): keys, _ = expressions(bindings) capacity_tail = 'client_memory_limit;' in header.read_text() + queue_tail = 'compute_timeout_ms;' in header.read_text() # Linux 7.2 folds the historical extension structs into the base payloads. # Compare the same wire prefixes and offsets while keeping the C oracle frozen. if 'struct drm_asahi_cmd_render_with_prior {' not in header.read_text(): translated = [] for key, expr in keys: kind, _, field = key.partition(':') - if capacity_tail and key == 'drm_asahi_params_global:size': - # Compare the frozen reply prefix; validate the entire known additive tail below. - expr = 'offsetof(struct drm_asahi_params_global, memory_limit)' if key == "drm_asahi_vm_create:pad": expr = "offsetof(struct drm_asahi_vm_create, flags)" base = None @@ -90,6 +88,19 @@ def c_layout(header, bindings, output, tag): else: expr = f'offsetof(struct {base}, {field})' translated.append((key, expr)) keys = translated + # The two additive tails apply to every header that carries them, including + # one that still has the historical extension structs. Compare the frozen + # reply prefixes; validate the entire known additive tails below. + tails = {} + if capacity_tail: + tails['drm_asahi_params_global:size'] = \ + 'offsetof(struct drm_asahi_params_global, memory_limit)' + if queue_tail: + tails['drm_asahi_queue_create:size'] = \ + 'offsetof(struct drm_asahi_queue_create, compute_timeout_ms)' + if tails: + keys = [(key, tails[key]) if key in tails else (key, expr) + for key, expr in keys] source = output / (tag + '.c') lines = ['#include ', '#include ', '#include ' + json.dumps(str(header)), 'int main(void) {'] @@ -101,6 +112,14 @@ def c_layout(header, bindings, output, tag): '_Static_assert(sizeof(((struct drm_asahi_params_global *)0)->memory_limit) == 8, "capacity width");', '_Static_assert(sizeof(((struct drm_asahi_params_global *)0)->client_memory_limit) == 8, "client capacity width");', ] + if queue_tail: + lines[3:3] = [ + '_Static_assert(sizeof(struct drm_asahi_queue_create) == 48, "queue time limit reply size");', + '_Static_assert(offsetof(struct drm_asahi_queue_create, compute_timeout_ms) == 40, "queue time limit offset");', + '_Static_assert(sizeof(((struct drm_asahi_queue_create *)0)->compute_timeout_ms) == 4, "queue time limit width");', + '_Static_assert(offsetof(struct drm_asahi_queue_create, pad) == 44, "queue time limit pad offset");', + '_Static_assert(sizeof(((struct drm_asahi_queue_create *)0)->pad) == 4, "queue time limit pad width");', + ] lines += [f'printf("{key}=%llu\\n", (unsigned long long)({expr}));' for key, expr in keys] source.write_text('\n'.join(lines + ['return 0;', '}']) + '\n') return run_measurements(shlex.split(os.environ.get('CC', 'cc')) + From b5ca79e11523076ea081cf6c705531c668c3c5d4 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Thu, 12 Nov 2026 19:52:49 -0500 Subject: [PATCH 2/3] drm/asahi: g17: don't fail work waiting behind a command inside its allowance With a compute allowance, one client's command may run for up to compute_timeout_max_ms (30 s by default), and another client's command, render or compute, can wait on the GPU until it ends. The waiting job's watchdog defers it behind the other VM's older work at most eight polls (MAX_RENEWALS, 16 s) and then fails it: a compositor frame that waited behind a 20 s compute command of another client failed with ETIMEDOUT after about 18 s, and the compositor lost its device. A job's watchdog now records when its allowance ends in the packet, and the cross-VM heads carry whether that job is still inside it. A waiting job whose blocker is live and inside its allowance is renewed with a new decision, Excuse, which does not use its eight renewals. Once the blocker completes, or its allowance has passed, the existing rules apply again, with what the job has left of its eight renewals for the blocker's own failure and teardown. Unchanged: - A blocker that already failed excuses nothing, as before. - The oldest unfinished work is never excused: it has no blocker. - A blocker under the default allowance excuses nothing, so clients that do not ask for an allowance see the decisions they saw before. Render and compute watchdogs decide through the same attribution, and both see compute heads as blockers, so a render job waiting behind a long compute command is covered too. The excuse is bounded on the waiting job's side: at most compute_timeout_max_ms in polls (15 with the default 30 s), however many blockers take turns. This rule therefore renews a job at most 15 + 8 times by default, 46 s of renewals. A job's own allowance and its other deferrals are counted separately and come on top, and each renewal re-arms a timer rather than an absolute deadline, so the time until a waiting job is failed (spared) can be longer than that. The watchdog lines for cross-VM decisions now print the excused polls and whether the blocker was inside its allowance, and are written at the first excuse, the first counted deferral and the final decision. Tests: KUnit and the host timeout tests cover a live blocker inside its allowance past eight renewals (excused), the excuse used up (renewals, then spared), a blocker past its allowance and a failed one (the decisions as before), no blocker and the oldest work (own), the bound, blockers taking turns (the counters are the waiting job's and are not reset), and when an allowance ends. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/g17/job.rs | 48 +++- drivers/gpu/drm/asahi/g17/queue/compute.rs | 1 + drivers/gpu/drm/asahi/g17/queue/render.rs | 1 + drivers/gpu/drm/asahi/g17/runtime/backend.rs | 30 +-- .../drm/asahi/g17/runtime/compute_control.rs | 20 +- drivers/gpu/drm/asahi/g17/runtime/render.rs | 2 + drivers/gpu/drm/asahi/g17/timeout.rs | 215 ++++++++++++++++-- include/uapi/drm/asahi_drm.h | 9 +- 8 files changed, 276 insertions(+), 50 deletions(-) diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index 8e71c0d25d69d..eccce9ea3dc84 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -4,7 +4,7 @@ use super::{command::Validated, completion::Completion, context::Context}; use crate::driver::AsahiDrmDevice; -use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU8, Ordering}; +use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU64, AtomicU8, Ordering}; use kernel::{ bindings, c_str, cpu::CpuId, @@ -128,6 +128,9 @@ pub(crate) struct Packet { cancelled: AtomicBool, early: AtomicI32, backlog: AtomicBool, + /// When the job's watchdog allowance ends (host ns, [`super::timeout::Allowance::end`]): + /// zero under the default allowance, `u64::MAX` for a longer one not yet placed. + allowance_end: AtomicU64, } impl Packet { @@ -160,11 +163,18 @@ impl Packet { cancelled: AtomicBool::new(false), early: AtomicI32::new(0), backlog: AtomicBool::new(false), + allowance_end: AtomicU64::new(0), }, GFP_KERNEL, )?) } + /// Its own watchdog still renews the job for the allowance its queue asked for at `now` + /// ([`super::timeout::Head::excused`]). + pub(crate) fn within_allowance(&self, now: u64) -> bool { + now < self.allowance_end.load(Ordering::Relaxed) + } + pub(crate) fn engine(&self) -> Engine { match &self.command { Validated::Render { .. } => Engine::Render, @@ -350,8 +360,12 @@ pub(crate) trait Backend: Send + Sync { fn publish(&self, packet: Arc) -> Result; fn timed_out(&self, packet: Arc) -> sched::Status; fn cancel(&self, packet: Arc); - /// Transfer a timeout to an outstanding off-lock graph transaction, if any. - fn defer_timeout(&self, packet: &Arc, renewals: u8) -> bool; + /// Transfer a timeout to an outstanding off-lock graph transaction, if any, or decide it + /// against other VMs' older work ([`super::timeout::decide`]). `renewals` and `excused` + /// count the job's earlier [`super::timeout::Decision::Defer`] and + /// [`super::timeout::Decision::Excuse`] results; the job is renewed on either. + fn defer_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> super::timeout::Decision; /// A long job of another queue is running beside `packet`. fn long_job_beside(&self, _packet: &Arc) -> bool { false } /// A published job's deadline may be renewed while other work retires @@ -372,6 +386,9 @@ struct Job { struct Timeout { backend: Arc, victim_renewals: u8, + /// Polls excused behind other VMs' live work inside its allowance + /// ([`super::timeout::Decision::Excuse`]). + excused: u32, execution_renewals: u8, shared_renewals: u8, allowance: super::timeout::Allowance, @@ -398,6 +415,7 @@ impl sched::JobImpl for Job { Timeout { backend: self.backend.clone(), victim_renewals: 0, + excused: 0, execution_renewals: 0, shared_renewals: 0, allowance: super::timeout::Allowance::new(self.allowance_ms), @@ -410,7 +428,12 @@ impl sched::JobImpl for Job { return true; } // Within the allowance its queue asked for. Past it, the deferrals below still apply. - if job.allowance.renew(job.packet.is_published(), super::now_ns()) { + let within = job.allowance.renew(job.packet.is_published(), super::now_ns()); + // Other VMs' watchdogs read the end to excuse work waiting behind this one. + if let Some(end) = job.allowance.end() { + job.packet.allowance_end.store(end, Ordering::Relaxed); + } + if within { return true; } if super::timeout::renew_execution( @@ -426,11 +449,17 @@ impl sched::JobImpl for Job { if super::timeout::renew_shared(job.packet.is_published(), shared, &mut job.shared_renewals) { return true; } - let deferred = job.backend.defer_timeout(&job.packet, job.victim_renewals); - if deferred { - job.victim_renewals = job.victim_renewals.saturating_add(1); + match job.backend.defer_timeout(&job.packet, job.victim_renewals, job.excused) { + super::timeout::Decision::Defer => { + job.victim_renewals = job.victim_renewals.saturating_add(1); + true + } + super::timeout::Decision::Excuse => { + job.excused = job.excused.saturating_add(1); + true + } + super::timeout::Decision::Own | super::timeout::Decision::Spare => false, } - deferred } fn prepare(job: &mut Self) -> Option { @@ -549,6 +578,9 @@ impl Scheduler { } else { None }; + if self.allowance_ms > super::timeout::DEFAULT_ALLOWANCE_MS { + packet.allowance_end.store(u64::MAX, Ordering::Relaxed); + } let mut job = self.entity.new_job( 1, Job { diff --git a/drivers/gpu/drm/asahi/g17/queue/compute.rs b/drivers/gpu/drm/asahi/g17/queue/compute.rs index 99d862a4f71ca..8042e2026451b 100644 --- a/drivers/gpu/drm/asahi/g17/queue/compute.rs +++ b/drivers/gpu/drm/asahi/g17/queue/compute.rs @@ -909,6 +909,7 @@ impl Queue { active.observed.is_some() }, failed: self.quarantined, + excused: active.packet.within_allowance(crate::g17::now_ns()), }) } diff --git a/drivers/gpu/drm/asahi/g17/queue/render.rs b/drivers/gpu/drm/asahi/g17/queue/render.rs index 16e9d0e265571..c68448f4503c3 100644 --- a/drivers/gpu/drm/asahi/g17/queue/render.rs +++ b/drivers/gpu/drm/asahi/g17/queue/render.rs @@ -1667,6 +1667,7 @@ impl Pair { published: active.published, finished: false, failed: self.quarantined, + excused: active.packet.within_allowance(crate::g17::now_ns()), }) } diff --git a/drivers/gpu/drm/asahi/g17/runtime/backend.rs b/drivers/gpu/drm/asahi/g17/runtime/backend.rs index 705325bf6d153..b17fe9eb07d11 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/backend.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/backend.rs @@ -386,19 +386,20 @@ impl job::Backend for Backend { .any(|other| other.owner() != Some(self.owner) && other.long_job_running()) } - fn defer_timeout(&self, packet: &Arc, renewals: u8) -> bool { + fn defer_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> crate::g17::timeout::Decision { if packet.engine() == Engine::Compute { - return self.defer_compute_timeout(packet, renewals); + return self.defer_compute_timeout(packet, renewals, excused); } use crate::g17::timeout::{self, Decision}; // As for compute, a final retirement pass precedes the attribution, and all completion // callbacks run after releasing the device mutex. Without a collector the attribution // goes ahead without that pass, as it did before it existed. let mut deferred = DeferredBatch::worker().ok(); - let result = (|| -> Result { + let result = (|| -> Result { let mut state = self.shared.state.lock(); let Some(firmware) = (*state).as_deref_mut() else { - return Ok(false); + return Ok(Decision::Own); }; if let Some(deferred) = deferred.as_mut() { firmware.scan_renders(deferred)?; @@ -409,15 +410,17 @@ impl job::Backend for Backend { "Render watchdog owner={} behind its context's older node {:#x} (own {:#x}); the firmware progresses: renewed\n", self.owner, older, own); } - return Ok(true); + return Ok(Decision::Defer); } // Borrowed-graph and queued-retirement deferrals share the renewal bound. if timeout::transaction_grace(renewals) && firmware.queues.defer_render_timeout(packet, ETIMEDOUT) { - return Ok(true); + return Ok(Decision::Defer); } - let verdict = firmware.queues.render_timeout_decision(packet, renewals); + let excuse = timeout::Excuse { used: excused, limit: timeout::excuse_limit( + crate::module_parameters::compute_timeout_max_ms.value()) }; + let verdict = firmware.queues.render_timeout_decision(packet, renewals, excuse); let decision = verdict.map_or(Decision::Own, |verdict| verdict.decision); // As compute does: work still blocked behind another VM's older unfinished work // after the whole grace is innocent, so its failure must stay non-sticky. The pair's @@ -426,19 +429,20 @@ impl job::Backend for Backend { packet.completion.classify(crate::g17::recovery::Class::Spared); } if let Some(verdict) = verdict { - if let Some(blocker) = verdict.reported_blocker(renewals) { + if let Some(blocker) = verdict.reported_blocker(renewals, excuse) { dev_warn!(self.shared.dev.as_ref(), - "Render watchdog owner={} vm={} publication={} renewals={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={}\n", - self.owner, verdict.own.vm, verdict.own.published, renewals, decision, - blocker.vm, blocker.published, u8::from(blocker.failed)); + "Render watchdog owner={} vm={} publication={} renewals={} excused={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={} blocker_in_allowance={}\n", + self.owner, verdict.own.vm, verdict.own.published, renewals, excused, + decision, blocker.vm, blocker.published, u8::from(blocker.failed), + u8::from(blocker.excused)); } } - Ok(decision == Decision::Defer) + Ok(decision) })(); if let Some(deferred) = deferred.as_mut() { deferred.finish(); } - matches!(result, Ok(true)) + result.unwrap_or(Decision::Own) } fn try_early(&self, packet: &Arc) -> bool { diff --git a/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs index dfa343a9ac275..922f4340aae3f 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs @@ -9,10 +9,11 @@ use kernel::{prelude::*, sync::Arc}; impl Backend { /// A final retirement pass precedes bounded cross-VM timeout attribution. /// All completion callbacks run after releasing the device mutex. - pub(super) fn defer_compute_timeout(&self, packet: &Arc, renewals: u8) -> bool { + pub(super) fn defer_compute_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> crate::g17::timeout::Decision { use crate::g17::{recovery::Class, timeout::{self, Decision}}; let Ok(mut deferred) = DeferredBatch::worker() else { - return false; + return Decision::Own; }; let result = (|| -> Result { let mut state = self.shared.state.lock(); @@ -39,22 +40,25 @@ impl Backend { .filter(|other| other.owner() != Some(self.owner)) .filter_map(|other| other.timeout_head()); let others = Iterator::chain(computes, firmware.queues.render_timeout_heads()); - let verdict = timeout::attribute(own, renewals, others); + let excuse = timeout::Excuse { used: excused, limit: timeout::excuse_limit( + crate::module_parameters::compute_timeout_max_ms.value()) }; + let verdict = timeout::attribute(own, renewals, excuse, others); let decision = verdict.decision; if decision == Decision::Spare { firmware.queues.compute(self.owner).ok_or(EIO)? .classify(&|_| Class::Spared); } - if let Some(blocker) = verdict.reported_blocker(renewals) { + if let Some(blocker) = verdict.reported_blocker(renewals, excuse) { dev_warn!(self.shared.dev.as_ref(), - "Compute watchdog owner={} vm={} publication={} renewals={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={}\n", - self.owner, own.vm, own.published, renewals, decision, - blocker.vm, blocker.published, u8::from(blocker.failed)); + "Compute watchdog owner={} vm={} publication={} renewals={} excused={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={} blocker_in_allowance={}\n", + self.owner, own.vm, own.published, renewals, excused, decision, + blocker.vm, blocker.published, u8::from(blocker.failed), + u8::from(blocker.excused)); } Ok(decision) })(); deferred.finish(); - matches!(result, Ok(Decision::Defer)) + result.unwrap_or(Decision::Own) } /// The older node of the packet's context ahead of its own, with its own word, when the diff --git a/drivers/gpu/drm/asahi/g17/runtime/render.rs b/drivers/gpu/drm/asahi/g17/runtime/render.rs index 4dac5f5230237..0218e48da42b3 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/render.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render.rs @@ -1271,6 +1271,7 @@ impl Registry { &self, packet: &Arc, renewals: u8, + excuse: crate::g17::timeout::Excuse, ) -> Option { let own = self .render @@ -1291,6 +1292,7 @@ impl Registry { crate::g17::timeout::attribute( own, renewals, + excuse, Iterator::chain(self.render_timeout_heads(), computes), ) }) diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index d42f4d1e7d856..9367b9c57efe3 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -3,6 +3,9 @@ //! Bounded isolation for a compute watchdog blocked behind another VM's older kick. //! Publication order is evidence of a possible blocker, not proof of a firmware //! dependency. Renewals are bounded; the oldest unfinished queue is never excused. +//! A blocker that runs inside the watchdog allowance its queue asked for excuses +//! the work behind it for that long ([`decide`]), bounded by the longest allowance +//! the system permits ([`excuse_limit`]). pub(crate) const MAX_RENEWALS: u8 = 8; @@ -20,8 +23,8 @@ const EXECUTION_RENEWALS: u8 = 4; /// How long an innocent client's publication waits for a recovery to settle and its /// spared work to drain, instead of failing with EBUSY: the watchdog defers work behind -/// another VM at most this long before it settles it (`classify`; ray work gets the -/// execution renewals on top). +/// another VM's failed work at most this long before it settles it (`classify`; ray work +/// gets the execution renewals on top). A failed blocker excuses nothing ([`decide`]). pub(crate) const RECOVERY_WAIT_MS: u32 = POLL_MS * (1 + EXECUTION_RENEWALS as u32 + MAX_RENEWALS as u32); @@ -103,6 +106,30 @@ impl Allowance { let since = *self.head_since.get_or_insert(now.saturating_sub(poll)); now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 } + + /// When the allowance ends (host ns), once an expiry placed the job, for an allowance + /// longer than the default: until then the watchdog renews the job for the asking. A + /// default allowance renews at most the poll after a wait in publication and excuses + /// nobody ([`Head::excused`]). + pub(crate) fn end(&self) -> Option { + let since = self.head_since.filter(|_| self.ms > DEFAULT_ALLOWANCE_MS)?; + Some(since.saturating_add(u64::from(self.ms) * 1_000_000)) + } +} + +/// How many polls a job may be excused behind other VMs' live work inside those blockers' +/// allowances ([`decide`]): the longest allowance a queue may get (`maximum_ms`, the +/// `compute_timeout_max_ms` module parameter) in whole polls, 15 by default. +pub(crate) fn excuse_limit(maximum_ms: u32) -> u32 { + allowance_ms(u32::MAX, maximum_ms).div_ceil(POLL_MS) +} + +/// The polls a waiting job was excused so far (`used`) and the most it may be (`limit`, +/// [`excuse_limit`]). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Excuse { + pub(crate) used: u32, + pub(crate) limit: u32, } pub(crate) fn renew_execution(published: bool, enabled: bool, renewals: &mut u8) -> bool { @@ -153,12 +180,19 @@ pub(crate) struct Head { /// Its queue already failed (quarantined) and holds the GPU only until the firmware drains /// it. No later watchdog of its own will settle it. pub(crate) failed: bool, + /// Its own watchdog still renews it for the allowance its queue asked for + /// ([`Allowance::end`]): a long command running as allowed, not a hung one. + pub(crate) excused: bool, } #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) enum Decision { Own, + /// Renewed: counts against [`MAX_RENEWALS`]. Defer, + /// Renewed behind a live blocker inside its allowance: counts against [`Excuse::limit`] + /// only. + Excuse, Spare, } @@ -176,10 +210,27 @@ pub(crate) fn blocker(own: Head, others: impl Iterator) -> Option) -> Decision { +/// +/// A live blocker that its own watchdog still renews for its queue's allowance +/// ([`Head::excused`]) is not hung: its command may run that long, and work queued behind it +/// on the GPU waits for it. Such work is renewed without using any more of its MAX_RENEWALS, +/// so that what is left of that grace covers the blocker's own failure and teardown once the +/// allowance has passed; then the rules above apply again. +/// +/// The excuse is bounded by the waiting job's [`Excuse::limit`], the longest allowance the +/// system permits in polls, however many blockers take turns. This attribution therefore +/// renews a job at most `limit + MAX_RENEWALS` times: 23 with the default 30 s maximum, 46 s +/// of renewals after the expiry that first deferred it. That bounds this rule only. A job's +/// own allowance and its other deferrals ([`renew_execution`], [`renew_shared`]) are counted +/// separately and come on top, and each renewal re-arms a timer rather than an absolute +/// deadline, so the time until the job is failed (spared) can exceed 46 s. +pub(crate) fn decide(own: Head, renewals: u8, excuse: Excuse, blocker: Option) -> Decision { match blocker { _ if own.finished => Decision::Own, None => Decision::Own, + Some(blocker) if blocker.excused && !blocker.failed && excuse.used < excuse.limit => { + Decision::Excuse + } Some(_) if renewals < MAX_RENEWALS => Decision::Defer, Some(blocker) if blocker.failed => Decision::Own, Some(_) => Decision::Spare, @@ -195,16 +246,22 @@ pub(crate) struct Verdict { } impl Verdict { - /// The blocker a watchdog line names: at the first deferral and once the grace has run out. - pub(crate) fn reported_blocker(&self, renewals: u8) -> Option { - self.blocker.filter(|_| renewals == 0 || self.decision != Decision::Defer) + /// The blocker a watchdog line names: at the first excuse, at the first counted deferral + /// and once the grace has run out. + pub(crate) fn reported_blocker(&self, renewals: u8, excuse: Excuse) -> Option { + self.blocker.filter(|_| match self.decision { + Decision::Excuse => excuse.used == 0, + Decision::Defer => renewals == 0, + Decision::Own | Decision::Spare => true, + }) } } -pub(crate) fn attribute(own: Head, renewals: u8, others: impl Iterator) -> Verdict { +pub(crate) fn attribute(own: Head, renewals: u8, excuse: Excuse, + others: impl Iterator) -> Verdict { let blocker = blocker(own, others); Verdict { - decision: decide(own, renewals, blocker), + decision: decide(own, renewals, excuse, blocker), own, blocker, } @@ -351,14 +408,20 @@ mod tests { } assert!(background_stalled(Some(since), u64::MAX)); } + /// No excuse used yet, under the default maximum allowance. + const FRESH: Excuse = Excuse { used: 0, limit: 15 }; fn classify(own: Head, renewals: u8, others: impl Iterator) -> Decision { - attribute(own, renewals, others).decision + attribute(own, renewals, FRESH, others).decision } fn head(vm: u64, published: u64, finished: bool) -> Head { - Head { vm, published, finished, failed: false } + Head { vm, published, finished, failed: false, excused: false } } fn failed(vm: u64, published: u64) -> Head { - Head { vm, published, finished: false, failed: true } + Head { vm, published, finished: false, failed: true, excused: false } + } + /// A live head whose own watchdog still renews it for its allowance. + fn excused(vm: u64, published: u64) -> Head { + Head { excused: true, ..head(vm, published, false) } } #[test] fn ray_query_execution_is_bounded_at_ten_seconds() { @@ -492,19 +555,133 @@ mod tests { #[test] fn the_watchdog_line_names_the_blocker_at_the_first_deferral_and_the_end() { let own = head(9, 103, false); - let verdict = attribute(own, 0, [failed(5, 100)].into_iter()); + let verdict = attribute(own, 0, FRESH, [failed(5, 100)].into_iter()); let expected = Verdict { decision: Decision::Defer, own, blocker: Some(failed(5, 100)) }; assert_eq!(verdict, expected); - assert_eq!(verdict.reported_blocker(0), Some(failed(5, 100))); - assert_eq!(attribute(own, 1, [failed(5, 100)].into_iter()).reported_blocker(1), None); - let last = attribute(own, MAX_RENEWALS, [failed(5, 100)].into_iter()); + assert_eq!(verdict.reported_blocker(0, FRESH), Some(failed(5, 100))); + let renewed = attribute(own, 1, FRESH, [failed(5, 100)].into_iter()); + assert_eq!(renewed.reported_blocker(1, FRESH), None); + let last = attribute(own, MAX_RENEWALS, FRESH, [failed(5, 100)].into_iter()); assert_eq!(last.decision, Decision::Own); - assert_eq!(last.reported_blocker(MAX_RENEWALS), Some(failed(5, 100))); - let spared = attribute(own, MAX_RENEWALS, [head(5, 100, false)].into_iter()); + assert_eq!(last.reported_blocker(MAX_RENEWALS, FRESH), Some(failed(5, 100))); + let spared = attribute(own, MAX_RENEWALS, FRESH, [head(5, 100, false)].into_iter()); assert_eq!(spared.decision, Decision::Spare); - assert_eq!(spared.reported_blocker(MAX_RENEWALS), Some(head(5, 100, false))); + assert_eq!(spared.reported_blocker(MAX_RENEWALS, FRESH), Some(head(5, 100, false))); // Nothing blocks: nothing to report. - assert_eq!(attribute(own, 0, [].into_iter()).reported_blocker(0), None); + assert_eq!(attribute(own, 0, FRESH, [].into_iter()).reported_blocker(0, FRESH), None); + // An excuse is named at its first poll, and so is the first counted deferral after it. + let blocker = excused(5, 100); + let first = attribute(own, 0, FRESH, [blocker].into_iter()); + assert_eq!(first.decision, Decision::Excuse); + assert_eq!(first.reported_blocker(0, FRESH), Some(blocker)); + let later = Excuse { used: 3, ..FRESH }; + let excused_again = attribute(own, 0, later, [blocker].into_iter()); + assert_eq!(excused_again.reported_blocker(0, later), None); + let past = attribute(own, 0, later, [head(5, 100, false)].into_iter()); + assert_eq!(past.decision, Decision::Defer); + assert_eq!(past.reported_blocker(0, later), Some(head(5, 100, false))); + } + #[test] + fn the_excuse_is_bounded_by_the_longest_allowance() { + assert_eq!(excuse_limit(30_000), 15); + assert_eq!(excuse_limit(29_000), 15); + assert_eq!(excuse_limit(0), 1); + // The default maximum gives 15 + 8 renewals behind other VMs' live work: 46 s of them. + assert_eq!(POLL_MS * (excuse_limit(30_000) + u32::from(MAX_RENEWALS)), 46_000); + } + #[test] + fn blockers_taking_turns_do_not_extend_the_excuse_or_the_renewals() { + // Two other clients' commands, each inside its allowance, are ahead by turns for + // longer than the excuse lasts. The counters belong to the waiting job: it is renewed + // limit + MAX_RENEWALS times in all, whichever blocker is ahead, and then spared. + let own = head(9, 103, false); + let (mut renewals, mut excuse, mut polls) = (0u8, FRESH, 0u32); + loop { + let ahead = if polls % 2 == 0 { excused(5, 100) } else { excused(6, 101) }; + match attribute(own, renewals, excuse, [ahead].into_iter()).decision { + Decision::Excuse => excuse.used += 1, + Decision::Defer => renewals += 1, + decision => { + assert_eq!(decision, Decision::Spare); + break; + } + } + polls += 1; + assert!(polls < 100); + } + assert_eq!(excuse.used, FRESH.limit); + assert_eq!(renewals, MAX_RENEWALS); + assert_eq!(polls, FRESH.limit + u32::from(MAX_RENEWALS)); + // Renewals spent before an excuse are not given back by it. + let spent = attribute(own, 2, FRESH, [excused(5, 100)].into_iter()); + assert_eq!(spent.decision, Decision::Excuse); + let after = Excuse { used: FRESH.limit, ..FRESH }; + assert_eq!(attribute(own, 2, after, [head(5, 100, false)].into_iter()).decision, + Decision::Defer); + assert_eq!(attribute(own, MAX_RENEWALS, after, [head(5, 100, false)].into_iter()) + .decision, Decision::Spare); + } + #[test] + fn a_live_blocker_inside_its_allowance_defers_past_the_renewals() { + // A compositor frame behind another client's 20 s compute command: its own renewals + // stay untouched while the command runs inside its allowance. + let own = head(9, 103, false); + for used in 0..FRESH.limit { + let excuse = Excuse { used, ..FRESH }; + assert_eq!(attribute(own, 0, excuse, [excused(5, 100)].into_iter()).decision, + Decision::Excuse); + // Past eight renewals from an earlier blocker too. + assert_eq!(attribute(own, MAX_RENEWALS, excuse, [excused(5, 100)].into_iter()) + .decision, Decision::Excuse); + } + // The excuse is used up: the renewals apply, then the work is spared. + let used = Excuse { used: FRESH.limit, ..FRESH }; + assert_eq!(attribute(own, 0, used, [excused(5, 100)].into_iter()).decision, + Decision::Defer); + assert_eq!(attribute(own, MAX_RENEWALS, used, [excused(5, 100)].into_iter()).decision, + Decision::Spare); + } + #[test] + fn a_blocker_past_its_allowance_gets_todays_decision() { + let own = head(9, 103, false); + let past = head(5, 100, false); + for renewals in 0..MAX_RENEWALS { + assert_eq!(classify(own, renewals, [past].into_iter()), Decision::Defer); + } + assert_eq!(classify(own, MAX_RENEWALS, [past].into_iter()), Decision::Spare); + } + #[test] + fn a_failed_blocker_excuses_nothing() { + let own = head(9, 103, false); + let failed_in_allowance = Head { excused: true, ..failed(5, 100) }; + for renewals in 0..MAX_RENEWALS { + assert_eq!(classify(own, renewals, [failed_in_allowance].into_iter()), Decision::Defer); + } + assert_eq!(classify(own, MAX_RENEWALS, [failed_in_allowance].into_iter()), Decision::Own); + // A live blocker is chosen before a failed one, excused or not. + let others = [failed_in_allowance, excused(7, 101)]; + assert_eq!(classify(own, MAX_RENEWALS, others.into_iter()), Decision::Excuse); + } + #[test] + fn no_blocker_or_finished_work_is_own() { + assert_eq!(classify(head(9, 103, false), 0, [].into_iter()), Decision::Own); + // The oldest unfinished work is never excused, not even by later excused work. + assert_eq!(classify(head(9, 100, false), 0, [excused(5, 101)].into_iter()), Decision::Own); + assert_eq!(classify(head(9, 103, true), 0, [excused(5, 100)].into_iter()), Decision::Own); + // Same-VM work does not excuse either. + assert_eq!(classify(head(5, 103, false), MAX_RENEWALS, [excused(5, 100)].into_iter()), + Decision::Own); + } + #[test] + fn only_a_placed_long_allowance_has_an_end() { + let poll = u64::from(POLL_MS) * MS; + let mut default = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!default.renew(true, 10_000 * MS)); + assert_eq!(default.end(), None); + let mut long = Allowance::new(20_000); + assert_eq!(long.end(), None); + assert!(long.renew(true, 10_000 * MS)); + assert_eq!(long.end(), Some(10_000 * MS - poll + 20_000 * MS)); } #[test] fn same_vm_finished_and_later_work_do_not_block() { diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index f41bf829e7b7d..2cf1efb39db94 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -1154,8 +1154,13 @@ struct drm_asahi_queue_create { * it is fed. The driver does not shield the other clients from such a * queue: they share the GPU with it as the firmware schedules them, * and a command of another client may have to wait until a long - * command ends, which can be longer than that client's own allowance - * and the deferrals it gets. Userspace should ask for an allowance + * command ends. The watchdog does not fail such a waiting command, + * render or compute, while the earlier command it waits behind runs + * inside its allowance; afterwards the usual deferrals apply. A + * waiting command is excused this way for at most the longest + * allowance the system permits in total, so a command that keeps + * waiting behind other clients' work longer than that and its own + * deferrals is still failed. Userspace should ask for an allowance * only for queues whose commands need one, and keep those commands as * short as it can. */ From 772bb5b12cb1ce2974f877034aeb751c6ac53f68 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Thu, 12 Nov 2026 22:10:06 -0500 Subject: [PATCH 3/3] drm/asahi: g17: count an allowance against the shared renewals On the G17 GPUs other than the M5, the watchdog renews a published job for up to fourteen polls while the GPU keeps completing other work (renew_shared), which bounds a job under the default allowance at thirty seconds on a shared GPU. These renewals were tried once a queue's allowance had passed, so the two added up: with the longest allowance (30 s by default) a command on a busy GPU could run for about 58 s, and a 40 s command could complete there instead of failing at about 30 s as it does on the M5, where the shared renewals do not apply. A poll that an allowance renews beyond what the default allowance renews now counts as one of the job's shared renewals. The two run from the same start, and a job is renewed for the longer of its allowance and the thirty seconds sharing gives any job, not for their sum. The one poll every allowance renews after a wait in publication is not counted, so asking never leaves a job fewer polls than the default. For an ordinary compute command that never finishes, polled every two seconds: - With an allowance of 30 s it is failed one poll after 30 s on a shared GPU or alone, on every G17, instead of after about 58 s on a busy G17 other than the M5. - With a shorter allowance on a shared GPU it is failed at thirty seconds, as a command without an allowance is. - Under the default allowance nothing changes. The renewals and deferrals that do not depend on sharing still come on top, on every G17 as before: the extra polls of ray and acceleration- structure work, and the deferrals of work waiting behind other work. A poll that runs late also still moves the estimated start, as the comment of compute_timeout_ms says; that comment now also says that the shared renewals run alongside an allowance. Tests: the timeout unit tests (KUnit and host) apply the allowance and the shared renewals in the watchdog's order to a job that never finishes: under the default allowance it fails at 2 s alone and 30 s on a shared GPU; at 30 s alone and shared with 30000; at 6 s alone and 30 s shared with 6000; at 30 s shared with 2001 and 20000; at 40 s with 40000; and after a wait in publication every allowance from the default to 30000 fails at the same poll on a shared GPU. This was not run on a G17 other than the M5 Max, where the shared renewals do not apply. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/g17/job.rs | 6 +- drivers/gpu/drm/asahi/g17/timeout.rs | 82 +++++++++++++++++++++++++++- include/uapi/drm/asahi_drm.h | 6 +- 3 files changed, 91 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index eccce9ea3dc84..4fe94f23fd597 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -427,13 +427,17 @@ impl sched::JobImpl for Job { if finished || job.packet.completion.take_replay_timeout() { return true; } - // Within the allowance its queue asked for. Past it, the deferrals below still apply. + // Within the allowance its queue asked for. Past it, the deferrals below still apply; + // the shared renewals run alongside an allowance rather than after it. + let placing = job.allowance.placing_after_wait(job.packet.is_published()); let within = job.allowance.renew(job.packet.is_published(), super::now_ns()); // Other VMs' watchdogs read the end to excuse work waiting behind this one. if let Some(end) = job.allowance.end() { job.packet.allowance_end.store(end, Ordering::Relaxed); } if within { + super::timeout::count_allowance_poll(job.allowance.asked() && !placing, + &mut job.shared_renewals); return true; } if super::timeout::renew_execution( diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index 9367b9c57efe3..e45c3ee3c626e 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -107,12 +107,23 @@ impl Allowance { now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 } + /// Whether the queue asked for an allowance longer than the default. + pub(crate) fn asked(&self) -> bool { + self.ms > DEFAULT_ALLOWANCE_MS + } + + /// Whether a [`Self::renew`] at an expiry that finds the job `published` places it after a + /// wait in publication: the one poll even the default allowance renews. + pub(crate) fn placing_after_wait(&self, published: bool) -> bool { + published && self.waited && self.head_since.is_none() + } + /// When the allowance ends (host ns), once an expiry placed the job, for an allowance /// longer than the default: until then the watchdog renews the job for the asking. A /// default allowance renews at most the poll after a wait in publication and excuses /// nobody ([`Head::excused`]). pub(crate) fn end(&self) -> Option { - let since = self.head_since.filter(|_| self.ms > DEFAULT_ALLOWANCE_MS)?; + let since = self.head_since.filter(|_| self.asked())?; Some(since.saturating_add(u64::from(self.ms) * 1_000_000)) } } @@ -171,6 +182,19 @@ pub(crate) fn renew_shared(published: bool, shared: bool, renewals: &mut u8) -> true } +/// Counts a poll that an allowance renewed beyond what the default allowance renews +/// (`beyond_default`) as one of the job's shared renewals. The two run from the same start +/// instead of one after the other: where [`renew_shared`] applies, a job is renewed for the +/// longer of its allowance and the thirty seconds sharing gives any job, not for their sum (58 s +/// with the default maximum). The one poll every allowance renews after a wait in publication +/// ([`Allowance::placing_after_wait`]) is not counted, so that asking never leaves a job fewer +/// polls than the default. +pub(crate) fn count_allowance_poll(beyond_default: bool, renewals: &mut u8) { + if beyond_default { + *renewals = renewals.saturating_add(1).min(SHARED_RENEWALS); + } +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct Head { /// The owning VM's [`crate::mmu::Vm::id`], one per VM status; the watchdog lines print it. @@ -467,6 +491,62 @@ mod tests { assert!(!renew_shared(true, true, &mut renewals)); assert_eq!(POLL_MS * (u32::from(renewals) + 1), 30_000); } + /// When an ordinary job that never finishes is failed (ms), on a GPU that is `shared` + /// throughout or not at all, by the allowance and the shared renewals in the order the + /// watchdog applies them, with no execution renewal or deferral. With `waited`, the expiry + /// at 2 s finds it still in publication (a deferral keeps it) and the next one published. + fn fails_at(allowance_ms: u32, shared: bool, waited: bool) -> u32 { + let mut allowance = Allowance::new(allowance_ms); + let mut renewals = 0; + let mut expiry = POLL_MS; + if waited { + assert!(!allowance.renew(false, u64::from(expiry) * MS)); + expiry += POLL_MS; + } + loop { + let placing = allowance.placing_after_wait(true); + if allowance.renew(true, u64::from(expiry) * MS) { + count_allowance_poll(allowance.asked() && !placing, &mut renewals); + } else if !renew_shared(true, shared, &mut renewals) { + return expiry; + } + expiry += POLL_MS; + assert!(expiry < 1_000_000); + } + } + #[test] + fn an_allowance_and_the_shared_renewals_overlap() { + // Under the default allowance: one poll alone, thirty seconds on a shared GPU. + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, false, false), 2_000); + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, true, false), 30_000); + // The longest allowance is the limit with or without sharing: not 30 s + 28 s. + assert_eq!(fails_at(30_000, false, false), 30_000); + assert_eq!(fails_at(30_000, true, false), 30_000); + // A shorter one never leaves a job worse off than the default on a shared GPU. + assert_eq!(fails_at(6_000, false, false), 6_000); + assert_eq!(fails_at(6_000, true, false), 30_000); + assert_eq!(fails_at(20_000, true, false), 30_000); + assert_eq!(fails_at(2_001, true, false), 30_000); + // Above thirty seconds (a raised maximum) the allowance is the limit. + assert_eq!(fails_at(40_000, true, false), 40_000); + // After a wait in publication every allowance is placed at the first published expiry + // (4 s), and the poll renewed there is not counted: still never fewer than the default. + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, false, true), 6_000); + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, true, true), 34_000); + for ms in [2_001, 6_000, 20_000, 30_000] { + assert_eq!(fails_at(ms, true, true), 34_000); + } + assert_eq!(fails_at(6_000, false, true), 10_000); + assert_eq!(fails_at(30_000, false, true), 34_000); + // Nothing is counted for a poll the default renews too. + let mut renewals = 0; + count_allowance_poll(false, &mut renewals); + assert_eq!(renewals, 0); + for _ in 0..40 { + count_allowance_poll(true, &mut renewals); + } + assert_eq!(renewals, SHARED_RENEWALS); + } #[test] fn a_stopped_gpu_or_unpublished_job_keeps_the_deadline() { let mut renewals = 0; diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index 2cf1efb39db94..46be28c8ed4a1 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -1147,7 +1147,11 @@ struct drm_asahi_queue_create { * - Once the allowance has passed, the deferrals the watchdog grants * any command still apply, as they do by default: a command it takes * to be waiting behind other clients' work, or one of a kind it - * gives extra polls, runs on until those are used up as well. + * gives extra polls, runs on until those are used up as well. The + * polls it renews any command for while the GPU keeps completing + * other work (up to 30 s in all, on the GPUs where it does that) + * run alongside the allowance, not after it: a command is renewed + * for the longer of the two. * * The allowance is per command, and nothing bounds the sum: a queue * that is kept full of long commands occupies the GPU for as long as