From 55f9630711dbb6657718a71121803a7e8bec065f Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Tue, 10 Nov 2026 21:17:58 -0500 Subject: [PATCH 1/8] drm/asahi: GEM_RESIDENCY, keep an object's memory resident On G17, reclaim may release the backing of VM-private objects of idle VMs, which the VM's next job then waits to get back. A client that would rather not pay that for some of its objects (a server holding model weights between requests, Vulkan's memory priority 1.0) had no way to say which. DRM_IOCTL_ASAHI_GEM_RESIDENCY sets or clears a per-object keep flag, reported by DRM_ASAHI_FEATURE_GEM_RESIDENCY (bit 35) on G17. The flag is a priority among the objects that reclaim may release, and no more: - Which reclaim releases GPU memory does not change. The VM shrinkers are not memcg-aware, so only reclaim on behalf of the whole system calls them (kswapd, direct reclaim, a request to the root memory cgroup, drop_caches). Reclaim inside a memory cgroup below the root releases no GPU memory, kept or not. - Which VMs it takes from does not change: an idle one, with no job, binding change or other use in progress or in the last two seconds, and past the backoff that follows a restoration. The memory of a VM in use stays, kept or not. - Within such a VM, kept objects are released after the others, and only when those left the scan target unmet. - Contents advised DONTNEED are purged as before, kept or not. The request is therefore a hint. Userspace cannot rely on a kept object being resident, and a client that stays active, or whose memory cgroup is the one running out of memory, keeps its GPU memory whether it asks or not. A user VM now has two shrinkers. The existing one purges advised contents and releases the idle objects that are not kept, as before. The new one counts the kept objects and has a batch no scan target can reach, so the shrinker core calls its scan only when the target reaches everything it counted: at the last two of the thirteen reclaim priorities, or the last three once scans deferred by earlier passes have built up. Its scan releases what the first shrinker's does and then, only if that did not reach the target, the kept objects. It is registered after the first, so a reclaim pass that reaches it has asked the VM's first shrinker before. Reclaim comes to those priorities when the earlier ones did not free what it was asked for: under memory pressure that nothing else relieves, for a request that names more than the system has to spare, and at once with drop_caches. Kept objects are not a second pass of the first shrinker's scan because the shrinker core hides the reclaim priority from a scan: counted there, they would be released at any priority whenever a VM had nothing else to give, and left uncounted, a VM with only kept objects would never be scanned at all. The flag is changed under the object's reservation, which idle reclaim holds for the VM it walks, so a request takes effect for every reclaim pass that starts after the ioctl returns; backing already released comes back with the next job, as before. Shared, imported and user-memory objects are never released and accept the flag without effect on them. Kept objects count against the memory limits as before. Their sizes are also totalled for the device (user_kept in the gpu_memory sysfs file) until the object is freed. Tested: KUnit in QEMU (arm64, 16K pages), 546 tests, 544 passed, 2 skipped (unrelated configuration); the new asahi_memacct test kept_bytes_follow_the_object checks that the kept total changes only on a change of state, is left as it is by a transfer to another client, is cleared by a free and ignores kernel objects and imports. KUnit does not reach the shrinkers: what reclaim does with kept objects is covered by the selftest that follows, which passes on the J716c (M5 Max MacBook Pro), the case that runs every shrinker at the last priority included. There, with a Vulkan driver that maps memory priority 1.0 to this ioctl, the memory priority and pageable allocation cases of the Vulkan CTS give 430 passed of 430. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/driver.rs | 2 + drivers/gpu/drm/asahi/file.rs | 31 ++++ drivers/gpu/drm/asahi/gem.rs | 38 +++++ drivers/gpu/drm/asahi/memacct.rs | 64 +++++++- drivers/gpu/drm/asahi/mmu/residency.rs | 211 ++++++++++++++++++------- drivers/gpu/drm/asahi/sysfs.c | 12 +- include/uapi/drm/asahi_drm.h | 79 +++++++++ 7 files changed, 379 insertions(+), 58 deletions(-) diff --git a/drivers/gpu/drm/asahi/driver.rs b/drivers/gpu/drm/asahi/driver.rs index 52c018cbd1dc1..d32336c1b151d 100644 --- a/drivers/gpu/drm/asahi/driver.rs +++ b/drivers/gpu/drm/asahi/driver.rs @@ -142,6 +142,8 @@ impl drm::driver::Driver for AsahiDrmDriver { ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_madvise), (ASAHI_GEM_USERPTR, drm_asahi_gem_userptr, ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_userptr), + (ASAHI_GEM_RESIDENCY, drm_asahi_gem_residency, + ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_residency), } } diff --git a/drivers/gpu/drm/asahi/file.rs b/drivers/gpu/drm/asahi/file.rs index 9fe3de9109731..cc1298edfe00d 100644 --- a/drivers/gpu/drm/asahi/file.rs +++ b/drivers/gpu/drm/asahi/file.rs @@ -568,6 +568,9 @@ impl File { if Self::supports_user_memory(reg_data) { params.features |= uapi::DRM_ASAHI_FEATURE_USERPTR as u64; } + if Self::supports_residency_control(reg_data) { + params.features |= uapi::DRM_ASAHI_FEATURE_GEM_RESIDENCY as u64; + } let size = core::mem::size_of::().min(data.size.try_into()?); @@ -997,6 +1000,34 @@ impl File { Ok(0) } + /// Whether this GPU takes residency requests (DRM_IOCTL_ASAHI_GEM_RESIDENCY): only G17 user + /// VMs release idle memory under pressure. + fn supports_residency_control(reg_data: &driver::AsahiDrmRegistrationData<'_>) -> bool { + reg_data.gpu.as_any().is::() + } + + /// IOCTL: gem_residency: Keep an object's memory resident, or return it to the default + /// policy. + pub(crate) fn gem_residency( + _device: &AsahiDrmDevice, + reg_data: &driver::AsahiDrmRegistrationData<'_>, + data: &mut uapi::drm_asahi_gem_residency, + file: &DrmFile, + ) -> Result { + if !Self::supports_residency_control(reg_data) { + return Err(EOPNOTSUPP); + } + if data.flags & !uapi::drm_asahi_gem_residency_flags_DRM_ASAHI_GEM_RESIDENCY_KEEP != 0 { + return Err(EINVAL); + } + let gem = gem::Object::lookup_handle(file, data.handle)?; + gem::set_keep_resident( + &gem, + data.flags & uapi::drm_asahi_gem_residency_flags_DRM_ASAHI_GEM_RESIDENCY_KEEP != 0, + )?; + Ok(0) + } + /// IOCTL: gem_mmap_offset: Assign an mmap offset to a GEM object. pub(crate) fn gem_mmap_offset( device: &AsahiDrmDevice, diff --git a/drivers/gpu/drm/asahi/gem.rs b/drivers/gpu/drm/asahi/gem.rs index 3c3b667e61af5..40487a77e97ea 100644 --- a/drivers/gpu/drm/asahi/gem.rs +++ b/drivers/gpu/drm/asahi/gem.rs @@ -31,6 +31,7 @@ use kernel::{ use core::ops::Range; use core::sync::atomic::{ + AtomicBool, AtomicI64, AtomicU64, AtomicUsize, @@ -65,6 +66,10 @@ pub(crate) struct AsahiObject { kernel_mappings: AtomicU64, /// Memory accounting and process attribution of the held backing. charge: memacct::Charge, + /// Userspace asked to keep the backing resident (DRM_IOCTL_ASAHI_GEM_RESIDENCY): idle + /// reclaim of the object's VM releases it after the VM's other objects, and only if those + /// did not meet the scan target. Changed and read under the object's reservation. + keep_resident: AtomicBool, /// The pinned pages of a user-memory object (DRM_IOCTL_ASAHI_GEM_USERPTR), its backing. /// The shmem layer unmaps them from the device before this is dropped. user_memory: Option, @@ -602,6 +607,7 @@ impl DriverObject for AsahiObject { exportable: args.exportable, kernel: args.kernel, kernel_mappings: AtomicU64::new(0), + keep_resident: AtomicBool::new(false), charge: args.charge, user_memory: args.user_memory, timestamp_lock <- new_mutex!((), "Asahi timestamp stores"), @@ -692,6 +698,16 @@ impl AsahiObject { && self.kernel_mappings.load(Ordering::Acquire) == 0 } + /// Whether userspace asked to keep this object's backing resident. Idle reclaim releases + /// the backing of such an object after the VM's other objects, and only if those did not + /// meet the scan target. Purge of contents userspace advised away does not look at this: + /// the advice says the contents are not needed. + /// + /// The caller holds the object's reservation. + pub(crate) fn keep_resident(&self) -> bool { + self.keep_resident.load(Ordering::Relaxed) + } + /// Whether this object is backed by the creating process's own memory. pub(crate) fn is_user_memory(&self) -> bool { self.user_memory.is_some() @@ -727,6 +743,28 @@ pub(crate) fn madvise(gem: &Object, dontneed: bool) -> Result { Ok(u32::from(gem.madvise(i32::from(dontneed)))) } +/// Keep (or stop keeping) the backing of a user object resident: once this returns, idle reclaim +/// treats the object as kept (released after the VM's other objects, where those did not meet +/// the scan target) or as any other. This is an order among the objects idle reclaim may +/// release, not an exemption. Backing already released stays released until the VM's next job +/// or binding restores it. Kernel objects are refused. +pub(crate) fn set_keep_resident(gem: &Object, keep: bool) -> Result { + if gem.kernel { + return Err(EINVAL); + } + // SAFETY: A live object's reservation pointer is set at creation and never changes. + let resv = unsafe { (*gem.as_raw()).resv }; + // SAFETY: The reservation of a live object. Idle reclaim tests the flag while it holds the + // reservation of the object's VM, which a VM-private object shares, so after the unlock no + // reclaim pass can still act on the old value; other objects are never reclaimed. + unsafe { kernel::bindings::dma_resv_lock(resv, core::ptr::null_mut()) }; + gem.keep_resident.store(keep, Ordering::Relaxed); + gem.charge.set_kept(keep); + // SAFETY: Locked above. + unsafe { kernel::bindings::dma_resv_unlock(resv) }; + Ok(()) +} + /// Large user BOs may release clean page cache but never swap out the rest of the /// system per folio; they get a quick ENOMEM when they cannot fit. Keep ordinary /// resource allocations and kernel-owned backing on their existing policy. 256 MiB diff --git a/drivers/gpu/drm/asahi/memacct.rs b/drivers/gpu/drm/asahi/memacct.rs index a61be841bff11..b1e2d4fe093b1 100644 --- a/drivers/gpu/drm/asahi/memacct.rs +++ b/drivers/gpu/drm/asahi/memacct.rs @@ -21,7 +21,9 @@ //! counter is back to its own value before `__mmdrop()` checks it. //! - Each client (DRM file) and the device keep totals of created and held bytes, //! reported in fdinfo (`drm-total-system`, `drm-resident-system`) and in the -//! `gpu_memory` sysfs file next to `agx_stats`. +//! `gpu_memory` sysfs file next to `agx_stats`. The device also totals the bytes of +//! objects that userspace asked to keep resident (which idle reclaim releases after +//! the other objects of their VM), in `gpu_memory`. //! - A new user object is admitted only while its client stays within a per-client //! limit and the device within a device-wide limit, both fractions of RAM. //! A refused allocation fails with `ENOMEM`, which userspace reports as @@ -66,13 +68,16 @@ pub(crate) struct Ledger { pub(crate) limit: AtomicU64, /// Per-client limit on committed bytes (0: none). pub(crate) client_limit: AtomicU64, + /// Size of the live user objects kept resident at userspace's request. + pub(crate) user_kept: AtomicU64, } // Keep the Rust half of the C sysfs layout contract checked as well. const _: () = { assert!(core::mem::offset_of!(Ledger, refused) == 24); assert!(core::mem::offset_of!(Ledger, client_limit) == 40); - assert!(core::mem::size_of::() == 48); + assert!(core::mem::offset_of!(Ledger, user_kept) == 48); + assert!(core::mem::size_of::() == 56); }; /// The ledger of the (single) AGX GPU. @@ -145,6 +150,7 @@ impl Ledger { refused: AtomicU64::new(0), limit: AtomicU64::new(0), client_limit: AtomicU64::new(0), + user_kept: AtomicU64::new(0), } } @@ -271,6 +277,8 @@ pub(crate) struct Charge { bytes: u64, /// Whether the bytes are currently counted as held by the GPU. held: AtomicBool, + /// Whether the bytes are currently counted as kept resident. + kept: AtomicBool, } // SAFETY: `ARef` may be dropped on any thread and the counters are atomic. @@ -286,6 +294,7 @@ impl Charge { kind: Kind::None, bytes: 0, held: AtomicBool::new(false), + kept: AtomicBool::new(false), } } @@ -295,6 +304,7 @@ impl Charge { kind: Kind::Kernel(ledger), bytes, held: AtomicBool::new(false), + kept: AtomicBool::new(false), } } @@ -310,6 +320,7 @@ impl Charge { kind: Kind::User { commit, mm }, bytes, held: AtomicBool::new(false), + kept: AtomicBool::new(false), } } @@ -387,11 +398,31 @@ impl Charge { self.apply(held); } } + + /// Records that userspace asked (or no longer asks) to keep the object resident. Only user + /// objects are counted, in the device's total, which a transfer to another client leaves + /// as it is. + /// + /// Idempotent: only a change of state is counted. + pub(crate) fn set_kept(&self, kept: bool) { + let Kind::User { commit, .. } = &self.kind else { + return; + }; + if self.kept.swap(kept, Ordering::AcqRel) == kept { + return; + } + if kept { + commit.ledger.user_kept.fetch_add(self.bytes, Ordering::AcqRel); + } else { + sub(&commit.ledger.user_kept, self.bytes); + } + } } impl Drop for Charge { fn drop(&mut self) { // Uncharge before `kind` (and with it the mm reference) is dropped. + self.set_kept(false); self.set_held(false); } } @@ -548,6 +579,35 @@ mod tests { Ok(()) } + static KEPT: Ledger = Ledger::new(); + + #[test] + fn kept_bytes_follow_the_object() -> Result { + KEPT.set_limits(Limits { device: 0, client: 0 }); + let a = Client::new()?; + let b = Client::new()?; + let mut charge = Charge::user(KEPT.admit(&a, 8 * MIB)?, None); + charge.set_kept(true); + charge.set_kept(true); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 8 * MIB); + charge.set_kept(false); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 0); + charge.set_kept(true); + // A move to another client leaves the device's count as it is. + charge.transfer(&b)?; + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 8 * MIB); + // Freeing a kept object stops counting it. + drop(charge); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 0); + // Kernel objects and imports are never counted as kept. + let k = Charge::kernel(&KEPT, MIB); + k.set_kept(true); + let i = Charge::none(); + i.set_kept(true); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 0); + Ok(()) + } + #[test] fn imports_are_not_counted() { let charge = Charge::none(); diff --git a/drivers/gpu/drm/asahi/mmu/residency.rs b/drivers/gpu/drm/asahi/mmu/residency.rs index a83068a5c9e9f..7d36f2bee910e 100644 --- a/drivers/gpu/drm/asahi/mmu/residency.rs +++ b/drivers/gpu/drm/asahi/mmu/residency.rs @@ -243,11 +243,46 @@ impl Drop for Lease { /// Registration lifetime belongs to the file's VM handle, not the VM clone /// retained by the callback. Unregister synchronously before dropping the clone. +/// +/// A VM has two shrinkers. The first purges advised contents and releases the +/// idle objects userspace did not ask to keep, in step with the rest of +/// reclaim. The second releases the objects userspace asked to keep resident +/// (DRM_IOCTL_ASAHI_GEM_RESIDENCY) as well, after the others, and the shrinker +/// core calls its scan only for a pass that is to scan everything it counted: +/// see `KEPT_BATCH`. +/// +/// Neither is memcg-aware: reclaim inside a memory cgroup below the root does +/// not call them, so only reclaim on behalf of the whole system releases GPU +/// memory. Both release idle backing only of a VM that passes the gate's +/// test (`Gate::try_reclaim`): no lease held, none released for +/// `RECLAIM_QUIET_NS`, and the backoff after a restoration over. Userspace's +/// request to keep an object is thus an order among the objects that these +/// shrinkers may release, and no more. pub(crate) struct VmShrinker { vm: crate::mmu::Vm, raw: *mut kernel::bindings::shrinker, + kept: *mut kernel::bindings::shrinker, } +/// The batch of the shrinker of kept objects: more than a scan target can be. +/// +/// The shrinker core scans a shrinker when the target it worked out reaches the +/// batch, or reaches everything the shrinker counted. The target is the count +/// scaled down by the reclaim priority (at most twice the count, at the last +/// priority), so with this batch only the second case is left: the target of +/// the last two of the thirteen priorities, or of the last three once scans +/// deferred by earlier passes have built up. A pass at an earlier priority +/// does not scan kept objects. +/// +/// Reclaim works down the priorities only as far as it must to free what it +/// was asked for. It comes to the last ones when the page cache, the other +/// shrinkers and this VM's first one did not give that much at the +/// priorities before: under memory pressure that nothing else relieves, and +/// also for a request that names more than the system has to spare (a reclaim +/// request to the root memory cgroup, say). Dropping the slab caches +/// (`drop_caches`) runs every shrinker at the last priority at once. +const KEPT_BATCH: kernel::ffi::c_long = kernel::ffi::c_long::MAX; + // SAFETY: shrinker core serializes registration lifetime; callback VM operations // use its residency/reservation locks. raw is never dereferenced after free. unsafe impl Send for VmShrinker {} @@ -261,6 +296,7 @@ impl VmShrinker { Self { vm: vm.clone(), raw: core::ptr::null_mut(), + kept: core::ptr::null_mut(), }, GFP_KERNEL, )?; @@ -271,14 +307,29 @@ impl VmShrinker { return Err(ENOMEM); } owner.raw = raw; - // SAFETY: Newly allocated/unregistered shrinker is exclusively owned. + // SAFETY: As above. + let kept = + unsafe { kernel::bindings::shrinker_alloc(0, c_str!("aurora-kept-vm").as_char_ptr()) }; + if kept.is_null() { + return Err(ENOMEM); + } + owner.kept = kept; + // SAFETY: Newly allocated/unregistered shrinkers are exclusively owned. The + // shrinker of kept objects is registered second: a reclaim pass that reaches it + // has asked this VM's first shrinker before. unsafe { (*raw).count_objects = Some(Self::count); (*raw).scan_objects = Some(Self::scan); (*raw).private_data = core::ptr::from_ref(&*owner).cast_mut().cast(); (*raw).seeks = 2; (*raw).batch = 256; + (*kept).count_objects = Some(Self::count_kept); + (*kept).scan_objects = Some(Self::scan_kept); + (*kept).private_data = core::ptr::from_ref(&*owner).cast_mut().cast(); + (*kept).seeks = 2; + (*kept).batch = KEPT_BATCH; kernel::bindings::shrinker_register(raw); + kernel::bindings::shrinker_register(kept); } Ok(owner) } @@ -295,7 +346,21 @@ impl VmShrinker { ) -> kernel::ffi::c_ulong { // SAFETY: The registration owns this stable allocation until callbacks drain. let owner = unsafe { &*((*raw).private_data as *const Self) }; - owner.vm.idle_reclaim_count() + owner.vm.idle_reclaim_count(false) + } + + /// Count the reclaimable pages of the objects userspace asked to keep resident. + /// + /// # Safety + /// + /// As for [`Self::count`]. + unsafe extern "C" fn count_kept( + raw: *mut kernel::bindings::shrinker, + _sc: *mut kernel::bindings::shrink_control, + ) -> kernel::ffi::c_ulong { + // SAFETY: The registration owns this stable allocation until callbacks drain. + let owner = unsafe { &*((*raw).private_data as *const Self) }; + owner.vm.idle_reclaim_count(true) } /// Reclaim idle pages. @@ -312,16 +377,35 @@ impl VmShrinker { let owner = unsafe { &*((*raw).private_data as *const Self) }; // SAFETY: The shrinker core supplies a live initialized control. let target = unsafe { (*sc).nr_to_scan }; - owner.vm.reclaim_idle_pages(target) + owner.vm.reclaim_idle_pages(target, false) + } + + /// Reclaim idle pages for a pass that scans everything: the objects userspace asked to + /// keep resident too, after the others. + /// + /// # Safety + /// + /// As for [`Self::scan`]. + unsafe extern "C" fn scan_kept( + raw: *mut kernel::bindings::shrinker, + sc: *mut kernel::bindings::shrink_control, + ) -> kernel::ffi::c_ulong { + // SAFETY: Both pointers are supplied by the registered shrinker core. + let owner = unsafe { &*((*raw).private_data as *const Self) }; + // SAFETY: The shrinker core supplies a live initialized control. + let target = unsafe { (*sc).nr_to_scan }; + owner.vm.reclaim_idle_pages(target, true) } } impl Drop for VmShrinker { fn drop(&mut self) { - if !self.raw.is_null() { - // SAFETY: This owns the sole registration; free drains callbacks - // before the embedded VM and callback private_data can disappear. - unsafe { kernel::bindings::shrinker_free(self.raw) }; + for raw in [self.kept, self.raw] { + if !raw.is_null() { + // SAFETY: This owns the sole registration; free drains callbacks + // before the embedded VM and callback private_data can disappear. + unsafe { kernel::bindings::shrinker_free(raw) }; + } } } } @@ -356,7 +440,10 @@ impl Vm { .transpose() } - pub(crate) fn idle_reclaim_count(&self) -> usize { + /// The pages reclaim could release now: with `kept`, those of the idle objects userspace + /// asked to keep resident; without, those of the other idle objects and of every object + /// whose contents userspace advised away, kept or not. + pub(crate) fn idle_reclaim_count(&self, kept: bool) -> usize { let Some(gate) = self.residency.as_ref() else { return 0; }; @@ -386,7 +473,12 @@ impl Vm { (advised || object.has_sg_table_locked()) && object.unleased_pages_releasable_locked() }; - if object.idle_reclaim_candidate() && (advised || idle) && releasable { + let counted = if advised { + !kept + } else { + idle && object.keep_resident() == kept + }; + if object.idle_reclaim_candidate() && counted && releasable { pages = pages.saturating_add(object.size() / kernel::page::PAGE_SIZE); } Ok(true) @@ -475,7 +567,10 @@ impl Vm { purged } - pub(crate) fn reclaim_idle_pages(&self, target: usize) -> usize { + /// Releases idle backing until `target` pages are free: contents userspace advised away + /// first, then the objects it did not ask to keep resident and, only with `kept` and only + /// if those did not reach the target, the objects it did. + pub(crate) fn reclaim_idle_pages(&self, target: usize, kept: bool) -> usize { let Some(gate) = self.residency.as_ref() else { return 0; }; @@ -508,50 +603,58 @@ impl Vm { }; let contexts = shared.contexts_naming_root(inner.ttb(), inner.uat_inner.m4_adopted); let mut unpinned = 0usize; - // SAFETY: Exclusive residency and the private reservation exclude jobs, - // binding edits and retirement. Callbacks change PTEs/flags only; all - // logical aliases stay linked and retain their borrowed BO/GEM. - let _ = unsafe { inner.for_each_private_bo(|inner, bo| { - if unpinned >= target { return Ok(false); } - let object = bo.obj(); - if !object.idle_reclaim_candidate() { return Ok(true); } - { - // Avoid inversion against preparation (VmBoInner -> resv). - let Some(segments) = bo.data().segments.try_lock() else { return Ok(true); }; - let releasable = if segments.is_some() { - object.sole_sg_lease_releasable_locked() - } else { - // A competing lease may have made the previous release fail after we - // dropped our segments. Retry that table, but do not count empty backing. - object.has_sg_table_locked() && object.unleased_pages_releasable_locked() - }; - if !releasable { return Ok(true); } - } - // The association retains SG backing until *all* aliases have - // been invalidated and their device translations flushed. A - // contended cursor or PTE failure retains backing and the touched - // INVALIDATED prefix for restoration or a later reclaim attempt. - inner.invalidate_private_bo_mappings(bo, |inner, start, range| { - reclaim.mark_changed(); - let end = start.checked_add(range).ok_or(EOVERFLOW)?; - let _change = inner.mapping_mutation(); - inner.page_table.discard_partial_map(start..end)?; - inner.tlbi_context_mask(start, range as usize, contexts); - Ok(()) - })?; - { - let Some(mut segments) = bo.data().segments.try_lock() else { return Ok(true); }; - *segments = None; - } - // Every private GPUVA alias was removed from the device page - // tables above. Kernel mappings and other SG owners are excluded; - // the C/Rust backing helper rechecks exclusion before release. - if object.release_idle_pages_locked().is_ok() { - crate::gem::set_mapping_held(object, false); - unpinned = unpinned.saturating_add(object.size() / kernel::page::PAGE_SIZE); - } - Ok(unpinned < target) - }) }; + // Kept objects go in a second pass, when the first left the target unmet. + for keep_pass in [false, true] { + if unpinned >= target || (keep_pass && !kept) { break; } + // SAFETY: Exclusive residency and the private reservation exclude jobs, + // binding edits and retirement. Callbacks change PTEs/flags only; all + // logical aliases stay linked and retain their borrowed BO/GEM. + let _ = unsafe { inner.for_each_private_bo(|inner, bo| { + if unpinned >= target { return Ok(false); } + let object = bo.obj(); + if !object.idle_reclaim_candidate() || object.keep_resident() != keep_pass { + return Ok(true); + } + { + // Avoid inversion against preparation (VmBoInner -> resv). + let Some(segments) = bo.data().segments.try_lock() else { return Ok(true); }; + let releasable = if segments.is_some() { + object.sole_sg_lease_releasable_locked() + } else { + // A competing lease may have made the previous release fail after we + // dropped our segments. Retry that table, but do not count empty backing. + object.has_sg_table_locked() && object.unleased_pages_releasable_locked() + }; + if !releasable { return Ok(true); } + } + // The association retains SG backing until *all* aliases have + // been invalidated and their device translations flushed. A + // contended cursor or PTE failure retains backing and the touched + // INVALIDATED prefix for restoration or a later reclaim attempt. + inner.invalidate_private_bo_mappings(bo, |inner, start, range| { + reclaim.mark_changed(); + let end = start.checked_add(range).ok_or(EOVERFLOW)?; + let _change = inner.mapping_mutation(); + inner.page_table.discard_partial_map(start..end)?; + inner.tlbi_context_mask(start, range as usize, contexts); + Ok(()) + })?; + { + let Some(mut segments) = bo.data().segments.try_lock() else { + return Ok(true); + }; + *segments = None; + } + // Every private GPUVA alias was removed from the device page + // tables above. Kernel mappings and other SG owners are excluded; + // the C/Rust backing helper rechecks exclusion before release. + if object.release_idle_pages_locked().is_ok() { + crate::gem::set_mapping_held(object, false); + unpinned = unpinned.saturating_add(object.size() / kernel::page::PAGE_SIZE); + } + Ok(unpinned < target) + }) }; + } drop(shared); drop(inner); diff --git a/drivers/gpu/drm/asahi/sysfs.c b/drivers/gpu/drm/asahi/sysfs.c index 0b2bca6571129..1e1b4fd07ea1e 100644 --- a/drivers/gpu/drm/asahi/sysfs.c +++ b/drivers/gpu/drm/asahi/sysfs.c @@ -42,6 +42,7 @@ struct asahi_gpu_memory { u64 refused; u64 limit; u64 client_limit; + u64 user_kept; }; struct asahi_stats_attribute { @@ -182,7 +183,11 @@ static ssize_t agx_stats_show(struct device *dev, /* * GPU memory held by the driver, in bytes: user objects created (committed) * and held by the GPU (resident), held kernel objects, the limits that admit - * user objects (0: none) and the number of refused creations. + * user objects (0: none), the number of refused creations, and the size of + * the user objects that userspace asked to keep resident (user_kept), whether + * their memory is held at the moment or not. The request is a hint: when + * reclaim releases idle memory of a VM, its kept objects go after the others + * (see DRM_IOCTL_ASAHI_GEM_RESIDENCY). */ static ssize_t gpu_memory_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -204,6 +209,8 @@ static ssize_t gpu_memory_show(struct device *dev, (unsigned long long)READ_ONCE(mem->client_limit)); n += scnprintf(buf + n, PAGE_SIZE - n, "refused %llu\n", (unsigned long long)READ_ONCE(mem->refused)); + n += scnprintf(buf + n, PAGE_SIZE - n, "user_kept %llu\n", + (unsigned long long)READ_ONCE(mem->user_kept)); return n; } @@ -228,7 +235,8 @@ int asahi_sysfs_register(struct device *dev, const void *snapshot, BUILD_BUG_ON(sizeof(struct asahi_qualified_stats) != 56); BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, refused) != 24); BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, client_limit) != 40); - BUILD_BUG_ON(sizeof(struct asahi_gpu_memory) != 48); + BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, user_kept) != 48); + BUILD_BUG_ON(sizeof(struct asahi_gpu_memory) != 56); BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, pstate) != 16); BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, busy_ns) != 40); diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index c51b239fb4917..c2eaf7af9d1b0 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -102,6 +102,15 @@ enum drm_asahi_ioctl_id { * memory (DRM_ASAHI_FEATURE_USERPTR). */ DRM_ASAHI_GEM_USERPTR, + + /** + * @DRM_ASAHI_GEM_RESIDENCY: Ask the kernel to release an object's + * memory after that of the other objects of its VM when it reclaims + * the VM's idle memory, or withdraw that request + * (DRM_ASAHI_FEATURE_GEM_RESIDENCY). A hint, see struct + * drm_asahi_gem_residency. + */ + DRM_ASAHI_GEM_RESIDENCY, }; #define DRM_ASAHI_MAX_CLUSTERS 64 @@ -495,6 +504,13 @@ enum drm_asahi_feature { */ #define DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION ((1ULL) << 34) +/* + * DRM_ASAHI_FEATURE_GEM_RESIDENCY: DRM_IOCTL_ASAHI_GEM_RESIDENCY is + * available. A feature bit like those of enum drm_asahi_feature, defined + * here like bit 32. + */ +#define DRM_ASAHI_FEATURE_GEM_RESIDENCY ((1ULL) << 35) + /* * DRM_ASAHI_FEATURE_USERPTR: DRM_IOCTL_ASAHI_GEM_USERPTR is available. A * feature bit like those of enum drm_asahi_feature, defined here like bit 32. @@ -832,6 +848,68 @@ struct drm_asahi_gem_madvise { __u32 pad; }; +/** + * enum drm_asahi_gem_residency_flags - Flags for DRM_IOCTL_ASAHI_GEM_RESIDENCY + */ +enum drm_asahi_gem_residency_flags { + /** + * @DRM_ASAHI_GEM_RESIDENCY_KEEP: When reclaim releases idle memory of + * the object's VM, release this object's memory after that of the + * VM's objects without the flag, and only if theirs was not enough. + * Without the flag, the object returns to the default policy. + */ + DRM_ASAHI_GEM_RESIDENCY_KEEP = (1L << 0), +}; + +/** + * struct drm_asahi_gem_residency - Arguments passed to + * DRM_IOCTL_ASAHI_GEM_RESIDENCY + * + * When the system reclaims memory, the kernel may release the memory of the + * VM-private objects of an idle VM. It brings the memory back before the + * VM's next job runs, which then waits for it. + * DRM_ASAHI_GEM_RESIDENCY_KEEP is a priority among the objects that may be + * released this way. It changes the order in which they go, not whether + * reclaim gets to them: + * + * - Only reclaim on behalf of the whole system releases GPU memory. Reclaim + * inside a memory cgroup below the root, such as that of a cgroup at its + * memory limit, releases none, kept or not. + * - Reclaim releases memory of a VM only while the VM is idle: no job, + * binding change or other use of the VM is in progress, the last one (or + * the VM's creation) is at least two seconds back, and after a job had to + * wait for released memory, a pause that grows with the time that took + * (between 1 and 30 seconds) is over. The memory of a VM that is not idle + * stays, kept or not. + * - Of an idle VM's objects, a reclaim pass releases those without the flag + * first, and kept objects only if that left the amount the pass set out + * to free unmet. + * - Contents userspace gave up with DRM_ASAHI_GEM_MADV_DONTNEED are purged + * as before, kept or not. + * + * The request is therefore a hint. Userspace must not rely on a kept object + * being resident, and a client that stays active, or whose memory cgroup is + * the one that runs out of memory, keeps its GPU memory whether it asked or + * not. + * + * The request applies once this ioctl returns; memory already released stays + * so until the VM's next job or binding brings it back. Other objects + * (shared, imported or GEM_USERPTR ones) are never released and accept the + * flag without effect on them. + * + * Kept objects still count against the memory limits as before; the device + * total of their sizes is reported as user_kept in the GPU's gpu_memory + * sysfs file. Available when DRM_ASAHI_FEATURE_GEM_RESIDENCY is set; + * otherwise the ioctl fails with EOPNOTSUPP or EINVAL. + */ +struct drm_asahi_gem_residency { + /** @handle: Handle of the object. */ + __u32 handle; + + /** @flags: Combination of drm_asahi_gem_residency_flags flags. */ + __u32 flags; +}; + /** * enum drm_asahi_bind_flags - Flags for GEM binding */ @@ -2281,6 +2359,7 @@ enum { DRM_IOCTL_ASAHI_SUBMIT = DRM_IOCTL_ASAHI(W, SUBMIT, submit), DRM_IOCTL_ASAHI_GEM_MADVISE = DRM_IOCTL_ASAHI(WR, GEM_MADVISE, gem_madvise), DRM_IOCTL_ASAHI_GEM_USERPTR = DRM_IOCTL_ASAHI(WR, GEM_USERPTR, gem_userptr), + DRM_IOCTL_ASAHI_GEM_RESIDENCY = DRM_IOCTL_ASAHI(W, GEM_RESIDENCY, gem_residency), }; #if defined(__cplusplus) From cb8258cba2e25ff53874dd216512d428db4347c8 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Wed, 11 Nov 2026 12:44:09 -0500 Subject: [PATCH 2/8] selftests: drm/asahi: GEM_RESIDENCY A kselftest harness program for the render node that exercises DRM_IOCTL_ASAHI_GEM_RESIDENCY without submitting GPU work: - a VM-private object: a keep request moves user_kept (gpu_memory sysfs file) by the object's size, once for a repeated request, a clear request moves it back, and freeing a kept object stops counting it; - unknown flags fail with EINVAL and an unknown handle with ENOENT, and neither changes user_kept; - a shared object takes the flag and is counted the same way; - light_reclaim_spares_kept_objects: of two idle bound objects, reclaim releases the one that is not kept first, never the kept one while the other is resident; the kept one stays through 16 further reclaim requests; once its request is withdrawn it is released too; - kept_objects_stay_purgeable: a kept object advised DONTNEED is purged by reclaim; - requests_serialize_with_reclaim: while a thread sets and clears the request on one object as fast as it can and reclaim runs, an object beside it that is not kept is released, which shows that reclaim got to the VM during the race, a kept object beside it is not, the state a request that has returned found is the one reclaim leaves, and user_kept counts each object once; - kept_objects_go_under_urgent_pressure: when every shrinker is run at the last reclaim priority, a kept object is released as well, without being purged. The four cases that make the kernel reclaim act on the whole system. They need root and run only when asked for in the environment: ASAHI_RESIDENCY_RECLAIM=1 for the three that write to memory.reclaim of the root memory cgroup, ASAHI_RESIDENCY_DROP_CACHES=1 for the last one, which writes 2 to /proc/sys/vm/drop_caches. A memory.reclaim request names an amount, not a priority: the kernel works down its reclaim priorities until it has reclaimed that much, and where nothing else can be reclaimed it ends at the last ones, at which kept objects go like everything else. The three cases therefore bring the memory to reclaim: a sparse temporary file of 2 GiB in the working directory, read once into the page cache, from which requests of 4 MiB are usually met within the first priorities. The objects have 2^19 pages (8 GiB with 16K pages), the least a VM must have for the lightest priority to scan its shrinker. The cases skip without four times the objects' size of available memory, in a working directory on tmpfs and in a memory cgroup with memory.min or memory.low protection. When a kept object was released, a case has to tell whether reclaim had leave to release it. It does so in one of two ways, and says which: - From the record of the shrinker passes, where it can have one: the events vmscan:mm_shrink_slab_start and mm_shrink_slab_end in a trace instance (tracefs at /sys/kernel/tracing). The shrinker core calls a VM's shrinker of kept objects only in a pass whose target reaches everything that shrinker counted, and the start event shows both numbers. The events name a shrinker by its pointer, so a case knows its own VM's shrinker by what it counts, the pages of the VM's idle kept objects: each case adds a random 0 to 1023 pages to its kept object, and takes for its VM's shrinker the one that never counted anything else. Leave to release the object is a pass that called that shrinker before the case first saw the object gone and had not ended when the case last saw it resident; a pass ends with the end event of the same shrinker from the same task. With such a pass the case is skipped ("as it may"), with their number and priority; without one it fails, whatever passes other VMs' shrinkers or other times had. If more than one shrinker in the record counts what the test's VM counts and a pass called one of them, the case is skipped as "inconclusive", which claims nothing about the release. The record is used when no event was lost and it shows the driver's other shrinker at work, which it knows by the names of the scan functions. - Otherwise from signs of the environment, a heuristic: whether the kernel met every request, whether 512 MiB of the file were still cached after each, and whether kswapd or direct reclaim scanned pages meanwhile. A kept object released with one of these signs skips the case, with the sign and the word that it is a heuristic; without one the case fails. The signs do not show which priorities reclaim reached: reclaim that comes to the last ones without leaving a sign fails a case that the record would skip. kept_objects_go_under_urgent_pressure also requires, where it has the record, a pass that called its VM's shrinker of kept objects before the object went. The trace instance belongs to the program: main() makes it before the cases fork, named asahi-residency-, and removes it when the program exits or is ended by a signal, so a case that fails, times out or is killed leaves nothing behind. The signals that end the program wait while the instance is set up or removed, and the handler makes only async-signal-safe calls, on paths made beforehand. An instance of that name that exists already is left alone, and the cases then go by the heuristic. The result of every series of reclaim requests is checked for an error (a failed request, or a client total that could not be read) before it is taken for a released object. The program is listed in the selftests' TARGETS. It skips where renderD128 is not this driver's node or lacks the feature. Tested on the J716c (M5 Max MacBook Pro), built there with gcc against this tree's UAPI header: - unprivileged, without the two variables: pass 3, skip 4; - as root with both variables: pass 7. The object that is not kept went with the first request and the kept one stayed through 16 more; the object no longer kept and the advised one went with the next request each. In the race, 11 million residency requests ran while reclaim released the control object, with its 11th request, and the toggled one; the kept object stayed, also through 16 further requests. At least 1995 MiB of the cold cache's 2048 were left. The record had one shrinker with the count of the test's VM in each case: 51 and 70 passes over it in these two cases, none of which called it, and in the urgent case one that did, at priority 0; - as root with ASAHI_RESIDENCY_RECLAIM=1, right afterwards and once more later: pass 6, skip 1, with similar figures; - as root with ASAHI_RESIDENCY_RECLAIM=1 and tracefs hidden from the program (a private mount over /sys/kernel/tracing): pass 6, skip 1, by the heuristic; - five changed copies of the program, light_reclaim_spares_kept_objects only. One that never asks to keep the object fails, with the record and without. One that runs every shrinker at the last priority between the case's two phases is skipped with "as it may: ... 1, down to priority 0 (traced)". One that does not keep the object while such a pass calls another client's shrinker of kept objects fails (no shrinker with the count of the test's VM), and so does one in which such a pass called a shrinker with that count and ended before the case first looked. One with a second client whose kept object has the same size is skipped with "inconclusive: 2 shrinkers ...". Without the record all five fail; - a case killed with its process group while it records, the program ended by SIGTERM, and an instance of its name made beforehand: the program removes its instance in the first two and leaves the foreign one alone, with the case passing by the heuristic. Sent SIGTERM at a random moment of its first 0.3 s, 30 times, it left no instance. Developed with Claude Code. Signed-off-by: Erik Bourget --- tools/testing/selftests/Makefile | 1 + .../drivers/gpu/asahi-residency/Makefile | 5 + .../drivers/gpu/asahi-residency/residency.c | 1397 +++++++++++++++++ 3 files changed, 1403 insertions(+) create mode 100644 tools/testing/selftests/drivers/gpu/asahi-residency/Makefile create mode 100644 tools/testing/selftests/drivers/gpu/asahi-residency/residency.c diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 5ec4864f5eb22..8394e597c5db6 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -25,6 +25,7 @@ TARGETS += drivers/dma-buf TARGETS += drivers/gpu/apple-afkv2 TARGETS += drivers/gpu/apple-dcp TARGETS += drivers/gpu/asahi-m4 +TARGETS += drivers/gpu/asahi-residency TARGETS += drivers/gpu/asahi-userptr TARGETS += drivers/ntsync TARGETS += drivers/s390x/uvdevice diff --git a/tools/testing/selftests/drivers/gpu/asahi-residency/Makefile b/tools/testing/selftests/drivers/gpu/asahi-residency/Makefile new file mode 100644 index 0000000000000..a4c170ceb9df1 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/asahi-residency/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-only OR MIT +CFLAGS += -O2 -Wall $(KHDR_INCLUDES) +LDLIBS += -lpthread +TEST_GEN_PROGS := residency +include ../../../lib.mk diff --git a/tools/testing/selftests/drivers/gpu/asahi-residency/residency.c b/tools/testing/selftests/drivers/gpu/asahi-residency/residency.c new file mode 100644 index 0000000000000..7a8fc7a36ccd0 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/asahi-residency/residency.c @@ -0,0 +1,1397 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * DRM_IOCTL_ASAHI_GEM_RESIDENCY on the GPU's render node: what it accepts and + * refuses, the accounting (user_kept in the GPU's gpu_memory sysfs file, a + * device total: run it while no other client changes the residency of its + * objects) and what reclaim does with kept objects. No GPU work is submitted. + * Skipped where the render node is not this driver's or lacks the feature. + * + * The cases that make the kernel reclaim memory act on the whole system. + * They need root and run only when asked for in the environment: + * + * ASAHI_RESIDENCY_RECLAIM=1 runs light_reclaim_spares_kept_objects, + * kept_objects_stay_purgeable and requests_serialize_with_reclaim, which + * write to the memory.reclaim file of the root memory cgroup. + * + * ASAHI_RESIDENCY_DROP_CACHES=1 runs kept_objects_go_under_urgent_pressure, + * which writes 2 to /proc/sys/vm/drop_caches. That runs every shrinker of + * the system at the last reclaim priority: it drops the dentry and inode + * caches, with the page cache of the files they stood for, and the idle + * memory of every GPU client. + * + * A request to the root memory cgroup names an amount, not a priority. The + * kernel starts at its lightest reclaim priority and goes on to the more + * urgent ones until it has reclaimed that much, anywhere in the system, and + * runs the system's shrinkers at each priority it comes to. Where little + * else can be reclaimed, a request of a few MiB ends at the last priorities, + * at which the kept objects of idle VMs are released like everything else + * the system can give. The cases therefore bring the memory to reclaim + * themselves: a sparse temporary file in the working directory, read once, + * which leaves COLD_BYTES of clean page cache that nothing else uses. As + * long as plenty of it is left, a request usually finds its pages there + * within the first few priorities. + * + * Usually is not always, and the system reclaims of its own accord as well, + * so a case in which a kept object was released has to find out whether + * reclaim had leave to release it. There are two ways, and the cases say + * which one they went by: + * + * - The record of the shrinker passes (preferred). The driver releases kept + * objects from a shrinker of their own, one per VM, which the shrinker + * core calls only in a pass whose target reaches everything that shrinker + * counted. The program records the vmscan:mm_shrink_slab_start and + * mm_shrink_slab_end events in a trace instance of its own (tracefs at + * /sys/kernel/tracing, with kernel symbols). The events name a shrinker by + * a pointer, which says nothing of the VM, so a case knows its own VM's + * shrinker by what it counts: the pages of the VM's idle kept objects, + * which the case makes a number of its own (see own_pages()). A kept + * object that went is a failure unless a pass called that shrinker before + * the case first saw the object gone and had not ended when the case last + * saw it resident; with such a pass the case is skipped, since the + * release was allowed. Where more than one shrinker of + * the record counts what the test's VM counts and a pass called one of + * them, the record cannot tell whose pass it was: the case is skipped as + * inconclusive, which says nothing about the release. + * - Signs of the environment (the fallback, where the passes cannot be + * recorded). After every request the cases note whether the kernel met it + * and whether the file still has COLD_FLOOR in the cache, and at the end + * whether kswapd or direct reclaim scanned pages meanwhile. A case in + * which a kept object was released after a request that was not met, with + * less of the file left, or while the system reclaimed as well, is + * skipped with that reason. These are heuristics: they make it likely, and + * do not show, that reclaim went to its last priorities, and reclaim that + * went there with every request met and the cache left would fail the + * case. + * + * These cases also need four times their objects' size of available memory + * (objects of RECLAIM_PAGES pages: 8 GiB each with 16K pages), a working + * directory on a file system with a page cache, and a memory cgroup without + * memory.min or memory.low protection, which would keep reclaim away from + * the cold cache. + */ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../../kselftest_harness.h" + +#ifndef __NR_cachestat +#define __NR_cachestat 451 +#endif + +#define PAGE 0x4000UL + +/* + * The idle pages a VM must have for the lightest reclaim pass to release + * some: the shrinker core scans a shrinker once the count it reported, + * divided by 2^12 at that priority and doubled, reaches the shrinker's batch + * (256 pages for this driver). + */ +#define RECLAIM_PAGES (256UL << 11) + +/* The time a VM must have been left alone before reclaim touches it, and more. */ +#define IDLE_SECONDS 3 + +/* What one reclaim request asks for. */ +#define REQUEST "4M" + +/* The clean page cache the reclaim cases bring, and how much of it must be left. */ +#define COLD_BYTES (2ULL << 30) +#define COLD_FLOOR (COLD_BYTES / 4) + +/* The most requests a case makes to see reclaim release an object. */ +#define MAX_REQUESTS 128 + +/* The requests through which a kept object has to stay resident. */ +#define SPARED_REQUESTS 16 + +static int ioc(int fd, unsigned long cmd, void *arg) +{ + int ret; + + do { + ret = ioctl(fd, cmd, arg); + } while (ret == -1 && (errno == EINTR || errno == EAGAIN)); + return ret ? -errno : 0; +} + +/* A field of the GPU's gpu_memory sysfs file. */ +static long long gpu_memory_field(const char *key) +{ + char line[256]; + long long value = -1; + size_t n = strlen(key); + FILE *f = fopen("/sys/class/drm/renderD128/device/gpu_memory", "r"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (!strncmp(line, key, n) && line[n] == ' ') + value = atoll(line + n + 1); + fclose(f); + return value; +} + +/* drm-resident-system of the client behind @fd, in bytes: the memory it holds. */ +static long long client_resident(int fd) +{ + char path[64], line[256], unit[8]; + long long value = -1; + FILE *f; + + snprintf(path, sizeof(path), "/proc/self/fdinfo/%d", fd); + f = fopen(path, "r"); + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) { + unit[0] = 0; + if (sscanf(line, "drm-resident-system: %lld %7s", &value, unit) < 1) + continue; + if (!strcmp(unit, "KiB")) + value <<= 10; + else if (!strcmp(unit, "MiB")) + value <<= 20; + break; + } + fclose(f); + return value; +} + +/* MemAvailable in bytes. */ +static long long mem_available(void) +{ + char line[256]; + long long kb = -1; + FILE *f = fopen("/proc/meminfo", "r"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "MemAvailable: %lld kB", &kb) == 1) + break; + fclose(f); + return kb < 0 ? -1 : kb << 10; +} + +static int write_string(const char *path, const char *text) +{ + int fd = open(path, O_WRONLY | O_CLOEXEC); + int ret; + + if (fd < 0) + return -errno; + ret = write(fd, text, strlen(text)) < 0 ? -errno : 0; + close(fd); + return ret; +} + +#define MEMORY_RECLAIM "/sys/fs/cgroup/memory.reclaim" + +/* The bytes of the cold cache file @fd that are in the page cache, or -errno. */ +static long long cold_cached(int fd) +{ + struct cachestat_range range = { .off = 0, .len = COLD_BYTES }; + struct cachestat cs; + + if (syscall(__NR_cachestat, fd, &range, &cs, 0)) + return -errno; + return (long long)cs.nr_cache * sysconf(_SC_PAGESIZE); +} + +/* + * Creates the cold cache: an unnamed sparse file of COLD_BYTES in the working + * directory, read once from end to end. Reading a hole leaves clean pages in + * the page cache without writing to the disk, and pages read once are the + * first ones reclaim takes. Returns the file, which holds the cache until it + * is closed, or -1 and the reason in @why. + */ +static int cold_cache(const char **why) +{ + static char buf[1 << 20]; + struct statfs fs; + int fd = open(".", O_TMPFILE | O_RDWR | O_CLOEXEC, 0600); + + *why = "cannot create a temporary file in the working directory"; + if (fd < 0) + return -1; + if (fstatfs(fd, &fs) || fs.f_type == TMPFS_MAGIC || + fs.f_type == RAMFS_MAGIC) { + *why = "the working directory is in memory: reclaim cannot drop its files"; + goto fail; + } + if (ftruncate(fd, COLD_BYTES)) + goto fail; + *why = "cannot read the temporary file into the page cache"; + for (uint64_t off = 0; off < COLD_BYTES; off += sizeof(buf)) + if (pread(fd, buf, sizeof(buf), off) != sizeof(buf)) + goto fail; + if (cold_cached(fd) < (long long)(COLD_BYTES - COLD_BYTES / 8)) { + *why = "the file system does not keep a file that was read in the page cache"; + goto fail; + } + return fd; +fail: + close(fd); + return -1; +} + +/* + * The pages kswapd and direct reclaim have scanned since boot: reclaim the + * system does of its own accord, at priorities the test knows nothing about. + */ +static long long scanned_unasked(void) +{ + char line[256]; + long long value, sum = 0; + FILE *f = fopen("/proc/vmstat", "r"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "pgscan_kswapd %lld", &value) == 1 || + sscanf(line, "pgscan_direct %lld", &value) == 1) + sum += value; + fclose(f); + return sum; +} + +/* + * Whether the memory cgroup of this process, or one above it, is protected + * from reclaim (memory.min, memory.low): reclaim then passes the cold cache + * by until it has tried everything else. + */ +static int cgroup_protected(void) +{ + char path[512] = "/sys/fs/cgroup", line[400], file[600], value[32]; + FILE *f = fopen("/proc/self/cgroup", "r"); + size_t root = strlen(path); + char *end; + + if (!f) + return 0; + while (fgets(line, sizeof(line), f)) + if (!strncmp(line, "0::", 3)) + snprintf(path + root, sizeof(path) - root, "%s", line + 3); + fclose(f); + path[strcspn(path, "\n")] = 0; + for (end = path + strlen(path); end > path + root; end = strrchr(path, '/')) { + static const char * const names[] = { "memory.min", "memory.low" }; + + *end = 0; + for (int i = 0; i < 2; i++) { + snprintf(file, sizeof(file), "%s/%s", path, names[i]); + f = fopen(file, "r"); + if (!f) + continue; + if (fgets(value, sizeof(value), f) && strcmp(value, "0\n")) { + fclose(f); + return 1; + } + fclose(f); + } + } + return 0; +} + +static double seconds(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return ts.tv_sec + ts.tv_nsec / 1e9; +} + +/* + * A number of pages that a case adds to the size of its kept object, drawn + * anew by every case: 0 to 1023 GPU pages. It makes what the VM's shrinker + * of kept objects counts a number that another VM's is unlikely to count, + * which is how the case finds its own shrinker in the record. + */ +static uint64_t own_pages(void) +{ + uint16_t r = 0; + + if (getrandom(&r, sizeof(r), 0) != sizeof(r)) + r = getpid() ^ time(NULL); + return r % 1024; +} + +/* + * The record of the shrinker core's passes over the shrinkers: a trace + * instance with the events that the core emits before it decides whether to + * call a shrinker and after it is done with it. + * + * The instance belongs to the program, not to a case. main() makes it before + * the cases fork, under a name with its process ID, and removes it when the + * program exits or is told to end, whatever became of the cases: a case + * that fails, times out or is killed leaves nothing to clean up. An + * instance of that name that exists already is somebody else's and is left + * alone; the cases then go without a record. A case only empties the buffer + * and switches the events on and off. + */ +#define TRACE_EVENTS "/events/vmscan/mm_shrink_slab_" + +static char trace_dir[80]; +/* The enable files of the two events, made with trace_dir. */ +static char trace_enable[2][160]; +static pid_t trace_owner; +/* The signals that end the program, blocked while the instance changes hands. */ +static const int trace_signals[] = { SIGINT, SIGTERM, SIGHUP, SIGQUIT }; +static sigset_t trace_sigset; + +/* Whether this case is recording. */ +static int tracing; + +static int trace_write(const char *file, const char *text) +{ + char path[160]; + + snprintf(path, sizeof(path), "%s%s", trace_dir, file); + return write_string(path, text); +} + +static void trace_events(const char *on) +{ + write_string(trace_enable[0], on); + write_string(trace_enable[1], on); +} + +/* + * Only async-signal-safe calls, on paths made beforehand, from here: it also + * runs from a signal handler. The signals that end the program are blocked + * meanwhile, so that it never runs twice at once. + */ +static void trace_remove(void) +{ + static const struct timespec pause = { .tv_nsec = 50000000 }; + sigset_t old; + + if (!trace_dir[0] || getpid() != trace_owner) + return; + sigprocmask(SIG_BLOCK, &trace_sigset, &old); + if (trace_dir[0]) { + trace_events("0"); + /* A case that was just killed may still hold a file of it. */ + for (int i = 0; i < 40 && rmdir(trace_dir) && errno == EBUSY; i++) + nanosleep(&pause, NULL); + trace_dir[0] = 0; + } + sigprocmask(SIG_SETMASK, &old, NULL); +} + +/* The signal stays blocked until the handler returns, and then ends the program. */ +static void trace_remove_on(int sig) +{ + trace_remove(); + signal(sig, SIG_DFL); + raise(sig); +} + +/* + * Makes the instance, if the reclaim cases were asked for. Its timestamps + * are on the clock of seconds(). + */ +static void trace_create(void) +{ + const char *reclaim = getenv("ASAHI_RESIDENCY_RECLAIM"); + const char *drop = getenv("ASAHI_RESIDENCY_DROP_CACHES"); + struct sigaction sa = { .sa_handler = trace_remove_on }; + sigset_t old; + + if (geteuid() || !((reclaim && !strcmp(reclaim, "1")) || + (drop && !strcmp(drop, "1")))) + return; + snprintf(trace_dir, sizeof(trace_dir), + "/sys/kernel/tracing/instances/asahi-residency-%d", getpid()); + snprintf(trace_enable[0], sizeof(trace_enable[0]), + "%s" TRACE_EVENTS "start/enable", trace_dir); + snprintf(trace_enable[1], sizeof(trace_enable[1]), + "%s" TRACE_EVENTS "end/enable", trace_dir); + sigemptyset(&trace_sigset); + for (unsigned int i = 0; i < ARRAY_SIZE(trace_signals); i++) + sigaddset(&trace_sigset, trace_signals[i]); + sa.sa_mask = trace_sigset; + + /* + * A signal waits until the instance is set up: until the handlers are + * in place, and until no file of the instance is open, which would + * keep the handler from removing it. + */ + sigprocmask(SIG_BLOCK, &trace_sigset, &old); + if (mkdir(trace_dir, 0700)) { + trace_dir[0] = 0; + } else { + trace_owner = getpid(); + atexit(trace_remove); + for (unsigned int i = 0; i < ARRAY_SIZE(trace_signals); i++) + sigaction(trace_signals[i], &sa, NULL); + if (trace_write("/trace_clock", "mono") || + trace_write("/buffer_size_kb", "4096")) + trace_remove(); + } + sigprocmask(SIG_SETMASK, &old, NULL); +} + +/* Starts the record of a case afresh. */ +static void trace_start(void) +{ + char path[160]; + int fd; + + if (!trace_dir[0]) + return; + snprintf(path, sizeof(path), "%s/trace", trace_dir); + fd = open(path, O_WRONLY | O_TRUNC | O_CLOEXEC); + if (fd < 0) + return; + close(fd); + if (write_string(trace_enable[0], "1") || + write_string(trace_enable[1], "1")) { + trace_events("0"); + return; + } + tracing = 1; +} + +static void trace_stop(void) +{ + if (!tracing) + return; + trace_events("0"); + tracing = 0; +} + +/* One of the driver's shrinker events. */ +struct shrink_event { + /* mm_shrink_slab_end, not _start. */ + int end; + /* Of a shrinker of kept objects, not of a VM's other shrinker. */ + int kept; + double time; + /* The shrinker, as the event prints its pointer. */ + char shrinker[24]; + /* The task that makes the pass. */ + int pid; + /* Of a start event: what the shrinker counted, the target, the priority. */ + long items, target; + int priority; +}; + +/* + * Reads a line of the record. Returns 1 for an event of one of the driver's + * shrinkers, which are told by the names of their scan functions (scan_kept + * of VmShrinker for the kept objects of a VM, VmShrinker's other one for the + * rest), 0 for any other line and -1 for an event that does not read as + * expected. + */ +static int shrink_event(const char *line, struct shrink_event *e) +{ + const char *at = strstr(line, ": mm_shrink_slab_"); + const char *nid, *p, *items, *target, *priority; + size_t n; + + if (line[0] == '#' || !at) + return 0; + for (p = at; p > line && p[-1] != ' '; p--) + ; + e->time = strtod(p, NULL); + at += strlen(": mm_shrink_slab_"); + e->end = !strncmp(at, "end: ", 5); + if (!e->end && strncmp(at, "start: ", 7)) + return 0; + at = strchr(at, ' ') + 1; + /* " : nid: ..." */ + nid = strstr(at, ": nid: "); + if (!nid) + return -1; + for (p = nid; p > at && p[-1] != ' '; p--) + ; + n = nid - p; + if (p == at || !n || n >= sizeof(e->shrinker)) + return -1; + if (!memmem(at, p - at, "VmShrinker", strlen("VmShrinker"))) + return 0; + e->kept = !!memmem(at, p - at, "scan_kept", strlen("scan_kept")); + memcpy(e->shrinker, p, n); + e->shrinker[n] = 0; + p = strstr(nid, " pid: "); + if (!p) + return -1; + e->pid = atoi(p + strlen(" pid: ")); + if (e->end) + return 1; + items = strstr(nid, " cache items "); + target = strstr(nid, " total_scan "); + priority = strstr(nid, " priority "); + if (!items || !target || !priority) + return -1; + e->items = atol(items + strlen(" cache items ")); + e->target = atol(target + strlen(" total_scan ")); + e->priority = atoi(priority + strlen(" priority ")); + return 1; +} + +/* What a case knows of its own VM for reading the record. */ +struct own_vm { + /* + * What the VM's shrinker of kept objects can have counted since the + * record began: the pages of the idle kept objects, in the one or two + * combinations the case brings about. + */ + long counts[2]; + /* + * When the case last saw its kept objects resident (taken before it + * looked) and first saw one gone (taken after it looked); 0 and + * infinity while it did not. + */ + double held_at, gone_at; +}; + +/* What the record has about the shrinkers of kept objects. */ +struct kept_record { + /* Passes over the VMs' other shrinker: the record has the driver's events. */ + int others; + /* Shrinkers that only ever counted what the test's VM counts. */ + int shrinkers; + /* Passes over those, and the ones among them that called the shrinker. */ + int passes, called; + /* The called ones that could have released the object that went. */ + int released; + /* The most urgent priority among those, or among the called ones. */ + int priority; +}; + +#define MAX_SHRINKERS 64 +#define MAX_CALLS 256 + +/* + * Reads the record for the case whose VM is @vm. The shrinker core calls a + * shrinker when the target of the pass (total_scan) reaches the shrinker's + * batch or everything it counted (cache items); the batch of a shrinker of + * kept objects is out of reach, which leaves the second. + * + * The record names shrinkers by pointer. The one of the test's VM is among + * those that never counted anything but @vm->counts; there is one such + * shrinker unless another VM's happens to count the same. + * + * A pass could have released an object that the case saw resident at + * held_at and gone at gone_at if it called the shrinker before gone_at and + * had not ended by held_at. A pass has ended with the next end event of the + * same shrinker from the same task. + * + * Returns 0, or -1 where there is no record, events were lost or it has + * more shrinkers or calls than this keeps track of. + */ +static int read_record(const struct own_vm *vm, struct kept_record *r) +{ + static struct { + char shrinker[24]; + int foreign, passes; + } seen[MAX_SHRINKERS]; + static struct { + /* pid: the task, and once the record is read, whether it is in the interval. */ + int shrinker, pid, priority; + double start, end; + } calls[MAX_CALLS]; + int nseen = 0, ncalls = 0, whole = 0, k; + long in_buffer, written; + char path[160], line[1024]; + struct shrink_event e; + FILE *f; + + memset(r, 0, sizeof(*r)); + r->priority = -1; + if (!tracing) + return -1; + snprintf(path, sizeof(path), "%s/trace", trace_dir); + f = fopen(path, "r"); + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) { + int ret = shrink_event(line, &e); + + if (sscanf(line, "# entries-in-buffer/entries-written: %ld/%ld", + &in_buffer, &written) == 2) + whole = in_buffer == written; + if (ret < 0) + whole = 0; + if (ret <= 0) + continue; + if (!e.kept) { + r->others += !e.end; + continue; + } + for (k = 0; k < nseen && strcmp(seen[k].shrinker, e.shrinker); k++) + ; + if (e.end) { + for (int c = ncalls - 1; c >= 0; c--) { + if (calls[c].shrinker != k || calls[c].pid != e.pid || + !isinf(calls[c].end)) + continue; + calls[c].end = e.time; + break; + } + continue; + } + if (k == nseen) { + if (nseen == MAX_SHRINKERS) { + whole = 0; + break; + } + memset(&seen[k], 0, sizeof(seen[k])); + strcpy(seen[k].shrinker, e.shrinker); + nseen++; + } + seen[k].passes++; + if (e.items != vm->counts[0] && e.items != vm->counts[1]) + seen[k].foreign = 1; + if (e.target < e.items) + continue; + if (ncalls == MAX_CALLS) { + whole = 0; + break; + } + calls[ncalls].shrinker = k; + calls[ncalls].pid = e.pid; + calls[ncalls].priority = e.priority; + calls[ncalls].start = e.time; + calls[ncalls].end = INFINITY; + ncalls++; + } + fclose(f); + if (!whole) + return -1; + for (k = 0; k < nseen; k++) { + if (seen[k].foreign) + continue; + r->shrinkers++; + r->passes += seen[k].passes; + } + /* The timestamps of the record are rounded to a microsecond. */ + for (int c = 0; c < ncalls; c++) { + calls[c].pid = calls[c].start <= vm->gone_at + 2e-6 && + calls[c].end >= vm->held_at - 2e-6; + if (seen[calls[c].shrinker].foreign) + continue; + r->called++; + r->released += calls[c].pid; + } + for (int c = 0; c < ncalls; c++) { + if (seen[calls[c].shrinker].foreign || + (r->released && !calls[c].pid)) + continue; + if (r->priority < 0 || calls[c].priority < r->priority) + r->priority = calls[c].priority; + } + return 0; +} + +/* The reclaim requests of a case, and whether the kernel could meet them lightly. */ +struct pressure { + /* The cold cache file. */ + int cold; + /* The requests made. */ + int requests; + /* Those the kernel did not meet in full. */ + int unmet; + /* The least of the cold cache that was left after a request. */ + long long cold_low; + /* scanned_unasked() when the case began. */ + long long unasked; + /* For the record of the shrinker passes. */ + struct own_vm vm; +}; + +/* + * Asks the root memory cgroup to reclaim REQUEST. The kernel returns EAGAIN + * when it could not reclaim that much at any priority, which is counted; + * any other error is returned. + */ +static int reclaim_some(struct pressure *p) +{ + int ret = write_string(MEMORY_RECLAIM, REQUEST); + long long cached = cold_cached(p->cold); + + p->requests++; + if (ret == -EAGAIN) { + p->unmet++; + ret = 0; + } + if (cached < p->cold_low) + p->cold_low = cached; + return ret; +} + +/* + * A heuristic, for where the shrinker passes cannot be recorded: a sign that + * reclaim may have gone to its last priorities since the case began, or + * NULL. With every request met, COLD_FLOOR of the cold cache left after each + * and no other reclaim in the system, none is likely to have gone beyond the + * first few priorities; that is not shown. + */ +static const char *urgent_reclaim_signs(const struct pressure *p) +{ + if (p->unmet) + return "the kernel could not meet a reclaim request"; + if (p->cold_low < (long long)COLD_FLOOR) + return "the cold cache ran low"; + if (scanned_unasked() != p->unasked) + return "kswapd or direct reclaim ran meanwhile"; + return NULL; +} + +/* + * What to make of a kept object that went between p->vm.held_at and + * p->vm.gone_at. Returns 0 if reclaim had no leave to release it, so that + * the case fails, and 1 if the case is to be skipped; either way @why says + * on what grounds. + * + * By the record of the shrinker passes where there is one that shows the + * driver's shrinkers at work (the cases ask once reclaim has released an + * object that is not kept). Reclaim had leave if, in that interval, a pass + * called the test VM's shrinker of kept objects: "as it may". If the record + * has several shrinkers that count what the test's VM counts and a pass + * called one of them, it does not tell whether that was the test's: + * "inconclusive", which claims nothing about the release. With no such pass + * for any of them, reclaim had no leave. + * + * By the heuristic otherwise, and the words say so. + */ +static int kept_could_go(const struct pressure *p, char *why, size_t size) +{ + struct kept_record r; + const char *sign; + + if (!read_record(&p->vm, &r) && r.others) { + if (!r.released) + snprintf(why, size, + "no reclaim pass called the shrinker of this VM's kept objects before it went (traced: %d shrinkers of kept objects count what this VM's counts, with %d passes and %d calls at other times)", + r.shrinkers, r.passes, r.called); + else if (r.shrinkers == 1) + snprintf(why, size, + "as it may: reclaim passes that called the shrinker of this VM's kept objects before it went: %d, down to priority %d (traced)", + r.released, r.priority); + else + snprintf(why, size, + "inconclusive: %d shrinkers of kept objects count what this VM's counts, and %d passes called one of them before it went (traced)", + r.shrinkers, r.released); + return r.released > 0; + } + sign = urgent_reclaim_signs(p); + if (sign) + snprintf(why, size, + "as it may: %s (a heuristic: the shrinker passes were not traced)", + sign); + else + snprintf(why, size, + "no sign of urgent reclaim (a heuristic: the shrinker passes were not traced)"); + return sign != NULL; +} + +/* For the log of a case that passed: what the record of the shrinker passes has. */ +static const char *trace_note(const struct pressure *p, char *note, size_t size) +{ + struct kept_record r; + + if (!read_record(&p->vm, &r) && r.others) + snprintf(note, size, + "reclaim passes over the shrinker of this VM's kept objects: %d, of which %d called it (traced, %d such shrinker%s)", + r.passes, r.called, r.shrinkers, r.shrinkers == 1 ? "" : "s"); + else + snprintf(note, size, "the shrinker passes were not traced"); + return note; +} + +static int residency(int fd, uint32_t handle, uint32_t flags) +{ + struct drm_asahi_gem_residency args = { .handle = handle, .flags = flags }; + + return ioc(fd, DRM_IOCTL_ASAHI_GEM_RESIDENCY, &args); +} + +/* Gives advice for @handle; returns whether its contents are retained, or -errno. */ +static int madvise_object(int fd, uint32_t handle, uint32_t madv) +{ + struct drm_asahi_gem_madvise args = { .handle = handle, .madv = madv }; + int ret = ioc(fd, DRM_IOCTL_ASAHI_GEM_MADVISE, &args); + + return ret ? ret : (int)args.retained; +} + +static int bind(int fd, uint32_t vm, uint32_t handle, uint64_t addr, + uint64_t size, uint32_t flags) +{ + struct drm_asahi_gem_bind_op op = { + .flags = flags, .handle = handle, .range = size, .addr = addr, + }; + struct drm_asahi_vm_bind args = { + .vm_id = vm, .num_binds = 1, .stride = sizeof(op), + .userptr = (uintptr_t)&op, + }; + + return ioc(fd, DRM_IOCTL_ASAHI_VM_BIND, &args); +} + +static void close_handle(int fd, uint32_t handle) +{ + struct drm_gem_close args = { .handle = handle }; + + ioc(fd, DRM_IOCTL_GEM_CLOSE, &args); +} + +/* Whether the driver behind @fd is this one. */ +static int is_asahi(int fd) +{ + char name[16] = ""; + struct drm_version version = { + .name = name, .name_len = sizeof(name) - 1, + }; + + return !ioc(fd, DRM_IOCTL_VERSION, &version) && !strcmp(name, "aurora"); +} + +FIXTURE(asahi) +{ + int fd; + /* The cold cache file of a reclaim case, or -1. */ + int cold; + struct drm_asahi_params_global params; + uint32_t vm; + uint64_t va, va_end; +}; + +FIXTURE_SETUP(asahi) +{ + struct drm_asahi_get_params get = { + .pointer = (uintptr_t)&self->params, + .size = sizeof(self->params), + }; + + self->cold = -1; + self->fd = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC); + if (self->fd < 0 || !is_asahi(self->fd)) + SKIP(return, "renderD128 is not an asahi render node"); + memset(&self->params, 0, sizeof(self->params)); + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GET_PARAMS, &get)); + if (!(self->params.features & DRM_ASAHI_FEATURE_GEM_RESIDENCY)) + SKIP(return, "no DRM_ASAHI_FEATURE_GEM_RESIDENCY"); + + uint64_t kernel = self->params.vm_kernel_min_size > (32ULL << 30) ? + self->params.vm_kernel_min_size : 32ULL << 30; + struct drm_asahi_vm_create vm = { + .kernel_start = self->params.vm_end - kernel, + .kernel_end = self->params.vm_end, + }; + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_VM_CREATE, &vm)); + self->vm = vm.vm_id; + self->va = self->params.vm_start; + self->va_end = vm.kernel_start; +} + +FIXTURE_TEARDOWN(asahi) +{ + trace_stop(); + close(self->fd); + /* Closing the unnamed file gives its page cache back. */ + if (self->cold >= 0) + close(self->cold); +} + +/* + * Creates a VM-private object of @size bytes in @vm and binds it, which + * gives it its memory. The driver keeps ranges of every VM for its own + * mappings, where a bind fails with EINVAL, and GET_PARAMS does not list + * them: the object goes to the first address after *@va, in steps of 4 GiB + * and below @va_end, at which it binds, and *@va moves past it. Returns its + * handle, or 0. + */ +static uint32_t bound_object(int fd, uint32_t vm, uint64_t *va, uint64_t va_end, + uint64_t size) +{ + struct drm_asahi_gem_create bo = { + .size = size, .flags = DRM_ASAHI_GEM_VM_PRIVATE, .vm_id = vm, + }; + + if (ioc(fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)) + return 0; + for (;;) { + *va = (*va + (4ULL << 30)) & ~((4ULL << 30) - 1); + if (*va + size > va_end) + break; + if (!bind(fd, vm, bo.handle, *va, size, + DRM_ASAHI_BIND_READ | DRM_ASAHI_BIND_WRITE)) { + *va += size; + return bo.handle; + } + } + close_handle(fd, bo.handle); + return 0; +} + +/* + * Prepares a case that asks the root memory cgroup for reclaim, with objects + * of @bytes in all: checks that it was asked for and can run here, and + * creates the cold cache, whose file goes to *@cold. Returns NULL, or why + * the case is skipped. + */ +static const char *reclaim_setup(struct pressure *p, uint64_t bytes, int *cold) +{ + const char *run = getenv("ASAHI_RESIDENCY_RECLAIM"); + const char *why; + + if (!run || strcmp(run, "1")) + return "set ASAHI_RESIDENCY_RECLAIM=1 to ask the root memory cgroup for reclaim"; + if (geteuid()) + return "reclaim requests need root"; + if (access(MEMORY_RECLAIM, W_OK)) + return "no " MEMORY_RECLAIM; + if (mem_available() < (long long)(4 * bytes + 2 * COLD_BYTES)) + return "not enough available memory for the test objects"; + if (cgroup_protected()) + return "the test's memory cgroup has memory.min or memory.low protection"; + *cold = cold_cache(&why); + if (*cold < 0) + return why; + memset(p, 0, sizeof(*p)); + p->cold = *cold; + p->cold_low = cold_cached(*cold); + p->unasked = scanned_unasked(); + p->vm.gone_at = INFINITY; + trace_start(); + return NULL; +} + +/* + * Notes, for the record, an observation of the case's kept objects that + * began at @before: whether they were all resident (see struct own_vm). + */ +static void note_kept(struct own_vm *vm, double before, int intact) +{ + if (intact) { + vm->held_at = before; + vm->gone_at = INFINITY; + } else if (isinf(vm->gone_at)) { + vm->gone_at = seconds(); + } +} + +/* + * What the client behind @fd holds. Its kept objects are all resident while + * that is @resident or @kept, which is noted for the record. + */ +static long long watch_resident(struct pressure *p, int fd, long long resident, + long long kept) +{ + double before = seconds(); + long long now = client_resident(fd); + + note_kept(&p->vm, before, now == resident || now == kept); + return now; +} + +/* + * Makes reclaim requests, at most @most, for as long as the client behind + * @fd holds @resident bytes. @kept is what it holds once everything but its + * kept objects has gone. Returns what it holds then, or a negative value: + * -errno of a request that failed, or -1 where what the client holds could + * not be read. The callers assert that it is not negative before they take + * the result for a sign of anything. + */ +static long long reclaim_while(struct pressure *p, int fd, long long resident, + long long kept, int most) +{ + long long now = watch_resident(p, fd, resident, kept); + + for (int i = 0; i < most && now == resident; i++) { + int ret = reclaim_some(p); + + if (ret) + return ret; + now = watch_resident(p, fd, resident, kept); + } + return now; +} + +/* The kept total follows a VM-private object: once per object, until freed. */ +TEST_F(asahi, kept_size_is_counted_once) +{ + const uint64_t size = 16 * PAGE; + struct drm_asahi_gem_create bo = { + .size = size, .flags = DRM_ASAHI_GEM_VM_PRIVATE, .vm_id = self->vm, + }; + long long kept0; + + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)); + kept0 = gpu_memory_field("user_kept"); + ASSERT_GE(kept0, 0); + ASSERT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + ASSERT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0 + (long long)size, gpu_memory_field("user_kept")); + ASSERT_EQ(0, residency(self->fd, bo.handle, 0)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + ASSERT_EQ(0, residency(self->fd, bo.handle, 0)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + /* Freeing a kept object stops counting it. */ + ASSERT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0 + (long long)size, gpu_memory_field("user_kept")); + close_handle(self->fd, bo.handle); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); +} + +TEST_F(asahi, bad_arguments) +{ + struct drm_asahi_gem_create bo = { + .size = PAGE, .flags = DRM_ASAHI_GEM_VM_PRIVATE, .vm_id = self->vm, + }; + long long kept0 = gpu_memory_field("user_kept"); + + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)); + EXPECT_EQ(-EINVAL, residency(self->fd, bo.handle, 1u << 1)); + EXPECT_EQ(-EINVAL, residency(self->fd, bo.handle, + DRM_ASAHI_GEM_RESIDENCY_KEEP | 1u << 31)); + EXPECT_EQ(-ENOENT, residency(self->fd, 0xffffff, + DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + close_handle(self->fd, bo.handle); +} + +/* An object that is never released takes the flag all the same. */ +TEST_F(asahi, shared_objects_take_the_flag) +{ + const uint64_t size = 2 * PAGE; + struct drm_asahi_gem_create bo = { .size = size }; + long long kept0 = gpu_memory_field("user_kept"); + + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)); + EXPECT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0 + (long long)size, gpu_memory_field("user_kept")); + EXPECT_EQ(0, residency(self->fd, bo.handle, 0)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + close_handle(self->fd, bo.handle); +} + +/* + * Of two idle objects, reclaim releases the one that is not kept first, at + * any priority. Reclaim that does not come to the kept objects (see the top + * of this file for how the case knows) leaves the kept one resident; once + * the request is withdrawn, it is released as well. The kept object is the + * larger one, so that what the client holds tells which one stayed. + */ +TEST_F_TIMEOUT(asahi, light_reclaim_spares_kept_objects, 180) +{ + const uint64_t size = RECLAIM_PAGES * sysconf(_SC_PAGESIZE); + const uint64_t kept_size = size + (1024 + own_pages()) * PAGE; + const long long both = kept_size + size; + struct pressure p; + const char *why = reclaim_setup(&p, both, &self->cold); + long long resident; + char reason[320]; + uint32_t a, b; + int first; + + if (why) + SKIP(return, "%s", why); + p.vm.counts[0] = p.vm.counts[1] = kept_size / sysconf(_SC_PAGESIZE); + a = bound_object(self->fd, self->vm, &self->va, self->va_end, kept_size); + b = bound_object(self->fd, self->vm, &self->va, self->va_end, size); + ASSERT_NE(0, a); + ASSERT_NE(0, b); + ASSERT_EQ(both, client_resident(self->fd)); + ASSERT_EQ(0, residency(self->fd, a, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + + sleep(IDLE_SECONDS); + resident = reclaim_while(&p, self->fd, both, kept_size, MAX_REQUESTS); + first = p.requests; + ASSERT_GE(resident, 0); + /* The shrinkers ran: something went, and never the kept object alone. */ + ASSERT_NE(both, resident); + ASSERT_NE((long long)size, resident); + if (resident == (long long)kept_size) { + resident = reclaim_while(&p, self->fd, kept_size, kept_size, + SPARED_REQUESTS); + ASSERT_GE(resident, 0); + } + if (resident != (long long)kept_size) { + if (kept_could_go(&p, reason, sizeof(reason))) + SKIP(return, "the kept object went after %d requests, %s", + p.requests, reason); + TH_LOG("the kept object went after %d requests, without leave: %s", + p.requests, reason); + } + ASSERT_EQ((long long)kept_size, resident); + TH_LOG("the object that is not kept went with request %d", first); + TH_LOG("the kept one stayed through %d more, %lld MiB of the cold cache left; %s", + p.requests - first, p.cold_low >> 20, + trace_note(&p, reason, sizeof(reason))); + + ASSERT_EQ(0, residency(self->fd, a, 0)); + first = p.requests; + resident = reclaim_while(&p, self->fd, kept_size, kept_size, + MAX_REQUESTS); + ASSERT_GE(resident, 0); + EXPECT_EQ(0, resident); + TH_LOG("the object no longer kept went after %d requests", + p.requests - first); +} + +/* Contents given up with DONTNEED are purged by reclaim, kept or not. */ +TEST_F_TIMEOUT(asahi, kept_objects_stay_purgeable, 180) +{ + const uint64_t size = RECLAIM_PAGES * sysconf(_SC_PAGESIZE); + struct pressure p; + const char *why = reclaim_setup(&p, size, &self->cold); + long long resident; + uint32_t a; + + if (why) + SKIP(return, "%s", why); + a = bound_object(self->fd, self->vm, &self->va, self->va_end, size); + ASSERT_NE(0, a); + ASSERT_EQ(0, residency(self->fd, a, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + ASSERT_EQ(1, madvise_object(self->fd, a, DRM_ASAHI_GEM_MADV_DONTNEED)); + + sleep(IDLE_SECONDS); + resident = reclaim_while(&p, self->fd, size, size, MAX_REQUESTS); + ASSERT_GE(resident, 0); + EXPECT_EQ(0, resident); + TH_LOG("purged after %d requests", p.requests); + EXPECT_EQ(0, madvise_object(self->fd, a, DRM_ASAHI_GEM_MADV_WILLNEED)); +} + +struct toggler { + int fd; + uint32_t handle; + int stop; + int failed; + long requests; +}; + +static void *toggle_residency(void *arg) +{ + struct toggler *t = arg; + + while (!__atomic_load_n(&t->stop, __ATOMIC_RELAXED)) { + if (residency(t->fd, t->handle, DRM_ASAHI_GEM_RESIDENCY_KEEP) || + residency(t->fd, t->handle, 0)) + t->failed = 1; + t->requests += 2; + } + return NULL; +} + +/* The objects of the race case, as bits of what the client holds. */ +#define TOGGLED 1 +#define CONTROL 2 +#define KEPT 4 + +/* + * Which objects a client that holds @resident bytes has resident, as a mask + * of the bits above, or -1. @sizes: their sizes, in the order of the bits. + */ +static int resident_objects(long long resident, const long long sizes[3]) +{ + for (int mask = 0; mask < 8; mask++) { + long long sum = 0; + + for (int i = 0; i < 3; i++) + if (mask & (1 << i)) + sum += sizes[i]; + if (sum == resident) + return mask; + } + return -1; +} + +/* + * Residency requests racing with reclaim. One thread sets and clears the + * request on one object as fast as it can while reclaim runs. An object + * beside it that is not kept (the control) has to be released during the + * race, which shows that reclaim got to this VM between the requests; a kept + * object beside it has to stay, as long as reclaim does not come to the kept + * objects (see the top of this file for how the case knows). Whatever a + * request that has returned found (the object resident or already released), + * reclaim leaves that way, and the kept total counts each object once. The + * three objects differ in size, so that what the client holds tells which of + * them are resident. + */ +TEST_F_TIMEOUT(asahi, requests_serialize_with_reclaim, 180) +{ + const uint64_t size = RECLAIM_PAGES * sysconf(_SC_PAGESIZE); + const uint64_t small = 1024 * PAGE; + const long long sizes[3] = { + size + 2 * small, size, small + own_pages() * PAGE, + }; + struct toggler t = { .fd = self->fd }; + long long kept0 = gpu_memory_field("user_kept"); + struct pressure p; + const char *why = reclaim_setup(&p, sizes[0] + sizes[1] + sizes[2], + &self->cold); + int resident = TOGGLED | CONTROL | KEPT, found, error = 0; + int control_gone = 0; + char reason[320]; + uint32_t control, kept; + pthread_t thread; + double end, before; + + if (why) + SKIP(return, "%s", why); + /* The kept object alone, or with the toggled one while that is kept. */ + p.vm.counts[0] = sizes[2] / sysconf(_SC_PAGESIZE); + p.vm.counts[1] = (sizes[0] + sizes[2]) / sysconf(_SC_PAGESIZE); + t.handle = bound_object(self->fd, self->vm, &self->va, self->va_end, sizes[0]); + control = bound_object(self->fd, self->vm, &self->va, self->va_end, sizes[1]); + kept = bound_object(self->fd, self->vm, &self->va, self->va_end, sizes[2]); + ASSERT_NE(0, t.handle); + ASSERT_NE(0, control); + ASSERT_NE(0, kept); + ASSERT_EQ(0, residency(self->fd, kept, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + kept0 += sizes[2]; + ASSERT_EQ(0, pthread_create(&thread, NULL, toggle_residency, &t)); + /* + * From before the VM counts as idle until a second after reclaim + * released the control object, for ten seconds at most. + */ + sleep(IDLE_SECONDS - 2); + end = seconds() + 10; + while (seconds() < end && !error && resident >= 0 && (resident & KEPT)) { + error = reclaim_some(&p); + before = seconds(); + resident = resident_objects(client_resident(self->fd), sizes); + note_kept(&p.vm, before, resident >= 0 && (resident & KEPT)); + if (resident >= 0 && !(resident & CONTROL) && !control_gone) { + control_gone = p.requests; + end = seconds() + 1; + } + usleep(100000); + } + __atomic_store_n(&t.stop, 1, __ATOMIC_RELAXED); + ASSERT_EQ(0, pthread_join(thread, NULL)); + ASSERT_EQ(0, error); + ASSERT_GE(resident, 0); + EXPECT_EQ(0, t.failed); + EXPECT_GT(t.requests, 0); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + + /* The toggled object is kept from here on, resident or not. */ + ASSERT_EQ(0, residency(self->fd, t.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + before = seconds(); + found = resident_objects(client_resident(self->fd), sizes); + note_kept(&p.vm, before, found >= 0 && (found & KEPT)); + resident = found; + for (int i = 0; i < SPARED_REQUESTS && resident == found; i++) { + ASSERT_EQ(0, reclaim_some(&p)); + before = seconds(); + resident = resident_objects(client_resident(self->fd), sizes); + note_kept(&p.vm, before, (found & KEPT) && + (resident | CONTROL) == (found | CONTROL)); + } + /* What the client holds was read, and is a sum of the objects' sizes. */ + ASSERT_GE(found, 0); + ASSERT_GE(resident, 0); + /* A kept object that went, during the race or after it. */ + if (!(found & KEPT) || (resident | CONTROL) != (found | CONTROL)) { + if (kept_could_go(&p, reason, sizeof(reason))) + SKIP(return, "a kept object went after %d requests, %s", + p.requests, reason); + TH_LOG("a kept object went after %d requests, without leave: %s", + p.requests, reason); + ASSERT_TRUE(found & KEPT); + ASSERT_EQ(found | CONTROL, resident | CONTROL); + } + /* Reclaim got to this VM during the race. */ + ASSERT_NE(0, control_gone); + TH_LOG("%ld requests raced with reclaim; the control object went with request %d", + t.requests, control_gone); + TH_LOG("the toggled object was %s, %lld MiB of the cold cache left; %s", + found & TOGGLED ? "left" : "released", p.cold_low >> 20, + trace_note(&p, reason, sizeof(reason))); + EXPECT_EQ(kept0 + sizes[0], gpu_memory_field("user_kept")); + close_handle(self->fd, t.handle); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + close_handle(self->fd, kept); + close_handle(self->fd, control); +} + +/* + * Urgent reclaim (here: every shrinker run at the last priority) releases + * kept objects too. Where the shrinker passes can be recorded, the record + * has to show a pass that called the shrinker of this VM's kept objects + * between the last look at the object and the first one that found it gone. + */ +TEST_F(asahi, kept_objects_go_under_urgent_pressure) +{ + const uint64_t size = 1024 * PAGE; + const uint64_t kept_size = size + own_pages() * PAGE; + const char *run = getenv("ASAHI_RESIDENCY_DROP_CACHES"); + long long kept0 = gpu_memory_field("user_kept"); + struct own_vm vm = { + .counts = { kept_size / sysconf(_SC_PAGESIZE), + kept_size / sysconf(_SC_PAGESIZE) }, + }; + struct kept_record r; + long long resident; + uint32_t a, b; + + if (!run || strcmp(run, "1")) + SKIP(return, "set ASAHI_RESIDENCY_DROP_CACHES=1 to drop the system's caches"); + if (geteuid()) + SKIP(return, "dropping caches needs root"); + a = bound_object(self->fd, self->vm, &self->va, self->va_end, kept_size); + b = bound_object(self->fd, self->vm, &self->va, self->va_end, size); + ASSERT_NE(0, a); + ASSERT_NE(0, b); + ASSERT_EQ(0, residency(self->fd, a, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + + sleep(IDLE_SECONDS); + trace_start(); + vm.held_at = seconds(); + ASSERT_EQ((long long)(kept_size + size), client_resident(self->fd)); + ASSERT_EQ(0, write_string("/proc/sys/vm/drop_caches", "2")); + resident = client_resident(self->fd); + vm.gone_at = seconds(); + EXPECT_EQ(0, resident); + if (!resident && !read_record(&vm, &r) && r.others) { + if (r.shrinkers > 1) { + TH_LOG( +"inconclusive: %d shrinkers of kept objects count what this VM's counts, and %d passes called one of them before the object went (traced)", + r.shrinkers, r.released); + } else { + EXPECT_GT(r.released, 0); + TH_LOG( +"reclaim passes that called the shrinker of this VM's kept objects before it went: %d, down to priority %d (traced)", + r.released, r.priority); + } + } + /* Released, not purged: the contents come back with the next binding. */ + EXPECT_EQ(1, madvise_object(self->fd, a, DRM_ASAHI_GEM_MADV_WILLNEED)); + EXPECT_EQ(kept0 + (long long)kept_size, gpu_memory_field("user_kept")); +} + +/* + * The trace instance is the program's: made before the cases fork and + * removed when the program exits, however its cases ended. + */ +int main(int argc, char **argv) +{ + trace_create(); + return test_harness_run(argc, argv); +} From cf5eaf225a1d837581df60060f4617da28afb9c6 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Tue, 10 Nov 2026 21:31:58 -0500 Subject: [PATCH 3/8] drm/asahi: g17: M5: report the performance state, its frequency and busy time agx_stats on the M5 reported utilisation only: pstate, the frequencies and busy_ns read "unsupported", so there was no way to tell whether the GPU runs at full clock during a workload. The M5 firmware reports its performance state in its statistics ring: tag 0x1b names the state now in effect (words 1 and 3, an index into the device tree's operating points, 1 = 338 MHz to 13 = 1620 MHz), and tag 0x19 the interval that just ended (start in words 3-4, end in words 1-2, state in word 8). The decoder now takes the state from 0x1b, time stamped by the end of the preceding 0x19, and the reader turns it into achieved_frequency_hz through the board's performance state table. A report whose two state words differ, or that names state 0 or a state beyond the table, is not used; a gate makes the state stale, as before. Its utilisation records (tag 0x13, every 8 ms) carry their own timer timestamps. busy_ns now integrates the device utilisation over the window each record closes. A record after a gap of more than two periods (the GPU was gated or records were lost) or with an earlier timestamp only sets a new baseline, as the legacy integrator does after a power change, so an idle gap is never credited. requested_frequency_hz, power_mw and temperature_raw stay unsupported. Documentation/ABI/testing/sysfs-driver-asahi-agx-stats and Documentation/gpu/asahi-usage-stats.rst describe the M5 values. Checked on the J716c (M5 Max MacBook Pro) with a single-thread loop of dependent integer operations, whose rate is proportional to the clock (54.0 M iterations/s at 1620 MHz): - one 1.4 s job that fills the GPU: every agx_stats read during it (12, 100 ms apart) gave pstate 13, achieved_frequency_hz 1620000000 and 100 % device utilisation. - 60 jobs of 20000 iterations, 30 ms apart: 48 ran in about 1792 us (335 MHz against the 338 MHz of state 1) and 12 in under 600 us. Of the agx_stats reads during the run (1 ms apart) that found a current state, 48 gave pstate 1 with achieved_frequency_hz 338000000 and 11 gave 13 with 1620000000; the others found the GPU gated. - three jobs of 300 ms at full clock, 300 ms apart, twice: busy_ns advanced by 871 and 863 ms for 905 and 919 ms of work, over 1.8 s (one 8 ms window is lost at each power-up). Tests: the telemetry unit tests (g17-telemetry-tests.rs on the host, 9 passed, and KUnit asahi_g17_telemetry) cover the M5 decode from firmware records, the busy windows, gates, gaps and other firmwares; g17-telemetry-integration-tests.py passes and still fails each of its mutations. Developed with Claude Code. Signed-off-by: Erik Bourget --- .../ABI/testing/sysfs-driver-asahi-agx-stats | 31 +++ Documentation/gpu/asahi-usage-stats.rst | 28 ++- drivers/gpu/drm/asahi/g17/channel.rs | 9 +- drivers/gpu/drm/asahi/g17/mod.rs | 9 +- drivers/gpu/drm/asahi/g17/stats.rs | 39 +++- drivers/gpu/drm/asahi/g17/telemetry.rs | 220 +++++++++++++++++- drivers/gpu/drm/asahi/sysfs.c | 22 +- drivers/gpu/drm/asahi/sysfs.h | 4 + .../asahi/g17-telemetry-integration-tests.py | 47 ++-- 9 files changed, 360 insertions(+), 49 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats b/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats index 36b417856a03b..282d38614a091 100644 --- a/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats +++ b/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats @@ -122,6 +122,37 @@ Description: frequency. No qualified busy-time duration, power unit or temperature field is available. + M5 firmware (the M5 command set of the shared G17 backend) uses + the same snapshot and keys, with these differences: + + pstate_status, pstate + Supported. The firmware reports the performance state + in effect in tag 0x1b, words 1 and 3, which must + agree: an index 1..13 into the board's performance + state table. The end of the interval record before it + (tag 0x19, words 1 and 2) is its firmware timestamp. + A report that names state 0 or a state beyond the + table removes pstate validity. The state reads stale + while the GPU cores are gated. + + achieved_frequency_hz + The frequency of pstate in the board's performance + state table, in Hz, while pstate_status is valid. + + busy_ns Cumulative u64 since the driver was bound. The tag + 0x13 records of this firmware carry a timestamp in + 24 MHz ticks (words 1 and 2). Each record credits the + window since the previous record of its epoch at its + device utilisation. A record more than two 8 ms + periods after the previous one, or with an earlier + timestamp, only starts a new window, so time the GPU + spent gated is never credited. MEASURED on T6050 + (J716c): 871 and 863 ms for three 300 ms compute jobs + at full clock (905 and 919 ms of work). + + requested_frequency_hz, power_mw, temperature_raw + Unsupported. + The host uses a 16 ms age/averaging-window limit for V25G83 and 32 ms for V26A428 (two nominal periods). These are conservative host policies, not firmware guarantees about packet duration. diff --git a/Documentation/gpu/asahi-usage-stats.rst b/Documentation/gpu/asahi-usage-stats.rst index e18208c7d25ea..2eb2166984465 100644 --- a/Documentation/gpu/asahi-usage-stats.rst +++ b/Documentation/gpu/asahi-usage-stats.rst @@ -109,11 +109,37 @@ ring on powered cores becomes stale rather than fabricating a zero-utilisation sample; only gated cores read as zero. Readers must not convert these percentages into engine nanoseconds or cycles. +M5 performance state and busy time +---------------------------------- + +The M5 firmware runs on the same backend and decoder, and its statistics ring +carries more. ``agx_stats`` reports there: + +* ``pstate``: the performance state in effect, from the firmware's state + reports (tag ``0x1b``, words 1 and 3, which must agree). It is an index into + the board's performance state table, 1 to 13. The interval record before a + report (tag ``0x19``) gives its firmware timestamp. A report that names state + 0 or a state beyond the table is not used, and ``pstate_status`` reads + ``stale`` while the cores are gated. +* ``achieved_frequency_hz``: the frequency of that state in the board's table, + while ``pstate_status`` is ``valid``. +* ``busy_ns``: the M5 utilisation records carry their own timestamps (words 1 + and 2, 24 MHz ticks), so each record closes a window of known length, which + is credited at the record's device utilisation. A record that follows the + previous one by more than two 8 ms periods, or whose timestamp went back, + only starts a new window: time the GPU spent gated is never credited. The + total keeps counting across power epochs. + +``requested_frequency_hz``, ``power_mw`` and ``temperature_raw`` stay +unsupported. ``busy_ns`` is device-wide, as on G13 and G14; it does not change +fdinfo's per-client ``drm-engine-*`` counters. + Deferred shared power telemetry ------------------------------ Current pstate and requested/achieved frequency remain unsupported for these -firmware graphs. The shared-power snapshot ABI is not sufficiently specified +firmware graphs (the M5 firmware reports its state in the statistics ring, see +above). The shared-power snapshot ABI is not sufficiently specified to implement a read-only reader. For **each** of G16S and G17P, it still needs: * The firmware-version-to-layout mapping, containing allocation and bounds, diff --git a/drivers/gpu/drm/asahi/g17/channel.rs b/drivers/gpu/drm/asahi/g17/channel.rs index d65ed18b6c786..9fc50fbad6308 100644 --- a/drivers/gpu/drm/asahi/g17/channel.rs +++ b/drivers/gpu/drm/asahi/g17/channel.rs @@ -92,11 +92,16 @@ pub(crate) struct Rings { } impl Rings { - pub(crate) fn new(graph: super::fw::variant::InitGraph) -> Result { + /// `m5`: the firmware runs the M5 protocol, whose statistics records are decoded too. + pub(crate) fn new(graph: super::fw::variant::InitGraph, m5: bool) -> Result { Ok(Self { quarantined: [[false; 2]; 2], reports: super::reports::Reports::new()?, - telemetry: super::telemetry::Telemetry::default(), + telemetry: if m5 { + super::telemetry::Telemetry::m5() + } else { + super::telemetry::Telemetry::default() + }, max_age_ns: super::stats::max_age_ns(graph), suspended: false, discard_statistics: [false; 2], diff --git a/drivers/gpu/drm/asahi/g17/mod.rs b/drivers/gpu/drm/asahi/g17/mod.rs index 88610416beab1..3bda0843956d4 100644 --- a/drivers/gpu/drm/asahi/g17/mod.rs +++ b/drivers/gpu/drm/asahi/g17/mod.rs @@ -1818,7 +1818,10 @@ impl Gpu { )?; let stats = Arc::new( crate::stats::StatsSnapshot { - g17_reader: Some(stats::Reader(shared.clone())), + g17_reader: Some(stats::Reader( + shared.clone(), + cfg.m5.map(|board| board.perf_states), + )), ..Default::default() }, GFP_KERNEL, @@ -2036,7 +2039,7 @@ impl Gpu { shared.feed.clone(), )?, seen_submissions: 0, - rings: channel::Rings::new(shared.variant.graph)?, + rings: channel::Rings::new(shared.variant.graph, shared.variant.m5_commands)?, recovery: recovery::State::new(), effort: power::Effort::new(), hold: power::Hold::new(), @@ -2240,7 +2243,7 @@ impl Gpu { shared.feed.clone(), )?, seen_submissions: 0, - rings: channel::Rings::new(shared.variant.graph)?, + rings: channel::Rings::new(shared.variant.graph, shared.variant.m5_commands)?, recovery: recovery::State::new(), effort: power::Effort::new(), hold: power::Hold::new(), diff --git a/drivers/gpu/drm/asahi/g17/stats.rs b/drivers/gpu/drm/asahi/g17/stats.rs index 887d966a04de3..d21519621fd02 100644 --- a/drivers/gpu/drm/asahi/g17/stats.rs +++ b/drivers/gpu/drm/asahi/g17/stats.rs @@ -7,13 +7,30 @@ use super::{fw::variant::InitGraph, telemetry::Readout, Shared}; use core::sync::atomic::Ordering; use kernel::sync::Arc; -pub(crate) struct Reader(pub(super) Arc); +/// The device's shared state, and the M5 board's performance states, whose frequency a +/// reported state index runs at. +pub(crate) struct Reader( + pub(super) Arc, + pub(super) Option<&'static [super::fw::m5::initdata::M5PerfState; 14]>, +); impl Reader { pub(crate) fn read(&self) -> Readout { + let mut view = self.read_firmware(); + if view.pstate_status == super::telemetry::VALID { + view.achieved_frequency_hz = self + .1 + .and_then(|states| states.get(view.pstate as usize)) + .map_or(0, |state| u64::from(state.0) * 1_000_000); + } + view + } + + fn read_firmware(&self) -> Readout { + let m5 = self.0.variant.m5_commands; let mut state = self.0.state.lock(); let Some(firmware) = state.as_deref_mut() else { - return qualify(self.0.variant.graph, Readout::default()); + return qualify(self.0.variant.graph, m5, Readout::default()); }; if self.0.crashed.load(Ordering::Acquire) || firmware.recovery.pending() { firmware.rings.invalidate_statistics(); @@ -34,6 +51,7 @@ impl Reader { } qualify( self.0.variant.graph, + m5, firmware.rings.read_statistics(super::now_ns()), ) } @@ -55,13 +73,15 @@ pub(super) const fn max_age_ns(graph: InitGraph) -> u64 { } } -/// Whether the firmware emits the logical pstate record (tag 0x10). V25G83 -/// and V26A428 accept the tag but never emit it, and no other record carries -/// the current pstate, so it is unsupported by these firmwares. A firmware -/// that is shown to emit it gets `true` here and reports valid or stale. -pub(super) const fn reports_pstate(graph: InitGraph) -> bool { +/// Whether the firmware reports its performance state. V25G83 and V26A428 +/// accept the logical pstate record (tag 0x10) but never emit it, and no other +/// record of theirs carries the current pstate, so it is unsupported by these +/// firmwares. M5 firmware (`m5`) reports it in its own records +/// ([`super::telemetry`]). A firmware that is shown to report it gets `true` +/// here and reports valid or stale. +pub(super) const fn reports_pstate(graph: InitGraph, m5: bool) -> bool { match graph { - InitGraph::V25G83 | InitGraph::G17P => false, + InitGraph::V25G83 | InitGraph::G17P => m5, } } @@ -69,9 +89,10 @@ pub(super) const fn reports_pstate(graph: InitGraph) -> bool { /// and the debugfs diagnostic still keep any tag 0x10 that does arrive. pub(super) fn qualify( graph: InitGraph, + m5: bool, mut view: super::telemetry::Readout, ) -> super::telemetry::Readout { - if !reports_pstate(graph) { + if !reports_pstate(graph, m5) { view.pstate_status = super::telemetry::UNSUPPORTED; view.pstate = 0; view.pstate_host_ns = 0; diff --git a/drivers/gpu/drm/asahi/g17/telemetry.rs b/drivers/gpu/drm/asahi/g17/telemetry.rs index e57f91b5bea02..27d8bef8c5383 100644 --- a/drivers/gpu/drm/asahi/g17/telemetry.rs +++ b/drivers/gpu/drm/asahi/g17/telemetry.rs @@ -1,9 +1,13 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT //! Qualified V25G83/V26A428 statistics, decoded from complete copied records. -//! No record measures busy time, power, temperature or achieved frequency. -//! A/B table selection is absent, so neither nominal candidate is exported as -//! a requested or achieved frequency. All state is protected by the device mutex. +//! No record of these firmwares measures busy time, power, temperature or achieved +//! frequency. A/B table selection is absent, so neither nominal candidate is exported +//! as a requested or achieved frequency. +//! +//! M5 firmware reports the performance state the GPU runs at (tags 0x19 and 0x1b), and +//! its utilisation records carry their own timestamps, from which busy time is +//! integrated. All state is protected by the device mutex. pub(super) const UNAVAILABLE: u32 = 0; pub(super) const VALID: u32 = 1; @@ -26,13 +30,31 @@ pub(crate) struct Readout { pub(crate) pstate_host_ns: u64, pub(crate) util_samples: u64, pub(crate) pstate_timestamp: u64, + /// Utilisation-weighted busy time since the device was created, while `busy_status` is + /// [`VALID`] (M5 only). + pub(crate) busy_ns: u64, + /// Frequency of `pstate` while `pstate_status` is [`VALID`], from the board's performance + /// state table ([`super::stats::Reader`]); zero where the table is unknown. + pub(crate) achieved_frequency_hz: u64, + pub(crate) busy_status: u32, + pub(crate) _pad: u32, } const _: () = { assert!(core::mem::offset_of!(Readout, util_host_ns) == 24); - assert!(core::mem::size_of::() == 56); + assert!(core::mem::offset_of!(Readout, busy_ns) == 56); + assert!(core::mem::size_of::() == 80); }; +/// Interval of the M5 utilisation record (tag 0x13), in timer ticks: 8 ms at 24 MHz. +const M5_UTIL_PERIOD_TICKS: u64 = 192_000; +/// A utilisation record more than two intervals after the previous one follows a gap +/// the firmware did not report (gated or lost records): it only sets the baseline. +const M5_UTIL_MAX_WINDOW_TICKS: u64 = 2 * M5_UTIL_PERIOD_TICKS; +/// Highest M5 performance state: the device tree's operating points 1 to 13 (0 is off). +const M5_MAX_PSTATE: u32 = 13; +const TIMER_HZ: u64 = 24_000_000; + #[derive(Default)] pub(super) struct Telemetry { view: Readout, @@ -49,9 +71,25 @@ pub(super) struct Telemetry { /// gates them a few milliseconds after the last work completes and emits no /// utilisation record while gated, so a gated GPU is idle, not unknown. gated: bool, + /// The M5 firmware's records are decoded ([`Self::m5`]). + m5: bool, + /// Timestamp of the latest M5 utilisation record in this epoch: the start of the window + /// the next one closes. + busy_ts: Option, + busy_ns: u64, + /// End of the latest M5 performance state interval (tag 0x19), in timer ticks. + pstate_interval_end: Option, } impl Telemetry { + /// Decoder for M5 firmware: its performance state and busy time records. + pub(super) fn m5() -> Self { + Self { + m5: true, + ..Self::default() + } + } + /// End an epoch. Preserve the reason that formerly valid data is stale, /// but never carry an average or a timestamp baseline into a new epoch. pub(super) fn invalidate(&mut self) { @@ -67,6 +105,9 @@ impl Telemetry { self.delivery_ns = None; self.delivery_intervals = [None; 2]; self.gated = false; + // Busy time keeps counting across epochs; only the window start is lost. + self.busy_ts = None; + self.pstate_interval_end = None; } /// The cores were observed power-gated. End the averaging epoch exactly as @@ -129,6 +170,12 @@ impl Telemetry { for (sum, value) in self.sums.iter_mut().zip(values) { *sum += u64::from(value); } + if self.m5 { + self.integrate_busy( + u64::from(words[1]) | (u64::from(words[2]) << 32), + words[6], + ); + } self.view.util_samples += 1; self.view.util_host_ns = host_ns; self.view.util_status = VALID; @@ -150,10 +197,45 @@ impl Telemetry { self.view.pstate_status = if words[6] <= 10 { VALID } else { UNAVAILABLE }; self.view.pstate = words[6]; } + // M5: the interval [words 3-4, words 1-2] that just ended, run at the state in + // word 8. Its end timestamps the state report that follows it. + 0x19 if self.m5 => { + self.pstate_interval_end = Some(u64::from(words[1]) | (u64::from(words[2]) << 32)); + } + // M5: the performance state now in effect, an index into the device tree's + // operating points, in words 1 and 3. A report whose two words differ is not + // used. + 0x1b if self.m5 => { + let pstate = words[1]; + if pstate != words[3] || pstate == 0 || pstate > M5_MAX_PSTATE { + self.view.pstate_status = UNAVAILABLE; + return; + } + self.view.pstate = pstate; + self.view.pstate_status = VALID; + self.view.pstate_host_ns = host_ns; + self.view.pstate_timestamp = self.pstate_interval_end.unwrap_or(0); + } _ => {} } } + /// Credits the window an M5 utilisation record closes, at its device utilisation. The + /// window starts at the previous record of the epoch; a record after a longer gap than + /// [`M5_UTIL_MAX_WINDOW_TICKS`] (the GPU was gated or records were lost) or one whose + /// timestamp went back only sets the baseline, so an idle gap is never credited. + fn integrate_busy(&mut self, timestamp: u64, device: u32) { + if let Some(last) = self.busy_ts { + let window = timestamp.wrapping_sub(last); + if timestamp > last && window <= M5_UTIL_MAX_WINDOW_TICKS { + // At most 384000 ticks x 100: no overflow. + let ns = window * u64::from(device) * 1_000_000_000 / (100 * TIMER_HZ); + self.busy_ns = self.busy_ns.saturating_add(ns); + } + } + self.busy_ts = Some(timestamp); + } + pub(super) fn read(&self, now_ns: u64, max_age_ns: u64) -> Readout { let mut view = self.view; let limit = max_age_ns.saturating_add(self.delivery_slack()); @@ -173,6 +255,12 @@ impl Telemetry { // A pstate report describes a state transition, with no qualified // cadence. Its validity ends on replacement or epoch invalidation, // rather than an invented periodic-report deadline. + if self.m5 { + view.busy_status = VALID; + view.busy_ns = self.busy_ns; + } else { + view.busy_status = UNSUPPORTED; + } view } } @@ -201,7 +289,7 @@ mod tests { fn spec_u1_u2_u3_both_firmware_policies() { for age in [AGE, 2 * AGE] { let mut stats = Telemetry::default(); - assert_eq!(stats.read(0, age), Readout::default()); + assert_eq!(stats.read(0, age), NOTHING); stats.record(&U1, 100, age); let view = stats.read(100, age); assert_eq!((view.tiler, view.renderer, view.device), (25, 75, 50)); @@ -358,6 +446,126 @@ mod tests { words[0] = tag; stats.record(&words, 1, AGE); } - assert_eq!(stats.read(1, AGE), Readout::default()); + assert_eq!(stats.read(1, AGE), NOTHING); + } + + /// M5 firmware records: the utilisation of three 8 ms windows of a full-GPU compute + /// job, a performance state interval and the state report after it. + const M5_U: [[u32; 18]; 3] = [ + [ + 0x13, 0x433a6047, 0x5c, 0, 0, 0x64, 0x64, 0x36, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, + ], + [ + 0x13, 0x433d4e4f, 0x5c, 0, 0, 0x64, 0x64, 0x36, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, + ], + [ + 0x13, 0x43403c9b, 0x5c, 0, 0, 0x64, 0x64, 0x36, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, + ], + ]; + const M5_INTERVAL: [u32; 18] = [ + 0x19, 0x3c042f8f, 0x59, 0x3c040ecf, 0x59, 1, 0, 9, 0xd, 0xd, 9, 0x514, 0, 6, 0, 0, 0, 0, + ]; + const M5_STATE: [u32; 18] = [ + 0x1b, 0xd, 0x7f, 0xd, 0x7f, 0, 1, 9, 0, 0, 9, 0x514, 0, 6, 0, 0, 0, 0, + ]; + const NOTHING: Readout = Readout { + util_status: UNAVAILABLE, + pstate_status: UNAVAILABLE, + tiler: 0, + renderer: 0, + device: 0, + pstate: 0, + util_host_ns: 0, + pstate_host_ns: 0, + util_samples: 0, + pstate_timestamp: 0, + busy_ns: 0, + achieved_frequency_hz: 0, + busy_status: UNSUPPORTED, + _pad: 0, + }; + + fn ticks_ns(ticks: u64) -> u64 { + ticks * 1_000_000_000 / 24_000_000 + } + + #[test] + fn m5_state_reports_give_the_pstate() { + let mut stats = Telemetry::m5(); + stats.record(&M5_INTERVAL, 100, AGE); + assert_eq!(stats.read(100, AGE).pstate_status, UNAVAILABLE); + stats.record(&M5_STATE, 101, AGE); + let view = stats.read(101, AGE); + assert_eq!((view.pstate_status, view.pstate), (VALID, 13)); + assert_eq!(view.pstate_timestamp, 0x59_3c04_2f8f); + assert_eq!(view.pstate_host_ns, 101); + let mut low = M5_STATE; + (low[1], low[3]) = (1, 1); + stats.record(&low, 102, AGE); + assert_eq!(stats.read(102, AGE).pstate, 1); + // Two different states in one report, off, or beyond the table: not a state. + for (a, b) in [(1, 13), (0, 0), (14, 14)] { + let mut words = M5_STATE; + (words[1], words[3]) = (a, b); + stats.record(&words, 103, AGE); + assert_eq!(stats.read(103, AGE).pstate_status, UNAVAILABLE); + } + // Gating ends the epoch: the state is no longer current. + stats.record(&M5_STATE, 104, AGE); + stats.gate(); + assert_eq!(stats.read(105, AGE).pstate_status, STALE); + } + + #[test] + fn m5_busy_time_integrates_utilisation_windows() { + let mut stats = Telemetry::m5(); + assert_eq!( + (stats.read(0, AGE).busy_status, stats.read(0, AGE).busy_ns), + (VALID, 0) + ); + // The first record only sets the baseline; the next two close one window each. + for (n, words) in M5_U.iter().enumerate() { + stats.record(words, 100 + n as u64, AGE); + } + // Each window is rounded down to whole nanoseconds on its own. + let windows = ticks_ns(u64::from(M5_U[1][1] - M5_U[0][1])) + + ticks_ns(u64::from(M5_U[2][1] - M5_U[1][1])); + assert_eq!(stats.read(102, AGE).busy_ns, windows); + assert!((15_900_000..16_100_000).contains(&stats.read(102, AGE).busy_ns)); + // A window is credited at its device utilisation. + let mut half = M5_U[2]; + half[1] += 192_000; + half[6] = 50; + stats.record(&half, 103, AGE); + assert_eq!(stats.read(103, AGE).busy_ns, windows + 4_000_000); + // A gate, a long gap or a timestamp going back only sets a new baseline. + let before = stats.read(103, AGE).busy_ns; + stats.gate(); + let mut after = half; + after[1] += 192_000; + stats.record(&after, 104, AGE); + assert_eq!(stats.read(104, AGE).busy_ns, before); + after[1] += 3 * 192_000; + stats.record(&after, 105, AGE); + assert_eq!(stats.read(105, AGE).busy_ns, before); + after[1] -= 1000; + stats.record(&after, 106, AGE); + assert_eq!(stats.read(106, AGE).busy_ns, before); + // An epoch boundary keeps the total. + stats.invalidate(); + assert_eq!(stats.read(107, AGE).busy_ns, before); + } + + #[test] + fn other_firmwares_ignore_the_m5_records() { + let mut stats = Telemetry::default(); + for words in [M5_INTERVAL, M5_STATE, M5_U[0], M5_U[1]] { + stats.record(&words, 1, AGE); + } + let view = stats.read(1, AGE); + assert_eq!( + (view.pstate_status, view.busy_status, view.busy_ns), + (UNAVAILABLE, UNSUPPORTED, 0) + ); } } diff --git a/drivers/gpu/drm/asahi/sysfs.c b/drivers/gpu/drm/asahi/sysfs.c index 0b2bca6571129..d32ee864e104c 100644 --- a/drivers/gpu/drm/asahi/sysfs.c +++ b/drivers/gpu/drm/asahi/sysfs.c @@ -99,11 +99,20 @@ static ssize_t agx_qualified_stats_show(struct asahi_stats_attribute *stats, cha n += sysfs_emit_at(buf, n, "pstate %u\n", snap.pstate); else n += sysfs_emit_at(buf, n, "pstate unsupported\n"); - /* No A/B selector, packet duration, measured power or temperature. */ - n += sysfs_emit_at(buf, n, "requested_frequency_hz unsupported\n" - "achieved_frequency_hz unsupported\n" - "busy_ns unsupported\n" - "power_mw unsupported\n" + /* No A/B selector, measured power or temperature. */ + n += sysfs_emit_at(buf, n, "requested_frequency_hz unsupported\n"); + /* M5 reports the state the GPU runs at: its frequency is the achieved one. */ + if (snap.pstate_status == 1 && snap.achieved_frequency_hz) + n += sysfs_emit_at(buf, n, "achieved_frequency_hz %llu\n", + (unsigned long long)snap.achieved_frequency_hz); + else + n += sysfs_emit_at(buf, n, "achieved_frequency_hz unsupported\n"); + if (snap.busy_status == 1) + n += sysfs_emit_at(buf, n, "busy_ns %llu\n", + (unsigned long long)snap.busy_ns); + else + n += sysfs_emit_at(buf, n, "busy_ns unsupported\n"); + n += sysfs_emit_at(buf, n, "power_mw unsupported\n" "temperature_raw unsupported\n"); n += sysfs_emit_at(buf, n, "utilisation_source firmware-statistics-tag-0x13\n" "utilisation_samples %llu\n" @@ -225,7 +234,8 @@ int asahi_sysfs_register(struct device *dev, const void *snapshot, return -ENODEV; BUILD_BUG_ON(offsetof(struct asahi_qualified_stats, util_host_ns) != 24); - BUILD_BUG_ON(sizeof(struct asahi_qualified_stats) != 56); + BUILD_BUG_ON(offsetof(struct asahi_qualified_stats, busy_ns) != 56); + BUILD_BUG_ON(sizeof(struct asahi_qualified_stats) != 80); BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, refused) != 24); BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, client_limit) != 40); BUILD_BUG_ON(sizeof(struct asahi_gpu_memory) != 48); diff --git a/drivers/gpu/drm/asahi/sysfs.h b/drivers/gpu/drm/asahi/sysfs.h index 12bcce7d4e7fb..3433ebbcebfd3 100644 --- a/drivers/gpu/drm/asahi/sysfs.h +++ b/drivers/gpu/drm/asahi/sysfs.h @@ -22,6 +22,10 @@ struct asahi_qualified_stats { u64 pstate_host_ns; u64 util_samples; u64 pstate_timestamp; + u64 busy_ns; + u64 achieved_frequency_hz; + u32 busy_status; + u32 pad; }; typedef void (*asahi_stats_read_fn)(const void *snapshot, struct asahi_qualified_stats *out); diff --git a/tools/asahi/g17-telemetry-integration-tests.py b/tools/asahi/g17-telemetry-integration-tests.py index a9a0f7c4446f0..8dde658062689 100644 --- a/tools/asahi/g17-telemetry-integration-tests.py +++ b/tools/asahi/g17-telemetry-integration-tests.py @@ -146,7 +146,7 @@ def item(source, signature): impl State { fn lock(&self)->std::sync::MutexGuard<'_,Option>> { self.0.lock().unwrap() } } - struct Variant { graph:fw::variant::InitGraph } + struct Variant { graph:fw::variant::InitGraph, m5_commands:bool } struct Shared { state:State, dev:crate::device::Device, variant:Variant, crashed:std::sync::atomic::AtomicBool } mod reports; ''' @@ -154,10 +154,10 @@ def item(source, signature): fn reader(graph:fw::variant::InitGraph)->stats::Reader { stats::Reader(std::sync::Arc::new(Shared { state:State(std::sync::Mutex::new(Some(Box::new(Firmware { - rings:channel::Rings::new(graph).unwrap(), + rings:channel::Rings::new(graph,false).unwrap(), init:initdata::InitData::new(),powered:true,epoch:0,recovery:Recovery(false), - })))),dev:crate::device::Device,variant:Variant { graph },crashed:false.into(), - })) + })))),dev:crate::device::Device,variant:Variant { graph, m5_commands:false },crashed:false.into(), + }),None) } #[test] @@ -255,11 +255,11 @@ def item(source, signature): let dev=crate::device::Device; for graph in [InitGraph::V25G83,InitGraph::G17P] { let init=initdata::InitData::new(); - let mut rings=Rings::new(graph).unwrap(); + let mut rings=Rings::new(graph,false).unwrap(); for words in [HIST_IDLE,HIST_ACTIVE,HIST_OTHER,TRANSITION] { init.push(Secondary,words); rings.drain_statistics(&dev,&init,Secondary); - assert_eq!(rings.read_statistics(1),telemetry::Readout::default()); + assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::UNSUPPORTED,..Default::default() }); assert!(report(&rings).contains(&format!("words={:08x?}",words))); } } @@ -271,7 +271,7 @@ def item(source, signature): let dev=crate::device::Device; for start in [0,5,7] { let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); for &role in &[Primary,Secondary] { for cursor in &init.cursors[role as usize] { cursor.store(start,Ordering::Relaxed); } for words in [HIST_OTHER,HIST_ACTIVE,TRANSITION] { init.push(role,words); } @@ -286,7 +286,7 @@ def item(source, signature): let line=text.lines().find(|line|line.starts_with(&format!("role={} tag={:#x} ",role,tag))).unwrap(); assert!(line.contains("count=2 host_ns=1 admitted=1 last_admitted=true"),"{}",line); } } - assert_eq!(rings.read_statistics(1),telemetry::Readout::default()); + assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::UNSUPPORTED,..Default::default() }); } } @@ -295,7 +295,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); let util=[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]; let pstate=[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]; rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],true,1); @@ -326,21 +326,22 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; for graph in [InitGraph::V25G83,InitGraph::G17P] { - assert!(!stats::reports_pstate(graph)); - let empty=stats::qualify(graph,telemetry::Readout::default()); + assert!(!stats::reports_pstate(graph,false)); + assert!(stats::reports_pstate(graph,true)); + let empty=stats::qualify(graph,false,telemetry::Readout::default()); assert_eq!((empty.util_status,empty.pstate_status),(telemetry::UNAVAILABLE,telemetry::UNSUPPORTED)); let init=initdata::InitData::new(); - let mut rings=Rings::new(graph).unwrap(); + let mut rings=Rings::new(graph,false).unwrap(); init.push(Primary,[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]); // Even a decodable tag 0x10 does not override the firmware's match data. init.push(Primary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); rings.drain_statistics(&dev,&init,Primary); assert_eq!(rings.read_statistics(1).pstate_status,telemetry::VALID); - let view=stats::qualify(graph,rings.read_statistics(1)); + let view=stats::qualify(graph,false,rings.read_statistics(1)); assert_eq!((view.util_status,view.device,view.pstate_status,view.pstate,view.pstate_timestamp), (telemetry::VALID,97,telemetry::UNSUPPORTED,0,0)); rings.invalidate_statistics(); - let view=stats::qualify(graph,rings.read_statistics(1)); + let view=stats::qualify(graph,false,rings.read_statistics(1)); assert_eq!((view.util_status,view.pstate_status),(telemetry::STALE,telemetry::UNSUPPORTED)); } } @@ -350,7 +351,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::V25G83).unwrap(); + let mut rings=Rings::new(InitGraph::V25G83,false).unwrap(); let util=[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]; // Three 8 ms records per producer update, one update every 24 ms. for n in 0..3u64 { @@ -370,7 +371,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],false,0); init.push(Secondary,HIST_IDLE); rings.drain_statistics(&dev,&init,Secondary); @@ -390,13 +391,13 @@ def item(source, signature): let graph=InitGraph::G17P; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(graph).unwrap(); + let mut rings=Rings::new(graph,false).unwrap(); init.push(Secondary,HIST_IDLE); rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],true,1); init.push(Secondary,HIST_ACTIVE); let shared=std::sync::Arc::new(Shared { state:State(std::sync::Mutex::new(Some(Box::new(Firmware { rings,init:initdata::InitData::new(),powered:true,epoch:0,recovery:Recovery(false), - })))),dev,variant:Variant { graph },crashed:false.into() }); + })))),dev,variant:Variant { graph, m5_commands:false },crashed:false.into() }); let view=reports::View(shared.clone()); struct View<'a>(&'a reports::View); impl std::fmt::Display for View<'_> { @@ -414,7 +415,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); let sample=|ta,render,device|[0x13,0,0,ta,render,0xdeadbeef,device,0,0,0,0,0,0,0,0,0,0,0]; init.push(Primary,sample(25,75,50)); init.push(Secondary,sample(35,85,65)); @@ -438,7 +439,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::V25G83).unwrap(); + let mut rings=Rings::new(InitGraph::V25G83,false).unwrap(); let words=[0x13,0,0,25,75,0,50,0,0,0,0,0,0,0,0,0,0,0]; init.push(Primary,words); rings.drain_statistics(&dev,&init,Primary); @@ -467,7 +468,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); init.push(Primary,[0x10,0x1000,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0]); init.push(Secondary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); rings.drain_statistics(&dev,&init,Primary); @@ -687,7 +688,9 @@ def main(): header = (RUST_HEADER.replace('M5_STATISTICS_TYPES', m5_types).replace('M5_ASSERTS_GPU_POWER', m5_power) .replace('INITDATA_FAMILY', family)) - reader_rs = 'pub(crate) struct Reader(pub(super) Arc);\n' + item(stats_rs, 'impl Reader') + reader_rs = ('pub(crate) struct Reader(pub(super) Arc, ' + 'pub(super) Option<&\'static [(u32, [u32; 8], [u32; 8]); 14]>);\n' + + item(stats_rs, 'impl Reader').replace('super::fw::m5::initdata::M5PerfState', '(u32, [u32; 8], [u32; 8])')) (work / 'rings.rs').write_text(header + 'mod stats { use super::{fw::variant::InitGraph, telemetry::Readout, Shared};\n' + 'use std::sync::{Arc,atomic::Ordering};\n' + reader_rs + '\n' + policy + '\n}\nmod telemetry;\nmod channel;\n' + RUST_TESTS) From ff734bcd45dafb8773d73d70a8cbde9294ff4a53 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Tue, 10 Nov 2026 21:45:34 -0500 Subject: [PATCH 4/8] rust: drm: file: print the engines' frequency in fdinfo Documentation/gpu/drm-usage-stats.rst defines drm-maxfreq-, the highest frequency of an engine, and drivers that know it also print the frequency the engine runs at as drm-curfreq-. A Rust driver could only print the busy time. DriverFile::engine_frequency returns the current frequency, if the driver knows it, and the highest, in Hz; show_fdinfo prints them for every engine engine_stats emits. All engines get the same pair, which suits a device whose engines share one clock. The default, None, prints nothing, so drivers that do not implement it are unchanged. Built for arm64; the next commit uses it. Developed with Claude Code. Signed-off-by: Erik Bourget --- rust/kernel/drm/file.rs | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/rust/kernel/drm/file.rs b/rust/kernel/drm/file.rs index 81f0a90e4a517..055440f5c52ea 100644 --- a/rust/kernel/drm/file.rs +++ b/rust/kernel/drm/file.rs @@ -49,6 +49,16 @@ pub trait DriverFile { /// Engine names must be stable, unique and contain no whitespace or colons. /// The default emits no engine keys. See `Documentation/gpu/drm-usage-stats.rst`. fn engine_stats(&self, _emit: impl FnMut(&crate::str::CStr, u64)) {} + + /// The frequency the engines of [`Self::engine_stats`] run at now, if known, and their + /// highest, in Hz, for this client's fdinfo (`drm-curfreq-` and + /// `drm-maxfreq-` beside each `drm-engine-`). The default, `None`, + /// prints neither. Every engine gets the same pair: this suits a device whose engines + /// share one clock, and a driver with independently clocked engines needs a per-engine + /// form instead. + fn engine_frequency(&self) -> Option<(Option, u64)> { + None + } } /// An open DRM File. @@ -141,6 +151,7 @@ impl File { // `driver_priv` was set by `open_callback()`. The seq_file read holds a reference to the // file, so `postclose_callback()` cannot run concurrently. let file = unsafe { File::::from_raw(raw_file) }; + let frequency = file.inner().engine_frequency(); file.inner().engine_stats(|name, ns| { // SAFETY: The DRM core owns printer throughout this callback; the format and // name are NUL-terminated, and the vararg types match %s and %llu. @@ -152,6 +163,29 @@ impl File { ns as core::ffi::c_ulonglong, ) }; + let Some((current, max)) = frequency else { + return; + }; + if let Some(current) = current { + // SAFETY: As above. + unsafe { + bindings::drm_printf( + printer, + c"drm-curfreq-%s:\t%llu Hz\n".as_char_ptr(), + name.as_char_ptr(), + current as core::ffi::c_ulonglong, + ) + }; + } + // SAFETY: As above. + unsafe { + bindings::drm_printf( + printer, + c"drm-maxfreq-%s:\t%llu Hz\n".as_char_ptr(), + name.as_char_ptr(), + max as core::ffi::c_ulonglong, + ) + }; }); let Some(stats) = file.inner().memory_stats() else { return; From 17eca40c50cd44d127152c9d2bbc544700d22e6b Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Tue, 10 Nov 2026 21:45:34 -0500 Subject: [PATCH 5/8] drm/asahi: g17: M5: report the GPU frequency in fdinfo Tools that read a client's fdinfo (gputop, nvtop) see its engine busy time but not the clock the GPU runs at, which agx_stats now reports on the M5. Every engine of a client's fdinfo gets drm-curfreq-, the achieved frequency of the performance state the firmware reports, while it reports one (it does not while the GPU is gated), and drm-maxfreq-, the highest frequency in the board's table (1620 MHz on the J716c). One clock drives the vertex, fragment and compute engines. Each read drains the statistics rings under the device mutex, as an agx_stats read does. GPUs without a board performance table (all but the M5) print neither key. Documentation/gpu/asahi-usage-stats.rst said that drm-maxfreq-* was left out on purpose, because the engine times must not be scaled by a frequency. That still holds and drm-cycles-* stays absent: the document now says that the two keys show the device's clock at the time of the read, and that an fdinfo read on the M5 takes the device mutex for them. On the J716c (M5 Max MacBook Pro), the fdinfo of a client running a 1.4 s job that fills the GPU reads, for each of vertex, fragment and compute, drm-curfreq-: 1620000000 Hz and drm-maxfreq-: 1620000000 Hz; that of an idle client with the GPU gated has the drm-maxfreq lines only. Developed with Claude Code. Signed-off-by: Erik Bourget --- Documentation/gpu/asahi-usage-stats.rst | 28 +++++++++++++++++++------ drivers/gpu/drm/asahi/file.rs | 12 ++++++++++- drivers/gpu/drm/asahi/g17/stats.rs | 12 +++++++++++ drivers/gpu/drm/asahi/stats.rs | 6 ++++++ tools/asahi/legacy-frequency-tests.rs | 15 +++++++++++++ 5 files changed, 66 insertions(+), 7 deletions(-) diff --git a/Documentation/gpu/asahi-usage-stats.rst b/Documentation/gpu/asahi-usage-stats.rst index 2eb2166984465..8bd90b2733d9c 100644 --- a/Documentation/gpu/asahi-usage-stats.rst +++ b/Documentation/gpu/asahi-usage-stats.rst @@ -51,11 +51,25 @@ successful stages that retired before a later failure remain counted. An unexecuted G17 replay contributes only when it eventually retires successfully. There is no submit-to-completion wall-time fallback in the current backends. -``drm-cycles-*`` and ``drm-maxfreq-*`` are intentionally absent. The timestamp -clock is independent of the variable GPU execution clock. Treating command timer -ticks as GPU busy cycles, or multiplying time by a sampled/maximum GPU frequency, -would give incorrect frequency-weighted utilization. Per-client execution-cycle -counters would require a separate hardware measurement source. +``drm-cycles-*`` is intentionally absent. The timestamp clock is independent of +the variable GPU execution clock. Treating command timer ticks as GPU busy +cycles, or multiplying time by a sampled/maximum GPU frequency, would give +incorrect frequency-weighted utilization. Per-client execution-cycle counters +would require a separate hardware measurement source. + +On the M5 each engine also has the frequency keys:: + + drm-curfreq-compute: 1620000000 Hz + drm-maxfreq-compute: 1620000000 Hz + +``drm-curfreq-`` is ``achieved_frequency_hz`` of ``agx_stats`` (see +below): the frequency of the performance state the firmware reports. It is +absent while the firmware reports none, as when the GPU is gated. +``drm-maxfreq-`` is the highest frequency in the board's table. One clock +drives the three engines, so both values are the device's, the same for every +engine and every client. They show the clock at the time of the read and are +not a weight for the engine times: without ``drm-cycles-*`` there is nothing to +scale. Other GPUs print neither key. Accounting cost --------------- @@ -66,7 +80,9 @@ per valid stage, with no new locks or allocations. The native backends reuse existing timestamp reads. Classic completion callbacks read two internal cells per stage; commands without application timestamps also execute two existing timestamp microsequence operations per stage. Reading fdinfo loads three counters -and does not traverse queues, wait for fences or access hardware. +and does not traverse queues, wait for fences or access hardware. On the M5 a +read also takes the device mutex once and drains the statistics rings, as a read +of ``agx_stats`` does, for the frequency keys. G16S and G17P device utilisation ------------------------------ diff --git a/drivers/gpu/drm/asahi/file.rs b/drivers/gpu/drm/asahi/file.rs index 9fe3de9109731..be5e70ffaed37 100644 --- a/drivers/gpu/drm/asahi/file.rs +++ b/drivers/gpu/drm/asahi/file.rs @@ -378,6 +378,8 @@ pub(crate) struct File { /// Largest size this file has asked GET_PARAMS for the global parameters with /// ([`gpu::Gpu::admit_vm`]). params_size: core::sync::atomic::AtomicU64, + /// The device's statistics, for the GPU frequency in fdinfo. + stats: Option>, /// The queues were drained once for an exiting task. drained: AtomicBool, } @@ -401,9 +403,10 @@ impl drm::file::DriverFile for File { let user_range = gpu.user_range(); let mem = memacct::Client::new()?; let usage = Arc::new(crate::usage::Client::new(), GFP_KERNEL)?; + let stats = gpu.stats_snapshot(); mod_dev_dbg!(device, "[File {}]: DRM device opened\n", id); - KBox::pin_init(File::new(id, user_range, mem, usage), GFP_KERNEL) + KBox::pin_init(File::new(id, user_range, mem, usage, stats), GFP_KERNEL) }) } @@ -428,6 +431,11 @@ impl drm::file::DriverFile for File { } } + /// One clock drives every engine. + fn engine_frequency(&self) -> Option<(Option, u64)> { + self.stats.as_ref()?.engine_frequency() + } + fn as_raw(&self) -> *mut bindings::drm_file { todo!() } @@ -442,6 +450,7 @@ impl File { user_range: Range, mem: Arc, usage: Arc, + stats: Option>, ) -> impl PinInit { // SAFETY: The initializer supplies aligned, pinned storage for Self. // XArray::new is infallible; each pinned array is initialized in place, @@ -473,6 +482,7 @@ impl File { addr_of_mut!((*slot).mem).write(mem); addr_of_mut!((*slot).usage).write(usage); addr_of_mut!((*slot).params_size).write(core::sync::atomic::AtomicU64::new(0)); + addr_of_mut!((*slot).stats).write(stats); addr_of_mut!((*slot).drained).write(AtomicBool::new(false)); Ok(()) }) diff --git a/drivers/gpu/drm/asahi/g17/stats.rs b/drivers/gpu/drm/asahi/g17/stats.rs index d21519621fd02..40688ccab0c8e 100644 --- a/drivers/gpu/drm/asahi/g17/stats.rs +++ b/drivers/gpu/drm/asahi/g17/stats.rs @@ -26,6 +26,18 @@ impl Reader { view } + /// The frequency of the state the GPU runs at, while the firmware reports one, and the + /// highest frequency of the board's states, in Hz; `None` without the board's performance + /// states. + pub(crate) fn frequency(&self) -> Option<(Option, u64)> { + let max = u64::from(self.1?.iter().map(|state| state.0).max()?) * 1_000_000; + let view = self.read(); + let current = (view.pstate_status == super::telemetry::VALID + && view.achieved_frequency_hz != 0) + .then_some(view.achieved_frequency_hz); + Some((current, max)) + } + fn read_firmware(&self) -> Readout { let m5 = self.0.variant.m5_commands; let mut state = self.0.state.lock(); diff --git a/drivers/gpu/drm/asahi/stats.rs b/drivers/gpu/drm/asahi/stats.rs index b8636d4c5eb70..d8f33c932afe5 100644 --- a/drivers/gpu/drm/asahi/stats.rs +++ b/drivers/gpu/drm/asahi/stats.rs @@ -84,6 +84,12 @@ impl StatsSnapshot { self.frequencies_hz.get(pstate as usize).copied() } + /// The frequency the GPU runs at now, if the firmware reports it, and its highest, in + /// Hz, for fdinfo (`drm-curfreq-*`, `drm-maxfreq-*`): M5 only. + pub(crate) fn engine_frequency(&self) -> Option<(Option, u64)> { + self.g17_reader.as_ref()?.frequency() + } + /// Bump the completed-submission counter (called from the queue /// completion path in `JobFence::command_complete`, NOT from the stats /// channel). diff --git a/tools/asahi/legacy-frequency-tests.rs b/tools/asahi/legacy-frequency-tests.rs index 18f1e4d33632e..84baee6ad1e16 100644 --- a/tools/asahi/legacy-frequency-tests.rs +++ b/tools/asahi/legacy-frequency-tests.rs @@ -9,6 +9,11 @@ mod prelude { mod g17 { pub mod stats { pub struct Reader; + impl Reader { + pub fn frequency(&self) -> Option<(Option, u64)> { + Some((None, 1_620_000_000)) + } + } } } #[path = "../../drivers/gpu/drm/asahi/stats.rs"] @@ -27,3 +32,13 @@ fn firmware_order_zero_and_invalid_indices() { assert_eq!(snapshot.requested_frequency(u32::MAX), None); assert_eq!(stats::StatsSnapshot::default().requested_frequency(0), None); } + +#[test] +fn engine_frequency_needs_the_g17_reader() { + assert_eq!(stats::StatsSnapshot::default().engine_frequency(), None); + let snapshot = stats::StatsSnapshot { + g17_reader: Some(g17::stats::Reader), + ..Default::default() + }; + assert_eq!(snapshot.engine_frequency(), Some((None, 1_620_000_000))); +} From a2180913e4ffe2744783d0fd3d8b85cbc526103f Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Tue, 10 Nov 2026 21:35:32 -0500 Subject: [PATCH 6/8] drm/asahi: g17: per-queue watchdog allowance for compute commands The scheduler's watchdog polls a G17 queue every two seconds and takes a compute command that a poll finds still running at the head of its queue to have hung: on the M5 it renews nothing, and the command fails about two seconds after it got there. A compute job that needs longer, a long prefill of an inference engine for example, has to be cut into small command buffers with few in flight, and still fails when other clients share the GPU: two processes each submitting a 1.1 s compute job that fills the GPU at the same moment ran side by side, one finished after 1.57 s and the other was killed 2.0 s after its submission, with about 0.3 s of its own work left. drm_asahi_queue_create gets compute_timeout_ms, advertised by DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT (bit 36): a watchdog allowance for each compute command of the queue. The driver clamps it (at least the default of one poll, at most the compute_timeout_max_ms module parameter, 30000 by default) and returns the allowance in effect. At a poll, the watchdog now first renews a command that is within its allowance. Render commands keep the default. It is an allowance, not a deadline: - The watchdog still polls every two seconds, so a command may run up to one poll longer than its allowance. - The start of a command is known only approximately. The scheduler arms the watchdog when the job ahead completes, and again when a later job is pushed, so a job's first expiry comes at least one poll after it reached the head of the queue, later if pushes postponed it; the time is counted from one poll before that expiry. A job still in publication at an expiry (waiting for capacity) has not started: its time counts from the first expiry that finds it published. - Past the allowance, the renewals and deferrals that false_timeout() already grants still apply, in the same order as before: the execution renewals of ray and acceleration-structure work, the shared-work renewals (not on the M5) and the deferral of work behind another VM's earlier work. Under the default allowance the first expiry of a published job is not renewed by it, as before. The one change there: a job that was still in publication at an earlier expiry now gets one poll from the expiry that first finds it published, where before it could fail at that expiry however recently it had been published. There is no bound on the sum of the allowances: a queue kept full of long commands occupies the GPU for as long as it is fed. Nor are the other clients shielded from such a queue: they share the GPU with it as the firmware schedules them, and a command of another client can wait until a long command ends (the numbers are below). The struct grows at its end: a caller passing the old size gets the default, and a kernel without the feature ignores the field. Measured on the J716c (M5 Max MacBook Pro), with compute jobs of a single-thread loop of known length. Before this change: - one job of 2.13 s or 2.41 s: killed at 2.04-2.05 s (ETIMEDOUT, the client sees a lost device); one of 1.86 s: completes. - four jobs of 0.93 s submitted back to back on one queue, with and without timestamp queries: all complete, the last 3.71 s after its submission; two of 1.6 s: the second completes 3.23 s after its submission. Jobs queued behind their own queue do not lose time. - one 1.1 s job on each of two queues of one process: both complete (2.24 s for the second), both queues share one firmware queue. With it, and a Vulkan driver that passes a requested allowance on: - no request: a 2.13 s job is killed at 2.04 s, as before. - 30000 requested: jobs of 5 s and 20 s complete; a 40 s job is killed 30.7 s after its submission, as it is when 600000 is requested. - three 16 s jobs submitted back to back on one queue: all complete, 16.0, 32.0 and 48.0 s after the first submission. - two processes each submitting a 1.1 s job that fills the GPU at the same moment: both complete, after 1.93 s and 2.37 s. Sharing, measured with one client keeping a queue full of compute jobs that fill the GPU (640 threads), of 1.5 s under the default allowance and of 5 s and 20 s with 30000 requested, while a second client submits a short compute job every 100 ms (a single thread of 5 ms, or 5 ms of work filling the GPU) under the default allowance: - On the integration/aurora of 2026-11-09 (about 1000 jobs of each kind beside the long jobs): the second client's jobs took a median of 6 to 9 ms and at most 105 ms, whatever the length of the first client's jobs. The desktop compositor's render jobs, about 1 ms on an idle GPU, took up to 0.7 s beside jobs of any of the three lengths, and 4 to 5 of them completed per second beside the 5 s and 20 s jobs against 9 to 10 beside the 1.5 s jobs and 43 idle; six of its jobs waited between 1.4 s and more than 2 s beside the long jobs, none beside the 1.5 s jobs. - On the integration/aurora this change is based on (about 730 jobs of the kind that fills the GPU): most took 6 ms, but six waited until about the end of the first client's running job: 1.5 s beside a 1.5 s job, and 3.4, 4.5, 5.7, 8.3 and 18.4 s beside 20 s jobs (the 4.5, 8.3 and 18.4 s ones with the first client's queue at low priority, which did not help). The second client's watchdog deferred the ones past two seconds behind the other VM's work; the last one ran out of deferrals after 18 s and failed with ETIMEDOUT, 2.3 s before the first client's job ended. So on the current tree a queue with a long allowance can hold up another client's command for as long as its own command runs, and past 18 s that client's work fails. Under the default allowance the same wait is bounded by the two seconds a command may run. Tests: the timeout tests (g17-ray-query-timeout-tests.sh on the host, 23 passed, and KUnit) cover the requested allowances, the default, a 30 s allowance counted from the head of the queue and ended at the first poll past it, a first expiry delayed by later pushes, and a job that waited in publication. The asahi-m4 layout probe compares the frozen queue_create bindings as a prefix of the grown struct and asserts the new tail. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/asahi_core.rs | 4 + drivers/gpu/drm/asahi/file.rs | 11 ++ drivers/gpu/drm/asahi/g17/job.rs | 17 ++- drivers/gpu/drm/asahi/g17/mod.rs | 3 +- drivers/gpu/drm/asahi/g17/queue/mod.rs | 22 +++- drivers/gpu/drm/asahi/g17/timeout.rs | 101 ++++++++++++++++++ drivers/gpu/drm/asahi/queue/mod.rs | 7 ++ include/uapi/drm/asahi_drm.h | 51 +++++++++ .../selftests/drivers/gpu/asahi-m4/abi.py | 25 ++++- 9 files changed, 235 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/asahi/asahi_core.rs b/drivers/gpu/drm/asahi/asahi_core.rs index 8dcfc3ad8ca40..6b840e524cf04 100644 --- a/drivers/gpu/drm/asahi/asahi_core.rs +++ b/drivers/gpu/drm/asahi/asahi_core.rs @@ -182,6 +182,10 @@ module! { default: 512, description: "G13/G14: GPU event-drain utilization minimum (0..1024)", }, + compute_timeout_max_ms: u32 { + default: 30000, + description: "G17: longest compute watchdog allowance a queue may set, ms (floor 2000)", + }, g15_deadline_ms: u32 { default: 5000, description: "G15: host deadline of a published prefix without progress, ms (min 100)", diff --git a/drivers/gpu/drm/asahi/file.rs b/drivers/gpu/drm/asahi/file.rs index 9fe3de9109731..c1dc902822f8b 100644 --- a/drivers/gpu/drm/asahi/file.rs +++ b/drivers/gpu/drm/asahi/file.rs @@ -102,6 +102,10 @@ pub(crate) const DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION: u64 = /// command may wait for it in firmware. pub(crate) const DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION: u32 = uapi::drm_asahi_compute_flags_DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION as u32; +/// Queues take a watchdog allowance for their compute commands +/// (`drm_asahi_queue_create::compute_timeout_ms`). +pub(crate) const DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT: u64 = + uapi::DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT as u64; /// The tiling half waits only on the render and compute prefixes of the command's stage-prefix /// fields, its fragment half on the whole ones. #[allow(dead_code)] @@ -1584,6 +1588,10 @@ impl File { data.flags, ); + if data.pad != 0 { + cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); + return Err(EINVAL); + } if data.flags != 0 || data.priority > uapi::drm_asahi_priority_DRM_ASAHI_PRIORITY_REALTIME { cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); return Err(EINVAL); @@ -1655,6 +1663,9 @@ impl File { )?; queue.reserve_render_scs(data.render_scs_layout)?; + if data.compute_timeout_ms != 0 { + data.compute_timeout_ms = queue.set_compute_timeout(data.compute_timeout_ms)?; + } data.queue_id = resv.index().try_into()?; resv.fill(Arc::pin_init( diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index 8d2e5f443fe14..8e71c0d25d69d 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -365,6 +365,8 @@ pub(crate) trait Backend: Send + Sync { struct Job { backend: Arc, packet: Arc, + /// The watchdog allowance of the job ([`super::timeout::Allowance`]), in ms. + allowance_ms: u32, } struct Timeout { @@ -372,6 +374,7 @@ struct Timeout { victim_renewals: u8, execution_renewals: u8, shared_renewals: u8, + allowance: super::timeout::Allowance, packet: Arc, } @@ -397,6 +400,7 @@ impl sched::JobImpl for Job { victim_renewals: 0, execution_renewals: 0, shared_renewals: 0, + allowance: super::timeout::Allowance::new(self.allowance_ms), packet: self.packet.clone(), } } @@ -405,6 +409,10 @@ impl sched::JobImpl for Job { if finished || job.packet.completion.take_replay_timeout() { return true; } + // Within the allowance its queue asked for. Past it, the deferrals below still apply. + if job.allowance.renew(job.packet.is_published(), super::now_ns()) { + return true; + } if super::timeout::renew_execution( job.packet.is_published(), job.packet.command.long_job(), &mut job.execution_renewals, ) { @@ -457,6 +465,7 @@ pub(crate) struct Scheduler { entity: sched::Entity>, _scheduler: sched::Scheduler>, backend: Arc, + allowance_ms: u32, } #[derive(Clone)] @@ -486,7 +495,11 @@ impl Fences { } impl Scheduler { - pub(crate) fn new(dev: &AsahiDrmDevice, backend: Arc, engine: Engine) -> Result { + /// `allowance_ms`: the watchdog allowance of each job ([`super::timeout::allowance_ms`]); + /// the watchdog still polls every [`TIMEOUT_MS`]. + pub(crate) fn new(dev: &AsahiDrmDevice, backend: Arc, engine: Engine, allowance_ms: u32) + -> Result + { let (credits, name) = match engine { Engine::Render => ( RENDER_CREDITS.checked_mul(backend.render_pairs()).ok_or(EOVERFLOW)?, @@ -513,6 +526,7 @@ impl Scheduler { entity, _scheduler: scheduler, backend, + allowance_ms, }) } @@ -540,6 +554,7 @@ impl Scheduler { Job { backend: self.backend.clone(), packet: packet.clone(), + allowance_ms: self.allowance_ms, }, )?; if dependencies.len() > MERGED_DEPENDENCIES { diff --git a/drivers/gpu/drm/asahi/g17/mod.rs b/drivers/gpu/drm/asahi/g17/mod.rs index 88610416beab1..7ec5e7808e929 100644 --- a/drivers/gpu/drm/asahi/g17/mod.rs +++ b/drivers/gpu/drm/asahi/g17/mod.rs @@ -2458,7 +2458,8 @@ impl gpu::Gpu for Gpu { u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_PER_COMMAND_INPUT_SYNCS) | u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS) | u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS) - | sync_output::FEATURE_CUMULATIVE_SIGNALS; + | sync_output::FEATURE_CUMULATIVE_SIGNALS + | crate::file::DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT; let variant = self.shared.variant; // The M5 command set has no AS-build flag: its validation refuses one. if !variant.m5_commands { diff --git a/drivers/gpu/drm/asahi/g17/queue/mod.rs b/drivers/gpu/drm/asahi/g17/queue/mod.rs index 47a506a205615..41af18adbc517 100644 --- a/drivers/gpu/drm/asahi/g17/queue/mod.rs +++ b/drivers/gpu/drm/asahi/g17/queue/mod.rs @@ -89,6 +89,9 @@ pub(crate) struct Queue { backend: Arc, render: Option>, compute: Option>, + /// The watchdog allowance of each compute command, set before the first one + /// ([`crate::queue::Queue::set_compute_timeout`]). + compute_allowance_ms: u32, fences: FenceContexts, allocation_failure: Fence, range: Range, @@ -127,7 +130,12 @@ impl Queue { window, variant, } = frontend; - let render = Some(Scheduler::new(dev, backend.clone(), Engine::Render)?); + let render = Some(Scheduler::new( + dev, + backend.clone(), + Engine::Render, + super::timeout::DEFAULT_ALLOWANCE_MS, + )?); Ok(Self { dev: dev.into(), context, @@ -135,6 +143,7 @@ impl Queue { backend, render, compute: None, + compute_allowance_ms: super::timeout::DEFAULT_ALLOWANCE_MS, fences, allocation_failure, range, @@ -158,6 +167,7 @@ impl Queue { &self.dev, self.backend.clone(), Engine::Compute, + self.compute_allowance_ms, )?); } Ok(()) @@ -513,6 +523,16 @@ impl Queue { } impl crate::queue::Queue for Queue { + fn set_compute_timeout(&mut self, requested_ms: u32) -> Result { + // The compute scheduler takes the allowance when the first compute command creates it. + if self.compute.is_some() { + return Err(EBUSY); + } + let maximum = crate::module_parameters::compute_timeout_max_ms.value(); + self.compute_allowance_ms = super::timeout::allowance_ms(requested_ms, maximum); + Ok(self.compute_allowance_ms) + } + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { let pages = [ super::fw::compute::ScratchRequest::new(layouts[0])?.pages(), diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index 3da5e255408b5..d42f4d1e7d856 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -52,6 +52,59 @@ pub(crate) fn as_build(flags: u32) -> Option { if flags & !AS_BUILD != 0 { None } else { Some(flags & AS_BUILD != 0) } } +/// The watchdog allowance of a compute command unless its queue asked for +/// another (`drm_asahi_queue_create::compute_timeout_ms`): one watchdog poll. +pub(crate) const DEFAULT_ALLOWANCE_MS: u32 = POLL_MS; + +/// The allowance a queue gets for its compute commands when it asks for +/// `requested_ms`, given the longest the system allows (`maximum_ms`, the +/// `compute_timeout_max_ms` module parameter): never below the default. +pub(crate) fn allowance_ms(requested_ms: u32, maximum_ms: u32) -> u32 { + requested_ms.min(maximum_ms).max(DEFAULT_ALLOWANCE_MS) +} + +/// A job's watchdog allowance: how long the watchdog renews it for the asking, +/// counted from about when it reached the head of its queue. +/// +/// It is not a deadline. The watchdog only sees a job at its expiries, and +/// does not know when the job started: the scheduler arms a queue's watchdog +/// when a job reaches the head (the job ahead of it completed) and again +/// whenever a later job of the queue is pushed, and the watchdog expires one +/// [`POLL_MS`] later. So a job's first expiry comes at least one poll after it +/// reached the head, later if pushes postponed it, and the time is counted +/// from one poll before that expiry. A job still in publication at an expiry +/// (waiting for capacity) has not started: its time counts from the first +/// expiry that finds it published. Either way a job never loses time to the +/// jobs ahead of it, and it may get up to one poll more than its allowance. +/// Under the default allowance it is never renewed. A job past its allowance +/// is not failed for that alone: the caller's other renewals and deferrals +/// are tried next. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Allowance { + ms: u32, + /// When the job reached the head, once an expiry placed it. + head_since: Option, + /// An expiry found the job still in publication. + waited: bool, +} + +impl Allowance { + pub(crate) const fn new(ms: u32) -> Self { + Self { ms, head_since: None, waited: false } + } + + /// Whether the job is within its allowance at the expiry at `now` (host ns). + pub(crate) fn renew(&mut self, published: bool, now: u64) -> bool { + if !published { + self.waited = true; + return false; + } + let poll = if self.waited { 0 } else { u64::from(POLL_MS) * 1_000_000 }; + let since = *self.head_since.get_or_insert(now.saturating_sub(poll)); + now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 + } +} + pub(crate) fn renew_execution(published: bool, enabled: bool, renewals: &mut u8) -> bool { if !published || !enabled || *renewals >= EXECUTION_RENEWALS { return false; @@ -201,6 +254,54 @@ pub(crate) fn behind_own_head(published: bool, older: Option, progressing: #[cfg_attr(not(test), kernel::macros::kunit_tests(asahi_g17_execution_allowance))] mod tests { use super::*; + const MS: u64 = 1_000_000; + #[test] + fn a_queue_gets_the_allowance_it_asks_for_within_bounds() { + assert_eq!(allowance_ms(0, 30_000), DEFAULT_ALLOWANCE_MS); + assert_eq!(allowance_ms(500, 30_000), DEFAULT_ALLOWANCE_MS); + assert_eq!(allowance_ms(20_000, 30_000), 20_000); + assert_eq!(allowance_ms(u32::MAX, 30_000), 30_000); + // A maximum below the default still leaves the default. + assert_eq!(allowance_ms(20_000, 0), DEFAULT_ALLOWANCE_MS); + } + #[test] + fn the_default_allowance_renews_nothing() { + let mut allowance = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!allowance.renew(true, 100_000 * MS)); + // Not after a wait in publication either. + let mut allowance = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!allowance.renew(false, 100_000 * MS)); + assert!(allowance.renew(true, 102_000 * MS)); + assert!(!allowance.renew(true, 104_000 * MS)); + } + #[test] + fn a_long_allowance_runs_from_the_head_of_the_queue() { + // Reached the head at 10 s (the job ahead completed): first expiry at 12 s. + let start = 10_000 * MS; + let poll = u64::from(POLL_MS) * MS; + let mut allowance = Allowance::new(30_000); + let mut expiry = start + poll; + while allowance.renew(true, expiry) { + expiry += poll; + } + assert_eq!(allowance.head_since, Some(start)); + // Killed at the first poll at or past 30 s after it reached the head. + assert_eq!(expiry - start, 30_000 * MS); + // A first expiry delayed by later pushes only moves the count later. + let mut allowance = Allowance::new(30_000); + assert!(allowance.renew(true, start + poll + 500 * MS)); + assert_eq!(allowance.head_since, Some(start + 500 * MS)); + } + #[test] + fn a_job_waiting_in_publication_starts_at_its_first_published_expiry() { + let mut allowance = Allowance::new(30_000); + assert!(!allowance.renew(false, 10_000 * MS)); + assert_eq!(allowance.head_since, None); + assert!(allowance.renew(true, 12_000 * MS)); + assert_eq!(allowance.head_since, Some(12_000 * MS)); + assert!(allowance.renew(true, 41_999 * MS)); + assert!(!allowance.renew(true, 42_000 * MS)); + } #[test] fn selects_compute_both_render_halves_and_builds() { assert!(!select(false, [false; 2], false)); diff --git a/drivers/gpu/drm/asahi/queue/mod.rs b/drivers/gpu/drm/asahi/queue/mod.rs index 525a2ffffbbfe..a37bb48fa613f 100644 --- a/drivers/gpu/drm/asahi/queue/mod.rs +++ b/drivers/gpu/drm/asahi/queue/mod.rs @@ -70,6 +70,13 @@ fn can_submit_inline(dependencies_ready: bool, previous_status: Option) -> /// Trait implemented by all versioned queues. pub(crate) trait Queue: Send + Sync { + /// Sets the watchdog allowance of the queue's compute commands to `requested_ms` + /// (`drm_asahi_queue_create::compute_timeout_ms`, nonzero) and returns the allowance in + /// effect. Only queues of GPUs that advertise DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT take one. + fn set_compute_timeout(&mut self, _requested_ms: u32) -> Result { + Err(EINVAL) + } + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { if layouts != [0; 2] { return Err(EINVAL); diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index c51b239fb4917..f41bf829e7b7d 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -495,6 +495,13 @@ enum drm_asahi_feature { */ #define DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION ((1ULL) << 34) +/* + * DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT: drm_asahi_queue_create::compute_timeout_ms + * sets the watchdog allowance of the queue's compute commands. A feature bit + * like those of enum drm_asahi_feature, defined here like bit 32. + */ +#define DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT ((1ULL) << 36) + /* * DRM_ASAHI_FEATURE_USERPTR: DRM_IOCTL_ASAHI_GEM_USERPTR is available. A * feature bit like those of enum drm_asahi_feature, defined here like bit 32. @@ -1112,6 +1119,50 @@ struct drm_asahi_queue_create { * Requires DRM_ASAHI_FEATURE_RENDER_RAY_QUERY; zero otherwise. */ __u64 render_scs_layout[2]; + + /** + * @compute_timeout_ms: Watchdog allowance of each compute command of + * this queue, in milliseconds. Zero keeps the driver's default. + * Requires DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT; zero otherwise. + * + * The driver's watchdog polls a queue every two seconds. By default it + * takes a compute command that a poll finds still running at the head + * of the queue (every earlier compute command of the queue has + * completed) to have hung, and fails it, and the queue's VM with it. + * With an allowance, the watchdog lets the command run on at each poll + * until the allowance has passed. For a nonzero request the driver + * raises a shorter allowance to its default of one poll, lowers a + * longer one to the maximum the system allows, and returns the + * allowance in effect here; a request of 0 is returned unchanged. + * + * This is an allowance, not a deadline: + * + * - The watchdog acts at its polls only, so a command may run up to + * one poll longer than its allowance. + * - The start of a command is known only approximately. The time is + * counted from one poll before the first poll that finds the command + * at the head of the queue. That is no earlier than the command got + * there and may be later: commands submitted to the queue in the + * meantime postpone that poll. + * - Once the allowance has passed, the deferrals the watchdog grants + * any command still apply, as they do by default: a command it takes + * to be waiting behind other clients' work, or one of a kind it + * gives extra polls, runs on until those are used up as well. + * + * The allowance is per command, and nothing bounds the sum: a queue + * that is kept full of long commands occupies the GPU for as long as + * it is fed. The driver does not shield the other clients from such a + * queue: they share the GPU with it as the firmware schedules them, + * and a command of another client may have to wait until a long + * command ends, which can be longer than that client's own allowance + * and the deferrals it gets. Userspace should ask for an allowance + * only for queues whose commands need one, and keep those commands as + * short as it can. + */ + __u32 compute_timeout_ms; + + /** @pad: MBZ */ + __u32 pad; }; /** diff --git a/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py b/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py index 7b9bbce635e6e..f9da7cd001c91 100644 --- a/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py +++ b/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py @@ -62,15 +62,13 @@ def run_measurements(command, source, binary): def c_layout(header, bindings, output, tag): keys, _ = expressions(bindings) capacity_tail = 'client_memory_limit;' in header.read_text() + queue_tail = 'compute_timeout_ms;' in header.read_text() # Linux 7.2 folds the historical extension structs into the base payloads. # Compare the same wire prefixes and offsets while keeping the C oracle frozen. if 'struct drm_asahi_cmd_render_with_prior {' not in header.read_text(): translated = [] for key, expr in keys: kind, _, field = key.partition(':') - if capacity_tail and key == 'drm_asahi_params_global:size': - # Compare the frozen reply prefix; validate the entire known additive tail below. - expr = 'offsetof(struct drm_asahi_params_global, memory_limit)' if key == "drm_asahi_vm_create:pad": expr = "offsetof(struct drm_asahi_vm_create, flags)" base = None @@ -90,6 +88,19 @@ def c_layout(header, bindings, output, tag): else: expr = f'offsetof(struct {base}, {field})' translated.append((key, expr)) keys = translated + # The two additive tails apply to every header that carries them, including + # one that still has the historical extension structs. Compare the frozen + # reply prefixes; validate the entire known additive tails below. + tails = {} + if capacity_tail: + tails['drm_asahi_params_global:size'] = \ + 'offsetof(struct drm_asahi_params_global, memory_limit)' + if queue_tail: + tails['drm_asahi_queue_create:size'] = \ + 'offsetof(struct drm_asahi_queue_create, compute_timeout_ms)' + if tails: + keys = [(key, tails[key]) if key in tails else (key, expr) + for key, expr in keys] source = output / (tag + '.c') lines = ['#include ', '#include ', '#include ' + json.dumps(str(header)), 'int main(void) {'] @@ -101,6 +112,14 @@ def c_layout(header, bindings, output, tag): '_Static_assert(sizeof(((struct drm_asahi_params_global *)0)->memory_limit) == 8, "capacity width");', '_Static_assert(sizeof(((struct drm_asahi_params_global *)0)->client_memory_limit) == 8, "client capacity width");', ] + if queue_tail: + lines[3:3] = [ + '_Static_assert(sizeof(struct drm_asahi_queue_create) == 48, "queue time limit reply size");', + '_Static_assert(offsetof(struct drm_asahi_queue_create, compute_timeout_ms) == 40, "queue time limit offset");', + '_Static_assert(sizeof(((struct drm_asahi_queue_create *)0)->compute_timeout_ms) == 4, "queue time limit width");', + '_Static_assert(offsetof(struct drm_asahi_queue_create, pad) == 44, "queue time limit pad offset");', + '_Static_assert(sizeof(((struct drm_asahi_queue_create *)0)->pad) == 4, "queue time limit pad width");', + ] lines += [f'printf("{key}=%llu\\n", (unsigned long long)({expr}));' for key, expr in keys] source.write_text('\n'.join(lines + ['return 0;', '}']) + '\n') return run_measurements(shlex.split(os.environ.get('CC', 'cc')) + From b5ca79e11523076ea081cf6c705531c668c3c5d4 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Thu, 12 Nov 2026 19:52:49 -0500 Subject: [PATCH 7/8] drm/asahi: g17: don't fail work waiting behind a command inside its allowance With a compute allowance, one client's command may run for up to compute_timeout_max_ms (30 s by default), and another client's command, render or compute, can wait on the GPU until it ends. The waiting job's watchdog defers it behind the other VM's older work at most eight polls (MAX_RENEWALS, 16 s) and then fails it: a compositor frame that waited behind a 20 s compute command of another client failed with ETIMEDOUT after about 18 s, and the compositor lost its device. A job's watchdog now records when its allowance ends in the packet, and the cross-VM heads carry whether that job is still inside it. A waiting job whose blocker is live and inside its allowance is renewed with a new decision, Excuse, which does not use its eight renewals. Once the blocker completes, or its allowance has passed, the existing rules apply again, with what the job has left of its eight renewals for the blocker's own failure and teardown. Unchanged: - A blocker that already failed excuses nothing, as before. - The oldest unfinished work is never excused: it has no blocker. - A blocker under the default allowance excuses nothing, so clients that do not ask for an allowance see the decisions they saw before. Render and compute watchdogs decide through the same attribution, and both see compute heads as blockers, so a render job waiting behind a long compute command is covered too. The excuse is bounded on the waiting job's side: at most compute_timeout_max_ms in polls (15 with the default 30 s), however many blockers take turns. This rule therefore renews a job at most 15 + 8 times by default, 46 s of renewals. A job's own allowance and its other deferrals are counted separately and come on top, and each renewal re-arms a timer rather than an absolute deadline, so the time until a waiting job is failed (spared) can be longer than that. The watchdog lines for cross-VM decisions now print the excused polls and whether the blocker was inside its allowance, and are written at the first excuse, the first counted deferral and the final decision. Tests: KUnit and the host timeout tests cover a live blocker inside its allowance past eight renewals (excused), the excuse used up (renewals, then spared), a blocker past its allowance and a failed one (the decisions as before), no blocker and the oldest work (own), the bound, blockers taking turns (the counters are the waiting job's and are not reset), and when an allowance ends. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/g17/job.rs | 48 +++- drivers/gpu/drm/asahi/g17/queue/compute.rs | 1 + drivers/gpu/drm/asahi/g17/queue/render.rs | 1 + drivers/gpu/drm/asahi/g17/runtime/backend.rs | 30 +-- .../drm/asahi/g17/runtime/compute_control.rs | 20 +- drivers/gpu/drm/asahi/g17/runtime/render.rs | 2 + drivers/gpu/drm/asahi/g17/timeout.rs | 215 ++++++++++++++++-- include/uapi/drm/asahi_drm.h | 9 +- 8 files changed, 276 insertions(+), 50 deletions(-) diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index 8e71c0d25d69d..eccce9ea3dc84 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -4,7 +4,7 @@ use super::{command::Validated, completion::Completion, context::Context}; use crate::driver::AsahiDrmDevice; -use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU8, Ordering}; +use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU64, AtomicU8, Ordering}; use kernel::{ bindings, c_str, cpu::CpuId, @@ -128,6 +128,9 @@ pub(crate) struct Packet { cancelled: AtomicBool, early: AtomicI32, backlog: AtomicBool, + /// When the job's watchdog allowance ends (host ns, [`super::timeout::Allowance::end`]): + /// zero under the default allowance, `u64::MAX` for a longer one not yet placed. + allowance_end: AtomicU64, } impl Packet { @@ -160,11 +163,18 @@ impl Packet { cancelled: AtomicBool::new(false), early: AtomicI32::new(0), backlog: AtomicBool::new(false), + allowance_end: AtomicU64::new(0), }, GFP_KERNEL, )?) } + /// Its own watchdog still renews the job for the allowance its queue asked for at `now` + /// ([`super::timeout::Head::excused`]). + pub(crate) fn within_allowance(&self, now: u64) -> bool { + now < self.allowance_end.load(Ordering::Relaxed) + } + pub(crate) fn engine(&self) -> Engine { match &self.command { Validated::Render { .. } => Engine::Render, @@ -350,8 +360,12 @@ pub(crate) trait Backend: Send + Sync { fn publish(&self, packet: Arc) -> Result; fn timed_out(&self, packet: Arc) -> sched::Status; fn cancel(&self, packet: Arc); - /// Transfer a timeout to an outstanding off-lock graph transaction, if any. - fn defer_timeout(&self, packet: &Arc, renewals: u8) -> bool; + /// Transfer a timeout to an outstanding off-lock graph transaction, if any, or decide it + /// against other VMs' older work ([`super::timeout::decide`]). `renewals` and `excused` + /// count the job's earlier [`super::timeout::Decision::Defer`] and + /// [`super::timeout::Decision::Excuse`] results; the job is renewed on either. + fn defer_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> super::timeout::Decision; /// A long job of another queue is running beside `packet`. fn long_job_beside(&self, _packet: &Arc) -> bool { false } /// A published job's deadline may be renewed while other work retires @@ -372,6 +386,9 @@ struct Job { struct Timeout { backend: Arc, victim_renewals: u8, + /// Polls excused behind other VMs' live work inside its allowance + /// ([`super::timeout::Decision::Excuse`]). + excused: u32, execution_renewals: u8, shared_renewals: u8, allowance: super::timeout::Allowance, @@ -398,6 +415,7 @@ impl sched::JobImpl for Job { Timeout { backend: self.backend.clone(), victim_renewals: 0, + excused: 0, execution_renewals: 0, shared_renewals: 0, allowance: super::timeout::Allowance::new(self.allowance_ms), @@ -410,7 +428,12 @@ impl sched::JobImpl for Job { return true; } // Within the allowance its queue asked for. Past it, the deferrals below still apply. - if job.allowance.renew(job.packet.is_published(), super::now_ns()) { + let within = job.allowance.renew(job.packet.is_published(), super::now_ns()); + // Other VMs' watchdogs read the end to excuse work waiting behind this one. + if let Some(end) = job.allowance.end() { + job.packet.allowance_end.store(end, Ordering::Relaxed); + } + if within { return true; } if super::timeout::renew_execution( @@ -426,11 +449,17 @@ impl sched::JobImpl for Job { if super::timeout::renew_shared(job.packet.is_published(), shared, &mut job.shared_renewals) { return true; } - let deferred = job.backend.defer_timeout(&job.packet, job.victim_renewals); - if deferred { - job.victim_renewals = job.victim_renewals.saturating_add(1); + match job.backend.defer_timeout(&job.packet, job.victim_renewals, job.excused) { + super::timeout::Decision::Defer => { + job.victim_renewals = job.victim_renewals.saturating_add(1); + true + } + super::timeout::Decision::Excuse => { + job.excused = job.excused.saturating_add(1); + true + } + super::timeout::Decision::Own | super::timeout::Decision::Spare => false, } - deferred } fn prepare(job: &mut Self) -> Option { @@ -549,6 +578,9 @@ impl Scheduler { } else { None }; + if self.allowance_ms > super::timeout::DEFAULT_ALLOWANCE_MS { + packet.allowance_end.store(u64::MAX, Ordering::Relaxed); + } let mut job = self.entity.new_job( 1, Job { diff --git a/drivers/gpu/drm/asahi/g17/queue/compute.rs b/drivers/gpu/drm/asahi/g17/queue/compute.rs index 99d862a4f71ca..8042e2026451b 100644 --- a/drivers/gpu/drm/asahi/g17/queue/compute.rs +++ b/drivers/gpu/drm/asahi/g17/queue/compute.rs @@ -909,6 +909,7 @@ impl Queue { active.observed.is_some() }, failed: self.quarantined, + excused: active.packet.within_allowance(crate::g17::now_ns()), }) } diff --git a/drivers/gpu/drm/asahi/g17/queue/render.rs b/drivers/gpu/drm/asahi/g17/queue/render.rs index 16e9d0e265571..c68448f4503c3 100644 --- a/drivers/gpu/drm/asahi/g17/queue/render.rs +++ b/drivers/gpu/drm/asahi/g17/queue/render.rs @@ -1667,6 +1667,7 @@ impl Pair { published: active.published, finished: false, failed: self.quarantined, + excused: active.packet.within_allowance(crate::g17::now_ns()), }) } diff --git a/drivers/gpu/drm/asahi/g17/runtime/backend.rs b/drivers/gpu/drm/asahi/g17/runtime/backend.rs index 705325bf6d153..b17fe9eb07d11 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/backend.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/backend.rs @@ -386,19 +386,20 @@ impl job::Backend for Backend { .any(|other| other.owner() != Some(self.owner) && other.long_job_running()) } - fn defer_timeout(&self, packet: &Arc, renewals: u8) -> bool { + fn defer_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> crate::g17::timeout::Decision { if packet.engine() == Engine::Compute { - return self.defer_compute_timeout(packet, renewals); + return self.defer_compute_timeout(packet, renewals, excused); } use crate::g17::timeout::{self, Decision}; // As for compute, a final retirement pass precedes the attribution, and all completion // callbacks run after releasing the device mutex. Without a collector the attribution // goes ahead without that pass, as it did before it existed. let mut deferred = DeferredBatch::worker().ok(); - let result = (|| -> Result { + let result = (|| -> Result { let mut state = self.shared.state.lock(); let Some(firmware) = (*state).as_deref_mut() else { - return Ok(false); + return Ok(Decision::Own); }; if let Some(deferred) = deferred.as_mut() { firmware.scan_renders(deferred)?; @@ -409,15 +410,17 @@ impl job::Backend for Backend { "Render watchdog owner={} behind its context's older node {:#x} (own {:#x}); the firmware progresses: renewed\n", self.owner, older, own); } - return Ok(true); + return Ok(Decision::Defer); } // Borrowed-graph and queued-retirement deferrals share the renewal bound. if timeout::transaction_grace(renewals) && firmware.queues.defer_render_timeout(packet, ETIMEDOUT) { - return Ok(true); + return Ok(Decision::Defer); } - let verdict = firmware.queues.render_timeout_decision(packet, renewals); + let excuse = timeout::Excuse { used: excused, limit: timeout::excuse_limit( + crate::module_parameters::compute_timeout_max_ms.value()) }; + let verdict = firmware.queues.render_timeout_decision(packet, renewals, excuse); let decision = verdict.map_or(Decision::Own, |verdict| verdict.decision); // As compute does: work still blocked behind another VM's older unfinished work // after the whole grace is innocent, so its failure must stay non-sticky. The pair's @@ -426,19 +429,20 @@ impl job::Backend for Backend { packet.completion.classify(crate::g17::recovery::Class::Spared); } if let Some(verdict) = verdict { - if let Some(blocker) = verdict.reported_blocker(renewals) { + if let Some(blocker) = verdict.reported_blocker(renewals, excuse) { dev_warn!(self.shared.dev.as_ref(), - "Render watchdog owner={} vm={} publication={} renewals={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={}\n", - self.owner, verdict.own.vm, verdict.own.published, renewals, decision, - blocker.vm, blocker.published, u8::from(blocker.failed)); + "Render watchdog owner={} vm={} publication={} renewals={} excused={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={} blocker_in_allowance={}\n", + self.owner, verdict.own.vm, verdict.own.published, renewals, excused, + decision, blocker.vm, blocker.published, u8::from(blocker.failed), + u8::from(blocker.excused)); } } - Ok(decision == Decision::Defer) + Ok(decision) })(); if let Some(deferred) = deferred.as_mut() { deferred.finish(); } - matches!(result, Ok(true)) + result.unwrap_or(Decision::Own) } fn try_early(&self, packet: &Arc) -> bool { diff --git a/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs index dfa343a9ac275..922f4340aae3f 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs @@ -9,10 +9,11 @@ use kernel::{prelude::*, sync::Arc}; impl Backend { /// A final retirement pass precedes bounded cross-VM timeout attribution. /// All completion callbacks run after releasing the device mutex. - pub(super) fn defer_compute_timeout(&self, packet: &Arc, renewals: u8) -> bool { + pub(super) fn defer_compute_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> crate::g17::timeout::Decision { use crate::g17::{recovery::Class, timeout::{self, Decision}}; let Ok(mut deferred) = DeferredBatch::worker() else { - return false; + return Decision::Own; }; let result = (|| -> Result { let mut state = self.shared.state.lock(); @@ -39,22 +40,25 @@ impl Backend { .filter(|other| other.owner() != Some(self.owner)) .filter_map(|other| other.timeout_head()); let others = Iterator::chain(computes, firmware.queues.render_timeout_heads()); - let verdict = timeout::attribute(own, renewals, others); + let excuse = timeout::Excuse { used: excused, limit: timeout::excuse_limit( + crate::module_parameters::compute_timeout_max_ms.value()) }; + let verdict = timeout::attribute(own, renewals, excuse, others); let decision = verdict.decision; if decision == Decision::Spare { firmware.queues.compute(self.owner).ok_or(EIO)? .classify(&|_| Class::Spared); } - if let Some(blocker) = verdict.reported_blocker(renewals) { + if let Some(blocker) = verdict.reported_blocker(renewals, excuse) { dev_warn!(self.shared.dev.as_ref(), - "Compute watchdog owner={} vm={} publication={} renewals={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={}\n", - self.owner, own.vm, own.published, renewals, decision, - blocker.vm, blocker.published, u8::from(blocker.failed)); + "Compute watchdog owner={} vm={} publication={} renewals={} excused={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={} blocker_in_allowance={}\n", + self.owner, own.vm, own.published, renewals, excused, decision, + blocker.vm, blocker.published, u8::from(blocker.failed), + u8::from(blocker.excused)); } Ok(decision) })(); deferred.finish(); - matches!(result, Ok(Decision::Defer)) + result.unwrap_or(Decision::Own) } /// The older node of the packet's context ahead of its own, with its own word, when the diff --git a/drivers/gpu/drm/asahi/g17/runtime/render.rs b/drivers/gpu/drm/asahi/g17/runtime/render.rs index 4dac5f5230237..0218e48da42b3 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/render.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render.rs @@ -1271,6 +1271,7 @@ impl Registry { &self, packet: &Arc, renewals: u8, + excuse: crate::g17::timeout::Excuse, ) -> Option { let own = self .render @@ -1291,6 +1292,7 @@ impl Registry { crate::g17::timeout::attribute( own, renewals, + excuse, Iterator::chain(self.render_timeout_heads(), computes), ) }) diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index d42f4d1e7d856..9367b9c57efe3 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -3,6 +3,9 @@ //! Bounded isolation for a compute watchdog blocked behind another VM's older kick. //! Publication order is evidence of a possible blocker, not proof of a firmware //! dependency. Renewals are bounded; the oldest unfinished queue is never excused. +//! A blocker that runs inside the watchdog allowance its queue asked for excuses +//! the work behind it for that long ([`decide`]), bounded by the longest allowance +//! the system permits ([`excuse_limit`]). pub(crate) const MAX_RENEWALS: u8 = 8; @@ -20,8 +23,8 @@ const EXECUTION_RENEWALS: u8 = 4; /// How long an innocent client's publication waits for a recovery to settle and its /// spared work to drain, instead of failing with EBUSY: the watchdog defers work behind -/// another VM at most this long before it settles it (`classify`; ray work gets the -/// execution renewals on top). +/// another VM's failed work at most this long before it settles it (`classify`; ray work +/// gets the execution renewals on top). A failed blocker excuses nothing ([`decide`]). pub(crate) const RECOVERY_WAIT_MS: u32 = POLL_MS * (1 + EXECUTION_RENEWALS as u32 + MAX_RENEWALS as u32); @@ -103,6 +106,30 @@ impl Allowance { let since = *self.head_since.get_or_insert(now.saturating_sub(poll)); now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 } + + /// When the allowance ends (host ns), once an expiry placed the job, for an allowance + /// longer than the default: until then the watchdog renews the job for the asking. A + /// default allowance renews at most the poll after a wait in publication and excuses + /// nobody ([`Head::excused`]). + pub(crate) fn end(&self) -> Option { + let since = self.head_since.filter(|_| self.ms > DEFAULT_ALLOWANCE_MS)?; + Some(since.saturating_add(u64::from(self.ms) * 1_000_000)) + } +} + +/// How many polls a job may be excused behind other VMs' live work inside those blockers' +/// allowances ([`decide`]): the longest allowance a queue may get (`maximum_ms`, the +/// `compute_timeout_max_ms` module parameter) in whole polls, 15 by default. +pub(crate) fn excuse_limit(maximum_ms: u32) -> u32 { + allowance_ms(u32::MAX, maximum_ms).div_ceil(POLL_MS) +} + +/// The polls a waiting job was excused so far (`used`) and the most it may be (`limit`, +/// [`excuse_limit`]). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Excuse { + pub(crate) used: u32, + pub(crate) limit: u32, } pub(crate) fn renew_execution(published: bool, enabled: bool, renewals: &mut u8) -> bool { @@ -153,12 +180,19 @@ pub(crate) struct Head { /// Its queue already failed (quarantined) and holds the GPU only until the firmware drains /// it. No later watchdog of its own will settle it. pub(crate) failed: bool, + /// Its own watchdog still renews it for the allowance its queue asked for + /// ([`Allowance::end`]): a long command running as allowed, not a hung one. + pub(crate) excused: bool, } #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) enum Decision { Own, + /// Renewed: counts against [`MAX_RENEWALS`]. Defer, + /// Renewed behind a live blocker inside its allowance: counts against [`Excuse::limit`] + /// only. + Excuse, Spare, } @@ -176,10 +210,27 @@ pub(crate) fn blocker(own: Head, others: impl Iterator) -> Option) -> Decision { +/// +/// A live blocker that its own watchdog still renews for its queue's allowance +/// ([`Head::excused`]) is not hung: its command may run that long, and work queued behind it +/// on the GPU waits for it. Such work is renewed without using any more of its MAX_RENEWALS, +/// so that what is left of that grace covers the blocker's own failure and teardown once the +/// allowance has passed; then the rules above apply again. +/// +/// The excuse is bounded by the waiting job's [`Excuse::limit`], the longest allowance the +/// system permits in polls, however many blockers take turns. This attribution therefore +/// renews a job at most `limit + MAX_RENEWALS` times: 23 with the default 30 s maximum, 46 s +/// of renewals after the expiry that first deferred it. That bounds this rule only. A job's +/// own allowance and its other deferrals ([`renew_execution`], [`renew_shared`]) are counted +/// separately and come on top, and each renewal re-arms a timer rather than an absolute +/// deadline, so the time until the job is failed (spared) can exceed 46 s. +pub(crate) fn decide(own: Head, renewals: u8, excuse: Excuse, blocker: Option) -> Decision { match blocker { _ if own.finished => Decision::Own, None => Decision::Own, + Some(blocker) if blocker.excused && !blocker.failed && excuse.used < excuse.limit => { + Decision::Excuse + } Some(_) if renewals < MAX_RENEWALS => Decision::Defer, Some(blocker) if blocker.failed => Decision::Own, Some(_) => Decision::Spare, @@ -195,16 +246,22 @@ pub(crate) struct Verdict { } impl Verdict { - /// The blocker a watchdog line names: at the first deferral and once the grace has run out. - pub(crate) fn reported_blocker(&self, renewals: u8) -> Option { - self.blocker.filter(|_| renewals == 0 || self.decision != Decision::Defer) + /// The blocker a watchdog line names: at the first excuse, at the first counted deferral + /// and once the grace has run out. + pub(crate) fn reported_blocker(&self, renewals: u8, excuse: Excuse) -> Option { + self.blocker.filter(|_| match self.decision { + Decision::Excuse => excuse.used == 0, + Decision::Defer => renewals == 0, + Decision::Own | Decision::Spare => true, + }) } } -pub(crate) fn attribute(own: Head, renewals: u8, others: impl Iterator) -> Verdict { +pub(crate) fn attribute(own: Head, renewals: u8, excuse: Excuse, + others: impl Iterator) -> Verdict { let blocker = blocker(own, others); Verdict { - decision: decide(own, renewals, blocker), + decision: decide(own, renewals, excuse, blocker), own, blocker, } @@ -351,14 +408,20 @@ mod tests { } assert!(background_stalled(Some(since), u64::MAX)); } + /// No excuse used yet, under the default maximum allowance. + const FRESH: Excuse = Excuse { used: 0, limit: 15 }; fn classify(own: Head, renewals: u8, others: impl Iterator) -> Decision { - attribute(own, renewals, others).decision + attribute(own, renewals, FRESH, others).decision } fn head(vm: u64, published: u64, finished: bool) -> Head { - Head { vm, published, finished, failed: false } + Head { vm, published, finished, failed: false, excused: false } } fn failed(vm: u64, published: u64) -> Head { - Head { vm, published, finished: false, failed: true } + Head { vm, published, finished: false, failed: true, excused: false } + } + /// A live head whose own watchdog still renews it for its allowance. + fn excused(vm: u64, published: u64) -> Head { + Head { excused: true, ..head(vm, published, false) } } #[test] fn ray_query_execution_is_bounded_at_ten_seconds() { @@ -492,19 +555,133 @@ mod tests { #[test] fn the_watchdog_line_names_the_blocker_at_the_first_deferral_and_the_end() { let own = head(9, 103, false); - let verdict = attribute(own, 0, [failed(5, 100)].into_iter()); + let verdict = attribute(own, 0, FRESH, [failed(5, 100)].into_iter()); let expected = Verdict { decision: Decision::Defer, own, blocker: Some(failed(5, 100)) }; assert_eq!(verdict, expected); - assert_eq!(verdict.reported_blocker(0), Some(failed(5, 100))); - assert_eq!(attribute(own, 1, [failed(5, 100)].into_iter()).reported_blocker(1), None); - let last = attribute(own, MAX_RENEWALS, [failed(5, 100)].into_iter()); + assert_eq!(verdict.reported_blocker(0, FRESH), Some(failed(5, 100))); + let renewed = attribute(own, 1, FRESH, [failed(5, 100)].into_iter()); + assert_eq!(renewed.reported_blocker(1, FRESH), None); + let last = attribute(own, MAX_RENEWALS, FRESH, [failed(5, 100)].into_iter()); assert_eq!(last.decision, Decision::Own); - assert_eq!(last.reported_blocker(MAX_RENEWALS), Some(failed(5, 100))); - let spared = attribute(own, MAX_RENEWALS, [head(5, 100, false)].into_iter()); + assert_eq!(last.reported_blocker(MAX_RENEWALS, FRESH), Some(failed(5, 100))); + let spared = attribute(own, MAX_RENEWALS, FRESH, [head(5, 100, false)].into_iter()); assert_eq!(spared.decision, Decision::Spare); - assert_eq!(spared.reported_blocker(MAX_RENEWALS), Some(head(5, 100, false))); + assert_eq!(spared.reported_blocker(MAX_RENEWALS, FRESH), Some(head(5, 100, false))); // Nothing blocks: nothing to report. - assert_eq!(attribute(own, 0, [].into_iter()).reported_blocker(0), None); + assert_eq!(attribute(own, 0, FRESH, [].into_iter()).reported_blocker(0, FRESH), None); + // An excuse is named at its first poll, and so is the first counted deferral after it. + let blocker = excused(5, 100); + let first = attribute(own, 0, FRESH, [blocker].into_iter()); + assert_eq!(first.decision, Decision::Excuse); + assert_eq!(first.reported_blocker(0, FRESH), Some(blocker)); + let later = Excuse { used: 3, ..FRESH }; + let excused_again = attribute(own, 0, later, [blocker].into_iter()); + assert_eq!(excused_again.reported_blocker(0, later), None); + let past = attribute(own, 0, later, [head(5, 100, false)].into_iter()); + assert_eq!(past.decision, Decision::Defer); + assert_eq!(past.reported_blocker(0, later), Some(head(5, 100, false))); + } + #[test] + fn the_excuse_is_bounded_by_the_longest_allowance() { + assert_eq!(excuse_limit(30_000), 15); + assert_eq!(excuse_limit(29_000), 15); + assert_eq!(excuse_limit(0), 1); + // The default maximum gives 15 + 8 renewals behind other VMs' live work: 46 s of them. + assert_eq!(POLL_MS * (excuse_limit(30_000) + u32::from(MAX_RENEWALS)), 46_000); + } + #[test] + fn blockers_taking_turns_do_not_extend_the_excuse_or_the_renewals() { + // Two other clients' commands, each inside its allowance, are ahead by turns for + // longer than the excuse lasts. The counters belong to the waiting job: it is renewed + // limit + MAX_RENEWALS times in all, whichever blocker is ahead, and then spared. + let own = head(9, 103, false); + let (mut renewals, mut excuse, mut polls) = (0u8, FRESH, 0u32); + loop { + let ahead = if polls % 2 == 0 { excused(5, 100) } else { excused(6, 101) }; + match attribute(own, renewals, excuse, [ahead].into_iter()).decision { + Decision::Excuse => excuse.used += 1, + Decision::Defer => renewals += 1, + decision => { + assert_eq!(decision, Decision::Spare); + break; + } + } + polls += 1; + assert!(polls < 100); + } + assert_eq!(excuse.used, FRESH.limit); + assert_eq!(renewals, MAX_RENEWALS); + assert_eq!(polls, FRESH.limit + u32::from(MAX_RENEWALS)); + // Renewals spent before an excuse are not given back by it. + let spent = attribute(own, 2, FRESH, [excused(5, 100)].into_iter()); + assert_eq!(spent.decision, Decision::Excuse); + let after = Excuse { used: FRESH.limit, ..FRESH }; + assert_eq!(attribute(own, 2, after, [head(5, 100, false)].into_iter()).decision, + Decision::Defer); + assert_eq!(attribute(own, MAX_RENEWALS, after, [head(5, 100, false)].into_iter()) + .decision, Decision::Spare); + } + #[test] + fn a_live_blocker_inside_its_allowance_defers_past_the_renewals() { + // A compositor frame behind another client's 20 s compute command: its own renewals + // stay untouched while the command runs inside its allowance. + let own = head(9, 103, false); + for used in 0..FRESH.limit { + let excuse = Excuse { used, ..FRESH }; + assert_eq!(attribute(own, 0, excuse, [excused(5, 100)].into_iter()).decision, + Decision::Excuse); + // Past eight renewals from an earlier blocker too. + assert_eq!(attribute(own, MAX_RENEWALS, excuse, [excused(5, 100)].into_iter()) + .decision, Decision::Excuse); + } + // The excuse is used up: the renewals apply, then the work is spared. + let used = Excuse { used: FRESH.limit, ..FRESH }; + assert_eq!(attribute(own, 0, used, [excused(5, 100)].into_iter()).decision, + Decision::Defer); + assert_eq!(attribute(own, MAX_RENEWALS, used, [excused(5, 100)].into_iter()).decision, + Decision::Spare); + } + #[test] + fn a_blocker_past_its_allowance_gets_todays_decision() { + let own = head(9, 103, false); + let past = head(5, 100, false); + for renewals in 0..MAX_RENEWALS { + assert_eq!(classify(own, renewals, [past].into_iter()), Decision::Defer); + } + assert_eq!(classify(own, MAX_RENEWALS, [past].into_iter()), Decision::Spare); + } + #[test] + fn a_failed_blocker_excuses_nothing() { + let own = head(9, 103, false); + let failed_in_allowance = Head { excused: true, ..failed(5, 100) }; + for renewals in 0..MAX_RENEWALS { + assert_eq!(classify(own, renewals, [failed_in_allowance].into_iter()), Decision::Defer); + } + assert_eq!(classify(own, MAX_RENEWALS, [failed_in_allowance].into_iter()), Decision::Own); + // A live blocker is chosen before a failed one, excused or not. + let others = [failed_in_allowance, excused(7, 101)]; + assert_eq!(classify(own, MAX_RENEWALS, others.into_iter()), Decision::Excuse); + } + #[test] + fn no_blocker_or_finished_work_is_own() { + assert_eq!(classify(head(9, 103, false), 0, [].into_iter()), Decision::Own); + // The oldest unfinished work is never excused, not even by later excused work. + assert_eq!(classify(head(9, 100, false), 0, [excused(5, 101)].into_iter()), Decision::Own); + assert_eq!(classify(head(9, 103, true), 0, [excused(5, 100)].into_iter()), Decision::Own); + // Same-VM work does not excuse either. + assert_eq!(classify(head(5, 103, false), MAX_RENEWALS, [excused(5, 100)].into_iter()), + Decision::Own); + } + #[test] + fn only_a_placed_long_allowance_has_an_end() { + let poll = u64::from(POLL_MS) * MS; + let mut default = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!default.renew(true, 10_000 * MS)); + assert_eq!(default.end(), None); + let mut long = Allowance::new(20_000); + assert_eq!(long.end(), None); + assert!(long.renew(true, 10_000 * MS)); + assert_eq!(long.end(), Some(10_000 * MS - poll + 20_000 * MS)); } #[test] fn same_vm_finished_and_later_work_do_not_block() { diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index f41bf829e7b7d..2cf1efb39db94 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -1154,8 +1154,13 @@ struct drm_asahi_queue_create { * it is fed. The driver does not shield the other clients from such a * queue: they share the GPU with it as the firmware schedules them, * and a command of another client may have to wait until a long - * command ends, which can be longer than that client's own allowance - * and the deferrals it gets. Userspace should ask for an allowance + * command ends. The watchdog does not fail such a waiting command, + * render or compute, while the earlier command it waits behind runs + * inside its allowance; afterwards the usual deferrals apply. A + * waiting command is excused this way for at most the longest + * allowance the system permits in total, so a command that keeps + * waiting behind other clients' work longer than that and its own + * deferrals is still failed. Userspace should ask for an allowance * only for queues whose commands need one, and keep those commands as * short as it can. */ From 772bb5b12cb1ce2974f877034aeb751c6ac53f68 Mon Sep 17 00:00:00 2001 From: Erik Bourget Date: Thu, 12 Nov 2026 22:10:06 -0500 Subject: [PATCH 8/8] drm/asahi: g17: count an allowance against the shared renewals On the G17 GPUs other than the M5, the watchdog renews a published job for up to fourteen polls while the GPU keeps completing other work (renew_shared), which bounds a job under the default allowance at thirty seconds on a shared GPU. These renewals were tried once a queue's allowance had passed, so the two added up: with the longest allowance (30 s by default) a command on a busy GPU could run for about 58 s, and a 40 s command could complete there instead of failing at about 30 s as it does on the M5, where the shared renewals do not apply. A poll that an allowance renews beyond what the default allowance renews now counts as one of the job's shared renewals. The two run from the same start, and a job is renewed for the longer of its allowance and the thirty seconds sharing gives any job, not for their sum. The one poll every allowance renews after a wait in publication is not counted, so asking never leaves a job fewer polls than the default. For an ordinary compute command that never finishes, polled every two seconds: - With an allowance of 30 s it is failed one poll after 30 s on a shared GPU or alone, on every G17, instead of after about 58 s on a busy G17 other than the M5. - With a shorter allowance on a shared GPU it is failed at thirty seconds, as a command without an allowance is. - Under the default allowance nothing changes. The renewals and deferrals that do not depend on sharing still come on top, on every G17 as before: the extra polls of ray and acceleration- structure work, and the deferrals of work waiting behind other work. A poll that runs late also still moves the estimated start, as the comment of compute_timeout_ms says; that comment now also says that the shared renewals run alongside an allowance. Tests: the timeout unit tests (KUnit and host) apply the allowance and the shared renewals in the watchdog's order to a job that never finishes: under the default allowance it fails at 2 s alone and 30 s on a shared GPU; at 30 s alone and shared with 30000; at 6 s alone and 30 s shared with 6000; at 30 s shared with 2001 and 20000; at 40 s with 40000; and after a wait in publication every allowance from the default to 30000 fails at the same poll on a shared GPU. This was not run on a G17 other than the M5 Max, where the shared renewals do not apply. Developed with Claude Code. Signed-off-by: Erik Bourget --- drivers/gpu/drm/asahi/g17/job.rs | 6 +- drivers/gpu/drm/asahi/g17/timeout.rs | 82 +++++++++++++++++++++++++++- include/uapi/drm/asahi_drm.h | 6 +- 3 files changed, 91 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index eccce9ea3dc84..4fe94f23fd597 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -427,13 +427,17 @@ impl sched::JobImpl for Job { if finished || job.packet.completion.take_replay_timeout() { return true; } - // Within the allowance its queue asked for. Past it, the deferrals below still apply. + // Within the allowance its queue asked for. Past it, the deferrals below still apply; + // the shared renewals run alongside an allowance rather than after it. + let placing = job.allowance.placing_after_wait(job.packet.is_published()); let within = job.allowance.renew(job.packet.is_published(), super::now_ns()); // Other VMs' watchdogs read the end to excuse work waiting behind this one. if let Some(end) = job.allowance.end() { job.packet.allowance_end.store(end, Ordering::Relaxed); } if within { + super::timeout::count_allowance_poll(job.allowance.asked() && !placing, + &mut job.shared_renewals); return true; } if super::timeout::renew_execution( diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index 9367b9c57efe3..e45c3ee3c626e 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -107,12 +107,23 @@ impl Allowance { now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 } + /// Whether the queue asked for an allowance longer than the default. + pub(crate) fn asked(&self) -> bool { + self.ms > DEFAULT_ALLOWANCE_MS + } + + /// Whether a [`Self::renew`] at an expiry that finds the job `published` places it after a + /// wait in publication: the one poll even the default allowance renews. + pub(crate) fn placing_after_wait(&self, published: bool) -> bool { + published && self.waited && self.head_since.is_none() + } + /// When the allowance ends (host ns), once an expiry placed the job, for an allowance /// longer than the default: until then the watchdog renews the job for the asking. A /// default allowance renews at most the poll after a wait in publication and excuses /// nobody ([`Head::excused`]). pub(crate) fn end(&self) -> Option { - let since = self.head_since.filter(|_| self.ms > DEFAULT_ALLOWANCE_MS)?; + let since = self.head_since.filter(|_| self.asked())?; Some(since.saturating_add(u64::from(self.ms) * 1_000_000)) } } @@ -171,6 +182,19 @@ pub(crate) fn renew_shared(published: bool, shared: bool, renewals: &mut u8) -> true } +/// Counts a poll that an allowance renewed beyond what the default allowance renews +/// (`beyond_default`) as one of the job's shared renewals. The two run from the same start +/// instead of one after the other: where [`renew_shared`] applies, a job is renewed for the +/// longer of its allowance and the thirty seconds sharing gives any job, not for their sum (58 s +/// with the default maximum). The one poll every allowance renews after a wait in publication +/// ([`Allowance::placing_after_wait`]) is not counted, so that asking never leaves a job fewer +/// polls than the default. +pub(crate) fn count_allowance_poll(beyond_default: bool, renewals: &mut u8) { + if beyond_default { + *renewals = renewals.saturating_add(1).min(SHARED_RENEWALS); + } +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct Head { /// The owning VM's [`crate::mmu::Vm::id`], one per VM status; the watchdog lines print it. @@ -467,6 +491,62 @@ mod tests { assert!(!renew_shared(true, true, &mut renewals)); assert_eq!(POLL_MS * (u32::from(renewals) + 1), 30_000); } + /// When an ordinary job that never finishes is failed (ms), on a GPU that is `shared` + /// throughout or not at all, by the allowance and the shared renewals in the order the + /// watchdog applies them, with no execution renewal or deferral. With `waited`, the expiry + /// at 2 s finds it still in publication (a deferral keeps it) and the next one published. + fn fails_at(allowance_ms: u32, shared: bool, waited: bool) -> u32 { + let mut allowance = Allowance::new(allowance_ms); + let mut renewals = 0; + let mut expiry = POLL_MS; + if waited { + assert!(!allowance.renew(false, u64::from(expiry) * MS)); + expiry += POLL_MS; + } + loop { + let placing = allowance.placing_after_wait(true); + if allowance.renew(true, u64::from(expiry) * MS) { + count_allowance_poll(allowance.asked() && !placing, &mut renewals); + } else if !renew_shared(true, shared, &mut renewals) { + return expiry; + } + expiry += POLL_MS; + assert!(expiry < 1_000_000); + } + } + #[test] + fn an_allowance_and_the_shared_renewals_overlap() { + // Under the default allowance: one poll alone, thirty seconds on a shared GPU. + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, false, false), 2_000); + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, true, false), 30_000); + // The longest allowance is the limit with or without sharing: not 30 s + 28 s. + assert_eq!(fails_at(30_000, false, false), 30_000); + assert_eq!(fails_at(30_000, true, false), 30_000); + // A shorter one never leaves a job worse off than the default on a shared GPU. + assert_eq!(fails_at(6_000, false, false), 6_000); + assert_eq!(fails_at(6_000, true, false), 30_000); + assert_eq!(fails_at(20_000, true, false), 30_000); + assert_eq!(fails_at(2_001, true, false), 30_000); + // Above thirty seconds (a raised maximum) the allowance is the limit. + assert_eq!(fails_at(40_000, true, false), 40_000); + // After a wait in publication every allowance is placed at the first published expiry + // (4 s), and the poll renewed there is not counted: still never fewer than the default. + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, false, true), 6_000); + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, true, true), 34_000); + for ms in [2_001, 6_000, 20_000, 30_000] { + assert_eq!(fails_at(ms, true, true), 34_000); + } + assert_eq!(fails_at(6_000, false, true), 10_000); + assert_eq!(fails_at(30_000, false, true), 34_000); + // Nothing is counted for a poll the default renews too. + let mut renewals = 0; + count_allowance_poll(false, &mut renewals); + assert_eq!(renewals, 0); + for _ in 0..40 { + count_allowance_poll(true, &mut renewals); + } + assert_eq!(renewals, SHARED_RENEWALS); + } #[test] fn a_stopped_gpu_or_unpublished_job_keeps_the_deadline() { let mut renewals = 0; diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index 2cf1efb39db94..46be28c8ed4a1 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -1147,7 +1147,11 @@ struct drm_asahi_queue_create { * - Once the allowance has passed, the deferrals the watchdog grants * any command still apply, as they do by default: a command it takes * to be waiting behind other clients' work, or one of a kind it - * gives extra polls, runs on until those are used up as well. + * gives extra polls, runs on until those are used up as well. The + * polls it renews any command for while the GPU keeps completing + * other work (up to 30 s in all, on the GPUs where it does that) + * run alongside the allowance, not after it: a command is renewed + * for the longer of the two. * * The allowance is per command, and nothing bounds the sum: a queue * that is kept full of long commands occupies the GPU for as long as