diff --git a/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats b/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats index 36b417856a03b..282d38614a091 100644 --- a/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats +++ b/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats @@ -122,6 +122,37 @@ Description: frequency. No qualified busy-time duration, power unit or temperature field is available. + M5 firmware (the M5 command set of the shared G17 backend) uses + the same snapshot and keys, with these differences: + + pstate_status, pstate + Supported. The firmware reports the performance state + in effect in tag 0x1b, words 1 and 3, which must + agree: an index 1..13 into the board's performance + state table. The end of the interval record before it + (tag 0x19, words 1 and 2) is its firmware timestamp. + A report that names state 0 or a state beyond the + table removes pstate validity. The state reads stale + while the GPU cores are gated. + + achieved_frequency_hz + The frequency of pstate in the board's performance + state table, in Hz, while pstate_status is valid. + + busy_ns Cumulative u64 since the driver was bound. The tag + 0x13 records of this firmware carry a timestamp in + 24 MHz ticks (words 1 and 2). Each record credits the + window since the previous record of its epoch at its + device utilisation. A record more than two 8 ms + periods after the previous one, or with an earlier + timestamp, only starts a new window, so time the GPU + spent gated is never credited. MEASURED on T6050 + (J716c): 871 and 863 ms for three 300 ms compute jobs + at full clock (905 and 919 ms of work). + + requested_frequency_hz, power_mw, temperature_raw + Unsupported. + The host uses a 16 ms age/averaging-window limit for V25G83 and 32 ms for V26A428 (two nominal periods). These are conservative host policies, not firmware guarantees about packet duration. diff --git a/Documentation/gpu/asahi-usage-stats.rst b/Documentation/gpu/asahi-usage-stats.rst index e18208c7d25ea..8bd90b2733d9c 100644 --- a/Documentation/gpu/asahi-usage-stats.rst +++ b/Documentation/gpu/asahi-usage-stats.rst @@ -51,11 +51,25 @@ successful stages that retired before a later failure remain counted. An unexecuted G17 replay contributes only when it eventually retires successfully. There is no submit-to-completion wall-time fallback in the current backends. -``drm-cycles-*`` and ``drm-maxfreq-*`` are intentionally absent. The timestamp -clock is independent of the variable GPU execution clock. Treating command timer -ticks as GPU busy cycles, or multiplying time by a sampled/maximum GPU frequency, -would give incorrect frequency-weighted utilization. Per-client execution-cycle -counters would require a separate hardware measurement source. +``drm-cycles-*`` is intentionally absent. The timestamp clock is independent of +the variable GPU execution clock. Treating command timer ticks as GPU busy +cycles, or multiplying time by a sampled/maximum GPU frequency, would give +incorrect frequency-weighted utilization. Per-client execution-cycle counters +would require a separate hardware measurement source. + +On the M5 each engine also has the frequency keys:: + + drm-curfreq-compute: 1620000000 Hz + drm-maxfreq-compute: 1620000000 Hz + +``drm-curfreq-`` is ``achieved_frequency_hz`` of ``agx_stats`` (see +below): the frequency of the performance state the firmware reports. It is +absent while the firmware reports none, as when the GPU is gated. +``drm-maxfreq-`` is the highest frequency in the board's table. One clock +drives the three engines, so both values are the device's, the same for every +engine and every client. They show the clock at the time of the read and are +not a weight for the engine times: without ``drm-cycles-*`` there is nothing to +scale. Other GPUs print neither key. Accounting cost --------------- @@ -66,7 +80,9 @@ per valid stage, with no new locks or allocations. The native backends reuse existing timestamp reads. Classic completion callbacks read two internal cells per stage; commands without application timestamps also execute two existing timestamp microsequence operations per stage. Reading fdinfo loads three counters -and does not traverse queues, wait for fences or access hardware. +and does not traverse queues, wait for fences or access hardware. On the M5 a +read also takes the device mutex once and drains the statistics rings, as a read +of ``agx_stats`` does, for the frequency keys. G16S and G17P device utilisation ------------------------------ @@ -109,11 +125,37 @@ ring on powered cores becomes stale rather than fabricating a zero-utilisation sample; only gated cores read as zero. Readers must not convert these percentages into engine nanoseconds or cycles. +M5 performance state and busy time +---------------------------------- + +The M5 firmware runs on the same backend and decoder, and its statistics ring +carries more. ``agx_stats`` reports there: + +* ``pstate``: the performance state in effect, from the firmware's state + reports (tag ``0x1b``, words 1 and 3, which must agree). It is an index into + the board's performance state table, 1 to 13. The interval record before a + report (tag ``0x19``) gives its firmware timestamp. A report that names state + 0 or a state beyond the table is not used, and ``pstate_status`` reads + ``stale`` while the cores are gated. +* ``achieved_frequency_hz``: the frequency of that state in the board's table, + while ``pstate_status`` is ``valid``. +* ``busy_ns``: the M5 utilisation records carry their own timestamps (words 1 + and 2, 24 MHz ticks), so each record closes a window of known length, which + is credited at the record's device utilisation. A record that follows the + previous one by more than two 8 ms periods, or whose timestamp went back, + only starts a new window: time the GPU spent gated is never credited. The + total keeps counting across power epochs. + +``requested_frequency_hz``, ``power_mw`` and ``temperature_raw`` stay +unsupported. ``busy_ns`` is device-wide, as on G13 and G14; it does not change +fdinfo's per-client ``drm-engine-*`` counters. + Deferred shared power telemetry ------------------------------ Current pstate and requested/achieved frequency remain unsupported for these -firmware graphs. The shared-power snapshot ABI is not sufficiently specified +firmware graphs (the M5 firmware reports its state in the statistics ring, see +above). The shared-power snapshot ABI is not sufficiently specified to implement a read-only reader. For **each** of G16S and G17P, it still needs: * The firmware-version-to-layout mapping, containing allocation and bounds, diff --git a/drivers/gpu/drm/asahi/asahi_core.rs b/drivers/gpu/drm/asahi/asahi_core.rs index 8dcfc3ad8ca40..6b840e524cf04 100644 --- a/drivers/gpu/drm/asahi/asahi_core.rs +++ b/drivers/gpu/drm/asahi/asahi_core.rs @@ -182,6 +182,10 @@ module! { default: 512, description: "G13/G14: GPU event-drain utilization minimum (0..1024)", }, + compute_timeout_max_ms: u32 { + default: 30000, + description: "G17: longest compute watchdog allowance a queue may set, ms (floor 2000)", + }, g15_deadline_ms: u32 { default: 5000, description: "G15: host deadline of a published prefix without progress, ms (min 100)", diff --git a/drivers/gpu/drm/asahi/driver.rs b/drivers/gpu/drm/asahi/driver.rs index b0d3af382ff7d..e301e837c5dd9 100644 --- a/drivers/gpu/drm/asahi/driver.rs +++ b/drivers/gpu/drm/asahi/driver.rs @@ -142,6 +142,8 @@ impl drm::driver::Driver for AsahiDrmDriver { ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_madvise), (ASAHI_GEM_USERPTR, drm_asahi_gem_userptr, ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_userptr), + (ASAHI_GEM_RESIDENCY, drm_asahi_gem_residency, + ioctl::AUTH | ioctl::RENDER_ALLOW, crate::file::File::gem_residency), } } diff --git a/drivers/gpu/drm/asahi/file.rs b/drivers/gpu/drm/asahi/file.rs index 9fe3de9109731..522c784017032 100644 --- a/drivers/gpu/drm/asahi/file.rs +++ b/drivers/gpu/drm/asahi/file.rs @@ -102,6 +102,10 @@ pub(crate) const DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION: u64 = /// command may wait for it in firmware. pub(crate) const DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION: u32 = uapi::drm_asahi_compute_flags_DRM_ASAHI_COMPUTE_VISIBLE_COMPLETION as u32; +/// Queues take a watchdog allowance for their compute commands +/// (`drm_asahi_queue_create::compute_timeout_ms`). +pub(crate) const DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT: u64 = + uapi::DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT as u64; /// The tiling half waits only on the render and compute prefixes of the command's stage-prefix /// fields, its fragment half on the whole ones. #[allow(dead_code)] @@ -378,6 +382,8 @@ pub(crate) struct File { /// Largest size this file has asked GET_PARAMS for the global parameters with /// ([`gpu::Gpu::admit_vm`]). params_size: core::sync::atomic::AtomicU64, + /// The device's statistics, for the GPU frequency in fdinfo. + stats: Option>, /// The queues were drained once for an exiting task. drained: AtomicBool, } @@ -401,9 +407,10 @@ impl drm::file::DriverFile for File { let user_range = gpu.user_range(); let mem = memacct::Client::new()?; let usage = Arc::new(crate::usage::Client::new(), GFP_KERNEL)?; + let stats = gpu.stats_snapshot(); mod_dev_dbg!(device, "[File {}]: DRM device opened\n", id); - KBox::pin_init(File::new(id, user_range, mem, usage), GFP_KERNEL) + KBox::pin_init(File::new(id, user_range, mem, usage, stats), GFP_KERNEL) }) } @@ -428,6 +435,11 @@ impl drm::file::DriverFile for File { } } + /// One clock drives every engine. + fn engine_frequency(&self) -> Option<(Option, u64)> { + self.stats.as_ref()?.engine_frequency() + } + fn as_raw(&self) -> *mut bindings::drm_file { todo!() } @@ -442,6 +454,7 @@ impl File { user_range: Range, mem: Arc, usage: Arc, + stats: Option>, ) -> impl PinInit { // SAFETY: The initializer supplies aligned, pinned storage for Self. // XArray::new is infallible; each pinned array is initialized in place, @@ -473,6 +486,7 @@ impl File { addr_of_mut!((*slot).mem).write(mem); addr_of_mut!((*slot).usage).write(usage); addr_of_mut!((*slot).params_size).write(core::sync::atomic::AtomicU64::new(0)); + addr_of_mut!((*slot).stats).write(stats); addr_of_mut!((*slot).drained).write(AtomicBool::new(false)); Ok(()) }) @@ -568,6 +582,9 @@ impl File { if Self::supports_user_memory(reg_data) { params.features |= uapi::DRM_ASAHI_FEATURE_USERPTR as u64; } + if Self::supports_residency_control(reg_data) { + params.features |= uapi::DRM_ASAHI_FEATURE_GEM_RESIDENCY as u64; + } let size = core::mem::size_of::().min(data.size.try_into()?); @@ -997,6 +1014,34 @@ impl File { Ok(0) } + /// Whether this GPU takes residency requests (DRM_IOCTL_ASAHI_GEM_RESIDENCY): only G17 user + /// VMs release idle memory under pressure. + fn supports_residency_control(reg_data: &driver::AsahiDrmRegistrationData<'_>) -> bool { + reg_data.gpu.as_any().is::() + } + + /// IOCTL: gem_residency: Keep an object's memory resident, or return it to the default + /// policy. + pub(crate) fn gem_residency( + _device: &AsahiDrmDevice, + reg_data: &driver::AsahiDrmRegistrationData<'_>, + data: &mut uapi::drm_asahi_gem_residency, + file: &DrmFile, + ) -> Result { + if !Self::supports_residency_control(reg_data) { + return Err(EOPNOTSUPP); + } + if data.flags & !uapi::drm_asahi_gem_residency_flags_DRM_ASAHI_GEM_RESIDENCY_KEEP != 0 { + return Err(EINVAL); + } + let gem = gem::Object::lookup_handle(file, data.handle)?; + gem::set_keep_resident( + &gem, + data.flags & uapi::drm_asahi_gem_residency_flags_DRM_ASAHI_GEM_RESIDENCY_KEEP != 0, + )?; + Ok(0) + } + /// IOCTL: gem_mmap_offset: Assign an mmap offset to a GEM object. pub(crate) fn gem_mmap_offset( device: &AsahiDrmDevice, @@ -1584,6 +1629,10 @@ impl File { data.flags, ); + if data.pad != 0 { + cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); + return Err(EINVAL); + } if data.flags != 0 || data.priority > uapi::drm_asahi_priority_DRM_ASAHI_PRIORITY_REALTIME { cls_pr_debug!(Errors, "queue_create: Invalid arguments\n"); return Err(EINVAL); @@ -1655,6 +1704,9 @@ impl File { )?; queue.reserve_render_scs(data.render_scs_layout)?; + if data.compute_timeout_ms != 0 { + data.compute_timeout_ms = queue.set_compute_timeout(data.compute_timeout_ms)?; + } data.queue_id = resv.index().try_into()?; resv.fill(Arc::pin_init( diff --git a/drivers/gpu/drm/asahi/g17/channel.rs b/drivers/gpu/drm/asahi/g17/channel.rs index d65ed18b6c786..9fc50fbad6308 100644 --- a/drivers/gpu/drm/asahi/g17/channel.rs +++ b/drivers/gpu/drm/asahi/g17/channel.rs @@ -92,11 +92,16 @@ pub(crate) struct Rings { } impl Rings { - pub(crate) fn new(graph: super::fw::variant::InitGraph) -> Result { + /// `m5`: the firmware runs the M5 protocol, whose statistics records are decoded too. + pub(crate) fn new(graph: super::fw::variant::InitGraph, m5: bool) -> Result { Ok(Self { quarantined: [[false; 2]; 2], reports: super::reports::Reports::new()?, - telemetry: super::telemetry::Telemetry::default(), + telemetry: if m5 { + super::telemetry::Telemetry::m5() + } else { + super::telemetry::Telemetry::default() + }, max_age_ns: super::stats::max_age_ns(graph), suspended: false, discard_statistics: [false; 2], diff --git a/drivers/gpu/drm/asahi/g17/job.rs b/drivers/gpu/drm/asahi/g17/job.rs index 8d2e5f443fe14..4fe94f23fd597 100644 --- a/drivers/gpu/drm/asahi/g17/job.rs +++ b/drivers/gpu/drm/asahi/g17/job.rs @@ -4,7 +4,7 @@ use super::{command::Validated, completion::Completion, context::Context}; use crate::driver::AsahiDrmDevice; -use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU8, Ordering}; +use core::sync::atomic::{AtomicBool, AtomicI32, AtomicU64, AtomicU8, Ordering}; use kernel::{ bindings, c_str, cpu::CpuId, @@ -128,6 +128,9 @@ pub(crate) struct Packet { cancelled: AtomicBool, early: AtomicI32, backlog: AtomicBool, + /// When the job's watchdog allowance ends (host ns, [`super::timeout::Allowance::end`]): + /// zero under the default allowance, `u64::MAX` for a longer one not yet placed. + allowance_end: AtomicU64, } impl Packet { @@ -160,11 +163,18 @@ impl Packet { cancelled: AtomicBool::new(false), early: AtomicI32::new(0), backlog: AtomicBool::new(false), + allowance_end: AtomicU64::new(0), }, GFP_KERNEL, )?) } + /// Its own watchdog still renews the job for the allowance its queue asked for at `now` + /// ([`super::timeout::Head::excused`]). + pub(crate) fn within_allowance(&self, now: u64) -> bool { + now < self.allowance_end.load(Ordering::Relaxed) + } + pub(crate) fn engine(&self) -> Engine { match &self.command { Validated::Render { .. } => Engine::Render, @@ -350,8 +360,12 @@ pub(crate) trait Backend: Send + Sync { fn publish(&self, packet: Arc) -> Result; fn timed_out(&self, packet: Arc) -> sched::Status; fn cancel(&self, packet: Arc); - /// Transfer a timeout to an outstanding off-lock graph transaction, if any. - fn defer_timeout(&self, packet: &Arc, renewals: u8) -> bool; + /// Transfer a timeout to an outstanding off-lock graph transaction, if any, or decide it + /// against other VMs' older work ([`super::timeout::decide`]). `renewals` and `excused` + /// count the job's earlier [`super::timeout::Decision::Defer`] and + /// [`super::timeout::Decision::Excuse`] results; the job is renewed on either. + fn defer_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> super::timeout::Decision; /// A long job of another queue is running beside `packet`. fn long_job_beside(&self, _packet: &Arc) -> bool { false } /// A published job's deadline may be renewed while other work retires @@ -365,13 +379,19 @@ pub(crate) trait Backend: Send + Sync { struct Job { backend: Arc, packet: Arc, + /// The watchdog allowance of the job ([`super::timeout::Allowance`]), in ms. + allowance_ms: u32, } struct Timeout { backend: Arc, victim_renewals: u8, + /// Polls excused behind other VMs' live work inside its allowance + /// ([`super::timeout::Decision::Excuse`]). + excused: u32, execution_renewals: u8, shared_renewals: u8, + allowance: super::timeout::Allowance, packet: Arc, } @@ -395,8 +415,10 @@ impl sched::JobImpl for Job { Timeout { backend: self.backend.clone(), victim_renewals: 0, + excused: 0, execution_renewals: 0, shared_renewals: 0, + allowance: super::timeout::Allowance::new(self.allowance_ms), packet: self.packet.clone(), } } @@ -405,6 +427,19 @@ impl sched::JobImpl for Job { if finished || job.packet.completion.take_replay_timeout() { return true; } + // Within the allowance its queue asked for. Past it, the deferrals below still apply; + // the shared renewals run alongside an allowance rather than after it. + let placing = job.allowance.placing_after_wait(job.packet.is_published()); + let within = job.allowance.renew(job.packet.is_published(), super::now_ns()); + // Other VMs' watchdogs read the end to excuse work waiting behind this one. + if let Some(end) = job.allowance.end() { + job.packet.allowance_end.store(end, Ordering::Relaxed); + } + if within { + super::timeout::count_allowance_poll(job.allowance.asked() && !placing, + &mut job.shared_renewals); + return true; + } if super::timeout::renew_execution( job.packet.is_published(), job.packet.command.long_job(), &mut job.execution_renewals, ) { @@ -418,11 +453,17 @@ impl sched::JobImpl for Job { if super::timeout::renew_shared(job.packet.is_published(), shared, &mut job.shared_renewals) { return true; } - let deferred = job.backend.defer_timeout(&job.packet, job.victim_renewals); - if deferred { - job.victim_renewals = job.victim_renewals.saturating_add(1); + match job.backend.defer_timeout(&job.packet, job.victim_renewals, job.excused) { + super::timeout::Decision::Defer => { + job.victim_renewals = job.victim_renewals.saturating_add(1); + true + } + super::timeout::Decision::Excuse => { + job.excused = job.excused.saturating_add(1); + true + } + super::timeout::Decision::Own | super::timeout::Decision::Spare => false, } - deferred } fn prepare(job: &mut Self) -> Option { @@ -457,6 +498,7 @@ pub(crate) struct Scheduler { entity: sched::Entity>, _scheduler: sched::Scheduler>, backend: Arc, + allowance_ms: u32, } #[derive(Clone)] @@ -486,7 +528,11 @@ impl Fences { } impl Scheduler { - pub(crate) fn new(dev: &AsahiDrmDevice, backend: Arc, engine: Engine) -> Result { + /// `allowance_ms`: the watchdog allowance of each job ([`super::timeout::allowance_ms`]); + /// the watchdog still polls every [`TIMEOUT_MS`]. + pub(crate) fn new(dev: &AsahiDrmDevice, backend: Arc, engine: Engine, allowance_ms: u32) + -> Result + { let (credits, name) = match engine { Engine::Render => ( RENDER_CREDITS.checked_mul(backend.render_pairs()).ok_or(EOVERFLOW)?, @@ -513,6 +559,7 @@ impl Scheduler { entity, _scheduler: scheduler, backend, + allowance_ms, }) } @@ -535,11 +582,15 @@ impl Scheduler { } else { None }; + if self.allowance_ms > super::timeout::DEFAULT_ALLOWANCE_MS { + packet.allowance_end.store(u64::MAX, Ordering::Relaxed); + } let mut job = self.entity.new_job( 1, Job { backend: self.backend.clone(), packet: packet.clone(), + allowance_ms: self.allowance_ms, }, )?; if dependencies.len() > MERGED_DEPENDENCIES { diff --git a/drivers/gpu/drm/asahi/g17/mod.rs b/drivers/gpu/drm/asahi/g17/mod.rs index 88610416beab1..e3db5fee61534 100644 --- a/drivers/gpu/drm/asahi/g17/mod.rs +++ b/drivers/gpu/drm/asahi/g17/mod.rs @@ -1818,7 +1818,10 @@ impl Gpu { )?; let stats = Arc::new( crate::stats::StatsSnapshot { - g17_reader: Some(stats::Reader(shared.clone())), + g17_reader: Some(stats::Reader( + shared.clone(), + cfg.m5.map(|board| board.perf_states), + )), ..Default::default() }, GFP_KERNEL, @@ -2036,7 +2039,7 @@ impl Gpu { shared.feed.clone(), )?, seen_submissions: 0, - rings: channel::Rings::new(shared.variant.graph)?, + rings: channel::Rings::new(shared.variant.graph, shared.variant.m5_commands)?, recovery: recovery::State::new(), effort: power::Effort::new(), hold: power::Hold::new(), @@ -2240,7 +2243,7 @@ impl Gpu { shared.feed.clone(), )?, seen_submissions: 0, - rings: channel::Rings::new(shared.variant.graph)?, + rings: channel::Rings::new(shared.variant.graph, shared.variant.m5_commands)?, recovery: recovery::State::new(), effort: power::Effort::new(), hold: power::Hold::new(), @@ -2458,7 +2461,8 @@ impl gpu::Gpu for Gpu { u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_PER_COMMAND_INPUT_SYNCS) | u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_EXACT_PRIOR_BARRIERS) | u64::from(uapi::drm_asahi_feature_DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS) - | sync_output::FEATURE_CUMULATIVE_SIGNALS; + | sync_output::FEATURE_CUMULATIVE_SIGNALS + | crate::file::DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT; let variant = self.shared.variant; // The M5 command set has no AS-build flag: its validation refuses one. if !variant.m5_commands { diff --git a/drivers/gpu/drm/asahi/g17/queue/compute.rs b/drivers/gpu/drm/asahi/g17/queue/compute.rs index 99d862a4f71ca..8042e2026451b 100644 --- a/drivers/gpu/drm/asahi/g17/queue/compute.rs +++ b/drivers/gpu/drm/asahi/g17/queue/compute.rs @@ -909,6 +909,7 @@ impl Queue { active.observed.is_some() }, failed: self.quarantined, + excused: active.packet.within_allowance(crate::g17::now_ns()), }) } diff --git a/drivers/gpu/drm/asahi/g17/queue/mod.rs b/drivers/gpu/drm/asahi/g17/queue/mod.rs index 47a506a205615..41af18adbc517 100644 --- a/drivers/gpu/drm/asahi/g17/queue/mod.rs +++ b/drivers/gpu/drm/asahi/g17/queue/mod.rs @@ -89,6 +89,9 @@ pub(crate) struct Queue { backend: Arc, render: Option>, compute: Option>, + /// The watchdog allowance of each compute command, set before the first one + /// ([`crate::queue::Queue::set_compute_timeout`]). + compute_allowance_ms: u32, fences: FenceContexts, allocation_failure: Fence, range: Range, @@ -127,7 +130,12 @@ impl Queue { window, variant, } = frontend; - let render = Some(Scheduler::new(dev, backend.clone(), Engine::Render)?); + let render = Some(Scheduler::new( + dev, + backend.clone(), + Engine::Render, + super::timeout::DEFAULT_ALLOWANCE_MS, + )?); Ok(Self { dev: dev.into(), context, @@ -135,6 +143,7 @@ impl Queue { backend, render, compute: None, + compute_allowance_ms: super::timeout::DEFAULT_ALLOWANCE_MS, fences, allocation_failure, range, @@ -158,6 +167,7 @@ impl Queue { &self.dev, self.backend.clone(), Engine::Compute, + self.compute_allowance_ms, )?); } Ok(()) @@ -513,6 +523,16 @@ impl Queue { } impl crate::queue::Queue for Queue { + fn set_compute_timeout(&mut self, requested_ms: u32) -> Result { + // The compute scheduler takes the allowance when the first compute command creates it. + if self.compute.is_some() { + return Err(EBUSY); + } + let maximum = crate::module_parameters::compute_timeout_max_ms.value(); + self.compute_allowance_ms = super::timeout::allowance_ms(requested_ms, maximum); + Ok(self.compute_allowance_ms) + } + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { let pages = [ super::fw::compute::ScratchRequest::new(layouts[0])?.pages(), diff --git a/drivers/gpu/drm/asahi/g17/queue/render.rs b/drivers/gpu/drm/asahi/g17/queue/render.rs index 16e9d0e265571..c68448f4503c3 100644 --- a/drivers/gpu/drm/asahi/g17/queue/render.rs +++ b/drivers/gpu/drm/asahi/g17/queue/render.rs @@ -1667,6 +1667,7 @@ impl Pair { published: active.published, finished: false, failed: self.quarantined, + excused: active.packet.within_allowance(crate::g17::now_ns()), }) } diff --git a/drivers/gpu/drm/asahi/g17/runtime/backend.rs b/drivers/gpu/drm/asahi/g17/runtime/backend.rs index 705325bf6d153..b17fe9eb07d11 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/backend.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/backend.rs @@ -386,19 +386,20 @@ impl job::Backend for Backend { .any(|other| other.owner() != Some(self.owner) && other.long_job_running()) } - fn defer_timeout(&self, packet: &Arc, renewals: u8) -> bool { + fn defer_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> crate::g17::timeout::Decision { if packet.engine() == Engine::Compute { - return self.defer_compute_timeout(packet, renewals); + return self.defer_compute_timeout(packet, renewals, excused); } use crate::g17::timeout::{self, Decision}; // As for compute, a final retirement pass precedes the attribution, and all completion // callbacks run after releasing the device mutex. Without a collector the attribution // goes ahead without that pass, as it did before it existed. let mut deferred = DeferredBatch::worker().ok(); - let result = (|| -> Result { + let result = (|| -> Result { let mut state = self.shared.state.lock(); let Some(firmware) = (*state).as_deref_mut() else { - return Ok(false); + return Ok(Decision::Own); }; if let Some(deferred) = deferred.as_mut() { firmware.scan_renders(deferred)?; @@ -409,15 +410,17 @@ impl job::Backend for Backend { "Render watchdog owner={} behind its context's older node {:#x} (own {:#x}); the firmware progresses: renewed\n", self.owner, older, own); } - return Ok(true); + return Ok(Decision::Defer); } // Borrowed-graph and queued-retirement deferrals share the renewal bound. if timeout::transaction_grace(renewals) && firmware.queues.defer_render_timeout(packet, ETIMEDOUT) { - return Ok(true); + return Ok(Decision::Defer); } - let verdict = firmware.queues.render_timeout_decision(packet, renewals); + let excuse = timeout::Excuse { used: excused, limit: timeout::excuse_limit( + crate::module_parameters::compute_timeout_max_ms.value()) }; + let verdict = firmware.queues.render_timeout_decision(packet, renewals, excuse); let decision = verdict.map_or(Decision::Own, |verdict| verdict.decision); // As compute does: work still blocked behind another VM's older unfinished work // after the whole grace is innocent, so its failure must stay non-sticky. The pair's @@ -426,19 +429,20 @@ impl job::Backend for Backend { packet.completion.classify(crate::g17::recovery::Class::Spared); } if let Some(verdict) = verdict { - if let Some(blocker) = verdict.reported_blocker(renewals) { + if let Some(blocker) = verdict.reported_blocker(renewals, excuse) { dev_warn!(self.shared.dev.as_ref(), - "Render watchdog owner={} vm={} publication={} renewals={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={}\n", - self.owner, verdict.own.vm, verdict.own.published, renewals, decision, - blocker.vm, blocker.published, u8::from(blocker.failed)); + "Render watchdog owner={} vm={} publication={} renewals={} excused={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={} blocker_in_allowance={}\n", + self.owner, verdict.own.vm, verdict.own.published, renewals, excused, + decision, blocker.vm, blocker.published, u8::from(blocker.failed), + u8::from(blocker.excused)); } } - Ok(decision == Decision::Defer) + Ok(decision) })(); if let Some(deferred) = deferred.as_mut() { deferred.finish(); } - matches!(result, Ok(true)) + result.unwrap_or(Decision::Own) } fn try_early(&self, packet: &Arc) -> bool { diff --git a/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs index dfa343a9ac275..922f4340aae3f 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/compute_control.rs @@ -9,10 +9,11 @@ use kernel::{prelude::*, sync::Arc}; impl Backend { /// A final retirement pass precedes bounded cross-VM timeout attribution. /// All completion callbacks run after releasing the device mutex. - pub(super) fn defer_compute_timeout(&self, packet: &Arc, renewals: u8) -> bool { + pub(super) fn defer_compute_timeout(&self, packet: &Arc, renewals: u8, excused: u32) + -> crate::g17::timeout::Decision { use crate::g17::{recovery::Class, timeout::{self, Decision}}; let Ok(mut deferred) = DeferredBatch::worker() else { - return false; + return Decision::Own; }; let result = (|| -> Result { let mut state = self.shared.state.lock(); @@ -39,22 +40,25 @@ impl Backend { .filter(|other| other.owner() != Some(self.owner)) .filter_map(|other| other.timeout_head()); let others = Iterator::chain(computes, firmware.queues.render_timeout_heads()); - let verdict = timeout::attribute(own, renewals, others); + let excuse = timeout::Excuse { used: excused, limit: timeout::excuse_limit( + crate::module_parameters::compute_timeout_max_ms.value()) }; + let verdict = timeout::attribute(own, renewals, excuse, others); let decision = verdict.decision; if decision == Decision::Spare { firmware.queues.compute(self.owner).ok_or(EIO)? .classify(&|_| Class::Spared); } - if let Some(blocker) = verdict.reported_blocker(renewals) { + if let Some(blocker) = verdict.reported_blocker(renewals, excuse) { dev_warn!(self.shared.dev.as_ref(), - "Compute watchdog owner={} vm={} publication={} renewals={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={}\n", - self.owner, own.vm, own.published, renewals, decision, - blocker.vm, blocker.published, u8::from(blocker.failed)); + "Compute watchdog owner={} vm={} publication={} renewals={} excused={} decision={:?} behind earlier work of another VM: blocker_vm={} blocker_publication={} blocker_failed={} blocker_in_allowance={}\n", + self.owner, own.vm, own.published, renewals, excused, decision, + blocker.vm, blocker.published, u8::from(blocker.failed), + u8::from(blocker.excused)); } Ok(decision) })(); deferred.finish(); - matches!(result, Ok(Decision::Defer)) + result.unwrap_or(Decision::Own) } /// The older node of the packet's context ahead of its own, with its own word, when the diff --git a/drivers/gpu/drm/asahi/g17/runtime/render.rs b/drivers/gpu/drm/asahi/g17/runtime/render.rs index 4dac5f5230237..0218e48da42b3 100644 --- a/drivers/gpu/drm/asahi/g17/runtime/render.rs +++ b/drivers/gpu/drm/asahi/g17/runtime/render.rs @@ -1271,6 +1271,7 @@ impl Registry { &self, packet: &Arc, renewals: u8, + excuse: crate::g17::timeout::Excuse, ) -> Option { let own = self .render @@ -1291,6 +1292,7 @@ impl Registry { crate::g17::timeout::attribute( own, renewals, + excuse, Iterator::chain(self.render_timeout_heads(), computes), ) }) diff --git a/drivers/gpu/drm/asahi/g17/stats.rs b/drivers/gpu/drm/asahi/g17/stats.rs index 887d966a04de3..40688ccab0c8e 100644 --- a/drivers/gpu/drm/asahi/g17/stats.rs +++ b/drivers/gpu/drm/asahi/g17/stats.rs @@ -7,13 +7,42 @@ use super::{fw::variant::InitGraph, telemetry::Readout, Shared}; use core::sync::atomic::Ordering; use kernel::sync::Arc; -pub(crate) struct Reader(pub(super) Arc); +/// The device's shared state, and the M5 board's performance states, whose frequency a +/// reported state index runs at. +pub(crate) struct Reader( + pub(super) Arc, + pub(super) Option<&'static [super::fw::m5::initdata::M5PerfState; 14]>, +); impl Reader { pub(crate) fn read(&self) -> Readout { + let mut view = self.read_firmware(); + if view.pstate_status == super::telemetry::VALID { + view.achieved_frequency_hz = self + .1 + .and_then(|states| states.get(view.pstate as usize)) + .map_or(0, |state| u64::from(state.0) * 1_000_000); + } + view + } + + /// The frequency of the state the GPU runs at, while the firmware reports one, and the + /// highest frequency of the board's states, in Hz; `None` without the board's performance + /// states. + pub(crate) fn frequency(&self) -> Option<(Option, u64)> { + let max = u64::from(self.1?.iter().map(|state| state.0).max()?) * 1_000_000; + let view = self.read(); + let current = (view.pstate_status == super::telemetry::VALID + && view.achieved_frequency_hz != 0) + .then_some(view.achieved_frequency_hz); + Some((current, max)) + } + + fn read_firmware(&self) -> Readout { + let m5 = self.0.variant.m5_commands; let mut state = self.0.state.lock(); let Some(firmware) = state.as_deref_mut() else { - return qualify(self.0.variant.graph, Readout::default()); + return qualify(self.0.variant.graph, m5, Readout::default()); }; if self.0.crashed.load(Ordering::Acquire) || firmware.recovery.pending() { firmware.rings.invalidate_statistics(); @@ -34,6 +63,7 @@ impl Reader { } qualify( self.0.variant.graph, + m5, firmware.rings.read_statistics(super::now_ns()), ) } @@ -55,13 +85,15 @@ pub(super) const fn max_age_ns(graph: InitGraph) -> u64 { } } -/// Whether the firmware emits the logical pstate record (tag 0x10). V25G83 -/// and V26A428 accept the tag but never emit it, and no other record carries -/// the current pstate, so it is unsupported by these firmwares. A firmware -/// that is shown to emit it gets `true` here and reports valid or stale. -pub(super) const fn reports_pstate(graph: InitGraph) -> bool { +/// Whether the firmware reports its performance state. V25G83 and V26A428 +/// accept the logical pstate record (tag 0x10) but never emit it, and no other +/// record of theirs carries the current pstate, so it is unsupported by these +/// firmwares. M5 firmware (`m5`) reports it in its own records +/// ([`super::telemetry`]). A firmware that is shown to report it gets `true` +/// here and reports valid or stale. +pub(super) const fn reports_pstate(graph: InitGraph, m5: bool) -> bool { match graph { - InitGraph::V25G83 | InitGraph::G17P => false, + InitGraph::V25G83 | InitGraph::G17P => m5, } } @@ -69,9 +101,10 @@ pub(super) const fn reports_pstate(graph: InitGraph) -> bool { /// and the debugfs diagnostic still keep any tag 0x10 that does arrive. pub(super) fn qualify( graph: InitGraph, + m5: bool, mut view: super::telemetry::Readout, ) -> super::telemetry::Readout { - if !reports_pstate(graph) { + if !reports_pstate(graph, m5) { view.pstate_status = super::telemetry::UNSUPPORTED; view.pstate = 0; view.pstate_host_ns = 0; diff --git a/drivers/gpu/drm/asahi/g17/telemetry.rs b/drivers/gpu/drm/asahi/g17/telemetry.rs index e57f91b5bea02..27d8bef8c5383 100644 --- a/drivers/gpu/drm/asahi/g17/telemetry.rs +++ b/drivers/gpu/drm/asahi/g17/telemetry.rs @@ -1,9 +1,13 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT //! Qualified V25G83/V26A428 statistics, decoded from complete copied records. -//! No record measures busy time, power, temperature or achieved frequency. -//! A/B table selection is absent, so neither nominal candidate is exported as -//! a requested or achieved frequency. All state is protected by the device mutex. +//! No record of these firmwares measures busy time, power, temperature or achieved +//! frequency. A/B table selection is absent, so neither nominal candidate is exported +//! as a requested or achieved frequency. +//! +//! M5 firmware reports the performance state the GPU runs at (tags 0x19 and 0x1b), and +//! its utilisation records carry their own timestamps, from which busy time is +//! integrated. All state is protected by the device mutex. pub(super) const UNAVAILABLE: u32 = 0; pub(super) const VALID: u32 = 1; @@ -26,13 +30,31 @@ pub(crate) struct Readout { pub(crate) pstate_host_ns: u64, pub(crate) util_samples: u64, pub(crate) pstate_timestamp: u64, + /// Utilisation-weighted busy time since the device was created, while `busy_status` is + /// [`VALID`] (M5 only). + pub(crate) busy_ns: u64, + /// Frequency of `pstate` while `pstate_status` is [`VALID`], from the board's performance + /// state table ([`super::stats::Reader`]); zero where the table is unknown. + pub(crate) achieved_frequency_hz: u64, + pub(crate) busy_status: u32, + pub(crate) _pad: u32, } const _: () = { assert!(core::mem::offset_of!(Readout, util_host_ns) == 24); - assert!(core::mem::size_of::() == 56); + assert!(core::mem::offset_of!(Readout, busy_ns) == 56); + assert!(core::mem::size_of::() == 80); }; +/// Interval of the M5 utilisation record (tag 0x13), in timer ticks: 8 ms at 24 MHz. +const M5_UTIL_PERIOD_TICKS: u64 = 192_000; +/// A utilisation record more than two intervals after the previous one follows a gap +/// the firmware did not report (gated or lost records): it only sets the baseline. +const M5_UTIL_MAX_WINDOW_TICKS: u64 = 2 * M5_UTIL_PERIOD_TICKS; +/// Highest M5 performance state: the device tree's operating points 1 to 13 (0 is off). +const M5_MAX_PSTATE: u32 = 13; +const TIMER_HZ: u64 = 24_000_000; + #[derive(Default)] pub(super) struct Telemetry { view: Readout, @@ -49,9 +71,25 @@ pub(super) struct Telemetry { /// gates them a few milliseconds after the last work completes and emits no /// utilisation record while gated, so a gated GPU is idle, not unknown. gated: bool, + /// The M5 firmware's records are decoded ([`Self::m5`]). + m5: bool, + /// Timestamp of the latest M5 utilisation record in this epoch: the start of the window + /// the next one closes. + busy_ts: Option, + busy_ns: u64, + /// End of the latest M5 performance state interval (tag 0x19), in timer ticks. + pstate_interval_end: Option, } impl Telemetry { + /// Decoder for M5 firmware: its performance state and busy time records. + pub(super) fn m5() -> Self { + Self { + m5: true, + ..Self::default() + } + } + /// End an epoch. Preserve the reason that formerly valid data is stale, /// but never carry an average or a timestamp baseline into a new epoch. pub(super) fn invalidate(&mut self) { @@ -67,6 +105,9 @@ impl Telemetry { self.delivery_ns = None; self.delivery_intervals = [None; 2]; self.gated = false; + // Busy time keeps counting across epochs; only the window start is lost. + self.busy_ts = None; + self.pstate_interval_end = None; } /// The cores were observed power-gated. End the averaging epoch exactly as @@ -129,6 +170,12 @@ impl Telemetry { for (sum, value) in self.sums.iter_mut().zip(values) { *sum += u64::from(value); } + if self.m5 { + self.integrate_busy( + u64::from(words[1]) | (u64::from(words[2]) << 32), + words[6], + ); + } self.view.util_samples += 1; self.view.util_host_ns = host_ns; self.view.util_status = VALID; @@ -150,10 +197,45 @@ impl Telemetry { self.view.pstate_status = if words[6] <= 10 { VALID } else { UNAVAILABLE }; self.view.pstate = words[6]; } + // M5: the interval [words 3-4, words 1-2] that just ended, run at the state in + // word 8. Its end timestamps the state report that follows it. + 0x19 if self.m5 => { + self.pstate_interval_end = Some(u64::from(words[1]) | (u64::from(words[2]) << 32)); + } + // M5: the performance state now in effect, an index into the device tree's + // operating points, in words 1 and 3. A report whose two words differ is not + // used. + 0x1b if self.m5 => { + let pstate = words[1]; + if pstate != words[3] || pstate == 0 || pstate > M5_MAX_PSTATE { + self.view.pstate_status = UNAVAILABLE; + return; + } + self.view.pstate = pstate; + self.view.pstate_status = VALID; + self.view.pstate_host_ns = host_ns; + self.view.pstate_timestamp = self.pstate_interval_end.unwrap_or(0); + } _ => {} } } + /// Credits the window an M5 utilisation record closes, at its device utilisation. The + /// window starts at the previous record of the epoch; a record after a longer gap than + /// [`M5_UTIL_MAX_WINDOW_TICKS`] (the GPU was gated or records were lost) or one whose + /// timestamp went back only sets the baseline, so an idle gap is never credited. + fn integrate_busy(&mut self, timestamp: u64, device: u32) { + if let Some(last) = self.busy_ts { + let window = timestamp.wrapping_sub(last); + if timestamp > last && window <= M5_UTIL_MAX_WINDOW_TICKS { + // At most 384000 ticks x 100: no overflow. + let ns = window * u64::from(device) * 1_000_000_000 / (100 * TIMER_HZ); + self.busy_ns = self.busy_ns.saturating_add(ns); + } + } + self.busy_ts = Some(timestamp); + } + pub(super) fn read(&self, now_ns: u64, max_age_ns: u64) -> Readout { let mut view = self.view; let limit = max_age_ns.saturating_add(self.delivery_slack()); @@ -173,6 +255,12 @@ impl Telemetry { // A pstate report describes a state transition, with no qualified // cadence. Its validity ends on replacement or epoch invalidation, // rather than an invented periodic-report deadline. + if self.m5 { + view.busy_status = VALID; + view.busy_ns = self.busy_ns; + } else { + view.busy_status = UNSUPPORTED; + } view } } @@ -201,7 +289,7 @@ mod tests { fn spec_u1_u2_u3_both_firmware_policies() { for age in [AGE, 2 * AGE] { let mut stats = Telemetry::default(); - assert_eq!(stats.read(0, age), Readout::default()); + assert_eq!(stats.read(0, age), NOTHING); stats.record(&U1, 100, age); let view = stats.read(100, age); assert_eq!((view.tiler, view.renderer, view.device), (25, 75, 50)); @@ -358,6 +446,126 @@ mod tests { words[0] = tag; stats.record(&words, 1, AGE); } - assert_eq!(stats.read(1, AGE), Readout::default()); + assert_eq!(stats.read(1, AGE), NOTHING); + } + + /// M5 firmware records: the utilisation of three 8 ms windows of a full-GPU compute + /// job, a performance state interval and the state report after it. + const M5_U: [[u32; 18]; 3] = [ + [ + 0x13, 0x433a6047, 0x5c, 0, 0, 0x64, 0x64, 0x36, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, + ], + [ + 0x13, 0x433d4e4f, 0x5c, 0, 0, 0x64, 0x64, 0x36, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, + ], + [ + 0x13, 0x43403c9b, 0x5c, 0, 0, 0x64, 0x64, 0x36, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, + ], + ]; + const M5_INTERVAL: [u32; 18] = [ + 0x19, 0x3c042f8f, 0x59, 0x3c040ecf, 0x59, 1, 0, 9, 0xd, 0xd, 9, 0x514, 0, 6, 0, 0, 0, 0, + ]; + const M5_STATE: [u32; 18] = [ + 0x1b, 0xd, 0x7f, 0xd, 0x7f, 0, 1, 9, 0, 0, 9, 0x514, 0, 6, 0, 0, 0, 0, + ]; + const NOTHING: Readout = Readout { + util_status: UNAVAILABLE, + pstate_status: UNAVAILABLE, + tiler: 0, + renderer: 0, + device: 0, + pstate: 0, + util_host_ns: 0, + pstate_host_ns: 0, + util_samples: 0, + pstate_timestamp: 0, + busy_ns: 0, + achieved_frequency_hz: 0, + busy_status: UNSUPPORTED, + _pad: 0, + }; + + fn ticks_ns(ticks: u64) -> u64 { + ticks * 1_000_000_000 / 24_000_000 + } + + #[test] + fn m5_state_reports_give_the_pstate() { + let mut stats = Telemetry::m5(); + stats.record(&M5_INTERVAL, 100, AGE); + assert_eq!(stats.read(100, AGE).pstate_status, UNAVAILABLE); + stats.record(&M5_STATE, 101, AGE); + let view = stats.read(101, AGE); + assert_eq!((view.pstate_status, view.pstate), (VALID, 13)); + assert_eq!(view.pstate_timestamp, 0x59_3c04_2f8f); + assert_eq!(view.pstate_host_ns, 101); + let mut low = M5_STATE; + (low[1], low[3]) = (1, 1); + stats.record(&low, 102, AGE); + assert_eq!(stats.read(102, AGE).pstate, 1); + // Two different states in one report, off, or beyond the table: not a state. + for (a, b) in [(1, 13), (0, 0), (14, 14)] { + let mut words = M5_STATE; + (words[1], words[3]) = (a, b); + stats.record(&words, 103, AGE); + assert_eq!(stats.read(103, AGE).pstate_status, UNAVAILABLE); + } + // Gating ends the epoch: the state is no longer current. + stats.record(&M5_STATE, 104, AGE); + stats.gate(); + assert_eq!(stats.read(105, AGE).pstate_status, STALE); + } + + #[test] + fn m5_busy_time_integrates_utilisation_windows() { + let mut stats = Telemetry::m5(); + assert_eq!( + (stats.read(0, AGE).busy_status, stats.read(0, AGE).busy_ns), + (VALID, 0) + ); + // The first record only sets the baseline; the next two close one window each. + for (n, words) in M5_U.iter().enumerate() { + stats.record(words, 100 + n as u64, AGE); + } + // Each window is rounded down to whole nanoseconds on its own. + let windows = ticks_ns(u64::from(M5_U[1][1] - M5_U[0][1])) + + ticks_ns(u64::from(M5_U[2][1] - M5_U[1][1])); + assert_eq!(stats.read(102, AGE).busy_ns, windows); + assert!((15_900_000..16_100_000).contains(&stats.read(102, AGE).busy_ns)); + // A window is credited at its device utilisation. + let mut half = M5_U[2]; + half[1] += 192_000; + half[6] = 50; + stats.record(&half, 103, AGE); + assert_eq!(stats.read(103, AGE).busy_ns, windows + 4_000_000); + // A gate, a long gap or a timestamp going back only sets a new baseline. + let before = stats.read(103, AGE).busy_ns; + stats.gate(); + let mut after = half; + after[1] += 192_000; + stats.record(&after, 104, AGE); + assert_eq!(stats.read(104, AGE).busy_ns, before); + after[1] += 3 * 192_000; + stats.record(&after, 105, AGE); + assert_eq!(stats.read(105, AGE).busy_ns, before); + after[1] -= 1000; + stats.record(&after, 106, AGE); + assert_eq!(stats.read(106, AGE).busy_ns, before); + // An epoch boundary keeps the total. + stats.invalidate(); + assert_eq!(stats.read(107, AGE).busy_ns, before); + } + + #[test] + fn other_firmwares_ignore_the_m5_records() { + let mut stats = Telemetry::default(); + for words in [M5_INTERVAL, M5_STATE, M5_U[0], M5_U[1]] { + stats.record(&words, 1, AGE); + } + let view = stats.read(1, AGE); + assert_eq!( + (view.pstate_status, view.busy_status, view.busy_ns), + (UNAVAILABLE, UNSUPPORTED, 0) + ); } } diff --git a/drivers/gpu/drm/asahi/g17/timeout.rs b/drivers/gpu/drm/asahi/g17/timeout.rs index 3da5e255408b5..e45c3ee3c626e 100644 --- a/drivers/gpu/drm/asahi/g17/timeout.rs +++ b/drivers/gpu/drm/asahi/g17/timeout.rs @@ -3,6 +3,9 @@ //! Bounded isolation for a compute watchdog blocked behind another VM's older kick. //! Publication order is evidence of a possible blocker, not proof of a firmware //! dependency. Renewals are bounded; the oldest unfinished queue is never excused. +//! A blocker that runs inside the watchdog allowance its queue asked for excuses +//! the work behind it for that long ([`decide`]), bounded by the longest allowance +//! the system permits ([`excuse_limit`]). pub(crate) const MAX_RENEWALS: u8 = 8; @@ -20,8 +23,8 @@ const EXECUTION_RENEWALS: u8 = 4; /// How long an innocent client's publication waits for a recovery to settle and its /// spared work to drain, instead of failing with EBUSY: the watchdog defers work behind -/// another VM at most this long before it settles it (`classify`; ray work gets the -/// execution renewals on top). +/// another VM's failed work at most this long before it settles it (`classify`; ray work +/// gets the execution renewals on top). A failed blocker excuses nothing ([`decide`]). pub(crate) const RECOVERY_WAIT_MS: u32 = POLL_MS * (1 + EXECUTION_RENEWALS as u32 + MAX_RENEWALS as u32); @@ -52,6 +55,94 @@ pub(crate) fn as_build(flags: u32) -> Option { if flags & !AS_BUILD != 0 { None } else { Some(flags & AS_BUILD != 0) } } +/// The watchdog allowance of a compute command unless its queue asked for +/// another (`drm_asahi_queue_create::compute_timeout_ms`): one watchdog poll. +pub(crate) const DEFAULT_ALLOWANCE_MS: u32 = POLL_MS; + +/// The allowance a queue gets for its compute commands when it asks for +/// `requested_ms`, given the longest the system allows (`maximum_ms`, the +/// `compute_timeout_max_ms` module parameter): never below the default. +pub(crate) fn allowance_ms(requested_ms: u32, maximum_ms: u32) -> u32 { + requested_ms.min(maximum_ms).max(DEFAULT_ALLOWANCE_MS) +} + +/// A job's watchdog allowance: how long the watchdog renews it for the asking, +/// counted from about when it reached the head of its queue. +/// +/// It is not a deadline. The watchdog only sees a job at its expiries, and +/// does not know when the job started: the scheduler arms a queue's watchdog +/// when a job reaches the head (the job ahead of it completed) and again +/// whenever a later job of the queue is pushed, and the watchdog expires one +/// [`POLL_MS`] later. So a job's first expiry comes at least one poll after it +/// reached the head, later if pushes postponed it, and the time is counted +/// from one poll before that expiry. A job still in publication at an expiry +/// (waiting for capacity) has not started: its time counts from the first +/// expiry that finds it published. Either way a job never loses time to the +/// jobs ahead of it, and it may get up to one poll more than its allowance. +/// Under the default allowance it is never renewed. A job past its allowance +/// is not failed for that alone: the caller's other renewals and deferrals +/// are tried next. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Allowance { + ms: u32, + /// When the job reached the head, once an expiry placed it. + head_since: Option, + /// An expiry found the job still in publication. + waited: bool, +} + +impl Allowance { + pub(crate) const fn new(ms: u32) -> Self { + Self { ms, head_since: None, waited: false } + } + + /// Whether the job is within its allowance at the expiry at `now` (host ns). + pub(crate) fn renew(&mut self, published: bool, now: u64) -> bool { + if !published { + self.waited = true; + return false; + } + let poll = if self.waited { 0 } else { u64::from(POLL_MS) * 1_000_000 }; + let since = *self.head_since.get_or_insert(now.saturating_sub(poll)); + now.saturating_sub(since) < u64::from(self.ms) * 1_000_000 + } + + /// Whether the queue asked for an allowance longer than the default. + pub(crate) fn asked(&self) -> bool { + self.ms > DEFAULT_ALLOWANCE_MS + } + + /// Whether a [`Self::renew`] at an expiry that finds the job `published` places it after a + /// wait in publication: the one poll even the default allowance renews. + pub(crate) fn placing_after_wait(&self, published: bool) -> bool { + published && self.waited && self.head_since.is_none() + } + + /// When the allowance ends (host ns), once an expiry placed the job, for an allowance + /// longer than the default: until then the watchdog renews the job for the asking. A + /// default allowance renews at most the poll after a wait in publication and excuses + /// nobody ([`Head::excused`]). + pub(crate) fn end(&self) -> Option { + let since = self.head_since.filter(|_| self.asked())?; + Some(since.saturating_add(u64::from(self.ms) * 1_000_000)) + } +} + +/// How many polls a job may be excused behind other VMs' live work inside those blockers' +/// allowances ([`decide`]): the longest allowance a queue may get (`maximum_ms`, the +/// `compute_timeout_max_ms` module parameter) in whole polls, 15 by default. +pub(crate) fn excuse_limit(maximum_ms: u32) -> u32 { + allowance_ms(u32::MAX, maximum_ms).div_ceil(POLL_MS) +} + +/// The polls a waiting job was excused so far (`used`) and the most it may be (`limit`, +/// [`excuse_limit`]). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Excuse { + pub(crate) used: u32, + pub(crate) limit: u32, +} + pub(crate) fn renew_execution(published: bool, enabled: bool, renewals: &mut u8) -> bool { if !published || !enabled || *renewals >= EXECUTION_RENEWALS { return false; @@ -91,6 +182,19 @@ pub(crate) fn renew_shared(published: bool, shared: bool, renewals: &mut u8) -> true } +/// Counts a poll that an allowance renewed beyond what the default allowance renews +/// (`beyond_default`) as one of the job's shared renewals. The two run from the same start +/// instead of one after the other: where [`renew_shared`] applies, a job is renewed for the +/// longer of its allowance and the thirty seconds sharing gives any job, not for their sum (58 s +/// with the default maximum). The one poll every allowance renews after a wait in publication +/// ([`Allowance::placing_after_wait`]) is not counted, so that asking never leaves a job fewer +/// polls than the default. +pub(crate) fn count_allowance_poll(beyond_default: bool, renewals: &mut u8) { + if beyond_default { + *renewals = renewals.saturating_add(1).min(SHARED_RENEWALS); + } +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct Head { /// The owning VM's [`crate::mmu::Vm::id`], one per VM status; the watchdog lines print it. @@ -100,12 +204,19 @@ pub(crate) struct Head { /// Its queue already failed (quarantined) and holds the GPU only until the firmware drains /// it. No later watchdog of its own will settle it. pub(crate) failed: bool, + /// Its own watchdog still renews it for the allowance its queue asked for + /// ([`Allowance::end`]): a long command running as allowed, not a hung one. + pub(crate) excused: bool, } #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) enum Decision { Own, + /// Renewed: counts against [`MAX_RENEWALS`]. Defer, + /// Renewed behind a live blocker inside its allowance: counts against [`Excuse::limit`] + /// only. + Excuse, Spare, } @@ -123,10 +234,27 @@ pub(crate) fn blocker(own: Head, others: impl Iterator) -> Option) -> Decision { +/// +/// A live blocker that its own watchdog still renews for its queue's allowance +/// ([`Head::excused`]) is not hung: its command may run that long, and work queued behind it +/// on the GPU waits for it. Such work is renewed without using any more of its MAX_RENEWALS, +/// so that what is left of that grace covers the blocker's own failure and teardown once the +/// allowance has passed; then the rules above apply again. +/// +/// The excuse is bounded by the waiting job's [`Excuse::limit`], the longest allowance the +/// system permits in polls, however many blockers take turns. This attribution therefore +/// renews a job at most `limit + MAX_RENEWALS` times: 23 with the default 30 s maximum, 46 s +/// of renewals after the expiry that first deferred it. That bounds this rule only. A job's +/// own allowance and its other deferrals ([`renew_execution`], [`renew_shared`]) are counted +/// separately and come on top, and each renewal re-arms a timer rather than an absolute +/// deadline, so the time until the job is failed (spared) can exceed 46 s. +pub(crate) fn decide(own: Head, renewals: u8, excuse: Excuse, blocker: Option) -> Decision { match blocker { _ if own.finished => Decision::Own, None => Decision::Own, + Some(blocker) if blocker.excused && !blocker.failed && excuse.used < excuse.limit => { + Decision::Excuse + } Some(_) if renewals < MAX_RENEWALS => Decision::Defer, Some(blocker) if blocker.failed => Decision::Own, Some(_) => Decision::Spare, @@ -142,16 +270,22 @@ pub(crate) struct Verdict { } impl Verdict { - /// The blocker a watchdog line names: at the first deferral and once the grace has run out. - pub(crate) fn reported_blocker(&self, renewals: u8) -> Option { - self.blocker.filter(|_| renewals == 0 || self.decision != Decision::Defer) + /// The blocker a watchdog line names: at the first excuse, at the first counted deferral + /// and once the grace has run out. + pub(crate) fn reported_blocker(&self, renewals: u8, excuse: Excuse) -> Option { + self.blocker.filter(|_| match self.decision { + Decision::Excuse => excuse.used == 0, + Decision::Defer => renewals == 0, + Decision::Own | Decision::Spare => true, + }) } } -pub(crate) fn attribute(own: Head, renewals: u8, others: impl Iterator) -> Verdict { +pub(crate) fn attribute(own: Head, renewals: u8, excuse: Excuse, + others: impl Iterator) -> Verdict { let blocker = blocker(own, others); Verdict { - decision: decide(own, renewals, blocker), + decision: decide(own, renewals, excuse, blocker), own, blocker, } @@ -201,6 +335,54 @@ pub(crate) fn behind_own_head(published: bool, older: Option, progressing: #[cfg_attr(not(test), kernel::macros::kunit_tests(asahi_g17_execution_allowance))] mod tests { use super::*; + const MS: u64 = 1_000_000; + #[test] + fn a_queue_gets_the_allowance_it_asks_for_within_bounds() { + assert_eq!(allowance_ms(0, 30_000), DEFAULT_ALLOWANCE_MS); + assert_eq!(allowance_ms(500, 30_000), DEFAULT_ALLOWANCE_MS); + assert_eq!(allowance_ms(20_000, 30_000), 20_000); + assert_eq!(allowance_ms(u32::MAX, 30_000), 30_000); + // A maximum below the default still leaves the default. + assert_eq!(allowance_ms(20_000, 0), DEFAULT_ALLOWANCE_MS); + } + #[test] + fn the_default_allowance_renews_nothing() { + let mut allowance = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!allowance.renew(true, 100_000 * MS)); + // Not after a wait in publication either. + let mut allowance = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!allowance.renew(false, 100_000 * MS)); + assert!(allowance.renew(true, 102_000 * MS)); + assert!(!allowance.renew(true, 104_000 * MS)); + } + #[test] + fn a_long_allowance_runs_from_the_head_of_the_queue() { + // Reached the head at 10 s (the job ahead completed): first expiry at 12 s. + let start = 10_000 * MS; + let poll = u64::from(POLL_MS) * MS; + let mut allowance = Allowance::new(30_000); + let mut expiry = start + poll; + while allowance.renew(true, expiry) { + expiry += poll; + } + assert_eq!(allowance.head_since, Some(start)); + // Killed at the first poll at or past 30 s after it reached the head. + assert_eq!(expiry - start, 30_000 * MS); + // A first expiry delayed by later pushes only moves the count later. + let mut allowance = Allowance::new(30_000); + assert!(allowance.renew(true, start + poll + 500 * MS)); + assert_eq!(allowance.head_since, Some(start + 500 * MS)); + } + #[test] + fn a_job_waiting_in_publication_starts_at_its_first_published_expiry() { + let mut allowance = Allowance::new(30_000); + assert!(!allowance.renew(false, 10_000 * MS)); + assert_eq!(allowance.head_since, None); + assert!(allowance.renew(true, 12_000 * MS)); + assert_eq!(allowance.head_since, Some(12_000 * MS)); + assert!(allowance.renew(true, 41_999 * MS)); + assert!(!allowance.renew(true, 42_000 * MS)); + } #[test] fn selects_compute_both_render_halves_and_builds() { assert!(!select(false, [false; 2], false)); @@ -250,14 +432,20 @@ mod tests { } assert!(background_stalled(Some(since), u64::MAX)); } + /// No excuse used yet, under the default maximum allowance. + const FRESH: Excuse = Excuse { used: 0, limit: 15 }; fn classify(own: Head, renewals: u8, others: impl Iterator) -> Decision { - attribute(own, renewals, others).decision + attribute(own, renewals, FRESH, others).decision } fn head(vm: u64, published: u64, finished: bool) -> Head { - Head { vm, published, finished, failed: false } + Head { vm, published, finished, failed: false, excused: false } } fn failed(vm: u64, published: u64) -> Head { - Head { vm, published, finished: false, failed: true } + Head { vm, published, finished: false, failed: true, excused: false } + } + /// A live head whose own watchdog still renews it for its allowance. + fn excused(vm: u64, published: u64) -> Head { + Head { excused: true, ..head(vm, published, false) } } #[test] fn ray_query_execution_is_bounded_at_ten_seconds() { @@ -303,6 +491,62 @@ mod tests { assert!(!renew_shared(true, true, &mut renewals)); assert_eq!(POLL_MS * (u32::from(renewals) + 1), 30_000); } + /// When an ordinary job that never finishes is failed (ms), on a GPU that is `shared` + /// throughout or not at all, by the allowance and the shared renewals in the order the + /// watchdog applies them, with no execution renewal or deferral. With `waited`, the expiry + /// at 2 s finds it still in publication (a deferral keeps it) and the next one published. + fn fails_at(allowance_ms: u32, shared: bool, waited: bool) -> u32 { + let mut allowance = Allowance::new(allowance_ms); + let mut renewals = 0; + let mut expiry = POLL_MS; + if waited { + assert!(!allowance.renew(false, u64::from(expiry) * MS)); + expiry += POLL_MS; + } + loop { + let placing = allowance.placing_after_wait(true); + if allowance.renew(true, u64::from(expiry) * MS) { + count_allowance_poll(allowance.asked() && !placing, &mut renewals); + } else if !renew_shared(true, shared, &mut renewals) { + return expiry; + } + expiry += POLL_MS; + assert!(expiry < 1_000_000); + } + } + #[test] + fn an_allowance_and_the_shared_renewals_overlap() { + // Under the default allowance: one poll alone, thirty seconds on a shared GPU. + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, false, false), 2_000); + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, true, false), 30_000); + // The longest allowance is the limit with or without sharing: not 30 s + 28 s. + assert_eq!(fails_at(30_000, false, false), 30_000); + assert_eq!(fails_at(30_000, true, false), 30_000); + // A shorter one never leaves a job worse off than the default on a shared GPU. + assert_eq!(fails_at(6_000, false, false), 6_000); + assert_eq!(fails_at(6_000, true, false), 30_000); + assert_eq!(fails_at(20_000, true, false), 30_000); + assert_eq!(fails_at(2_001, true, false), 30_000); + // Above thirty seconds (a raised maximum) the allowance is the limit. + assert_eq!(fails_at(40_000, true, false), 40_000); + // After a wait in publication every allowance is placed at the first published expiry + // (4 s), and the poll renewed there is not counted: still never fewer than the default. + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, false, true), 6_000); + assert_eq!(fails_at(DEFAULT_ALLOWANCE_MS, true, true), 34_000); + for ms in [2_001, 6_000, 20_000, 30_000] { + assert_eq!(fails_at(ms, true, true), 34_000); + } + assert_eq!(fails_at(6_000, false, true), 10_000); + assert_eq!(fails_at(30_000, false, true), 34_000); + // Nothing is counted for a poll the default renews too. + let mut renewals = 0; + count_allowance_poll(false, &mut renewals); + assert_eq!(renewals, 0); + for _ in 0..40 { + count_allowance_poll(true, &mut renewals); + } + assert_eq!(renewals, SHARED_RENEWALS); + } #[test] fn a_stopped_gpu_or_unpublished_job_keeps_the_deadline() { let mut renewals = 0; @@ -391,19 +635,133 @@ mod tests { #[test] fn the_watchdog_line_names_the_blocker_at_the_first_deferral_and_the_end() { let own = head(9, 103, false); - let verdict = attribute(own, 0, [failed(5, 100)].into_iter()); + let verdict = attribute(own, 0, FRESH, [failed(5, 100)].into_iter()); let expected = Verdict { decision: Decision::Defer, own, blocker: Some(failed(5, 100)) }; assert_eq!(verdict, expected); - assert_eq!(verdict.reported_blocker(0), Some(failed(5, 100))); - assert_eq!(attribute(own, 1, [failed(5, 100)].into_iter()).reported_blocker(1), None); - let last = attribute(own, MAX_RENEWALS, [failed(5, 100)].into_iter()); + assert_eq!(verdict.reported_blocker(0, FRESH), Some(failed(5, 100))); + let renewed = attribute(own, 1, FRESH, [failed(5, 100)].into_iter()); + assert_eq!(renewed.reported_blocker(1, FRESH), None); + let last = attribute(own, MAX_RENEWALS, FRESH, [failed(5, 100)].into_iter()); assert_eq!(last.decision, Decision::Own); - assert_eq!(last.reported_blocker(MAX_RENEWALS), Some(failed(5, 100))); - let spared = attribute(own, MAX_RENEWALS, [head(5, 100, false)].into_iter()); + assert_eq!(last.reported_blocker(MAX_RENEWALS, FRESH), Some(failed(5, 100))); + let spared = attribute(own, MAX_RENEWALS, FRESH, [head(5, 100, false)].into_iter()); assert_eq!(spared.decision, Decision::Spare); - assert_eq!(spared.reported_blocker(MAX_RENEWALS), Some(head(5, 100, false))); + assert_eq!(spared.reported_blocker(MAX_RENEWALS, FRESH), Some(head(5, 100, false))); // Nothing blocks: nothing to report. - assert_eq!(attribute(own, 0, [].into_iter()).reported_blocker(0), None); + assert_eq!(attribute(own, 0, FRESH, [].into_iter()).reported_blocker(0, FRESH), None); + // An excuse is named at its first poll, and so is the first counted deferral after it. + let blocker = excused(5, 100); + let first = attribute(own, 0, FRESH, [blocker].into_iter()); + assert_eq!(first.decision, Decision::Excuse); + assert_eq!(first.reported_blocker(0, FRESH), Some(blocker)); + let later = Excuse { used: 3, ..FRESH }; + let excused_again = attribute(own, 0, later, [blocker].into_iter()); + assert_eq!(excused_again.reported_blocker(0, later), None); + let past = attribute(own, 0, later, [head(5, 100, false)].into_iter()); + assert_eq!(past.decision, Decision::Defer); + assert_eq!(past.reported_blocker(0, later), Some(head(5, 100, false))); + } + #[test] + fn the_excuse_is_bounded_by_the_longest_allowance() { + assert_eq!(excuse_limit(30_000), 15); + assert_eq!(excuse_limit(29_000), 15); + assert_eq!(excuse_limit(0), 1); + // The default maximum gives 15 + 8 renewals behind other VMs' live work: 46 s of them. + assert_eq!(POLL_MS * (excuse_limit(30_000) + u32::from(MAX_RENEWALS)), 46_000); + } + #[test] + fn blockers_taking_turns_do_not_extend_the_excuse_or_the_renewals() { + // Two other clients' commands, each inside its allowance, are ahead by turns for + // longer than the excuse lasts. The counters belong to the waiting job: it is renewed + // limit + MAX_RENEWALS times in all, whichever blocker is ahead, and then spared. + let own = head(9, 103, false); + let (mut renewals, mut excuse, mut polls) = (0u8, FRESH, 0u32); + loop { + let ahead = if polls % 2 == 0 { excused(5, 100) } else { excused(6, 101) }; + match attribute(own, renewals, excuse, [ahead].into_iter()).decision { + Decision::Excuse => excuse.used += 1, + Decision::Defer => renewals += 1, + decision => { + assert_eq!(decision, Decision::Spare); + break; + } + } + polls += 1; + assert!(polls < 100); + } + assert_eq!(excuse.used, FRESH.limit); + assert_eq!(renewals, MAX_RENEWALS); + assert_eq!(polls, FRESH.limit + u32::from(MAX_RENEWALS)); + // Renewals spent before an excuse are not given back by it. + let spent = attribute(own, 2, FRESH, [excused(5, 100)].into_iter()); + assert_eq!(spent.decision, Decision::Excuse); + let after = Excuse { used: FRESH.limit, ..FRESH }; + assert_eq!(attribute(own, 2, after, [head(5, 100, false)].into_iter()).decision, + Decision::Defer); + assert_eq!(attribute(own, MAX_RENEWALS, after, [head(5, 100, false)].into_iter()) + .decision, Decision::Spare); + } + #[test] + fn a_live_blocker_inside_its_allowance_defers_past_the_renewals() { + // A compositor frame behind another client's 20 s compute command: its own renewals + // stay untouched while the command runs inside its allowance. + let own = head(9, 103, false); + for used in 0..FRESH.limit { + let excuse = Excuse { used, ..FRESH }; + assert_eq!(attribute(own, 0, excuse, [excused(5, 100)].into_iter()).decision, + Decision::Excuse); + // Past eight renewals from an earlier blocker too. + assert_eq!(attribute(own, MAX_RENEWALS, excuse, [excused(5, 100)].into_iter()) + .decision, Decision::Excuse); + } + // The excuse is used up: the renewals apply, then the work is spared. + let used = Excuse { used: FRESH.limit, ..FRESH }; + assert_eq!(attribute(own, 0, used, [excused(5, 100)].into_iter()).decision, + Decision::Defer); + assert_eq!(attribute(own, MAX_RENEWALS, used, [excused(5, 100)].into_iter()).decision, + Decision::Spare); + } + #[test] + fn a_blocker_past_its_allowance_gets_todays_decision() { + let own = head(9, 103, false); + let past = head(5, 100, false); + for renewals in 0..MAX_RENEWALS { + assert_eq!(classify(own, renewals, [past].into_iter()), Decision::Defer); + } + assert_eq!(classify(own, MAX_RENEWALS, [past].into_iter()), Decision::Spare); + } + #[test] + fn a_failed_blocker_excuses_nothing() { + let own = head(9, 103, false); + let failed_in_allowance = Head { excused: true, ..failed(5, 100) }; + for renewals in 0..MAX_RENEWALS { + assert_eq!(classify(own, renewals, [failed_in_allowance].into_iter()), Decision::Defer); + } + assert_eq!(classify(own, MAX_RENEWALS, [failed_in_allowance].into_iter()), Decision::Own); + // A live blocker is chosen before a failed one, excused or not. + let others = [failed_in_allowance, excused(7, 101)]; + assert_eq!(classify(own, MAX_RENEWALS, others.into_iter()), Decision::Excuse); + } + #[test] + fn no_blocker_or_finished_work_is_own() { + assert_eq!(classify(head(9, 103, false), 0, [].into_iter()), Decision::Own); + // The oldest unfinished work is never excused, not even by later excused work. + assert_eq!(classify(head(9, 100, false), 0, [excused(5, 101)].into_iter()), Decision::Own); + assert_eq!(classify(head(9, 103, true), 0, [excused(5, 100)].into_iter()), Decision::Own); + // Same-VM work does not excuse either. + assert_eq!(classify(head(5, 103, false), MAX_RENEWALS, [excused(5, 100)].into_iter()), + Decision::Own); + } + #[test] + fn only_a_placed_long_allowance_has_an_end() { + let poll = u64::from(POLL_MS) * MS; + let mut default = Allowance::new(DEFAULT_ALLOWANCE_MS); + assert!(!default.renew(true, 10_000 * MS)); + assert_eq!(default.end(), None); + let mut long = Allowance::new(20_000); + assert_eq!(long.end(), None); + assert!(long.renew(true, 10_000 * MS)); + assert_eq!(long.end(), Some(10_000 * MS - poll + 20_000 * MS)); } #[test] fn same_vm_finished_and_later_work_do_not_block() { diff --git a/drivers/gpu/drm/asahi/gem.rs b/drivers/gpu/drm/asahi/gem.rs index 3c3b667e61af5..40487a77e97ea 100644 --- a/drivers/gpu/drm/asahi/gem.rs +++ b/drivers/gpu/drm/asahi/gem.rs @@ -31,6 +31,7 @@ use kernel::{ use core::ops::Range; use core::sync::atomic::{ + AtomicBool, AtomicI64, AtomicU64, AtomicUsize, @@ -65,6 +66,10 @@ pub(crate) struct AsahiObject { kernel_mappings: AtomicU64, /// Memory accounting and process attribution of the held backing. charge: memacct::Charge, + /// Userspace asked to keep the backing resident (DRM_IOCTL_ASAHI_GEM_RESIDENCY): idle + /// reclaim of the object's VM releases it after the VM's other objects, and only if those + /// did not meet the scan target. Changed and read under the object's reservation. + keep_resident: AtomicBool, /// The pinned pages of a user-memory object (DRM_IOCTL_ASAHI_GEM_USERPTR), its backing. /// The shmem layer unmaps them from the device before this is dropped. user_memory: Option, @@ -602,6 +607,7 @@ impl DriverObject for AsahiObject { exportable: args.exportable, kernel: args.kernel, kernel_mappings: AtomicU64::new(0), + keep_resident: AtomicBool::new(false), charge: args.charge, user_memory: args.user_memory, timestamp_lock <- new_mutex!((), "Asahi timestamp stores"), @@ -692,6 +698,16 @@ impl AsahiObject { && self.kernel_mappings.load(Ordering::Acquire) == 0 } + /// Whether userspace asked to keep this object's backing resident. Idle reclaim releases + /// the backing of such an object after the VM's other objects, and only if those did not + /// meet the scan target. Purge of contents userspace advised away does not look at this: + /// the advice says the contents are not needed. + /// + /// The caller holds the object's reservation. + pub(crate) fn keep_resident(&self) -> bool { + self.keep_resident.load(Ordering::Relaxed) + } + /// Whether this object is backed by the creating process's own memory. pub(crate) fn is_user_memory(&self) -> bool { self.user_memory.is_some() @@ -727,6 +743,28 @@ pub(crate) fn madvise(gem: &Object, dontneed: bool) -> Result { Ok(u32::from(gem.madvise(i32::from(dontneed)))) } +/// Keep (or stop keeping) the backing of a user object resident: once this returns, idle reclaim +/// treats the object as kept (released after the VM's other objects, where those did not meet +/// the scan target) or as any other. This is an order among the objects idle reclaim may +/// release, not an exemption. Backing already released stays released until the VM's next job +/// or binding restores it. Kernel objects are refused. +pub(crate) fn set_keep_resident(gem: &Object, keep: bool) -> Result { + if gem.kernel { + return Err(EINVAL); + } + // SAFETY: A live object's reservation pointer is set at creation and never changes. + let resv = unsafe { (*gem.as_raw()).resv }; + // SAFETY: The reservation of a live object. Idle reclaim tests the flag while it holds the + // reservation of the object's VM, which a VM-private object shares, so after the unlock no + // reclaim pass can still act on the old value; other objects are never reclaimed. + unsafe { kernel::bindings::dma_resv_lock(resv, core::ptr::null_mut()) }; + gem.keep_resident.store(keep, Ordering::Relaxed); + gem.charge.set_kept(keep); + // SAFETY: Locked above. + unsafe { kernel::bindings::dma_resv_unlock(resv) }; + Ok(()) +} + /// Large user BOs may release clean page cache but never swap out the rest of the /// system per folio; they get a quick ENOMEM when they cannot fit. Keep ordinary /// resource allocations and kernel-owned backing on their existing policy. 256 MiB diff --git a/drivers/gpu/drm/asahi/memacct.rs b/drivers/gpu/drm/asahi/memacct.rs index a61be841bff11..b1e2d4fe093b1 100644 --- a/drivers/gpu/drm/asahi/memacct.rs +++ b/drivers/gpu/drm/asahi/memacct.rs @@ -21,7 +21,9 @@ //! counter is back to its own value before `__mmdrop()` checks it. //! - Each client (DRM file) and the device keep totals of created and held bytes, //! reported in fdinfo (`drm-total-system`, `drm-resident-system`) and in the -//! `gpu_memory` sysfs file next to `agx_stats`. +//! `gpu_memory` sysfs file next to `agx_stats`. The device also totals the bytes of +//! objects that userspace asked to keep resident (which idle reclaim releases after +//! the other objects of their VM), in `gpu_memory`. //! - A new user object is admitted only while its client stays within a per-client //! limit and the device within a device-wide limit, both fractions of RAM. //! A refused allocation fails with `ENOMEM`, which userspace reports as @@ -66,13 +68,16 @@ pub(crate) struct Ledger { pub(crate) limit: AtomicU64, /// Per-client limit on committed bytes (0: none). pub(crate) client_limit: AtomicU64, + /// Size of the live user objects kept resident at userspace's request. + pub(crate) user_kept: AtomicU64, } // Keep the Rust half of the C sysfs layout contract checked as well. const _: () = { assert!(core::mem::offset_of!(Ledger, refused) == 24); assert!(core::mem::offset_of!(Ledger, client_limit) == 40); - assert!(core::mem::size_of::() == 48); + assert!(core::mem::offset_of!(Ledger, user_kept) == 48); + assert!(core::mem::size_of::() == 56); }; /// The ledger of the (single) AGX GPU. @@ -145,6 +150,7 @@ impl Ledger { refused: AtomicU64::new(0), limit: AtomicU64::new(0), client_limit: AtomicU64::new(0), + user_kept: AtomicU64::new(0), } } @@ -271,6 +277,8 @@ pub(crate) struct Charge { bytes: u64, /// Whether the bytes are currently counted as held by the GPU. held: AtomicBool, + /// Whether the bytes are currently counted as kept resident. + kept: AtomicBool, } // SAFETY: `ARef` may be dropped on any thread and the counters are atomic. @@ -286,6 +294,7 @@ impl Charge { kind: Kind::None, bytes: 0, held: AtomicBool::new(false), + kept: AtomicBool::new(false), } } @@ -295,6 +304,7 @@ impl Charge { kind: Kind::Kernel(ledger), bytes, held: AtomicBool::new(false), + kept: AtomicBool::new(false), } } @@ -310,6 +320,7 @@ impl Charge { kind: Kind::User { commit, mm }, bytes, held: AtomicBool::new(false), + kept: AtomicBool::new(false), } } @@ -387,11 +398,31 @@ impl Charge { self.apply(held); } } + + /// Records that userspace asked (or no longer asks) to keep the object resident. Only user + /// objects are counted, in the device's total, which a transfer to another client leaves + /// as it is. + /// + /// Idempotent: only a change of state is counted. + pub(crate) fn set_kept(&self, kept: bool) { + let Kind::User { commit, .. } = &self.kind else { + return; + }; + if self.kept.swap(kept, Ordering::AcqRel) == kept { + return; + } + if kept { + commit.ledger.user_kept.fetch_add(self.bytes, Ordering::AcqRel); + } else { + sub(&commit.ledger.user_kept, self.bytes); + } + } } impl Drop for Charge { fn drop(&mut self) { // Uncharge before `kind` (and with it the mm reference) is dropped. + self.set_kept(false); self.set_held(false); } } @@ -548,6 +579,35 @@ mod tests { Ok(()) } + static KEPT: Ledger = Ledger::new(); + + #[test] + fn kept_bytes_follow_the_object() -> Result { + KEPT.set_limits(Limits { device: 0, client: 0 }); + let a = Client::new()?; + let b = Client::new()?; + let mut charge = Charge::user(KEPT.admit(&a, 8 * MIB)?, None); + charge.set_kept(true); + charge.set_kept(true); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 8 * MIB); + charge.set_kept(false); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 0); + charge.set_kept(true); + // A move to another client leaves the device's count as it is. + charge.transfer(&b)?; + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 8 * MIB); + // Freeing a kept object stops counting it. + drop(charge); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 0); + // Kernel objects and imports are never counted as kept. + let k = Charge::kernel(&KEPT, MIB); + k.set_kept(true); + let i = Charge::none(); + i.set_kept(true); + assert_eq!(KEPT.user_kept.load(Ordering::Relaxed), 0); + Ok(()) + } + #[test] fn imports_are_not_counted() { let charge = Charge::none(); diff --git a/drivers/gpu/drm/asahi/mmu/residency.rs b/drivers/gpu/drm/asahi/mmu/residency.rs index a83068a5c9e9f..7d36f2bee910e 100644 --- a/drivers/gpu/drm/asahi/mmu/residency.rs +++ b/drivers/gpu/drm/asahi/mmu/residency.rs @@ -243,11 +243,46 @@ impl Drop for Lease { /// Registration lifetime belongs to the file's VM handle, not the VM clone /// retained by the callback. Unregister synchronously before dropping the clone. +/// +/// A VM has two shrinkers. The first purges advised contents and releases the +/// idle objects userspace did not ask to keep, in step with the rest of +/// reclaim. The second releases the objects userspace asked to keep resident +/// (DRM_IOCTL_ASAHI_GEM_RESIDENCY) as well, after the others, and the shrinker +/// core calls its scan only for a pass that is to scan everything it counted: +/// see `KEPT_BATCH`. +/// +/// Neither is memcg-aware: reclaim inside a memory cgroup below the root does +/// not call them, so only reclaim on behalf of the whole system releases GPU +/// memory. Both release idle backing only of a VM that passes the gate's +/// test (`Gate::try_reclaim`): no lease held, none released for +/// `RECLAIM_QUIET_NS`, and the backoff after a restoration over. Userspace's +/// request to keep an object is thus an order among the objects that these +/// shrinkers may release, and no more. pub(crate) struct VmShrinker { vm: crate::mmu::Vm, raw: *mut kernel::bindings::shrinker, + kept: *mut kernel::bindings::shrinker, } +/// The batch of the shrinker of kept objects: more than a scan target can be. +/// +/// The shrinker core scans a shrinker when the target it worked out reaches the +/// batch, or reaches everything the shrinker counted. The target is the count +/// scaled down by the reclaim priority (at most twice the count, at the last +/// priority), so with this batch only the second case is left: the target of +/// the last two of the thirteen priorities, or of the last three once scans +/// deferred by earlier passes have built up. A pass at an earlier priority +/// does not scan kept objects. +/// +/// Reclaim works down the priorities only as far as it must to free what it +/// was asked for. It comes to the last ones when the page cache, the other +/// shrinkers and this VM's first one did not give that much at the +/// priorities before: under memory pressure that nothing else relieves, and +/// also for a request that names more than the system has to spare (a reclaim +/// request to the root memory cgroup, say). Dropping the slab caches +/// (`drop_caches`) runs every shrinker at the last priority at once. +const KEPT_BATCH: kernel::ffi::c_long = kernel::ffi::c_long::MAX; + // SAFETY: shrinker core serializes registration lifetime; callback VM operations // use its residency/reservation locks. raw is never dereferenced after free. unsafe impl Send for VmShrinker {} @@ -261,6 +296,7 @@ impl VmShrinker { Self { vm: vm.clone(), raw: core::ptr::null_mut(), + kept: core::ptr::null_mut(), }, GFP_KERNEL, )?; @@ -271,14 +307,29 @@ impl VmShrinker { return Err(ENOMEM); } owner.raw = raw; - // SAFETY: Newly allocated/unregistered shrinker is exclusively owned. + // SAFETY: As above. + let kept = + unsafe { kernel::bindings::shrinker_alloc(0, c_str!("aurora-kept-vm").as_char_ptr()) }; + if kept.is_null() { + return Err(ENOMEM); + } + owner.kept = kept; + // SAFETY: Newly allocated/unregistered shrinkers are exclusively owned. The + // shrinker of kept objects is registered second: a reclaim pass that reaches it + // has asked this VM's first shrinker before. unsafe { (*raw).count_objects = Some(Self::count); (*raw).scan_objects = Some(Self::scan); (*raw).private_data = core::ptr::from_ref(&*owner).cast_mut().cast(); (*raw).seeks = 2; (*raw).batch = 256; + (*kept).count_objects = Some(Self::count_kept); + (*kept).scan_objects = Some(Self::scan_kept); + (*kept).private_data = core::ptr::from_ref(&*owner).cast_mut().cast(); + (*kept).seeks = 2; + (*kept).batch = KEPT_BATCH; kernel::bindings::shrinker_register(raw); + kernel::bindings::shrinker_register(kept); } Ok(owner) } @@ -295,7 +346,21 @@ impl VmShrinker { ) -> kernel::ffi::c_ulong { // SAFETY: The registration owns this stable allocation until callbacks drain. let owner = unsafe { &*((*raw).private_data as *const Self) }; - owner.vm.idle_reclaim_count() + owner.vm.idle_reclaim_count(false) + } + + /// Count the reclaimable pages of the objects userspace asked to keep resident. + /// + /// # Safety + /// + /// As for [`Self::count`]. + unsafe extern "C" fn count_kept( + raw: *mut kernel::bindings::shrinker, + _sc: *mut kernel::bindings::shrink_control, + ) -> kernel::ffi::c_ulong { + // SAFETY: The registration owns this stable allocation until callbacks drain. + let owner = unsafe { &*((*raw).private_data as *const Self) }; + owner.vm.idle_reclaim_count(true) } /// Reclaim idle pages. @@ -312,16 +377,35 @@ impl VmShrinker { let owner = unsafe { &*((*raw).private_data as *const Self) }; // SAFETY: The shrinker core supplies a live initialized control. let target = unsafe { (*sc).nr_to_scan }; - owner.vm.reclaim_idle_pages(target) + owner.vm.reclaim_idle_pages(target, false) + } + + /// Reclaim idle pages for a pass that scans everything: the objects userspace asked to + /// keep resident too, after the others. + /// + /// # Safety + /// + /// As for [`Self::scan`]. + unsafe extern "C" fn scan_kept( + raw: *mut kernel::bindings::shrinker, + sc: *mut kernel::bindings::shrink_control, + ) -> kernel::ffi::c_ulong { + // SAFETY: Both pointers are supplied by the registered shrinker core. + let owner = unsafe { &*((*raw).private_data as *const Self) }; + // SAFETY: The shrinker core supplies a live initialized control. + let target = unsafe { (*sc).nr_to_scan }; + owner.vm.reclaim_idle_pages(target, true) } } impl Drop for VmShrinker { fn drop(&mut self) { - if !self.raw.is_null() { - // SAFETY: This owns the sole registration; free drains callbacks - // before the embedded VM and callback private_data can disappear. - unsafe { kernel::bindings::shrinker_free(self.raw) }; + for raw in [self.kept, self.raw] { + if !raw.is_null() { + // SAFETY: This owns the sole registration; free drains callbacks + // before the embedded VM and callback private_data can disappear. + unsafe { kernel::bindings::shrinker_free(raw) }; + } } } } @@ -356,7 +440,10 @@ impl Vm { .transpose() } - pub(crate) fn idle_reclaim_count(&self) -> usize { + /// The pages reclaim could release now: with `kept`, those of the idle objects userspace + /// asked to keep resident; without, those of the other idle objects and of every object + /// whose contents userspace advised away, kept or not. + pub(crate) fn idle_reclaim_count(&self, kept: bool) -> usize { let Some(gate) = self.residency.as_ref() else { return 0; }; @@ -386,7 +473,12 @@ impl Vm { (advised || object.has_sg_table_locked()) && object.unleased_pages_releasable_locked() }; - if object.idle_reclaim_candidate() && (advised || idle) && releasable { + let counted = if advised { + !kept + } else { + idle && object.keep_resident() == kept + }; + if object.idle_reclaim_candidate() && counted && releasable { pages = pages.saturating_add(object.size() / kernel::page::PAGE_SIZE); } Ok(true) @@ -475,7 +567,10 @@ impl Vm { purged } - pub(crate) fn reclaim_idle_pages(&self, target: usize) -> usize { + /// Releases idle backing until `target` pages are free: contents userspace advised away + /// first, then the objects it did not ask to keep resident and, only with `kept` and only + /// if those did not reach the target, the objects it did. + pub(crate) fn reclaim_idle_pages(&self, target: usize, kept: bool) -> usize { let Some(gate) = self.residency.as_ref() else { return 0; }; @@ -508,50 +603,58 @@ impl Vm { }; let contexts = shared.contexts_naming_root(inner.ttb(), inner.uat_inner.m4_adopted); let mut unpinned = 0usize; - // SAFETY: Exclusive residency and the private reservation exclude jobs, - // binding edits and retirement. Callbacks change PTEs/flags only; all - // logical aliases stay linked and retain their borrowed BO/GEM. - let _ = unsafe { inner.for_each_private_bo(|inner, bo| { - if unpinned >= target { return Ok(false); } - let object = bo.obj(); - if !object.idle_reclaim_candidate() { return Ok(true); } - { - // Avoid inversion against preparation (VmBoInner -> resv). - let Some(segments) = bo.data().segments.try_lock() else { return Ok(true); }; - let releasable = if segments.is_some() { - object.sole_sg_lease_releasable_locked() - } else { - // A competing lease may have made the previous release fail after we - // dropped our segments. Retry that table, but do not count empty backing. - object.has_sg_table_locked() && object.unleased_pages_releasable_locked() - }; - if !releasable { return Ok(true); } - } - // The association retains SG backing until *all* aliases have - // been invalidated and their device translations flushed. A - // contended cursor or PTE failure retains backing and the touched - // INVALIDATED prefix for restoration or a later reclaim attempt. - inner.invalidate_private_bo_mappings(bo, |inner, start, range| { - reclaim.mark_changed(); - let end = start.checked_add(range).ok_or(EOVERFLOW)?; - let _change = inner.mapping_mutation(); - inner.page_table.discard_partial_map(start..end)?; - inner.tlbi_context_mask(start, range as usize, contexts); - Ok(()) - })?; - { - let Some(mut segments) = bo.data().segments.try_lock() else { return Ok(true); }; - *segments = None; - } - // Every private GPUVA alias was removed from the device page - // tables above. Kernel mappings and other SG owners are excluded; - // the C/Rust backing helper rechecks exclusion before release. - if object.release_idle_pages_locked().is_ok() { - crate::gem::set_mapping_held(object, false); - unpinned = unpinned.saturating_add(object.size() / kernel::page::PAGE_SIZE); - } - Ok(unpinned < target) - }) }; + // Kept objects go in a second pass, when the first left the target unmet. + for keep_pass in [false, true] { + if unpinned >= target || (keep_pass && !kept) { break; } + // SAFETY: Exclusive residency and the private reservation exclude jobs, + // binding edits and retirement. Callbacks change PTEs/flags only; all + // logical aliases stay linked and retain their borrowed BO/GEM. + let _ = unsafe { inner.for_each_private_bo(|inner, bo| { + if unpinned >= target { return Ok(false); } + let object = bo.obj(); + if !object.idle_reclaim_candidate() || object.keep_resident() != keep_pass { + return Ok(true); + } + { + // Avoid inversion against preparation (VmBoInner -> resv). + let Some(segments) = bo.data().segments.try_lock() else { return Ok(true); }; + let releasable = if segments.is_some() { + object.sole_sg_lease_releasable_locked() + } else { + // A competing lease may have made the previous release fail after we + // dropped our segments. Retry that table, but do not count empty backing. + object.has_sg_table_locked() && object.unleased_pages_releasable_locked() + }; + if !releasable { return Ok(true); } + } + // The association retains SG backing until *all* aliases have + // been invalidated and their device translations flushed. A + // contended cursor or PTE failure retains backing and the touched + // INVALIDATED prefix for restoration or a later reclaim attempt. + inner.invalidate_private_bo_mappings(bo, |inner, start, range| { + reclaim.mark_changed(); + let end = start.checked_add(range).ok_or(EOVERFLOW)?; + let _change = inner.mapping_mutation(); + inner.page_table.discard_partial_map(start..end)?; + inner.tlbi_context_mask(start, range as usize, contexts); + Ok(()) + })?; + { + let Some(mut segments) = bo.data().segments.try_lock() else { + return Ok(true); + }; + *segments = None; + } + // Every private GPUVA alias was removed from the device page + // tables above. Kernel mappings and other SG owners are excluded; + // the C/Rust backing helper rechecks exclusion before release. + if object.release_idle_pages_locked().is_ok() { + crate::gem::set_mapping_held(object, false); + unpinned = unpinned.saturating_add(object.size() / kernel::page::PAGE_SIZE); + } + Ok(unpinned < target) + }) }; + } drop(shared); drop(inner); diff --git a/drivers/gpu/drm/asahi/queue/mod.rs b/drivers/gpu/drm/asahi/queue/mod.rs index 525a2ffffbbfe..a37bb48fa613f 100644 --- a/drivers/gpu/drm/asahi/queue/mod.rs +++ b/drivers/gpu/drm/asahi/queue/mod.rs @@ -70,6 +70,13 @@ fn can_submit_inline(dependencies_ready: bool, previous_status: Option) -> /// Trait implemented by all versioned queues. pub(crate) trait Queue: Send + Sync { + /// Sets the watchdog allowance of the queue's compute commands to `requested_ms` + /// (`drm_asahi_queue_create::compute_timeout_ms`, nonzero) and returns the allowance in + /// effect. Only queues of GPUs that advertise DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT take one. + fn set_compute_timeout(&mut self, _requested_ms: u32) -> Result { + Err(EINVAL) + } + fn reserve_render_scs(&mut self, layouts: [u64; 2]) -> Result { if layouts != [0; 2] { return Err(EINVAL); diff --git a/drivers/gpu/drm/asahi/stats.rs b/drivers/gpu/drm/asahi/stats.rs index b8636d4c5eb70..d8f33c932afe5 100644 --- a/drivers/gpu/drm/asahi/stats.rs +++ b/drivers/gpu/drm/asahi/stats.rs @@ -84,6 +84,12 @@ impl StatsSnapshot { self.frequencies_hz.get(pstate as usize).copied() } + /// The frequency the GPU runs at now, if the firmware reports it, and its highest, in + /// Hz, for fdinfo (`drm-curfreq-*`, `drm-maxfreq-*`): M5 only. + pub(crate) fn engine_frequency(&self) -> Option<(Option, u64)> { + self.g17_reader.as_ref()?.frequency() + } + /// Bump the completed-submission counter (called from the queue /// completion path in `JobFence::command_complete`, NOT from the stats /// channel). diff --git a/drivers/gpu/drm/asahi/sysfs.c b/drivers/gpu/drm/asahi/sysfs.c index 0b2bca6571129..ac2139d1ebb9b 100644 --- a/drivers/gpu/drm/asahi/sysfs.c +++ b/drivers/gpu/drm/asahi/sysfs.c @@ -42,6 +42,7 @@ struct asahi_gpu_memory { u64 refused; u64 limit; u64 client_limit; + u64 user_kept; }; struct asahi_stats_attribute { @@ -99,11 +100,20 @@ static ssize_t agx_qualified_stats_show(struct asahi_stats_attribute *stats, cha n += sysfs_emit_at(buf, n, "pstate %u\n", snap.pstate); else n += sysfs_emit_at(buf, n, "pstate unsupported\n"); - /* No A/B selector, packet duration, measured power or temperature. */ - n += sysfs_emit_at(buf, n, "requested_frequency_hz unsupported\n" - "achieved_frequency_hz unsupported\n" - "busy_ns unsupported\n" - "power_mw unsupported\n" + /* No A/B selector, measured power or temperature. */ + n += sysfs_emit_at(buf, n, "requested_frequency_hz unsupported\n"); + /* M5 reports the state the GPU runs at: its frequency is the achieved one. */ + if (snap.pstate_status == 1 && snap.achieved_frequency_hz) + n += sysfs_emit_at(buf, n, "achieved_frequency_hz %llu\n", + (unsigned long long)snap.achieved_frequency_hz); + else + n += sysfs_emit_at(buf, n, "achieved_frequency_hz unsupported\n"); + if (snap.busy_status == 1) + n += sysfs_emit_at(buf, n, "busy_ns %llu\n", + (unsigned long long)snap.busy_ns); + else + n += sysfs_emit_at(buf, n, "busy_ns unsupported\n"); + n += sysfs_emit_at(buf, n, "power_mw unsupported\n" "temperature_raw unsupported\n"); n += sysfs_emit_at(buf, n, "utilisation_source firmware-statistics-tag-0x13\n" "utilisation_samples %llu\n" @@ -182,7 +192,11 @@ static ssize_t agx_stats_show(struct device *dev, /* * GPU memory held by the driver, in bytes: user objects created (committed) * and held by the GPU (resident), held kernel objects, the limits that admit - * user objects (0: none) and the number of refused creations. + * user objects (0: none), the number of refused creations, and the size of + * the user objects that userspace asked to keep resident (user_kept), whether + * their memory is held at the moment or not. The request is a hint: when + * reclaim releases idle memory of a VM, its kept objects go after the others + * (see DRM_IOCTL_ASAHI_GEM_RESIDENCY). */ static ssize_t gpu_memory_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -204,6 +218,8 @@ static ssize_t gpu_memory_show(struct device *dev, (unsigned long long)READ_ONCE(mem->client_limit)); n += scnprintf(buf + n, PAGE_SIZE - n, "refused %llu\n", (unsigned long long)READ_ONCE(mem->refused)); + n += scnprintf(buf + n, PAGE_SIZE - n, "user_kept %llu\n", + (unsigned long long)READ_ONCE(mem->user_kept)); return n; } @@ -225,10 +241,12 @@ int asahi_sysfs_register(struct device *dev, const void *snapshot, return -ENODEV; BUILD_BUG_ON(offsetof(struct asahi_qualified_stats, util_host_ns) != 24); - BUILD_BUG_ON(sizeof(struct asahi_qualified_stats) != 56); + BUILD_BUG_ON(offsetof(struct asahi_qualified_stats, busy_ns) != 56); + BUILD_BUG_ON(sizeof(struct asahi_qualified_stats) != 80); BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, refused) != 24); BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, client_limit) != 40); - BUILD_BUG_ON(sizeof(struct asahi_gpu_memory) != 48); + BUILD_BUG_ON(offsetof(struct asahi_gpu_memory, user_kept) != 48); + BUILD_BUG_ON(sizeof(struct asahi_gpu_memory) != 56); BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, pstate) != 16); BUILD_BUG_ON(offsetof(struct asahi_stats_snapshot, busy_ns) != 40); diff --git a/drivers/gpu/drm/asahi/sysfs.h b/drivers/gpu/drm/asahi/sysfs.h index 12bcce7d4e7fb..3433ebbcebfd3 100644 --- a/drivers/gpu/drm/asahi/sysfs.h +++ b/drivers/gpu/drm/asahi/sysfs.h @@ -22,6 +22,10 @@ struct asahi_qualified_stats { u64 pstate_host_ns; u64 util_samples; u64 pstate_timestamp; + u64 busy_ns; + u64 achieved_frequency_hz; + u32 busy_status; + u32 pad; }; typedef void (*asahi_stats_read_fn)(const void *snapshot, struct asahi_qualified_stats *out); diff --git a/include/uapi/drm/asahi_drm.h b/include/uapi/drm/asahi_drm.h index c51b239fb4917..ab9e92b80c23b 100644 --- a/include/uapi/drm/asahi_drm.h +++ b/include/uapi/drm/asahi_drm.h @@ -102,6 +102,15 @@ enum drm_asahi_ioctl_id { * memory (DRM_ASAHI_FEATURE_USERPTR). */ DRM_ASAHI_GEM_USERPTR, + + /** + * @DRM_ASAHI_GEM_RESIDENCY: Ask the kernel to release an object's + * memory after that of the other objects of its VM when it reclaims + * the VM's idle memory, or withdraw that request + * (DRM_ASAHI_FEATURE_GEM_RESIDENCY). A hint, see struct + * drm_asahi_gem_residency. + */ + DRM_ASAHI_GEM_RESIDENCY, }; #define DRM_ASAHI_MAX_CLUSTERS 64 @@ -495,6 +504,20 @@ enum drm_asahi_feature { */ #define DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION ((1ULL) << 34) +/* + * DRM_ASAHI_FEATURE_GEM_RESIDENCY: DRM_IOCTL_ASAHI_GEM_RESIDENCY is + * available. A feature bit like those of enum drm_asahi_feature, defined + * here like bit 32. + */ +#define DRM_ASAHI_FEATURE_GEM_RESIDENCY ((1ULL) << 35) + +/* + * DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT: drm_asahi_queue_create::compute_timeout_ms + * sets the watchdog allowance of the queue's compute commands. A feature bit + * like those of enum drm_asahi_feature, defined here like bit 32. + */ +#define DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT ((1ULL) << 36) + /* * DRM_ASAHI_FEATURE_USERPTR: DRM_IOCTL_ASAHI_GEM_USERPTR is available. A * feature bit like those of enum drm_asahi_feature, defined here like bit 32. @@ -832,6 +855,68 @@ struct drm_asahi_gem_madvise { __u32 pad; }; +/** + * enum drm_asahi_gem_residency_flags - Flags for DRM_IOCTL_ASAHI_GEM_RESIDENCY + */ +enum drm_asahi_gem_residency_flags { + /** + * @DRM_ASAHI_GEM_RESIDENCY_KEEP: When reclaim releases idle memory of + * the object's VM, release this object's memory after that of the + * VM's objects without the flag, and only if theirs was not enough. + * Without the flag, the object returns to the default policy. + */ + DRM_ASAHI_GEM_RESIDENCY_KEEP = (1L << 0), +}; + +/** + * struct drm_asahi_gem_residency - Arguments passed to + * DRM_IOCTL_ASAHI_GEM_RESIDENCY + * + * When the system reclaims memory, the kernel may release the memory of the + * VM-private objects of an idle VM. It brings the memory back before the + * VM's next job runs, which then waits for it. + * DRM_ASAHI_GEM_RESIDENCY_KEEP is a priority among the objects that may be + * released this way. It changes the order in which they go, not whether + * reclaim gets to them: + * + * - Only reclaim on behalf of the whole system releases GPU memory. Reclaim + * inside a memory cgroup below the root, such as that of a cgroup at its + * memory limit, releases none, kept or not. + * - Reclaim releases memory of a VM only while the VM is idle: no job, + * binding change or other use of the VM is in progress, the last one (or + * the VM's creation) is at least two seconds back, and after a job had to + * wait for released memory, a pause that grows with the time that took + * (between 1 and 30 seconds) is over. The memory of a VM that is not idle + * stays, kept or not. + * - Of an idle VM's objects, a reclaim pass releases those without the flag + * first, and kept objects only if that left the amount the pass set out + * to free unmet. + * - Contents userspace gave up with DRM_ASAHI_GEM_MADV_DONTNEED are purged + * as before, kept or not. + * + * The request is therefore a hint. Userspace must not rely on a kept object + * being resident, and a client that stays active, or whose memory cgroup is + * the one that runs out of memory, keeps its GPU memory whether it asked or + * not. + * + * The request applies once this ioctl returns; memory already released stays + * so until the VM's next job or binding brings it back. Other objects + * (shared, imported or GEM_USERPTR ones) are never released and accept the + * flag without effect on them. + * + * Kept objects still count against the memory limits as before; the device + * total of their sizes is reported as user_kept in the GPU's gpu_memory + * sysfs file. Available when DRM_ASAHI_FEATURE_GEM_RESIDENCY is set; + * otherwise the ioctl fails with EOPNOTSUPP or EINVAL. + */ +struct drm_asahi_gem_residency { + /** @handle: Handle of the object. */ + __u32 handle; + + /** @flags: Combination of drm_asahi_gem_residency_flags flags. */ + __u32 flags; +}; + /** * enum drm_asahi_bind_flags - Flags for GEM binding */ @@ -1112,6 +1197,59 @@ struct drm_asahi_queue_create { * Requires DRM_ASAHI_FEATURE_RENDER_RAY_QUERY; zero otherwise. */ __u64 render_scs_layout[2]; + + /** + * @compute_timeout_ms: Watchdog allowance of each compute command of + * this queue, in milliseconds. Zero keeps the driver's default. + * Requires DRM_ASAHI_FEATURE_COMPUTE_TIMEOUT; zero otherwise. + * + * The driver's watchdog polls a queue every two seconds. By default it + * takes a compute command that a poll finds still running at the head + * of the queue (every earlier compute command of the queue has + * completed) to have hung, and fails it, and the queue's VM with it. + * With an allowance, the watchdog lets the command run on at each poll + * until the allowance has passed. For a nonzero request the driver + * raises a shorter allowance to its default of one poll, lowers a + * longer one to the maximum the system allows, and returns the + * allowance in effect here; a request of 0 is returned unchanged. + * + * This is an allowance, not a deadline: + * + * - The watchdog acts at its polls only, so a command may run up to + * one poll longer than its allowance. + * - The start of a command is known only approximately. The time is + * counted from one poll before the first poll that finds the command + * at the head of the queue. That is no earlier than the command got + * there and may be later: commands submitted to the queue in the + * meantime postpone that poll. + * - Once the allowance has passed, the deferrals the watchdog grants + * any command still apply, as they do by default: a command it takes + * to be waiting behind other clients' work, or one of a kind it + * gives extra polls, runs on until those are used up as well. The + * polls it renews any command for while the GPU keeps completing + * other work (up to 30 s in all, on the GPUs where it does that) + * run alongside the allowance, not after it: a command is renewed + * for the longer of the two. + * + * The allowance is per command, and nothing bounds the sum: a queue + * that is kept full of long commands occupies the GPU for as long as + * it is fed. The driver does not shield the other clients from such a + * queue: they share the GPU with it as the firmware schedules them, + * and a command of another client may have to wait until a long + * command ends. The watchdog does not fail such a waiting command, + * render or compute, while the earlier command it waits behind runs + * inside its allowance; afterwards the usual deferrals apply. A + * waiting command is excused this way for at most the longest + * allowance the system permits in total, so a command that keeps + * waiting behind other clients' work longer than that and its own + * deferrals is still failed. Userspace should ask for an allowance + * only for queues whose commands need one, and keep those commands as + * short as it can. + */ + __u32 compute_timeout_ms; + + /** @pad: MBZ */ + __u32 pad; }; /** @@ -2281,6 +2419,7 @@ enum { DRM_IOCTL_ASAHI_SUBMIT = DRM_IOCTL_ASAHI(W, SUBMIT, submit), DRM_IOCTL_ASAHI_GEM_MADVISE = DRM_IOCTL_ASAHI(WR, GEM_MADVISE, gem_madvise), DRM_IOCTL_ASAHI_GEM_USERPTR = DRM_IOCTL_ASAHI(WR, GEM_USERPTR, gem_userptr), + DRM_IOCTL_ASAHI_GEM_RESIDENCY = DRM_IOCTL_ASAHI(W, GEM_RESIDENCY, gem_residency), }; #if defined(__cplusplus) diff --git a/rust/kernel/drm/file.rs b/rust/kernel/drm/file.rs index 81f0a90e4a517..055440f5c52ea 100644 --- a/rust/kernel/drm/file.rs +++ b/rust/kernel/drm/file.rs @@ -49,6 +49,16 @@ pub trait DriverFile { /// Engine names must be stable, unique and contain no whitespace or colons. /// The default emits no engine keys. See `Documentation/gpu/drm-usage-stats.rst`. fn engine_stats(&self, _emit: impl FnMut(&crate::str::CStr, u64)) {} + + /// The frequency the engines of [`Self::engine_stats`] run at now, if known, and their + /// highest, in Hz, for this client's fdinfo (`drm-curfreq-` and + /// `drm-maxfreq-` beside each `drm-engine-`). The default, `None`, + /// prints neither. Every engine gets the same pair: this suits a device whose engines + /// share one clock, and a driver with independently clocked engines needs a per-engine + /// form instead. + fn engine_frequency(&self) -> Option<(Option, u64)> { + None + } } /// An open DRM File. @@ -141,6 +151,7 @@ impl File { // `driver_priv` was set by `open_callback()`. The seq_file read holds a reference to the // file, so `postclose_callback()` cannot run concurrently. let file = unsafe { File::::from_raw(raw_file) }; + let frequency = file.inner().engine_frequency(); file.inner().engine_stats(|name, ns| { // SAFETY: The DRM core owns printer throughout this callback; the format and // name are NUL-terminated, and the vararg types match %s and %llu. @@ -152,6 +163,29 @@ impl File { ns as core::ffi::c_ulonglong, ) }; + let Some((current, max)) = frequency else { + return; + }; + if let Some(current) = current { + // SAFETY: As above. + unsafe { + bindings::drm_printf( + printer, + c"drm-curfreq-%s:\t%llu Hz\n".as_char_ptr(), + name.as_char_ptr(), + current as core::ffi::c_ulonglong, + ) + }; + } + // SAFETY: As above. + unsafe { + bindings::drm_printf( + printer, + c"drm-maxfreq-%s:\t%llu Hz\n".as_char_ptr(), + name.as_char_ptr(), + max as core::ffi::c_ulonglong, + ) + }; }); let Some(stats) = file.inner().memory_stats() else { return; diff --git a/tools/asahi/g17-telemetry-integration-tests.py b/tools/asahi/g17-telemetry-integration-tests.py index a9a0f7c4446f0..8dde658062689 100644 --- a/tools/asahi/g17-telemetry-integration-tests.py +++ b/tools/asahi/g17-telemetry-integration-tests.py @@ -146,7 +146,7 @@ def item(source, signature): impl State { fn lock(&self)->std::sync::MutexGuard<'_,Option>> { self.0.lock().unwrap() } } - struct Variant { graph:fw::variant::InitGraph } + struct Variant { graph:fw::variant::InitGraph, m5_commands:bool } struct Shared { state:State, dev:crate::device::Device, variant:Variant, crashed:std::sync::atomic::AtomicBool } mod reports; ''' @@ -154,10 +154,10 @@ def item(source, signature): fn reader(graph:fw::variant::InitGraph)->stats::Reader { stats::Reader(std::sync::Arc::new(Shared { state:State(std::sync::Mutex::new(Some(Box::new(Firmware { - rings:channel::Rings::new(graph).unwrap(), + rings:channel::Rings::new(graph,false).unwrap(), init:initdata::InitData::new(),powered:true,epoch:0,recovery:Recovery(false), - })))),dev:crate::device::Device,variant:Variant { graph },crashed:false.into(), - })) + })))),dev:crate::device::Device,variant:Variant { graph, m5_commands:false },crashed:false.into(), + }),None) } #[test] @@ -255,11 +255,11 @@ def item(source, signature): let dev=crate::device::Device; for graph in [InitGraph::V25G83,InitGraph::G17P] { let init=initdata::InitData::new(); - let mut rings=Rings::new(graph).unwrap(); + let mut rings=Rings::new(graph,false).unwrap(); for words in [HIST_IDLE,HIST_ACTIVE,HIST_OTHER,TRANSITION] { init.push(Secondary,words); rings.drain_statistics(&dev,&init,Secondary); - assert_eq!(rings.read_statistics(1),telemetry::Readout::default()); + assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::UNSUPPORTED,..Default::default() }); assert!(report(&rings).contains(&format!("words={:08x?}",words))); } } @@ -271,7 +271,7 @@ def item(source, signature): let dev=crate::device::Device; for start in [0,5,7] { let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); for &role in &[Primary,Secondary] { for cursor in &init.cursors[role as usize] { cursor.store(start,Ordering::Relaxed); } for words in [HIST_OTHER,HIST_ACTIVE,TRANSITION] { init.push(role,words); } @@ -286,7 +286,7 @@ def item(source, signature): let line=text.lines().find(|line|line.starts_with(&format!("role={} tag={:#x} ",role,tag))).unwrap(); assert!(line.contains("count=2 host_ns=1 admitted=1 last_admitted=true"),"{}",line); } } - assert_eq!(rings.read_statistics(1),telemetry::Readout::default()); + assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::UNSUPPORTED,..Default::default() }); } } @@ -295,7 +295,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); let util=[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]; let pstate=[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]; rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],true,1); @@ -326,21 +326,22 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; for graph in [InitGraph::V25G83,InitGraph::G17P] { - assert!(!stats::reports_pstate(graph)); - let empty=stats::qualify(graph,telemetry::Readout::default()); + assert!(!stats::reports_pstate(graph,false)); + assert!(stats::reports_pstate(graph,true)); + let empty=stats::qualify(graph,false,telemetry::Readout::default()); assert_eq!((empty.util_status,empty.pstate_status),(telemetry::UNAVAILABLE,telemetry::UNSUPPORTED)); let init=initdata::InitData::new(); - let mut rings=Rings::new(graph).unwrap(); + let mut rings=Rings::new(graph,false).unwrap(); init.push(Primary,[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]); // Even a decodable tag 0x10 does not override the firmware's match data. init.push(Primary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); rings.drain_statistics(&dev,&init,Primary); assert_eq!(rings.read_statistics(1).pstate_status,telemetry::VALID); - let view=stats::qualify(graph,rings.read_statistics(1)); + let view=stats::qualify(graph,false,rings.read_statistics(1)); assert_eq!((view.util_status,view.device,view.pstate_status,view.pstate,view.pstate_timestamp), (telemetry::VALID,97,telemetry::UNSUPPORTED,0,0)); rings.invalidate_statistics(); - let view=stats::qualify(graph,rings.read_statistics(1)); + let view=stats::qualify(graph,false,rings.read_statistics(1)); assert_eq!((view.util_status,view.pstate_status),(telemetry::STALE,telemetry::UNSUPPORTED)); } } @@ -350,7 +351,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::V25G83).unwrap(); + let mut rings=Rings::new(InitGraph::V25G83,false).unwrap(); let util=[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]; // Three 8 ms records per producer update, one update every 24 ms. for n in 0..3u64 { @@ -370,7 +371,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],false,0); init.push(Secondary,HIST_IDLE); rings.drain_statistics(&dev,&init,Secondary); @@ -390,13 +391,13 @@ def item(source, signature): let graph=InitGraph::G17P; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(graph).unwrap(); + let mut rings=Rings::new(graph,false).unwrap(); init.push(Secondary,HIST_IDLE); rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],true,1); init.push(Secondary,HIST_ACTIVE); let shared=std::sync::Arc::new(Shared { state:State(std::sync::Mutex::new(Some(Box::new(Firmware { rings,init:initdata::InitData::new(),powered:true,epoch:0,recovery:Recovery(false), - })))),dev,variant:Variant { graph },crashed:false.into() }); + })))),dev,variant:Variant { graph, m5_commands:false },crashed:false.into() }); let view=reports::View(shared.clone()); struct View<'a>(&'a reports::View); impl std::fmt::Display for View<'_> { @@ -414,7 +415,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); let sample=|ta,render,device|[0x13,0,0,ta,render,0xdeadbeef,device,0,0,0,0,0,0,0,0,0,0,0]; init.push(Primary,sample(25,75,50)); init.push(Secondary,sample(35,85,65)); @@ -438,7 +439,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::V25G83).unwrap(); + let mut rings=Rings::new(InitGraph::V25G83,false).unwrap(); let words=[0x13,0,0,25,75,0,50,0,0,0,0,0,0,0,0,0,0,0]; init.push(Primary,words); rings.drain_statistics(&dev,&init,Primary); @@ -467,7 +468,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P).unwrap(); + let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); init.push(Primary,[0x10,0x1000,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0]); init.push(Secondary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); rings.drain_statistics(&dev,&init,Primary); @@ -687,7 +688,9 @@ def main(): header = (RUST_HEADER.replace('M5_STATISTICS_TYPES', m5_types).replace('M5_ASSERTS_GPU_POWER', m5_power) .replace('INITDATA_FAMILY', family)) - reader_rs = 'pub(crate) struct Reader(pub(super) Arc);\n' + item(stats_rs, 'impl Reader') + reader_rs = ('pub(crate) struct Reader(pub(super) Arc, ' + 'pub(super) Option<&\'static [(u32, [u32; 8], [u32; 8]); 14]>);\n' + + item(stats_rs, 'impl Reader').replace('super::fw::m5::initdata::M5PerfState', '(u32, [u32; 8], [u32; 8])')) (work / 'rings.rs').write_text(header + 'mod stats { use super::{fw::variant::InitGraph, telemetry::Readout, Shared};\n' + 'use std::sync::{Arc,atomic::Ordering};\n' + reader_rs + '\n' + policy + '\n}\nmod telemetry;\nmod channel;\n' + RUST_TESTS) diff --git a/tools/asahi/legacy-frequency-tests.rs b/tools/asahi/legacy-frequency-tests.rs index 18f1e4d33632e..84baee6ad1e16 100644 --- a/tools/asahi/legacy-frequency-tests.rs +++ b/tools/asahi/legacy-frequency-tests.rs @@ -9,6 +9,11 @@ mod prelude { mod g17 { pub mod stats { pub struct Reader; + impl Reader { + pub fn frequency(&self) -> Option<(Option, u64)> { + Some((None, 1_620_000_000)) + } + } } } #[path = "../../drivers/gpu/drm/asahi/stats.rs"] @@ -27,3 +32,13 @@ fn firmware_order_zero_and_invalid_indices() { assert_eq!(snapshot.requested_frequency(u32::MAX), None); assert_eq!(stats::StatsSnapshot::default().requested_frequency(0), None); } + +#[test] +fn engine_frequency_needs_the_g17_reader() { + assert_eq!(stats::StatsSnapshot::default().engine_frequency(), None); + let snapshot = stats::StatsSnapshot { + g17_reader: Some(g17::stats::Reader), + ..Default::default() + }; + assert_eq!(snapshot.engine_frequency(), Some((None, 1_620_000_000))); +} diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 5ec4864f5eb22..8394e597c5db6 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -25,6 +25,7 @@ TARGETS += drivers/dma-buf TARGETS += drivers/gpu/apple-afkv2 TARGETS += drivers/gpu/apple-dcp TARGETS += drivers/gpu/asahi-m4 +TARGETS += drivers/gpu/asahi-residency TARGETS += drivers/gpu/asahi-userptr TARGETS += drivers/ntsync TARGETS += drivers/s390x/uvdevice diff --git a/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py b/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py index 7b9bbce635e6e..f9da7cd001c91 100644 --- a/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py +++ b/tools/testing/selftests/drivers/gpu/asahi-m4/abi.py @@ -62,15 +62,13 @@ def run_measurements(command, source, binary): def c_layout(header, bindings, output, tag): keys, _ = expressions(bindings) capacity_tail = 'client_memory_limit;' in header.read_text() + queue_tail = 'compute_timeout_ms;' in header.read_text() # Linux 7.2 folds the historical extension structs into the base payloads. # Compare the same wire prefixes and offsets while keeping the C oracle frozen. if 'struct drm_asahi_cmd_render_with_prior {' not in header.read_text(): translated = [] for key, expr in keys: kind, _, field = key.partition(':') - if capacity_tail and key == 'drm_asahi_params_global:size': - # Compare the frozen reply prefix; validate the entire known additive tail below. - expr = 'offsetof(struct drm_asahi_params_global, memory_limit)' if key == "drm_asahi_vm_create:pad": expr = "offsetof(struct drm_asahi_vm_create, flags)" base = None @@ -90,6 +88,19 @@ def c_layout(header, bindings, output, tag): else: expr = f'offsetof(struct {base}, {field})' translated.append((key, expr)) keys = translated + # The two additive tails apply to every header that carries them, including + # one that still has the historical extension structs. Compare the frozen + # reply prefixes; validate the entire known additive tails below. + tails = {} + if capacity_tail: + tails['drm_asahi_params_global:size'] = \ + 'offsetof(struct drm_asahi_params_global, memory_limit)' + if queue_tail: + tails['drm_asahi_queue_create:size'] = \ + 'offsetof(struct drm_asahi_queue_create, compute_timeout_ms)' + if tails: + keys = [(key, tails[key]) if key in tails else (key, expr) + for key, expr in keys] source = output / (tag + '.c') lines = ['#include ', '#include ', '#include ' + json.dumps(str(header)), 'int main(void) {'] @@ -101,6 +112,14 @@ def c_layout(header, bindings, output, tag): '_Static_assert(sizeof(((struct drm_asahi_params_global *)0)->memory_limit) == 8, "capacity width");', '_Static_assert(sizeof(((struct drm_asahi_params_global *)0)->client_memory_limit) == 8, "client capacity width");', ] + if queue_tail: + lines[3:3] = [ + '_Static_assert(sizeof(struct drm_asahi_queue_create) == 48, "queue time limit reply size");', + '_Static_assert(offsetof(struct drm_asahi_queue_create, compute_timeout_ms) == 40, "queue time limit offset");', + '_Static_assert(sizeof(((struct drm_asahi_queue_create *)0)->compute_timeout_ms) == 4, "queue time limit width");', + '_Static_assert(offsetof(struct drm_asahi_queue_create, pad) == 44, "queue time limit pad offset");', + '_Static_assert(sizeof(((struct drm_asahi_queue_create *)0)->pad) == 4, "queue time limit pad width");', + ] lines += [f'printf("{key}=%llu\\n", (unsigned long long)({expr}));' for key, expr in keys] source.write_text('\n'.join(lines + ['return 0;', '}']) + '\n') return run_measurements(shlex.split(os.environ.get('CC', 'cc')) + diff --git a/tools/testing/selftests/drivers/gpu/asahi-residency/Makefile b/tools/testing/selftests/drivers/gpu/asahi-residency/Makefile new file mode 100644 index 0000000000000..a4c170ceb9df1 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/asahi-residency/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-only OR MIT +CFLAGS += -O2 -Wall $(KHDR_INCLUDES) +LDLIBS += -lpthread +TEST_GEN_PROGS := residency +include ../../../lib.mk diff --git a/tools/testing/selftests/drivers/gpu/asahi-residency/residency.c b/tools/testing/selftests/drivers/gpu/asahi-residency/residency.c new file mode 100644 index 0000000000000..7a8fc7a36ccd0 --- /dev/null +++ b/tools/testing/selftests/drivers/gpu/asahi-residency/residency.c @@ -0,0 +1,1397 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * DRM_IOCTL_ASAHI_GEM_RESIDENCY on the GPU's render node: what it accepts and + * refuses, the accounting (user_kept in the GPU's gpu_memory sysfs file, a + * device total: run it while no other client changes the residency of its + * objects) and what reclaim does with kept objects. No GPU work is submitted. + * Skipped where the render node is not this driver's or lacks the feature. + * + * The cases that make the kernel reclaim memory act on the whole system. + * They need root and run only when asked for in the environment: + * + * ASAHI_RESIDENCY_RECLAIM=1 runs light_reclaim_spares_kept_objects, + * kept_objects_stay_purgeable and requests_serialize_with_reclaim, which + * write to the memory.reclaim file of the root memory cgroup. + * + * ASAHI_RESIDENCY_DROP_CACHES=1 runs kept_objects_go_under_urgent_pressure, + * which writes 2 to /proc/sys/vm/drop_caches. That runs every shrinker of + * the system at the last reclaim priority: it drops the dentry and inode + * caches, with the page cache of the files they stood for, and the idle + * memory of every GPU client. + * + * A request to the root memory cgroup names an amount, not a priority. The + * kernel starts at its lightest reclaim priority and goes on to the more + * urgent ones until it has reclaimed that much, anywhere in the system, and + * runs the system's shrinkers at each priority it comes to. Where little + * else can be reclaimed, a request of a few MiB ends at the last priorities, + * at which the kept objects of idle VMs are released like everything else + * the system can give. The cases therefore bring the memory to reclaim + * themselves: a sparse temporary file in the working directory, read once, + * which leaves COLD_BYTES of clean page cache that nothing else uses. As + * long as plenty of it is left, a request usually finds its pages there + * within the first few priorities. + * + * Usually is not always, and the system reclaims of its own accord as well, + * so a case in which a kept object was released has to find out whether + * reclaim had leave to release it. There are two ways, and the cases say + * which one they went by: + * + * - The record of the shrinker passes (preferred). The driver releases kept + * objects from a shrinker of their own, one per VM, which the shrinker + * core calls only in a pass whose target reaches everything that shrinker + * counted. The program records the vmscan:mm_shrink_slab_start and + * mm_shrink_slab_end events in a trace instance of its own (tracefs at + * /sys/kernel/tracing, with kernel symbols). The events name a shrinker by + * a pointer, which says nothing of the VM, so a case knows its own VM's + * shrinker by what it counts: the pages of the VM's idle kept objects, + * which the case makes a number of its own (see own_pages()). A kept + * object that went is a failure unless a pass called that shrinker before + * the case first saw the object gone and had not ended when the case last + * saw it resident; with such a pass the case is skipped, since the + * release was allowed. Where more than one shrinker of + * the record counts what the test's VM counts and a pass called one of + * them, the record cannot tell whose pass it was: the case is skipped as + * inconclusive, which says nothing about the release. + * - Signs of the environment (the fallback, where the passes cannot be + * recorded). After every request the cases note whether the kernel met it + * and whether the file still has COLD_FLOOR in the cache, and at the end + * whether kswapd or direct reclaim scanned pages meanwhile. A case in + * which a kept object was released after a request that was not met, with + * less of the file left, or while the system reclaimed as well, is + * skipped with that reason. These are heuristics: they make it likely, and + * do not show, that reclaim went to its last priorities, and reclaim that + * went there with every request met and the cache left would fail the + * case. + * + * These cases also need four times their objects' size of available memory + * (objects of RECLAIM_PAGES pages: 8 GiB each with 16K pages), a working + * directory on a file system with a page cache, and a memory cgroup without + * memory.min or memory.low protection, which would keep reclaim away from + * the cold cache. + */ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../../kselftest_harness.h" + +#ifndef __NR_cachestat +#define __NR_cachestat 451 +#endif + +#define PAGE 0x4000UL + +/* + * The idle pages a VM must have for the lightest reclaim pass to release + * some: the shrinker core scans a shrinker once the count it reported, + * divided by 2^12 at that priority and doubled, reaches the shrinker's batch + * (256 pages for this driver). + */ +#define RECLAIM_PAGES (256UL << 11) + +/* The time a VM must have been left alone before reclaim touches it, and more. */ +#define IDLE_SECONDS 3 + +/* What one reclaim request asks for. */ +#define REQUEST "4M" + +/* The clean page cache the reclaim cases bring, and how much of it must be left. */ +#define COLD_BYTES (2ULL << 30) +#define COLD_FLOOR (COLD_BYTES / 4) + +/* The most requests a case makes to see reclaim release an object. */ +#define MAX_REQUESTS 128 + +/* The requests through which a kept object has to stay resident. */ +#define SPARED_REQUESTS 16 + +static int ioc(int fd, unsigned long cmd, void *arg) +{ + int ret; + + do { + ret = ioctl(fd, cmd, arg); + } while (ret == -1 && (errno == EINTR || errno == EAGAIN)); + return ret ? -errno : 0; +} + +/* A field of the GPU's gpu_memory sysfs file. */ +static long long gpu_memory_field(const char *key) +{ + char line[256]; + long long value = -1; + size_t n = strlen(key); + FILE *f = fopen("/sys/class/drm/renderD128/device/gpu_memory", "r"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (!strncmp(line, key, n) && line[n] == ' ') + value = atoll(line + n + 1); + fclose(f); + return value; +} + +/* drm-resident-system of the client behind @fd, in bytes: the memory it holds. */ +static long long client_resident(int fd) +{ + char path[64], line[256], unit[8]; + long long value = -1; + FILE *f; + + snprintf(path, sizeof(path), "/proc/self/fdinfo/%d", fd); + f = fopen(path, "r"); + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) { + unit[0] = 0; + if (sscanf(line, "drm-resident-system: %lld %7s", &value, unit) < 1) + continue; + if (!strcmp(unit, "KiB")) + value <<= 10; + else if (!strcmp(unit, "MiB")) + value <<= 20; + break; + } + fclose(f); + return value; +} + +/* MemAvailable in bytes. */ +static long long mem_available(void) +{ + char line[256]; + long long kb = -1; + FILE *f = fopen("/proc/meminfo", "r"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "MemAvailable: %lld kB", &kb) == 1) + break; + fclose(f); + return kb < 0 ? -1 : kb << 10; +} + +static int write_string(const char *path, const char *text) +{ + int fd = open(path, O_WRONLY | O_CLOEXEC); + int ret; + + if (fd < 0) + return -errno; + ret = write(fd, text, strlen(text)) < 0 ? -errno : 0; + close(fd); + return ret; +} + +#define MEMORY_RECLAIM "/sys/fs/cgroup/memory.reclaim" + +/* The bytes of the cold cache file @fd that are in the page cache, or -errno. */ +static long long cold_cached(int fd) +{ + struct cachestat_range range = { .off = 0, .len = COLD_BYTES }; + struct cachestat cs; + + if (syscall(__NR_cachestat, fd, &range, &cs, 0)) + return -errno; + return (long long)cs.nr_cache * sysconf(_SC_PAGESIZE); +} + +/* + * Creates the cold cache: an unnamed sparse file of COLD_BYTES in the working + * directory, read once from end to end. Reading a hole leaves clean pages in + * the page cache without writing to the disk, and pages read once are the + * first ones reclaim takes. Returns the file, which holds the cache until it + * is closed, or -1 and the reason in @why. + */ +static int cold_cache(const char **why) +{ + static char buf[1 << 20]; + struct statfs fs; + int fd = open(".", O_TMPFILE | O_RDWR | O_CLOEXEC, 0600); + + *why = "cannot create a temporary file in the working directory"; + if (fd < 0) + return -1; + if (fstatfs(fd, &fs) || fs.f_type == TMPFS_MAGIC || + fs.f_type == RAMFS_MAGIC) { + *why = "the working directory is in memory: reclaim cannot drop its files"; + goto fail; + } + if (ftruncate(fd, COLD_BYTES)) + goto fail; + *why = "cannot read the temporary file into the page cache"; + for (uint64_t off = 0; off < COLD_BYTES; off += sizeof(buf)) + if (pread(fd, buf, sizeof(buf), off) != sizeof(buf)) + goto fail; + if (cold_cached(fd) < (long long)(COLD_BYTES - COLD_BYTES / 8)) { + *why = "the file system does not keep a file that was read in the page cache"; + goto fail; + } + return fd; +fail: + close(fd); + return -1; +} + +/* + * The pages kswapd and direct reclaim have scanned since boot: reclaim the + * system does of its own accord, at priorities the test knows nothing about. + */ +static long long scanned_unasked(void) +{ + char line[256]; + long long value, sum = 0; + FILE *f = fopen("/proc/vmstat", "r"); + + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) + if (sscanf(line, "pgscan_kswapd %lld", &value) == 1 || + sscanf(line, "pgscan_direct %lld", &value) == 1) + sum += value; + fclose(f); + return sum; +} + +/* + * Whether the memory cgroup of this process, or one above it, is protected + * from reclaim (memory.min, memory.low): reclaim then passes the cold cache + * by until it has tried everything else. + */ +static int cgroup_protected(void) +{ + char path[512] = "/sys/fs/cgroup", line[400], file[600], value[32]; + FILE *f = fopen("/proc/self/cgroup", "r"); + size_t root = strlen(path); + char *end; + + if (!f) + return 0; + while (fgets(line, sizeof(line), f)) + if (!strncmp(line, "0::", 3)) + snprintf(path + root, sizeof(path) - root, "%s", line + 3); + fclose(f); + path[strcspn(path, "\n")] = 0; + for (end = path + strlen(path); end > path + root; end = strrchr(path, '/')) { + static const char * const names[] = { "memory.min", "memory.low" }; + + *end = 0; + for (int i = 0; i < 2; i++) { + snprintf(file, sizeof(file), "%s/%s", path, names[i]); + f = fopen(file, "r"); + if (!f) + continue; + if (fgets(value, sizeof(value), f) && strcmp(value, "0\n")) { + fclose(f); + return 1; + } + fclose(f); + } + } + return 0; +} + +static double seconds(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return ts.tv_sec + ts.tv_nsec / 1e9; +} + +/* + * A number of pages that a case adds to the size of its kept object, drawn + * anew by every case: 0 to 1023 GPU pages. It makes what the VM's shrinker + * of kept objects counts a number that another VM's is unlikely to count, + * which is how the case finds its own shrinker in the record. + */ +static uint64_t own_pages(void) +{ + uint16_t r = 0; + + if (getrandom(&r, sizeof(r), 0) != sizeof(r)) + r = getpid() ^ time(NULL); + return r % 1024; +} + +/* + * The record of the shrinker core's passes over the shrinkers: a trace + * instance with the events that the core emits before it decides whether to + * call a shrinker and after it is done with it. + * + * The instance belongs to the program, not to a case. main() makes it before + * the cases fork, under a name with its process ID, and removes it when the + * program exits or is told to end, whatever became of the cases: a case + * that fails, times out or is killed leaves nothing to clean up. An + * instance of that name that exists already is somebody else's and is left + * alone; the cases then go without a record. A case only empties the buffer + * and switches the events on and off. + */ +#define TRACE_EVENTS "/events/vmscan/mm_shrink_slab_" + +static char trace_dir[80]; +/* The enable files of the two events, made with trace_dir. */ +static char trace_enable[2][160]; +static pid_t trace_owner; +/* The signals that end the program, blocked while the instance changes hands. */ +static const int trace_signals[] = { SIGINT, SIGTERM, SIGHUP, SIGQUIT }; +static sigset_t trace_sigset; + +/* Whether this case is recording. */ +static int tracing; + +static int trace_write(const char *file, const char *text) +{ + char path[160]; + + snprintf(path, sizeof(path), "%s%s", trace_dir, file); + return write_string(path, text); +} + +static void trace_events(const char *on) +{ + write_string(trace_enable[0], on); + write_string(trace_enable[1], on); +} + +/* + * Only async-signal-safe calls, on paths made beforehand, from here: it also + * runs from a signal handler. The signals that end the program are blocked + * meanwhile, so that it never runs twice at once. + */ +static void trace_remove(void) +{ + static const struct timespec pause = { .tv_nsec = 50000000 }; + sigset_t old; + + if (!trace_dir[0] || getpid() != trace_owner) + return; + sigprocmask(SIG_BLOCK, &trace_sigset, &old); + if (trace_dir[0]) { + trace_events("0"); + /* A case that was just killed may still hold a file of it. */ + for (int i = 0; i < 40 && rmdir(trace_dir) && errno == EBUSY; i++) + nanosleep(&pause, NULL); + trace_dir[0] = 0; + } + sigprocmask(SIG_SETMASK, &old, NULL); +} + +/* The signal stays blocked until the handler returns, and then ends the program. */ +static void trace_remove_on(int sig) +{ + trace_remove(); + signal(sig, SIG_DFL); + raise(sig); +} + +/* + * Makes the instance, if the reclaim cases were asked for. Its timestamps + * are on the clock of seconds(). + */ +static void trace_create(void) +{ + const char *reclaim = getenv("ASAHI_RESIDENCY_RECLAIM"); + const char *drop = getenv("ASAHI_RESIDENCY_DROP_CACHES"); + struct sigaction sa = { .sa_handler = trace_remove_on }; + sigset_t old; + + if (geteuid() || !((reclaim && !strcmp(reclaim, "1")) || + (drop && !strcmp(drop, "1")))) + return; + snprintf(trace_dir, sizeof(trace_dir), + "/sys/kernel/tracing/instances/asahi-residency-%d", getpid()); + snprintf(trace_enable[0], sizeof(trace_enable[0]), + "%s" TRACE_EVENTS "start/enable", trace_dir); + snprintf(trace_enable[1], sizeof(trace_enable[1]), + "%s" TRACE_EVENTS "end/enable", trace_dir); + sigemptyset(&trace_sigset); + for (unsigned int i = 0; i < ARRAY_SIZE(trace_signals); i++) + sigaddset(&trace_sigset, trace_signals[i]); + sa.sa_mask = trace_sigset; + + /* + * A signal waits until the instance is set up: until the handlers are + * in place, and until no file of the instance is open, which would + * keep the handler from removing it. + */ + sigprocmask(SIG_BLOCK, &trace_sigset, &old); + if (mkdir(trace_dir, 0700)) { + trace_dir[0] = 0; + } else { + trace_owner = getpid(); + atexit(trace_remove); + for (unsigned int i = 0; i < ARRAY_SIZE(trace_signals); i++) + sigaction(trace_signals[i], &sa, NULL); + if (trace_write("/trace_clock", "mono") || + trace_write("/buffer_size_kb", "4096")) + trace_remove(); + } + sigprocmask(SIG_SETMASK, &old, NULL); +} + +/* Starts the record of a case afresh. */ +static void trace_start(void) +{ + char path[160]; + int fd; + + if (!trace_dir[0]) + return; + snprintf(path, sizeof(path), "%s/trace", trace_dir); + fd = open(path, O_WRONLY | O_TRUNC | O_CLOEXEC); + if (fd < 0) + return; + close(fd); + if (write_string(trace_enable[0], "1") || + write_string(trace_enable[1], "1")) { + trace_events("0"); + return; + } + tracing = 1; +} + +static void trace_stop(void) +{ + if (!tracing) + return; + trace_events("0"); + tracing = 0; +} + +/* One of the driver's shrinker events. */ +struct shrink_event { + /* mm_shrink_slab_end, not _start. */ + int end; + /* Of a shrinker of kept objects, not of a VM's other shrinker. */ + int kept; + double time; + /* The shrinker, as the event prints its pointer. */ + char shrinker[24]; + /* The task that makes the pass. */ + int pid; + /* Of a start event: what the shrinker counted, the target, the priority. */ + long items, target; + int priority; +}; + +/* + * Reads a line of the record. Returns 1 for an event of one of the driver's + * shrinkers, which are told by the names of their scan functions (scan_kept + * of VmShrinker for the kept objects of a VM, VmShrinker's other one for the + * rest), 0 for any other line and -1 for an event that does not read as + * expected. + */ +static int shrink_event(const char *line, struct shrink_event *e) +{ + const char *at = strstr(line, ": mm_shrink_slab_"); + const char *nid, *p, *items, *target, *priority; + size_t n; + + if (line[0] == '#' || !at) + return 0; + for (p = at; p > line && p[-1] != ' '; p--) + ; + e->time = strtod(p, NULL); + at += strlen(": mm_shrink_slab_"); + e->end = !strncmp(at, "end: ", 5); + if (!e->end && strncmp(at, "start: ", 7)) + return 0; + at = strchr(at, ' ') + 1; + /* " : nid: ..." */ + nid = strstr(at, ": nid: "); + if (!nid) + return -1; + for (p = nid; p > at && p[-1] != ' '; p--) + ; + n = nid - p; + if (p == at || !n || n >= sizeof(e->shrinker)) + return -1; + if (!memmem(at, p - at, "VmShrinker", strlen("VmShrinker"))) + return 0; + e->kept = !!memmem(at, p - at, "scan_kept", strlen("scan_kept")); + memcpy(e->shrinker, p, n); + e->shrinker[n] = 0; + p = strstr(nid, " pid: "); + if (!p) + return -1; + e->pid = atoi(p + strlen(" pid: ")); + if (e->end) + return 1; + items = strstr(nid, " cache items "); + target = strstr(nid, " total_scan "); + priority = strstr(nid, " priority "); + if (!items || !target || !priority) + return -1; + e->items = atol(items + strlen(" cache items ")); + e->target = atol(target + strlen(" total_scan ")); + e->priority = atoi(priority + strlen(" priority ")); + return 1; +} + +/* What a case knows of its own VM for reading the record. */ +struct own_vm { + /* + * What the VM's shrinker of kept objects can have counted since the + * record began: the pages of the idle kept objects, in the one or two + * combinations the case brings about. + */ + long counts[2]; + /* + * When the case last saw its kept objects resident (taken before it + * looked) and first saw one gone (taken after it looked); 0 and + * infinity while it did not. + */ + double held_at, gone_at; +}; + +/* What the record has about the shrinkers of kept objects. */ +struct kept_record { + /* Passes over the VMs' other shrinker: the record has the driver's events. */ + int others; + /* Shrinkers that only ever counted what the test's VM counts. */ + int shrinkers; + /* Passes over those, and the ones among them that called the shrinker. */ + int passes, called; + /* The called ones that could have released the object that went. */ + int released; + /* The most urgent priority among those, or among the called ones. */ + int priority; +}; + +#define MAX_SHRINKERS 64 +#define MAX_CALLS 256 + +/* + * Reads the record for the case whose VM is @vm. The shrinker core calls a + * shrinker when the target of the pass (total_scan) reaches the shrinker's + * batch or everything it counted (cache items); the batch of a shrinker of + * kept objects is out of reach, which leaves the second. + * + * The record names shrinkers by pointer. The one of the test's VM is among + * those that never counted anything but @vm->counts; there is one such + * shrinker unless another VM's happens to count the same. + * + * A pass could have released an object that the case saw resident at + * held_at and gone at gone_at if it called the shrinker before gone_at and + * had not ended by held_at. A pass has ended with the next end event of the + * same shrinker from the same task. + * + * Returns 0, or -1 where there is no record, events were lost or it has + * more shrinkers or calls than this keeps track of. + */ +static int read_record(const struct own_vm *vm, struct kept_record *r) +{ + static struct { + char shrinker[24]; + int foreign, passes; + } seen[MAX_SHRINKERS]; + static struct { + /* pid: the task, and once the record is read, whether it is in the interval. */ + int shrinker, pid, priority; + double start, end; + } calls[MAX_CALLS]; + int nseen = 0, ncalls = 0, whole = 0, k; + long in_buffer, written; + char path[160], line[1024]; + struct shrink_event e; + FILE *f; + + memset(r, 0, sizeof(*r)); + r->priority = -1; + if (!tracing) + return -1; + snprintf(path, sizeof(path), "%s/trace", trace_dir); + f = fopen(path, "r"); + if (!f) + return -1; + while (fgets(line, sizeof(line), f)) { + int ret = shrink_event(line, &e); + + if (sscanf(line, "# entries-in-buffer/entries-written: %ld/%ld", + &in_buffer, &written) == 2) + whole = in_buffer == written; + if (ret < 0) + whole = 0; + if (ret <= 0) + continue; + if (!e.kept) { + r->others += !e.end; + continue; + } + for (k = 0; k < nseen && strcmp(seen[k].shrinker, e.shrinker); k++) + ; + if (e.end) { + for (int c = ncalls - 1; c >= 0; c--) { + if (calls[c].shrinker != k || calls[c].pid != e.pid || + !isinf(calls[c].end)) + continue; + calls[c].end = e.time; + break; + } + continue; + } + if (k == nseen) { + if (nseen == MAX_SHRINKERS) { + whole = 0; + break; + } + memset(&seen[k], 0, sizeof(seen[k])); + strcpy(seen[k].shrinker, e.shrinker); + nseen++; + } + seen[k].passes++; + if (e.items != vm->counts[0] && e.items != vm->counts[1]) + seen[k].foreign = 1; + if (e.target < e.items) + continue; + if (ncalls == MAX_CALLS) { + whole = 0; + break; + } + calls[ncalls].shrinker = k; + calls[ncalls].pid = e.pid; + calls[ncalls].priority = e.priority; + calls[ncalls].start = e.time; + calls[ncalls].end = INFINITY; + ncalls++; + } + fclose(f); + if (!whole) + return -1; + for (k = 0; k < nseen; k++) { + if (seen[k].foreign) + continue; + r->shrinkers++; + r->passes += seen[k].passes; + } + /* The timestamps of the record are rounded to a microsecond. */ + for (int c = 0; c < ncalls; c++) { + calls[c].pid = calls[c].start <= vm->gone_at + 2e-6 && + calls[c].end >= vm->held_at - 2e-6; + if (seen[calls[c].shrinker].foreign) + continue; + r->called++; + r->released += calls[c].pid; + } + for (int c = 0; c < ncalls; c++) { + if (seen[calls[c].shrinker].foreign || + (r->released && !calls[c].pid)) + continue; + if (r->priority < 0 || calls[c].priority < r->priority) + r->priority = calls[c].priority; + } + return 0; +} + +/* The reclaim requests of a case, and whether the kernel could meet them lightly. */ +struct pressure { + /* The cold cache file. */ + int cold; + /* The requests made. */ + int requests; + /* Those the kernel did not meet in full. */ + int unmet; + /* The least of the cold cache that was left after a request. */ + long long cold_low; + /* scanned_unasked() when the case began. */ + long long unasked; + /* For the record of the shrinker passes. */ + struct own_vm vm; +}; + +/* + * Asks the root memory cgroup to reclaim REQUEST. The kernel returns EAGAIN + * when it could not reclaim that much at any priority, which is counted; + * any other error is returned. + */ +static int reclaim_some(struct pressure *p) +{ + int ret = write_string(MEMORY_RECLAIM, REQUEST); + long long cached = cold_cached(p->cold); + + p->requests++; + if (ret == -EAGAIN) { + p->unmet++; + ret = 0; + } + if (cached < p->cold_low) + p->cold_low = cached; + return ret; +} + +/* + * A heuristic, for where the shrinker passes cannot be recorded: a sign that + * reclaim may have gone to its last priorities since the case began, or + * NULL. With every request met, COLD_FLOOR of the cold cache left after each + * and no other reclaim in the system, none is likely to have gone beyond the + * first few priorities; that is not shown. + */ +static const char *urgent_reclaim_signs(const struct pressure *p) +{ + if (p->unmet) + return "the kernel could not meet a reclaim request"; + if (p->cold_low < (long long)COLD_FLOOR) + return "the cold cache ran low"; + if (scanned_unasked() != p->unasked) + return "kswapd or direct reclaim ran meanwhile"; + return NULL; +} + +/* + * What to make of a kept object that went between p->vm.held_at and + * p->vm.gone_at. Returns 0 if reclaim had no leave to release it, so that + * the case fails, and 1 if the case is to be skipped; either way @why says + * on what grounds. + * + * By the record of the shrinker passes where there is one that shows the + * driver's shrinkers at work (the cases ask once reclaim has released an + * object that is not kept). Reclaim had leave if, in that interval, a pass + * called the test VM's shrinker of kept objects: "as it may". If the record + * has several shrinkers that count what the test's VM counts and a pass + * called one of them, it does not tell whether that was the test's: + * "inconclusive", which claims nothing about the release. With no such pass + * for any of them, reclaim had no leave. + * + * By the heuristic otherwise, and the words say so. + */ +static int kept_could_go(const struct pressure *p, char *why, size_t size) +{ + struct kept_record r; + const char *sign; + + if (!read_record(&p->vm, &r) && r.others) { + if (!r.released) + snprintf(why, size, + "no reclaim pass called the shrinker of this VM's kept objects before it went (traced: %d shrinkers of kept objects count what this VM's counts, with %d passes and %d calls at other times)", + r.shrinkers, r.passes, r.called); + else if (r.shrinkers == 1) + snprintf(why, size, + "as it may: reclaim passes that called the shrinker of this VM's kept objects before it went: %d, down to priority %d (traced)", + r.released, r.priority); + else + snprintf(why, size, + "inconclusive: %d shrinkers of kept objects count what this VM's counts, and %d passes called one of them before it went (traced)", + r.shrinkers, r.released); + return r.released > 0; + } + sign = urgent_reclaim_signs(p); + if (sign) + snprintf(why, size, + "as it may: %s (a heuristic: the shrinker passes were not traced)", + sign); + else + snprintf(why, size, + "no sign of urgent reclaim (a heuristic: the shrinker passes were not traced)"); + return sign != NULL; +} + +/* For the log of a case that passed: what the record of the shrinker passes has. */ +static const char *trace_note(const struct pressure *p, char *note, size_t size) +{ + struct kept_record r; + + if (!read_record(&p->vm, &r) && r.others) + snprintf(note, size, + "reclaim passes over the shrinker of this VM's kept objects: %d, of which %d called it (traced, %d such shrinker%s)", + r.passes, r.called, r.shrinkers, r.shrinkers == 1 ? "" : "s"); + else + snprintf(note, size, "the shrinker passes were not traced"); + return note; +} + +static int residency(int fd, uint32_t handle, uint32_t flags) +{ + struct drm_asahi_gem_residency args = { .handle = handle, .flags = flags }; + + return ioc(fd, DRM_IOCTL_ASAHI_GEM_RESIDENCY, &args); +} + +/* Gives advice for @handle; returns whether its contents are retained, or -errno. */ +static int madvise_object(int fd, uint32_t handle, uint32_t madv) +{ + struct drm_asahi_gem_madvise args = { .handle = handle, .madv = madv }; + int ret = ioc(fd, DRM_IOCTL_ASAHI_GEM_MADVISE, &args); + + return ret ? ret : (int)args.retained; +} + +static int bind(int fd, uint32_t vm, uint32_t handle, uint64_t addr, + uint64_t size, uint32_t flags) +{ + struct drm_asahi_gem_bind_op op = { + .flags = flags, .handle = handle, .range = size, .addr = addr, + }; + struct drm_asahi_vm_bind args = { + .vm_id = vm, .num_binds = 1, .stride = sizeof(op), + .userptr = (uintptr_t)&op, + }; + + return ioc(fd, DRM_IOCTL_ASAHI_VM_BIND, &args); +} + +static void close_handle(int fd, uint32_t handle) +{ + struct drm_gem_close args = { .handle = handle }; + + ioc(fd, DRM_IOCTL_GEM_CLOSE, &args); +} + +/* Whether the driver behind @fd is this one. */ +static int is_asahi(int fd) +{ + char name[16] = ""; + struct drm_version version = { + .name = name, .name_len = sizeof(name) - 1, + }; + + return !ioc(fd, DRM_IOCTL_VERSION, &version) && !strcmp(name, "aurora"); +} + +FIXTURE(asahi) +{ + int fd; + /* The cold cache file of a reclaim case, or -1. */ + int cold; + struct drm_asahi_params_global params; + uint32_t vm; + uint64_t va, va_end; +}; + +FIXTURE_SETUP(asahi) +{ + struct drm_asahi_get_params get = { + .pointer = (uintptr_t)&self->params, + .size = sizeof(self->params), + }; + + self->cold = -1; + self->fd = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC); + if (self->fd < 0 || !is_asahi(self->fd)) + SKIP(return, "renderD128 is not an asahi render node"); + memset(&self->params, 0, sizeof(self->params)); + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GET_PARAMS, &get)); + if (!(self->params.features & DRM_ASAHI_FEATURE_GEM_RESIDENCY)) + SKIP(return, "no DRM_ASAHI_FEATURE_GEM_RESIDENCY"); + + uint64_t kernel = self->params.vm_kernel_min_size > (32ULL << 30) ? + self->params.vm_kernel_min_size : 32ULL << 30; + struct drm_asahi_vm_create vm = { + .kernel_start = self->params.vm_end - kernel, + .kernel_end = self->params.vm_end, + }; + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_VM_CREATE, &vm)); + self->vm = vm.vm_id; + self->va = self->params.vm_start; + self->va_end = vm.kernel_start; +} + +FIXTURE_TEARDOWN(asahi) +{ + trace_stop(); + close(self->fd); + /* Closing the unnamed file gives its page cache back. */ + if (self->cold >= 0) + close(self->cold); +} + +/* + * Creates a VM-private object of @size bytes in @vm and binds it, which + * gives it its memory. The driver keeps ranges of every VM for its own + * mappings, where a bind fails with EINVAL, and GET_PARAMS does not list + * them: the object goes to the first address after *@va, in steps of 4 GiB + * and below @va_end, at which it binds, and *@va moves past it. Returns its + * handle, or 0. + */ +static uint32_t bound_object(int fd, uint32_t vm, uint64_t *va, uint64_t va_end, + uint64_t size) +{ + struct drm_asahi_gem_create bo = { + .size = size, .flags = DRM_ASAHI_GEM_VM_PRIVATE, .vm_id = vm, + }; + + if (ioc(fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)) + return 0; + for (;;) { + *va = (*va + (4ULL << 30)) & ~((4ULL << 30) - 1); + if (*va + size > va_end) + break; + if (!bind(fd, vm, bo.handle, *va, size, + DRM_ASAHI_BIND_READ | DRM_ASAHI_BIND_WRITE)) { + *va += size; + return bo.handle; + } + } + close_handle(fd, bo.handle); + return 0; +} + +/* + * Prepares a case that asks the root memory cgroup for reclaim, with objects + * of @bytes in all: checks that it was asked for and can run here, and + * creates the cold cache, whose file goes to *@cold. Returns NULL, or why + * the case is skipped. + */ +static const char *reclaim_setup(struct pressure *p, uint64_t bytes, int *cold) +{ + const char *run = getenv("ASAHI_RESIDENCY_RECLAIM"); + const char *why; + + if (!run || strcmp(run, "1")) + return "set ASAHI_RESIDENCY_RECLAIM=1 to ask the root memory cgroup for reclaim"; + if (geteuid()) + return "reclaim requests need root"; + if (access(MEMORY_RECLAIM, W_OK)) + return "no " MEMORY_RECLAIM; + if (mem_available() < (long long)(4 * bytes + 2 * COLD_BYTES)) + return "not enough available memory for the test objects"; + if (cgroup_protected()) + return "the test's memory cgroup has memory.min or memory.low protection"; + *cold = cold_cache(&why); + if (*cold < 0) + return why; + memset(p, 0, sizeof(*p)); + p->cold = *cold; + p->cold_low = cold_cached(*cold); + p->unasked = scanned_unasked(); + p->vm.gone_at = INFINITY; + trace_start(); + return NULL; +} + +/* + * Notes, for the record, an observation of the case's kept objects that + * began at @before: whether they were all resident (see struct own_vm). + */ +static void note_kept(struct own_vm *vm, double before, int intact) +{ + if (intact) { + vm->held_at = before; + vm->gone_at = INFINITY; + } else if (isinf(vm->gone_at)) { + vm->gone_at = seconds(); + } +} + +/* + * What the client behind @fd holds. Its kept objects are all resident while + * that is @resident or @kept, which is noted for the record. + */ +static long long watch_resident(struct pressure *p, int fd, long long resident, + long long kept) +{ + double before = seconds(); + long long now = client_resident(fd); + + note_kept(&p->vm, before, now == resident || now == kept); + return now; +} + +/* + * Makes reclaim requests, at most @most, for as long as the client behind + * @fd holds @resident bytes. @kept is what it holds once everything but its + * kept objects has gone. Returns what it holds then, or a negative value: + * -errno of a request that failed, or -1 where what the client holds could + * not be read. The callers assert that it is not negative before they take + * the result for a sign of anything. + */ +static long long reclaim_while(struct pressure *p, int fd, long long resident, + long long kept, int most) +{ + long long now = watch_resident(p, fd, resident, kept); + + for (int i = 0; i < most && now == resident; i++) { + int ret = reclaim_some(p); + + if (ret) + return ret; + now = watch_resident(p, fd, resident, kept); + } + return now; +} + +/* The kept total follows a VM-private object: once per object, until freed. */ +TEST_F(asahi, kept_size_is_counted_once) +{ + const uint64_t size = 16 * PAGE; + struct drm_asahi_gem_create bo = { + .size = size, .flags = DRM_ASAHI_GEM_VM_PRIVATE, .vm_id = self->vm, + }; + long long kept0; + + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)); + kept0 = gpu_memory_field("user_kept"); + ASSERT_GE(kept0, 0); + ASSERT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + ASSERT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0 + (long long)size, gpu_memory_field("user_kept")); + ASSERT_EQ(0, residency(self->fd, bo.handle, 0)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + ASSERT_EQ(0, residency(self->fd, bo.handle, 0)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + /* Freeing a kept object stops counting it. */ + ASSERT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0 + (long long)size, gpu_memory_field("user_kept")); + close_handle(self->fd, bo.handle); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); +} + +TEST_F(asahi, bad_arguments) +{ + struct drm_asahi_gem_create bo = { + .size = PAGE, .flags = DRM_ASAHI_GEM_VM_PRIVATE, .vm_id = self->vm, + }; + long long kept0 = gpu_memory_field("user_kept"); + + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)); + EXPECT_EQ(-EINVAL, residency(self->fd, bo.handle, 1u << 1)); + EXPECT_EQ(-EINVAL, residency(self->fd, bo.handle, + DRM_ASAHI_GEM_RESIDENCY_KEEP | 1u << 31)); + EXPECT_EQ(-ENOENT, residency(self->fd, 0xffffff, + DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + close_handle(self->fd, bo.handle); +} + +/* An object that is never released takes the flag all the same. */ +TEST_F(asahi, shared_objects_take_the_flag) +{ + const uint64_t size = 2 * PAGE; + struct drm_asahi_gem_create bo = { .size = size }; + long long kept0 = gpu_memory_field("user_kept"); + + ASSERT_EQ(0, ioc(self->fd, DRM_IOCTL_ASAHI_GEM_CREATE, &bo)); + EXPECT_EQ(0, residency(self->fd, bo.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + EXPECT_EQ(kept0 + (long long)size, gpu_memory_field("user_kept")); + EXPECT_EQ(0, residency(self->fd, bo.handle, 0)); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + close_handle(self->fd, bo.handle); +} + +/* + * Of two idle objects, reclaim releases the one that is not kept first, at + * any priority. Reclaim that does not come to the kept objects (see the top + * of this file for how the case knows) leaves the kept one resident; once + * the request is withdrawn, it is released as well. The kept object is the + * larger one, so that what the client holds tells which one stayed. + */ +TEST_F_TIMEOUT(asahi, light_reclaim_spares_kept_objects, 180) +{ + const uint64_t size = RECLAIM_PAGES * sysconf(_SC_PAGESIZE); + const uint64_t kept_size = size + (1024 + own_pages()) * PAGE; + const long long both = kept_size + size; + struct pressure p; + const char *why = reclaim_setup(&p, both, &self->cold); + long long resident; + char reason[320]; + uint32_t a, b; + int first; + + if (why) + SKIP(return, "%s", why); + p.vm.counts[0] = p.vm.counts[1] = kept_size / sysconf(_SC_PAGESIZE); + a = bound_object(self->fd, self->vm, &self->va, self->va_end, kept_size); + b = bound_object(self->fd, self->vm, &self->va, self->va_end, size); + ASSERT_NE(0, a); + ASSERT_NE(0, b); + ASSERT_EQ(both, client_resident(self->fd)); + ASSERT_EQ(0, residency(self->fd, a, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + + sleep(IDLE_SECONDS); + resident = reclaim_while(&p, self->fd, both, kept_size, MAX_REQUESTS); + first = p.requests; + ASSERT_GE(resident, 0); + /* The shrinkers ran: something went, and never the kept object alone. */ + ASSERT_NE(both, resident); + ASSERT_NE((long long)size, resident); + if (resident == (long long)kept_size) { + resident = reclaim_while(&p, self->fd, kept_size, kept_size, + SPARED_REQUESTS); + ASSERT_GE(resident, 0); + } + if (resident != (long long)kept_size) { + if (kept_could_go(&p, reason, sizeof(reason))) + SKIP(return, "the kept object went after %d requests, %s", + p.requests, reason); + TH_LOG("the kept object went after %d requests, without leave: %s", + p.requests, reason); + } + ASSERT_EQ((long long)kept_size, resident); + TH_LOG("the object that is not kept went with request %d", first); + TH_LOG("the kept one stayed through %d more, %lld MiB of the cold cache left; %s", + p.requests - first, p.cold_low >> 20, + trace_note(&p, reason, sizeof(reason))); + + ASSERT_EQ(0, residency(self->fd, a, 0)); + first = p.requests; + resident = reclaim_while(&p, self->fd, kept_size, kept_size, + MAX_REQUESTS); + ASSERT_GE(resident, 0); + EXPECT_EQ(0, resident); + TH_LOG("the object no longer kept went after %d requests", + p.requests - first); +} + +/* Contents given up with DONTNEED are purged by reclaim, kept or not. */ +TEST_F_TIMEOUT(asahi, kept_objects_stay_purgeable, 180) +{ + const uint64_t size = RECLAIM_PAGES * sysconf(_SC_PAGESIZE); + struct pressure p; + const char *why = reclaim_setup(&p, size, &self->cold); + long long resident; + uint32_t a; + + if (why) + SKIP(return, "%s", why); + a = bound_object(self->fd, self->vm, &self->va, self->va_end, size); + ASSERT_NE(0, a); + ASSERT_EQ(0, residency(self->fd, a, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + ASSERT_EQ(1, madvise_object(self->fd, a, DRM_ASAHI_GEM_MADV_DONTNEED)); + + sleep(IDLE_SECONDS); + resident = reclaim_while(&p, self->fd, size, size, MAX_REQUESTS); + ASSERT_GE(resident, 0); + EXPECT_EQ(0, resident); + TH_LOG("purged after %d requests", p.requests); + EXPECT_EQ(0, madvise_object(self->fd, a, DRM_ASAHI_GEM_MADV_WILLNEED)); +} + +struct toggler { + int fd; + uint32_t handle; + int stop; + int failed; + long requests; +}; + +static void *toggle_residency(void *arg) +{ + struct toggler *t = arg; + + while (!__atomic_load_n(&t->stop, __ATOMIC_RELAXED)) { + if (residency(t->fd, t->handle, DRM_ASAHI_GEM_RESIDENCY_KEEP) || + residency(t->fd, t->handle, 0)) + t->failed = 1; + t->requests += 2; + } + return NULL; +} + +/* The objects of the race case, as bits of what the client holds. */ +#define TOGGLED 1 +#define CONTROL 2 +#define KEPT 4 + +/* + * Which objects a client that holds @resident bytes has resident, as a mask + * of the bits above, or -1. @sizes: their sizes, in the order of the bits. + */ +static int resident_objects(long long resident, const long long sizes[3]) +{ + for (int mask = 0; mask < 8; mask++) { + long long sum = 0; + + for (int i = 0; i < 3; i++) + if (mask & (1 << i)) + sum += sizes[i]; + if (sum == resident) + return mask; + } + return -1; +} + +/* + * Residency requests racing with reclaim. One thread sets and clears the + * request on one object as fast as it can while reclaim runs. An object + * beside it that is not kept (the control) has to be released during the + * race, which shows that reclaim got to this VM between the requests; a kept + * object beside it has to stay, as long as reclaim does not come to the kept + * objects (see the top of this file for how the case knows). Whatever a + * request that has returned found (the object resident or already released), + * reclaim leaves that way, and the kept total counts each object once. The + * three objects differ in size, so that what the client holds tells which of + * them are resident. + */ +TEST_F_TIMEOUT(asahi, requests_serialize_with_reclaim, 180) +{ + const uint64_t size = RECLAIM_PAGES * sysconf(_SC_PAGESIZE); + const uint64_t small = 1024 * PAGE; + const long long sizes[3] = { + size + 2 * small, size, small + own_pages() * PAGE, + }; + struct toggler t = { .fd = self->fd }; + long long kept0 = gpu_memory_field("user_kept"); + struct pressure p; + const char *why = reclaim_setup(&p, sizes[0] + sizes[1] + sizes[2], + &self->cold); + int resident = TOGGLED | CONTROL | KEPT, found, error = 0; + int control_gone = 0; + char reason[320]; + uint32_t control, kept; + pthread_t thread; + double end, before; + + if (why) + SKIP(return, "%s", why); + /* The kept object alone, or with the toggled one while that is kept. */ + p.vm.counts[0] = sizes[2] / sysconf(_SC_PAGESIZE); + p.vm.counts[1] = (sizes[0] + sizes[2]) / sysconf(_SC_PAGESIZE); + t.handle = bound_object(self->fd, self->vm, &self->va, self->va_end, sizes[0]); + control = bound_object(self->fd, self->vm, &self->va, self->va_end, sizes[1]); + kept = bound_object(self->fd, self->vm, &self->va, self->va_end, sizes[2]); + ASSERT_NE(0, t.handle); + ASSERT_NE(0, control); + ASSERT_NE(0, kept); + ASSERT_EQ(0, residency(self->fd, kept, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + kept0 += sizes[2]; + ASSERT_EQ(0, pthread_create(&thread, NULL, toggle_residency, &t)); + /* + * From before the VM counts as idle until a second after reclaim + * released the control object, for ten seconds at most. + */ + sleep(IDLE_SECONDS - 2); + end = seconds() + 10; + while (seconds() < end && !error && resident >= 0 && (resident & KEPT)) { + error = reclaim_some(&p); + before = seconds(); + resident = resident_objects(client_resident(self->fd), sizes); + note_kept(&p.vm, before, resident >= 0 && (resident & KEPT)); + if (resident >= 0 && !(resident & CONTROL) && !control_gone) { + control_gone = p.requests; + end = seconds() + 1; + } + usleep(100000); + } + __atomic_store_n(&t.stop, 1, __ATOMIC_RELAXED); + ASSERT_EQ(0, pthread_join(thread, NULL)); + ASSERT_EQ(0, error); + ASSERT_GE(resident, 0); + EXPECT_EQ(0, t.failed); + EXPECT_GT(t.requests, 0); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + + /* The toggled object is kept from here on, resident or not. */ + ASSERT_EQ(0, residency(self->fd, t.handle, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + before = seconds(); + found = resident_objects(client_resident(self->fd), sizes); + note_kept(&p.vm, before, found >= 0 && (found & KEPT)); + resident = found; + for (int i = 0; i < SPARED_REQUESTS && resident == found; i++) { + ASSERT_EQ(0, reclaim_some(&p)); + before = seconds(); + resident = resident_objects(client_resident(self->fd), sizes); + note_kept(&p.vm, before, (found & KEPT) && + (resident | CONTROL) == (found | CONTROL)); + } + /* What the client holds was read, and is a sum of the objects' sizes. */ + ASSERT_GE(found, 0); + ASSERT_GE(resident, 0); + /* A kept object that went, during the race or after it. */ + if (!(found & KEPT) || (resident | CONTROL) != (found | CONTROL)) { + if (kept_could_go(&p, reason, sizeof(reason))) + SKIP(return, "a kept object went after %d requests, %s", + p.requests, reason); + TH_LOG("a kept object went after %d requests, without leave: %s", + p.requests, reason); + ASSERT_TRUE(found & KEPT); + ASSERT_EQ(found | CONTROL, resident | CONTROL); + } + /* Reclaim got to this VM during the race. */ + ASSERT_NE(0, control_gone); + TH_LOG("%ld requests raced with reclaim; the control object went with request %d", + t.requests, control_gone); + TH_LOG("the toggled object was %s, %lld MiB of the cold cache left; %s", + found & TOGGLED ? "left" : "released", p.cold_low >> 20, + trace_note(&p, reason, sizeof(reason))); + EXPECT_EQ(kept0 + sizes[0], gpu_memory_field("user_kept")); + close_handle(self->fd, t.handle); + EXPECT_EQ(kept0, gpu_memory_field("user_kept")); + close_handle(self->fd, kept); + close_handle(self->fd, control); +} + +/* + * Urgent reclaim (here: every shrinker run at the last priority) releases + * kept objects too. Where the shrinker passes can be recorded, the record + * has to show a pass that called the shrinker of this VM's kept objects + * between the last look at the object and the first one that found it gone. + */ +TEST_F(asahi, kept_objects_go_under_urgent_pressure) +{ + const uint64_t size = 1024 * PAGE; + const uint64_t kept_size = size + own_pages() * PAGE; + const char *run = getenv("ASAHI_RESIDENCY_DROP_CACHES"); + long long kept0 = gpu_memory_field("user_kept"); + struct own_vm vm = { + .counts = { kept_size / sysconf(_SC_PAGESIZE), + kept_size / sysconf(_SC_PAGESIZE) }, + }; + struct kept_record r; + long long resident; + uint32_t a, b; + + if (!run || strcmp(run, "1")) + SKIP(return, "set ASAHI_RESIDENCY_DROP_CACHES=1 to drop the system's caches"); + if (geteuid()) + SKIP(return, "dropping caches needs root"); + a = bound_object(self->fd, self->vm, &self->va, self->va_end, kept_size); + b = bound_object(self->fd, self->vm, &self->va, self->va_end, size); + ASSERT_NE(0, a); + ASSERT_NE(0, b); + ASSERT_EQ(0, residency(self->fd, a, DRM_ASAHI_GEM_RESIDENCY_KEEP)); + + sleep(IDLE_SECONDS); + trace_start(); + vm.held_at = seconds(); + ASSERT_EQ((long long)(kept_size + size), client_resident(self->fd)); + ASSERT_EQ(0, write_string("/proc/sys/vm/drop_caches", "2")); + resident = client_resident(self->fd); + vm.gone_at = seconds(); + EXPECT_EQ(0, resident); + if (!resident && !read_record(&vm, &r) && r.others) { + if (r.shrinkers > 1) { + TH_LOG( +"inconclusive: %d shrinkers of kept objects count what this VM's counts, and %d passes called one of them before the object went (traced)", + r.shrinkers, r.released); + } else { + EXPECT_GT(r.released, 0); + TH_LOG( +"reclaim passes that called the shrinker of this VM's kept objects before it went: %d, down to priority %d (traced)", + r.released, r.priority); + } + } + /* Released, not purged: the contents come back with the next binding. */ + EXPECT_EQ(1, madvise_object(self->fd, a, DRM_ASAHI_GEM_MADV_WILLNEED)); + EXPECT_EQ(kept0 + (long long)kept_size, gpu_memory_field("user_kept")); +} + +/* + * The trace instance is the program's: made before the cases fork and + * removed when the program exits, however its cases ended. + */ +int main(int argc, char **argv) +{ + trace_create(); + return test_harness_run(argc, argv); +}