From 783ffb64652d63f6708f7f3f95b7b31747bfc52a Mon Sep 17 00:00:00 2001 From: Diljit Singh Date: Fri, 13 Nov 2026 13:35:34 -0500 Subject: [PATCH] drm/asahi: g17: A18 Pro: report the performance state, its frequency and busy time The A18 Pro's 26A428 firmware sends the same statistics records as the M5's: the performance state in effect (tag 0x1b, after an interval record, tag 0x19) and utilisation records that carry their own timestamps, one every 16 ms on the A18 Pro. Only the M5 decoded them, so agx_stats on the A18 Pro reported pstate, achieved_frequency_hz and busy_ns as unsupported, and its fdinfo had no frequency keys. Describe what each firmware sends beyond utilisation in a Protocol (highest state, utilisation period) chosen from the init graph: 25G83 sends neither, the A18 Pro has states 1-10 and a 16 ms period, the M5 states 1-13 and an 8 ms period. The A18 Pro's state frequencies come from the device tree operating points its performance table A names. The records do not name the table, but at its top state, where the GPU runs while work is in flight, a sustained FP32 FMA load completes faster than table B's frequency there allows. Tag 0x10 is no longer decoded on a firmware that reports its state in tags 0x19 and 0x1b. While the GPU cores are gated the GPU is off: report state 0, valid, at 0 Hz in agx_stats and drm-curfreq-*, instead of a stale state, on the A18 Pro and the M5 alike. Power-up still starts a new epoch, stale until the firmware reports a state again. Signed-off-by: Diljit Singh --- .../ABI/testing/sysfs-driver-asahi-agx-stats | 75 ++++--- Documentation/gpu/asahi-usage-stats.rst | 71 ++++--- drivers/gpu/drm/asahi/g17/channel.rs | 8 +- drivers/gpu/drm/asahi/g17/mod.rs | 36 +++- drivers/gpu/drm/asahi/g17/stats.rs | 80 ++++++-- drivers/gpu/drm/asahi/g17/telemetry.rs | 187 +++++++++++++----- drivers/gpu/drm/asahi/stats.rs | 2 +- drivers/gpu/drm/asahi/sysfs.c | 13 +- .../asahi/g17-telemetry-integration-tests.py | 109 +++++++--- 9 files changed, 413 insertions(+), 168 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats b/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats index 282d38614a091..eab183c793ab8 100644 --- a/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats +++ b/Documentation/ABI/testing/sysfs-driver-asahi-agx-stats @@ -86,14 +86,15 @@ Description: older than the age limit below or its epoch ended. unsupported is a fixed property of the firmware, taken from the driver's per-firmware match data, and - never changes at runtime: the V25G83 and V26A428 - firmwares accept tag 0x10 but never emit it, and no - other record carries the current pstate, so - pstate_status always reads unsupported on G16S and - G17P. utilisation_status is never unsupported on - these firmwares. Numeric metrics are printed only - while their capability is valid; otherwise their - value is the string unsupported. + never changes at runtime: the V25G83 firmware accepts + tag 0x10 but never emits it, and no other record of + it carries the current pstate, so pstate_status + always reads unsupported on G16S. V26A428 reports + its state in other records (see below). + utilisation_status is never unsupported on these + firmwares. Numeric metrics are printed only while + their capability is valid; otherwise their value is + the string unsupported. util_tiler_percent, util_renderer_percent, util_device_percent Unsigned integer percent (0..100), from tag 0x13 @@ -107,7 +108,9 @@ Description: pstate Logical selector 0..10 from tag 0x10 word 6, on a firmware whose match data says it emits tag 0x10 - (none of the current G16S/G17P firmwares do). Words 1 + (none of the current G16S/G17P firmwares do). A + firmware that reports its state in tags 0x19 and + 0x1b (V26A428) never has tag 0x10 decoded. Words 1 and 2 form a little-endian u64 firmware timestamp. The first timestamp establishes a baseline. Invalid selectors and backwards timestamps remove pstate @@ -116,39 +119,53 @@ Description: requested_frequency_hz, achieved_frequency_hz, busy_ns, power_mw, temperature_raw - Always unsupported on these firmwares. The statistics - packet does not identify performance table A or B; - a selector is neither a host request nor a measured - frequency. No qualified busy-time duration, power unit - or temperature field is available. + Always unsupported on V25G83 (G16S). No record of it + carries a state, a qualified busy-time duration, a + power unit or a temperature. - M5 firmware (the M5 command set of the shared G17 backend) uses - the same snapshot and keys, with these differences: + The V26A428 firmware, which the A18 Pro (T8140, G17P) and M5 + run, uses the same snapshot and keys, with these differences: pstate_status, pstate Supported. The firmware reports the performance state in effect in tag 0x1b, words 1 and 3, which must - agree: an index 1..13 into the board's performance - state table. The end of the interval record before it - (tag 0x19, words 1 and 2) is its firmware timestamp. - A report that names state 0 or a state beyond the - table removes pstate validity. The state reads stale - while the GPU cores are gated. + agree: an index into the performance state table, + 1..10 on the A18 Pro, 1..13 on M5. The end of the + interval record before it (tag 0x19, words 1 and 2) + is its firmware timestamp. A report that names state + 0 or a state beyond the table removes pstate + validity. While the GPU cores are gated, pstate reads + valid 0: the GPU is off. Power-up starts a new epoch, + stale until the next report. The driver asks for full + performance while work is in flight, which the + firmware takes as a floor, so an A18 Pro at work + normally reads its top state, 10; a lower state there + is a firmware limit, such as a thermal one. achieved_frequency_hz - The frequency of pstate in the board's performance - state table, in Hz, while pstate_status is valid. + The frequency of pstate in Hz, while pstate_status is + valid, and 0 for state 0. M5: the board's performance + state table. A18 Pro: the device tree operating point + that the state's entry of performance table A names. + Its records do not name the table; at its top state a + sustained FP32 FMA load completes faster than table + B's frequency there allows, so it runs table A. busy_ns Cumulative u64 since the driver was bound. The tag 0x13 records of this firmware carry a timestamp in - 24 MHz ticks (words 1 and 2). Each record credits the - window since the previous record of its epoch at its - device utilisation. A record more than two 8 ms + 24 MHz ticks (words 1 and 2), one every 8 ms on M5 + and every 16 ms on the A18 Pro. Each record credits + the window since the previous record of its epoch at + its device utilisation. A record more than two periods after the previous one, or with an earlier timestamp, only starts a new window, so time the GPU - spent gated is never credited. MEASURED on T6050 + spent gated is never credited, and a load sparser + than one period reads low. MEASURED on T6050 (J716c): 871 and 863 ms for three 300 ms compute jobs - at full clock (905 and 919 ms of work). + at full clock (905 and 919 ms of work). MEASURED on + T8140 (J700), 24 traces of GPU stress loads: within + about 20% of the union of the commands' own execution + intervals for loads busy more than half the time. requested_frequency_hz, power_mw, temperature_raw Unsupported. diff --git a/Documentation/gpu/asahi-usage-stats.rst b/Documentation/gpu/asahi-usage-stats.rst index 8bd90b2733d9c..1c6a8ea517e4e 100644 --- a/Documentation/gpu/asahi-usage-stats.rst +++ b/Documentation/gpu/asahi-usage-stats.rst @@ -57,15 +57,16 @@ cycles, or multiplying time by a sampled/maximum GPU frequency, would give incorrect frequency-weighted utilization. Per-client execution-cycle counters would require a separate hardware measurement source. -On the M5 each engine also has the frequency keys:: +On the M5 and the A18 Pro each engine also has the frequency keys:: drm-curfreq-compute: 1620000000 Hz drm-maxfreq-compute: 1620000000 Hz ``drm-curfreq-`` is ``achieved_frequency_hz`` of ``agx_stats`` (see -below): the frequency of the performance state the firmware reports. It is -absent while the firmware reports none, as when the GPU is gated. -``drm-maxfreq-`` is the highest frequency in the board's table. One clock +below): the frequency of the performance state the firmware reports, 0 Hz while +the GPU cores are gated. It is absent while no state is known, as right after +the GPU powers up. ``drm-maxfreq-`` is the highest frequency of the +performance states. One clock drives the three engines, so both values are the device's, the same for every engine and every client. They show the clock at the time of the read and are not a weight for the engine times: without ``drm-cycles-*`` there is nothing to @@ -80,9 +81,9 @@ per valid stage, with no new locks or allocations. The native backends reuse existing timestamp reads. Classic completion callbacks read two internal cells per stage; commands without application timestamps also execute two existing timestamp microsequence operations per stage. Reading fdinfo loads three counters -and does not traverse queues, wait for fences or access hardware. On the M5 a -read also takes the device mutex once and drains the statistics rings, as a read -of ``agx_stats`` does, for the frequency keys. +and does not traverse queues, wait for fences or access hardware. On the M5 and +the A18 Pro a read also takes the device mutex once and drains the statistics +rings, as a read of ``agx_stats`` does, for the frequency keys. G16S and G17P device utilisation ------------------------------ @@ -125,26 +126,39 @@ ring on powered cores becomes stale rather than fabricating a zero-utilisation sample; only gated cores read as zero. Readers must not convert these percentages into engine nanoseconds or cycles. -M5 performance state and busy time ----------------------------------- +Performance state and busy time on V26A428 (A18 Pro and M5) +----------------------------------------------------------- -The M5 firmware runs on the same backend and decoder, and its statistics ring -carries more. ``agx_stats`` reports there: +The V26A428 firmware, which the A18 Pro (T8140, G17P) and the M5 run on the same +backend and decoder, carries more in its statistics ring. ``agx_stats`` reports +there: * ``pstate``: the performance state in effect, from the firmware's state reports (tag ``0x1b``, words 1 and 3, which must agree). It is an index into - the board's performance state table, 1 to 13. The interval record before a - report (tag ``0x19``) gives its firmware timestamp. A report that names state - 0 or a state beyond the table is not used, and ``pstate_status`` reads - ``stale`` while the cores are gated. -* ``achieved_frequency_hz``: the frequency of that state in the board's table, - while ``pstate_status`` is ``valid``. -* ``busy_ns``: the M5 utilisation records carry their own timestamps (words 1 - and 2, 24 MHz ticks), so each record closes a window of known length, which - is credited at the record's device utilisation. A record that follows the - previous one by more than two 8 ms periods, or whose timestamp went back, - only starts a new window: time the GPU spent gated is never credited. The - total keeps counting across power epochs. + the performance state table: 1 to 10 on the A18 Pro, 1 to 13 on the M5. The + interval record before a report (tag ``0x19``) gives its firmware timestamp. + A report that names state 0 or a state beyond the table is not used. While + the cores are gated the GPU is off and ``pstate`` reads a valid 0; power-up + starts a new epoch, ``stale`` until the next report. +* ``achieved_frequency_hz``: the frequency of that state while + ``pstate_status`` is ``valid``, 0 for state 0. The M5 takes it from its + board's table. The A18 Pro takes the device tree operating point that the + state's entry of its performance table A names: its records do not name the + table, but at its top state a sustained FP32 FMA load completes faster than + table B's frequency there allows. +* ``busy_ns``: the utilisation records carry their own timestamps (words 1 and + 2, 24 MHz ticks), one every 8 ms on the M5 and every 16 ms on the A18 Pro, so + each record closes a window of known length, which is credited at the + record's device utilisation. A record that follows the previous one by more + than two periods, or whose timestamp went back, only starts a new window: + time the GPU spent gated is never credited, and a load sparser than one + period reads low. The total keeps counting across power epochs. + +The driver asks for full performance while work is in flight, which the +firmware takes as a floor for its state. An A18 Pro at work therefore normally +reports its top state; a lower one is a firmware limit, such as a thermal one. +The tag ``0x19`` records are periodic samples, not busy intervals, and do not +contribute to ``busy_ns``. ``requested_frequency_hz``, ``power_mw`` and ``temperature_raw`` stay unsupported. ``busy_ns`` is device-wide, as on G13 and G14; it does not change @@ -153,10 +167,11 @@ fdinfo's per-client ``drm-engine-*`` counters. Deferred shared power telemetry ------------------------------ -Current pstate and requested/achieved frequency remain unsupported for these -firmware graphs (the M5 firmware reports its state in the statistics ring, see -above). The shared-power snapshot ABI is not sufficiently specified -to implement a read-only reader. For **each** of G16S and G17P, it still needs: +Current pstate and requested/achieved frequency remain unsupported on G16S +(V25G83); V26A428 reports its state in the statistics ring, see above. The +shared-power snapshot ABI is not sufficiently specified to implement a +read-only reader, which G16S would need, and which would also name the A18 +Pro's table directly. For **each** of G16S and G17P, it still needs: * The firmware-version-to-layout mapping, containing allocation and bounds, offsets, widths and stride of the generation/seqlock word, live pstate, @@ -176,7 +191,7 @@ A future implementation needs bounded seqlock retries and lifecycle invalidation before publishing these fields. Tag ``0x10`` remains decoded for diagnostics but is not emitted by the currently qualified firmware graphs. A dormant pstate record, a host request or a command timer cannot provide achieved frequency. -``drm-curfreq-gpu`` must wait for a qualified shared snapshot. +On G16S, ``drm-curfreq-*`` must wait for a qualified shared snapshot. Hardware acceptance ------------------- diff --git a/drivers/gpu/drm/asahi/g17/channel.rs b/drivers/gpu/drm/asahi/g17/channel.rs index 9fc50fbad6308..5276e7758f22a 100644 --- a/drivers/gpu/drm/asahi/g17/channel.rs +++ b/drivers/gpu/drm/asahi/g17/channel.rs @@ -92,16 +92,12 @@ pub(crate) struct Rings { } impl Rings { - /// `m5`: the firmware runs the M5 protocol, whose statistics records are decoded too. + /// `m5`: the firmware runs the M5 protocol ([`super::stats::protocol`]). pub(crate) fn new(graph: super::fw::variant::InitGraph, m5: bool) -> Result { Ok(Self { quarantined: [[false; 2]; 2], reports: super::reports::Reports::new()?, - telemetry: if m5 { - super::telemetry::Telemetry::m5() - } else { - super::telemetry::Telemetry::default() - }, + telemetry: super::telemetry::Telemetry::new(super::stats::protocol(graph, m5)), max_age_ns: super::stats::max_age_ns(graph), suspended: false, discard_statistics: [false; 2], diff --git a/drivers/gpu/drm/asahi/g17/mod.rs b/drivers/gpu/drm/asahi/g17/mod.rs index e3db5fee61534..ec19e0b80a1e3 100644 --- a/drivers/gpu/drm/asahi/g17/mod.rs +++ b/drivers/gpu/drm/asahi/g17/mod.rs @@ -1816,12 +1816,10 @@ impl Gpu { directory.read_only_file(c"hardware_data", reports::HardwareDataView(shared.clone())), GFP_KERNEL, )?; + let frequencies = Self::state_frequencies(dev, cfg, variant, id.num_clusters)?; let stats = Arc::new( crate::stats::StatsSnapshot { - g17_reader: Some(stats::Reader( - shared.clone(), - cfg.m5.map(|board| board.perf_states), - )), + g17_reader: Some(stats::Reader(shared.clone(), frequencies)), ..Default::default() }, GFP_KERNEL, @@ -1932,6 +1930,36 @@ impl Gpu { cells.as_slice().try_into().ok() } + /// The frequency of each performance state the firmware reports + /// ([`stats::reports_pstate`]): M5's from its board, the A18 Pro's from the operating + /// points its performance table A names. The A18 Pro's records do not name the table, but + /// at its top state, where it runs while work is in flight, a sustained FP32 FMA load + /// completes faster than table B's frequency there allows. `None` where the firmware + /// reports no state. + fn state_frequencies( + dev: &AsahiDrmDevice, + cfg: &'static Config, + variant: &fw::variant::Variant, + clusters: u32, + ) -> Result> { + if !stats::reports_pstate(variant.graph, variant.m5_commands) { + return Ok(None); + } + if let Some(board) = cfg.m5 { + return Ok(stats::StateFrequencies::new( + board.perf_states.iter().map(|state| state.0), + )); + } + let perf = cfg.perf()?; + let opps = initdata::InitData::opps(dev, cfg.opps, clusters)?; + let mut mhz = [0u32; fw::initdata::PERF_STATES]; + for (slot, &index) in mhz.iter_mut().zip(perf.states_a.iter()) { + let opp = opps.get(usize::from(index)).ok_or(EINVAL)?; + *slot = u32::try_from(opp.freq_hz / 1_000_000)?; + } + Ok(stats::StateFrequencies::new(mhz.into_iter())) + } + /// Returns the highest advertised operating-point frequency without changing table order. fn max_frequency_khz(dev: &AsahiDrmDevice) -> Result { let node = dev.as_ref().as_ref().of_node().ok_or(EIO)?; diff --git a/drivers/gpu/drm/asahi/g17/stats.rs b/drivers/gpu/drm/asahi/g17/stats.rs index 40688ccab0c8e..5484c68fa62f7 100644 --- a/drivers/gpu/drm/asahi/g17/stats.rs +++ b/drivers/gpu/drm/asahi/g17/stats.rs @@ -7,12 +7,44 @@ use super::{fw::variant::InitGraph, telemetry::Readout, Shared}; use core::sync::atomic::Ordering; use kernel::sync::Arc; -/// The device's shared state, and the M5 board's performance states, whose frequency a -/// reported state index runs at. -pub(crate) struct Reader( - pub(super) Arc, - pub(super) Option<&'static [super::fw::m5::initdata::M5PerfState; 14]>, -); +/// Frequency of each performance state the firmware reports, in MHz; state 0 is off. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct StateFrequencies { + mhz: [u32; Self::MAX], + states: usize, +} + +impl StateFrequencies { + const MAX: usize = 16; + + /// `mhz` holds states 0 to `mhz.len() - 1`; `None` for more than [`Self::MAX`] states. + pub(crate) fn new(mhz: impl ExactSizeIterator) -> Option { + let states = mhz.len(); + if states > Self::MAX { + return None; + } + let mut table = [0; Self::MAX]; + for (slot, value) in table.iter_mut().zip(mhz) { + *slot = value; + } + Some(Self { mhz: table, states }) + } + + /// The frequency of `pstate` in Hz; `None` beyond the table or where it is unknown (0). + fn hz(&self, pstate: u32) -> Option { + let mhz = *self.mhz[..self.states].get(pstate as usize)?; + (mhz != 0).then_some(u64::from(mhz) * 1_000_000) + } + + fn max_hz(&self) -> Option { + let mhz = *self.mhz[..self.states].iter().max()?; + (mhz != 0).then_some(u64::from(mhz) * 1_000_000) + } +} + +/// The device's shared state, and the frequency of each performance state the firmware +/// reports. +pub(crate) struct Reader(pub(super) Arc, pub(super) Option); impl Reader { pub(crate) fn read(&self) -> Readout { @@ -20,20 +52,21 @@ impl Reader { if view.pstate_status == super::telemetry::VALID { view.achieved_frequency_hz = self .1 - .and_then(|states| states.get(view.pstate as usize)) - .map_or(0, |state| u64::from(state.0) * 1_000_000); + .as_ref() + .and_then(|states| states.hz(view.pstate)) + .unwrap_or(0); } view } /// The frequency of the state the GPU runs at, while the firmware reports one, and the - /// highest frequency of the board's states, in Hz; `None` without the board's performance - /// states. + /// highest frequency of its states, in Hz; `None` without the states' frequencies. pub(crate) fn frequency(&self) -> Option<(Option, u64)> { - let max = u64::from(self.1?.iter().map(|state| state.0).max()?) * 1_000_000; + let max = self.1.as_ref()?.max_hz()?; let view = self.read(); + // State 0 is off: the gated GPU runs at no frequency. let current = (view.pstate_status == super::telemetry::VALID - && view.achieved_frequency_hz != 0) + && (view.achieved_frequency_hz != 0 || view.pstate == 0)) .then_some(view.achieved_frequency_hz); Some((current, max)) } @@ -85,18 +118,25 @@ pub(super) const fn max_age_ns(graph: InitGraph) -> u64 { } } -/// Whether the firmware reports its performance state. V25G83 and V26A428 -/// accept the logical pstate record (tag 0x10) but never emit it, and no other -/// record of theirs carries the current pstate, so it is unsupported by these -/// firmwares. M5 firmware (`m5`) reports it in its own records -/// ([`super::telemetry`]). A firmware that is shown to report it gets `true` -/// here and reports valid or stale. -pub(super) const fn reports_pstate(graph: InitGraph, m5: bool) -> bool { +/// The records `graph`'s firmware sends beyond utilisation; `m5`: it runs the M5 protocol, +/// with M5's performance states. +pub(super) const fn protocol(graph: InitGraph, m5: bool) -> super::telemetry::Protocol { match graph { - InitGraph::V25G83 | InitGraph::G17P => m5, + InitGraph::G17P if m5 => super::telemetry::Protocol::M5, + InitGraph::G17P => super::telemetry::Protocol::A18_PRO, + InitGraph::V25G83 => super::telemetry::Protocol::NONE, } } +/// Whether the firmware reports its performance state. V25G83 and V26A428 accept +/// the logical pstate record (tag 0x10) but never emit it. V26A428, on the A18 Pro +/// as on M5, reports the state in its own records instead (tags 0x19 and 0x1b, +/// [`protocol`]); V25G83 sends no record carrying it, so it is unsupported there. +/// A firmware that reports it reads valid or stale. +pub(super) const fn reports_pstate(graph: InitGraph, m5: bool) -> bool { + protocol(graph, m5).reports_pstate() +} + /// Apply the firmware's static capabilities to a decoded readout. The decoder /// and the debugfs diagnostic still keep any tag 0x10 that does arrive. pub(super) fn qualify( diff --git a/drivers/gpu/drm/asahi/g17/telemetry.rs b/drivers/gpu/drm/asahi/g17/telemetry.rs index 27d8bef8c5383..23f66a7fcced0 100644 --- a/drivers/gpu/drm/asahi/g17/telemetry.rs +++ b/drivers/gpu/drm/asahi/g17/telemetry.rs @@ -1,13 +1,12 @@ // SPDX-License-Identifier: GPL-2.0-only OR MIT //! Qualified V25G83/V26A428 statistics, decoded from complete copied records. -//! No record of these firmwares measures busy time, power, temperature or achieved -//! frequency. A/B table selection is absent, so neither nominal candidate is exported -//! as a requested or achieved frequency. +//! No record of these firmwares measures power or temperature. //! -//! M5 firmware reports the performance state the GPU runs at (tags 0x19 and 0x1b), and -//! its utilisation records carry their own timestamps, from which busy time is -//! integrated. All state is protected by the device mutex. +//! The 26A428 firmware (the A18 Pro's, and the M5's) reports the performance state the +//! GPU runs at (tags 0x19 and 0x1b), and its utilisation records carry their own +//! timestamps, from which busy time is integrated ([`Protocol`]). 25G83 sends neither. +//! All state is protected by the device mutex. pub(super) const UNAVAILABLE: u32 = 0; pub(super) const VALID: u32 = 1; @@ -31,10 +30,10 @@ pub(crate) struct Readout { pub(crate) util_samples: u64, pub(crate) pstate_timestamp: u64, /// Utilisation-weighted busy time since the device was created, while `busy_status` is - /// [`VALID`] (M5 only). + /// [`VALID`] ([`Protocol::util_period_ticks`]). pub(crate) busy_ns: u64, - /// Frequency of `pstate` while `pstate_status` is [`VALID`], from the board's performance - /// state table ([`super::stats::Reader`]); zero where the table is unknown. + /// Frequency of `pstate` while `pstate_status` is [`VALID`], from the frequency of each + /// state ([`super::stats::Reader`]); zero for state 0 or where it is unknown. pub(crate) achieved_frequency_hz: u64, pub(crate) busy_status: u32, pub(crate) _pad: u32, @@ -46,15 +45,52 @@ const _: () = { assert!(core::mem::size_of::() == 80); }; -/// Interval of the M5 utilisation record (tag 0x13), in timer ticks: 8 ms at 24 MHz. -const M5_UTIL_PERIOD_TICKS: u64 = 192_000; -/// A utilisation record more than two intervals after the previous one follows a gap -/// the firmware did not report (gated or lost records): it only sets the baseline. -const M5_UTIL_MAX_WINDOW_TICKS: u64 = 2 * M5_UTIL_PERIOD_TICKS; -/// Highest M5 performance state: the device tree's operating points 1 to 13 (0 is off). -const M5_MAX_PSTATE: u32 = 13; const TIMER_HZ: u64 = 24_000_000; +/// What a firmware reports beyond utilisation. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct Protocol { + /// Highest performance state its state records (tags 0x19 and 0x1b) name; zero where it + /// sends none. State 0 is off. + pub(crate) max_pstate: u32, + /// Interval of its utilisation record (tag 0x13), in timer ticks, whose timestamps busy + /// time is integrated from; zero where it is not integrated. + pub(crate) util_period_ticks: u64, +} + +impl Protocol { + /// 25G83: utilisation only. + pub(crate) const NONE: Self = Self { + max_pstate: 0, + util_period_ticks: 0, + }; + /// M5: the device tree's operating points 1 to 13, utilisation every 8 ms. + pub(crate) const M5: Self = Self { + max_pstate: 13, + util_period_ticks: 192_000, + }; + /// A18 Pro: states 1 to 10 of its eleven-state performance tables, utilisation every + /// 16 ms. + pub(crate) const A18_PRO: Self = Self { + max_pstate: 10, + util_period_ticks: 384_000, + }; + + pub(crate) const fn reports_pstate(self) -> bool { + self.max_pstate != 0 + } + + fn integrates_busy(self) -> bool { + self.util_period_ticks != 0 + } + + /// A utilisation record more than two intervals after the previous one follows a gap the + /// firmware did not report (gated or lost records): it only sets the baseline. + fn max_window_ticks(self) -> u64 { + 2 * self.util_period_ticks + } +} + #[derive(Default)] pub(super) struct Telemetry { view: Readout, @@ -71,21 +107,21 @@ pub(super) struct Telemetry { /// gates them a few milliseconds after the last work completes and emits no /// utilisation record while gated, so a gated GPU is idle, not unknown. gated: bool, - /// The M5 firmware's records are decoded ([`Self::m5`]). - m5: bool, - /// Timestamp of the latest M5 utilisation record in this epoch: the start of the window - /// the next one closes. + /// The records decoded beyond utilisation. + protocol: Protocol, + /// Timestamp of the latest utilisation record in this epoch: the start of the window the + /// next one closes. busy_ts: Option, busy_ns: u64, - /// End of the latest M5 performance state interval (tag 0x19), in timer ticks. + /// End of the latest performance state interval (tag 0x19), in timer ticks. pstate_interval_end: Option, } impl Telemetry { - /// Decoder for M5 firmware: its performance state and busy time records. - pub(super) fn m5() -> Self { + /// Decoder for a firmware that also sends `protocol`'s records. + pub(super) fn new(protocol: Protocol) -> Self { Self { - m5: true, + protocol, ..Self::default() } } @@ -170,7 +206,7 @@ impl Telemetry { for (sum, value) in self.sums.iter_mut().zip(values) { *sum += u64::from(value); } - if self.m5 { + if self.protocol.integrates_busy() { self.integrate_busy( u64::from(words[1]) | (u64::from(words[2]) << 32), words[6], @@ -183,7 +219,9 @@ impl Telemetry { self.view.renderer = (self.sums[1] / self.view.util_samples) as u32; self.view.device = (self.sums[2] / self.view.util_samples) as u32; } - 0x10 => { + // The logical pstate record, which no current firmware emits. A firmware that + // reports its state in tags 0x19 and 0x1b never has it decoded. + 0x10 if !self.protocol.reports_pstate() => { let timestamp = u64::from(words[1]) | (u64::from(words[2]) << 32); if self.timestamp.is_some_and(|last| timestamp < last) { self.invalidate(); @@ -197,17 +235,16 @@ impl Telemetry { self.view.pstate_status = if words[6] <= 10 { VALID } else { UNAVAILABLE }; self.view.pstate = words[6]; } - // M5: the interval [words 3-4, words 1-2] that just ended, run at the state in - // word 8. Its end timestamps the state report that follows it. - 0x19 if self.m5 => { + // 26A428: the interval [words 3-4, words 1-2] that just ended. Its end + // timestamps the state report that follows it. + 0x19 if self.protocol.reports_pstate() => { self.pstate_interval_end = Some(u64::from(words[1]) | (u64::from(words[2]) << 32)); } - // M5: the performance state now in effect, an index into the device tree's - // operating points, in words 1 and 3. A report whose two words differ is not - // used. - 0x1b if self.m5 => { + // 26A428: the performance state now in effect, in words 1 and 3. A report whose + // two words differ is not used. + 0x1b if self.protocol.reports_pstate() => { let pstate = words[1]; - if pstate != words[3] || pstate == 0 || pstate > M5_MAX_PSTATE { + if pstate != words[3] || pstate == 0 || pstate > self.protocol.max_pstate { self.view.pstate_status = UNAVAILABLE; return; } @@ -220,15 +257,15 @@ impl Telemetry { } } - /// Credits the window an M5 utilisation record closes, at its device utilisation. The - /// window starts at the previous record of the epoch; a record after a longer gap than - /// [`M5_UTIL_MAX_WINDOW_TICKS`] (the GPU was gated or records were lost) or one whose + /// Credits the window a utilisation record closes, at its device utilisation. The window + /// starts at the previous record of the epoch; a record after a longer gap than + /// [`Protocol::max_window_ticks`] (the GPU was gated or records were lost) or one whose /// timestamp went back only sets the baseline, so an idle gap is never credited. fn integrate_busy(&mut self, timestamp: u64, device: u32) { if let Some(last) = self.busy_ts { let window = timestamp.wrapping_sub(last); - if timestamp > last && window <= M5_UTIL_MAX_WINDOW_TICKS { - // At most 384000 ticks x 100: no overflow. + if timestamp > last && window <= self.protocol.max_window_ticks() { + // At most two 16 ms periods (768000 ticks) x 100 x 10^9: no overflow. let ns = window * u64::from(device) * 1_000_000_000 / (100 * TIMER_HZ); self.busy_ns = self.busy_ns.saturating_add(ns); } @@ -254,8 +291,13 @@ impl Telemetry { } // A pstate report describes a state transition, with no qualified // cadence. Its validity ends on replacement or epoch invalidation, - // rather than an invented periodic-report deadline. - if self.m5 { + // rather than an invented periodic-report deadline. Gated cores run + // at no state: off (0) is a measurement here, as idle is above. + if self.gated && self.protocol.reports_pstate() { + view.pstate_status = VALID; + view.pstate = 0; + } + if self.protocol.integrates_busy() { view.busy_status = VALID; view.busy_ns = self.busy_ns; } else { @@ -491,7 +533,7 @@ mod tests { #[test] fn m5_state_reports_give_the_pstate() { - let mut stats = Telemetry::m5(); + let mut stats = Telemetry::new(Protocol::M5); stats.record(&M5_INTERVAL, 100, AGE); assert_eq!(stats.read(100, AGE).pstate_status, UNAVAILABLE); stats.record(&M5_STATE, 101, AGE); @@ -510,15 +552,19 @@ mod tests { stats.record(&words, 103, AGE); assert_eq!(stats.read(103, AGE).pstate_status, UNAVAILABLE); } - // Gating ends the epoch: the state is no longer current. + // Gated cores are off: state 0. Power-up ends the epoch: the state before it is + // no longer current. stats.record(&M5_STATE, 104, AGE); stats.gate(); - assert_eq!(stats.read(105, AGE).pstate_status, STALE); + let view = stats.read(105, AGE); + assert_eq!((view.pstate_status, view.pstate), (VALID, 0)); + stats.invalidate(); + assert_eq!(stats.read(106, AGE).pstate_status, STALE); } #[test] fn m5_busy_time_integrates_utilisation_windows() { - let mut stats = Telemetry::m5(); + let mut stats = Telemetry::new(Protocol::M5); assert_eq!( (stats.read(0, AGE).busy_status, stats.read(0, AGE).busy_ns), (VALID, 0) @@ -556,6 +602,57 @@ mod tests { assert_eq!(stats.read(107, AGE).busy_ns, before); } + #[test] + fn a18_pro_state_reports_give_the_pstate() { + let mut stats = Telemetry::new(Protocol::A18_PRO); + let mut interval = M5_INTERVAL; + let mut state = M5_STATE; + (state[1], state[3]) = (10, 10); + stats.record(&interval, 100, AGE); + stats.record(&state, 101, AGE); + let view = stats.read(101, AGE); + assert_eq!((view.pstate_status, view.pstate), (VALID, 10)); + assert_eq!(view.pstate_timestamp, 0x59_3c04_2f8f); + // Its table ends at state 10. + (state[1], state[3]) = (11, 11); + stats.record(&state, 102, AGE); + assert_eq!(stats.read(102, AGE).pstate_status, UNAVAILABLE); + // Tag 0x10 is not this firmware's state report. + interval[1] += 1; + stats.record(&interval, 103, AGE); + stats.record(&P1, 103, AGE); + assert_eq!(stats.read(103, AGE).pstate_status, UNAVAILABLE); + (state[1], state[3]) = (1, 1); + stats.record(&state, 104, AGE); + assert_eq!(stats.read(104, AGE).pstate, 1); + // Gated cores are off. + stats.gate(); + let view = stats.read(105, AGE); + assert_eq!((view.pstate_status, view.pstate), (VALID, 0)); + } + + #[test] + fn a18_pro_busy_time_integrates_16_ms_windows() { + let mut stats = Telemetry::new(Protocol::A18_PRO); + let mut words = M5_U[0]; + stats.record(&words, 100, AGE); + // One full 16 ms window at 100%, then one at 25%. + words[1] += 384_000; + stats.record(&words, 101, AGE); + assert_eq!(stats.read(101, AGE).busy_ns, 16_000_000); + words[1] += 384_000; + words[6] = 25; + stats.record(&words, 102, AGE); + assert_eq!(stats.read(102, AGE).busy_ns, 20_000_000); + // Up to two periods is one window; longer is a gap the GPU spent gated. + words[1] += 768_000; + stats.record(&words, 103, AGE); + assert_eq!(stats.read(103, AGE).busy_ns, 28_000_000); + words[1] += 768_001; + stats.record(&words, 104, AGE); + assert_eq!(stats.read(104, AGE).busy_ns, 28_000_000); + } + #[test] fn other_firmwares_ignore_the_m5_records() { let mut stats = Telemetry::default(); diff --git a/drivers/gpu/drm/asahi/stats.rs b/drivers/gpu/drm/asahi/stats.rs index d8f33c932afe5..e0f68f1436980 100644 --- a/drivers/gpu/drm/asahi/stats.rs +++ b/drivers/gpu/drm/asahi/stats.rs @@ -85,7 +85,7 @@ impl StatsSnapshot { } /// The frequency the GPU runs at now, if the firmware reports it, and its highest, in - /// Hz, for fdinfo (`drm-curfreq-*`, `drm-maxfreq-*`): M5 only. + /// Hz, for fdinfo (`drm-curfreq-*`, `drm-maxfreq-*`): V26A428 (A18 Pro and M5) only. pub(crate) fn engine_frequency(&self) -> Option<(Option, u64)> { self.g17_reader.as_ref()?.frequency() } diff --git a/drivers/gpu/drm/asahi/sysfs.c b/drivers/gpu/drm/asahi/sysfs.c index ac2139d1ebb9b..f7004b10a400e 100644 --- a/drivers/gpu/drm/asahi/sysfs.c +++ b/drivers/gpu/drm/asahi/sysfs.c @@ -91,8 +91,8 @@ static ssize_t agx_qualified_stats_show(struct asahi_stats_attribute *stats, cha } /* * unsupported comes from the firmware match data (g17::stats): V25G83 - * and V26A428 never emit a pstate record, so it is not a transient - * state. unavailable keeps meaning "no qualified sample yet". + * never emits a pstate record, so it is not a transient state. + * unavailable keeps meaning "no qualified sample yet". */ n += sysfs_emit_at(buf, n, "pstate_status %s\n", asahi_stats_quality(snap.pstate_status)); @@ -100,10 +100,13 @@ static ssize_t agx_qualified_stats_show(struct asahi_stats_attribute *stats, cha n += sysfs_emit_at(buf, n, "pstate %u\n", snap.pstate); else n += sysfs_emit_at(buf, n, "pstate unsupported\n"); - /* No A/B selector, measured power or temperature. */ + /* No host frequency request, measured power or temperature. */ n += sysfs_emit_at(buf, n, "requested_frequency_hz unsupported\n"); - /* M5 reports the state the GPU runs at: its frequency is the achieved one. */ - if (snap.pstate_status == 1 && snap.achieved_frequency_hz) + /* + * V26A428 reports the state the GPU runs at: its frequency is the + * achieved one. State 0 is off (gated cores) and runs at 0 Hz. + */ + if (snap.pstate_status == 1 && (snap.achieved_frequency_hz || !snap.pstate)) n += sysfs_emit_at(buf, n, "achieved_frequency_hz %llu\n", (unsigned long long)snap.achieved_frequency_hz); else diff --git a/tools/asahi/g17-telemetry-integration-tests.py b/tools/asahi/g17-telemetry-integration-tests.py index 8dde658062689..5420559c2fb84 100644 --- a/tools/asahi/g17-telemetry-integration-tests.py +++ b/tools/asahi/g17-telemetry-integration-tests.py @@ -184,7 +184,8 @@ def item(source, signature): } let view=reader.read(); assert_eq!((view.util_status,view.device,view.util_samples),(telemetry::VALID,0,1)); - assert_eq!(view.pstate_status,telemetry::UNSUPPORTED); + // 26A428 reports its state in tags 0x19/0x1b, none of which arrived; 25G83 never does. + assert_eq!(view.pstate_status,match graph { InitGraph::G17P=>telemetry::UNAVAILABLE, _=>telemetry::UNSUPPORTED }); NOW.with(|now|now.set(1)); } } @@ -259,7 +260,9 @@ def item(source, signature): for words in [HIST_IDLE,HIST_ACTIVE,HIST_OTHER,TRANSITION] { init.push(Secondary,words); rings.drain_statistics(&dev,&init,Secondary); - assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::UNSUPPORTED,..Default::default() }); + // 26A428 integrates busy time from its utilisation records; none arrived. + let busy=match graph { InitGraph::G17P=>telemetry::VALID, _=>telemetry::UNSUPPORTED }; + assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:busy,..Default::default() }); assert!(report(&rings).contains(&format!("words={:08x?}",words))); } } @@ -286,7 +289,7 @@ def item(source, signature): let line=text.lines().find(|line|line.starts_with(&format!("role={} tag={:#x} ",role,tag))).unwrap(); assert!(line.contains("count=2 host_ns=1 admitted=1 last_admitted=true"),"{}",line); } } - assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::UNSUPPORTED,..Default::default() }); + assert_eq!(rings.read_statistics(1),telemetry::Readout { busy_status:telemetry::VALID,..Default::default() }); } } @@ -295,7 +298,8 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); + // The decoder's tag 0x10 path: only a firmware without 26A428's state records decodes it. + let mut rings=Rings::new(InitGraph::V25G83,false).unwrap(); let util=[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]; let pstate=[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]; rings.observe_statistics_power(&dev,&init,&[Primary,Secondary],true,1); @@ -322,28 +326,62 @@ def item(source, signature): } #[test] - fn current_firmwares_report_pstate_unsupported_and_keep_utilisation() { + fn v25g83_reports_pstate_unsupported_and_keeps_utilisation() { use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; - for graph in [InitGraph::V25G83,InitGraph::G17P] { - assert!(!stats::reports_pstate(graph,false)); - assert!(stats::reports_pstate(graph,true)); - let empty=stats::qualify(graph,false,telemetry::Readout::default()); - assert_eq!((empty.util_status,empty.pstate_status),(telemetry::UNAVAILABLE,telemetry::UNSUPPORTED)); - let init=initdata::InitData::new(); - let mut rings=Rings::new(graph,false).unwrap(); - init.push(Primary,[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]); - // Even a decodable tag 0x10 does not override the firmware's match data. - init.push(Primary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); - rings.drain_statistics(&dev,&init,Primary); - assert_eq!(rings.read_statistics(1).pstate_status,telemetry::VALID); - let view=stats::qualify(graph,false,rings.read_statistics(1)); - assert_eq!((view.util_status,view.device,view.pstate_status,view.pstate,view.pstate_timestamp), - (telemetry::VALID,97,telemetry::UNSUPPORTED,0,0)); - rings.invalidate_statistics(); - let view=stats::qualify(graph,false,rings.read_statistics(1)); - assert_eq!((view.util_status,view.pstate_status),(telemetry::STALE,telemetry::UNSUPPORTED)); - } + // 26A428 reports its state on the A18 Pro and on M5; 25G83 never does. + assert!(!stats::reports_pstate(InitGraph::V25G83,false)); + assert!(stats::reports_pstate(InitGraph::G17P,false)); + assert!(stats::reports_pstate(InitGraph::G17P,true)); + let graph=InitGraph::V25G83; + let empty=stats::qualify(graph,false,telemetry::Readout::default()); + assert_eq!((empty.util_status,empty.pstate_status),(telemetry::UNAVAILABLE,telemetry::UNSUPPORTED)); + let init=initdata::InitData::new(); + let mut rings=Rings::new(graph,false).unwrap(); + init.push(Primary,[0x13,0,0,8,97,0,97,0,0,0,0,0,0,0,0,0,0,0]); + // Even a decodable tag 0x10 does not override the firmware's match data. + init.push(Primary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); + rings.drain_statistics(&dev,&init,Primary); + assert_eq!(rings.read_statistics(1).pstate_status,telemetry::VALID); + let view=stats::qualify(graph,false,rings.read_statistics(1)); + assert_eq!((view.util_status,view.device,view.pstate_status,view.pstate,view.pstate_timestamp), + (telemetry::VALID,97,telemetry::UNSUPPORTED,0,0)); + rings.invalidate_statistics(); + let view=stats::qualify(graph,false,rings.read_statistics(1)); + assert_eq!((view.util_status,view.pstate_status),(telemetry::STALE,telemetry::UNSUPPORTED)); + } + + #[test] + fn a18_pro_reports_its_state_through_the_reader() { + use {fw::variant::InitGraph,Role::*}; + // States 0 to 10; an interval record, then the state in words 1 and 3. + let mhz=[0,338,492,618,796,928,1056,1170,1278,1338,1470]; + let reader=stats::Reader(reader(InitGraph::G17P).0,stats::StateFrequencies::new(mhz.into_iter())); + let push=|words|reader.0.state.lock().as_ref().unwrap().init.push(Secondary,words); + // Tag 0x10 is not this firmware's state report. + push([0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); + assert_eq!(reader.read().pstate_status,telemetry::UNAVAILABLE); + assert_eq!(reader.frequency(),Some((None,1_470_000_000))); + push([0x19,0x2000,0x40,0x1000,0x40,0,0,0,0,0,0,0,0,0,0,0,0,0]); + push([0x1b,7,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0]); + let view=reader.read(); + assert_eq!((view.pstate_status,view.pstate,view.pstate_timestamp,view.achieved_frequency_hz), + (telemetry::VALID,7,0x40_0000_2000,1_170_000_000)); + assert_eq!(reader.frequency(),Some((Some(1_170_000_000),1_470_000_000))); + // A state beyond the table is not a state. + push([0x1b,11,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0]); + assert_eq!(reader.read().pstate_status,telemetry::UNAVAILABLE); + push([0x1b,10,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0]); + assert_eq!(reader.read().achieved_frequency_hz,1_470_000_000); + // Gated cores run at no state: off, 0 Hz. + reader.0.state.lock().as_mut().unwrap().powered=false; + let view=reader.read(); + assert_eq!((view.pstate_status,view.pstate,view.achieved_frequency_hz),(telemetry::VALID,0,0)); + assert_eq!(reader.frequency(),Some((Some(0),1_470_000_000))); + // Power-up starts a new epoch: no state until the firmware reports one. + reader.0.state.lock().as_mut().unwrap().powered=true; + assert_eq!(reader.read().pstate_status,telemetry::STALE); + assert_eq!(reader.frequency(),Some((None,1_470_000_000))); } #[test] @@ -468,7 +506,7 @@ def item(source, signature): use {channel::Rings,fw::variant::InitGraph,Role::*}; let dev=crate::device::Device; let init=initdata::InitData::new(); - let mut rings=Rings::new(InitGraph::G17P,false).unwrap(); + let mut rings=Rings::new(InitGraph::V25G83,false).unwrap(); init.push(Primary,[0x10,0x1000,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0]); init.push(Secondary,[0x10,0x2500,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0]); rings.drain_statistics(&dev,&init,Primary); @@ -587,7 +625,17 @@ def item(source, signature): assert(strstr(now,status==2?"utilisation_status stale\n":"utilisation_status unavailable\n")); } } - /* G16S/G17P: pstate unsupported by firmware, utilisation still valid. */ + /* 26A428 reports the state the GPU runs at; state 0 (gated cores) runs at 0 Hz. */ + current=(struct asahi_qualified_stats){.util_status=1,.pstate_status=1,.pstate=10,.achieved_frequency_hz=1470000000}; + agx_stats_show(NULL,&stats.attr,now); + assert(strstr(now,"pstate_status valid\npstate 10\nrequested_frequency_hz unsupported\nachieved_frequency_hz 1470000000\n")); + current.pstate=0;current.achieved_frequency_hz=0; + agx_stats_show(NULL,&stats.attr,now); + assert(strstr(now,"pstate 0\nrequested_frequency_hz unsupported\nachieved_frequency_hz 0\n")); + current.pstate_status=2; + agx_stats_show(NULL,&stats.attr,now); + assert(strstr(now,"pstate unsupported\nrequested_frequency_hz unsupported\nachieved_frequency_hz unsupported\n")); + /* G16S: pstate unsupported by firmware, utilisation still valid. */ current=(struct asahi_qualified_stats){.util_status=1,.pstate_status=3,.tiler=30,.renderer=80,.device=57,.pstate=0, .util_samples=19,.util_host_ns=123}; agx_stats_show(NULL,&stats.attr,now); @@ -674,7 +722,7 @@ def main(): assert 'for &role in firmware.roles()' in stats_rs stats_rs = stats_rs.replace('for &role in firmware.roles()', 'for &role in &[]', 1) policy = '\n'.join(item(stats_rs, signature) for signature in - ['pub(super) const fn max_age_ns', 'pub(super) const fn reports_pstate', + ['pub(super) const fn max_age_ns', 'pub(super) const fn protocol(', 'pub(super) const fn reports_pstate', 'pub(super) fn qualify(']) # The rings take the family's statistics tags and control power rule from # InitData; build those production methods into the fixture's InitData. @@ -688,9 +736,10 @@ def main(): header = (RUST_HEADER.replace('M5_STATISTICS_TYPES', m5_types).replace('M5_ASSERTS_GPU_POWER', m5_power) .replace('INITDATA_FAMILY', family)) - reader_rs = ('pub(crate) struct Reader(pub(super) Arc, ' - 'pub(super) Option<&\'static [(u32, [u32; 8], [u32; 8]); 14]>);\n' - + item(stats_rs, 'impl Reader').replace('super::fw::m5::initdata::M5PerfState', '(u32, [u32; 8], [u32; 8])')) + reader_rs = ('#[derive(Clone, Copy, Debug, PartialEq, Eq)]\n' + item(stats_rs, 'pub(crate) struct StateFrequencies') + + '\n' + item(stats_rs, 'impl StateFrequencies') + '\n' + + 'pub(crate) struct Reader(pub(super) Arc, pub(super) Option);\n' + + item(stats_rs, 'impl Reader')) (work / 'rings.rs').write_text(header + 'mod stats { use super::{fw::variant::InitGraph, telemetry::Readout, Shared};\n' + 'use std::sync::{Arc,atomic::Ordering};\n' + reader_rs + '\n' + policy + '\n}\nmod telemetry;\nmod channel;\n' + RUST_TESTS)