Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,13 @@ jobs:

# The portable configurations the README advertises, so they cannot rot.
portable:
name: no_std + wasm32
name: no_std + wasm32 + bare metal
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown
targets: wasm32-unknown-unknown, thumbv7em-none-eabihf, riscv32imac-unknown-none-elf
- uses: Swatinem/rust-cache@v2
- name: no_std + alloc
run: cargo check -p rusty_zstd --no-default-features --features alloc
Expand All @@ -84,6 +84,19 @@ jobs:
run: cargo check -p rusty_zstd --target wasm32-unknown-unknown --no-default-features --features alloc
- name: profile feature builds
run: cargo check -p rusty_zstd --features profile
# BARE METAL. A `no-std` category on the registry is not a claim; a target
# that compiles is. Both of these failed with 204 errors until the census
# counters stopped requiring 64-bit atomics -- and every one of those errors
# was a counter, not a line of codec. `census64` makes the counter a
# zero-sized stub where `AtomicU64` does not exist, so the instrument costs
# the crate a diagnostic on those parts instead of costing it the parts.
#
# Cortex-M4F and RV32IMAC are what rusty_RTOS (Kairos) consumes the codec on,
# for on-chip OTA payloads and trace capture. Neither has 64-bit atomics.
- name: bare metal (Cortex-M4F, no 64-bit atomics)
run: cargo check -p rusty_zstd --target thumbv7em-none-eabihf --no-default-features --features alloc
- name: bare metal (RV32IMAC, no 64-bit atomics)
run: cargo check -p rusty_zstd --target riscv32imac-unknown-none-elf --no-default-features --features alloc

lint:
runs-on: ubuntu-latest
Expand Down
45 changes: 45 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,51 @@ based on [Keep a Changelog](https://keepachangelog.com/); this project uses

## [Unreleased]

### Fixed -- the census counters cost the crate BARE METAL; they no longer do

`cargo check -p rusty_zstd --no-default-features --features alloc` failed with
**204 errors** on `thumbv7em-none-eabihf` (Cortex-M4F) and the same on
`riscv32imac-unknown-none-elf`. Every one of them was
`cannot find AtomicU64 in atomic`, and every one was an INSTRUMENT: reload
counts, decode band histograms, kernel reach, walk exits, copy bytes. Not one
was a line of codec. Neither part has 64-bit atomics.

This was found by `rusty_RTOS`'s house gate, which consumes the crate for
on-chip OTA payloads and trace capture, and it is the reason its plan says a
`no-std` category on the registry is not a claim: **the target that compiles is
the claim.** The plan estimated eight statics from a capped rustc report; the
real count is 427 fully-qualified uses plus six imports across sixteen files.

`census64` is now the one seam. On every target that HAS 64-bit atomics it is
`pub use core::sync::atomic::AtomicU64` -- the same type, not a wrapper -- so
the public statics keep their published type and the emitted code is unchanged.
**Proved, not asserted:** the board over every shipping kernel, fill and finder
is identical to the pre-change assembly in all thirty-two columns, and the
identity gates still read GOLD `2F6594F7EEDBD12B` / 59,680,638 and LDM
`57BE83EA4E1199E8` / 57,796,847.

On a part without them it is a zero-sized stub: the statics leave BSS entirely
and every `fetch_add` folds away.

**Why a stub rather than `portable-atomic`.** That crate would keep the census
READABLE there, and it is the other honest answer. But on a core with no 64-bit
atomic instruction it needs a critical-section implementation, and a library
that turns that feature on conscripts every downstream firmware's interrupt
policy so a diagnostic counter can increment. The seam is one type wide, so a
firmware that does want the count can supply `portable_atomic::AtomicU64`.

**A zero there means "not measurable on this target", never "measured zero"**,
which is the same trap this crate's own rules warn about -- so it is published,
not buried: `census64::CENSUS_LIVE` is `false` on such a build. The reverse
mistake would be far worse (a stub selected on a HOSTED target would make every
count-based verdict fiction while every gate still passed), so it is gated two
ways: a compile-time size check, and a unit test that asserts on BEHAVIOUR
rather than on the constant -- it counts to 42 and reads it back.

CI gains the rung, in the job that already guards the portable configurations:
`--target thumbv7em-none-eabihf` and `--target riscv32imac-unknown-none-elf`,
both `--no-default-features --features alloc`, on every push.

### Measured -- what the encode campaign bought: 1.08-1.27x at L3-L12 (2026-09-09)

Every verdict in the sections below is an INSTRUCTION COUNT, because this box
Expand Down
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,8 +347,13 @@ implementation cannot leak into the dependency graph.
- **Byte-identity where it is owed.** The XXH64 checksum is gated against the
published XXH64 vectors and a GOLD oracle; every SIMD kernel is gated against
its scalar twin; the Huffman and FSE table decoders are gated against C's.
- **`no_std + alloc` and `wasm32-unknown-unknown` build in CI**, so the portable
configuration cannot rot.
- **`no_std + alloc`, `wasm32-unknown-unknown` and two bare-metal targets build
in CI**, so the portable configuration cannot rot. The bare-metal rungs are
Cortex-M4F and RV32IMAC, neither of which has 64-bit atomics: a `no-std`
category is not a claim, a target that compiles is. Note that the measurement
counters need `AtomicU64`, so on a part without it they compile to a
zero-sized stub and `census64::CENSUS_LIVE` reads `false` -- a zero from a
counter there means "not measurable on this target", never "measured zero".

Fetch the oracle with `pwsh scripts/fetch-oracle.ps1`, or point
`RUSTY_ZSTD_ORACLE` at any `zstd` binary whose `--version` says 1.5.7. Details:
Expand All @@ -363,6 +368,8 @@ Fetch the oracle with `pwsh scripts/fetch-oracle.ps1`, or point
| macOS (x86-64 / aarch64) | ✅ builds + tests |
| `wasm32-unknown-unknown` | ✅ builds (`std` and `alloc`) |
| `no_std + alloc` | ✅ builds |
| `thumbv7em-none-eabihf` (Cortex-M4F) | ✅ builds (`no_std + alloc`) |
| `riscv32imac-unknown-none-elf` | ✅ builds (`no_std + alloc`) |

AVX2 and NEON kernels are selected at **runtime**. On a CPU without them, the
scalar twins run and the output is identical.
Expand Down
4 changes: 2 additions & 2 deletions crates/rusty_zstd/src/bit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ pub(crate) struct BitRev<'a> {

/// Executed `BitRev::reload` calls. See the note inside `reload`.
#[cfg(feature = "profile")]
pub static RELOAD_CALLS: core::sync::atomic::AtomicU64 = core::sync::atomic::AtomicU64::new(0);
pub static RELOAD_CALLS: crate::census64::AtomicU64 = crate::census64::AtomicU64::new(0);

/// Executed refills -- reloads that reached the container load rather than
/// taking one of the four early-outs. See the note inside `reload`.
#[cfg(feature = "profile")]
pub static RELOAD_REFILLS: core::sync::atomic::AtomicU64 = core::sync::atomic::AtomicU64::new(0);
pub static RELOAD_REFILLS: crate::census64::AtomicU64 = crate::census64::AtomicU64::new(0);

/// Read and clear the refill counter.
#[cfg(feature = "profile")]
Expand Down
183 changes: 183 additions & 0 deletions crates/rusty_zstd/src/census64.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
//! The census counter's 64-bit atomic, and what it becomes on a part that has
//! no 64-bit atomics.
//!
//! WHY THIS MODULE EXISTS. Every measurement instrument in this crate counts
//! into a `static AtomicU64`: reload calls, decode band histograms, kernel
//! reach, walk exits, copy bytes. That is deliberate and the codec skills
//! depend on it -- a count is the only instrument that separates "this kernel
//! does not help" from "this arm is wired to nothing".
//!
//! It also cost the crate two targets. `core::sync::atomic::AtomicU64` does
//! not exist on a Cortex-M4F (`thumbv7em-none-eabihf`) or on RV32
//! (`riscv32imac-unknown-none-elf`), so `--no-default-features --features
//! alloc` failed with 204 errors on both -- every one of them a counter, not a
//! line of codec. A `no-std` label on the registry is not a claim; a target
//! that compiles is.
//!
//! WHAT IT DOES. On every target that HAS 64-bit atomics this is exactly
//! `core::sync::atomic::AtomicU64`, re-exported. Same type, same ABI, no
//! wrapper, so the public statics keep their published type and the emitted
//! code is byte-for-byte what it was.
//!
//! On a target that does not, it is a zero-sized stub whose operations do
//! nothing and whose loads read 0. The statics stop occupying BSS, the
//! `fetch_add`s vanish, and the codec keeps its targets.
//!
//! WHY A STUB AND NOT `portable-atomic`. That crate is the other honest
//! answer, and it would keep the census READABLE on those parts. But on a
//! core with no 64-bit atomic instruction it needs a critical-section
//! implementation, and a library that turns that feature on forces the choice
//! onto every downstream firmware. A codec should not conscript the
//! application's interrupt policy so that a diagnostic counter can increment.
//! Anyone who does want the count on such a part can supply
//! `portable_atomic::AtomicU64` here; the seam is one type wide.
//!
//! HONESTY. A counter that silently reads zero is exactly the "stale
//! instrument" this crate's own rules warn about, so the fact is published
//! rather than buried: [`CENSUS_LIVE`] is `false` on such a target, and any
//! gate or report that asserts on a census must consult it before believing a
//! zero. **Zero here means "not measurable on this target", never "measured
//! zero".**
//!
//! The reverse mistake would be worse and is gated below: if this stub were
//! ever selected on a HOSTED target, every counter in the crate would read 0
//! and every census-based verdict would silently become fiction. The unit test
//! fails the build if that happens.

/// Whether the census counters in this build can actually count.
///
/// `true` wherever `AtomicU64` exists (every hosted target this crate ships
/// on). `false` on a part without 64-bit atomics, where every counter is a
/// stub and every reader returns 0. Check this before believing a zero.
pub const CENSUS_LIVE: bool = cfg!(target_has_atomic = "64");

#[cfg(target_has_atomic = "64")]
pub use core::sync::atomic::AtomicU64;

#[cfg(not(target_has_atomic = "64"))]
pub use stub::AtomicU64;

#[cfg(not(target_has_atomic = "64"))]
mod stub {
use core::sync::atomic::Ordering;

/// A do-nothing stand-in for `core::sync::atomic::AtomicU64` on a target
/// that has no 64-bit atomics. Zero-sized: a `static` of this type costs
/// no BSS and every operation folds away.
///
/// It is `Sync` for the same reason `()` is: there is no state to race
/// on. Loads return 0, and [`super::CENSUS_LIVE`] is `false` so a reader
/// can tell that apart from a real zero.
#[derive(Debug, Default)]
pub struct AtomicU64(());

impl AtomicU64 {
/// The value is discarded: this build cannot hold one.
#[inline(always)]
pub const fn new(_v: u64) -> Self {
Self(())
}

/// Always 0. See [`super::CENSUS_LIVE`].
#[inline(always)]
pub fn load(&self, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn store(&self, _v: u64, _order: Ordering) {}

/// Always 0 (the previous value this build never held).
#[inline(always)]
pub fn swap(&self, _v: u64, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn fetch_add(&self, _v: u64, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn fetch_sub(&self, _v: u64, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn fetch_max(&self, _v: u64, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn fetch_min(&self, _v: u64, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn fetch_or(&self, _v: u64, _order: Ordering) -> u64 {
0
}

#[inline(always)]
pub fn fetch_and(&self, _v: u64, _order: Ordering) -> u64 {
0
}

/// Always reports the stored value as 0, so a compare against
/// anything else fails -- the same shape a real CAS has when it loses.
#[inline(always)]
pub fn compare_exchange(
&self,
current: u64,
_new: u64,
_success: Ordering,
_failure: Ordering,
) -> Result<u64, u64> {
if current == 0 {
Ok(0)
} else {
Err(0)
}
}

#[inline(always)]
pub fn compare_exchange_weak(
&self,
current: u64,
new: u64,
success: Ordering,
failure: Ordering,
) -> Result<u64, u64> {
self.compare_exchange(current, new, success, failure)
}
}
}

/// The selected type must be a REAL 64-bit atomic wherever one exists. If a cfg
/// edit ever let the stub through here, this fails the build rather than the
/// census. (The stub is zero-sized; the real one is eight bytes.)
#[cfg(target_has_atomic = "64")]
const _: () = [(); 1][(core::mem::size_of::<AtomicU64>() != 8) as usize];

#[cfg(test)]
mod tests {
/// The census must actually COUNT anywhere tests run. Asserted on behaviour,
/// not on the `CENSUS_LIVE` constant: if a cfg edit ever selected the stub on
/// a hosted target, every counter in the crate would read 0, `kreach_gate`
/// would see a perfectly-routed 0/0 everywhere, and the whole instrument
/// would report success while measuring nothing.
#[test]
fn census_counts_wherever_tests_run() {
assert!(super::CENSUS_LIVE == cfg!(target_has_atomic = "64"));
let c = super::AtomicU64::new(0);
c.fetch_add(7, core::sync::atomic::Ordering::Relaxed);
c.fetch_add(35, core::sync::atomic::Ordering::Relaxed);
assert_eq!(
c.swap(0, core::sync::atomic::Ordering::Relaxed),
42,
"the AtomicU64 stub was selected on a target that HAS 64-bit atomics: \
every census counter in this crate now reads 0 and every count-based \
verdict from it is fiction"
);
}
}
Loading
Loading