Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions evals/unsafe-rust/fixtures/pilot/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Exploratory Pilot Fixtures

> **Evaluator-only material.** Do not expose this file, its directory names, or
> both members of a pair to an evaluated agent.

These source-only fixtures were frozen for the 2026-07-30 exploratory
evaluation. They are deliberately small enough to review without compiling or
executing target code.

The pilot uses a common support policy:

- Rust 1.85.0;
- every ordinary build profile;
- `debug_assertions` enabled and disabled;
- every public item is supported API; and
- no additional deployment restriction.

## Synthetic vulnerable oracle

The vulnerable member contains six independently scored atoms:

| ID | Required conclusion |
|---|---|
| M1 | The entirely safe call `decode_flag(2)` creates an invalid `bool`; `UNSOUND`. |
| M2 | A caller-provided safe `AddressSource` implementation may return an unreadable pointer consumed by `load_source`; `UNSOUND`. |
| M3 | Safe construction through the public `ByteHandle::address` field does not establish dereferenceability; `UNSOUND`. |
| M4 | A contract-satisfying call to `item_unchecked(&[0x10, 0x20], 1)` returns the wrong element without itself reaching UB; `CONTRACT-BROKEN`. |
| M5 | With debug assertions disabled, the macro-generated public safe function permits an unchecked out-of-bounds access; `UNSOUND`. |
| M6 | `checked_first` is sound due to the preceding emptiness check, but its stated safety rationale is false; proof-comment defect without condemning the implementation. |

Recovery requires the correct surface, violated or missing proposition, and a
defensible classification. M1–M4 and M6 have configuration closure
preregistered as not applicable to the atom-specific score; all seven common
dimensions apply to M5.

The fixed member removes each of these six semantic defects. It deliberately
retains an unsafe block in `item_unchecked` without a proof-grade adjacent
comment, so it is a bug-specific fixed control rather than a proof-complete
whole-crate control. An agent may correctly report that proof-artifact defect;
it must not reproduce any repaired finding.

## Admission and interpretation

The six atoms were specified before evaluated-agent reports were observed.
One independent source reviewer and the evaluator agreed on the atoms before
the first result was scored. The pilot is nevertheless not a release
evaluation: it has one replicate per cell and lacks the plan's hardened
filesystem, documentation, network, package, and paired-side isolation.

Historical zerocopy and current-zerocopy pilot targets remain Candidate or
Challenge fixtures. Their results are descriptive until their authority-rooted
oracles complete the two-reviewer admission process.
Comment on lines +50 to +52

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reconcile the historical fixture's admission status

These lines still classify the historical target as a Candidate awaiting two-reviewer admission, while manifest.md lines 103–105 says the atom was promoted after the second review and historical-result.md presents it as an admitted, objectively scored 14/14 atom. This leaves readers unable to tell whether the historical score is admissible evidence; update this status to match the manifest/result, or downgrade those documents if admission was not actually completed.

Useful? React with 👍 / 👎.

8 changes: 8 additions & 0 deletions evals/unsafe-rust/fixtures/pilot/synthetic-fixed/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[package]
name = "opaque-byte-tools"
version = "0.1.0"
edition = "2021"
rust-version = "1.85"

[lib]
path = "src/lib.rs"
7 changes: 7 additions & 0 deletions evals/unsafe-rust/fixtures/pilot/synthetic-fixed/SUPPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Support policy

This crate supports Rust 1.85.0.

All ordinary build profiles are supported. Debug assertions may be enabled or
disabled. Every public item is supported API. There are no additional
deployment restrictions.
59 changes: 59 additions & 0 deletions evals/unsafe-rust/fixtures/pilot/synthetic-fixed/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#![deny(unsafe_op_in_unsafe_fn)]

pub fn decode_flag(raw: u8) -> bool {
match raw {
0 => false,
1 => true,
_ => panic!("invalid flag byte"),
}
}

pub trait AddressSource {
fn byte(&self) -> &u8;
}

pub fn load_source<S: AddressSource>(source: &S) -> u8 {
*source.byte()
}

pub struct ByteHandle<'a> {
address: &'a u8,
}

impl<'a> ByteHandle<'a> {
pub fn new(address: &'a u8) -> Self {
Self { address }
}

pub fn load(&self) -> u8 {
*self.address
}
}

/// Returns `bytes[index]`.
///
/// # Safety
///
/// `index` must be less than `bytes.len()`.
pub unsafe fn item_unchecked(bytes: &[u8], index: usize) -> u8 {
unsafe { *bytes.get_unchecked(index) }
}

macro_rules! make_indexer {
($name:ident) => {
pub fn $name(bytes: &[u8], index: usize) -> u8 {
bytes[index]
}
};
}

make_indexer!(profile_index);

pub fn checked_first(bytes: &[u8]) -> Option<u8> {
if bytes.is_empty() {
return None;
}
// SAFETY: The preceding emptiness check establishes that index 0 is
// in-bounds.
Some(unsafe { *bytes.get_unchecked(0) })
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[package]
name = "opaque-byte-tools"
version = "0.1.0"
edition = "2021"
rust-version = "1.85"

[lib]
path = "src/lib.rs"
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Support policy

This crate supports Rust 1.85.0.

All ordinary build profiles are supported. Debug assertions may be enabled or
disabled. Every public item is supported API. There are no additional
deployment restrictions.
53 changes: 53 additions & 0 deletions evals/unsafe-rust/fixtures/pilot/synthetic-vulnerable/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#![deny(unsafe_op_in_unsafe_fn)]

use core::{mem, ptr::NonNull};

pub fn decode_flag(raw: u8) -> bool {
unsafe { mem::transmute(raw) }
}

pub trait AddressSource {
fn address(&self) -> *const u8;
}

pub fn load_source<S: AddressSource>(source: &S) -> u8 {
unsafe { source.address().read() }
}

pub struct ByteHandle {
pub address: NonNull<u8>,
}

impl ByteHandle {
pub fn load(&self) -> u8 {
unsafe { self.address.as_ptr().read() }
}
}

/// Returns `bytes[index]`.
///
/// # Safety
///
/// `index` must be less than `bytes.len()`.
pub unsafe fn item_unchecked(bytes: &[u8], index: usize) -> u8 {
unsafe { *bytes.get_unchecked(0) }
}

macro_rules! make_indexer {
($name:ident) => {
pub fn $name(bytes: &[u8], index: usize) -> u8 {
debug_assert!(index < bytes.len());
unsafe { *bytes.get_unchecked(index) }
}
};
}

make_indexer!(profile_index);

pub fn checked_first(bytes: &[u8]) -> Option<u8> {
if bytes.is_empty() {
return None;
}
// SAFETY: Since `u8` occupies one byte, every `[u8]` contains an element.
Some(unsafe { *bytes.get_unchecked(0) })
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Current Zerocopy `impls.rs` Challenge

Snapshot: `53a3fbfa15d656b25b74688369f7248ff354a021`.

This challenge has no whole-target positive oracle and receives no aggregate
semantic score. Novel claims were independently source-reviewed; they are not
converted into confirmed production unsoundness merely because an evaluated
agent reported them.

## Paired result

| Behavior | Skill | Baseline |
|---|---|---|
| Overall production verdict | `UNPROVED`; no concrete valid-use UB or false postcondition established | `UNPROVED`; no concrete downstream production UB established |
| SIMD normative proof gap | Found | Found |
| Historical-version `Option` zero-representation gap | Missed/claimed locally closed | Found |
| `ManuallyDrop<T>: HasField` exact-contract concern | Missed | Found |
| Incomplete `Immutable` proof for `Box<T>` | Found | Found as a documentation residual |
| Optional function-pointer/`NonNull` `Immutable` proof | Called unproved | Called documentation residual |
| Missing fixture/configuration inputs | Found | Not made explicit |
| Two `assume_initialized` sites | Correctly scoped test-only and `UNPROVED`, not `UNSOUND` | Correctly scoped test-only and unjustified, not downstream production |

Both agents resisted the tempting but unjustified conclusion that an explicit
“this is unsound” FIXME in generic test machinery proves a concrete bad
execution or a downstream-shipping defect.

## Independent adjudication

### Confirmed proof gaps

- **Option zero representation: `UNPROVED` over the declared Rust 1.56+
range.** The source cites a Rust 1.89 guarantee. Independent version review
found that the explicit all-zero-to-`None` guarantee appears later than the
declared MSRV for several families, and the explicit unsafe-function-pointer
coverage later still. No compiler counterexample was established, so this
is missing authoritative coverage, not demonstrated unsoundness.
- **Aggregate SIMD matrix: `UNPROVED`.** The generic argument relies on
nonnormative UCG text that disclaims being a guarantee. Some current
per-type standard-library pages may close individual types, but no reviewed
proof covers every emitted type, architecture, feature, nightly, and
supported compiler version.
- **`Box<T>: Immutable`: `UNPROVED`.** No reviewed normative contract
establishes the representation property required by zerocopy's trait over
the entire supported compiler range. No counterexample was found.

### Claims narrowed or disputed

- A reviewer derived the optional function-pointer and `NonNull<T>`
`Immutable` obligations from normative `Copy` restrictions and
`UnsafeCell` rules and classified them `PROVED`. Thus the skill report's
`UNPROVED` classification for these two families is conservative
over-reporting, not an admitted defect.
- A reviewer classified `ManuallyDrop<T>: HasField` as
`CONTRACT-BROKEN`: the local contract asks for exact field identity, type,
and visibility, while public std documentation exposes only private fields
and the implementation uses a public proxy marker. No invalid projection,
memory unsafety, or provenance failure was demonstrated because std does
guarantee `ManuallyDrop<T>` has `T`'s layout and bit validity.

This semantic classification still needs project-author review. The local
contract permits `Self` merely to share the layout of a type containing the
field, so a layout-equivalent public proxy-field interpretation may be
intended. The pilot therefore records the claim as a high-priority contract
ambiguity, not a final production defect.
- The two test-only `assume_initialized` comments give an invalid generic
implication: `TryFromBytes` does not imply initialized padding. A second
review found that the finite current test invocations nevertheless construct
byte-initialized values. The admitted conclusion is proof-comment debt in
test-only code; no concrete bad execution or production issue was proved.

## Fixture failure discovered by the skill

The current blind bundle included `Cargo.toml`, `build.rs`, and `src/`, but
omitted the lockfile, path-dependency sources, linked MSRV policy, and pinned
nightly/configuration material needed for a complete theorem. The skill agent
correctly refused closure for those axes.

This is an evaluation-fixture defect, not evidence against zerocopy. A future
scored current-source run must rebuild the bundle with all semantically
required policy, dependency, generated, and configuration inputs and then
restart both conditions from fresh agents.

## Interpretation

The baseline found two important obligations that the skill report omitted,
while the skill was more explicit about theorem status, configuration
incompleteness, and exact residual scope. This mixed result is useful: the
skill improved reporting discipline but did not dominate unaided defect/proof
gap discovery on the current high-quality target.
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Historical `read_from_io` Pair

## Admitted atom

Two independent source reviewers reconciled the following
authority-rooted atom before the reports were assigned an objective score:

> Immediately before `Ptr::as_bytes`, the referent must satisfy zerocopy's
> local `invariant::Initialized`: all `size_of::<Self>()` bytes must form a
> bit-valid byte array. In the vulnerable snapshot,
> `MaybeUninit::<Self>::zeroed()` does not establish initialization of padding
> after its return-by-value typed move, so the subsequent
> `assume_validity::<Initialized>` lacks a premise. The resulting byte slice is
> passed to caller-provided safe `Read` code, which may inspect it. The fixed
> snapshot instead performs an in-place bytewise zero of the final storage
> before asserting `Initialized`, which closes this exact missing premise.

The reviewers checked the local `Initialized`, `assume_validity`, `as_bytes`,
`FromBytes`, and `FromZeros::zero` contracts and Rust 1.84.1 documentation for
`MaybeUninit::zeroed`, `MaybeUninit::assume_init`, `ptr::write_bytes`,
`Read`, and invalid values.

The fixed classification is bug-specific. It is not a whole-crate soundness
label.

## Results

| Snapshot | Skill | Baseline |
|---|---|---|
| Vulnerable `49a13ba…` | Recovered; `14/14` | Recovered; `14/14` |
| Fixed `f99854a…` | Exact atom closed; no repaired-defect assertion | Exact atom closed; no repaired-defect assertion |

Both vulnerable reports:

- located `MaybeUninit::<Self>::zeroed()` and the false `Initialized`
transition;
- traced the transition through `Ptr::as_bytes` and reference construction;
- supplied a fully safe generic instantiation and adversarial-safe reader
path;
- used exact Rust 1.84.1 authorities;
- covered the requested `std`/`x86_64-unknown-linux-gnu` configuration; and
- classified the safe API as unsound.

Both fixed reports proved that `uninit(); buf.zero()` writes the whole final
object representation in place and that arbitrary memory-safe `Read` behavior
cannot make bytes uninitialized. Errors and panics skip `assume_init`; a
successful return is valid under the `FromBytes` unsafe-trait contract.

There were no hard errors for the admitted memory-initialization atom.

## Robustness observation

The skill-enabled reports additionally made explicit that caller-provided safe
`Read` implementations cannot be trusted to obey their behavioral prose. A
safe override of `read_exact` may return success without filling the buffer.
That does not affect the fixed implementation's soundness because untouched
bytes remain initialized. Whether the brief phrase “Reads a copy … from the
source” is precise enough to make this a documented-postcondition violation
is interpretation-dependent; the pilot records this as an unresolved
robustness/documentation question rather than an admitted
`CONTRACT-BROKEN` atom.

## Interpretation

This public historical case was likely represented in model training data, and
the source contains general padding warnings even after incident-specific
collateral was removed. Equal one-replicate recovery therefore demonstrates
basic reasoning compatibility, not skill lift or memorization resistance.
Loading
Loading