-
Notifications
You must be signed in to change notification settings - Fork 170
Add unsafe Rust skill V1 and preserve predecessor pilot #3520
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
joshlf
wants to merge
1
commit into
main
Choose a base branch
from
G6f4nmapfoz5ur4camwh2yhyazexc2lct
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Exploratory Pilot Fixtures | ||
|
|
||
| > **Evaluator-only material.** Do not expose this file, its directory names, or | ||
| > both members of a pair to an evaluated agent. | ||
|
|
||
| These source-only fixtures were frozen for the 2026-07-30 exploratory | ||
| evaluation. They are deliberately small enough to review without compiling or | ||
| executing target code. | ||
|
|
||
| The pilot uses a common support policy: | ||
|
|
||
| - Rust 1.85.0; | ||
| - every ordinary build profile; | ||
| - `debug_assertions` enabled and disabled; | ||
| - every public item is supported API; and | ||
| - no additional deployment restriction. | ||
|
|
||
| ## Synthetic vulnerable oracle | ||
|
|
||
| The vulnerable member contains six independently scored atoms: | ||
|
|
||
| | ID | Required conclusion | | ||
| |---|---| | ||
| | M1 | The entirely safe call `decode_flag(2)` creates an invalid `bool`; `UNSOUND`. | | ||
| | M2 | A caller-provided safe `AddressSource` implementation may return an unreadable pointer consumed by `load_source`; `UNSOUND`. | | ||
| | M3 | Safe construction through the public `ByteHandle::address` field does not establish dereferenceability; `UNSOUND`. | | ||
| | M4 | A contract-satisfying call to `item_unchecked(&[0x10, 0x20], 1)` returns the wrong element without itself reaching UB; `CONTRACT-BROKEN`. | | ||
| | M5 | With debug assertions disabled, the macro-generated public safe function permits an unchecked out-of-bounds access; `UNSOUND`. | | ||
| | M6 | `checked_first` is sound due to the preceding emptiness check, but its stated safety rationale is false; proof-comment defect without condemning the implementation. | | ||
|
|
||
| Recovery requires the correct surface, violated or missing proposition, and a | ||
| defensible classification. M1–M4 and M6 have configuration closure | ||
| preregistered as not applicable to the atom-specific score; all seven common | ||
| dimensions apply to M5. | ||
|
|
||
| The fixed member removes each of these six semantic defects. It deliberately | ||
| retains an unsafe block in `item_unchecked` without a proof-grade adjacent | ||
| comment, so it is a bug-specific fixed control rather than a proof-complete | ||
| whole-crate control. An agent may correctly report that proof-artifact defect; | ||
| it must not reproduce any repaired finding. | ||
|
|
||
| ## Admission and interpretation | ||
|
|
||
| The six atoms were specified before evaluated-agent reports were observed. | ||
| One independent source reviewer and the evaluator agreed on the atoms before | ||
| the first result was scored. The pilot is nevertheless not a release | ||
| evaluation: it has one replicate per cell and lacks the plan's hardened | ||
| filesystem, documentation, network, package, and paired-side isolation. | ||
|
|
||
| Historical zerocopy and current-zerocopy pilot targets remain Candidate or | ||
| Challenge fixtures. Their results are descriptive until their authority-rooted | ||
| oracles complete the two-reviewer admission process. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| [package] | ||
| name = "opaque-byte-tools" | ||
| version = "0.1.0" | ||
| edition = "2021" | ||
| rust-version = "1.85" | ||
|
|
||
| [lib] | ||
| path = "src/lib.rs" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # Support policy | ||
|
|
||
| This crate supports Rust 1.85.0. | ||
|
|
||
| All ordinary build profiles are supported. Debug assertions may be enabled or | ||
| disabled. Every public item is supported API. There are no additional | ||
| deployment restrictions. |
59 changes: 59 additions & 0 deletions
59
evals/unsafe-rust/fixtures/pilot/synthetic-fixed/src/lib.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| #![deny(unsafe_op_in_unsafe_fn)] | ||
|
|
||
| pub fn decode_flag(raw: u8) -> bool { | ||
| match raw { | ||
| 0 => false, | ||
| 1 => true, | ||
| _ => panic!("invalid flag byte"), | ||
| } | ||
| } | ||
|
|
||
| pub trait AddressSource { | ||
| fn byte(&self) -> &u8; | ||
| } | ||
|
|
||
| pub fn load_source<S: AddressSource>(source: &S) -> u8 { | ||
| *source.byte() | ||
| } | ||
|
|
||
| pub struct ByteHandle<'a> { | ||
| address: &'a u8, | ||
| } | ||
|
|
||
| impl<'a> ByteHandle<'a> { | ||
| pub fn new(address: &'a u8) -> Self { | ||
| Self { address } | ||
| } | ||
|
|
||
| pub fn load(&self) -> u8 { | ||
| *self.address | ||
| } | ||
| } | ||
|
|
||
| /// Returns `bytes[index]`. | ||
| /// | ||
| /// # Safety | ||
| /// | ||
| /// `index` must be less than `bytes.len()`. | ||
| pub unsafe fn item_unchecked(bytes: &[u8], index: usize) -> u8 { | ||
| unsafe { *bytes.get_unchecked(index) } | ||
| } | ||
|
|
||
| macro_rules! make_indexer { | ||
| ($name:ident) => { | ||
| pub fn $name(bytes: &[u8], index: usize) -> u8 { | ||
| bytes[index] | ||
| } | ||
| }; | ||
| } | ||
|
|
||
| make_indexer!(profile_index); | ||
|
|
||
| pub fn checked_first(bytes: &[u8]) -> Option<u8> { | ||
| if bytes.is_empty() { | ||
| return None; | ||
| } | ||
| // SAFETY: The preceding emptiness check establishes that index 0 is | ||
| // in-bounds. | ||
| Some(unsafe { *bytes.get_unchecked(0) }) | ||
| } |
8 changes: 8 additions & 0 deletions
8
evals/unsafe-rust/fixtures/pilot/synthetic-vulnerable/Cargo.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| [package] | ||
| name = "opaque-byte-tools" | ||
| version = "0.1.0" | ||
| edition = "2021" | ||
| rust-version = "1.85" | ||
|
|
||
| [lib] | ||
| path = "src/lib.rs" |
7 changes: 7 additions & 0 deletions
7
evals/unsafe-rust/fixtures/pilot/synthetic-vulnerable/SUPPORT.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # Support policy | ||
|
|
||
| This crate supports Rust 1.85.0. | ||
|
|
||
| All ordinary build profiles are supported. Debug assertions may be enabled or | ||
| disabled. Every public item is supported API. There are no additional | ||
| deployment restrictions. |
53 changes: 53 additions & 0 deletions
53
evals/unsafe-rust/fixtures/pilot/synthetic-vulnerable/src/lib.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| #![deny(unsafe_op_in_unsafe_fn)] | ||
|
|
||
| use core::{mem, ptr::NonNull}; | ||
|
|
||
| pub fn decode_flag(raw: u8) -> bool { | ||
| unsafe { mem::transmute(raw) } | ||
| } | ||
|
|
||
| pub trait AddressSource { | ||
| fn address(&self) -> *const u8; | ||
| } | ||
|
|
||
| pub fn load_source<S: AddressSource>(source: &S) -> u8 { | ||
| unsafe { source.address().read() } | ||
| } | ||
|
|
||
| pub struct ByteHandle { | ||
| pub address: NonNull<u8>, | ||
| } | ||
|
|
||
| impl ByteHandle { | ||
| pub fn load(&self) -> u8 { | ||
| unsafe { self.address.as_ptr().read() } | ||
| } | ||
| } | ||
|
|
||
| /// Returns `bytes[index]`. | ||
| /// | ||
| /// # Safety | ||
| /// | ||
| /// `index` must be less than `bytes.len()`. | ||
| pub unsafe fn item_unchecked(bytes: &[u8], index: usize) -> u8 { | ||
| unsafe { *bytes.get_unchecked(0) } | ||
| } | ||
|
|
||
| macro_rules! make_indexer { | ||
| ($name:ident) => { | ||
| pub fn $name(bytes: &[u8], index: usize) -> u8 { | ||
| debug_assert!(index < bytes.len()); | ||
| unsafe { *bytes.get_unchecked(index) } | ||
| } | ||
| }; | ||
| } | ||
|
|
||
| make_indexer!(profile_index); | ||
|
|
||
| pub fn checked_first(bytes: &[u8]) -> Option<u8> { | ||
| if bytes.is_empty() { | ||
| return None; | ||
| } | ||
| // SAFETY: Since `u8` occupies one byte, every `[u8]` contains an element. | ||
| Some(unsafe { *bytes.get_unchecked(0) }) | ||
| } |
89 changes: 89 additions & 0 deletions
89
evals/unsafe-rust/runs/2026-07-30-exploratory-pilot/current-result.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Current Zerocopy `impls.rs` Challenge | ||
|
|
||
| Snapshot: `53a3fbfa15d656b25b74688369f7248ff354a021`. | ||
|
|
||
| This challenge has no whole-target positive oracle and receives no aggregate | ||
| semantic score. Novel claims were independently source-reviewed; they are not | ||
| converted into confirmed production unsoundness merely because an evaluated | ||
| agent reported them. | ||
|
|
||
| ## Paired result | ||
|
|
||
| | Behavior | Skill | Baseline | | ||
| |---|---|---| | ||
| | Overall production verdict | `UNPROVED`; no concrete valid-use UB or false postcondition established | `UNPROVED`; no concrete downstream production UB established | | ||
| | SIMD normative proof gap | Found | Found | | ||
| | Historical-version `Option` zero-representation gap | Missed/claimed locally closed | Found | | ||
| | `ManuallyDrop<T>: HasField` exact-contract concern | Missed | Found | | ||
| | Incomplete `Immutable` proof for `Box<T>` | Found | Found as a documentation residual | | ||
| | Optional function-pointer/`NonNull` `Immutable` proof | Called unproved | Called documentation residual | | ||
| | Missing fixture/configuration inputs | Found | Not made explicit | | ||
| | Two `assume_initialized` sites | Correctly scoped test-only and `UNPROVED`, not `UNSOUND` | Correctly scoped test-only and unjustified, not downstream production | | ||
|
|
||
| Both agents resisted the tempting but unjustified conclusion that an explicit | ||
| “this is unsound” FIXME in generic test machinery proves a concrete bad | ||
| execution or a downstream-shipping defect. | ||
|
|
||
| ## Independent adjudication | ||
|
|
||
| ### Confirmed proof gaps | ||
|
|
||
| - **Option zero representation: `UNPROVED` over the declared Rust 1.56+ | ||
| range.** The source cites a Rust 1.89 guarantee. Independent version review | ||
| found that the explicit all-zero-to-`None` guarantee appears later than the | ||
| declared MSRV for several families, and the explicit unsafe-function-pointer | ||
| coverage later still. No compiler counterexample was established, so this | ||
| is missing authoritative coverage, not demonstrated unsoundness. | ||
| - **Aggregate SIMD matrix: `UNPROVED`.** The generic argument relies on | ||
| nonnormative UCG text that disclaims being a guarantee. Some current | ||
| per-type standard-library pages may close individual types, but no reviewed | ||
| proof covers every emitted type, architecture, feature, nightly, and | ||
| supported compiler version. | ||
| - **`Box<T>: Immutable`: `UNPROVED`.** No reviewed normative contract | ||
| establishes the representation property required by zerocopy's trait over | ||
| the entire supported compiler range. No counterexample was found. | ||
|
|
||
| ### Claims narrowed or disputed | ||
|
|
||
| - A reviewer derived the optional function-pointer and `NonNull<T>` | ||
| `Immutable` obligations from normative `Copy` restrictions and | ||
| `UnsafeCell` rules and classified them `PROVED`. Thus the skill report's | ||
| `UNPROVED` classification for these two families is conservative | ||
| over-reporting, not an admitted defect. | ||
| - A reviewer classified `ManuallyDrop<T>: HasField` as | ||
| `CONTRACT-BROKEN`: the local contract asks for exact field identity, type, | ||
| and visibility, while public std documentation exposes only private fields | ||
| and the implementation uses a public proxy marker. No invalid projection, | ||
| memory unsafety, or provenance failure was demonstrated because std does | ||
| guarantee `ManuallyDrop<T>` has `T`'s layout and bit validity. | ||
|
|
||
| This semantic classification still needs project-author review. The local | ||
| contract permits `Self` merely to share the layout of a type containing the | ||
| field, so a layout-equivalent public proxy-field interpretation may be | ||
| intended. The pilot therefore records the claim as a high-priority contract | ||
| ambiguity, not a final production defect. | ||
| - The two test-only `assume_initialized` comments give an invalid generic | ||
| implication: `TryFromBytes` does not imply initialized padding. A second | ||
| review found that the finite current test invocations nevertheless construct | ||
| byte-initialized values. The admitted conclusion is proof-comment debt in | ||
| test-only code; no concrete bad execution or production issue was proved. | ||
|
|
||
| ## Fixture failure discovered by the skill | ||
|
|
||
| The current blind bundle included `Cargo.toml`, `build.rs`, and `src/`, but | ||
| omitted the lockfile, path-dependency sources, linked MSRV policy, and pinned | ||
| nightly/configuration material needed for a complete theorem. The skill agent | ||
| correctly refused closure for those axes. | ||
|
|
||
| This is an evaluation-fixture defect, not evidence against zerocopy. A future | ||
| scored current-source run must rebuild the bundle with all semantically | ||
| required policy, dependency, generated, and configuration inputs and then | ||
| restart both conditions from fresh agents. | ||
|
|
||
| ## Interpretation | ||
|
|
||
| The baseline found two important obligations that the skill report omitted, | ||
| while the skill was more explicit about theorem status, configuration | ||
| incompleteness, and exact residual scope. This mixed result is useful: the | ||
| skill improved reporting discipline but did not dominate unaided defect/proof | ||
| gap discovery on the current high-quality target. |
68 changes: 68 additions & 0 deletions
68
evals/unsafe-rust/runs/2026-07-30-exploratory-pilot/historical-result.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # Historical `read_from_io` Pair | ||
|
|
||
| ## Admitted atom | ||
|
|
||
| Two independent source reviewers reconciled the following | ||
| authority-rooted atom before the reports were assigned an objective score: | ||
|
|
||
| > Immediately before `Ptr::as_bytes`, the referent must satisfy zerocopy's | ||
| > local `invariant::Initialized`: all `size_of::<Self>()` bytes must form a | ||
| > bit-valid byte array. In the vulnerable snapshot, | ||
| > `MaybeUninit::<Self>::zeroed()` does not establish initialization of padding | ||
| > after its return-by-value typed move, so the subsequent | ||
| > `assume_validity::<Initialized>` lacks a premise. The resulting byte slice is | ||
| > passed to caller-provided safe `Read` code, which may inspect it. The fixed | ||
| > snapshot instead performs an in-place bytewise zero of the final storage | ||
| > before asserting `Initialized`, which closes this exact missing premise. | ||
|
|
||
| The reviewers checked the local `Initialized`, `assume_validity`, `as_bytes`, | ||
| `FromBytes`, and `FromZeros::zero` contracts and Rust 1.84.1 documentation for | ||
| `MaybeUninit::zeroed`, `MaybeUninit::assume_init`, `ptr::write_bytes`, | ||
| `Read`, and invalid values. | ||
|
|
||
| The fixed classification is bug-specific. It is not a whole-crate soundness | ||
| label. | ||
|
|
||
| ## Results | ||
|
|
||
| | Snapshot | Skill | Baseline | | ||
| |---|---|---| | ||
| | Vulnerable `49a13ba…` | Recovered; `14/14` | Recovered; `14/14` | | ||
| | Fixed `f99854a…` | Exact atom closed; no repaired-defect assertion | Exact atom closed; no repaired-defect assertion | | ||
|
|
||
| Both vulnerable reports: | ||
|
|
||
| - located `MaybeUninit::<Self>::zeroed()` and the false `Initialized` | ||
| transition; | ||
| - traced the transition through `Ptr::as_bytes` and reference construction; | ||
| - supplied a fully safe generic instantiation and adversarial-safe reader | ||
| path; | ||
| - used exact Rust 1.84.1 authorities; | ||
| - covered the requested `std`/`x86_64-unknown-linux-gnu` configuration; and | ||
| - classified the safe API as unsound. | ||
|
|
||
| Both fixed reports proved that `uninit(); buf.zero()` writes the whole final | ||
| object representation in place and that arbitrary memory-safe `Read` behavior | ||
| cannot make bytes uninitialized. Errors and panics skip `assume_init`; a | ||
| successful return is valid under the `FromBytes` unsafe-trait contract. | ||
|
|
||
| There were no hard errors for the admitted memory-initialization atom. | ||
|
|
||
| ## Robustness observation | ||
|
|
||
| The skill-enabled reports additionally made explicit that caller-provided safe | ||
| `Read` implementations cannot be trusted to obey their behavioral prose. A | ||
| safe override of `read_exact` may return success without filling the buffer. | ||
| That does not affect the fixed implementation's soundness because untouched | ||
| bytes remain initialized. Whether the brief phrase “Reads a copy … from the | ||
| source” is precise enough to make this a documented-postcondition violation | ||
| is interpretation-dependent; the pilot records this as an unresolved | ||
| robustness/documentation question rather than an admitted | ||
| `CONTRACT-BROKEN` atom. | ||
|
|
||
| ## Interpretation | ||
|
|
||
| This public historical case was likely represented in model training data, and | ||
| the source contains general padding warnings even after incident-specific | ||
| collateral was removed. Equal one-replicate recovery therefore demonstrates | ||
| basic reasoning compatibility, not skill lift or memorization resistance. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These lines still classify the historical target as a Candidate awaiting two-reviewer admission, while
manifest.mdlines 103–105 says the atom was promoted after the second review andhistorical-result.mdpresents it as an admitted, objectively scored14/14atom. This leaves readers unable to tell whether the historical score is admissible evidence; update this status to match the manifest/result, or downgrade those documents if admission was not actually completed.Useful? React with 👍 / 👎.