diff --git a/CHANGELOG.md b/CHANGELOG.md index 484bfaf..0efbd68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,13 +5,13 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.1.2] — 2026-06-30 +## [0.1.2] - 2026-06-30 ### Clarified -- **`family_id` scope boundary** (`python/bundle_spec/taxonomy.py` docstring + `README.md`): documented that `family_id` is the **validator / node-software** vulnerability-class vocabulary, and that on-chain contract / DeFi-economic findings (AMM value-extraction, oracle/TWAP manipulation, liquidation, governance capture, bridge-contract message abuse, ERC-4626/reentrancy logic) are **out of scope** — a separate DeFi-economic namespace, never `family_id` values, cross-mapped in the VII `ledger/family_map.csv`. Records the canonical-vocabulary ratification (the 11-value `FamilyId` enum, unchanged). **No enum/wire change** — `BUNDLE_VERSION` unchanged; doc-only patch. +- **`family_id` scope boundary** (`python/bundle_spec/taxonomy.py` docstring + `README.md`): documented that `family_id` is the **validator / node-software** vulnerability-class vocabulary, and that on-chain contract / DeFi-economic findings (AMM value-extraction, oracle/TWAP manipulation, liquidation, governance capture, bridge-contract message abuse, ERC-4626/reentrancy logic) are **out of scope**: a separate DeFi-economic namespace, never `family_id` values, cross-mapped in the VII `ledger/family_map.csv`. Records the canonical-vocabulary ratification (the 11-value `FamilyId` enum, unchanged). **No enum/wire change**: `BUNDLE_VERSION` unchanged; doc-only patch. -## [0.1.1] — 2026-05-13 +## [0.1.1] - 2026-05-13 ### Clarified @@ -21,7 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`tools/validate_bundle.py`**: release-cert validator CLI that walks bundle directories + asserts each manifest validates against `BundleManifest` + asserts flag-honest file presence per the forward-only `BundleFiles` semantic. Per-bundle release-cert JSON output when `--cert-out` supplied. Smoke-tested against the 5 reference example bundles (5/5 pass). -## [0.1.0] — 2026-05-06 +## [0.1.0] - 2026-05-06 ### Added @@ -39,11 +39,11 @@ Initial release of Bundle v1. `arrow_schema::Schema` constants), same `compute_genome_id` output (verified by shared test vectors). JSON Schema embedded via `include_str!()`. -- **JSON Schema artefacts** in `schema/` — generated from the +- **JSON Schema artefacts** in `schema/`, generated from the Pydantic models via `tools/regen_schema.py`. Stable formatter (sorted keys, 2-space indent, trailing newline). CI verifies no drift between committed schema and Pydantic source. -- **Five reference example bundles** in `examples/` — +- **Five reference example bundles** in `examples/`: `sui_F10_multi_get_objects_amp` (attack), `sui_F14_devinspect_tokio_wedge` (attack), `sui_BENIGN_reproducer_pipeline` (benign), @@ -51,9 +51,9 @@ Initial release of Bundle v1. `solana_BENIGN_organic_rpc` (benign). All `lab-tls-fronted` fidelity, `self-owned` authorisation, with post-term cleartext pcap stripped per the public-release audit. -- **`tools/regen_schema.py`** — regenerates JSON Schema from the +- **`tools/regen_schema.py`**: regenerates JSON Schema from the canonical Pydantic models. -- **`tools/strip_pcap.py`** — drops post-term cleartext pcap from +- **`tools/strip_pcap.py`**: drops post-term cleartext pcap from a bundle and re-emits the manifest with `packets_pcap=False`, for safe public release of lab-tls-fronted bundles. @@ -61,11 +61,11 @@ Initial release of Bundle v1. - **0.1.x**: bug-fix-only patches on the current schema. - **0.2.0**: reserved for the upcoming `provenance.substrate` × - `provenance.traffic_origin` decomposition (additive — existing + `provenance.traffic_origin` decomposition (additive: existing `provenance.fidelity_class` becomes derived). Tracked as the *next test of additive extensibility* in upstream methodology documentation. -- **1.0.0**: deferred to first set of external citations — not +- **1.0.0**: deferred to first set of external citations, not calendar-bound. ### Cross-language guarantees pinned by tests diff --git a/README.md b/README.md index f3f9849..a644f41 100644 --- a/README.md +++ b/README.md @@ -15,23 +15,30 @@ This repository contains the **canonical normative specification** ``` / -├── manifest.json — BundleManifest (Pydantic-validated) -├── packets.pcap — packet capture, snaplen-256 by default -├── host.parquet — host telemetry time-series -├── app.parquet — application metric scrape time-series -├── protocol.parquet — consensus / protocol signal time-series -└── responses.parquet — per-request response semantics +├── manifest.json - BundleManifest (Pydantic-validated) +├── packets.pcap - packet capture, snaplen-256 by default +├── host.parquet - host telemetry time-series +├── app.parquet - application metric scrape time-series +├── protocol.parquet - consensus / protocol signal time-series +└── responses.parquet - per-request response semantics ``` License: **MIT**. The format is intended to be adopted on third-party -data without coordination — a researcher can produce bundles against +data without coordination. A researcher can produce bundles against their own infrastructure, train against them, and publish results citing the format. Adoption is the point. +## Adopting the standard + +- [Adoption guide](docs/adoption-guide.md): how to classify a validator-integrity + finding into the eleven-family vocabulary. +- [Methodology](docs/methodology.md): how the Validator Integrity Index is built, + and what it does and does not yet claim. + ## Why a format? Adversarial research on blockchain infrastructure has historically -been one-off — each project builds its own capture pipeline, its own +been one-off. Each project builds its own capture pipeline, its own labelling conventions, its own evaluation methodology. That makes results incomparable. @@ -105,8 +112,8 @@ Identity, taxonomy, time window, attack parameters, provenance. ``` The `family_id` is a **chain-agnostic** vulnerability-class label -(11 values: 10 attack families + benign) — **validator / node-software -classes only**; on-chain contract / DeFi-economic findings are a +(11 values: 10 attack families + benign), covering **validator / node-software +classes only**. On-chain contract / DeFi-economic findings are a separate out-of-scope namespace (cross-mapped in the VII `ledger/family_map.csv`). The `primitive_id` is the **chain-specific** implementation. This two-level taxonomy is the @@ -114,7 +121,7 @@ load-bearing decomposition for cross-chain holdout. ### packets.pcap -Raw packet capture. Default snaplen 256 bytes (TCP/IP headers only — +Raw packet capture. Default snaplen 256 bytes (TCP/IP headers only, so no payload bytes leak). Producers can use larger snaplens when they control the privacy posture. @@ -134,7 +141,7 @@ All four files share a monotonic `t_ns` column (nanoseconds since Schemas are stable; new columns require a `BUNDLE_VERSION` bump. -## Quick start — Python +## Quick start: Python ```bash pip install bundle-spec @@ -160,7 +167,7 @@ table = pq.read_table("crp_abc123/responses.parquet") assert table.schema.equals(responses_schema()) # or a superset, additive ``` -## Quick start — Rust +## Quick start: Rust ```toml [dependencies] @@ -180,7 +187,7 @@ println!("{} {} {:?}", m.family_id, m.primitive_id, m.ground_truth_label); See `python/bundle_spec/bundle_v1.py` (Pydantic models with docstrings) and `schema/bundle_v1.schema.json` (generated). Both are -the same content in different forms — the Pydantic model is +the same content in different forms. The Pydantic model is canonical. Required structured-enum fields: @@ -213,7 +220,7 @@ must be byte-for-byte equivalent. CI runs `tools/compare_arrow_schemas.py` to check. The `genome_id` canonicalisation (sorted-key JSON, SHA-256, first 16 -hex chars) is also pinned by a cross-language determinism test — a +hex chars) is also pinned by a cross-language determinism test: a fixed input must produce the same hex string in both languages. ## Examples @@ -247,12 +254,12 @@ re-typing, removing) require a major bump and a new module path SemVer for `bundle-spec` (this package): -- **0.1.x** — bug-fix-only patches on the current schema. -- **0.2.0** — reserved for the upcoming `provenance.substrate` × +- **0.1.x**: bug-fix-only patches on the current schema. +- **0.2.0**: reserved for the upcoming `provenance.substrate` × `provenance.traffic_origin` decomposition (additive: existing `provenance.fidelity_class` will be derived from the new pair via a model validator; existing readers stay compatible). -- **1.0.0** — deferred to first set of external citations. +- **1.0.0**: deferred to first set of external citations. ## Reproducibility @@ -277,6 +284,6 @@ network dependency. ## License -MIT — see `LICENSE`. Code, schemas, and example metadata are MIT- +MIT. See `LICENSE`. Code, schemas, and example metadata are MIT- licensed and free to use without restriction. The example bundle **data** (Parquet files, packet captures) is also MIT-licensed. diff --git a/docs/adoption-guide.md b/docs/adoption-guide.md new file mode 100644 index 0000000..1f43de9 --- /dev/null +++ b/docs/adoption-guide.md @@ -0,0 +1,74 @@ +# Adopting the standard: classify a validator-integrity finding + +The Bundle v1 format and its family vocabulary are published so that adversarial research on validator +infrastructure becomes comparable: one on-disk shape, one labelling scheme, one evaluation methodology, +adoptable on your own data without coordination. This guide shows how to classify a node or client +integrity finding into the vocabulary. It is deliberately short. + +## The eleven families + +| family | what it is | how to tell it apart | +|---|---|---| +| `response_amp` | a small request that returns disproportionately large response bytes | the cost lands on the server's outbound bandwidth | +| `compute_amp` | a small request that triggers disproportionate server-side CPU | parser pathologies, verification fallbacks, sync-over-async wedges | +| `memory_amp` | unbounded accumulation of retained state in process memory | it grows over time, not per request | +| `connection_exhaustion` | a per-connection or per-subscription resource that is never released on disconnect | the leak is the open handle, not the payload | +| `consensus_abuse` | validator-authenticated misbehaviour inside consensus | it presupposes validator key material | +| `gossip_abuse` | unauthenticated peer-to-peer or gossip abuse via public peer endpoints | state-sync floods, request amplification | +| `auth_bypass` | missing or broken authentication on a control, admin, or first-party API | the surface should have required credentials | +| `rate_limiter_bypass` | a defect in the rate-limiter's own logic | fail-open, burst-doubling, or key-collision, not a response-size issue | +| `service_misconfig` | exploitation of a misconfigured co-located daemon | Redis, Grafana, Prometheus, SSH, and similar | +| `reconnaissance` | discovery and target-mapping traffic itself | it is the prior stage, not the exploit | +| `benign` | no attack shape | it trains the "not malicious" boundary | + +## How to classify a finding + +Work top to bottom. The first rule that matches gives the family. + +1. If it is only discovery or mapping traffic, with no exploitation, it is `reconnaissance`. +2. If it requires validator key material to misbehave inside consensus, it is `consensus_abuse`. +3. If it is unauthenticated abuse of a public peer-to-peer or gossip endpoint, it is `gossip_abuse`. +4. If it defeats authentication on a surface that should need credentials, it is `auth_bypass`. +5. If the bug is in the rate-limiter's own logic, it is `rate_limiter_bypass`. +6. If it is a misconfigured co-located service being exploited, it is `service_misconfig`. +7. Otherwise it is a resource-amplification shape. Pick by the resource that dominates: large response + bytes from a small request is `response_amp`; large CPU from a small request is `compute_amp`; + unbounded retained memory over time is `memory_amp`; a per-connection or per-subscription leak is + `connection_exhaustion`. +8. If there is no attack shape, it is `benign`. + +A finding can touch more than one mechanism. Label it by the dominant one, meaning the mechanism the +mitigation lives in. A response-size cap fixes `response_amp`; a fix in the limiter logic is +`rate_limiter_bypass`. + +## A worked example, using a public NullRabbit advisory + +Take advisory NR-2026-001 (Solana). An unauthenticated `getMultipleAccounts` JSON-RPC call returns far +more response bytes than the request costs to send, so one caller can drive disproportionate server +egress. + +Walk the rules in order. Is it discovery only? No. Does it need consensus key material? No. Is it +peer-to-peer or gossip? No, it is the public JSON-RPC surface. Does it defeat authentication? No. Is it a +rate-limiter logic bug? No. Is it a co-located service? No. It is a resource-amplification shape, and the +dominant cost is response bytes, so it is `response_amp`. + +For contrast, the same advisory's `simulateTransaction` finding drives disproportionate CPU rather than +bytes, so it is `compute_amp`. Same endpoint family, different family label, because the dominant resource +differs. That is the discipline the vocabulary is meant to enforce. + +## Producing bundles + +To contribute measurements, capture each run in the Bundle v1 shape: a manifest, a packet capture, and the +four time-series modalities (host, app, protocol, responses), as specified in this repository. The public +sample dataset provides ground-truth examples to validate your pipeline against. + +## Citing and contributing + +Cite the format using the `CITATION.cff` file in this repository. The format is MIT-licensed, so you can +adopt it on your own infrastructure and publish results citing it. To propose a new family or refine a +definition, open an issue. The vocabulary is versioned and evolves by proposal, not by decree. + +## Scope + +This vocabulary is for node and client software findings only. On-chain contract, protocol, and +DeFi-economic findings use a separate vocabulary and are out of scope here. diff --git a/docs/methodology.md b/docs/methodology.md new file mode 100644 index 0000000..46ab130 --- /dev/null +++ b/docs/methodology.md @@ -0,0 +1,70 @@ +# How the Validator Integrity Index is built + +*A companion to the Bundle v1 format specification and the NullRabbit disclosure record.* + +NullRabbit Labs measures one thing: the probability that a given blockchain validator or node client +commits a slashable fault or is remotely exploitable. This note explains how the underlying finding +record is built, and just as importantly, what the index does and does not yet claim. A rating layer is +either trusted or it is nothing, so the method and its honest limits are the product. + +## 1. Canonicalise, don't count + +Adversarial research surfaces the same underlying weakness many times: across chains, across ports, and +across client implementations. Counting each surfacing as a separate finding inflates a number without +adding information. Instead, one written rule collapses repeat surfacings of the same root weakness into a +single canonical record, keyed to the specific client and component. The index therefore measures +distinct, real weaknesses rather than research volume. + +## 2. Provenance: ours, or a reproduction of someone else's public disclosure + +Every record is marked as one of two kinds. The first is original, meaning a weakness NullRabbit +discovered or first measured. The second is public-replication, meaning a faithful reproduction of an +already-public disclosure such as a CVE, a GHSA advisory, or a named third-party audit. The distinction is +explicit, so that a re-statement of public work is never mistaken for a novel finding, and so the line +between what is disclosure-gated and what is freely publishable is never in doubt. + +## 3. One vocabulary for every finding + +Every weakness is classified into one of eleven chain-agnostic families defined by the Bundle v1 format: +response amplification, compute amplification, memory amplification, connection exhaustion, consensus +abuse, gossip abuse, authentication bypass, rate-limiter bypass, service misconfiguration, reconnaissance, +and a benign class. The vocabulary is defined by mechanism, not by chain. A finding in a Solana RPC path +and a structurally identical finding in a Cosmos RPC path receive the same family label. That is what +makes cross-client comparison meaningful, and what lets a finding reported by an outside team drop into +the same scheme. The [adoption guide](adoption-guide.md) shows how to classify a finding. + +## 4. The disclosure record, including the rejections + +For every disclosed finding, NullRabbit records what was sent to the vendor and what the vendor actually +said back. That includes the disclosures that were ignored, dismissed, or declined, and the findings +NullRabbit itself later retracted. This record is published deliberately. Showing where vendors +disagreed, where they went silent, and where we were wrong is precisely what makes the rest of the record +credible. + +## 5. What the index does not claim + +Honesty about limits is part of the method, not a footnote. + +First, the index is procedural, not yet empirical. Today its credibility rests on a clean, documented +method and an honest disclosure record, not on a large body of vendor-confirmed, externally-credited +validations. Those accrue over time as disclosures are acknowledged. Until then, the per-client ratings +are not presented as externally proven. + +Second, there is no cross-chain generalisation claim. The research behind the index does not yet reliably +generalise a detector trained on one chain to an unseen chain. That limitation is reported plainly, and +in-distribution results are never presented as cross-chain ones. + +Third, self-graded severity is a liability, not an asset. Where a severity is our own assessment rather +than a vendor's, it is treated as provisional and calibrated against the disclosed evidence. + +## 6. Scope + +The index covers node and client software only: the daemon a validator runs, spanning consensus, RPC, +peer-to-peer, and co-located services. On-chain contract, protocol, and DeFi-economic findings are a +separate measurement with a separate vocabulary, and they are deliberately not folded in. + +## Where the standard lives + +The Bundle v1 format and the family vocabulary are published under the MIT license in this repository. The +format is designed to be adopted on third-party data without coordination. See the +[adoption guide](adoption-guide.md). diff --git a/examples/README.md b/examples/README.md index cb7a1dc..bc6f623 100644 --- a/examples/README.md +++ b/examples/README.md @@ -16,18 +16,18 @@ attack/benign × cross-chain × mechanism design space. ``` example_/ -├── manifest.json — BundleManifest (schema-validated) -├── pcap_pre_termination.pcap — pre-term TLS pcap (no cleartext leak) -├── host.parquet — host telemetry -├── app.parquet — Prometheus scrape (often empty for these +├── manifest.json - BundleManifest (schema-validated) +├── pcap_pre_termination.pcap - pre-term TLS pcap (no cleartext leak) +├── host.parquet - host telemetry +├── app.parquet - Prometheus scrape (often empty for these │ bundles; the scrape didn't run during │ the capture window) -├── protocol.parquet — chain-protocol signals -└── responses.parquet — per-request semantics +├── protocol.parquet - chain-protocol signals +└── responses.parquet - per-request semantics ``` `packets.pcap` (the post-term cleartext loopback) has been **stripped** -from each example — it would contain HTTP/JSON-RPC bodies, which we +from each example, since it would contain HTTP/JSON-RPC bodies, which we deliberately don't publish in this reference set. The manifest records this via `files.packets_pcap=False`. @@ -36,7 +36,7 @@ The pre-term TLS pcap is preserved. Per a full content audit contain only: - TLS handshake records (encrypted payload after EncryptedExtensions) - Loopback IP addresses (127.0.0.1, 127.0.0.2) -- ALPN values (h2 / http/1.1 only — no surprising entries) +- ALPN values (h2 / http/1.1 only, no surprising entries) No SNI (clients use IP-only target URLs), no plaintext JSON-RPC tokens, no customer/engagement metadata, no filesystem paths. @@ -45,11 +45,11 @@ tokens, no customer/engagement metadata, no filesystem paths. All five bundles share the same provenance posture: -- `target_authorisation`: `self-owned` — captured against operator- +- `target_authorisation`: `self-owned`, captured against operator- owned localnet validators. - `target_env`: `localnet-sui-multinode4` (Sui side) or `localnet-solana-test-validator-v2.2.16` (Solana side). -- `fidelity_class`: `lab-tls-fronted` — full reproducer producing +- `fidelity_class`: `lab-tls-fronted`, a full reproducer producing genuine traffic against a TLS-terminating nginx in front of a localnet validator, captured at the pre-term wire vantage. @@ -77,7 +77,7 @@ println!("{} {:?}", m.primitive_id, m.ground_truth_label); ## License -The example bundles are MIT-licensed alongside the spec — see the +The example bundles are MIT-licensed alongside the spec. See the top-level `LICENSE`. Use freely. The data was produced entirely on operator-owned localnet infrastructure with no third-party traffic involved. diff --git a/nr-bundles-public-dataset-card.md b/nr-bundles-public-dataset-card.md index e4baadc..a66f2fb 100644 --- a/nr-bundles-public-dataset-card.md +++ b/nr-bundles-public-dataset-card.md @@ -22,7 +22,7 @@ A curated, multi-modal dataset of blockchain validator infrastructure under atta ## What this dataset is -Most security datasets on the Hub are either application-layer (smart contract vulnerabilities, exploit code snippets) or generic (enterprise network traffic, malware signatures). The infrastructure layer where blockchain validators actually run — the RPC surface, the gossip layer, the consensus daemon, the indexer — has no shared, ML-trainable representation. This dataset is the first public sample of one. +Most security datasets on the Hub are either application-layer (smart contract vulnerabilities, exploit code snippets) or generic (enterprise network traffic, malware signatures). The infrastructure layer where blockchain validators actually run (the RPC surface, the gossip layer, the consensus daemon, the indexer) has no shared, ML-trainable representation. This dataset is the first public sample of one. Each entry is a **bundle**: a multi-modal observation of validator infrastructure under a single workload. Bundles capture network packets, host telemetry, application metrics, and observed responses simultaneously, under a chain-agnostic schema. The format is open and specified at [`nr-bundle-spec`](https://github.com/NullRabbitLabs/nr-bundle-spec) (MIT-licensed). Anyone can produce v1 bundles for any chain. @@ -43,7 +43,7 @@ Each bundle is a directory containing a manifest plus up to six modality files. | `responses.parquet` | RPC/API response metadata (no bodies) | Bundles where the validator served RPC traffic during the capture window (manifest `responses_parquet=True`); **absent or zero-rows for passive-workload benigns** (`sui_BENIGN_passive_fullnode`, `solana_BENIGN_validator_passive`) | | `vectors.parquet` | Computed feature vectors derived from the above | **Not populated in this release.** Reserved schema slot; consumers should derive feature vectors at inference time using the bundle-spec reference parser plus their own feature pipeline | -The manifest pins `primitive_id`, `family_id`, `traffic_source`, `fidelity_class`, `target_authorisation`, and provenance hashes for every file in the bundle. The schema is chain-agnostic — no field assumes a specific blockchain. The `BundleFiles` block (`packets_pcap`, `host_parquet`, `app_parquet`, `protocol_parquet`, `responses_parquet`, `vectors_parquet`) is the authoritative present/absent contract; readers should check the manifest before reading each modality file. +The manifest pins `primitive_id`, `family_id`, `traffic_source`, `fidelity_class`, `target_authorisation`, and provenance hashes for every file in the bundle. The schema is chain-agnostic: no field assumes a specific blockchain. The `BundleFiles` block (`packets_pcap`, `host_parquet`, `app_parquet`, `protocol_parquet`, `responses_parquet`, `vectors_parquet`) is the authoritative present/absent contract; readers should check the manifest before reading each modality file. Bundles in this release carry one of two `fidelity_class` values: `lab` (controlled reproducer environment, no TLS termination) and `lab-tls-fronted` (controlled reproducer with an nginx TLS-termination front; pre-term TLS pcap is retained, post-term cleartext pcap is dropped). All bundles have `target_authorisation: self-owned` (every validator targeted was operated by NullRabbit). @@ -51,9 +51,9 @@ Full schema: [`nr-bundle-spec` v0.1.0](https://github.com/NullRabbitLabs/nr-bund ## Family taxonomy -Bundles are classified by **family**, defined by attack mechanism rather than by chain. Because the format and the mechanism are shared across chains, cross-chain generalisation becomes **testable**: you can hold out an entire chain and measure whether a model trained on the others recovers a family. On this corpus it does **not** transfer yet — held-out-chain family macro-F1 is **0.17 (Sui held out) / 0.35 (Solana held out)**, against a 7-class random floor of ~0.14. Cross-chain transfer is the open question the shared format lets you pose and measure, not a result this dataset demonstrates; closing the gap needs more chains per family, not more features. +Bundles are classified by **family**, defined by attack mechanism rather than by chain. Because the format and the mechanism are shared across chains, cross-chain generalisation becomes **testable**: you can hold out an entire chain and measure whether a model trained on the others recovers a family. On this corpus it does **not** transfer yet: held-out-chain family macro-F1 is **0.17 (Sui held out) / 0.35 (Solana held out)**, against a 7-class random floor of ~0.14. Cross-chain transfer is the open question the shared format lets you pose and measure, not a result this dataset demonstrates; closing the gap needs more chains per family, not more features. -These are **validator / node-software** vulnerability classes only — on-chain contract / DeFi-economic findings are a separate, out-of-scope namespace and are never used here. The full taxonomy defines eleven families (ten attack classes plus `benign`). Seven are populated in this release: +These are **validator / node-software** vulnerability classes only. On-chain contract / DeFi-economic findings are a separate, out-of-scope namespace and are never used here. The full taxonomy defines eleven families (ten attack classes plus `benign`). Seven are populated in this release: | Family | Mechanism | |---|---| @@ -161,13 +161,13 @@ for bundle_dir in sorted(Path(local).glob("crp_*")): (bundle_dir / "manifest.json").read_text() ) print(manifest.primitive_id, manifest.family_id, manifest.fidelity_class) - # Each modality is a Parquet file — read with pyarrow / pandas / polars + # Each modality is a Parquet file, read with pyarrow / pandas / polars # as the modality demands. See nr-bundle-spec for per-modality schemas. # Check manifest.files. before reading; bundles do # not contain modalities that were not captured during the workload. ``` -`datasets.load_dataset("NullRabbit/nr-bundles-public")` does **not** work on this dataset — HF's auto-loader assumes a single tabular schema, which doesn't match the multi-modal bundle layout. Use `snapshot_download` + the bundle-spec parsers as shown above. +`datasets.load_dataset("NullRabbit/nr-bundles-public")` does **not** work on this dataset: HF's auto-loader assumes a single tabular schema, which doesn't match the multi-modal bundle layout. Use `snapshot_download` + the bundle-spec parsers as shown above. For schema, validation, and reference parsers (Python + Rust), see [`nr-bundle-spec`](https://github.com/NullRabbitLabs/nr-bundle-spec). A validator CLI is available at `tools/validate_bundle.py`. @@ -205,4 +205,4 @@ Related: Research enquiries: security@nullrabbit.ai -Format extensions, schema feedback, or bundles produced by external researchers against the open spec — open an issue at `nr-bundle-spec`. +Format extensions, schema feedback, or bundles produced by external researchers against the open spec, open an issue at `nr-bundle-spec`. diff --git a/rust/bundle_spec/README.md b/rust/bundle_spec/README.md index 6712bbf..678bda2 100644 --- a/rust/bundle_spec/README.md +++ b/rust/bundle_spec/README.md @@ -1,6 +1,6 @@ -# bundle-spec — Rust reference parser +# bundle-spec: Rust reference parser -Reference Rust parser for [Bundle v1](https://github.com/NullRabbitLabs/nr-bundle-spec) — +Reference Rust parser for [Bundle v1](https://github.com/NullRabbitLabs/nr-bundle-spec), the canonical multi-modal capture format for adversarial blockchain validator research. @@ -29,16 +29,16 @@ println!("{} {:?}", manifest.primitive_id, manifest.family_id); ## What's in the crate -- `bundle_spec::manifest` — `BundleManifest`, `Provenance`, +- `bundle_spec::manifest`: `BundleManifest`, `Provenance`, `BundleFiles`, plus the `Posture` / `GroundTruthLabel` / `TrafficSource` / `FidelityClass` / `TargetAuthorisation` enums. -- `bundle_spec::primitive` — `PrimitiveDescriptor`, `ParameterSpec`. -- `bundle_spec::taxonomy` — `FamilyId` enum + `family_definitions()`. -- `bundle_spec::parquet` — `Schema` constants for the four time- +- `bundle_spec::primitive`: `PrimitiveDescriptor`, `ParameterSpec`. +- `bundle_spec::taxonomy`: `FamilyId` enum + `family_definitions()`. +- `bundle_spec::parquet`: `Schema` constants for the four time- series modalities + the optional `vectors` slot. -- `bundle_spec::genome_id` — `compute_genome_id` helper, byte- +- `bundle_spec::genome_id`: `compute_genome_id` helper, byte- identical with the Python reference parser (pinned by tests). -- `bundle_spec::validation` — `validate_manifest_json` / +- `bundle_spec::validation`: `validate_manifest_json` / `validate_primitive_json` for structural JSON Schema validation against the embedded schema artefacts. @@ -58,4 +58,4 @@ on the load-bearing surfaces: ## License -MIT — see the repo's top-level `LICENSE`. +MIT. See the repo's top-level `LICENSE`.