v0.2.0: adversarial hardening — uniform rejection contract across all 7 lineages

.github/workflows/verify.yml

@@ -20,35 +20,35 @@ jobs:
       matrix:
         lineage: [c, cpp, rust, go, d, haskell, ocaml]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
 
       - name: Set up Go
         if: matrix.lineage == 'go'
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
         with:
           go-version: '1.22'
           cache-dependency-path: go/go.mod
 
       - name: Set up Rust
         if: matrix.lineage == 'rust'
-        uses: dtolnay/rust-toolchain@stable
+        uses: dtolnay/rust-toolchain@4be7066ada62dd38de10e7b70166bc74ed198c30 # stable (2026-07-15)
 
       - name: Set up D
         if: matrix.lineage == 'd'
-        uses: dlang-community/setup-dlang@v2
+        uses: dlang-community/setup-dlang@d7d85fcde7c4cd5f9a6618fce1bccc316e1e910b # v2
         with:
           compiler: dmd-latest
 
       - name: Set up Haskell
         if: matrix.lineage == 'haskell'
-        uses: haskell-actions/setup@v2
+        uses: haskell-actions/setup@cd0d9bdd65b20557f41bea4dbe43d0b5fbbfe553 # v2.11.0
         with:
           ghc-version: '9.6'
           cabal-version: 'latest'
 
       - name: Set up OCaml
         if: matrix.lineage == 'ocaml'
-        uses: ocaml/setup-ocaml@v3
+        uses: ocaml/setup-ocaml@15d660006c1d3110d77c34b7faa3bddefe8b82f0 # v3.7.0
         with:
           ocaml-compiler: '5.2'
 
@@ -109,7 +109,7 @@ jobs:
           test -x "${{ matrix.lineage }}/bin/baion_canon_hash"
           printf '%s' '{"b":1,"a":[1,2]}' | "${{ matrix.lineage }}/bin/baion_canon_hash"
 
-      - uses: actions/upload-artifact@v4
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: cli-${{ matrix.lineage }}
           path: ${{ matrix.lineage }}/bin/baion_canon_hash
@@ -120,9 +120,9 @@ jobs:
     needs: build
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
 
-      - uses: actions/download-artifact@v4
+      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
         with:
           pattern: cli-*
 
@@ -138,3 +138,8 @@ jobs:
 
       - name: Run cross-lineage verifier
         run: ./verify_all_lineages.sh
+
+      # Agreement-only sweep over generated danger-zone cases: any
+      # accept/reject split or hash split across the seven CLIs fails.
+      - name: Run differential probe
+        run: python3 conformance/differential_probe.py

.gitignore

@@ -9,3 +9,4 @@
 /*/.dub/
 *.o
 *.a
+conformance/__pycache__/

README.md

@@ -9,7 +9,9 @@ Seven independent implementations of the same canonicalization contract — C, C
 ./verify_all_lineages.sh
 ```
 
-`build_all.sh` builds each lineage with its native toolchain, runs its test suite, and places the CLI at `<lineage>/bin/baion_canon_hash` — the layout the verifier requires. `verify_all_lineages.sh` then pipes the same JSON documents into every CLI and diffs the hex, and additionally asserts that all seven **uniformly reject** inputs outside the supported domain (see below). All seven lineages must be present — a missing binary fails the run, and one byte of disagreement anywhere fails the run. Success prints `PASS: 7/7 lineages produced identical output`.
+**Prerequisites** (one toolchain per lineage): a C compiler + `make`; CMake ≥ 3.20 + a C++17 compiler; Rust (`cargo`); Go ≥ 1.22; D (`dmd` + `dub`); GHC ≥ 9.6 + `cabal` (aeson ≥ 2.2 is fetched by cabal); OCaml ≥ 5.x + `dune` with `yojson`, `digestif`, `alcotest` (via opam); `python3` for the conformance tooling. A successful run ends with a 7/7 PASS table from `build_all.sh` and `PASS: 7/7 lineages agree ...` from the verifier, exit 0.
+
+`build_all.sh` builds each lineage with its native toolchain, runs its test suite, and places the CLI at `<lineage>/bin/baion_canon_hash` — the layout the verifier requires. `verify_all_lineages.sh` then feeds every vector in the conformance corpus (`conformance/accept.jsonl` + `conformance/reject.jsonl`) to every CLI: accept vectors must hash to the corpus-pinned SHA-256 in all seven lineages, and reject vectors must be **uniformly refused** (see below). `conformance/differential_probe.py` additionally sweeps generated danger-zone cases (number bands, escape forms, document framing) and fails on any disagreement; both run in CI. All seven lineages must be present — a missing binary fails the run, and one byte of disagreement anywhere fails the run. `conformance/fuzz_agreement.py` is the third layer: a seeded randomized fuzzer (structured boundary-biased documents, byte mutations, raw garbage) that asserts seven-way agreement on every generated input.
 
 ## Why this exists
 
@@ -22,15 +24,22 @@ The interesting engineering is in the edge cases: key ordering, number formattin
 Cross-lineage byte-identity is enforced and tested for:
 
 - objects (member names must be **unique** — see below), arrays, strings (full UTF-8, including multi-byte and escaped control characters **except U+0000**), booleans, null
-- integers within the IEEE-754 exact range (±2⁵³)
-- floats whose canonical form is pinned by the conformance vectors (including integer-valued floats, which serialize without a trailing `.0` per RFC 8785 §3.2.2.3 — `1.0` canonicalizes to `1` in every lineage, enforced by the verifier)
+- integers in the closed interval [−2⁵³, +2⁵³] (i.e. |n| ≤ 9007199254740992). Every integer in this interval is exactly representable in IEEE-754 binary64. Note this is one wider than JavaScript's `MAX_SAFE_INTEGER` (2⁵³ − 1): ±2⁵³ itself is admitted because it is exact and unambiguous *within this domain* — the neighboring value 2⁵³ + 1 (the first integer that would silently round to it) is rejected, so no two accepted integer tokens can collide
+- floats in the **plain-decimal domain**: zero, or magnitude in [10⁻⁶, 10²¹), written without exponent notation; the canonical form is ECMAScript `ToString` (shortest round-trip digits, plain decimal) per RFC 8785 §3.2.2.3 — including for values above 2⁵³ that enter via fraction tokens, where shortest-digits and the exact integer value differ (`65219416364867774.9377591` canonicalizes to `65219416364867780`, not `…776`) — `1.0` → `1`, `-0.0` → `0`, `0.1` → `0.1`, in every lineage, enforced by the pinned corpus
 
-**Uniformly rejected:** two input classes are refused with a nonzero exit by all seven CLIs, and the verifier asserts the rejection is uniform:
+**Uniformly rejected:** these input classes are refused with a nonzero exit by all seven CLIs, and the verifier asserts the rejection is uniform:
 
 - *Strings containing U+0000* (as the escape `\u0000` or a raw NUL byte). One lineage cannot represent embedded NUL losslessly, and accepting it anywhere would allow silent canonicalization collisions. A literal backslash followed by the text `u0000` (JSON `\\u0000`) is not a NUL and canonicalizes normally.
 - *Objects with duplicate member names*, at any nesting depth. RFC 8259 leaves duplicate-name behavior undefined, and the seven JSON ecosystems genuinely diverge (keep-first, keep-last, keep-both) — so object member names must be unique. Duplicates are detected on the *decoded* name: `{"a":1,"\u0061":2}` is rejected because `\u0061` decodes to `a`.
-
-**Known exclusions** (documented honestly because they are where seven ecosystems genuinely differ): number formatting outside the pinned vectors — very large integers beyond 2⁵³, negative zero, and scientific-notation thresholds — is not yet normalized across all seven lineages and must not be relied on. Non-finite numbers (NaN, ±Inf) are not valid JSON and are rejected or nulled per lineage test suites. If your data stays in the supported domain, the byte-identity guarantee holds; the conformance fixture is the authoritative definition of that domain.
+- *Numbers outside the plain-decimal domain*: exponent notation (`1e2` is rejected even though the value is in range — spell it `100`), integers beyond ±2⁵³, and fractions below 10⁻⁶ or at/above 10²¹. Seven number formatters genuinely disagree in exponent territory; the supported domain is exactly where byte-identity is provable, and inside it the output is normalized rather than excluded.
+- *Unpaired surrogate escapes* (a `\ud800`–`\udbff` escape not immediately followed by a low half, or a lone `\udc00`–`\udfff`): not Unicode scalar values, and ecosystems differ on replacement behavior. A literal backslash followed by surrogate text (`\\ud800`) is ordinary content.
+- *Anything other than exactly one JSON document*: a leading UTF-8 BOM, trailing non-whitespace, concatenated documents, trailing commas, or empty input.
+- *Invalid UTF-8*: stray continuation bytes, truncated sequences, overlong encodings, encoded surrogates, and code points above U+10FFFF. One ecosystem silently replaces invalid bytes with U+FFFD before hashing — accepting invalid UTF-8 anywhere would allow silent canonicalization collisions.
+- *Raw control characters* (U+0000–U+001F) inside string literals — RFC 8259 requires them escaped (`\t`, `\u001f`). Between tokens, only the four JSON whitespace bytes (tab, LF, CR, space) are accepted.
+- *Malformed escapes and tokens*: an escape other than the eight RFC 8259 escapes or `\u` + exactly 4 hex digits; number tokens with leading zeros (`0635`), a bare trailing dot (`0.`), or attached junk (`2-`); literals not spelled exactly `null`/`true`/`false`.
+- *JSON extensions* some parsers tolerate: comments (`//`, `/* */`), unquoted member names, and `NaN`/`Infinity`/`-Infinity` literals.
+
+**Known exclusions:** none beyond the rejection classes above. The conformance corpus (`conformance/accept.jsonl`, pinned hashes; `conformance/reject.jsonl`, uniform rejections; regenerated by `conformance/gen_corpus.py`, which refuses to pin any case the seven CLIs disagree on) is the authoritative definition of the supported domain. If your data stays in the supported domain, the byte-identity guarantee holds.
 
 ## Layout
 

c/include/baion/canonical_json.h

@@ -22,4 +22,61 @@ int baion_reject_u0000(const char* input, size_t len);
  * escaped-form duplicates. */
 int baion_reject_duplicate_keys(const cJSON* root);
 
+/* Pre-parse scan of raw JSON input bytes: returns BAION_OK if the first
+ * bytes are NOT a UTF-8 byte-order mark (EF BB BF), else BAION_ERR_PARSE.
+ * Must run BEFORE cJSON parsing — cJSON silently skips a leading BOM, so a
+ * BOM-prefixed document and its BOM-free twin would hash identically while
+ * being byte-distinct on the wire. */
+int baion_reject_bom(const char* input, size_t len);
+
+/* Pre-parse LEXICAL scan for raw control bytes: returns BAION_OK if the input
+ * carries no raw byte < 0x20 inside string literals and no raw byte < 0x20
+ * other than TAB/LF/CR between tokens, else BAION_ERR_PARSE. Must run BEFORE
+ * cJSON parsing — cJSON accepts raw controls inside strings and skips any
+ * byte <= 0x20 between tokens as whitespace, both of which RFC 8259 forbids,
+ * so C would otherwise accept documents the sibling lineages reject. Escaped
+ * forms (the backslash-t and backslash-u001F spellings) stay accepted: they
+ * are escape TEXT, not raw bytes, so this byte-level check cannot
+ * false-positive on them. */
+int baion_reject_raw_controls(const char* input, size_t len);
+
+/* Pre-parse scan of raw input bytes: returns BAION_OK if the whole byte
+ * stream is well-formed UTF-8 per RFC 3629, else BAION_ERR_PARSE. Rejects
+ * continuation bytes without a lead, truncated sequences, overlong encodings
+ * (0xC0/0xC1 leads, 0xE0 0x80-0x9F, 0xF0 0x80-0x8F), encoded surrogates
+ * (0xED 0xA0-0xBF), and values above U+10FFFF (0xF4 0x90+, 0xF5-0xFF leads).
+ * Must run BEFORE cJSON parsing — cJSON copies string bytes through
+ * unexamined, so C would otherwise hash byte streams the sibling lineages
+ * reject at decode time. */
+int baion_reject_invalid_utf8(const char* input, size_t len);
+
+/* Pre-parse LEXICAL scan of escape shape inside string literals: returns
+ * BAION_OK if every backslash is followed by one of the eight single-char
+ * escapes (quote, backslash, slash, b, f, n, r, t) or by 'u' + exactly 4 hex
+ * digits, else BAION_ERR_PARSE. Must run BEFORE cJSON parsing — cJSON's hex
+ * decoding tolerates some short backslash-u forms. Surrogate PAIRING validity
+ * is out of scope here; this scan judges lexical shape only. */
+int baion_reject_malformed_escapes(const char* input, size_t len);
+
+/* Pre-parse LEXICAL scan of RFC 8259 number token shape: returns BAION_OK if
+ * every number token is optional '-', then '0' or [1-9] digits (no leading
+ * zeros), then optional '.' followed by at least one digit (no bare trailing
+ * dot), else BAION_ERR_PARSE. Exponent text is not judged here — that
+ * verdict belongs to baion_reject_number_domain. Must run BEFORE cJSON
+ * parsing — cJSON accepts leading zeros and bare trailing dots that the
+ * sibling lineages reject. */
+int baion_reject_number_grammar(const char* input, size_t len);
+
+/* Pre-parse LEXICAL scan of raw JSON number tokens: returns BAION_OK if every
+ * number token in the input is inside the plain-decimal domain, else
+ * BAION_ERR_PARSE. Must run BEFORE cJSON parsing — cJSON collapses "100" and
+ * "1e2" onto the same double, so only the raw token spelling can distinguish
+ * them. Out of domain: any exponent notation (e/E); integer tokens (no '.')
+ * whose magnitude exceeds 2^53 (digit-string compare, never via double);
+ * fraction tokens whose value v has (v != 0 && |v| < 1e-6) or |v| >= 1e21.
+ * Precondition: input[len] == '\0' (the fraction check hands the token
+ * suffix to strtod, which stops at the token's non-number delimiter or at
+ * that terminator). */
+int baion_reject_number_domain(const char* input, size_t len);
+
 #endif /* BAION_CANONICAL_JSON_H */