RAG Converter uses
rusty_json_turbofor JSON. It makes personal and work files AI-readable without them leaving the machine: the whole conversion runs as WebAssembly in the browser tab, with nothing uploaded and nothing to install.
rusty_json_turbo is serde_json, forked and made fast the house way: the same public API, the same bytes out, the same errors at the same line and column -- gated byte-identical against upstream on every commit -- with speed earned one measured brick at a time. Pure Rust, no C, no
*-sys, no FFI,no_std + alloccapable, dual MIT / Apache-2.0 exactly as upstream. The library keeps the crate nameserde_json, souse serde_json::...is the whole migration.
Part of Remade With Rust by Mata Network -- the JSON layer under every public API the house ships. Jump to the ecosystem
Status: five bricks landed, and the numbers below are quoted net of the measurement bias that made the first drafts of them look better than they were. What exists today:
- The fork, at serde_json 1.0.151 (
afdf6fc), unchanged in behaviour. Every upstream test passes against it. - The oracle. Upstream serde_json is linked in-process next to the fork,
and a differential test proves, over the whole corpus plus a few hundred edge
documents and every number token in it: identical compact and pretty bytes,
equal
Value, identical error text with line, column and category, identicalf64/f32bit patterns. It runs in CI on every push and under every feature flag with both arms on. - The harness. A paired ABBA benchmark over json-benchmark's three files,
arms alternating lead, a null arm for the floor, a method line on every
table. Numbers land in
corpus/LEDGER.mdand nowhere else first. - The instruments. A content census that counts where a document's bytes actually go, a scan-only ceiling that bounds every value-construction change, and deterministic work counters with an A/B knob — so a change can be decided by counting the work it removed, on a machine too busy to trust a clock.
- Five landed bricks, below, and four refuted ones with the arithmetic that killed each — a refutation is a result and it is written down.
- The plan.
docs/plans/fast_mission.md: the brick catalog, the gates, the milestones, the decisions.
| serde_json (upstream) | rusty_json_turbo | |
|---|---|---|
| Public API | the reference | identical, item for item, every feature flag |
| Output bytes, errors, float bits | the reference | byte-identical by contract, gated per commit |
C / *-sys in the dependency tree |
none | none |
unsafe |
12 sites | 12 sites inherited; M2 moves them to one audited island |
no_std + alloc, wasm32 |
yes | yes -- compiled on 9 targets, and both tested: the suite runs on wasm32-wasip1, no_std + alloc runs in four feature configurations, and the browser target's output bytes are checksum-matched to native |
| License | MIT OR Apache-2.0 | MIT OR Apache-2.0 |
| Speed | the baseline | 1.21x-1.23x on string-heavy stringify, 1.21x on key-heavy struct parse, 1.26x on typed streams over a reader, 1.89x on RawValue over a reader — each net of instantiation bias, each with its method line |
No number enters this README without a method line, and each brick's ledger row lands before its sentence here.
And every ours-against-upstream figure here is quoted NET of instantiation bias, which is the correction most forks never make. At the M0 commit this fork's source was character-for-character upstream's, and the same in-process comparison still read from 0.807x to 1.004x depending on the cell — two builds of the same crate get laid out and inlined differently, and for the serializer that was worth 19%. So the first drafts of the table below said 1.43x and 1.53x; the honest figures are 1.21x-1.23x. The bias itself moves about 5% between reference builds, so anything inside 5% of 1.00 is reported as unchanged rather than as a win.
| cell | reported | bias at identical source | net |
|---|---|---|---|
| twitter, struct stringify | 1.53x | 0.807x | 1.234x |
| twitter, DOM stringify | 1.43x | 0.840x | 1.214x |
| citm_catalog, struct parse | 1.22x | 0.979x | 1.212x |
| citm_catalog, DOM parse | 1.09x | 1.004x | 1.091x |
| citm_catalog, struct stringify | 1.12x | 0.965x | 1.075x |
| twitter, DOM parse | 1.04x | 1.003x | 1.050x |
| canada, struct parse | 1.04x | 1.000x | 1.036x |
| twitter, struct parse | 0.99x | 1.001x | 0.988x — no gain, and said so |
Ten of twelve cells are genuine gains; they are just smaller than the raw
comparison suggested, and the ordering changes — citm_catalog struct parse
turns out to be one of the best cells rather than a middling one, because it
had almost no bias to give back.
The M0 baseline is in corpus/LEDGER.md with its pin,
pairs, window, null-arm floor and machine: upstream against itself (the floor,
medians within 2.3%), ours against upstream (identical source; one cell shows a
3% link-layout bias, which is now that cell's floor), and ours against
simd-json 0.18 and sonic-rs 0.5 on all twelve json-benchmark cells. Two honest
readings from that table: upstream serde_json already beats simd-json on eleven
of twelve cells under this method, and sonic-rs's 1.5-3.6x DOM-parse lead is
its arena Value, not its scanner.
There are two performance stories here and they must not be added together in the reader's head, so they get two tables. Table 1 is what you can have today, from the house stack alone, with no code change anywhere. Table 2 is what this project is actually for — the changes to the JSON code itself, each of which lands with its own ledger row and its own byte-identical gate.
Add rusty_json_turbo-alloc
to your binary and declare it. That is the whole diff: not one line of JSON
code touched. Pinned to one core, paired, ABBA-interleaved, allocation counts
proven identical on both sides. > 1 means the house allocator is faster.
| workload | allocations per parse | platform | rusty_alloc | ratio | window |
|---|---|---|---|---|---|
| twitter, DOM parse | 20,834 | 459 MB/s | 690 MB/s | 1.53x | 2 s |
| citm_catalog, DOM parse | 39,339 | 697 MB/s | 1013 MB/s | 1.46x | 2 s |
| canada, DOM parse | 56,061 | 376 MB/s | 499 MB/s | 1.32x | 250 ms |
| twitter, struct parse | 2,762 | 880 MB/s | 1024 MB/s | 1.17x | 250 ms |
| citm_catalog, struct parse | 2,544 | 1326 MB/s | 1465 MB/s | 1.12x | 250 ms |
| canada, struct parse | 485 | 723 MB/s | 768 MB/s | 1.07x | 2 s |
| any file, stringify | 0 | 1545 MB/s | 1476 MB/s | 1.00x | 2 s |
The last row is the control, and it is why the others are believable.
A stringify into a pre-sized buffer allocates zero times, so the allocator
cannot touch it — and it doesn't. The effect sorts with the allocation count,
which a deterministic census proves is identical under both allocators. The
window column matters: at 250 ms samples twitter DOM parse reads 2.0x rather
than 1.53x, so about a quarter of the short-window figure is per-process
warm-up and the longer, smaller number is the honest one; the 250 ms rows are
therefore upper bounds on their own effect. canada, struct parse at 1.07x is
inside the cross-binary layout band (0.976–1.070x, measured on the
zero-allocation cells) and so is not claimable as an allocator result.
The full oracle passes under rusty_alloc and under its hardened secure
profile: no output byte changes, and secure keeps nearly the whole win.
Two things Table 1 is not. It is not a win of this crate over
serde_json: at this milestone the JSON code is upstream's, so upstream linked
against rusty_alloc gets exactly the same thing, and none of it counts toward
the speed gate, which compares like-for-like allocators. And it is a Windows
measurement, where Rust's platform allocator is HeapAlloc; glibc's malloc has
tcache and fastbins and should close much of the gap, so the Linux number is an
open question rather than an extrapolation.
Changes to the JSON code itself. One has landed. Each row ships only with a measured ledger entry and a byte-identical gate, and any row that does not pay is reverted with the reason recorded. The rows marked repriced or promoted were re-ranked by measurement before being built, which is the whole point of having instruments first.
| Upgrade | Targets | Mechanism | Status |
|---|---|---|---|
| Whitespace off the per-byte path | every token boundary | skip a whitespace run in one walk of the slice instead of a Result<Option<u8>> round trip per byte |
✅ landed — see below |
| Wide whitespace scan | pretty-printed input | eight bytes per step (SWAR), with a short-run peel and the scalar walk kept as the oracle | ✅ landed — see below |
| SIMD whitespace scan | pretty-printed input | an SSE2/AVX2 twin of the above, in a separate crate so the parser never writes unsafe |
❌ built, then switched OFF by default — measured against upstream it loses on 13 of 15 cells. A #[target_feature] function cannot be inlined, so reaching the island costs more inlining than the width buys. Reachable with --features accel; see corpus/LEDGER.md |
| Map as a sorted vector | DOM parse | replace Map's BTreeMap with a sorted Vec — same sorted iteration order, so it is a drop-in |
promoted. The median object arity is 2, 97.7% of citm_catalog's objects hold eight keys or fewer, and a BTreeMap node is built for eleven pairs — so ~80% of each node is unused. An unpaired probe puts the whole value-model change around 1.45x on citm_catalog (median of nine runs, spread 0.97x–1.74x). That decides the brick is worth building; it is not a figure to quote, and a paired A/B will set the real one |
Arena Value (additional type) |
DOM parse | interned keys and inline short strings beside the sorted-vector map — the shape that gives the fastest competitor its 1.5–3.6x DOM lead | planned, v1.x, and the ceiling is now measured. The scan column builds nothing, so dom-parse above it is the price of building a Value: 70%-85% of a DOM parse, a 3.29x ceiling on citm_catalog and 6.89x on the widest S4 fixture. Key reuse is 80.6x on citm, 142x on twitter and 1,418.7x on frame-telemetry — 46,816 allocations for 33 distinct names |
| SIMD string scan | string-heavy input | 16/32-byte twin of the existing 8-byte SWAR quote/backslash/control scan | demoted — mean string run is 19 bytes, 98.3% already zero-copy, and upstream is already 8-byte SWAR with memchr2 |
| Escape-mask writer | stringify | per-chunk "needs escape" mask, one write per clean run | ✅ landed — see below |
| 8-digit integer / fraction parse | number-heavy input | validate and convert eight ASCII digits in one 8-byte load, per-digit tail | ✅ landed — see below |
| Key dispatch + derive handshake | struct parse | length-bucketed match, then a field-index handshake across the serde seam, replacing a linear memcmp ladder per key |
planned |
| Buffered reader | from_reader |
an internal window reusing the slice scanners, instead of one iterator call per byte | ✅ landed — the reader-to-slice gap closed from 1.55x to 1.22x on citm_catalog, 1.28x to 1.14x on twitter; and you no longer want a BufReader |
| ASCII fast-path UTF-8 validation | from_slice |
validate the ASCII run wide, walk only non-ASCII tails | reframed — 18,099 short validations per twitter parse, but validating once up front is not byte-identical, so that form is rejected |
| Sink specialisation | stringify | fold separators into adjacent writes; write integers and floats into spare capacity | promoted — measured 2.6 bytes per sink call on citm_catalog, about 7.3 calls per key; it must win on call count alone, and the count says there is room |
| Correctly-rounded float parse | float-heavy input | core's Eisel-Lemire, replacing the vendored bignum path | planned, v1.x, opt-in (it changes output) |
One price worth knowing before you set a feature flag: preserve_order costs
you 1.16x-1.45x on DOM parsing. Measured as two binaries with one flag
between them, 15 pairs, pinned, ABBA — IndexMap loses to BTreeMap on every
DOM cell, unanimously, at z = +3.87, while struct-parse (which builds no
Map and so cannot be reached by the change) stays flat as the control. The
cost is the hashing: hashing two keys to build a two-element map is pure
overhead against a BTreeMap that stores both in one node. Insertion order is
worth having — it is just not free, and now the price is written down.
If you hold JSON you do not want to parse — routing it, storing it, forwarding
it — RawValue is the right tool, and over a reader upstream builds it by
pushing every consumed byte into a Vec, one at a time, through next()
and discard(). Since this fork's reader holds a window, those bytes are
already contiguous: the capture does not need accumulating, it needs
addressing. So begin_raw_buffering records where the value starts, fill
declines to slide the window while a capture is live, and the text is taken in
one copy at the end.
Priced on a 10,000-document NDJSON log stream (2.6 MB), release, best-of-7, all four arms in one process:
| arm | before | after |
|---|---|---|
Value over a reader |
157 MB/s | 165 MB/s |
RawValue over a reader |
257 MB/s | 466 MB/s |
RawValue over a slice |
1,074 MB/s | 1,048 MB/s |
RawValue over a reader, upstream serde_json |
259 MB/s | 246 MB/s |
1.81x on the target arm, and the control held — the slice arm is in the
same process on the same file and this change provably cannot reach it
(SliceRead's capture was already a range), and it moved 0.98x. Against
upstream in the same process, 1.009x → 0.528x: 1.89x faster. The
reader-to-slice gap the pricing test existed to find closes from 4.18x to
2.25x.
Growing and returning the window changes no output at all, so a byte-identical gate cannot see it; the work counters can, and do: 10,000 captures, 2,603,127 bytes, zero window growths on that stream, against 2 growths and 1 return on a single 60 KB value.
A log stream is the opposite shape from the benchmark corpus — 10,000 documents of ~260 bytes rather than one large blob — and it is all reader, so it is where the buffered reader pays most. 15 pairs, leading arm rotated:
| arm | ours | upstream | ratio |
|---|---|---|---|
StreamDeserializer / reader, typed struct |
221 MB/s | 176 | 1.26x faster |
StreamDeserializer / reader, Value |
156 MB/s | 140 | 1.11x faster |
per-document from_slice, typed |
273 MB/s | 274 | parity |
StreamDeserializer / slice, typed |
275 MB/s | 279 | parity |
Two findings from that cell worth having: StreamDeserializer costs nothing
against slicing the stream yourself (0.993x-1.020x across four
configurations), and per-document cost is almost entirely fixed rather than
proportional to size — the pretty variant is 325 bytes per document against
the minified 261 and is faster per document, because the extra bytes are
whitespace the wide scanner eats cheaply while the setup is the same either
way.
citm_catalog.json is 71.0% whitespace, and skipping it was 36% of the
time to parse that file into a Value — measured, not guessed, by running the
same document with the skippable whitespace removed. It went through
peek()/discard() one byte at a time. It now walks the run in one pass, eight
bytes per step where the run is long enough to pay for it.
Measured in one binary with one environment variable between the arms, so the
two arms cannot differ by code layout. 21 pairs, pinned, ABBA, on a quiet
machine. > 1 means the new path is faster; the session's null-arm floor was
0.983x–1.021x.
| file | workload | before | after | ratio |
|---|---|---|---|---|
| citm_catalog | scan | 1,565 MB/s | 2,105 MB/s | 1.345x |
| citm_catalog | struct parse | 1,171 MB/s | 1,461 MB/s | 1.246x |
| citm_catalog | DOM parse | 664 MB/s | 722 MB/s | 1.095x |
| canada | scan | 1,006 MB/s | 1,062 MB/s | 1.053x |
| scan | 1,472 MB/s | 1,524 MB/s | 1.042x | |
| struct parse | 791 MB/s | 813 MB/s | 1.036x | |
| any file | stringify | — | — | 0.997x–1.008x (control, unmoved) |
Eight digits per step (canada.json is 90.08% number bytes):
| file | workload | before | after | ratio |
|---|---|---|---|---|
| canada | struct parse | 555 MB/s | 604 MB/s | 1.083x |
| canada | DOM parse | 296 MB/s | 309 MB/s | 1.043x |
| citm_catalog | struct parse | 1,361 MB/s | 1,383 MB/s | 1.017x |
| any workload | — | — | at the floor (1.5% number bytes) | |
| stringify ×6, scan ×3 | — | — | — | 0.999x–1.010x (control, unmoved) |
Eight bytes per escape step, on stringify (twitter is 58.1% string; the
escape hit rate is 0.334%):
| file | workload | before | after | ratio |
|---|---|---|---|---|
| struct stringify | 1,774 MB/s | 2,153 MB/s | 1.208x | |
| DOM stringify | 1,837 MB/s | 2,202 MB/s | 1.200x | |
| citm_catalog | struct stringify | 1,870 MB/s | 2,007 MB/s | 1.068x |
| citm_catalog | DOM stringify | 1,261 MB/s | 1,306 MB/s | 1.028x |
| parse cells ×2 | — | — | — | 0.995x–1.002x (control, unmoved) |
Both twitter rows won every one of 61 paired runs bar one, and a best-of-N
statistic agrees with the median on every row. The controls are parse cells
here — stringify cells were the control for every parse brick so far, and the
roles simply swap. canada is not quoted: it holds 90 string bytes in 2.25 MB,
so there is nothing for this to win.
The counter that proves it is switched on, which no output gate can:
scanner steps fall from 367,917 to 101,382 on twitter and 221,379 to 108,825
on citm_catalog, while the count of fragments written stays identical — same
output, less walking. And note the opposite tuning to the whitespace scan:
that one peels four bytes scalar first because 46% of its runs are a single
byte; this one has no peel, because a string scan runs the length of a whole
string. Same technique, opposite shape, because the census said so.
The gradient across the three files is the corpus census read back:
canada is 90.08% number and moves most, citm_catalog is 7.35% and moves a
little, twitter is 1.5% and does not move. Nine control cells stayed inside 1%.
The step is exact, not approximate — below a bound where eight more digits
provably cannot overflow a u64, the chunk takes the same branch the
byte-at-a-time loop would have taken, including the digit at which a long number
switches to the slow float path.
Against upstream serde_json, net of instantiation bias. The middle column
is what a naive ours-vs-upstream run reports; the right column is what our code
actually changed. They differ because two crate instantiations get laid out and
inlined differently: at a commit where our source was byte-identical to
upstream, the same comparison already read 0.807x on twitter struct stringify
and 1.004x on citm_catalog DOM parse. That per-cell floor is divided out
here.
| file | workload | raw ratio | net of bias |
|---|---|---|---|
| struct stringify | 1.53x | 1.23x | |
| DOM stringify | 1.43x | 1.21x | |
| citm_catalog | struct parse | 1.22x | 1.21x |
| citm_catalog | DOM parse | 1.09x | 1.09x |
| citm_catalog | struct stringify | 1.12x | 1.08x |
| DOM parse | 1.04x | 1.05x | |
| canada | struct parse | 1.04x | 1.04x |
| citm_catalog | DOM stringify | 1.14x | 1.03x |
| canada | struct stringify | 1.03x | 1.02x |
| canada | DOM parse | 1.01x | 1.02x |
| struct parse | 0.99x | 0.99x | |
| canada | DOM stringify | 1.01x | ~1.00x |
The bias is almost entirely in the stringify column — every parse cell's
floor sits between 0.979x and 1.004x, while three of four stringify cells sit
between 0.807x and 0.938x. The serializer is the tighter code path and far more
sensitive to layout, which is exactly why the raw stringify figures looked so
much better than the raw parse ones. Measured with tools/biasvs.ps1: two
binaries interleaved round by round, each normalised against its own upstream
arm so drift cancels; 9 rounds, 9 pairs each, pinned.
The correction has its own error bar of about 5%, because the bias itself moves between builds of a reference that should be equivalent. Treat anything within 5% of 1.00 as unchanged, and read these to two significant figures at most.
The brick figures elsewhere in this README are not affected. Those come from knob A/B runs — one binary, one crate instance, one environment variable between the arms — so there is no instantiation difference to bias them. That is why this project measures bricks that way.
Two cells are flat and both are understood. twitter struct parse is
at 0.99x: a ceiling probe made key dispatch completely free and bought 0.6%,
so the derive is already near-optimal and there is nothing there to win. And
twitter DOM parse is 1.05x against an ambition of 1.8x, because DOM parse
makes roughly one allocation per 31 input bytes — short strings dominate the
count, map nodes dominate the bytes, and about 80% of every node is unused at a
median arity of two. Closing that needs a different Value type, not another
brick.
One honest caveat on the absolute MB/s above. They were taken before
the corpus was pinned platform-independent. There was no root .gitattributes,
so a core.autocrlf=true checkout inflated twitter.json from 631,514 to
646,995 bytes -- 15,481 carriage returns, every one counted as whitespace --
and citm_catalog.json by 50,468. No ratio in this README is affected:
both arms of every paired run read the same file, so 1.208x is 1.208x either
way. The absolute MB/s figures are on the 2.4% larger documents and will read
slightly differently now, and the byte-percentages have been recomputed. The
manifest was regenerated and is now verified on all three CI runners; the full
account is in corpus/LEDGER.md under M1-D.
A four-digit step was built on top of it twice and reverted twice, and
the second time is the one worth reading. At the fraction call site it hits
110,984 times out of 111,080 and removes a third of canada's peek
calls — and is still 2.5% slower (61 pairs, 51 wins, best-of-N agreeing,
two stringify controls flat). The fold has a fixed cost that does not shrink
with width: four scalar digit steps are already about as cheap as one fold, so
break-even sits above four digits. Removed work is not saved time. Full
evidence in corpus/LEDGER.md.
Every row above 1.04x won 21 of 21 paired runs. canada.json's 1.053x is
not a whitespace result — that file has 33 whitespace bytes in 2.25 MB. It is
the entry point now costing one load and one test where it used to build a
Result<Option<u8>>, so minified documents gain too, which is where most JSON
on a wire lives. The four stringify cells never reach this code and did not
move, which is what makes the rest believable.
Two regressions were found and fixed on the way, both by files that
cannot benefit. Returning only an index made the caller re-load the byte it
had just examined and cost 4% on canada; and reaching for an eight-byte step
on a one-byte run — 46% of twitter's whitespace runs — cost 10% on twitter
scan. Neither was visible in the output: both arms stayed byte-identical
throughout. Keeping a file in the corpus that a change must not help is what
caught them. Full history, counts and method line:
corpus/LEDGER.md.
A fork, not a wrapper and not a rewrite: the upstream tree lives at this
repository's root so git merge upstream/master stays a plain merge, and
every divergence is one row in
docs/UPSTREAM-CHANGES.md. Upstream is also the
oracle: it is a crate, so it links into the harness on every developer
machine and CI runner, and the correctness gate is a unit test rather than a
rig. Bricks that need nothing house-specific are offered upstream as pull
requests.
The companion fork of serde / serde_core / serde_derive lives at
Remade-With-Rust/serde (branch
turbo). Its crate names and versions are unchanged and it is consumed only as
a [patch.crates-io] overlay, so every impl Serialize in the ecosystem keeps
working.
Remade With Rust is an initiative by Mata Network to rebuild essential C and C++ tools in Rust — for the memory safety, the predictable performance, and the freedom of a permissive license. Each project is a reimplementation, not a fork: same wire protocols and file formats, new code you can actually depend on.
We build the core to production grade and open-source it so the community can extend it. No copyleft. No surprises. Just the tools we rely on, made faster and safer.
| Project | What it is |
|---|---|
| 🎬 remade_ffmpeg_rs | Our FFmpeg alternative. Drop-in ffmpeg and ffprobe binaries — demux → decode → filter → encode → mux, rebuilt as composable Rust crates with zero GPL/LGPL. Apache-2.0. |
| 🧠 FFAI | Our sister project: media for AI. "The AI media toolkit, remade with rust." Embedded ASR + TTS (Mercury), OCR (Carmenta) and vision-language captioning (Argus) behind an ffmpeg-style, swap-by-name architecture — no Python, no CUDA. MIT OR Apache-2.0. |
| 🌐 Mata Network | The home page. "Stop sacrificing your privacy for convenience." Sovereign, self-hostable privacy infrastructure — wallet & identity, password manager, contact manager, and a browser extension that stops information leaking as you browse. Remade With Rust is its open-source arm. |
→ All projects: github.com/Remade-With-Rust
At M0 the library carries upstream's twelve unsafe sites unchanged. The
mission plan's M2 moves every one of them behind a single -accel island (or
replaces it with safe code the measurement says costs nothing), so that the
core can take #![forbid(unsafe_code)] and every kernel keeps its scalar twin
as the oracle and the fallback. Until then: no new unsafe anywhere, and the
workspace denies it outside this crate.
[dependencies]
# Drop-in: the library's crate name is `serde_json`, so this line alone swaps it.
serde_json = { package = "rusty_json_turbo", version = "0.2" }
# or, keeping the package name in code:
rusty_json_turbo = "0.2" # then `use serde_json::...` as before| Feature | Default | What it adds |
|---|---|---|
std |
yes | io::Read/Write entry points, std::error::Error |
alloc |
-- | no_std with a heap: default-features = false, features = ["alloc"] |
preserve_order |
-- | Map keeps insertion order (indexmap) |
float_roundtrip |
-- | exact-round-trip float parsing (~2x slower on floats) |
arbitrary_precision |
-- | Number keeps the original digits |
raw_value |
-- | RawValue |
unbounded_depth |
-- | disable_recursion_limit |
profile |
-- | house instruments (stage profiler, reach census); zero-sized when off |
Every feature has the semantics upstream documents at docs.rs/serde_json; upstream's own guide applies verbatim. MSRV 1.85.
The library never sets #[global_allocator] — deliberately. A program may
declare exactly one, so it belongs to the binary, and this crate is a drop-in
for serde_json, which sits in nearly every Rust dependency graph. To take
Table 1's speedup, add the seam to your binary:
[dependencies]
rusty_json_turbo-alloc = "0.1"#[global_allocator]
static ALLOC: rusty_json_turbo_alloc::Alloc = rusty_json_turbo_alloc::Alloc;| Crate | Role |
|---|---|
rusty_json_turbo |
← you are here — the library: a drop-in serde_json, byte-identical by contract |
rusty_json_turbo-alloc |
the rusty_alloc seam, for the binary — Table 1's speedup in one line |
rusty_json_turbo-accel |
the SIMD island, arriving with the kernels (M3). Not yet published |
rusty_json_turbo-cli · -bench |
the rjson command line and the measurement harness. Not published: one is a deliverable, the other links the oracle |
The family shares one version, bumped together, so pinning one pins them all.
rjson (never jq) is consumer #1 of the library and never the only one:
rjson validate file.json # exit 0 if valid; errors carry line and column
rjson pretty file.json # re-emit pretty-printed
rjson minify - # from stdin, compact
rjson isa # which kernel arm this process resolved. upstream serde_json 1.0.151, at the root so merges stay plain
├── src/ tests/ build.rs fuzz/
├── crates/
│ ├── rusty_json_turbo-alloc/ the rusty_alloc seam (one pin)
│ ├── rusty_json_turbo-cli/ rjson (allocator declared here)
│ └── rusty_json_turbo-bench/ the ORACLE + the paired benchmark; links upstream; never published
├── corpus/ JSONCORP data, HASHES.txt, fetch.ps1, LEDGER.md
├── docs/plans/fast_mission.md the mission plan; docs/UPSTREAM-CHANGES.md the divergence map
└── tools/pinbench.ps1 pinned, High-priority runner for rjson-bench
cargo test -p rusty_json_turbo # upstream's suite against the fork
cargo test -p rusty_json_turbo-bench # the differential oracle over JSONCORP
cargo build --release -p rusty_json_turbo-bench [--features competitors]
powershell -File tools/pinbench.ps1 -- null --all # the floor
powershell -File tools/pinbench.ps1 -- bench --all # ours vs upstream
target/release/rjson-bench diff-oracle path/to/*.json # the gate, on your own filesCompiled means cargo check in three feature configurations. Tested
means the real test suite runs there. The distinction is the point: a check
cannot catch a wrong answer, and the places a JSON parser gets a different
answer on a different target are real ones -- usize width in the recursion
guard and the reader's window arithmetic, float formatting, and byte order
anywhere a multi-byte load is used, which this crate does on the whitespace
scan, the escape scan and the eight-digit number fold.
| Platform | Status |
|---|---|
| Linux x86_64 (gnu, musl) | compiled; full suite tested in CI |
| Windows x86_64 | compiled; full suite tested in CI |
| macOS aarch64 | compiled; full suite tested in CI, and the per-arch census runs there |
| Linux aarch64 (gnu) | compiled; suite, oracle, census and no_std probe run under qemu-user in CI (correctness only -- a qemu timing is meaningless and none is taken) |
| Linux aarch64 (musl), Windows aarch64, macOS x86_64 | compiled in CI |
wasm32-wasip1 |
full suite tested under wasmtime: 249 library tests, the differential oracle over the whole corpus, the soak, and every brick gate |
wasm32-unknown-unknown (the browser target) |
compiled, and its output bytes are checksum-compared with native -- see below |
no_std + alloc |
compiled on every target above, and RUN in four feature configurations, natively and on wasm |
no_std with no std at all (aarch64-unknown-none) |
compiled in CI |
Two of those deserve spelling out, because they are the ones most projects claim without checking.
The differential oracle passes on wasm32. Output bytes, Value, error
text with line and column, float bit patterns and StreamDeserializer offsets
-- all identical to upstream serde_json, over 94 documents, on a 32-bit
target.
The browser target is checked by checksum, not by assertion.
wasm32-wasip1 has a platform underneath it; a browser does not. So
crates/rusty_json_turbo-wasmdemo embeds three corpus documents, parses and
re-serializes them inside wasm32-unknown-unknown, and its FNV-1a of the
output bytes must equal native's exactly -- including 2,063,469 bytes of
re-serialized canada, which is 2.25 MB of floating point and the one value
family where a difference would be a silent wrong answer rather than a crash.
The parser is linked alloc-only there, and the module is asserted to need no
host imports at all.
A per-arch census guards the thing no output gate can see. The wide
scanners are 8-byte SWAR behind length guards, so an architecture where those
guards never passed would produce completely correct output at the speed of
the per-byte fallback. The census asserts they are reached, and measured, every
counter reading on wasm32 is byte-for-byte identical to x86_64.
NEON and simd128 twins are deliberately not shipped, and the reason is
measured rather than asserted: the corpus's mean whitespace run is 7.2 bytes
with the longest at 29, so a 16-byte vector kernel has the same problem AVX2
had at 32 -- which lost on every cell but one. Plus there is no aarch64
hardware here to measure on, and this project does not ship unmeasured
optimisations. The full argument is in corpus/LEDGER.md.
Licensed under either of Apache License, Version 2.0 or MIT license at your
option, exactly as upstream serde_json; the LICENSE-APACHE and LICENSE-MIT
files are upstream's. Attribution and corpus provenance: NOTICE.md.
No GPL/LGPL and no C anywhere in the dependency tree, enforced with
cargo-deny.
Mata Network builds sovereign, self-hostable privacy infrastructure — "stop sacrificing your privacy for convenience": wallet & identity, a password manager, a contact manager, and a browser extension that stops your information leaking as you browse.
Remade With Rust is our open-source home for the permissively-licensed building blocks that work depends on — including remade_ffmpeg_rs (the FFmpeg alternative) and FFAI (the AI media toolkit).