Repository navigation
perf: check a large JSON Lines file on several cores - #68
Merged
Merged
Conversation
A raw newline cannot sit inside a JSON string, and JSON Lines has no comments, so every raw newline ends a record and is a safe place to cut. - Cut a JSON Lines file of 1 MB or more at newlines into one part per core its chunk of files leaves free, and check the parts at once. - Each part is parsed within the whole text from its first byte, so offsets, lines and columns are the file's own, and a byte order mark counts only at the start. - Join the parts in order and stop at the first syntax error, as one pass does. 78 MB of nested records check 3x faster on 12 cores; the output is unchanged. - The check fuzz target now also compares the parts with one pass.
simonvanlierde
enabled auto-merge (squash)
October 19, 2026 15:01
* origin/main: perf(schema): check a top-level array one element at a time (#67) # Conflicts: # .cargo/mutants.toml # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Files were already checked on every core, but one large JSON Lines file still ran on one. A raw newline cannot sit inside a JSON string, and JSON Lines has no comments, so every raw newline ends a record and is a safe place to cut.
jsonvet::line_partscuts the text after the first newline at or past each equal share.parse_linesparses one part within the whole text, starting at its first byte, so offsets need no shifting and a byte order mark counts only at offset 0.join_linesputs the parts back in order and stops at the first syntax error, as one pass does. All three are#[doc(hidden)], likeparse.threads / files in the chunk), through the existingparallel().Measured with
hyperfine -N, 10 runs, release builds of main and this branch, 12 cores:--schemaThe long-string files gain less, because the 16-byte string scan already makes their parse cheap and reading the file (19 ms for
cat) stays on one thread. With--schema, checking the values against the schema stays on one thread and dominates. Peak memory is unchanged (104 MB and 0.40 GB).Output is byte-identical to main, exit code included, on seven files in text and SARIF, with and without
--schema: 2,000 lines with duplicate keys, unsafe integers, lone surrogates, CRLF lines and a byte order mark spread through them; the same with a syntax error at line 1,501, a blank line, or a line too deep for--schema; and the three large files above.Tests: a property test that the parts give the same problems and events as one pass for any part count from 1 to 8 (it failed with the byte order mark check left on for every part, and with the stop at the first syntax error removed); a test of where the cuts land, which caught a cut one line late; a CLI test comparing one part with eight across those cases; and a test that only JSON Lines of 1 MB or more is split. The
checkfuzz target now compares the parts with one pass too: 5 minutes each ofcheck,diff_serde_jsonanddiff_json5, then 5 more ofcheckafter the cut fix, no crash.cargo mutantson the touched functions: 31 caught, 14 unviable, two speed-only mutants excluded in.cargo/mutants.toml.