Skip to content

perf: check a large JSON Lines file on several cores - #68

Merged
simonvanlierde merged 4 commits into
mainfrom
perf/jsonl-split
Oct 19, 2026
Merged

simonvanlierde merged 4 commits into
mainfrom
perf/jsonl-split

Conversation

@simonvanlierde

@simonvanlierde simonvanlierde commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Files were already checked on every core, but one large JSON Lines file still ran on one. A raw newline cannot sit inside a JSON string, and JSON Lines has no comments, so every raw newline ends a record and is a safe place to cut.

  • jsonvet::line_parts cuts the text after the first newline at or past each equal share. parse_lines parses one part within the whole text, starting at its first byte, so offsets need no shifting and a byte order mark counts only at offset 0. join_lines puts the parts back in order and stops at the first syntax error, as one pass does. All three are #[doc(hidden)], like parse.
  • The CLI splits a JSON Lines file of 1 MB or more into one part per core its chunk of files leaves free (threads / files in the chunk), through the existing parallel().

Measured with hyperfine -N, 10 runs, release builds of main and this branch, 12 cores:

Input main branch
78 MB, 400,000 nested records 264 ms ± 3 89 ms ± 4
100 MB, records that are mostly one long string 83 ms ± 2 64 ms ± 2
102 MB, the same with problems in 1 line in 40 94 ms ± 2 73 ms ± 2
102 MB with --schema 581 ms ± 4 566 ms ± 5

The long-string files gain less, because the 16-byte string scan already makes their parse cheap and reading the file (19 ms for cat) stays on one thread. With --schema, checking the values against the schema stays on one thread and dominates. Peak memory is unchanged (104 MB and 0.40 GB).

Output is byte-identical to main, exit code included, on seven files in text and SARIF, with and without --schema: 2,000 lines with duplicate keys, unsafe integers, lone surrogates, CRLF lines and a byte order mark spread through them; the same with a syntax error at line 1,501, a blank line, or a line too deep for --schema; and the three large files above.

Tests: a property test that the parts give the same problems and events as one pass for any part count from 1 to 8 (it failed with the byte order mark check left on for every part, and with the stop at the first syntax error removed); a test of where the cuts land, which caught a cut one line late; a CLI test comparing one part with eight across those cases; and a test that only JSON Lines of 1 MB or more is split. The check fuzz target now compares the parts with one pass too: 5 minutes each of check, diff_serde_json and diff_json5, then 5 more of check after the cut fix, no crash. cargo mutants on the touched functions: 31 caught, 14 unviable, two speed-only mutants excluded in .cargo/mutants.toml.

A raw newline cannot sit inside a JSON string, and JSON Lines has no
comments, so every raw newline ends a record and is a safe place to cut.

- Cut a JSON Lines file of 1 MB or more at newlines into one part per
  core its chunk of files leaves free, and check the parts at once.
- Each part is parsed within the whole text from its first byte, so
  offsets, lines and columns are the file's own, and a byte order mark
  counts only at the start.
- Join the parts in order and stop at the first syntax error, as one pass
  does. 78 MB of nested records check 3x faster on 12 cores; the output
  is unchanged.
- The check fuzz target now also compares the parts with one pass.
* origin/main:
  perf(parse): index object keys by stored 32-bit SipHash and reuse key lists (#66)
  feat: clearer syntax errors, encoding names and did-you-mean (#65)

# Conflicts:
#	CHANGELOG.md
#	crates/jsonvet/tests/properties.rs
@simonvanlierde
simonvanlierde enabled auto-merge (squash) October 19, 2026 15:01
simonvanlierde and others added 2 commits October 19, 2026 15:03
* origin/main:
  perf(schema): check a top-level array one element at a time (#67)

# Conflicts:
#	.cargo/mutants.toml
#	CHANGELOG.md
@simonvanlierde
simonvanlierde merged commit 81f425b into main Oct 19, 2026
25 checks passed
@simonvanlierde
simonvanlierde deleted the perf/jsonl-split branch October 19, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant