Skip to content

feat(html): incremental HTML parser over lexbor, with a pause at every script - #68

Merged
x-ror merged 2 commits into
masterfrom
feat/html-lexbor
Nov 18, 2026
Merged

x-ror merged 2 commits into
masterfrom
feat/html-lexbor

Conversation

@x-ror

@x-ror x-ror commented Nov 18, 2026

Copy link
Copy Markdown
Owner

What changed

pkg/html binds lexbor and wraps it in an incremental Parser (#10):

  • feed/finish take network bytes; pump parses until the next parser-blocking script, the end of the available input, or the end of the document;
  • when pump returns a script, the tree holds exactly what precedes it, as a browser's parser would have it;
  • write inserts document.write text at the insertion point, before the rest of the input, and scripts inside it pause in turn.

lava html [--scripts] <file|-> parses a file (or stdin) in 16 KiB pieces and prints the serialized document. --scripts reports each script at the moment the parser pauses for it. No script is executed.

How the pause works without changing lexbor

lexbor has no pause and no document.write: its tree builder leaves both as TODO at </script> (html/tree/insertion_mode/text.c). It does not need them, because the caller holds the input:

  • pump feeds lexbor slices that never run past the first > after a </script (any case);
  • a small C shim (pkg/html/shim/larva_html.c) wraps lexbor's token callback and records the script element an end tag actually closed. A </script> inside a comment or a JS string is ignored, because lexbor never tokenizes it as that end tag.

The shim also wraps the few lexbor calls that are inline functions or struct reads, so no lexbor struct layout is mirrored in Odin.

Dependency (§2)

lexbor, Apache-2.0, which fits both the AGPL core and the commercial extensions. Added to NOTICE.

  • Pinned to commit e609d94 and checked against the tarball's sha256.
  • Built from source into the gitignored .deps/ by scripts/build-lexbor.sh, which build-native-deps.sh now calls. Needs cmake and network on the first build.
  • Not the distro package: liblexbor-dev is not installed on the dev machines or the CI image. Not vendored: the source is ~22 MB.

Tried by hand

  • A small page with an external script, an inline script holding "</scr" + "ipt>", a </script> inside a comment, and an upper-case <SCRIPT>: three pauses, in order, none on the comment.
  • en.wikipedia.org/wiki/HTML (803 KB): 5 pauses for the 5 <script tags in the source, 0.01 s and 24 MB max RSS (/usr/bin/time, one run on a 16-core dev machine).
  • The document.write path was shown with a C prototype of the same feeding scheme; the CLI does not call write.

Not done

  • No tests and no gates: paused for this phase on the owner's instruction. odin check cmd/lava passes and bin/lava builds.
  • Charset sniffing: input is taken as UTF-8.
  • Fetching a URL (larva fetch), script execution, and a DOM API for JS.
  • An end tag with a quoted > in an attribute (</script x=">">) is cut too early and the pause lands late; documented in parser.odin.
  • darwin/windows: the foreign import points at .deps on Linux/macOS; no lexbor build exists for Windows yet.

🤖 Generated with Claude Code

x-ror and others added 2 commits November 18, 2026 21:20
…y script

pkg/html binds lexbor (Apache-2.0, pinned commit, built from source into .deps/
by scripts/build-lexbor.sh) and wraps it in an incremental Parser:

- feed/finish take network bytes; pump parses until the next parser-blocking
  script, the end of the available input, or the end of the document;
- when pump returns a script, the tree holds exactly what precedes it, as a
  browser's parser would have it;
- write inserts document.write text at the insertion point, before the rest of
  the input.

lexbor has no pause and no document.write and does not need them: we feed it
slices that never run past a `</script>` end tag, and a small C shim
(pkg/html/shim) wraps lexbor's token callback to learn which script element
that end tag closed.

`lava html [--scripts] <file|->` parses a file in 16 KiB pieces and prints the
serialized document; --scripts reports each script where the parser pauses.

Not done: charset sniffing (input is taken as UTF-8), fetching a URL, script
execution, a DOM API. No tests: paused for this phase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t; hide lexbor

From the quality and native-safety reviews of the first version:

- A script was silently lost when an end tag had a quoted `>` in an attribute
  (`</script a=">">`): the slice ended early, the next one closed two scripts,
  and the hook kept only the last. The cut now skips quoted attribute values as
  the tokenizer does, and the shim counts closes: two in one slice is a failure,
  never a dropped script.
- Consecutive writes came out reversed (each was inserted at the same point).
  Written text now has its own buffer and an insertion cursor that advances.
- One unterminated `</script` made parsing quadratic and held back the rest of
  the page (16 MiB in 16 KiB reads: 3.17 s). The cut scan resumes where it
  stopped, compaction is amortized, and an unterminated candidate holds back at
  most 4 KiB. Same input now: 0.05 s.
- lexbor no longer leaks into the API: the bindings are private, callers see
  Parser and Node. Off Linux, init fails with a message instead of a link error.
- The feed loop moved from the CLI into pkg/html (parse_all); serialize returns a
  string; attr copies; one ASCII helper replaces the `| 0x20` folding that also
  matched 0x0F as '/'.
- Docs: layering table, ARCHITECTURE 3.6, README build requirements, make help.

FFI declarations stay hand-written rather than generated by odin-c-bindgen:
a dozen procs, and everything lexbor exposes as inline functions, macros or
struct fields goes through the shim, which a generator cannot bind.

No tests: paused for this phase. Checked by hand: odin check on linux, windows
and darwin targets; the three cases above through `lava html` and a probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@x-ror

x-ror commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Pushed 5b41d0a, which addresses a quality review and a native-safety review of the first commit.

Bugs, each confirmed by the reviewer with a probe and re-checked by hand after the fix:

Bug Before After
<script>A()</script a=">"><script>B()</script> only B handed out, A lost both, in order
write("<i>1</i>") then write("<b>2</b>") <b>2</b><i>1</i> <i>1</i><b>2</b>; a nested write lands after them
one </script with no >, 16 MiB in 16 KiB reads 3.17 s 0.05 s, at most 4 KiB held back

Two scripts closing in one slice is now a parse failure, never a dropped script.

Structure:

  • lexbor no longer leaks into the API: the bindings are private, callers see Parser and Node. Off Linux, init fails with a message instead of a link error (lexbor_other.odin).
  • The feed loop moved from the CLI into pkg/html (parse_all). serialize returns a string, attr copies.
  • One ASCII helper replaces the | 0x20 folding, which also matched byte 0x0F as /.
  • lava html rejects unknown flags; --chunk N replaces the fixed 16 KiB.
  • Docs: layering table in CLAUDE.md, docs/ARCHITECTURE.md §3.6, README build requirements, make help.

Bindings stay hand-written, not generated with odin-c-bindgen: a dozen procs, and everything lexbor exposes as inline functions, macros or struct fields goes through the C shim, which a generator cannot bind.

Not done from the reviews:

  • GitHub archive tarballs are not guaranteed byte-stable. If one changes, the sha256 check fails closed and the build stops.
  • Two concurrent build-lexbor.sh runs can clobber .deps/lexbor-src.

Still no tests or gates. odin check passes on linux, windows and darwin targets.

@x-ror
x-ror merged commit 22bc600 into master Nov 18, 2026
5 checks passed
@x-ror
x-ror deleted the feat/html-lexbor branch November 18, 2026 19:41
@x-ror

x-ror commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Pushed 87cf007, which closes the two items the reviews left open:

  • Tarball stability. lexbor is now fetched with git fetch --depth 1 <commit> and checked by its commit id instead of a sha256 over a GitHub archive tarball, which GitHub does not promise to keep byte-identical. Needs git instead of curl; README updated.
  • Concurrent builds. build-lexbor.sh holds flock on .deps/lexbor.lock and builds in a private mktemp dir removed on any exit. Checked by hand: two runs started together from an empty .deps built lexbor once.

Still no tests or gates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant