Repository navigation
feat(html): incremental HTML parser over lexbor, with a pause at every script - #68
Merged
Merged
Conversation
…y script pkg/html binds lexbor (Apache-2.0, pinned commit, built from source into .deps/ by scripts/build-lexbor.sh) and wraps it in an incremental Parser: - feed/finish take network bytes; pump parses until the next parser-blocking script, the end of the available input, or the end of the document; - when pump returns a script, the tree holds exactly what precedes it, as a browser's parser would have it; - write inserts document.write text at the insertion point, before the rest of the input. lexbor has no pause and no document.write and does not need them: we feed it slices that never run past a `</script>` end tag, and a small C shim (pkg/html/shim) wraps lexbor's token callback to learn which script element that end tag closed. `lava html [--scripts] <file|->` parses a file in 16 KiB pieces and prints the serialized document; --scripts reports each script where the parser pauses. Not done: charset sniffing (input is taken as UTF-8), fetching a URL, script execution, a DOM API. No tests: paused for this phase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t; hide lexbor From the quality and native-safety reviews of the first version: - A script was silently lost when an end tag had a quoted `>` in an attribute (`</script a=">">`): the slice ended early, the next one closed two scripts, and the hook kept only the last. The cut now skips quoted attribute values as the tokenizer does, and the shim counts closes: two in one slice is a failure, never a dropped script. - Consecutive writes came out reversed (each was inserted at the same point). Written text now has its own buffer and an insertion cursor that advances. - One unterminated `</script` made parsing quadratic and held back the rest of the page (16 MiB in 16 KiB reads: 3.17 s). The cut scan resumes where it stopped, compaction is amortized, and an unterminated candidate holds back at most 4 KiB. Same input now: 0.05 s. - lexbor no longer leaks into the API: the bindings are private, callers see Parser and Node. Off Linux, init fails with a message instead of a link error. - The feed loop moved from the CLI into pkg/html (parse_all); serialize returns a string; attr copies; one ASCII helper replaces the `| 0x20` folding that also matched 0x0F as '/'. - Docs: layering table, ARCHITECTURE 3.6, README build requirements, make help. FFI declarations stay hand-written rather than generated by odin-c-bindgen: a dozen procs, and everything lexbor exposes as inline functions, macros or struct fields goes through the shim, which a generator cannot bind. No tests: paused for this phase. Checked by hand: odin check on linux, windows and darwin targets; the three cases above through `lava html` and a probe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
Pushed 5b41d0a, which addresses a quality review and a native-safety review of the first commit. Bugs, each confirmed by the reviewer with a probe and re-checked by hand after the fix:
Two scripts closing in one slice is now a parse failure, never a dropped script. Structure:
Bindings stay hand-written, not generated with odin-c-bindgen: a dozen procs, and everything lexbor exposes as inline functions, macros or struct fields goes through the C shim, which a generator cannot bind. Not done from the reviews:
Still no tests or gates. |
Owner
Author
|
Pushed 87cf007, which closes the two items the reviews left open:
Still no tests or gates. |
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
pkg/htmlbinds lexbor and wraps it in an incrementalParser(#10):feed/finishtake network bytes;pumpparses until the next parser-blocking script, the end of the available input, or the end of the document;pumpreturns a script, the tree holds exactly what precedes it, as a browser's parser would have it;writeinsertsdocument.writetext at the insertion point, before the rest of the input, and scripts inside it pause in turn.lava html [--scripts] <file|->parses a file (or stdin) in 16 KiB pieces and prints the serialized document.--scriptsreports each script at the moment the parser pauses for it. No script is executed.How the pause works without changing lexbor
lexbor has no pause and no
document.write: its tree builder leaves both asTODOat</script>(html/tree/insertion_mode/text.c). It does not need them, because the caller holds the input:pumpfeeds lexbor slices that never run past the first>after a</script(any case);pkg/html/shim/larva_html.c) wraps lexbor's token callback and records the script element an end tag actually closed. A</script>inside a comment or a JS string is ignored, because lexbor never tokenizes it as that end tag.The shim also wraps the few lexbor calls that are inline functions or struct reads, so no lexbor struct layout is mirrored in Odin.
Dependency (§2)
lexbor, Apache-2.0, which fits both the AGPL core and the commercial extensions. Added to
NOTICE.e609d94and checked against the tarball's sha256..deps/byscripts/build-lexbor.sh, whichbuild-native-deps.shnow calls. Needscmakeand network on the first build.liblexbor-devis not installed on the dev machines or the CI image. Not vendored: the source is ~22 MB.Tried by hand
"</scr" + "ipt>", a</script>inside a comment, and an upper-case<SCRIPT>: three pauses, in order, none on the comment.en.wikipedia.org/wiki/HTML(803 KB): 5 pauses for the 5<scripttags in the source, 0.01 s and 24 MB max RSS (/usr/bin/time, one run on a 16-core dev machine).document.writepath was shown with a C prototype of the same feeding scheme; the CLI does not callwrite.Not done
odin check cmd/lavapasses andbin/lavabuilds.larva fetch), script execution, and a DOM API for JS.>in an attribute (</script x=">">) is cut too early and the pause lands late; documented inparser.odin..depson Linux/macOS; no lexbor build exists for Windows yet.🤖 Generated with Claude Code