WIP groundwork: flat token cursor for the parser [+0.3% as-is; needs lexer fusion]#57
Draft
xmakro wants to merge 5 commits into
Draft
WIP groundwork: flat token cursor for the parser [+0.3% as-is; needs lexer fusion]#57xmakro wants to merge 5 commits into
xmakro wants to merge 5 commits into
Conversation
…pre-flattened buffer Replaces the parser's tree-walking TokenCursor with a single flat, pre-materialized token buffer and an integer cursor: bump is an index increment, lookahead is direct indexing, parser snapshots and lazy token-stream replay share the buffer via two refcount bumps, and whole delimited sequences skip in one step through an open/close match table. Skipped invisible delimiters are retained as entries (filtered during consumption) so tree-level lookahead and nesting-depth queries keep their exact semantics; the buffer records per-entry depth for that. Correctness is fully validated: the compiler bootstraps itself and the complete ui suite passes with zero new diffs. Current placement regresses instruction counts (+0.29 percent geomean, syn check +0.96 percent): flattening at Parser construction adds an eager copy pass per parsed stream, where the old cursor was lazy, and every macro expansion output re-parse pays it. The profitable endgame, in order: (1) fuse flat emission into the lexer, which today produces tokens linearly and then builds the tree this change un-builds — that also deletes the per-delimited-group tree allocations; (2) make expansion-output re-parses flat-first as well, which is where derive-heavy crates spend most of their parsing. Groundwork PR; not for merging as is.
The lexer now lexes straight into the parser's flat token buffer (FlatEntry entries + open/close match table), never materializing the token tree it used to build: delimited groups become an open entry, the contents, and a close entry patched into the match table. This deletes the per-delimited-group tree allocations and the eager flatten pass at Parser construction for every primary parse (crate root, mod files, include!), which was the structural regression left by the previous commit. Depth and spacing bookkeeping reproduce FlatTokenCursor::new byte-for-byte; all delimiter-recovery diagnostics keep their exact behavior (every recovery path still ends in Err, so only the diagnostic side effects matter there). The few consumers that need a real TokenStream (proc-macro from_str, cmdline attrs, fake token streams for diagnostics) rebuild the tree from the flat buffer via flat_range_to_stream, which also replaces the parser's private rebuild_flat_subtree. Expansion-output re-parses still go through Parser::new -> FlatTokenCursor::new; making those flat-first is the next step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tt fragments splice as slices transcribe() now builds the expansion output directly in the parser's flat form through a FlatSink (open/close entries with match-table patching), so the result-stack machinery disappears and ParserAnyMacro re-parses expansion output with no flatten pass. Captured $x:tt fragments become FlatTt: a slice of the source buffer for delimited groups (two refcount bumps, spliced into the output by memcpy with depth/match rebasing) or a by-value token for single tokens, which may be unglued halves no slice can represent. The maybe_use_metavar_location share/respan heuristic carries over: splice verbatim on no-collision, otherwise rewrite the boundary delimiter spans after splicing. AST-fragment metavars emit their invisible delimiters straight into the sink and splice their token streams via the shared tree-flatten path (FlatTokenCursor::new is now a trivial wrapper over FlatSink::splice_stream). The rare mbe attr/derive rule paths materialize a tree at the boundary for now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…flat mbe matching TokenStream gains a second backing: a lazily materialized view of one nesting level of a flat token buffer (TokenStreamInner::Flat with a OnceLock for the trees). Macro-invocation DelimArgs are parsed as such views (two refcount bumps instead of rebuilding the tree), while attributes and macro-definition bodies stay eager so long-lived nodes do not pin token buffers. All tree-level accessors materialize transparently on first use, and equality/hash/encode/stable-hash operate on the materialized trees, so behavior and hashes are unchanged for every consumer that actually needs a tree. FlatTokenCursor gains an exclusive end bound (index/end shrunk to u32 to keep the cursor at 24 bytes) so a parser can run over a sub-range of a shared buffer and see Eof at the range end. The mbe matcher and Parser::new both recognize view-backed streams and parse straight from the buffer — the per-invocation flatten pass and the args tree rebuild are gone; the doc-comment desugaring pre-pass falls back to the eager path in the rare case a doc comment appears in macro arguments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dents scan Two findings from adversarial review of the lazy-view commits: The derived Debug on the view-backed TokenStream (and FlatTokenSlice) printed the entire underlying buffer — a whole crate's tokens — for every unexpanded macro call's arguments, making -Zunpretty=ast-tree quadratic in crate size (2.4MB -> 1.49GB on 400 invocations). Manual impls now print only the viewed range, in the pre-existing format. The pre-expansion KeywordIdents lint iterated every macro invocation's argument stream, silently materializing the tree the lazy views exist to avoid — the optimization only survived for macro-generated invocations. check_tokens now scans view entries in place; delimiter entries reset the dollar state exactly like the tree walk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Strategic rearchitecture groundwork, benchmarked and banked rather than merged.
What it does: replaces the parser's tree-walking TokenCursor with a flat pre-materialized token buffer + integer cursor. bump = index increment; look_ahead = direct indexing (no cursor clone even for deep lookahead); Parser::clone (mbe black-box, ~30 diagnostic snapshots) = two refcount bumps instead of a stack Vec clone; lazy token-stream capture replay reads a shared slice; big-tt skips are O(1) via an open/close match table; Parser shrinks by 16 bytes. Skipped invisible delimiters stay in the buffer as entries (filtered on consumption) so tree-level lookahead and depth queries keep byte-exact semantics.
Correctness: bootstraps the compiler and passes the full ui suite with zero new diffs — the risky part of the rearchitecture is retired.
Performance as placed today: +0.29% geomean (syn check +0.96%). Root cause is structural: the tree is still built by the lexer and then flattened at Parser construction (an extra eager copy pass the lazy cursor never paid), and every expansion-output re-parse pays it too, which is most of the parsing in derive-heavy crates.
The profitable path, in order:
Measurement: ThinLTO stage2, glibc, instructions:u, full scenario, measured on top of the whole perf stack.