Port upstream 4942: drive the token walk with a linear walker#159
Open
Renakoni wants to merge 1 commit into
Open
Port upstream 4942: drive the token walk with a linear walker#159Renakoni wants to merge 1 commit into
Renakoni wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the remaining piece of upstream marktext#4942 (the algorithmic half of the marktext#4887 large-file freeze fix).
Why only this piece
The vendored Muya already carries the rest of #4942 from the earlier large-file investigation:
cloneStateTree(schema-specific clone replacingstructuredClone), the revision-keyed reference-definition cache inInlineRenderer, and ajson-changepayload with no eagerdocclone at all (more aggressive than upstream's lazy getter — nothing in this app reads it). The one piece still missing was the token-walk driver, and it is the dominant remaining parse cost.What changes
lexBlockdrove muya's token augmentation throughMarked.walkTokens, whose driverconcats every callback result into one accumulator array — cloning that array once per token, O(tokens²) on large documents. Our callbacks return nothing, so the accumulation is pure waste. A local depth-first walker with the same visit order (tables' header/row cells, list items, generic childtokens) replaces it.This is the same hotspot the on-device profile pointed at (~19.7 s inside
walkTokensfor a ~2 MB document on Android).Timing on the same synthetic shape used upstream (Node, this machine; on-device is strictly slower):
Marked.walkTokensalonelexBlockwith the linear walkerVerification
walkTokensLinear.spec.ts(ported from upstream): identical visit coverage vsMarked.walkTokenson nested constructs (table cells, nested lists, blockquote children), and augmentation (headingStyle,codeBlockStyle) reaching tokens only visible through child arrays.node_modules/@muyajs/coreresynced.pnpm typecheck,pnpm lint(tracked code),pnpm buildall clean.