Skip to content

Port upstream 4942: drive the token walk with a linear walker#159

Open
Renakoni wants to merge 1 commit into
mainfrom
sync/upstream-4942-linear-token-walk
Open

Port upstream 4942: drive the token walk with a linear walker#159
Renakoni wants to merge 1 commit into
mainfrom
sync/upstream-4942-linear-token-walk

Conversation

@Renakoni

Copy link
Copy Markdown
Owner

Ports the remaining piece of upstream marktext#4942 (the algorithmic half of the marktext#4887 large-file freeze fix).

Why only this piece

The vendored Muya already carries the rest of #4942 from the earlier large-file investigation: cloneStateTree (schema-specific clone replacing structuredClone), the revision-keyed reference-definition cache in InlineRenderer, and a json-change payload with no eager doc clone at all (more aggressive than upstream's lazy getter — nothing in this app reads it). The one piece still missing was the token-walk driver, and it is the dominant remaining parse cost.

What changes

lexBlock drove muya's token augmentation through Marked.walkTokens, whose driver concats every callback result into one accumulator array — cloning that array once per token, O(tokens²) on large documents. Our callbacks return nothing, so the accumulation is pure waste. A local depth-first walker with the same visit order (tables' header/row cells, list items, generic child tokens) replaces it.

This is the same hotspot the on-device profile pointed at (~19.7 s inside walkTokens for a ~2 MB document on Android).

Timing on the same synthetic shape used upstream (Node, this machine; on-device is strictly slower):

document Marked.walkTokens alone full lexBlock with the linear walker
835 KB (8k sections: heading + paragraph + list) 18.9 s 0.117 s

Verification

  • New walkTokensLinear.spec.ts (ported from upstream): identical visit coverage vs Marked.walkTokens on nested constructs (table cells, nested lists, blockquote children), and augmentation (headingStyle, codeBlockStyle) reaching tokens only visible through child arrays.
  • Muya app-gate suite: 1445/1445.
  • Root unit tests (incl. the vendor-sync contract): 577/577; node_modules/@muyajs/core resynced.
  • pnpm typecheck, pnpm lint (tracked code), pnpm build all clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant