feat: lazy-load node descriptions for md-folder documents - #82
Merged
Merged
Conversation
The md-folder backend (one .md file per node) read and parsed every node's full body into memory the instant a document was opened, and kept it there for the whole session — for a very large model, that's the dominant memory cost, paid up front regardless of how much of the model the user actually looks at. Second step (after #81) of scaling the model to very large sizes. deserializeFromMdFolder gains a `lazy` option: node metadata (frontmatter) still loads eagerly as before, but bodies are left unset and their file paths returned separately (bodyPaths) instead. diagramStore tracks which nodes are still pending (pendingBodyNodeIds) and hydrateNode fetches one node's body on demand (wired to the properties panel opening a node) via a new single-file read path for both Electron (reusing the existing file:read IPC) and web (a new readOneFileFromHandle in webFolder.ts). Nothing in the app tolerated a partially-loaded model before this: the Elements panel search and Quick Search (Cmd/Ctrl+P) both scan every node's description synchronously. Rather than build a permanent in-memory search index (which would just duplicate the same text and give up most of the memory win), the existing instant search keeps working as-is on label/type/ technology (already resident) and simply won't match text in an unopened node's body — degraded, not broken. A new explicit "Search descriptions" action (scanDescriptions) does an opt-in scan of only the still-pending nodes for a query; matches get merged into the live model so opening them next is instant, non-matches are read and discarded rather than cached. The one correctness hazard this had to avoid: saveDocument always writes every node's description back to disk, so serializing an unhydrated node's description (undefined) verbatim would silently wipe its saved content on the next save. Rather than a riskier byte-patching write path, saveDocument transiently re-reads any still-pending node's file just for the purposes of building the write payload — never merged into the live model, so a save can't defeat the laziness, and the serialize/write code path itself is completely unchanged. Scope is the `md` document source only — `fs` (single JSON file) and `ls` (localStorage) are inherently all-or-nothing blobs with nothing to split. The AI query layer (ai/queryLanguage.ts) still sees undefined descriptions for unhydrated nodes on a lazy doc; that's a known, deliberately deferred follow-up rather than something fixed here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tomasz-zajac-oss
pushed a commit
that referenced
this pull request
Sep 23, 2026
deserializeFromMdFolder now returns { data, bodyPaths? } since #82.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.mdfile per node) read and parsed every node's full body into memory the instant a document was opened and kept it there for the whole session, regardless of how much of the model was actually looked at.deserializeFromMdFoldergains alazyoption — node metadata (frontmatter) still loads eagerly, but bodies are left unset and their file paths returned separately (bodyPaths).diagramStoretracks pending nodes (pendingBodyNodeIds);hydrateNodefetches one node's body on demand, wired to the properties panel opening a node, via a new single-file read path for both Electron (reuses the existingfile:readIPC) and web (newreadOneFileFromHandleinwebFolder.ts).saveDocumentwrites every node's description back to disk on every save, so serializing an unhydrated node'sdescription(undefined) verbatim would silently wipe its saved content on the very next save. Rather than a riskier byte-patching write path,saveDocumenttransiently re-reads any still-pending node's file just to build the write payload — never merged into the live model, so a save can't defeat the laziness, and the serialize/write code path itself is unchanged.mddocument source only —fs(single JSON file) andls(localStorage) are all-or-nothing blobs with nothing to split.ai/queryLanguage.tsstill seesundefineddescriptions for unhydrated nodes on a lazy doc.Test plan
npm run typecheckpassesnpm run test— 379/379 tests pass, including:tests/lazyNodeHydration.test.ts(new) —loadDiagrammarks pending nodes correctly,hydrateNodemerges + clears pending without touching undo,scanDescriptionsfinds matches in unopened nodes and discards non-matchestests/mdFolderDocument.test.ts— updated round-trip test plus a new lazy round-trip safety test: load lazily, save without ever hydrating a node, reload, confirm its description survivedtests/mdFolderPersistence.test.ts— updated fordeserializeFromMdFolder's new{ data, bodyPaths }return shapenpm run dev:webon the default (non-lazy,ls) sample model: confirmed zero visual/behavioral regression — description renders normally (no "Loading…" placeholder), no stray "Search descriptions" affordance appears, no console errors. (The native "Open folder" OS dialog can't be driven by headless automation, so the md-folder lazy-load path itself is covered by the integration tests above rather than an end-to-end UI run.)🤖 Generated with Claude Code