feat(sync): schema v3 nested Y.Map metadata — lazy migration#52
Open
kavinsood wants to merge 1 commit into
Open
feat(sync): schema v3 nested Y.Map metadata — lazy migration#52kavinsood wants to merge 1 commit into
kavinsood wants to merge 1 commit into
Conversation
Introduces schema v3 metadata model: file metadata entries are written as nested Y.Maps instead of opaque JSON objects, giving field-level CRDT resolution and eliminating whole-object tombstones on mtime updates. ## Core changes ### src/sync/fileMeta.ts (new) Unified dual-shape helper module for v2 (flat) and v3 (nested Y.Map) metadata. Provides type-safe decoders, read helpers, write helpers, lazy conversion, incremental diff, and semantic change types. Single authoritative interface — no call site accesses metadata directly. ### src/sync/schema.ts (new) Pure, Obsidian-free SCHEMA_VERSION = 3 constant. Importable in tests without dragging in the obsidian dependency. ### Metadata model - All writes produce nested Y.Maps via ensureNestedMetaEntry + create helpers - Reads dual-decode both flat (v2) and nested (v3) shapes everywhere - Lazy on-write conversion: untouched flat entries remain flat indefinitely - No eager migration, no distributed migration storm - sys.schemaVersion bumped to 3 on first v3 client connect (markSchemaV3) - migrateSchemaToV2 reverted to write flat v2 objects (was incorrectly writing nested maps, creating v3 shapes under v2 schema marker) ### Semantic observer (observeMetaChanges) Single shared meta.observeDeep handler on VaultSync. Uses event paths for O(k) incremental diff instead of O(N) full snapshot on every change. Dispatches MetaChangeBatch with origin + isLocal to all subscribers. DiskMirror and witness tracker consume semantic changes directly. ### DiskMirror - Replaced shallow meta.observe with observeMetaChanges subscription - Correctly handles nested field mutations (deletedAt, path, mtime) - Skips local-origin batches (isLocal=true) to prevent local writes feeding back as remote file operations - Suppresses disk.rename.observed remoteOrigin flag via _pendingRemoteRenameNewPaths when handleRemoteRename runs - Normalizes all paths from semantic change events before disk ops ### Server - SERVER_MIN/MAX_SCHEMA_VERSION bumped to 3 - countActivePathsInDoc / computeDocStats dual-read v2 and v3 metadata - documentSummary debug response includes flatMetaEntries, nestedMetaEntries, invalidMetaEntries shape counters - readMetaPath / isMetaDeleted helper methods for dual-shape reads ### Analyzer (orphan-after-rename rule) New remoteOrigin exemption: disk.rename.observed events with remoteOrigin:true (set by main.ts when DiskMirror's pending rename set contains the new path) are not flagged as orphans. Passive receiver devices correctly produce disk renames without CRDT rename events — the CRDT rename was initiated by the other device. ## Tests ### New test suites (430 assertions across 10 suites): - tests/file-meta-decode.ts — 112: decoder, type guards, helpers - tests/file-meta-lazy-write.ts — 34: no-storm proof, concurrent convergence - tests/meta-observer-integration.ts — 40: nested mutations fire semantic changes; origin filtering proven local vs remote; incremental diff correct - tests/meta-v3-schema-gate-and-stats.ts — 47: schema gate imports real server constants; mixed metadata stats; realistic vault round-trip - tests/meta-diskmirror-integration.ts — 52: real DiskMirror integration with spied handlers; proves remote nested delete/rename/revive trigger correct disk ops; proves local changes are ignored ### Updated: - tests/disk-mirror-observer.ts: added observeMetaChanges to fakeVaultSync - tests/v2-offline-rename-regressions.mjs: use getMetaPath/getMetaDeletedAt helpers (restore now writes nested Y.Maps, flat property access broke) - tests/run-regressions.mjs: added meta-diskmirror-integration to suite ## QA scenario (S15) Two-vault CDP scenario on ~/temenos + ~/temenos-b against the deployed kavin-yaos.ripplor.workers.dev server (SQL storage, schema v3): Phase 1 create: hash match ✓ (19037d3bbde3) Phase 2 rename: hash match ✓ (dc952595d28f), old path gone ✓ Phase 3 delete: file gone on B ✓ Phase 4 revive: hash match ✓ (7593fdbb0b82) Phase 5 mtime-only: B disk hash unchanged (f8f98eccff4a = f8f98eccff4a) ✓ Phase 6 schema: schemaVersion A=3 B=3 ✓ Both analyzer passes: 0 hard failures. Exit 0. Server post-run: flatMeta=1403, nestedMeta=5 (only touched entries converted). Full CI: npm ci + npm ci --prefix server + npm run build + npm run test:ci + npm run test:regressions (73 suites) + npm --prefix server run typecheck — all clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces YAOS schema v3 metadata storage.
File metadata is now written as nested Yjs maps instead of opaque JSON objects. This gives field-level CRDT behavior for metadata updates — renaming a file on device A while device B updates the mtime no longer causes one device's change to silently overwrite the other. mtime-only saves no longer generate a whole-object tombstone for every file save.
The migration is lazy: existing flat v2 metadata remains readable indefinitely and is converted one entry at a time when touched by v3 writes. There is no full-vault metadata migration.
Dependency: This PR targets
fix/sql-storage-migration(PR #51). It should only be merged after that PR is merged and baked. After that merge, this branch rebases ontomain— it is a single clean commit.What changed
Data model
meta[fileId]entries are nowY.Map { path, mtime?, device? }instead of opaqueFileMetaJSON objectsY.Map { path, deletedAt }— same minimal shape, nestedsys.schemaVersionbumped to 3 on first v3 client connect — no metadata loopNew
src/sync/fileMeta.tsSingle authoritative interface for reading and writing metadata. Dual-shape decoders, read helpers (
getMetaPath,getMetaMtimeetc.), write helpers (createNestedActiveMeta,ensureNestedMetaEntry), incremental semantic diff, and shape statistics. No call site accesses metadata entries directly.New
src/sync/schema.tsSCHEMA_VERSION = 3in a pure, Obsidian-free module. Importable in tests and server code without the Obsidian dependency.Semantic observer (
observeMetaChanges)Single shared
meta.observeDeephandler onVaultSync. Uses Yjs event paths for O(k) incremental diff (k = affected file IDs) instead of O(N) full-map scan on every metadata change. DispatchesMetaChangeBatch { origin, isLocal, changes }to consumers.DiskMirror and the witness tracker subscribe to
observeMetaChanges— they no longer use shallowmeta.observe(), which would have silently dropped all nested field mutations.DiskMirror
observeMetaChangesfor remote nested delete/rename/reviveisLocal: true) — local metadata writes never feed back as remote file operationsconsumeRemoteRename(newPath): boolean— consume-on-use pattern (matches existingconsumeDeleteSuppression) marks rename as DiskMirror-originated beforeapp.fileManager.renameFileis called, consumed by the vault rename handlerqueueRenamewhenconsumeRemoteRenamereturns true — passive receiver renames never re-enter CRDTServer
SERVER_MIN/MAX_SCHEMA_VERSIONbumped to 3 — old v2 clients rejecteddocumentSummarydebug response includesflatMetaEntries,nestedMetaEntries,invalidMetaEntriesshape counterscountActivePathsInDoc/computeDocStatsdual-read both v2 and v3 shapes — server can cold-boot a v2 persisted room safelyAnalyzer
orphan-after-renamerule gained aremoteOriginexemption: passive receiver devices callhandleRemoteRenamewhich triggers a disk-level rename, emittingdisk.rename.observed. This correctly has nocrdt.file.renamedcounterpart (the CRDT rename was on the other device). TheremoteOrigin: trueflag in the event data exempts it.Tests
file-meta-decode.tsfile-meta-lazy-write.tsmeta-observer-integration.tsmeta-v3-schema-gate-and-stats.tsmeta-diskmirror-integration.tsQA scenario — S15
Two-vault CDP scenario on
~/temenos+~/temenos-bagainst the production server (SQL storage, schema v3):Both analyzer passes: 0 hard failures. Exit 0.
Server post-run:
flatMeta: 1403,nestedMeta: 5— exactly the 5 files touched during the scenario lazily converted. 1,403 entries remain flat.Safety
update_requiredbefore the room DO is wokenMerge order
fix/sql-storage-migration) and bakemain(single commit, trivial)Review checklist
.path,.mtime,.deletedAtaccess outside helper/testsentry.set(key, undefined)meta.observe()dependency for nested changes