Skip to content

feat(documents): two-way whole-file mirror engine and the FileTarget port - #3116

Open
leemour wants to merge 2 commits into
mainfrom
feat/file-mirror
Open

leemour wants to merge 2 commits into
mainfrom
feat/file-mirror

Conversation

@leemour

@leemour leemour commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Task

Step 1 of the ydisk two-way folder sync: a whole-file mirror engine in cli-core, with the FileTarget port ydisk will implement for the Disk.

Why

drive's pushDocuments is local-first, writes text only, reads the remote once per file and only reports deletions. A file store needs bytes both ways, one listing per run, deletions into trashes and renames by id. Rewriting pushDocuments would put drive's Docs/Sheets merge at risk, so the file engine is new and shares the layer around it.

What

  • syncFileMirror (src/services/files/mirror.ts): three-way comparison per path (local now, remote now, last sync).

    • The same content on both sides is adopted. A first sync of identical trees moves nothing.
    • Different content on both sides is a conflict and is never overwritten. keep settles it: local, remote, or both, which downloads the remote version as name (conflict YYYY-MM-DD <remote>).ext and sends both.
    • Deletions go to the remote trash (with the expected sha256/md5) or to <root>/.wirecat-trash/YYYY-MM-DD/. A guard stops the run before any write at more than 50 deletions or 25% of the mirror, unless allowDeletions covers it. A dry run reports tooManyDeletions.
    • A remote rename, found by id, renames the local file. A local rename is a deletion plus a new file.
    • Every write is conditional on the listing. A crash at any step is settled by the next listing: same content is adopted, and a file gone from both sides is forgotten. outcome_unknown is not retried within the run.
    • Uploads and downloads stream. A download whose hash differs from the listing never lands.
    • A dry run writes nothing on either side.
  • A deletion must be proven. A file counts as deleted only when all of these hold:

    • the last sync saw it;
    • that side's listing is complete (list returns complete);
    • that side's root is the same folder: the remote root id matches, and the local folder has the recorded device and inode and is not empty while the state has files.

    Anything else is reported in unknown, nothing is deleted for it, and the run is partial. A broken root stops the whole run.

  • Local versions (src/services/files/versions.ts). fileVersions(dir) keeps a local file's content before the sync overwrites or deletes it. The content is stored by sha256 under the profile's state folder, and every change is added to an append-only log. An upload logs the remote version it replaced. Retention: versions stay 30 days, and the last version of a deleted file stays for good. restoreFileVersion puts a version back.

  • FileTarget (src/documents/file-mirror-model.ts): identity, list(root) → { rootId, files, complete } | undefined, download(path, into), upload(path, from, expect), trash(path, expect), ensureFolder(path), revision?().

  • Also exported: syncFileMirrors (the lock and mirror.partial, as syncMirrors does), fileSyncState, fileMirrorStatus, and fileRemoteChanges, which lists only when the remote revision moved.

  • Shared changes. drive's behaviour is unchanged.

    • jsonState and eachMirror are pulled out of fileMirrorState and syncMirrors.
    • scanLocal reports size and mtimeMs, and reuses a known hash when both match.
    • replaceLocal takes bytes or a stream writer and an expected content hash, and can create a file only while it is absent.
    • watchMirrors takes localChanges.
    • The guard's action gains delete and move.
    • conflictReport adds sizes and leaves out an empty url.
  • Docs: docs/dev/DOCUMENTS.md, "Plain files, both ways". Changelog fragment: changes/feat-file-mirror.md.

Testing

  • src/services/files/mirror.test.ts (58 tests). Besides the cases below, it covers deletion proof (incomplete listing, failed listing, a local root that is gone, empty or replaced, a remote root that is gone or replaced, a file never synced, the partial issue) and versions (a pull, keep: "remote", a crossed deletion, an upload's replaced version, restore, retention, the dry run).
  • The same file runs on a temporary folder, against a fake remote that keeps content on disk and is strict about expect. It covers:
    • every row of the three-way table, and the first sync;
    • keep local, remote and both;
    • both trashes and the deletion guard;
    • renames on both sides;
    • a crash before and after each kind of write;
    • outcome_unknown followed by adopt;
    • read-back mismatches;
    • binary files and 0-byte files;
    • a 50 MB file streamed both ways, never read whole;
    • the dry run and the revision poll.
  • mirror-watch.test.ts gains a localChanges case.
  • The existing mirror, push and sync tests pass unchanged.
  • Locally: typecheck, lint, docs:check and the full vitest suite. One unrelated test, notes-meaning-filters, timed out under load and passes alone.

🤖 Generated with Claude Code

leemour and others added 2 commits November 18, 2026 19:01
…port

A CLI for a file store (ydisk next) can keep a local folder and a remote
folder the same, whole files at a time. drive's pushDocuments is local-first,
text-only and reads the remote once per file, so the file engine is new and
shares the layer around it.

- syncFileMirror: three-way comparison per path (local now, remote now, last
  sync) over one remote listing; same content is adopted, different changes
  are conflicts settled by keep local|remote|both (both keeps a dated copy);
  deletions go to the remote trash (expected sha256/md5) or
  <root>/.wirecat-trash/, behind a 50-file / 25% guard; a remote rename found
  by id renames the local file; every write is conditional, so a crash at any
  step is settled by the next listing; outcome_unknown is never retried in
  the run; downloads and uploads stream; a dry run writes nothing.
- FileTarget port (list with root id, download, upload, trash, ensureFolder,
  optional revision), fileSyncState, fileMirrorStatus, syncFileMirrors and
  fileRemoteChanges, which lists only when the remote revision moved.
- Shared changes: jsonState and eachMirror extracted from fileMirrorState and
  syncMirrors; scanLocal reports size and mtime and reuses a known hash;
  replaceLocal takes bytes or a stream writer, an expected content hash, and
  creates only while absent; watchMirrors takes localChanges; the guard's
  action gains delete and move; conflictReport adds sizes and drops an empty
  url. drive's behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocal versions

Owner rulings on #3116.

- A file counts as deleted only when the last sync saw it, the side's
  listing is complete (FileTarget.list returns `complete`), and the side's
  root is the same folder: the remote root id, the local folder's device and
  inode, and a local folder that is not empty while the state has files (an
  unmounted drive). Anything else is reported in `unknown` and the run is
  partial; a broken root makes the run do nothing.
- fileVersions: before a sync overwrites or deletes a local file, its content
  is kept by sha256 under the profile's state folder, with an append-only
  log per mirror of every change both ways (an upload logs the remote
  version it replaced). Retention: 30 days, and a deleted file's last
  version for good. restoreFileVersion puts one back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant