Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions changes/feat-file-mirror.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
### Added

- **`syncFileMirror` and the `FileTarget` port in `@wirecat/cli-core/documents`: two-way sync of plain files.** A CLI
for a file store (Yandex Disk next) keeps a local folder and a remote folder the same, whole files at a time,
binary and large files included, streamed both ways. A change on one side is copied, the same content on both is
adopted, different changes are a conflict that `keep` (`local`, `remote`, `both`) settles; deletions go to the
remote's trash and to `<mirror>/.wirecat-trash/`, and a run that would delete more than 50 files or a quarter of the
mirror stops first. Every write is conditional on the last listing, so a crash at any step is settled by the next
run. `syncFileMirrors`, `fileSyncState`, `fileMirrorStatus` and `fileRemoteChanges` (the watch lists only when the
remote's revision moved) complete it ([DOCUMENTS.md](../docs/dev/DOCUMENTS.md#sync-plain-files-with-a-provider)).
- **A file mirror deletes only what it can prove was deleted.** The last sync saw the file, the side's listing is
complete, and the side's root is the same folder (the remote root's id; the local folder's device and inode, and
not empty while the state has files, as an unmounted drive is). Anything else is reported as `unknown` and the run
is partial.
- **`fileVersions`: a local version store for mirrors.** Before a sync overwrites or deletes a local file, its content
is kept by sha256 under the profile's state folder, with an append-only log of every change both ways; versions stay
30 days, and a deleted file's last version stays for good. `restoreFileVersion` puts one back.
- **`watchMirrors` takes `localChanges` instead of `kindOf` and `stateOf`**, for an engine with its own state. A
document mirror passes `kindOf` and `stateOf` as before.
- **`scanLocal` reports each file's `size` and `mtimeMs`** and takes an optional lookup that reuses a known hash when
both match, so a sync does not read unchanged files again. `conflictReport` adds `local_size` and `<remote>_size`
when a conflict has them. A mirror guard's `action` can also be `delete` and `move`.

### Changed — may break callers

- **`PushConflict.url` is optional, and `MirrorWatchOptions` is a type, not an interface.** A file conflict has no
URL, so `conflictReport` leaves `url` out when there is none; a document conflict still has one. Code that reads
`conflict.url` as a string, or extends `MirrorWatchOptions` with `interface … extends`, needs a small change. drive
needs none.
75 changes: 75 additions & 0 deletions docs/dev/DOCUMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,81 @@ for file events the system dropped; and all of them at the full run (`fullMs`, 3
late. It yields one event per cycle with its triggers and mirrors, and a failed cycle as an error whose mirrors are
tried again at the next poll. The CLI takes the lock inside `sync`, so a manual sync can run between cycles.

<a id="sync-plain-files-with-a-provider"></a>

### Plain files, both ways

A remote folder that keeps files as they are (Yandex Disk, a WebDAV share) uses the whole-file engine,
`syncFileMirror`, instead of `pushDocuments`: it copies bytes, never merges text, and reads the remote with one
listing per run instead of one read per file. The provider implements `FileTarget`:

```ts
interface FileTarget {
identity(): Promise<string>
list(root: string, signal?: AbortSignal): Promise<{ rootId: string; files: RemoteFile[]; complete: boolean } | undefined>
download(path: string, into: string, signal?: AbortSignal): Promise<void>
upload(path: string, from: string, expect: Expect, signal?: AbortSignal): Promise<RemoteFile>
trash(path: string, expect: { sha256: string; md5?: string }, signal?: AbortSignal): Promise<void>
ensureFolder(path: string, signal?: AbortSignal): Promise<void>
revision?(signal?: AbortSignal): Promise<string>
}
type Expect = { absent: true } | { sha256: string; md5?: string }
```

`list` returns each file's path relative to `root`, its `id` (kept across a move and an overwrite), `size`, `sha256`
(the plain hash of the content), `md5` and `modifiedAt`, plus the root folder's own id and `complete`, false when a page was skipped or a cap was hit. The other calls take the
remote path, `root` joined with the relative path. `download` and `upload` stream a local file, so a file of any
size takes the same memory. Every write is conditional on what the listing saw: `upload` and `trash` refuse with a
`conflict` error when the remote file is not as `expect` says; a provider without an atomic check reads the file
just before the write. An editing command reuses the same call, `upload(path, file, { sha256: whatWasDownloaded })`.

1. Each path is compared three ways: the local file now, the remote file now and the last sync (`FileEntry` in the
state). A change on one side is copied to the other. The same content on both sides is adopted without a
transfer, so a first sync of two identical trees moves nothing. Different content on both sides, or a deletion
on one side against a change on the other, is a conflict: never overwritten, reported with each side's time and
size, and the run ends with a `mirror.partial` issue. `keep` settles conflicts on a later run: `local` and
`remote` pick a side; `both` downloads the remote version beside the local one as
`name (conflict YYYY-MM-DD <remote>).ext`, then sends both.
2. A deletion crosses sides into a trash: the remote's (`trash`, with the expected hashes) or
`<root>/.wirecat-trash/YYYY-MM-DD/`, which is hidden and so never synced. A run that would delete more than 50
files, or more than a quarter of the mirror, stops before any write with `mirror.too-many-deletions` unless
`allowDeletions` covers the count.
3. A remote file found at a new path with the same `id` is renamed locally. A local rename is a deletion and a new
file.
4. A local file whose size and mtime match the state is not hashed again (`scanLocal`'s `known` lookup). A local
write goes through a hidden temporary file and lands only if the file still has the hash the run read, or is
still absent for a new file; a download whose content does not match the listing's `sha256` never lands. An
upload that reads back with another hash is a conflict.
5. A crash at any step is settled by the next run's listing: the same content on both sides is adopted, a file gone
from both is forgotten, so nothing is written twice and nothing is lost. `pending` in the state only names a
download's temporary file, removed at the next start, and a write that ended `outcome_unknown`, which is never
retried within the run. The state is saved at most once a second and at the end.
6. A file counts as deleted on a side only when the last sync saw it, the side's listing is complete, and the side's
root is the same folder: the remote root has the id it had at the first sync; the local root exists, is a folder
with the device and inode recorded at the first sync, and is not empty while the state has files (an unmounted
drive reads as empty). A file missing from an incomplete listing is reported in `unknown` and nothing is deleted
for it. When a root fails, the run does nothing and reports every file of the state in `unknown`. Either way the
run is partial. A file the last sync did not see is never a deletion. Hidden remote files are skipped as local
ones are.
7. `dryRun` lists the remote and returns the `plan`; it writes nothing on either side and saves no state.

`versions` (`fileVersions(dir)`, with `dir` under the profile's state folder, one per mirror, never inside the
mirrored folder) keeps the content of every local file before the sync overwrites or deletes it (a pull, `keep:
"remote"`, a crossed deletion) in `<dir>/objects/<sha256>`, and appends each change to `<dir>/log.jsonl`: path,
event (`created`, `changed`, `deleted`, `renamed`), direction, old and new hash, size and time. An upload logs the
remote version it replaced as `old`. Retention is one rule, applied after each sync: a content stays while a log
event newer than 30 days names it, and the last content of a path whose newest event is a deletion stays for good.
`versions.list(path)` lists a path's events with `available` (bytes still kept); `restoreFileVersion(root, path,
sha256, versions)` puts one back, keeping the current file first, and the next sync sends it.

`guard` wraps every write with its `action` (`create`, `update`, `folder`, `delete`, `move`) and `side` (`local` or
`remote`), so the CLI maps overwrites and both trashes to its delete permission. `syncFileMirrors` runs the engine
for each mirror under one lock, as `syncMirrors` does; `fileSyncState(dir, mirror, principal)` keeps the state in
`<dir>/<id>.json`; `fileMirrorStatus` lists local changes and deletions without asking the remote. For
`watchMirrors`, pass `localChanges` (the mirrors whose `fileMirrorStatus` lists anything) instead of `kindOf` and
`stateOf`, and `remoteChanges: fileRemoteChanges(target)`: it reads the target's `revision` at each poll and lists
only when it moved; a target without one is listed at every poll.

For documents with structure, `markdownBlocks` (async; the parser loads on first use) parses Markdown into `Block`s
(headings, paragraphs, bullet, numbered and checklist items, code, quotes, tables, images, rules) with inline marks.
The provider maps its own document into the same blocks and writes only `diffBlocks(current, next)`, so unchanged
Expand Down
90 changes: 90 additions & 0 deletions src/documents/file-mirror-model.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
/** A file in the remote folder, as one listing returns it. */
export interface RemoteFile {
/** Relative to the listed folder, with `/` separators. */
path: string
/** Stays the same across a move or an overwrite, so a remote rename is found by it. */
id: string
size: number
/** The plain sha256 of the content. A file without one (still being processed) is left for the next run. */
sha256?: string
md5?: string
modifiedAt: string
}

/**
* What a remote write expects to find at the path, from the last listing: nothing, or the file with these hashes.
* A target that cannot check the condition atomically checks it just before the write.
*/
export type Expect = { absent: true } | { sha256: string; md5?: string }

/**
* A remote folder of plain files. Writes take the remote path (`root` joined with the relative path) and refuse with
* a `conflict` CliError when the file is not what `expect` says.
*/
export interface FileTarget {
identity(): Promise<string>
/**
* Every file under `root`, with the root folder's own id; undefined when the folder does not exist. `complete` is
* false when any page was skipped or a cap was hit: a file missing from such a listing is not taken as deleted.
*/
list(
root: string,
signal?: AbortSignal,
): Promise<{ rootId: string; files: RemoteFile[]; complete: boolean } | undefined>
/** Streams the file into `into`, a new local file. */
download(path: string, into: string, signal?: AbortSignal): Promise<void>
/** Streams the local file `from` to `path` and returns the file as the remote now has it. */
upload(path: string, from: string, expect: Expect, signal?: AbortSignal): Promise<RemoteFile>
/** Moves the file to the remote's trash, where the owner can restore it. */
trash(path: string, expect: Exclude<Expect, { absent: true }>, signal?: AbortSignal): Promise<void>
/** Creates the folder and its parents; a folder that exists is fine. */
ensureFolder(path: string, signal?: AbortSignal): Promise<void>
/** A value that moves with every change on the remote, when the provider has one: a watch reads it before listing. */
revision?(signal?: AbortSignal): Promise<string>
}

/** The last sync of one path: the base of the three-way comparison. */
export interface FileEntry {
id: string
size: number
sha256: string
md5?: string
remoteModified: string
localMtimeMs: number
localSize: number
syncedAt: string
}

export interface FileMirrorState {
version: 1
principal: string
root: string
target: string
/** Set at the first sync; a later listing with another id means the remote folder was moved or replaced. */
rootId?: string
/** The local folder's device and inode at the first sync: another folder at the same path is not this mirror. */
localRoot?: { dev: number; ino: number }
entries: Record<string, FileEntry>
/**
* Saved before each write. After a crash the next listing settles the write (same content on both sides is
* adopted); this only names the temporary file to remove and the operation to look up.
*/
pending: Record<string, { op: FileStepKind; operationId: string }>
}

export interface FileMirrorStatePort {
read(): FileMirrorState
save(state: FileMirrorState): void
}

export type FileStepKind =
| "upload"
| "download"
| "adopt"
| "trash-remote"
| "trash-local"
| "rename-local"
| "keep-both"
| "forget"
| "conflict"
| "unknown"
30 changes: 30 additions & 0 deletions src/documents/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ export { documentIngestion, type IngestionOptions } from "../services/documents/
export { type DocumentKnowledge, documentKnowledge } from "../services/documents/knowledge.js"
export { blocksMarkdown, inlineMarkdown, spliceMarkdown } from "../services/documents/markdown-splice.js"
export {
type MirrorLocalCheck,
type MirrorTrigger,
type MirrorWatchEvent,
type MirrorWatchOptions,
Expand All @@ -28,6 +29,7 @@ export {
export type { SyncOptions } from "../services/documents/ports.js"
export { extractDocumentMemories, type ImportedSource, type ProposalOptions } from "../services/documents/proposals.js"
export {
type MirrorAction,
type PushConflict,
type PushOptions,
type PushResult,
Expand All @@ -36,8 +38,36 @@ export {
scanLocal,
} from "../services/documents/push.js"
export { type SyncResult, syncDocuments } from "../services/documents/sync.js"
export {
type FileMirrorOptions,
type FileMirrorResult,
type FileMirrorSyncOptions,
type FileStep,
fileMirrorStatus,
fileRemoteChanges,
fileSyncState,
syncFileMirror,
syncFileMirrors,
} from "../services/files/mirror.js"
export {
type FileVersion,
type FileVersions,
fileVersions,
restoreFileVersion,
type VersionDirection,
type VersionEvent,
} from "../services/files/versions.js"
export { type Block, blockKey, blockShape, type Inline, type ListKind, normalizeInlines } from "./blocks.js"
export { catalogFiles } from "./catalog.js"
export type {
Expect,
FileEntry,
FileMirrorState,
FileMirrorStatePort,
FileStepKind,
FileTarget,
RemoteFile,
} from "./file-mirror-model.js"
export { htmlToMarkdown, markdownToHtml } from "./html-markdown.js"
export { markdownBlocks, type PlacedBlock, placedMarkdownBlocks } from "./markdown-blocks.js"
export type {
Expand Down
3 changes: 3 additions & 0 deletions src/documents/push-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ export interface LocalDocument {
absolute: string
hash: string
kind: string
/** Set by `scanLocal`, so the next scan can skip hashing a file whose size and mtime did not change. */
size?: number
mtimeMs?: number
}
export interface Pushed<S> {
id: string
Expand Down
7 changes: 7 additions & 0 deletions src/errors/reasons.ts
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,13 @@ const OTHERS: Entry[] = [
["mirror.none", "not_found", "the profile has no mirrors; add one with mirrors add"],
["mirror.not-found", "not_found", "no such mirror in this profile; see mirrors list"],
validation("mirror.not-folder", "the path to mirror is not a folder"),
[
"mirror.too-many-deletions",
"confirmation_required",
"the sync would delete more files than the guard allows; check them with --dry-run and allow them explicitly",
],
["upload.mismatch", "conflict", "the uploaded file reads back with other content; the local file changed meanwhile"],
["download.mismatch", "invalid_response", "the downloaded file has other content than the listing said"],
["catalog.unavailable", "configuration_error", "the store has no shared catalog for this account"],
["catalog.not-found", "not_found", "the file or folder is not in the local catalog; run catalog import or sync"],
validation("catalog.depth-invalid", "tree depth must be between 0 and 50"),
Expand Down
22 changes: 19 additions & 3 deletions src/services/documents/mirror-watch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ const mirrorAt = (id: string, files: Record<string, string>): Mirror => {
* A watch on a clock that jumps to each wake-up. `at(ms, act)` runs `act` once the clock passes `ms`; the watch stops
* after `cycles` cycles. The sync records the local files as pushed, as a real sync leaves them.
*/
const harness = (mirrors: Mirror[], cycles: number) => {
const harness = (mirrors: Mirror[], cycles: number, localChanges?: (mirrors: Mirror[]) => Promise<string[]>) => {
let clock = 0
const stop = new AbortController()
const stateDir = mkdtempSync(join(tmpdir(), "mirror-watch-state-"))
Expand Down Expand Up @@ -54,8 +54,7 @@ const harness = (mirrors: Mirror[], cycles: number) => {
const options = {
mirrors,
signal: stop.signal,
kindOf,
stateOf,
...(localChanges ? { localChanges } : { kindOf, stateOf }),
remoteChanges: async () => {
const found = remote
remote = []
Expand Down Expand Up @@ -180,6 +179,23 @@ describe("watchMirrors", () => {
expect(watch.clock()).toBe(1_800_000)
})

it("asks the engine's own check, when given one, whether saved files differ", async () => {
const a = mirrorAt("a", { "one.md": "one" })
const b = mirrorAt("b", { "two.md": "two" })
const asked: string[][] = []
const watch = harness([a, b], 2, async (chosen) => {
asked.push(chosen.map((mirror) => mirror.id))
return chosen.filter((mirror) => mirror.id === "b").map((mirror) => mirror.id)
})
watch.at(1_000, () => watch.save(a, "one.md", "one, edited"))
watch.at(10_000, () => watch.save(b, "two.md", "two, edited"))
expect(await watch.run()).toEqual([
{ triggers: ["start"], mirrors: ["a", "b"] },
{ triggers: ["local"], mirrors: ["b"] },
])
expect(asked).toEqual([["a"], ["b"]])
})

it("hears a saved file through the real file watcher", async () => {
const a = mirrorAt("a", { "one.md": "one" })
const stop = new AbortController()
Expand Down
13 changes: 10 additions & 3 deletions src/services/documents/mirror-watch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,17 @@ export type MirrorWatchEvent<T> =
| { triggers: MirrorTrigger[]; mirrors: string[]; result: T }
| { triggers: MirrorTrigger[]; mirrors: string[]; error: CliError }

export interface MirrorWatchOptions<S> {
/**
* How a watch tells which mirrors' local files differ from their state: a document push's kinds and state, or the
* engine's own check.
*/
export type MirrorLocalCheck<S> =
| { kindOf(path: string): string | undefined; stateOf(mirror: Mirror): PushStatePort<S> }
| { localChanges(mirrors: Mirror[]): Promise<string[]> }

export type MirrorWatchOptions<S> = MirrorLocalCheck<S> & {
mirrors: Mirror[]
signal: AbortSignal
kindOf(path: string): string | undefined
stateOf(mirror: Mirror): PushStatePort<S>
/** Ids of the mirrors whose remote files changed since the last call, or `all` when the provider cannot tell. */
remoteChanges?(signal: AbortSignal): Promise<string[] | "all">
pollMs?: number
Expand Down Expand Up @@ -87,6 +93,7 @@ export async function* watchMirrors<S, T>(
const differs = async (ids: Iterable<string>) => {
const chosen = [...ids].flatMap((id) => byId.get(id) ?? [])
try {
if ("localChanges" in options) return await options.localChanges(chosen)
const items = await mirrorStatus(chosen, options.kindOf, options.stateOf)
return items.filter((item) => item.create.length || item.update.length).map((item) => item.id)
} catch {
Expand Down
Loading
Loading