From d1758e71e2f291db48fe5d15dd8f38ef8cf924f0 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 11:41:26 +0300 Subject: [PATCH 1/6] docs(adr): propose ADR-0049 merge on partial clones (P37c) Co-Authored-By: Claude Sonnet 5.5 --- ROADMAP.md | 18 +++- docs/adr/0049-merge-on-partial-clones.md | 117 +++++++++++++++++++++++ docs/adr/README.md | 1 + 3 files changed, 131 insertions(+), 5 deletions(-) create mode 100644 docs/adr/0049-merge-on-partial-clones.md diff --git a/ROADMAP.md b/ROADMAP.md index acdaf7e2..3ae32668 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -554,11 +554,18 @@ made push, verify and gc ask "is this object present?" instead of "is this path in the filter?", which also fixed gc and fast-forward failing after an ordinary fetch on a partial clone. 37b-ii (built) added sparse widen on a lazy clone: the view closure is filled by id before switch, fast-forward, undo and -`sparse set`, and a by-id reply is capped and split on refusal. Merge, rebase and sessions on partial clones are directions -with constraints, each needing its own design record; `export` and `sparse -disable` stay refused. Design and open questions in +`sparse set`, and a by-id reply is capped and split on refusal. `export` and +`sparse disable` stay refused. Design and open questions in [ADR-0048](docs/adr/0048-presence-based-partial-clones.md). +**P37c — merge on partial clones (proposed).** A tree-recursive three-way merge +that carries a subtree by id where the three sides agree, checks protection +rules before carrying, never drops wraps for subtrees it does not enumerate, +and fills (lazy) or refuses (otherwise) only the region where all three sides +differ. Clean merges first, then conflicts, then cherry-pick/rebase; sessions +are undecided. Design in +[ADR-0049](docs/adr/0049-merge-on-partial-clones.md). + ## Completed phases (usability-first ordering) | Phase | Goal | Demoable outcome | ADR | @@ -758,8 +765,9 @@ scale-&-reach horizon): Threading gap-tolerance through `three_way`/replay/`ws::harvest` individually is the follow-on if the "backfill to a full clone first" workaround proves too coarse in practice. - **Now scoped as P37b (ADR-0048, Proposed):** lifted one operation at a - time once push, verify and gc are presence-based. + **Now scoped as P37b (ADR-0048) and P37c (ADR-0049, Proposed):** push, + verify and gc are presence-based (built); merge is designed in ADR-0049, + with replay following it and sessions undecided. - **Blob-size/object-count clone filters (P27).** `sc clone --filter` is prefix-only, matching the sparse path-boundary machinery already built; git's `--filter=blob:limit` size-based filtering is a separate axis, diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md new file mode 100644 index 00000000..edfd457b --- /dev/null +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -0,0 +1,117 @@ +# ADR-0049: Merge on partial clones (P37c) + +- **Status:** Proposed +- **Date:** 2026-10-17 +- **Phase:** 37c +- **Builds on:** ADR-0048 (presence-based partial clones), ADR-0047 (want-by-id fetch), ADR-0012 (three-way merge), ADR-0025 (protected merge and replay), ADR-0037 (partial clone) + +## Context + +ADR-0048 left merge, cherry-pick/rebase and sessions refused on a partial clone and listed what a correct implementation must respect. This record turns the merge half of that list into a design, after reading the code. Replay and sessions follow from it and are sketched at the end. + +Today `sc merge` on a partial clone fast-forwards or adopts (no tree is rebuilt) and otherwise stops at `repo.rs:1349` with `PartialCloneUnsupported`. The reason is how a real merge works: + +1. `three_way_files` (`merge.rs:191`) flattens all three trees to `path -> (blob id, mode, perms)` maps, merges path by path, and returns a flat `files` list. +2. `merge_with_identity` (`repo.rs:1385-1530`) routes that list through the protection rules (a carried plain file under a union rule is re-encrypted), rebuilds the whole tree with `write_tree_with_perms`, unions the wraps (`wrapped_carry` plus fresh wraps), prunes the wrap map to blobs reachable in the merged tree, materializes, and commits a two-parent snapshot. +3. A conflict leaves marker files and a persisted decided root (`merge_state`), and completion (`sc commit`) rebuilds from the working tree plus that decided root. + +Every step reads every file of both sides. A partial clone does not hold them, so the flatten fails with a raw `NotFound` (hence the explicit refusal). + +Almost all of that work is unnecessary. For a subtree where `ours == theirs`, `base == theirs`, or `base == ours` **by tree id**, the answer is known without reading anything below it. A merge only has to look inside a subtree where all three sides differ, and a partial clone typically has the changed region in its view. That is also how a tree-recursive merge becomes proportional to the change instead of to the repository. + +ADR-0048 withdrew an earlier tree-recursive proposal because carrying a subtree by id is not always safe. This record keeps the idea and adds the checks that make it safe. + +## Decisions + +### 1. A tree-recursive three-way merge, with id-carry only where it is provably equal + +Add `merge::merge_trees` next to the flatten implementation. It merges one directory level at a time over the union of entry names: + +| Situation at a name | Result | +|---|---| +| `ours == theirs` (tree or blob id, mode and perms) | take ours | +| `base == theirs` | take ours | +| `base == ours` | take theirs | +| all three differ, all blobs | the existing file merge (diff3, protected-arm rules), unchanged | +| all three differ, all trees | recurse | +| kinds differ (file on one side, directory on another) | flatten **both** subtrees at that name and use the existing path rules; needs both present | + +A subtree is read only in the recurse and flatten rows. In every other row its id is carried and its content is never read, so it may be absent. + +The new function returns a merged tree id built bottom-up (so no `write_tree_with_perms` over a flat list), the set of conflicted paths, sidecars, and the wrap information described below. Verdicts are by presence-independent id equality, so the result does not depend on which subtrees the clone happens to hold, except where a row needs content. + +### 2. Rows that need content are filled or refused, never failed raw + +The recurse and flatten rows need all three subtrees present. A callback `ensure_present(&[ObjectId]) -> Result<()>` is passed in, so `merge.rs` stays network-free and `Store::get` stays network-free (the ADR-0047 rule): + +- lazy and online: the callback fills the missing ids by id (`Filler`, same batching and splitting as ADR-0048); +- otherwise: it fails with `GapInView` naming the path and the `sc backfill` prefix, before anything is written. + +Merge becomes an allow-listed network site for lazy clones, like the 37b-ii materializer sites. The fill touches only the region where all three sides differ, which is the region the merge is about. + +### 3. The carry shortcut must respect protection rules + +Carrying a subtree by id skips the step in `repo.rs:1395-1405` that encrypts a plain file under a union rule. A subtree may be carried from side S only if no rule in the **union** of the sides' rules newly covers a path under it, relative to S's own rules. A rule `R` newly covers a subtree at path `P` when `R` is not in S's rule set and either `R.prefix` is a prefix of `P` (it covers the whole subtree) or `R.prefix` lies inside `P` (it covers part of it). + +When a rule newly covers a candidate subtree, that subtree is not carried: it is treated as a recurse row (read, route through the rules, re-encrypt). That needs the subtree present, so it follows decision 2. Rule sets are small, so the check is cheap; the common case (no new rule) carries. + +### 4. Wraps are never dropped for a subtree that is not enumerated + +`protection.wrapped` maps blob ids to wrapped DEKs. Today the merged map is rebuilt from the files that survive and pruned to blobs reachable in the merged tree. A carried subtree is never enumerated, so its blobs' wraps would be pruned, which permanently loses the ability of any recipient to decrypt that ciphertext. This is the same defect class as the commit-graft wrap bug fixed in P27 (`repo.rs:680-695`). + +The rule for a partial merge is the one the graft already uses: **union in every wrap the inputs held** (ours, theirs, base) for ids the merge could not enumerate, and prune only against what it did enumerate. Over-retention of a wrap for a blob that is no longer reachable is harmless (history keeps the ciphertext and the wraps anyway; rotation is not erasure). Under-retention is data loss. A later `sc rewrap` prunes. + +### 5. Conflicts in a partial merge + +A conflict can only arise in a recurse or flatten row, which is already present. So conflict materialization writes markers only for present, in-view paths. A conflict outside the sparse view is refused with the `sc sparse set` hint the existing code uses (`repo.rs:1590`). + +Completion needs more than today: + +- the completion commit must graft from the **decided root** persisted in `merge_state`, not from HEAD's tree, because the decided root is what carries theirs-side changes outside the view (`graft_out_of_sparse` currently grafts from one parent only, `repo.rs:667`); +- the commit's guard at `repo.rs:411` (`merge/pick completion` refused on a partial clone) is replaced by that graft; +- `sc conflicts` and `sc resolve` (`conflicts.rs:147`, `:263`) flatten whole trees and need the same gap tolerance; +- gc's decided-root walk is already presence-based (ADR-0048, 37b-i). + +Because completion is the riskiest piece, it is a separate phase after the clean case. + +### 6. Secrets, rules and tombstones are unaffected + +`merge_secrets` and the rule/tombstone union work on snapshot metadata, not trees, so they are unchanged. + +### 7. Acceptance: a differential test, not a code-review argument + +`merge_trees` is checked against the flatten implementation. On full clones, over randomly generated base/ours/theirs trees (varied depth, shared subtrees, protected and plain files, rule sets that newly cover subtrees, deletions, mode changes, file/directory kind changes), both must produce the same decrypted file set, the same conflict set and the same sidecars. Bytes are compared for carried objects; freshly sealed output is compared after decryption, because sealing is randomized (ADR-0043). A second differential runs the same merges on a partial clone and a full clone and compares the same things. A full-clone-only test cannot catch the wrap-pruning divergence in decision 4, so the partial/full comparison also compares the merged `protection.wrapped` coverage: every blob reachable in the merged tree must have a wrap whenever the full clone's merge has one. + +The new engine is first wired only for partial clones. Full clones keep the flatten path until the differential has run clean for a release; switching them is a separate, optional step. + +## Phasing + +- **37c-i, engine.** `merge_trees` (pure, store-backed), the rule-aware carry check, wrap union for unenumerated subtrees, the `ensure_present` callback, and the differential tests (full-vs-full and partial-vs-full). No behavior change: nothing calls it yet. +- **37c-ii, clean merge on a partial clone.** Wire `merge_with_identity` to the engine when the clone is partial. A conflict is still refused (clean merges only), with a message that says so. Demo: two branches touching disjoint directories merge on a `--filter` clone without fetching either side's out-of-view content; a merge that touches an out-of-view conflict region fills it on a lazy clone and is refused on a non-lazy one. +- **37c-iii, conflicts.** Markers for present in-view conflicts, completion grafting from the decided root, `sc conflicts` and `sc resolve` on a partial clone, abort. +- **37c-iv, replay.** `cherry-pick` and `rebase` call the same engine (`replay.rs:176` is the only guard that blocks them; `three_way_files` is already their merge). Replay adds its own constraints: protected re-encryption under the replayed rules and the resumable-rebase state. +- **Sessions** (`sc ws`, `sc work`): undecided. They materialize whole checkouts per agent and auto-merge on harvest. Whether they should work on a partial clone at all is a product question, not an engine one. + +## Consequences + +- A partial clone can merge without ever holding the other side's whole tree, which is the point of a partial clone. +- The same engine is a candidate to make full-clone merges proportional to the change. This ADR does not take that step. +- A new network site (merge) exists for lazy clones, bounded to the all-differ region. +- The merged snapshot on a partial clone can carry more wraps than the same merge on a full clone. That is deliberate and documented (decision 4). +- No wire change. + +## Open questions + +1. **Tree id equality of rebuilt subtrees.** A tree built by `merge_trees` must hash exactly like the same tree built by `write_tree_with_perms` from a flat list (entry order, directory modes and perms, the empty-subtree case in ADR-0048 open question 8). The differential test compares ids for the no-conflict, no-encrypt case. +2. **Kind changes (file vs directory).** Decision 1 sends them through the flatten row. If real histories hit this often on partial clones, a dedicated row is worth adding; until then it needs both subtrees present. +3. **Rule check cost and shape.** The "newly covers" test assumes prefix rules. If rules ever grow beyond prefixes the check changes. +4. **Wrap over-retention growth.** Repeated partial merges accumulate wraps for dead blobs until a `sc rewrap`. Probably fine; measure before adding a prune. +5. **Switching full clones to the new engine.** Out of scope; decide after the differential has soaked. +6. **Interaction with private branches.** Merging a public branch INTO a private one routes through `merge_into_private`, a separate path. It stays refused on a partial clone (ADR-0048 q10). + +## Alternatives considered + +- **Fetch everything a merge might need first (`backfill --all`, then merge).** Correct and simple, and it is what the refusal message says today. It defeats the partial clone: a partial clone merging is exactly the case that wants to avoid the full transfer. +- **Make the flatten gap-tolerant and skip absent files.** Skipping absent paths silently drops them from the merged tree. Rejected: this is the failure the commit graft's id-carry exists to prevent. +- **Server-side merge.** Still deferred (ROADMAP). It would also solve this but needs the server to hold keys for protected content or to refuse protected paths. +- **Carry every unchanged-by-id subtree and ignore protection rules.** This is the unsafe version ADR-0048 withdrew: a union rule newly covering a carried plain subtree would leave plaintext in the merged snapshot. diff --git a/docs/adr/README.md b/docs/adr/README.md index 532aa790..8769ce48 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -59,6 +59,7 @@ supersedes the old one and update the **Status** line of the superseded record. | [0046](0046-wal-bucket-remotes.md) | Bucket remotes via an immutable-pack WAL + one CAS'd manifest | Accepted | 36 | | [0047](0047-want-by-id-fetch-and-gap-fill.md) | Want-by-id fetch and explicit gap fill for partial clones | Accepted (37a built) | 37 | | [0048](0048-presence-based-partial-clones.md) | Presence-based walks for partial clones | Accepted (37b-i, 37b-ii built) | 37b | +| [0049](0049-merge-on-partial-clones.md) | Merge on partial clones | Proposed | 37c | See [`ROADMAP.md`](../../ROADMAP.md) for the phase sequence. ADR numbers are assigned in creation order, so 0017 — the secret scanner — sequences as phase P5, From b01cb121b17c93158accc36bee40081104d57348 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 11:46:05 +0300 Subject: [PATCH 2/6] docs(adr): rework ADR-0049 after design review (leaf key, policy-intersecting enumeration, wrap union, pull-based engine, clean merges first) Co-Authored-By: Claude Sonnet 5.5 --- docs/adr/0049-merge-on-partial-clones.md | 140 +++++++++++++---------- 1 file changed, 79 insertions(+), 61 deletions(-) diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index edfd457b..8fd0a8f0 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -3,115 +3,133 @@ - **Status:** Proposed - **Date:** 2026-10-17 - **Phase:** 37c -- **Builds on:** ADR-0048 (presence-based partial clones), ADR-0047 (want-by-id fetch), ADR-0012 (three-way merge), ADR-0025 (protected merge and replay), ADR-0037 (partial clone) +- **Builds on:** ADR-0048 (presence-based partial clones), ADR-0047 (want-by-id fetch), ADR-0012 (three-way merge), ADR-0025 (protected merge and replay), ADR-0043 (randomized sealing) ## Context -ADR-0048 left merge, cherry-pick/rebase and sessions refused on a partial clone and listed what a correct implementation must respect. This record turns the merge half of that list into a design, after reading the code. Replay and sessions follow from it and are sketched at the end. +ADR-0048 left merge, cherry-pick/rebase and sessions refused on a partial clone and listed what a correct implementation must respect. This record turns the merge half of that list into a design. Replay and sessions follow from it and are sketched at the end. A first draft of this record was reviewed against the source (Codex Sol, adversarial, read-only); it found five design flaws, all confirmed in the code and all fixed below. The corrections are folded into the decisions, and the review's counterexamples are required test cases. Today `sc merge` on a partial clone fast-forwards or adopts (no tree is rebuilt) and otherwise stops at `repo.rs:1349` with `PartialCloneUnsupported`. The reason is how a real merge works: 1. `three_way_files` (`merge.rs:191`) flattens all three trees to `path -> (blob id, mode, perms)` maps, merges path by path, and returns a flat `files` list. -2. `merge_with_identity` (`repo.rs:1385-1530`) routes that list through the protection rules (a carried plain file under a union rule is re-encrypted), rebuilds the whole tree with `write_tree_with_perms`, unions the wraps (`wrapped_carry` plus fresh wraps), prunes the wrap map to blobs reachable in the merged tree, materializes, and commits a two-parent snapshot. -3. A conflict leaves marker files and a persisted decided root (`merge_state`), and completion (`sc commit`) rebuilds from the working tree plus that decided root. +2. `merge_with_identity` (`repo.rs:1385-1530`) routes that list through the protection rules (any plain survivor under a union rule is encrypted, whatever side it came from, `repo.rs:1403`), rebuilds the whole tree with `write_tree_with_perms`, unions the wraps, prunes them to blobs reachable in the merged tree, materializes, and commits a two-parent snapshot. +3. A conflict leaves marker files, a persisted decided root (`merge_state`) and sidecars; plaintext of protected content never enters the CAS (`repo.rs:1420`, `:1427`). Completion (`sc commit`) rebuilds from the working tree plus the decided root. -Every step reads every file of both sides. A partial clone does not hold them, so the flatten fails with a raw `NotFound` (hence the explicit refusal). +Every step reads every file of both sides, and a partial clone does not hold them. But for a subtree where the three sides agree **by tree id**, the answer is known without reading below it, so a merge only has to look inside the subtrees that differ. That is the idea ADR-0048 withdrew as unsafe as first written. It is safe only with the restrictions here. -Almost all of that work is unnecessary. For a subtree where `ours == theirs`, `base == theirs`, or `base == ours` **by tree id**, the answer is known without reading anything below it. A merge only has to look inside a subtree where all three sides differ, and a partial clone typically has the changed region in its view. That is also how a tree-recursive merge becomes proportional to the change instead of to the repository. +## Decisions -ADR-0048 withdrew an earlier tree-recursive proposal because carrying a subtree by id is not always safe. This record keeps the idea and adds the checks that make it safe. +### 1. Leaf equality is one rule, shared by both engines -## Decisions +The flatten merge resolves a plain path by **blob id alone** (`merge.rs:229-249`), and a protected path by `(blob id, PROTECTED)` (`merge.rs:312`). A mode-only change therefore loses today: base and ours hold blob X at `0644`, theirs holds X at `0755`; flatten sees ours' id equal to theirs' and keeps `0644`. A tree-level shortcut that compares whole entries would pick `0755`, so the two engines would disagree on the same merge. + +Rather than keep two behaviors, 37c-i changes the flatten leaf comparison to the entry key `(blob id, mode, perms & PROTECTED)` and uses that key in both engines. This is a deliberate, small behavior change on full clones: a mode-only change on one side now merges (it is currently dropped). `RANDOMIZED` stays out of the key because it is a property of the blob. ADR-0012 gets an amendment note and a regression test (the 0644/0755 example). + +### 2. A tree-recursive merge that carries only what it can prove safe -### 1. A tree-recursive three-way merge, with id-carry only where it is provably equal +Add `merge::merge_trees` beside the flatten implementation. It works one directory level at a time over the union of entry names. A subtree is **carried by id, unread**, when: -Add `merge::merge_trees` next to the flatten implementation. It merges one directory level at a time over the union of entry names: +- it is identical (same tree id) on all three sides, or +- the three sides do not all differ (`ours == theirs`, `base == theirs`, or `base == ours`, by tree id), **and** the subtree is policy-disjoint (decision 3). + +Everything else is **enumerated**: read, flattened for that subtree, and resolved by the existing leaf rules (diff3, the protected arm, delete/modify), unchanged. Absence is explicit: | Situation at a name | Result | |---|---| -| `ours == theirs` (tree or blob id, mode and perms) | take ours | -| `base == theirs` | take ours | -| `base == ours` | take theirs | -| all three differ, all blobs | the existing file merge (diff3, protected-arm rules), unchanged | -| all three differ, all trees | recurse | -| kinds differ (file on one side, directory on another) | flatten **both** subtrees at that name and use the existing path rules; needs both present | +| tree on all three sides, all differ | recurse | +| tree on some sides, absent (or a blob) on others | enumerate the subtrees that exist and apply the per-path rules, so delete/modify and add/add keep their flatten meaning | +| blob on one side and a tree on another | enumerate both and apply the per-path rules; if that yields both `x` and `x/child`, the existing tree writer's error stands (this is unchanged) | +| blobs on all three, all differ | the existing leaf merge | + +### 3. Policy-intersecting subtrees are always enumerated + +A subtree is policy-disjoint when no rule in the **union** of the sides' rules covers it or lies inside it (`protect::matching_prefix` semantics: boundary-aware, longest prefix wins). A subtree that intersects a union rule and is not identical on all three sides is enumerated, never carried. + +The first draft tested only whether a rule was *new* relative to the side being carried. That is not enough: the rule can already be on that side while plaintext sits under it. `protect` records the rule in a policy-only snapshot and encrypts in a second commit (`protect_ops.rs:85-105`), and a partial-clone commit encrypts only the sparse view and grafts the rest unchanged (`repo.rs:667`). The full merge encrypts every plain survivor under a union rule (`repo.rs:1403`) and would repair it; a carry would preserve it. -A subtree is read only in the recurse and flatten rows. In every other row its id is carried and its content is never read, so it may be absent. +One accepted divergence: a subtree identical on all three sides is carried even under a rule, so a merge does not force a partial clone to fetch every protected prefix. Plaintext under a rule that is identical on all three sides was already in both parents; the merge introduces nothing. A full merge would repair it and a partial one leaves it. A `sc verify` check for that state is a follow-on. -The new function returns a merged tree id built bottom-up (so no `write_tree_with_perms` over a flat list), the set of conflicted paths, sidecars, and the wrap information described below. Verdicts are by presence-independent id equality, so the result does not depend on which subtrees the clone happens to hold, except where a row needs content. +Recipient-list changes alone do not require re-wrapping a carried ciphertext: the full merge already carries ciphertext unchanged and unions the wraps it knows (`repo.rs:1413`, `merge.rs:454`). -### 2. Rows that need content are filled or refused, never failed raw +### 4. Wraps: keep the complete input union whenever anything was carried -The recurse and flatten rows need all three subtrees present. A callback `ensure_present(&[ObjectId]) -> Result<()>` is passed in, so `merge.rs` stays network-free and `Store::get` stays network-free (the ADR-0047 rule): +`protection.wrapped` maps blob ids to wrapped DEKs. The full merge rebuilds it from surviving files and prunes to reachable blobs. A carried subtree is never enumerated, so pruning would drop the wraps of blobs inside it, which permanently loses every recipient's ability to decrypt them (the defect class the commit-graft fix closed, `repo.rs:699`). -- lazy and online: the callback fills the missing ids by id (`Filler`, same batching and splitting as ADR-0048); -- otherwise: it fails with `GapInView` naming the path and the `sc backfill` prefix, before anything is written. +A per-id rule ("prune what you enumerated") is unsound: a blob id seen as deleted at one path may survive under a carried subtree elsewhere, since ids alias across paths. So the rule is coarse: **if the merge carried any subtree, the merged map is the complete union of the input wraps** (ours, theirs, base, with the existing ours/theirs/base precedence and per-recipient dedup), plus fresh wraps. If it carried nothing, the existing prune applies. Union resurrecting a wrap for a recipient revoked since is the intended P16 behavior (tombstones gate future seals, not historical wraps; ADR-0026). -Merge becomes an allow-listed network site for lazy clones, like the 37b-ii materializer sites. The fill touches only the region where all three sides differ, which is the region the merge is about. +Over-retention costs size: wraps are inline in the snapshot, and an object above `MAX_OBJECT_SIZE` fails transfer. Nothing prunes dead entries today (`sc rewrap` refuses partial clones and only removes ids it upgrades, `rewrap.rs:164`, `:308`). So the growth is real and bounded by the repo's protected blob count; it is measured before any prune is designed. -### 3. The carry shortcut must respect protection rules +### 5. The engine is pull-based; filling happens outside the store lock -Carrying a subtree by id skips the step in `repo.rs:1395-1405` that encrypts a plain file under a union rule. A subtree may be carried from side S only if no rule in the **union** of the sides' rules newly covers a path under it, relative to S's own rules. A rule `R` newly covers a subtree at path `P` when `R` is not in S's rule set and either `R.prefix` is a prefix of `P` (it covers the whole subtree) or `R.prefix` lies inside `P` (it covers part of it). +`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(ids)`: the subtrees it must read and does not hold. The caller drops the lock, fills those ids (lazy and online) or fails with `GapInView` naming the path and the `sc backfill` prefix (otherwise), and calls again. Each round is idempotent, and progress is by presence. -When a rule newly covers a candidate subtree, that subtree is not carried: it is treated as a recurse row (read, route through the rules, re-encrypt). That needs the subtree present, so it follows decision 2. Rule sets are small, so the check is cheap; the common case (no new rule) carries. +The merged tree can also carry a subtree that is absent **inside the sparse view** (the fetch case from ADR-0048). So after the merge and before materializing, the caller runs `ensure_view(merged_root, sparse)`, as every other materializer site does. The claim that network traffic is bounded to the differing region is therefore "the differing region, the rule-intersecting region, and the view's closure". -### 4. Wraps are never dropped for a subtree that is not enumerated +Failure contract: a fill that fails midway leaves only immutable CAS additions. No ref, spec, working-tree file or merge state is written before the last fill and `ensure_view` have succeeded. -`protection.wrapped` maps blob ids to wrapped DEKs. Today the merged map is rebuilt from the files that survive and pruned to blobs reachable in the merged tree. A carried subtree is never enumerated, so its blobs' wraps would be pruned, which permanently loses the ability of any recipient to decrypt that ciphertext. This is the same defect class as the commit-graft wrap bug fixed in P27 (`repo.rs:680-695`). +### 6. The result keeps plaintext out of the CAS -The rule for a partial merge is the one the graft already uses: **union in every wrap the inputs held** (ours, theirs, base) for ids the merge could not enumerate, and prune only against what it did enumerate. Over-retention of a wrap for a blob that is no longer reachable is harmless (history keeps the ciphertext and the wraps anyway; rotation is not erasure). Under-retention is data loss. A later `sc rewrap` prunes. +`TreeMerge` is not a bare tree id. It returns what `FileMerge` returns today (the enumerated `MergedFile` list with `needs_encrypt` plaintext, sidecars, conflicts, `wrapped_carry`) **plus** the carried subtrees as `(path, TreeEntry)` pairs. The existing pipeline (`repo.rs:1385-1530`) handles the enumerated files unchanged (routing through union rules, encryption, plaintext never written to the CAS). The carried entries are overlaid by id when the final tree is assembled. No new code path touches plaintext. -### 5. Conflicts in a partial merge +### 7. Tree semantics, stated plainly -A conflict can only arise in a recurse or flatten row, which is already present. So conflict materialization writes markers only for present, in-view paths. A conflict outside the sparse view is refused with the `sc sparse set` hint the existing code uses (`repo.rs:1590`). +Flatten drops empty directories and directory metadata and rebuilds directories with mode `0755`/perms `0` (`worktree.rs:348`, `vfs/lib.rs:175`); a carried subtree keeps whatever it has, including an empty directory entry that `graft_out_of_sparse` can produce (`worktree.rs:313`). Equivalence is defined on **file sets, modes and perms**, not on tree ids. A merged tree on a partial clone may differ from the full-clone merge by an empty directory entry. The differential test compares file sets, not ids. -Completion needs more than today: +### 8. Conflicts are deferred until their contract is specified -- the completion commit must graft from the **decided root** persisted in `merge_state`, not from HEAD's tree, because the decided root is what carries theirs-side changes outside the view (`graft_out_of_sparse` currently grafts from one parent only, `repo.rs:667`); -- the commit's guard at `repo.rs:411` (`merge/pick completion` refused on a partial clone) is replaced by that graft; -- `sc conflicts` and `sc resolve` (`conflicts.rs:147`, `:263`) flatten whole trees and need the same gap tolerance; -- gc's decided-root walk is already presence-based (ADR-0048, 37b-i). +A conflict needs a state contract this record does not yet fix, and the review showed why it is not a detail: -Because completion is the riskiest piece, it is a separate phase after the clean case. +- a conflicted merge's protected content merges produce plaintext that must live only on disk, with the CAS-safe decided root carrying everything else (`repo.rs:1420-1520`, `:1636`); +- an in-view conflict can coexist with a **clean protected content merge outside the view**; today that clean plaintext is written straight to disk, and a decided root alone does not preserve it, so completion would lose it; +- newly sealed decided blobs' wraps must survive a process restart, but merge state stores only the root (`merge_state.rs:68`) and completion reconstructs protection from the parents (`repo.rs:359`); +- presence does not imply the sparse view: a conflict outside the view needs the existing `sc sparse set` refusal, and abort must preflight and fill before it deletes sidecars (today it deletes them first, `repo.rs:1666`); +- `sc conflicts` and `sc resolve` flatten whole trees and need path-directed lookup (`conflicts.rs:147`, `:263`, `:329`). -### 6. Secrets, rules and tombstones are unaffected +37c-i and 37c-ii therefore support **clean merges only**: any conflict on a partial clone is refused with a message saying so, before anything is written. 37c-iii amends this record with the conflict contract (a CAS-safe decided root, ephemeral conflict plaintext, persisted fresh wraps or an explicit refusal of the conflict-plus-out-of-view-protected-merge combination) before it builds anything. -`merge_secrets` and the rule/tombstone union work on snapshot metadata, not trees, so they are unchanged. +### 9. Acceptance: differential tests built from real operations -### 7. Acceptance: a differential test, not a code-review argument +`merge_trees` is checked against the flatten implementation, with both on the shared leaf key of decision 1. The comparison is on **decrypted file sets, modes, perms, conflict sets, sidecars, effective rules and tombstones, the recipient set each protected blob is wrapped for, and actual decryption by each relevant identity**, not on wrap-map key coverage (which can pass with a missing recipient or a wrong DEK). Carried objects compare by bytes; freshly sealed output after decryption, because sealing is randomized. Cases are generated through real operations (`protect`, `grant`, `revoke`, commit, fetch) as well as arbitrary trees, and gaps are randomized independently of the sparse view and the filter. Required named cases: -`merge_trees` is checked against the flatten implementation. On full clones, over randomly generated base/ours/theirs trees (varied depth, shared subtrees, protected and plain files, rule sets that newly cover subtrees, deletions, mode changes, file/directory kind changes), both must produce the same decrypted file set, the same conflict set and the same sidecars. Bytes are compared for carried objects; freshly sealed output is compared after decryption, because sealing is randomized (ADR-0043). A second differential runs the same merges on a partial clone and a full clone and compares the same things. A full-clone-only test cannot catch the wrap-pruning divergence in decision 4, so the partial/full comparison also compares the merged `protection.wrapped` coverage: every blob reachable in the merged tree must have a wrap whenever the full clone's merge has one. +- the mode-only change (0644/0755) from decision 1; +- plaintext under an existing rule on the carried side (decision 3), including the interrupted-`protect` state and the partial-clone graft state; +- shared blob ids across a deleted path and a carried path (decision 4); +- crossed revokes and regrants; nested prefix overrides (longest prefix wins); +- randomized equal-plaintext independent seals; +- every absence and kind combination in decision 2's table; +- an empty-directory entry on a carried side; +- merge then gc, and an interrupted fill then retry. -The new engine is first wired only for partial clones. Full clones keep the flatten path until the differential has run clean for a release; switching them is a separate, optional step. +A second differential runs each merge on a partial clone and a full clone. The new engine is wired only for partial clones first; full clones keep the flatten path (with the decision 1 leaf key) until the differential has run clean for a release. ## Phasing -- **37c-i, engine.** `merge_trees` (pure, store-backed), the rule-aware carry check, wrap union for unenumerated subtrees, the `ensure_present` callback, and the differential tests (full-vs-full and partial-vs-full). No behavior change: nothing calls it yet. -- **37c-ii, clean merge on a partial clone.** Wire `merge_with_identity` to the engine when the clone is partial. A conflict is still refused (clean merges only), with a message that says so. Demo: two branches touching disjoint directories merge on a `--filter` clone without fetching either side's out-of-view content; a merge that touches an out-of-view conflict region fills it on a lazy clone and is refused on a non-lazy one. -- **37c-iii, conflicts.** Markers for present in-view conflicts, completion grafting from the decided root, `sc conflicts` and `sc resolve` on a partial clone, abort. -- **37c-iv, replay.** `cherry-pick` and `rebase` call the same engine (`replay.rs:176` is the only guard that blocks them; `three_way_files` is already their merge). Replay adds its own constraints: protected re-encryption under the replayed rules and the resumable-rebase state. -- **Sessions** (`sc ws`, `sc work`): undecided. They materialize whole checkouts per agent and auto-merge on harvest. Whether they should work on a partial clone at all is a product question, not an engine one. +- **37c-i, engine.** The leaf-key fix on the flatten merge (with its ADR-0012 amendment and test), `merge_trees` returning `TreeMerge` or `Needs`, the policy-intersection test, the wrap-union rule, and the differential tests. Nothing calls the new engine yet except tests. +- **37c-ii, clean merge on a partial clone.** `merge_with_identity` uses the engine when the clone is partial: fill loop outside the lock, `ensure_view(merged_root)`, then the existing pipeline. A conflict is refused up front. Demo: two branches touching disjoint directories merge on a `--filter` clone without fetching either side's out-of-view content; a merge whose differing region is outside the view fills it on a lazy clone and is refused on a non-lazy one. +- **37c-iii, conflicts.** First an amendment to this record that fixes decision 8's contract, then markers for present in-view conflicts, completion from the decided root, abort, `sc conflicts` and `sc resolve`. +- **37c-iv, replay.** `cherry-pick` and `rebase` call the same engine; `replay.rs:176` is the only guard that blocks them, and `three_way_files` is already their merge. Replay adds protected re-encryption under the replayed rules and the resumable-rebase state. +- **Sessions** (`sc ws`, `sc work`): undecided. They materialize whole checkouts per agent and auto-merge on harvest. Whether they should work on a partial clone at all is a product question. ## Consequences -- A partial clone can merge without ever holding the other side's whole tree, which is the point of a partial clone. -- The same engine is a candidate to make full-clone merges proportional to the change. This ADR does not take that step. -- A new network site (merge) exists for lazy clones, bounded to the all-differ region. -- The merged snapshot on a partial clone can carry more wraps than the same merge on a full clone. That is deliberate and documented (decision 4). +- A partial clone can merge without holding the other side's whole tree, which is what a partial clone is for. A merge whose differing region intersects a protection rule fetches that region. +- Mode-only changes now merge on full clones too (decision 1). This is a deliberate fix and is called out in the release notes. +- A merged snapshot that carried subtrees holds the complete input wrap union, so it can be larger than the same merge on a full clone. +- Merge becomes an allow-listed network site for lazy clones. - No wire change. ## Open questions -1. **Tree id equality of rebuilt subtrees.** A tree built by `merge_trees` must hash exactly like the same tree built by `write_tree_with_perms` from a flat list (entry order, directory modes and perms, the empty-subtree case in ADR-0048 open question 8). The differential test compares ids for the no-conflict, no-encrypt case. -2. **Kind changes (file vs directory).** Decision 1 sends them through the flatten row. If real histories hit this often on partial clones, a dedicated row is worth adding; until then it needs both subtrees present. -3. **Rule check cost and shape.** The "newly covers" test assumes prefix rules. If rules ever grow beyond prefixes the check changes. -4. **Wrap over-retention growth.** Repeated partial merges accumulate wraps for dead blobs until a `sc rewrap`. Probably fine; measure before adding a prune. -5. **Switching full clones to the new engine.** Out of scope; decide after the differential has soaked. -6. **Interaction with private branches.** Merging a public branch INTO a private one routes through `merge_into_private`, a separate path. It stays refused on a partial clone (ADR-0048 q10). +1. **A `sc verify` check for plaintext under a rule** (decision 3's accepted divergence). +2. **Wrap growth.** Measure before designing a prune; `sc rewrap` on a partial clone is refused today. +3. **Switching full clones to the new engine.** Out of scope; decide after the differential has soaked. +4. **Merging a public branch INTO a private one** routes through `merge_into_private` and stays refused on a partial clone (ADR-0048 q10). +5. **Empty-directory normalization.** Decision 7 accepts the divergence; revisit if it shows up in real histories. ## Alternatives considered -- **Fetch everything a merge might need first (`backfill --all`, then merge).** Correct and simple, and it is what the refusal message says today. It defeats the partial clone: a partial clone merging is exactly the case that wants to avoid the full transfer. -- **Make the flatten gap-tolerant and skip absent files.** Skipping absent paths silently drops them from the merged tree. Rejected: this is the failure the commit graft's id-carry exists to prevent. -- **Server-side merge.** Still deferred (ROADMAP). It would also solve this but needs the server to hold keys for protected content or to refuse protected paths. -- **Carry every unchanged-by-id subtree and ignore protection rules.** This is the unsafe version ADR-0048 withdrew: a union rule newly covering a carried plain subtree would leave plaintext in the merged snapshot. +- **Fetch everything a merge might need first (`backfill --all`, then merge).** Correct and simple, and what the refusal message says today. It defeats the partial clone. +- **Make the flatten gap-tolerant and skip absent files.** Skipping absent paths silently drops them from the merged tree, the failure the commit graft's id-carry exists to prevent. +- **Carry every unchanged-by-id subtree and ignore protection rules.** The unsafe version ADR-0048 withdrew. +- **A "newly covered by a rule" carry test.** The first draft of this record. Unsound, because a side can already hold plaintext under its own rule (decision 3). +- **Server-side merge.** Still deferred (ROADMAP). It needs the server to hold keys for protected content or to refuse protected paths. From dfa0b75249493b6fc5cbea31fdaae44443d7e9b8 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 11:48:02 +0300 Subject: [PATCH 3/6] docs(adr): ADR-0049 separate content and mode, reject file/dir collisions, state merge ordering Co-Authored-By: Claude Sonnet 5.5 --- docs/adr/0049-merge-on-partial-clones.md | 38 ++++++++++++++++-------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index 8fd0a8f0..9b5a1c62 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -19,11 +19,18 @@ Every step reads every file of both sides, and a partial clone does not hold the ## Decisions -### 1. Leaf equality is one rule, shared by both engines +### 1. Content and mode are merged as separate dimensions, in both engines -The flatten merge resolves a plain path by **blob id alone** (`merge.rs:229-249`), and a protected path by `(blob id, PROTECTED)` (`merge.rs:312`). A mode-only change therefore loses today: base and ours hold blob X at `0644`, theirs holds X at `0755`; flatten sees ours' id equal to theirs' and keeps `0644`. A tree-level shortcut that compares whole entries would pick `0755`, so the two engines would disagree on the same merge. +The flatten merge resolves a plain path by **blob id alone** (`merge.rs:229-249`) and a protected path by `(blob id, PROTECTED)` (`merge.rs:312`); the mode of the result then simply follows whichever side's entry was kept. A mode-only change therefore loses today: base and ours hold blob X at `0644`, theirs holds X at `0755`; the ids agree, ours is kept, and `0755` is dropped. A tree-level shortcut compares whole entries and would pick `0755`, so the two engines would disagree on the same merge. -Rather than keep two behaviors, 37c-i changes the flatten leaf comparison to the entry key `(blob id, mode, perms & PROTECTED)` and uses that key in both engines. This is a deliberate, small behavior change on full clones: a mode-only change on one side now merges (it is currently dropped). `RANDOMIZED` stays out of the key because it is a property of the blob. ADR-0012 gets an amendment note and a regression test (the 0644/0755 example). +A single composite key `(id, mode, perms)` is not the answer. It turns a metadata change into a false content divergence: base `(X,0644)`, ours `(Y,0644)`, theirs `(Y,0755)` agree on Y, yet every composite key differs, which would demand an identity for protected Y, report a conflict for identical binary Y, and could re-seal unchanged ciphertext (`merge.rs:283`, `:377`, `:399`). + +So 37c-i separates the dimensions, in **both** engines: + +- **Content** resolves exactly as today: by blob id (and the `PROTECTED` bit for protected paths). Nothing about content resolution, diff3, sealing or wraps changes. +- **Mode** is merged afterwards, independently, as a three-way on the modes of the entries that exist: equal on both sides, take it; one side equal to base, take the other's; both sides changed differently, executable if either is (today's rule for that case, kept as the tiebreak). Example: base `(X,0755)`, ours `(X,0644)`, theirs `(Y,0755)` resolves content to Y (ours is unchanged from base by id, so theirs wins) and mode to `0644` (ours changed it, theirs did not), so ours' executable removal is kept. + +This is a deliberate, small behavior change on full clones: a mode-only change on one side now merges. It also makes the tree-level shortcuts below exact, because identical tree ids imply equal content and equal mode at every leaf, so "carry the differing side" is the per-dimension three-way answer. ADR-0012 gets an amendment note, and the cases above are regression tests (mode-only, identical-content/different-mode on binary and protected-keyless content, executable removal). ### 2. A tree-recursive merge that carries only what it can prove safe @@ -38,16 +45,18 @@ Everything else is **enumerated**: read, flattened for that subtree, and resolve |---|---| | tree on all three sides, all differ | recurse | | tree on some sides, absent (or a blob) on others | enumerate the subtrees that exist and apply the per-path rules, so delete/modify and add/add keep their flatten meaning | -| blob on one side and a tree on another | enumerate both and apply the per-path rules; if that yields both `x` and `x/child`, the existing tree writer's error stands (this is unchanged) | +| blob on one side and a tree on another | enumerate both and apply the per-path rules, then apply the collision check below | | blobs on all three, all differ | the existing leaf merge | +**File/directory collisions are rejected up front, in both engines.** The existing path does not reject them: with an empty base, ours adding file `x` and theirs adding `x/child`, the leaf merge reports two clean additions, the writer emits both a blob and a tree named `x` (`vfs/lib.rs:155`, `:175`; `Tree::new` only sorts duplicates, `object.rs:134`), and materialization can write `x` and then fail creating `x/` after changing files (`worktree.rs:839`). 37c-i makes the merge fail before any output write when a resolved name is a file and also a directory, or a file is an ancestor of another resolved path, including across the boundary between carried subtrees and enumerated files. The error names the path. This is a small fix to full-clone merge too. + ### 3. Policy-intersecting subtrees are always enumerated -A subtree is policy-disjoint when no rule in the **union** of the sides' rules covers it or lies inside it (`protect::matching_prefix` semantics: boundary-aware, longest prefix wins). A subtree that intersects a union rule and is not identical on all three sides is enumerated, never carried. +A subtree is policy-disjoint when no rule in the **union** of the sides' rules covers it or lies inside it, checked in both directions at `/` boundaries (`protect::matching_prefix` semantics: boundary-aware, longest prefix wins). Every rule counts, including rules present on only one side and rules whose recipients have all been revoked: tombstones gate future seals and do not remove the prefix (`protect.rs:10`, `:30`). A subtree that intersects a union rule and is not identical on all three sides is enumerated, never carried. The first draft tested only whether a rule was *new* relative to the side being carried. That is not enough: the rule can already be on that side while plaintext sits under it. `protect` records the rule in a policy-only snapshot and encrypts in a second commit (`protect_ops.rs:85-105`), and a partial-clone commit encrypts only the sparse view and grafts the rest unchanged (`repo.rs:667`). The full merge encrypts every plain survivor under a union rule (`repo.rs:1403`) and would repair it; a carry would preserve it. -One accepted divergence: a subtree identical on all three sides is carried even under a rule, so a merge does not force a partial clone to fetch every protected prefix. Plaintext under a rule that is identical on all three sides was already in both parents; the merge introduces nothing. A full merge would repair it and a partial one leaves it. A `sc verify` check for that state is a follow-on. +One accepted divergence: a subtree identical on all three sides is carried even under a rule, so a merge does not force a partial clone to fetch every protected prefix. Plaintext under a rule that is identical on all three sides was already in both parents; the merge introduces nothing. A full merge would repair it and a partial one leaves it. A `sc verify` check for that state is a follow-on. The differential test (decision 9) excludes this state from its protection-perms equivalence on purpose. Recipient-list changes alone do not require re-wrapping a carried ciphertext: the full merge already carries ciphertext unchanged and unions the wraps it knows (`repo.rs:1413`, `merge.rs:454`). @@ -65,11 +74,13 @@ Over-retention costs size: wraps are inline in the snapshot, and an object above The merged tree can also carry a subtree that is absent **inside the sparse view** (the fetch case from ADR-0048). So after the merge and before materializing, the caller runs `ensure_view(merged_root, sparse)`, as every other materializer site does. The claim that network traffic is bounded to the differing region is therefore "the differing region, the rule-intersecting region, and the view's closure". -Failure contract: a fill that fails midway leaves only immutable CAS additions. No ref, spec, working-tree file or merge state is written before the last fill and `ensure_view` have succeeded. +The order of a partial merge is: merge registries and files (a `Needs` result loops back through fill and recomputes, with no output writes during the retries) → refuse conflicts and collisions → encrypt → assemble the final root and wraps → `ensure_view(merged_root)` and the materialize preflight → materialize → snapshot and ref move. The repo lock stays held throughout; only the store mutex is released around fills. + +Failure contract: fetched objects persist (fill rounds write immutable CAS additions, so "before anything is written" cannot mean literally nothing), and so may the merged tree objects of the output step. No ref, spec, working-tree file or merge state is written until `ensure_view` and the preflight have succeeded. ### 6. The result keeps plaintext out of the CAS -`TreeMerge` is not a bare tree id. It returns what `FileMerge` returns today (the enumerated `MergedFile` list with `needs_encrypt` plaintext, sidecars, conflicts, `wrapped_carry`) **plus** the carried subtrees as `(path, TreeEntry)` pairs. The existing pipeline (`repo.rs:1385-1530`) handles the enumerated files unchanged (routing through union rules, encryption, plaintext never written to the CAS). The carried entries are overlaid by id when the final tree is assembled. No new code path touches plaintext. +`TreeMerge` is not a bare tree id. It returns what `FileMerge` returns today (the enumerated `MergedFile` list with `needs_encrypt` plaintext, sidecars, conflicts, `wrapped_carry`) **plus** the carried subtrees as `(path, TreeEntry)` pairs. The existing pipeline (`repo.rs:1385-1530`) handles the enumerated files unchanged (routing through union rules, encryption, plaintext never written to the CAS). The carried entries are overlaid by id when the final tree is assembled: a recursive overlay rebuilds each ancestor directory from the enumerated files at that level, inserts the carried entries verbatim (id, mode, perms untouched), and keeps the enumerated siblings, with the collision check above run over the combined names. No new code path touches plaintext. ### 7. Tree semantics, stated plainly @@ -85,13 +96,14 @@ A conflict needs a state contract this record does not yet fix, and the review s - presence does not imply the sparse view: a conflict outside the view needs the existing `sc sparse set` refusal, and abort must preflight and fill before it deletes sidecars (today it deletes them first, `repo.rs:1666`); - `sc conflicts` and `sc resolve` flatten whole trees and need path-directed lookup (`conflicts.rs:147`, `:263`, `:329`). -37c-i and 37c-ii therefore support **clean merges only**: any conflict on a partial clone is refused with a message saying so, before anything is written. 37c-iii amends this record with the conflict contract (a CAS-safe decided root, ephemeral conflict plaintext, persisted fresh wraps or an explicit refusal of the conflict-plus-out-of-view-protected-merge combination) before it builds anything. +37c-i and 37c-ii therefore support **clean merges only**: any conflict on a partial clone is refused with a message saying so, before any merge-output, merge-state or working-tree write. This is implementable because `three_way_files` computes the conflict set without inserting objects (`merge.rs:191-438`), and a secret-registry conflict already fails before the file merge (`merge.rs:162`). 37c-iii amends this record with the conflict contract (a CAS-safe decided root, ephemeral conflict plaintext, persisted fresh wraps or an explicit refusal of the conflict-plus-out-of-view-protected-merge combination) before it builds anything. ### 9. Acceptance: differential tests built from real operations -`merge_trees` is checked against the flatten implementation, with both on the shared leaf key of decision 1. The comparison is on **decrypted file sets, modes, perms, conflict sets, sidecars, effective rules and tombstones, the recipient set each protected blob is wrapped for, and actual decryption by each relevant identity**, not on wrap-map key coverage (which can pass with a missing recipient or a wrong DEK). Carried objects compare by bytes; freshly sealed output after decryption, because sealing is randomized. Cases are generated through real operations (`protect`, `grant`, `revoke`, commit, fetch) as well as arbitrary trees, and gaps are randomized independently of the sparse view and the filter. Required named cases: +`merge_trees` is checked against the flatten implementation, with both using the content/mode split of decision 1. The comparison is on **decrypted file sets, modes, perms, conflict sets, sidecars, effective rules and tombstones, the recipient set each protected blob is wrapped for, and actual decryption by each relevant identity**, not on wrap-map key coverage (which can pass with a missing recipient or a wrong DEK). Carried objects compare by bytes; freshly sealed output after decryption, because sealing is randomized. Cases are generated through real operations (`protect`, `grant`, `revoke`, commit, fetch) as well as arbitrary trees, and gaps are randomized independently of the sparse view and the filter. Required named cases: -- the mode-only change (0644/0755) from decision 1; +- the mode and content cases from decision 1 (mode-only change, identical content with different mode on binary and on protected-keyless content, executable removal); +- file/directory collisions, including across a carried/enumerated boundary; - plaintext under an existing rule on the carried side (decision 3), including the interrupted-`protect` state and the partial-clone graft state; - shared blob ids across a deleted path and a carried path (decision 4); - crossed revokes and regrants; nested prefix overrides (longest prefix wins); @@ -104,7 +116,7 @@ A second differential runs each merge on a partial clone and a full clone. The n ## Phasing -- **37c-i, engine.** The leaf-key fix on the flatten merge (with its ADR-0012 amendment and test), `merge_trees` returning `TreeMerge` or `Needs`, the policy-intersection test, the wrap-union rule, and the differential tests. Nothing calls the new engine yet except tests. +- **37c-i, engine.** The content/mode split and the collision check on the flatten merge (with the ADR-0012 amendment and tests), `merge_trees` returning `TreeMerge` or `Needs`, the policy-intersection test, the wrap-union rule, and the differential tests. Nothing calls the new engine yet except tests. - **37c-ii, clean merge on a partial clone.** `merge_with_identity` uses the engine when the clone is partial: fill loop outside the lock, `ensure_view(merged_root)`, then the existing pipeline. A conflict is refused up front. Demo: two branches touching disjoint directories merge on a `--filter` clone without fetching either side's out-of-view content; a merge whose differing region is outside the view fills it on a lazy clone and is refused on a non-lazy one. - **37c-iii, conflicts.** First an amendment to this record that fixes decision 8's contract, then markers for present in-view conflicts, completion from the decided root, abort, `sc conflicts` and `sc resolve`. - **37c-iv, replay.** `cherry-pick` and `rebase` call the same engine; `replay.rs:176` is the only guard that blocks them, and `three_way_files` is already their merge. Replay adds protected re-encryption under the replayed rules and the resumable-rebase state. @@ -113,7 +125,7 @@ A second differential runs each merge on a partial clone and a full clone. The n ## Consequences - A partial clone can merge without holding the other side's whole tree, which is what a partial clone is for. A merge whose differing region intersects a protection rule fetches that region. -- Mode-only changes now merge on full clones too (decision 1). This is a deliberate fix and is called out in the release notes. +- Mode-only changes now merge on full clones too (decision 1), and a file/directory collision is rejected before any write instead of failing during materialization. Both are deliberate fixes and are called out in the release notes. - A merged snapshot that carried subtrees holds the complete input wrap union, so it can be larger than the same merge on a full clone. - Merge becomes an allow-listed network site for lazy clones. - No wire change. From 0951e3e0e7693c4b38dc073bd8c5b81910c29ac8 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 11:52:21 +0300 Subject: [PATCH 4/6] docs: ROADMAP P37c fill scope names the policy and view regions Co-Authored-By: Claude Sonnet 5.5 --- ROADMAP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 3ae32668..3c368f6f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -561,8 +561,8 @@ clone: the view closure is filled by id before switch, fast-forward, undo and **P37c — merge on partial clones (proposed).** A tree-recursive three-way merge that carries a subtree by id where the three sides agree, checks protection rules before carrying, never drops wraps for subtrees it does not enumerate, -and fills (lazy) or refuses (otherwise) only the region where all three sides -differ. Clean merges first, then conflicts, then cherry-pick/rebase; sessions +and fills (lazy) or refuses (otherwise) the region where the sides differ, +the region that intersects a protection rule, and the sparse view's closure. Clean merges first, then conflicts, then cherry-pick/rebase; sessions are undecided. Design in [ADR-0049](docs/adr/0049-merge-on-partial-clones.md). From b9573b21968df682b4dbe151ccebd32707fec749 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 11:58:28 +0300 Subject: [PATCH 5/6] docs(adr): ADR-0049 Needs() carries the traversal path for the GapInView refusal Co-Authored-By: Claude Sonnet 5.5 --- docs/adr/0049-merge-on-partial-clones.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index 9b5a1c62..68fdcd2d 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -70,7 +70,7 @@ Over-retention costs size: wraps are inline in the snapshot, and an object above ### 5. The engine is pull-based; filling happens outside the store lock -`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(ids)`: the subtrees it must read and does not hold. The caller drops the lock, fills those ids (lazy and online) or fails with `GapInView` naming the path and the `sc backfill` prefix (otherwise), and calls again. Each round is idempotent, and progress is by presence. +`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(missing)`, where each entry is `(ObjectId, path)`: the subtree it must read and does not hold, with the path at which the traversal found it (an id alone does not identify a path, since subtrees alias, and an absent root has no path to recover from tree data; the root is reported as the empty path). The caller drops the lock, fills those ids (lazy and online) or fails with `GapInView` using the first reported path and its filter prefix (otherwise), and calls again. Each round is idempotent, and progress is by presence. The merged tree can also carry a subtree that is absent **inside the sparse view** (the fetch case from ADR-0048). So after the merge and before materializing, the caller runs `ensure_view(merged_root, sparse)`, as every other materializer site does. The claim that network traffic is bounded to the differing region is therefore "the differing region, the rule-intersecting region, and the view's closure". From 09ddc568c6eeb36fc0921b6aef45df10a2da8529 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 17 Oct 2026 12:04:40 +0300 Subject: [PATCH 6/6] docs(adr): ADR-0049 Needs covers blobs, mode vote skips absent entries, carried-root variant Co-Authored-By: Claude Sonnet 5.5 --- docs/adr/0049-merge-on-partial-clones.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/0049-merge-on-partial-clones.md b/docs/adr/0049-merge-on-partial-clones.md index 68fdcd2d..0793484c 100644 --- a/docs/adr/0049-merge-on-partial-clones.md +++ b/docs/adr/0049-merge-on-partial-clones.md @@ -28,7 +28,7 @@ A single composite key `(id, mode, perms)` is not the answer. It turns a metadat So 37c-i separates the dimensions, in **both** engines: - **Content** resolves exactly as today: by blob id (and the `PROTECTED` bit for protected paths). Nothing about content resolution, diff3, sealing or wraps changes. -- **Mode** is merged afterwards, independently, as a three-way on the modes of the entries that exist: equal on both sides, take it; one side equal to base, take the other's; both sides changed differently, executable if either is (today's rule for that case, kept as the tiebreak). Example: base `(X,0755)`, ours `(X,0644)`, theirs `(Y,0755)` resolves content to Y (ours is unchanged from base by id, so theirs wins) and mode to `0644` (ours changed it, theirs did not), so ours' executable removal is kept. +- **Mode** is merged afterwards, independently, as a three-way on the modes of the entries that exist. It only votes between entries that are both present: when one side has no entry (a delete/modify conflict, where content keeps the surviving side), the result takes the **surviving entry's mode**, and a deletion contributes no mode. Between two present entries: equal on both sides, take it; one side equal to base, take the other's; both sides changed differently, executable if either is (today's rule for that case, kept as the tiebreak). Example: base `(X,0755)`, ours `(X,0644)`, theirs `(Y,0755)` resolves content to Y (ours is unchanged from base by id, so theirs wins) and mode to `0644` (ours changed it, theirs did not), so ours' executable removal is kept. This is a deliberate, small behavior change on full clones: a mode-only change on one side now merges. It also makes the tree-level shortcuts below exact, because identical tree ids imply equal content and equal mode at every leaf, so "carry the differing side" is the per-dimension three-way answer. ADR-0012 gets an amendment note, and the cases above are regression tests (mode-only, identical-content/different-mode on binary and protected-keyless content, executable removal). @@ -70,7 +70,7 @@ Over-retention costs size: wraps are inline in the snapshot, and an object above ### 5. The engine is pull-based; filling happens outside the store lock -`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(missing)`, where each entry is `(ObjectId, path)`: the subtree it must read and does not hold, with the path at which the traversal found it (an id alone does not identify a path, since subtrees alias, and an absent root has no path to recover from tree data; the root is reported as the empty path). The caller drops the lock, fills those ids (lazy and online) or fails with `GapInView` using the first reported path and its filter prefix (otherwise), and calls again. Each round is idempotent, and progress is by presence. +`Filler` takes the store mutex (`fill.rs:283`), and the current merge engine runs while holding it (`repo.rs:1362`). A callback from inside the merge would deadlock. Instead `merge_trees` returns either a finished `TreeMerge` or `Needs(missing)`, where each entry is `(ObjectId, path)`: a subtree **or blob** it must read and does not hold (the leaf merge and the encryption step read blob bytes, so a differing or rule-intersecting region outside the filter needs its blobs too), with the path at which the traversal found it (an id alone does not identify a path, since subtrees alias, and an absent root has no path to recover from tree data; the root is reported as the empty path). The caller drops the lock, fills those ids (lazy and online; blobs are ordinary by-id requests, batched and split like trees) or fails with `GapInView` using the first reported path and its filter prefix (otherwise), and calls again. Each round is idempotent, and progress is by presence. The merged tree can also carry a subtree that is absent **inside the sparse view** (the fetch case from ADR-0048). So after the merge and before materializing, the caller runs `ensure_view(merged_root, sparse)`, as every other materializer site does. The claim that network traffic is bounded to the differing region is therefore "the differing region, the rule-intersecting region, and the view's closure". @@ -80,7 +80,7 @@ Failure contract: fetched objects persist (fill rounds write immutable CAS addit ### 6. The result keeps plaintext out of the CAS -`TreeMerge` is not a bare tree id. It returns what `FileMerge` returns today (the enumerated `MergedFile` list with `needs_encrypt` plaintext, sidecars, conflicts, `wrapped_carry`) **plus** the carried subtrees as `(path, TreeEntry)` pairs. The existing pipeline (`repo.rs:1385-1530`) handles the enumerated files unchanged (routing through union rules, encryption, plaintext never written to the CAS). The carried entries are overlaid by id when the final tree is assembled: a recursive overlay rebuilds each ancestor directory from the enumerated files at that level, inserts the carried entries verbatim (id, mode, perms untouched), and keeps the enumerated siblings, with the collision check above run over the combined names. No new code path touches plaintext. +`TreeMerge` is not a bare tree id. It returns what `FileMerge` returns today (the enumerated `MergedFile` list with `needs_encrypt` plaintext, sidecars, conflicts, `wrapped_carry`) **plus** the carried subtrees as `(path, TreeEntry)` pairs. The existing pipeline (`repo.rs:1385-1530`) handles the enumerated files unchanged (routing through union rules, encryption, plaintext never written to the CAS). The carried entries are overlaid by id when the final tree is assembled. A carry at the **root** (the empty path) has no enclosing entry or ancestor directory, so it is its own variant: `TreeMerge` holds either a carried root id, which is the merged root as is (this is what two branches with disjoint secret-registry changes, or a secrets-only commit merged with a file-changing sibling, produce when the root ids agree), or a list of `(path, TreeEntry)` carries to overlay. Otherwise: a recursive overlay rebuilds each ancestor directory from the enumerated files at that level, inserts the carried entries verbatim (id, mode, perms untouched), and keeps the enumerated siblings, with the collision check above run over the combined names. No new code path touches plaintext. ### 7. Tree semantics, stated plainly