[aien-spark]

Stacked on #90, retarget to main after #90 merges. Base is feat/48-git-bridge, so this diff shows only cut 2.

Closes #48 (with #90): all three acceptance conditions are met once both merge.

What cut 2 adds

• WAL record kind 10 GitBridgeImported (binary codec only). Open PR feat(b4): revision state machine Private..Canonical (ADR 0003 B4) (#28) #88 claims kind 9, so this takes 10; nothing here depends on 9.
• Persistent map in the store (synapse-store/src/git_bridge.rs): Git blob to (BlobDigest, size), Git tree to TreeDigest (canonical C1 node stored), Git commit to (ChangesetId, TreeDigest) and back, tags, references, HEAD. Imported revision ID = SHA-256 over tree, parents and the original header bytes; the Git ID is rechecked at every export. Rebuilt on replay and checkpoint load; compaction keeps the blobs it names. B1 revisions may name an imported commit as a parent. record_git_import is the only writer and checks every entry first.
• Import (synapse-git/src/bridge.rs): the whole history is converted in memory before the first write. Anything with no exact native form (gitlinks, legacy 100664, padded 040000, non-UTF-8 or C1-refused names such as .git, empty subtrees, non-canonical trees) is GitObjectNotRepresentable (new refusal code). Unsafe reference names are PathInvalid. Both leave the store unchanged. Importing the same history again writes nothing.
• Export from the store: every imported object is rebuilt and must hash to its imported Git ID. Native revisions export deterministically (author = attestation DID, fixed committer, time 0 +0000, Synapse-Changeset-Id/Synapse-Change-Id trailers). A native child exports on its parent's result tree.
• feat(git): H1 cut 1, native Git objects, packfiles, full-history read and export (#48) #90 re-review items folded in: (a) fan-out directories under objects/ must be real directories (refs/ already were); (b) the standalone pack indexer honours the ReadLimits total budget (and resolves late reference-delta bases with a waiter list); (c) peak RSS against the budget is stated in the doc.

Red before green (evidence/h1-git-bridge/red-green/)

cut2-red.txt: 7 stubs, each removing one behaviour, each fails its test by assertion. S1 canonical-form check skipped, S2 headers dropped on import, S3 map not rebuilt on checkpoint load, S4 native commit time from the clock, S5 reference names left to the store (objects written first), S6 compaction drops bridge blobs, S7 native result trees not remembered. rereview-red.txt: N1 fan-out check removed, N3 indexer budget removed, both fail. cut2-green.txt: passing runs.

Commands and counts

• cargo test --locked --workspace -j 4: 371 passed, 0 failed, 3 ignored (the ignored are real-repo tests that need a mirror path). synapse-git tests: h1_bridge_import 8 + 1 ignored, h1_git_bridge 6 + 1 ignored, h1_hostile_repo 9, h1_read_limits 4, plus 2 new codec/ID unit tests in synapse-store.
• cargo clippy --locked --workspace --all-targets -- -D warnings: clean. cargo fmt --all --check: clean.
• Crumb: recompiled and crumb verify OK in a clean detached worktree with the tool pinned by crumb.lock (aien-architecture d0f0b95).

Real repositories through the store (mirror clones; import into a fresh store, reopen, export from the store, compare every reference and rev-list --objects --all against the export, git fsck --full --strict). Release test binary, /usr/bin/time -v:

Repository	Objects	Refs	Records	Import	Export	Peak RSS	Result
aien-dev/synapse (main 3d78efe)	1,915	75	183	7.5 s	1.0 s	197,116 KiB	identical IDs and refs, fsck strict clean
aien-dev/omega (main f4a9849)	29,636	633	2,433	123.3 s	5.8 s	976,824 KiB	identical IDs and refs, fsck strict clean

Omega peak RSS is 0.9316 times the 1 GiB default read budget (cut 1: 1.063 times). The budget bounds resolved object bytes, not the whole process. The times are measured, not profiled, and no section 6 performance claim is made.

Limits (details in docs/H1_GIT_BRIDGE.md section 4)

• A detached HEAD commit is imported, but the export's HEAD names refs/heads/main.
• Native export needs base_tree to be empty, imported, or the result of a native parent exported in the same call. A base registered only in the API tree store is TreeNodeNotFound. Native refs are passed in by the caller and not stored. Native commits carry no signature.
• If an I/O failure hits part way through the writes, the records already written stay. Re-import skips them and resumes.
• Check-then-open race: O_NOFOLLOW only guards the last path component, so a directory swapped for a link between the check and the open is not caught. Fixing it needs openat walks.
• Packs are stored-block zlib with no deltas (omega export 374,713,444 bytes). SHA-1 repositories only.

CI note: the "AIEN Verification" check fails because synapse is not enrolled in the Spark pipeline (main fails it too). CodeQL is checked on the head after each push.

🤖 Generated with Claude Code

[aien-spark]

Independent review (session omega repo issues, reviewer did not author)

Head reviewed: b69db2d, diffed against feat/48-git-bridge (28d783c).

Verdict: CHANGES REQUESTED. Two blocking findings. I confirmed both with throwaway probe tests (not pushed). Everything else I checked holds up.

Blocking

B1. A hostile repository can get a partial import that never finishes. This breaks the claim "a refused repository leaves the store unchanged" (bridge.rs:8-9, bridge.rs:157-160, doc 1b "Import").
import_history only checks blob size against max_blob_bytes before it starts writing (bridge.rs:190). It never checks the encoded size of each record against max_wal_record_bytes. A Commit record carries the whole commit after its link lines (bridge.rs:309), and a Tag record carries rest. One tree is also a single Objects entry, with nothing splitting it. The frame cap is checked only inside append_revision_record (check_frame_cap, revision.rs:601), and that runs after every blob and tree record has already been written. With the default limits the gap is real: ReadLimits::max_object_bytes is 1 GiB (repo.rs:99) but max_wal_record_bytes is 128 MiB (lib.rs:202). So a repository with one commit whose message is 129 MiB to 1 GiB (or one tag, or one very wide tree) reads fine, converts fine, writes all its objects, and then gets refused QuotaPayloadTooLarge. Importing again fails the same way at the same record. The blobs already written stay pinned for good, because compaction keeps every blob the bridge map names (lib.rs:1663).
Probe: limits max_blob_bytes 4 KiB, max_total_changeset_bytes 16 KiB, max_wal_record_bytes 64 KiB, and one commit with a 100 KiB message. Result: Err(QuotaPayloadTooLarge), committed_seq 0 to 1, map holds 2 objects.
Fix: in the conversion phase, compute every record's encoded size (or bound headers, rest and node bytes) and refuse before the first write. A test should go with it. The same check would catch a custom config where max_blob_bytes > max_total_changeset_bytes: the flush at bridge.rs:413 sends an oversized blob to the store alone, and the store's max_total_changeset_bytes check (git_bridge.rs:569) refuses it mid-import.

B2. record_git_import lets a single record poison the map for good (git_bridge.rs:361-375, 398-404, 448-456).
The doc says the store refuses "a Git ID mapped again to different content" before writing. The check only compares against mappings that are already committed. It never compares inside the entry being checked, and it never checks a Git ID against content the store holds. Example: an Objects entry with X -> blob A and X -> blob B is accepted, and apply keeps the last one (git_bridge.rs:294). After that, export_store fails GitObjectCorrupt every time (B's bytes do not hash to X). A correct re-import is refused too, as a conflict with X -> B, and no record can ever replace a mapping. So that repository can never be exported or repaired. The same goes for a blob mapped under any wrong Git ID, and for two commit Git IDs with the same imported revision ID (by_changeset is overwritten at git_bridge.rs:305).
Probe: the entry above, sent straight to record_git_import, gave Ok(Some(WalSeq(1))), the map shows X -> B, and the export gives Err(GitObjectCorrupt).
Your own crafted-entry test frames this exact caller ("a replica or a buggy importer"). Fix: refuse duplicate Git IDs inside one entry, and check each Git ID at write time. All the inputs are on hand: blob bytes are provided or visible; tree IDs can be rebuilt from the node plus the child IDs already in the map, the same way git_tree_of_node does it; commit and tag IDs from their links plus stored bytes. The SHA-1 helper is already in synapse-core.

Non-blocking

• N1 (PLAUSIBLE, by design): replay does not re-run check. GitBridgeLedger::apply is total (git_bridge.rs:286), so a well-formed but crafted kind 10 frame gets past the semantic checks. The export re-hash catches forged blob, tree, commit and tag IDs, but a Refs or Tag target Tag(oid) is written without any check (bridge.rs:494). A crafted record could therefore export a dangling ref. Ref names are checked again at export (bridge.rs:862), so a crafted WAL cannot write outside the export directory. This matches how kinds 7 and 8 replay.
• N2 (PLAUSIBLE, existed before this PR): try_resolve_at holds a chain of up to 4,096 inflated delta streams (pack.rs:500-525). The streams themselves are not counted in max_total_object_bytes, only declared result sizes. Copy instructions can make a stream several times larger than its result. Worth a sentence in the memory paragraph of the doc.
• N3: a tree File entry that names a blob which is visible but not mapped by the bridge has no size check (git_bridge.rs:410-428, known_size is None). Git IDs are unaffected (Git trees carry no sizes), but the C1 node can hold the wrong size.

Verified OK

• Kind 10 codec: decode is strict. It uses tagged targets, r.vec bounded by remaining bytes, tree nodes decoded through decode_tree_node (which re-encodes through TreeNodeEncoder, so canonical form is enforced), strictly ascending ref names, and refuses trailing bytes. Every truncation is refused (unit test). There is no JSON form. Kinds 9 and 11 still decode as UnknownTag, and the checkpoint revision log accepts only 7, 8 and 10 (codec.rs:1266). Nothing depends on 9 or 11.
• Persistence: kind 10 records stay in revision_log. The state root covers them through record_digest_input (revision.rs:144). A checkpoint load rebuilds the map (lib.rs:2241), and compaction keeps the bridge blobs.
• Export rebuilds every imported blob, tree, commit and tag from stored bytes and checks each hash against its imported ID (add_checked). It never trusts a stored ID except Tag(oid) targets (N1). Native export is deterministic: BTreeMap object set, fixed time and committer, the hash maps are used only for lookups.
• Hostile repo: modes, names, padded 040000, ordering and duplicates are refused through the canonical rebuild comparison (bridge.rs:251-271). Unsafe ref names are refused before any write. objects/, its fan-out directories and pack/ must be real directories. The standalone indexer checks the declared total before resolving anything, and the waiter list resolves late reference-delta bases.
• SHA-1 comes from the sha1 crate (synapse-core/src/lib.rs:333), SHA-256 from sha2. No hand-rolled crypto.
• Memory figures: all four aien-calc receipts in doc section 4 verify as MATCH (calc:f4b7563320d5, calc:b42240ae452f, calc:3499282d5a7c, calc:15f1957f5764), and I recomputed 976,824 KiB / 1 GiB = 0.931572 and 976,824 KiB / 374,301,457 B = 2.67236 independently. These are consistent with the stated 0.9316 and 2.672.

Tests rerun

• cargo test --locked --workspace -j 4 (nice 19, quietlock clear): 371 passed, 0 failed, 3 ignored. This matches the PR body. h1_bridge_import 8 + 1 ignored, h1_git_bridge 6 + 1 ignored, h1_hostile_repo 9, h1_read_limits 4.
• Red check: I disabled the fan-out real_dir guard (repo.rs:277). h1_hostile_linked_object_fanout_directory_is_refused FAILED and the other 8 passed. I then restored the guard.
• Two probe tests for B1 and B2 (results above), deleted afterwards.

Not checked

• The real-repository runs (the ignored tests that need mirror clones), and the timing and RSS numbers themselves (only the arithmetic).
• Whether the native mutation rule (an upsert replaces ancestor files and everything below; a delete removes only a file) matches the B1 tree store byte for byte. B1 revisions carry no result tree to compare against.
• Clippy, fmt and crumb verify.
• The openat race (already listed as a limit).

🤖 Generated with Claude Code

[aien-spark]

Review of b69db2d addressed in ceae255 (fixes in d0e9a83, crumb recompile in ceae255). Both blockers fixed red before green, plus N3 and the export tag check.

B1 partial import. New SynapseStore::record_git_imports(repo, records): a batch of more than one is first run record by record against a copy of the map (earlier records of the batch applied), with every check the write does: codec, blob caps, max_total_changeset_bytes, digests, the map checks below, and the exact WAL frame size (metadata encoded as the write encodes it, plus inline blobs) against max_wal_record_bytes. Only then are the records written, each checked again against the live map. record_git_import is a batch of one. The importer builds every record first and makes one call; Objects batching also keeps an upper bound of the frame under the cap. A commit, tag or single tree over the cap, or a blob over max_total_changeset_bytes, is now QuotaPayloadTooLarge with committed seq still 0.

B2 map poisoning. GitBridgeLedger::check now refuses a Git ID twice in one entry and a second Git ID for one commit's content, and re-hashes every newly mapped Git ID with the workspace SHA-1 (GitOid::compute_git_object): blob from its bytes (provided or read from the store), tree from its C1 node rebuilt as a Git tree by the new synapse_core::git_tree_payload_of_node (export uses the same function), commit from tree and parent lines plus stored headers, tag from its object line plus stored bytes. Mismatch is GitObjectCorrupt. The repo map keeps reverse maps (digest to Git ID) for that.

Non-blocking. N3: a tree file entry must name a blob mapped by a Git blob of the same repository, with the same size (BlobNotFound / BlobDigestMismatch); subtrees, commit trees and targets must also be imported into the same repository. Export: ImportedIds keeps the tag set, so a Tag(oid) target that is not an imported tag is GitObjectCorrupt. N1 (replay trusts kind 10, as kinds 7 and 8) and N2 (delta streams not counted) are written up as limits in docs/H1_GIT_BRIDGE.md.

Red (evidence/h1-git-bridge/red-green/review96-red.txt)

• Baseline: the two new tests, unchanged, against b69db2d both fail: "one Git ID mapped to two blobs in one entry: accepted (Some(WalSeq(1)))" and "commit record over max_wal_record_bytes: nothing written" (left WalSeq(1), right WalSeq(0)), your two probes.
• Eight stubs on the fixed code, each alone, all fail by assertion: R1 dry run skips frame size, R2 importer writes record by record (both: nothing-written assertion); R3 to R6 blob, tree, commit, tag Git ID not re-hashed (each: "... under a wrong Git ID: accepted"); R7 tree file blob need only be visible (wrong refusal code); R8 export passes any Tag(oid) through (unit test).
• No stub for the duplicate-ID and same-content-commit checks: the re-hash already refuses every case they catch; they are kept so the refusal names the fault.

Green (review96-green.txt)

• cargo test --locked --workspace -j 4: 374 passed, 0 failed, 3 ignored (was 371; +2 integration, +1 unit). Clippy -D warnings and fmt --check clean. Crumb recompiled and crumb verify OK with the pinned tool (crumb.lock d0f0b95) in a clean detached worktree.
• Both new tests then import correctly into the same store after the refusals (full repository under default limits; the fixture into the repository that took the refusals) and export with identical IDs, fsck strict clean.
• Real repositories (release, /usr/bin/time -v, same mirrors): synapse 1,915 objects, 75 refs, 183 records, import 8.1 s, export 0.9 s, peak RSS 197,056 KiB; omega 29,636 objects, 633 refs, 2,433 records, import 133.7 s, export 15.1 s, peak RSS 977,812 KiB. Identical IDs and refs, fsck strict clean. The host was loaded (load average about 14.5 on 20 cores), so the times are not a clean comparison with the earlier run; peak RSS is 0.101% above it (calc:f87ba615b0d5).

Limits. An I/O failure during the write phase can still stop a batch part way (a re-import finishes it). The dry run copies the whole bridge map while it runs. CodeQL default setup does not run on this PR (base is not the default branch). "AIEN Verification" fails because synapse is not pipeline-enrolled (main fails it too).

[aien-spark]

Independent review (session omega repo issues, reviewer did not author): re-review

Head reviewed: ceae255 (fixes in d0e9a83), compared with b69db2d.

Verdict: APPROVE-READY (no blocking findings). Both blockers are fixed. I confirmed each fix in the code, with the test suite, with red checks, and by rerunning my original two probes.

B1 (partial import): fixed

• import_history now builds every record and makes one call to record_git_imports (bridge.rs, end of import_history). For a batch of more than one record, the store first runs every record against a clone of the map with the batch's earlier records applied. That dry run covers codec, blob caps, max_total_changeset_bytes, digests, every check in GitBridgeLedger::check, and the frame length against max_wal_record_bytes (git_bridge.rs:686-701). Only after that does it write anything.
• git_import_frame_len matches the real v2 payload layout in wal2::encode_record exactly: 4 + meta + 4 + Σ(32 + 4 + len). On v1 the metadata length equals what encode_v1_payload caps.
• My original probe (commit with a 100 KiB message, 64 KiB frame cap) now gives QuotaPayloadTooLarge with committed_seq still 0 and no map.

B2 (map poisoning): fixed

• A Git ID used twice in one entry is refused. A second Git ID for the same commit content is refused. Every newly mapped Git ID is re-hashed with GitOid::compute_git_object (the sha1 crate): blobs from provided or stored bytes, trees through the shared git_tree_payload_of_node, commits and tags from their rebuilt link lines plus stored bytes.
• My probe: duplicate X → GitObjectCorrupt; X mapped to the wrong blob → GitObjectCorrupt; the correct mapping afterwards → Ok(Some(WalSeq(1))).
• Extras also confirmed: a tree's file entries must name a blob mapped in the same repository with the same size, and subtrees, commit trees and targets must be imported into the same repository (target_oid). Export refuses a Tag(oid) target that is not an imported tag (ImportedIds::tags). The reverse maps are rebuilt by apply, so they survive checkpoint load and replay.

New holes in the batch path: looked for, none found

• Validation and write cannot disagree. The live pass runs the same prepare_git_import on the same records in the same order. The dry run's sim_visible holds exactly the blobs that earlier records put inline, and those are exactly the blobs the live blob_first_visible_seq gains, since every other mapped blob was already visible. So inline sets, frame lengths and refusals match. All of it runs under &mut self, so nothing can interleave.
• The dry run calls read_blob only for a newly mapped Git ID whose blob is not provided. A blob that is visible only inside the simulation would have to be under a second Git ID for the same bytes, and the re-hash refuses that anyway. So the store copy and the real map cannot drift into a false accept.
• The single-record fast path (no copy) is safe. v2 checks the frame cap before writing (revision.rs:601). v1 caps the payload in encode_v1_payload before any blob file is written (revision.rs:609-611). An import with no objects is just a lone Refs record, which takes this path.
• Remaining partial-write cases are I/O failure or check_writable flipping mid-batch. Both are documented, and a re-import finishes the batch.

Non-blocking

• The dry run clones the whole GitBridgeLedger (every repository's maps and every tree node) on every multi-record call. For omega the peak RSS barely moved (977,812 vs 976,824 KiB, receipt calc:f87ba615b0d5 verifies as MATCH). On a store holding many imported repositories, the cost grows with the whole bridge, not with this import. A per-repository overlay would avoid that. This is already listed as a limit.
• N1 (replay trusts kind 10) and N2 (delta streams not counted in the budget) are now written up as limits in the doc, which is fine.

Tests rerun

• cargo test --locked --workspace -j 4 (nice 19, quietlock clear): 374 passed, 0 failed, 3 ignored. This matches the builder.
• Red checks, each on its own and restored afterwards:
  • Dry run disabled (sim always None): h1_import_refuses_a_record_over_the_store_limits_before_writing FAILED (nothing-written assertion).
  • verify_git_id forced to accept: h1_store_refuses_git_ids_that_do_not_match_their_content FAILED ("a blob under another blob's Git ID: accepted").
• My two original probes, rerun on ceae255: both now refused with nothing written (results above). Probe files deleted.

Not checked

• The real-repository runs and the timings themselves.
• Clippy, fmt and crumb verify.
• Whether the native mutation rule matches the B1 tree store (same as before).

🤖 Generated with Claude Code

[aien-spark]

This was generated by AI during triage.

The rebase train (tools/aien-local-ci/rebase-train.sh) could not move this pull request onto the latest
main by itself: rebase onto origin/main conflicts. It resolves only .crumb files (generated by the crumb compiler) on its own.

Files that need a person (or an agent) to resolve:

• crates/synapse-git/src/inflate.rs
• crates/synapse-git/src/lib.rs
• crates/synapse-git/src/pack.rs
• crates/synapse-git/src/repo.rs
• docs/H1_GIT_BRIDGE.md
• evidence/h1-git-bridge/SHA256SUMS

Rebase the branch onto main by hand, resolve these files, and push. The needs-rebase label was added;
the train tries again on every run and removes the label once a rebase goes through.

[drakestapleton]

Dependency audit: prerequisite #90 is already merged as 2f06c455a5af8a5db7e0230c3692f08db21f61f2. Route #96 to current-main integration and a fresh exact-head signed check, preserving its remaining source-review and acceptance requirements.

All-PR merge desk: aien-dev/aien-architecture-mv#190. Existing implementation ownership is retained; this is not a merge approval.

[aien-spark]

parked: GPU-stack focus; label returns when resumed