[mickvandijke]

Follow-up to #219 (ADR-0006). Closes the remaining safety gaps in file manifests across Windows, macOS and Linux. Four commits do the work and a fifth fixes what review found:

1. Portable names (811435a). Folding missed a merge APFS makes: ẞ lowercases to ß while ß becomes ss, so straẞe and strasse passed as two entries but are one file on a Mac. The key now applies full Unicode case folding after uppercasing, and drops the code points HFS+ ignores (ab / a<U+200C>b). It was checked against every BMP character in the Windows XP and Windows 7 NTFS upcase tables and against every cased character on a live case-insensitive APFS volume, and misses no merge either makes. Components also reject C1 controls, which some terminals act on when entry names are printed, and bidi override and isolate controls (invoice<U+202E>fdp.exe). Joiners and directional marks stay allowed.
2. Extraction containment (75a4815). Extraction used to check directories by path and then create, re-read and rename the file by path. A process able to write inside the output root could swap a checked directory for a symlink in between and get the file written wherever the link pointed. The ADR listed this as the remaining hardening step. Every name is now resolved from directories held open. On Unix they are descriptors and every operation is *at (via rustix). On Windows the chain from the root down is held without delete sharing, so it cannot be renamed, deleted or replaced by a junction. The final step is a no-replace rename. SHA-256 is computed while writing (new Client::file_download_to_writer), and the file is synced before it takes its name. Overwrite also never replaces a file written earlier in the same extraction, so a long name and its Windows 8.3 alias cannot silently become one file.
3. Size budget (ee8c6c7). About a thousand embedded root maps (64 KiB each) can exceed the 64 MiB manifest limit, which was only checked after every upload was paid for. Now each file is uploaded only if the largest entry it can become still fits. The larger root-map reference is used only when it fits, and the shrunk map is kept otherwise.
4. Crash safety (e31cb8a). A crash, kill, power cut or second Ctrl-C used to lose every DataMap uploaded so far, and for private uploads those DataMaps exist nowhere else. The builder now writes a CRC-checked journal (<id>.ant.partial) to the upload history and flushes each entry before paying for the next upload. The journal becomes the ordinary record when the build ends. A leftover journal lists as an interrupted upload, and show, link, export and download all accept its id.
5. Review fixes (4c3c545), from a high-effort code review of this PR:
   • The size budget counts uploads at their embedded map, so embedded-mode builds that cannot fit are refused before paying.
   • History records written under earlier rules are repaired on load instead of becoming unreadable.
   • Ids are claimed through the journal itself, so records and builds never share one.
   • The history directory is synced before a journal is removed.
   • In-progress ids are refused by the CLI.
   • Extraction no longer fails an entry after its file is in place.
   • The fsync runs off the async workers.
   • A late canonicalize failure keeps the partial result.

Linear issue

Closes V2-1478

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none. Uploads, downloads and chunk stores go through the existing client paths. The upload path is unchanged; the download is the same chunk fetch and decrypt loop, writing into a file opened by the caller.
• Storage: none on the network. .ant bytes are unchanged. Adds a local upload-history file, the build journal <id>.ant.partial (magic C1 41 4E 4A, version 1, frames of little-endian u32 length + CRC-32 + named msgpack), specified in ADR-0006. Older clients list only .ant files and ignore journals. The path rules are stricter: names containing C1 or bidi controls, and entries that collide only under the new key (ẞ/ss, HFS+ ignorables), were accepted before and are now rejected at build, decode and extract time. The upload history is the exception. A record written by an earlier client is repaired when it is loaded, rather than refused: names are made portable, and the later of two colliding entries extracts under its content address. Recorded uploads therefore stay reachable.
• API: breaking (pre-1.0). Removed the path-based extract::prepare_target. New public fields BuildOptions::history, BuildResult::{history_id, history_error} and UploadRecord::state, plus the RecordState enum (re-exported). Struct literals must set the new fields. Added Client::file_download_to_writer. CLI changes:
  • ant manifest create records the build itself through the journal, and refuses to start, before paying anything, if the upload history cannot be written. Previously it warned after uploading.
  • ant manifest list marks in-progress and interrupted uploads, and its JSON gains state. Commands that take a record id refuse an upload still in progress, and warn that an interrupted one is partial.
  • ant file upload records claim their id through the same journal mechanism, with an unchanged file layout.
  • download --overwrite no longer replaces a file written earlier in the same extraction.

Semver impact

• breaking
• feature
• fix

Test evidence

• cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, and cargo clippy -p ant-core --target wasm32-unknown-unknown --no-default-features --features browser-wasm -- -D warnings: all clean.
• cargo test --lib --all: 862 passed. New tests:
  • Path rules: C1 and bidi controls rejected, joiners and marks kept, HFS+-ignorable collisions, ß/ẞ/ss.
  • Containment: writes beneath the root and refuses existing names; overwrite replaces only a pre-existing regular file; an uncommitted temporary file is removed; a name that appears during the download is not replaced, including at the rename itself; links at every depth are never followed; a directory swapped for a symlink mid-download cannot redirect the file; a second name for a file this extraction wrote (a hard link standing in for an 8.3 alias) is not replaced.
  • Size budget: the bound covers a real shrunk map; every real upload map is at least the counted smallest; a long-path build that would fit as addresses but not embedded is refused before paying; the fixed part is exact to within the array-header growth; admission and reference choice under a full budget.
  • Journal: an interrupted record holds exactly the entries written, with the later record of a name winning; it lists as in progress while locked; a torn or damaged last frame keeps every earlier frame; finishing writes the record and removes the journal; journals and records never share an id; a leftover journal beside its record is ignored, as is one cut short before its opening; recording a single upload leaves no journal.
  • History: a record written under looser rules loads repaired with no entry lost.
  • The streaming SHA-256 writer.
• cargo test -p ant-cli --bin ant: 32 passed, including in-progress ids being refused and interrupted ones used.
• cargo test -p ant-core --features test-utils --test e2e_manifest -- --test-threads=1: 7 passed in 286 s against the in-process testnet and Anvil, all through the new extraction path including SHA-256 checks. The failed-build test now shows the first paid upload in the in-progress journal before the build continues, the partial result recorded by the builder itself, and no record for a build cancelled before any upload.
• Folding was checked with a probe that rebuilds the default NTFS upcase tables (from ntfs-3g's ntfs_upcase_table_build, XP and Win7 variants) and compares every BMP character. It also creates a file for every cased character on this Mac's case-insensitive APFS and records which names APFS merges. Result: 0 misses on all three; the same probe found the ẞ gap in the previous fold.
• Windows: the containment module is type-checked and clippy-clean for x86_64-pc-windows-msvc in an isolated crate with the same windows-sys, tempfile and rand versions. ant-core's C dependencies do not cross-compile here, and CI has no Windows runner, so the Windows path has not been run.
• python3 scripts/adr-governance.py: passes.
• Review: /code-review high on this PR raised 10 findings, which I checked against the code. Nine were real and are fixed in 4c3c545. The tenth, that removing prepare_target is breaking, is already declared below.

New dependency

No new crates in Cargo.lock. Three crates that were already in the tree become direct ant-core dependencies:

• unicase 2.9, for full Unicode case folding (already present via webrtc).
• crc32fast 1.5, for journal frame checksums (already present via flate2).
• rustix 1.1 with fs, Unix only, for directory-relative file operations (already present via tempfile).

The existing Windows windows-sys 0.61 dependency gains the Win32_Storage_FileSystem feature, used for file identity.

ADR

docs/adr/ADR-0006-file-manifests-and-links.md (status Proposed, amended by this PR: portable path rules and the measured folding, the size budget, the upload journal format and states, handle-relative containment, and the test list).

Mitigation / rollback

Revert the PR. Nothing on the network changes. Journals already written stay as *.ant.partial files in the upload history, where older clients ignore them. Manifests rejected by the stricter path rules become readable again.

🤖 Generated with Claude Code