[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job (merged) · #740 the restore job, the cross-backend matrix and backup remove-target (merged) · this PR: extenddb backup sync, --prune-missing, --adopt-store and the sqlite-memory startup sync · next: #820 the store-maintenance options, #821 migrate-legacy.
Closes #746 (part of #624).

Based on main at 887b6e26 (#840). Rebased from main at b8ad9cd5 (32 commits, #474's BackupGsi/BackupLsi manifest types among them) with no conflict; the change set is the same (47 files, +12,923 −883), and fmt, clippy 1.97.0 in the three feature sets, the engine, app, server, core and storage --lib suites, SQLite --lib backup, PostgreSQL backup_restore and the sync, prune and adopt wire cells on the SQLite and PostgreSQL builds pass on the rebased tree. Before that, based on main at b8ad9cd5 (#740 merged as 76b9710d, then #792), rebased with git rebase --onto origin/main 9803ab7f from #740's head 9803ab7f, which this PR was stacked on; #740 merged with the content of its later head 15f38c72, 143 files away from 9803ab7f. The rebase conflicted in one file, the design's revision log: main's item 33 is kept as main has it (it gained the sentence on migration 8) and this PR's item 34 follows it unchanged, so no item is renumbered; the seven other files both sides changed (backup/tests.rs, Cassandra's backup_engine.rs direct tests, the SQLite and trait backup_catalog.rs, the differences document, the admin guide and the upgrade manual) merged without a conflict, and the commit is the same size as before (44 files, +10,195 −758). Before that, it was rebased from #740's earlier head 6bdd1177 onto that stack after its later revisions and main's eight commits under it: PR 10 now meets #740's bounded store calls, its DeleteBackup that answers DELETED once the DELETING write commits, its CreateBackup that checks the maintenance claim before any row, its signal listener opened before every command's catalog, its purge_rows and its MongoDB deployment of tests/test_backup_job.py, and is adapted to each (below). The rebase conflicted in 14 files, each resolved by keeping both sides' intent: crates/app/src/cmd_backup.rs (#740's run with sync added; sync takes the run's signal handle), crates/engine/src/backup.rs (DeleteBackup's second fence logged and answered DELETED), backup/job.rs, backup/maintenance.rs (the fence threaded through purge_rows; the skip of FAILED maintenance rows dropped, #700 writes no such row), backup/reconciler.rs, backup/runtime.rs (both sides' ConfiguredStore fields; the fence's read through BoundedCalls), backup/tests.rs, crates/storage-cassandra/src/test_hooks.rs, crates/storage-postgres/src/backup_catalog.rs, docs/design/15-backup-restore.md (this PR's revision item renumbered 34 after #740's 28 to 33), docs/differences-from-dynamodb.md, docs/manuals/05-admin-guide.md, docs/manuals/06-developer-test-guide.md and tests/test_backup_job.py (the sync cells then skipped on the MongoDB deployment; they now run there, below). The design's PR 10 is split in three (revision item 34): this PR is the catalog-loss recovery, #820 the options that rewrite or delete rows of other stores, #821 the legacy-backup migration. No migration, no schema change, no new dependency; Cargo.lock is unchanged. The identity file stays format version 1: its one new member is #[serde(default)], so this release and #740's read each other's files.

What

Storage trait (crates/storage). Two BackupCatalog methods, with no default body. insert_backup_row(&BackupRow) -> bool, beside claim_backup: insert the row as given, unowned (owner_generation 0), its status as given, completed_at and heartbeat_at on the database's clock; false when a row with its key exists, or with its backup id on the three backends that index it. The trait doc says that Cassandra's check is per ARN spelling and that sync closes the gap, and that on Cassandra a stored row equal to this one counts as inserted. forget_available_backup_row(&BackupKey) -> ForgetRow: remove an AVAILABLE row outright under the row's lock with start_backup_delete's in-use rule, leaving no tombstone and touching no object; InUse(table) and Status(status) leave the row. MaintenanceClaim gains heartbeat_within(bound) and heartbeat_parks_before_its_bound(), with JobClaim's defaults (the whole call bounded; false); SQLite overrides both, its bound starting once the beat holds the engine write lock, and true. StorageConfig::database_location keeps the reason a location names no file.

Core (crates/core). layout.rs gains EXPORTS_COMPONENT, CHANGELOG_COMPONENT, StoreKey and classify_store_key(key), which tells a key outside every backup prefix, a key under a reserved component (exports/ first, changelog third), three components that are not a backup prefix (another id shape, a component no ARN carries) with the reason, and a backup prefix with its account, table and id; parse_store_prefix(key) is its Option form. Only the 14-digit id shape is a backup prefix, and the triple is validated through BackupArn::new.

PostgreSQL, SQLite, MongoDB, Cassandra. insert_backup_row: PostgreSQL INSERT ... ON CONFLICT DO NOTHING with no conflict target (the key and the unique backup_id both refuse it), on the pool, with no advisory lock (a job that holds the id's lock and has not inserted its row yet finds this one and is refused). SQLite INSERT ... ON CONFLICT DO NOTHING (so a NOT NULL or CHECK violation is an error, not a false) under the write lock and BEGIN IMMEDIATE, without a job slot. MongoDB the claim's pipeline upsert filtered on owner_generation: {$exists: false}, writing owner_generation: 0 and $$NOW for the two stamps, E11000 (the unique backup_id) meaning a row exists; row_fields takes the status, and claim_backup passes CREATING. Cassandra the backups_by_arn lightweight transaction IF NOT EXISTS, then the LOCAL_QUORUM batch of backups_by_account, backups_by_table and the backup_index_by_status entry under the row's status (write_index_entries, the batch claim_backup wrote inline, now shared); an unknown outcome or a NotMatched answer is re-read at LOCAL_SERIAL and settled as inserted when the row read is this one (its entries then written, which completes an insert that stopped between its two writes), as existing when it is another; no row after a lost answer is sent once more (then an error), and no row after NotMatched answers false (a legacy in-catalog backup holds the ARN); a failed entry batch is sent once more. forget_available_backup_row shares one body with start_backup_delete on every backend, with a remove flag: PostgreSQL and SQLite start_backup_delete_tx(key, remove) (the row lock, the status check, the in-use rule, then UPDATE ... DELETING or DELETE); MongoDB start_backup_delete_tx with delete_one, its exhaustion error naming DeleteBackup or the --prune-missing removal; Cassandra start_backup_delete_lwt(key, remove), one loop under one condition (IF backup_status = 'AVAILABLE' AND active_restores = null) through conditional_write, where a removal's unknown outcome is settled by a LOCAL_SERIAL read (no row: removed, then the entries; a row: the loop runs again) and an entry batch that fails after the removal is logged and answered Removed. SQLite's claim_backup keeps INSERT OR IGNORE, with a comment saying why it ignores only a key conflict.

Engine (crates/engine/src/backup/). New sync.rs, written against BackupCatalog and &dyn BackupStore, which the CLI and the server both call:

• sync reads the configured root's rows of every status and every store id first, then lists the whole root and handles each prefix as the listing passes it (the store lists in key order; sync holds one prefix's objects and the prefix of each manifest it listed, and keeps of the rows only each one's status, store id and table and the keys of the configured store's AVAILABLE rows), checking it against the rows before it reads anything under it. A listing that fails part-way keeps the rows inserted before it and makes no missing-manifest report; a stop asked through SyncTarget::stop ends the run before the next prefix, and SyncTarget::claim, the claim the command holds, is re-checked with one beat right before each insert, a claim found lost ending the run there with that prefix not adopted (the startup sync passes None). A prefix with no row is verified by reconciler::verify_prefix (split out of probe_manifest, which maps to StatusFields and, like the restore's manifest check, accepts a manifest whose ARN differs from the row's in the region alone, BackupArn::names_same_backup): the manifest and sidecar within their bounds, the digest, the reader's acceptance, and every data file the manifest lists at its recorded size, from the listing already held. The manifest's ARN must name the listed account, table and backup id; a manifest of another region is adopted under this server's region's spelling of the ARN (BackupArn::in_region) and logged, so ListBackups names this server's region and DescribeBackup, RestoreTableFromBackup and DeleteBackup find the backup under either spelling, while the manifest keeps its own; another partition is skipped (no request can address it). Before the insert the row is read under both spellings of the ARN (this server's region and the manifest's) through the engine's BackupArnSpellings, and a backup the manifest's spelling, another region's, has a row for is skipped; a row under this server's spelling is left to the insert, which answers false for it or, on Cassandra, completes its own insert that stopped before its entries. The adopted row is the one a CreateBackup leaves: the manifest's name, size, item total and source_table, the ARN in this server's region, the id's instant as created_at, the configured store's identity. An insert that answers false is re-read by its ARN so the report says which row it met, with no listing of the rows of its status. A failed read of one prefix's manifest or sidecar (SkipReason::Store) or a failed catalog call (SkipReason::Catalog) is reported for that prefix and the sync goes on; SyncReport::transient_skips counts them. The report also lists the AVAILABLE rows whose manifest the listing lacks.
• prune_missing heads each such row's manifest once more and removes the row outright (forget_available_backup_row), deleting no object; it reports InUse and a row another writer moved, checks the claim and the signal before each row, and re-checks the claim with one beat right before each removal (a claim found lost ends the loop with that row left). A head that fails keeps the row (PruneOutcome::Failed): the store has not said whether the manifest is back. PruneOutcome::needs_another_run names the outcomes a later run may change (Failed, InUse), which the command exits 1 for.
• startup_sync is the in-memory catalog's sync: when the store is healthy and the sync pending, bounded by STARTUP_SYNC_TIMEOUT (60 s); a run that stops before it listed the whole root, reaches the bound, or skipped a prefix for a store or catalog error stays pending, those skips logged at WARN on every run and the other skips at INFO until a run has listed the whole root and logged its line of counts (a run cut at the bound or stopped by an error names no prefix, so the next run names them), and at DEBUG after, with the line of counts. Every run lists the root from its first key, so the startup sync completes only while listing the whole root fits in the bound; past it the sync stays pending and is retried every cycle; on both store kinds the listing is collected whole (BoundedCalls::list, one ObjectMeta per object held until the run ends) before the run sees its first key, so a listing that outlasts the bound reaches no prefix. Its store calls run as the checks run theirs (ConfiguredStore::calls_within), so on a filesystem root an abandoned walk holds the root's call slot and is marked in flight until it ends, and the check and the duties are refused at once rather than starting a call beside it (BoundedCalls now also marks a parked call whose caller drops it before its bound); the mark records which caller's call has not returned (InFlightMark) and the refusal names it ("an abandoned startup sync's store call has not returned"), and since that refusal records the store unreachable, CreateBackup, DeleteBackup and restores are refused until the walk ends, after which the first cycle checks the store healthy and runs the sync again. Taking the pending sync is the one gate.

store_identity.rs: the file's ephemeral_catalog member; adopt(store, kind, root, catalog, backend, ephemeral, rule, claim) first runs the check the server runs on the root as it will be once this catalog owns it (compare and the manifest probe) and refuses a root this catalog cannot use with nothing written (AdoptRefusal::Unusable), then re-checks claim (the command's maintenance claim; None for the startup adoption) with one beat right before the rewrite, AdoptRefusal::ClaimLost with nothing written when it is lost, then rewrites the file's catalog_id and backend with a plain put, keeping store_id and created_at_ms, reads it back (AdoptRefusal::RaceLost when another write landed over it), refuses when this catalog's rows for the root carry another store id, and logs an extenddb::audit intent line before the put and the audit line after the re-read with its outcome; AdoptRefusal::NoFile carries the rows' store ids, so a root that lost its file under rows is identity file missing and a fresh one identity file unreadable rather than unreachable; every store call of the adoption that fails in transport names its step (at_step), the manifest probe of a root this catalog already owns included; establish of an in-memory catalog adopts a root whose file names another catalog and carries the marker (AdoptRule::EphemeralOwner), and recheck never adopts. runtime.rs: ConfiguredStore::with_ephemeral_catalog and the pending startup sync (take, retry); BackupRuntime::with_region; and StoreFence (ConfiguredStore::fence), the deletion fence: it shares the store's health and reads the catalog id once, and its check re-reads extenddb-store.json with one get under the store's check bound, in a call slot and with an in-flight mark of its own on a filesystem root (so a deletion holding the root's slot does not hold it), refuses with no read on a store that is not healthy, and when the file names another catalog records ForeignCatalog on the store and refuses, and when it names this catalog under another store id than the one the store's check settled (a root replaced by another root of the catalog; the store's id is now shared with its fences) records IdMismatch and refuses; a root with no file, a file the store refuses to serve and one that does not parse are refused and recorded as the next check would record them (FileMissingWithRows, Denied, FileUnreadable, Validate), so CreateBackup answers the unhealthy-store ValidationException; only a read that fails in transport refuses and leaves the health as it was (CreateBackup then answers InternalServerError). It runs before DeleteBackup's DELETING write and before purge_row removes a dead CREATING row or moves an AVAILABLE one, so a refusal there for a root another catalog owns, a root with no file, or a file the store refuses to serve or that does not parse leaves the row as read with its objects (DeleteBackup answers the unhealthy-store 400 ValidationException, and DescribeBackup still answers AVAILABLE; a fence read that fails in transport there is a store failure, and the request goes on to the DELETING write), and again right before each delete_prefix of a backup's objects: finish_backup_delete (now taking the fence and answering FinishError; an adoption between the two reads leaves the row DELETING with nothing deleted and DeleteBackup answering DELETED, as for any deletion that fails after the DELETING write, which this catalog's reconciler finishes only if this catalog owns the root again), the job's fail_row (FailurePath::Fenced), the reconciler's FAILED-prefix and DELETED-prefix duties, and purge_row; before the reconciler removes a tombstone; in CreateBackup before claim_backup, so a fenced create writes no row and no object, and in the backup job right before its first put, so a root adopted after the request gets no object of the job (FAILED with the fence's reason). DeleteBackup's log line for a deletion the fence refused after the DELETING write is worded from what the fence found (fenced_deletion_note). The restore deletes nothing in the store and has no fence, and neither has --prune-missing's removal of a row, which deletes no object; StoreFence's type doc names every call site and says so. reconciler.rs: StoreCalls and verify_prefix are pub(super), the cycle runs a pending startup sync after a healthy store check, within its bound, and it re-tests the store's health after each store duty, so a fence refusal stops the duties after it, the dead-owner duty included; the SQLite command's prelude logs a line of its own instead of "a maintenance claim is held". maintenance.rs: HeldClaim confirms the claim by the job heartbeat's rule (job/heartbeat.rs): each beat bounded by one period through heartbeat_within, a beat that answers Lost, fails or outlives the bound recording the loss, the claim read lost once its last confirmed beat (the take standing for the first) is two periods old, a beat in flight counting only where heartbeat_parks_before_its_bound (SQLite), and the timer's ticks one period apart from the take; HeldClaim::recheck sends one beat, under the same bound, and records a loss as the timer does (a claim already read lost answers at once). purge_row takes the fence and the claim, re-checks the claim right before it removes a dead CREATING row and right before it deletes that row's prefix, and its comment says why nothing adopts a prefix between the two steps while each command reads its claim held, and what remains (a call already sent when the process was stopped); rows of other statuses keep their row through the deletion, so they need no re-check; restore.rs: OTHER_STORE and the missing-manifest error line say what an operator can do in this release; a missing sidecar's line says to put the prefix back, since --prune-missing removes only a row whose manifest is gone.

Server (crates/server). backup_store.rs: Caller::SyncCommand, as Command except that a defaulted SQLite root whose file names another catalog, while the catalog has no rows for it, is configured and unhealthy rather than the warning that leaves it unconfigured (PR 7's leniency, which the server and purge keep); the ephemeral flag from backend == "sqlite" and database_file().is_none(); under the exception the unhealthy line says the root is the shared default one with no rows, configured for backup sync; the warning for a defaulted root another catalog owns names backup sync --adopt-store; a command's unhealthy-store line is its own, without the server's sentence about starting and /health, and backup sync logs a root another catalog owns at WARN, not ERROR. serve.rs: the region on the runtime, and the startup sync of a filesystem store before the routes serve, within the engine's bound.

App (crates/app). extenddb backup sync [--adopt-store] [--prune-missing]: opened as serve opens the catalog (on SQLite under the serve lock; plain sync there runs no dead-owner resolution, and --prune-missing and --adopt-store run resolve_dead_owners before their wait, as purge does), its serving registration named extenddb backup sync; prints a header naming the store, the adopted ARNs, each skipped prefix with its reason, the AVAILABLE rows whose manifest is gone (with "run with --prune-missing to remove the rows, which deletes no object" when the option is absent), and a count line; exits 0 once it ran, 1 on a refusal (an unhealthy store, a root another catalog owns without --adopt-store, a held or lost claim), a listing that failed part-way, a stop, a row --prune-missing left for a later run (a store or catalog call failed, a failed head included, or a restore is reading the backup, with "run the command again once the restore ends"; a row whose manifest is back or that another writer moved is reported with exit 0), or a prefix skipped for a store or catalog error, 6 and 3 on the catalog refusals. The missing-manifest heading says the rows were AVAILABLE when the catalog was read, since a DeleteBackup on another instance can move one during the run. On PostgreSQL, MongoDB and Cassandra every sync takes the maintenance claim and heartbeats it, plain sync without the wait; --prune-missing and --adopt-store also wait for running jobs as purge does. Every sync installs the signal listener and stops before its next prefix. Every sync reads the identity file once more at the end and fails naming the owner if it changed (owner_after_sync), plain or adopting; a run with a store or catalog skip exits 1 (complete_run) and its count line says incomplete (run_word); a row left for a later run exits 1 (PruneCounts); app unit tests cover these decisions, and the wire cells test_an_adopter_whose_put_lands_during_the_sync_ends_it_with_exit_1 and test_a_prefix_the_store_cannot_read_leaves_the_sync_incomplete pin their call sites. The adoption runs on the store's bounded calls (store_identity::adopt_configured): each store call under the 5 s bound, the catalog's reads and the claim's beat beside it, and a store that stops answering ends the command naming the step that did not finish. A stopped or failed run prints "(not examined)" under the missing-manifest header, and a stopped prune's count line says "stopped". Plain sync's refusal of a root another catalog owns, --adopt-store's output and its help say to stop every server of the catalog the file names before adopting (a running one acts on the root until a later check that can read its catalog, its fence excepted), and the refusal, on SQLite's default root only, that every database file of a directory shares that root. resolve_dead_owners (purge's and the claimed options') now prints the store-backed restores it failed and removed, those it failed, and the targets it left for backup remove-target. --adopt-store's help says it waits for running jobs. A defaulted root's ConfiguredStore carries defaulted once its health is settled too, which the refusal reads. purge passes the store's fence to purge_row. Every backup subcommand refuses an in-memory SQLite database before it opens a catalog, and reports a location that does not parse with its parse error. The clap help names sync. destroy's output names backup sync --adopt-store.

Harness and matrix. tests/test_backup_job.py, on the SQLite, PostgreSQL and MongoDB deployments (MongoDB's row edits run mongosh in the replica set's container, EXTENDDB_TEST_MONGODB_CONTAINER, which integration-mongodb.yml's backup-store job now sets; the cells that need a second catalog skip on MongoDB, whose catalog database name is fixed per replica set; "PostgreSQL" below reads "beside a server", PostgreSQL and MongoDB): test_sync_rebuilds_dropped_rows_and_adopts_only_verified_prefixes, test_prune_missing_and_the_unhealthy_store_refusals, test_a_second_maintenance_command_fails_at_once (PostgreSQL; on SQLite the serve lock refuses a second writing command first; --adopt-store on a root the catalog owns waits for the same live job, "Waiting for running jobs to finish", and a signal stops it before any change), test_prune_missing_on_a_host_local_root_deletes_no_object and test_sync_is_refused_while_purge_holds_the_claim (PostgreSQL), test_adopt_store_makes_this_catalog_the_owner (a second catalog of the build's backend, its PostgreSQL databases dropped after), and test_the_previous_owner_deletes_nothing_under_an_adopted_root (PostgreSQL: the previous owner's server keeps running with its next check an hour away, its DeleteBackup of the adopted backup is refused, the bytes stay and DescribeBackup there still answers AVAILABLE, its /health then reports owned by another catalog, and the adopting catalog's server is healthy and restores the backup), and test_sigterm_during_the_prefix_loop_stops_sync_and_keeps_its_rows (SIGTERM after the first two prefixes' rows are back, 10,000 skipped copies still to go: exit 1, "stopped", the rows kept, and the next sync runs at once and adopts nothing more), test_a_prefix_the_store_cannot_read_leaves_the_sync_incomplete (a row-less prefix whose manifest is mode 000: the "store could not be read" skip, exit 1 "the run is incomplete", and the next sync adopts it once the mode is restored), and test_an_adopter_whose_put_lands_during_the_sync_ends_it_with_exit_1 (a second catalog's --adopt-store with 10,000 skipped copies under the root; once "Adopted the root:" is printed and while the sync is in its prefix loop the identity file is rewritten to name a third catalog, and the command exits 1 naming it and the store id). test_plain_sync_on_sqlite_leaves_a_stopped_servers_restore_to_resume (SQLite: SIGTERM mid-restore, plain sync exits 0 and fails nothing, the restart resumes the restore to ACTIVE with every item); test_sync_rebuilds_dropped_rows_and_adopts_only_verified_prefixes now expects the DELETING and CREATING rows left as they were on both deployments; test_adopt_store_makes_this_catalog_the_owner asserts the default-root sentence absent for a configured root. tests/test_backup_store_config.py: test_an_in_memory_server_restarted_onto_its_root_serves_the_previous_starts_backups, test_adopt_store_takes_the_default_root_a_re_created_catalog_finds (a store-backed backup on the default root, destroy and init, plain sync exit 1 naming --adopt-store and the shared default root, --adopt-store adopting the backup with the store id kept, the restarted server describing and listing it: the command's Caller::SyncCommand), and backup sync in both tuples of test_a_refused_catalog_ends_the_backup_commands_with_the_serve_exit_codes (exit 6 while a migration holds the serving key, 3 on a catalog behind the binary). devtools/backup-restore-matrix copies the source's prefix and runs extenddb backup sync on the target (on SQLite with the target's server stopped) in place of seed_row, checks the adopted row's name and size against the manifest, and gains --adopt-store (no copy: the target's store is the source's root; after the adoption the restarted source must report owned by another catalog); matrix-nightly.yml runs it from SQLite to PostgreSQL and back, and its comments and requirements-matrix.txt's say what the script now does. devtools/backup-kill-points.py, whose Cassandra restore points seeded the target's row with the matrix's seed_row, now copies the prefix and runs extenddb backup sync beside the Cassandra target's server, requiring exit 0 and the one backup adopted, as the matrix's adopt_backup does; its comments and the nightly's Cassandra build step say so. Engine cells: the claim's two-period rule (a timer that stops), a beat that hangs bounded by its period, a beat in flight confirming only where it parks (a paused clock: a beat on schedule that never returns, and a process stopped three periods between beats), a beat that panics leaving no flight, the re-check counting as a beat and recording a loss, and the re-check before purge's removal of a dead CREATING row and its prefix, sync's insert, --prune-missing's removal and --adopt-store's rewrite, each with a claim lost there. The startup sync's log rule: a first run whose listing fails and one cut at its bound name no prefix, the next run names the lasting skip at INFO and the run after it at DEBUG (a_startup_sync_whose_first_run_fails_logs_its_skips_at_info_on_the_next, ..._is_cut_at_its_bound_...). The region rule at the restore: a row over a manifest that spells the backup in another region restores (a_row_over_a_manifest_of_another_region_restores) and resumes (a_resume_whose_store_documents_fail_verification_is_failed). A stop at the second of three prefixes, inside the listing's loop, makes no missing-manifest report. The fence's CreateBackup cell has a third case, the filesystem store's error for a file at mode 000, recorded as failed validation. SQLite: the_maintenance_claims_beat_is_bounded_from_the_write_lock. The four backends' insert_backup_row_inserts_an_unowned_row_only_when_absent cells insert a row whose created_at is a year before the insert and assert completed_at and heartbeat_at within 60 s of the insert.

Docs and texts. The admin guide: a backup sync paragraph (what it adopts and never adopts, its output and memory (the rows' read, measured), its exit codes, its refusals, the claim and what a killed sync costs, the signal, the SQLite serve lock, the partial report of other roots, the case with no way out but putting the prefix back), the identity paragraph (--adopt-store, the default-root exception and the default root every database file of a directory shares, the marker, the deletion fence (before the row change and before the deletion, what a late refusal leaves, a root with no file, an S3 server error recorded as failed validation, the S3 gap, a stale check, the failed job's orphan prefix), the previous owner's window and the rule to stop every server of the catalog a file names before adopting its root), the --adopt-store paragraph (the same rule), the :memory: row (the 60 s bound and the listing it must fit, held whole in memory; an abandoned run's filesystem walk that runs to its end while every check is refused on it and the store is unreachable, CreateBackup, DeleteBackup and restores refused, until the first cycle after it runs the sync again, with a smaller root or a durable catalog as the remedy for a root too large to list in 60 s; /health silent until a filesystem startup sync ends and a signal then ending the process at once, the BackupNotFound window until the startup sync's log line and, for a prefix that met a store or catalog error, until a later cycle's line, the backup a server killed inside a DeleteBackup serves again, the prefixes a stopped server had not finished writing or deleting (a killed CreateBackup, a failed backup, a deletion cut after its manifest), which no command removes, the startup sync's "not adopted: not a complete backup" line that names them, and the rule to delete by hand only a prefix with no extenddb-manifest.json and no row in ListBackups, and a new in-memory server started only after the old one has exited) and the in-memory refusal, the registrations list. The sync paragraph also states the claim's heartbeat rule and its re-checks with the bound that remains, and what plain sync (no resolution) and the two claimed options (purge's resolutions, a RUNNING store-backed restore failed) do on SQLite with what a stopped server left; the :memory: row gives the remedy for a root an earlier development build's in-memory server wrote (its file has no ephemeral_catalog: remove it, and the next start writes a new one and adopts the backups). The sync paragraph also gives the per-prefix memory term (about 350 bytes per prefix at one object per prefix), says the report lists every prefix the catalog already has a row for, that the objects a pruned row's prefix still holds stay until an operator deletes the prefix by hand (purge deletes only the prefixes of rows), and that on PostgreSQL the next maintenance command takes a killed command's claim over at once; the identity paragraph gives the fence's store-id rule. The upgrade manual's rollback paragraph (--adopt-store after the re-upgrade, the defaulted SQLite root's warning until then, the serve lock or the claim for sync), the differences ("Where backups live", where an adopted backup of another region keeps the manifest's region in its source table's ARN, in DescribeBackup's SourceTableDetails.TableArn, ListBackups' BackupSummaries[].TableArn and the restore's RestoreSummary.SourceTableArn), usage (cross-backend restore, where --adopt-store of the source's root moves the root to the target's catalog: stop the source's servers first) and limits (backup contents) rows, the developer test guide's matrix paragraph. Design: revision item 34 (the fence included), the PR 10 section's note of the split, the PR 9b paragraph and the doc-row table's migrate-legacy row; 4(c) (the previous owner acts until a later check that can read its catalog unless the fence stops it), the PR 10 section's host-local acceptance line (amended to the rule implemented), 4(e) (--move-store is #820's), 4(f), 4(g), decision 6, section 7's memory paragraph, the PR 10 contract, 5.3's failure step (OTHER_STORE's reason as this PR words it) and the testing summary's kill-point line edited with item 34, which also states the claim's rule, the re-checks and plain sync on SQLite. The comments that gave the Cassandra orphan backup_index_by_status sweep and the MongoDB orphan target-collection sweep to backup sync say no sweep removes them yet, and the MongoDB error line in remove_restore_target, which an operator reads, now says the collections it names can be dropped by hand. Cassandra's module doc says that a status entry under an earlier status stays until a sweep exists, and names the path that writes one by design (a DeleteBackup between insert_backup_row's transaction and its entry batch); the trait doc and the sync module doc qualify "no two rows of one backup id" as under this server's or the manifest's spelling of the ARN (item 34 as under the manifest's account, table and id); sync's sentence for an insert that met a row it cannot re-read names an earlier release's in-catalog backup at the ARN; STARTUP_SYNC_TIMEOUT's doc states the listing collected whole on both store kinds, the abandoned filesystem walk and what it costs until it ends; destroy's sentence says that on the default SQLite root the re-created catalog's server logs the warning and leaves the store unconfigured. The backup job's line for a job failed on an unhealthy store says its objects are never deleted while another catalog has adopted the root.

Why

Design 4(c), 4(f), 4(g) and the PR 10 section. The store is the source of truth and the row a cache; until this PR nothing rebuilt the cache, so a lost or re-created catalog lost its view of every store-backed backup, and sqlite-memory lost it at every restart. The split keeps this PR to the commands that touch no row of another store: plain sync and --prune-missing act only on the configured store's rows, --adopt-store rewrites one file and then syncs, and the startup sync is the same sync. --move-store, --forget-store and --forget-backup rewrite or delete rows of other stores, a different set of invariants with two more trait methods per backend (#820); migrate-legacy is a separate data path (#821).

Choices to weigh

• Caller::SyncCommand, the one exception to PR 7's defaulted-root leniency. On SQLite's defaulted root a catalog re-created by destroy and init finds the root's file naming the old catalog and no rows of its own, and PR 7 (item 29) makes that a warning that leaves the store unconfigured, so CreateBackup keeps working on a server that never configured backups. Without the exception backup sync --adopt-store would answer "no store" on exactly the root it exists for. The exception is scoped to the backup sync command and to a root another catalog owns; the server and purge keep the warning. The alternative is to require an explicit [backup.store] naming <dir>/backups for the adoption: that is a configured root, unhealthy owned by another catalog, which --adopt-store adopts, so it is a remedy too, at the cost of one configuration line the operator has to know to write. The default root is shared by every database file in its directory, so with either choice --adopt-store there takes the root from whichever catalog wrote its file; the refusal, the server's warning and the admin guide say so.
• The startup sync runs on every start of an in-memory catalog with a store, not only after an adoption. The catalog is new at every start, so there is no row to compare with and nothing to decide; on a fresh root the sync is one listing. A filesystem store syncs before the listener, an S3 store in the reconciler's cycle after its first healthy check, since an S3 check never holds the listener; every run is bounded at 60 s, and one past the bound, or one that met a store or catalog error on a prefix, runs again in the next cycle. Until a run ends, requests for the earlier backups answer BackupNotFound (with /health possibly already healthy); the :memory: row says so and names the log line that ends it. The adoption that precedes it is inside establish and only for a file carrying ephemeral_catalog: true; recheck never adopts, so two in-memory servers on one root (a non-goal) do not take it from each other every cycle, and a durable catalog's root stays unhealthy for an in-memory server.
• --prune-missing removes rows and deletes no object. The manifest probe sees only the newest AVAILABLE row's manifest, so a host-local root that holds the newest backup is healthy while it lacks older ones. A prune that moved those rows to DELETING or DELETED would let the other host's deletion duties delete their bytes. Removing the row outright, as 4(g) does for --forget-store, leaves nothing for a duty to act on; the row is gone for every instance, so the backup is BackupNotFound on every instance until a sync on the host that holds the bytes adopts it again; the design's host-local acceptance line is amended to that rule (no reconciler deletes a prefix because its own root lacks the bytes, and --prune-missing there removes the row and deletes nothing), and test_prune_missing_on_a_host_local_root_deletes_no_object pins it. The cost: a row pruned because its prefix was deleted by hand leaves that prefix's other objects for the operator to delete by hand, since purge deletes only the prefixes of rows and every later sync reports the prefix as not a complete backup. The alternative was to prune only ARNs the operator names; the option would then need an argument, which the design does not give it.
• The dead end that follows. When the newest AVAILABLE row's own manifest is gone, the probe marks the store unhealthy, so plain sync and --prune-missing both refuse, and the row cannot be pruned. The way out in this PR is to put the prefix (or its manifest) back; Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820's --forget-backup will be the other. The admin guide states it; test_prune_missing_and_the_unhealthy_store_refusals pins both refusals and the recovery once the manifest is back. This predates the PR (the probe is PR 7's) and is the same condition the host-local case shows, so the PR does not change it.
• Plain sync takes the maintenance claim beside servers, with no wait. The insert-if-absent orders it against a CreateBackup on any instance (an_insert_that_meets_a_row_leaves_it_and_says_which), but not against purge, which removes a dead CREATING row before it deletes the prefix, and that prefix can hold a complete manifest (the job died after writing it); a sync between the two steps would adopt bytes purge then deletes. With the claim the two never act at once while each reads its claim held (test_sync_is_refused_while_purge_holds_the_claim), and 4(f) now names plain sync. On MongoDB and Cassandra a claim is taken over on its heartbeat's age alone, so a command stopped past stale_job_after_secs (SIGSTOP, a paused VM) can resume after another took its claim: the claim reads lost once no beat confirmed it for two periods, and the command sends one beat right before each step it cannot undo and stops there when the claim is lost. What remains is a stop after that beat confirmed the claim, or with the step's call already sent: the call lands when the process resumes. The cost is that CreateBackup and RestoreTableFromBackup are refused while a sync runs, as during purge (a refused CreateBackup writes no row: the request checks the claim before any). A sync killed while it holds the claim keeps every instance's jobs refused until stale_job_after_secs passes (300 s by default); the admin guide says so. A prefix purge leaves after a failed deletion is a backup with no row: when it holds a complete manifest a later sync adopts it and the next purge deletes it; an incomplete one stays too, for the operator to delete by hand. On SQLite it runs under the serve lock, which is all the design's rule that every writing command refuses to run beside a server asks: the lock keeps every server and every other command off the file, so nothing changes between the row read and the insert, and plain sync runs no dead-owner resolution. Purge's resolutions (ReconcilerState::for_maintenance_holder) would fail a store-backed restore a stopped server left RUNNING and remove its target, which the next server start resumes instead; plain sync skips the rows a stopped server left and the next start resolves them. --prune-missing and --adopt-store take the claim on SQLite and run those resolutions before their wait, as purge does (that prelude logs a line of its own, "run by a maintenance command before its own work"), and print what they did.
• The rows are read once, before the listing; the listing streams. The design's "checks each prefix against the catalog as the listing reaches it" holds with one read of the root's rows up front (read_backup_row needs a full ARN, which sync has only after reading the manifest, and Cassandra keys by ARN); the read under the ARN's two spellings and the insert-if-absent then decide against rows that land later and rows of other roots, on Cassandra too. The listing is consumed as it streams with no store trait change, but the filesystem store builds its whole listing in memory first, so on a filesystem root the peak barely moved (98 MiB at 300,000 objects against 102 before). Sync keeps of each row only what it uses, but every backend's list_backup_rows collects its whole result first, so the read of the rows is the larger cost on a catalog with many rows of the root, tombstones included: about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of source_table_details_json), 610 MiB at 25,000 rows and 1,144 MiB at 50,000 against 47 MiB with none, measured on the SQLite debug build of this tree on a 16-core x86_64 Linux host with glibc's default arenas; backup list peaks the same, and with a one-object prefix per row sync now peaks at 1,145 MiB where it held the rows beside the listing at 1,181. Streaming the rows would change every backend's list_backup_rows; the design, the admin guide and section 7 state the cost instead.
• The deletion fence narrows the previous owner's window; it does not close it. After an adoption the previous owner's store stays healthy until a later check that can read its catalog finds it, at the earliest one reconciler interval (60 s by default) after its cycle in progress ends, and in that window its DeleteBackup, its reconciler and its failed jobs would delete bytes under the new owner's AVAILABLE rows, which would then leave the new owner's store unhealthy with no way out in this release. The fence re-reads the identity file right before each deletion (and before a CreateBackup's row and again before its job's first put), so the window shrinks to the gap between one get and the call after it (on S3 a listing and a DeleteObjects under the SDK's retries, seconds when throttled), and the first refusal marks the store unhealthy for later requests (a check that read the file before the adoption and ends after the refusal records it healthy until the next cycle: deletions stay fenced, restores are accepted). DeleteBackup and purge run it before they change the row as well as before the deletion, so a DeleteBackup refused for a root another catalog owns, a root with no file, or a file the store refuses to serve or that does not parse answers 400 and leaves the row AVAILABLE with its objects, and DescribeBackup never answers that refusal with DELETED; the cost is two small gets per CreateBackup (the request's and the job's) and two per DeleteBackup. A put already past the read still lands, which is why the texts state the precondition (stop every server of the previous catalog first) rather than promise the fence enforces it. A root with no file, or a file the store refuses to serve or that does not parse, is recorded as the next check would record it, so the store is unhealthy at once and CreateBackup answers the unhealthy-store ValidationException rather than a 500 on a store /health reports healthy; a read that fails in transport (the store unreachable, the read past its bound) refuses the deletion (it is retried, as a failed delete_prefix is) and changes no health; an S3 server error the SDK's retries did not clear (a 500 InternalError, a 503 SlowDown) is not one: the S3 store reports it as it reports other refusals, so the fence records failed validation, as the store's check does, and the store is unhealthy until a check passes (one reconciler interval, 60 s by default), refusing CreateBackup, DeleteBackup and restores in that time. The fence reads more often than the check (twice per CreateBackup and per DeleteBackup), so a throttled bucket meets it more often; classing 5xx answers as transport would change the S3 store's error mapping for every caller, which this PR leaves as it is, and the admin guide states the case; at DeleteBackup's first read it is a store failure, so the request writes DELETING and answers DELETED and the second read refuses the deletion. A refusal at the second read (an adoption between the two) leaves the previous owner's row DELETING with nothing deleted, and DeleteBackup answers DELETED, as for any deletion that fails after its DELETING write; that catalog's reconciler acts on no row of its now unhealthy store, and finishes the deletion only if the catalog owns the root again. A failed job of the previous owner leaves its partial prefix under the adopted root, which nothing of the new owner removes; the admin guide says so. The fence keeps the catalog id it read first: a catalog re-created beside a running process is then refused every deletion and CreateBackup until the process restarts, the safe direction, which the comment states.
• Rows of other roots are reported in part. Sync reports the prefixes whose backup id a row of another root holds. The rows of the configured root under another store id cannot reach it (such a store is unhealthy, which sync refuses), so the report has no section for them. A listing of every root's rows needs list_backup_rows_by_root, which Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820 adds with the options that act on them.

Testing done

On this tree (main at 887b6e26 with this PR's changes, among them the in-flight mark that names its caller, the adoption's probe step, the startup sync's documented walk and the region adoption's three fields; 47 files, +13,388 −907 against main), debug builds through a shared target with every tracked file touched first: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 701 passed (backup:: 338; 3 cells extended: the cycle cut at its bound now holds the walk and lets it return, asserting the check refused on it with the text naming the startup sync's call, the store unreachable, and the next cycle adopting the backup and finishing the deletion; the adoption's step cell gains the probe of an owned root, on a head that never answers; the restore's region cell adopts its prefix through sync with the manifest's source table in the other region and asserts the three fields), app 50, server 375. Mutants, each a targeted edit undone by the reverse edit, against the cells named: the refusal's text fixed to "check" whatever the caller (killed by the cycle cell); the probe's error mapped without its step (killed by the step cell); the cycle running the startup sync once per process (killed by the cycle cell); the adopted row's table_arn rebuilt in the server's region, and the restore's SourceTableArn rebuilt from the backup's ARN (each killed by the restore's region cell, the first surviving the earlier region cell, which reads no field). One edit that disabled the re-arm in the run's timeout branch survived the cycle cell: there the listing call's own 60 s bound ends the run first (Err(Listing), which re-arms), as the skip-lines cell's comment says; the "left pending" assertion covers the re-arm on that path. No wire cell asserts a text this pass changed.

On the tree before this pass (main at 887b6e26 with this PR's changes, among them the startup sync's rule for its skip lines, the region cells at the restore and the resume, the stop inside the listing's loop, the fence's filesystem case, DeleteBackup's doc and the texts; 47 files, +13,210 −884 against main), debug builds through a shared target with every tracked file touched first: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 701 passed (3 new cells, 3 extended), app 50, server 375, core 1087 (1 ignored). Mutants, each a targeted edit undone by the reverse edit with the file's md5 checked, each against the cells named for it, 7 of 7 killed: the skip lines' rule decided at the take (the flag set when the sync is taken; and, with the flag as it is, only the process's first run logging at INFO, against each of the two new cells alone), the flag never set, the restore's manifest check and the resume's requiring the row's region, a missing-manifest report made at a stop inside the listing's loop, and the fence taking the filesystem store's I/O error for a transport failure. Wire on the sqlite-memory build, Python 3.10 with the moto variables: the --adopt-store cells, the in-memory restart cell and the region cell, 5 passed (-n 3). Binary check on that build, an in-memory catalog over a filesystem root holding one backup and a copy of its prefix under another id, with an account directory at mode 000: the startup sync before the listener and the first reconciler cycle's each logged one WARN line ("startup sync of the in-memory catalog stopped: the store's listing failed: read directory .../999999999999: Permission denied (os error 13); no row was written; the next reconciler cycle runs it again") and no prefix line; once the directory was readable again the next cycle logged at INFO "backup store startup sync: 123456789012/Music/01794287129941-5c08ebdc/ not adopted: the manifest names arn:aws:dynamodb:us-east-1:123456789012:table/Music/backup/01794287129940-5c08ebdc, whose account, table and backup id are not the prefix's" and "1 backup(s) adopted, 1 prefix(es) skipped", and DescribeBackup of the backup answered AVAILABLE. The S3 server error at the fence (Choices to weigh) is read from the code; moto injects no faults.

What CI does not run. No Cassandra deployment runs tests/test_backup_job.py, so backup sync, --prune-missing and --adopt-store have no wire leg on Cassandra (its engine fakes and the crate's direct cells cover the trait methods). No CI cell runs backup sync against an S3 store: the wire cells, the in-memory restart cell and the matrix use filesystem roots, and the engine cells an in-memory store. A sync against moto was run by hand on 84bc263 with the PostgreSQL build: a dropped row re-adopted with exit 0, plain sync refusing a root whose file was rewritten to name another catalog and naming --adopt-store, and --adopt-store adopting it with the store id kept and ephemeral_catalog: false written; an in-memory server restarted over a moto root adopted it in its first cycle and synced its 3 backups. The nightly matrix (matrix-nightly.yml, which runs the changed devtools/backup-restore-matrix and devtools/backup-kill-points.py, its MongoDB and Cassandra targets and the four kill-point jobs among them) runs nowhere else; main's last two scheduled runs failed in restore-kill-points-mongodb.

Nightly matrix on this branch: started on 4812e3e8, https://github.com/ExtendDB/extenddb/actions/runs/42775123844 (result to follow).

On the tree before the rebase onto 887b6e26 (based on b8ad9cd5; the fence's recorded problems, its own slot and the job's fence; the adoption's check before the rewrite and its bounded calls; the region adoption; the startup sync's log levels, take and walk; every sync's last read; the MongoDB wire coverage; 47 files, +12,923 −883 against main), debug builds through a shared target with every tracked file touched first: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 489 passed (12 new cells, 8 rewritten to the new rules), app 50, server 343, core 803. Mutants, each a targeted edit with the file restored from its copy and its digest checked, each against the cells named for it: 25 of 26 killed (the fence's missing and denied reads unrecorded, the fence in the calls' slot, no fence in the job, the adoption without its probe, unbounded, comparing the catalog id alone, without the rows' ids on a missing file or without the two-ids refusal; the startup sync's retry at INFO, its listing unbounded, its run without the take, a dropped call left unmarked; the row under the manifest's region, the probe requiring the same region, the missing-manifest filter without the store id, the fence reading the catalog id on each call, the prune's stop before the first row only and its audit line removed, the re-check unbounded and the trait default unbounded, the timer counting from its first run, the fenced deletion's line naming an adoption for a missing file; in the app the prune exiting 0 for a row left and complete for a transient skip); the 26th (the cache's early return removed, get_or_init keeping it) is equivalent. Wire, by name with the moto variables, Python 3.10: SQLite 13 passed, 4 skipped (-n 4, with tests/test_backup_store_config.py's in-memory and adoption cells); PostgreSQL 14 passed, 1 skipped (the SQLite-only cell); MongoDB, on a private mongo:7 replica set with EXTENDDB_TEST_MONGODB_CONTAINER set, 10 passed, 4 skipped (the three cells with a second catalog and the SQLite-only cell). At the binary level the restore's manifest check requiring the row's region (PostgreSQL build) fails the region cell at its restore. Binary checks on PostgreSQL with a filesystem root and the reconciler an hour away: with extenddb-store.json removed, CreateBackup on a store /health reported healthy answered 400 ValidationException "... unhealthy (identity file missing) ..." twice (13 and 3 ms), and /health then reported the store unhealthy; at mode 000 the same, "unhealthy (failed validation)", the reason the filesystem store gives a file it may not read; a CreateBackup sent 1 s into a DeleteBackup of a prefix of 300,000 objects (the DeleteBackup answering DELETED after 5.6 s, the prefix gone after 74 s) answered 200 CREATING in 81 ms and its backup became AVAILABLE.

On an earlier tree (the claim's rule and re-checks, plain sync on SQLite, the kill-point script, the texts; 45 files, +11,351 −822 against main), debug builds through a shared target with every tracked file touched first: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 477 passed (9 new), app 49, server 343, storage 236; SQLite --lib backup 64 and the new maintenance-claim cell; PostgreSQL backup_restore 58; MongoDB live_insert_backup_row_inserts_an_unowned_row_only_when_absent and Cassandra test_insert_backup_row_inserts_an_unowned_row_only_when_absent (--exact, test-hooks) pass. Mutants, each a targeted edit, run and the file restored with its md5 checked, all 16 killed: in the engine the two-period rule off, a beat in flight confirming on every backend, the timer's beat unbounded, recheck recording no loss, and each of the five re-checks removed (purge's two, sync's, the prune's, the adoption's); SQLite's maintenance beat bounded over the whole call and its parks false; SQLite's and PostgreSQL's insert writing created_at as both clock columns; and at the binary level (SQLite debug builds, the targeted wire cells) backup sync configured as Caller::Command, plain SQLite sync running the dead-owner resolutions again, and the default-root sentence on every root. Wire, tests/test_backup_store_config.py tests/test_backup_job.py by name with the moto variables, Python 3.10: on the SQLite build 37 passed, 8 skipped (-n 4; the two new cells are SQLite-only; the default-root cell failed on the first build, whose ConfiguredStore did not carry defaulted for a settled health, and tests/test_backup_store_config.py passed whole on the corrected build, 15 passed, 4 skipped), on the PostgreSQL build 27 passed, 18 skipped (-n 2). devtools/backup-kill-points.py restore cassandra on a private Cassandra node (the script's port changed in a scratch copy only), with a SQLite source: restore_after_download, restore_after_file_commit, restore_cassandra_before_progress (first and second reach), restore_between_failed_and_removal and sigterm, 6 of 6 passed, each target's backup sync exiting 0 with the backup adopted; restore sqlite restore_after_download passed. The lines below are carried from the earlier trees, as each says.

On the tree rebased onto main at b8ad9cd5, debug builds: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 468 passed, app 49, server 343, storage 236; SQLite --lib backup 64; PostgreSQL backup_restore 58 (--test-threads=2); the wire suites tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 35 passed, 8 skipped (-n 4). The lines below are carried from the earlier trees, as each says.

Re-measured on the tree rebased onto #740's 94cd324e, debug builds (the move to 9803ab7f, whose two conflicts were text in runtime.rs and the design, was checked with fmt, clippy and the engine, server and app suites: engine 466, server 310, app 49): cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets --features extenddb-storage-cassandra/test-hooks -- -D warnings clean; engine --lib 464 passed (with and without backup-test-hooks: #740's 423 plus this PR's 42, less the dropped wait cell), server 310, app 49, core 801; PostgreSQL backup_restore 58; SQLite --lib 278 (1 ignored), its -- backup restore_job_tests 72, backup_job_panic and restore_job_resume 1 + 1; MongoDB live live_ 67 (--test-threads=1); Cassandra direct_integration (test-hooks, --exact, under the shared node's lock) the five insert and three forget cells named below, 8 passed; the wire suites as below (35 passed, 8 skipped on SQLite; 27 passed, 16 skipped on PostgreSQL). The mutant at the rebase's DeleteBackup change (its first fence never refusing) is killed by delete_backup_on_a_root_another_catalog_adopted_deletes_nothing. Everything else below is carried from the tree before the rebase, as each line says.

The counts below were measured on the final tree, and the binaries are debug builds of it, except where a line says the earlier tree (the first commit of this PR, before the changes to sync's claim, prune, partition and spelling rules, the streaming listing and the startup bound) or b58d879 (the tree before the last changes: the prune exit for a row in use, the fence's store-id rule, and the cells and texts listed above; they change no code of the core and storage crates, where the trait's change is a doc comment, so the core, PostgreSQL, SQLite and MongoDB counts below are b58d879's).

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets --features extenddb-storage-cassandra/test-hooks -- -D warnings clean; cargo +1.88.0 check --locked --workspace --all-targets clean with no warning on b58d879.
• Engine --lib: 431 passed before the rebase (464 on it), among them the deletion fence at each call site, each on a store whose last check passed and a root another catalog adopted since (DeleteBackup refused before its DELETING write with no delete_prefix, the row left AVAILABLE, and a second request refused at the gate with no read; an adoption between DeleteBackup's two reads leaving the row DELETING with nothing deleted and DELETED answered; purge refused before it changes the row; the reconciler's DELETING, FAILED-prefix, DELETED-prefix and tombstone duties with the file rewritten right after the cycle's check read it, each acting on nothing; CreateBackup refused before any row or put; the job's failure path writing FAILED and deleting nothing; purge deleting nothing), each leaving the store unhealthy owned by another catalog; the fence's read failing alone (a MemStore failure of the identity key) at DeleteBackup, CreateBackup, the job's failure path, purge and each reconciler deletion, each refused with nothing deleted and the health left, no row moved but DeleteBackup's, which writes DELETING and answers DELETED; a stalled read refused at the bound and a root with no identity file refused (DeleteBackup answering DELETED with the row DELETING in both); a fence refusal in the DELETING duty stopping the dead-owner duty before it reads the foreign root; the startup sync through reconciler::cycle on a root listing that never answers, cut at 60 s with the DELETING row finished in the same cycle and the sync pending; the next sync completing its own torn insert (a Fake knob for Cassandra's rule) while a row under the other spelling still skips; a key ending in the manifest's name below a prefix not hiding a missing manifest; the marker cell (a durable --adopt-store of a marked file writes ephemeral_catalog: false, an in-memory establish on that file answers ForeignCatalog and writes nothing, and an in-memory recheck of a lost file writes it back marked); and the sync cells (every adopt and skip reason, another account, table or partition in the manifest, a row of another root under the canonical spelling, a prefix under another table with a row's backup id, a CreateBackup racing the insert, an insert that fails, a stop before a prefix, a listing failing part-way, a failed row read, only AVAILABLE rows reported missing, --prune-missing's outcomes with no object deleted and no tombstone, the adoption and its lost race, the in-memory adoption with the marker true and false, the startup sync through reconciler::cycle with a store error keeping it pending and with a recheck finding the root foreign, the abandoned startup sync, the dead-owner probe refusing a manifest of another region), and: four manifests whose sidecar matches and that the reader refuses (a source_table of another table, a creation time that is not the id's instant, a ../ data-file key, a size total that is not the files' sum), each skipped as "the manifest is refused"; a failed read of the row under the ARN's spellings (a Fake knob) skipping the prefix as a catalog error with no insert, and the next sync adopting it; a Fake mode with rows unique by ARN alone (Cassandra's rule) and two prefixes of one id under two tables, the second skipped on the row check; --prune-missing with a failed head (a MemStore knob) keeping the row as Failed, and needs_another_run true for Failed and InUse only; the startup sync with an incomplete prefix left done; a cycle with a pending startup sync on a stalled listing returning at the shutdown, well before the bound, with the sync pending; the fence refusal in the DELETING duty also leaving the prefixes of a stale FAILED row and a DELETED row unlisted (a MemStore list record); adopt(.., AdoptRule::EphemeralOwner) on a durable file refused with nothing written, and an in-memory establish on it reading the file once; the fence refusing a file of this catalog under another store id and recording IdMismatch; server --lib 305 before the rebase, 310 on it (backup_store with the SyncCommand cells, the exception's scope among them); app 49 (the adopting sync's last read, the transient-skip exit); core --lib 800 before the rebase, 801 on it, with the layout cells (every reserved component, the id grid is_backup_id accepts and refuses, traversal, separators, control and non-ASCII characters).
• On the live services: PostgreSQL backup_restore 51 passed (58 on the rebased tree); SQLite's backup_job_panic and restore_job_resume, which drive the engine's job and reconciler, pass; SQLite --lib 268 (measured before the deletion fence, as MongoDB's and Cassandra's below are, and 278 on the rebased tree; the fence changes no storage crate, and none of them depends on the engine); MongoDB live live_ 63 with --test-threads=1 (67 on the rebased tree); Cassandra direct_integration (--features test-hooks, under the shared node's lock, --exact) test_an_insert_over_an_in_catalog_backup_answers_false_and_writes_no_entry (a backups_by_arn row with no store kind at the ARN: false, no list or status entry) on the final tree, and the insert cells (with the completed torn insert, the unknown-outcome cells through a test-hooks seam on conditional_write, and the failed entry batch) and the start_backup_delete cell, which on each backend also covers forget_available_backup_row (removed and listed nowhere, the in-use and status refusals, a missing row; on Cassandra the three index entries read directly).
• Mutants, each a targeted edit, run, and the file restored (those of the earlier rules on the earlier tree): 6 in layout.rs; the four backends' inserts (the insert that overwrites, the one that ignores backup_id uniqueness, a wrong generation or stamp, on Cassandra the list rows and the status entry skipped or misplaced, the settlement ignoring the generation, the store or the status, any row read after a lost answer counted as inserted, the single attempt, NotMatched not completing a torn insert, the batch not re-sent); forget_available_backup_row without the in-use rule (PostgreSQL, SQLite), marking DELETING (MongoDB), leaving its index entries (Cassandra); the engine's sync, adoption and startup-sync rules (the account, table and partition checks, the spelling read, missing manifests of any status, the startup sync on an unhealthy store or with a transient skip marked done, an insert error, the stop, the prune deleting bytes, the other-table reason, the probe comparing only the backup id, and the earlier rules); the SyncCommand exception removed; the deletion fence removed at each of its six call sites (finish_backup_delete, the reconciler's prefix deletion and its tombstone removal, fail_row, CreateBackup, purge_row), the fence not recording ForeignCatalog, the fence reading the file on a store already unhealthy, adopt keeping the file's marker, and recheck rewriting a lost file without the marker; and at the binary level against the wire cells: finish_backup_delete's fence removed (the previous owner's DeleteBackup then succeeds), the row check before the manifest read removed, the startup sync before the listener removed, the in-memory CLI refusal removed, --adopt-store taking no claim, plain sync taking no claim beside servers, and the prune through start_backup_delete and finish_backup_delete. Each was caught at the assertion it targets. On the final tree, each against the whole crate suite: the spelling skip for either spelling; no fence before DeleteBackup's DELETING write, or before purge's row change; the dead-owner duty after a refusal; missing-manifest keys of any status; any key ending in the manifest's name; the startup sync's timeout removed; the fence's read error passing, its read unbounded, a root with no file passing; the adopting sync's last read accepting another owner and a transient skip exiting 0 (app cells); stop: None in the CLI's sync (the wire SIGTERM cell); PostgreSQL's shared body without FOR UPDATE, Cassandra's shared condition as IF EXISTS and without the in-use pre-check, and Cassandra's forget with the entry error returned, Unknown never or always counted as removed, and no lost-answer exhaustion check. All 21 were killed (a 22nd, the wait on a FAILED maintenance row, targeted code the rebase dropped). On the final tree, 16 more, each a targeted edit, run, and the file restored with its md5 checked: in the engine, against its whole --lib suite, the reader's validation replaced by a plain parse, a failed spelling read treated as no row, a failed prune head falling through to the removal, Incomplete counted as transient, the row check's insert after an adoption removed, InUse not counted for another run, the cycle's shutdown select replaced by awaiting the sync, the three health re-tests before the FAILED-prefix, DELETED-prefix and tombstone duties removed, the marker guard in establish and the one in adopt each removed, and the fence's store-id comparison removed; Cassandra's NotMatched arm with is_none_or (killed by the in-catalog backup cell, which the four earlier insert cells let pass); and at the binary level (PostgreSQL debug builds, the targeted wire cell) the adopting sync's last read removed, complete_run answering Ok, --adopt-store's wait gated on --prune-missing alone, and sync's catalog refusal turned into a plain error (exit 1 where 6 is expected). All 16 were killed at the assertion each targets.
• Wire, by name with the moto variables, on Python 3.10: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 35 passed, 8 skipped (the PostgreSQL-only cells; pytest -n 4), on the PostgreSQL build 27 passed, 16 skipped (the SQLite-only cells; pytest -n 2, since -n 4 exhausts the server's connection slots).
• Storage, for the shared delete and forget body and Cassandra's forget: PostgreSQL backup_restore's two row-lock cells (start_backup_delete_waits_... and the new forget_available_backup_row_waits_for_the_row_lock_and_then_sees_the_registration) and start_backup_delete_checks_...; SQLite --lib -- backup restore_job_tests 63 passed (72 on the rebased tree); MongoDB's two live_ cells of the shared body; Cassandra --exact under the lock, 6 passed, among them test_an_unknown_forget_outcome_is_settled_by_a_serial_read and test_a_forget_whose_entry_removal_fails_answers_removed (seams fail_forget_writes and fail_index_entry_removals) and test_a_forget_clears_a_finished_restores_registration_and_removes_the_row.
• Binary checks on b58d879 (SQLite): SIGTERM one second into a sync of 2,000 prefixes: exit 1, "stopped"; the in-memory refusal and the parse errors of file://example.com/... and file:x.sqlite?bogus=1; peak RSS 45 MiB empty, 66 MiB at 100,000 objects, 98 MiB at 300,000 (no rows); with rows of the root, 610 MiB at 25,000 and 1,144 MiB at 50,000 rows of a five-GSI table, 148 MiB at 50,000 rows whose source description is {}. On the earlier tree: SQLite (a writing sync beside a server exits 6; destroy and init, then plain sync refusing and --adopt-store adopting the root and its backup, which the restarted server describes, lists and restores; a prefix copied into another deployment's root adopted and restored there), the sqlite-memory shape on a filesystem and on an S3 store (moto: the reconciler's first cycle adopts the bucket's root and syncs; Describe and Restore of the previous start's ARN), and PostgreSQL beside its server (a SQLite deployment's backup copied in, adopted and restored; the volume unmounted refusing sync and --prune-missing with nothing written; --adopt-store of a SQLite root and of an in-memory catalog's root).
• devtools/backup-restore-matrix, one invocation per cell: sqlite→sqlite, sqlite→postgres, postgres→sqlite, postgres→postgres, the two --duplicate-key cells into SQLite and PostgreSQL, sqlite→postgres and postgres→sqlite --vector, and sqlite→postgres and postgres→sqlite --adopt-store: 10 of 10 PASSED (25 to 33 s each) on the earlier tree; on b58d879 sqlite→postgres, postgres→postgres, sqlite→postgres --adopt-store and postgres→sqlite PASSED again (28 to 33 s), the PostgreSQL targets' syncs now under the maintenance claim beside the server. On b58d879 sqlite→mongodb also PASSED, the target's sync taking the claim beside its server and adopting the backup. postgres→cassandra could not run on this host: the shared Cassandra node refuses CREATE ROLE ... PASSWORD (CassandraRoleManager doesn't support PASSWORD), which is the environment's, not the change's; it passed on an earlier tree of this PR. The nightly runs the MongoDB and Cassandra target cells on a schedule, not on this PR.

Not in this PR

• Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820: backup sync --move-store, --forget-store and --forget-backup, with BackupCatalog::list_backup_rows_by_root and rewrite_store, and the full report of the rows of other roots, with their acceptance cells (--move-store's resolutions and refusals, --forget-backup's refusals, the store's return after --forget-store, a CreateBackup racing --move-store). restore.rs's OTHER_STORE reason and the reconciler's module doc now say what an operator can do without --move-store (configure the store the record names and restore again).
• Backup/restore: backup migrate-legacy #821: extenddb backup migrate-legacy <arn>|--all, with its acceptance cells.
• The Cassandra sweep of orphan backup_index_by_status entries and the MongoDB sweep of the data collections a failed restore-target removal leaves; the code says no sweep removes them yet.
• The acceptance line "a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler has finished the one and failed the other" is PR 9's wait (wait_for_quiet) and its engine tests; this PR's claimed options reuse it unchanged.

Breaking changes

None. The identity file's new member defaults to false, so #740's binary reads a file this release wrote and this release reads #740's. extenddb backup list, purge and remove-target now refuse an in-memory SQLite database (exit 1), where they opened a new, empty catalog.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com

[LeeroyHannigan]

I built the head on SQLite, PostgreSQL, MongoDB, Cassandra and sqlite-memory and ran the servers and the commands in a container as uid 1000. I ran backup sync, --prune-missing and --adopt-store beside running servers on the four backends and against an S3 store. I ran the fence, the claim and the startup sync with two catalogs on one root, with killed, stopped and signalled commands. I merged main at 0cfb0d9 (no conflict). On the merge I ran clippy 1.97.0 with both CI feature sets and the 1.88.0 check of the feature-gated code. The engine, core, storage, config, app, SQLite and server suites pass on it. All 49 CI jobs at this head pass. I ran 49 mutants of sync.rs, maintenance.rs, runtime.rs, store_identity.rs, reconciler.rs, layout.rs, the SQLite catalog, the server's store config and cmd_backup.rs against the suites. That is 50 runs, with one mutant run against two suites. The suites killed 34 runs. Of the 16 that survived, one is equivalent and 14 are inline as test gaps. A PostgreSQL wire cell in CI asserts what the last one breaks, but I did not run that cell. I compared the wire answers with the service for the cursor shapes, another region's spelling of a backup ARN and CreateBackup.

What matched the description:

• The two trait methods behave the same on the four backends within the trait doc, with Cassandra's equal-row exception.
• Sync's report, its exit codes 0, 1, 3 and 6, and the serve lock on SQLite.
• DescribeBackup and ListBackups of an adopted row equal the CreateBackup record field for field on the four backends and S3.
• Restores of adopted rows reach ACTIVE with all items on the four backends and S3.
• The maintenance claim and its heartbeat rule. The next command takes a killed sync's claim over at once on PostgreSQL, and after stale_job_after_secs on MongoDB and Cassandra.
• A sync stopped past two periods loses its claim and stops before its next insert.
• SIGTERM stops a sync before its next prefix and releases the claim, and a second command fails at once naming the holder.
• --prune-missing removes the row and no object, the backup answers BackupNotFoundException after it, and the next sync re-adopts it once the manifest is back.
• --adopt-store and the previous owner's fence hold on PostgreSQL, MongoDB, SQLite and Cassandra.
• The fence refuses the previous owner's DeleteBackup before the row changes, the bytes stay, the store goes unhealthy and the gate then refuses CreateBackup. Main deletes the bytes under the other catalog's root for the same requests.
• The in-memory startup sync, the ephemeral_catalog marker and the restart after SIGKILL serve the previous start's backups.
• A CreateBackup job and a restore in flight across an adoption complete as described.
• A second account cannot list, describe, restore or delete the first account's adopted backups.

The five CodeQL alerts point at assertion messages inside #[cfg(test)] in layout.rs, and at a stdout print in cmd_backup.rs of an account id. The printed ARN beside it already carries that id.

Each item is inline. One is Blocking: CreateBackup answers 500 on a store /health reports healthy, where main and the service answer 200. Nine are Major. Three are defects: the --adopt-store rewrite before the probe, the startup sync's log volume on a retry, and the other-region adoption. Six are properties the description claims that no test or CI job covers. Thirteen are Minor. Once the Blocking item and the nine Major items are fixed, with their tests, I will approve. The Minor items can land in the same revision.

[LeeroyHannigan]

Blocking: CreateBackup answers 500 here on a store /health reports healthy. The fence's check returns a problem it does not record, so refusal() is None and internal() answers. Two states reach it. With extenddb-store.json missing or at mode 000 (PostgreSQL, an S3 root and a filesystem root), CreateBackup answered 500 InternalServerError "Internal server error" twice each. Only after the next check did it answer 400 ValidationException "The filesystem backup store configured on this server is unhealthy (identity file missing)". A DeleteBackup of a 300,000-object prefix held the filesystem store's one call slot. A CreateBackup sent 1 s into it took 5015 ms to fail with 500. The file was readable the whole time, and the requests sent once the walk ended answered 200. Main answers 200 CREATING in both states: four CreateBackup requests sent during the same 5 s deletion took 5 to 32 ms and answered 200. The service answers CreateBackup of an ACTIVE table 200. The PR's cell a_fence_whose_read_fails_refuses_each_call_and_leaves_the_health (tests.rs line 21939) pins the 500. When the read finds no file or cannot read it, record the store's health as the next check would, and answer the unhealthy-store ValidationException. Read the file outside the deletions' call slot, or run this fence in the job right before its first put. Keep 500 for a read that failed in transport.

[robinnsc]

Fixed as you proposed. StoreFence::check (crates/engine/src/backup/runtime.rs:888-951) now records on the store what the next check would record: FileMissingWithRows for a root with no file (:941), and the read's own problem for a file the store refuses to serve or that does not parse; only a read that failed in transport records nothing (:924). CreateBackup answers refusal(), the unhealthy-store ValidationException, for every recorded problem and keeps the 500 for transport (crates/engine/src/backup/job.rs:350-357): a missing file answers unhealthy (identity file missing), an S3 AccessDenied answers unhealthy (permission denied), and a filesystem file at mode 000 answers unhealthy (failed validation), because the filesystem store reports EACCES as an I/O error. I also did both of the other options: the fence reads in a call slot of its own (fence_slot and fence_in_flight, runtime.rs:311-312 and :648-649), so a deletion holding the root's call slot does not hold it, and the job runs the fence again right before its first put (job.rs:481-489). Tests in crates/engine/src/backup/tests.rs: create_backup_on_a_root_whose_identity_file_is_missing_or_denied_is_refused_as_unhealthy (:23017, the three states, then refused at the gate with no read), create_backup_during_a_long_deletion_answers_at_once (:23093, CREATING in zero time 1 s into a 4 s deletion), a_root_adopted_after_the_request_gets_no_object_of_the_job (:23139), and the cell that pinned the 500, renamed a_fence_whose_read_fails_in_transport_refuses_each_call_and_leaves_the_health (:22805), which now covers transport only.

[LeeroyHannigan]

The probe runs after the put at line 713. It reads only this catalog's newest AVAILABLE row and one head. The store id is the same before and after the put, so the probe gives the same answer before the rewrite. With one AVAILABLE row of this catalog whose manifest is gone, adopt answered AdoptedUnhealthy after its put rewrote the file to name this catalog. I also ran it with two catalogs on one root. A owned the root and made its newest backup, B adopted the root and B's server deleted that backup, and then A ran --adopt-store. A's command exited 1 with "rewrote the identity file, so the root is taken from its previous owner, but the sync did not run". A's plain sync and --prune-missing then refused (store unhealthy, manifest missing), and B's refused (owned by another catalog). Neither catalog could sync, prune or delete under the root until B adopted it back. Run compare and manifest_probe before the claim's re-check and the put, so adopt refuses a root this catalog cannot use with nothing written. No test names AdoptedUnhealthy or that sentence, and the engine suite passes (477 tests) with the probe call removed. Add that cell.

[robinnsc]

adopt now runs compare and manifest_probe on the root as it will be once this catalog owns it, before the claim's re-check and the put (crates/engine/src/backup/store_identity.rs:721-726; the put is at :740). A root this catalog cannot use is refused as AdoptRefusal::Unusable with nothing written and the file still naming its owner, so AdoptedUnhealthy and the command's rewrote the identity file ... but the sync did not run branch are gone. adopt_store_refuses_a_root_this_catalog_cannot_use_with_nothing_written (crates/engine/src/backup/tests.rs:21168) asserts no put, the file unchanged and no audit intent line, and kills the mutant with the probe call removed. The wire cell test_adopt_store_checks_the_root_before_it_writes (tests/test_backup_job.py:2675) asserts exit 1 with the refusal, the file's bytes unchanged and the row still AVAILABLE, then the adoption and a plain sync once the manifest is back.

[LeeroyHannigan]

While a startup sync stays pending, each reconciler cycle runs it again and writes this line for each prefix of the root again. An ARN mismatch gets a second line from the WARN at line 632. On a 3,002-prefix root with one manifest at mode 000 and a 1 s cycle, 6 runs in 12 s wrote 36,053 log lines. 18,000 of them were WARN and 18,041 INFO, and the log had 0 new lines in the 5 s after I restored the mode. With the default 60 s cycle a root of N backups writes N lines per minute until the error clears. Log the per-prefix lines on the first run only, or at DEBUG on a retry (rearm knows the sync ran before). Keep the WARN for the transient skips and the end line's counts, and add a cell asserting the line count of a second run.

[robinnsc]

The lasting skips are now logged at INFO only until a run of the startup sync has reported (listed the whole root and written its line of counts), and at DEBUG after that (crates/engine/src/backup/sync.rs:1006 and :1040-1063); the transient skips stay at WARN and the line of counts stays at INFO. I keyed this on a flag set after the line of counts (record_startup_sync_report, sync.rs:1080; crates/engine/src/backup/runtime.rs:542-549) rather than on the first run, because a first run cut at its bound or ended by an error names no prefix, and its reasons would then never reach the log at INFO. The second WARN in adopt_prefix for an ARN mismatch or another partition is removed; the report carries the reason. a_second_startup_sync_run_logs_only_the_transient_skips_and_the_counts (crates/engine/src/backup/tests.rs:21637) asserts that the second run adds one INFO line (the counts), one WARN and three DEBUG lines, and a_startup_sync_whose_first_run_fails_logs_its_skips_at_info_on_the_next and a_startup_sync_whose_first_run_is_cut_at_its_bound_logs_its_skips_at_info_on_the_next (tests.rs:21833 and :21838) cover a first run that does not report.

[LeeroyHannigan]

Sync adopts a manifest of another region under that region's ARN. I copied two prefixes of a us-west-2 server under a us-east-1 PostgreSQL server's root and synced them (exit 0, this WARN line twice). ListBackups and ListBackups with TableName on the us-east-1 server then listed both with us-west-2 BackupArns. DescribeBackup and RestoreTableFromBackup with the us-east-1 spelling answered 400 BackupNotFoundException "Backup not found: " twice each. The same server answered both spellings 200 for the backups it made, and main does the same for its own backups. The ListBackups answers I measured on the service all name the endpoint's region. The service also finds a backup under another region's spelling of its ARN: DescribeBackup answers 200 and echoes the spelling sent. Adopt the row under this server's region and let the restore accept a manifest that names another region's spelling of the row's ARN. Or skip such a manifest with a reason, as for another partition. State the result in docs/differences-from-dynamodb.md either way.

[robinnsc]

Sync now adopts such a manifest under this server's region: the row's ARN is the manifest's rebuilt with BackupArn::in_region (crates/engine/src/backup/sync.rs:681-684; crates/core/src/backup_format/layout.rs:503), and the manifest keeps its own spelling. The restore's two manifest checks and the dead-owner probe accept a manifest that differs from the row in the region alone (names_same_backup, layout.rs:493; crates/engine/src/backup/restore.rs:251 and :1345; crates/engine/src/backup/reconciler.rs:1464). The Where backups live row of docs/differences-from-dynamodb.md (line 127) states the result, including that DescribeBackup answers this server's spelling where the service echoes the spelling sent, and that the source table's ARN keeps the manifest's region. Covered by sync_adopts_a_backup_of_another_region_under_this_servers_region (crates/engine/src/backup/tests.rs:20246), the_dead_owner_probe_accepts_another_region_and_refuses_another_partition (:22312), a_row_over_a_manifest_of_another_region_restores (:20312), a resume case in a_resume_whose_store_documents_fail_verification_is_failed (:17818), and the wire cell test_sync_adopts_another_regions_prefix_under_this_servers_region (tests/test_backup_job.py:2595), which checks ListBackups with and without TableName, DescribeBackup under both spellings, and a restore and a delete by the us-west-2 spelling.

[LeeroyHannigan]

No test covers the store id half of this filter. With && row.store.store_id == identity.store_id removed, AVAILABLE rows of the root under another store id become --prune-missing candidates. The engine suite still passes (477 tests). The other_id row in sync_reports_missing_manifests_and_rows_of_other_stores_and_changes_none has a complete prefix at tests.rs line 20403, so the filter is never reached. Give that row no manifest under the root and assert it is absent from missing_manifests. Then run prune_missing over the report and assert the row stays.

[robinnsc]

Extended sync_reports_missing_manifests_and_rows_of_other_stores_and_changes_none (crates/engine/src/backup/tests.rs:20577) with other_id_gone, an AVAILABLE row under store id store-0 with no prefix under the root. The cell asserts that missing_manifests is exactly the two rows of this store id (:20634), so other_id_gone is absent, then runs prune_missing over the report and asserts two outcomes with other_id_gone still AVAILABLE under store-0 (:20657-20673). It kills the mutant with && row.store.store_id == identity.store_id removed from the filter (crates/engine/src/backup/sync.rs:443).

[LeeroyHannigan]

Neither refusal of this first read has a test. With row_ids: Vec::new() at line 669 the engine suite passes (477 tests): the only cell on NoFile (tests.rs line 20777) asserts the fresh-root case. Under that change an operator whose root lost its file under rows is told "plain sync creates the file" for a root plain sync refuses. With the MultipleRowIds branch disabled, the suite passes too. adopt then answers AlreadyOwned, and the command records the store healthy and syncs a root the check holds unhealthy. Both branches need the file changed between the command's check and this read. Add a cell that removes the file under a row of the root before adopt and asserts NoFile { row_ids: [..] } and FileMissingWithRows. Add a second with rows under two store ids and a file naming this catalog, asserting MultipleRowIds with nothing written.

[robinnsc]

Added adopt_stores_first_read_refuses_a_missing_file_under_rows_and_rows_of_two_ids (crates/engine/src/backup/tests.rs:21212). Its first case runs adopt on a root with no file under an AVAILABLE row of store-1 and asserts NoFile { row_ids: [store-1] }, FileMissingWithRows([store-1]) from into_problem, and nothing written. Its second case puts rows under store-1 and store-9 beside a file naming this catalog and asserts MultipleRowIds([store-1, store-9]) with no put and the file unchanged. The two cases kill the row_ids: Vec::new() mutant and the mutant with the MultipleRowIds branch disabled (crates/engine/src/backup/store_identity.rs:676-682).

[LeeroyHannigan]

No test asserts this audit line: with it removed the engine suite passes (477 tests), and no wire cell looks for it. The only audit assertion in tests/test_backup_job.py (line 2372) is the adoption's. Assert this line in the --prune-missing wire cell as the adoption cell does.

[robinnsc]

test_prune_missing_and_the_unhealthy_store_refusals now asserts exactly one extenddb::audit line containing removed an AVAILABLE row whose manifest is not under the root, the pruned row's ARN and backup sync --prune-missing (tests/test_backup_job.py:1983-1987). In the engine, prune_missing_stops_after_a_removal_when_asked (crates/engine/src/backup/tests.rs:20911) counts the line once, which kills the mutant with the line at crates/engine/src/backup/sync.rs:906-914 removed.

[LeeroyHannigan]

No test covers this take: running the sync whether or not take_startup_sync answered true passes the engine suite (477 tests). The insert-if-absent orders two concurrent runs on an in-memory catalog. The cost is a doubled listing and a report line saying a row landed after the read. Add a cell that takes the pending sync and asserts that a second startup_sync answers None with no listing.

[robinnsc]

I removed the earlier startup_sync_pending() read, so take_startup_sync is the one gate (crates/engine/src/backup/sync.rs:997-1001; crates/engine/src/backup/runtime.rs:535-537); with that read in place the mutant could not be reached from a test. a_taken_startup_sync_is_not_run_by_a_second_caller (crates/engine/src/backup/tests.rs:21846) takes the pending sync, then asserts that startup_sync answers None with no listing of the root, no row read, no insert and the sync not handed back. It kills the mutant that runs the sync without the take.

[LeeroyHannigan]

This wait has no deadline. When the timer counts its ticks from the task's first run instead of the take (line 169), the resumed beat never starts. The cell then spins until an outside timeout stops it (300 s here, with the 476 other cells passed). In CI it would hold run-tests until the job's 15-minute bound instead of failing at an assertion. Bound the wait, with a yield count and an assertion or tokio::time::timeout, so the cell fails naming the beat that did not start.

[robinnsc]

The wait is now bounded at 1,000 yields, with an assertion naming the beat (crates/engine/src/backup/maintenance.rs:1218-1228). With the timer counting its ticks from the task's first run, a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks (:1184) fails at parks false: the resumed beat did not start within 1000 yields in 0.01 s instead of spinning.

[LeeroyHannigan]

This doc and the admin guide (docs/manuals/05-admin-guide.md line 174, "writes a CreateBackup's first object") place the fence at CreateBackup's first put. The code runs it at the request, before claim_backup (job.rs lines 331 to 346), and the job's puts are not fenced. A CreateBackup job I paused before its manifest while another catalog adopted the root completed that manifest under the new owner's root. The new owner's next sync adopted it, as the description says of a put past the read. Say "before CreateBackup's row" here and in the guide, unless the fence moves into the job. Also, "right before each delete_prefix" is one read before two deletions in finish_backup_delete and purge_row (maintenance.rs lines 617 to 624).

[robinnsc]

I moved the fence into the job as well: CreateBackup reads the file at the request, before its row (crates/engine/src/backup/job.rs:338-357), and the job reads it again right before its first put (job.rs:481-489), a refusal failing the row with no object written. Puts after the first are not fenced; the StoreFence doc's last paragraph says the fence narrows the window and does not close it. The doc (crates/engine/src/backup/runtime.rs:770-782) and the admin guide (docs/manuals/05-admin-guide.md:174) now name both reads, and say that purge reads the file once before its row change and its two deletions rather than right before each delete_prefix; finish_backup_delete likewise runs one fence before its two deletions (crates/engine/src/backup.rs:1431-1455). a_root_adopted_after_the_request_gets_no_object_of_the_job (crates/engine/src/backup/tests.rs:23139) adopts the root between the two reads and asserts the row FAILED with the fence's reason, no put and no delete; it kills the mutant without the job's fence.

[LeeroyHannigan]

Round 2, at 84bc263. I checked the 23 items of my round-1 review against this head, and this head fixes all 23. Item 8's MongoDB sync cells now run through mongosh in the CI job's container, and the PR names the Cassandra gap. All 64 checks pass at this head, the matrix jobs included. I ran cargo test -p extenddb-engine --lib backup:: (335 passed), the core layout tests (17 passed) and the app's cmd_backup tests (4 passed).

The changes since round 1 bring four new items: one Major and three Minor. Each one is in code or text that this push changed.

1. Major: docs/design/15-backup-restore.md line 595, and the same sentence in docs/manuals/05-admin-guide.md line 143 (the path row). The design describes a startup sync abandoned at its bound. It says the walk "runs to its end on the blocking pool, holding the listing it builds, and the next cycle starts another walk". At this head the walk holds the root's call slot and is marked in flight (runtime.rs 697 and 1139). The next cycle's check fails at once with "a call of an earlier check has not returned" (runtime.rs 1004 to 1011). That check records the store unreachable. So startup_sync does not run (sync.rs 995), and no second walk starts. The cell a_cycles_startup_sync_cut_at_its_bound_leaves_its_walk_in_flight (tests.rs 21447) asserts this. Neither document says that the store stays unhealthy for the rest of an abandoned walk, with CreateBackup, DeleteBackup and restores refused. Fix: state the behaviour and its consequence in both places, or re-arm the walk after a cut run.

2. Minor: sync.rs line 1001. first_run is startup_sync_takes() == 1. A first run cut at the bound logs no per-prefix line, and the second run logs the lasting skips at DEBUG. So an operator never sees at INFO which prefixes the sync skips for good. Fix: key the INFO pass on whether a completed run logged the lines. Add a cell where the first run is cut and the second one completes.

3. Minor: runtime.rs lines 1004 to 1011, 1198 to 1203 and 1296 to 1301, and sync.rs line 1006.

   • The refusal text "a call of an earlier check has not returned" is the same whatever set the in-flight mark. After an abandoned startup sync it names a check that did not exist. Fix: record which caller's call is in flight, and name it.
   • On an S3 store BoundedCalls::list collects the whole root listing before its first key (try_collect), so the startup sync holds every ObjectMeta of the root in memory. At b42dbfa it streamed the S3 listing. Fix: state this in the STARTUP_SYNC_TIMEOUT doc, or stream the listing page by page under the bound.

4. Minor: store_identity.rs lines 684 to 687. On the AlreadyOwned path the probe's error maps to AdoptRefusal::Problem without at_step. A store that stops answering during that probe ends --adopt-store with "the adoption's store call did not finish within 5 s" and no step. The other store calls of the adoption name their step, and the commit message says this one does too. Fix: map it through at_step("the manifest probe of", ...), and add the case to adopt_store_on_a_store_that_stops_answering_names_the_step.

[robinnsc]

New head: 4812e3e8, on main at 887b6e26 (#840). Each inline comment is answered in its thread with the change, where it is on this head, and the test that covers it. The nightly matrix is running on this head: https://github.com/ExtendDB/extenddb/actions/runs/42775123844. The description's testing section links it and names what CI does not run: no Cassandra leg for tests/test_backup_job.py, and no backup sync against an S3 store, for which it describes a manual moto run.

Other changes: DeleteBackup's doc and the comment above its first fence now say that a root another catalog owns, a missing identity file, a file the store refuses to serve and one that does not parse are refused with 400, the row left AVAILABLE with its objects, and that only a transport failure goes on to the DELETING write. An S3 server error the SDK's retries did not clear (500 InternalError, 503 SlowDown) is recorded at the fence as failed validation, which the admin guide and the description now state; I left s3.rs's error mapping alone, because classing 5xx as transport would change it for every caller. The admin guide also says that a prefix an in-memory server had not finished writing or deleting stays under the root with no row, to be deleted by hand.

[LeeroyHannigan]

Round 3, at 4812e3e. This head fixes round-2 item 2. The INFO report now keys on a run that reported. Two new cells cover a first run that fails and a first run cut at its bound. Items 1, 3 and 4 are not fixed, and item 1 is Major. The changes since round 2 bring two new Minor items. Each item has an inline comment below. I ran cargo test -p extenddb-engine --lib backup:: (338 passed), and all 64 checks pass at this head.

Once you fix the Major item, I will approve. The Minor items can land in the same revision.

[LeeroyHannigan]

Major, from round 2, not fixed. This item still says that after a startup sync cut at its bound "the next cycle starts another walk". The admin guide's path row (05-admin-guide.md line 143) says the same. The code is the same as at 84bc263. The abandoned walk keeps the in-flight mark, so the next check fails and records the store unhealthy (runtime.rs lines 1003 to 1012). startup_sync does not run on an unhealthy store (sync.rs line 996). a_cycles_startup_sync_cut_at_its_bound_leaves_its_walk_in_flight (tests.rs line 21563) asserts this and passes. Neither document says that the store stays unhealthy until the abandoned walk ends, with CreateBackup, DeleteBackup and restores refused. Fix: state the behaviour and its consequence in both places, or re-arm the walk after a cut run, with a test.

[robinnsc]

Documented rather than re-armed. The walk has no cancel flag, so clearing the mark after a cut run would let each cycle start another walk of a root the last one has not finished listing, each holding its own listing on the blocking pool; the mark is what keeps that at one. At fa0bbd9 the sentence is replaced in docs/design/15-backup-restore.md:595 (item 34), docs/manuals/05-admin-guide.md:143 (the path row, with the refusal text quoted) and the STARTUP_SYNC_TIMEOUT doc at crates/engine/src/backup/sync.rs:165-187: on both store kinds the whole listing is collected before the first key; on S3 the abandoned requests end with the dropped future; on a filesystem store the walk runs to its end holding the listing and the root's call slot, so no second walk starts, each cycle's check is refused at once and records the store unreachable, CreateBackup, DeleteBackup and restores are refused and the sync does not run until the first cycle after the walk ends, which runs it again from the first key; a root too large to list within the bound is never synced by an in-memory server. The timeout WARN at sync.rs:1039-1052 says the same. a_cycles_startup_sync_cut_at_its_bound_leaves_its_walk_in_flight (tests.rs:21618) now holds the root listing instead of stalling it for good: it keeps the earlier assertions, then asserts the second cycle's health is Unhealthy(Transport("an abandoned startup sync's store call has not returned")) with reason unreachable and no insert, releases the hold, and asserts the third cycle finds the store healthy, the sync no longer pending, sync_arn(1) AVAILABLE and the DELETING row's deletion finished, manifest then prefix.

[LeeroyHannigan]

Minor, new. This row now says to delete by hand a prefix that a stopped server had not finished writing or deleting. It does not say how to find such a prefix. It also does not say how to tell it from the prefix of a CreateBackup that the running server is still writing. That prefix has no manifest until its last put either. The startup sync names each such prefix in its "not adopted: not a complete backup" line (sync.rs lines 1048 to 1053). A running CreateBackup has a row that ListBackups returns. Fix: name that log line, and say to delete only a prefix that has no extenddb-manifest.json and no row in ListBackups. The Major item's sentence is in this row too.

[robinnsc]

docs/manuals/05-admin-guide.md:143 at fa0bbd9 names the log line (backup store startup sync: <prefix> not adopted: not a complete backup: ...) and says to delete by hand only a prefix that has no extenddb-manifest.json and whose backup ListBackups does not return, since a CreateBackup the running server is still writing has no manifest until its last put either and its row is in ListBackups with status CREATING. The Major item's sentence in the same row is replaced as described in that thread. Text only, no code change.

[LeeroyHannigan]

Minor, from round 2, not fixed. The STARTUP_SYNC_TIMEOUT doc says that only a filesystem store builds the whole listing before the first key. On S3 the startup sync's listing also goes through BoundedCalls::list, which runs try_collect (runtime.rs line 1302). So it holds every ObjectMeta of the root, and it reaches no prefix when the listing outlasts the bound. After an abandoned startup sync, the refusal "a call of an earlier check has not returned" (runtime.rs line 1011) names a check that never ran. Fix: state the S3 case in this doc, or stream the listing under the bound, and name the caller whose call is in flight.

[robinnsc]

Both parts done at fa0bbd9. The STARTUP_SYNC_TIMEOUT doc (crates/engine/src/backup/sync.rs:165-187) now states that on both store kinds the listing goes through BoundedCalls::list, which collects the store's stream into a Vec of ObjectMeta held in memory until the run ends, so a listing that outlasts the bound reaches no prefix; S3's abandoned requests end with the dropped future, the filesystem walk does not stop. The listing is not streamed. The in-flight mark is now InFlightMark (runtime.rs:1123, a Mutex<Option<&'static str>> holding the BoundedCalls caller name), set by MarkIfAbandoned (runtime.rs:1179) and cleared by ClearOnDrop (runtime.rs:1159); the refusal is InFlightMark::refusal(caller) (runtime.rs:1149), "an abandoned {caller} has not returned", so a check refused after an abandoned startup sync reads "an abandoned startup sync's store call has not returned" and after an abandoned check "an abandoned check has not returned", and check_configured_store's INFO line (runtime.rs:1008-1014) names the caller too. The check cell asserts the check text (tests.rs:9521) and a_cycles_startup_sync_cut_at_its_bound_leaves_its_walk_in_flight asserts the startup sync text (tests.rs:21685); a refusal that returned the check text whatever the caller fails the second.

[LeeroyHannigan]

Minor, from round 2, not fixed. On the AlreadyOwned path the manifest probe's error still maps to AdoptRefusal::Problem without at_step. The other store calls of the adoption name their step (lines 675, 714, 743 and 752). The commit message says that a store that stops answering "ends the command naming the step that did not finish". Fix: map it through at_step("the manifest probe of", ...), and add the case to adopt_store_on_a_store_that_stops_answering_names_the_step.

[robinnsc]

crates/engine/src/backup/store_identity.rs:687 at fa0bbd9 maps the AlreadyOwned probe's error through at_step("the manifest probe of", problem), so the refusal reads "the manifest probe of extenddb-store.json: the adoption's store call did not finish within 5 s". adopt_store_on_a_store_that_stops_answering_names_the_step (tests.rs:21314) gains the case "the probe of an owned root" (tests.rs:21324): the identity file names this catalog, the catalog holds an AVAILABLE row of the store, and the head of that row's manifest never answers (a new MemStore::stall_heads_of hook, tests.rs:949); it asserts the step text, the 5 s bound, no put, and the call left in flight. Reverting the mapping to .map_err(AdoptRefusal::Problem) fails that case.

[LeeroyHannigan]

Minor, new. The new sentence says that DescribeBackup's SourceTableDetails.TableArn and the restore's RestoreSummary.SourceTableArn answer in the manifest's region. ListBackups' BackupSummaries[].TableArn comes from the same stored JSON (BackupRow::summary, crates/storage/src/backup_catalog.rs lines 429 to 447), so it does the same, and the sentence leaves it out. No cell checks the region of any of the three fields for a backup adopted from another region. Fix: add ListBackups' TableArn to the sentence. In one of the adoption cells, write the manifest's source table in the other region and assert the three fields.

[robinnsc]

docs/differences-from-dynamodb.md:127 at fa0bbd9 names DescribeBackup's SourceTableDetails.TableArn, ListBackups' BackupSummaries[].TableArn and the restore's RestoreSummary.SourceTableArn as answering in the manifest's region. a_row_over_a_manifest_of_another_region_restores (tests.rs:20323) is now an adoption cell: the seeded row is removed, the manifest's backup_arn and source_table.table_arn are rewritten in eu-west-1 with the sidecar recomputed, sync adopts the prefix under this server's region, and the cell asserts DescribeBackup's source_table_details.table_arn, ListBackups' BackupSummaries[0].TableArn (with BackupArn in this server's region) and the restore response's RestoreSummary.SourceTableArn, all equal to the other region's table ARN, and the restore still reaches ACTIVE with every item. Rebuilding the adopted row's table_arn in the server's region fails it at the Describe assertion; rebuilding the restore's source_table_arn from the backup's ARN fails it at SourceTableArn.

[robinnsc]

Pushed fa0bbd9 (47 files against main 887b6e2; 7 files, +227 −72 against 4812e3e). Your review on 84bc263 arrived while the changes of the next push were being made, so that push did not see it. Each of the five open items is answered in its thread: the Major one is documented rather than re-armed, with the reason there, and the four Minor ones are fixed as proposed. On this head cargo test -p extenddb-engine --lib passes 701 (338 under backup::), app 50 and server 375 pass, fmt and clippy with -D warnings are clean including the storage test-hook features, and the nightly matrix passed on the branch: https://github.com/ExtendDB/extenddb/actions/runs/42775123844.

[LeeroyHannigan]

Round 4, at fa0bbd9. This head fixes all five round-3 items. For the Major, design item 34 and the admin guide's path row now describe the abandoned walk as the code runs it. The walk holds the root's call slot and the in-flight mark, and the server refuses each check until the walk ends. The first healthy cycle after it syncs again from the first key. I agree with your reason not to re-arm the sync: the filesystem walk has no cancel flag.

One new Minor item has an inline comment. It is not a condition for my approval.

[LeeroyHannigan]

Minor, new. The clause "(once the abandoned listing of the root has returned; until then the store is unreachable)" depends on store.kind alone. So every filesystem run cut at the bound prints it. A run cut after its listing ended has no abandoned listing. The run lists the root first (line 502). Then it reads the manifests through the bounded calls and inserts the rows through the catalog (line 739), which the bounded calls do not cover. After a cut in a catalog call, nothing sets the mark. After a cut in a manifest read, the mark clears when that read returns. The next check then runs as usual. The case that the clause is for can end in the Err arm instead (lines 1103 to 1107). Your PR body says that in the cycle cell the listing's own 60 s bound ends the run first, with Err(Listing). That line does not say that the store is unreachable. No cell checks the text of the timeout branch. Fix: after either arm, choose the clause on store.check_call_in_flight(), not on the store kind. Word it for any store call of the run that is still running. Change the comment at lines 1036 to 1038 to match.