[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job (merged) · #740 the restore job, the cross-backend matrix and backup remove-target (merged) · #825 backup sync, --prune-missing, --adopt-store · #855 the CI filters, the open measurement cells and the backup and restore manual · #880 backup sync --move-store, --forget-store, --forget-backup · this PR: backup migrate-legacy · next: #748 export and import through the store, #749 the in-catalog reader's removal.
Closes #821 (part of #624).

Stacked on #880 (feat/backup-store-maintenance, head 537097af), itself on #855 (feat/backup-docs, 925509bd) and #825 (be84a7eb) on main at b8ad9cd5 (#740 merged as 76b9710d, then #792); opened as a draft against feat/backup-store-maintenance. Rebased from #880's earlier head c96cb472 with git rebase --onto 537097af c96cb472. Two files conflicted: the admin guide's [backup] table, where main changed the heartbeat_interval_secs row (on PostgreSQL, MongoDB and Cassandra a job's beat in flight no longer counts) and this PR the stale_job_after_secs row, so each row is taken from the side that changed it; and the upgrade manual's [backup.store] paragraph, which keeps this PR's migrate-legacy sentences with main's "Migration 8" (restore_progress) twice. The design's revision log merged without a conflict; item 37 follows #880's 36, so no item is renumbered. The commit is the same size as before (38 files, +11,610 −311). One commit. No migration, no schema change, no new dependency; Cargo.lock is unchanged. This is the third of the three PRs the design's PR 10 ships as (revision item 34). Every binary and measurement below is a debug build of the tree before the rebase (c96cb472 plus this diff), unless a line says otherwise.

What

Format (crates/core). SnapshotDescriptor::KIND_LEGACY_IN_CATALOG = legacy_in_catalog, whose doc says the marker is empty, that the copy has the legacy create's consistency (a point-in-time snapshot on PostgreSQL and SQLite; on MongoDB an $out of the live collection and on Cassandra a paged read of the live table, neither point-in-time), and that source_backend and source_extenddb_version name the binary that wrote the copy (the in-catalog creator's version was never recorded). No format version change and no fixture byte change; a unit test parses and round-trips a manifest of that kind. is_valid_backup_name is exported, so the command refuses a name the manifest would refuse before it writes anything.

The definition (crates/storage/src/backup_definition.rs). source_table_from_legacy(source, vector_state, region, account), beside restore_input: the table the legacy restore creates, by one rule on every backend. It runs ensure_restorable, refuses LegacyVectorState::Unknown (a SQLite backup with no definition), and otherwise takes restore_input with no overrides (a backup with no definition keeps its keys, its key attribute definitions and the 5/5 fallback) plus the definition's table class, TTL and stream and SseDescription::from_specification of its SSE specification. The vector state is [] on PostgreSQL and SQLite with no names (PostgreSQL with no definition and a NULL or zero snapshot too) and null on MongoDB and Cassandra. Tests: an exhaustive property over every combination of key schema, billing, recorded throughput, GSI shapes, LSI, four SSE specifications, on-demand throughput, class, TTL and stream and no definition, times two vector states and four override sets, that restore_input_from_manifest(source_table_from_legacy(s)) restores what s.restore_input restores; the one difference, a KMSMasterKeyId not carried, has a test of its own.

Storage trait (crates/storage/src/backup_catalog.rs). LegacyBackupReader gains items(account, arn) -> LegacyBackupItems (the name, the recorded source, the vector state and a 'static item stream), list_all() -> Vec<LegacyBackupKey> (the in-catalog backups of every account that are not DELETED, by creation time then ARN; a row that does not decode is an error), list_deleted() (the DELETED ones, which are always a DeleteBackup's), holder_status(account, arn) -> Option<BackupStatus> (the status, DELETED included, of an in-catalog backup at the ARN that the replacement did not replace), finish_replacement(account, arn) -> bool (MongoDB's finish of a replacement that committed and stopped before its drop; false elsewhere, the trait's default), remove_deleted(account, arn) -> bool (the removal of a DELETED in-catalog backup that is not a replacement's, conditioned on both, once nothing is left under its prefix) and replace_with_index_row(row) -> Replaced { Replaced, AlreadyReplaced, NotAvailable(String), InUse(String) }; an index row of the id under another ARN or store is AlreadyExists with nothing written (Cassandra sees only the ARN's own partition, which the trait doc says, and the engine covers). BackupCatalog gains row_holding_backup_id(account, backup_id), the row of the id whatever its ARN, table and store, the key the inserts conflict on; Cassandra, which indexes no backup id, answers None, and the doc says why. The trait doc names the replacement as its second write beside the restore's registration. MaintenanceClaim gains heartbeat_within(bound) and heartbeat_parks_before_its_bound(), with JobClaim's defaults; SQLite's starts the bound once the beat holds its write lock.

PostgreSQL, SQLite, MongoDB, Cassandra. Each stream is a frozen copy of the backend's restore read: PostgreSQL a REPEATABLE READ READ ONLY catalog transaction whose first statement re-checks AVAILABLE, owned by a spawned task sending through a one-slot channel; SQLite the restore's keyset batches (500 rows or 4 MiB), each in its own read transaction re-checking AVAILABLE; MongoDB a cursor on _backup_<backup_id> with batch_size(DEFAULT_FETCH_BATCH) and legacy_item_from_document, a copy of document_to_item, and Unsupported for a document with no backup_id; Cassandra the buckets in order, each read in pages of LEGACY_ITEMS_PAGE = 8 rows by item_index. The replacement: PostgreSQL one transaction (the backups row FOR UPDATE, the CREATING-target check, the insert statement insert_backup_row used, now insert_unowned_row, then the items, the definition and the backups row deleted, so a DELETED backups row is always a DeleteBackup's); SQLite the same under the write lock and BEGIN IMMEDIATE; MongoDB start_replacement, one session transaction ($inc lock_seq while AVAILABLE, the CREATING-target check, the upsert, DELETING with migrated_to_store: true), then finish_legacy_delete's drop and DELETED, a DELETING document with an index row of the same ARN and store being finished as AlreadyReplaced, and a commit whose answer is lost settled by replacement_landed (the marked document and the index document read back: the call's own replacement answers Replaced); Cassandra replace_legacy_row, the row converted in place by one lightweight transaction IF backup_status = 'AVAILABLE' AND store_kind = null AND active_restores = null after creating_restore_target, then write_index_entries (sent once more on a failure) and cleanup_backup_payload, an unknown outcome settled by read_index_row_serial, NotMatched read again up to three attempts, and a converted row without its entries finished by the next call, its entries written only for a row the serial read finds AVAILABLE (one deleted since gets its buckets deleted and NotAvailable). holder_status and list_deleted: the backups rows on PostgreSQL and SQLite; the documents without the marker on MongoDB, whose finish_replacement finishes a marked DELETING document whatever its index row holds; the rows with store_kind null on Cassandra. remove_deleted: DELETE ... AND backup_status = 'DELETED' on PostgreSQL and SQLite (SQLite under the write lock), MongoDB's delete_one of a DELETED document without the marker, and on Cassandra a lightweight transaction IF backup_status = 'DELETED' AND store_kind = null, a lost answer settled by a read of the partition. row_holding_backup_id: the backup_index row by (account_id, backup_id) on PostgreSQL and SQLite, the index document by the same fields on MongoDB. PostgreSQL's maintenance take, with the advisory lock taken, now refuses while the backup_maintenance row's heartbeat is younger than stale_job_after_secs (the session closed, AlreadyExists naming the holder and the window), as the other backends' take does; the module doc says why. Cassandra's conversion writes the insert's columns, created_at among them, which a unit cell compares with the insert's column set. Four frozen paths change, each to order a request wholly before or after a replacement: PostgreSQL's delete reads the status under its FOR UPDATE lock and answers not found for anything but AVAILABLE; MongoDB's reads a DELETING document marked migrated_to_store as not found, in delete and in the lock's miss; Cassandra's DELETING claim carries AND store_kind = null, and so does Cassandra's legacy restore registration (register_restore(.., in_catalog)), whose false re-reads through load_backup and answers not found for an index row. SQLite's are unchanged: the command holds the serve lock. Module docs name the stream and the replacement. Fake implements the seven reader methods and row_holding_backup_id with fault knobs, its holder_status answering None for a marked backup as MongoDB does; PostgreSQL's test-hooks gain legacy_items_pause (after the stream's AVAILABLE re-check, before its first read); Cassandra's test-hooks gain fail_convert_writes, stop_convert_after_row, pause_before_convert and pause_before_legacy_restore_registration; MongoDB's test liveness gains replacement_commit_unknown.

Engine (crates/engine/src/backup/migrate_legacy.rs). migrate(storage, &Migration, Selection::{Arn, All}, on_backup) -> MigrateReport, written against BackupCatalog + LegacyBackupReader and the configured store. --all takes list_all, then list_deleted; <arn> is resolved through holder_status under its two spellings (no listing of every account), then finish_replacement, then the catalog row (AVAILABLE "already in the store", DELETING or DELETED "deleted", after the replacement's call; another store's or a CREATING or FAILED row refused). Per backup, in this order: the id (no instant: refused before anything is read, the text naming 0.1.x and the way out); the table name by the store-prefix rules and the partition; a DELETING backup handed to finish_replacement; a catalog row of the id under either spelling (read_catalog_row: one of the configured store under the stored spelling is finished, reported by its status; any other refuses the backup before anything is deleted, since the prefix carries no region), then a row of the id under any ARN (row_holding_backup_id), refused the same way; the status (CREATING and DELETING skipped, named); the open (Unsupported refused); the name; ensure_restorable ("the legacy restore refuses it too"); the unknown vector state. Then the copy: a listing of the prefix, whose objects with no row are an earlier run's when the manifest there, if any, names this backup (StoreCalls::manifest_arn; one that names another backup or does not read refuses the backup with the prefix left, which on Cassandra is what tells another row's prefix), and are deleted through the fence; the fence before the first put; writer::write_data_files with the item validator under the instance's limits; the count check (its text names PostgreSQL's case, a wrong recorded count, and the way out); the three documents and the manifest (unordered, legacy_in_catalog, no marker); verify_prefix and the second count check; the claim re-checked; replace_with_index_row with the row built as sync builds it (the manifest's name, size, item total and source_table, the id's instant as created_at, unowned). A refusal or failure after the first put deletes the copy through the fence; AlreadyExists at the replacement keeps it, with a note. After a stream error or a short count the command re-reads holder_status and reports Skipped ("deleted during the copy") when the backup is no longer AVAILABLE. Before each step that cannot be undone (the manifest's put, delete_copy, the leftover's deletion, clean_deleted's deletion, every replacement call, finish_replacement and the record's removal) it calls HeldClaim::recheck(); a failure stops the run with what it wrote left, and the stop says so. A DELETED backup goes to clean_deleted: a catalog row of the id under either spelling or any ARN refuses it (one under its own ARN and AVAILABLE means the backup is in the catalog again, after a backup sync of an earlier release adopted a stopped run's copy: the refusal names the row and the choice, DeleteBackup the ARN again or keep it, in --all too and with no audit line; any other may own the prefix), then the listing, the fence, the copy's deletion when its manifest, if any, names this backup, the listing again, and once that lists nothing remove_deleted, each with a note; so list_deleted shrinks, and a DELETED in-catalog row whose id carries an instant is a prefix not yet confirmed clean. --all reports only a deleted backup it did something for, and an id with no instant is never listed in the store nor removed. Outcome::{Migrated, Finished, InStore, Deleted, Refused, NotFound, Skipped, Failed}, carried on BackupOutcome with in_catalog_kept (false for a deleted backup and after a replacement call whose answer did not come) and wrote (the run deleted or replaced anything for it; finish reads it from holder_status, clean_deleted from what it removed); MigrateReport::stopped for a lost claim, a signal, an unhealthy or fenced store, a failed listing, a catalog call and a replacement whose call fails, and for a signal received while the last replacement ran. The writer (job/writer.rs) runs under a ClaimWatch (confirms_the_claim, the loss token, the reason): Pipeline<'a, W: ?Sized = HeartbeatHandle>, so the job's Pipeline<'_> is unchanged, and impl ClaimWatch for HeldClaim, whose timer cancels a lost_signal token when it records the loss. maintenance.rs: HeldClaim records its last confirmed beat and the beat in flight, bounds each beat with heartbeat_within(interval) (a timeout is a loss), and lost() also answers once the last confirmed beat is two periods old with no beat on schedule in flight within its bound (a parked SQLite beat confirms while it waits). A beat is on schedule when it started within 1.1 periods of the last confirmed one (the beats fall one period apart from the take, interval_at), so a process stopped and resumed starts a late beat that confirms nothing; the flight is marked by a drop guard (InFlight), cleared by a timer that ends mid-beat, and a confirmation is recorded before the flight ends. recheck() is one beat under the period's bound that counts as a confirmed beat or records the loss as the timer does; purge's loop and the waits and loops of backup sync's options read the same lost(). is_item_refusal shares refusal_reason's words through one constant. put_document is generic over the watch. Under backup-test-hooks the writer's after_data_file and the command's before_manifest and after_manifest kill points pause it as they pause a job.

Sync, Describe, Delete, Restore and ListBackups. sync.rs: SkipReason::InCatalog { backup_arn, status } ("an in-catalog backup (STATUS) holds this ARN; run extenddb backup migrate-legacy ", and for DELETED "an in-catalog backup with this ARN was deleted; this prefix is a copy a stopped migration left, which extenddb backup migrate-legacy deletes"), not transient, from holder_status under both spellings before the row read under them; sync, finish_group, adopt_prefix and startup_sync now need LegacyBackupReader, which every caller's storage has. backup.rs: DescribeBackup, DeleteBackup and RestoreTableFromBackup read the catalog row once more after a legacy not-found (RestoreTableFromBackup after the source read's and after the registration's, then taking the row's path, restore_from_row), and ListBackups reads the legacy reader before the catalog index, then drops an in-catalog summary whose ARN a listed catalog row has.

App (crates/app/src/cmd_backup/migrate_legacy.rs, a child of cmd_backup). BackupCommand::MigrateLegacy { arn, all } with a required ArgGroup (usage migrate-legacy <ARN|--all>, exit 2 for neither or both) and <arn> checked by a clap value_parser (backup_arn_arg, exit 2 before anything is opened), Mode::Write, the registration name extenddb backup migrate-legacy. The command needs Identity::Known, takes the claim on every backend, starts HeldClaim, runs wait_for_jobs over the configured identity (on SQLite after resolve_dead_owners, with a zero window), runs the migration, releases the claim, prints one line per backup with its notes indented (outcome_lines; "Its in-catalog backup is kept" after a refused or failed line only where in_catalog_kept holds, so never for a deleted backup nor after a replacement call whose answer did not come) and the summary line (N migrated, N finished, N already in the store, N deleted, N refused, N skipped, N failed), logs one extenddb::audit line per backup the run deleted or replaced anything for (audited, by wrote; none for an outcome that wrote nothing), and exits 0 only when every backup reached is in the store or deleted. The refusal with no store configured says the command needs one. backup_arn_arg returns the bare reason, so clap's error states it once. The subcommand's help names the item check's and the legacy restore's refusals and --all's clean-up after deleted backups; lib.rs's help and the module doc name the command.

Harness. tests/test_backup_job.py: three new cells, one extended (below); helpers _catalog_rows, _in_catalog_rows, _copy_in_catalog_backup, _keep_in_catalog_backup, _put_back_in_catalog_backup, _typed_canonical, _migrate. tests/test_backup_store_config.py: the serve-lock cell runs migrate-legacy beside a SQLite server.

Docs and texts. The backup manual's Legacy backups section: "Migrating legacy backups into the store" in place of "This release has no command that converts a legacy backup" (the two forms; what is kept; the check and the one-step replacement per backend; the cost: the claim, every backup and restore job answering LimitExceededException, the wait, SQLite stopped, a killed command's claim held until its heartbeat is older than stale_job_after_secs (PostgreSQL too; SQLite's serve lock takes it), PostgreSQL's held REPEATABLE READ transaction and the vacuum it holds back; <arn> to migrate in pieces; sample output, and that the log lines share standard output; the outcomes, the notes and exit (audit lines only for a write, "deleted" with the record's removal, "deleted during the copy", where "Its in-catalog backup is kept" is said, the claim re-checked before each step that cannot be undone); the refusals, every 0.1.x backup first, with the list's lead-in qualified and the list completed (the count check, a row of the id under another ARN or a manifest of another backup under the prefix, a CREATING or FAILED row of the configured store, the partition, AlreadyExists at the replacement, a deleted backup an AVAILABLE row holds again); the KMS sentence (the stored specification loses the key id, and no answer reports one before or after); the restore-and-back-up way out, kept from the earlier text; what a migrated backup reports; a stopped run, sync's two skips and a deleted backup's copy and record (the guarantee rests on the record, which --all removes once the prefix is clean, so --all is run after the last DeleteBackup of an in-catalog backup before the release that removes the reader); MongoDB's and Cassandra's stopped replacements, and running <arn> before deleting a Cassandra backup whose conversion stopped; Cassandra's recurring --all, which exits 1 while a create is in flight; memory, measured on this change's earlier revision, with Cassandra's table shape); the vector-index sentence corrected; the Strategy section's Cassandra paragraph and the catalog-lost paragraph point to it. The upgrade manual: the store paragraph, the 0.1.x paragraph (refused on their ids), the rollback paragraph (a backup migrated after the upgrade is adopted by --adopt-store like any other backup this release wrote) and --all run after the last DeleteBackup of an in-catalog backup, and why. The admin guide: the command's paragraph (the leftover rule by id and manifest, the record's removal, the re-check, the line kinds, the audit lines, memory) and sample output with its refused line, the registration names, and the stale_job_after_secs row (a dead maintenance command's claim, on PostgreSQL too). The differences doc: "Where backups live" and the maintenance-claim row. Design: revision item 37, with sections 3, 4(c), 4(f), 4(g) and 7, PR 10's claim paragraph (PostgreSQL's take needs the stale window), the PR 10 paragraph and transitions, the PR 14 lines (its migration also refuses while a DELETED in-catalog row of a DeleteBackup whose id carries an instant exists, naming --all; a DELETING one names DeleteBackup again) and section 10's upgrade-manual row edited with it.

Why

Design section 3, 4(f), 4(g) and the PR 10 section's migrate-legacy paragraph, and PR 14's precondition: before the in-catalog reader is removed, operators need a way to move the backups it holds into the store without losing their ARNs. Until now the only way was a restore and a new backup, which gives a new ARN, a new creation time and, for a table that still exists, an interim table. Cassandra keeps writing in-catalog backups until PR 14 settles its backup path, so there the command is a recurring step.

Choices to weigh

• One replacement step per backend, not "insert the row, then the reader's delete". DeleteBackup does not read the maintenance claim. In the design's order a DeleteBackup that landed between the AVAILABLE check and the insert was answered DELETED for a backup that then came back as the migrated row, and a restore target in CREATING made the delete refuse after the insert, leaving both rows and two ListBackups entries. On Cassandra the order cannot work at all: the legacy row and the index row share the backups_by_arn key, and the frozen delete removes the list rows the migrated row uses and leaves a DELETED row that every later insert answers false for. The cost is one write method on the frozen trait (as the restore's registration already is) and four small changes to frozen paths (three deletes and Cassandra's legacy registration), each pinned by a test and a mutant; PR 14 removes all of it with the trait.
• Every 0.1.x backup is refused. The released fixtures carry 13-digit ids (v0.1.14 is the newest tag, and the shared 14-digit id came after it), and the store format's ids carry their creation instant, which the manifest parser checks. So the command migrates only in-catalog backups taken by this release line: without a store, with kind = "disabled", and on Cassandra. The texts say so plainly, and the refusal names the way out. The pre-0.0.4 SQLite refusal is in practice a subset of this one; it stays as a defensive check, reachable through a hand-seeded row, with its own SQLite cell.
• The source table is what the legacy restore creates. One rule on every backend, restore_input with no overrides plus the features DescribeBackup reports, so a migrated backup restores the table the in-catalog backup restored. The design refused only the pre-0.0.4 SQLite case; a backup with no definition on PostgreSQL, MongoDB and Cassandra migrates with its keys and the 5/5 fallback, as its legacy restore creates it. What the legacy restore refuses (vector indexes, a multi-part key) is refused here too, so the migration lifts no refusal. The SSE specification's KMSMasterKeyId does not survive the manifest's SSE description; the manual names it.
• Extra refusals the design does not list: a table name the store-prefix rules refuse, a backup name the manifest refuses (the name survives with the ARN or not at all), a MongoDB backup with no item collection, and an ARN a row of another store holds. Each keeps the in-catalog backup and writes nothing.
• The claim on every backend, and the deletion fence. The design's invariants say every writing command takes the maintenance claim, and 4(f)'s list did not name this one. On SQLite it costs nothing under the serve lock and gives one code path. The fence runs before the first put, as CreateBackup's does, and before each deletion of a copy. The cost, stated in the texts: every backup and restore job answers LimitExceededException for the run, and on SQLite the server is stopped. <arn> migrates in pieces.
• Always rewrite after a crash. Objects under the prefix with no row are deleted through the fence and the copy is written again. Reusing a complete copy would save one copy after a crash and needs a second rule; one rule is simpler to state and to test. Meanwhile backup sync skips such a prefix, naming the command, so a sync between the crash and the rerun cannot give the backup two rows. Cassandra already skipped it through the insert's false.
• A deleted backup never comes back from a copy. A stopped run's copy can outlive its in-catalog backup: a DeleteBackup of the legacy ARN does not read the claim, and the copy's deletion can fail. So the in-catalog row in any status, DELETED included, bars sync's adoption, and the command deletes the copy of a DELETED in-catalog backup that no catalog row holds. Telling a DeleteBackup's DELETED row from a migration's needs one mark per backend: the replacement deletes the backups row on PostgreSQL and SQLite (the frozen delete keeps its tombstone), MongoDB's migrated_to_store marker already tells them apart, and Cassandra's converted row is an index row. list_deleted lists every DeleteBackup's tombstone, so --all lists one prefix per deleted in-catalog backup with an instant in its id (none of a 0.1.x release's). Once that prefix lists nothing and no catalog row holds the id, --all removes the tombstone (remove_deleted), so the list shrinks instead of growing with every DeleteBackup, and a tombstone left means a prefix not yet confirmed clean. The guarantee therefore rests on those tombstones until --all has cleaned them: PR 14's migration is to refuse while one with an instant-bearing id exists, and the manuals say to run --all after the last DeleteBackup of an in-catalog backup. A row of the id under any ARN keeps the prefix, since the inserts conflict on (account_id, backup_id) and a prefix carries no region: row_holding_backup_id on PostgreSQL, SQLite and MongoDB, and on Cassandra, which can read only an ARN's partition, the manifest under the prefix, which must name this backup.
• No maintenance command acts after another took its claim over. A maintenance claim whose heartbeat hangs (a half-open connection), or whose process was stopped and resumed, used to stay "confirmed" to the writer and to the check before the replacement while the stale window let a second command take the claim over and act on the same prefix; and PostgreSQL's take succeeded as soon as the advisory lock was free, so a command whose catalog session ended could still run until its next beat while a second replaced the backup (the first then put its manifest over the second's copy, failed its re-read and deleted the prefix, leaving an AVAILABLE row with an empty prefix). Three changes hold one invariant on every backend, a holder stops acting within two periods of its last confirmed beat and a taker needs a heartbeat older than stale_job_after_secs, which the config holds to at least three periods: HeldClaim bounds each beat by a period and reads a claim with no beat confirmed within two periods as lost, counting a beat in flight only when it started on schedule (SQLite's parked beat confirming while it waits); PostgreSQL's take refuses while the row's heartbeat is younger than the window, as the other backends' take does; and the command calls HeldClaim::recheck(), one live beat, before each step that cannot be undone, since the job's other fences (generation-conditioned writes, a prefix of its own) have no counterpart here. The cost: after a killed command on PostgreSQL the next one fails, naming the holder, until the window passes (Breaking changes). The bound and the on-schedule rule also apply to purge and the options of backup sync, which read the same lost().
• The count is checked twice, against the recorded count. A DeleteBackup during the read ends the SQLite, MongoDB and Cassandra streams short; PostgreSQL reads a pinned snapshot. The manifest's table_size_bytes is the sum of the copied items' sizes, as the job records it, so BackupSizeBytes and TableSizeBytes can differ from what the in-catalog backup reported; the name, ARN, id, creation times and item count do not.
• --all goes on past a refused, skipped or failed backup and exits 1 at the end with the counts; a global condition (claim lost, a failed re-check, signal, store unhealthy or fenced, listing or catalog failure, a replacement whose answer did not come) stops the run at once.
• A Cassandra conversion that stopped after its row is an index row, which list_all does not list, so --all does not finish it; migrate-legacy <arn> does (its "already in the store" path runs the replacement once more). Until then the backup is served by Describe, List and Restore, and backup list, which reads status entries, does not show it. The manual says so.
• ListBackups dedupes by ARN. It reads the legacy reader before the catalog index, so a replacement that commits between the two reads lists the backup in both, and the filter keeps the catalog row's summary. The other way to list both is a backup sync without the in-catalog skip (a feat(backup): backup sync, --adopt-store and --prune-missing [backup/restore 10] #825 or feat(backup): backup sync --move-store, --forget-store, --forget-backup [backup/restore 10b] #880 build during a rolling upgrade) adopting a stopped run's copy before the migration replaced it. DescribeBackup, DeleteBackup and RestoreTableFromBackup read the catalog row once more after a legacy not-found, for the same window.
• The stream reads are frozen copies, not calls into the data path, so the data path stays free to change before PR 14. Cassandra's bucket read is paged (8 rows of a 64-row bucket), where the restore's copy reads a whole bucket, about 1.7 GiB at the line cap; the restore's copy stays as it was.

Testing done

On the tree rebased onto 537097af (main at b8ad9cd5), debug builds, every tracked file touched: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 565 passed, app 63, server 343, storage 241; SQLite --lib backup 75; PostgreSQL backup_restore 74 (--test-threads=2); the wire suites tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 47 passed, 13 skipped (-n 4), as before the rebase.

Measured on the tree before the rebase, debug builds, a 16-core Xeon Platinum 8124M host (30 GiB, Linux 5.10), the services local.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, and with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; cargo +1.88.0 check --locked --workspace --all-targets clean.
• Core: a_manifest_of_the_legacy_in_catalog_kind_parses_and_round_trips; --lib backup_format 181 passed, 1 ignored. Storage: --lib 237 passed, with the four source_table_from_legacy cells (the exhaustive property; the described features and the legacy identity; the KMS key id; the refusals and the unknown vector state).
• Backends, against the live services: SQLite legacy_migration 7 cells (the stream across both batch bounds with 1,201 small and three 3 MiB items; a stream ending at a DeleteBackup after its batch; list_all across accounts; the replacement's insert and removal in one step, the backups row gone, an adopted copy finished, another store refused, in-use and unavailable refused with nothing written; holder_status and list_deleted telling a delete from a replacement, remove_deleted of a DeleteBackup's row only and once, row_holding_backup_id by the id and the account); PostgreSQL legacy_migration 11 cells (the stream from its pinned snapshot while another transaction deletes every item row; the same replacement and holder cells; a legacy delete queued behind a replacement answers not found; a real replacement and a DeleteBackup queued on each other in both orders, the replacement's insert or the delete's item deletion held behind a table lock, each asserting the answers, so a replacement without FOR UPDATE fails at an answer; a DeleteBackup committed while the stream is paused after its AVAILABLE re-check, the stream reading every item of the pinned snapshot), and the dead-holder claim cell refusing the take while the row's heartbeat is young; MongoDB 9 live cells (2,500 items past a cursor batch; a drop ending a stream; the replacement; a replacement stopped before its drop finished by the next call, DeleteBackup leaving it; the same stop with the index row removed since, finished by finish_replacement; holder_status and list_deleted; a commit whose answer is lost answering Replaced; the in-use and unavailable refusals) and the unit cells of lock_miss (a migrated document) and legacy_key (a missing field is an error); Cassandra 9 direct cells on a node of their own (200 items over four buckets in pages of 8; list_all; the conversion in place, the legacy describe, list and delete no longer seeing it; in-use and unavailable; a lost answer either way, a stop after the row finished by the next call, and a DeleteBackup paused between its read and its claim answering not found; a legacy restore paused between its DDL and its registration while the row is converted, answering not found with its target removed; a registration landing between the conversion's read and its transaction, InUse; a stopped conversion of a backup deleted since, its buckets deleted and no entry written; holder_status and list_deleted), run with the 6 existing cells that reach the changed delete, registration, restore_source, insert and abandoned-create paths (15 passed), holder_status_and_list_deleted covering remove_deleted (an index row and an AVAILABLE row left, the DELETED one removed once), and the unit cell comparing the conversion's columns with the insert's. The PostgreSQL backup_restore suite: 74 passed; SQLite --lib: 289 passed; MongoDB live live_: 80 passed, --lib backup: 31 passed.
• Engine --lib: 563 passed, 41 of them in backup::tests::migrate_legacy: the copy under the ARN and its row; each refusal before the copy writing nothing; a refused item deleting the copy and --all going on; a short or failing read; the fence on a root another catalog adopted, before a leftover's deletion, at a copy's deletion after its manifest and at a deleted backup's copy; a copy that does not read back; a store serving another backup's copy (the manifest's ARN) or another count (the second count check); the items' size and the region and creation-time notes; a copy an earlier run left; a backup whose row is in the store; <arn> and the finish by the row's status; a row under the other spelling; AlreadyExists keeping the copy; a replacement that fails; deleted or in use at the replacement; CREATING and DELETING skipped; MongoDB's stopped replacement finished whatever its row holds; a deleted backup's whole and partial copies (the sync skip, --all and <arn> deleting them, a row's prefix left, nothing listed for an id with no instant); a signal, a lost claim, a stalled heartbeat before the replacement, a signal during a slow copy and during the replacement; the claim's loss token; the ARN's spellings; sync's skip under both spellings; Describe, Delete, Restore and ListBackups racing a replacement; the ListBackups dedupe; a claim taken over since its last beat, which stops each step that cannot be undone (the manifest, the leftover's deletion, delete_copy, clean_deleted's deletion, the record's removal, the replacement, finish's and finish_replacement), and one taken over during the copy, caught before the next such step; a lost claim read before each backup; a row of the id under any ARN keeping the prefix, and with no id index (Cassandra's case) a manifest of another backup keeping it; a deleted backup's record removed only once its prefix is clean (not while a deletion is refused, and after the listing again); a deleted backup an AVAILABLE row holds again, refused rather than called deleted; a read a DeleteBackup ended, and a backup deleted before its open, skipped as deleted while --all goes on; a restore whose registration meets a replacement taking the row's path; sync's skip of an in-catalog backup and its copy's manifest both under another region; Fake's holder_status answering None for a marked backup. Six cells of HeldClaim (a hanging beat lost after two periods and a parked one confirmed; a stopped timer lost after two periods; a beat that starts after a gap confirming nothing, parked or not; a beat on schedule confirming only within its bound; a beat that panics leaving no flight; recheck() counting as a beat or recording the loss). The job, writer, sync and purge cells pass unchanged; describe_backup_of_another_accounts_id_is_not_found now counts the second row read. App --lib: 63 passed (the outcome lines and notes, the audit lines, the ARN check, a refused or failed line of each kind saying the in-catalog backup is kept only where it is, an audit line only after a write).
• Wire, Python 3.10, moto variables: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 47 passed, 13 skipped (-n 4), on the PostgreSQL build 42 passed, 18 skipped (-n 2); the migrate cells on the MongoDB build (a replica set of their own) 2 passed, 1 skipped. New: test_migrate_legacy_carries_in_catalog_backups_into_the_store_under_their_arns (303 items with a GSI and NS, B, nested M and L, BOOL and NULL values; two in-catalog backups; <arn> spelt in another region, --all, reruns, "1 already in the store"; the manifests; the in-catalog backups rows gone with no item or definition row; ListBackups once each, DescribeBackup as before, the restore's items and GSI, DeleteBackup; beside the server on PostgreSQL and MongoDB), test_migrate_legacy_refuses_a_backup_with_no_instant_or_an_item_618_let_in_and_keeps_it (--all past a BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 backup and, on SQLite and PostgreSQL, a 13-digit copy of a backup's rows; the three refused items logged; both kept and AVAILABLE; the 13-digit one restored and backed up again into the store) and test_a_backup_deleted_after_a_stopped_migration_never_comes_back (SQLite, PostgreSQL: a malformed <arn> exits 2 with no claim taken; two migrations put back as a stop after the copy leaves the catalog, one prefix cut to its data files; sync adopts neither, before and after DeleteBackup of both; --all deletes both copies, each line with its note, "2 deleted", exit 0; both stay BackupNotFound). Extended: test_a_second_maintenance_command_fails_at_once (PostgreSQL: migrate-legacy --all refused while --prune-missing holds the claim, then holding it while it waits for a live job, backup purge --confirm refused beside it, a SIGTERM stopping it before any change) and the serve-lock cell (exit 6 beside a SQLite server).
• Binary-level checks before the wire cells (SQLite, PostgreSQL; this diff's first form over feat(backup): backup sync --move-store, --forget-store, --forget-backup [backup/restore 10b] #880's earlier head 11544ecd): init with kind = "disabled", two CreateBackups, the store configured, the command run (refused with exit 6 beside a SQLite server, nothing written), boto3 afterwards: 33 and 30 checks, none failed.
• SIGKILL at each kill point, on SQLite and PostgreSQL builds with backup-test-hooks, a 40,000-item table in two data files (this diff's first form): after the first data file, before the manifest and after it, the in-catalog backup was whole with no row; after the manifest a plain backup sync skipped the prefix naming the command and wrote no row; each rerun deleted the copy, migrated 40,000 items and removed the in-catalog backup; a restore of the last held every item. On PostgreSQL the killed command's advisory lock ended with its session, so that first form's rerun took the claim at once; on this tree the rerun takes it once the dead holder's heartbeat is older than stale_job_after_secs (the takeover runs below). On this tree's backup-test-hooks builds: SIGKILL after the manifest and after the first data file, then DeleteBackup of the in-catalog backup, then --all, <arn> and a plain sync: the copy deleted ("1 deleted"), nothing adopted, the backup BackupNotFound, on SQLite and PostgreSQL, and on PostgreSQL with an S3 store (moto); on PostgreSQL a DeleteBackup while the run was paused after its manifest, with the prefix made unwritable: the run skipped with its note and exit 1, sync adopted nothing, --all then deleted the copy; a SIGTERM at each point exits 1 (after the manifest the replacement completes first); the kill-after-manifest rerun migrates. Cassandra's stop after the conversion is the direct cell above (no binary kill point and no Cassandra wire leg).
• Mutants, each a targeted one-occurrence edit, the named tests run, the original bytes written back and checked by hash: 32 in core, the storage crate and the four backends (the kind's text; the definition untrimmed, the unknown state accepted, ensure_restorable ignored, null written as [], the class or the SSE description dropped; a batch or bucket or page boundary skipped, a stream truncated or read outside its snapshot, the per-batch AVAILABLE check dropped; the in-use check skipped, deletion before the insert, Cassandra's store_kind left null or the buckets deleted before the conversion, a lost answer read as applied; each frozen delete's new check dropped; DELETED rows or index rows listed), 10 in the engine (the id refusal, the first count check and both, the fence before the first put, --all stopping at a refusal, the leftover kept, the re-read skipped, the sync skip, the dedupe, the loss token), 3 on the wire (a run with refusals exiting 0, the <ARN>/--all group optional, the wait skipped); and since: 18 in the backends (the replacement leaving a DELETED row on PostgreSQL or SQLite, list_deleted or holder_status filtering wrongly on each backend, PostgreSQL's replacement without FOR UPDATE, MongoDB's holder_status ignoring the marker, finish_replacement a no-op, lock_miss without the marker, the lost commit not settled, list_all taking a default for a missing field, Cassandra's registration without store_kind = null, the conversion without active_restores = null, a deleted row's finish writing entries), 29 in the engine (a deleted backup's prefix adopted by sync, not cleaned, not listed, cleaned without the fence or over a row; MongoDB's stopped replacement not finished; HeldClaim with no bound and no two-period clause, either alone, or no parked clause; the row read under one spelling; the copy deleted on AlreadyExists; the finish ignoring the row's status or calling for a CREATING row; each of the four re-reads dropped; ListBackups reading the index first; the fence before a leftover's and a copy's deletion; the second count check; the claim check before the replacement; the manifest's ARN check; table_size_bytes not set; the copy kept on a signal; the region and the creation-time notes; the finish's InUse and NotAvailable answers; the signal after the last replacement), 3 in the app (the notes not printed, the in-store finish not audited, the in-catalog suffix on every refusal), 2 on the wire (<arn> not checked before the claim, deleted backups not cleaned): 97, all killed; and on this tree 46 more, each a targeted edit with the named cell run and the bytes written back and checked by hash, all killed: 4 in HeldClaim (any late beat confirming, the flight cleared without the guard, the re-check recording no loss or not counting as a beat), 8 re-checks dropped (before the manifest, the replacement, the leftover's deletion, delete_copy, clean_deleted's deletion, the record's removal, finish's replacement, finish_replacement), 4 of the id rule (the lookup dropped in the migration and in clean_deleted, the manifest check dropped at the leftover and at a deleted backup's copy), 3 of the record's removal (never, before the prefix is known clean, without the listing again), 3 of the kept and skipped reports (no re-read after a short or failed read, in_catalog_kept always true, a lost replacement answer not recorded), one each for the deleted backup an AVAILABLE row holds, the audit of a write, the restore's third read, the leftover's wrote, MongoDB's stopped replacement answering Deleted and the five engine cells of other findings, 3 in the app (the suffix on every line, an audit line without a write, the clap error doubled) and 11 in the backends (PostgreSQL's take of a young dead row, its stream's AVAILABLE re-check outside the snapshot, its replacement without FOR UPDATE; remove_deleted of any status on each of the four; row_holding_backup_id of any account on PostgreSQL, SQLite and MongoDB; Cassandra's conversion without created_at): 143 in all.
• Takeover runs on this tree's backup-test-hooks builds, 45,000 items padded to 1,000 bytes, stale_job_after_secs = 45, a second run on another config (32 MiB data files): on PostgreSQL a first run paused before its manifest had its claim session ended (pg_terminate_backend) right after a beat; the second was refused 12 times, naming the holder, and migrated 51.6 s after the beat; the first, released, exited 1 at its re-check; the row AVAILABLE with its six objects, a restore holding 45,000 of 45,000 items. On MongoDB (a replica set of its own) the first was stopped (SIGSTOP) 52 s at the same point, the second migrated, the replica set was paused (docker pause) as the first resumed and was released; the first exited 1 ("no heartbeat confirmed the maintenance claim within two periods"), the prefix unchanged throughout, the restore 45,000 of 45,000. On the head before this change both sequences ended with an AVAILABLE row whose prefix was empty. On SQLite: a whole copy left with the in-catalog backup AVAILABLE, DeleteBackup, sync adopting nothing, --all with the prefix made unwritable (exit 1, the record kept, no "Its in-catalog backup is kept"), --all again (the copy deleted, the record removed, "1 deleted"), a third --all ("0 deleted", no audit line), sync, <arn> refused (no record and no row), DescribeBackup BackupNotFound.
• Memory and time of the command (design section 7, the manual and the admin guide state it), measured on this change's earlier revision, whose read paths and writer this tree keeps (the re-checks add one claim beat per step that cannot be undone): in-catalog backups of the job's measurement shape (a composite key, two GSIs, a 200-byte pad), each migrated in its own run with the server stopped, ru_maxrss of the command through wait4, two runs each:

Backend	1,000 items	100,000 items	250,000 items
PostgreSQL	56.1 MiB, 0.9 s	56.1 MiB, 8.2 to 8.3 s
SQLite, a 224 MB database file	55.0 MiB, 0.1 s	110.8 to 110.9 MiB, 7.0 s
SQLite, a 557 MB database file	67.9 MiB, 0.1 s		116.9 to 117.1 MiB, 16.8 to 17.2 s

An independent measurement of the same shape on a host of the same description gave 68.4 to 74.9 MiB on PostgreSQL beside a running server and, on SQLite, 81.0 MiB for 1,000 items over a 562 MB file and 121.2 and 148.6 MiB for 250,000. So PostgreSQL is flat in the item count at 56 to 75 MiB, and SQLite's excess (55 to 149 MiB measured) is the page cache of the command's connections (PRAGMA cache_size = -32000 per connection), bounded by about 32 MB times the pool's connections (pool_size, 10 by default), its small-backup figure following the database file's size. The earlier run of this PR measured MongoDB and Cassandra flat at 56.1 and 55.2 MiB for 1,000 and 100,000 items (12.4 and 10.6 s for 100,000; Cassandra without GSIs, whose asynchronous queue over 100,000 writes timed the node out); their read paths are unchanged by this revision.

Acceptance lines of #821 (the design's PR 10 acceptance list) and where each is tested:

Line	Tests
migrate-legacy on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone	engine an_in_catalog_backup_is_copied_under_its_arn_and_replaced_by_its_row; the backends' replacement cells (no non-DELETED row, no items, definition, collection or buckets); wire test_migrate_legacy_carries_in_catalog_backups_... (SQLite, PostgreSQL, MongoDB; the in-catalog rows read on SQLite and PostgreSQL). Those cells' backups are taken by this tree's in-catalog create, the frozen copy of the PR 0 (#384) code each backend keeps, so they stand in for the PR 0 binary's; an independent run on this change's earlier form (head 27070b6c) also migrated, on SQLite, a GSI table's backup with B and NS values that main 94b5f689 (catalog migration 5) took, after migrate --yes: 40 items, DescribeBackup differing only in BackupSizeBytes, the restore holding 40 items with 40 GSI hits
migrate-legacy refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason	engine each_refusal_before_the_copy_... and a_refused_item_deletes_the_copy_...; storage a_copy_refuses_what_the_legacy_restore_refuses_and_an_unknown_vector_state and SQLite's stream cell (the definition row removed: Unknown); wire test_migrate_legacy_refuses_... (the 13-digit id on SQLite and PostgreSQL, #618 on all three)
a second maintenance command while one holds the claim fails at once	wire test_a_second_maintenance_command_fails_at_once (PostgreSQL, both ways)
a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler has finished the one and failed the other	not a cell of its own: migrate-legacy waits through the shared wait_for_jobs (cmd_backup.rs), the function whose wire cell test_a_maintenance_command_waits_out_a_deleting_row_and_a_dead_restore runs it from sync --forget-backup; the migrate cells' second-command cell shows the command waiting for a live job, and an independent binary run on PostgreSQL waited for a running legacy restore of a 60,000-item backup and migrated after it
on SQLite every writing command refuses to run beside a server	wire, the serve-lock cell (exit 6)
PR 10 paragraph: the re-read and count check before the row	engine a_read_short_..., a_copy_that_does_not_read_back_...
4(c): the fence before the first put	engine migrate_legacy_on_a_root_another_catalog_adopted_writes_nothing
transitions: legacy rows replaced by an AVAILABLE row	the backends' replacement cells (the row unowned, completed_at set)
4(g): sync skips a prefix an in-catalog backup holds	engine sync_skips_a_prefix_... and a_backup_deleted_after_a_stopped_copy_never_comes_back; wire test_a_backup_deleted_after_a_stopped_migration_never_comes_back; SIGKILL after the manifest, then backup sync (SQLite, PostgreSQL)

Cassandra has no wire leg (no pytest fixture runs it); its share is the direct cells above and the memory run.

Not in this PR

• Backup/restore: legacy reader removal #749: removing the in-catalog reader (PR 14), whose refusal counts the rows that are not DELETED, as revision item 37 records.
• Carrying a backup with a 13-digit id (every 0.1.x backup) into the store format: refused by design, since the format's ids carry their creation instant; the way out is a restore and a new backup.
• A Cassandra wire leg (direct tests only).
• On Cassandra, a conversion stopped after its lightweight transaction and then a DeleteBackup of the migrated backup, whose index row is then removed, leaves the item buckets in the account's keyspace: nothing deletes them until PR 14 drops backup_items. The manual says to run migrate-legacy <arn> before deleting such a backup.

Breaking changes

None for a released build's data. backup sync skips a prefix whose ARN an in-catalog backup holds, in any status, where it could adopt it before; no release wrote such a prefix. ListBackups lists an ARN both readers hold once. On PostgreSQL, MongoDB and Cassandra an in-catalog DeleteBackup that read the backup before a replacement committed answers BackupNotFoundException, where it could have answered DELETED for a backup that lives on in the store. On PostgreSQL the second of two concurrent DeleteBackups of one in-catalog backup now answers BackupNotFoundException (the status is read under the row lock), where it answered DELETED. On Cassandra a legacy restore whose registration lands after a migration converted its backup answers BackupNotFoundException. A maintenance command (purge, migrate-legacy, the options of backup sync) whose heartbeat does not confirm the claim within two periods, counting a beat in flight only when it started on schedule, now stops as for a lost claim. On PostgreSQL a maintenance command (purge, remove-target, migrate-legacy, the writing forms of backup sync) started after another was killed, or after the other's catalog session ended, now fails at once, naming the holder, until the dead holder's heartbeat is older than [backup] stale_job_after_secs (300 s by default), as on MongoDB and Cassandra; before, it took the claim as soon as the advisory lock was free. A script that reruns a killed command at once on PostgreSQL must retry or wait for the window. SQLite is unchanged: under the serve lock the next command takes the claim at once.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com