From fa0bbd984bf88782c7fe5ffd88b6d8bf430cbd95 Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Tue, 10 Nov 2026 11:05:49 +0000 Subject: [PATCH] feat(backup): backup sync, --adopt-store and --prune-missing `extenddb backup sync` rebuilds the catalog's rows from the configured backup store. It reads the configured root's rows first (every status, every store id the catalog records for the root), then lists the whole root and handles each `///` prefix as the listing passes it: the store lists in key order, so sync holds one prefix's objects and the prefix of each manifest it listed, and of the rows only each one's status, store id and table and the keys of the configured store's AVAILABLE rows. Every backend reads the rows whole before sync sees the first, so the read is the larger cost: about 22.5 KiB of peak memory per row of a table with five GSIs (1,144 MiB at 50,000 rows, measured on a SQLite debug build), which the admin guide and section 7 state. A prefix with a row is skipped before anything under it is read. A prefix with no row is verified as the reconciler's dead-owner duty verifies one (`verify_prefix`, split out of `probe_manifest`: the manifest and sidecar within their bounds, the digest, the reader's acceptance, every data file at its recorded size from the listing already held), and its manifest's ARN must name the prefix's account, table and backup id. A manifest whose ARN is in another region is adopted under this server's region's spelling of the ARN and logged, so ListBackups names this server's region and DescribeBackup, RestoreTableFromBackup and DeleteBackup find the backup under either spelling, as the service does; the manifest keeps its spelling, and the dead-owner probe and the restore accept a manifest whose ARN differs from the row's in the region alone (`BackupArn::names_same_backup`, with `BackupArn::in_region`). One of another partition is skipped, since every request refuses such an ARN. Before the insert the row is read under both spellings of the ARN (this server's region and the manifest's), and the prefix is skipped when the manifest's spelling, another region's, has a row; a row under this server's spelling is left to the insert. The prefix then gets an AVAILABLE row through the new `BackupCatalog::insert_backup_row`: unowned, in the configured store's identity, with the manifest's name, size, item total and source table, the ARN in this server's region and the id's instant as `created_at`. A prefix with a row of any status, without a manifest or sidecar, with a sidecar or data file that does not match, whose manifest names another backup, or under `exports/` or a table's `changelog/`, is reported with its reason. An insert that answers false (a CreateBackup that landed after the read, a row of another root, account or table holding the id, on Cassandra a claim's row whose list entries are missing) is re-read by its ARN and reported, with no listing of the rows of its status. The report also names the AVAILABLE rows whose manifest is gone (the rows that were AVAILABLE when the catalog was read). On PostgreSQL, MongoDB and Cassandra every sync takes the maintenance claim: plain sync without the wait for running jobs, since its insert-if-absent orders it against a CreateBackup; the claim orders it against `purge`, which removes a dead CREATING row before it deletes a prefix that can hold a complete manifest. `HeldClaim` confirms the claim by the job heartbeat's rule: each beat is bounded by one period (`MaintenanceClaim::heartbeat_within`, whose bound on SQLite starts once the beat holds the write lock), the claim reads lost once its last confirmed beat is two periods old, and a beat in flight counts only where it parks before its bound (`heartbeat_parks_before_its_bound`, SQLite). `HeldClaim::recheck` sends one beat right before each step that cannot be undone (sync's insert, `--prune-missing`'s removal, `--adopt-store`'s rewrite of the identity file, purge's removal of a dead CREATING row and the deletion of its prefix) and the command stops there when the claim is lost, so a command stopped past `stale_job_after_secs` on MongoDB or Cassandra, whose claim another command took over, stops before its next such step; a call already sent when it was stopped still lands. On SQLite every form runs under the serve lock, and plain sync runs no dead-owner resolution: the lock already keeps every server and command off the file, so the CREATING and DELETING rows and the RUNNING restore a stopped server left are skipped and its next start resumes the restore. `--prune-missing` and `--adopt-store` take the claim there and run purge's resolutions first, which fail such a restore and remove its target; every command that runs them prints what they did, the failed restores included. Every sync stops on a signal before its next prefix and exits 1. It exits 1 on a refusal (an unhealthy or unchecked store, naming the problem, with no row changed), on a listing that failed part-way (the rows inserted before it stay), and when a prefix was skipped for a store read or a catalog call that failed, since the run is incomplete, and when `--prune-missing` left a row a later run may remove; it exits 0 otherwise, skipped prefixes included. The serving registration is named `extenddb backup sync`. A sync killed while it holds the claim keeps every instance's jobs refused until `stale_job_after_secs` passes, as a killed purge does, and the admin guide says so. `--prune-missing` waits for running jobs as purge does and removes each AVAILABLE row whose manifest is gone outright, through the new `BackupCatalog::forget_available_backup_row` (the row's lock, the status check and the in-use rule of `start_backup_delete`, then the delete), and deletes nothing in the store. A row whose manifest's `head` fails is kept; that row, a row a restore is reading ("run the command again once the restore ends") and one whose removal failed exit 1, while a row whose manifest is back or that another writer moved is reported with exit 0 (`PruneOutcome::needs_another_run`). The objects a removed row's prefix still holds stay until an operator deletes the prefix by hand: purge deletes only the prefixes of rows. No tombstone is left for a deletion duty, so on host-local roots a prune on the instance that lacks a backup's bytes leaves them on the other host; the row is gone for every instance, so the backup is BackupNotFound everywhere until a sync on the host that holds the bytes adopts it again, the design's host-local acceptance line as amended. On PostgreSQL, SQLite and Cassandra `start_backup_delete` and `forget_available_backup_row` share one body with a `remove` flag, as on MongoDB, so the existing race cells cover both; Cassandra's forget settles an unknown outcome with a `LOCAL_SERIAL` read and answers removed when only the entry batch after the removal fails (logged). `--adopt-store` is the way a durable catalog takes a root whose `extenddb-store.json` names another catalog: under the claim, `store_identity::adopt` first runs the check the server runs on the root as it will be once this catalog owns it (`compare` and the manifest probe) and refuses a root this catalog cannot use with nothing written (`AdoptRefusal::Unusable`, which replaces the state of a rewrite whose check then failed); then it logs an intent line, rewrites the file's catalog id and backend with a plain put, keeping the store id and creation time, reads it back and logs the outcome under `extenddb::audit` (adopted or race lost), refuses when this catalog's rows for the root carry another store id, and syncs. The command runs it on the store's bounded calls (`adopt_configured`): each store call under the 5 s bound, the catalog's reads and the claim's beat beside it, and a store that stops answering ends the command naming the step that did not finish, the manifest probe of a root this catalog already owns included. Every sync, plain or adopting, then reads the file once more and fails naming the owner if another catalog adopted the root during the run. On a root this catalog owns it says so and syncs. For `backup sync` a defaulted SQLite root whose file names another catalog is configured and unhealthy (`Caller::SyncCommand`) rather than the warning that leaves it unconfigured, so `--adopt-store` can adopt the root a catalog re-created by `destroy` and `init` finds; the server and purge keep the warning, and the warning and `destroy` name `backup sync --adopt-store`; `backup sync` logs a root another catalog owns at WARN, since the adoption or the refusal that follows says the rest. Plain sync's refusal of a root another catalog owns says that every database file of a directory shares the default root only when the root is that default one. The previous owner of an adopted root acts on it until a later check that can read its catalog, at the earliest one reconciler interval after its cycle in progress ends, so every deletion from the store now runs a fence first: `StoreFence` re-reads `extenddb-store.json` (one `get` through the store's bounded calls, under its check bound and, on a filesystem root, in a call slot and with an in-flight mark of its own, so a deletion holding the root's slot for seconds does not hold it) right before the `delete_prefix` of `finish_backup_delete` (DeleteBackup and the reconciler's DELETING duty), the backup job's failure path, the FAILED-prefix and DELETED-prefix duties and purge, before the reconciler removes a tombstone, before a CreateBackup's row, and in the backup job right before its first put, a refusal there failing the row with no object written. DeleteBackup and purge also run it before they change the row, so a DeleteBackup refused for a root another catalog owns, a root with no file, or a file the store refuses to serve or that does not parse answers 400 and leaves the row AVAILABLE with its objects, and DescribeBackup never answers it with DELETED; an adoption between the two reads leaves the row DELETING with nothing deleted and the request answered DELETED, as for any deletion that fails after the DELETING write, finished only if this catalog owns the root again. When the file names another catalog it refuses, records `ForeignCatalog` so the store is unhealthy at once (a file naming this catalog under another store id than the store's check settled is refused and recorded as `IdMismatch`), and the request answers the unhealthy store's ValidationException; the reconciler's cycle re-tests the health after each store duty and runs none after the refusal, the dead-owner duty included. A root with no file, a file the store refuses to serve and one that does not parse are refused and recorded as the next check would record them (`FileMissingWithRows`, `Denied`, `FileUnreadable`, `Validate`), so CreateBackup answers the unhealthy-store ValidationException for them; only a read that fails in transport refuses the call and leaves the health to the next check: CreateBackup then answers InternalServerError, and DeleteBackup takes it, before its DELETING write, as a store failure: it writes DELETING, answers DELETED, and its second read refuses the deletion, logged with what the fence found (an adoption only for a file naming another catalog). A write already past the read still lands, so the refusal, the command's output and the admin guide say to stop every server of the catalog the file names before adopting, and the admin guide says to start a new in-memory server only after the old one has exited. An S3 server error the SDK's retries did not clear (a 503 SlowDown) is not a transport failure at the fence: the store reports it as it reports other refusals, so the fence records `failed validation`, as the check does, and the store stays unhealthy until a check passes. `insert_backup_row` on every backend: PostgreSQL `INSERT ... ON CONFLICT DO NOTHING` with no conflict target, on the pool; SQLite `INSERT ... ON CONFLICT DO NOTHING` under the write lock and `BEGIN IMMEDIATE`, without a job slot; MongoDB the claim's pipeline upsert filtered on an absent `owner_generation`, with `owner_generation: 0` and E11000 meaning a row exists (`row_fields` takes the status); Cassandra the `backups_by_arn` lightweight transaction `IF NOT EXISTS` and then the batch of the list rows and the status entry under the row's status (shared with `claim_backup`, sent once more on a failure), an unknown outcome or a `NotMatched` answer settled by a `LOCAL_SERIAL` re-read, which counts a stored row equal to this one as inserted and writes its entries, so the next sync completes an insert that stopped between its two writes; no row after a lost answer is sent once more, and no row after `NotMatched` answers false (a legacy in-catalog backup holds the ARN; a direct cell inserts over such a row). Cassandra has no `backup_id` uniqueness, so its insert-if-absent is per ARN spelling, and sync's read under both spellings closes the gap. `layout.rs` gains `EXPORTS_COMPONENT`, `CHANGELOG_COMPONENT`, `classify_store_key` and `parse_store_prefix`, which accepts only the 14-digit id shape and validates through `BackupArn::new`. An in-memory SQLite catalog is new at every start. The identity file gains `ephemeral_catalog` (`#[serde(default)]`), written true by a server whose SQLite database names no file; such a server's `store_identity::establish` adopts a root whose file names another catalog and carries the marker, never in `recheck`, and a root a durable catalog owns stays unhealthy. The server then syncs with no claim, on every start with a store configured: before it serves for a filesystem store, in the reconciler's cycle after a healthy check for an S3 store. Each run is bounded at 60 s; one that stops early, reaches the bound or met a store or catalog error on a prefix stays pending for the next cycle. Every run lists the root from its first key, so the startup sync completes only while listing the whole root fits in the bound; past it the sync stays pending and is retried every cycle; on both store kinds the listing is collected whole, one `ObjectMeta` per object held until the run ends, before the run sees its first key, so a listing that outlasts the bound reaches no prefix. Its store calls run as the checks run theirs (`ConfiguredStore::calls_within`), so on a filesystem store an abandoned run's walk of the root holds the root's call slot and is marked in flight until it ends, and the check and the duties after it are refused at once instead of starting a call beside it; a parked call whose caller drops it before its bound is marked the same way. The mark records which caller's call has not returned (`InFlightMark`), and the refusal names it ("an abandoned startup sync's store call has not returned"); that refusal records the store unreachable, so CreateBackup, DeleteBackup and restores are refused until the walk ends, and the first cycle after it checks the store healthy and runs the sync again. Until a run has listed the whole root and logged its line of counts, each run logs every skipped prefix (a run cut at the bound or stopped by an error names none, so the next run names them); after that a run of a sync left pending logs the store and catalog skips at WARN and the rest at DEBUG, with the line of counts. Taking the pending sync is the one gate, so of two callers one runs it. Every `extenddb backup` command refuses an in-memory database before it opens a catalog, exit 1, saying why; a location that does not parse is reported with its parse error. Wire cells, on the SQLite, PostgreSQL and MongoDB deployments (MongoDB's row edits run `mongosh` in the replica set's container, which the MongoDB backup-store job names; the cells with a second catalog skip there, since a second MongoDB catalog needs a replica set of its own; no Cassandra job runs the file): dropped rows rebuilt and Describe, List and Restore working; every skip reason above; `--prune-missing` removing the row, deleting no object and logging its audit line; an empty root, a root holding only the identity file and one with another store id refused with no row changed and nothing written; the missing newest manifest refused until it is back; on SQLite sync beside a server exit 6; `--adopt-store` from a second catalog, the first catalog's server then reporting the store owned by another catalog, and on PostgreSQL that server, still running with its last check healthy, refusing a DeleteBackup of the adopted backup, leaving its bytes and its row AVAILABLE; beside a server (PostgreSQL, MongoDB) a second maintenance command failing at once while `--prune-missing` waits, a plain sync failing at once while purge holds the claim, and `--prune-missing` on a host-local root leaving the other host's bytes, which a sync there adopts again; a SIGTERM during the prefix loop exiting 1 with the rows inserted so far kept and the claim free; and the in-memory restart serving the previous start's backup; an identity file rewritten to name a third catalog while an adopting sync is in its prefix loop ending the command with exit 1, naming that catalog; a row-less prefix whose manifest the store cannot read reported and the run exiting 1 as incomplete, its summary line saying so, then adopted once it can be read; beside a server `--adopt-store` waiting for a running job; another region's prefixes adopted under this server's region, listed in it, described by either spelling, restored and deleted by the other region's; `--adopt-store` on a root whose file names another catalog refused with nothing written while the newest backup's manifest is gone, and adopting once it is back; a plain sync whose root's file is rewritten during its prefix loop exiting 1 naming the catalog; and `backup sync` exiting 6 and 3 on the catalog refusals; on SQLite plain sync after a SIGTERM mid-restore leaving the RUNNING record, which the next start resumes to ACTIVE with every item, and a store-backed backup on the default root after `destroy` and `init` refused by plain sync and adopted by `--adopt-store` (the command's `Caller::SyncCommand`). App unit tests cover the decisions behind the last read, the transient-skip exit, the summary line's word and the prune's exit for a row left for a later run (`PruneCounts`); the wire cells pin their call sites. Engine cells cover the claim's two-period rule, its beat bound, a beat in flight on a backend whose beat parks and one whose beat does not (a paused clock, a process stopped between beats), each re-check site with a claim lost at it, and a claim that implements only `heartbeat` failing a re-check at one period through the trait's default bound. Engine cells also cover the fence: a missing identity file, an S3 denial and a filesystem read refused at mode 000 recorded and CreateBackup refused as unhealthy, the fence's own slot (a CreateBackup 1 s into a 4 s deletion answering at once), the job's fence before its first put, the catalog id kept for the process and the fenced deletion's log line; the adoption's refusals with nothing written (`Unusable`, a missing file under rows, rows of two store ids, a re-read under another store id) and its bound naming the step, the manifest probe of an owned root included; the region adoption, with the source table's ARN answered in the manifest's region by DescribeBackup, ListBackups and the restore's summary, and a restore and a resume over a manifest that spells the backup in another region; a prune stopped after its first removal; a stop inside the listing's loop making no missing-manifest report; a second startup sync run's log lines, and a first run that fails or is cut at its bound naming no prefix and the next naming them at INFO; the taken startup sync; and the walk an abandoned startup sync leaves in flight, the check refused on it naming the sync's call, and the sync run by the first cycle after the walk returns. A SQLite cell covers the maintenance claim's beat parked behind the write lock, and the four backends' insert cells now insert a row a year older than the insert, so the two clock columns are told apart from `created_at`. `devtools/backup-restore-matrix` adopts the copied prefix with `backup sync` in place of the hand-written `backup_index` row, gains an `--adopt-store` cell shape, and the nightly runs it from SQLite to PostgreSQL and back. `devtools/backup-kill-points.py`, which seeded a Cassandra kill point's row with the matrix's removed `seed_row`, adopts the copied prefix with `backup sync` beside the target's server, as the matrix does. Docs: the admin guide's `backup sync` paragraph (what it adopts and never adopts, the output, the exit codes, the refusals, the claim, the SQLite serve lock, the partial report of other roots, the case with no way out but putting the prefix back), the identity paragraph (`--adopt-store`, the default-root exception and the shared default root, the marker, the deletion fence (where it runs, what it records, its own slot) and the rule to stop the previous owner's servers first), the adoption's check before the rewrite and its bounded calls, the startup sync's log levels, the `:memory:` row (with the remedy for a root an earlier development build's in-memory server wrote; the prefixes a stopped server had not finished writing or deleting, which no command removes, the startup sync's "not adopted: not a complete backup" line that names them, and the rule to delete by hand only a prefix with no `extenddb-manifest.json` and no row in ListBackups; and what an abandoned filesystem walk costs until it ends, with a smaller root or a durable catalog for a root too large to list in 60 s) and the in-memory refusal, the claim's heartbeat rule and its re-checks, and what plain sync, `--prune-missing` and `--adopt-store` do on SQLite with what a stopped server left; the upgrade manual's rollback paragraph; the differences (an adopted backup of another region, whose source table ARN keeps the manifest's region in DescribeBackup, ListBackups and the restore's summary), usage and limits rows; the developer test guide's matrix paragraph; and design revision item 34 with the PR 10 section's note of the split, and 4(c), 4(e), 4(f), 4(g), decision 6, section 7, 5.3's `OTHER_STORE` step and the testing summary's kill-point line edited with it. The code comments that gave the Cassandra orphan-entry sweep and the MongoDB orphan-collection sweep to `backup sync` say no sweep removes them yet, and the MongoDB error line names the collections for an operator to drop. The backup job's line for a job failed on an unhealthy store says its objects stay while another catalog has adopted the root. Not here: `--move-store`, `--forget-store`, `--forget-backup`, `list_backup_rows_by_root`, `rewrite_store` and the full report of the rows of other roots (#820); `migrate-legacy` (#821). Co-authored-by: Lee Signed-off-by: Scott Robinson --- .github/workflows/integration-mongodb.yml | 2 + .github/workflows/matrix-nightly.yml | 29 +- crates/app/src/cmd_backup.rs | 921 +++- crates/app/src/cmd_destroy.rs | 7 +- crates/app/src/lib.rs | 3 +- crates/config/src/lib.rs | 10 + crates/core/src/backup_format/layout.rs | 424 ++ crates/core/src/backup_format/mod.rs | 12 +- crates/engine/src/backup.rs | 159 +- crates/engine/src/backup/job.rs | 135 +- crates/engine/src/backup/job/failure.rs | 22 +- crates/engine/src/backup/job/writer.rs | 5 +- crates/engine/src/backup/maintenance.rs | 557 ++- crates/engine/src/backup/reconciler.rs | 360 +- crates/engine/src/backup/restore.rs | 26 +- crates/engine/src/backup/runtime.rs | 639 ++- crates/engine/src/backup/store_identity.rs | 475 +- crates/engine/src/backup/sync.rs | 1110 +++++ crates/engine/src/backup/tests.rs | 4369 ++++++++++++++++- crates/engine/src/lib.rs | 4 +- crates/server/src/backup_store.rs | 240 +- crates/server/src/serve.rs | 16 +- .../storage-cassandra/src/backup_catalog.rs | 607 ++- crates/storage-cassandra/src/test_hooks.rs | 159 +- .../tests/direct/backup_engine.rs | 784 +++ crates/storage-mongodb/src/backup_catalog.rs | 117 +- crates/storage-mongodb/src/backup_engine.rs | 4 +- .../storage-mongodb/src/backup_live_tests.rs | 219 + crates/storage-postgres/src/backup_catalog.rs | 145 +- .../storage-postgres/tests/backup_restore.rs | 306 +- crates/storage-sqlite/src/backup.rs | 233 + crates/storage-sqlite/src/backup_catalog.rs | 209 +- crates/storage-sqlite/src/config.rs | 4 + crates/storage/src/backup_catalog.rs | 103 +- crates/storage/src/config.rs | 13 + devtools/backup-kill-points.py | 43 +- devtools/backup-restore-matrix | 260 +- docs/design/15-backup-restore.md | 32 +- docs/differences-from-dynamodb.md | 2 +- docs/dynamodb-limits.md | 2 +- docs/manuals/03-usage-guide.md | 2 +- docs/manuals/05-admin-guide.md | 28 +- docs/manuals/06-developer-test-guide.md | 36 +- docs/manuals/07-upgrade-manual.md | 2 +- requirements-matrix.txt | 2 +- tests/test_backup_job.py | 1293 ++++- tests/test_backup_store_config.py | 165 +- 47 files changed, 13388 insertions(+), 907 deletions(-) create mode 100644 crates/engine/src/backup/sync.rs diff --git a/.github/workflows/integration-mongodb.yml b/.github/workflows/integration-mongodb.yml index 83708ef5f..6b67d6cfd 100644 --- a/.github/workflows/integration-mongodb.yml +++ b/.github/workflows/integration-mongodb.yml @@ -230,6 +230,8 @@ jobs: env: EXTENDDB_BINARY: ./target/release/extenddb EXTENDDB_TEST_MONGODB_URI: mongodb://127.0.0.1:27023/?replicaSet=rs0&directConnection=true + # The sync cells edit backup_index documents with mongosh in it. + EXTENDDB_TEST_MONGODB_CONTAINER: extenddb-runtests-mongo-27023 run: python3 -m pytest tests/test_backup_job.py -v -rP -p no:cacheprovider - name: Stop the job suite's replica set if: always() diff --git a/.github/workflows/matrix-nightly.yml b/.github/workflows/matrix-nightly.yml index b7951da97..c28edcc50 100644 --- a/.github/workflows/matrix-nightly.yml +++ b/.github/workflows/matrix-nightly.yml @@ -5,10 +5,12 @@ name: Cross-backend restore matrix (nightly) # The cross-backend restore matrix (design PR 9b): a backup taken on one # backend restores, through the store, into another. Each cell starts a source # and a target server from the per-backend binaries, copies the source's backup -# prefix under a root the target owns, seeds the target's `backup_index` row as -# `backup sync --adopt-store` will (PR 10), restores and compares. The cells are -# heavy (two servers each, four backends, PostgreSQL + a MongoDB replica set + -# Cassandra) so they run nightly rather than on every pull request. +# prefix under a root the target owns, adopts it into the target's catalog with +# `extenddb backup sync`, restores and compares; the --adopt-store cells point +# the target at the source's root instead and adopt the root with `backup sync +# --adopt-store`. The cells are heavy (two servers each, four backends, +# PostgreSQL + a MongoDB replica set + Cassandra) so they run nightly rather +# than on every pull request. on: schedule: @@ -71,12 +73,12 @@ jobs: python-version: "3.11" - name: Install Python dependencies # requirements-matrix.txt adds the MongoDB and Cassandra drivers the - # script's seed and teardown use on those backends. + # script's setup and teardown use on those backends. run: pip install -r requirements.txt -r requirements-matrix.txt - name: Install the PostgreSQL client - # The matrix script seeds and tears down every PostgreSQL source and - # target through the host's psql; the runner image is not assumed to + # The matrix script tears down every PostgreSQL source and target + # through the host's psql; the runner image is not assumed to # carry one. run: | sudo apt-get update @@ -171,6 +173,14 @@ jobs: --work "matrix-logs/$s-$t-vector" || rc=1 done done + # The adoption of a root another catalog owns, both ways between + # SQLite and PostgreSQL. + for pair in "sqlite postgres" "postgres sqlite"; do + set -- $pair + echo "=== $1 -> $2 --adopt-store ===" + python3 devtools/backup-restore-matrix --source "$1" --target "$2" --adopt-store \ + --work "matrix-logs/$1-$2-adopt" || rc=1 + done exit $rc - name: Upload the matrix logs on failure @@ -435,8 +445,9 @@ jobs: # Cassandra has no SnapshotEngine, so its CreateBackup is in-catalog # and no backup taken on it restores through the store-backed job: the # script takes each point's backup on a SQLite source server - # (`--source-bin`, a plain SQLite build) and adopts it into the - # Cassandra target's catalog, as the matrix does. One target + # (`--source-bin`, a plain SQLite build), copies its prefix under + # the Cassandra target's root and adopts it with `backup sync` + # beside the target's server, as the matrix does. One target # directory, two sequential builds. run: | cargo build --release -p extenddb --no-default-features --features sqlite diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs index 2ddc308b2..99a8f1551 100644 --- a/crates/app/src/cmd_backup.rs +++ b/crates/app/src/cmd_backup.rs @@ -87,28 +87,26 @@ //! backends purge runs beside live servers; the maintenance claim is the //! exclusion. //! -//! Every command opens the catalog through the backend's factory, under its -//! own name (`extenddb backup list`, `extenddb backup purge`, `extenddb -//! backup remove-target`), and none runs the startup recovery a server runs -//! (`ServerComponentsOptions::startup_recovery` off): that work belongs to -//! the servers the command runs beside, and a command that ran it could exit -//! with it half done. On PostgreSQL, MongoDB and Cassandra the command holds -//! a serving registration for its run, as a server does, so a migration -//! waits for it, and the registration carries the command's name, which the -//! migration's wait prints (PostgreSQL's session `application_name`, -//! MongoDB's `extenddb_servers` document, Cassandra's `extenddb_servers` -//! row). The command releases the registration when it ends, whatever the -//! outcome, so a migration does not wait for a row or document to expire. A -//! catalog the -//! factory refuses (a migration running, the catalog at another migration -//! than this binary's) ends the command with the message and the exit code -//! `serve` gives for it: 6 while a migration runs, 3 for a catalog behind -//! this binary. A registration lost during the command does not stop it: -//! no supervisor runs in a command, so a command that outlasts a lost -//! registration runs beside a migration that started after the loss. The -//! signal listener is up before the open in every subcommand, so a Ctrl-C -//! during the open or the identity read ends the command once the open -//! returns, with the registration released. +//! Every command opens the catalog through the backend's factory, under its own +//! name (`extenddb backup list`, `extenddb backup purge`, `extenddb backup +//! remove-target`, `extenddb backup sync`), and none runs the startup recovery +//! a server runs (`ServerComponentsOptions::startup_recovery` off): that work +//! belongs to the servers the command runs beside, and a command that ran it +//! could exit with it half done. On PostgreSQL, MongoDB and Cassandra the +//! command holds a serving registration for its run, as a server does, so a +//! migration waits for it, and the registration carries the command's name, +//! which the migration's wait prints (PostgreSQL's session `application_name`, +//! MongoDB's `extenddb_servers` document, Cassandra's `extenddb_servers` row). +//! The command releases the registration when it ends, whatever the outcome, so +//! a migration does not wait for a row or document to expire. A catalog the +//! factory refuses (a migration running, the catalog at another migration than +//! this binary's) ends the command with the message and the exit code `serve` +//! gives for it: 6 while a migration runs, 3 for a catalog behind this binary. +//! A registration lost during the command does not stop it: no supervisor runs +//! in a command, so a command that outlasts a lost registration runs beside a +//! migration that started after the loss. The signal listener is up before the +//! open in every subcommand, so a Ctrl-C during the open or the identity read +//! ends the command once the open returns, with the registration released. //! //! `extenddb destroy` removes the databases and leaves the store untouched: //! the store is the copy that survives destroy, and this command is the one @@ -131,6 +129,60 @@ //! of what it removed (`extenddb_engine::maintenance::remove_target` states //! each step), and logs the removal under `extenddb::audit` with the //! account, the table and the backup's ARN. +//! +//! `backup sync` rebuilds the catalog from the configured store: it gives an +//! AVAILABLE row to every backup prefix of the store that has a verified +//! manifest and no row, and reports the prefixes it skips with the reason +//! and the AVAILABLE rows whose manifest is gone (`extenddb_engine::sync` +//! states what it adopts and never adopts). It opens the catalog as `serve` +//! does (on SQLite under the serve lock) and needs the store healthy: an +//! unhealthy store refuses it, naming the problem, with no row changed. On +//! PostgreSQL, MongoDB and Cassandra every sync takes the maintenance claim +//! and heartbeats it, so no purge removes a row and deletes its bytes +//! between this sync's read and its insert (the claim re-checked with a +//! beat right before each insert, `--prune-missing`'s removal and +//! `--adopt-store`'s rewrite: `extenddb_engine::sync`'s module doc gives +//! the bound that remains); plain sync waits for no job (its +//! insert-if-absent orders it against a CreateBackup), and while it holds +//! the claim the jobs refuse, as during purge. `--prune-missing` and +//! `--adopt-store` also wait for running jobs as purge does, on SQLite +//! after the dead-owner resolutions purge runs, which fail a store-backed +//! restore a stopped server left RUNNING and remove its target. Plain sync +//! on SQLite runs no dead-owner resolution: the serve lock already keeps +//! every server and every other command off the file, so nothing changes +//! between its row read and its insert, and the rows a stopped server left +//! (a CREATING or DELETING backup, a RUNNING restore) are rows the sync +//! skips and leaves for the next server start, which resolves the backups +//! and resumes the restore. A first `SIGINT` or `SIGTERM` stops any +//! sync before its next prefix (or the wait at once), releases the claim +//! and exits 1. `--prune-missing` removes each AVAILABLE row whose manifest +//! is gone outright and deletes no object, and reports a row a restore is +//! reading or another writer moved. `--adopt-store` is the way a durable +//! catalog takes a root whose identity file names another catalog (a server +//! on an in-memory database adopts, at startup, a root another in-memory +//! catalog wrote): it rewrites the file's owner +//! (`extenddb_engine::store_identity::adopt`), logged under +//! `extenddb::audit`, syncs, then reads the file once more and fails naming +//! the owner if another catalog adopted the root meanwhile; on a root this +//! catalog already owns it says so and syncs. Every server of the catalog +//! the file names is to be stopped before the adoption: the previous owner +//! acts on the root until a later check that can read its catalog finds +//! the store owned by another catalog, except where its deletion fence +//! stops it first (a re-read of the file before each deletion, before +//! DeleteBackup's and purge's row change and before a CreateBackup's first +//! put, which leaves a gap of one read). The refusal and the output say so. Sync +//! exits 0 once it ran, skipped +//! prefixes included, and 1 on a refusal, a stop, a listing that failed, a +//! row `--prune-missing` left for a later run (a store or catalog call +//! failed, or a restore is reading the backup), or a prefix skipped for a +//! store or catalog error (the run is incomplete; a second run adopts it). +//! A row `--prune-missing` leaves because its manifest is back or another +//! writer moved it is reported, and the exit stays 0. +//! +//! Every `backup` command refuses an in-memory SQLite database (a +//! `[storage.sqlite] path` naming no file) before it opens the catalog: it +//! would open a new, empty catalog that ends with the command. A server on +//! such a database syncs with its store itself at every start. use std::sync::Arc; use std::time::{Duration, Instant}; @@ -190,6 +242,24 @@ enum BackupCommand { /// The table's name table: String, }, + /// Give a catalog row to every backup of the configured backup store that has none: each + /// prefix with a verified manifest (its sidecar matching, every data file present at its + /// size, its ARN naming the prefix's account, table and backup id) gets an AVAILABLE row. + /// Reports the prefixes it skips and the AVAILABLE rows whose manifest is gone. Refuses on + /// an unhealthy store. Beside servers (PostgreSQL, MongoDB, Cassandra) it takes the + /// maintenance claim; on SQLite it needs the server stopped. + Sync { + /// Make this catalog the owner of a store root whose extenddb-store.json names another + /// catalog, then sync. Stop every server of the catalog the file names first. Takes the + /// maintenance claim first and waits for running jobs. + #[arg(long)] + adopt_store: bool, + /// Also remove the AVAILABLE rows whose manifest is gone from the store. Each row is + /// removed outright and no object is deleted (the bytes may be on another host's root + /// of the same path). Takes the maintenance claim first and waits for running jobs. + #[arg(long)] + prune_missing: bool, + }, } /// How often the waiting purge re-reads the rows. @@ -202,6 +272,7 @@ const WAIT_POLL: Duration = Duration::from_secs(2); const LIST: &str = "extenddb backup list"; const PURGE: &str = "extenddb backup purge"; const REMOVE_TARGET: &str = "extenddb backup remove-target"; +const SYNC: &str = "extenddb backup sync"; pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> { if !std::path::Path::new(&args.config).exists() { @@ -211,6 +282,23 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> ); } let app_config = config::load(&args.config)?; + // An in-memory database is a new catalog for each process that opens + // it: a command would read an empty catalog, and a writing one would + // record its work in a catalog that ends with it (an adopted root + // owned by a catalog nothing can open again). A location that does not + // parse is not in memory: its own error is printed, as `serve` does. + if app_config.storage.backend == "sqlite" { + match app_config.storage.database_location() { + Ok(Some(_)) => {} + Ok(None) => anyhow::bail!( + "the SQLite database is in memory ([storage.sqlite] path names no file), so \ + this command would open a new, empty catalog that ends with the command; a \ + server on an in-memory database syncs its catalog with the configured backup \ + store itself at every start" + ), + Err(e) => anyhow::bail!("{e}"), + } + } // The signal listener is up before the catalog is opened, in every // subcommand: a signal during the open or the identity read is acted on // once the open returns, with the serving registration released, where @@ -242,6 +330,7 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> BackupCommand::List { .. } => (LIST, Mode::Read), BackupCommand::Purge { confirm: false } => (PURGE, Mode::Read), BackupCommand::Purge { confirm: true } => (PURGE, Mode::Write), + BackupCommand::Sync { .. } => (SYNC, Mode::Sync), BackupCommand::RemoveTarget { .. } => unreachable!("handled above"), }; let opening = open(&app_config, &args.config, release, command, mode).await?; @@ -289,6 +378,18 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> anyhow::bail!("{reason}") } (BackupCommand::RemoveTarget { .. }, _) => unreachable!("handled above"), + ( + BackupCommand::Sync { + adopt_store, + prune_missing, + }, + Opening::Store(opened), + ) => { + let done = sync(&opened, &app_config, adopt_store, prune_missing, interrupt).await; + opened.close().await; + done + } + (BackupCommand::Sync { .. }, Opening::NoStore(reason)) => anyhow::bail!("{reason}"), } } @@ -312,6 +413,10 @@ enum Mode { /// and which the command must not exit with half done; the store's /// check may write the identity file of an empty root. Write, + /// As `Write`, for `backup sync`: a defaulted root another catalog owns + /// is configured, unhealthy, rather than left unconfigured with the + /// warning (`extenddb_server::backup_store::Caller::SyncCommand`). + Sync, /// As `serve` opens it, for a command that needs no store /// (`remove-target`): the store is described as `Read` leaves it, so /// neither the default root nor an identity file is created. @@ -401,6 +506,7 @@ async fn open_catalog( extenddb_server::backup_store::Caller::ReadOnlyCommand } Mode::Write => extenddb_server::backup_store::Caller::Command, + Mode::Sync => extenddb_server::backup_store::Caller::SyncCommand, }, ) .await; @@ -488,7 +594,7 @@ async fn open( Err(problem) => Identity::Problem(problem), } } - (Mode::Write, StoreHealth::Checking) => { + (Mode::Write | Mode::Sync, StoreHealth::Checking) => { let outcome = extenddb_engine::check_configured_store(store, &**storage, backend).await; store.record_check(outcome.clone()); match outcome { @@ -496,13 +602,13 @@ async fn open( Err(problem) => Identity::Problem(problem), } } - (Mode::Write, StoreHealth::Healthy) => match store.identity() { + (Mode::Write | Mode::Sync, StoreHealth::Healthy) => match store.identity() { Some(identity) => Identity::Known(identity), None => Identity::Problem(StoreProblem::Catalog( "the store's id is unknown".to_owned(), )), }, - (Mode::Write, StoreHealth::Unhealthy(problem)) => Identity::Problem(problem), + (Mode::Write | Mode::Sync, StoreHealth::Unhealthy(problem)) => Identity::Problem(problem), }; Ok(Opening::Store(Opened { catalog, identity })) } @@ -718,6 +824,7 @@ async fn purge( ), }; let store = &*configured.store; + let fence = configured.fence(); print_plan(opened).await?; let stale_after = Duration::from_secs(backup.stale_job_after_secs); @@ -820,6 +927,7 @@ async fn purge( let outcome = purge_rows( storage, store, + &fence, &rows, owner_window, &claim, @@ -938,6 +1046,21 @@ async fn resolve_dead_owners(opened: &Opened) { for table in &report.failed_targets_removed { println!("removed the restore target {table}: its restore had failed"); } + for table in &report.manifest_restores_failed_and_removed { + println!( + "failed the restore into {table} and removed the table: its owner is dead, \ + and a maintenance command resumes no restore" + ); + } + for (table, reason) in &report.manifest_restores_failed { + println!("failed the restore into {table} ({reason}): its owner is dead"); + } + for table in &report.targets_left_for_operator { + println!( + "left the restore target {table}: its restore failed with target_not_empty; \ + remove it with backup remove-target" + ); + } for table in &report.targets_left_to_owner { println!( "left the table {table}: it is no longer the restore target its record names" @@ -1034,6 +1157,567 @@ async fn remove_restore_target( Ok(()) } +/// `backup sync [--adopt-store] [--prune-missing]` (module doc). +#[allow(clippy::too_many_lines)] +async fn sync( + opened: &Opened, + app_config: &config::AppConfig, + adopt_store: bool, + prune_missing: bool, + mut interrupt: Interrupt, +) -> anyhow::Result<()> { + let storage = &*opened.catalog.storage; + let configured = opened.store(); + println!("=== extenddb backup sync ==="); + // The identity sync writes rows under, or the root to adopt. + let mut adopting = false; + let known = match &opened.identity { + Identity::Known(identity) => { + println!( + "Store: {} {} (store id {})", + identity.store_kind, identity.store_root, identity.store_id + ); + if adopt_store { + println!( + "The root's {} already names this catalog: there is nothing to adopt.", + extenddb_engine::STORE_IDENTITY_KEY + ); + } + Some(identity.clone()) + } + Identity::Problem(problem @ StoreProblem::ForeignCatalog { .. }) => { + println!( + "Store: {} {} (owned by another catalog: {problem})", + configured.kind, configured.root + ); + if !adopt_store { + anyhow::bail!( + "the {} backup store {} is owned by another catalog: {problem}. Sync changes \ + no row of a root another catalog owns. Stop every server of the catalog the \ + file names, then run with --adopt-store to make this catalog its owner: a \ + running server of that catalog acts on the root until a later check that \ + can read its catalog finds it owned by another catalog (its deletion fence refuses a deletion once the \ + file names another catalog, but a write already past the fence's read \ + lands){}", + configured.kind, + configured.root, + if configured.is_defaulted() { + ". The default root beside a SQLite database file is shared by every \ + database file in its directory, and the adoption takes it from \ + whichever catalog wrote the file" + } else { + "" + } + ); + } + adopting = true; + None + } + Identity::Problem(problem) => anyhow::bail!( + "the {} backup store {} is unhealthy: {problem}; sync changes no row while the store \ + is unhealthy", + configured.kind, + configured.root + ), + Identity::Empty => anyhow::bail!( + "the {} backup store {} has no {}: its check did not run", + configured.kind, + configured.root, + extenddb_engine::STORE_IDENTITY_KEY + ), + }; + + // On SQLite the command holds the serve lock, which already excludes + // every server and every other command, so plain sync takes no claim + // and resolves nothing a stopped server left: its rows are skipped, and + // the next server start resolves them (and resumes a store-backed + // restore, which the dead-owner resolutions would fail). Elsewhere every sync takes the + // maintenance claim, so no purge removes a row between this sync's row + // read and its insert and then deletes the bytes it adopted; plain sync + // waits for no job, since its insert-if-absent orders it against a + // CreateBackup. The options wait for running jobs as purge does. + let sqlite = opened.catalog.backend == "sqlite"; + let waits = adopt_store || prune_missing; + let mut claim = None; + if !sqlite || waits { + let backup = &app_config.backup; + let stale_after = Duration::from_secs(backup.stale_job_after_secs); + let holder = format!( + "extenddb backup sync on {} (pid {})", + extenddb_server::backup_store::hostname(), + std::process::id() + ); + let taken = storage + .maintenance_claim(&holder, stale_after) + .await + .map_err(|e| match e { + StorageError::AlreadyExists(m) => anyhow::anyhow!( + "{m}; one maintenance command runs at a time, and a claim whose holder died \ + expires after stale_job_after_secs ({} s)", + backup.stale_job_after_secs + ), + other => anyhow::anyhow!("cannot take the maintenance claim: {other}"), + })?; + println!("Took the maintenance claim as {holder:?}."); + let held = HeldClaim::start(taken, JOB_HEARTBEAT_INTERVAL); + if waits + && let Err(refusal) = wait_for_jobs( + opened, + configured, + known.as_ref(), + &held, + &mut interrupt, + stale_after, + ) + .await + { + release(held).await; + return Err(refusal); + } + claim = Some(held); + } + + let done = sync_held( + opened, + app_config, + known, + adopting, + prune_missing, + claim.as_ref(), + &interrupt, + ) + .await; + if let Some(claim) = claim { + release(claim).await; + } + done +} + +/// The wait `--adopt-store` and `--prune-missing` run under the claim, as +/// purge's: on SQLite after the dead-owner resolutions with a zero window. +/// The refusal is the command's error, the claim still held. +async fn wait_for_jobs( + opened: &Opened, + configured: &ConfiguredStore, + known: Option<&StoreIdentity>, + claim: &HeldClaim, + interrupt: &mut Interrupt, + stale_after: Duration, +) -> anyhow::Result<()> { + let storage = &*opened.catalog.storage; + // The identity the wait reads rows of: the file's id, before the + // adoption as after it. + let wait_identity = match known { + Some(identity) => identity.clone(), + None => { + let file = tokio::time::timeout( + configured.check_timeout(), + extenddb_engine::store_identity::read_identity_file(&*configured.store), + ) + .await + .map_err(|_| { + anyhow::anyhow!( + "the store's {} could not be read within {} s", + extenddb_engine::STORE_IDENTITY_KEY, + configured.check_timeout().as_secs() + ) + })? + .map_err(|e| anyhow::anyhow!("{e}"))? + .ok_or_else(|| { + anyhow::anyhow!( + "the root's {} is gone; run the command again", + extenddb_engine::STORE_IDENTITY_KEY + ) + })?; + StoreIdentity { + store_kind: configured.kind, + store_root: configured.root.clone(), + store_id: file.store_id, + } + } + }; + let owner_window = if opened.catalog.backend == "sqlite" { + resolve_dead_owners(opened).await; + Duration::ZERO + } else { + stale_after + }; + let started = Instant::now(); + let outcome = wait_for_quiet( + storage, + &wait_identity, + claim, + Wait { + owner_window, + limit: stale_after * 2, + poll: WAIT_POLL, + }, + interrupt, + |remaining| { + println!( + "Waiting for running jobs to finish: {}", + remaining.join("; ") + ) + }, + ) + .await; + match outcome { + WaitOutcome::Quiet => Ok(()), + WaitOutcome::Busy(remaining) => Err(anyhow::anyhow!( + "jobs were still running after waiting {} s (2 * stale_job_after_secs): {}. The \ + maintenance claim is released; retry once they finish", + started.elapsed().as_secs(), + remaining.join("; ") + )), + WaitOutcome::Lost(reason) => Err(anyhow::anyhow!( + "stopped before any change: {reason}; retry once the claim is free" + )), + WaitOutcome::CatalogError(e) => Err(anyhow::anyhow!( + "stopped before any change: {e}. The maintenance claim is released; retry once the \ + catalog answers" + )), + WaitOutcome::Interrupted(signal) => Err(anyhow::anyhow!( + "stopped before any change: {signal} received while waiting. The maintenance claim \ + is released" + )), + } +} + +/// The adoption, the sync and the prune, with the claim held (on +/// PostgreSQL, MongoDB and Cassandra) or under the serve lock (SQLite); the +/// caller releases the claim. +#[allow(clippy::too_many_lines)] +async fn sync_held( + opened: &Opened, + app_config: &config::AppConfig, + known: Option, + adopting: bool, + prune_missing: bool, + claim: Option<&HeldClaim>, + interrupt: &Interrupt, +) -> anyhow::Result<()> { + let storage = &*opened.catalog.storage; + let configured = opened.store(); + let store = &*configured.store; + let catalog_id = storage + .catalog_id() + .await + .map_err(|e| anyhow::anyhow!("the catalog could not be read: {e}"))?; + let identity = match known { + Some(identity) => identity, + None => { + debug_assert!(adopting); + // The store's calls under the check's bound, as the server's + // check runs them: a store that stops answering ends the + // adoption at the call that did not finish, named in the + // refusal, and the catalog's reads and the claim's beat run + // beside the bound. + let adopted = extenddb_engine::store_identity::adopt_configured( + configured, + storage, + &opened.catalog.backend, + claim, + ) + .await + .map_err(|refusal| { + anyhow::anyhow!("--adopt-store did not adopt the root: {refusal}") + })?; + let identity = match adopted { + extenddb_engine::store_identity::Adoption::Adopted { + identity, + previous_catalog_id, + } => { + println!( + "Adopted the root: {} named catalog {previous_catalog_id} and now names \ + this catalog (store id {}). A server of catalog {previous_catalog_id} \ + that still runs acts on the root until a later check that can read its \ + catalog finds it owned by another catalog; its deletion fence refuses a deletion and a \ + CreateBackup once it re-reads the file. Stop every such server before \ + an adoption.", + extenddb_engine::STORE_IDENTITY_KEY, + identity.store_id + ); + identity + } + extenddb_engine::store_identity::Adoption::AlreadyOwned(identity) => { + println!( + "The root's {} names this catalog now: there is nothing to adopt.", + extenddb_engine::STORE_IDENTITY_KEY + ); + identity + } + }; + configured.record_check(Ok(identity.clone())); + identity + } + }; + if let Some(reason) = claim.and_then(HeldClaim::lost) { + anyhow::bail!("stopped before the sync: {reason}; retry once the claim is free"); + } + + let stop = || { + claim.and_then(HeldClaim::lost).or_else(|| { + interrupt + .signalled() + .map(|signal| format!("{signal} received")) + }) + }; + let report = extenddb_engine::sync::sync( + storage, + extenddb_engine::sync::SyncTarget { + store, + identity: &identity, + metrics: None, + region: Some(&app_config.server.region), + stop: Some(&stop), + claim, + }, + ) + .await; + let report = match report { + Ok(report) => report, + Err(extenddb_engine::sync::SyncError::Listing { error, partial }) => { + print_sync_report(&partial, prune_missing, false); + anyhow::bail!( + "the store's listing failed: {error}. {} backup(s) listed before the failure \ + were adopted, and no AVAILABLE row was examined for a missing manifest; run the \ + command again", + partial.adopted.len() + ); + } + Err(e) => anyhow::bail!("{e}"), + }; + print_sync_report(&report, prune_missing, report.stopped.is_none()); + if let Some(reason) = &report.stopped { + anyhow::bail!( + "stopped: {reason}. {} backup(s) were adopted before the stop, and no AVAILABLE row \ + was examined for a missing manifest; run the command again for the rest", + report.adopted.len() + ); + } + + let mut counts = PruneCounts::default(); + let mut stopped = None; + if prune_missing && !report.missing_manifests.is_empty() { + let claim = claim.ok_or_else(|| { + anyhow::anyhow!("--prune-missing runs only under the maintenance claim") + })?; + let (outcomes, stop_reason) = extenddb_engine::sync::prune_missing( + storage, + store, + &configured.root, + &report.missing_manifests, + claim, + &stop, + ) + .await; + println!(); + println!("--- --prune-missing (rows removed; no object is deleted):"); + for (arn, outcome) in &outcomes { + println!(" {arn}: {outcome}"); + } + counts = PruneCounts::of(&outcomes); + stopped = stop_reason; + } + + // The file once more: every sync started from a file naming this + // catalog (the command's check read it, or the adoption wrote it), and + // an adopter whose put landed during the sync is detected here, the + // rows this sync inserted being then rows of a root this catalog does + // not own. + let via = configured.calls("identity file re-read"); + let file = extenddb_engine::store_identity::read_identity_file(&via).await; + let file = (!via.timed_out()).then_some(file); + owner_after_sync(file, &catalog_id, configured.check_timeout())?; + + println!(); + println!( + "=== extenddb backup sync {}: {} backup(s) adopted, {} prefix(es) skipped, {} \ + AVAILABLE row(s) whose manifest is gone{} ===", + run_word(&report, stopped.is_some()), + report.adopted.len(), + report.skipped.len(), + report.missing_manifests.len(), + match (prune_missing, &stopped) { + (false, _) => String::new(), + (true, None) => format!(" ({} removed)", counts.removed), + (true, Some(_)) => format!(" ({} removed before the stop)", counts.removed), + }, + ); + if let Some(reason) = stopped { + anyhow::bail!("--prune-missing stopped before the remaining rows: {reason}"); + } + counts.left_for_a_later_run()?; + complete_run(&report) +} + +/// The word of the summary line: `stopped` when the prune stopped before +/// its remaining rows, `incomplete` when a prefix was skipped for a store or +/// catalog error (the run then exits 1, [`complete_run`]), `complete` +/// otherwise. +fn run_word(report: &extenddb_engine::sync::SyncReport, stopped: bool) -> &'static str { + if stopped { + "stopped" + } else if report.transient_skips() > 0 { + "incomplete" + } else { + "complete" + } +} + +/// What `--prune-missing` did, counted over its outcomes. +#[derive(Debug, Default, PartialEq, Eq)] +struct PruneCounts { + /// Rows removed. + removed: usize, + /// Rows left for a later run (`PruneOutcome::needs_another_run`). + left: usize, + /// Of those, rows a restore is reading. + in_use: usize, +} + +impl PruneCounts { + fn of(outcomes: &[(String, extenddb_engine::sync::PruneOutcome)]) -> Self { + let mut counts = Self::default(); + for (_, outcome) in outcomes { + if *outcome == extenddb_engine::sync::PruneOutcome::Pruned { + counts.removed += 1; + } else if outcome.needs_another_run() { + counts.left += 1; + if matches!(outcome, extenddb_engine::sync::PruneOutcome::InUse(_)) { + counts.in_use += 1; + } + } + } + counts + } + + /// The exit of a run that left a row for a later run: exit 1 with the + /// error naming the count and, when a restore is reading one of the + /// backups, when to run the command again; `Ok` when no row was left. + fn left_for_a_later_run(&self) -> anyhow::Result<()> { + if self.left == 0 { + return Ok(()); + } + anyhow::bail!( + "{} row(s) whose manifest is gone were left for a later run (see above){}", + self.left, + if self.in_use > 0 { + format!( + "; {} of them are being restored from: run the command again once the \ + restore ends", + self.in_use + ) + } else { + "; run the command again once the store and the catalog answer".to_owned() + } + ) + } +} + +/// The sync's last read of `extenddb-store.json` (`file`, `None` when the +/// read did not finish within `bound`): `Ok` when it still names this +/// catalog (`catalog_id`), and otherwise the error naming what it found, so +/// an adopter whose put landed during the sync is reported, whether the +/// sync adopted the root or started from a file that named this catalog. +fn owner_after_sync( + file: Option< + Result< + Option, + extenddb_engine::StoreProblem, + >, + >, + catalog_id: &str, + bound: Duration, +) -> anyhow::Result<()> { + match file { + Some(Ok(Some(file))) if file.catalog_id == catalog_id => Ok(()), + Some(Ok(Some(file))) => anyhow::bail!( + "after the sync, {} names catalog {} (store id {}): another catalog adopted the root \ + during the sync, and the rows this sync inserted are this catalog's rows of a root \ + it does not own", + extenddb_engine::STORE_IDENTITY_KEY, + file.catalog_id, + file.store_id + ), + Some(Ok(None)) => anyhow::bail!( + "after the sync, the root has no {}", + extenddb_engine::STORE_IDENTITY_KEY + ), + Some(Err(problem)) => anyhow::bail!( + "after the sync, {} could not be read: {problem}", + extenddb_engine::STORE_IDENTITY_KEY + ), + None => anyhow::bail!( + "after the sync, {} could not be read within {} s", + extenddb_engine::STORE_IDENTITY_KEY, + bound.as_secs() + ), + } +} + +/// A sync that ran to its end is complete only when no prefix was skipped +/// for a store or catalog error; otherwise the command exits 1, since a +/// later run adopts those prefixes. +fn complete_run(report: &extenddb_engine::sync::SyncReport) -> anyhow::Result<()> { + let transient = report.transient_skips(); + if transient > 0 { + anyhow::bail!( + "the run is incomplete: {transient} prefix(es) were skipped for a store or catalog \ + error (see above); run the command again once the store and the catalog answer" + ); + } + Ok(()) +} + +/// The report of a sync: what it adopted, what it skipped and why, the +/// AVAILABLE rows whose manifest is gone (`examined` is false when the run +/// ended before the end of the listing, which leaves them unexamined). +fn print_sync_report( + report: &extenddb_engine::sync::SyncReport, + prune_missing: bool, + examined: bool, +) { + println!(); + println!("--- Adopted (an AVAILABLE row inserted for each):"); + if report.adopted.is_empty() { + println!(" (none)"); + } + for adopted in &report.adopted { + match &adopted.foreign { + Some(note) => println!(" {} ({note})", adopted.backup_arn), + None => println!(" {}", adopted.backup_arn), + } + } + println!(); + println!("--- Skipped (not adopted):"); + if report.skipped.is_empty() { + println!(" (none)"); + } + for skipped in &report.skipped { + println!(" {}: {}", skipped.prefix, skipped.reason); + } + println!(); + println!( + "--- AVAILABLE rows whose manifest is not in the store (the rows were AVAILABLE when the \ + catalog was read):" + ); + if !examined { + println!(" (not examined)"); + } else if report.missing_manifests.is_empty() { + println!(" (none)"); + } + for key in &report.missing_manifests { + println!(" {} (account {})", key.backup_arn, key.account_id); + } + if !report.missing_manifests.is_empty() && !prune_missing { + println!( + " (left AVAILABLE; run with --prune-missing to remove the rows, which deletes no \ + object)" + ); + } +} + #[cfg(test)] mod tests { use std::sync::Arc; @@ -1043,7 +1727,192 @@ mod tests { use async_trait::async_trait; use extenddb_storage::hooks::{Reregistration, ServingRegistrar}; - use super::{release_registration, release_registration_within}; + use super::{ + PruneCounts, complete_run, owner_after_sync, release_registration, + release_registration_within, run_word, + }; + + /// The identity file as a store holds it, naming `catalog_id`. + fn file(catalog_id: &str) -> extenddb_engine::StoreIdentityFile { + extenddb_engine::StoreIdentityFile { + format_version: 1, + store_id: "store-1".to_owned(), + catalog_id: catalog_id.to_owned(), + backend: "postgres".to_owned(), + created_at_ms: 0, + ephemeral_catalog: false, + } + } + + /// The adopting sync's last read: the file still naming this catalog + /// passes; another catalog (an adopter whose put landed during the + /// sync), no file, an unreadable file and a read past the bound each + /// fail, the first naming the new owner. + #[test] + fn the_adopting_syncs_last_read_fails_unless_the_file_names_this_catalog() { + let bound = Duration::from_secs(10); + owner_after_sync(Some(Ok(Some(file("catalog-1")))), "catalog-1", bound).unwrap(); + let err = owner_after_sync(Some(Ok(Some(file("adopter")))), "catalog-1", bound) + .unwrap_err() + .to_string(); + assert!( + err.starts_with( + "after the sync, extenddb-store.json names catalog adopter (store id \ + store-1): another catalog adopted the root during the sync" + ), + "{err}" + ); + let err = owner_after_sync(Some(Ok(None)), "catalog-1", bound) + .unwrap_err() + .to_string(); + assert_eq!(err, "after the sync, the root has no extenddb-store.json"); + let err = owner_after_sync( + Some(Err(extenddb_engine::StoreProblem::Transport( + "down".to_owned(), + ))), + "catalog-1", + bound, + ) + .unwrap_err() + .to_string(); + assert!( + err.starts_with("after the sync, extenddb-store.json could not be read: "), + "{err}" + ); + let err = owner_after_sync(None, "catalog-1", bound) + .unwrap_err() + .to_string(); + assert_eq!( + err, + "after the sync, extenddb-store.json could not be read within 10 s" + ); + } + + /// A sync that skipped a prefix for a store or catalog error exits 1, + /// naming the count, and its summary line reads `incomplete`; skips of + /// any other reason leave it complete. + #[test] + fn a_sync_with_a_transient_skip_is_incomplete() { + use extenddb_engine::sync::{SkipReason, SkippedPrefix, SyncReport}; + let skipped = |reason| SkippedPrefix { + prefix: "123456789012/T/01700000000000-0000000a/".to_owned(), + reason, + }; + let lasting = || { + vec![ + skipped(SkipReason::Incomplete("no manifest".to_owned())), + skipped(SkipReason::RowExists("a row".to_owned())), + ] + }; + let complete = SyncReport { + skipped: lasting(), + ..SyncReport::default() + }; + complete_run(&complete).unwrap(); + assert_eq!(run_word(&complete, false), "complete"); + assert_eq!(run_word(&complete, true), "stopped"); + for reason in [ + SkipReason::Store("get failed".to_owned()), + SkipReason::Catalog("insert failed".to_owned()), + ] { + let mut skips = lasting(); + skips.push(skipped(reason)); + let report = SyncReport { + skipped: skips, + ..SyncReport::default() + }; + let err = complete_run(&report).unwrap_err().to_string(); + assert!( + err.starts_with( + "the run is incomplete: 1 prefix(es) were skipped for a store or \ + catalog error" + ), + "{err}" + ); + assert_eq!(run_word(&report, false), "incomplete"); + assert_eq!(run_word(&report, true), "stopped"); + } + } + + /// `--prune-missing`'s exit: a row left for a later run (a restore + /// reading the backup, a failed store or catalog call) exits 1 naming + /// the count and, for a restore, when to run the command again; rows + /// removed, whose manifest is back, whose status changed or that are + /// gone leave nothing for a later run and exit 0. + #[test] + fn a_prune_that_left_a_row_for_a_later_run_exits_1() { + use extenddb_engine::sync::PruneOutcome; + use extenddb_storage::backup_catalog::BackupStatus; + let arn = |n: u32| format!("arn:aws:dynamodb:us-east-1:123456789012:table/T/backup/{n}"); + let settled = vec![ + (arn(1), PruneOutcome::Pruned), + (arn(2), PruneOutcome::ManifestBack), + (arn(3), PruneOutcome::Status(BackupStatus::Deleting)), + (arn(4), PruneOutcome::Gone), + (arn(5), PruneOutcome::Pruned), + ]; + let counts = PruneCounts::of(&settled); + assert_eq!( + counts, + PruneCounts { + removed: 2, + left: 0, + in_use: 0 + } + ); + counts.left_for_a_later_run().unwrap(); + assert_eq!(PruneCounts::default().left_for_a_later_run().ok(), Some(())); + + let mut with_restore = settled.clone(); + with_restore.push((arn(6), PruneOutcome::InUse("T-restored".to_owned()))); + let counts = PruneCounts::of(&with_restore); + assert_eq!( + counts, + PruneCounts { + removed: 2, + left: 1, + in_use: 1 + } + ); + let err = counts.left_for_a_later_run().unwrap_err().to_string(); + assert_eq!( + err, + "1 row(s) whose manifest is gone were left for a later run (see above); 1 of them \ + are being restored from: run the command again once the restore ends" + ); + + let mut with_failure = settled; + with_failure.push(( + arn(7), + PruneOutcome::Failed("the manifest's head failed: down".to_owned()), + )); + with_failure.push((arn(8), PruneOutcome::InUse("T-r".to_owned()))); + let counts = PruneCounts::of(&with_failure); + assert_eq!( + counts, + PruneCounts { + removed: 2, + left: 2, + in_use: 1 + } + ); + let err = counts.left_for_a_later_run().unwrap_err().to_string(); + assert!( + err.starts_with("2 row(s) whose manifest is gone were left"), + "{err}" + ); + assert!(err.ends_with("once the restore ends"), "{err}"); + let counts = PruneCounts::of(&[( + arn(9), + PruneOutcome::Failed("the row could not be removed: down".to_owned()), + )]); + let err = counts.left_for_a_later_run().unwrap_err().to_string(); + assert_eq!( + err, + "1 row(s) whose manifest is gone were left for a later run (see above); run the \ + command again once the store and the catalog answer" + ); + } /// A registrar that counts its releases, each taking `takes`. struct Counting { diff --git a/crates/app/src/cmd_destroy.rs b/crates/app/src/cmd_destroy.rs index 3966d929b..38ebc6adf 100755 --- a/crates/app/src/cmd_destroy.rs +++ b/crates/app/src/cmd_destroy.rs @@ -96,9 +96,10 @@ pub async fn run(args: DestroyArgs) -> anyhow::Result<()> { println!("WARNING: This will permanently destroy ALL data in both databases."); println!( "The backup store ([backup.store]) is left untouched; `extenddb backup purge --confirm` \ - deletes its backups. Its extenddb-store.json names this catalog: a catalog re-created \ - with `init` needs that file removed (or a new root), or the store is reported owned \ - by another catalog." + deletes its backups. Its extenddb-store.json names this catalog, so a catalog \ + re-created with `init` finds the store owned by another catalog (on the default \ + SQLite root, the server's warning, which leaves the store unconfigured) until \ + `extenddb backup sync --adopt-store` makes it the owner and gives its backups rows." ); println!(); diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index 30870c3bb..6dfcc54c0 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -69,7 +69,8 @@ enum Command { Init(cmd_init::InitArgs), /// Tear down a extenddb deployment (the databases; the backup store is left untouched) Destroy(cmd_destroy::DestroyArgs), - /// List or purge the backups of the configured backup store, or remove a restore target + /// List, purge or sync the backups of the configured backup store, or remove a restore + /// target Backup(cmd_backup::BackupArgs), /// Validate a extenddb deployment Verify(cmd_verify::VerifyArgs), diff --git a/crates/config/src/lib.rs b/crates/config/src/lib.rs index ccdb5833d..c04dd0ccd 100644 --- a/crates/config/src/lib.rs +++ b/crates/config/src/lib.rs @@ -186,6 +186,16 @@ impl StorageConfig { self.config.database_file() } + /// [`Self::database_file`] with the reason a location names no usable + /// file ([`extenddb_storage::config::StorageConfig::database_location`]). + /// + /// # Errors + /// + /// The backend's reason the location is not usable. + pub fn database_location(&self) -> Result, String> { + self.config.database_location() + } + /// Get a reference to the underlying trait object for factory calls. pub fn as_trait(&self) -> &(dyn extenddb_storage::config::StorageConfig + 'static) { &*self.config diff --git a/crates/core/src/backup_format/layout.rs b/crates/core/src/backup_format/layout.rs index 5c2f87ff2..204f00688 100644 --- a/crates/core/src/backup_format/layout.rs +++ b/crates/core/src/backup_format/layout.rs @@ -47,6 +47,16 @@ pub const FILES_MANIFEST_FILE: &str = "manifest-files.json"; /// sees a manifest. pub const DATA_PREFIX: &str = "data/"; +/// The first component of a store key that is reserved for exports +/// (`/exports/...`). No account id is spelled this way (account ids +/// are digits), and [`classify_store_key`] names it rather than relying on +/// that. +pub const EXPORTS_COMPONENT: &str = "exports"; + +/// The third component of a store key that is reserved for the change-log +/// archive (`//
/changelog/...`). It is not a backup id. +pub const CHANGELOG_COMPONENT: &str = "changelog"; + /// Longest table name `DynamoDB` accepts, kept here as a path-safety bound: /// a longer component would exceed common filesystem name limits. The /// engine's configurable minimum length is not enforced here; a shorter name @@ -473,6 +483,33 @@ impl BackupArn { pub fn backup_id(&self) -> &str { &self.backup_id } + + /// Whether `other` names this backup: the same partition, account, + /// table and backup id, in any region. A manifest a server of another + /// region wrote names the backup in that region, and the row of the + /// backup carries the region of the server that holds it; readers that + /// compare the two accept a difference in the region alone. + #[must_use] + pub fn names_same_backup(&self, other: &Self) -> bool { + self.partition == other.partition + && self.account_id == other.account_id + && self.table_name == other.table_name + && self.backup_id == other.backup_id + } + + /// This backup's ARN spelled in `region`, or `None` when `region` is + /// not one an ARN can carry ([`check_backup_arn`]'s alphabet). + #[must_use] + pub fn in_region(&self, region: &str) -> Option { + Self::new( + &self.partition, + region, + &self.account_id, + &self.table_name, + &self.backup_id, + ) + .ok() + } } impl std::fmt::Display for BackupArn { @@ -489,6 +526,133 @@ impl std::str::FromStr for BackupArn { } } +// --------------------------------------------------------------------------- +// Store keys +// --------------------------------------------------------------------------- + +/// What a key of a store listing is, relative to the backup prefixes +/// [`BackupArn::store_prefix`] forms. See [`classify_store_key`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum StoreKey<'a> { + /// Fewer than four components: a file at the root (the store's identity + /// file), or one directly in an account's or a table's directory. No + /// object of a backup has such a key. + Outside, + /// Under a reserved component, the one named: [`EXPORTS_COMPONENT`] first, + /// or [`CHANGELOG_COMPONENT`] third. + Reserved(&'static str), + /// Under `///`, whose three components are not the prefix of a + /// backup a manifest can describe. + NotABackup { + /// `///`, a slice of the key. + prefix: &'a str, + /// Why, with the untrusted components cut to a bounded length. + reason: String, + }, + /// Under the prefix of a backup. + Backup { + /// `/
//`, a slice of the key, equal to + /// [`BackupArn::store_prefix`] of every ARN with these components. + prefix: &'a str, + /// The account id component. + account_id: &'a str, + /// The table name component. + table_name: &'a str, + /// The backup id component, in the 14-digit shape + /// [`format_backup_id`] produces. + backup_id: &'a str, + }, +} + +/// Classify a key of a store listing (relative to the store's root): under a +/// backup prefix `/
//`, under a reserved component, +/// under three components that are not a backup prefix, or outside every +/// prefix. +/// +/// A backup prefix is one whose components [`BackupArn::new`] accepts, in any +/// partition and region (so a component carrying a separator, surrounding +/// whitespace, a control or non-ASCII character, or a traversal component +/// such as `..` is refused), and whose id has the 14-digit shape: only that +/// shape carries the instant a manifest's `backup_creation_date_time` must +/// equal, so a prefix under an earlier build's id (13 digits, with or without +/// a suffix) holds no manifest this build can read. +#[must_use] +pub fn classify_store_key(key: &str) -> StoreKey<'_> { + let mut parts = key.splitn(4, '/'); + let account_id = parts.next().unwrap_or_default(); + let table_name = parts.next(); + if account_id == EXPORTS_COMPONENT && table_name.is_some() { + return StoreKey::Reserved(EXPORTS_COMPONENT); + } + let (Some(table_name), Some(backup_id), Some(_)) = (table_name, parts.next(), parts.next()) + else { + return StoreKey::Outside; + }; + if backup_id == CHANGELOG_COMPONENT { + return StoreKey::Reserved(CHANGELOG_COMPONENT); + } + let prefix = &key[..account_id.len() + table_name.len() + backup_id.len() + 3]; + match store_prefix_problem(account_id, table_name, backup_id) { + Some(reason) => StoreKey::NotABackup { prefix, reason }, + None => StoreKey::Backup { + prefix, + account_id, + table_name, + backup_id, + }, + } +} + +/// The account id, table name and backup id of the backup prefix `key` is +/// under, or `None` when [`classify_store_key`] finds it outside every backup +/// prefix, under a reserved component, or under components that are not a +/// backup prefix. +#[must_use] +pub fn parse_store_prefix(key: &str) -> Option<(&str, &str, &str)> { + match classify_store_key(key) { + StoreKey::Backup { + account_id, + table_name, + backup_id, + .. + } => Some((account_id, table_name, backup_id)), + StoreKey::Outside | StoreKey::Reserved(_) | StoreKey::NotABackup { .. } => None, + } +} + +/// Why three path components are not a backup prefix, or `None` when they +/// are. The rules are [`BackupArn::new`]'s; the partition and region given to +/// it are placeholders that pass, since a prefix carries neither. +fn store_prefix_problem(account_id: &str, table_name: &str, backup_id: &str) -> Option { + const PARTITION: &str = "aws"; + const REGION: &str = "us-east-1"; + if BackupArn::new(PARTITION, REGION, account_id, table_name, backup_id).is_ok() { + return backup_id_epoch_millis(backup_id).is_none().then(|| { + format!( + "backup id {:?} has an earlier build's shape, which no manifest carries", + truncated(backup_id) + ) + }); + } + // Name the class of the refusal without the placeholder partition and + // region, which `BackupArn::new`'s own message would show. + let formatted = format_backup_arn(PARTITION, REGION, account_id, table_name, backup_id); + Some(match check_backup_arn(&formatted) { + Err(BackupArnError::Id) => format!("{:?} is not a backup id", truncated(backup_id)), + Err(BackupArnError::Invalid) => format!( + "account id {:?}, table name {:?} and backup id {:?} are not the components of a \ + backup ARN", + truncated(account_id), + truncated(table_name), + truncated(backup_id) + ), + Ok(_) => match check_store_prefix_table_name(table_name) { + Err(e) => e.to_string(), + Ok(()) => "a component has surrounding whitespace".to_owned(), + }, + }) +} + #[cfg(test)] mod tests { use super::*; @@ -525,6 +689,32 @@ mod tests { assert_eq!(ARN.parse::().unwrap(), arn); } + /// Two spellings of one backup differ in the region alone; a change in + /// the partition, account, table or id names another backup. + #[test] + fn same_backup_in_another_region() { + let arn = BackupArn::parse(ARN).unwrap(); + let west = arn.in_region("us-west-2").unwrap(); + assert_eq!( + west.to_arn_string(), + "arn:aws:dynamodb:us-west-2:123456789012:table/Music/backup/01489602797149-73d8d5bc" + ); + assert!(arn.names_same_backup(&west)); + assert!(west.names_same_backup(&arn)); + assert!(arn.names_same_backup(&arn)); + for other in [ + "arn:aws-cn:dynamodb:us-east-1:123456789012:table/Music/backup/01489602797149-73d8d5bc", + "arn:aws:dynamodb:us-east-1:123456789013:table/Music/backup/01489602797149-73d8d5bc", + "arn:aws:dynamodb:us-east-1:123456789012:table/Songs/backup/01489602797149-73d8d5bc", + "arn:aws:dynamodb:us-east-1:123456789012:table/Music/backup/01489602797149-73d8d5bd", + ] { + let other = BackupArn::parse(other).unwrap(); + assert!(!arn.names_same_backup(&other), "{other}"); + } + assert_eq!(arn.in_region("us east"), None, "outside the ARN alphabet"); + assert_eq!(arn.in_region(""), None); + } + #[test] fn other_partitions_parse() { for arn in [ @@ -797,5 +987,239 @@ mod tests { assert_eq!(SUMMARY_MANIFEST_FILE, "manifest-summary.json"); assert_eq!(FILES_MANIFEST_FILE, "manifest-files.json"); assert_eq!(DATA_PREFIX, "data/"); + assert_eq!(EXPORTS_COMPONENT, "exports"); + assert_eq!(CHANGELOG_COMPONENT, "changelog"); + } + + const ID: &str = "01489602797149-73d8d5bc"; + + fn not_a_backup(key: &str) -> String { + match classify_store_key(key) { + StoreKey::NotABackup { prefix, reason } => { + assert!(key.starts_with(prefix), "{key:?}: {prefix:?}"); + assert_eq!(prefix.matches('/').count(), 3, "{key:?}: {prefix:?}"); + assert!(prefix.ends_with('/'), "{key:?}: {prefix:?}"); + assert_eq!(parse_store_prefix(key), None, "{key:?}"); + reason + } + other => panic!("{key:?}: expected NotABackup, got {other:?}"), + } + } + + #[test] + fn store_keys_under_a_backup_prefix_parse_to_its_components() { + for name in [ + EXTENDDB_MANIFEST_FILE, + MANIFEST_CHECKSUM_FILE, + "data/000001.json.gz", + "data/a/b/c", + ] { + let key = format!("123456789012/Music/{ID}/{name}"); + assert_eq!( + classify_store_key(&key), + StoreKey::Backup { + prefix: &format!("123456789012/Music/{ID}/"), + account_id: "123456789012", + table_name: "Music", + backup_id: ID, + }, + "{key}" + ); + assert_eq!( + parse_store_prefix(&key), + Some(("123456789012", "Music", ID)), + "{key}" + ); + } + // The prefix is the one `BackupArn::store_prefix` forms, whatever the + // ARN's partition and region. + for (partition, region, table) in [ + ("aws", "us-east-1", "Music"), + ("aws-cn", "cn-north-1", "My.Table-2"), + ("aws-us-gov", "us-gov-west-1", "a_b"), + ] { + let arn = BackupArn::new(partition, region, "000000000001", table, ID).unwrap(); + let key = format!("{}{EXTENDDB_MANIFEST_FILE}", arn.store_prefix()); + let StoreKey::Backup { prefix, .. } = classify_store_key(&key) else { + panic!("{key}"); + }; + assert_eq!(prefix, arn.store_prefix()); + assert_eq!( + parse_store_prefix(&key), + Some((arn.account_id(), arn.table_name(), arn.backup_id())) + ); + } + // A table named like a reserved component is a table: `exports` is + // reserved only first and `changelog` only third. + for table in [EXPORTS_COMPONENT, CHANGELOG_COMPONENT] { + let key = format!("123/{table}/{ID}/{EXTENDDB_MANIFEST_FILE}"); + assert_eq!(parse_store_prefix(&key), Some(("123", table, ID)), "{key}"); + } + } + + #[test] + fn store_keys_outside_every_prefix_and_under_reserved_components_are_skipped() { + for key in [ + "", + "extenddb-store.json", + "123456789012", + "123456789012/Music", + "123456789012/Music/file", + &format!("123456789012/Music/{ID}"), + // A file named like a reserved component, not under it. + EXPORTS_COMPONENT, + "123456789012/Music/changelog", + ] { + assert_eq!(classify_store_key(key), StoreKey::Outside, "{key:?}"); + assert_eq!(parse_store_prefix(key), None, "{key:?}"); + } + for key in [ + "exports/", + "exports/x", + "exports/123456789012/Music", + &format!("exports/123456789012/{ID}/{EXTENDDB_MANIFEST_FILE}"), + &format!("exports/Music/{ID}/{EXTENDDB_MANIFEST_FILE}"), + ] { + assert_eq!( + classify_store_key(key), + StoreKey::Reserved(EXPORTS_COMPONENT), + "{key:?}" + ); + assert_eq!(parse_store_prefix(key), None, "{key:?}"); + } + for key in [ + "123456789012/Music/changelog/", + "123456789012/Music/changelog/000001.json.gz", + &format!("123456789012/Music/changelog/{ID}/{EXTENDDB_MANIFEST_FILE}"), + ] { + assert_eq!( + classify_store_key(key), + StoreKey::Reserved(CHANGELOG_COMPONENT), + "{key:?}" + ); + assert_eq!(parse_store_prefix(key), None, "{key:?}"); + } + // The reserved words are matched whole and by case: another + // component in their place is an account or an id like any other. + for key in [ + "Exports/Music/x/y", + "exportsx/Music/x/y", + "123/Music/Changelog/x", + "123/Music/changelogs/x", + ] { + not_a_backup(key); + } + } + + /// The id grid of `id_shape_grid_and_error_classes`, applied to the third + /// component: only the 14-digit shape is a backup prefix; the shapes an + /// earlier build issued are refused with their own reason, and every + /// other string is not a backup id. + #[test] + fn store_prefix_ids_are_the_fourteen_digit_shape_only() { + let mut ids: Vec = Vec::new(); + for millis_len in 0..=16usize { + for suffix_len in 0..=10usize { + ids.push(format!( + "{}-{}", + "1".repeat(millis_len), + "a".repeat(suffix_len) + )); + } + ids.push("1".repeat(millis_len)); + } + ids.extend( + [ + "01489602797149-73D8D5BC", + "01489602797149-73d8d5bg", + "0148960279714x-73d8d5bc", + "01489602797149_73d8d5bc", + "01489602797149-73d8d5bc-0", + "-01489602797149-73d8d5bc", + "a4b1d6c2-0000-4000-8000-000000000000", + ] + .map(str::to_owned), + ); + let (mut adopted, mut earlier, mut refused) = (0, 0, 0); + for id in &ids { + let key = format!("123456789012/Music/{id}/{EXTENDDB_MANIFEST_FILE}"); + if backup_id_epoch_millis(id).is_some() { + assert_eq!( + parse_store_prefix(&key), + Some(("123456789012", "Music", id.as_str())), + "{id}" + ); + adopted += 1; + } else if is_backup_id(id) { + let reason = not_a_backup(&key); + assert!(reason.contains("earlier build's shape"), "{id}: {reason}"); + earlier += 1; + } else { + let reason = not_a_backup(&key); + // An empty component is refused as a malformed path before + // the id rule is reached. + let expected = if id.is_empty() { + "are not the components of a backup ARN" + } else { + "not a backup id" + }; + assert!(reason.contains(expected), "{id}: {reason}"); + refused += 1; + } + } + // Every class was reached: the 14-digit form; 13 digits with a + // suffix and 13 bare digits; the rest. + assert_eq!((adopted, earlier), (1, 2), "{ids:?}"); + assert!(refused > 100, "{refused}"); + } + + #[test] + fn store_prefix_components_with_traversal_or_separators_are_refused() { + let long = "a".repeat(MAX_TABLE_NAME_LEN + 1); + for key in [ + // Traversal components. + format!("123/../{ID}/x"), + format!("123/./{ID}/x"), + format!("123/..../{ID}/x"), + format!("../Music/{ID}/x"), + format!("./Music/{ID}/x"), + "123/Music/../x".to_owned(), + "123/Music/./x".to_owned(), + // Empty components. + format!("/Music/{ID}/x"), + format!("123//{ID}/x"), + "123/Music//x".to_owned(), + // Separators and characters a component cannot carry. + format!("12:3/Music/{ID}/x"), + format!("123/Mu:sic/{ID}/x"), + format!("123/Music/{ID}:x/x"), + format!("12\\3/Music/{ID}/x"), + format!("123/Mu\\sic/{ID}/x"), + format!("123/Music/{ID}\\x/x"), + format!("123/Mu sic/{ID}/x"), + format!("123/Music\t/{ID}/x"), + format!("123/Music/{ID} /x"), + format!("123/Music/{ID}\n/x"), + format!(" 123/Music/{ID}/x"), + format!("123/Mu\u{0}sic/{ID}/x"), + format!("123/Mus\u{e9}c/{ID}/x"), + // An account is digits; a table name is at most 255 bytes. + format!("12345678901x/Music/{ID}/x"), + format!("arn:aws/Music/{ID}/x"), + format!("123/{long}/{ID}/x"), + ] { + let reason = not_a_backup(&key); + assert!(!reason.is_empty(), "{key:?}"); + // The placeholder partition and region never reach the reason. + assert!(!reason.contains("us-east-1"), "{key:?}: {reason}"); + } + // The reasons name the rule where one rule applies. + assert!(not_a_backup(&format!("123/../{ID}/x")).contains("is only dots")); + assert!(not_a_backup(&format!("123/Music/{ID}\n/x")).contains("surrounding whitespace")); + assert!(not_a_backup(&format!("123/{long}/{ID}/x")).contains("longer than 255")); + // The reason is bounded however long the components are. + let huge = "x".repeat(100_000); + let reason = not_a_backup(&format!("{huge}/{huge}/{huge}/x")); + assert!(reason.len() < 1_000, "{}", reason.len()); } } diff --git a/crates/core/src/backup_format/mod.rs b/crates/core/src/backup_format/mod.rs index 94491a911..4516cd00e 100644 --- a/crates/core/src/backup_format/mod.rs +++ b/crates/core/src/backup_format/mod.rs @@ -83,11 +83,13 @@ pub use documents::{BackupDocuments, ManifestHeader}; pub use error::{FormatError, JsonError}; pub use key_order::{KeyOrder, check_key_value, hash_key_text, shown}; pub use layout::{ - BACKUP_ID_MILLIS_DIGITS, BackupArn, BackupArnError, BackupArnParts, DATA_PREFIX, - EXTENDDB_MANIFEST_FILE, FILES_MANIFEST_FILE, MANIFEST_CHECKSUM_FILE, MAX_BACKUP_ARN_LEN, - MAX_TABLE_NAME_LEN, SUMMARY_MANIFEST_FILE, backup_creation_date_time, backup_id_epoch_millis, - check_backup_arn, check_store_prefix_table_name, data_file_key, format_backup_arn, - format_backup_id, is_backup_id, is_region, parse_backup_arn, trim_backup_arn, + BACKUP_ID_MILLIS_DIGITS, BackupArn, BackupArnError, BackupArnParts, CHANGELOG_COMPONENT, + DATA_PREFIX, EXPORTS_COMPONENT, EXTENDDB_MANIFEST_FILE, FILES_MANIFEST_FILE, + MANIFEST_CHECKSUM_FILE, MAX_BACKUP_ARN_LEN, MAX_TABLE_NAME_LEN, SUMMARY_MANIFEST_FILE, + StoreKey, backup_creation_date_time, backup_id_epoch_millis, check_backup_arn, + check_store_prefix_table_name, classify_store_key, data_file_key, format_backup_arn, + format_backup_id, is_backup_id, is_region, parse_backup_arn, parse_store_prefix, + trim_backup_arn, }; pub use manifest::{ BackupGsi, BackupLsi, CURRENT_FORMAT_VERSION, DataFileEntry, DataFileReader, ExtenddbManifest, diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs index 9d01ada6f..213d3be53 100755 --- a/crates/engine/src/backup.rs +++ b/crates/engine/src/backup.rs @@ -41,7 +41,7 @@ pub use maintenance::HeldClaim; pub use restore::{OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, TARGET_NOT_EMPTY}; pub use runtime::{ BackupRuntime, ConfiguredStore, JobSettings, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, - StoreHealth, StoreHealthReport, check_configured_store, read_identity_bounded, + StoreFence, StoreHealth, StoreHealthReport, check_configured_store, read_identity_bounded, }; pub use store_identity::{STORE_IDENTITY_KEY, StoreIdentityFile, StoreProblem}; @@ -1236,18 +1236,31 @@ async fn delete_backup( /// ([`BackupCatalog::start_backup_delete`]), so a restore's registration and /// this order wholly one before the other: a registration that won leaves /// this `BackupInUseException` naming its target. A DELETING row (a retried -/// DeleteBackup) resumes the deletion. Then the deletion itself +/// DeleteBackup) resumes the deletion. The deletion fence +/// ([`StoreFence`]) runs before the DELETING write and again right before +/// the first deletion. At the first, a root it finds owned by another +/// catalog, a root with no identity file, a file the store refuses to +/// serve and a file that does not parse are recorded on the store and +/// refused with the row as it was (and its objects kept), answered as the +/// unhealthy store's `ValidationException`; a read that fails in transport +/// (the store unreachable, or the read past its bound) is a store failure +/// and the request goes on to the DELETING write, the second fence then +/// refusing the deletion. A refusal at the second (another catalog adopted +/// the root in between, or the read still fails) leaves the row +/// DELETING with nothing deleted, which the reconciler finishes once its +/// own fence passes (for an adopted root, only if this catalog owns the +/// root again). Then the deletion itself /// ([`finish_backup_delete`], through the store's bounded calls, /// [`ConfiguredStore::calls`]): the manifest first, so the prefix is no /// longer a complete backup to any reader, then the rest of the prefix, /// then the row to DELETED once a listing of the prefix returns nothing. The /// request answers the description with `BackupStatus` DELETED once the -/// DELETING write has committed, whatever the deletion does: every reader -/// already reports the backup deleted from that write on, and the service -/// has no counterpart for a failed object deletion. If a step fails (the -/// store not answering, a listing that still shows an object) the row stays -/// DELETING, the failure is logged, and the reconciler finishes the deletion -/// at its next cycle. +/// DELETING write has committed, whatever the deletion does, the second +/// fence's refusal included: every reader already reports the backup +/// deleted from that write on, and the service has no counterpart for a +/// failed object deletion. If a step fails (the store not answering, a +/// listing that still shows an object) the row stays DELETING, the failure +/// is logged, and the reconciler finishes the deletion at its next cycle. /// /// The description is built before the DELETING write: a row whose /// `source_table_details_json` this build cannot read is answered as an @@ -1278,6 +1291,38 @@ async fn delete_catalog_backup( } let store = runtime.require_healthy_store()?; let mut desc = row.description().map_err(storage_err_to_dynamo)?; + // The fence before the DELETING write as well as before the deletion: a + // root it finds owned by another catalog (or by this one under another + // store id) is refused here with the row as it was, so the API never + // answers that refusal with a DELETED backup. A root with no file, a + // file the store refuses to serve and one that does not parse are + // recorded on the store and refused here the same way. A read that + // fails in transport (the store unreachable, or the read past its + // bound) records nothing and is a store failure: the request goes on as + // it does when the store does not answer the deletion, and the fence + // before the deletion refuses it, leaving the row DELETING for the + // reconciler, whose deletion runs the fence again. + let fence = store.fence(); + if let Err(problem) = fence.check_catalog(storage).await { + if let Some(refusal) = fence.refusal() { + tracing::error!( + "DeleteBackup {arn_text}: refused before the row was changed; nothing was \ + deleted: {problem}" + ); + return Err(refusal); + } + if let StoreProblem::Catalog(e) = &problem { + tracing::error!("DeleteBackup {arn_text}: the catalog id could not be read: {e}"); + return Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )); + } + tracing::warn!( + "DeleteBackup {arn_text}: the deletion fence could not read the store's identity \ + file before the DELETING write ({problem}); the deletion is fenced again before \ + any object is deleted" + ); + } match storage .start_backup_delete(&key) .await @@ -1301,18 +1346,75 @@ async fn delete_catalog_backup( )))); } } - if let Err(e) = finish_backup_delete(storage, &store.calls("deletion"), arn).await { - tracing::error!( - "DeleteBackup {arn_text}: the row is DELETING but its objects could not be deleted; \ - the reconciler retries: {e}" - ); + if let Err(e) = finish_backup_delete(storage, &store.calls("deletion"), &fence, arn).await { + match e { + FinishError::Fenced(problem) => tracing::error!( + "DeleteBackup {arn_text}: the row is DELETING and no object was deleted; {}", + fenced_deletion_note(&problem) + ), + FinishError::Failed(e) => tracing::error!( + "DeleteBackup {arn_text}: the row is DELETING but its objects could not be \ + deleted; the reconciler retries: {e}" + ), + } } desc.backup_details.backup_status = BackupStatus::Deleted.as_str().to_owned(); Ok(desc) } +/// The rest of the log line for a deletion the fence refused: what the +/// fence found (`problem`) and when the deletion finishes. Only a file that +/// names another catalog is an adoption; a file that could not be read, or +/// is missing, says so, and an unhealthy store names its problem. +fn fenced_deletion_note(problem: &StoreProblem) -> String { + match problem { + StoreProblem::ForeignCatalog { .. } => format!( + "another catalog adopted the root after the first fence ({problem}); the deletion \ + finishes if this catalog owns the root again" + ), + StoreProblem::Transport(_) + | StoreProblem::Denied(_) + | StoreProblem::Validate(_) + | StoreProblem::FileUnreadable(_) + | StoreProblem::FileMissingWithRows(_) => format!( + "the store's identity file could not be read ({problem}); the reconciler finishes \ + the deletion once it can" + ), + StoreProblem::IdMismatch { .. } + | StoreProblem::MultipleRowIds(_) + | StoreProblem::CreateRaceLost { .. } + | StoreProblem::ManifestMissing { .. } + | StoreProblem::Catalog(_) => format!( + "the store is unhealthy ({problem}); the reconciler finishes the deletion once the \ + store's check passes" + ), + } +} + +/// Why [`finish_backup_delete`] left the row DELETING. +#[derive(Debug)] +pub(crate) enum FinishError { + /// The deletion fence refused before any object was deleted + /// ([`StoreFence`]): the store is not healthy, or its identity file + /// names another catalog or could not be read. + Fenced(StoreProblem), + /// A store or catalog call failed, or the prefix still lists an object. + Failed(String), +} + +impl std::fmt::Display for FinishError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Fenced(problem) => write!(f, "the deletion was refused: {problem}"), + Self::Failed(e) => f.write_str(e), + } + } +} + /// The deletion of a DELETING row's objects and the row's move to DELETED: -/// `extenddb-manifest.json` first (a `delete_prefix` on its exact key, so the +/// the deletion fence (`fence`, which re-reads `extenddb-store.json` and +/// refuses when the root is no longer this catalog's), then +/// `extenddb-manifest.json` (a `delete_prefix` on its exact key, so the /// prefix is no longer a complete backup to any reader), then the rest of the /// prefix, then, once a listing of the prefix returns nothing, /// `set_backup_deleted`. Shared by DeleteBackup and the reconciler's duty for @@ -1322,42 +1424,50 @@ async fn delete_catalog_backup( /// /// # Errors /// -/// A store failure, a listing that still shows objects (a put that -/// overlapped the deletion, which the next run removes), or a catalog write -/// failure; the row stays DELETING for the next run. +/// The fence's refusal, with nothing deleted; a store failure, a listing +/// that still shows objects (a put that overlapped the deletion, which the +/// next run removes), or a catalog write failure. The row stays DELETING +/// for the next run. pub(crate) async fn finish_backup_delete( storage: &S, store: &dyn extenddb_backup_store::BackupStore, + fence: &StoreFence, arn: &BackupArn, -) -> Result { +) -> Result { let prefix = arn.store_prefix(); let manifest_key = format!( "{prefix}{}", extenddb_core::backup_format::EXTENDDB_MANIFEST_FILE ); + fence + .check_catalog(storage) + .await + .map_err(FinishError::Fenced)?; + let failed = + |what: &str, e: &dyn std::fmt::Display| FinishError::Failed(format!("{what}: {e}")); store .delete_prefix(&manifest_key) .await - .map_err(|e| format!("deleting the manifest: {e}"))?; + .map_err(|e| failed("deleting the manifest", &e))?; store .delete_prefix(&prefix) .await - .map_err(|e| format!("deleting the prefix: {e}"))?; + .map_err(|e| failed("deleting the prefix", &e))?; let remaining = store .list(&prefix) .try_next() .await - .map_err(|e| format!("listing the prefix after the deletion: {e}"))?; + .map_err(|e| failed("listing the prefix after the deletion", &e))?; if let Some(object) = remaining { - return Err(format!( + return Err(FinishError::Failed(format!( "the prefix still lists {} after the deletion", object.key - )); + ))); } storage .set_backup_deleted(&BackupKey::from_arn(arn)) .await - .map_err(|e| format!("marking the row DELETED: {e}")) + .map_err(|e| failed("marking the row DELETED", &e)) } /// Overrides refused outright. `VectorIndexOverride` is not in the botocore @@ -1910,6 +2020,7 @@ pub mod reconciler; mod restore; mod runtime; pub mod store_identity; +pub mod sync; #[cfg(test)] mod tests; diff --git a/crates/engine/src/backup/job.rs b/crates/engine/src/backup/job.rs index 90cdd7e5a..bca4db180 100644 --- a/crates/engine/src/backup/job.rs +++ b/crates/engine/src/backup/job.rs @@ -7,8 +7,12 @@ //! The task ([`start`]) opens the table snapshot, mints the backup id from //! the millisecond the open returned, reads the table's TTL, builds the //! manifest's `source_table` from the snapshot's description -//! ([`source_table_schema`]), inserts the CREATING row under the job's -//! claim (`claim_backup`), answers the request through a one-shot channel +//! ([`source_table_schema`]), runs the deletion fence (a root another +//! catalog adopted since the store's last check is refused before the row, +//! `StoreFence`; a fence read that recorded a problem answers the +//! unhealthy-store refusal, one that failed in transport an internal +//! error), inserts the CREATING row under the job's claim +//! (`claim_backup`), answers the request through a one-shot channel //! (`BackupDetails` CREATING, or the refusal) and then goes on as the job //! ([`run`]) with the same owned values: nothing crosses a thread or a //! channel between the request and the job but the response (the @@ -29,7 +33,11 @@ //! `CreateBackup` waiting in the open: its connection closed 8.2 s after it //! was sent), so there the refusal's effect is the row not written. //! -//! The job: the data files through the pipeline ([`writer`]), then the +//! The job: the fence once more, right before the first put (a root +//! adopted between the request's read and the job's first write gets no +//! object of this job's; the refusal fails the row with the fence's reason +//! and makes no store write), the data files through the pipeline +//! ([`writer`]), then the //! three documents `BackupDocuments::build` returns in its order, a re-check //! of the claim, the manifest (the commit record), and the AVAILABLE write //! with the size, the count and the manifest's `source_table` @@ -47,7 +55,7 @@ //! period, ends the job with no further store or catalog access. Any other //! failure, and the shutdown token (reason `shutdown`), runs the failure //! path ([`failure::fail_row`]): FAILED, then the prefix's deletion only -//! when that write matched. +//! when that write matched and the fence passed. //! //! Each job records `BackupJobs` and `BackupJobDuration` (microseconds, //! from the snapshot's open to the end of its last catalog or store call: @@ -93,7 +101,7 @@ use extenddb_storage::{ use tokio::sync::oneshot; use tokio::time::Instant; -use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings, Spool}; +use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings, Spool, StoreFence}; use super::{create_backup_storage_err_to_dynamo, storage_err_to_dynamo}; use failure::{FailurePath, Written}; use heartbeat::{HeartbeatHandle, HeartbeatMetric}; @@ -115,6 +123,10 @@ pub(crate) const SHUTDOWN: &str = "shutdown"; pub(crate) struct JobEnv { pub configured: Arc, pub store: Arc, + /// The store's deletion fence, which CreateBackup runs before its row, + /// the job right before its first put and the failure path before its + /// deletion. + pub fence: StoreFence, pub identity: StoreIdentity, pub bucket: String, pub key_prefix: String, @@ -144,6 +156,7 @@ impl JobEnv { Ok(Self { configured: Arc::clone(store), store: Arc::clone(&store.store), + fence: store.fence(), identity, bucket: store.bucket.clone(), key_prefix: store.key_prefix.clone(), @@ -210,6 +223,9 @@ pub(crate) struct Job { key_order: Option, limits: RestorableLimits, env: JobEnv, + /// This instance's catalog id, which the fence before the first put and + /// the failure path's fence compare the identity file with. + catalog_id: String, /// The id's instant, the row's `created_at`. created_at: f64, started: Instant, @@ -319,6 +335,26 @@ where owner_generation: UNOWNED_GENERATION, heartbeat_at: 0.0, }; + // The fence before the row: a root another catalog adopted since the + // store's last check gets no row from this catalog, and the store is + // marked unhealthy. A read that recorded a problem (the file names + // another catalog, is missing, or cannot be served) answers as the + // unhealthy store the next request meets; only a read that failed in + // transport, which recorded nothing, answers as a failed store call. + // The job runs the fence again right before its first put. + let catalog_id = env + .fence + .catalog_id(storage) + .await + .map_err(|e| internal(&format!("the catalog id could not be read: {e}")))?; + if let Err(problem) = env.fence.check(&catalog_id).await { + return Err(env.fence.refusal().unwrap_or_else(|| { + internal(&format!( + "{table_name}: the store's identity file could not be read before the row: \ + {problem}" + )) + })); + } let claimed_at = Instant::now(); // The insert too is selected against the token: a server that is // stopping inserts no row it would have to fail at once. @@ -346,6 +382,7 @@ where key_order, limits: *limits, env: env.clone(), + catalog_id, created_at, started, claimed_at, @@ -395,6 +432,7 @@ pub(crate) async fn run(job: Job) -> BackupJobOutcome { key_order, limits, env, + catalog_id, created_at, started, claimed_at, @@ -439,31 +477,40 @@ pub(crate) async fn run(job: Job) -> BackupJobOutcome { }; let tracker = key_order.as_ref().map(KeyRangeTracker::new); let mut item_count = 0; - let end = match writer::write_data_files(&pipeline, items, key_schema, limits, tracker).await { - Ok(files) => { - item_count = files.item_count; - // Both figures are the stream's: the items written and their - // DynamoDB sizes summed, not the statistics pass's (which can - // be stale, or 0 on a fresh table). - source_table.item_count = i64::try_from(files.item_count).unwrap_or(i64::MAX); - source_table.table_size_bytes = i64::try_from(files.item_bytes).unwrap_or(i64::MAX); - let header = ManifestHeader { - backup_arn: arn.clone(), - backup_name, - source_backend: env.source_backend.clone(), - source_extenddb_version: env.source_extenddb_version.clone(), - snapshot: info, - item_order: if key_order.is_some() { - ItemOrder::KeyOrdered - } else { - ItemOrder::Unordered - }, - source_table, - }; - write_documents(&pipeline, &claim, header, files).await - } - Err(stop) => Err(stop), - }; + let end = async { + // The fence right before the first put (`StoreFence`'s doc): the + // request's read was before the row, and a root another catalog + // adopted since gets no object of this job's. The refusal, recorded + // or not, fails the row with no store write. + env.fence.check(&catalog_id).await.map_err(|problem| { + Stop::StoreUnhealthy(format!( + "the store's identity file was re-read before the first write and refused \ + it ({problem}); no object was written" + )) + })?; + let files = writer::write_data_files(&pipeline, items, key_schema, limits, tracker).await?; + item_count = files.item_count; + // Both figures are the stream's: the items written and their + // DynamoDB sizes summed, not the statistics pass's (which can + // be stale, or 0 on a fresh table). + source_table.item_count = i64::try_from(files.item_count).unwrap_or(i64::MAX); + source_table.table_size_bytes = i64::try_from(files.item_bytes).unwrap_or(i64::MAX); + let header = ManifestHeader { + backup_arn: arn.clone(), + backup_name, + source_backend: env.source_backend.clone(), + source_extenddb_version: env.source_extenddb_version.clone(), + snapshot: info, + item_order: if key_order.is_some() { + ItemOrder::KeyOrdered + } else { + ItemOrder::Unordered + }, + source_table, + }; + write_documents(&pipeline, &claim, header, files).await + } + .await; let (outcome, bytes) = match end { Ok(bytes) => { tracing::info!( @@ -480,14 +527,15 @@ pub(crate) async fn run(job: Job) -> BackupJobOutcome { (BackupJobOutcome::Failed, 0) } Err(Stop::StoreUnhealthy(reason)) => { - // The store is not to be called: FAILED with the store's reason, - // the prefix left to the FAILED-prefix duty once the store is - // healthy again. + // The store is not to be written: FAILED with the store's + // reason, the prefix left to the FAILED-prefix duty once the + // store is healthy again. fail_without_store_access(&*claim, &arn, &reason).await; tracing::error!( - "backup {arn_text}: FAILED ({reason}); no store call was made and its objects \ - are deleted by the reconciler once the store's check passes and the row's \ - heartbeat is stale" + "backup {arn_text}: FAILED ({reason}); no store write was made and its objects, \ + if any, are deleted by the reconciler once the store's check passes and the \ + row's heartbeat is stale (never, while another catalog has adopted the root: \ + the objects then stay under it)" ); (BackupJobOutcome::Failed, 0) } @@ -497,10 +545,23 @@ pub(crate) async fn run(job: Job) -> BackupJobOutcome { Stop::Fail(reason) => (reason, BackupJobOutcome::Failed), Stop::ClaimLost(_) | Stop::StoreUnhealthy(_) => unreachable!("matched above"), }; - match failure::fail_row(&*claim, &*env.store, &prefix, &reason, metrics).await { + match failure::fail_row( + &*claim, + &*env.store, + &env.fence, + &catalog_id, + &prefix, + &reason, + metrics, + ) + .await + { FailurePath::Deleted => tracing::error!( "backup {arn_text}: FAILED ({reason}); its objects were deleted" ), + FailurePath::Fenced(why) => tracing::error!( + "backup {arn_text}: FAILED ({reason}); its objects were not deleted: {why}" + ), FailurePath::DeleteFailed(e) => tracing::error!( "backup {arn_text}: FAILED ({reason}); its objects could not be deleted and \ the reconciler deletes them once the row's heartbeat is stale: {e}" diff --git a/crates/engine/src/backup/job/failure.rs b/crates/engine/src/backup/job/failure.rs index c0a2eba2d..27920fa62 100644 --- a/crates/engine/src/backup/job/failure.rs +++ b/crates/engine/src/backup/job/failure.rs @@ -31,14 +31,18 @@ //! reconciler's (AVAILABLE from a complete manifest, FAILED otherwise). //! //! The backup job's failure path ([`fail_row`]) is the FAILED write -//! followed, only when it matched, by `delete_prefix` of the backup's -//! prefix; the holder of the claim owns the prefix, so a job whose FAILED -//! write did not match never deletes. A deletion that fails leaves the +//! followed, only when it matched, by the deletion fence and then +//! `delete_prefix` of the backup's prefix; the holder of the claim owns the +//! prefix, so a job whose FAILED write did not match never deletes, and a +//! root another catalog adopted while the job ran is left as it is (the +//! fence marks the store unhealthy). A deletion that fails leaves the //! prefix to the reconciler, which deletes a FAILED row's prefix once its //! heartbeat is stale. The restore job's failure path, FAILED then the //! target's removal under the same gate, is `super::super::restore::fail_restore`. use extenddb_backup_store::BackupStore; + +use super::super::StoreFence; use extenddb_core::metrics::MetricsCollector; use extenddb_storage::JobClaim; use extenddb_storage::backup_catalog::{JobStatus, StatusFields, WriteOutcome}; @@ -115,16 +119,21 @@ pub(crate) enum FailurePath { Deleted, /// FAILED written; the deletion failed and is the reconciler's. DeleteFailed(String), + /// FAILED written; the deletion fence refused, and nothing was + /// deleted. + Fenced(String), /// FAILED not written by this claim (the row is another's, or the claim /// connection failed): no store access was made. NoStoreAccess(String), } -/// The failure path (module doc): FAILED with `reason`, then the deletion -/// of `prefix` when the write matched. +/// The failure path (module doc): FAILED with `reason`, then, when the +/// write matched, the fence for `catalog_id` and the deletion of `prefix`. pub(crate) async fn fail_row( claim: &dyn JobClaim, store: &dyn BackupStore, + fence: &StoreFence, + catalog_id: &str, prefix: &str, reason: &str, metrics: Option<&MetricsCollector>, @@ -138,6 +147,9 @@ pub(crate) async fn fail_row( Written::Matched => {} Written::NotMatched(why) | Written::Error(why) => return FailurePath::NoStoreAccess(why), } + if let Err(problem) = fence.check(catalog_id).await { + return FailurePath::Fenced(problem.to_string()); + } match store.delete_prefix(prefix).await { Ok(_) => FailurePath::Deleted, Err(e) => { diff --git a/crates/engine/src/backup/job/writer.rs b/crates/engine/src/backup/job/writer.rs index ba479a168..5a967bc82 100644 --- a/crates/engine/src/backup/job/writer.rs +++ b/crates/engine/src/backup/job/writer.rs @@ -109,8 +109,9 @@ pub(crate) enum Stop { /// or catalog access. ClaimLost(String), /// The store, as the runtime holds it, is unhealthy or is no longer the - /// store the row records ([`Pipeline::store_check`]): FAILED with this - /// reason and no store call, the prefix's deletion included. + /// store the row records ([`Pipeline::store_check`]), or the fence's + /// read right before the first put refused it: FAILED with this reason + /// and no store write, the prefix's deletion included. StoreUnhealthy(String), } diff --git a/crates/engine/src/backup/maintenance.rs b/crates/engine/src/backup/maintenance.rs index 5536c36fb..afd3f62bf 100644 --- a/crates/engine/src/backup/maintenance.rs +++ b/crates/engine/src/backup/maintenance.rs @@ -11,8 +11,11 @@ //! backend measures the heartbeat's age, PostgreSQL the lock first) sees it //! held through a wait and a deletion phase of any length. A heartbeat that //! finds the claim lost (another holder took it over after this one looked -//! dead), or fails, is recorded for the command to stop at before its next -//! destructive step. Dropping the handle without `release` stops the timer +//! dead), fails or outlives its one-period bound is recorded, a claim with +//! no beat confirmed within two periods reads lost too (the type's doc), +//! and the command stops before its next destructive step; right before +//! each step that cannot be undone it sends one beat of its own +//! ([`HeldClaim::recheck`]). Dropping the handle without `release` stops the timer //! (the claim then expires by the stale rule, as a dead holder's does); //! `release` stops it, waits for a heartbeat in flight, then releases. //! @@ -59,44 +62,146 @@ use futures::TryStreamExt; use tokio::sync::watch; use tokio::time::Instant; +use super::StoreFence; + /// A maintenance claim with its heartbeat on a timer. +/// +/// The claim is confirmed by the job's rule (`job/heartbeat.rs`): each beat +/// is bounded by one period ([`MaintenanceClaim::heartbeat_within`]), and a +/// beat that answers `Lost`, fails or outlives its bound records the loss; +/// between beats the claim counts as held while the last confirmed beat is +/// at most two periods old (one for the timer to fire, one for the beat to +/// return). A beat in flight confirms the claim only where the backend's +/// beat parks before its bound +/// ([`MaintenanceClaim::heartbeat_parks_before_its_bound`], SQLite, where +/// it waits on this process's own write lock and the serve lock keeps every +/// other process off the file), for as long as it waits; elsewhere a beat +/// in flight says nothing about the record. Past that, [`Self::lost`] +/// answers as for a lost claim: a process stopped and resumed (SIGSTOP, a +/// paused VM), a timer that stopped running and a beat that hangs all read +/// as lost once two periods pass with no confirmed beat. Every backend's +/// take needs the record's heartbeat older than the stale window, which the +/// config holds to at least three periods, so a holder that reads its claim +/// held has not been taken over. +/// +/// The passive reading is checked before each prefix, row and wait poll; +/// before each step that cannot be undone the command calls +/// [`Self::recheck`], one beat it waits for. What remains is a stop between +/// the re-check's confirmation and the step's own call reaching the catalog +/// or the store, or with that call already sent: the call lands once the +/// process resumes, though another holder may have taken the claim over +/// during the stop. pub struct HeldClaim { claim: Arc, lost: Arc>>, + /// The instant of the last confirmed beat; the take stands for the + /// first. + last_confirmed: Arc>, + /// Whether a beat (the timer's or a re-check's) is in flight. + in_flight: Arc>, + /// Whether the claim's beat can wait outside its bound: a beat in + /// flight confirms the claim only where it can. + parks: bool, + period: Duration, timer: tokio::task::JoinHandle<()>, } +/// Marks a beat in flight for as long as it lives, and clears the mark when +/// dropped: by the beat's return, or by the timer task's end mid-beat (a +/// panic in the backend's beat, an abort). +struct InFlight<'a>(&'a Mutex); + +impl<'a> InFlight<'a> { + fn mark(flag: &'a Mutex) -> Self { + *flag.lock().unwrap_or_else(PoisonError::into_inner) = true; + Self(flag) + } +} + +impl Drop for InFlight<'_> { + fn drop(&mut self) { + *self.0.lock().unwrap_or_else(PoisonError::into_inner) = false; + } +} + +/// Why a beat did not confirm the claim, `None` when it did. +fn beat_failure( + outcome: Option>, + bound: Duration, +) -> Option { + match outcome { + Some(Ok(Heartbeat::Confirmed)) => None, + Some(Ok(Heartbeat::Lost)) => { + Some("the maintenance claim was lost (another holder took it over)".to_owned()) + } + Some(Err(e)) => Some(format!("the maintenance claim's heartbeat failed: {e}")), + None => Some(format!( + "the maintenance claim's heartbeat did not return within {} s", + bound.as_secs_f64() + )), + } +} + impl HeldClaim { /// Hold `claim`, heartbeating it every `interval` from now until /// [`HeldClaim::release`] or the drop (the caller passes the jobs' /// heartbeat period, `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL`, /// which every stale window the config accepts is at least three of). + /// Now stands for the first confirmed beat: the take just stamped the + /// record's heartbeat, so the instant is at or after it by the take's + /// own duration. #[must_use] pub fn start(claim: Box, interval: Duration) -> Self { let claim: Arc = Arc::from(claim); let lost = Arc::new(Mutex::new(None)); + let taken = Instant::now(); + let last_confirmed = Arc::new(Mutex::new(taken)); + let in_flight = Arc::new(Mutex::new(false)); + let parks = claim.heartbeat_parks_before_its_bound(); let timer = tokio::spawn({ let claim = Arc::clone(&claim); let lost = Arc::clone(&lost); + let last_confirmed = Arc::clone(&last_confirmed); + let in_flight = Arc::clone(&in_flight); async move { - let mut ticks = tokio::time::interval(interval); - ticks.tick().await; + // The beats fall one period apart from the take, whenever + // the task first runs. + let mut ticks = tokio::time::interval_at(taken + interval, interval); + ticks.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); loop { ticks.tick().await; - let reason = match claim.heartbeat().await { - Ok(Heartbeat::Confirmed) => continue, - Ok(Heartbeat::Lost) => { - "the maintenance claim was lost (another holder took it over)" - .to_owned() + let failure = { + let _beat = InFlight::mark(&in_flight); + let failure = + beat_failure(claim.heartbeat_within(interval).await, interval); + // The confirmation is recorded before the flight + // ends, so a reader never sees neither. + if failure.is_none() { + *last_confirmed + .lock() + .unwrap_or_else(PoisonError::into_inner) = Instant::now(); } - Err(e) => format!("the maintenance claim's heartbeat failed: {e}"), + failure }; - *lost.lock().unwrap_or_else(PoisonError::into_inner) = Some(reason); + let Some(reason) = failure else { + continue; + }; + lost.lock() + .unwrap_or_else(PoisonError::into_inner) + .get_or_insert(reason); return; } } }); - Self { claim, lost, timer } + Self { + claim, + lost, + last_confirmed, + in_flight, + parks, + period: interval, + timer, + } } /// The claim's fencing token. @@ -105,14 +210,76 @@ impl HeldClaim { self.claim.generation() } - /// Why the claim is no longer held, once a heartbeat said so; `None` - /// while every heartbeat has confirmed it. + /// Why the claim is no longer held (the type's doc): a heartbeat that + /// answered `Lost`, failed or outlived its bound, or no beat confirmed + /// within two periods and, where the beat parks, none in flight. + /// `None` while the claim is confirmed. #[must_use] pub fn lost(&self) -> Option { - self.lost + if let Some(reason) = self + .lost .lock() .unwrap_or_else(PoisonError::into_inner) .clone() + { + return Some(reason); + } + let since = self + .last_confirmed + .lock() + .unwrap_or_else(PoisonError::into_inner) + .elapsed(); + if since <= 2 * self.period { + return None; + } + if self.parks + && *self + .in_flight + .lock() + .unwrap_or_else(PoisonError::into_inner) + { + return None; + } + Some(format!( + "no heartbeat confirmed the maintenance claim within two periods (the last {:.0} s \ + ago)", + since.as_secs_f64() + )) + } + + /// One beat outside the timer, under the period's bound, before a step + /// that cannot be undone: `Ok` when it confirmed the claim (it counts + /// as a confirmed beat), otherwise the reason the claim is lost, which + /// is recorded as the timer records a loss. A claim [`Self::lost`] + /// already reads lost answers at once, with no beat. + /// + /// # Errors + /// + /// The reason, as [`Self::lost`] gives one. + pub async fn recheck(&self) -> Result<(), String> { + if let Some(reason) = self.lost() { + return Err(reason); + } + let failure = { + let _beat = InFlight::mark(&self.in_flight); + let failure = beat_failure(self.claim.heartbeat_within(self.period).await, self.period); + if failure.is_none() { + *self + .last_confirmed + .lock() + .unwrap_or_else(PoisonError::into_inner) = Instant::now(); + } + failure + }; + match failure { + None => Ok(()), + Some(reason) => Err(self + .lost + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get_or_insert(reason) + .clone()), + } } /// Stop the heartbeat, wait for one in flight to end, and release the @@ -325,8 +492,12 @@ async fn busy_jobs( Ok(busy) } -/// Delete one row's objects and the row: the manifest first, then the -/// prefix, then the row through the status machine to a tombstone and out. +/// Delete one row's objects and the row: the deletion fence first +/// (`fence`, which re-reads `extenddb-store.json` and refuses, marking the +/// store unhealthy, once another catalog has adopted the root), before the +/// row of a CREATING or AVAILABLE row is changed and again right before the +/// first deletion, then the manifest, then the prefix, then the row through +/// the status machine to a tombstone and out. /// The row is `row` as the command read it; the status another deleter may /// have moved it to since is read from `start_backup_delete`'s answer, and /// each state the row can be in when the command reaches it ends with no @@ -337,13 +508,37 @@ async fn busy_jobs( /// live one): the row goes first, through the dead-owner removal on the /// catalog's clock, and the prefix only once the row is gone, so an owner /// that heartbeated after the wait leaves the row and its objects together -/// rather than a row over a deleted prefix. A prefix left by a failure -/// between the two steps holds no manifest (a CREATING backup has none -/// yet), so nothing adopts it. +/// rather than a row over a deleted prefix. Nothing adopts the prefix +/// between the two steps while this command holds the claim: every `backup +/// sync` takes the maintenance claim purge holds (on SQLite both run under +/// the serve lock), so no sync reads the rows after the removal and lists +/// the prefix before its deletion. `claim` is re-checked with a beat +/// ([`HeldClaim::recheck`]) right before the CREATING row's removal and +/// right before its prefix's deletion: a command stopped (SIGSTOP, a paused +/// VM) past `stale_job_after_secs` on MongoDB or Cassandra, where another +/// command takes a claim over on the heartbeat's age alone, finds its claim +/// lost there and leaves the row, or the prefix, as it is. What the +/// re-check does not cover is a stop after it confirmed the claim and +/// before the removal or deletion it precedes reached the catalog or the +/// store, or with that call already sent (the type's doc). The rows of +/// other statuses keep their row through the deletion, and sync skips a +/// prefix with a row, so their deletion needs no re-check: a second deleter +/// runs the same idempotent sequence. A +/// prefix a failed deletion leaves after the row's removal is reported as +/// left; if its job had written the manifest before it died (the state the +/// dead-owner duty resolves to AVAILABLE), a later sync adopts it as a +/// complete backup, which the next purge deletes. /// /// # Errors /// -/// The reason the row was left, for the operator: a FAILED or CREATING +/// The reason the row was left, for the operator: the claim lost at a +/// re-check (the CREATING row unchanged, or removed with its prefix left), +/// the fence's refusal +/// (nothing deleted; before the row was changed it is as read, and from the +/// second fence a CREATING row is already removed with its prefix left and +/// an AVAILABLE one is DELETING, which a later purge or the reconciler +/// finishes only once this catalog owns the root again), a FAILED or +/// CREATING /// status another writer moved an AVAILABLE row to (neither follows /// AVAILABLE; the row is left for a look), a restore reading the backup, a /// CREATING row whose owner heartbeated after the wait, a store or catalog @@ -351,12 +546,26 @@ async fn busy_jobs( pub async fn purge_row( storage: &S, store: &dyn BackupStore, + fence: &StoreFence, row: &BackupRow, stale_after: Duration, + claim: &HeldClaim, ) -> Result<(), String> { let arn = BackupArn::parse(&row.backup_arn) .map_err(|e| format!("the row's ARN does not parse: {e}"))?; let key = row.key(); + // The fence before the catalog write as well as before the deletion: a + // refusal here leaves the row as it was read. + if matches!(row.status, BackupStatus::Creating | BackupStatus::Available) { + fence.check_catalog(storage).await.map_err(|problem| { + format!("the deletion was refused: {problem}; the row is unchanged") + })?; + } + if row.status == BackupStatus::Creating { + claim.recheck().await.map_err(|reason| { + format!("stopped before the CREATING row's removal: {reason}; the row is unchanged") + })?; + } if row.status == BackupStatus::Creating && !storage .remove_dead_creating_backup_row(&key, stale_after) @@ -389,6 +598,21 @@ pub async fn purge_row( } } let prefix = arn.store_prefix(); + fence + .check_catalog(storage) + .await + .map_err(|problem| format!("the deletion was refused: {problem}"))?; + if status == BackupStatus::Creating { + // No row names the prefix now: a sync holding a claim taken over + // from this command could adopt it, so the claim is confirmed by a + // beat right before the deletion. + claim.recheck().await.map_err(|reason| { + format!( + "stopped before the prefix's deletion: {reason}; the CREATING row is removed \ + and its objects are left (a later sync adopts them if its manifest is complete)" + ) + })?; + } store .delete_prefix(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")) .await @@ -459,14 +683,17 @@ pub enum Stop { Interrupted(&'static str), } -/// The deletion phase: [`purge_row`] on each of `rows` in order, stopping -/// before the next row (never inside one) once a heartbeat has found the -/// claim lost or failed, or a signal has been received. A row that could not +/// The deletion phase: [`purge_row`] on each of `rows` in order, each behind +/// the deletion fence (`fence`), stopping before the next row (never inside +/// one) once a heartbeat has found the claim lost or failed, or a signal has +/// been received. A row that could not /// be deleted is skipped with its reason and the loop goes on; `deleted` is /// called with each row deleted, for the command's audit line and output. +#[allow(clippy::too_many_arguments)] pub async fn purge_rows( storage: &S, store: &dyn BackupStore, + fence: &StoreFence, rows: &[BackupRow], stale_after: Duration, claim: &HeldClaim, @@ -483,7 +710,7 @@ pub async fn purge_rows( outcome.stopped = Some(Stop::Interrupted(signal)); break; } - match purge_row(storage, store, row, stale_after).await { + match purge_row(storage, store, fence, row, stale_after, claim).await { Ok(()) => { deleted(row); outcome.deleted.push(row.backup_arn.clone()); @@ -841,4 +1068,284 @@ mod tests { "{err}" ); } + + /// A claim whose heartbeat never answers (a half-open connection), or + /// one that parks before its bound as SQLite's does, as `parks` says. + struct StalledBeat { + parks: bool, + } + + impl MaintenanceClaim for StalledBeat { + fn generation(&self) -> u64 { + 1 + } + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(std::future::pending()) + } + fn heartbeat_within( + &self, + bound: Duration, + ) -> BoxFuture<'_, Option>> { + if self.parks { + // Parked on the write lock: the bound has not started. + Box::pin(std::future::pending()) + } else { + Box::pin(async move { tokio::time::timeout(bound, self.heartbeat()).await.ok() }) + } + } + fn heartbeat_parks_before_its_bound(&self) -> bool { + self.parks + } + fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> { + Box::pin(async { Ok(()) }) + } + } + + /// A heartbeat that hangs is bounded by one period: the claim stays + /// confirmed through the first beat's bound and reads lost once it + /// passes (two periods after the take), naming the bound. A beat parked + /// before its bound (SQLite's) keeps the claim confirmed for as long as + /// it waits. + #[tokio::test(start_paused = true)] + async fn a_heartbeat_that_hangs_loses_the_claim_after_two_periods() { + let period = Duration::from_secs(1); + let held = HeldClaim::start(Box::new(StalledBeat { parks: false }), period); + tokio::time::sleep(Duration::from_millis(1_900)).await; + assert_eq!(held.lost(), None, "within two periods"); + tokio::time::sleep(Duration::from_millis(200)).await; + let reason = held.lost().expect("lost after the beat's bound"); + assert!(reason.contains("did not return within 1 s"), "{reason}"); + + let parked = HeldClaim::start(Box::new(StalledBeat { parks: true }), period); + tokio::time::sleep(Duration::from_secs(60)).await; + assert_eq!(parked.lost(), None, "a parked beat confirms while it waits"); + } + + /// A timer that stops running without recording a loss (here, aborted) + /// leaves no beat in flight: the claim reads lost once its last + /// confirmed beat is two periods old, though no beat said so. + #[tokio::test(start_paused = true)] + async fn a_timer_that_stops_loses_the_claim_after_two_periods() { + let fake = Arc::new(FakeClaim::default()); + let held = held_over(&fake, Duration::from_secs(1)); + tokio::time::sleep(Duration::from_millis(1_500)).await; + held.timer.abort(); + tokio::time::sleep(Duration::from_millis(1_000)).await; + assert_eq!( + held.lost(), + None, + "the beat at 1 s is under two periods old" + ); + tokio::time::sleep(Duration::from_millis(1_100)).await; + let reason = held.lost().expect("no beat within two periods"); + assert!(reason.contains("within two periods"), "{reason}"); + } + + /// A claim whose beat never returns and ignores its bound (a timer task + /// stopped mid-beat), recording that it started, and parking before its + /// bound (SQLite's) or not, as `parks` says. + struct HangingBeat { + parks: bool, + started: Arc, + } + + impl MaintenanceClaim for HangingBeat { + fn generation(&self) -> u64 { + 1 + } + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(std::future::pending()) + } + fn heartbeat_within( + &self, + _bound: Duration, + ) -> BoxFuture<'_, Option>> { + self.started + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Box::pin(std::future::pending()) + } + fn heartbeat_parks_before_its_bound(&self) -> bool { + self.parks + } + fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> { + Box::pin(async { Ok(()) }) + } + } + + /// A beat in flight confirms the claim only where the backend's beat + /// parks before its bound. On a paused clock: a beat on schedule that + /// never returns, and a process stopped for three periods and resumed + /// (the clock advanced with the timer unpolled, its missed tick's beat + /// started at once on resume), read lost two periods after the take on + /// a backend whose beat does not park (PostgreSQL, MongoDB, Cassandra), + /// with that beat in flight; where it parks (SQLite) the claim reads + /// held while it waits. + #[tokio::test(start_paused = true)] + async fn a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks() { + let period = Duration::from_secs(1); + for parks in [false, true] { + let started = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let held = HeldClaim::start( + Box::new(HangingBeat { + parks, + started: Arc::clone(&started), + }), + period, + ); + tokio::time::sleep(Duration::from_millis(1_900)).await; + assert_eq!(started.load(std::sync::atomic::Ordering::SeqCst), 1); + assert_eq!(held.lost(), None, "parks {parks}: within two periods"); + tokio::time::sleep(Duration::from_millis(200)).await; + if parks { + assert_eq!(held.lost(), None, "a parked beat confirms while it waits"); + } else { + let reason = held.lost().expect("a beat in flight confirms nothing"); + assert!(reason.contains("within two periods"), "{reason}"); + } + + // Stopped between beats and resumed. + let started = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let held = HeldClaim::start( + Box::new(HangingBeat { + parks, + started: Arc::clone(&started), + }), + period, + ); + tokio::time::advance(3 * period).await; + // The missed tick's beat starts within a few polls of the + // timer task; a timer that counts its ticks from its first run + // rather than the take would never start it, and the cell + // fails here by name rather than spinning. + let mut yields = 0; + while started.load(std::sync::atomic::Ordering::SeqCst) == 0 { + yields += 1; + assert!( + yields < 1_000, + "parks {parks}: the resumed beat did not start within {yields} yields" + ); + tokio::task::yield_now().await; + } + if parks { + assert_eq!(held.lost(), None, "SQLite: the resumed beat parks"); + } else { + let reason = held.lost().expect("the resumed beat confirms nothing"); + assert!(reason.contains("within two periods"), "{reason}"); + } + } + } + + /// A claim whose beat never answers and that implements `heartbeat` + /// alone, so its `heartbeat_within` is the trait's default (what + /// PostgreSQL, MongoDB and Cassandra use). + struct PendingBeat; + + impl MaintenanceClaim for PendingBeat { + fn generation(&self) -> u64 { + 1 + } + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(std::future::pending()) + } + fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> { + Box::pin(async { Ok(()) }) + } + } + + /// The re-check's beat and the timer's run under one period's bound + /// through the trait's default `heartbeat_within`: on a claim whose + /// beat never returns, `recheck` fails after exactly one period naming + /// the bound, and the timer's own beat, due at one period and bounded + /// by one, records the same loss at two periods. Under an unbounded + /// beat neither would return, and sync's insert, the prune's removal + /// and the adoption's rewrite would hang behind the re-check. + #[tokio::test(start_paused = true)] + async fn a_recheck_on_a_pending_beat_fails_at_one_period_through_the_default_bound() { + let period = Duration::from_secs(1); + let held = HeldClaim::start(Box::new(PendingBeat), period); + let started = Instant::now(); + // A guard far past the bound, so an unbounded re-check fails here + // by name instead of hanging the suite. + let reason = tokio::time::timeout(10 * period, held.recheck()) + .await + .expect("the re-check returns within its one-period bound") + .expect_err("the beat never returns"); + assert_eq!(started.elapsed(), period, "one period, no more"); + assert!(reason.contains("did not return within 1 s"), "{reason}"); + assert_eq!(held.lost().as_deref(), Some(reason.as_str())); + + // The timer alone: its beat at one period is bounded by one. + let held = HeldClaim::start(Box::new(PendingBeat), period); + tokio::time::sleep(Duration::from_millis(1_900)).await; + assert_eq!( + held.lost(), + None, + "the first beat is still within its bound" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + let reason = held.lost().expect("the beat outlived its bound"); + assert!(reason.contains("did not return within 1 s"), "{reason}"); + } + + /// A claim whose beat panics, on SQLite's terms (it parks). + struct PanickingBeat; + + impl MaintenanceClaim for PanickingBeat { + fn generation(&self) -> u64 { + 1 + } + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async { panic!("the backend's beat panicked") }) + } + fn heartbeat_parks_before_its_bound(&self) -> bool { + true + } + fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> { + Box::pin(async { Ok(()) }) + } + } + + /// A timer task that ends mid-beat (the beat panicked) clears its + /// flight on the way out, so the claim reads lost two periods after + /// the last confirmed beat even where a beat in flight would park. + #[tokio::test(start_paused = true)] + async fn a_beat_that_panics_leaves_no_flight() { + let held = HeldClaim::start(Box::new(PanickingBeat), Duration::from_secs(1)); + tokio::time::sleep(Duration::from_millis(1_500)).await; + assert_eq!(held.lost(), None); + tokio::time::sleep(Duration::from_millis(600)).await; + let reason = held.lost().expect("no flight left by the panicked beat"); + assert!(reason.contains("within two periods"), "{reason}"); + } + + /// The re-check before a step that cannot be undone: a beat that + /// confirms counts as a confirmed beat; one that finds the claim lost + /// records the loss as the timer would, and a claim already read as + /// lost answers without a beat. + #[tokio::test(start_paused = true)] + async fn a_recheck_confirms_the_claim_or_records_its_loss() { + let fake = Arc::new(FakeClaim::default()); + let held = held_over(&fake, Duration::from_secs(1)); + held.timer.abort(); + tokio::time::sleep(Duration::from_millis(1_500)).await; + let before = fake.beats(); + held.recheck().await.expect("confirmed"); + assert_eq!(fake.beats(), before + 1); + tokio::time::sleep(Duration::from_millis(1_500)).await; + assert_eq!(held.lost(), None, "the re-check counts as a beat"); + tokio::time::sleep(Duration::from_millis(600)).await; + assert!(held.lost().is_some(), "two periods after the re-check"); + let before = fake.beats(); + assert!(held.recheck().await.is_err()); + assert_eq!(fake.beats(), before, "a claim read as lost gets no beat"); + + let fake = Arc::new(FakeClaim::default()); + let held = held_over(&fake, Duration::from_secs(1)); + *fake.lost.lock().unwrap() = true; + let reason = held.recheck().await.expect_err("taken over"); + assert!(reason.contains("another holder took it over"), "{reason}"); + assert_eq!(held.lost().as_deref(), Some(reason.as_str())); + *fake.lost.lock().unwrap() = false; + assert_eq!(held.recheck().await, Err(reason), "a loss stays recorded"); + } } diff --git a/crates/engine/src/backup/reconciler.rs b/crates/engine/src/backup/reconciler.rs index 05eb212db..70605ea5e 100644 --- a/crates/engine/src/backup/reconciler.rs +++ b/crates/engine/src/backup/reconciler.rs @@ -42,6 +42,13 @@ //! one thread and not one per cycle. The check is also selected against //! the shutdown token, so a cycle that is checking when the server stops //! returns at once with no further duty and the drain finishes. +//! - With the startup sync of an in-memory catalog pending and the store +//! healthy after the check: the sync (`super::sync::startup_sync`), +//! bounded by `super::sync::STARTUP_SYNC_TIMEOUT` (60 s) and selected +//! against the shutdown token, so a store that stalls during it delays +//! the duties below by at most the bound; one that stops before it has +//! listed the whole root, reaches the bound, or skips a prefix for a store +//! or catalog error stays pending for the next cycle. //! - Over the records `list_dead_restore_jobs` returns: a RUNNING restore //! job record whose `heartbeat_at` is older than `stale_job_after` (on //! the catalog's clock) has a dead owner. One of kind `legacy` is marked @@ -67,9 +74,10 @@ //! unclaimed write and its target left for the operator, as below (the //! command's wait is satisfied either way: the record is no longer //! RUNNING); a record of a store other than the -//! configured one (no store configured included) is failed naming -//! `backup sync --move-store`, its target removed by the FAILED-target -//! duty next; while the configured store is unhealthy, or not yet checked, +//! configured one (no store configured included) is failed with +//! [`OTHER_STORE`] (configure that store and restore again; no command +//! re-points a record yet), its target removed by the FAILED-target duty +//! next; while the configured store is unhealthy, or not yet checked, //! the record is skipped (one line per cycle, naming which); with no free //! job permit it is skipped to the next cycle; then the backend's //! `claim_restore_dead` is the test and the takeover (a fresh generation, @@ -145,6 +153,16 @@ //! nothing and whose `completed_at` is older than `failed_row_retention` //! (the completed clock) is removed. Every age is the database's //! comparison; the engine holds no clock. +//! - Each deletion of a prefix and each removal of a tombstone, the +//! DELETING rows' included, runs the deletion fence first +//! ([`super::StoreFence`]): `extenddb-store.json` is re-read, and a +//! file that names another catalog (a root adopted since this cycle's +//! check) marks the store unhealthy with nothing deleted. The cycle +//! re-tests the store's health after each store duty, so the duties +//! after the refusal (the dead-owner duty's reads and writes among +//! them) do not run on the root; the refusing duty's later rows are +//! refused with no read of the file (the FAILED-prefix and +//! DELETED-prefix duties still list each prefix first). //! - Not while the maintenance claim is held, on every backend: the //! backend's `sweep_restore_registrations` (a no-op but on Cassandra); //! the progress-row sweep (`sweep_restore_progress` with the table ids of @@ -210,7 +228,7 @@ use futures::TryStreamExt; use super::job::JobEnv; use super::job::failure::{Written, write_status}; use super::restore::{self, OTHER_STORE, TARGET_NOT_EMPTY}; -use super::{BackupRuntime, ConfiguredStore}; +use super::{BackupRuntime, ConfiguredStore, StoreFence}; /// The two durations a reconciler reads from `[backup]`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -476,10 +494,18 @@ where ..CycleReport::default() }; if report.maintenance_held { - tracing::info!( - "backup reconciler: a maintenance claim is held; running only the duties that \ - resolve rows whose owner is dead, the store's check and the DELETING rows" - ); + if state.maintenance_held.is_some() { + tracing::info!( + "backup reconciler: run by a maintenance command before its own work; running \ + only the duties that resolve rows whose owner is dead, the store's check and \ + the DELETING rows" + ); + } else { + tracing::info!( + "backup reconciler: a maintenance claim is held; running only the duties that \ + resolve rows whose owner is dead, the store's check and the DELETING rows" + ); + } } // The store's check first, so every duty of this cycle that reads the @@ -534,6 +560,22 @@ where } } report.store_healthy = Some(store.is_healthy()); + // The startup sync of an in-memory catalog, once its store is + // healthy: an S3 store's first check is this cycle's, and a sync + // the server's start did not finish is left pending for here + // (`super::sync::startup_sync`). + if store.startup_sync_pending() { + let sync = super::sync::startup_sync(&**storage, runtime); + match &state.shutdown { + Some(shutdown) => tokio::select! { + _ = sync => {} + () = shutdown.cancelled() => return Ok(report), + }, + None => { + sync.await; + } + } + } } let mut store_not_ready_logged = false; @@ -669,17 +711,32 @@ where store: &*store.store, metrics: runtime.metrics().map(|m| &**m), }; - report.deletions_finished = finish_deleting_rows(&**storage, store, &identity).await?; - (report.creating_resolved, report.creating_left_to_owner) = - resolve_dead_creating_rows(&**storage, calls, &identity, stale_job_after).await?; - if !report.maintenance_held { + let fence = store.fence(); + // A fence refusal makes the store unhealthy (a root another catalog + // adopted since this cycle's check): the store duties after it do + // not run on the root, the dead-owner duty's reads included. + let healthy = || store.is_healthy(); + report.deletions_finished = + finish_deleting_rows(&**storage, store, &fence, &identity).await?; + if healthy() { + (report.creating_resolved, report.creating_left_to_owner) = + resolve_dead_creating_rows(&**storage, calls, &identity, stale_job_after).await?; + } + if !report.maintenance_held && healthy() { let retention = runtime.job_settings().failed_row_retention; + let duty = Duty { + storage: &**storage, + store: calls, + fence: &fence, + }; report.failed_prefixes_deleted = - delete_stale_failed_prefixes(&**storage, calls, &identity, stale_job_after).await?; - report.deleted_prefixes_swept = - sweep_deleted_prefixes(&**storage, calls, &identity).await?; - report.tombstones_removed = - remove_tombstones(&**storage, calls, &identity, retention).await?; + delete_stale_failed_prefixes(duty, &identity, stale_job_after).await?; + if healthy() { + report.deleted_prefixes_swept = sweep_deleted_prefixes(duty, &identity).await?; + } + if healthy() { + report.tombstones_removed = remove_tombstones(duty, &identity, retention).await?; + } } } @@ -1331,13 +1388,15 @@ async fn resolve_dead_creating_rows( /// a store that refuses the reconciler's calls moves the metric even when /// every cycle completes. #[derive(Clone, Copy)] -struct StoreCalls<'a> { - store: &'a dyn BackupStore, - metrics: Option<&'a MetricsCollector>, +pub(super) struct StoreCalls<'a> { + pub(super) store: &'a dyn BackupStore, + pub(super) metrics: Option<&'a MetricsCollector>, } impl StoreCalls<'_> { - fn failed(self, op: &str, e: StoreError) -> StoreError { + /// `e`, with `BackupStoreErrors` recorded under `op` when metrics are + /// collected. + pub(super) fn failed(self, op: &str, e: StoreError) -> StoreError { if let Some(metrics) = self.metrics { metrics.record_backup_store_error(op); } @@ -1389,10 +1448,100 @@ enum Probe { Incomplete(String), } -/// The manifest probe of the dead-owner duty (module doc). A store error -/// other than an absent object is the `Err`. +/// The manifest probe of the dead-owner duty (module doc): the checks of +/// [`verify_prefix`], with the manifest's ARN required to name the row's +/// backup (its partition included; the region may differ, as it does for a +/// backup `backup sync` adopted from a manifest another region's server +/// wrote, `BackupArn::names_same_backup`) before the prefix is listed. A +/// store error other than an absent object is the `Err`. async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result { let prefix = arn.store_prefix(); + let manifest = match read_verified_manifest(store, &prefix).await? { + Verified::Complete(manifest) => manifest, + Verified::Incomplete(why) => return Ok(Probe::Incomplete(why)), + }; + match manifest.backup_arn() { + Ok(found) if found.names_same_backup(arn) => {} + Ok(found) => { + return Ok(Probe::Incomplete(format!( + "the manifest names {}, not the row's backup", + found.to_arn_string() + ))); + } + Err(e) => { + return Ok(Probe::Incomplete(format!( + "the manifest's ARN is refused: {e}" + ))); + } + } + let listed = store.list_sizes(&prefix).await?; + if let Some(why) = missing_data_file(&manifest, &prefix, &listed) { + return Ok(Probe::Incomplete(why)); + } + Ok(Probe::Complete(StatusFields { + size_bytes: Some(manifest.backup_size_bytes), + item_count: Some(manifest_item_count(&manifest)), + failure_reason: None, + source_table_details_json: serde_json::to_value(&manifest.source_table).ok(), + completed: true, + })) +} + +/// What [`verify_prefix`] found under a prefix. +pub(super) enum Verified { + /// A manifest whose sidecar agrees, that the reader accepts, and whose + /// every data file is listed under the prefix at its recorded size. + Complete(Box), + /// No manifest, or one without its sidecar or not matching it, or one + /// the reader refuses, or a data file missing or of another size; the + /// text says which. + Incomplete(String), +} + +/// The checks a prefix must pass before anything reads it as a backup, in +/// order: `extenddb-manifest.json` read within [`MANIFEST_READ_BOUND`], its +/// sidecar within [`SIDECAR_READ_BOUND`], the sidecar's digest equal to the +/// manifest's, the manifest accepted by +/// [`ExtenddbManifest::from_json_bytes`] (which also checks that its ARN, +/// `source_table` and creation instant agree), then every `data_files` key +/// present under the prefix with its `bytes`. `listed` is the prefix's +/// objects (key to size) when the caller already listed them (sync, which +/// lists the whole root); `None` lists the prefix here. The manifest's ARN +/// is the caller's to compare: the dead-owner duty requires the row's ARN, +/// sync the listed path's components. +/// +/// # Errors +/// +/// A store error other than an absent object. +pub(super) async fn verify_prefix( + store: StoreCalls<'_>, + prefix: &str, + listed: Option<&std::collections::HashMap>, +) -> Result { + let manifest = match read_verified_manifest(store, prefix).await? { + Verified::Complete(manifest) => manifest, + incomplete @ Verified::Incomplete(_) => return Ok(incomplete), + }; + let listed_here; + let listed = match listed { + Some(listed) => listed, + None => { + listed_here = store.list_sizes(prefix).await?; + &listed_here + } + }; + Ok(match missing_data_file(&manifest, prefix, listed) { + Some(why) => Verified::Incomplete(why), + None => Verified::Complete(manifest), + }) +} + +/// The first three checks of [`verify_prefix`]: the manifest and its +/// sidecar read within their bounds, the digest, the reader's acceptance. +async fn read_verified_manifest( + store: StoreCalls<'_>, + prefix: &str, +) -> Result { let manifest = match store .read_object( &format!("{prefix}{EXTENDDB_MANIFEST_FILE}"), @@ -1401,9 +1550,9 @@ async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result bytes, - ReadObject::Absent => return Ok(Probe::Incomplete("no manifest".to_owned())), + ReadObject::Absent => return Ok(Verified::Incomplete("no manifest".to_owned())), ReadObject::Oversize => { - return Ok(Probe::Incomplete(format!( + return Ok(Verified::Incomplete(format!( "a manifest over {MANIFEST_READ_BOUND} bytes" ))); } @@ -1417,69 +1566,55 @@ async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result bytes, ReadObject::Absent => { - return Ok(Probe::Incomplete( + return Ok(Verified::Incomplete( "a manifest without its sidecar".to_owned(), )); } ReadObject::Oversize => { - return Ok(Probe::Incomplete(format!( + return Ok(Verified::Incomplete(format!( "a sidecar over {SIDECAR_READ_BOUND} bytes" ))); } }; if let Err(e) = verify_manifest_checksum(&manifest, &sidecar) { - return Ok(Probe::Incomplete(format!( + return Ok(Verified::Incomplete(format!( "the sidecar does not match: {e}" ))); } - let manifest = match ExtenddbManifest::from_json_bytes(&manifest) { - Ok(manifest) => manifest, - Err(e) => return Ok(Probe::Incomplete(format!("the manifest is refused: {e}"))), - }; - match manifest.backup_arn() { - Ok(found) if found == *arn => {} - Ok(found) => { - return Ok(Probe::Incomplete(format!( - "the manifest names {}, not the row's backup", - found.to_arn_string() - ))); - } - Err(e) => { - return Ok(Probe::Incomplete(format!( - "the manifest's ARN is refused: {e}" - ))); - } - } - let listed = store.list_sizes(&prefix).await?; - for file in &manifest.data_files { - match listed.get(&format!("{prefix}{}", file.key)) { - Some(size) if *size == file.bytes => {} - Some(size) => { - return Ok(Probe::Incomplete(format!( - "data file {} has {size} bytes in the store and {} in the manifest", - file.key, file.bytes - ))); - } - None => { - return Ok(Probe::Incomplete(format!( - "data file {} is not in the store", - file.key - ))); - } - } - } - let item_count = manifest + Ok(match ExtenddbManifest::from_json_bytes(&manifest) { + Ok(manifest) => Verified::Complete(Box::new(manifest)), + Err(e) => Verified::Incomplete(format!("the manifest is refused: {e}")), + }) +} + +/// The first `data_files` entry `listed` lacks or holds at another size, as +/// the reason a prefix is incomplete; `None` when every one is there. +fn missing_data_file( + manifest: &ExtenddbManifest, + prefix: &str, + listed: &std::collections::HashMap, +) -> Option { + manifest + .data_files + .iter() + .find_map(|file| match listed.get(&format!("{prefix}{}", file.key)) { + Some(size) if *size == file.bytes => None, + Some(size) => Some(format!( + "data file {} has {size} bytes in the store and {} in the manifest", + file.key, file.bytes + )), + None => Some(format!("data file {} is not in the store", file.key)), + }) +} + +/// The manifest's item total, the sum over its data files. +pub(super) fn manifest_item_count(manifest: &ExtenddbManifest) -> i64 { + let total = manifest .data_files .iter() .map(|f| f.item_count) .sum::(); - Ok(Probe::Complete(StatusFields { - size_bytes: Some(manifest.backup_size_bytes), - item_count: Some(i64::try_from(item_count).unwrap_or(i64::MAX)), - failure_reason: None, - source_table_details_json: serde_json::to_value(&manifest.source_table).ok(), - completed: true, - })) + i64::try_from(total).unwrap_or(i64::MAX) } /// The most bytes a manifest read collects: a manifest this server wrote is @@ -1527,15 +1662,50 @@ pub(super) async fn read_object( Ok(ReadObject::Bytes(bytes)) } +/// What the duties that delete a prefix or remove a tombstone act through: +/// the catalog, the store's calls, and the deletion fence each runs right +/// before its deletion or removal. +struct Duty<'a, S: ?Sized> { + storage: &'a S, + store: StoreCalls<'a>, + fence: &'a StoreFence, +} + +impl Clone for Duty<'_, S> { + fn clone(&self) -> Self { + *self + } +} + +impl Copy for Duty<'_, S> {} + +impl Duty<'_, S> { + /// The fence before `row`'s deletion or removal: `true` when it passed; + /// a refusal is logged and acts on nothing. + async fn fenced(self, row: &BackupRow, status: &str, what: &str) -> bool { + match self.fence.check_catalog(self.storage).await { + Ok(()) => true, + Err(problem) => { + tracing::error!( + "did not {what} of {status} backup {} (account {}): {problem}", + row.backup_arn, + row.account_id + ); + false + } + } + } +} + /// The FAILED-prefix duty (module doc): FAILED rows whose heartbeat is older /// than `stale_job_after`, their prefix deleted while it lists objects. async fn delete_stale_failed_prefixes( - storage: &S, - store: StoreCalls<'_>, + duty: Duty<'_, S>, identity: &StoreIdentity, stale_job_after: Duration, ) -> Result, StorageError> { - let rows: Vec = storage + let rows: Vec = duty + .storage .list_backup_rows_older_than( BackupStatus::Failed, identity, @@ -1546,7 +1716,7 @@ async fn delete_stale_failed_prefixes( .await?; let mut deleted = Vec::new(); for row in rows { - if delete_prefix_if_listed(store, &row, "FAILED").await { + if delete_prefix_if_listed(duty, &row, "FAILED").await { deleted.push(row.backup_arn); } } @@ -1556,26 +1726,32 @@ async fn delete_stale_failed_prefixes( /// The DELETED-prefix duty (module doc): DELETED rows whose prefix still /// lists objects, the prefix deleted. async fn sweep_deleted_prefixes( - storage: &S, - store: StoreCalls<'_>, + duty: Duty<'_, S>, identity: &StoreIdentity, ) -> Result, StorageError> { - let rows: Vec = storage + let rows: Vec = duty + .storage .list_backup_rows(Some(BackupStatus::Deleted), identity) .try_collect() .await?; let mut swept = Vec::new(); for row in rows { - if delete_prefix_if_listed(store, &row, "DELETED").await { + if delete_prefix_if_listed(duty, &row, "DELETED").await { swept.push(row.backup_arn); } } Ok(swept) } -/// Delete `row`'s prefix when it lists objects; returns whether a deletion -/// ran. A store failure is logged and left for the next cycle. -async fn delete_prefix_if_listed(store: StoreCalls<'_>, row: &BackupRow, status: &str) -> bool { +/// Delete `row`'s prefix when it lists objects, once the fence passed; +/// returns whether a deletion ran. A store failure or the fence's refusal +/// is logged and left for the next cycle. +async fn delete_prefix_if_listed( + duty: Duty<'_, S>, + row: &BackupRow, + status: &str, +) -> bool { + let store = duty.store; let prefix = match BackupArn::parse(&row.backup_arn) { Ok(arn) => arn.store_prefix(), Err(e) => { @@ -1597,6 +1773,9 @@ async fn delete_prefix_if_listed(store: StoreCalls<'_>, row: &BackupRow, status: return false; } } + if !duty.fenced(row, status, "delete the prefix").await { + return false; + } match store.delete_prefix(&prefix).await { Ok(n) => { tracing::info!( @@ -1622,11 +1801,11 @@ async fn delete_prefix_if_listed(store: StoreCalls<'_>, row: &BackupRow, status: /// `completed_at` is older than `retention` and whose prefix lists nothing /// are removed. async fn remove_tombstones( - storage: &S, - store: StoreCalls<'_>, + duty: Duty<'_, S>, identity: &StoreIdentity, retention: Duration, ) -> Result, StorageError> { + let Duty { storage, store, .. } = duty; let mut removed = Vec::new(); for status in [BackupStatus::Failed, BackupStatus::Deleted] { let rows: Vec = storage @@ -1657,6 +1836,14 @@ async fn remove_tombstones( continue; } } + // The empty listing is a fact about this catalog's root only + // while the root is this catalog's. + if !duty + .fenced(&row, &status.to_string(), "remove the row") + .await + { + continue; + } match storage.remove_backup_row(&row.key()).await { Ok(true) => { tracing::info!( @@ -1686,6 +1873,7 @@ async fn remove_tombstones( async fn finish_deleting_rows( storage: &S, store: &super::ConfiguredStore, + fence: &StoreFence, identity: &extenddb_storage::backup_catalog::StoreIdentity, ) -> Result, StorageError> { let rows: Vec<_> = storage @@ -1704,7 +1892,7 @@ async fn finish_deleting_rows( continue; } }; - match super::finish_backup_delete(storage, &store.calls("deletion"), &arn).await { + match super::finish_backup_delete(storage, &store.calls("deletion"), fence, &arn).await { Ok(true) => { tracing::info!( "finished the deletion of backup {} (account {})", diff --git a/crates/engine/src/backup/restore.rs b/crates/engine/src/backup/restore.rs index 23d774482..2615ab367 100644 --- a/crates/engine/src/backup/restore.rs +++ b/crates/engine/src/backup/restore.rs @@ -7,7 +7,7 @@ //! The request ([`request`]) runs once `check_catalog_row_for_restore` has //! accepted the backup's row: it takes the job permit, reads the sidecar and //! the manifest from the store (a missing one is `BackupNotFoundException` -//! and an error line naming `backup sync --prune-missing`; each read +//! and an error line naming the remedies; each read //! collects at most its bound, [`super::reconciler::SIDECAR_READ_BOUND`] and //! [`super::reconciler::MANIFEST_READ_BOUND`], and an object over it is a //! `ValidationException` naming a corrupt prefix, as a bad sidecar is), @@ -146,7 +146,7 @@ pub const TARGET_NOT_EMPTY: &str = "target_not_empty"; /// The reason the reconciler writes on a RUNNING record whose store is not /// the configured one. -pub const OTHER_STORE: &str = "the restore reads a store other than the configured one; backup sync --move-store re-points it"; +pub const OTHER_STORE: &str = "the restore reads a store other than the configured one; configure that store in [backup.store] and restore again"; /// What the request resolved before it spawned the task. pub(crate) struct RestoreRequestParts { @@ -199,10 +199,20 @@ where let read = async |name: &str, bound: u64| -> Result, DynamoDbError> { match super::reconciler::read_object(&*env.store, &format!("{prefix}{name}"), bound).await { Ok(ReadObject::Bytes(bytes)) => Ok(bytes), + Ok(ReadObject::Absent) if name == EXTENDDB_MANIFEST_FILE => { + tracing::error!( + "backup {arn_text} (account {account_id}): its row is AVAILABLE but the store \ + holds no {name} under its prefix; put the prefix back, or, unless it is the \ + store's newest backup (whose missing manifest makes the store unhealthy), \ + remove the row with backup sync --prune-missing" + ); + Err(backup_not_found(&arn_text)) + } Ok(ReadObject::Absent) => { tracing::error!( "backup {arn_text} (account {account_id}): its row is AVAILABLE but the store \ - holds no {name} under its prefix; backup sync --prune-missing marks the row" + holds no {name} under its prefix; put the prefix back (backup sync \ + --prune-missing removes only a row whose manifest is gone)" ); Err(backup_not_found(&arn_text)) } @@ -230,14 +240,15 @@ where })?; // The parser ran the version gate and the identity rule (the ARN's // table is `source_table.table_name`); what is left is that the - // manifest's backup is the row's. + // manifest's backup is the row's, in whichever region the manifest + // spells it (a backup adopted from another region's server). let manifest = ExtenddbManifest::from_json_bytes(&manifest_bytes).map_err(|e| { DynamoDbError::ValidationException(format!( "Backup {arn_text} has a manifest this server cannot restore from: {e}" )) })?; match manifest.backup_arn() { - Ok(named) if named == *arn => {} + Ok(named) if named.names_same_backup(arn) => {} Ok(named) => { return Err(DynamoDbError::ValidationException(format!( "Backup {arn_text} has a manifest that names another backup, {}", @@ -1319,7 +1330,8 @@ type Prepared = ( ); /// The manifest of a (manifest, sidecar) pair that verifies and names -/// `arn`; the `Err` says what failed. +/// `arn`'s backup (in any region, `BackupArn::names_same_backup`); the +/// `Err` says what failed. fn verified_manifest( manifest_bytes: &[u8], sidecar_bytes: &[u8], @@ -1330,7 +1342,7 @@ fn verified_manifest( let manifest = ExtenddbManifest::from_json_bytes(manifest_bytes) .map_err(|e| format!("the manifest is refused: {e}"))?; match manifest.backup_arn() { - Ok(named) if named == *arn => Ok(manifest), + Ok(named) if named.names_same_backup(arn) => Ok(manifest), Ok(named) => Err(format!( "the manifest names another backup, {}", named.to_arn_string() diff --git a/crates/engine/src/backup/runtime.rs b/crates/engine/src/backup/runtime.rs index f29b468c7..7ee653fa6 100644 --- a/crates/engine/src/backup/runtime.rs +++ b/crates/engine/src/backup/runtime.rs @@ -29,17 +29,25 @@ //! mount answers, which dropping the future cannot end, so a filesystem //! root's calls run as tasks of their own that the abandoned call does not //! take with it, and the store remembers that an abandoned call is in flight -//! ([`ConfiguredStore::check_call_in_flight`]): no call starts while one is +//! ([`ConfiguredStore::check_call_in_flight`]; a call is abandoned at its +//! bound or when its caller drops it first, as a run bounded as a whole +//! does): no call starts while one is //! (the next [`check_configured_store`] records the same `unreachable` the -//! abandoned one did), so a mount that stays stalled parks one thread, not -//! one per cycle. A filesystem root's calls also run one at a time: a call +//! abandoned one did, its text naming the caller whose call has not +//! returned), so a mount that stays stalled parks one thread, not one per +//! cycle. A filesystem root's calls also run one at a time: a call //! that finds another running (a DeleteBackup beside a check, two //! DeleteBackups together) waits for it under its own bound, so calls that //! start before the first reaches its bound park no thread of their own //! and answer at the bound with the same transport failure. The deletions //! DeleteBackup and the reconciler run against the store go through the //! same adapter ([`BoundedCalls`]), under the same bound and rules. The -//! thread costs nothing at exit: the binary ends its runtime with a bounded +//! deletion fence's read of the identity file ([`StoreFence`]) runs under +//! the same bound but in a slot and with an in-flight mark of its own, so a +//! deletion that holds the calls' slot for seconds (a prefix of many +//! objects) does not hold a CreateBackup's fence read behind it; a mount +//! that stays stalled parks at most two threads, one per slot. The thread +//! costs nothing at exit: the binary ends its runtime with a bounded //! shutdown that detaches it. use std::collections::HashMap; @@ -176,6 +184,10 @@ pub struct BackupRuntime { /// of the same request meets while the first registration is still in /// flight ([`Self::begin_restore_registration`]). restore_registrations: Arc>>, + /// The server's region, which `backup sync` compares an adopted ARN's + /// region with (a backup from another region is adopted and logged); + /// `None` until set, when only the partition is compared. + region: Option, } /// The registration outcome a waiting retry reads: `None` until the @@ -266,20 +278,40 @@ pub struct ConfiguredStore { /// manifests starts with ([`Self::with_manifest_names`]; empty until /// set). pub key_prefix: String, - store_id: RwLock>, - health: RwLock, + /// The id the last passing check settled; shared with every + /// [`StoreFence`] of the store, which compares the file's id with it. + store_id: Arc>>, + /// Shared with every [`StoreFence`] of the store, which records the + /// problem it finds here. + health: Arc>, + /// This instance's catalog id, read once by the first fence check that + /// needs it and kept for the process. A catalog re-created under a + /// running process (`destroy` and `init` beside it, which MongoDB and + /// Cassandra allow) gets a new id the cache does not see; the fence then + /// finds a root that names the new catalog foreign and refuses every + /// deletion and CreateBackup until the process restarts, which is the + /// safe direction. + catalog_id: Arc>, /// Whether the store's `validate` has passed; the checks run it first /// until it has. validated: AtomicBool, - /// Whether a call on a filesystem root abandoned at its bound is still - /// running (module doc): set when the call is abandoned, cleared by the - /// call's own task when the mount answers. Never set on an S3 store. - check_call_in_flight: Arc, + /// The call on a filesystem root abandoned at its bound that is still + /// running (module doc), named by its caller: set when the call is + /// abandoned, cleared by the call's own task when the mount answers. + /// Never set on an S3 store. + check_call_in_flight: Arc, /// The one slot a filesystem root's calls run in (module doc): a call /// waits for it under its bound, held by the call's task until the /// store answers, so callers that start before the first call reaches /// its bound park no thread of their own. Unused on an S3 store. call_slot: Arc, + /// The deletion fence's own in-flight mark and slot (module doc): its + /// read of the identity file never waits behind a deletion or a check, + /// and a fence read abandoned at its bound refuses the next fence reads + /// at once, as the checks' mark refuses the checks. Unused on an S3 + /// store. + fence_in_flight: Arc, + fence_slot: Arc, /// Whether the store is the defaulted SQLite root beside the database /// file, whose first failed check with no rows for it is a warning that /// leaves the store unconfigured ([`ConfiguredStore::set_aside`]) rather @@ -288,6 +320,19 @@ pub struct ConfiguredStore { /// Set once a defaulted root's first check failed with no rows for it: /// the runtime then answers as a server with no store configured. set_aside: AtomicBool, + /// Whether this instance's catalog is an in-memory SQLite database: the + /// identity file its checks write says so, and its checks adopt a root + /// another such catalog owned (`store_identity`'s module doc). + ephemeral_catalog: bool, + /// Whether the startup sync of an in-memory catalog is still to run + /// ([`crate::backup::sync::startup_sync`]): set with + /// `ephemeral_catalog`, cleared when a sync over a healthy store has + /// listed the whole root. + startup_sync_pending: AtomicBool, + /// Whether a run of the startup sync has reported: listed the whole + /// root and logged its line of counts. Until one has, a run logs each + /// prefix it skipped at INFO; after, at DEBUG. + startup_sync_reported: AtomicBool, } /// The restore spool of an S3 store. @@ -370,13 +415,19 @@ impl ConfiguredStore { root, store, spool, - store_id: RwLock::new(store_id), - health: RwLock::new(health), + store_id: Arc::new(RwLock::new(store_id)), + health: Arc::new(RwLock::new(health)), + catalog_id: Arc::new(std::sync::OnceLock::new()), validated: AtomicBool::new(validated), - check_call_in_flight: Arc::new(AtomicBool::new(false)), + check_call_in_flight: Arc::new(InFlightMark::default()), call_slot: Arc::new(Semaphore::new(1)), + fence_in_flight: Arc::new(InFlightMark::default()), + fence_slot: Arc::new(Semaphore::new(1)), defaulted: false, set_aside: AtomicBool::new(false), + ephemeral_catalog: false, + startup_sync_pending: AtomicBool::new(false), + startup_sync_reported: AtomicBool::new(false), } } @@ -398,13 +449,19 @@ impl ConfiguredStore { root, store, spool, - store_id: RwLock::new(None), - health: RwLock::new(StoreHealth::Checking), + store_id: Arc::new(RwLock::new(None)), + health: Arc::new(RwLock::new(StoreHealth::Checking)), + catalog_id: Arc::new(std::sync::OnceLock::new()), validated: AtomicBool::new(false), - check_call_in_flight: Arc::new(AtomicBool::new(false)), + check_call_in_flight: Arc::new(InFlightMark::default()), call_slot: Arc::new(Semaphore::new(1)), + fence_in_flight: Arc::new(InFlightMark::default()), + fence_slot: Arc::new(Semaphore::new(1)), defaulted: false, set_aside: AtomicBool::new(false), + ephemeral_catalog: false, + startup_sync_pending: AtomicBool::new(false), + startup_sync_reported: AtomicBool::new(false), } } @@ -450,6 +507,54 @@ impl ConfiguredStore { self } + /// The same store for an instance whose catalog is an in-memory SQLite + /// database (`ephemeral_catalog`): its checks write the marker and + /// adopt a root another in-memory catalog owned, and the startup sync + /// is pending ([`crate::backup::sync::startup_sync`]). + #[must_use] + pub fn with_ephemeral_catalog(mut self, ephemeral_catalog: bool) -> Self { + self.ephemeral_catalog = ephemeral_catalog; + self.startup_sync_pending = AtomicBool::new(ephemeral_catalog); + self + } + + /// Whether this instance's catalog is an in-memory SQLite database. + #[must_use] + pub fn ephemeral_catalog(&self) -> bool { + self.ephemeral_catalog + } + + /// Whether the startup sync is still to run. + #[must_use] + pub fn startup_sync_pending(&self) -> bool { + self.startup_sync_pending.load(Ordering::Acquire) + } + + /// Take the pending startup sync: `true` once, for the caller that runs + /// it, so two callers (the server's start and a reconciler cycle) never + /// run it at once. A caller whose sync did not list the whole root + /// hands it back with [`Self::retry_startup_sync`]. + pub(crate) fn take_startup_sync(&self) -> bool { + self.startup_sync_pending.swap(false, Ordering::AcqRel) + } + + /// Whether a run of the startup sync has reported (listed the whole root + /// and logged its line of counts), so a later run logs its lasting skips + /// at DEBUG. + pub(crate) fn startup_sync_reported(&self) -> bool { + self.startup_sync_reported.load(Ordering::Acquire) + } + + /// Record that a run of the startup sync has reported. + pub(crate) fn record_startup_sync_report(&self) { + self.startup_sync_reported.store(true, Ordering::Release); + } + + /// Hand a taken startup sync back, for the next reconciler cycle. + pub(crate) fn retry_startup_sync(&self) { + self.startup_sync_pending.store(true, Ordering::Release); + } + /// The store's identity, once a check has read its id. #[must_use] pub fn identity(&self) -> Option { @@ -526,36 +631,27 @@ impl ConfiguredStore { } Err(problem) => StoreHealth::Unhealthy(problem), }; - let mut health = self - .health - .write() - .unwrap_or_else(std::sync::PoisonError::into_inner); - if health.category() == next.category() { - *health = next; - return false; - } - match (&*health, &next) { - (StoreHealth::Checking, StoreHealth::Healthy) => tracing::info!( - "backup store ({}) {} passed its check (store id {}); it serves restores and \ - deletes of the backups it holds", - self.kind, - self.root, - self.identity().map(|i| i.store_id).unwrap_or_default() - ), - (_, StoreHealth::Healthy) => { - tracing::info!("backup store ({}) is healthy again", self.kind); - } - (_, StoreHealth::Unhealthy(problem)) => tracing::error!( - "backup store ({}) {} is unhealthy: {problem}; restore and delete of a backup \ - in the store are refused until the check passes; the check runs again each \ - reconciler cycle", - self.kind, - self.root - ), - (_, StoreHealth::Checking) => {} + transition(self.kind, &self.root, &self.health, next, || { + self.identity().map(|i| i.store_id).unwrap_or_default() + }) + } + + /// The deletion fence of this store ([`StoreFence`]), sharing its + /// health. + #[must_use] + pub fn fence(&self) -> StoreFence { + StoreFence { + store: Arc::clone(&self.store), + kind: self.kind, + root: self.root.clone(), + health: Arc::clone(&self.health), + store_id: Arc::clone(&self.store_id), + catalog_id: Arc::clone(&self.catalog_id), + in_flight: Arc::clone(&self.fence_in_flight), + call_slot: Arc::clone(&self.fence_slot), + park: self.kind == StoreKind::Filesystem, + bound: self.check_timeout(), } - *health = next; - true } /// Whether the store's `validate` has passed. @@ -578,7 +674,14 @@ impl ConfiguredStore { /// Whether a call abandoned at its bound has not returned (module doc). #[must_use] pub fn check_call_in_flight(&self) -> bool { - self.check_call_in_flight.load(Ordering::Acquire) + self.check_call_in_flight.caller().is_some() + } + + /// Whether a deletion fence's read abandoned at its bound has not + /// returned (its own mark, [`StoreFence`]). + #[must_use] + pub fn fence_read_in_flight(&self) -> bool { + self.fence_in_flight.caller().is_some() } /// The store as a check, a command's identity read or a deletion calls @@ -586,13 +689,24 @@ impl ConfiguredStore { /// in-flight rule (module doc), and nothing else under the bound, so a /// slow catalog read beside the calls does not count against it. `what` /// names the caller in the transport failure a call that does not finish - /// is recorded as ("the check did not finish within 5 s"). - pub(crate) fn calls(&self, what: &'static str) -> BoundedCalls<'_> { + /// is recorded as ("the check did not finish within 5 s") and, on a + /// filesystem root, in the refusal of every call that follows while the + /// abandoned one has not returned ("an abandoned check has not + /// returned"). + pub fn calls(&self, what: &'static str) -> BoundedCalls<'_> { + self.calls_within(what, self.check_timeout()) + } + + /// [`Self::calls`] under `bound` in place of the check's: the startup + /// sync's calls, whose listing of the whole root runs under the sync's + /// own bound, in the same slot and under the same in-flight rule as the + /// checks and the deletions. + pub(crate) fn calls_within(&self, what: &'static str, bound: Duration) -> BoundedCalls<'_> { BoundedCalls { store: &self.store, in_flight: &self.check_call_in_flight, slot: &self.call_slot, - bound: self.check_timeout(), + bound, what, // A filesystem root's calls are blocking threads, which dropping // the future cannot end; an S3 store's are the SDK's requests, @@ -603,6 +717,259 @@ impl ConfiguredStore { } } +/// Set `health` to `next`, logging a transition between categories once +/// ([`ConfiguredStore::record_check`]); `store_id` names the id in the +/// first healthy line. Returns whether the category changed. +fn transition( + kind: StoreKind, + root: &str, + health: &RwLock, + next: StoreHealth, + store_id: impl FnOnce() -> String, +) -> bool { + let mut health = health + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if health.category() == next.category() { + *health = next; + return false; + } + match (&*health, &next) { + (StoreHealth::Checking, StoreHealth::Healthy) => tracing::info!( + "backup store ({kind}) {root} passed its check (store id {}); it serves restores \ + and deletes of the backups it holds", + store_id() + ), + (_, StoreHealth::Healthy) => { + tracing::info!("backup store ({kind}) is healthy again"); + } + (_, StoreHealth::Unhealthy(problem)) => tracing::error!( + "backup store ({kind}) {root} is unhealthy: {problem}; restore and delete of a \ + backup in the store are refused until the check passes; the check runs again each \ + reconciler cycle" + ), + (_, StoreHealth::Checking) => {} + } + *health = next; + true +} + +/// The `ValidationException` an operation that touches a store of `kind` +/// answers while its health is not [`StoreHealth::Healthy`]; `None` when +/// it is. +fn health_refusal(kind: StoreKind, health: &StoreHealth) -> Option { + match health { + StoreHealth::Healthy => None, + StoreHealth::Checking => Some(DynamoDbError::ValidationException(format!( + "The {kind} backup store configured on this server has not passed a check yet (its \ + first check is pending); the operation is refused until it does" + ))), + StoreHealth::Unhealthy(problem) => Some(DynamoDbError::ValidationException(format!( + "The {kind} backup store configured on this server is unhealthy ({}); the operation \ + is refused until the store's check passes", + problem.category() + ))), + } +} + +/// The deletion fence of the configured store: a re-read of +/// `extenddb-store.json` right before a call that deletes from the store or +/// acts on its rows because of what the store lists, and before a +/// CreateBackup's writes. Its call sites: DeleteBackup's DELETING write and +/// its deletion of the backup's objects; purge's row change (the removal of +/// a dead CREATING row, the DELETING write) and its deletion of the prefix, +/// one read before the two; the reconciler's deletion of a DELETING row's +/// objects, its FAILED-prefix and DELETED-prefix duties and its removal of +/// tombstones; the backup job's failure path; CreateBackup's request, +/// before its row (a fast refusal); and the backup job, right before its +/// first put. `backup sync --prune-missing`'s removal of a row is not +/// fenced: it deletes no object, and a root it does not own leaves it no +/// bytes to lose. +/// +/// A store's health is the last check's, and the check runs once per +/// reconciler cycle; a root another catalog adopted since +/// (`extenddb backup sync --adopt-store`, or a second in-memory server's +/// startup) would otherwise be written and deleted under the new owner's +/// rows until this instance's next check. The fence reads the file (one +/// `get` under the store's bound, through [`BoundedCalls`] in the fence's +/// own slot, so on a filesystem root it waits behind another fence read and +/// never behind a deletion, and parks no second thread beside a fence read +/// abandoned at its bound) and, when it names another catalog, records +/// [`StoreProblem::ForeignCatalog`] on the store, so every operation that +/// needs a healthy store is refused from then on, and refuses the call. A +/// file that names this catalog under another store id than the one the +/// store's check settled (a root replaced by another root of the same +/// catalog) is recorded and refused the same way, as +/// [`StoreProblem::IdMismatch`]. A root with no file is recorded as the +/// next check would record it, [`StoreProblem::FileMissingWithRows`] with +/// the store's settled id (every fenced call acts on a row of this catalog +/// for the root), and a file the store refuses to serve or that does not +/// parse as the problem the read names ([`StoreProblem::Denied`], +/// [`StoreProblem::Validate`], [`StoreProblem::FileUnreadable`]); each +/// refuses the call. Only a read that failed in transport (the store +/// unreachable, the read past its bound, a fence read abandoned earlier +/// still in flight) refuses the call without changing the health: the next +/// check decides. A store that is not healthy refuses at once, with no +/// read. DeleteBackup takes a refusal that recorded nothing, at its read +/// before the DELETING write, as a store failure: it writes DELETING and +/// answers DELETED, and its read before the deletion refuses the deletion +/// (`delete_catalog_backup`). CreateBackup answers a refusal that recorded +/// a problem as [`BackupRuntime::require_healthy_store`] answers an +/// unhealthy store, and one that recorded nothing as an internal error. +/// +/// It narrows the window to the gap between one read and the call that +/// follows it; it does not close it. A put of the file that lands in that +/// gap is not seen. On S3 that gap is more than one request: a +/// `delete_prefix` is a listing and then a `DeleteObjects`, each under the +/// SDK's retries, so a throttled listing keeps it open for seconds. A +/// check that read the file before the adoption and finishes after a fence +/// recorded [`StoreProblem::ForeignCatalog`] records its own outcome over +/// it, healthy, until the next cycle's check; each deletion is still +/// fenced in that time, but a restore is accepted. +#[derive(Clone)] +pub struct StoreFence { + store: Arc, + kind: StoreKind, + root: String, + health: Arc>, + store_id: Arc>>, + catalog_id: Arc>, + /// The fence's own in-flight mark and call slot, shared by every fence + /// of the store and by nothing else ([`BoundedCalls`]). + in_flight: Arc, + call_slot: Arc, + park: bool, + bound: Duration, +} + +impl StoreFence { + /// This instance's catalog id: read from `catalog` once, then the + /// store's copy. + /// + /// # Errors + /// + /// [`StoreProblem::Catalog`] when the catalog cannot be read. + pub async fn catalog_id( + &self, + catalog: &C, + ) -> Result { + if let Some(id) = self.catalog_id.get() { + return Ok(id.clone()); + } + let id = catalog + .catalog_id() + .await + .map_err(|e| StoreProblem::Catalog(e.to_string()))?; + Ok(self.catalog_id.get_or_init(|| id).clone()) + } + + /// The fence for `catalog`'s id ([`Self::check`]). + /// + /// # Errors + /// + /// As [`Self::check`], or [`StoreProblem::Catalog`] when the id cannot + /// be read. + pub async fn check_catalog( + &self, + catalog: &C, + ) -> Result<(), StoreProblem> { + let catalog_id = self.catalog_id(catalog).await?; + self.check(&catalog_id).await + } + + /// The fence (type doc) for the catalog whose id is `catalog_id`. + /// + /// # Errors + /// + /// The store's problem when it is not healthy; + /// [`StoreProblem::ForeignCatalog`], recorded on the store, when the + /// file names another catalog, and [`StoreProblem::IdMismatch`], + /// recorded, when it names this one under another store id; + /// [`StoreProblem::FileMissingWithRows`], recorded, when the root has no + /// file; the read's problem, recorded, when the store refused the read + /// or the file does not parse; and the read's transport failure, not + /// recorded, when the file could not be read within the bound or the + /// store could not be reached. + pub async fn check(&self, catalog_id: &str) -> Result<(), StoreProblem> { + match &*self + .health + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + { + StoreHealth::Healthy => {} + StoreHealth::Checking => { + return Err(StoreProblem::Transport( + "the store's first check is pending".to_owned(), + )); + } + StoreHealth::Unhealthy(problem) => return Err(problem.clone()), + } + let via = BoundedCalls { + store: &self.store, + in_flight: &self.in_flight, + slot: &self.call_slot, + bound: self.bound, + what: "identity file read", + park: self.park, + timed_out: AtomicBool::new(false), + }; + let read = store_identity::read_identity_file(&via).await; + if via.timed_out() { + return Err(via.did_not_finish_problem()); + } + let known = || { + self.store_id + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + }; + let problem = match read { + // The store could not be reached, or an earlier fence read is + // still in flight: the next check decides. + Err(transport @ StoreProblem::Transport(_)) => return Err(transport), + // The store refused the read, or the file does not parse: what + // the next check's read would record. + Err(problem) => problem, + Ok(Some(file)) if file.catalog_id != catalog_id => StoreProblem::ForeignCatalog { + file_catalog_id: file.catalog_id, + catalog_id: catalog_id.to_owned(), + }, + Ok(Some(file)) => match known() { + Some(known) if file.store_id != known => StoreProblem::IdMismatch { + file_id: file.store_id, + row_ids: vec![known], + }, + _ => return Ok(()), + }, + // No file: the next check finds the rows of this catalog for + // the root (the fenced call's row among them) under no file. + Ok(None) => StoreProblem::FileMissingWithRows(known().into_iter().collect()), + }; + transition( + self.kind, + &self.root, + &self.health, + StoreHealth::Unhealthy(problem.clone()), + String::new, + ); + Err(problem) + } + + /// The refusal an operation answers once the fence refused it: the + /// store's health as the fence left it, as + /// [`BackupRuntime::require_healthy_store`] answers it; `None` while the + /// store is still healthy (the read failed in transport, which the next + /// check decides), when the caller answers as for a failed store call. + pub(crate) fn refusal(&self) -> Option { + let health = self + .health + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone(); + health_refusal(self.kind, &health) + } +} + /// The check of the configured store a reconciler cycle, and a command that /// needs the store's identity, run and record through /// [`ConfiguredStore::record_check`]: `validate` first while it has not @@ -638,16 +1005,13 @@ pub async fn check_configured_store( catalog: &C, backend: &str, ) -> Result { - if store.check_call_in_flight() { + if let Some(caller) = store.check_call_in_flight.caller() { tracing::info!( - "backup store ({}) check was not started: a call of the check abandoned at an \ - earlier cycle has not returned (the store is still not answering); the next cycle \ - checks again", + "backup store ({}) check was not started: an abandoned {caller} has not returned \ + (the store is still not answering); the next cycle checks again", store.kind ); - return Err(StoreProblem::Transport( - "a call of an earlier check has not returned".to_owned(), - )); + return Err(StoreProblem::Transport(InFlightMark::refusal(caller))); } let via = store.calls("check"); let outcome = check_through(store, &via, catalog, backend).await; @@ -677,10 +1041,22 @@ async fn check_through( } } match store.identity() { - Some(identity) => store_identity::recheck(via, &identity, catalog, backend) + Some(identity) => { + store_identity::recheck(via, &identity, catalog, backend, store.ephemeral_catalog) + .await + .map(|()| identity) + } + None => { + store_identity::establish( + via, + store.kind, + &store.root, + catalog, + backend, + store.ephemeral_catalog, + ) .await - .map(|()| identity), - None => store_identity::establish(via, store.kind, &store.root, catalog, backend).await, + } } } @@ -708,12 +1084,13 @@ pub async fn read_identity_bounded( read } -/// The configured store as the checks, a command's identity read and the -/// deletions call it (module doc): each call runs under `bound`, and a call +/// The configured store as the checks, a command's identity read, the +/// deletion fence and the deletions call it (module doc): each call runs under `bound`, and a call /// that does not finish is a transport failure naming `what` and the bound. /// On a filesystem root (`park`) each call runs as a task of its own, owning -/// the store handle, so a call abandoned at its bound leaves the task -/// running until the mount answers, marked in flight until then; a call that +/// the store handle, so a call abandoned at its bound, or whose caller +/// drops it before the bound, leaves the task running until the mount +/// answers, marked in flight until then; a call that /// finds the mark set (a call abandoned earlier has not returned) is refused /// at once as a transport failure, so no second thread is parked beside the /// first. The root's calls run one at a time (`slot`): a call that finds @@ -724,11 +1101,13 @@ pub async fn read_identity_bounded( /// returns, and finishing when the first does. On an S3 /// store the call is the SDK's request, which the dropped future ends, so /// nothing is marked and nothing waits. The check calls `validate`, `get`, -/// `head` and `put_if_absent`, the deletions `delete_prefix` and `list`; -/// `put` is forwarded for the trait's sake and bounded the same way. -pub(crate) struct BoundedCalls<'a> { +/// `head` and `put_if_absent`, and `put` when an in-memory catalog adopts +/// the root (`store_identity::adopt`) or `--adopt-store` rewrites the file; +/// the fence calls `get`; the deletions call `delete_prefix` and `list`; the +/// startup sync calls `list`, `get` and `head`. +pub struct BoundedCalls<'a> { store: &'a Arc, - in_flight: &'a Arc, + in_flight: &'a Arc, slot: &'a Arc, bound: std::time::Duration, what: &'static str, @@ -736,13 +1115,49 @@ pub(crate) struct BoundedCalls<'a> { timed_out: AtomicBool, } +/// The in-flight mark of a filesystem root's calls (module doc): the caller +/// whose abandoned call still runs ([`BoundedCalls`]'s `what`: the check, a +/// deletion, the startup sync's store call, the fence's identity file +/// read), or nothing. The refusal of the next call names it. +#[derive(Default)] +pub(crate) struct InFlightMark(std::sync::Mutex>); + +impl InFlightMark { + /// The caller whose abandoned call has not returned, if any. + fn caller(&self) -> Option<&'static str> { + *self + .0 + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + } + + fn set(&self, what: &'static str) { + *self + .0 + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(what); + } + + fn clear(&self) { + *self + .0 + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; + } + + /// The text a call refused on the mark carries, naming `caller`. + fn refusal(caller: &str) -> String { + format!("an abandoned {caller} has not returned") + } +} + /// The call task's end, by any exit: clears the in-flight mark under the /// call's own lock, so a caller that abandons the call at its bound marks it /// only while the task still runs (a task that ended between the timeout's /// last poll and the caller's mark would otherwise leave the mark set with /// nothing left to clear it, and every later check refused until a restart). struct ClearOnDrop { - in_flight: Arc, + in_flight: Arc, running: Arc>, } @@ -753,13 +1168,40 @@ impl Drop for ClearOnDrop { .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); *running = false; - self.in_flight.store(false, Ordering::Release); + self.in_flight.clear(); + } +} + +/// The caller's side of a parked call: dropped while armed, it marks the +/// call in flight if its task still runs, under the task's lock (a task +/// that has already ended, its guard having cleared the mark, is not marked +/// again). +struct MarkIfAbandoned { + in_flight: Arc, + running: Arc>, + what: &'static str, + armed: bool, +} + +impl Drop for MarkIfAbandoned { + fn drop(&mut self) { + if !self.armed { + return; + } + let running = self + .running + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if *running { + self.in_flight.set(self.what); + } } } impl BoundedCalls<'_> { /// Whether a call of this adapter was abandoned at the bound. - pub(crate) fn timed_out(&self) -> bool { + #[must_use] + pub fn timed_out(&self) -> bool { self.timed_out.load(Ordering::Acquire) } @@ -795,9 +1237,9 @@ impl BoundedCalls<'_> { + Send + 'static, { - if self.in_flight.load(Ordering::Acquire) { + if let Some(caller) = self.in_flight.caller() { return Box::pin(std::future::ready(Err(StoreError::Transport { - source: "a call of an earlier check has not returned".into(), + source: InFlightMark::refusal(caller).into(), }))); } let store = Arc::clone(self.store); @@ -830,7 +1272,7 @@ impl BoundedCalls<'_> { let permit = permit.expect("the store's call slot is never closed"); let running = Arc::new(std::sync::Mutex::new(true)); let clear = ClearOnDrop { - in_flight, + in_flight: Arc::clone(&in_flight), running: Arc::clone(&running), }; let task = tokio::spawn(async move { @@ -838,20 +1280,23 @@ impl BoundedCalls<'_> { let _clear = clear; call(store).await }); + // Marks the call in flight when this future ends before the + // task: at the bound below, or dropped by its caller (a run + // bounded as a whole, such as the startup sync, whose own + // timeout fires before the call's). + let mut abandoned = MarkIfAbandoned { + in_flight, + running, + what: self.what, + armed: true, + }; match tokio::time::timeout(bound, task).await { Ok(joined) => { + abandoned.armed = false; joined.map_err(|e| StoreError::Other(format!("store call task failed: {e}")))? } Err(_) => { - // Under the task's lock: a task that has already ended - // (its guard cleared the mark) is not marked again. - let running = running - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - if *running { - self.in_flight.store(true, Ordering::Release); - } - drop(running); + drop(abandoned); self.timed_out.store(true, Ordering::Release); Err(self.did_not_finish()) } @@ -936,6 +1381,7 @@ impl BackupRuntime { source_extenddb_version: env!("CARGO_PKG_VERSION").to_owned(), limits: Arc::new(LimitsConfig::default()), restore_registrations: Arc::default(), + region: None, } } @@ -1081,6 +1527,19 @@ impl BackupRuntime { (&self.source_backend, &self.source_extenddb_version) } + /// The same runtime with the server's region. + #[must_use] + pub fn with_region(mut self, region: &str) -> Self { + self.region = Some(region.to_owned()); + self + } + + /// The server's region, once set. + #[must_use] + pub fn region(&self) -> Option<&str> { + self.region.as_deref() + } + /// Spawn a job task under the tracker [`Self::drain_jobs`] waits on. /// Public for the server's test of its shutdown sequence, which spawns a /// task that outlives the drain; the engine's own jobs come through @@ -1265,19 +1724,9 @@ impl BackupRuntime { .as_ref() .filter(|store| !store.is_set_aside()) .expect("every caller checks that a store is configured first"); - match store.health() { - StoreHealth::Healthy => Ok(store), - StoreHealth::Checking => Err(DynamoDbError::ValidationException(format!( - "The {} backup store configured on this server has not passed a check yet (its \ - first check is pending); the operation is refused until it does", - store.kind - ))), - StoreHealth::Unhealthy(problem) => Err(DynamoDbError::ValidationException(format!( - "The {} backup store configured on this server is unhealthy ({}); the operation \ - is refused until the store's check passes", - store.kind, - problem.category() - ))), + match health_refusal(store.kind, &store.health()) { + None => Ok(store), + Some(refusal) => Err(refusal), } } } diff --git a/crates/engine/src/backup/store_identity.rs b/crates/engine/src/backup/store_identity.rs index b9f846eca..45ca0d8d7 100644 --- a/crates/engine/src/backup/store_identity.rs +++ b/crates/engine/src/backup/store_identity.rs @@ -39,6 +39,37 @@ //! An unhealthy store is not a startup refusal: the caller logs the problem, //! reports it on `/health`, refuses the operations that touch the store, and //! runs the check again each cycle until it passes. +//! +//! A root whose file names another catalog changes owner only by +//! [`adopt`]: `extenddb backup sync --adopt-store`, under the maintenance +//! claim, for a root any other catalog owns; and a server whose catalog is +//! an in-memory SQLite database, for a root whose file another in-memory +//! catalog wrote. Such a catalog, and with it its `catalog_id`, is new at +//! every start, so the file it writes carries `ephemeral_catalog: true` +//! ([`StoreIdentityFile::ephemeral_catalog`]), and [`establish`] in such a +//! server adopts a root whose file carries the marker (the previous start's +//! catalog, or another in-memory server's), and only such a root: one a +//! durable catalog owns stays unhealthy there as anywhere. [`recheck`], +//! which runs once the identity is known, never adopts, so two in-memory +//! servers pointed at one root do not take it back and forth: the later +//! start owns it and the earlier finds it owned by another catalog. Until +//! that check, the previous owner acts on the root as its own, except that +//! its deletion fence (`super::StoreFence`) re-reads the file before each +//! deletion from the store and before a CreateBackup's first put, refuses +//! when the file names another catalog, and marks the store unhealthy; a +//! write already past the fence's read still lands. So every server of the +//! catalog a file names is to be stopped before `--adopt-store` takes its +//! root, and a new in-memory server started only after the old one has +//! exited. The +//! rewrite is a plain `put` (the store trait has no conditional replace), +//! so two adopters race and the last writer wins. Each re-reads the file +//! after its put, which detects another adopter's put that landed between +//! the two, and `backup sync --adopt-store` reads the file once more after +//! its sync, which detects one that landed during the sync; a put that +//! lands after both reads is found by the loser's next check, which finds +//! the store owned by another catalog. The audit line is written after the +//! re-read with what it found, and an intent line before the put, so a put +//! whose caller stopped waiting for it is on record too. use extenddb_backup_store::{BackupStore, StoreError, byte_stream_from}; use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE}; @@ -47,6 +78,8 @@ use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind}; use futures::TryStreamExt; use serde::{Deserialize, Serialize}; +use super::maintenance::HeldClaim; + /// The key of the identity file, at the root, under no `/` prefix. pub const STORE_IDENTITY_KEY: &str = "extenddb-store.json"; @@ -72,11 +105,17 @@ pub struct StoreIdentityFile { pub backend: String, /// When the file was created, epoch milliseconds on the creating host. pub created_at_ms: u64, + /// Whether the owning catalog is an in-memory SQLite database, new at + /// every start of its server (module doc). Absent in a file an earlier + /// build wrote, which no in-memory catalog of that build could adopt + /// again, so it reads as `false`. + #[serde(default)] + pub ephemeral_catalog: bool, } impl StoreIdentityFile { /// A new identity for `catalog_id`. - fn new(catalog_id: &str, backend: &str) -> Self { + fn new(catalog_id: &str, backend: &str, ephemeral_catalog: bool) -> Self { Self { format_version: STORE_IDENTITY_FORMAT_VERSION, store_id: uuid::Uuid::new_v4().to_string(), @@ -85,6 +124,7 @@ impl StoreIdentityFile { created_at_ms: std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .map_or(0, |d| u64::try_from(d.as_millis()).unwrap_or(u64::MAX)), + ephemeral_catalog, } } @@ -259,6 +299,17 @@ impl From for StoreProblem { /// Read the identity file, or `None` when the root has none. At most /// [`STORE_IDENTITY_MAX_BYTES`] are read; a larger object is /// [`StoreProblem::FileUnreadable`]. +/// +/// # Errors +/// +/// The store's error as a [`StoreProblem`], or a file that does not parse. +pub async fn read_identity_file( + store: &dyn BackupStore, +) -> Result, StoreProblem> { + read_file(store).await +} + +/// [`read_identity_file`]. async fn read_file(store: &dyn BackupStore) -> Result, StoreProblem> { let mut body = match store.get(STORE_IDENTITY_KEY).await { Ok(body) => body, @@ -296,7 +347,11 @@ async fn row_ids( /// The startup check: `validate` has passed; read the file, create it when /// the root is empty of both file and rows, compare it with the rows and /// with `catalog_id`, then probe the newest manifest. Returns the store's -/// identity, the rows' and the file's. +/// identity, the rows' and the file's. `backend` and `ephemeral_catalog` +/// describe this catalog in a file the check writes; a catalog with +/// `ephemeral_catalog` set (an in-memory SQLite database) also adopts a +/// root whose file another such catalog wrote (module doc, +/// [`AdoptRule::EphemeralOwner`]). /// /// # Errors /// @@ -307,6 +362,7 @@ pub async fn establish( root: &str, catalog: &C, backend: &str, + ephemeral_catalog: bool, ) -> Result { let catalog_id = catalog .catalog_id() @@ -319,7 +375,7 @@ pub async fn establish( return Err(StoreProblem::FileMissingWithRows(row_ids)); } None => { - let ours = StoreIdentityFile::new(&catalog_id, backend); + let ours = StoreIdentityFile::new(&catalog_id, backend, ephemeral_catalog); let created = store .put_if_absent(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes())) .await? @@ -344,16 +400,42 @@ pub async fn establish( store id {}", file.store_id ); - } else { + } else if file.catalog_id == catalog_id { tracing::info!( "backup store ({kind}) root already carried {STORE_IDENTITY_KEY} (store id \ {}), created by another instance of this catalog", file.store_id ); + } else { + tracing::info!( + "backup store ({kind}) root already carried {STORE_IDENTITY_KEY} (store id \ + {}), created by catalog {} while this check created it", + file.store_id, + file.catalog_id + ); } file } }; + if ephemeral_catalog && file.catalog_id != catalog_id && file.ephemeral_catalog { + return match adopt( + store, + kind, + root, + catalog, + backend, + ephemeral_catalog, + AdoptRule::EphemeralOwner, + None, + ) + .await + { + Ok(Adoption::Adopted { identity, .. } | Adoption::AlreadyOwned(identity)) => { + Ok(identity) + } + Err(refusal) => Err(refusal.into_problem(&catalog_id)), + }; + } let identity = compare(&file, &catalog_id, row_ids, kind, root)?; manifest_probe(store, &identity, catalog).await?; Ok(identity) @@ -386,6 +468,366 @@ pub async fn read_identity( } } +/// Which owners [`adopt`] takes a root from. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AdoptRule { + /// Any other catalog: `extenddb backup sync --adopt-store`, an + /// operator's decision. + Command, + /// Only a catalog whose file carries `ephemeral_catalog`: the startup of + /// a server whose own catalog is in memory (module doc). + EphemeralOwner, +} + +impl AdoptRule { + /// The `action` of the audit line an adoption writes. + fn action(self) -> &'static str { + match self { + Self::Command => "backup sync --adopt-store", + Self::EphemeralOwner => "startup adoption by an in-memory catalog", + } + } +} + +/// What [`adopt`] did. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Adoption { + /// The file named `previous_catalog_id` and now names this catalog; the + /// store passed the check that follows (`compare`, the manifest probe). + Adopted { + identity: StoreIdentity, + previous_catalog_id: String, + }, + /// The file already named this catalog: nothing was written, and the + /// store passed the check as [`establish`] runs it. + AlreadyOwned(StoreIdentity), +} + +/// Why [`adopt`] did not leave the root this catalog's and healthy. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum AdoptRefusal { + /// The store could not be read or written, the file could not be + /// parsed, the catalog could not be read, or (the file already this + /// catalog's) the check found another problem. + Problem(StoreProblem), + /// The root has no identity file: there is no owner to take it from. + /// `row_ids` are the store ids this catalog's rows for the root carry: + /// with none, the root is a fresh one that the next check (plain sync's + /// included) gives a file; with some, the root lost its file under the + /// rows. + NoFile { row_ids: Vec }, + /// The rule is [`AdoptRule::EphemeralOwner`] and the file's catalog is + /// not an in-memory one. + DurableOwner { file_catalog_id: String }, + /// This catalog's rows for the root carry an id other than the file's: + /// the adopted root would be unhealthy at once (an identity mismatch). + RowsCarryOtherIds { + file_id: String, + row_ids: Vec, + }, + /// After the put, the file names another catalog or holds another store + /// id: a concurrent adopter wrote after this one (last writer wins). + RaceLost { + file_catalog_id: String, + file_store_id: String, + }, + /// The root, once this catalog's, would fail the check the server runs + /// on it (the manifest probe: this catalog's newest AVAILABLE row of + /// the root has no manifest under it); nothing was written, and the + /// file still names `file_catalog_id`. + Unusable { + file_catalog_id: String, + problem: StoreProblem, + }, + /// The maintenance claim the command holds was found lost right before + /// the rewrite (the reason); nothing was written. + ClaimLost(String), +} + +impl AdoptRefusal { + /// The store problem the refusal leaves, for a check that records one: + /// a root another catalog still owns is [`StoreProblem::ForeignCatalog`]. + #[must_use] + pub fn into_problem(self, catalog_id: &str) -> StoreProblem { + match self { + Self::Problem(problem) | Self::Unusable { problem, .. } => problem, + Self::NoFile { row_ids } if !row_ids.is_empty() => { + StoreProblem::FileMissingWithRows(row_ids) + } + Self::NoFile { .. } => StoreProblem::FileUnreadable(format!( + "{STORE_IDENTITY_KEY} was removed while the root was being adopted; the next \ + check creates it" + )), + Self::DurableOwner { file_catalog_id } + | Self::RaceLost { + file_catalog_id, .. + } => StoreProblem::ForeignCatalog { + file_catalog_id, + catalog_id: catalog_id.to_owned(), + }, + Self::RowsCarryOtherIds { file_id, row_ids } => { + StoreProblem::IdMismatch { file_id, row_ids } + } + Self::ClaimLost(reason) => StoreProblem::Catalog(reason), + } + } +} + +impl std::fmt::Display for AdoptRefusal { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Problem(problem) => write!(f, "{problem}"), + Self::NoFile { row_ids } if row_ids.is_empty() => write!( + f, + "the root has no {STORE_IDENTITY_KEY}, so no catalog owns it and there is \ + nothing to adopt; plain sync creates the file" + ), + Self::NoFile { row_ids } => write!( + f, + "the root has no {STORE_IDENTITY_KEY}, and this catalog's rows for it carry \ + store id {}: the root lost its file under the rows, and there is no owner to \ + adopt it from", + row_ids.join(", ") + ), + Self::DurableOwner { file_catalog_id } => write!( + f, + "{STORE_IDENTITY_KEY} names catalog {file_catalog_id}, which is not an \ + in-memory catalog" + ), + Self::RowsCarryOtherIds { file_id, row_ids } => write!( + f, + "{STORE_IDENTITY_KEY} holds store id {file_id}, and this catalog's rows for the \ + root carry {}: adopting the root would leave it unhealthy (an identity \ + mismatch); nothing was written", + row_ids.join(", ") + ), + Self::RaceLost { + file_catalog_id, + file_store_id, + } => write!( + f, + "another writer replaced {STORE_IDENTITY_KEY} after this catalog's write: it \ + now names catalog {file_catalog_id} with store id {file_store_id}" + ), + Self::Unusable { + file_catalog_id, + problem, + } => write!( + f, + "the root would be unhealthy once this catalog owns it ({problem}); nothing was \ + written, and {STORE_IDENTITY_KEY} still names catalog {file_catalog_id}" + ), + Self::ClaimLost(reason) => write!( + f, + "stopped before the rewrite of {STORE_IDENTITY_KEY}: {reason}; nothing was \ + written" + ), + } + } +} + +/// Make this catalog the root's owner: run the check [`establish`] runs on +/// the root as it will be once this catalog owns it (`compare` with this +/// catalog's rows, the manifest probe of its newest AVAILABLE row), then +/// rewrite `extenddb-store.json`'s `catalog_id` (and `backend` and +/// `ephemeral_catalog`, which describe the owner) with a plain `put`, +/// keeping `store_id` and `created_at_ms`, and re-read the file. The check +/// reads only this catalog's rows and the store, which the rewrite does not +/// change, so its answer before the rewrite is its answer after it, and a +/// root this catalog could not use is refused with nothing written +/// ([`AdoptRefusal::Unusable`]). `rule` says which owners it takes the +/// root from. The rewrite is logged under the `extenddb::audit` target with +/// the old and the new catalog id. `claim`, the maintenance claim +/// `--adopt-store` holds, is re-checked with a beat right before the +/// rewrite, which is skipped when the claim is found lost +/// ([`AdoptRefusal::ClaimLost`]); the startup adoption of an in-memory +/// catalog holds none and passes `None`. +/// +/// A file that already names this catalog is not written +/// ([`Adoption::AlreadyOwned`]). Nothing is written when the root has no +/// file, when the rule refuses the owner, when this catalog's rows for the +/// root carry an id other than the file's, or when the check fails. +/// +/// # Errors +/// +/// [`AdoptRefusal`], each case with what was and was not written. +#[allow(clippy::too_many_arguments)] +pub async fn adopt( + store: &dyn BackupStore, + kind: StoreKind, + root: &str, + catalog: &C, + backend: &str, + ephemeral_catalog: bool, + rule: AdoptRule, + claim: Option<&HeldClaim>, +) -> Result { + let catalog_id = catalog + .catalog_id() + .await + .map_err(|e| AdoptRefusal::Problem(StoreProblem::Catalog(e.to_string())))?; + let row_ids = catalog + .read_store_ids(kind, root) + .await + .map_err(|e| AdoptRefusal::Problem(StoreProblem::Catalog(e.to_string())))?; + let file = read_file(store) + .await + .map_err(|problem| AdoptRefusal::Problem(at_step("the first read of", problem)))? + .ok_or_else(|| AdoptRefusal::NoFile { + row_ids: row_ids.clone(), + })?; + if file.catalog_id == catalog_id { + if row_ids.len() > 1 { + return Err(AdoptRefusal::Problem(StoreProblem::MultipleRowIds(row_ids))); + } + let identity = + compare(&file, &catalog_id, row_ids, kind, root).map_err(AdoptRefusal::Problem)?; + manifest_probe(store, &identity, catalog) + .await + .map_err(|problem| AdoptRefusal::Problem(at_step("the manifest probe of", problem)))?; + return Ok(Adoption::AlreadyOwned(identity)); + } + if rule == AdoptRule::EphemeralOwner && !file.ephemeral_catalog { + return Err(AdoptRefusal::DurableOwner { + file_catalog_id: file.catalog_id, + }); + } + if row_ids.iter().any(|id| *id != file.store_id) { + return Err(AdoptRefusal::RowsCarryOtherIds { + file_id: file.store_id, + row_ids, + }); + } + let ours = StoreIdentityFile { + catalog_id: catalog_id.clone(), + backend: backend.to_owned(), + ephemeral_catalog, + ..file.clone() + }; + // The check of the root as this catalog's, before anything is written: + // the file's id with this catalog's rows (which carry it, or there are + // none), and the manifest of this catalog's newest AVAILABLE row. + let unusable = |problem: StoreProblem| match problem { + // The probe's `head` failed in transport: the store's problem, not + // the root's. + StoreProblem::Transport(_) => { + AdoptRefusal::Problem(at_step("the manifest probe before the rewrite of", problem)) + } + problem => AdoptRefusal::Unusable { + file_catalog_id: file.catalog_id.clone(), + problem, + }, + }; + let identity = compare(&ours, &catalog_id, row_ids.clone(), kind, root).map_err(&unusable)?; + manifest_probe(store, &identity, catalog) + .await + .map_err(&unusable)?; + if let Some(claim) = claim { + claim.recheck().await.map_err(AdoptRefusal::ClaimLost)?; + } + tracing::info!( + target: "extenddb::audit", + action = rule.action(), + outcome = "rewriting", + store_kind = %kind, + store_root = %root, + store_id = %file.store_id, + previous_catalog_id = %file.catalog_id, + catalog_id = %catalog_id, + "rewriting {STORE_IDENTITY_KEY} to name this catalog" + ); + store + .put(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes())) + .await + .map_err(|e| { + AdoptRefusal::Problem(at_step( + "the rewrite (whose outcome is unknown; the audit log's intent line says it was \ + started) of", + e.into(), + )) + })?; + let reread = read_file(store) + .await + .map_err(|problem| { + AdoptRefusal::Problem(at_step("the re-read after the rewrite of", problem)) + })? + .ok_or_else(|| AdoptRefusal::NoFile { + row_ids: row_ids.clone(), + })?; + let won = reread.catalog_id == catalog_id && reread.store_id == file.store_id; + tracing::info!( + target: "extenddb::audit", + action = rule.action(), + outcome = if won { "adopted" } else { "race lost" }, + store_kind = %kind, + store_root = %root, + store_id = %file.store_id, + previous_catalog_id = %file.catalog_id, + catalog_id = %catalog_id, + file_catalog_id = %reread.catalog_id, + "rewrote {STORE_IDENTITY_KEY}; its re-read names catalog {}", + reread.catalog_id + ); + if !won { + return Err(AdoptRefusal::RaceLost { + file_catalog_id: reread.catalog_id, + file_store_id: reread.store_id, + }); + } + tracing::info!( + "backup store ({kind}) {root}: adopted from catalog {} (store id {})", + file.catalog_id, + identity.store_id + ); + Ok(Adoption::Adopted { + identity, + previous_catalog_id: file.catalog_id, + }) +} + +/// [`adopt`] as `backup sync --adopt-store` runs it, on `configured`'s +/// bounded calls ([`super::ConfiguredStore::calls`]): each store call under +/// the store's check bound, so a store that stops answering ends the +/// adoption at the call that did not finish, the refusal naming the step +/// ([`at_step`]), while the catalog's reads and the claim's beat run beside +/// the bound and are not cut by it. +/// +/// # Errors +/// +/// As [`adopt`]. +pub async fn adopt_configured( + configured: &super::ConfiguredStore, + catalog: &C, + backend: &str, + claim: Option<&HeldClaim>, +) -> Result { + adopt( + &configured.calls("adoption's store call"), + configured.kind, + &configured.root, + catalog, + backend, + false, + AdoptRule::Command, + claim, + ) + .await +} + +/// A store problem of [`adopt`] with the step it stopped at, for a store +/// that stopped answering (the command's bounded calls end the call that +/// did not finish, and the refusal then names it): the transport failure's +/// text gains `step` and the file's name; every other problem names its +/// condition already. +fn at_step(step: &str, problem: StoreProblem) -> StoreProblem { + match problem { + StoreProblem::Transport(text) => { + StoreProblem::Transport(format!("{step} {STORE_IDENTITY_KEY}: {text}")) + } + other => other, + } +} + /// The file against the catalog: it must name this catalog, and the rows /// for the root, when there are any, must carry its id. fn compare( @@ -420,7 +862,7 @@ fn compare( /// manifest is under the root. A root that lost its file while the catalog /// has no rows for it gets the file back, written with the identity's id /// (`put_if_absent`, so a file another instance wrote first wins and is then -/// compared like any other). +/// compared like any other). It never adopts (module doc). /// /// # Errors /// @@ -430,6 +872,7 @@ pub async fn recheck( identity: &StoreIdentity, catalog: &C, backend: &str, + ephemeral_catalog: bool, ) -> Result<(), StoreProblem> { let catalog_id = catalog .catalog_id() @@ -444,7 +887,7 @@ pub async fn recheck( None => { let ours = StoreIdentityFile { store_id: identity.store_id.clone(), - ..StoreIdentityFile::new(&catalog_id, backend) + ..StoreIdentityFile::new(&catalog_id, backend, ephemeral_catalog) }; store .put_if_absent(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes())) @@ -524,13 +967,30 @@ mod tests { #[test] fn the_file_round_trips_and_refuses_other_shapes() { - let file = StoreIdentityFile::new("cat-1", "postgres"); + let file = StoreIdentityFile::new("cat-1", "postgres", false); assert_eq!(file.format_version, 1); assert_eq!(file.catalog_id, "cat-1"); assert_eq!(file.backend, "postgres"); assert!(!file.store_id.is_empty()); let parsed = StoreIdentityFile::parse(&file.to_bytes()).unwrap(); assert_eq!(parsed, file); + // The marker round-trips, and a file without it (an earlier build's) + // reads as a durable catalog's. + let memory = StoreIdentityFile::new("cat-2", "sqlite", true); + assert!( + StoreIdentityFile::parse(&memory.to_bytes()) + .unwrap() + .ephemeral_catalog + ); + let earlier = serde_json::json!({ + "format_version": 1, "store_id": "s", "catalog_id": "c", "backend": "sqlite", + "created_at_ms": 0 + }); + assert!( + !StoreIdentityFile::parse(earlier.to_string().as_bytes()) + .unwrap() + .ephemeral_catalog + ); let v2 = serde_json::json!({ "format_version": 2, "store_id": "s", "catalog_id": "c", "backend": "b", @@ -634,6 +1094,7 @@ mod tests { catalog_id: "c-1".to_owned(), backend: "sqlite".to_owned(), created_at_ms: 0, + ephemeral_catalog: false, }; let ok = compare(&file, "c-1", vec![], StoreKind::Filesystem, "/r").unwrap(); assert_eq!(ok.store_id, "s-1"); diff --git a/crates/engine/src/backup/sync.rs b/crates/engine/src/backup/sync.rs new file mode 100644 index 000000000..0d8e8cb30 --- /dev/null +++ b/crates/engine/src/backup/sync.rs @@ -0,0 +1,1110 @@ +// SPDX-License-Identifier: Apache-2.0 + +//! `backup sync`: the catalog rebuilt from the store, for the backups the +//! catalog has no row for, and the report of the rows whose backup the store +//! no longer holds. +//! +//! The store is the source of truth and the row a cache (design 4(g)). +//! [`sync`] reads the rows first: every row of the configured kind and root, +//! of every status and every store id the catalog records for the root. It +//! keeps of each row only what it uses (the status, store id and table by +//! `(account_id, backup_id)`, and the key of each AVAILABLE row of the +//! configured store id) and drops the rest as the row is read; the +//! backends' `list_backup_rows` collect their whole result first, so the +//! read peaks above what is kept. It then lists the configured +//! store's whole root (the store trait has no delimiter listing) and handles +//! each `/
//` prefix ([`classify_store_key`]) as +//! the listing passes it: the store lists in key order, so the keys of one +//! prefix arrive together, and sync holds the objects of one prefix with +//! their sizes and the prefix of each manifest it listed, never the whole +//! listing. A filesystem store builds its own listing in memory +//! before it returns the first key (the store trait's doc gives its cost); +//! an S3 store lists a page at a time. Keys outside every backup prefix +//! (`extenddb-store.json` at the root) are passed over; the reserved +//! components (`exports/` first, `changelog` third) and three components +//! that are not a backup prefix (an id of an earlier build's shape, a +//! component no ARN can carry) are reported as skipped, never read. +//! +//! A prefix with a row of any status (CREATING, DELETING, DELETED, FAILED, +//! AVAILABLE) is skipped with the row's status as the reason, so sync never +//! modifies a row nor adopts a prefix a row names. A prefix with no row is +//! read and verified as the dead-owner duty verifies one +//! (`reconciler::verify_prefix`: the manifest and its sidecar within their +//! bounds, the digest, the reader's acceptance, every listed data file at +//! its recorded size, from the prefix's objects already listed), and the +//! manifest's ARN must name the listed account, table and backup id. A +//! manifest whose ARN is in another partition than [`ARN_PARTITION`] is +//! skipped: every request refuses such an ARN before it reads a row, so no +//! request could address the row. A manifest whose ARN is in another region +//! than this server's is adopted under this server's region's spelling of +//! the ARN (the manifest keeps the spelling it carries) and logged: the row +//! then carries the ARN this server issues, as the rows of the backups it +//! made do, so ListBackups names this server's region and DescribeBackup, +//! RestoreTableFromBackup and DeleteBackup find the backup under either +//! spelling, as they do for a backup this server made; the readers that +//! compare a manifest's ARN with the row's accept a difference in the +//! region alone (`BackupArn::names_same_backup`). Before the insert the row +//! is read under each spelling of the ARN (this server's region and the +//! manifest's), and a backup the manifest's spelling has a row for is +//! skipped: Cassandra keys its rows by the ARN and has no unique backup id, +//! so there the read is what keeps a second row of the backup id out of +//! the catalog under the manifest's spelling (a row of another root under +//! a third region's spelling is read by neither). A row of the same account +//! and id under another table that the rows read did not return (a row of +//! another root, or one whose list entries are missing) is not seen by it; +//! within one run, the row check keeps a second prefix of an id this run +//! adopted out. A row under the spelling the insert writes is left to the +//! insert. A +//! verified prefix gets an AVAILABLE row through +//! [`BackupCatalog::insert_backup_row`], unowned, in the configured store's +//! identity, with the manifest's name, size, item total and +//! `source_table`, the ARN in this server's region, and the id's instant +//! as `created_at`. An insert that +//! answers `false` found a row that landed since the rows were read (a +//! CreateBackup on another instance), a row of another root, account or +//! table holding the backup id, or, on Cassandra, a row whose list and +//! status entries are missing and that is not this insert's (a +//! CreateBackup's claim that stopped between its two writes); the row is +//! re-read to say which, and is left as it is. On Cassandra an earlier +//! insert of this same row that stopped between the row and its entries +//! (a sync killed there, or an entry batch that failed twice) is not in +//! the listings the rows are read from, so the next sync of the prefix +//! reaches the insert, which writes the entries and answers `true` (the +//! trait's doc), and the report lists the backup as adopted again. A failed +//! read of one prefix's manifest or sidecar ([`SkipReason::Store`]), or a +//! failed catalog call for one prefix ([`SkipReason::Catalog`]), is +//! reported for that prefix and sync goes on; a later run adopts it once +//! the call succeeds ([`SyncReport::transient_skips`]). +//! +//! A listing that fails stops sync: the rows inserted for the prefixes the +//! listing passed before the failure stay (each was verified), the prefix +//! it was in is not read (its keys may be cut short), and no +//! missing-manifest report is made, since the listing did not reach every +//! prefix. A stop asked through [`SyncTarget::stop`] (the command's +//! signal) is honoured before each prefix the same way. +//! +//! The report also names the AVAILABLE rows of the configured store whose +//! manifest the listing did not hold. [`prune_missing`] (`--prune-missing`, +//! under the maintenance claim) removes each such row outright through +//! [`BackupCatalog::forget_available_backup_row`], the row's lock, status +//! check and in-use rule with the removal, and deletes nothing in the +//! store: the bytes may be under another host's root of the same path, and +//! no DELETING or DELETED tombstone is left for a deletion duty to act on +//! them. A later sync where the bytes are adopts them again. Without +//! `--prune-missing` such rows stay AVAILABLE. +//! +//! The command takes the maintenance claim for every sync on PostgreSQL, +//! MongoDB and Cassandra, where it runs beside servers (design 4(f)), and +//! runs under the serve lock on SQLite; so no purge runs between its row +//! read and its insert while the claim is held. The claim is re-checked +//! with a beat ([`SyncTarget::claim`], `HeldClaim::recheck`) right before +//! each insert, and `--prune-missing` re-checks it right before each +//! removal: on MongoDB and Cassandra a command stopped (SIGSTOP, a paused +//! VM) past `stale_job_after_secs` can have its claim taken over by a +//! purge, and the beat finds it lost and stops the run before the write. +//! The bound that remains is a stop after the re-check confirmed the claim +//! and before the insert or removal reached the catalog, or with that call +//! already sent: it lands once the process resumes. A row it inserts is +//! AVAILABLE and unowned +//! (`owner_generation` 0); no reconciler duty reads an AVAILABLE row's +//! generation, so the row is handled as a created one. +//! +//! [`startup_sync`] is the sync a server whose catalog is an in-memory +//! SQLite database runs itself once its store is healthy: the catalog is +//! new at every start, so it is how such a server serves the backups of its +//! previous starts. The check that made the store healthy adopted a root an +//! earlier in-memory catalog owned (`store_identity`'s module doc). Each run +//! is bounded by [`STARTUP_SYNC_TIMEOUT`], and its store calls run as the +//! store's checks run theirs (`BoundedCalls`, under the same bound): on a +//! filesystem root the listing's walk holds the root's call slot and, once +//! abandoned at the bound, is marked in flight until it returns, so no +//! second walk starts beside the first: each cycle's check is refused at +//! once on the mark, naming the startup sync's call, and records the store +//! `unreachable` (CreateBackup, DeleteBackup and restores refused, the +//! deletions left waiting), and the sync, which runs only on a healthy +//! store, runs again on the first cycle after the walk ends. A run that +//! does not finish, or +//! that skipped a prefix for a store or catalog error, is left pending and +//! runs again in a later reconciler cycle. Every run lists from the first +//! key, so the startup sync completes only while listing the whole root +//! fits in the bound; past that it stays pending and is retried every +//! cycle, and the backups beyond what a run reaches stay unserved. Until a +//! run finishes, requests +//! for the earlier backups answer `BackupNotFoundException`; the log line +//! "startup sync of the in-memory catalog: N backup(s) adopted" ends that +//! window. Each run that lists the whole root ends with the line of +//! counts, and a run cut at the bound or stopped by an error logs one WARN +//! line instead. Until a run has logged its line of counts, a run logs each +//! prefix it skipped (WARN for a store or catalog error, INFO otherwise); +//! a later run of a sync left pending logs the transient skips at WARN +//! again and the rest at DEBUG, so a root of many prefixes with one that +//! fails costs one WARN line per cycle, not one line per prefix. + +use std::collections::{HashMap, HashSet}; +use std::time::Duration; + +use extenddb_backup_store::{BackupStore, StoreError}; +use extenddb_core::backup_format::{ + BackupArn, CHANGELOG_COMPONENT, EXPORTS_COMPONENT, EXTENDDB_MANIFEST_FILE, StoreKey, + backup_creation_date_time, classify_store_key, +}; +use extenddb_core::metrics::MetricsCollector; +use extenddb_storage::BackupCatalog; +use extenddb_storage::backup_catalog::{ + BackupKey, BackupRow, BackupStatus, ForgetRow, StoreIdentity, StoreKind, UNOWNED_GENERATION, + is_backup_not_found, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::util::ARN_PARTITION; +use futures::TryStreamExt; + +use super::BackupRuntime; +use super::maintenance::HeldClaim; +use super::reconciler::{StoreCalls, Verified, manifest_item_count, verify_prefix}; + +/// The bound of one run of [`startup_sync`], before the server serves and in +/// each reconciler cycle, and of each of its store calls. A run abandoned +/// at the bound keeps the rows it inserted, and the next run skips those +/// prefixes on the row check, but every run lists the root from its first +/// key: a root whose listing alone takes longer than the bound is never +/// synced, and the sync stays pending and is retried every cycle. On both +/// store kinds the whole listing is collected before the first key reaches +/// the run (`BoundedCalls::list` collects the store's stream into a `Vec` +/// of `ObjectMeta`, one per object of the root, held in memory until the +/// run ends; the filesystem store also builds its listing whole before it +/// yields), so a listing that outlasts the bound reaches no prefix. On an +/// S3 store the abandoned listing's requests end with the dropped future. +/// On a filesystem store the listing is not stopped: the walk runs to its +/// end on the blocking pool, holding the listing it builds and the root's +/// call slot, marked in flight (module doc); until it ends, each cycle's +/// check is refused at once, naming the startup sync's call, and records +/// the store `unreachable`, so CreateBackup, DeleteBackup and restores are +/// refused, the deletions wait, and the sync, which runs only on a healthy +/// store, does not run; once the walk ends, the next cycle's check passes +/// and the sync runs again, from the first key. A root too large to list +/// within the bound is never synced by an in-memory server: the operator +/// gives it a smaller root or a durable catalog. +pub const STARTUP_SYNC_TIMEOUT: Duration = Duration::from_secs(60); + +/// Asked before each prefix: why to stop, or `None` to go on. +pub type Stop<'a> = &'a (dyn Fn() -> Option + Sync); + +/// The store a sync reads and the identity its rows are written under. +#[derive(Clone, Copy)] +pub struct SyncTarget<'a> { + /// The configured store. + pub store: &'a dyn BackupStore, + /// The configured store's identity, its id the identity file's. + pub identity: &'a StoreIdentity, + /// The collector failed store calls are recorded in; `None` in a + /// command. + pub metrics: Option<&'a MetricsCollector>, + /// The server's region: an adopted row carries the manifest's ARN + /// rebuilt in it (logged when the manifest's region differs), and the + /// row is looked up under both spellings before the insert. `None` + /// adopts the ARN as the manifest spells it and reads that spelling + /// alone. + pub region: Option<&'a str>, + /// Asked before each prefix; `None` never stops. + pub stop: Option>, + /// The maintenance claim the command holds, re-checked with a beat + /// ([`HeldClaim::recheck`]) right before each insert; a claim found lost + /// there stops the run as `stop` does, the prefix not adopted. `None` + /// (the startup sync of an in-memory catalog, which holds no claim) + /// re-checks nothing. + pub claim: Option<&'a HeldClaim>, +} + +/// What a sync did. +#[derive(Debug, Default)] +pub struct SyncReport { + /// The backups that got an AVAILABLE row, in listing order. + pub adopted: Vec, + /// The prefixes that did not, in listing order, with the reason. + pub skipped: Vec, + /// The keys of the AVAILABLE rows of the configured store whose + /// manifest the listing does not hold; empty when the run stopped + /// before the end of the listing. + pub missing_manifests: Vec, + /// Why the run stopped before the end of the listing, when it did. + pub stopped: Option, +} + +impl SyncReport { + /// The prefixes skipped for a store or catalog error, which a later run + /// may adopt. + #[must_use] + pub fn transient_skips(&self) -> usize { + self.skipped + .iter() + .filter(|s| s.reason.is_transient()) + .count() + } +} + +/// A backup [`sync`] gave an AVAILABLE row. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct AdoptedBackup { + /// The ARN as the row carries it: in the server's region when one is + /// given, otherwise as the manifest spells it. + pub backup_arn: String, + /// Set when the manifest's ARN names another region than the server's: + /// the sentence the log carries, naming the manifest's spelling. + pub foreign: Option, +} + +/// A prefix [`sync`] did not adopt. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SkippedPrefix { + /// The prefix, as the store's keys spell it (`exports/` for that + /// component). + pub prefix: String, + pub reason: SkipReason, +} + +/// Why a prefix was not adopted. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum SkipReason { + /// Under a reserved component, the one named. + Reserved(&'static str), + /// Three components that are not a backup prefix; why. + NotABackup(String), + /// The catalog has a row for the account and backup id: its status, + /// store id, and its table when that is not the prefix's. + HasRow { + status: BackupStatus, + store_id: String, + other_table: Option, + }, + /// A row the catalog's read had not returned holds the backup: what + /// the read under the ARN's spellings, or the insert's re-read, found. + RowExists(String), + /// The prefix failed verification; which check. + Incomplete(String), + /// The manifest names another backup than the path. + ArnMismatch(String), + /// The manifest's ARN is in another partition: no request to this + /// server can address it. + OtherPartition(String), + /// A read of the prefix's manifest or sidecar failed; the store's + /// error. + Store(String), + /// A catalog call for the prefix (the read under the ARN's spellings, + /// the insert) failed; the catalog's error. + Catalog(String), +} + +impl SkipReason { + /// Whether a later run may adopt the prefix without anything changing + /// in the store: a store or catalog call failed. + #[must_use] + pub fn is_transient(&self) -> bool { + matches!(self, Self::Store(_) | Self::Catalog(_)) + } +} + +impl std::fmt::Display for SkipReason { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Reserved(component) if *component == EXPORTS_COMPONENT => { + f.write_str("reserved for exports") + } + Self::Reserved(_) => f.write_str("reserved for the change-log archive"), + Self::NotABackup(why) => write!(f, "not a backup prefix: {why}"), + Self::HasRow { + status, + store_id, + other_table: None, + } => write!( + f, + "the catalog has a row for it ({status}, store id {store_id})" + ), + Self::HasRow { + status, + store_id, + other_table: Some(table), + } => write!( + f, + "the catalog has a row for its backup id under table {table} ({status}, store \ + id {store_id}); a backup id names one backup" + ), + Self::RowExists(what) => f.write_str(what), + Self::Incomplete(why) => write!(f, "not a complete backup: {why}"), + Self::ArnMismatch(why) | Self::OtherPartition(why) => f.write_str(why), + Self::Store(e) => write!(f, "the store could not be read: {e}"), + Self::Catalog(e) => write!(f, "the catalog could not be read or written: {e}"), + } + } +} + +/// Why a sync stopped before it reached the end of the listing. +#[derive(Debug)] +pub enum SyncError { + /// The store's listing failed; `partial` is what the run did with the + /// prefixes the listing passed before (no missing-manifest report). + Listing { + error: StoreError, + partial: SyncReport, + }, + /// The catalog's rows could not be read; nothing was listed or written. + Catalog(StorageError), +} + +impl std::fmt::Display for SyncError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Listing { error, partial } if partial.adopted.is_empty() => { + write!(f, "the store's listing failed: {error}; no row was written") + } + Self::Listing { error, partial } => write!( + f, + "the store's listing failed: {error}; {} backup(s) listed before it were adopted", + partial.adopted.len() + ), + Self::Catalog(e) => write!( + f, + "the catalog's rows could not be read: {e}; no row was written" + ), + } + } +} + +impl std::error::Error for SyncError {} + +/// The prefix a listed key belongs to, and what it is. +enum Group { + /// Under a reserved component. + Reserved(&'static str), + /// Under three components that are not a backup prefix. + NotABackup(String), + /// Under a backup prefix: its components and the objects listed so far + /// with their sizes. + Backup { + account_id: String, + table_name: String, + backup_id: String, + objects: HashMap, + }, +} + +/// The prefix of `key` and an empty group for it; `None` for a key outside +/// every backup prefix. +fn group_of(key: &str) -> Option<(String, Group)> { + match classify_store_key(key) { + StoreKey::Outside => None, + StoreKey::Reserved(component) => { + Some((reserved_prefix(key, component), Group::Reserved(component))) + } + StoreKey::NotABackup { prefix, reason } => { + Some((prefix.to_owned(), Group::NotABackup(reason))) + } + StoreKey::Backup { + prefix, + account_id, + table_name, + backup_id, + } => Some(( + prefix.to_owned(), + Group::Backup { + account_id: account_id.to_owned(), + table_name: table_name.to_owned(), + backup_id: backup_id.to_owned(), + objects: HashMap::new(), + }, + )), + } +} + +/// The prefix a key under a reserved component is reported under: +/// `exports/`, or `/
/changelog/`. +fn reserved_prefix(key: &str, component: &str) -> String { + if component == EXPORTS_COMPONENT { + return format!("{EXPORTS_COMPONENT}/"); + } + let mut parts = key.splitn(3, '/'); + let account = parts.next().unwrap_or_default(); + let table = parts.next().unwrap_or_default(); + format!("{account}/{table}/{CHANGELOG_COMPONENT}/") +} + +/// The rows of the configured kind and root, every status and every store +/// id the catalog records for the root, kept as only what sync uses: the +/// row check's status, store id and table by `(account_id, backup_id)`, +/// and the keys of the configured store id's AVAILABLE rows (the +/// missing-manifest candidates). Each row is dropped as it is read, its +/// parsed `source_table_details_json` with it. +async fn rows_of_root( + storage: &S, + identity: &StoreIdentity, +) -> Result<(Known, Vec), StorageError> { + let mut ids = storage + .read_store_ids(identity.store_kind, &identity.store_root) + .await?; + if !ids.contains(&identity.store_id) { + ids.push(identity.store_id.clone()); + } + let mut known = Known::new(); + let mut available = Vec::new(); + for id in ids { + let of_id = StoreIdentity { + store_id: id, + ..identity.clone() + }; + let mut listed = storage.list_backup_rows(None, &of_id); + while let Some(row) = listed.try_next().await? { + if row.status == BackupStatus::Available && row.store.store_id == identity.store_id { + available.push(row.key()); + } + let BackupRow { + account_id, + backup_id, + table_name, + status, + store, + .. + } = row; + known.insert( + (account_id, backup_id), + (status, store.store_id, table_name), + ); + } + } + Ok((known, available)) +} + +/// What the row check knows of a row: its status, store id and table. +type Known = HashMap<(String, String), (BackupStatus, String, String)>; + +/// Adopt the backups of the configured store that have no row (module doc). +/// +/// # Errors +/// +/// [`SyncError`]: the catalog's read failed (nothing written), or the +/// listing failed (the rows inserted before it stay). +pub async fn sync( + storage: &S, + target: SyncTarget<'_>, +) -> Result { + let calls = StoreCalls { + store: target.store, + metrics: target.metrics, + }; + let identity = target.identity; + let (mut known, available) = rows_of_root(storage, identity) + .await + .map_err(SyncError::Catalog)?; + + let mut report = SyncReport::default(); + // The prefixes whose manifest the listing held: one string per listed + // manifest. + let mut manifests: HashSet = HashSet::new(); + let mut open: Option<(String, Group)> = None; + let mut listing = calls.store.list(""); + loop { + let next = match listing.try_next().await { + Ok(next) => next, + Err(e) => { + return Err(SyncError::Listing { + error: calls.failed("list", e), + partial: report, + }); + } + }; + let Some(meta) = next else { break }; + let Some((prefix, group)) = group_of(&meta.key) else { + continue; + }; + if open.as_ref().is_none_or(|(open, _)| *open != prefix) { + // The listing has passed the open prefix: every key of it is + // listed. + if let Some((done, group)) = open.take() + && let Some(reason) = + finish_group(storage, calls, target, &mut known, &mut report, done, group).await + { + report.stopped = Some(reason); + return Ok(report); + } + open = Some((prefix.clone(), group)); + } + if let Some((_, Group::Backup { objects, .. })) = &mut open { + if meta.key.len() == prefix.len() + EXTENDDB_MANIFEST_FILE.len() + && meta.key.ends_with(EXTENDDB_MANIFEST_FILE) + { + manifests.insert(prefix); + } + objects.insert(meta.key, meta.size); + } + } + if let Some((done, group)) = open.take() + && let Some(reason) = + finish_group(storage, calls, target, &mut known, &mut report, done, group).await + { + report.stopped = Some(reason); + return Ok(report); + } + + report.missing_manifests = available + .into_iter() + .filter(|key| { + !BackupArn::parse(&key.backup_arn) + .is_ok_and(|arn| manifests.contains(&arn.store_prefix())) + }) + .collect(); + Ok(report) +} + +/// One prefix the listing has passed: the stop, then the row check, then +/// the verification and the insert. Returns why the run stops, when the +/// stop asked it to (the prefix is then left unread). +async fn finish_group( + storage: &S, + calls: StoreCalls<'_>, + target: SyncTarget<'_>, + known: &mut Known, + report: &mut SyncReport, + prefix: String, + group: Group, +) -> Option { + if let Some(reason) = target.stop.and_then(|stop| stop()) { + return Some(reason); + } + let (account_id, table_name, backup_id, objects) = match group { + Group::Reserved(component) => { + report.skipped.push(SkippedPrefix { + prefix, + reason: SkipReason::Reserved(component), + }); + return None; + } + Group::NotABackup(why) => { + report.skipped.push(SkippedPrefix { + prefix, + reason: SkipReason::NotABackup(why), + }); + return None; + } + Group::Backup { + account_id, + table_name, + backup_id, + objects, + } => (account_id, table_name, backup_id, objects), + }; + // The row check, before anything under the prefix is read. + let key = (account_id, backup_id); + if let Some((status, store_id, row_table)) = known.get(&key) { + report.skipped.push(SkippedPrefix { + prefix, + reason: SkipReason::HasRow { + status: *status, + store_id: store_id.clone(), + other_table: (*row_table != table_name).then(|| row_table.clone()), + }, + }); + return None; + } + let outcome = adopt_prefix( + storage, + calls, + target, + &prefix, + (&key.0, &table_name, &key.1), + &objects, + ) + .await; + match outcome { + Ok(adopted) => { + known.insert( + key, + ( + BackupStatus::Available, + target.identity.store_id.clone(), + table_name, + ), + ); + report.adopted.push(adopted); + } + Err(NotAdopted::Skipped(reason)) => report.skipped.push(SkippedPrefix { prefix, reason }), + Err(NotAdopted::Stopped(reason)) => return Some(reason), + } + None +} + +/// Why [`adopt_prefix`] inserted no row. +enum NotAdopted { + /// The prefix is reported as skipped, and the run goes on. + Skipped(SkipReason), + /// The claim was found lost right before the insert: the run stops, + /// the prefix not adopted. + Stopped(String), +} + +impl From for NotAdopted { + fn from(reason: SkipReason) -> Self { + Self::Skipped(reason) + } +} + +/// Verify one prefix with no row and insert its row. +async fn adopt_prefix( + storage: &S, + calls: StoreCalls<'_>, + target: SyncTarget<'_>, + prefix: &str, + (account_id, table_name, backup_id): (&str, &str, &str), + objects: &HashMap, +) -> Result { + let manifest = match verify_prefix(calls, prefix, Some(objects)).await { + Ok(Verified::Complete(manifest)) => manifest, + Ok(Verified::Incomplete(why)) => return Err(SkipReason::Incomplete(why).into()), + Err(e) => return Err(SkipReason::Store(e.to_string()).into()), + }; + let arn = manifest + .backup_arn() + .map_err(|e| SkipReason::ArnMismatch(format!("the manifest's ARN is refused: {e}")))?; + // The skip reasons are the report's; the caller logs or prints them + // (the startup sync once per prefix, the command in its report). + if arn.account_id() != account_id + || arn.table_name() != table_name + || arn.backup_id() != backup_id + { + return Err(SkipReason::ArnMismatch(format!( + "the manifest names {}, whose account, table and backup id are not the prefix's", + arn.to_arn_string() + )) + .into()); + } + let backup_arn = arn.to_arn_string(); + if arn.partition() != ARN_PARTITION { + return Err(SkipReason::OtherPartition(format!( + "the manifest names {backup_arn}, in partition {}, and no request to this server \ + can address an ARN outside partition {ARN_PARTITION}", + arn.partition() + )) + .into()); + } + let created_at = backup_creation_date_time(backup_id).ok_or_else(|| { + SkipReason::NotABackup(format!("backup id {backup_id:?} carries no instant")) + })?; + // The row's ARN: the manifest's rebuilt in this server's region, so the + // row reads as one of a backup this server made (module doc); the + // manifest's own spelling when no region is given or the region is + // outside the ARN alphabet. + let row_arn = target + .region + .and_then(|region| arn.in_region(region)) + .unwrap_or_else(|| arn.clone()); + let foreign = foreign_note(&arn, target.region); + // A row under a spelling other than the row's: on Cassandra the insert + // decides per spelling only (the trait's doc). A row under the row's + // own spelling is left to the insert, which answers `false` for it, or, + // on Cassandra, completes it when it is an earlier insert of this same + // row that stopped before its list and status entries. + match row_under_a_spelling(storage, &arn, target.region).await { + Ok(Some((spelling, row))) if spelling != row_arn => { + return Err(SkipReason::RowExists(describe_row(target, &row)).into()); + } + Ok(_) => {} + Err(e) => return Err(SkipReason::Catalog(e.to_string()).into()), + } + let source_table_details_json = serde_json::to_value(&manifest.source_table) + .map_err(|e| SkipReason::Incomplete(format!("the source table does not serialize: {e}")))?; + let row = BackupRow { + backup_id: backup_id.to_owned(), + account_id: account_id.to_owned(), + table_name: table_name.to_owned(), + backup_arn: row_arn.to_arn_string(), + backup_name: manifest.backup_name.clone(), + status: BackupStatus::Available, + store: target.identity.clone(), + size_bytes: manifest.backup_size_bytes, + item_count: manifest_item_count(&manifest), + created_at, + completed_at: None, + failure_reason: None, + source_table_details_json, + owner_generation: UNOWNED_GENERATION, + heartbeat_at: 0.0, + }; + // The insert is the step that cannot be undone: a row of a prefix a + // purge holding a claim taken over from this command is deleting would + // be AVAILABLE over deleted bytes. + if let Some(claim) = target.claim + && let Err(reason) = claim.recheck().await + { + return Err(NotAdopted::Stopped(format!( + "{reason} (before the insert for {prefix}, which was not adopted)" + ))); + } + match storage.insert_backup_row(&row).await { + Ok(true) => {} + Ok(false) => { + return Err( + SkipReason::RowExists(existing_row(storage, target, &row_arn).await).into(), + ); + } + Err(e) => return Err(SkipReason::Catalog(e.to_string()).into()), + } + match &foreign { + Some(note) => tracing::warn!( + "backup sync: adopted {} (account {account_id}), {note}; requests find it under \ + either spelling of its ARN", + row.backup_arn + ), + None => tracing::info!( + "backup sync: adopted {} (account {account_id})", + row.backup_arn + ), + } + Ok(AdoptedBackup { + backup_arn: row.backup_arn, + foreign, + }) +} + +/// The row under `arn` rebuilt in `region` (the spelling every request tries +/// first) or under `arn` as the manifest writes it, in that order, with the +/// spelling it was found under. +async fn row_under_a_spelling( + storage: &S, + arn: &BackupArn, + region: Option<&str>, +) -> Result, StorageError> { + let written = arn.to_arn_string(); + if let Some(region) = region + && let Ok(parts) = extenddb_storage::util::check_backup_arn(&written) + { + let spellings = super::BackupArnSpellings::new(&parts, region); + return super::read_catalog_row(storage, &spellings).await; + } + Ok(storage + .read_backup_row(arn) + .await? + .map(|row| (arn.clone(), row))) +} + +/// The sentence for a manifest whose ARN is in another region than the +/// server's, or `None`. +fn foreign_note(arn: &BackupArn, region: Option<&str>) -> Option { + region.filter(|region| arn.region() != *region).map(|_| { + format!( + "whose manifest spells its ARN in region {}, not this server's", + arn.region() + ) + }) +} + +/// What holds the backup id when the insert answered `false`: the row is +/// re-read by the ARN the insert wrote, so the report says which row it +/// met. +async fn existing_row( + storage: &S, + target: SyncTarget<'_>, + row_arn: &BackupArn, +) -> String { + match storage.read_backup_row(row_arn).await { + Ok(Some(row)) => describe_row(target, &row), + Ok(None) => format!( + "a row with backup id {} exists under another account, another table or another \ + spelling of the ARN, or an earlier release's in-catalog backup holds this ARN; it \ + is left as it is", + row_arn.backup_id() + ), + Err(e) => format!("a row exists for it (its re-read failed: {e}); it is left as it is"), + } +} + +/// The report's sentence for a row the rows read at the start did not +/// return: one of another store, or one of this store that landed after +/// the read (a CreateBackup, say) or whose list entries a write did not +/// complete (a CreateBackup's claim on Cassandra). The row itself says +/// which store; the two cases of this store read the same, since telling +/// them apart would list every row of the row's status for each such +/// prefix. +fn describe_row(target: SyncTarget<'_>, row: &BackupRow) -> String { + if row.store != *target.identity { + return format!( + "a row of another store holds it ({}; {} {}, store id {}); it is left as it is", + row.status, row.store.store_kind, row.store.store_root, row.store.store_id + ); + } + format!( + "a row for it ({}) exists that the catalog's read at the start did not return (it \ + landed after the read, or its list entries were never written); it is not a row this \ + sync's insert writes, so it is left as it is", + row.status + ) +} + +/// What [`prune_missing`] did with one row. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum PruneOutcome { + /// The row is gone; nothing in the store was deleted. + Pruned, + /// The manifest is under the prefix again; the row was left. + ManifestBack, + /// A restore target registered from the backup is CREATING (its name); + /// the row was left. + InUse(String), + /// The row was not AVAILABLE any more (its status); left as it is. + Status(BackupStatus), + /// The row was gone. + Gone, + /// A store or catalog call failed (why); the row was left. + Failed(String), +} + +impl std::fmt::Display for PruneOutcome { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Pruned => f.write_str("removed: the row is gone, and no object was deleted"), + Self::ManifestBack => f.write_str("left: the manifest is under its prefix again"), + Self::InUse(table) => write!( + f, + "left: the restore target {table} is being restored from it" + ), + Self::Status(status) => write!(f, "left: the row is {status} now"), + Self::Gone => f.write_str("left: the row is gone"), + Self::Failed(why) => write!(f, "left: {why}"), + } + } +} + +impl PruneOutcome { + /// Whether the row is still AVAILABLE with its manifest gone and a later + /// run may remove it: a store or catalog call failed + /// ([`Self::Failed`]), or a restore is reading the backup + /// ([`Self::InUse`]). The command exits 1 when any row ends so. + /// [`Self::ManifestBack`], [`Self::Status`] and [`Self::Gone`] leave + /// nothing for a later run, and the command exits 0 for them. + #[must_use] + pub fn needs_another_run(&self) -> bool { + matches!(self, Self::InUse(_) | Self::Failed(_)) + } +} + +/// `--prune-missing`: for each AVAILABLE row whose manifest the listing did +/// not hold, `head` the manifest once more, then remove the row outright +/// ([`BackupCatalog::forget_available_backup_row`], which refuses a backup a +/// restore is reading and a row another writer moved). A `head` that fails +/// leaves the row ([`PruneOutcome::Failed`]): the store has not said whether +/// the manifest is back, so the row is kept and a later run decides. +/// Nothing in the store is deleted (module doc). The caller holds the +/// maintenance claim (`claim`), re-checked with a beat right before each +/// removal: a claim found lost there ends the loop with the reason, that +/// row left as it is. `stop` +/// is asked before each row and returns why to stop (the claim lost, a +/// signal), which ends the loop with the reason. Each removal is logged +/// under the `extenddb::audit` target, with `store_root` the configured +/// store's root. +pub async fn prune_missing( + storage: &S, + store: &dyn BackupStore, + store_root: &str, + rows: &[BackupKey], + claim: &HeldClaim, + stop: &dyn Fn() -> Option, +) -> (Vec<(String, PruneOutcome)>, Option) { + let mut done = Vec::new(); + for row in rows { + if let Some(reason) = stop() { + return (done, Some(reason)); + } + let outcome = match prune_one(storage, store, row, claim).await { + Ok(outcome) => outcome, + Err(reason) => return (done, Some(reason)), + }; + if outcome == PruneOutcome::Pruned { + tracing::info!( + target: "extenddb::audit", + action = "backup sync --prune-missing", + account_id = %row.account_id, + backup_arn = %row.backup_arn, + store_root = %store_root, + "removed an AVAILABLE row whose manifest is not under the root; no object was \ + deleted" + ); + } + done.push((row.backup_arn.clone(), outcome)); + } + (done, None) +} + +/// One row of [`prune_missing`]; `Err` is the claim found lost right +/// before the removal, which ends the loop. +async fn prune_one( + storage: &S, + store: &dyn BackupStore, + row: &BackupKey, + claim: &HeldClaim, +) -> Result { + let arn = match BackupArn::parse(&row.backup_arn) { + Ok(arn) => arn, + Err(e) => { + return Ok(PruneOutcome::Failed(format!( + "the row's ARN does not parse: {e}" + ))); + } + }; + match store + .head(&format!("{}{EXTENDDB_MANIFEST_FILE}", arn.store_prefix())) + .await + { + Ok(Some(_)) => return Ok(PruneOutcome::ManifestBack), + Ok(None) => {} + Err(e) => { + return Ok(PruneOutcome::Failed(format!( + "the manifest's head failed: {e}" + ))); + } + } + claim + .recheck() + .await + .map_err(|reason| format!("{reason} (before the removal of {})", row.backup_arn))?; + Ok(match storage.forget_available_backup_row(row).await { + Ok(ForgetRow::Removed) => PruneOutcome::Pruned, + Ok(ForgetRow::InUse(table)) => PruneOutcome::InUse(table), + Ok(ForgetRow::Status(status)) => PruneOutcome::Status(status), + Err(e) if is_backup_not_found(&e) => PruneOutcome::Gone, + Err(e) => PruneOutcome::Failed(format!("the row could not be removed: {e}")), + }) +} + +/// Hands a taken startup sync back when the run ends without listing the +/// whole root, or with a prefix a later run may adopt; dropped with its +/// future included. +struct Rearm<'a> { + store: &'a super::ConfiguredStore, + done: bool, +} + +impl Drop for Rearm<'_> { + fn drop(&mut self) { + if !self.done { + self.store.retry_startup_sync(); + } + } +} + +/// The startup sync of a server whose catalog is an in-memory SQLite +/// database (module doc): once, when the configured store is healthy and +/// the sync is pending ([`super::ConfiguredStore::with_ephemeral_catalog`]), +/// bounded by [`STARTUP_SYNC_TIMEOUT`]. `None` when nothing ran (no store, +/// not pending, not healthy, or another caller is running it) and when a +/// run reached the bound (logged; the sync stays pending). A run that +/// stops before it listed the whole root (a listing or catalog failure, the +/// bound, or its future dropped at a shutdown), or that skipped a prefix for +/// a store or catalog error, leaves the sync pending for the next +/// reconciler cycle. The server calls it before it serves (the filesystem +/// store, whose check ran then) and every reconciler cycle calls it after +/// the store's check (the S3 store, whose first check runs there, and a sync +/// left pending). +pub async fn startup_sync( + storage: &S, + runtime: &BackupRuntime, +) -> Option> { + let store = runtime.configured_store()?; + let identity = store.identity().filter(|_| store.is_healthy())?; + // The take is the one gate: of two callers that find the sync pending, + // one runs it. + if !store.take_startup_sync() { + return None; + } + // Until a run has reported (listed the whole root and logged its line + // of counts), the lasting skips are logged at INFO, so their reasons + // reach the log when the first run was cut at its bound or ended in an + // error. + let first_report = !store.startup_sync_reported(); + let mut rearm = Rearm { store, done: false }; + // The store's calls as the checks run them (module doc): the listing's + // walk under the run's bound, in the root's call slot, marked in flight + // once abandoned. + let via = store.calls_within("startup sync's store call", STARTUP_SYNC_TIMEOUT); + let run = sync( + storage, + SyncTarget { + store: &via, + identity: &identity, + metrics: runtime.metrics().map(|m| &**m), + region: runtime.region(), + stop: None, + claim: None, + }, + ); + let Ok(outcome) = tokio::time::timeout(STARTUP_SYNC_TIMEOUT, run).await else { + // On a filesystem store the abandoned listing's walk still runs: + // until it returns, each cycle's check is refused and the store is + // unreachable, so the sync runs again on the first cycle after it. + tracing::warn!( + "backup store ({}) startup sync of the in-memory catalog did not finish within {} \ + s; the backups it adopted keep their rows, and a later reconciler cycle runs it \ + again from the first key of the root{}", + store.kind, + STARTUP_SYNC_TIMEOUT.as_secs(), + if store.kind == StoreKind::Filesystem { + " (once the abandoned listing of the root has returned; until then the store is \ + unreachable)" + } else { + "" + } + ); + return None; + }; + match &outcome { + Ok(report) => { + let transient = report.transient_skips(); + rearm.done = transient == 0; + let mut reserved = 0usize; + for skipped in &report.skipped { + if matches!(skipped.reason, SkipReason::Reserved(_)) { + reserved += 1; + } else if skipped.reason.is_transient() { + tracing::warn!( + "backup store startup sync: {} not adopted: {}; the next reconciler \ + cycle tries it again", + skipped.prefix, + skipped.reason + ); + } else if first_report { + tracing::info!( + "backup store startup sync: {} not adopted: {}", + skipped.prefix, + skipped.reason + ); + } else { + // An earlier run reported and named the prefix; the + // line of counts below says how many. + tracing::debug!( + "backup store startup sync: {} not adopted: {}", + skipped.prefix, + skipped.reason + ); + } + } + tracing::info!( + "backup store ({}) startup sync of the in-memory catalog: {} backup(s) adopted, \ + {} prefix(es) skipped ({reserved} of them reserved for exports or the \ + change-log archive){}", + store.kind, + report.adopted.len(), + report.skipped.len(), + if transient == 0 { + String::new() + } else { + format!( + "; {transient} prefix(es) met a store or catalog error, and the next \ + reconciler cycle runs the sync again" + ) + } + ); + store.record_startup_sync_report(); + } + Err(e) => tracing::warn!( + "backup store ({}) startup sync of the in-memory catalog stopped: {e}; a later \ + reconciler cycle runs it again", + store.kind + ), + } + Some(outcome) +} diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs index f9187cf1b..8dc9c9f1a 100644 --- a/crates/engine/src/backup/tests.rs +++ b/crates/engine/src/backup/tests.rs @@ -21,9 +21,10 @@ use extenddb_core::types::{ TimeToLiveDescription, TimeToLiveStatus, UpdateTableInput, }; use extenddb_storage::backup_catalog::{ - AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, - JobRow, JobStatus, RestoreJobRow, RestoreKind, RestoreRegistrationRow, RestoreTargetKey, - StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, + AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, + JobClaim, JobRow, JobStatus, RestoreJobRow, RestoreKind, RestoreRegistrationRow, + RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, + UNOWNED_GENERATION, WriteOutcome, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; @@ -297,6 +298,37 @@ struct Fake { /// removing anything: the owner's lock (PostgreSQL's per-session /// advisory lock) is still held by a frozen owner's session. owner_lock_held: Mutex, + /// Rows `insert_backup_row` lands just before it decides, by ARN: a + /// CreateBackup on another instance whose row lands between sync's read + /// of the rows and its insert. + lands_before_insert: Mutex>, + /// ARNs `list_backup_rows` does not return though the row exists: a + /// Cassandra row whose list and status entries were never written. + unlisted: Mutex>, + /// When set, `insert_backup_row` that meets an unowned row under the + /// same ARN equal to the given one in every column the insert takes + /// answers `true` and lists it again (`unlisted` loses it): Cassandra's + /// completion of an insert that stopped before its entries. + completes_own_insert: std::sync::atomic::AtomicBool, + /// Every row `insert_backup_row` received, inserted or not. + insert_attempts: Mutex>, + /// The status filter of every `list_backup_rows` call, in order. + row_listings: Mutex>>, + /// When set, `insert_backup_row` fails with this error and writes + /// nothing. + insert_error: Mutex>, + /// Every key `forget_available_backup_row` received. + forgets_seen: Mutex>, + /// Preset answers of `forget_available_backup_row`, by ARN; the row is + /// left as the test set it. + forget_answers: Mutex>, + /// When set, `read_backup_row` fails with this error (every call). + read_row_error: Mutex>, + /// When set, rows are unique by ARN alone, as on Cassandra: + /// `insert_backup_row` answers `false` only for a row under the same + /// ARN, and `read_backup_row` finds a row by its ARN; otherwise by + /// backup id, as on PostgreSQL, SQLite and MongoDB. + unique_by_arn: std::sync::atomic::AtomicBool, } /// A closure the `Fake` runs inside `sweep_restore_progress`. @@ -883,10 +915,52 @@ struct MemStore { /// When set, `delete_prefix` takes this long before it answers (a slow /// store's deletion). delete_delay: Mutex>, + /// When set, every `delete_prefix` waits until it is cleared, counted + /// in `stalls`, while every other call answers: a deletion parked in a + /// mount that stopped answering beside reads of the identity file that + /// do not. + stall_deletes: std::sync::atomic::AtomicBool, /// When set, `list` of this prefix fails as a transport error. fail_lists_of: Mutex>, + /// When set, a `list` of the whole root fails as a transport error + /// once it has yielded the keys that sort before this one: a listing + /// that stops part-way through. + fail_root_list_at: Mutex>, /// When set, `delete_prefix` of this prefix fails as a transport error. fail_deletes_of: Mutex>, + /// When set, the identity file becomes these bytes right after the next + /// `get` of it has read the current ones: a root another catalog adopts + /// between a store's check and what follows it. + identity_after_read: Mutex>>, + /// When set, a `list` of the whole root never yields (a listing that + /// stalls); a `list` of any prefix answers. + stall_root_list: std::sync::atomic::AtomicBool, + /// While set, a `list` of the whole root waits, and answers once it is + /// cleared (a walk of a slow mount that returns). + hold_root_list: std::sync::atomic::AtomicBool, + /// When set, the `get`s of the identity file after the first `n` (the + /// count) fail as a transport error, or never answer (`stall`); every + /// other call answers. The deletion fence's read failing alone. + identity_get_failure: Mutex>, + /// When set, `head` of this key fails as a transport error; every other + /// call answers. + fail_heads_of: Mutex>, + /// When set, `head` of this key never answers; every other call answers. + stall_heads_of: Mutex>, + /// Every `list`'s prefix, in order. + lists: Mutex>, +} + +/// How `MemStore::identity_get_failure` fails a read of the identity file. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum IdentityFailure { + Transport, + Stall, + /// The store refuses the read (an `AccessDenied` on S3). + Denied, + /// The read fails with the filesystem store's shape for a file it may + /// not open (a file at mode 000): an I/O error, not a denial. + Unreadable, } /// What `MemStore::on_put` runs after a stored `put`, with the object's key. @@ -1104,6 +1178,41 @@ impl extenddb_backup_store::BackupStore for MemStore { self.gets.lock().unwrap().push(key.to_owned()); self.stall().await; self.check()?; + if key == STORE_IDENTITY_KEY { + let failure = { + let mut slot = self.identity_get_failure.lock().unwrap(); + match slot.as_mut() { + Some((0, failure)) => Some(*failure), + Some((pass, _)) => { + *pass -= 1; + None + } + None => None, + } + }; + match failure { + Some(IdentityFailure::Transport) => { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "identity get refused".to_owned(), + ))), + }); + } + Some(IdentityFailure::Stall) => futures::future::pending::<()>().await, + Some(IdentityFailure::Denied) => { + return Err(extenddb_backup_store::StoreError::PermissionDenied( + format!("GetObject {key}: permission denied"), + )); + } + Some(IdentityFailure::Unreadable) => { + return Err(extenddb_backup_store::StoreError::Io { + context: format!("open {key}"), + source: std::io::Error::from_raw_os_error(13), + }); + } + None => {} + } + } if self .fail_gets_under .lock() @@ -1160,6 +1269,11 @@ impl extenddb_backup_store::BackupStore for MemStore { .get(key) .cloned() .ok_or(extenddb_backup_store::StoreError::NotFound)?; + if key == STORE_IDENTITY_KEY + && let Some(next) = self.identity_after_read.lock().unwrap().take() + { + self.insert(key, &next); + } if Self::is_data_file(key) && let Some(n) = self.fail_get_after_bytes.lock().unwrap().take() { @@ -1201,6 +1315,16 @@ impl extenddb_backup_store::BackupStore for MemStore { Box::pin(async move { self.stall().await; self.check()?; + if self.fail_heads_of.lock().unwrap().as_deref() == Some(key) { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError("head refused".to_owned()))), + }); + } + if self.stall_heads_of.lock().unwrap().as_deref() == Some(key) { + self.stalls + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + std::future::pending::<()>().await; + } Ok(self.objects.lock().unwrap().get(key).map(|bytes| { extenddb_backup_store::ObjectMeta { key: key.to_owned(), @@ -1217,9 +1341,35 @@ impl extenddb_backup_store::BackupStore for MemStore { prefix: &'a str, ) -> BoxStream<'a, Result> { + self.lists.lock().unwrap().push(prefix.to_owned()); if let Err(e) = self.check() { return Box::pin(futures::stream::once(async move { Err(e) })); } + if prefix.is_empty() + && self + .stall_root_list + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin(futures::stream::pending()); + } + if prefix.is_empty() + && self + .hold_root_list + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin( + futures::stream::once(async move { + while self + .hold_root_list + .load(std::sync::atomic::Ordering::Acquire) + { + tokio::time::sleep(Duration::from_millis(10)).await; + } + self.list(prefix) + }) + .flatten(), + ); + } if self.fail_lists_of.lock().unwrap().as_deref() == Some(prefix) { return Box::pin(futures::stream::once(async move { Err(extenddb_backup_store::StoreError::Transport { @@ -1227,12 +1377,19 @@ impl extenddb_backup_store::BackupStore for MemStore { }) })); } - let metas: Vec<_> = self + let fail_at = self + .fail_root_list_at + .lock() + .unwrap() + .clone() + .filter(|_| prefix.is_empty()); + let mut metas: Vec<_> = self .objects .lock() .unwrap() .iter() .filter(|(k, _)| Self::matches(k, prefix)) + .take_while(|(k, _)| fail_at.as_ref().is_none_or(|at| *k < at)) .map(|(k, v)| { Ok(extenddb_backup_store::ObjectMeta { key: k.clone(), @@ -1242,6 +1399,11 @@ impl extenddb_backup_store::BackupStore for MemStore { }) }) .collect(); + if fail_at.is_some() { + metas.push(Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError("list stopped".to_owned()))), + })); + } Box::pin(futures::stream::iter(metas)) } @@ -1251,6 +1413,19 @@ impl extenddb_backup_store::BackupStore for MemStore { ) -> BoxedFuture<'a, Result> { Box::pin(async move { self.stall().await; + if self + .stall_deletes + .load(std::sync::atomic::Ordering::Acquire) + { + self.stalls + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + while self + .stall_deletes + .load(std::sync::atomic::Ordering::Acquire) + { + tokio::time::sleep(Duration::from_millis(10)).await; + } + } self.check()?; let delay = *self.delete_delay.lock().unwrap(); if let Some(delay) = delay { @@ -1287,8 +1462,16 @@ impl extenddb_backup_store::BackupStore for MemStore { } } -/// A runtime with `store` configured as `identity`, healthy. +/// A runtime with `store` configured as `identity`, healthy. A root with no +/// identity file gets one naming the `Fake`'s catalog, as the check that +/// made it healthy leaves it, so the deletion fence passes. fn runtime_on(mem: &Arc, identity: StoreIdentity) -> BackupRuntime { + if !mem.objects.lock().unwrap().contains_key(STORE_IDENTITY_KEY) { + mem.insert( + STORE_IDENTITY_KEY, + &identity_file(&identity.store_id, "catalog-1"), + ); + } runtime().with_store(ConfiguredStore::new( identity.store_kind, identity.store_root.clone(), @@ -1310,14 +1493,36 @@ fn runtime_unhealthy(mem: &Arc, problem: StoreProblem) -> BackupRuntim )) } +/// A deletion fence over a healthy store whose root's identity file names +/// the `Fake`'s catalog: it passes. For the tests that call a deleting step +/// with a store of their own. +fn passing_fence() -> super::StoreFence { + let mem = MemStore::default(); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::new(mem) as Arc, + None, + Ok(store()), + ) + .fence() +} + /// The identity file of `store()` for catalog `catalog_id`, as bytes. fn identity_file(store_id: &str, catalog_id: &str) -> Vec { + identity_file_of(store_id, catalog_id, false) +} + +/// [`identity_file`] with the in-memory catalog marker as given. +fn identity_file_of(store_id: &str, catalog_id: &str, ephemeral_catalog: bool) -> Vec { StoreIdentityFile { format_version: 1, store_id: store_id.to_owned(), catalog_id: catalog_id.to_owned(), backend: "test".to_owned(), created_at_ms: 0, + ephemeral_catalog, } .to_bytes() } @@ -1496,6 +1701,59 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(claim) }) } + /// The backends' unowned insert: the row as given, owner + /// `UNOWNED_GENERATION`, `completed_at` and `heartbeat_at` now; `false` + /// when a row with the backup id exists (PostgreSQL's, SQLite's and + /// MongoDB's unique `backup_id`, which also covers the key), or with + /// `unique_by_arn` a row under the ARN (Cassandra's), unless + /// `completes_own_insert` is set and the row is this insert's own. + fn insert_backup_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxedFuture<'a, Result> { + self.insert_attempts.lock().unwrap().push(row.clone()); + if let Some(e) = self.insert_error.lock().unwrap().clone() { + return Box::pin(async move { Err(StorageError::Internal(e)) }); + } + let mut rows = self.rows.lock().unwrap(); + for landed in self.lands_before_insert.lock().unwrap().drain(..) { + rows.insert(landed.backup_arn.clone(), landed); + } + if self + .completes_own_insert + .load(std::sync::atomic::Ordering::Acquire) + && rows.get(&row.backup_arn).is_some_and(|stored| { + stored.owner_generation == UNOWNED_GENERATION + && BackupRow { + owner_generation: row.owner_generation, + completed_at: row.completed_at, + heartbeat_at: row.heartbeat_at, + ..stored.clone() + } == *row + }) + { + self.unlisted.lock().unwrap().remove(&row.backup_arn); + return Box::pin(async move { Ok(true) }); + } + let exists = if self + .unique_by_arn + .load(std::sync::atomic::Ordering::Acquire) + { + rows.contains_key(&row.backup_arn) + } else { + rows.values().any(|r| r.backup_id == row.backup_id) + }; + if !exists { + let now = seconds_since_epoch(); + let mut inserted = row.clone(); + inserted.owner_generation = UNOWNED_GENERATION; + inserted.completed_at = Some(now); + inserted.heartbeat_at = now; + rows.insert(row.backup_arn.clone(), inserted); + } + Box::pin(async move { Ok(!exists) }) + } + /// The backends' registration: under the backup row's lock the row must /// be AVAILABLE (else `Backup not found`) and not in use (else /// `BackupInUse`), the name free (else `TableAlreadyExists`, or @@ -1665,6 +1923,7 @@ impl BackupCatalog for Fake { status: Option, store: &'a StoreIdentity, ) -> BoxStream<'a, Result> { + self.row_listings.lock().unwrap().push(status); if let Some(e) = self.list_error.lock().unwrap().clone() { return Box::pin(futures::stream::once(async move { Err(StorageError::Internal(e)) @@ -1677,6 +1936,7 @@ impl BackupCatalog for Fake { .values() .filter(|r| r.store == *store) .filter(|r| status.is_none_or(|s| r.status == s)) + .filter(|r| !self.unlisted.lock().unwrap().contains(&r.backup_arn)) .cloned() .collect(); rows.sort_by(|a, b| { @@ -1733,18 +1993,32 @@ impl BackupCatalog for Fake { } /// The backend's rule: the row of (account, id), and only when its - /// account, table and ARN equal the request's. + /// account, table and ARN equal the request's; with `unique_by_arn`, + /// the row of (account, ARN), as Cassandra keys it. fn read_backup_row<'a>( &'a self, arn: &'a BackupArn, ) -> BoxedFuture<'a, Result, StorageError>> { self.row_reads.lock().unwrap().push(arn.to_arn_string()); + if let Some(e) = self.read_row_error.lock().unwrap().clone() { + return Box::pin(async move { Err(StorageError::Internal(e)) }); + } + let by_arn = self + .unique_by_arn + .load(std::sync::atomic::Ordering::Acquire); let found = self .rows .lock() .unwrap() .values() - .find(|r| r.account_id == arn.account_id() && r.backup_id == arn.backup_id()) + .find(|r| { + r.account_id == arn.account_id() + && if by_arn { + r.backup_arn == arn.to_arn_string() + } else { + r.backup_id == arn.backup_id() + } + }) .filter(|r| r.table_name == arn.table_name() && r.backup_arn == arn.to_arn_string()) .cloned(); Box::pin(async move { Ok(found) }) @@ -1829,6 +2103,37 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(removed) }) } + /// The backends' outright removal, without the in-use rule; preset + /// `InUse` answers stand for it. A preset answer in `forget_answers` + /// leaves the row as the test set it; otherwise an AVAILABLE row goes, + /// another status is answered, and a missing row is `Backup not found`. + fn forget_available_backup_row<'a>( + &'a self, + key: &'a BackupKey, + ) -> BoxedFuture<'a, Result> { + self.forgets_seen.lock().unwrap().push(key.clone()); + let answer = self + .forget_answers + .lock() + .unwrap() + .get(&key.backup_arn) + .cloned(); + let mut rows = self.rows.lock().unwrap(); + let status = rows.get(&key.backup_arn).map(|r| r.status); + if answer.is_none() && status == Some(BackupStatus::Available) { + rows.remove(&key.backup_arn); + } + drop(rows); + Box::pin(async move { + match (answer, status) { + (Some(answer), _) => Ok(answer), + (None, Some(BackupStatus::Available)) => Ok(ForgetRow::Removed), + (None, Some(status)) => Ok(ForgetRow::Status(status)), + (None, None) => Err(StorageError::Validation("Backup not found: x".to_owned())), + } + }) + } + /// Remove a CREATING row whose heartbeat is older than `stale_after` /// (on this process's clock, which stands for the catalog's here). fn remove_dead_creating_backup_row<'a>( @@ -7421,7 +7726,11 @@ async fn a_store_backed_create_refuses_a_name_no_store_prefix_can_carry() { assert!(!msg.contains("not a backup ARN"), "{msg}"); } assert!(fake.rows.lock().unwrap().is_empty(), "no row was claimed"); - assert!(mem.keys().is_empty(), "nothing was written to the store"); + assert_eq!( + mem.keys(), + [STORE_IDENTITY_KEY], + "nothing was written to the store but the identity file" + ); assert_eq!(runtime.free_slots(), 4, "no slot was taken"); } @@ -7671,6 +7980,11 @@ fn held(holder: &Arc) -> maintenance::HeldClaim { ) } +/// A held claim whose heartbeats confirm, for the steps that re-check it. +fn confirmed_claim() -> maintenance::HeldClaim { + held(&Arc::new(FakeHolder::default())) +} + /// The claim object the `Fake`'s `maintenance_claim` hands out: `Forward` /// whose release also frees the `Fake`'s record, so a second command may /// take the claim once the first released it. @@ -7763,7 +8077,17 @@ async fn purge_finishes_a_row_another_deleter_moved_or_removed_meanwhile() { let mut outcomes = Vec::new(); for row in &read { - outcomes.push(maintenance::purge_row(&fake, &mem, row, Duration::from_secs(300)).await); + outcomes.push( + maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + row, + Duration::from_secs(300), + &confirmed_claim(), + ) + .await, + ); } assert_eq!( outcomes[0], @@ -7837,12 +8161,27 @@ async fn purge_removes_a_dead_creating_row_and_leaves_a_live_one() { let dead_row = fake.rows.lock().unwrap()[&dead].clone(); let live_row = fake.rows.lock().unwrap()[&live].clone(); assert_eq!( - maintenance::purge_row(&fake, &mem, &dead_row, window).await, + maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + &dead_row, + window, + &confirmed_claim() + ) + .await, Ok(()) ); - let err = maintenance::purge_row(&fake, &mem, &live_row, window) - .await - .unwrap_err(); + let err = maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + &live_row, + window, + &confirmed_claim(), + ) + .await + .unwrap_err(); assert!( err.contains("the CREATING row could not be removed"), "{err}" @@ -7854,7 +8193,15 @@ async fn purge_removes_a_dead_creating_row_and_leaves_a_live_one() { // moment has passed. tokio::time::sleep(Duration::from_millis(20)).await; assert_eq!( - maintenance::purge_row(&fake, &mem, &live_row, Duration::ZERO).await, + maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + &live_row, + Duration::ZERO, + &confirmed_claim() + ) + .await, Ok(()) ); assert!(fake.rows.lock().unwrap().is_empty()); @@ -8744,9 +9091,10 @@ async fn a_store_backed_restore_checks_the_target_name_after_the_row_and_before_ async fn establish_creates_the_identity_file_on_an_empty_root() { let mem = MemStore::default(); let fake = Fake::default(); - let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") - .await - .unwrap(); + let identity = + store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test", false) + .await + .unwrap(); assert_eq!(identity.store_kind, StoreKind::Filesystem); assert_eq!(identity.store_root, "/r"); assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]); @@ -8756,7 +9104,7 @@ async fn establish_creates_the_identity_file_on_an_empty_root() { assert_eq!(file.backend, "test"); assert_eq!(mem.puts.lock().unwrap().as_slice(), [STORE_IDENTITY_KEY]); // A second startup against the same root reads the file and uses its id. - let again = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") + let again = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test", false) .await .unwrap(); assert_eq!(again, identity); @@ -8774,8 +9122,8 @@ async fn two_instances_on_one_empty_root_end_with_one_id() { let mem = Arc::new(MemStore::default()); let fakes = [Fake::default(), Fake::default()]; let (a, b) = tokio::join!( - store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[0], "test"), - store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[1], "test"), + store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[0], "test", false), + store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[1], "test", false), ); let (a, b) = (a.unwrap(), b.unwrap()); assert_eq!(a, b); @@ -8797,7 +9145,7 @@ async fn a_file_naming_another_catalog_is_unhealthy_until_restored() { &identity_file("store-1", "another-catalog"), ); let fake = fake_with_row(BackupStatus::Available); - let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &*fake, "test") + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &*fake, "test", false) .await .unwrap_err(); assert_eq!( @@ -8812,7 +9160,7 @@ async fn a_file_naming_another_catalog_is_unhealthy_until_restored() { // Restored: the check passes. mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}"); - let identity = store_identity::recheck(&mem, &store(), &*fake, "test").await; + let identity = store_identity::recheck(&mem, &store(), &*fake, "test", false).await; assert_eq!(identity, Ok(())); } @@ -8826,7 +9174,7 @@ async fn a_non_object_at_the_identity_key_is_neither_readable_nor_creatable() { mem.key_taken_by_a_non_object .store(true, std::sync::atomic::Ordering::Release); let fake = Fake::default(); - let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test", false) .await .unwrap_err(); assert_eq!( @@ -8850,6 +9198,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { store().store_root.as_str(), &*fake, "test", + false, ) .await .unwrap_err(); @@ -8870,6 +9219,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { store().store_root.as_str(), &*fake, "test", + false, ) .await .unwrap_err(); @@ -8887,6 +9237,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { store().store_root.as_str(), &*fake, "test", + false, ) .await .unwrap(); @@ -8902,6 +9253,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { store().store_root.as_str(), &*fake, "test", + false, ) .await .unwrap_err(); @@ -8935,6 +9287,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &store().store_root, &*fake, "test", + false, ) .await .unwrap_err(); @@ -8952,6 +9305,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &store().store_root, &*fake, "test", + false, ) .await .unwrap(); @@ -8966,6 +9320,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &store().store_root, &*fake, "test", + false, ) .await .unwrap(); @@ -8980,7 +9335,7 @@ async fn a_create_race_lost_on_a_store_ignoring_the_condition_is_caught_by_the_r *mem.overwrite_after_conditional_put.lock().unwrap() = Some(identity_file("the-other-instance", "catalog-1")); let fake = Fake::default(); - let err = store_identity::establish(&mem, StoreKind::S3, "s3://b/", &fake, "test") + let err = store_identity::establish(&mem, StoreKind::S3, "s3://b/", &fake, "test", false) .await .unwrap_err(); match err { @@ -9013,6 +9368,7 @@ async fn an_unreachable_store_is_unhealthy_and_recovers_on_recheck() { &store().store_root, &*fake, "test", + false, ) .await .unwrap_err(); @@ -9162,7 +9518,7 @@ async fn a_stalled_store_check_is_abandoned_at_its_bound() { assert_eq!( configured.health(), StoreHealth::Unhealthy(StoreProblem::Transport( - "a call of an earlier check has not returned".to_owned() + "an abandoned check has not returned".to_owned() )) ); assert_eq!( @@ -9363,7 +9719,7 @@ fn record_check_reports_a_transition_only_across_categories() { async fn a_recheck_rewrites_a_lost_file_when_no_rows_exist() { let mem = MemStore::default(); let fake = Fake::default(); - store_identity::recheck(&mem, &store(), &fake, "test") + store_identity::recheck(&mem, &store(), &fake, "test", false) .await .unwrap(); let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(); @@ -9372,7 +9728,7 @@ async fn a_recheck_rewrites_a_lost_file_when_no_rows_exist() { // A file that changed under the running server (the root replaced) is // an id mismatch. mem.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1")); - let err = store_identity::recheck(&mem, &store(), &fake, "test") + let err = store_identity::recheck(&mem, &store(), &fake, "test", false) .await .unwrap_err(); assert_eq!( @@ -9455,7 +9811,7 @@ async fn an_oversized_identity_file_is_unreadable() { let mut big = identity_file("store-1", "catalog-1"); big.resize(store_identity::STORE_IDENTITY_MAX_BYTES + 1, b' '); mem.insert(STORE_IDENTITY_KEY, &big); - let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test", false) .await .unwrap_err(); assert!( @@ -9467,9 +9823,10 @@ async fn an_oversized_identity_file_is_unreadable() { let mut padded = identity_file("store-1", "catalog-1"); padded.resize(store_identity::STORE_IDENTITY_MAX_BYTES, b' '); mem.insert(STORE_IDENTITY_KEY, &padded); - let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") - .await - .unwrap(); + let identity = + store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test", false) + .await + .unwrap(); assert_eq!(identity.store_id, "store-1"); } @@ -9971,10 +10328,10 @@ async fn a_prefix_that_still_lists_an_object_is_left_deleting() { let store_handle = Refilling(MemStore::default()); let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]); let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap(); - let err = finish_backup_delete(&*fake, &store_handle, &parsed) + let err = finish_backup_delete(&*fake, &store_handle, &passing_fence(), &parsed) .await .unwrap_err(); - assert!(err.contains("still lists"), "{err}"); + assert!(err.to_string().contains("still lists"), "{err}"); assert_eq!( fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, BackupStatus::Deleting @@ -10084,9 +10441,16 @@ mod review_round_2 { seed(&fake, &mem, &arn, BackupStatus::Available); let prefix = BackupArn::parse(&arn).unwrap().store_prefix(); let row = fake.rows.lock().unwrap()[&arn].clone(); - maintenance::purge_row(&fake, &mem, &row, Duration::from_secs(300)) - .await - .unwrap(); + maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + &row, + Duration::from_secs(300), + &confirmed_claim(), + ) + .await + .unwrap(); assert_eq!( *mem.deletes.lock().unwrap(), vec![format!("{prefix}{MANIFEST_KEY_SUFFIX}"), prefix.clone()], @@ -10102,9 +10466,16 @@ mod review_round_2 { *mem.late_put_after_delete.lock().unwrap() = Some("data/late.json.gz".to_owned()); seed(&fake, &mem, &arn, BackupStatus::Available); let row = fake.rows.lock().unwrap()[&arn].clone(); - let err = maintenance::purge_row(&fake, &mem, &row, Duration::from_secs(300)) - .await - .unwrap_err(); + let err = maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + &row, + Duration::from_secs(300), + &confirmed_claim(), + ) + .await + .unwrap_err(); assert_eq!( err, format!("the prefix still lists {prefix}data/late.json.gz after the deletion") @@ -10126,9 +10497,16 @@ mod review_round_2 { .unwrap() .insert(arn.clone(), DeleteStart::InUse("Restored".to_owned())); let row = fake.rows.lock().unwrap()[&arn].clone(); - let err = maintenance::purge_row(&fake, &mem, &row, Duration::from_secs(300)) - .await - .unwrap_err(); + let err = maintenance::purge_row( + &fake, + &mem, + &passing_fence(), + &row, + Duration::from_secs(300), + &confirmed_claim(), + ) + .await + .unwrap_err(); assert_eq!(err, "a restore into Restored is reading it"); assert!(mem.deletes.lock().unwrap().is_empty(), "no delete ran"); assert_eq!(mem.keys().len(), 2, "the objects stay"); @@ -10378,6 +10756,7 @@ mod review_round_2 { let outcome = purge_rows( &fake, &mem, + &passing_fence(), &rows, Duration::from_secs(300), &claim, @@ -10409,6 +10788,7 @@ mod review_round_2 { let outcome = purge_rows( &fake, &mem, + &passing_fence(), &rows, Duration::from_secs(300), &claim, @@ -10443,6 +10823,7 @@ mod review_round_2 { let outcome = purge_rows( &fake, &mem, + &passing_fence(), &rest, Duration::from_secs(300), &claim, @@ -10522,8 +10903,9 @@ mod review_round_2 { } /// The deletions run under the store's bound and the in-flight rule: - /// DeleteBackup on a stalled store answers DELETED at the bound with the - /// row left DELETING, two DeleteBackups together park one thread between + /// DeleteBackup on a store whose deletions stall (its fence reads, in + /// their own slot, answer) answers DELETED at the bound with the row + /// left DELETING, two DeleteBackups together park one thread between /// them (the second waits for the first's slot and answers at its bound /// with nothing parked), and the reconciler's deletion of those rows /// returns at the bound too, with no second call parked beside the first. @@ -10539,7 +10921,8 @@ mod review_round_2 { ]); let runtime = runtime_on(&mem, store()); let configured = runtime.configured_store().unwrap(); - mem.set_stalled(true); + mem.stall_deletes + .store(true, std::sync::atomic::Ordering::Release); let started = tokio::time::Instant::now(); let (first, second) = tokio::time::timeout( Duration::from_secs(120), @@ -10581,7 +10964,8 @@ mod review_round_2 { // The mount answers: the parked call returns, the next cycle checks // and finishes the deletions. - mem.set_stalled(false); + mem.stall_deletes + .store(false, std::sync::atomic::Ordering::Release); tokio::time::sleep(Duration::from_millis(50)).await; assert!(!configured.check_call_in_flight()); let report = cycle_on(&fake, &runtime).await; @@ -11145,7 +11529,11 @@ async fn an_open_that_fails_answers_the_request_and_leaves_nothing() { wait_for_jobs(&runtime).await; assert!(fake.rows.lock().unwrap().is_empty()); assert!(fake.claims.lock().unwrap().is_empty()); - assert!(mem.keys().is_empty()); + assert_eq!( + mem.keys(), + [STORE_IDENTITY_KEY], + "no object but the identity file" + ); assert_eq!(runtime.free_slots(), 4); } @@ -11771,6 +12159,7 @@ async fn a_second_create_over_the_job_bound_is_limit_exceeded() { true, Some(Arc::clone(&gate)), ); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); let runtime = BackupRuntime::new(1) .with_store(ConfiguredStore::new( store().store_kind, @@ -12364,7 +12753,11 @@ async fn a_stopped_heartbeat_timer_starts_no_put() { ) ); assert!(mem.puts.lock().unwrap().is_empty()); - assert!(mem.keys().is_empty()); + assert_eq!( + mem.keys(), + [STORE_IDENTITY_KEY], + "no object but the identity file" + ); } /// The maintenance check runs before the permit and the row, so a slow @@ -15067,7 +15460,7 @@ async fn the_reconciler_resumes_a_dead_manifest_restore_with_its_spool_host() { /// The manifest arm's refusals: no free permit skips the record to the next /// cycle with no takeover; a held maintenance claim fails it `OWNER_DEAD` /// and removes its target like a legacy record; a record of another store -/// is FAILED naming `--move-store` and its target left to the FAILED-target +/// is FAILED with `OTHER_STORE` and its target left to the FAILED-target /// duty, which removes it next cycle; an unhealthy store skips it. #[tokio::test] async fn the_reconciler_skips_or_fails_a_dead_manifest_restore_it_cannot_resume() { @@ -17425,7 +17818,9 @@ async fn a_truncated_spooled_manifest_has_the_resume_read_the_store() { /// A resume whose store documents fail verification is FAILED with the /// reason, its target removed, and its duration recorded from the resume's /// start: a sidecar that does not match the manifest; a manifest naming -/// another backup under a sidecar that matches it. +/// another backup under a sidecar that matches it. A manifest that spells +/// the row's backup in another region (a backup sync adopted from another +/// region's prefix) verifies, and the resume completes. #[tokio::test] async fn a_resume_whose_store_documents_fail_verification_is_failed() { async fn resume_over(tamper: impl Fn(&MemStore, &str)) -> (Arc, BackupRuntime, LogLines) { @@ -17562,6 +17957,34 @@ async fn a_resume_whose_store_documents_fail_verification_is_failed() { .len(), 1 ); + + // The same backup spelled in another region, under a sidecar that + // matches it. + let (fake, _runtime, lines) = resume_over(|mem, prefix| { + let key = format!("{prefix}{EXTENDDB_MANIFEST_FILE}"); + let text = String::from_utf8(mem.objects.lock().unwrap()[&key].clone()).unwrap(); + let own = arn(ACCOUNT); + let respelled = text.replace(own.as_str(), &own.replace(REGION, "eu-west-1")); + assert_ne!(respelled, text); + let manifest = respelled.into_bytes(); + let sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&manifest) + ) + }; + mem.insert(&key, &manifest); + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + sidecar.as_bytes(), + ); + }) + .await; + assert!(lines.containing("FAILED on resume").is_empty()); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done, "{record:?}"); + assert_eq!(fake.restored_items("MusicRestored").len(), 4); } /// A same-host resume that fails before it runs (the spooled pair a crash @@ -19005,7 +19428,11 @@ async fn a_store_marked_unhealthy_or_renamed_mid_job_fails_the_row_with_no_store assert!(row.completed_at.is_some()); assert!(mem.puts.lock().unwrap().is_empty(), "no put"); assert!(mem.deletes.lock().unwrap().is_empty(), "no delete_prefix"); - assert!(mem.keys().is_empty()); + assert_eq!( + mem.keys(), + [STORE_IDENTITY_KEY], + "no object but the identity file" + ); assert_eq!( writes(&fake), [(JobStatus::Failed, Some(reason))], @@ -19185,7 +19612,16 @@ async fn a_re_read_showing_the_target_status_under_another_generation_is_not_mat fake.rows.lock().unwrap().insert(arn.clone(), r); let claim = fake.claim_on(&arn, 7); *fake.unknown_write.lock().unwrap() = Some(UnknownWrite::Lost); - let path = job::failure::fail_row(&*claim, &*mem, &prefix, "a reason", None).await; + let path = job::failure::fail_row( + &*claim, + &*mem, + &passing_fence(), + "catalog-1", + &prefix, + "a reason", + None, + ) + .await; match path { job::failure::FailurePath::NoStoreAccess(why) => { assert_eq!(why, "the row carries generation 9 (this claim's is 7)"); @@ -19601,3 +20037,3832 @@ async fn another_accounts_arn_in_another_region_is_access_denied_before_the_regi "{err:?}" ); } + +// --- backup sync (`sync`) ------------------------------------------------------ + +/// A backup ARN of `store()`'s account and table with a 14-digit id that +/// carries an instant, one per `n`. +fn sync_arn(n: u32) -> String { + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/0179197643{n:04}-0000{n:04}") +} + +/// The objects of a complete backup of `arn`, keys relative to the root: the +/// golden manifest's data files at their recorded sizes and the four +/// documents `BackupDocuments::build` writes, the source table renamed to +/// `arn`'s table, account, region and partition, the name `sync-`. +fn backup_objects(arn: &str) -> Vec<(String, Vec)> { + use extenddb_core::backup_format::{BackupDocuments, ExtenddbManifest, ManifestHeader}; + let golden = ExtenddbManifest::from_json_bytes(include_bytes!( + "../../../core/testdata/extenddb-manifest/v1.json" + )) + .unwrap(); + let parsed = BackupArn::parse(arn).unwrap(); + let mut source_table = golden.source_table.clone(); + source_table.table_name = parsed.table_name().to_owned(); + source_table.table_arn = format!( + "arn:{}:dynamodb:{}:{}:table/{}", + parsed.partition(), + parsed.region(), + parsed.account_id(), + parsed.table_name() + ); + let docs = BackupDocuments::build( + ManifestHeader { + backup_arn: parsed.clone(), + backup_name: format!("sync-{}", parsed.backup_id()), + source_backend: golden.source_backend.clone(), + source_extenddb_version: golden.source_extenddb_version.clone(), + snapshot: golden.snapshot.clone(), + item_order: golden.item_order, + source_table, + }, + golden + .data_files + .iter() + .cloned() + .map(|entry| (entry, None)) + .collect(), + "", + &store().store_root, + ) + .unwrap(); + let prefix = parsed.store_prefix(); + let mut objects: Vec<(String, Vec)> = golden + .data_files + .iter() + .map(|file| { + ( + format!("{prefix}{}", file.key), + vec![0u8; usize::try_from(file.bytes).unwrap()], + ) + }) + .collect(); + for (name, bytes) in docs.documents() { + objects.push((format!("{prefix}{name}"), bytes.to_vec())); + } + objects +} + +/// Write the objects of a backup of `arn` into `mem` with its manifest +/// changed by `edit` and a sidecar that matches the changed manifest: a +/// manifest whose digest checks and whose content the reader judges. +fn put_backup_with_manifest(mem: &MemStore, arn: &str, edit: impl Fn(&mut Value)) { + use sha2::Digest as _; + for (key, bytes) in backup_objects(arn) { + if key.ends_with("extenddb-manifest.sha256") { + continue; + } + if key.ends_with("extenddb-manifest.json") { + let mut manifest: Value = serde_json::from_slice(&bytes).unwrap(); + edit(&mut manifest); + let edited = serde_json::to_vec(&manifest).unwrap(); + let sidecar = format!( + "{} extenddb-manifest.json\n", + hex::encode(sha2::Sha256::digest(&edited)) + ); + mem.insert(&key.replace(".json", ".sha256"), sidecar.as_bytes()); + mem.insert(&key, &edited); + } else { + mem.insert(&key, &bytes); + } + } +} + +/// Write a complete backup of `arn` into `mem`. +fn put_backup(mem: &MemStore, arn: &str) { + for (key, bytes) in backup_objects(arn) { + mem.insert(&key, &bytes); + } +} + +fn prefix_of(arn: &str) -> String { + BackupArn::parse(arn).unwrap().store_prefix() +} + +/// A sync of `mem` as `identity`, in the tests' region. +fn sync_target<'a>(mem: &'a MemStore, identity: &'a StoreIdentity) -> super::sync::SyncTarget<'a> { + super::sync::SyncTarget { + store: mem, + identity, + metrics: None, + region: Some(REGION), + stop: None, + claim: None, + } +} + +/// The reasons of a report's skipped prefixes, by prefix. +fn skipped_by_prefix(report: &super::sync::SyncReport) -> HashMap { + report + .skipped + .iter() + .map(|s| (s.prefix.clone(), s.reason.clone())) + .collect() +} + +/// A prefix with a verified manifest and no row gets an AVAILABLE row with +/// the values CreateBackup's AVAILABLE write leaves: the manifest's name, +/// ARN, size, item total and `source_table`, the configured store's +/// identity, `created_at` the id's instant, unowned, `completed_at` and +/// `heartbeat_at` the catalog's clock. A second sync finds the row and +/// skips the prefix; a reconciler cycle then leaves the row as it is, as it +/// leaves a created one. +#[tokio::test] +async fn sync_adopts_a_verified_prefix_with_the_row_a_create_writes() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let adopted = sync_arn(1); + put_backup(&mem, &adopted); + let identity = store(); + + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!( + report.adopted, + [super::sync::AdoptedBackup { + backup_arn: adopted.clone(), + foreign: None, + }] + ); + assert!(report.skipped.is_empty(), "{:?}", report.skipped); + assert!(report.missing_manifests.is_empty()); + assert_eq!(report.stopped, None); + let manifest = stored_manifest(&mem, &adopted); + let parsed = BackupArn::parse(&adopted).unwrap(); + let row = fake.rows.lock().unwrap()[&adopted].clone(); + assert_eq!(row.status, BackupStatus::Available); + assert_eq!(row.backup_arn, adopted); + assert_eq!(row.account_id, parsed.account_id()); + assert_eq!(row.table_name, parsed.table_name()); + assert_eq!(row.backup_id, parsed.backup_id()); + assert_eq!(row.backup_name, manifest.backup_name); + assert_eq!(row.backup_name, format!("sync-{}", parsed.backup_id())); + assert_eq!(row.store, identity); + assert_eq!(row.size_bytes, manifest.backup_size_bytes); + assert_eq!(row.item_count, 100_000); + assert_eq!( + Some(row.created_at), + extenddb_core::backup_format::backup_creation_date_time(parsed.backup_id()) + ); + assert_eq!( + row.source_table_details_json, + serde_json::to_value(&manifest.source_table).unwrap() + ); + assert_eq!(row.owner_generation, UNOWNED_GENERATION); + assert!(row.completed_at.is_some()); + assert_eq!(row.failure_reason, None); + + let again = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(again.adopted.is_empty()); + assert_eq!( + skipped_by_prefix(&again), + HashMap::from([( + prefix_of(&adopted), + super::sync::SkipReason::HasRow { + status: BackupStatus::Available, + store_id: "store-1".to_owned(), + other_table: None, + } + )]) + ); + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 1); + + let runtime = runtime_on(&mem, store()); + let cycle = cycle_on(&fake, &runtime).await; + assert_eq!(cycle.store_healthy, Some(true)); + assert_eq!( + fake.rows.lock().unwrap()[&adopted], + row, + "the cycle left it" + ); +} + +/// A manifest whose ARN names another region than the server's is adopted +/// under the server's region's spelling of the ARN, and the report (and the +/// log) says so, naming the manifest's region; the manifest keeps the +/// spelling it carries. DescribeBackup finds the row under either spelling +/// and answers the server's, ListBackups lists it in the server's region, +/// and the dead-owner probe and the restore's manifest check accept the +/// manifest's spelling. With no region known, the row carries the +/// manifest's spelling and no region is compared. +#[tokio::test] +async fn sync_adopts_a_backup_of_another_region_under_this_servers_region() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let canonical = sync_arn(7); + let other = canonical.replace(REGION, "eu-west-1"); + put_backup(&mem, &other); + let identity = store(); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!(report.adopted.len(), 1, "{:?}", report.skipped); + assert_eq!(report.adopted[0].backup_arn, canonical); + let note = report.adopted[0].foreign.clone().unwrap(); + assert!(note.contains("eu-west-1"), "{note}"); + { + let rows = fake.rows.lock().unwrap(); + assert_eq!(rows[&canonical].backup_arn, canonical); + assert!( + !rows.contains_key(&other), + "no row under the manifest's spelling" + ); + } + assert_eq!( + stored_manifest(&mem, &other) + .backup_arn() + .unwrap() + .to_arn_string(), + other, + "the manifest keeps its spelling" + ); + assert_eq!( + fake.insert_attempts.lock().unwrap()[0].backup_arn, + canonical + ); + // Both spellings resolve to the row, which reads as the server's. + for spelling in [&canonical, &other] { + let (resolved, row) = read_catalog_row(&*fake, &own(spelling)) + .await + .unwrap() + .expect("found under either spelling"); + assert_eq!(resolved.to_arn_string(), canonical, "{spelling}"); + assert_eq!(row.backup_arn, canonical, "{spelling}"); + } + // The readers that compare a manifest's ARN with the row's. + let manifest = stored_manifest(&mem, &other); + let row_arn = BackupArn::parse(&canonical).unwrap(); + assert!(manifest.backup_arn().unwrap().names_same_backup(&row_arn)); + + let fake = Arc::new(Fake::default()); + let unknown_region = super::sync::SyncTarget { + region: None, + ..sync_target(&mem, &identity) + }; + let report = super::sync::sync(&*fake, unknown_region).await.unwrap(); + assert_eq!(report.adopted[0].foreign, None); + assert_eq!(report.adopted[0].backup_arn, other); + assert!(fake.rows.lock().unwrap().contains_key(&other)); +} + +/// A prefix copied from a server of another region, whose manifest spells +/// the backup's ARN and the source table's in that region, is adopted by +/// sync under this server's region: the row's `BackupArn` is this server's, +/// and the source table's ARN keeps the manifest's region in the three +/// fields that answer it (DescribeBackup's `SourceTableDetails.TableArn`, +/// ListBackups' `BackupSummaries[].TableArn` and the restore's +/// `RestoreSummary.SourceTableArn`). The request's manifest check and the +/// job's accept the manifest's spelling, and the restore reaches ACTIVE +/// with every item. A manifest of another backup is still refused +/// (`the_manifest_request_refuses_in_order_before_the_registration`). +#[tokio::test] +async fn a_row_over_a_manifest_of_another_region_restores() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs(2)]; + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + // The copied prefix alone: the row is sync's to write. + fake.rows.lock().unwrap().clear(); + let runtime = job_runtime(&mem); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let manifest_key = format!("{prefix}{EXTENDDB_MANIFEST_FILE}"); + let sidecar_key = format!("{prefix}{MANIFEST_CHECKSUM_FILE}"); + let other = arn(ACCOUNT).replace(REGION, "eu-west-1"); + assert_ne!(other, arn(ACCOUNT)); + let other_table = manifest.source_table.table_arn.replace(REGION, "eu-west-1"); + assert_ne!(other_table, manifest.source_table.table_arn); + let mut parsed: Value = + serde_json::from_slice(&mem.objects.lock().unwrap()[&manifest_key]).unwrap(); + parsed["backup_arn"] = json!(other); + parsed["source_table"]["table_arn"] = json!(other_table); + let bytes = serde_json::to_vec(&parsed).unwrap(); + let sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&bytes) + ) + }; + mem.insert(&manifest_key, &bytes); + mem.insert(&sidecar_key, sidecar.as_bytes()); + + let report = super::sync::sync(&*fake, sync_target(&mem, &store())) + .await + .unwrap(); + assert_eq!(report.adopted.len(), 1, "{:?}", report.skipped); + assert_eq!(report.adopted[0].backup_arn, arn(ACCOUNT)); + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap(); + assert_eq!(desc.backup_details.backup_arn, arn(ACCOUNT)); + assert_eq!(desc.source_table_details.table_arn, other_table); + let listed = list(&fake, json!({})).await.unwrap(); + let summaries = listed["BackupSummaries"].as_array().unwrap(); + assert_eq!(summaries.len(), 1); + assert_eq!(summaries[0]["BackupArn"], arn(ACCOUNT)); + assert_eq!(summaries[0]["TableArn"], other_table); + + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!( + response["TableDescription"]["RestoreSummary"]["SourceBackupArn"], + arn(ACCOUNT) + ); + assert_eq!( + response["TableDescription"]["RestoreSummary"]["SourceTableArn"], + other_table + ); + wait_for_jobs(&runtime).await; + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done, "{record:?}"); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.tables.lock().unwrap()["MusicRestored"].table_status, + TableStatus::Active + ); +} + +/// Every prefix 4(g) says sync never adopts is skipped with its reason, and +/// the run goes on to adopt the one complete prefix among them: no manifest, +/// a manifest without its sidecar, a sidecar that does not match, a data +/// file missing, a data file of another size, a manifest naming another +/// backup than the path, a manifest the store fails to serve, a manifest +/// whose sidecar matches and that the reader refuses (a `source_table` of +/// another table, a creation time that is not the id's instant, a data-file +/// key that leaves the prefix, a size total that is not the files' sum), +/// the reserved `changelog` and `exports/` components, an id of an earlier +/// build's shape, and a complete prefix with a row of each status. The +/// manifest of a prefix with a row is never read: the row check comes +/// first. +#[tokio::test] +async fn sync_skips_every_prefix_it_must_not_adopt() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + let manifest_name = "extenddb-manifest.json"; + let sidecar_name = "extenddb-manifest.sha256"; + + let good = sync_arn(1); + put_backup(&mem, &good); + let none = sync_arn(2); + mem.insert(&format!("{}data/000001.json.gz", prefix_of(&none)), b"x"); + let no_sidecar = sync_arn(3); + for (key, bytes) in backup_objects(&no_sidecar) { + if !key.ends_with(sidecar_name) { + mem.insert(&key, &bytes); + } + } + let bad_sidecar = sync_arn(4); + for (key, bytes) in backup_objects(&bad_sidecar) { + if key.ends_with(sidecar_name) { + mem.insert( + &key, + format!("{} {manifest_name}\n", "0".repeat(64)).as_bytes(), + ); + } else { + mem.insert(&key, &bytes); + } + } + let missing_file = sync_arn(5); + let mut objects = backup_objects(&missing_file); + let dropped = objects + .iter() + .position(|(k, _)| k.contains("/data/")) + .unwrap(); + objects.remove(dropped); + for (key, bytes) in objects { + mem.insert(&key, &bytes); + } + let short_file = sync_arn(6); + for (key, mut bytes) in backup_objects(&short_file) { + if key.contains("/data/") && bytes.len() > 1 { + bytes.pop(); + } + mem.insert(&key, &bytes); + } + // The documents and files of backup 8, under backup 7's prefix. + let mismatched = sync_arn(7); + for (key, bytes) in backup_objects(&sync_arn(8)) { + let moved = key.replacen(&prefix_of(&sync_arn(8)), &prefix_of(&mismatched), 1); + mem.insert(&moved, &bytes); + } + let unreadable = sync_arn(9); + put_backup(&mem, &unreadable); + *mem.fail_gets_under.lock().unwrap() = Some(prefix_of(&unreadable)); + let other_source = sync_arn(10); + put_backup_with_manifest(&mem, &other_source, |m| { + m["source_table"]["table_name"] = "Other".into(); + }); + let other_instant = sync_arn(11); + put_backup_with_manifest(&mem, &other_instant, |m| { + m["backup_creation_date_time"] = 1.5.into(); + }); + let traversal = sync_arn(12); + put_backup_with_manifest(&mem, &traversal, |m| { + m["data_files"][0]["key"] = "../../../../x/data/000001.json.gz".into(); + }); + let other_size = sync_arn(13); + put_backup_with_manifest(&mem, &other_size, |m| { + m["backup_size_bytes"] = 7.into(); + }); + mem.insert(&format!("{ACCOUNT}/Music/changelog/000001.json.gz"), b"x"); + mem.insert("exports/01791976430001/manifest-summary.json", b"{}"); + mem.insert( + &format!("{ACCOUNT}/Music/1791976431438/{manifest_name}"), + b"{}", + ); + let mut with_rows = Vec::new(); + for (n, status) in [ + (20, BackupStatus::Creating), + (21, BackupStatus::Deleting), + (22, BackupStatus::Deleted), + (23, BackupStatus::Failed), + (24, BackupStatus::Available), + ] { + let a = sync_arn(n); + put_backup(&mem, &a); + fake.rows.lock().unwrap().insert(a.clone(), row(&a, status)); + with_rows.push((a, status)); + } + let rows_before: Vec = fake.rows.lock().unwrap().values().cloned().collect(); + + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!( + report + .adopted + .iter() + .map(|a| a.backup_arn.clone()) + .collect::>(), + std::slice::from_ref(&good) + ); + let skipped = skipped_by_prefix(&report); + let incomplete = |arn: &str| match &skipped[&prefix_of(arn)] { + SkipReason::Incomplete(why) => why.clone(), + other => panic!("{arn}: {other:?}"), + }; + assert_eq!(incomplete(&none), "no manifest"); + assert_eq!(incomplete(&no_sidecar), "a manifest without its sidecar"); + assert!( + incomplete(&bad_sidecar).starts_with("the sidecar does not match"), + "{}", + incomplete(&bad_sidecar) + ); + assert!( + incomplete(&missing_file).contains("is not in the store"), + "{}", + incomplete(&missing_file) + ); + assert!( + incomplete(&short_file).contains("bytes in the store"), + "{}", + incomplete(&short_file) + ); + assert!( + matches!(&skipped[&prefix_of(&mismatched)], SkipReason::ArnMismatch(why) if why.contains(&sync_arn(8))), + "{:?}", + skipped[&prefix_of(&mismatched)] + ); + assert!( + matches!(&skipped[&prefix_of(&unreadable)], SkipReason::Store(e) if e.contains("get refused")), + "{:?}", + skipped[&prefix_of(&unreadable)] + ); + for (refused, says) in [ + (&other_source, "source_table.table_name is \"Other\""), + (&other_instant, "does not match the instant in backup_arn"), + (&traversal, "must start with the data/ prefix"), + ( + &other_size, + "backup_size_bytes is 7 but the data files total", + ), + ] { + let why = incomplete(refused); + assert!( + why.starts_with("the manifest is refused: ") && why.contains(says), + "{refused}: {why}" + ); + } + assert_eq!( + skipped[&format!("{ACCOUNT}/Music/changelog/")], + SkipReason::Reserved("changelog") + ); + assert_eq!(skipped["exports/"], SkipReason::Reserved("exports")); + assert!( + matches!(&skipped[&format!("{ACCOUNT}/Music/1791976431438/")], SkipReason::NotABackup(why) if why.contains("earlier build")), + "{:?}", + skipped[&format!("{ACCOUNT}/Music/1791976431438/")] + ); + for (a, status) in &with_rows { + assert_eq!( + skipped[&prefix_of(a)], + SkipReason::HasRow { + status: *status, + store_id: "store-1".to_owned(), + other_table: None, + }, + "{a}" + ); + assert!( + !mem.gets().iter().any(|k| k.starts_with(&prefix_of(a))), + "{a}: read before its row was checked" + ); + } + assert_eq!(report.skipped.len(), 19, "{:?}", report.skipped); + // One insert, the good prefix's; every row as it was. + assert_eq!( + fake.insert_attempts + .lock() + .unwrap() + .iter() + .map(|r| r.backup_arn.clone()) + .collect::>(), + std::slice::from_ref(&good) + ); + let rows = fake.rows.lock().unwrap(); + for before in &rows_before { + assert_eq!(rows[&before.backup_arn], *before); + } + assert_eq!(rows.len(), rows_before.len() + 1); +} + +/// The report names the AVAILABLE rows of the configured store whose +/// manifest the listing does not hold (no prefix at all, or a prefix +/// without its manifest), not one whose prefix is complete, nor a row of +/// the root under another store id, with its prefix complete (the prefix +/// is skipped: the row check covers every id of the root) or gone (it is +/// no candidate, and `--prune-missing` over the report leaves it). A +/// prefix whose backup id a row of another root holds is skipped naming +/// that store. Nothing is modified without `--prune-missing`. +#[tokio::test] +async fn sync_reports_missing_manifests_and_rows_of_other_stores_and_changes_none() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + let no_prefix = sync_arn(1); + let no_manifest = sync_arn(2); + let complete = sync_arn(3); + let other_id = sync_arn(4); + let other_root = sync_arn(5); + let other_id_gone = sync_arn(6); + for a in [ + &no_prefix, + &no_manifest, + &complete, + &other_id, + &other_id_gone, + ] { + fake.rows + .lock() + .unwrap() + .insert(a.clone(), row(a, BackupStatus::Available)); + } + for a in [&other_id, &other_id_gone] { + fake.rows.lock().unwrap().get_mut(a).unwrap().store.store_id = "store-0".to_owned(); + } + let mut elsewhere = row(&other_root, BackupStatus::Available); + elsewhere.store.store_root = "/srv/elsewhere".to_owned(); + fake.rows + .lock() + .unwrap() + .insert(other_root.clone(), elsewhere.clone()); + mem.insert( + &format!("{}data/000001.json.gz", prefix_of(&no_manifest)), + b"x", + ); + // A key that ends in the manifest's name below the prefix is not the + // prefix's manifest: the row is still reported. + mem.insert( + &format!("{}data/x-extenddb-manifest.json", prefix_of(&no_manifest)), + b"{}", + ); + for a in [&complete, &other_id, &other_root] { + put_backup(&mem, a); + } + let rows_before: Vec = fake.rows.lock().unwrap().values().cloned().collect(); + + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty(), "{:?}", report.adopted); + let mut missing: Vec = report + .missing_manifests + .iter() + .map(|r| r.backup_arn.clone()) + .collect(); + missing.sort(); + assert_eq!(missing, [no_prefix.clone(), no_manifest.clone()]); + let skipped = skipped_by_prefix(&report); + assert_eq!( + skipped[&prefix_of(&other_id)], + SkipReason::HasRow { + status: BackupStatus::Available, + store_id: "store-0".to_owned(), + other_table: None, + } + ); + assert!( + matches!(&skipped[&prefix_of(&other_root)], SkipReason::RowExists(what) if what.contains("of another store") && what.contains("/srv/elsewhere")), + "{:?}", + skipped[&prefix_of(&other_root)] + ); + { + let rows = fake.rows.lock().unwrap(); + assert_eq!(rows.len(), rows_before.len()); + for before in &rows_before { + assert_eq!(rows[&before.backup_arn], *before); + } + } + + // `--prune-missing` over the report removes the two candidates and + // leaves the row of the other store id, whose manifest is gone too. + let (outcomes, stopped) = super::sync::prune_missing( + &*fake, + &*mem, + &identity.store_root, + &report.missing_manifests, + &confirmed_claim(), + &|| None, + ) + .await; + assert_eq!(stopped, None); + assert_eq!(outcomes.len(), 2, "{outcomes:?}"); + let rows = fake.rows.lock().unwrap(); + assert!(!rows.contains_key(&no_prefix) && !rows.contains_key(&no_manifest)); + assert_eq!(rows[&other_id_gone].status, BackupStatus::Available); + assert_eq!(rows[&other_id_gone].store.store_id, "store-0"); +} + +/// The insert-if-absent decides against what the read of the rows did not +/// see, and each case leaves the row it met: a CreateBackup on another +/// instance whose CREATING row landed between the read and the insert; a +/// row the catalog holds that its listings do not return (Cassandra's +/// insert that ended between its two writes); a row of the backup id under +/// another account. The first two read the same: the row met is read by +/// its ARN alone, with no listing of the rows of its status. +#[tokio::test] +async fn an_insert_that_meets_a_row_leaves_it_and_says_which() { + use super::sync::SkipReason; + let identity = store(); + let reason = |report: &super::sync::SyncReport, arn: &str| match &skipped_by_prefix(report) + [&prefix_of(arn)] + { + SkipReason::RowExists(what) => what.clone(), + other => panic!("{arn}: {other:?}"), + }; + + // A CreateBackup racing the sync. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let racing = sync_arn(1); + put_backup(&mem, &racing); + let creating = row(&racing, BackupStatus::Creating); + fake.lands_before_insert + .lock() + .unwrap() + .push(creating.clone()); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty()); + let what = reason(&report, &racing); + assert!( + what.contains("(CREATING) exists that the catalog's read at the start did not return"), + "{what}" + ); + assert!( + !fake + .row_listings + .lock() + .unwrap() + .iter() + .any(Option::is_some), + "no listing of one status to word the line" + ); + assert_eq!(fake.rows.lock().unwrap()[&racing], creating); + + // A row the listings do not return. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let unlisted = sync_arn(2); + put_backup(&mem, &unlisted); + let available = row(&unlisted, BackupStatus::Available); + fake.rows + .lock() + .unwrap() + .insert(unlisted.clone(), available.clone()); + fake.unlisted.lock().unwrap().insert(unlisted.clone()); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty()); + let what = reason(&report, &unlisted); + assert!( + what.contains("(AVAILABLE) exists that the catalog's read at the start did not return"), + "{what}" + ); + assert_eq!(fake.rows.lock().unwrap()[&unlisted], available); + + // The backup id under another account. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let ours = sync_arn(3); + put_backup(&mem, &ours); + let theirs = ours.replace(ACCOUNT, "210987654321"); + let other_account = row(&theirs, BackupStatus::Available); + fake.rows + .lock() + .unwrap() + .insert(theirs.clone(), other_account.clone()); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty()); + let what = reason(&report, &ours); + assert!(what.contains("under another account"), "{what}"); + assert_eq!(fake.rows.lock().unwrap().len(), 1); + assert_eq!(fake.rows.lock().unwrap()[&theirs], other_account); +} + +/// A listing that fails stops the sync with no row written, and so does a +/// read of the rows that fails. +#[tokio::test] +async fn a_failed_listing_or_row_read_stops_sync_with_no_row_written() { + let identity = store(); + let mem = Arc::new(MemStore::default()); + put_backup(&mem, &sync_arn(1)); + let fake = Arc::new(Fake::default()); + *mem.fail_lists_of.lock().unwrap() = Some(String::new()); + let err = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap_err(); + assert!( + matches!(&err, super::sync::SyncError::Listing { partial, .. } if partial.adopted.is_empty()), + "{err}" + ); + assert!(err.to_string().contains("list refused"), "{err}"); + *mem.fail_lists_of.lock().unwrap() = None; + + *fake.list_error.lock().unwrap() = Some("pool exhausted".to_owned()); + let err = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap_err(); + assert!(matches!(err, super::sync::SyncError::Catalog(_)), "{err}"); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + assert!(fake.rows.lock().unwrap().is_empty()); +} + +/// `--prune-missing` over the rows a sync reported: a row whose manifest is +/// gone is removed outright, with no DELETING or DELETED tombstone and no +/// object deleted (the objects left under its prefix stay, as a host-local +/// root's bytes on another host would); a row a restore is reading, a row +/// another writer moved, a row whose manifest is back, a row gone meanwhile +/// and a row whose manifest's `head` fails are each reported and left; the +/// rows a restore is reading and the failed `head`'s row are the ones a +/// later run may remove (the command exits 1 for them). A stop asked before +/// the first row changes nothing. DeleteBackup's start is never called. +#[tokio::test] +async fn prune_missing_removes_rows_whose_manifest_is_gone_and_deletes_nothing() { + use super::sync::PruneOutcome; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gone = sync_arn(1); + let leftover = sync_arn(2); + let in_use = sync_arn(3); + let moved = sync_arn(4); + let back = sync_arn(5); + let removed = sync_arn(6); + let unanswered = sync_arn(7); + let rows: Vec = [ + &gone, + &leftover, + &in_use, + &moved, + &back, + &removed, + &unanswered, + ] + .into_iter() + .map(|a| row(a, BackupStatus::Available)) + .collect(); + for r in &rows { + fake.rows + .lock() + .unwrap() + .insert(r.backup_arn.clone(), r.clone()); + } + let leftover_key = format!("{}data/000001.json.gz", prefix_of(&leftover)); + mem.insert(&leftover_key, b"x"); + fake.forget_answers + .lock() + .unwrap() + .insert(in_use.clone(), ForgetRow::InUse("Target".to_owned())); + fake.rows.lock().unwrap().get_mut(&moved).unwrap().status = BackupStatus::Deleting; + mem.insert(&manifest_key(&back), b"{}"); + fake.rows.lock().unwrap().remove(&removed); + *mem.fail_heads_of.lock().unwrap() = Some(manifest_key(&unanswered)); + + let keys: Vec = rows.iter().map(BackupRow::key).collect(); + let root = store().store_root; + let (none, stopped) = + super::sync::prune_missing(&*fake, &*mem, &root, &keys, &confirmed_claim(), &|| { + Some("stopped".to_owned()) + }) + .await; + assert!(none.is_empty()); + assert_eq!(stopped.as_deref(), Some("stopped")); + assert!(fake.forgets_seen.lock().unwrap().is_empty()); + + let keys_before = mem.keys(); + let (outcomes, stopped) = + super::sync::prune_missing(&*fake, &*mem, &root, &keys, &confirmed_claim(), &|| None).await; + assert_eq!(stopped, None); + assert_eq!( + outcomes, + [ + (gone.clone(), PruneOutcome::Pruned), + (leftover.clone(), PruneOutcome::Pruned), + (in_use.clone(), PruneOutcome::InUse("Target".to_owned())), + (moved.clone(), PruneOutcome::Status(BackupStatus::Deleting)), + (back.clone(), PruneOutcome::ManifestBack), + (removed.clone(), PruneOutcome::Gone), + ( + unanswered.clone(), + PruneOutcome::Failed( + "the manifest's head failed: transport error: dispatch failure: head refused" + .to_owned() + ), + ), + ] + ); + assert_eq!( + outcomes + .iter() + .map(|(_, outcome)| outcome.needs_another_run()) + .collect::>(), + [false, false, true, false, false, false, true] + ); + assert!( + !fake + .forgets_seen + .lock() + .unwrap() + .iter() + .any(|key| key.backup_arn == unanswered), + "a failed head removes nothing" + ); + let after = fake.rows.lock().unwrap(); + assert!(!after.contains_key(&gone), "no tombstone is left"); + assert!(!after.contains_key(&leftover), "no tombstone is left"); + assert_eq!(after[&in_use].status, BackupStatus::Available); + assert_eq!(after[&moved].status, BackupStatus::Deleting); + assert_eq!(after[&back].status, BackupStatus::Available); + assert_eq!(after[&unanswered].status, BackupStatus::Available); + assert_eq!(mem.keys(), keys_before, "no object was deleted"); + assert!(mem.keys().contains(&leftover_key)); + assert!(fake.delete_starts_seen.lock().unwrap().is_empty()); +} + +/// A stop asked after the first removal (a SIGTERM during a long prune) +/// ends the loop before the next row: one `Pruned` outcome, the reason +/// returned, the later rows still AVAILABLE and never offered to the +/// removal. The removal is logged under the audit target, once. +#[tokio::test] +async fn prune_missing_stops_after_a_removal_when_asked() { + use super::sync::PruneOutcome; + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let arns: Vec = (1..=3).map(sync_arn).collect(); + for a in &arns { + fake.rows + .lock() + .unwrap() + .insert(a.clone(), row(a, BackupStatus::Available)); + } + let keys: Vec = arns + .iter() + .map(|a| row(a, BackupStatus::Available).key()) + .collect(); + let removals = || fake.forgets_seen.lock().unwrap().len(); + let stop = || (removals() >= 1).then(|| "SIGTERM".to_owned()); + let (outcomes, stopped) = super::sync::prune_missing( + &*fake, + &*mem, + &store().store_root, + &keys, + &confirmed_claim(), + &stop, + ) + .await; + assert_eq!(outcomes, [(arns[0].clone(), PruneOutcome::Pruned)]); + assert_eq!(stopped.as_deref(), Some("SIGTERM")); + assert_eq!(removals(), 1); + let rows = fake.rows.lock().unwrap(); + assert!(!rows.contains_key(&arns[0])); + for later in &arns[1..] { + assert_eq!(rows[later].status, BackupStatus::Available, "{later}"); + } + assert_eq!( + lines + .containing("removed an AVAILABLE row whose manifest is not under the root") + .len(), + 1 + ); +} + +/// `--adopt-store`'s rewrite: the file names this catalog, keeps its store +/// id and creation time, and is written once; the identity is the file's +/// id; the previous owner's next check finds the root owned by another +/// catalog. On a root this catalog owns nothing is written. A root whose +/// rows in this catalog carry another id, and a root with no file, are +/// refused with nothing written. +#[tokio::test] +async fn adopt_store_makes_this_catalog_the_owner_and_keeps_the_store_id() { + use super::store_identity::{AdoptRefusal, AdoptRule, Adoption, adopt}; + let file = StoreIdentityFile { + format_version: 1, + store_id: "store-7".to_owned(), + catalog_id: "previous".to_owned(), + backend: "postgres".to_owned(), + created_at_ms: 1234, + ephemeral_catalog: false, + }; + let mem = MemStore::default(); + mem.insert(STORE_IDENTITY_KEY, &file.to_bytes()); + let fake = Fake::default(); + let adopted = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &fake, + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap(); + let identity = StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/r".to_owned(), + store_id: "store-7".to_owned(), + }; + assert_eq!( + adopted, + Adoption::Adopted { + identity: identity.clone(), + previous_catalog_id: "previous".to_owned(), + } + ); + let now = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(); + assert_eq!( + now, + StoreIdentityFile { + catalog_id: "catalog-1".to_owned(), + backend: "sqlite".to_owned(), + ..file.clone() + } + ); + assert_eq!(mem.puts.lock().unwrap().as_slice(), [STORE_IDENTITY_KEY]); + // The previous owner's next check. + let previous = Fake::default(); + *previous.catalog_id.lock().unwrap() = Some("previous".to_owned()); + assert_eq!( + store_identity::recheck(&mem, &identity, &previous, "postgres", false).await, + Err(StoreProblem::ForeignCatalog { + file_catalog_id: "catalog-1".to_owned(), + catalog_id: "previous".to_owned(), + }) + ); + // Already this catalog's: nothing written. + let again = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &fake, + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap(); + assert_eq!(again, Adoption::AlreadyOwned(identity.clone())); + assert_eq!(mem.puts.lock().unwrap().len(), 1); + + // This catalog's rows for the root carry another id. + let mem = MemStore::default(); + mem.insert(STORE_IDENTITY_KEY, &file.to_bytes()); + let fake = fake_with_row(BackupStatus::Available); + let err = adopt( + &mem, + StoreKind::Filesystem, + &store().store_root, + &*fake, + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap_err(); + assert_eq!( + err, + AdoptRefusal::RowsCarryOtherIds { + file_id: "store-7".to_owned(), + row_ids: vec!["store-1".to_owned()], + } + ); + assert!(mem.puts.lock().unwrap().is_empty()); + assert_eq!( + StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(), + file + ); + + // No file. + let mem = MemStore::default(); + let err = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &Fake::default(), + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap_err(); + assert_eq!(err, AdoptRefusal::NoFile { row_ids: vec![] }); + assert!(matches!( + err.into_problem("catalog-1"), + StoreProblem::FileUnreadable(_) + )); + assert!(mem.keys().is_empty()); +} + +/// Two adopters of one root race through a plain put: the one whose write +/// is replaced before its re-read reports the race lost, naming the +/// catalog the file names now. +#[tokio::test] +async fn an_adoption_overwritten_before_its_re_read_reports_the_race_lost() { + use super::store_identity::{AdoptRefusal, AdoptRule, adopt}; + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-7", "previous")); + let third = identity_file("store-7", "third"); + let racer = Arc::clone(&mem); + *mem.on_put.lock().unwrap() = Some(Box::new(move |key: &str| { + if key == STORE_IDENTITY_KEY { + racer.insert(STORE_IDENTITY_KEY, &third); + } + })); + let err = adopt( + &*mem, + StoreKind::Filesystem, + "/r", + &Fake::default(), + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap_err(); + *mem.on_put.lock().unwrap() = None; + assert_eq!( + err, + AdoptRefusal::RaceLost { + file_catalog_id: "third".to_owned(), + file_store_id: "store-7".to_owned(), + } + ); + assert_eq!( + err.into_problem("catalog-1"), + StoreProblem::ForeignCatalog { + file_catalog_id: "third".to_owned(), + catalog_id: "catalog-1".to_owned(), + } + ); + // A re-read that names this catalog under another store id (the root + // replaced by another root of the same catalog between the put and the + // re-read) is lost too: the file is not the one this adoption wrote. + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-7", "previous")); + let replaced = identity_file("store-8", "catalog-1"); + let racer = Arc::clone(&mem); + *mem.on_put.lock().unwrap() = Some(Box::new(move |key: &str| { + if key == STORE_IDENTITY_KEY { + racer.insert(STORE_IDENTITY_KEY, &replaced); + } + })); + let err = adopt( + &*mem, + StoreKind::Filesystem, + "/r", + &Fake::default(), + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap_err(); + *mem.on_put.lock().unwrap() = None; + assert_eq!( + err, + AdoptRefusal::RaceLost { + file_catalog_id: "catalog-1".to_owned(), + file_store_id: "store-8".to_owned(), + } + ); +} + +/// `--adopt-store` runs the check the server runs on the root as it will +/// be once this catalog owns it before anything is written: with this +/// catalog's newest AVAILABLE row of the root missing its manifest, the +/// adoption is refused as `Unusable` naming the probe's problem, no put is +/// made, the file still names its owner, and no audit line says a rewrite +/// started. +#[tokio::test] +async fn adopt_store_refuses_a_root_this_catalog_cannot_use_with_nothing_written() { + use super::store_identity::{AdoptRefusal, AdoptRule, adopt}; + let (lines, _guard) = LogLines::capture(); + let mem = MemStore::default(); + let owned = identity_file("store-1", "previous"); + mem.insert(STORE_IDENTITY_KEY, &owned); + let fake = fake_with_row(BackupStatus::Available); + let err = adopt( + &mem, + StoreKind::Filesystem, + &store().store_root, + &*fake, + "sqlite", + false, + AdoptRule::Command, + Some(&confirmed_claim()), + ) + .await + .unwrap_err(); + assert!( + matches!(&err, AdoptRefusal::Unusable { file_catalog_id, problem: StoreProblem::ManifestMissing { .. } } + if file_catalog_id == "previous"), + "{err:?}" + ); + let text = err.to_string(); + assert!( + text.contains("nothing was written") && text.contains("still names catalog previous"), + "{text}" + ); + assert!(mem.puts.lock().unwrap().is_empty()); + assert_eq!(mem.objects.lock().unwrap()[STORE_IDENTITY_KEY], owned); + assert!(lines.containing("to name this catalog").is_empty()); + assert!(matches!( + err.into_problem("catalog-1"), + StoreProblem::ManifestMissing { .. } + )); +} + +/// The refusals of `adopt`'s first read, for a file changed between the +/// command's check and the read: a root whose file is gone under a row of +/// this catalog names the rows' ids (`FileMissingWithRows`, which plain +/// sync refuses too), and a file naming this catalog over rows under two +/// store ids is `MultipleRowIds`; neither writes anything. +#[tokio::test] +async fn adopt_stores_first_read_refuses_a_missing_file_under_rows_and_rows_of_two_ids() { + use super::store_identity::{AdoptRefusal, AdoptRule, adopt}; + let root = store().store_root; + let mem = MemStore::default(); + let fake = fake_with_row(BackupStatus::Available); + let err = adopt( + &mem, + StoreKind::Filesystem, + &root, + &*fake, + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap_err(); + assert_eq!( + err, + AdoptRefusal::NoFile { + row_ids: vec!["store-1".to_owned()] + } + ); + assert_eq!( + err.into_problem("catalog-1"), + StoreProblem::FileMissingWithRows(vec!["store-1".to_owned()]) + ); + assert!(mem.keys().is_empty(), "nothing was written"); + + let mem = MemStore::default(); + let file = identity_file("store-1", "catalog-1"); + mem.insert(STORE_IDENTITY_KEY, &file); + let fake = fake_with_row(BackupStatus::Available); + let mut other = row(&newer_arn(ACCOUNT), BackupStatus::Available); + other.store.store_id = "store-9".to_owned(); + fake.rows + .lock() + .unwrap() + .insert(other.backup_arn.clone(), other); + let err = adopt( + &mem, + StoreKind::Filesystem, + &root, + &*fake, + "sqlite", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap_err(); + let AdoptRefusal::Problem(StoreProblem::MultipleRowIds(mut ids)) = err else { + panic!("{err:?}"); + }; + ids.sort(); + assert_eq!(ids, ["store-1", "store-9"]); + assert!(mem.puts.lock().unwrap().is_empty()); + assert_eq!(mem.objects.lock().unwrap()[STORE_IDENTITY_KEY], file); +} + +/// `--adopt-store` on a store that stops answering ends at the store's +/// bound, at the call that did not finish, and the refusal names that +/// step: the first read of the file, the re-read after the rewrite (the +/// put landed), or, on a root this catalog already owns, the manifest +/// probe's `head`. The catalog's reads are not under the bound. +#[tokio::test(start_paused = true)] +async fn adopt_store_on_a_store_that_stops_answering_names_the_step() { + use super::store_identity::{AdoptRefusal, adopt_configured}; + for (cell, step, puts) in [ + ("the first read", "the first read of extenddb-store.json", 0), + ( + "the re-read", + "the re-read after the rewrite of extenddb-store.json", + 1, + ), + ( + "the probe of an owned root", + "the manifest probe of extenddb-store.json", + 0, + ), + ] { + let mem = Arc::new(MemStore::default()); + let owned = cell.contains("owned"); + let file_catalog = if owned { "catalog-1" } else { "previous" }; + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", file_catalog)); + let configured = ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ); + let fake = if owned { + // The probe `head`s the newest AVAILABLE row's manifest, and + // that call never answers. + let manifest_key = format!("{}{EXTENDDB_MANIFEST_FILE}", prefix_of(&arn(ACCOUNT))); + *mem.stall_heads_of.lock().unwrap() = Some(manifest_key); + fake_with_row(BackupStatus::Available) + } else { + failing_identity(&mem, puts, IdentityFailure::Stall); + Arc::new(Fake::default()) + }; + let started = tokio::time::Instant::now(); + let err = tokio::time::timeout( + Duration::from_secs(300), + adopt_configured(&configured, &*fake, "sqlite", None), + ) + .await + .expect("the adoption is bounded") + .unwrap_err(); + assert_eq!(started.elapsed(), configured.check_timeout(), "{cell}"); + let AdoptRefusal::Problem(StoreProblem::Transport(text)) = &err else { + panic!("{cell}: {err:?}"); + }; + assert!( + text.starts_with(step) && text.contains("did not finish within 5 s"), + "{cell}: {text}" + ); + assert!(err.to_string().contains(step), "{cell}: {err}"); + assert_eq!(mem.puts.lock().unwrap().len(), puts, "{cell}"); + assert!(configured.check_call_in_flight(), "{cell}: parked"); + } +} + +/// A server whose own catalog is in memory adopts, in its startup check, a +/// root whose file another in-memory catalog wrote, keeping the store id +/// and writing the marker; a root a durable catalog owns stays owned by +/// another catalog, its file untouched; a durable server never adopts a +/// marked root; and once the identity is known the re-check never adopts, +/// so a second in-memory server's later adoption leaves the first one's +/// store owned by another catalog. +#[tokio::test] +async fn an_in_memory_catalog_adopts_only_a_root_another_in_memory_catalog_owned() { + // Marked: adopted. + let mem = MemStore::default(); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "previous-start", true), + ); + let fake = Fake::default(); + let identity = + store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "sqlite", true) + .await + .unwrap(); + assert_eq!(identity.store_id, "store-7"); + let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(); + assert_eq!(file.catalog_id, "catalog-1"); + assert_eq!(file.store_id, "store-7"); + assert!(file.ephemeral_catalog); + + // A durable owner: unhealthy, nothing written. + let mem = MemStore::default(); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "durable", false), + ); + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "sqlite", true) + .await + .unwrap_err(); + assert_eq!( + err, + StoreProblem::ForeignCatalog { + file_catalog_id: "durable".to_owned(), + catalog_id: "catalog-1".to_owned(), + } + ); + assert!(mem.puts.lock().unwrap().is_empty()); + + // A durable server and a marked root: unhealthy, nothing written. + let mem = MemStore::default(); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "previous-start", true), + ); + let err = + store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "postgres", false) + .await + .unwrap_err(); + assert!(matches!(err, StoreProblem::ForeignCatalog { .. }), "{err}"); + assert!(mem.puts.lock().unwrap().is_empty()); + + // An empty root: the file an in-memory catalog creates carries the + // marker. + let mem = MemStore::default(); + store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "sqlite", true) + .await + .unwrap(); + let created = + StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(); + assert!(created.ephemeral_catalog); + + // The identity known: another in-memory server took the root since. + let mem = MemStore::default(); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "second-server", true), + ); + let known = StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/r".to_owned(), + store_id: "store-7".to_owned(), + }; + let err = store_identity::recheck(&mem, &known, &fake, "sqlite", true) + .await + .unwrap_err(); + assert!(matches!(err, StoreProblem::ForeignCatalog { .. }), "{err}"); + assert!(mem.puts.lock().unwrap().is_empty()); +} + +/// A runtime whose store `mem` is unchecked, for an in-memory catalog when +/// `ephemeral` is set (`store()`'s kind and root). +fn runtime_checking(mem: &Arc, ephemeral: bool) -> BackupRuntime { + runtime().with_store( + ConfiguredStore::checking( + store().store_kind, + store().store_root, + Arc::clone(mem) as Arc, + None, + ) + .with_ephemeral_catalog(ephemeral), + ) +} + +/// The startup of a server whose catalog is in memory, against the root +/// the previous start owned: the first reconciler cycle's check adopts the +/// root and the startup sync gives the previous start's backup its row +/// again, under the file's store id; the next cycle syncs no more. A root a +/// durable catalog owns stays unhealthy and nothing is synced, the sync +/// left pending. A listing that fails leaves the sync pending, and the next +/// cycle runs it. +#[tokio::test] +async fn the_startup_sync_of_an_in_memory_catalog_serves_the_previous_starts_backups() { + let previous = sync_arn(1); + + let mem = Arc::new(MemStore::default()); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "previous-start", true), + ); + put_backup(&mem, &previous); + // A prefix with no manifest: skipped, and no later run adopts it + // unless the store changes, so it leaves the sync done. + let incomplete = sync_arn(2); + mem.insert( + &format!("{}data/000001.json.gz", prefix_of(&incomplete)), + b"x", + ); + let fake = Arc::new(Fake::default()); + let runtime = runtime_checking(&mem, true); + let store_of = + |runtime: &BackupRuntime| runtime.configured_store().unwrap().startup_sync_pending(); + assert!(store_of(&runtime)); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + let row = fake.rows.lock().unwrap()[&previous].clone(); + assert_eq!(row.status, BackupStatus::Available); + assert_eq!(row.store.store_id, "store-7"); + assert!(!fake.rows.lock().unwrap().contains_key(&incomplete)); + assert!( + !store_of(&runtime), + "an incomplete prefix leaves the sync done" + ); + cycle_on(&fake, &runtime).await; + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 1, "synced once"); + + // A durable owner. + let mem = Arc::new(MemStore::default()); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "durable", false), + ); + put_backup(&mem, &previous); + let fake = Arc::new(Fake::default()); + let runtime = runtime_checking(&mem, true); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(false)); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + assert!(store_of(&runtime)); + + // The listing fails once. + let mem = Arc::new(MemStore::default()); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "previous-start", true), + ); + put_backup(&mem, &previous); + *mem.fail_lists_of.lock().unwrap() = Some(String::new()); + let fake = Arc::new(Fake::default()); + let runtime = runtime_checking(&mem, true); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert!(fake.rows.lock().unwrap().is_empty()); + assert!(store_of(&runtime)); + *mem.fail_lists_of.lock().unwrap() = None; + cycle_on(&fake, &runtime).await; + assert_eq!( + fake.rows.lock().unwrap()[&previous].status, + BackupStatus::Available + ); + assert!(!store_of(&runtime)); + + // A durable catalog has no startup sync. + let mem = Arc::new(MemStore::default()); + put_backup(&mem, &previous); + let fake = Arc::new(Fake::default()); + let runtime = runtime_checking(&mem, false); + assert!(!store_of(&runtime)); + cycle_on(&fake, &runtime).await; + assert!(fake.insert_attempts.lock().unwrap().is_empty()); +} + +/// The server's own call of the startup sync (before it serves, a +/// filesystem store whose check passed) runs it once; a run abandoned at +/// the server's bound (its future dropped while the store does not answer) +/// leaves it pending for the reconciler. +#[tokio::test] +async fn a_startup_sync_abandoned_at_its_bound_stays_pending() { + let mem = Arc::new(MemStore::default()); + put_backup(&mem, &sync_arn(1)); + let fake = Arc::new(Fake::default()); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + let pending = || runtime.configured_store().unwrap().startup_sync_pending(); + mem.set_stalled(true); + let abandoned = tokio::time::timeout( + Duration::from_millis(50), + super::sync::startup_sync(&*fake, &runtime), + ) + .await; + assert!(abandoned.is_err()); + assert!(mem.stalls.load(std::sync::atomic::Ordering::Acquire) > 0); + assert!(pending(), "left pending"); + let configured = runtime.configured_store().unwrap(); + assert!( + configured.check_call_in_flight(), + "the dropped run's call is marked in flight" + ); + mem.set_stalled(false); + for _ in 0..200 { + if !configured.check_call_in_flight() { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert!(!configured.check_call_in_flight(), "the call returned"); + let ran = super::sync::startup_sync(&*fake, &runtime).await; + assert!(matches!(ran, Some(Ok(ref report)) if report.adopted.len() == 1)); + assert!(!pending()); + assert!(super::sync::startup_sync(&*fake, &runtime).await.is_none()); +} + +/// A reconciler cycle whose startup sync meets a root listing that does +/// not answer within the sync's bound returns once that bound passes. The +/// listing runs as the store's bounded calls run a filesystem call, so the +/// abandoned walk is marked in flight: the cycle's store duties are refused +/// at once rather than starting a call beside it (the DELETING row stays +/// for a later cycle), the sync stays pending, and the next cycle's check +/// is refused at once with no second listing of the root, recording the +/// store unreachable with a text that names the startup sync's call. Once +/// the walk ends, the next cycle's check passes, the sync runs and adopts, +/// and the deletion is finished. +#[tokio::test(start_paused = true)] +async fn a_cycles_startup_sync_cut_at_its_bound_leaves_its_walk_in_flight() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let deleting = arn(ACCOUNT); + seed(&fake, &mem, &deleting, BackupStatus::Deleting); + put_backup(&mem, &sync_arn(1)); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + let configured = runtime.configured_store().unwrap(); + mem.hold_root_list + .store(true, std::sync::atomic::Ordering::Release); + let started = tokio::time::Instant::now(); + // A guard far past the bound, so a startup sync with no bound fails + // here instead of hanging. + let report = tokio::time::timeout( + super::sync::STARTUP_SYNC_TIMEOUT * 5, + cycle_on(&fake, &runtime), + ) + .await + .expect("the startup sync is bounded"); + let took = started.elapsed(); + assert_eq!( + report.store_healthy, + Some(true), + "{:?}", + configured.health() + ); + assert!(took >= super::sync::STARTUP_SYNC_TIMEOUT, "{took:?}"); + assert!( + took < super::sync::STARTUP_SYNC_TIMEOUT + Duration::from_secs(1), + "{took:?}: the duties after the sync waited for nothing" + ); + assert!(configured.check_call_in_flight(), "the walk is marked"); + assert!(report.deletions_finished.is_empty()); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!( + fake.rows.lock().unwrap()[&deleting].status, + BackupStatus::Deleting + ); + assert!(configured.startup_sync_pending(), "left pending"); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + let root_lists = || { + mem.lists + .lock() + .unwrap() + .iter() + .filter(|p| p.is_empty()) + .count() + }; + assert_eq!(root_lists(), 1); + + let started = tokio::time::Instant::now(); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(started.elapsed(), Duration::ZERO); + assert_eq!(report.store_healthy, Some(false)); + assert_eq!( + configured.health(), + StoreHealth::Unhealthy(StoreProblem::Transport( + "an abandoned startup sync's store call has not returned".to_owned() + )) + ); + assert_eq!(runtime.store_health().unwrap().reason, Some("unreachable")); + assert_eq!(root_lists(), 1, "no second walk beside the first"); + assert!(configured.startup_sync_pending()); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + + // The walk ends: its task clears the mark, and the next cycle's check + // passes, runs the sync and finishes the deletion. + mem.hold_root_list + .store(false, std::sync::atomic::Ordering::Release); + for _ in 0..200 { + if !configured.check_call_in_flight() { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + assert!(!configured.check_call_in_flight(), "the walk returned"); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.store_healthy, + Some(true), + "{:?}", + configured.health() + ); + assert!(!configured.startup_sync_pending(), "the sync ran"); + assert_eq!( + fake.rows.lock().unwrap()[&sync_arn(1)].status, + BackupStatus::Available, + "adopted by the later cycle's run" + ); + assert_eq!(report.deletions_finished, vec![deleting.clone()]); + assert_eq!( + mem.deletes.lock().unwrap().as_slice(), + [manifest_key(&deleting), prefix_of(&deleting)], + "the manifest first, then the prefix" + ); +} + +/// The startup sync's per-prefix lines: the first run, which lists the +/// whole root, logs each prefix it skipped (INFO, or WARN for a store +/// error); a second run of the sync left pending logs the store error's +/// WARN again and the other skips at DEBUG, so its INFO and WARN lines are +/// the transient skip's and the line of counts. +#[tokio::test] +async fn a_second_startup_sync_run_logs_only_the_transient_skips_and_the_counts() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + // Three prefixes skipped for good (a row of each) and one whose + // manifest the store fails to serve. + for n in 1..=3 { + let held = sync_arn(n); + put_backup(&mem, &held); + fake.rows + .lock() + .unwrap() + .insert(held.clone(), row(&held, BackupStatus::Available)); + } + let failing = sync_arn(4); + put_backup(&mem, &failing); + *mem.fail_gets_under.lock().unwrap() = Some(prefix_of(&failing)); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + let configured = runtime.configured_store().unwrap(); + let ours = |level| { + lines + .at_level(level) + .into_iter() + .filter(|m| m.contains("startup sync")) + .collect::>() + }; + + let first = super::sync::startup_sync(&*fake, &runtime).await; + assert!(matches!(first, Some(Ok(ref r)) if r.transient_skips() == 1)); + assert!(configured.startup_sync_pending(), "left pending"); + let first_info = ours(tracing::Level::INFO); + let first_warn = ours(tracing::Level::WARN); + assert_eq!( + first_info.len(), + 3 + 1, + "three skips and the counts: {first_info:?}" + ); + assert_eq!(first_warn.len(), 1, "{first_warn:?}"); + + let second = super::sync::startup_sync(&*fake, &runtime).await; + assert!(matches!(second, Some(Ok(ref r)) if r.skipped.len() == 4)); + let info = ours(tracing::Level::INFO); + let warn = ours(tracing::Level::WARN); + assert_eq!( + info.len() - first_info.len(), + 1, + "the line of counts alone: {:?}", + &info[first_info.len()..] + ); + assert!( + info.last().unwrap().contains("4 prefix(es) skipped"), + "{info:?}" + ); + assert_eq!( + warn.len() - first_warn.len(), + 1, + "the transient skip: {warn:?}" + ); + assert!( + warn.last().unwrap().contains(&prefix_of(&failing)), + "{warn:?}" + ); + assert_eq!( + ours(tracing::Level::DEBUG).len(), + 3, + "the second run's lasting skips" + ); + assert!(configured.startup_sync_reported()); +} + +/// Three startup sync runs over a root with one prefix skipped for good (a +/// row holds its backup) and one whose manifest the store fails to serve, +/// the first run ending before it reports: listing the root fails +/// (`cut_at_bound` false) or the listing stalls past +/// [`super::sync::STARTUP_SYNC_TIMEOUT`] (true). That run names no prefix. +/// The second run lists the whole root and is the first to report, so it +/// logs the lasting skip at INFO; the third logs it at DEBUG. The transient +/// skip is a WARN line in the second and third runs. +async fn startup_sync_logs_its_skips_at_info_until_a_run_has_reported(cut_at_bound: bool) { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let held = sync_arn(1); + put_backup(&mem, &held); + fake.rows + .lock() + .unwrap() + .insert(held.clone(), row(&held, BackupStatus::Available)); + let failing = sync_arn(2); + put_backup(&mem, &failing); + *mem.fail_gets_under.lock().unwrap() = Some(prefix_of(&failing)); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + let configured = runtime.configured_store().unwrap(); + let naming = |level, prefix: &str| { + lines + .at_level(level) + .into_iter() + .filter(|m| m.contains("startup sync") && m.contains(prefix)) + .count() + }; + + if cut_at_bound { + mem.hold_root_list + .store(true, std::sync::atomic::Ordering::Release); + let started = tokio::time::Instant::now(); + let first = tokio::time::timeout( + super::sync::STARTUP_SYNC_TIMEOUT * 5, + super::sync::startup_sync(&*fake, &runtime), + ) + .await + .expect("the startup sync is bounded"); + // The run's bound and its listing call's bound are one length: the + // run ends at the bound either way, with no report. + assert!(matches!(first, None | Some(Err(_))), "{first:?}"); + assert!(started.elapsed() >= super::sync::STARTUP_SYNC_TIMEOUT); + assert!( + configured.check_call_in_flight(), + "the abandoned walk is marked" + ); + mem.hold_root_list + .store(false, std::sync::atomic::Ordering::Release); + for _ in 0..200 { + if !configured.check_call_in_flight() { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + assert!(!configured.check_call_in_flight(), "the walk returned"); + } else { + *mem.fail_lists_of.lock().unwrap() = Some(String::new()); + let first = super::sync::startup_sync(&*fake, &runtime).await; + assert!(matches!(first, Some(Err(_))), "{first:?}"); + *mem.fail_lists_of.lock().unwrap() = None; + } + assert!(configured.startup_sync_pending(), "left pending"); + assert!(!configured.startup_sync_reported()); + let held_prefix = prefix_of(&held); + let failing_prefix = prefix_of(&failing); + for level in [ + tracing::Level::INFO, + tracing::Level::WARN, + tracing::Level::DEBUG, + ] { + assert_eq!(naming(level, &held_prefix), 0, "{level}: no prefix named"); + } + + let second = super::sync::startup_sync(&*fake, &runtime).await; + assert!( + matches!(second, Some(Ok(ref r)) if r.skipped.len() == 2), + "{second:?}" + ); + assert!(configured.startup_sync_reported()); + assert!(configured.startup_sync_pending(), "the transient skip"); + assert_eq!( + naming(tracing::Level::INFO, &held_prefix), + 1, + "the first run to report names the lasting skip at INFO" + ); + assert_eq!(naming(tracing::Level::DEBUG, &held_prefix), 0); + assert_eq!(naming(tracing::Level::WARN, &failing_prefix), 1); + + let third = super::sync::startup_sync(&*fake, &runtime).await; + assert!( + matches!(third, Some(Ok(ref r)) if r.skipped.len() == 2), + "{third:?}" + ); + assert_eq!( + naming(tracing::Level::INFO, &held_prefix), + 1, + "no second INFO line" + ); + assert_eq!(naming(tracing::Level::DEBUG, &held_prefix), 1); + assert_eq!(naming(tracing::Level::WARN, &failing_prefix), 2); +} + +#[tokio::test] +async fn a_startup_sync_whose_first_run_fails_logs_its_skips_at_info_on_the_next() { + startup_sync_logs_its_skips_at_info_until_a_run_has_reported(false).await; +} + +#[tokio::test(start_paused = true)] +async fn a_startup_sync_whose_first_run_is_cut_at_its_bound_logs_its_skips_at_info_on_the_next() { + startup_sync_logs_its_skips_at_info_until_a_run_has_reported(true).await; +} + +/// A startup sync another caller has taken is not run again: with the +/// pending sync taken, `startup_sync` answers `None` with no listing of the +/// root and no row read, and the sync is not handed back. +#[tokio::test] +async fn a_taken_startup_sync_is_not_run_by_a_second_caller() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + put_backup(&mem, &sync_arn(1)); + let fake = Arc::new(Fake::default()); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + let configured = runtime.configured_store().unwrap(); + assert!(configured.startup_sync_pending()); + assert!(configured.take_startup_sync(), "the first caller takes it"); + assert!(super::sync::startup_sync(&*fake, &runtime).await.is_none()); + assert!(mem.lists.lock().unwrap().is_empty(), "no listing"); + assert!(fake.row_listings.lock().unwrap().is_empty(), "no row read"); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + assert!(!configured.startup_sync_pending(), "not handed back"); +} + +/// A manifest the path does not name is skipped whatever component differs +/// (another account, another table), and so is one in another partition +/// whose components match the path: every request refuses an ARN outside +/// `aws` before it reads a row, so no request could address the row. +/// Nothing is inserted. +#[tokio::test] +async fn sync_skips_a_manifest_of_another_account_table_or_partition() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + let moved_under = |from: &str, to: &str| { + for (key, bytes) in backup_objects(from) { + mem.insert(&key.replacen(&prefix_of(from), &prefix_of(to), 1), &bytes); + } + }; + let account_path = sync_arn(1); + let other_account = account_path.replace(ACCOUNT, "210987654321"); + moved_under(&other_account, &account_path); + let table_path = sync_arn(2); + let other_table = table_path.replace("table/Music", "table/Other"); + moved_under(&other_table, &table_path); + let partition = sync_arn(3) + .replace("arn:aws:", "arn:aws-cn:") + .replace(REGION, "cn-north-1"); + put_backup(&mem, &partition); + assert_eq!(prefix_of(&partition), prefix_of(&sync_arn(3))); + + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty(), "{:?}", report.adopted); + let skipped = skipped_by_prefix(&report); + assert!( + matches!(&skipped[&prefix_of(&account_path)], SkipReason::ArnMismatch(why) if why.contains(&other_account)), + "{:?}", + skipped[&prefix_of(&account_path)] + ); + assert!( + matches!(&skipped[&prefix_of(&table_path)], SkipReason::ArnMismatch(why) if why.contains(&other_table)), + "{:?}", + skipped[&prefix_of(&table_path)] + ); + assert!( + matches!(&skipped[&prefix_of(&partition)], SkipReason::OtherPartition(why) + if why.contains("aws-cn") && why.contains("no request")), + "{:?}", + skipped[&prefix_of(&partition)] + ); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + assert!(fake.rows.lock().unwrap().is_empty()); +} + +/// Only an AVAILABLE row of the configured store is reported with a missing +/// manifest: a CREATING, DELETING, DELETED or FAILED row with no prefix is +/// not, so `--prune-missing` never examines it. +#[tokio::test] +async fn only_available_rows_are_reported_with_a_missing_manifest() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + for (n, status) in [ + (1, BackupStatus::Creating), + (2, BackupStatus::Failed), + (3, BackupStatus::Deleting), + (4, BackupStatus::Deleted), + (5, BackupStatus::Available), + ] { + let a = sync_arn(n); + fake.rows.lock().unwrap().insert(a.clone(), row(&a, status)); + } + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!( + report + .missing_manifests + .iter() + .map(|r| r.backup_arn.clone()) + .collect::>(), + [sync_arn(5)] + ); +} + +/// A prefix whose backup id a row of this root holds under another table +/// (the prefix copied under a second table directory) is skipped naming +/// the row's table, never read. +#[tokio::test] +async fn a_prefix_under_another_table_with_a_rows_backup_id_is_skipped_naming_the_table() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + let original = sync_arn(1); + fake.rows + .lock() + .unwrap() + .insert(original.clone(), row(&original, BackupStatus::Available)); + put_backup(&mem, &original); + let copy = original.replace("table/Music", "table/Copy"); + for (key, bytes) in backup_objects(&original) { + mem.insert( + &key.replacen(&prefix_of(&original), &prefix_of(©), 1), + &bytes, + ); + } + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + let skipped = skipped_by_prefix(&report); + assert_eq!( + skipped[&prefix_of(©)], + SkipReason::HasRow { + status: BackupStatus::Available, + store_id: "store-1".to_owned(), + other_table: Some("Music".to_owned()), + } + ); + assert!( + skipped[&prefix_of(©)] + .to_string() + .contains("under table Music"), + "{}", + skipped[&prefix_of(©)] + ); + assert!(!mem.gets().iter().any(|k| k.starts_with(&prefix_of(©)))); +} + +/// Before the insert the row is read under each spelling of the manifest's +/// ARN: a row of another root under the manifest's own spelling (another +/// region's) holds the backup though sync would write the row under this +/// server's region, and the prefix is skipped naming that store, with no +/// insert (on Cassandra the insert alone would not see it). A row of +/// another root under this server's spelling, the one the insert writes, +/// is met by the insert, which answers `false`, and the prefix is skipped +/// naming that store too. +#[tokio::test] +async fn sync_skips_a_backup_a_row_holds_under_another_spelling_of_its_arn() { + use super::sync::SkipReason; + let identity = store(); + let canonical = sync_arn(1); + let spelled = canonical.replace(REGION, "eu-west-1"); + for (cell, held_under, inserts) in [ + ("the manifest's spelling", &spelled, 0), + ("this server's spelling", &canonical, 1), + ] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut elsewhere = row(held_under, BackupStatus::Available); + elsewhere.store.store_root = "/srv/elsewhere".to_owned(); + fake.rows + .lock() + .unwrap() + .insert(held_under.clone(), elsewhere.clone()); + put_backup(&mem, &spelled); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty(), "{cell}"); + assert!( + matches!(&skipped_by_prefix(&report)[&prefix_of(&spelled)], SkipReason::RowExists(what) + if what.contains("of another store") && what.contains("/srv/elsewhere")), + "{cell}: {:?}", + report.skipped + ); + assert_eq!( + fake.insert_attempts.lock().unwrap().len(), + inserts, + "{cell}" + ); + let reads = fake.row_reads.lock().unwrap().clone(); + assert!( + reads.contains(&canonical) && (inserts == 1 || reads.contains(&spelled)), + "{cell}: the spellings were read in turn: {reads:?}" + ); + assert_eq!(fake.rows.lock().unwrap().len(), 1, "{cell}"); + assert_eq!(fake.rows.lock().unwrap()[held_under], elsewhere, "{cell}"); + } +} + +/// An adopted row whose list and status entries are gone (a Cassandra +/// insert that stopped between the row and its entries) is not in the rows +/// sync reads first, and the read under the manifest's own spelling finds +/// it: the next sync passes it to the insert, which completes it, and the +/// backup is adopted and listed again. A row under the other spelling of +/// the ARN still skips the prefix with no insert. +#[tokio::test] +async fn the_next_sync_completes_its_own_insert_that_stopped_before_its_entries() { + use super::sync::SkipReason; + let identity = store(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + fake.completes_own_insert + .store(true, std::sync::atomic::Ordering::Release); + let torn = sync_arn(1); + put_backup(&mem, &torn); + let first = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!(first.adopted.len(), 1); + fake.unlisted.lock().unwrap().insert(torn.clone()); + let stored = fake.rows.lock().unwrap()[&torn].clone(); + + let second = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!( + second.adopted, + [super::sync::AdoptedBackup { + backup_arn: torn.clone(), + foreign: None, + }], + "{:?}", + second.skipped + ); + assert!(second.skipped.is_empty(), "{:?}", second.skipped); + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 2); + assert!( + !fake.unlisted.lock().unwrap().contains(&torn), + "listed again" + ); + assert_eq!(fake.rows.lock().unwrap()[&torn], stored); + + // The other spelling: a row under the manifest's spelling (another + // region's) is unlisted; sync writes this server's spelling, so the + // row is not its own, and the prefix is skipped with no insert. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + fake.completes_own_insert + .store(true, std::sync::atomic::Ordering::Release); + let canonical = sync_arn(2); + let spelled = canonical.replace(REGION, "eu-west-1"); + let mut held = row(&spelled, BackupStatus::Available); + held.owner_generation = UNOWNED_GENERATION; + fake.rows + .lock() + .unwrap() + .insert(spelled.clone(), held.clone()); + fake.unlisted.lock().unwrap().insert(spelled.clone()); + put_backup(&mem, &spelled); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty()); + assert!( + matches!(&skipped_by_prefix(&report)[&prefix_of(&spelled)], SkipReason::RowExists(what) + if what.contains("the catalog's read at the start did not return")), + "{:?}", + report.skipped + ); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + assert_eq!(fake.rows.lock().unwrap()[&spelled], held); + assert!(!fake.rows.lock().unwrap().contains_key(&canonical)); +} + +/// An insert that fails is a catalog skip for its prefix, and sync goes on +/// to the next; the report counts it as transient, so the command exits 1 +/// and a startup sync stays pending. +#[tokio::test] +async fn an_insert_that_fails_is_reported_and_sync_goes_on() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + put_backup(&mem, &sync_arn(1)); + put_backup(&mem, &sync_arn(2)); + *fake.insert_error.lock().unwrap() = Some("pool exhausted".to_owned()); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty()); + let skipped = skipped_by_prefix(&report); + for n in [1, 2] { + assert!( + matches!(&skipped[&prefix_of(&sync_arn(n))], SkipReason::Catalog(e) if e.contains("pool exhausted")), + "{:?}", + skipped[&prefix_of(&sync_arn(n))] + ); + } + assert_eq!(report.transient_skips(), 2); + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 2); +} + +/// A stop asked before a prefix ends the run there: the prefixes before it +/// keep their rows, nothing after it is read, and no missing-manifest report +/// is made (the listing did not reach every prefix). The stop comes at the +/// second of three prefixes, so the run ends inside the listing's loop, not +/// at the last group. +#[tokio::test] +async fn a_stop_ends_sync_before_the_next_prefix_with_no_missing_manifest_report() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + for n in 1..=3 { + put_backup(&mem, &sync_arn(n)); + } + let gone = sync_arn(9); + fake.rows + .lock() + .unwrap() + .insert(gone.clone(), row(&gone, BackupStatus::Available)); + let asked = std::sync::atomic::AtomicUsize::new(0); + let stop = || { + (asked.fetch_add(1, std::sync::atomic::Ordering::AcqRel) >= 1) + .then(|| "SIGTERM received".to_owned()) + }; + let target = super::sync::SyncTarget { + stop: Some(&stop), + ..sync_target(&mem, &identity) + }; + let report = super::sync::sync(&*fake, target).await.unwrap(); + assert_eq!(report.stopped.as_deref(), Some("SIGTERM received")); + assert_eq!( + report + .adopted + .iter() + .map(|a| a.backup_arn.clone()) + .collect::>(), + [sync_arn(1)] + ); + assert!(report.missing_manifests.is_empty()); + for n in 2..=3 { + assert!( + !mem.gets() + .iter() + .any(|k| k.starts_with(&prefix_of(&sync_arn(n)))), + "prefix {n} not read" + ); + } +} + +/// A listing that fails part-way keeps the rows of the prefixes it passed +/// (each verified), reports them, and makes no missing-manifest report. The +/// prefix it was listing when it failed (here the second) is not read: its +/// keys may be cut short. +#[tokio::test] +async fn a_listing_that_fails_part_way_keeps_the_rows_before_it() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + for n in 1..=3 { + put_backup(&mem, &sync_arn(n)); + } + let gone = sync_arn(9); + fake.rows + .lock() + .unwrap() + .insert(gone.clone(), row(&gone, BackupStatus::Available)); + *mem.fail_root_list_at.lock().unwrap() = Some(prefix_of(&sync_arn(3))); + let err = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap_err(); + let super::sync::SyncError::Listing { partial, .. } = &err else { + panic!("{err}"); + }; + assert_eq!(partial.adopted.len(), 1, "{:?}", partial.adopted); + assert_eq!(partial.adopted[0].backup_arn, sync_arn(1)); + assert!(partial.missing_manifests.is_empty()); + assert!( + err.to_string().contains("1 backup(s) listed before it"), + "{err}" + ); + assert!(fake.rows.lock().unwrap().contains_key(&sync_arn(1))); + assert!(!fake.rows.lock().unwrap().contains_key(&sync_arn(2))); + assert!( + !mem.gets() + .iter() + .any(|k| k.starts_with(&prefix_of(&sync_arn(2)))) + ); +} + +/// The startup sync of an in-memory catalog stays pending when a prefix +/// was skipped for a store error, and the next cycle adopts it once its +/// reads answer; the prefixes adopted in the first run are skipped on the +/// row check then. +#[tokio::test] +async fn a_startup_sync_with_a_store_error_on_a_prefix_stays_pending() { + let mem = Arc::new(MemStore::default()); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "previous-start", true), + ); + let ok = sync_arn(1); + let flaky = sync_arn(2); + put_backup(&mem, &ok); + put_backup(&mem, &flaky); + *mem.fail_gets_under.lock().unwrap() = Some(prefix_of(&flaky)); + let fake = Arc::new(Fake::default()); + let runtime = runtime_checking(&mem, true); + let pending = || runtime.configured_store().unwrap().startup_sync_pending(); + cycle_on(&fake, &runtime).await; + assert!(fake.rows.lock().unwrap().contains_key(&ok)); + assert!(!fake.rows.lock().unwrap().contains_key(&flaky)); + assert!(pending(), "a store error leaves the sync pending"); + *mem.fail_gets_under.lock().unwrap() = None; + cycle_on(&fake, &runtime).await; + assert_eq!( + fake.rows.lock().unwrap()[&flaky].status, + BackupStatus::Available + ); + assert!(!pending()); + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 2); +} + +/// A startup sync never runs while the store is unhealthy, though its +/// identity is known: a recheck that found the root owned by another +/// catalog (a second in-memory server took it) leaves the sync pending and +/// nothing inserted. +#[tokio::test] +async fn a_startup_sync_does_not_run_while_a_recheck_finds_the_root_foreign() { + let mem = Arc::new(MemStore::default()); + put_backup(&mem, &sync_arn(1)); + let fake = Arc::new(Fake::default()); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + let configured = runtime.configured_store().unwrap(); + configured.record_check(Err(StoreProblem::ForeignCatalog { + file_catalog_id: "second-server".to_owned(), + catalog_id: "catalog-1".to_owned(), + })); + assert!(configured.identity().is_some(), "the identity is known"); + assert!(super::sync::startup_sync(&*fake, &runtime).await.is_none()); + assert!(configured.startup_sync_pending()); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); +} + +/// The dead-owner probe requires the manifest's ARN to name the row's +/// backup: a complete prefix whose manifest spells the row's backup in +/// another region makes the dead CREATING row AVAILABLE (sync adopts such a +/// manifest under this server's region, and the probe reads it the same +/// way), and one whose manifest names the backup id in another partition +/// leaves it FAILED. +#[tokio::test] +async fn the_dead_owner_probe_accepts_another_region_and_refuses_another_partition() { + for (cell, spelled, expected) in [ + ( + "another region", + sync_arn(1).replace(REGION, "eu-west-1"), + BackupStatus::Available, + ), + ( + "another partition", + sync_arn(1).replace("arn:aws:", "arn:aws-cn:"), + BackupStatus::Failed, + ), + ] { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let dead = sync_arn(1); + put_backup(&mem, &spelled); + assert_eq!(prefix_of(&spelled), prefix_of(&dead), "{cell}: one prefix"); + let mut creating = row(&dead, BackupStatus::Creating); + creating.heartbeat_at = 1234.5; + fake.rows.lock().unwrap().insert(dead.clone(), creating); + fake.dead.lock().unwrap().insert(dead.clone()); + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.creating_resolved, + [(dead.clone(), expected)], + "{cell}" + ); + assert_eq!(fake.rows.lock().unwrap()[&dead].status, expected, "{cell}"); + } +} + +// --- the deletion fence ------------------------------------------------------ + +/// The identity file of `store()` after catalog `adopter` adopted the root. +fn adopted_file() -> Vec { + identity_file("store-1", "adopter") +} + +/// The store `runtime` is configured with is unhealthy because its root's +/// identity file names `adopter`, as the fence recorded it. +fn assert_fenced(runtime: &BackupRuntime) { + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Unhealthy(StoreProblem::ForeignCatalog { + file_catalog_id: "adopter".to_owned(), + catalog_id: "catalog-1".to_owned(), + }) + ); +} + +/// DeleteBackup on a store whose last check passed, after another catalog +/// adopted the root: the fence re-reads the identity file before the +/// DELETING write and refuses. No object is deleted, the request is refused +/// as an unhealthy store's, the row stays AVAILABLE (so DescribeBackup does +/// not answer a refused request with DELETED), and the store is unhealthy, +/// so the next request is refused at the gate with no read. +#[tokio::test] +async fn delete_backup_on_a_root_another_catalog_adopted_deletes_nothing() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + mem.insert(STORE_IDENTITY_KEY, &adopted_file()); + let err = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap_err(); + assert!( + validation_message(err).contains("unhealthy (owned by another catalog)"), + "the refusal names the store's problem" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(objects_under(&mem, &arn(ACCOUNT)).len(), 2); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Available + ); + assert!( + fake.delete_starts_seen.lock().unwrap().is_empty(), + "no DELETING write" + ); + assert_fenced(&runtime); + let gets = mem.gets().len(); + assert!( + delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .is_err() + ); + assert_eq!(mem.gets().len(), gets, "refused at the gate"); + // The fence of a store already unhealthy refuses with no read. + let fence = runtime.configured_store().unwrap().fence(); + assert!(matches!( + fence.check("catalog-1").await, + Err(StoreProblem::ForeignCatalog { .. }) + )); + assert_eq!(mem.gets().len(), gets, "the fence read nothing"); +} + +/// The fence keeps the catalog id it read first for the process: a +/// catalog destroyed and re-created beside a running process gets a new +/// id, and a root whose file names the new catalog is foreign to this +/// process. DeleteBackup is refused with `ForeignCatalog` recorded and +/// nothing deleted, though the catalog now answers the new id. +#[tokio::test] +async fn the_fence_keeps_the_catalog_id_it_read_first() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + seed(&fake, &mem, &newer_arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + let desc = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap(); + assert_eq!(desc.backup_details.backup_status, "DELETED"); + let deletes = mem.deletes.lock().unwrap().len(); + assert!(deletes > 0, "the first deletion ran"); + + *fake.catalog_id.lock().unwrap() = Some("catalog-2".to_owned()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-2")); + let err = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))) + .await + .unwrap_err(); + assert!( + validation_message(err).contains("unhealthy (owned by another catalog)"), + "the refusal names the store's problem" + ); + assert_eq!( + mem.deletes.lock().unwrap().len(), + deletes, + "nothing deleted" + ); + assert_eq!(objects_under(&mem, &newer_arn(ACCOUNT)).len(), 2); + assert_eq!( + fake.rows.lock().unwrap()[&newer_arn(ACCOUNT)].status, + BackupStatus::Available + ); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Unhealthy(StoreProblem::ForeignCatalog { + file_catalog_id: "catalog-2".to_owned(), + catalog_id: "catalog-1".to_owned(), + }) + ); +} + +/// The log line of a deletion the fence refused after the DELETING write +/// says what the fence found: an adoption only for a file that names +/// another catalog, a read that failed or found no file as such (the +/// reconciler finishes it once it can), and any other problem as the +/// store's. +#[test] +fn a_fenced_deletions_line_is_worded_from_the_problem() { + let adopted = fenced_deletion_note(&StoreProblem::ForeignCatalog { + file_catalog_id: "adopter".to_owned(), + catalog_id: "catalog-1".to_owned(), + }); + assert!( + adopted.starts_with("another catalog adopted the root"), + "{adopted}" + ); + for problem in [ + StoreProblem::Transport("the fence did not finish within 5 s".to_owned()), + StoreProblem::FileMissingWithRows(vec!["store-1".to_owned()]), + StoreProblem::Denied("permission denied".to_owned()), + StoreProblem::FileUnreadable("not a store identity file".to_owned()), + ] { + let note = fenced_deletion_note(&problem); + assert!( + note.starts_with("the store's identity file could not be read") + && note.contains("the reconciler finishes the deletion once it can") + && !note.contains("adopted"), + "{note}" + ); + } + let note = fenced_deletion_note(&StoreProblem::IdMismatch { + file_id: "store-2".to_owned(), + row_ids: vec!["store-1".to_owned()], + }); + assert!(note.starts_with("the store is unhealthy"), "{note}"); + assert!(!note.contains("adopted"), "{note}"); +} + +/// The adoption lands between DeleteBackup's two fences: the first passes, +/// the row moves to DELETING, and the fence before the first `delete_prefix` +/// refuses. The request answers DELETED, as for any deletion that fails +/// after the DELETING write; nothing is deleted, the row stays DELETING, the +/// store is unhealthy; the reconciler's DELETING duty acts on no row of it. +#[tokio::test] +async fn an_adoption_between_delete_backups_two_fences_leaves_the_row_deleting() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + *mem.identity_after_read.lock().unwrap() = Some(adopted_file()); + let desc = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap(); + assert_eq!(desc.backup_details.backup_status, "DELETED"); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(objects_under(&mem, &arn(ACCOUNT)).len(), 2); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Deleting + ); + assert_fenced(&runtime); + let report = cycle_on(&fake, &runtime).await; + assert!(report.deletions_finished.is_empty()); + assert!(mem.deletes.lock().unwrap().is_empty()); +} + +/// Each reconciler duty that deletes a prefix or removes a tombstone, on a +/// root another catalog adopts right after the cycle's check read the +/// identity file: the fence re-reads it before the deletion or removal and +/// refuses. No `delete_prefix` is called, no row is moved or removed, and +/// the store is unhealthy at once. +#[tokio::test] +async fn each_reconciler_deletion_on_a_root_adopted_after_the_check_is_refused() { + let now = seconds_since_epoch(); + let settings = JobSettings { + failed_row_retention: Duration::from_secs(600), + ..job_settings() + }; + // (cell, status, heartbeat_at, completed_at, objects under the prefix) + for (cell, status, heartbeat_at, completed_at, objects) in [ + ("DELETING row", BackupStatus::Deleting, now, None, true), + ( + "stale FAILED row", + BackupStatus::Failed, + now - 1000.0, + Some(now), + true, + ), + ("DELETED row", BackupStatus::Deleted, now, Some(now), true), + ( + "tombstone", + BackupStatus::Failed, + now - 1000.0, + Some(now - 1000.0), + false, + ), + ] { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let backup = arn(ACCOUNT); + let mut r = row(&backup, status); + r.heartbeat_at = heartbeat_at; + r.completed_at = completed_at; + fake.rows.lock().unwrap().insert(backup.clone(), r); + if objects { + let prefix = BackupArn::parse(&backup).unwrap().store_prefix(); + mem.insert(&format!("{prefix}data/000001.json.gz"), b"x"); + } + let runtime = runtime_on(&mem, store()).with_job_settings(settings); + *mem.identity_after_read.lock().unwrap() = Some(adopted_file()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true), "{cell}: the check passed"); + assert!(report.deletions_finished.is_empty(), "{cell}"); + assert!(report.failed_prefixes_deleted.is_empty(), "{cell}"); + assert!(report.deleted_prefixes_swept.is_empty(), "{cell}"); + assert!(report.tombstones_removed.is_empty(), "{cell}"); + assert!(mem.deletes.lock().unwrap().is_empty(), "{cell}"); + assert_eq!( + objects_under(&mem, &backup).len(), + usize::from(objects), + "{cell}" + ); + assert_eq!(fake.rows.lock().unwrap()[&backup].status, status, "{cell}"); + assert!(fake.set_deleted.lock().unwrap().is_empty(), "{cell}"); + assert!(fake.removed_rows.lock().unwrap().is_empty(), "{cell}"); + assert_eq!( + mem.gets() + .iter() + .filter(|k| *k == STORE_IDENTITY_KEY) + .count(), + 2, + "{cell}: the check's read and the fence's" + ); + assert_fenced(&runtime); + } +} + +/// CreateBackup on a store whose last check passed, after another catalog +/// adopted the root: the fence before the row and the first put refuses. +/// No row is claimed, nothing is put, and the store is unhealthy. +#[tokio::test] +async fn create_backup_on_a_root_another_catalog_adopted_writes_nothing() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + let runtime = job_runtime(&mem); + mem.insert(STORE_IDENTITY_KEY, &adopted_file()); + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + assert!( + validation_message(err).contains("unhealthy (owned by another catalog)"), + "the refusal names the store's problem" + ); + wait_for_jobs(&runtime).await; + assert!(fake.claims.lock().unwrap().is_empty()); + assert!(fake.rows.lock().unwrap().is_empty()); + assert!(mem.puts.lock().unwrap().is_empty()); + assert_eq!(runtime.free_slots(), 4, "the slot is released"); + assert_fenced(&runtime); +} + +/// The backup job's failure path on a root another catalog adopted while +/// the job ran, after its first put: FAILED is written, the fence before +/// the deletion refuses, and no `delete_prefix` is called; the store is +/// unhealthy. +#[tokio::test] +async fn the_job_failure_path_on_a_root_adopted_while_the_job_ran_deletes_nothing() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + songs(3), + true, + Some(Arc::clone(&gate)), + ); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + // The adoption lands with the job's first stored object, and the puts + // after it fail. + let hook = Arc::clone(&mem); + *mem.on_put.lock().unwrap() = Some(Box::new(move |_key| { + hook.insert(STORE_IDENTITY_KEY, &adopted_file()); + hook.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + })); + gate.add_permits(3); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.rows.lock().unwrap()[&details.backup_arn].status, + BackupStatus::Failed + ); + assert!( + !mem.puts.lock().unwrap().is_empty(), + "the first put passed the job's fence" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_fenced(&runtime); +} + +/// Purge of a row on a root another catalog adopted after the command's +/// check: the fence before the first `delete_prefix` refuses, nothing is +/// deleted, the reason names the problem, and the store is unhealthy. +#[tokio::test] +async fn purge_on_a_root_another_catalog_adopted_deletes_nothing() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &adopted_file()); + let fake = Fake::default(); + let backup = arn_n(1); + seed(&fake, &mem, &backup, BackupStatus::Available); + let configured = ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ); + let read = fake.rows.lock().unwrap()[&backup].clone(); + let err = maintenance::purge_row( + &fake, + &*mem, + &configured.fence(), + &read, + Duration::from_secs(300), + &confirmed_claim(), + ) + .await + .unwrap_err(); + assert!( + err.starts_with("the deletion was refused: ") && err.contains("names catalog adopter"), + "{err}" + ); + assert!(err.ends_with("the row is unchanged"), "{err}"); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(objects_under(&mem, &backup).len(), 2); + assert!(fake.removed_rows.lock().unwrap().is_empty()); + assert!( + fake.delete_starts_seen.lock().unwrap().is_empty(), + "no DELETING write" + ); + assert_eq!(fake.rows.lock().unwrap()[&backup], read); + assert_eq!( + configured.health(), + StoreHealth::Unhealthy(StoreProblem::ForeignCatalog { + file_catalog_id: "adopter".to_owned(), + catalog_id: "catalog-1".to_owned(), + }) + ); +} + +/// The marker follows the owner. A durable catalog's `--adopt-store` of a +/// root an in-memory catalog owned writes `ephemeral_catalog: false`; an +/// in-memory server's startup check then finds the root owned by a durable +/// catalog and writes nothing; and the re-check of an in-memory server +/// whose root lost its file (no rows for it) writes the file back with the +/// marker. +#[tokio::test] +async fn the_marker_is_the_owners_after_an_adoption_and_a_rewrite() { + use super::store_identity::{AdoptRule, Adoption, adopt}; + let mem = MemStore::default(); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "in-memory-start", true), + ); + let durable = Fake::default(); + *durable.catalog_id.lock().unwrap() = Some("durable".to_owned()); + let adopted = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &durable, + "postgres", + false, + AdoptRule::Command, + None, + ) + .await + .unwrap(); + assert!(matches!(adopted, Adoption::Adopted { .. }), "{adopted:?}"); + let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(); + assert_eq!(file.catalog_id, "durable"); + assert!(!file.ephemeral_catalog, "the durable adopter's marker"); + + let puts = mem.puts.lock().unwrap().len(); + let in_memory = Fake::default(); + let err = store_identity::establish( + &mem, + StoreKind::Filesystem, + "/r", + &in_memory, + "sqlite", + true, + ) + .await + .unwrap_err(); + assert_eq!( + err, + StoreProblem::ForeignCatalog { + file_catalog_id: "durable".to_owned(), + catalog_id: "catalog-1".to_owned(), + } + ); + assert_eq!(mem.puts.lock().unwrap().len(), puts, "nothing written"); + assert_eq!( + StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(), + file + ); + + let lost = MemStore::default(); + let known = StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/r".to_owned(), + store_id: "store-7".to_owned(), + }; + store_identity::recheck(&lost, &known, &in_memory, "sqlite", true) + .await + .unwrap(); + let rewritten = + StoreIdentityFile::parse(&lost.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(); + assert_eq!(rewritten.store_id, "store-7"); + assert_eq!(rewritten.catalog_id, "catalog-1"); + assert!(rewritten.ephemeral_catalog, "the in-memory owner's marker"); +} + +// --- the deletion fence when its read fails ----------------------------------- + +/// A healthy store whose identity file reads fail after the first `pass`. +fn failing_identity(mem: &MemStore, pass: usize, failure: IdentityFailure) { + *mem.identity_get_failure.lock().unwrap() = Some((pass, failure)); +} + +/// The fence's read of the identity file fails in transport (the store +/// answers every other call) at each call site: DeleteBackup, CreateBackup, +/// the job's failure path and purge. Each call is refused as a failed store +/// call would be (CreateBackup answers `InternalServerError`, the one +/// fence answer that is not the unhealthy-store refusal), nothing is +/// deleted or put, and the store's health is left as the last check set +/// it. DeleteBackup, as for any store failure, goes on to the DELETING +/// write and answers DELETED, its deletion refused by the second fence and +/// the row left DELETING; the other calls move no row. +#[tokio::test] +async fn a_fence_whose_read_fails_in_transport_refuses_each_call_and_leaves_the_health() { + // DeleteBackup. + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + failing_identity(&mem, 0, IdentityFailure::Transport); + let desc = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap(); + assert_eq!(desc.backup_details.backup_status, "DELETED"); + assert!(mem.deletes.lock().unwrap().is_empty(), "DeleteBackup"); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Deleting + ); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Healthy + ); + + // CreateBackup. + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + let runtime = job_runtime(&mem); + failing_identity(&mem, 0, IdentityFailure::Transport); + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert!(fake.claims.lock().unwrap().is_empty(), "CreateBackup"); + assert!(mem.puts.lock().unwrap().is_empty()); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Healthy + ); + + // The job's failure path: the job's fence before its first put reads + // the file, a put fails, FAILED is written, the failure path's fence + // read fails, nothing is deleted. + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + songs(3), + true, + Some(Arc::clone(&gate)), + ); + let runtime = job_runtime(&mem); + // The request's fence and the job's pass; the failure path's fails. + failing_identity(&mem, 2, IdentityFailure::Transport); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + gate.add_permits(3); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.rows.lock().unwrap()[&details.backup_arn].status, + BackupStatus::Failed + ); + assert!(mem.deletes.lock().unwrap().is_empty(), "fail_row"); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Healthy + ); + + // Purge, of an AVAILABLE and of a dead CREATING row. + for status in [BackupStatus::Available, BackupStatus::Creating] { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let backup = arn_n(1); + seed(&fake, &mem, &backup, status); + let configured = ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ); + failing_identity(&mem, 0, IdentityFailure::Transport); + let read = fake.rows.lock().unwrap()[&backup].clone(); + let err = maintenance::purge_row( + &fake, + &*mem, + &configured.fence(), + &read, + Duration::from_secs(300), + &confirmed_claim(), + ) + .await + .unwrap_err(); + assert!(err.starts_with("the deletion was refused: "), "{err}"); + assert!(mem.deletes.lock().unwrap().is_empty(), "purge {status}"); + assert_eq!(fake.rows.lock().unwrap()[&backup], read, "purge {status}"); + assert!(fake.delete_starts_seen.lock().unwrap().is_empty()); + assert_eq!(configured.health(), StoreHealth::Healthy); + } +} + +/// The fence's read fails at each reconciler duty that deletes a prefix or +/// removes a tombstone (the cycle's check read the file first): nothing is +/// deleted, no row moves, and the store stays healthy. +#[tokio::test] +async fn each_reconciler_deletion_whose_fence_read_fails_is_refused_with_the_health_left() { + let now = seconds_since_epoch(); + let settings = JobSettings { + failed_row_retention: Duration::from_secs(600), + ..job_settings() + }; + for (cell, status, heartbeat_at, completed_at, objects) in [ + ("DELETING row", BackupStatus::Deleting, now, None, true), + ( + "stale FAILED row", + BackupStatus::Failed, + now - 1000.0, + Some(now), + true, + ), + ("DELETED row", BackupStatus::Deleted, now, Some(now), true), + ( + "tombstone", + BackupStatus::Failed, + now - 1000.0, + Some(now - 1000.0), + false, + ), + ] { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let backup = arn(ACCOUNT); + let mut r = row(&backup, status); + r.heartbeat_at = heartbeat_at; + r.completed_at = completed_at; + fake.rows.lock().unwrap().insert(backup.clone(), r); + if objects { + let prefix = BackupArn::parse(&backup).unwrap().store_prefix(); + mem.insert(&format!("{prefix}data/000001.json.gz"), b"x"); + } + let runtime = runtime_on(&mem, store()).with_job_settings(settings); + failing_identity(&mem, 1, IdentityFailure::Transport); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true), "{cell}: the check passed"); + assert!(report.deletions_finished.is_empty(), "{cell}"); + assert!(report.failed_prefixes_deleted.is_empty(), "{cell}"); + assert!(report.deleted_prefixes_swept.is_empty(), "{cell}"); + assert!(report.tombstones_removed.is_empty(), "{cell}"); + assert!(mem.deletes.lock().unwrap().is_empty(), "{cell}"); + assert_eq!(fake.rows.lock().unwrap()[&backup].status, status, "{cell}"); + assert!(fake.set_deleted.lock().unwrap().is_empty(), "{cell}"); + assert!(fake.removed_rows.lock().unwrap().is_empty(), "{cell}"); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Healthy, + "{cell}" + ); + } +} + +/// A root with no identity file under the catalog's rows (removed since +/// the last check) is recorded by the fence as the next check would record +/// it, `FileMissingWithRows` with the store's id, and DeleteBackup is +/// refused as an unhealthy store's: the row stays AVAILABLE, nothing is +/// deleted, and the next request is refused at the gate. +#[tokio::test] +async fn a_fence_on_a_root_with_no_identity_file_records_it_and_refuses() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + mem.objects.lock().unwrap().remove(STORE_IDENTITY_KEY); + let err = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap_err(); + assert!( + validation_message(err).contains("unhealthy (identity file missing)"), + "the refusal names the store's problem" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(objects_under(&mem, &arn(ACCOUNT)).len(), 2); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Available + ); + assert!(fake.delete_starts_seen.lock().unwrap().is_empty()); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Unhealthy(StoreProblem::FileMissingWithRows(vec![ + "store-1".to_owned() + ])) + ); +} + +/// CreateBackup on a store `/health` reports healthy whose identity file +/// was removed, that the store refuses to serve (an S3 `AccessDenied`), or +/// that a filesystem root may not open (mode 000, which the filesystem +/// store reports as an I/O error), since the last check: the fence before +/// the row records the problem the next check would record and CreateBackup +/// answers the unhealthy-store `ValidationException` naming it, with no row +/// claimed and nothing put; the next request is refused at the gate with no +/// read. +#[tokio::test] +async fn create_backup_on_a_root_whose_identity_file_is_missing_or_denied_is_refused_as_unhealthy() +{ + for (cell, reason, expected) in [ + ( + "missing", + "unhealthy (identity file missing)", + Some(StoreProblem::FileMissingWithRows(vec![ + "store-1".to_owned(), + ])), + ), + ( + "a denied read (S3 AccessDenied)", + "unhealthy (permission denied)", + Some(StoreProblem::Denied(format!( + "GetObject {STORE_IDENTITY_KEY}: permission denied" + ))), + ), + // The filesystem store's mode-000 file: a `Validate` problem whose + // text is the store error's. + ( + "a filesystem read refused (mode 000)", + "unhealthy (failed validation)", + None, + ), + ] { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + let runtime = job_runtime(&mem); + match cell { + "missing" => { + mem.objects.lock().unwrap().remove(STORE_IDENTITY_KEY); + } + "a denied read (S3 AccessDenied)" => { + failing_identity(&mem, 0, IdentityFailure::Denied); + } + _ => failing_identity(&mem, 0, IdentityFailure::Unreadable), + } + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + let message = validation_message(err); + assert!(message.contains(reason), "{cell}: {message}"); + wait_for_jobs(&runtime).await; + assert!(fake.claims.lock().unwrap().is_empty(), "{cell}"); + assert!(mem.puts.lock().unwrap().is_empty(), "{cell}"); + assert_eq!(runtime.free_slots(), 4, "{cell}: the slot is released"); + let health = runtime.configured_store().unwrap().health(); + match expected { + Some(expected) => { + assert_eq!(health, StoreHealth::Unhealthy(expected), "{cell}"); + } + None => assert!( + matches!( + &health, + StoreHealth::Unhealthy(StoreProblem::Validate(text)) + if text.contains(STORE_IDENTITY_KEY) + ), + "{cell}: {health:?}" + ), + } + let gets = mem.gets().len(); + assert!( + create_music_backup(&fake, &runtime).await.is_err(), + "{cell}" + ); + assert_eq!(mem.gets().len(), gets, "{cell}: refused at the gate"); + } +} + +/// The fence's read of the identity file has a slot of its own: a +/// DeleteBackup whose deletion holds the root's call slot for 4 s (a prefix +/// of many objects) does not hold a CreateBackup sent 1 s into it, which +/// answers CREATING at once and whose job completes; the deletion +/// completes too. With the fence in the calls' slot the CreateBackup +/// waited out the deletion. +#[tokio::test(start_paused = true)] +async fn create_backup_during_a_long_deletion_answers_at_once() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + let runtime = job_runtime(&mem); + *mem.delete_delay.lock().unwrap() = Some(Duration::from_secs(4)); + let started = tokio::time::Instant::now(); + let spellings = own(&arn(ACCOUNT)); + let delete = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &spellings); + let create = async { + tokio::time::sleep(Duration::from_secs(1)).await; + let sent = tokio::time::Instant::now(); + let created = create_music_backup(&fake, &runtime).await; + (created, sent.elapsed()) + }; + let (deleted, (created, took)) = futures::future::join(delete, create).await; + assert_eq!(took, Duration::ZERO, "the CreateBackup waited for nothing"); + let created = created.unwrap(); + assert_eq!(created.backup_status, "CREATING"); + assert_eq!(deleted.unwrap().backup_details.backup_status, "DELETED"); + assert!( + started.elapsed() >= Duration::from_secs(4), + "the deletion held the slot" + ); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.rows.lock().unwrap()[&created.backup_arn].status, + BackupStatus::Available + ); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Deleted + ); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Healthy + ); +} + +/// A root adopted between CreateBackup's fence, at the request, and the +/// job's first write: the job runs the fence again right before its first +/// put, which refuses. The row is FAILED with the fence's reason, no object +/// is put or deleted, and the store is unhealthy. +#[tokio::test] +async fn a_root_adopted_after_the_request_gets_no_object_of_the_job() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + let runtime = job_runtime(&mem); + // The request's read sees this catalog's file; the next read, the job's, + // sees the adopter's. + *mem.identity_after_read.lock().unwrap() = Some(adopted_file()); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.unwrap_or_default(); + assert!( + reason.contains("re-read before the first write") && reason.contains("adopter"), + "{reason}" + ); + assert!(mem.puts.lock().unwrap().is_empty(), "no object was put"); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!( + objects_under(&mem, &details.backup_arn), + Vec::::new() + ); + assert_fenced(&runtime); +} + +/// A fence whose read of the identity file never answers is refused at the +/// store's check bound: DeleteBackup answers DELETED once the bound passes +/// (the second fence is refused at once, the first read still parked), with +/// nothing deleted, the row DELETING and the health left. +#[tokio::test(start_paused = true)] +async fn a_fence_whose_read_stalls_is_refused_at_its_bound() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + let bound = runtime.configured_store().unwrap().check_timeout(); + failing_identity(&mem, 0, IdentityFailure::Stall); + let started = tokio::time::Instant::now(); + // A guard far past the bound, so a fence with no bound fails here + // instead of hanging. + let desc = tokio::time::timeout( + bound * 10, + delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))), + ) + .await + .expect("the fence's read is bounded") + .unwrap(); + assert_eq!(desc.backup_details.backup_status, "DELETED"); + assert!(started.elapsed() >= bound, "{:?}", started.elapsed()); + assert!(started.elapsed() < bound + Duration::from_secs(1)); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Deleting + ); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Healthy + ); +} + +/// A fence refusal in the cycle's DELETING duty makes the store unhealthy, +/// and the cycle runs no later store duty on the root: the dead-owner duty +/// reads nothing under the dead CREATING row's prefix and leaves the row, +/// and the FAILED-prefix, DELETED-prefix and tombstone duties list nothing +/// under a stale FAILED row's and a DELETED row's prefixes, both past their +/// windows. +#[tokio::test] +async fn a_fence_refusal_in_the_deleting_duty_stops_the_dead_owner_duty() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + let deleting = arn_n(1); + assert_ne!(dead, deleting); + seed(&fake, &mem, &deleting, BackupStatus::Deleting); + let stale_failed = arn_n(2); + let deleted = arn_n(3); + let now = seconds_since_epoch(); + for (a, status) in [ + (&stale_failed, BackupStatus::Failed), + (&deleted, BackupStatus::Deleted), + ] { + let mut r = row(a, status); + r.heartbeat_at = now - 1_000_000.0; + r.completed_at = Some(now - 1_000_000.0); + fake.rows.lock().unwrap().insert(a.clone(), r); + mem.insert(&format!("{}data/000001.json.gz", prefix_of(a)), b"x"); + } + let runtime = runtime_on(&mem, store()); + *mem.identity_after_read.lock().unwrap() = Some(adopted_file()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true), "the check passed"); + assert!(report.deletions_finished.is_empty()); + assert!(report.creating_resolved.is_empty()); + assert_fenced(&runtime); + let dead_prefix = prefix_of(&dead); + assert!( + !mem.gets().iter().any(|k| k.starts_with(&dead_prefix)), + "{:?}", + mem.gets() + ); + assert_eq!( + fake.rows.lock().unwrap()[&dead].status, + BackupStatus::Creating + ); + assert!(fake.takeovers.lock().unwrap().is_empty()); + assert!(mem.deletes.lock().unwrap().is_empty()); + for a in [&stale_failed, &deleted] { + let prefix = prefix_of(a); + assert!( + !mem.lists.lock().unwrap().contains(&prefix), + "{a}: listed after the refusal: {:?}", + mem.lists.lock().unwrap() + ); + assert!(fake.rows.lock().unwrap().contains_key(a.as_str())); + } +} + +/// A read of the row under the ARN's spellings that fails skips the prefix +/// as a catalog error, which a later run may adopt: no insert is sent, +/// since the read is what keeps a second row of the id out on Cassandra. +/// The next sync, with the read answering, adopts the prefix. +#[tokio::test] +async fn sync_skips_a_prefix_whose_row_read_fails_and_inserts_nothing() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let identity = store(); + let backup = sync_arn(1); + put_backup(&mem, &backup); + *fake.read_row_error.lock().unwrap() = Some("read timed out".to_owned()); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty(), "{:?}", report.adopted); + assert!( + matches!(&skipped_by_prefix(&report)[&prefix_of(&backup)], SkipReason::Catalog(e) if e.contains("read timed out")), + "{:?}", + report.skipped + ); + assert_eq!(report.transient_skips(), 1); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + assert!(fake.rows.lock().unwrap().is_empty()); + + *fake.read_row_error.lock().unwrap() = None; + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!(report.adopted.len(), 1); + assert_eq!( + fake.rows.lock().unwrap()[&backup].status, + BackupStatus::Available + ); +} + +/// On a catalog whose rows are unique by ARN alone (Cassandra's rule), two +/// prefixes of one account and backup id under two tables, each with a +/// complete manifest naming its own path, are not both adopted in one run: +/// the first gets its row, and the second is skipped on the row check as +/// the id's row under the other table. Without the row check the insert +/// would accept the second, since no backup-id uniqueness refuses it. +#[tokio::test] +async fn one_run_adopts_one_prefix_of_a_backup_id_where_rows_are_unique_by_arn() { + use super::sync::SkipReason; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + fake.unique_by_arn + .store(true, std::sync::atomic::Ordering::Release); + let identity = store(); + let music = sync_arn(1); + let other = music.replace("table/Music", "table/Other"); + put_backup(&mem, &music); + put_backup(&mem, &other); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + // `Music` sorts before `Other`, so the listing reaches it first. + assert_eq!( + report + .adopted + .iter() + .map(|a| a.backup_arn.clone()) + .collect::>(), + std::slice::from_ref(&music) + ); + assert_eq!( + skipped_by_prefix(&report)[&prefix_of(&other)], + SkipReason::HasRow { + status: BackupStatus::Available, + store_id: "store-1".to_owned(), + other_table: Some("Music".to_owned()), + } + ); + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 1); + assert!(!fake.rows.lock().unwrap().contains_key(&other)); +} + +/// A cycle whose startup sync meets a root listing that never answers +/// returns as soon as the shutdown token is cancelled, well before the +/// sync's own bound, with the sync left pending and no later duty run. +#[tokio::test(start_paused = true)] +async fn a_cycles_startup_sync_ends_at_the_shutdown_and_stays_pending() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let deleting = arn(ACCOUNT); + seed(&fake, &mem, &deleting, BackupStatus::Deleting); + put_backup(&mem, &sync_arn(1)); + let runtime = runtime().with_store( + ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + ) + .with_ephemeral_catalog(true), + ); + mem.stall_root_list + .store(true, std::sync::atomic::Ordering::Release); + let shutdown = extenddb_storage::CancellationToken::new(); + let cancel = shutdown.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_secs(5)).await; + cancel.cancel(); + }); + let started = tokio::time::Instant::now(); + // A guard past the bound, so a cycle that ignores the token fails here + // instead of hanging. + let report = tokio::time::timeout( + super::sync::STARTUP_SYNC_TIMEOUT * 5, + reconciler::cycle( + &fake, + &runtime, + Duration::from_secs(300), + "test", + &mut reconciler::ReconcilerState::with_shutdown(shutdown.clone()), + ), + ) + .await + .expect("the cycle ends") + .unwrap(); + let took = started.elapsed(); + assert!(shutdown.is_cancelled()); + assert!(took < super::sync::STARTUP_SYNC_TIMEOUT, "{took:?}"); + assert_eq!(report.store_healthy, Some(true)); + assert!(report.deletions_finished.is_empty(), "{report:?}"); + assert_eq!( + fake.rows.lock().unwrap()[&deleting].status, + BackupStatus::Deleting + ); + assert!( + runtime.configured_store().unwrap().startup_sync_pending(), + "left pending" + ); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); +} + +/// The adoption under the in-memory rule refuses a root whose file was +/// written by a durable catalog, with nothing written; the check of an +/// in-memory catalog does not hand such a root to the adoption at all (one +/// read of the file) and leaves it owned by another catalog. +#[tokio::test] +async fn the_in_memory_adoption_refuses_a_durable_owners_root() { + use super::store_identity::{AdoptRefusal, AdoptRule, adopt}; + let mem = MemStore::default(); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file_of("store-7", "durable", false), + ); + let fake = Fake::default(); + let err = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &fake, + "sqlite", + true, + AdoptRule::EphemeralOwner, + None, + ) + .await + .unwrap_err(); + assert_eq!( + err, + AdoptRefusal::DurableOwner { + file_catalog_id: "durable".to_owned(), + } + ); + assert!(mem.puts.lock().unwrap().is_empty()); + + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "sqlite", true) + .await + .unwrap_err(); + assert_eq!( + err, + StoreProblem::ForeignCatalog { + file_catalog_id: "durable".to_owned(), + catalog_id: "catalog-1".to_owned(), + } + ); + let file_reads = |mem: &MemStore| { + mem.gets() + .iter() + .filter(|k| *k == STORE_IDENTITY_KEY) + .count() + }; + assert_eq!(file_reads(&mem), 2, "the adoption's read, then the check's"); + assert!(mem.puts.lock().unwrap().is_empty()); +} + +/// The fence refuses a file that names this catalog under another store id +/// than the one the store's check settled (the root replaced by another +/// root of the same catalog), and records the mismatch on the store; a file +/// with the settled id passes. +#[tokio::test] +async fn a_fence_refuses_a_file_of_this_catalog_under_another_store_id() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed(&fake, &mem, &arn(ACCOUNT), BackupStatus::Available); + let runtime = runtime_on(&mem, store()); + let fence = runtime.configured_store().unwrap().fence(); + fence.check("catalog-1").await.unwrap(); + + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1")); + let mismatch = StoreProblem::IdMismatch { + file_id: "store-2".to_owned(), + row_ids: vec!["store-1".to_owned()], + }; + assert_eq!(fence.check("catalog-1").await, Err(mismatch.clone())); + assert_eq!( + runtime.configured_store().unwrap().health(), + StoreHealth::Unhealthy(mismatch) + ); + let err = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + .await + .unwrap_err(); + assert!( + matches!(&err, DynamoDbError::ValidationException(m) if m.contains("identity mismatch")), + "{err:?}" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!( + fake.rows.lock().unwrap()[&arn(ACCOUNT)].status, + BackupStatus::Available + ); +} + +// --- the claim re-checked before each step that cannot be undone ------------ + +/// A maintenance claim whose first `confirms` beats confirm and every later +/// one answers `Lost` (another holder took the claim over), counting the +/// beats. +struct ScriptedClaim { + confirms: usize, + beats: Arc, +} + +impl extenddb_storage::MaintenanceClaim for ScriptedClaim { + fn generation(&self) -> u64 { + 13 + } + fn heartbeat( + &self, + ) -> BoxedFuture<'_, Result> { + let n = self.beats.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Box::pin(async move { + Ok(if n < self.confirms { + extenddb_storage::backup_catalog::Heartbeat::Confirmed + } else { + extenddb_storage::backup_catalog::Heartbeat::Lost + }) + }) + } + fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> { + Box::pin(async { Ok(()) }) + } +} + +/// A held claim over a [`ScriptedClaim`] with a period long enough that the +/// timer sends no beat during the test: every beat counted is a re-check's. +fn scripted_claim( + confirms: usize, +) -> (maintenance::HeldClaim, Arc) { + let beats = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let held = maintenance::HeldClaim::start( + Box::new(ScriptedClaim { + confirms, + beats: Arc::clone(&beats), + }), + Duration::from_secs(3600), + ); + (held, beats) +} + +fn beats_of(beats: &std::sync::atomic::AtomicUsize) -> usize { + beats.load(std::sync::atomic::Ordering::SeqCst) +} + +/// Purge re-checks the claim with a beat right before it removes a dead +/// CREATING row and again right before it deletes that row's prefix, which +/// no row names once the row is gone: a claim another holder took over +/// (a command stopped past the stale window on MongoDB or Cassandra, whose +/// claim a sync then took) leaves the row and its objects when the first +/// beat finds it lost, and the objects when the second does, for a later +/// sync to adopt. Two beats with the claim kept delete both. A row of +/// another status keeps its row through the deletion and needs no beat. +#[tokio::test] +async fn purge_rechecks_the_claim_before_a_dead_creating_rows_removal_and_its_prefix() { + let window = Duration::from_secs(300); + // Lost at the first beat: the row and its objects stay. + let fake = Fake::default(); + let mem = MemStore::default(); + let dead = arn_n(1); + seed(&fake, &mem, &dead, BackupStatus::Creating); + let keys = mem.keys(); + let read = fake.rows.lock().unwrap()[&dead].clone(); + let (claim, beats) = scripted_claim(0); + let err = maintenance::purge_row(&fake, &mem, &passing_fence(), &read, window, &claim) + .await + .unwrap_err(); + assert!( + err.contains("stopped before the CREATING row's removal") + && err.contains("another holder took it over"), + "{err}" + ); + assert_eq!(beats_of(&beats), 1); + assert_eq!( + fake.rows.lock().unwrap()[&dead].status, + BackupStatus::Creating + ); + assert_eq!(mem.keys(), keys, "nothing deleted"); + assert!(mem.deletes.lock().unwrap().is_empty()); + + // Lost at the second beat: the row is removed, its objects stay. + let fake = Fake::default(); + let mem = MemStore::default(); + seed(&fake, &mem, &dead, BackupStatus::Creating); + let keys = mem.keys(); + let read = fake.rows.lock().unwrap()[&dead].clone(); + let (claim, beats) = scripted_claim(1); + let err = maintenance::purge_row(&fake, &mem, &passing_fence(), &read, window, &claim) + .await + .unwrap_err(); + assert!( + err.contains("stopped before the prefix's deletion") + && err.contains("its objects are left"), + "{err}" + ); + assert_eq!(beats_of(&beats), 2); + assert!(fake.rows.lock().unwrap().is_empty(), "the row was removed"); + assert_eq!(mem.keys(), keys, "the prefix is left whole"); + assert!(mem.deletes.lock().unwrap().is_empty()); + + // Both beats confirm: the row and the prefix are gone. + let fake = Fake::default(); + let mem = MemStore::default(); + seed(&fake, &mem, &dead, BackupStatus::Creating); + let read = fake.rows.lock().unwrap()[&dead].clone(); + let (claim, beats) = scripted_claim(2); + maintenance::purge_row(&fake, &mem, &passing_fence(), &read, window, &claim) + .await + .unwrap(); + assert_eq!(beats_of(&beats), 2); + assert!(fake.rows.lock().unwrap().is_empty()); + assert!(mem.keys().is_empty()); + + // An AVAILABLE row: no beat. + let fake = Fake::default(); + let mem = MemStore::default(); + seed(&fake, &mem, &dead, BackupStatus::Available); + let read = fake.rows.lock().unwrap()[&dead].clone(); + let (claim, beats) = scripted_claim(0); + maintenance::purge_row(&fake, &mem, &passing_fence(), &read, window, &claim) + .await + .unwrap(); + assert_eq!(beats_of(&beats), 0); + assert!(mem.keys().is_empty()); +} + +/// Sync re-checks the claim with a beat right before each insert: the +/// prefix whose beat finds the claim lost is not adopted, and the run stops +/// there as at a signal, with the rows inserted before it kept. With no +/// claim (the startup sync of an in-memory catalog) no beat is sent. +#[tokio::test] +async fn sync_rechecks_the_claim_before_each_insert_and_stops_once_it_is_lost() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let first = sync_arn(1); + let second = sync_arn(2); + put_backup(&mem, &first); + put_backup(&mem, &second); + let identity = store(); + let (claim, beats) = scripted_claim(1); + let target = super::sync::SyncTarget { + claim: Some(&claim), + ..sync_target(&mem, &identity) + }; + let report = super::sync::sync(&*fake, target).await.unwrap(); + assert_eq!(beats_of(&beats), 2, "one beat per insert"); + assert_eq!( + report + .adopted + .iter() + .map(|a| a.backup_arn.clone()) + .collect::>(), + std::slice::from_ref(&first) + ); + let stopped = report.stopped.clone().expect("the run stopped"); + assert!( + stopped.contains("another holder took it over") + && stopped.contains(&format!("before the insert for {}", prefix_of(&second))), + "{stopped}" + ); + assert!(report.skipped.is_empty(), "{:?}", report.skipped); + assert!(report.missing_manifests.is_empty()); + assert_eq!(fake.insert_attempts.lock().unwrap().len(), 1); + assert!(!fake.rows.lock().unwrap().contains_key(&second)); + + let fake = Arc::new(Fake::default()); + let report = super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert_eq!(report.adopted.len(), 2); + assert_eq!(beats_of(&beats), 2, "no claim, no beat"); +} + +/// `--prune-missing` re-checks the claim with a beat right before each +/// removal: a claim found lost there ends the loop with the reason, that +/// row and the rest left AVAILABLE and nothing forgotten after it. +#[tokio::test] +async fn prune_missing_rechecks_the_claim_before_each_removal() { + use super::sync::PruneOutcome; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let arns = [sync_arn(1), sync_arn(2), sync_arn(3)]; + for a in &arns { + fake.rows + .lock() + .unwrap() + .insert(a.clone(), row(a, BackupStatus::Available)); + } + let keys: Vec = arns + .iter() + .map(|a| row(a, BackupStatus::Available).key()) + .collect(); + let (claim, beats) = scripted_claim(1); + let root = store().store_root; + let (outcomes, stopped) = + super::sync::prune_missing(&*fake, &*mem, &root, &keys, &claim, &|| None).await; + assert_eq!(outcomes, [(arns[0].clone(), PruneOutcome::Pruned)]); + let stopped = stopped.expect("stopped at the second row"); + assert!( + stopped.contains("another holder took it over") + && stopped.contains(&format!("before the removal of {}", arns[1])), + "{stopped}" + ); + assert_eq!(beats_of(&beats), 2); + let forgets: Vec = fake + .forgets_seen + .lock() + .unwrap() + .iter() + .map(|k| k.backup_arn.clone()) + .collect(); + assert_eq!(forgets, [arns[0].clone()]); + let after = fake.rows.lock().unwrap(); + assert!(!after.contains_key(&arns[0])); + assert_eq!(after[&arns[1]].status, BackupStatus::Available); + assert_eq!(after[&arns[2]].status, BackupStatus::Available); +} + +/// `--adopt-store` re-checks the claim with a beat right before it rewrites +/// `extenddb-store.json`: a claim found lost leaves the file naming its +/// owner, with nothing put; a confirmed beat rewrites it. +#[tokio::test] +async fn adopt_store_rechecks_the_claim_before_the_rewrite() { + use super::store_identity::{AdoptRefusal, AdoptRule, Adoption, adopt}; + let file = StoreIdentityFile { + format_version: 1, + store_id: "store-7".to_owned(), + catalog_id: "previous".to_owned(), + backend: "postgres".to_owned(), + created_at_ms: 1234, + ephemeral_catalog: false, + }; + let mem = MemStore::default(); + mem.insert(STORE_IDENTITY_KEY, &file.to_bytes()); + let fake = Fake::default(); + let (claim, beats) = scripted_claim(0); + let err = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &fake, + "sqlite", + false, + AdoptRule::Command, + Some(&claim), + ) + .await + .unwrap_err(); + assert!( + matches!(&err, AdoptRefusal::ClaimLost(reason) if reason.contains("another holder")), + "{err:?}" + ); + assert!(err.to_string().contains("nothing was written"), "{err}"); + assert_eq!(beats_of(&beats), 1); + assert!(mem.puts.lock().unwrap().is_empty()); + assert_eq!( + StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap(), + file + ); + + let (claim, beats) = scripted_claim(1); + let adopted = adopt( + &mem, + StoreKind::Filesystem, + "/r", + &fake, + "sqlite", + false, + AdoptRule::Command, + Some(&claim), + ) + .await + .unwrap(); + assert!(matches!(adopted, Adoption::Adopted { .. }), "{adopted:?}"); + assert_eq!(beats_of(&beats), 1); + assert_eq!(mem.puts.lock().unwrap().as_slice(), [STORE_IDENTITY_KEY]); +} diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index edcde4faf..89cb73648 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -11,8 +11,8 @@ mod backup; pub use backup::{ BackupRuntime, ConfiguredStore, HeldClaim, JobSettings, KillPoint, LOCAL_CHECK_TIMEOUT, OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, - StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem, TARGET_NOT_EMPTY, - check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity, + StoreFence, StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem, TARGET_NOT_EMPTY, + check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity, sync, }; mod batch_get_item; mod batch_write_item; diff --git a/crates/server/src/backup_store.rs b/crates/server/src/backup_store.rs index 3f81615ef..c0046a032 100644 --- a/crates/server/src/backup_store.rs +++ b/crates/server/src/backup_store.rs @@ -41,7 +41,11 @@ //! refusal of `open` or `validate` a startup error, and an identity failure //! an unhealthy store (reported on `/health`, its rows' restores and deletes //! refused naming the kind and the category, rechecked each cycle). A -//! command that found no store learns why from [`NotConfigured`]. +//! command that found no store learns why from [`NotConfigured`]. `backup +//! sync` ([`Caller::SyncCommand`]) is the one exception to the leniency: a +//! defaulted root whose identity file names another catalog is configured +//! and unhealthy for it, so that `--adopt-store` can adopt the root a +//! catalog re-created by `destroy` and `init` finds. //! //! A filesystem store's `validate` and identity check run here, before the //! listener, bounded by [`LOCAL_CHECK_TIMEOUT`]: a root on a local disk @@ -65,6 +69,16 @@ //! refusal that is not a transport failure (a missing or denied bucket) then //! makes the store unhealthy rather than stopping startup. //! +//! A SQLite catalog with no database file (`:memory:`, the `sqlite-memory` +//! build's default) is new at every start, `catalog_id` included. Its +//! store's checks write `ephemeral_catalog: true` in an identity file they +//! create and adopt a root whose file another in-memory catalog wrote +//! (`extenddb_engine::store_identity`'s module doc), and its startup sync +//! is pending: the server runs it before it serves when the filesystem +//! store's check passed here, and the reconciler runs it after the first +//! check that passes otherwise (`extenddb_engine::sync::startup_sync`). A +//! root a durable catalog owns stays unhealthy there as anywhere. +//! //! Transport failures (the store unreachable: no region, no credentials, a //! refused connection) and failures of the identity check or the manifest //! probe (a denial the store reports while a check runs, say a directory @@ -113,6 +127,13 @@ pub enum Caller { /// the directory neither created nor locked; the checks run as a /// server's do. Command, + /// `backup sync`: as `Command`, except that a defaulted root whose + /// identity file names another catalog stays configured, unhealthy, + /// while the catalog has no rows for it, where `Command` and `Server` + /// take the warning path (module doc): it is the root a catalog + /// re-created by `destroy` and `init` finds, and `--adopt-store` is the + /// command that adopts it. + SyncCommand, /// A command that reads beside a server (`backup list`, the purge's dry /// run): as `Command`, and no check of the store runs, so a root is left /// as it is found (neither the default directory nor an identity file is @@ -148,22 +169,22 @@ impl std::fmt::Display for NotConfigured { match self { Self::Absent => f.write_str( "no backup store is configured: [backup.store] is absent and this backend has \ - no default root, so there are no store-backed backups to list or purge", + no default root, so there are no store-backed backups to list, purge or sync", ), Self::Disabled => f.write_str( "no backup store is configured: [backup.store] is disabled, so there are no \ - store-backed backups to list or purge", + store-backed backups to list, purge or sync", ), Self::DefaultRootMissing(root) => write!( f, "the default root {} does not exist yet: no server has run with this database \ - file, so there are no store-backed backups to list or purge", + file, so there are no store-backed backups to list, purge or sync", root.display() ), Self::DefaultRootUnusable { root, problem } => write!( f, "the default root {root} failed its check ({problem}) and the catalog has no \ - rows for it, so there are no store-backed backups to list or purge; the \ + rows for it, so there are no store-backed backups to list, purge or sync; the \ warning above says how to fix the root" ), } @@ -383,7 +404,16 @@ pub async fn configure_described( (StoreKind::S3, Some(s3)) => (s3.bucket.clone(), s3.prefix.clone()), _ => (opened.root.clone(), String::new()), }; - let configured = |store: ConfiguredStore| store.with_manifest_names(bucket, key_prefix); + // A SQLite database with no file is in memory: its catalog is new at + // every start, so the store's checks mark the identity file they write + // and adopt a root another in-memory catalog owned, and the startup + // sync is pending (module doc). + let ephemeral_catalog = backend == "sqlite" && database_file.is_none(); + let configured = |store: ConfiguredStore| { + store + .with_manifest_names(bucket, key_prefix) + .with_ephemeral_catalog(ephemeral_catalog) + }; // The spool and its lock: configuration errors on every backend. let (spool, spool_lock) = match (resolved.spool_path, caller) { @@ -391,7 +421,9 @@ pub async fn configure_described( let (spool, lock) = take_spool(&path)?; (Some(spool), Some(lock)) } - (Some(_), Caller::Command | Caller::ReadOnlyCommand) | (None, _) => (None, None), + (Some(_), Caller::Command | Caller::SyncCommand | Caller::ReadOnlyCommand) | (None, _) => { + (None, None) + } }; let spool_line = spool .as_ref() @@ -431,7 +463,15 @@ pub async fn configure_described( // here (`run_local_checks`). A `validate` refusal is a configuration // error; a store condition or the identity check's failure degrades the // store; on the defaulted root without rows either is the warning. - let outcome = match run_local_checks(&*opened.store, kind, &opened.root, catalog, backend).await + let outcome = match run_local_checks( + &*opened.store, + kind, + &opened.root, + catalog, + backend, + ephemeral_catalog, + ) + .await { LocalChecks::Done(outcome) => outcome, LocalChecks::Refused(e) => { @@ -483,7 +523,12 @@ pub async fn configure_described( opened.root, identity.store_id ), - Err(problem) if defaulted && !default_root_has_rows(catalog, &opened.root).await => { + Err(problem) + if defaulted + && !(caller == Caller::SyncCommand + && matches!(problem, StoreProblem::ForeignCatalog { .. })) + && !default_root_has_rows(catalog, &opened.root).await => + { let problem = problem.to_string(); warn_default_root_unusable(&opened.root, &problem); return Ok(StoreSetup::without_store( @@ -494,21 +539,55 @@ pub async fn configure_described( }, )); } - Err(problem) => tracing::error!( - "backup store: {kind} {} is unhealthy: {problem}; the server starts, restore and \ - delete of a backup in the store are refused, /health reports the backup_store \ - component unhealthy, and the check runs again each reconciler cycle{}", - opened.root, - rows_note(defaulted) - ), + Err(problem) => { + // The `backup sync` exception configures a defaulted root + // another catalog owns whether or not this catalog has rows for + // it; the clause says which. + let note = if defaulted + && caller == Caller::SyncCommand + && matches!(problem, StoreProblem::ForeignCatalog { .. }) + && !default_root_has_rows(catalog, &opened.root).await + { + SYNC_EXCEPTION_NOTE + } else { + rows_note(defaulted) + }; + if caller == Caller::Server { + tracing::error!( + "backup store: {kind} {} is unhealthy: {problem}; the server starts, restore \ + and delete of a backup in the store are refused, /health reports the \ + backup_store component unhealthy, and the check runs again each reconciler \ + cycle{note}", + opened.root, + ); + } else if caller == Caller::SyncCommand + && matches!(problem, StoreProblem::ForeignCatalog { .. }) + { + // `backup sync`'s own line: a root another catalog owns is + // the state `--adopt-store` acts on, so the line is a + // warning; the command's output names the owner and says + // whether it adopted the root or refused. + tracing::warn!( + "backup store: {kind} {} is owned by another catalog: {problem}{note}", + opened.root, + ); + } else { + // A command's own line: no server starts, and the command + // says what it does with an unhealthy store. + tracing::error!( + "backup store: {kind} {} is unhealthy: {problem}{note}", + opened.root, + ); + } + } } - let configured = configured(ConfiguredStore::new( - kind, - opened.root, - opened.store, - spool, - outcome, - )); + let configured = configured( + ConfiguredStore::new(kind, opened.root, opened.store, spool, outcome) + // Its health is settled here, so the flag changes no check; it + // tells a command's refusal that the root is the shared default + // one. + .defaulted(defaulted), + ); Ok(StoreSetup::with_store(runtime, configured, spool_lock)) } @@ -525,7 +604,8 @@ enum LocalChecks { } /// A filesystem store's startup checks: `validate`, then -/// [`store_identity::establish`], bounded by [`LOCAL_CHECK_TIMEOUT`] so a +/// [`store_identity::establish`] (which, for an in-memory catalog, adopts a +/// root another in-memory catalog owned), bounded by [`LOCAL_CHECK_TIMEOUT`] so a /// root on a mount that stops answering after `open` does not hold the /// listener. The store's synchronous filesystem calls run on the blocking /// pool, so the bound fires whichever call stalls; a timed-out check leaves @@ -536,11 +616,13 @@ async fn run_local_checks( root: &str, catalog: &C, backend: &str, + ephemeral_catalog: bool, ) -> LocalChecks { let checks = async { match store.validate().await { Ok(()) => LocalChecks::Done( - store_identity::establish(store, kind, root, catalog, backend).await, + store_identity::establish(store, kind, root, catalog, backend, ephemeral_catalog) + .await, ), // A store condition, not a configuration error: the probe write // or the root's resolution failed (module doc). @@ -583,6 +665,14 @@ fn rows_note(defaulted: bool) -> &'static str { } } +/// The clause of the unhealthy line under the `backup sync` exception +/// ([`Caller::SyncCommand`]): a defaulted root another catalog owns, with +/// no rows of this catalog. +const SYNC_EXCEPTION_NOTE: &str = " (the default root beside the database file, which every \ + database file in that directory shares; the catalog has no backup rows for it, and it is \ + configured for `backup sync`, so --adopt-store can adopt it from the catalog its \ + extenddb-store.json names)"; + /// The warning for a defaulted root whose creation, `open`, `validate` or /// identity check failed while the catalog has no rows for it: the store is /// left unconfigured, as the module doc says. @@ -592,9 +682,11 @@ fn warn_default_root_unusable(root: &str, problem: &str) { backup rows for it, so the store is unconfigured and CreateBackup writes in-catalog \ backups. Fix the root (it must be owned by this user with mode 0700, under directories \ owned by root or this user that others cannot write to, and its extenddb-store.json, \ - if any, must name this catalog: a catalog re-created by `destroy` and `init` needs the \ - old file removed or a new root), configure [backup.store] to choose another, or set \ - kind = \"disabled\" to silence this" + if any, must name this catalog: a catalog re-created by `destroy` and `init` adopts the \ + root and its backups with `extenddb backup sync --adopt-store`, once no server of the \ + catalog the file names runs, and the root is shared by every database file in its \ + directory), configure [backup.store] to choose another, or set kind = \"disabled\" to \ + silence this" ); } @@ -967,6 +1059,12 @@ mod tests { ) -> BoxedFuture<'a, Result, StorageError>> { unused() } + fn insert_backup_row<'a>( + &'a self, + _row: &'a BackupRow, + ) -> BoxedFuture<'a, Result> { + unused() + } fn claim_dead_backup<'a>( &'a self, _key: &'a BackupKey, @@ -1058,6 +1156,13 @@ mod tests { ) -> BoxedFuture<'a, Result> { unused() } + fn forget_available_backup_row<'a>( + &'a self, + _key: &'a BackupKey, + ) -> BoxedFuture<'a, Result> + { + unused() + } fn remove_dead_creating_backup_row<'a>( &'a self, _key: &'a BackupKey, @@ -1293,6 +1398,62 @@ mod tests { "{why:?}" ); assert!(why.to_string().contains("no rows for it"), "{why}"); + // `backup sync` is the exception: the same root is a configured, + // unhealthy store for it, which `--adopt-store` adopts; a purge + // takes the warning path as the server does. + let setup = configure_described( + &backup, + Some(&database), + "sqlite", + &without_rows, + Caller::SyncCommand, + ) + .await + .unwrap(); + assert_eq!(setup.not_configured, None); + assert_eq!( + setup + .runtime + .configured_store() + .expect("configured for sync") + .health() + .category(), + Some("owned by another catalog") + ); + let setup = configure_described( + &backup, + Some(&database), + "sqlite", + &without_rows, + Caller::Command, + ) + .await + .unwrap(); + assert!(setup.runtime.configured_store().is_none()); + // The exception is for a root another catalog owns only: another + // failure of the defaulted root takes the warning path for sync too. + std::fs::write(root.join(store_identity::STORE_IDENTITY_KEY), b"not json").unwrap(); + let setup = configure_described( + &backup, + Some(&database), + "sqlite", + &without_rows, + Caller::SyncCommand, + ) + .await + .unwrap(); + assert!(setup.runtime.configured_store().is_none()); + assert!( + matches!(&setup.not_configured, Some(NotConfigured::DefaultRootUnusable { problem, .. }) + if problem.contains("could not be read")), + "{:?}", + setup.not_configured + ); + std::fs::write( + root.join(store_identity::STORE_IDENTITY_KEY), + foreign.to_bytes(), + ) + .unwrap(); // The same file over rows for the root: a configured, unhealthy // store (`owned by another catalog`), not the warning. let setup = configure_described( @@ -1427,7 +1588,14 @@ mod tests { // fails here instead of hanging. let outcome = tokio::time::timeout( std::time::Duration::from_secs(120), - run_local_checks(&stalled, StoreKind::Filesystem, "/r", &catalog, "test"), + run_local_checks( + &stalled, + StoreKind::Filesystem, + "/r", + &catalog, + "test", + false, + ), ) .await .expect("the checks return at their bound"); @@ -1442,7 +1610,8 @@ mod tests { context: "write probe /r/.put-validate.tmp".to_owned(), source: std::io::Error::from_raw_os_error(28), })); - let outcome = run_local_checks(&io, StoreKind::Filesystem, "/r", &catalog, "test").await; + let outcome = + run_local_checks(&io, StoreKind::Filesystem, "/r", &catalog, "test", false).await; let LocalChecks::Done(Err(problem)) = outcome else { panic!("an I/O failure of the probe degrades the store"); }; @@ -1452,8 +1621,15 @@ mod tests { let denied = FakeStore(Validate::Fails(|| { StoreError::PermissionDenied("/r has mode 0755".to_owned()) })); - let outcome = - run_local_checks(&denied, StoreKind::Filesystem, "/r", &catalog, "test").await; + let outcome = run_local_checks( + &denied, + StoreKind::Filesystem, + "/r", + &catalog, + "test", + false, + ) + .await; assert!(matches!( outcome, LocalChecks::Refused(StoreError::PermissionDenied(_)) diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs index 8505c0b75..ea4293596 100644 --- a/crates/server/src/serve.rs +++ b/crates/server/src/serve.rs @@ -684,8 +684,22 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { .with_metrics(metrics.clone()) .with_shutdown(shutdown.child_token()) .with_source(&backend_name, build.version) - .with_limits(Arc::clone(&limits)), + .with_limits(Arc::clone(&limits)) + .with_region(&app_config.server.region), ); + // An in-memory catalog is new at every start: the server syncs it with + // the store before it serves, when the store's check passed here (a + // filesystem store), so the backups of the previous starts are served + // from the first request. The sync is bounded + // (`extenddb_engine::sync::STARTUP_SYNC_TIMEOUT`, 60 s); one that does + // not finish, or that met a store or catalog error on a prefix, is left + // to the reconciler's next cycle and logged (`backup_store` module doc). + if backup_runtime + .configured_store() + .is_some_and(extenddb_engine::ConfiguredStore::startup_sync_pending) + { + extenddb_engine::sync::startup_sync(&*storage, &backup_runtime).await; + } let backup_runtime_for_reconciler = Arc::clone(&backup_runtime); let state = AppState { storage: storage.clone(), diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs index 36f767dd7..61e7ee05e 100644 --- a/crates/storage-cassandra/src/backup_catalog.rs +++ b/crates/storage-cassandra/src/backup_catalog.rs @@ -38,9 +38,15 @@ //! plain batch after the lightweight transaction on the row. A process that //! dies between the two leaves an entry whose status the row no longer //! matches; every reader of the status table re-reads the row and skips such -//! an entry, and `remove_backup_row` deletes the entry under the status it -//! read, so the orphan costs a wasted read until then. The sweep that -//! removes orphan entries is `backup sync`'s, in a later release. +//! an entry, one wasted read each. `remove_backup_row` deletes the entries +//! under the status it read, so an entry under an earlier status stays until +//! a sweep of orphan entries exists, and no sweep removes them yet. One path +//! writes such an entry by design: `insert_backup_row`'s row is AVAILABLE +//! from its lightweight transaction on, so a DeleteBackup between that +//! transaction and the insert's entry batch moves a status entry the batch +//! has not written yet, and the batch then writes an AVAILABLE entry that no +//! later write removes (an insert completed by a later sync has the same +//! window). //! //! A conditional write whose error is the driver's `Io` or `Timeout`, or the //! server's `WriteTimeout` or `WriteFailure`, has an unknown outcome @@ -75,10 +81,10 @@ use cdrs_tokio::types::value::Value; use extenddb_core::backup_format::BackupArn; use extenddb_core::types::CreateTableInput; use extenddb_storage::backup_catalog::{ - BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, JobClaim, + JobRow, JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, - TimeBounds, WriteOutcome, billing_mode_name, new_generation, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, billing_mode_name, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -87,10 +93,15 @@ use futures::stream::BoxStream; use crate::CassandraEngine; use crate::engine::CassandraSession; -/// Attempts `start_backup_delete` makes when a restore registers or -/// unregisters between its in-use check and the DELETING claim. +/// Attempts `start_backup_delete` and `forget_available_backup_row` make +/// when a restore registers or unregisters between their in-use check and +/// their write. const DELETE_CLAIM_ATTEMPTS: usize = 3; +/// The condition of the DELETING write and of the outright removal: the row +/// is AVAILABLE and no restore is registered on it. +const DELETE_CLAIM_CONDITION: &str = "IF backup_status = 'AVAILABLE' AND active_restores = null"; + /// The `owner_generation` column value of a generation: the column is a /// signed 64-bit integer and `new_generation` never exceeds `i64::MAX`. pub(crate) fn generation_column(generation: u64) -> i64 { @@ -263,6 +274,23 @@ fn index_row_from_row(row: &Row) -> Result, StorageError> { })) } +/// Whether `read`, the row a serial read found, is the row +/// `insert_backup_row(row)` writes: unowned, with every column the insert +/// takes from `row` equal (`created_at` to the stored millisecond). +fn is_unowned_insert_of(read: &BackupRow, row: &BackupRow) -> bool { + read.owner_generation == UNOWNED_GENERATION + && read.backup_id == row.backup_id + && read.table_name == row.table_name + && read.backup_name == row.backup_name + && read.status == row.status + && read.store == row.store + && read.size_bytes == row.size_bytes + && read.item_count == row.item_count + && seconds_to_millis(read.created_at) == seconds_to_millis(row.created_at) + && read.failure_reason == row.failure_reason + && read.source_table_details_json == row.source_table_details_json +} + /// The rows `read` fetches, as a stream: the read runs once, on the first /// poll, and a failed read is one `Err` item. fn rows_stream<'a, F>(read: F) -> BoxStream<'a, Result> @@ -419,6 +447,9 @@ impl CassandraEngine { async fn remove_index_entries(&self, row: &BackupRow) -> Result<(), StorageError> { let catalog = self.catalog_keyspace(); let created_at = seconds_to_millis(row.created_at); + if let Some(error) = crate::test_hooks::index_entry_removal_error(&row.backup_arn) { + return Err(error); + } let batch = BatchQueryBuilder::new() .with_consistency(Consistency::LocalQuorum) .add_query( @@ -466,6 +497,107 @@ impl CassandraEngine { Ok(()) } + /// Write the three index entries of a `backups_by_arn` row this process + /// has just inserted, after it: its account listing, its table listing + /// and its status partition under `status`, in one logged batch. + async fn write_index_entries( + &self, + row: &BackupRow, + table_id: &str, + table_arn: &str, + status: BackupStatus, + ) -> Result<(), StorageError> { + let catalog = self.catalog_keyspace(); + let created_at = seconds_to_millis(row.created_at); + if let Some(error) = crate::test_hooks::index_entry_batch_error(&row.backup_arn) { + return Err(error); + } + let batch = BatchQueryBuilder::new() + .with_consistency(Consistency::LocalQuorum) + .add_query( + format!( + "INSERT INTO {catalog}.backups_by_account (account_id, created_at, \ + backup_arn, backup_name, table_id, table_name, table_arn, \ + backup_size_bytes) VALUES (?, ?, ?, ?, ?, ?, ?, ?)" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + created_at, + row.backup_arn.as_str(), + row.backup_name.as_str(), + table_id, + row.table_name.as_str(), + table_arn, + row.size_bytes + ), + ) + .add_query( + format!( + "INSERT INTO {catalog}.backups_by_table (account_id, table_name, \ + created_at, backup_arn, backup_name, table_id, table_arn, \ + backup_size_bytes) VALUES (?, ?, ?, ?, ?, ?, ?, ?)" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + row.table_name.as_str(), + created_at, + row.backup_arn.as_str(), + row.backup_name.as_str(), + table_id, + table_arn, + row.size_bytes + ), + ) + .add_query( + format!( + "INSERT INTO {catalog}.backup_index_by_status \ + (store_root, backup_status, backup_arn, account_id, store_kind, store_id) \ + VALUES (?, ?, ?, ?, ?, ?)" + ), + cdrs_tokio::query_values!( + row.store.store_root.as_str(), + status.as_str(), + row.backup_arn.as_str(), + row.account_id.as_str(), + row.store.store_kind.as_str(), + row.store.store_id.as_str() + ), + ); + self.session + .batch( + batch + .build() + .map_err(|e| StorageError::Internal(e.to_string()))?, + ) + .await + .map_err(|e| StorageError::Internal(format!("Backup index rows: {e}")))?; + Ok(()) + } + + /// [`Self::read_index_row`] at `LOCAL_SERIAL`, so a row a lightweight + /// transaction committed, or one whose round is in progress, is seen. + async fn read_index_row_serial( + &self, + account_id: &str, + backup_arn: &str, + ) -> Result, StorageError> { + let query = format!( + "SELECT * FROM {}.backups_by_arn WHERE account_id = ? AND backup_arn = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &query, + cdrs_tokio::query_values!(account_id, backup_arn), + "backup index row", + ) + .await?; + rows.first() + .map(index_row_from_row) + .transpose() + .map(|row| row.flatten().filter(|r| r.account_id == account_id)) + } + async fn read_index_row( &self, account_id: &str, @@ -537,6 +669,148 @@ impl CassandraEngine { } } + /// DeleteBackup's first step, and with `remove` the outright removal of + /// `forget_available_backup_row`, as one body: read the row, check its + /// status, run the in-use rule, then one lightweight transaction under + /// [`DELETE_CLAIM_CONDITION`] that writes DELETING or, with `remove`, + /// deletes the row (`Deleting` then means removed); then the row's index + /// entries. Every writer of `active_restores` is conditional, so the + /// condition orders the write wholly before or after a registration. + /// + /// A lost answer of the removal is settled by a serial read: no row + /// means removed, a row means the loop runs again (a lost answer on + /// every attempt is an error). A lost answer of the DELETING write is an + /// error (another delete may have written the same status). After the + /// removal a failed entry batch is logged: the row is gone, and every + /// reader of an entry re-reads the row and skips it. + async fn start_backup_delete_lwt( + &self, + key: &BackupKey, + remove: bool, + ) -> Result { + let catalog = self.catalog_keyspace(); + let write = if remove { + format!( + "DELETE FROM {catalog}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \ + {DELETE_CLAIM_CONDITION}" + ) + } else { + format!( + "UPDATE {catalog}.backups_by_arn SET backup_status = 'DELETING' \ + WHERE account_id = ? AND backup_arn = ? {DELETE_CLAIM_CONDITION}" + ) + }; + let mut attempt = 0; + loop { + let stored = self + .load_stored_backup(&key.account_id, &key.backup_arn) + .await? + .filter(|stored| !stored.is_in_catalog()) + .ok_or_else(|| not_found(&key.backup_arn))?; + let status = BackupStatus::parse(&stored.backup_status)?; + if status != BackupStatus::Available { + return Ok(DeleteStart::Status(status)); + } + // The in-use rule reads the target's status; a registration whose + // target is no longer being restored is cleared here so it cannot + // hold the row's lock. + if let Some(target) = self.creating_restore_target(&stored).await? { + return Ok(DeleteStart::InUse(target)); + } + // The removal's entries are keyed by the row's columns, so the + // row is read before it goes. + let removed_row = if remove { + Some( + self.read_index_row(&key.account_id, &key.backup_arn) + .await? + .ok_or_else(|| not_found(&key.backup_arn))?, + ) + } else { + None + }; + crate::test_hooks::before_delete_claim(&key.backup_arn).await; + let values = + || cdrs_tokio::query_values!(key.account_id.as_str(), key.backup_arn.as_str()); + let fault = if remove { + crate::test_hooks::forget_write_fault(&key.backup_arn) + } else { + None + }; + let outcome = if fault == Some(crate::test_hooks::ProgressWriteFault::Lost) { + WriteOutcome::Unknown + } else { + let outcome = conditional_write(&self.session, &write, values()).await?; + if fault.is_some() { + WriteOutcome::Unknown + } else { + outcome + } + }; + let applied = match outcome { + WriteOutcome::Matched => true, + WriteOutcome::NotMatched => false, + WriteOutcome::Unknown if remove => self + .read_index_row_serial(&key.account_id, &key.backup_arn) + .await? + .is_none(), + WriteOutcome::Unknown => { + return Err(StorageError::Internal(format!( + "the DELETING write of backup {} has an unknown outcome", + key.backup_arn + ))); + } + }; + let lost = outcome == WriteOutcome::Unknown; + if applied { + if let Some(row) = removed_row { + if let Err(error) = self.remove_index_entries(&row).await { + tracing::warn!( + "backup {}: the row is removed and its list and status entries \ + are not ({error}); readers verify against the row and skip them", + key.backup_arn + ); + } + } else { + let store = StoreIdentity { + store_kind: StoreKind::parse(stored.store_kind.as_deref().unwrap_or("")) + .unwrap_or(StoreKind::Filesystem), + store_root: stored.store_root.clone().unwrap_or_default(), + store_id: stored.store_id.clone().unwrap_or_default(), + }; + self.move_status_index_row(key, &store, "AVAILABLE", "DELETING") + .await; + } + return Ok(DeleteStart::Deleting); + } + attempt += 1; + if attempt >= DELETE_CLAIM_ATTEMPTS { + if lost { + return Err(StorageError::Internal(format!( + "the removal of backup {} has an unknown outcome on every attempt", + key.backup_arn + ))); + } + // A registration or another writer landed between the check + // and the write on every attempt. + let stored = self + .load_stored_backup(&key.account_id, &key.backup_arn) + .await? + .ok_or_else(|| not_found(&key.backup_arn))?; + let status = BackupStatus::parse(&stored.backup_status)?; + if status != BackupStatus::Available { + return Ok(DeleteStart::Status(status)); + } + let target = stored + .active_restores + .values() + .next() + .cloned() + .unwrap_or_default(); + return Ok(DeleteStart::InUse(target)); + } + } + } + /// The ARNs the status index lists for `store` under `status`. async fn status_index_arns( &self, @@ -968,65 +1242,13 @@ impl BackupCatalog for CassandraEngine { ))); } // The index rows, after the authoritative row. - let created_at = seconds_to_millis(row.created_at); - let batch = BatchQueryBuilder::new() - .with_consistency(Consistency::LocalQuorum) - .add_query( - format!( - "INSERT INTO {catalog}.backups_by_account (account_id, created_at, \ - backup_arn, backup_name, table_id, table_name, table_arn, \ - backup_size_bytes) VALUES (?, ?, ?, ?, ?, ?, ?, ?)" - ), - cdrs_tokio::query_values!( - row.account_id.as_str(), - created_at, - row.backup_arn.as_str(), - row.backup_name.as_str(), - source.table_id.as_str(), - row.table_name.as_str(), - source.table_arn.as_str(), - row.size_bytes - ), - ) - .add_query( - format!( - "INSERT INTO {catalog}.backups_by_table (account_id, table_name, \ - created_at, backup_arn, backup_name, table_id, table_arn, \ - backup_size_bytes) VALUES (?, ?, ?, ?, ?, ?, ?, ?)" - ), - cdrs_tokio::query_values!( - row.account_id.as_str(), - row.table_name.as_str(), - created_at, - row.backup_arn.as_str(), - row.backup_name.as_str(), - source.table_id.as_str(), - source.table_arn.as_str(), - row.size_bytes - ), - ) - .add_query( - format!( - "INSERT INTO {catalog}.backup_index_by_status \ - (store_root, backup_status, backup_arn, account_id, store_kind, store_id) \ - VALUES (?, 'CREATING', ?, ?, ?, ?)" - ), - cdrs_tokio::query_values!( - row.store.store_root.as_str(), - row.backup_arn.as_str(), - row.account_id.as_str(), - row.store.store_kind.as_str(), - row.store.store_id.as_str() - ), - ); - self.session - .batch( - batch - .build() - .map_err(|e| StorageError::Internal(e.to_string()))?, - ) - .await - .map_err(|e| StorageError::Internal(format!("Backup index rows: {e}")))?; + self.write_index_entries( + row, + &source.table_id, + &source.table_arn, + BackupStatus::Creating, + ) + .await?; Ok(Box::new(CassandraBackupClaim { engine: self.clone(), key: row.key(), @@ -1036,6 +1258,125 @@ impl BackupCatalog for CassandraEngine { }) } + fn insert_backup_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + Self::validate_account_id(&row.account_id)?; + let source = row.source_table()?; + let key_schema = serde_json::to_string(&source.key_schema) + .map_err(|e| StorageError::Internal(format!("Serialize key schema: {e}")))?; + let attribute_definitions = serde_json::to_string(&source.attribute_definitions) + .map_err(|e| StorageError::Internal(format!("Serialize attributes: {e}")))?; + let source_table_details_json = row.source_table_details_json.to_string(); + let catalog = self.catalog_keyspace(); + // `claim_backup`'s insert, unowned and complete. The partition is + // (`account_id`, `backup_arn`) and no table indexes the backup id, + // so a row for the id under another ARN spelling does not stop it. + let insert = format!( + "INSERT INTO {catalog}.backups_by_arn (account_id, backup_arn, backup_id, \ + backup_name, table_id, table_name, table_arn, backup_status, backup_type, \ + backup_size_bytes, item_count, key_schema, attribute_definitions, billing_mode, \ + table_created_at, created_at, completed_at, failure_reason, store_kind, \ + store_root, store_id, source_table_details_json, owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'USER', ?, ?, ?, ?, ?, ?, ?, toTimestamp(now()), \ + ?, ?, ?, ?, ?, ?, toTimestamp(now())) IF NOT EXISTS" + ); + let values = || { + cdrs_tokio::query_values!( + row.account_id.as_str(), + row.backup_arn.as_str(), + row.backup_id.as_str(), + row.backup_name.as_str(), + source.table_id.as_str(), + row.table_name.as_str(), + source.table_arn.as_str(), + row.status.as_str(), + row.size_bytes, + row.item_count, + key_schema.as_str(), + attribute_definitions.as_str(), + billing_mode_name(source.billing_mode), + seconds_to_millis(source.table_creation_date_time), + seconds_to_millis(row.created_at), + row.failure_reason.as_deref(), + row.store.store_kind.as_str(), + row.store.store_root.as_str(), + row.store.store_id.as_str(), + source_table_details_json.as_str(), + generation_column(UNOWNED_GENERATION) + ) + }; + // A lost answer, and an insert that finds a row, are settled by a + // serial read. The row as this call writes it counts as inserted: + // another `insert_backup_row` of the same verified prefix writes + // the same row, either call may count it, and one that stopped + // (or whose batch failed) between the row and its index entries + // left it with no entries, which this call then writes. Any other + // row is one that existed. No row after a lost answer means the + // proposal did not commit, so the write is sent once more, as + // design 4(e) allows; no row after `NotMatched` is a legacy + // in-catalog row at the ARN, which `read_index_row_serial` does + // not return. + let mut inserted = None; + for _ in 0..2 { + let fault = crate::test_hooks::insert_write_fault(&row.backup_arn); + let outcome = if fault == Some(crate::test_hooks::ProgressWriteFault::Lost) { + WriteOutcome::Unknown + } else { + let outcome = conditional_write(&self.session, &insert, values()).await?; + if fault.is_some() { + WriteOutcome::Unknown + } else { + outcome + } + }; + let read = || self.read_index_row_serial(&row.account_id, &row.backup_arn); + match outcome { + WriteOutcome::Matched => inserted = Some(true), + WriteOutcome::NotMatched => { + inserted = Some( + read() + .await? + .is_some_and(|read| is_unowned_insert_of(&read, row)), + ); + } + WriteOutcome::Unknown => { + inserted = read().await?.map(|read| is_unowned_insert_of(&read, row)); + } + } + if inserted.is_some() { + break; + } + } + let Some(inserted) = inserted else { + return Err(StorageError::Internal(format!( + "the insert of backup {} has an unknown outcome", + row.backup_arn + ))); + }; + if inserted { + // The index entries, after the authoritative row, the status + // entry under the row's status. The batch is idempotent + // upserts; a failed one is sent once more, and one that fails + // twice is an error the next insert of the row repairs. + if let Err(first) = self + .write_index_entries(row, &source.table_id, &source.table_arn, row.status) + .await + { + tracing::warn!( + "backup {}: index entries not written, sending them once more: {first}", + row.backup_arn + ); + self.write_index_entries(row, &source.table_id, &source.table_arn, row.status) + .await?; + } + } + Ok(inserted) + }) + } + fn list_backup_rows<'a>( &'a self, status: Option, @@ -1191,69 +1532,19 @@ impl BackupCatalog for CassandraEngine { &'a self, key: &'a BackupKey, ) -> BoxFuture<'a, Result> { + Box::pin(self.start_backup_delete_lwt(key, false)) + } + + fn forget_available_backup_row<'a>( + &'a self, + key: &'a BackupKey, + ) -> BoxFuture<'a, Result> { Box::pin(async move { - let mark_deleting = format!( - "UPDATE {}.backups_by_arn SET backup_status = 'DELETING' \ - WHERE account_id = ? AND backup_arn = ? \ - IF backup_status = 'AVAILABLE' AND active_restores = null", - self.catalog_keyspace() - ); - let mut attempt = 0; - loop { - let stored = self - .load_stored_backup(&key.account_id, &key.backup_arn) - .await? - .filter(|stored| !stored.is_in_catalog()) - .ok_or_else(|| not_found(&key.backup_arn))?; - let status = BackupStatus::parse(&stored.backup_status)?; - if status != BackupStatus::Available { - return Ok(DeleteStart::Status(status)); - } - // The in-use rule reads the target's status; a registration - // whose target is no longer being restored is cleared here so - // it cannot hold the row's lock. - if let Some(target) = self.creating_restore_target(&stored).await? { - return Ok(DeleteStart::InUse(target)); - } - crate::test_hooks::before_delete_claim(&key.backup_arn).await; - let result = crate::cassandra_util::query_lwt( - &self.session, - &mark_deleting, - cdrs_tokio::query_values!(key.account_id.as_str(), key.backup_arn.as_str()), - ) - .await?; - if crate::cassandra_util::lwt_applied(&result)? { - let store = StoreIdentity { - store_kind: StoreKind::parse(stored.store_kind.as_deref().unwrap_or("")) - .unwrap_or(StoreKind::Filesystem), - store_root: stored.store_root.clone().unwrap_or_default(), - store_id: stored.store_id.clone().unwrap_or_default(), - }; - self.move_status_index_row(key, &store, "AVAILABLE", "DELETING") - .await; - return Ok(DeleteStart::Deleting); - } - attempt += 1; - if attempt >= DELETE_CLAIM_ATTEMPTS { - // A registration landed between the check and the claim - // on every attempt: the backup is in use by it. - let stored = self - .load_stored_backup(&key.account_id, &key.backup_arn) - .await? - .ok_or_else(|| not_found(&key.backup_arn))?; - let status = BackupStatus::parse(&stored.backup_status)?; - if status != BackupStatus::Available { - return Ok(DeleteStart::Status(status)); - } - let target = stored - .active_restores - .values() - .next() - .cloned() - .unwrap_or_default(); - return Ok(DeleteStart::InUse(target)); - } - } + Ok(match self.start_backup_delete_lwt(key, true).await? { + DeleteStart::Deleting => ForgetRow::Removed, + DeleteStart::InUse(table) => ForgetRow::InUse(table), + DeleteStart::Status(status) => ForgetRow::Status(status), + }) }) } @@ -2404,4 +2695,62 @@ mod tests { let arns: Vec<&str> = rows.iter().map(|r| r.backup_arn.as_str()).collect(); assert_eq!(arns, ["z", "a", "b"]); } + + /// The settle of an insert whose answer was lost: the row read back is + /// this insert only when it is unowned and every column the insert takes + /// from the row is equal; the columns the insert sets itself + /// (`completed_at`, `heartbeat_at`) and the ARN, which the read was + /// keyed by, do not count. + #[test] + fn a_lost_insert_is_settled_only_by_the_row_it_writes() { + let row = BackupRow { + backup_id: "01757500000123-0badcafe".to_owned(), + account_id: "123456789012".to_owned(), + table_name: "Music".to_owned(), + backup_arn: + "arn:aws:dynamodb:us-east-1:123456789012:table/Music/backup/01757500000123-0badcafe" + .to_owned(), + backup_name: "nightly".to_owned(), + status: BackupStatus::Available, + store: StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/backups".to_owned(), + store_id: "store-1".to_owned(), + }, + size_bytes: 1234, + item_count: 5, + created_at: 1_757_500_000.123, + completed_at: None, + failure_reason: None, + source_table_details_json: serde_json::json!({ "table_name": "Music" }), + owner_generation: 99, + heartbeat_at: 0.0, + }; + let read = BackupRow { + owner_generation: UNOWNED_GENERATION, + completed_at: Some(1_757_500_001.0), + heartbeat_at: 1_757_500_001.0, + created_at: 1_757_500_000.123_4, + ..row.clone() + }; + assert!(is_unowned_insert_of(&read, &row)); + let differing: [fn(&mut BackupRow); 11] = [ + |r| r.owner_generation = 7, + |r| r.backup_id = "01757500000123-00000000".to_owned(), + |r| r.table_name = "Other".to_owned(), + |r| r.backup_name = "other".to_owned(), + |r| r.status = BackupStatus::Creating, + |r| r.store.store_id = "store-2".to_owned(), + |r| r.size_bytes = 1, + |r| r.item_count = 1, + |r| r.created_at = 1_757_500_000.124, + |r| r.failure_reason = Some("x".to_owned()), + |r| r.source_table_details_json = serde_json::json!({ "table_name": "Other" }), + ]; + for (i, change) in differing.iter().enumerate() { + let mut other = read.clone(); + change(&mut other); + assert!(!is_unowned_insert_of(&other, &row), "change {i}"); + } + } } diff --git a/crates/storage-cassandra/src/test_hooks.rs b/crates/storage-cassandra/src/test_hooks.rs index 42c48cf57..62f318c95 100644 --- a/crates/storage-cassandra/src/test_hooks.rs +++ b/crates/storage-cassandra/src/test_hooks.rs @@ -43,16 +43,26 @@ //! outcome; //! * a pause in a store-backed `bulk_load` between a batch's claim check and //! its writes, so a test can let a successor write and delete in that gap -//! and see the stalled batch's timestamp lose to the successor's state. +//! and see the stalled batch's timestamp lose to the successor's state; +//! * a fault on the lightweight transaction of `insert_backup_row`, and an +//! error on the logged batch that writes its index entries, so a test can +//! see the serial re-read settle a lost answer and an insert complete the +//! entries a stopped one left unwritten; +//! * a fault on the lightweight transaction of `forget_available_backup_row`, +//! and an error on the logged batch that removes a row's index entries, so +//! a test can see the serial re-read settle a lost removal and a removal +//! whose entries stay answer that the row is removed. use std::time::Duration; use extenddb_storage::error::StorageError; -/// How a test makes a progress lightweight transaction answer -/// `WriteOutcome::Unknown`: with the transaction run first (`Landed`: the -/// answer was lost on its way back) or not run at all (`Lost`: the proposal -/// never reached a replica). Each fault is taken by one write, in order. +/// How a test makes a lightweight transaction (a restore's progress write, +/// the insert of `insert_backup_row` or the removal of +/// `forget_available_backup_row`) answer `WriteOutcome::Unknown`: +/// with the transaction run first (`Landed`: the answer was lost on its way +/// back) or not run at all (`Lost`: the proposal never reached a replica). +/// Each fault is taken by one write, in order. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum ProgressWriteFault { Landed, @@ -114,6 +124,10 @@ mod armed { pub(super) restore_batch_size: HashMap, pub(super) progress_write_faults: HashMap>, pub(super) after_restore_claim_check: HashMap>, + pub(super) insert_write_faults: HashMap>, + pub(super) index_entry_batch_errors: HashMap, + pub(super) forget_write_faults: HashMap>, + pub(super) index_entry_removal_errors: HashMap, } pub(super) static HOOKS: LazyLock> = LazyLock::new(Mutex::default); @@ -284,6 +298,141 @@ pub fn fail_progress_writes(account_id: &str, faults: &[ProgressWriteFault]) { }); } +/// Make the next lightweight transactions of `insert_backup_row` for +/// `backup_arn` answer `WriteOutcome::Unknown`, one fault per write in the +/// order given, as [`fail_progress_writes`] does for progress writes. Writes +/// after the last fault run as usual. +#[cfg(feature = "test-hooks")] +pub fn fail_insert_writes(backup_arn: &str, faults: &[ProgressWriteFault]) { + armed::with_hooks(|hooks| { + hooks + .insert_write_faults + .insert(backup_arn.to_owned(), faults.iter().copied().collect()); + }); +} + +/// Make the next `count` logged batches that write the index entries of +/// `backup_arn` fail with an error before reaching the catalog. +#[cfg(feature = "test-hooks")] +pub fn fail_index_entry_batches(backup_arn: &str, count: usize) { + armed::with_hooks(|hooks| { + hooks + .index_entry_batch_errors + .insert(backup_arn.to_owned(), count); + }); +} + +/// Make the next lightweight transactions of `forget_available_backup_row` +/// for `backup_arn` answer `WriteOutcome::Unknown`, one fault per write in +/// the order given, as [`fail_insert_writes`] does for the insert. +#[cfg(feature = "test-hooks")] +pub fn fail_forget_writes(backup_arn: &str, faults: &[ProgressWriteFault]) { + armed::with_hooks(|hooks| { + hooks + .forget_write_faults + .insert(backup_arn.to_owned(), faults.iter().copied().collect()); + }); +} + +/// Make the next `count` logged batches that remove the index entries of +/// `backup_arn` fail with an error before reaching the catalog. +#[cfg(feature = "test-hooks")] +pub fn fail_index_entry_removals(backup_arn: &str, count: usize) { + armed::with_hooks(|hooks| { + hooks + .index_entry_removal_errors + .insert(backup_arn.to_owned(), count); + }); +} + +/// The fault the next removal lightweight transaction of `backup_arn` takes, +/// if a test armed one. +pub(crate) fn forget_write_fault(backup_arn: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + armed::with_hooks(|hooks| { + hooks + .forget_write_faults + .get_mut(backup_arn) + .and_then(std::collections::VecDeque::pop_front) + }) + } + #[cfg(not(feature = "test-hooks"))] + { + let _ = backup_arn; + None + } +} + +/// The error the next index-entry removal batch of `backup_arn` fails with, +/// if a test asked for one. +pub(crate) fn index_entry_removal_error(backup_arn: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + let injected = + armed::with_hooks( + |hooks| match hooks.index_entry_removal_errors.get_mut(backup_arn) { + Some(left) if *left > 0 => { + *left -= 1; + true + } + _ => false, + }, + ); + if injected { + return Some(StorageError::Internal( + "test hook: index entry removal failed before reaching the catalog".to_owned(), + )); + } + } + let _ = backup_arn; + None +} + +/// The fault the next insert lightweight transaction of `backup_arn` takes, +/// if a test armed one. +pub(crate) fn insert_write_fault(backup_arn: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + armed::with_hooks(|hooks| { + hooks + .insert_write_faults + .get_mut(backup_arn) + .and_then(std::collections::VecDeque::pop_front) + }) + } + #[cfg(not(feature = "test-hooks"))] + { + let _ = backup_arn; + None + } +} + +/// The error the next index-entry batch of `backup_arn` fails with, if a +/// test asked for one. +pub(crate) fn index_entry_batch_error(backup_arn: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + let injected = + armed::with_hooks( + |hooks| match hooks.index_entry_batch_errors.get_mut(backup_arn) { + Some(left) if *left > 0 => { + *left -= 1; + true + } + _ => false, + }, + ); + if injected { + return Some(StorageError::Internal( + "test hook: index entry batch failed before reaching the catalog".to_owned(), + )); + } + } + let _ = backup_arn; + None +} + /// The period the jobs of `account_id` heartbeat at. pub(crate) fn heartbeat_interval(account_id: &str) -> Duration { #[cfg(feature = "test-hooks")] diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs index bd68edef1..8302fd0e8 100644 --- a/crates/storage-cassandra/tests/direct/backup_engine.rs +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs @@ -2098,6 +2098,76 @@ fn arn_of(row: &BackupRow) -> BackupArn { BackupArn::parse(&row.backup_arn).unwrap() } +/// Delete `rows` from `backups_by_arn` and their entries from the status +/// index and the two listings, whatever their status, so a test that wrote +/// rows outside a claim leaves none in the shared catalog keyspace. +async fn remove_index_rows(engine: &CassandraEngine, rows: &[&BackupRow]) { + let catalog = engine.catalog_keyspace(); + for row in rows { + let created_at = (row.created_at * 1_000.0).round() as i64; + for (cql, values) in [ + ( + format!( + "DELETE FROM {catalog}.backups_by_arn WHERE account_id = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(row.account_id.as_str(), row.backup_arn.as_str()), + ), + ( + format!( + "DELETE FROM {catalog}.backups_by_account \ + WHERE account_id = ? AND created_at = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + created_at, + row.backup_arn.as_str() + ), + ), + ( + format!( + "DELETE FROM {catalog}.backups_by_table WHERE account_id = ? \ + AND table_name = ? AND created_at = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + row.table_name.as_str(), + created_at, + row.backup_arn.as_str() + ), + ), + ] { + engine + .session() + .query_with_values(&cql, values) + .await + .expect("remove a backup row"); + } + for status in [ + BackupStatus::Creating, + BackupStatus::Available, + BackupStatus::Deleting, + BackupStatus::Deleted, + BackupStatus::Failed, + ] { + engine + .session() + .query_with_values( + &format!( + "DELETE FROM {catalog}.backup_index_by_status \ + WHERE store_root = ? AND backup_status = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.store.store_root.as_str(), + status.as_str(), + row.backup_arn.as_str() + ), + ) + .await + .expect("remove a status entry"); + } + } +} + /// Run `cql` with `values` as a lightweight transaction and assert it applied. async fn apply(engine: &CassandraEngine, cql: &str, values: cdrs_tokio::query::QueryValues) { let result = @@ -2395,6 +2465,11 @@ async fn test_start_backup_delete_checks_the_status_and_the_in_use_rule_under_th engine.start_backup_delete(&key).await.unwrap(), DeleteStart::InUse("DelTarget".to_owned()) ); + assert_eq!( + engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::InUse("DelTarget".to_owned()), + "the outright removal keeps the in-use rule" + ); assert_eq!( engine .read_backup_row(&arn_of(&row)) @@ -2425,6 +2500,60 @@ async fn test_start_backup_delete_checks_the_status_and_the_in_use_rule_under_th DeleteStart::Status(BackupStatus::Deleting), "a retry finds the row DELETING" ); + assert_eq!( + engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Status(BackupStatus::Deleting), + "only an AVAILABLE row is removed outright" + ); + // An AVAILABLE row goes outright, read and listed nowhere, with no + // tombstone; a second removal finds no row. + let mut adopted = index_row(&account_id, "Music", "adopted", &store); + adopted.status = BackupStatus::Available; + assert!(engine.insert_backup_row(&adopted).await.expect("insert")); + assert_eq!( + engine + .forget_available_backup_row(&adopted.key()) + .await + .unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Removed + ); + assert!( + engine + .read_backup_row(&arn_of(&adopted)) + .await + .unwrap() + .is_none() + ); + for status in [None, Some(BackupStatus::Available)] { + let listed: Vec = engine + .list_backup_rows(status, &adopted.store) + .try_collect() + .await + .unwrap(); + assert!( + !listed.iter().any(|r| r.backup_arn == adopted.backup_arn), + "{status:?}" + ); + } + let by_account: Vec = engine + .list_account_backup_rows(&adopted.account_id, None, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + !by_account + .iter() + .any(|r| r.backup_arn == adopted.backup_arn) + ); + assert_eq!( + raw_index_entries(&engine, &adopted).await, + 0, + "the list and status entries went with the row" + ); + assert!(matches!( + engine.forget_available_backup_row(&adopted.key()).await, + Err(StorageError::Validation(m)) if m.contains("Backup not found") + )); assert!( !engine.remove_backup_row(&key).await.unwrap(), "DELETING is not a tombstone" @@ -3709,6 +3838,59 @@ async fn test_the_in_catalog_create_heartbeats_its_row_while_the_items_copy() { .expect("delete backup"); } +/// How many of `row`'s three index entries (`backups_by_account`, +/// `backups_by_table`, `backup_index_by_status` under its status) are in +/// the tables, read directly: every listing re-reads the authoritative row +/// and skips an entry whose row is gone, so only a direct read sees one. +async fn raw_index_entries(engine: &CassandraEngine, row: &BackupRow) -> usize { + let catalog = engine.catalog_keyspace(); + let created_at = (row.created_at * 1_000.0).round() as i64; + let mut found = 0; + for (cql, values) in [ + ( + format!( + "SELECT backup_arn FROM {catalog}.backups_by_account \ + WHERE account_id = ? AND created_at = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(row.account_id.as_str(), created_at, row.backup_arn.as_str()), + ), + ( + format!( + "SELECT backup_arn FROM {catalog}.backups_by_table WHERE account_id = ? \ + AND table_name = ? AND created_at = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + row.table_name.as_str(), + created_at, + row.backup_arn.as_str() + ), + ), + ( + format!( + "SELECT backup_arn FROM {catalog}.backup_index_by_status \ + WHERE store_root = ? AND backup_status = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.store.store_root.as_str(), + row.status.as_str(), + row.backup_arn.as_str() + ), + ), + ] { + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &cql, + values, + "index entry", + ) + .await + .expect("read an index entry"); + found += rows.len(); + } + found +} + #[cfg(feature = "test-hooks")] /// Items of `backup_arn`'s payload in `account_id`'s keyspace, all buckets. async fn payload_item_count(engine: &CassandraEngine, account_id: &str, backup_arn: &str) -> i64 { @@ -4890,6 +5072,608 @@ async fn test_read_store_ids_lists_the_distinct_ids_of_a_root() { ); } +/// `insert_backup_row`: the row lands as given, unowned, with `completed_at` +/// and `heartbeat_at` on the coordinator's clock, its status entry and both +/// list rows written (`list_backup_rows` reads the first, the account and +/// table listings the others); a second insert and an insert over a CREATING +/// row a claim holds answer `false` and change nothing; a claim over the +/// inserted row is refused. The rule is per ARN spelling: the same id under +/// another region is another row. +#[tokio::test] +async fn test_insert_backup_row_inserts_an_unowned_row_only_when_absent() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let mut adopted = index_row(&account_id, "Music", "adopted", &store); + adopted.status = BackupStatus::Available; + adopted.size_bytes = 1234; + adopted.item_count = 5; + // Ignored: the insert sets these three itself. + adopted.owner_generation = 99; + adopted.completed_at = None; + adopted.heartbeat_at = 0.0; + // A year before the id's instant: the two clock columns must be + // the database's clock at the insert, not the row's `created_at`. + adopted.created_at -= 365.0 * 86_400.0; + assert!(engine.insert_backup_row(&adopted).await.expect("insert")); + let stored = engine + .read_backup_row(&arn_of(&adopted)) + .await + .expect("read") + .expect("the row exists"); + assert_eq!(stored.owner_generation, UNOWNED_GENERATION); + let completed_at = stored.completed_at.expect("completed_at is set"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("the clock is after the epoch") + .as_secs_f64(); + for at in [completed_at, stored.heartbeat_at] { + assert!( + (at - now).abs() < 60.0, + "{at} is not the insert's instant {now}" + ); + } + assert!( + (stored.created_at - adopted.created_at).abs() < 1e-3, + "{} is the row's created_at {}", + stored.created_at, + adopted.created_at + ); + assert_eq!( + BackupRow { + owner_generation: 99, + completed_at: None, + heartbeat_at: 0.0, + created_at: adopted.created_at, + ..stored.clone() + }, + adopted, + "every other column is the row as given" + ); + + // A second insert of the key, with other values, writes nothing. + let mut again = adopted.clone(); + again.backup_name = "again".to_owned(); + again.size_bytes = 1; + again.status = BackupStatus::Failed; + assert!(!engine.insert_backup_row(&again).await.expect("insert")); + assert_eq!( + engine.read_backup_row(&arn_of(&adopted)).await.unwrap(), + Some(stored.clone()), + "the refused inserts changed nothing" + ); + let claimed = engine.claim_backup(&adopted).await.err(); + assert!( + matches!(claimed, Some(StorageError::AlreadyExists(_))), + "{claimed:?}" + ); + + // Over a CREATING row a live claim holds. + let creating = index_row(&account_id, "Music", "job", &store); + let claim = engine.claim_backup(&creating).await.expect("claim"); + let mut over = creating.clone(); + over.status = BackupStatus::Available; + assert!(!engine.insert_backup_row(&over).await.expect("insert")); + let job_row = engine + .read_backup_row(&arn_of(&creating)) + .await + .unwrap() + .unwrap(); + assert_eq!(job_row.status, BackupStatus::Creating); + assert_eq!(job_row.owner_generation, claim.generation()); + + // Read and listed like a created row. + let arns = + |rows: Vec| -> Vec { rows.into_iter().map(|r| r.backup_arn).collect() }; + let available: Vec = engine + .list_backup_rows(Some(BackupStatus::Available), &store) + .try_collect() + .await + .unwrap(); + assert_eq!(available, std::slice::from_ref(&stored)); + let creating_rows: Vec = engine + .list_backup_rows(Some(BackupStatus::Creating), &store) + .try_collect() + .await + .unwrap(); + assert_eq!( + arns(creating_rows), + std::slice::from_ref(&creating.backup_arn) + ); + for table in [None, Some("Music")] { + let listed: Vec = engine + .list_account_backup_rows(&account_id, table, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + listed.contains(&stored), + "{table:?}: {:?}", + arns(listed.clone()) + ); + } + drop(claim); + + // Cassandra indexes no backup id: another spelling of the ARN inserts. + let mut respelled = adopted.clone(); + respelled.backup_arn = + BackupArn::new("aws", "eu-west-1", &account_id, "Music", &adopted.backup_id) + .unwrap() + .to_arn_string(); + assert!(engine.insert_backup_row(&respelled).await.expect("insert")); + assert!( + engine + .read_backup_row(&arn_of(&respelled)) + .await + .unwrap() + .is_some() + ); + remove_index_rows(&engine, &[&adopted, &respelled, &creating]).await; +} + +/// Delete the three index entries of `row` and keep its `backups_by_arn` +/// row: the state an insert leaves when it stops, or its batch fails, +/// between the two writes. +async fn drop_index_entries(engine: &CassandraEngine, row: &BackupRow) { + let catalog = engine.catalog_keyspace(); + let created_at = (row.created_at * 1_000.0).round() as i64; + for (cql, values) in [ + ( + format!( + "DELETE FROM {catalog}.backups_by_account \ + WHERE account_id = ? AND created_at = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(row.account_id.as_str(), created_at, row.backup_arn.as_str()), + ), + ( + format!( + "DELETE FROM {catalog}.backups_by_table WHERE account_id = ? \ + AND table_name = ? AND created_at = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + row.table_name.as_str(), + created_at, + row.backup_arn.as_str() + ), + ), + ( + format!( + "DELETE FROM {catalog}.backup_index_by_status \ + WHERE store_root = ? AND backup_status = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + row.store.store_root.as_str(), + row.status.as_str(), + row.backup_arn.as_str() + ), + ), + ] { + engine + .session() + .query_with_values(&cql, values) + .await + .expect("remove an index entry"); + } +} + +/// Which of the three listings return `row`: the store's status listing +/// (`list_backup_rows`), the account listing and the table listing. +async fn listed_in(engine: &CassandraEngine, row: &BackupRow) -> [bool; 3] { + let by_status: Vec = engine + .list_backup_rows(Some(row.status), &row.store) + .try_collect() + .await + .expect("status listing"); + let mut listed = [by_status.iter().any(|r| r.backup_arn == row.backup_arn); 3]; + for (slot, table) in [(1, None), (2, Some(row.table_name.as_str()))] { + let rows: Vec = engine + .list_account_backup_rows(&row.account_id, table, &TimeBounds::default()) + .try_collect() + .await + .expect("account listing"); + listed[slot] = rows.iter().any(|r| r.backup_arn == row.backup_arn); + } + listed +} + +/// An insert that stopped between its row and its index entries leaves an +/// AVAILABLE row no listing returns. The next insert of the same row (the +/// next sync of the same verified prefix) finds the row its own, writes the +/// entries and answers `true`; an insert with other values still answers +/// `false` and writes nothing. +#[tokio::test] +async fn test_insert_backup_row_completes_an_insert_that_stopped_before_its_entries() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let mut row = index_row(&account_id, "Music", "torn", &store); + row.status = BackupStatus::Available; + row.size_bytes = 77; + row.item_count = 3; + assert!(engine.insert_backup_row(&row).await.expect("insert")); + drop_index_entries(&engine, &row).await; + assert_eq!( + listed_in(&engine, &row).await, + [false; 3], + "the torn state: the row is listed nowhere" + ); + + assert!( + engine.insert_backup_row(&row).await.expect("insert"), + "the second insert of the same row completes the first" + ); + assert_eq!(listed_in(&engine, &row).await, [true; 3]); + let stored = engine + .read_backup_row(&arn_of(&row)) + .await + .unwrap() + .expect("the row exists"); + assert_eq!(stored.owner_generation, UNOWNED_GENERATION); + + let mut other = row.clone(); + other.size_bytes = 1; + drop_index_entries(&engine, &row).await; + assert!( + !engine.insert_backup_row(&other).await.expect("insert"), + "a row with other values is one that existed" + ); + assert_eq!(listed_in(&engine, &row).await, [false; 3]); + remove_index_rows(&engine, &[&row]).await; +} + +/// An insert at an ARN an earlier release's in-catalog backup holds (a +/// `backups_by_arn` row with no store kind, which the index read does not +/// return) answers `false` and writes no list or status entry: the +/// `NotMatched` arm's serial read finds no index row, and that is not this +/// insert's row. +#[tokio::test] +async fn test_an_insert_over_an_in_catalog_backup_answers_false_and_writes_no_entry() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let mut row = index_row(&account_id, "Music", "legacy", &store); + row.status = BackupStatus::Available; + let catalog = engine.catalog_keyspace(); + engine + .session() + .query_with_values( + &format!( + "INSERT INTO {catalog}.backups_by_arn (account_id, backup_arn, backup_id, \ + backup_name, table_name, backup_status) VALUES (?, ?, ?, ?, ?, 'AVAILABLE')" + ), + cdrs_tokio::query_values!( + row.account_id.as_str(), + row.backup_arn.as_str(), + row.backup_id.as_str(), + "legacy", + "Music" + ), + ) + .await + .expect("an in-catalog backup's row"); + let inserted = engine.insert_backup_row(&row).await.expect("insert"); + let entries = raw_index_entries(&engine, &row).await; + remove_index_rows(&engine, &[&row]).await; + assert!( + !inserted, + "an insert over an in-catalog backup answered true" + ); + assert_eq!( + entries, 0, + "an insert over an in-catalog backup wrote entries" + ); +} + +/// An insert whose answer is lost is settled by a serial read: one whose +/// transaction ran finds its row and answers `true` with its entries +/// written; one whose transaction never ran finds no row, sends it once +/// more and answers `true`; one that finds a row a claim holds answers +/// `false` and changes nothing; two lost answers with no row are an error +/// and leave no row. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_an_unknown_insert_outcome_is_settled_by_a_serial_read() { + use extenddb_storage_cassandra::test_hooks::{ProgressWriteFault, fail_insert_writes}; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let available = |name: &str| { + let mut row = index_row(&account_id, "Music", name, &store); + row.status = BackupStatus::Available; + row + }; + + // The transaction ran; its answer was lost. + let landed = available("landed"); + fail_insert_writes(&landed.backup_arn, &[ProgressWriteFault::Landed]); + assert!(engine.insert_backup_row(&landed).await.expect("insert")); + assert_eq!(listed_in(&engine, &landed).await, [true; 3]); + + // The transaction never ran: the write is sent once more. + let lost = available("lost"); + fail_insert_writes(&lost.backup_arn, &[ProgressWriteFault::Lost]); + assert!(engine.insert_backup_row(&lost).await.expect("insert")); + assert_eq!(listed_in(&engine, &lost).await, [true; 3]); + let stored = engine.read_backup_row(&arn_of(&lost)).await.unwrap(); + assert_eq!(stored.map(|r| r.owner_generation), Some(UNOWNED_GENERATION)); + + // A row a claim holds is there: the insert is refused and changes nothing. + let creating = index_row(&account_id, "Music", "job", &store); + let claim = engine.claim_backup(&creating).await.expect("claim"); + let mut over = creating.clone(); + over.status = BackupStatus::Available; + fail_insert_writes(&over.backup_arn, &[ProgressWriteFault::Lost]); + assert!(!engine.insert_backup_row(&over).await.expect("insert")); + let job_row = engine + .read_backup_row(&arn_of(&creating)) + .await + .unwrap() + .unwrap(); + assert_eq!( + (job_row.status, job_row.owner_generation), + (BackupStatus::Creating, claim.generation()) + ); + assert!( + !listed_in(&engine, &over).await[0], + "no AVAILABLE status entry" + ); + drop(claim); + + // Two lost answers and no row: an unknown outcome is an error. + let unknown = available("unknown"); + fail_insert_writes( + &unknown.backup_arn, + &[ProgressWriteFault::Lost, ProgressWriteFault::Lost], + ); + let failed = engine + .insert_backup_row(&unknown) + .await + .expect_err("two lost answers and no row"); + assert!( + matches!(&failed, StorageError::Internal(m) if m.contains("unknown outcome")), + "{failed}" + ); + assert_eq!( + engine.read_backup_row(&arn_of(&unknown)).await.unwrap(), + None + ); + remove_index_rows(&engine, &[&landed, &lost, &creating, &unknown]).await; +} + +/// A failed batch of index entries is sent once more; one that fails twice +/// is an error that leaves the row unlisted, and the next insert of the row +/// writes the entries. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_failed_index_entry_batch_is_sent_once_more() { + use extenddb_storage_cassandra::test_hooks::fail_index_entry_batches; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let available = |name: &str| { + let mut row = index_row(&account_id, "Music", name, &store); + row.status = BackupStatus::Available; + row + }; + + let once = available("once"); + fail_index_entry_batches(&once.backup_arn, 1); + assert!(engine.insert_backup_row(&once).await.expect("insert")); + assert_eq!(listed_in(&engine, &once).await, [true; 3]); + + let twice = available("twice"); + fail_index_entry_batches(&twice.backup_arn, 2); + let failed = engine + .insert_backup_row(&twice) + .await + .expect_err("a batch that fails twice"); + assert!( + matches!(&failed, StorageError::Internal(m) if m.contains("index entry batch")), + "{failed}" + ); + assert!( + engine + .read_backup_row(&arn_of(&twice)) + .await + .unwrap() + .is_some() + ); + assert_eq!(listed_in(&engine, &twice).await, [false; 3]); + assert!(engine.insert_backup_row(&twice).await.expect("insert")); + assert_eq!(listed_in(&engine, &twice).await, [true; 3]); + remove_index_rows(&engine, &[&once, &twice]).await; +} + +/// A removal whose answer is lost is settled by a serial read: one whose +/// transaction ran finds no row and answers `Removed` with the entries +/// removed; one whose transaction never ran finds the row, runs once more and +/// answers `Removed`; a lost answer on every attempt is an error that leaves +/// the row AVAILABLE and listed. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_an_unknown_forget_outcome_is_settled_by_a_serial_read() { + use extenddb_storage::backup_catalog::ForgetRow; + use extenddb_storage_cassandra::test_hooks::{ProgressWriteFault, fail_forget_writes}; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let adopted = |name: &str| { + let mut row = index_row(&account_id, "Music", name, &store); + row.status = BackupStatus::Available; + row + }; + + for (name, fault) in [ + ("landed", ProgressWriteFault::Landed), + ("lost", ProgressWriteFault::Lost), + ] { + let row = adopted(name); + assert!(engine.insert_backup_row(&row).await.expect("insert")); + fail_forget_writes(&row.backup_arn, &[fault]); + assert_eq!( + engine + .forget_available_backup_row(&row.key()) + .await + .unwrap(), + ForgetRow::Removed, + "{name}" + ); + assert_eq!( + engine.read_backup_row(&arn_of(&row)).await.unwrap(), + None, + "{name}" + ); + assert_eq!(raw_index_entries(&engine, &row).await, 0, "{name}"); + } + + let unknown = adopted("unknown"); + assert!(engine.insert_backup_row(&unknown).await.expect("insert")); + fail_forget_writes(&unknown.backup_arn, &[ProgressWriteFault::Lost; 3]); + let failed = engine + .forget_available_backup_row(&unknown.key()) + .await + .expect_err("a lost answer on every attempt"); + assert!( + matches!(&failed, StorageError::Internal(m) if m.contains("unknown outcome")), + "{failed}" + ); + assert_eq!( + engine + .read_backup_row(&arn_of(&unknown)) + .await + .unwrap() + .map(|r| r.status), + Some(BackupStatus::Available) + ); + assert_eq!(listed_in(&engine, &unknown).await, [true; 3]); + remove_index_rows(&engine, &[&unknown]).await; +} + +/// A removal whose entry batch fails after the row is gone answers `Removed`: +/// the row is read and listed nowhere, the entries it left are skipped by +/// every listing, and a second removal finds no row. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_forget_whose_entry_removal_fails_answers_removed() { + use extenddb_storage::backup_catalog::ForgetRow; + use extenddb_storage_cassandra::test_hooks::fail_index_entry_removals; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let mut row = index_row(&account_id, "Music", "orphaned", &store); + row.status = BackupStatus::Available; + assert!(engine.insert_backup_row(&row).await.expect("insert")); + fail_index_entry_removals(&row.backup_arn, 1); + assert_eq!( + engine + .forget_available_backup_row(&row.key()) + .await + .unwrap(), + ForgetRow::Removed + ); + assert_eq!(engine.read_backup_row(&arn_of(&row)).await.unwrap(), None); + assert_eq!( + raw_index_entries(&engine, &row).await, + 3, + "the failed batch left the entries" + ); + assert_eq!( + listed_in(&engine, &row).await, + [false; 3], + "every listing skips an entry whose row is gone" + ); + assert!(matches!( + engine.forget_available_backup_row(&row.key()).await, + Err(StorageError::Validation(m)) if m.contains("Backup not found") + )); + remove_index_rows(&engine, &[&row]).await; +} + +/// The in-use rule runs before the removal's write and clears a registration +/// whose target is no longer being restored: a row whose `active_restores` +/// still names a finished (ACTIVE) target is removed at once. Without the +/// rule the write's `active_restores = null` condition refuses every attempt +/// and the removal answers `InUse` naming the finished table. +#[tokio::test] +async fn test_a_forget_clears_a_finished_restores_registration_and_removes_the_row() { + use extenddb_storage::backup_catalog::ForgetRow; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let mut row = index_row(&account_id, "Music", "finished", &store); + row.status = BackupStatus::Available; + assert!(engine.insert_backup_row(&row).await.expect("insert")); + let finished = create_active_table( + &engine, + &account_id, + CreateTableInput { + table_name: "FinishedTarget".to_owned(), + key_schema: vec![ks("id", KeyType::Hash)], + attribute_definitions: vec![attr("id", ScalarAttributeType::S)], + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + }, + ) + .await; + apply( + &engine, + &format!( + "UPDATE {}.backups_by_arn SET active_restores[?] = ? \ + WHERE account_id = ? AND backup_arn = ? IF EXISTS", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + finished.table_id.as_str(), + "FinishedTarget", + account_id.as_str(), + row.backup_arn.as_str() + ), + ) + .await; + assert_eq!( + engine + .forget_available_backup_row(&row.key()) + .await + .unwrap(), + ForgetRow::Removed + ); + assert_eq!(engine.read_backup_row(&arn_of(&row)).await.unwrap(), None); + assert_eq!(raw_index_entries(&engine, &row).await, 0); +} + /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is older /// than the window is removed outright, its status-index entry with it; one /// with a live owner, one that is not CREATING, and one that is gone are left, diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs index cfbfb83af..4dda98fb4 100644 --- a/crates/storage-mongodb/src/backup_catalog.rs +++ b/crates/storage-mongodb/src/backup_catalog.rs @@ -42,10 +42,10 @@ use std::time::Duration; use extenddb_core::backup_format::BackupArn; use extenddb_core::types::{CreateTableInput, RestoreSummary, TableDescription}; use extenddb_storage::backup_catalog::{ - BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, JobClaim, + JobRow, JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, - TimeBounds, WriteOutcome, new_generation, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::TryStreamExt; @@ -196,16 +196,17 @@ fn row_from_doc(doc: &Document) -> Result { }) } -/// The fields of `row` the claim's insert sets, status CREATING; the -/// generation and the heartbeat are added by the caller. -fn row_fields(row: &BackupRow) -> Result { +/// The fields of `row` an insert sets, with `status` (the claim's CREATING, +/// or the row's own for `insert_backup_row`); the generation, the heartbeat +/// and any completion are added by the caller. +fn row_fields(row: &BackupRow, status: BackupStatus) -> Result { let source = bson::to_bson(&row.source_table_details_json) .map_err(|e| StorageError::Internal(format!("serialize source_table_details_json: {e}")))?; Ok(doc! { "table_name": &row.table_name, "backup_arn": &row.backup_arn, "backup_name": &row.backup_name, - "status": BackupStatus::Creating.as_str(), + "status": status.as_str(), "store_kind": row.store.store_kind.as_str(), "store_root": &row.store.store_root, "store_id": &row.store.store_id, @@ -609,8 +610,14 @@ impl MongoEngine { /// DeleteBackup's first step as one session transaction: `$inc` the /// row's `lock_seq` (the lock), read the status, run the in-use rule, - /// write DELETING. A transaction that lost a write conflict is re-run. - async fn start_backup_delete_tx(&self, key: &BackupKey) -> Result { + /// write DELETING, or, with `remove`, delete the row + /// (`forget_available_backup_row`; `Deleting` then means removed). A + /// transaction that lost a write conflict is re-run. + async fn start_backup_delete_tx( + &self, + key: &BackupKey, + remove: bool, + ) -> Result { let index = self.backup_index(); let mut session = self.client.start_session().await.map_err(internal)?; let tx_options = self.write_transaction_options(); @@ -640,14 +647,22 @@ impl MongoEngine { } let mut filter = key_filter(key); filter.insert("status", BackupStatus::Available.as_str()); - index - .update_one( - filter, - doc! { "$set": { "status": BackupStatus::Deleting.as_str() } }, - ) - .session(&mut session) - .await - .map_err(TxErr::from)?; + if remove { + index + .delete_one(filter) + .session(&mut session) + .await + .map_err(TxErr::from)?; + } else { + index + .update_one( + filter, + doc! { "$set": { "status": BackupStatus::Deleting.as_str() } }, + ) + .session(&mut session) + .await + .map_err(TxErr::from)?; + } Ok(DeleteStart::Deleting) } .await; @@ -667,8 +682,12 @@ impl MongoEngine { } } Err(StorageError::Internal(format!( - "DeleteBackup of {} did not commit after {TRANSIENT_RETRY_ATTEMPTS} conflicting \ - attempts", + "{} of {} did not commit after {TRANSIENT_RETRY_ATTEMPTS} conflicting attempts", + if remove { + "The backup sync --prune-missing removal" + } else { + "DeleteBackup" + }, key.backup_arn ))) } @@ -841,7 +860,7 @@ impl BackupCatalog for MongoEngine { // server's clock on the heartbeat; an existing row never matches // the filter (every row carries a generation), so the upsert's // insert hits the unique index and the claim is refused. - let mut set = row_fields(row)?; + let mut set = row_fields(row, BackupStatus::Creating)?; set.insert("owner_generation", generation_field(generation)); set.insert("heartbeat_at", "$$NOW"); let set = literal_set(set); @@ -874,6 +893,38 @@ impl BackupCatalog for MongoEngine { }) } + fn insert_backup_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The claim's upsert, unowned: an existing row never matches the + // filter (every row carries a generation, this one 0), so the + // upsert's insert hits a unique index, (`account_id`, + // `backup_id`) or `backup_id`, and nothing is written. + let mut filter = key_filter(&row.key()); + filter.insert("owner_generation", doc! { "$exists": false }); + let mut set = row_fields(row, row.status)?; + if let Some(reason) = &row.failure_reason { + set.insert("failure_reason", reason); + } + set.insert("owner_generation", generation_field(UNOWNED_GENERATION)); + set.insert("heartbeat_at", "$$NOW"); + set.insert("completed_at", "$$NOW"); + let set = literal_set(set); + match self + .backup_index() + .update_one(filter, vec![doc! { "$set": set }]) + .upsert(true) + .await + { + Ok(result) => Ok(result.upserted_id.is_some()), + Err(e) if is_duplicate_key(&e) => Ok(false), + Err(e) => Err(internal(e)), + } + }) + } + fn list_backup_rows<'a>( &'a self, status: Option, @@ -968,7 +1019,20 @@ impl BackupCatalog for MongoEngine { &'a self, key: &'a BackupKey, ) -> BoxFuture<'a, Result> { - Box::pin(self.start_backup_delete_tx(key)) + Box::pin(self.start_backup_delete_tx(key, false)) + } + + fn forget_available_backup_row<'a>( + &'a self, + key: &'a BackupKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + Ok(match self.start_backup_delete_tx(key, true).await? { + DeleteStart::Deleting => ForgetRow::Removed, + DeleteStart::InUse(table) => ForgetRow::InUse(table), + DeleteStart::Status(status) => ForgetRow::Status(status), + }) + }) } fn set_backup_deleted<'a>( @@ -1115,8 +1179,8 @@ impl BackupCatalog for MongoEngine { // provenance keeps the source backup in use, whatever the // collection drops do, and the target counts as removed: a drop // that fails then is logged, naming the collections, and leaves - // them for `backup sync` (no record remains for a later cycle to - // retry from). + // them in the data database (no record remains for a later cycle + // to retry from, and no sweep removes them yet). let removed = self .tables() .find_one_and_delete(doc! { @@ -1143,7 +1207,8 @@ impl BackupCatalog for MongoEngine { tracing::error!( "the restore target {} (account {}) was removed from the catalog, but \ dropping its data collection {} or the index collections of table id {} \ - failed; they are left in the data database for `backup sync` to sweep: {e}", + failed; they are left in the data database, where nothing removes them, \ + and can be dropped by hand: {e}", key.table_name, key.account_id, crate::data::data_collection_name(&key.table_id), @@ -1553,7 +1618,7 @@ mod tests { #[test] fn a_row_round_trips_through_its_document() { let row = row(); - let mut doc = row_fields(&row).unwrap(); + let mut doc = row_fields(&row, row.status).unwrap(); doc.insert("account_id", &row.account_id); doc.insert("backup_id", &row.backup_id); doc.insert("owner_generation", generation_field(row.owner_generation)); @@ -1566,7 +1631,7 @@ mod tests { #[test] fn a_document_with_an_unknown_store_kind_or_status_is_internal() { let row = row(); - let mut doc = row_fields(&row).unwrap(); + let mut doc = row_fields(&row, row.status).unwrap(); doc.insert("account_id", &row.account_id); doc.insert("backup_id", &row.backup_id); doc.insert("owner_generation", 7_i64); diff --git a/crates/storage-mongodb/src/backup_engine.rs b/crates/storage-mongodb/src/backup_engine.rs index 2bdd0ba8a..17a4507eb 100644 --- a/crates/storage-mongodb/src/backup_engine.rs +++ b/crates/storage-mongodb/src/backup_engine.rs @@ -78,8 +78,8 @@ //! document is gone the target is removed (the name and the backup are //! free, and no record is left for a later cycle), so a drop that fails //! after it is logged with the collections' names and leaves them in the -//! data database for `backup sync` to sweep, as nothing in the catalog -//! points at them. +//! data database, where nothing in the catalog points at them and no sweep +//! removes them yet; the log line names them for an operator to drop. use std::time::{Duration, Instant}; diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs index 5919d51ef..4b2cb377e 100644 --- a/crates/storage-mongodb/src/backup_live_tests.rs +++ b/crates/storage-mongodb/src/backup_live_tests.rs @@ -754,6 +754,169 @@ async fn live_read_store_ids_lists_the_distinct_ids_of_a_root() { fx.finish().await; } +/// `insert_backup_row`: the row lands as given, unowned (the document +/// carries `owner_generation: 0`, which the claim's upsert filter relies on), +/// with `completed_at` and `heartbeat_at` on the server's clock; a second +/// insert, an insert over a CREATING row a claim holds, and an insert of the +/// id under another account (the unique `backup_id`) answer `false` and +/// change nothing; a claim over the inserted row is refused; the row is read +/// and listed like a created one. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_insert_backup_row_inserts_an_unowned_row_only_when_absent() { + let fx = Fixture::new().await; + let mut adopted = row(ACCOUNT, "Music", "adopted"); + adopted.status = BackupStatus::Available; + adopted.size_bytes = 1234; + adopted.item_count = 5; + // Ignored: the insert sets these three itself. + adopted.owner_generation = 99; + adopted.completed_at = None; + adopted.heartbeat_at = 0.0; + // A year before the id's instant: the two clock columns must be + // the database's clock at the insert, not the row's `created_at`. + adopted.created_at -= 365.0 * 86_400.0; + assert!(fx.engine.insert_backup_row(&adopted).await.expect("insert")); + let stored = fx + .engine + .read_backup_row(&arn_of(&adopted)) + .await + .expect("read") + .expect("the row exists"); + assert_eq!(stored.owner_generation, UNOWNED_GENERATION); + let completed_at = stored.completed_at.expect("completed_at is set"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("the clock is after the epoch") + .as_secs_f64(); + for at in [completed_at, stored.heartbeat_at] { + assert!( + (at - now).abs() < 60.0, + "{at} is not the insert's instant {now}" + ); + } + assert!( + (stored.created_at - adopted.created_at).abs() < 1e-3, + "{} is the row's created_at {}", + stored.created_at, + adopted.created_at + ); + assert_eq!( + BackupRow { + owner_generation: 99, + completed_at: None, + heartbeat_at: 0.0, + created_at: adopted.created_at, + ..stored.clone() + }, + adopted, + "every other column is the row as given" + ); + + // A second insert of the key, with other values, writes nothing. + let mut again = adopted.clone(); + again.backup_name = "again".to_owned(); + again.size_bytes = 1; + again.status = BackupStatus::Failed; + assert!(!fx.engine.insert_backup_row(&again).await.expect("insert")); + // The id under another account: the unique `backup_id`. + let mut elsewhere = adopted.clone(); + elsewhere.account_id = OTHER_ACCOUNT.to_owned(); + elsewhere.backup_arn = + extenddb_storage::util::backup_arn(REGION, OTHER_ACCOUNT, "Music", &adopted.backup_id); + assert!( + !fx.engine + .insert_backup_row(&elsewhere) + .await + .expect("insert") + ); + assert!( + fx.engine + .read_backup_row(&arn_of(&elsewhere)) + .await + .unwrap() + .is_none() + ); + assert_eq!( + fx.engine.read_backup_row(&arn_of(&adopted)).await.unwrap(), + Some(stored.clone()), + "the refused inserts changed nothing" + ); + let claimed = fx.engine.claim_backup(&adopted).await.err(); + assert!( + matches!(claimed, Some(StorageError::AlreadyExists(_))), + "{claimed:?}" + ); + + // Over a CREATING row a live claim holds. + let creating = row(ACCOUNT, "Music", "job"); + let claim = fx.engine.claim_backup(&creating).await.expect("claim"); + let mut over = creating.clone(); + over.status = BackupStatus::Available; + assert!(!fx.engine.insert_backup_row(&over).await.expect("insert")); + let job_row = fx + .engine + .read_backup_row(&arn_of(&creating)) + .await + .unwrap() + .unwrap(); + assert_eq!(job_row.status, BackupStatus::Creating); + assert_eq!(job_row.owner_generation, claim.generation()); + + // Read and listed like a created row. + let arns = + |rows: Vec| -> Vec { rows.into_iter().map(|r| r.backup_arn).collect() }; + let available: Vec = fx + .engine + .list_backup_rows(Some(BackupStatus::Available), &store()) + .try_collect() + .await + .unwrap(); + assert_eq!(available, std::slice::from_ref(&stored)); + let creating_rows: Vec = fx + .engine + .list_backup_rows(Some(BackupStatus::Creating), &store()) + .try_collect() + .await + .unwrap(); + assert_eq!( + arns(creating_rows), + std::slice::from_ref(&creating.backup_arn) + ); + for table in [None, Some("Music")] { + let listed: Vec = fx + .engine + .list_account_backup_rows(ACCOUNT, table, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + listed.contains(&stored), + "{table:?}: {:?}", + arns(listed.clone()) + ); + } + drop(claim); + + let document = fx + .engine + .backup_index() + .find_one(doc! { "backup_id": &adopted.backup_id }) + .await + .expect("read") + .expect("the document"); + assert_eq!(document.get_i64("owner_generation"), Ok(0), "{document}"); + assert_eq!( + fx.engine + .backup_index() + .count_documents(doc! { "backup_id": &adopted.backup_id }) + .await + .expect("count"), + 1 + ); + fx.finish().await; +} + /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is older /// than the window is removed outright; one with a live owner, one that is /// not CREATING, and one that is gone are left, answering `false`. @@ -1289,6 +1452,11 @@ async fn live_start_backup_delete_checks_the_status_and_the_in_use_rule_under_th fx.engine.start_backup_delete(&key).await.unwrap(), DeleteStart::InUse("del_target".to_owned()) ); + assert_eq!( + fx.engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::InUse("del_target".to_owned()), + "the outright removal keeps the in-use rule" + ); assert_eq!( fx.engine .read_backup_row(&arn_of(&row)) @@ -1314,6 +1482,57 @@ async fn live_start_backup_delete_checks_the_status_and_the_in_use_rule_under_th DeleteStart::Status(BackupStatus::Deleting), "a retry finds the row DELETING" ); + assert_eq!( + fx.engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Status(BackupStatus::Deleting), + "only an AVAILABLE row is removed outright" + ); + // An AVAILABLE row goes outright, read and listed nowhere, with no + // tombstone; a second removal finds no row. + let mut adopted = self::row(ACCOUNT, "Music", "adopted"); + adopted.status = BackupStatus::Available; + assert!(fx.engine.insert_backup_row(&adopted).await.expect("insert")); + assert_eq!( + fx.engine + .forget_available_backup_row(&adopted.key()) + .await + .unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Removed + ); + assert!( + fx.engine + .read_backup_row(&arn_of(&adopted)) + .await + .unwrap() + .is_none() + ); + for status in [None, Some(BackupStatus::Available)] { + let listed: Vec = fx + .engine + .list_backup_rows(status, &adopted.store) + .try_collect() + .await + .unwrap(); + assert!( + !listed.iter().any(|r| r.backup_arn == adopted.backup_arn), + "{status:?}" + ); + } + let by_account: Vec = fx + .engine + .list_account_backup_rows(&adopted.account_id, None, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + !by_account + .iter() + .any(|r| r.backup_arn == adopted.backup_arn) + ); + assert!(matches!( + fx.engine.forget_available_backup_row(&adopted.key()).await, + Err(StorageError::Validation(m)) if m.contains("Backup not found") + )); assert!( !fx.engine.remove_backup_row(&key).await.unwrap(), "DELETING is not a tombstone" diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs index d973eaf00..83453dff2 100644 --- a/crates/storage-postgres/src/backup_catalog.rs +++ b/crates/storage-postgres/src/backup_catalog.rs @@ -45,10 +45,10 @@ use std::time::Duration; use extenddb_core::backup_format::BackupArn; use extenddb_core::types::{CreateTableInput, RestoreSummary}; use extenddb_storage::backup_catalog::{ - BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, JobClaim, + JobRow, JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, - TimeBounds, WriteOutcome, new_generation, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -542,6 +542,47 @@ impl BackupCatalog for PostgresEngine { }) } + fn insert_backup_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // No conflict target: the primary key (`account_id`, + // `backup_id`) and the unique `backup_id` both make it a no-op. + // No advisory lock: a job that holds the id's lock and has not + // inserted its row yet finds this one and is refused, as it is by + // any row with the key. + let inserted = sqlx::query( + "INSERT INTO backup_index (backup_id, account_id, table_name, backup_arn, \ + backup_name, status, store_kind, store_root, store_id, size_bytes, item_count, \ + created_at, completed_at, failure_reason, source_table_details_json, \ + owner_generation, heartbeat_at) \ + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, to_timestamp($12), now(), \ + $13, $14, $15, now()) ON CONFLICT DO NOTHING", + ) + .bind(&row.backup_id) + .bind(&row.account_id) + .bind(&row.table_name) + .bind(&row.backup_arn) + .bind(&row.backup_name) + .bind(row.status.as_str()) + .bind(row.store.store_kind.as_str()) + .bind(&row.store.store_root) + .bind(&row.store.store_id) + .bind(row.size_bytes) + .bind(row.item_count) + .bind(row.created_at) + .bind(&row.failure_reason) + .bind(&row.source_table_details_json) + .bind(generation_column(UNOWNED_GENERATION)) + .execute(&self.pool) + .await + .map_err(db_err)? + .rows_affected(); + Ok(inserted == 1) + }) + } + fn list_backup_rows<'a>( &'a self, status: Option, @@ -631,42 +672,19 @@ impl BackupCatalog for PostgresEngine { &'a self, key: &'a BackupKey, ) -> BoxFuture<'a, Result> { + Box::pin(self.start_backup_delete_tx(key, false)) + } + + fn forget_available_backup_row<'a>( + &'a self, + key: &'a BackupKey, + ) -> BoxFuture<'a, Result> { Box::pin(async move { - let mut tx = self.pool.begin().await.map_err(db_err)?; - // FOR UPDATE is the contract a restore's registration of this - // row must meet: it locks the same row before its own in-use - // check, so the two order wholly one before the other (a - // registration that commits first leaves this `InUse`; one that - // waits behind this finds the row DELETING). `claim_restore` - // (`create_table.rs`, the manifest arm) takes this lock; the - // legacy registration locks the legacy `backups` row instead. - let status: Option = sqlx::query_scalar( - "SELECT status FROM backup_index WHERE account_id = $1 AND backup_id = $2 \ - FOR UPDATE", - ) - .bind(&key.account_id) - .bind(&key.backup_id) - .fetch_optional(&mut *tx) - .await - .map_err(db_err)?; - let status = BackupStatus::parse(&status.ok_or_else(|| not_found(&key.backup_arn))?)?; - if status != BackupStatus::Available { - return Ok(DeleteStart::Status(status)); - } - if let Some(table) = creating_restore_target(&mut *tx, &key.backup_arn).await? { - return Ok(DeleteStart::InUse(table)); - } - sqlx::query( - "UPDATE backup_index SET status = 'DELETING' \ - WHERE account_id = $1 AND backup_id = $2 AND status = 'AVAILABLE'", - ) - .bind(&key.account_id) - .bind(&key.backup_id) - .execute(&mut *tx) - .await - .map_err(db_err)?; - tx.commit().await.map_err(db_err)?; - Ok(DeleteStart::Deleting) + Ok(match self.start_backup_delete_tx(key, true).await? { + DeleteStart::Deleting => ForgetRow::Removed, + DeleteStart::InUse(table) => ForgetRow::InUse(table), + DeleteStart::Status(status) => ForgetRow::Status(status), + }) }) } @@ -1217,6 +1235,57 @@ const MAINTENANCE_TAKE_TRIES: u32 = 6; const MAINTENANCE_TAKE_PAUSE: Duration = Duration::from_millis(2); impl crate::PostgresEngine { + /// DeleteBackup's first step, and with `remove` the outright removal of + /// `forget_available_backup_row`, as one transaction: lock the row, read + /// its status, run the in-use rule, then write DELETING or, with + /// `remove`, delete the row (`Deleting` then means removed). One body, + /// so the two methods share the lock and the rule. + async fn start_backup_delete_tx( + &self, + key: &BackupKey, + remove: bool, + ) -> Result { + let mut tx = self.pool.begin().await.map_err(db_err)?; + // FOR UPDATE is the contract a restore's registration of this row + // must meet: it locks the same row before its own in-use check, so + // the two order wholly one before the other (a registration that + // commits first leaves this `InUse`; one that waits behind this + // finds the row DELETING, or no row). `claim_restore` + // (`create_table.rs`, the manifest arm) takes this lock; the legacy + // registration locks the legacy `backups` row instead. + let status: Option = sqlx::query_scalar( + "SELECT status FROM backup_index WHERE account_id = $1 AND backup_id = $2 \ + FOR UPDATE", + ) + .bind(&key.account_id) + .bind(&key.backup_id) + .fetch_optional(&mut *tx) + .await + .map_err(db_err)?; + let status = BackupStatus::parse(&status.ok_or_else(|| not_found(&key.backup_arn))?)?; + if status != BackupStatus::Available { + return Ok(DeleteStart::Status(status)); + } + if let Some(table) = creating_restore_target(&mut *tx, &key.backup_arn).await? { + return Ok(DeleteStart::InUse(table)); + } + let write = if remove { + "DELETE FROM backup_index \ + WHERE account_id = $1 AND backup_id = $2 AND status = 'AVAILABLE'" + } else { + "UPDATE backup_index SET status = 'DELETING' \ + WHERE account_id = $1 AND backup_id = $2 AND status = 'AVAILABLE'" + }; + sqlx::query(write) + .bind(&key.account_id) + .bind(&key.backup_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; + tx.commit().await.map_err(db_err)?; + Ok(DeleteStart::Deleting) + } + /// The lock step of `maintenance_claim` (the tries on one claim /// connection) with `tries` and `pause` in place of the take's own, so a /// test can hold the key's shared lock past the first try and release it diff --git a/crates/storage-postgres/tests/backup_restore.rs b/crates/storage-postgres/tests/backup_restore.rs index 85115d440..2a58b5ff0 100644 --- a/crates/storage-postgres/tests/backup_restore.rs +++ b/crates/storage-postgres/tests/backup_restore.rs @@ -2996,6 +2996,29 @@ mod catalog_index { ); return; } + waits_for_the_row_lock_and_then_sees_the_registration(false).await; + } + + /// `forget_available_backup_row` runs under the same lock: the outright + /// removal waits in the `SELECT ... FOR UPDATE` while a registration + /// holds the row, then answers `InUse` and leaves the row AVAILABLE. + /// Without the lock it would wait in its `DELETE` instead, and then + /// remove the row the committed registration reads. + #[tokio::test] + async fn forget_available_backup_row_waits_for_the_row_lock_and_then_sees_the_registration() { + if base_conn().is_none() { + eprintln!( + "SKIP forget_available_backup_row_waits_for_the_row_lock_and_then_sees_the_registration: no PostgreSQL" + ); + return; + } + waits_for_the_row_lock_and_then_sees_the_registration(true).await; + } + + /// The two cells above: `start_backup_delete`, or with `remove` + /// `forget_available_backup_row`, against a registration holding the + /// row's lock. + async fn waits_for_the_row_lock_and_then_sees_the_registration(remove: bool) { let s = scratch().await; let row = row(ACCOUNT, "Music", "nightly"); let key = row.key(); @@ -3049,7 +3072,24 @@ mod catalog_index { let engine = s.engine.clone(); let deleting = tokio::spawn(async move { let key = key; - engine.start_backup_delete(&key).await + if remove { + engine + .forget_available_backup_row(&key) + .await + .map(|answer| match answer { + extenddb_storage::backup_catalog::ForgetRow::Removed => { + DeleteStart::Deleting + } + extenddb_storage::backup_catalog::ForgetRow::InUse(table) => { + DeleteStart::InUse(table) + } + extenddb_storage::backup_catalog::ForgetRow::Status(status) => { + DeleteStart::Status(status) + } + }) + } else { + engine.start_backup_delete(&key).await + } }); // The delete is waiting on the registration's transaction, in its // `SELECT ... FOR UPDATE` (scoped to this database and that statement: @@ -3070,7 +3110,7 @@ mod catalog_index { } assert!( tokio::time::Instant::now() < deadline, - "start_backup_delete did not wait for the row lock" + "the delete step (remove: {remove}) did not wait for the row lock" ); assert!( !deleting.is_finished(), @@ -3090,7 +3130,7 @@ mod catalog_index { .await .expect("the delete finishes once the lock is released") .expect("task") - .expect("start_backup_delete"); + .expect("the delete step"); assert_eq!(answer, DeleteStart::InUse("lk_target".to_owned())); assert_eq!( s.engine @@ -3493,6 +3533,11 @@ mod catalog_index { s.engine.start_backup_delete(&key).await.unwrap(), DeleteStart::InUse("del_target".to_owned()) ); + assert_eq!( + s.engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::InUse("del_target".to_owned()), + "the outright removal keeps the in-use rule" + ); assert_eq!( s.engine .read_backup_row(&arn_of(&row)) @@ -3519,6 +3564,57 @@ mod catalog_index { DeleteStart::Status(BackupStatus::Deleting), "a retry finds the row DELETING" ); + assert_eq!( + s.engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Status(BackupStatus::Deleting), + "only an AVAILABLE row is removed outright" + ); + // An AVAILABLE row goes outright, read and listed nowhere, with no + // tombstone; a second removal finds no row. + let mut adopted = catalog_index_row(ACCOUNT, "Music", "adopted"); + adopted.status = BackupStatus::Available; + assert!(s.engine.insert_backup_row(&adopted).await.expect("insert")); + assert_eq!( + s.engine + .forget_available_backup_row(&adopted.key()) + .await + .unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Removed + ); + assert!( + s.engine + .read_backup_row(&arn_of(&adopted)) + .await + .unwrap() + .is_none() + ); + for status in [None, Some(BackupStatus::Available)] { + let listed: Vec = s + .engine + .list_backup_rows(status, &adopted.store) + .try_collect() + .await + .unwrap(); + assert!( + !listed.iter().any(|r| r.backup_arn == adopted.backup_arn), + "{status:?}" + ); + } + let by_account: Vec = s + .engine + .list_account_backup_rows(&adopted.account_id, None, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + !by_account + .iter() + .any(|r| r.backup_arn == adopted.backup_arn) + ); + assert!(matches!( + s.engine.forget_available_backup_row(&adopted.key()).await, + Err(StorageError::Validation(m)) if m.contains("Backup not found") + )); assert!( !s.engine.remove_backup_row(&key).await.unwrap(), "DELETING is not a tombstone" @@ -3946,6 +4042,150 @@ mod catalog_index { s.cleanup().await; } + /// `insert_backup_row`: the row lands as given, unowned, with + /// `completed_at` and `heartbeat_at` on the database's clock; a second + /// insert, an insert over a CREATING row a claim holds, and an insert of + /// the id under another account (the unique `backup_id`) answer `false` + /// and change nothing; a claim over the inserted row is refused; the row + /// is read and listed like a created one. + #[tokio::test] + async fn insert_backup_row_inserts_an_unowned_row_only_when_absent() { + if base_conn().is_none() { + eprintln!( + "SKIP insert_backup_row_inserts_an_unowned_row_only_when_absent: no PostgreSQL" + ); + return; + } + let s = scratch().await; + let mut adopted = catalog_index_row(ACCOUNT, "Music", "adopted"); + adopted.status = BackupStatus::Available; + adopted.size_bytes = 1234; + adopted.item_count = 5; + // Ignored: the insert sets these three itself. + adopted.owner_generation = 99; + adopted.completed_at = None; + adopted.heartbeat_at = 0.0; + // A year before the id's instant: the two clock columns must be + // the database's clock at the insert, not the row's `created_at`. + adopted.created_at -= 365.0 * 86_400.0; + assert!(s.engine.insert_backup_row(&adopted).await.expect("insert")); + let stored = s + .engine + .read_backup_row(&arn_of(&adopted)) + .await + .expect("read") + .expect("the row exists"); + assert_eq!(stored.owner_generation, UNOWNED_GENERATION); + let completed_at = stored.completed_at.expect("completed_at is set"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("the clock is after the epoch") + .as_secs_f64(); + for at in [completed_at, stored.heartbeat_at] { + assert!( + (at - now).abs() < 60.0, + "{at} is not the insert's instant {now}" + ); + } + assert_eq!( + BackupRow { + owner_generation: 99, + completed_at: None, + heartbeat_at: 0.0, + ..stored.clone() + }, + adopted, + "every other column is the row as given" + ); + + // A second insert of the key, with other values, writes nothing. + let mut again = adopted.clone(); + again.backup_name = "again".to_owned(); + again.size_bytes = 1; + again.status = BackupStatus::Failed; + assert!(!s.engine.insert_backup_row(&again).await.expect("insert")); + // The id under another account: the unique `backup_id`. + let mut elsewhere = adopted.clone(); + elsewhere.account_id = OTHER_ACCOUNT.to_owned(); + elsewhere.backup_arn = + extenddb_storage::util::backup_arn(REGION, OTHER_ACCOUNT, "Music", &adopted.backup_id); + assert!( + !s.engine + .insert_backup_row(&elsewhere) + .await + .expect("insert") + ); + assert!( + s.engine + .read_backup_row(&arn_of(&elsewhere)) + .await + .unwrap() + .is_none() + ); + assert_eq!( + s.engine.read_backup_row(&arn_of(&adopted)).await.unwrap(), + Some(stored.clone()), + "the refused inserts changed nothing" + ); + let claimed = s.engine.claim_backup(&adopted).await.err(); + assert!( + matches!(claimed, Some(StorageError::AlreadyExists(_))), + "{claimed:?}" + ); + + // Over a CREATING row a live claim holds. + let creating = row(ACCOUNT, "Music", "job"); + let claim = s.engine.claim_backup(&creating).await.expect("claim"); + let mut over = creating.clone(); + over.status = BackupStatus::Available; + assert!(!s.engine.insert_backup_row(&over).await.expect("insert")); + let job_row = s + .engine + .read_backup_row(&arn_of(&creating)) + .await + .unwrap() + .unwrap(); + assert_eq!(job_row.status, BackupStatus::Creating); + assert_eq!(job_row.owner_generation, claim.generation()); + + // Read and listed like a created row. + let arns = |rows: Vec| -> Vec { + rows.into_iter().map(|r| r.backup_arn).collect() + }; + let available: Vec = s + .engine + .list_backup_rows(Some(BackupStatus::Available), &store()) + .try_collect() + .await + .unwrap(); + assert_eq!(available, std::slice::from_ref(&stored)); + let creating_rows: Vec = s + .engine + .list_backup_rows(Some(BackupStatus::Creating), &store()) + .try_collect() + .await + .unwrap(); + assert_eq!( + arns(creating_rows), + std::slice::from_ref(&creating.backup_arn) + ); + for table in [None, Some("Music")] { + let listed: Vec = s + .engine + .list_account_backup_rows(ACCOUNT, table, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + listed.contains(&stored), + "{table:?}: {:?}", + arns(listed.clone()) + ); + } + drop(claim); + s.cleanup().await; + } + /// The maintenance claim: taken once, refused to a second holder naming /// the first, heartbeated, released; a holder whose session ended and /// whose heartbeat is older than the stale window is dead, so its claim @@ -5150,9 +5390,17 @@ mod store_identity { StoreKind::Filesystem, &label, &s.engine, - "postgres" + "postgres", + false + ), + establish( + &*store_b, + StoreKind::Filesystem, + &label, + &other, + "postgres", + false ), - establish(&*store_b, StoreKind::Filesystem, &label, &other, "postgres"), ); let (a, b) = (a.expect("instance a"), b.expect("instance b")); assert_eq!(a, b, "both instances use one id"); @@ -5174,7 +5422,7 @@ mod store_identity { .unwrap(), std::slice::from_ref(&a.store_id) ); - recheck(&*store_a, &a, engine, "postgres") + recheck(&*store_a, &a, engine, "postgres", false) .await .expect("still healthy"); } @@ -5214,6 +5462,7 @@ mod store_identity { label, &s.engine, "postgres", + false, ) .await .expect("instance a"); @@ -5236,15 +5485,22 @@ mod store_identity { ) .await .expect("available"); - recheck(&*store_a, &a, &s.engine, "postgres") + recheck(&*store_a, &a, &s.engine, "postgres", false) .await .expect("a is healthy"); // The second instance: its root has no file and the catalog has rows // for the label. - let err = establish(&*store_b, StoreKind::Filesystem, label, &other, "postgres") - .await - .unwrap_err(); + let err = establish( + &*store_b, + StoreKind::Filesystem, + label, + &other, + "postgres", + false, + ) + .await + .unwrap_err(); assert_eq!( err, StoreProblem::FileMissingWithRows(vec![a.store_id.clone()]) @@ -5261,9 +5517,16 @@ mod store_identity { .put(STORE_IDENTITY_KEY, byte_stream_from(file.to_bytes())) .await .unwrap(); - let err = establish(&*store_b, StoreKind::Filesystem, label, &other, "postgres") - .await - .unwrap_err(); + let err = establish( + &*store_b, + StoreKind::Filesystem, + label, + &other, + "postgres", + false, + ) + .await + .unwrap_err(); assert_eq!( err, StoreProblem::ManifestMissing { @@ -5271,16 +5534,23 @@ mod store_identity { } ); // The first instance is unaffected. - recheck(&*store_a, &a, &s.engine, "postgres") + recheck(&*store_a, &a, &s.engine, "postgres", false) .await .expect("a stays healthy"); // A shared root (the second instance pointed at the first's // directory) passes. let shared: Arc = Arc::new(FilesystemStore::open(root_a.path()).await.unwrap()); - let b = establish(&*shared, StoreKind::Filesystem, label, &other, "postgres") - .await - .expect("a shared root is healthy"); + let b = establish( + &*shared, + StoreKind::Filesystem, + label, + &other, + "postgres", + false, + ) + .await + .expect("a shared root is healthy"); assert_eq!(b, a); s.cleanup().await; } @@ -5305,6 +5575,7 @@ mod store_identity { catalog_id: "another-catalog".to_owned(), backend: "sqlite".to_owned(), created_at_ms: 0, + ephemeral_catalog: false, }; store .put(STORE_IDENTITY_KEY, byte_stream_from(foreign.to_bytes())) @@ -5316,6 +5587,7 @@ mod store_identity { &label, &s.engine, "postgres", + false, ) .await .unwrap_err(); diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs index cd5c55792..699023000 100644 --- a/crates/storage-sqlite/src/backup.rs +++ b/crates/storage-sqlite/src/backup.rs @@ -4750,6 +4750,11 @@ mod restore_tests { engine.start_backup_delete(&key).await.unwrap(), DeleteStart::InUse("del_target".to_owned()) ); + assert_eq!( + engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::InUse("del_target".to_owned()), + "the outright removal keeps the in-use rule" + ); assert_eq!( engine .read_backup_row(&arn_of(&row)) @@ -4776,6 +4781,55 @@ mod restore_tests { DeleteStart::Status(BackupStatus::Deleting), "a retry finds the row DELETING" ); + assert_eq!( + engine.forget_available_backup_row(&key).await.unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Status(BackupStatus::Deleting), + "only an AVAILABLE row is removed outright" + ); + // An AVAILABLE row goes outright, read and listed nowhere, with no + // tombstone; a second removal finds no row. + let mut adopted = self::row(ACCOUNT, "Music", "adopted"); + adopted.status = BackupStatus::Available; + assert!(engine.insert_backup_row(&adopted).await.expect("insert")); + assert_eq!( + engine + .forget_available_backup_row(&adopted.key()) + .await + .unwrap(), + extenddb_storage::backup_catalog::ForgetRow::Removed + ); + assert!( + engine + .read_backup_row(&arn_of(&adopted)) + .await + .unwrap() + .is_none() + ); + for status in [None, Some(BackupStatus::Available)] { + let listed: Vec = engine + .list_backup_rows(status, &adopted.store) + .try_collect() + .await + .unwrap(); + assert!( + !listed.iter().any(|r| r.backup_arn == adopted.backup_arn), + "{status:?}" + ); + } + let by_account: Vec = engine + .list_account_backup_rows(&adopted.account_id, None, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + !by_account + .iter() + .any(|r| r.backup_arn == adopted.backup_arn) + ); + assert!(matches!( + engine.forget_available_backup_row(&adopted.key()).await, + Err(StorageError::Validation(m)) if m.contains("Backup not found") + )); assert!( !engine.remove_backup_row(&key).await.unwrap(), "DELETING is not a tombstone" @@ -5206,6 +5260,144 @@ mod restore_tests { ); } + /// `insert_backup_row`: the row lands as given, unowned, with + /// `completed_at` and `heartbeat_at` on the database's clock; a + /// second insert, an insert over a CREATING row a claim holds, and an + /// insert of the id under another account (the unique `backup_id`) + /// answer `false` and change nothing; a claim over the inserted row + /// is refused; the row is read and listed like a created one. + #[tokio::test] + async fn insert_backup_row_inserts_an_unowned_row_only_when_absent() { + let engine = engine().await; + let mut adopted = row(ACCOUNT, "Music", "adopted"); + adopted.status = BackupStatus::Available; + adopted.size_bytes = 1234; + adopted.item_count = 5; + // Ignored: the insert sets these three itself. + adopted.owner_generation = 99; + adopted.completed_at = None; + adopted.heartbeat_at = 0.0; + // A year before the id's instant: the two clock columns must be + // the database's clock at the insert, not the row's `created_at`. + adopted.created_at -= 365.0 * 86_400.0; + assert!(engine.insert_backup_row(&adopted).await.expect("insert")); + let stored = engine + .read_backup_row(&arn_of(&adopted)) + .await + .expect("read") + .expect("the row exists"); + assert_eq!(stored.owner_generation, UNOWNED_GENERATION); + let completed_at = stored.completed_at.expect("completed_at is set"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("the clock is after the epoch") + .as_secs_f64(); + for at in [completed_at, stored.heartbeat_at] { + assert!( + (at - now).abs() < 60.0, + "{at} is not the insert's instant {now}" + ); + } + assert!( + (stored.created_at - adopted.created_at).abs() < 1e-3, + "{} is the row's created_at {}", + stored.created_at, + adopted.created_at + ); + assert_eq!( + BackupRow { + owner_generation: 99, + completed_at: None, + heartbeat_at: 0.0, + created_at: adopted.created_at, + ..stored.clone() + }, + adopted, + "every other column is the row as given" + ); + + // A second insert of the key, with other values, writes nothing. + let mut again = adopted.clone(); + again.backup_name = "again".to_owned(); + again.size_bytes = 1; + again.status = BackupStatus::Failed; + assert!(!engine.insert_backup_row(&again).await.expect("insert")); + // The id under another account: the unique `backup_id`. + let mut elsewhere = adopted.clone(); + elsewhere.account_id = OTHER_ACCOUNT.to_owned(); + elsewhere.backup_arn = extenddb_storage::util::backup_arn( + REGION, + OTHER_ACCOUNT, + "Music", + &adopted.backup_id, + ); + assert!(!engine.insert_backup_row(&elsewhere).await.expect("insert")); + assert!( + engine + .read_backup_row(&arn_of(&elsewhere)) + .await + .unwrap() + .is_none() + ); + assert_eq!( + engine.read_backup_row(&arn_of(&adopted)).await.unwrap(), + Some(stored.clone()), + "the refused inserts changed nothing" + ); + let claimed = engine.claim_backup(&adopted).await.err(); + assert!( + matches!(claimed, Some(StorageError::AlreadyExists(_))), + "{claimed:?}" + ); + + // Over a CREATING row a live claim holds. + let creating = row(ACCOUNT, "Music", "job"); + let claim = engine.claim_backup(&creating).await.expect("claim"); + let mut over = creating.clone(); + over.status = BackupStatus::Available; + assert!(!engine.insert_backup_row(&over).await.expect("insert")); + let job_row = engine + .read_backup_row(&arn_of(&creating)) + .await + .unwrap() + .unwrap(); + assert_eq!(job_row.status, BackupStatus::Creating); + assert_eq!(job_row.owner_generation, claim.generation()); + + // Read and listed like a created row. + let arns = |rows: Vec| -> Vec { + rows.into_iter().map(|r| r.backup_arn).collect() + }; + let available: Vec = engine + .list_backup_rows(Some(BackupStatus::Available), &store()) + .try_collect() + .await + .unwrap(); + assert_eq!(available, std::slice::from_ref(&stored)); + let creating_rows: Vec = engine + .list_backup_rows(Some(BackupStatus::Creating), &store()) + .try_collect() + .await + .unwrap(); + assert_eq!( + arns(creating_rows), + std::slice::from_ref(&creating.backup_arn) + ); + for table in [None, Some("Music")] { + let listed: Vec = engine + .list_account_backup_rows(ACCOUNT, table, &TimeBounds::default()) + .try_collect() + .await + .unwrap(); + assert!( + listed.contains(&stored), + "{table:?}: {:?}", + arns(listed.clone()) + ); + } + drop(claim); + } + /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat /// is older than the window is removed outright; one with a live /// owner, one that is not CREATING, and one that is gone are left, @@ -5723,6 +5915,47 @@ mod restore_tests { assert!(!check().await.expect("check")); } + /// The maintenance claim's beat queues behind the engine write lock, + /// and its bound starts once the beat holds it: a writer of this + /// process holding the lock three bounds long delays the beat, which + /// then confirms the claim; the claim says its beat parks before its + /// bound. A beat on a claim taken over answers `Lost` within the + /// bound. + #[tokio::test] + async fn the_maintenance_claims_beat_is_bounded_from_the_write_lock() { + let engine = engine().await; + let stale = Duration::from_secs(300); + let claim = engine + .maintenance_claim("purge on host-a", stale) + .await + .expect("claim"); + assert!(claim.heartbeat_parks_before_its_bound()); + let bound = Duration::from_millis(200); + let writer = engine.write_lock.lock().await; + let started = std::time::Instant::now(); + let (beat, ()) = tokio::join!(claim.heartbeat_within(bound), async move { + tokio::time::sleep(3 * bound).await; + drop(writer); + }); + assert!( + matches!(beat, Some(Ok(Heartbeat::Confirmed))), + "the beat parked on the lock confirms once it holds it: {beat:?}" + ); + assert!(started.elapsed() >= 3 * bound, "{:?}", started.elapsed()); + + sqlx::query("UPDATE backup_maintenance SET owner_generation = owner_generation + 1") + .execute(&engine.pool) + .await + .expect("take over"); + assert!( + matches!( + claim.heartbeat_within(bound).await, + Some(Ok(Heartbeat::Lost)) + ), + "a claim taken over is lost" + ); + } + /// An engine that holds the serve lock (a command that opened the /// file as `serve` does: `backup purge --confirm`) takes over a /// maintenance claim whose heartbeat is young: under the lock the diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs index 6bde52732..7e492387f 100644 --- a/crates/storage-sqlite/src/backup_catalog.rs +++ b/crates/storage-sqlite/src/backup_catalog.rs @@ -51,10 +51,10 @@ use extenddb_core::backup_format::BackupArn; use extenddb_core::types::{CreateTableInput, RestoreSummary}; use extenddb_storage::JobPermit; use extenddb_storage::backup_catalog::{ - BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, JobClaim, + JobRow, JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, - TimeBounds, WriteOutcome, new_generation, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -693,6 +693,56 @@ impl SqliteEngine { .await .map_err(db_err) } + + /// DeleteBackup's first step, and with `remove` the outright removal of + /// `forget_available_backup_row`, as one write transaction under the + /// write lock: read the status, run the in-use rule, then write DELETING + /// or, with `remove`, delete the row (`Deleting` then means removed). A + /// restore's registration runs its in-use check the same way, so it + /// orders wholly before or after either. One body, so the two methods + /// share the lock and the rule. + async fn start_backup_delete_tx( + &self, + key: &BackupKey, + remove: bool, + ) -> Result { + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + let status: Option = sqlx::query_scalar( + "SELECT status FROM backup_index WHERE account_id = ? AND backup_id = ?", + ) + .bind(&key.account_id) + .bind(&key.backup_id) + .fetch_optional(&mut *tx) + .await + .map_err(db_err)?; + let status = BackupStatus::parse(&status.ok_or_else(|| not_found(&key.backup_arn))?)?; + if status != BackupStatus::Available { + return Ok(DeleteStart::Status(status)); + } + if let Some(table) = creating_restore_target(&mut *tx, &key.backup_arn).await? { + return Ok(DeleteStart::InUse(table)); + } + let write = if remove { + "DELETE FROM backup_index \ + WHERE account_id = ? AND backup_id = ? AND status = 'AVAILABLE'" + } else { + "UPDATE backup_index SET status = 'DELETING' \ + WHERE account_id = ? AND backup_id = ? AND status = 'AVAILABLE'" + }; + sqlx::query(write) + .bind(&key.account_id) + .bind(&key.backup_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; + tx.commit().await.map_err(db_err)?; + Ok(DeleteStart::Deleting) + } } impl BackupCatalog for SqliteEngine { @@ -733,6 +783,7 @@ impl BackupCatalog for SqliteEngine { .begin_with("BEGIN IMMEDIATE") .await .map_err(db_err)?; + // OR IGNORE: typed values fail no NOT NULL or CHECK, so only a key conflict is ignored. let inserted = sqlx::query(&format!( "INSERT OR IGNORE INTO backup_index (backup_id, account_id, table_name, \ backup_arn, backup_name, status, store_kind, store_root, store_id, size_bytes, \ @@ -775,6 +826,57 @@ impl BackupCatalog for SqliteEngine { }) } + fn insert_backup_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // No slot: an unowned row has no job. A job that holds the id's + // slot and has not inserted its row yet finds this one and is + // refused, as it is by any row with the key. + let source_table = serde_json::to_string(&row.source_table_details_json) + .map_err(|e| StorageError::Internal(format!("source table details: {e}")))?; + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + // `ON CONFLICT DO NOTHING` covers the primary key (`account_id`, + // `backup_id`) and the unique `backup_id`, and no other + // constraint: a NOT NULL or CHECK violation is an error. + let inserted = sqlx::query(&format!( + "INSERT INTO backup_index (backup_id, account_id, table_name, \ + backup_arn, backup_name, status, store_kind, store_root, store_id, size_bytes, \ + item_count, created_at, completed_at, failure_reason, \ + source_table_details_json, owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, {NOW}, ?, ?, ?, {NOW}) \ + ON CONFLICT DO NOTHING" + )) + .bind(&row.backup_id) + .bind(&row.account_id) + .bind(&row.table_name) + .bind(&row.backup_arn) + .bind(&row.backup_name) + .bind(row.status.as_str()) + .bind(row.store.store_kind.as_str()) + .bind(&row.store.store_root) + .bind(&row.store.store_id) + .bind(row.size_bytes) + .bind(row.item_count) + .bind(format_epoch_seconds(row.created_at)) + .bind(&row.failure_reason) + .bind(&source_table) + .bind(generation_column(UNOWNED_GENERATION)) + .execute(&mut *tx) + .await + .map_err(db_err)? + .rows_affected(); + tx.commit().await.map_err(db_err)?; + Ok(inserted == 1) + }) + } + fn list_backup_rows<'a>( &'a self, status: Option, @@ -866,42 +968,19 @@ impl BackupCatalog for SqliteEngine { &'a self, key: &'a BackupKey, ) -> BoxFuture<'a, Result> { + Box::pin(self.start_backup_delete_tx(key, false)) + } + + fn forget_available_backup_row<'a>( + &'a self, + key: &'a BackupKey, + ) -> BoxFuture<'a, Result> { Box::pin(async move { - // Under the write lock, in one write transaction: a restore's - // registration runs its in-use check the same way, so the two - // order wholly one before the other. - let _writer = self.write_lock.lock().await; - let mut tx = self - .pool - .begin_with("BEGIN IMMEDIATE") - .await - .map_err(db_err)?; - let status: Option = sqlx::query_scalar( - "SELECT status FROM backup_index WHERE account_id = ? AND backup_id = ?", - ) - .bind(&key.account_id) - .bind(&key.backup_id) - .fetch_optional(&mut *tx) - .await - .map_err(db_err)?; - let status = BackupStatus::parse(&status.ok_or_else(|| not_found(&key.backup_arn))?)?; - if status != BackupStatus::Available { - return Ok(DeleteStart::Status(status)); - } - if let Some(table) = creating_restore_target(&mut *tx, &key.backup_arn).await? { - return Ok(DeleteStart::InUse(table)); - } - sqlx::query( - "UPDATE backup_index SET status = 'DELETING' \ - WHERE account_id = ? AND backup_id = ? AND status = 'AVAILABLE'", - ) - .bind(&key.account_id) - .bind(&key.backup_id) - .execute(&mut *tx) - .await - .map_err(db_err)?; - tx.commit().await.map_err(db_err)?; - Ok(DeleteStart::Deleting) + Ok(match self.start_backup_delete_tx(key, true).await? { + DeleteStart::Deleting => ForgetRow::Removed, + DeleteStart::InUse(table) => ForgetRow::InUse(table), + DeleteStart::Status(status) => ForgetRow::Status(status), + }) }) } @@ -1457,6 +1536,25 @@ struct SqliteMaintenanceClaim { generation: u64, } +impl SqliteMaintenanceClaim { + /// The heartbeat's `UPDATE`, with the engine write lock held. + async fn beat_under_the_lock(&self) -> Result { + let affected = sqlx::query(&format!( + "UPDATE backup_maintenance SET heartbeat_at = {NOW} WHERE owner_generation = ?" + )) + .bind(generation_column(self.generation)) + .execute(&self.engine.pool) + .await + .map_err(db_err)? + .rows_affected(); + Ok(if affected == 1 { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + } +} + impl MaintenanceClaim for SqliteMaintenanceClaim { fn generation(&self) -> u64 { self.generation @@ -1465,22 +1563,31 @@ impl MaintenanceClaim for SqliteMaintenanceClaim { fn heartbeat(&self) -> BoxFuture<'_, Result> { Box::pin(async move { let _writer = self.engine.write_lock.lock().await; - let affected = sqlx::query(&format!( - "UPDATE backup_maintenance SET heartbeat_at = {NOW} WHERE owner_generation = ?" - )) - .bind(generation_column(self.generation)) - .execute(&self.engine.pool) - .await - .map_err(db_err)? - .rows_affected(); - Ok(if affected == 1 { - Heartbeat::Confirmed - } else { - Heartbeat::Lost - }) + self.beat_under_the_lock().await + }) + } + + /// The bound starts once the beat holds the engine write lock, as the + /// job claim's does: a writer of this process holding the lock past a + /// period delays the beat and does not lose the claim. + fn heartbeat_within( + &self, + bound: Duration, + ) -> BoxFuture<'_, Option>> { + Box::pin(async move { + let _writer = self.engine.write_lock.lock().await; + tokio::time::timeout(bound, self.beat_under_the_lock()) + .await + .ok() }) } + /// The wait for the engine write lock is outside the bound (above): a + /// beat parked there confirms the claim for as long as it waits. + fn heartbeat_parks_before_its_bound(&self) -> bool { + true + } + fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> { Box::pin(async move { let _writer = self.engine.write_lock.lock().await; diff --git a/crates/storage-sqlite/src/config.rs b/crates/storage-sqlite/src/config.rs index 24987988f..f10c05cf2 100644 --- a/crates/storage-sqlite/src/config.rs +++ b/crates/storage-sqlite/src/config.rs @@ -79,6 +79,10 @@ impl StorageConfig for SqliteConfig { crate::serve_lock::database_file(&self.path).ok().flatten() } + fn database_location(&self) -> Result, String> { + crate::serve_lock::database_file(&self.path) + } + fn max_connections(&self) -> u32 { self.pool_size } diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs index 084c8d9c3..7a8a354b6 100644 --- a/crates/storage/src/backup_catalog.rs +++ b/crates/storage/src/backup_catalog.rs @@ -473,6 +473,18 @@ pub enum DeleteStart { Status(BackupStatus), } +/// What [`BackupCatalog::forget_available_backup_row`] did. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ForgetRow { + /// The row was AVAILABLE and is gone. + Removed, + /// A restore target registered from the backup is CREATING; its name. + /// The row is left. + InUse(String), + /// The row was not AVAILABLE; its status. The row is left. + Status(BackupStatus), +} + /// What a heartbeat found. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Heartbeat { @@ -864,7 +876,8 @@ pub trait BackupCatalog: Send + Sync { /// Insert `row` with status CREATING under a new claim, and return the /// claim. The claim precedes the insert or is the insert itself, so no - /// row ever exists unowned. + /// CREATING row ever exists unowned (the AVAILABLE rows + /// [`Self::insert_backup_row`] writes are unowned and have no job). /// /// # Errors /// @@ -875,6 +888,56 @@ pub trait BackupCatalog: Send + Sync { row: &'a BackupRow, ) -> BoxFuture<'a, Result, StorageError>>; + /// Insert `row` unowned, if no row with its key exists: `backup sync`'s + /// insert for a prefix it verified. The row is written as given (its + /// status, sync's being AVAILABLE, its store, sizes, `created_at`, + /// `failure_reason` and source table) except for three columns: + /// `owner_generation` is [`UNOWNED_GENERATION`], and `completed_at` and + /// `heartbeat_at` are the database's clock; the row's own values for + /// those are ignored. No claim is taken: an unowned row has no job. + /// Sync passes AVAILABLE; an unowned row of another status has no job to + /// move it (on SQLite an unowned CREATING row has no slot, and the next + /// cycle resolves it as dead). + /// + /// Returns `false`, writing nothing, when a row with the key exists, of + /// whatever status or store, with one exception on Cassandra: a stored + /// row equal to this one in every column the insert writes (unowned, the + /// same store, sizes and source table) counts as inserted, and its list + /// and status entries are written, which completes an insert that + /// stopped between the row and its entries. On PostgreSQL, SQLite and + /// MongoDB, which + /// index `backup_id` alone as unique, a row with the backup id under + /// another account also exists for this rule. Cassandra keys its rows + /// by the account and the ARN and indexes no backup id, so there the + /// rule is per ARN spelling: a row for the same id under another region + /// or partition does not count. `backup sync` closes that gap itself: + /// it inserts the row under this deployment's region's spelling of the + /// manifest's ARN, and before the insert it reads the row under that + /// spelling and the manifest's own and skips a backup the manifest's + /// spelling, when it names another region, has a row for, so sync never + /// gives an account two rows of one backup id under this deployment's or + /// the manifest's spelling of the ARN on any backend (on Cassandra a row + /// of another root under a third region's spelling is not read, and is + /// not seen). A row under this deployment's spelling, the one sync + /// inserts, is left to this method, which answers `false` for it or, on + /// Cassandra, completes it under the exception above. + /// + /// # Errors + /// + /// The backend's error for a failed write. On Cassandra an answer lost + /// in transit, and a `NotMatched` answer, is settled by a serial read of + /// the row: this row counts as inserted and another as existing. No row + /// after a lost answer means not written, and the write is sent once + /// more; no row after `NotMatched` answers `false` (a legacy in-catalog + /// backup holds the ARN, which the read does not return). An outcome + /// still unknown after that is an error. A failed batch of the entries + /// is sent once more before the error, and the next insert of the same + /// row writes them. + fn insert_backup_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result>; + /// Every row of `store` with `status`, or of every status for `None`. fn list_backup_rows<'a>( &'a self, @@ -934,6 +997,24 @@ pub trait BackupCatalog: Send + Sync { key: &'a BackupKey, ) -> BoxFuture<'a, Result>; + /// Remove an AVAILABLE row outright, leaving no DELETING or DELETED + /// tombstone, so no deletion duty ever acts on the backup's objects: + /// `backup sync --prune-missing`'s write for a row whose manifest the + /// configured root does not hold, whose bytes may be under another + /// host's root of the same path. Under the row's lock, as + /// [`Self::start_backup_delete`] takes it, the status check and the + /// in-use rule in one step with the removal: a row a CREATING restore + /// target reads is left ([`ForgetRow::InUse`]), and so is a row another + /// writer moved ([`ForgetRow::Status`]). The store is not touched. + /// + /// # Errors + /// + /// `Validation("Backup not found: ")` when there is no such row. + fn forget_available_backup_row<'a>( + &'a self, + key: &'a BackupKey, + ) -> BoxFuture<'a, Result>; + /// Remove a CREATING row whose owner is dead: `heartbeat_at` older than /// `stale_after` on the catalog's clock. `false` when the row is not /// CREATING, is gone, or has a live owner. The maintenance command's @@ -1224,6 +1305,26 @@ pub trait MaintenanceClaim: Send + Sync { /// still this holder's. The caller treats an `Err` as [`Heartbeat::Lost`]. fn heartbeat(&self) -> BoxFuture<'_, Result>; + /// [`Self::heartbeat`] under `bound`: `None` when the beat did not + /// return within it, which the holder treats as a lost claim. The + /// default bounds the whole call; SQLite, whose beat queues behind its + /// own process's engine write lock, measures the bound from the moment + /// the beat holds that lock, as [`JobClaim::heartbeat_within`] does. + fn heartbeat_within( + &self, + bound: Duration, + ) -> BoxFuture<'_, Option>> { + Box::pin(async move { tokio::time::timeout(bound, self.heartbeat()).await.ok() }) + } + + /// Whether [`Self::heartbeat_within`] can wait outside its bound before + /// the bound starts, as [`JobClaim::heartbeat_parks_before_its_bound`] + /// states it: `true` on SQLite only, where a beat parked on the write + /// lock confirms the claim for as long as it waits. + fn heartbeat_parks_before_its_bound(&self) -> bool { + false + } + /// Remove the record, conditioned on the generation; a record another /// holder took over is left to it. Idempotent. fn release(&self) -> BoxFuture<'_, Result<(), StorageError>>; diff --git a/crates/storage/src/config.rs b/crates/storage/src/config.rs index 21ff4914a..1136e4d13 100644 --- a/crates/storage/src/config.rs +++ b/crates/storage/src/config.rs @@ -69,6 +69,19 @@ pub trait StorageConfig: Send + Sync + std::fmt::Debug { None } + /// [`Self::database_file`] with the reason a configured location names + /// no usable file: `Ok(None)` for a server database and for an + /// in-memory one, `Err` for a location the backend cannot parse (which + /// [`Self::database_file`] also answers `None` for). The default is + /// `Ok` of [`Self::database_file`]. + /// + /// # Errors + /// + /// The backend's reason the location is not usable. + fn database_location(&self) -> Result, String> { + Ok(self.database_file()) + } + /// Clone this config into a boxed trait object. fn clone_box(&self) -> Box; diff --git a/devtools/backup-kill-points.py b/devtools/backup-kill-points.py index 3480201c8..1f1a2f6a8 100755 --- a/devtools/backup-kill-points.py +++ b/devtools/backup-kill-points.py @@ -292,10 +292,12 @@ def main() -> None: # format and no backup taken on a Cassandra server restores through the # store-backed job. A Cassandra kill point therefore takes its backup on a # SQLite source server (`--source-bin`, a SQLite build), copies the prefix -# under the Cassandra target's root and seeds the target's `backup_index` -# row as `devtools/backup-restore-matrix` does (and as `backup sync -# --adopt-store` will), then restores on the paused Cassandra server. The -# matrix script is imported for `copy_prefix`, `prefix_of` and `seed_row`. +# under the Cassandra target's root and adopts it with `extenddb backup +# sync` beside the target's server, as `devtools/backup-restore-matrix` +# does (the sync takes the maintenance claim and releases it before it +# exits, so the restore request that follows is not refused), then restores +# on the paused Cassandra server. The matrix script is imported for +# `copy_prefix` and `prefix_of`. # # `--second-reach` kills at the point's second reach instead of its first: # the first pause is released (the release file written, then removed once @@ -812,7 +814,7 @@ def _tamper_a_data_file(dep: Deployment, arn: str) -> str: def _matrix_module(): """`devtools/backup-restore-matrix` imported as a module, for the prefix - copy and the `backup_index` seed a Cassandra target needs.""" + copy a Cassandra target needs.""" import importlib.machinery import importlib.util @@ -831,27 +833,24 @@ def _matrix_module(): def _adopt_backup(source: Deployment, target: Deployment, arn: str) -> None: - """Copy the source's backup prefix under the target's root and seed the - target's `backup_index` row, the matrix's stand-in for `backup sync - --adopt-store`.""" - import json - import types - + """Copy the source's backup prefix under the target's root and adopt it + with `extenddb backup sync` beside the target's server, as the matrix's + `adopt_backup` does: the sync must exit 0 and name the one backup + adopted. Its output is kept in `sync.log`.""" mx = _matrix_module() src_prefix = mx.prefix_of(source.root, arn) dst_prefix = mx.prefix_of(target.root, arn) mx.copy_prefix(src_prefix, dst_prefix, target.root) - with open(os.path.join(dst_prefix, "extenddb-manifest.json")) as f: - manifest = json.load(f) - - def store_id() -> str: - with open(os.path.join(target.root, "extenddb-store.json")) as f: - return json.load(f)["store_id"] - - adapter = types.SimpleNamespace(backend=target.backend, root=target.root, work=target.work, - tag=target.tag, store_id=store_id) - mx.seed_row(adapter, arn, manifest, manifest["backup_size_bytes"], manifest["source_table"]["item_count"]) - _say(f"backup {arn} adopted by the {target.backend} target (prefix copied, backup_index row seeded)") + r = subprocess.run( + [target.bin, "backup", "sync", "--config", target.config], + cwd=target.work, capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=600, + ) + with open(os.path.join(target.work, "sync.log"), "a") as log: + log.write(f"$ extenddb backup sync: exit {r.returncode}\n{r.stdout}{r.stderr}\n") + if r.returncode != 0 or f" {arn}" not in r.stdout.splitlines() or "1 backup(s) adopted" not in r.stdout: + raise SystemExit(f"backup sync on the {target.backend} target did not adopt {arn} " + f"(exit {r.returncode}): {(r.stdout + r.stderr)[-3000:]}") + _say(f"backup {arn} adopted by the {target.backend} target (prefix copied, backup sync)") def _run_restore_point(backend: str, point: str, binary: str, items: int, work: str, diff --git a/devtools/backup-restore-matrix b/devtools/backup-restore-matrix index 589c52479..c4ff4ef2a 100755 --- a/devtools/backup-restore-matrix +++ b/devtools/backup-restore-matrix @@ -5,14 +5,17 @@ A cell proves that a backup taken on the source backend restores, through the store, into the target backend: the restored table has the source's items, key schema, secondary indexes, throughput, SSE and (where the target supports them) vector indexes, and its class is STANDARD. The target's catalog has no -`backup_index` row for the source's backup until `backup sync --adopt-store` -runs (PR 10); the cell stands in for that command by copying the prefix under -a root the target owns and seeding the row at the storage level, so it proves -the restore and not the adoption. `matrix-nightly.yml` runs the twelve plain -cells and the duplicate-key and vector cells through this committed script. +`backup_index` row for the source's backup until `extenddb backup sync` on +the target adopts the prefix the cell copies under a root the target owns, +so a cell proves the adoption and the restore. With --adopt-store the cell +copies nothing: the target's store path is the source's root, and `backup +sync --adopt-store` makes the target's catalog its owner, after which the +source's server finds the store owned by another catalog. +`matrix-nightly.yml` runs the twelve plain cells, the duplicate-key and +vector cells and two --adopt-store cells through this committed script. backup-restore-matrix --source --target - [--items N] [--vector] [--duplicate-key] [--work DIR] + [--items N] [--vector] [--duplicate-key] [--adopt-store] [--work DIR] [--bin-sqlite PATH] [--bin-postgres PATH] [--bin-mongodb PATH] [--bin-cassandra PATH] Starts a source server from the backend's binary with a filesystem store at @@ -20,15 +23,17 @@ its own root, creates a composite-key table with a GSI and an LSI (and, with --vector, a vector index), writes --items items carrying every attribute type, takes a backup and waits for AVAILABLE. Starts the target server with its own empty root (it writes its own extenddb-store.json), copies the -source's backup prefix under the target root, seeds the target's -backup_index row at the storage level with the target's store id (what -`backup sync --adopt-store` will do in PR 10; the cell proves the restore, -not the adoption), restores, waits for ACTIVE, compares the item set and the +source's backup prefix under the target root and runs `extenddb backup sync` +on the target (beside its server, or on SQLite with the server stopped, since +there the command takes the serve lock), which must adopt the backup, then +restores, waits for ACTIVE, compares the item set and the description (key schema, indexes, throughput, SSE; the table class STANDARD), queries each GSI, and tears both servers down by PID and drops their databases. With --duplicate-key the copied prefix is replaced by a hand-built unordered backup whose two data files share one key, and the cell passes -when the restore fails (the target gone, no table) instead. +when the restore fails (the target gone, no table) instead. --adopt-store +takes neither --duplicate-key nor a MongoDB source with a MongoDB target +(the two catalogs would share the backend's fixed names). The two servers connect to PostgreSQL at 127.0.0.1:55432 (postgres/devpass), MongoDB at 127.0.0.1:27117 and Cassandra at 127.0.0.1:9042, the coordinates @@ -104,16 +109,6 @@ def check(cond: bool, msg: str) -> None: raise CellFailed(msg) -def sqlite_module(): - """pysqlite3 when installed (an old sqlite3 library bundled with the - interpreter cannot read the catalog), else the standard module.""" - try: - import pysqlite3 as sqlite3 # type: ignore - except ImportError: - import sqlite3 # type: ignore - return sqlite3 - - def free_port() -> int: import socket @@ -589,135 +584,55 @@ def b64(value): raise TypeError(type(value)) -# --- the seed ------------------------------------------------------------------ +# --- the adoption -------------------------------------------------------------- -def seed_row(target: Deployment, arn: str, manifest: dict, size_bytes: int, item_count: int) -> None: - """Insert the target's `backup_index` row for the copied prefix, as - `backup sync` will (PR 10): AVAILABLE, the target's store identity, the - manifest's source table, `created_at` from the backup id.""" - backup_id = arn.rsplit("/", 1)[1] - table = arn.split(":table/")[1].split("/")[0] - created_ms = int(backup_id.split("-")[0]) - # `created_at` as the job writes it (the id's millisecond, six fraction - # digits on SQLite); `completed_at` and `heartbeat_at` are the seed's own - # instant, distinct from `created_at` as the job's AVAILABLE write and - # its last beat are. - created_iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(created_ms / 1000)) + f".{created_ms % 1000:03d}000Z" - now_ms = int(time.time() * 1000) - now_iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(now_ms / 1000)) + f".{now_ms % 1000:03d}Z" - store_root = os.path.realpath(target.root) - source_table = json.dumps(manifest["source_table"], separators=(",", ":")) - if target.backend == "postgres": - catalog = f"{target.tag}_catalog" - psql( - "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " - "store_kind, store_root, store_id, size_bytes, item_count, created_at, completed_at, " - "source_table_details_json, owner_generation, heartbeat_at) VALUES " - f"('{ACCOUNT}', '{backup_id}', '{table}', '{arn}', 'matrix', 'AVAILABLE', 'filesystem', " - f"'{store_root}', '{target.store_id()}', {size_bytes}, {item_count}, " - f"to_timestamp({created_ms} / 1000.0), now(), '{source_table.replace(chr(39), chr(39) * 2)}'::jsonb, 0, now())", - database=catalog, - ) - elif target.backend == "sqlite": - conn = sqlite_module().connect(os.path.join(target.work, "db.sqlite"), timeout=10) - try: - conn.execute( - "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " - "store_kind, store_root, store_id, size_bytes, item_count, created_at, completed_at, " - "source_table_details_json, owner_generation, heartbeat_at) VALUES (?, ?, ?, ?, 'matrix', 'AVAILABLE', " - "'filesystem', ?, ?, ?, ?, ?, ?, ?, 0, ?)", - (ACCOUNT, backup_id, table, arn, store_root, target.store_id(), size_bytes, item_count, - created_iso, now_iso, source_table, now_iso), - ) - conn.commit() - finally: - conn.close() - elif target.backend == "mongodb": - import datetime +def backup_sync(target: Deployment, *options: str) -> str: + """`extenddb backup sync [options]` on the target, its output logged to + `sync.log`: beside the server on PostgreSQL, MongoDB and Cassandra; on + SQLite, where the command takes the serve lock, with the server stopped + and started again after it.""" + if target.backend == "sqlite": + target.stop() + r = subprocess.run( + [target.bin, "backup", "sync", *options, "--config", target.config], + cwd=target.work, capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=600, + ) + with open(os.path.join(target.work, "sync.log"), "a") as log: + log.write(f"$ extenddb backup sync {' '.join(options)}: exit {r.returncode}\n{r.stdout}{r.stderr}\n") + if target.backend == "sqlite": + target.start() + check(r.returncode == 0, + f"backup sync {' '.join(options)} on the target exited {r.returncode}: {(r.stdout + r.stderr)[-3000:]}") + return r.stdout - from bson import Int64 - from pymongo import MongoClient - created = datetime.datetime.fromtimestamp(created_ms / 1000, tz=datetime.timezone.utc) - now = datetime.datetime.fromtimestamp(now_ms / 1000, tz=datetime.timezone.utc) - client = MongoClient(MONGODB_URI) - try: - client["extenddb_catalog"]["backup_index"].insert_one({ - "account_id": ACCOUNT, - "backup_id": backup_id, - "table_name": table, - "backup_arn": arn, - "backup_name": "matrix", - "status": "AVAILABLE", - "store_kind": "filesystem", - "store_root": store_root, - "store_id": target.store_id(), - "size_bytes": Int64(size_bytes), - "item_count": Int64(item_count), - "created_at": created, - "completed_at": now, - "source_table_details_json": manifest["source_table"], - "owner_generation": Int64(0), - "heartbeat_at": now, - "lock_seq": Int64(0), - }) - finally: - client.close() - elif target.backend == "cassandra": - # The catalog-index row (`store_kind` set) in `backups_by_arn`, plus - # its `backup_index_by_status` entry; DescribeTable and the restore - # re-read the authoritative row, so the two list rows are optional. - import datetime - - from cassandra.auth import PlainTextAuthProvider - from cassandra.cluster import Cluster - - st = manifest["source_table"] - created_dt = datetime.datetime.utcfromtimestamp(created_ms / 1000) - tct = st.get("table_creation_date_time") - table_created_dt = ( - datetime.datetime.utcfromtimestamp(tct) if tct else created_dt - ) - now_dt = datetime.datetime.utcnow() - cluster = Cluster( - ["127.0.0.1"], port=9042, - auth_provider=PlainTextAuthProvider("cassandra", "cassandra"), - ) - session = cluster.connect() - try: - insert = session.prepare( - "INSERT INTO extenddb_catalog.backups_by_arn (account_id, backup_arn, backup_id, " - "backup_name, table_id, table_name, table_arn, backup_status, backup_type, " - "backup_size_bytes, item_count, key_schema, attribute_definitions, billing_mode, " - "table_created_at, created_at, store_kind, store_root, store_id, " - "source_table_details_json, owner_generation, heartbeat_at) " - "VALUES (?, ?, ?, 'matrix', ?, ?, ?, 'AVAILABLE', 'USER', ?, ?, ?, ?, ?, ?, ?, " - "'filesystem', ?, ?, ?, 0, ?)" - ) - session.execute( - insert, - ( - ACCOUNT, arn, backup_id, st["table_id"], table, st["table_arn"], - size_bytes, item_count, - json.dumps(st["key_schema"], separators=(",", ":")), - json.dumps(st["attribute_definitions"], separators=(",", ":")), - str(st.get("billing_mode", "PAY_PER_REQUEST")), - table_created_dt, created_dt, store_root, target.store_id(), - source_table, now_dt, - ), - ) - status_entry = session.prepare( - "INSERT INTO extenddb_catalog.backup_index_by_status (store_root, backup_status, " - "backup_arn, account_id, store_kind, store_id) " - "VALUES (?, 'AVAILABLE', ?, ?, 'filesystem', ?)" - ) - session.execute(status_entry, (store_root, arn, ACCOUNT, target.store_id())) - finally: - cluster.shutdown() - else: - raise NotImplementedError(f"seed_row for {target.backend}") - say(f"target: backup_index row seeded for {arn} (store {target.store_id()})") +def store_health(d: Deployment) -> dict: + """The `backup_store` component of the server's /health.""" + return requests.get(f"{d.endpoint}/health", verify=False, timeout=5).json().get("components", {}).get("backup_store", {}) + + +def wait_store(d: Deployment, predicate, what: str, timeout: float = 60) -> dict: + deadline = time.monotonic() + timeout + while True: + component = store_health(d) + if predicate(component): + return component + check(time.monotonic() < deadline, f"the {d.role}'s store is not {what} after {timeout} s: {component}") + time.sleep(0.5) + + +def adopt_backup(target: Deployment, arn: str, adopt_store: bool) -> None: + """Sync the target's catalog with its root and require the one backup + adopted: a plain `backup sync` for a prefix copied under a root the target + owns, `--adopt-store` for the source's root itself.""" + out = backup_sync(target, *(["--adopt-store"] if adopt_store else [])) + if adopt_store: + check("Adopted the root: extenddb-store.json named catalog" in out, f"--adopt-store did not adopt the root: {out[-3000:]}") + check(f" {arn}" in out.splitlines() and "1 backup(s) adopted" in out, + f"backup sync did not adopt {arn}: {out[-3000:]}") + wait_store(target, lambda c: c.get("status") == "healthy", "healthy") + say(f"target: backup sync adopted {arn}" + (" with the root (--adopt-store)" if adopt_store else "")) # --- comparison --------------------------------------------------------------- @@ -786,7 +701,12 @@ def compare_descriptions(source: dict, target: dict) -> None: def run(args) -> int: - cell_tag = f"{args.source}-{args.target}" + ("-vector" if args.vector else "") + ("-dup" if args.duplicate_key else "") + cell_tag = ( + f"{args.source}-{args.target}" + + ("-vector" if args.vector else "") + + ("-dup" if args.duplicate_key else "") + + ("-adopt" if args.adopt_store else "") + ) work = args.work or os.path.join(tempfile.gettempdir(), "backup-restore-matrix", cell_tag) work = os.path.abspath(work) shutil.rmtree(work, ignore_errors=True) @@ -809,20 +729,41 @@ def run(args) -> int: manifest = json.load(open(os.path.join(source_prefix, "extenddb-manifest.json"))) source.stop() + if args.adopt_store: + # No copy: the target's store is the source's root, which names + # the source's catalog until the target adopts it. + target.root = source.root target.init() target.start() target.provision() - check(os.path.isfile(os.path.join(target.root, "extenddb-store.json")), "the target wrote no identity file") - target_prefix = prefix_of(target.root, arn) - copy_prefix(source_prefix, target_prefix, target.root) - if args.duplicate_key: - manifest = write_duplicate_key_prefix(target_prefix, manifest, args.vector) - say("target: the prefix now holds an unordered backup with one key in two files") - seed_row(target, arn, manifest, manifest["backup_size_bytes"], manifest["source_table"]["item_count"]) + if args.adopt_store: + with open(os.path.join(source.root, "extenddb-store.json")) as f: + owner = json.load(f)["catalog_id"] + wait_store(target, lambda c: c.get("reason") == "owned by another catalog", "owned by another catalog") + adopt_backup(target, arn, adopt_store=True) + with open(os.path.join(source.root, "extenddb-store.json")) as f: + check(json.load(f)["catalog_id"] != owner, "the root's file still names the source's catalog") + source.start("serve-after-adoption.log") + component = wait_store(source, lambda c: c.get("reason") == "owned by another catalog", + "owned by another catalog") + check(component.get("status") == "unhealthy", f"the source's store after the adoption: {component}") + say(f"source: after the adoption its store is {component}") + source.stop() + else: + check(os.path.isfile(os.path.join(target.root, "extenddb-store.json")), "the target wrote no identity file") + target_prefix = prefix_of(target.root, arn) + copy_prefix(source_prefix, target_prefix, target.root) + if args.duplicate_key: + manifest = write_duplicate_key_prefix(target_prefix, manifest, args.vector) + say("target: the prefix now holds an unordered backup with one key in two files") + adopt_backup(target, arn, adopt_store=False) c = target.client() described = c.describe_backup(BackupArn=arn)["BackupDescription"] - check(described["BackupDetails"]["BackupStatus"] == "AVAILABLE", "the seeded row does not describe AVAILABLE") + check(described["BackupDetails"]["BackupStatus"] == "AVAILABLE", "the adopted row does not describe AVAILABLE") + check(described["BackupDetails"]["BackupName"] == "matrix", f"the adopted row's name: {described['BackupDetails']}") + check(described["BackupDetails"]["BackupSizeBytes"] == manifest["backup_size_bytes"], + f"the adopted row's size: {described['BackupDetails']} against the manifest's {manifest['backup_size_bytes']}") started = time.monotonic() if args.vector and args.target == "mongodb": @@ -964,10 +905,17 @@ def main() -> None: p.add_argument("--items", type=int, default=1200) p.add_argument("--vector", action="store_true") p.add_argument("--duplicate-key", action="store_true") + p.add_argument("--adopt-store", action="store_true", + help="point the target's store at the source's root and adopt it with `backup sync --adopt-store`") p.add_argument("--work") for b in DEFAULT_BINS: p.add_argument(f"--bin-{b}") - sys.exit(run(p.parse_args())) + args = p.parse_args() + if args.adopt_store and args.duplicate_key: + p.error("--adopt-store copies no prefix, so it takes no --duplicate-key") + if args.adopt_store and args.source == "mongodb" and args.target == "mongodb": + p.error("--adopt-store keeps the source's catalog for its health check; two MongoDB catalogs share fixed names") + sys.exit(run(args)) if __name__ == "__main__": diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index 60cf5b9ce..06ac759b1 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -94,15 +94,15 @@ The invariants and state machines every later PR honours, by lettered part; the (b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, its progress (`file_index` and `items_applied`: per batch on SQLite and Cassandra, `file_index` per file on MongoDB; PostgreSQL records progress in `restore_progress` rows of the data database instead, 5.3 step 5), `failure_reason` and `completed_at` (PR 6's schema; `RestoreJobRow` carries every member). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `RestoreJobs{failed}` (section 9). -(c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10), and the previous owner then finds the store unhealthy and stops acting on it. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. The probe needs an AVAILABLE row, and until one exists the guard has a gap: two instances with host-local roots at one path and a copied `extenddb-store.json` (one id in both files, so the identity check passes on each) are not told apart, the dead-owner duty of one fails a killed instance's complete backup under the other directory (no manifest in its own) and the killed instance's restart deletes that backup's prefix, the only copy (measured on PostgreSQL, PR 8's review; revision item 32). The rule is stated in the admin guide: one shared root from the first backup on, and the identity file is never copied between roots. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. +(c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10; a server on an in-memory database adopts, at startup, a root whose file another in-memory catalog wrote, revision item 34). The previous owner keeps acting on the root until a later check that can read its catalog finds the store unhealthy, at the earliest one reconciler interval after the cycle in progress ends, unless its deletion fence stops it first: each deletion from the store, DeleteBackup's and purge's row change before it, and each CreateBackup before its row and again before its job's first put, re-reads the file and is refused, the store marked unhealthy at once, when the file names another catalog (revision item 34). A write already past the fence's read still lands, so every server of the previous catalog is stopped before an adoption. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. The probe needs an AVAILABLE row, and until one exists the guard has a gap: two instances with host-local roots at one path and a copied `extenddb-store.json` (one id in both files, so the identity check passes on each) are not told apart, the dead-owner duty of one fails a killed instance's complete backup under the other directory (no manifest in its own) and the killed instance's restart deletes that backup's prefix, the only copy (measured on PostgreSQL during PR 8; revision item 32). The rule is stated in the admin guide: one shared root from the first backup on, and the identity file is never copied between roots. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, #820, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. (d) Identities. The ARN encodes partition, region, account, table, and backup id, and the key prefix is derived from account, table and id deterministically by `BackupArn::store_prefix`, under the configured store's prefix, and no row stores it, so restore, sync, purge and the reconciler derive it the same way and `describe_backup(arn)` works from the store alone after a sync. The engine, the writer and sync all parse and format ARNs through one implementation, `BackupArn` in `extenddb_core::backup_format` (#394); the engine keeps #384's region handling: it resolves a backup under the ARN as written and under the ARN re-spelled in its own region, and refuses a partition other than its own. `backup_creation_date_time` and `created_at` are the id's instant, so the ARN, the manifest, the API's reported time and the ListBackups cursor agree by construction. One row is addressed by `BackupKey` (`account_id`, `backup_id`, `backup_arn`): PostgreSQL, SQLite and MongoDB key `backup_index` by (`account_id`, `backup_id`), Cassandra keys `backups_by_arn` by (`account_id`, `backup_arn`); no method takes a bare backup id, and the engine treats a row whose `account_id`, `table_name` or `backup_arn` (after that re-spelling) differs from the request's as absent. One target is addressed by `RestoreTargetKey` (`account_id`, `table_name`, `table_id`), the table id guarding against a table re-created under the name. -(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING backup row and every RUNNING store-backed restore record at startup (the restart ages every RUNNING restore's heartbeat to the epoch), and at runtime a row or record whose in-process slot no task holds (on SQLite the store-backed restore's claim holds the `restore:/
` slot from before the target exists to the task's end; the legacy restore holds no slot, and a legacy and a store-backed restore into one name exclude each other through the name's `tables` row under the write transaction (`BEGIN IMMEDIATE`: the CREATING-target check, then the unique name)); on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A dead RUNNING store-backed restore record is taken over through `claim_restore_dead(key, stale_after, spool_host)` under the same rules (PostgreSQL's name lock and SQLite's slot refuse a taker outright; MongoDB and Cassandra condition on the stale heartbeat, Cassandra's lightweight transaction also on the heartbeat it read and on `kind = manifest`), which writes the claimant's `spool_host` with the fresh generation and never takes over a legacy record; the successor resumes the restore from the recorded progress (5.3 step 7). A restore record's takeover differs from a backup row's in one write: `claim_restore_dead` sets `heartbeat_at` to the database's clock in the same conditional write as the generation and `spool_host` (PostgreSQL `now()`, SQLite `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, MongoDB `$$NOW`, Cassandra `toTimestamp(now())`), so a second instance's reconciler, whose cycle lands before the successor's first beat (one `heartbeat_interval_secs` after the takeover), finds the record live and is refused. The backup takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale window counts from the owner's last beat; nothing waits on a restore record's old heartbeat after a takeover (the FAILED-target duty ages on `COALESCE(completed_at, heartbeat_at)` and every FAILED write of a resume stamps `completed_at`; the maintenance claim waits on RUNNING, not on the beat). Per backend, the second taker is refused by: PostgreSQL the successor's advisory lock on the target name, then the fresh heartbeat; SQLite the successor's in-process slot (one process serves the file), and the record the startup step had aged is no longer listed as dead once the takeover refreshed its heartbeat; MongoDB the `$expr` staleness term of the takeover's own filter against the refreshed `heartbeat_at`; Cassandra the takeover's serial read, which sees the refreshed `restore_heartbeat_at` and stops before its lightweight transaction. So the two-taker argument of the backup takeover is, for a restore, "a second taker finds the record live"; the belt under it is that every progress write is fenced by the generation (a first taker's next write would match nothing and its job end with no table access) and that a resume's first file is `reapplying`, which covers a partial file on MongoDB and which Cassandra's upserts collapse, so a later change to either mechanism (a progress write without the generation, `reapplying` on a later file) is seen to matter here. The takeover on Cassandra also re-applies the target's `active_restores[table_id] = name` entry on the backup's row when it is absent (one serial read, then `UPDATE ... IF backup_status = 'AVAILABLE'`): the registration writes the target's row and its DDL before the entry, so an owner that died between the two left a resumable record whose backup nothing held in use, and `DeleteBackup` would have succeeded under the resumed load. A backup no longer AVAILABLE keeps no entry; the resumed load fails on the missing prefix and the target is removed. The engine's first confirmed beat of a fresh restore is the return of the one heartbeat the task sends right after `claim_restore` returns and before the reply (5.3 step 4), and of a resumed restore the instant `claim_restore_dead` returned, taken by the reconciler beside the takeover and handed to the job; a takeover's `heartbeat_at` was stamped inside that write, so the instant is at or after the beat it stands for by the write's own duration, and the catalog reads between a takeover and the resumed job's start count against the two-period window rather than widen it. The registration's own `heartbeat_at` stamp is the catalog's clock at some point of the registration, not its end: on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL commits a stale `heartbeat_at`; the job's first beat, sent before the reply, refreshes it. Two readers judge a RUNNING record's owner by the heartbeat alone and need that beat: the reconciler under a maintenance claim (a dead record is failed and its target removed, or failed `target_not_empty`) and `extenddb backup remove-target`; the lock fences only `claim_restore_dead`. A first beat that finds the claim lost, or fails, ends the task before the reply: the record is failed with reason "the claim was lost before the job started" (a write that matches nothing when the claim is another's) and the request is answered `InternalServerError`. What bounds a registration's duration per backend: PostgreSQL holds the `backup_index` row's `FOR UPDATE` lock and the target name's advisory lock through the registration, so `claim_restore_dead` is refused however long it takes; SQLite holds the in-process `restore:/
` slot from before the target exists; MongoDB runs the registration in a transaction bounded by `transactionLifetimeLimitSeconds` (default 60 s); Cassandra's registration writes are lightweight transactions bounded by the coordinator's `write_request_timeout_in_ms` and `cas_contention_timeout_in_ms` (Cassandra defaults 2 s and 1 s), and the manifest registration runs no heartbeat during the target's DDL; a registration whose DDL outlasts the stale window can lose its record to a takeover, and its refusal path then conditions the target's removal on its own generation and removes nothing (the registration answers an error naming the lost claim, and the record's owner finishes the restore). A registration that took two periods does not end the job at its first check, since the two-period rule counts from the first beat's return. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. PostgreSQL's lock rule applies to the resume's takeover alone: the two writes the reconciler makes to a dead RUNNING manifest record without resuming it, FAILED `OWNER_DEAD` (the target removed) or `target_not_empty` (the target left, 5.3 step 7) under a maintenance claim and FAILED `OTHER_STORE`, test the heartbeat alone on every backend, because the operator's command waits on the first and the second is a configuration this design forbids (4(c): instances of one catalog share one root), so neither waits for a stopped owner's session to end: both go through `fail_restore_job`, which conditions on RUNNING alone after `list_dead_restore_jobs` selected the record by heartbeat age, and on PostgreSQL they do not try the owner's advisory lock, so a stopped owner whose session lives keeps its record on the resume path and loses it on these two. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. While the server holds no serving registration (PostgreSQL, MongoDB and Cassandra) it does what the module doc of `crates/server/src/registration.rs` says: it refuses new requests, stops its background workers, the reconciler among them, and starts a new set once it registers again, and the work requests started goes on. Two points that doc leaves to this design: a reconciler cycle running at the stop spawns no resume once its own token has fired (the record is left for the next set's reconciler, which starts from a fresh state), and the backup and restore jobs, resumes included, run on the jobs' token, a child of the server's shutdown token and not of the workers', so a job running when the registration is lost goes on writing the catalog and the store until it ends or the server stops, beside a migration that may already have taken the catalog. Gating the running jobs on the registration (a job stopping while the server is unregistered) is GitHub issue #755. +(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No CREATING row exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); the AVAILABLE rows `backup sync` inserts are unowned (`owner_generation` 0) and have no job, and no duty reads an AVAILABLE row's generation (revision item 34); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING backup row and every RUNNING store-backed restore record at startup (the restart ages every RUNNING restore's heartbeat to the epoch), and at runtime a row or record whose in-process slot no task holds (on SQLite the store-backed restore's claim holds the `restore:/
` slot from before the target exists to the task's end; the legacy restore holds no slot, and a legacy and a store-backed restore into one name exclude each other through the name's `tables` row under the write transaction (`BEGIN IMMEDIATE`: the CREATING-target check, then the unique name)); on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A dead RUNNING store-backed restore record is taken over through `claim_restore_dead(key, stale_after, spool_host)` under the same rules (PostgreSQL's name lock and SQLite's slot refuse a taker outright; MongoDB and Cassandra condition on the stale heartbeat, Cassandra's lightweight transaction also on the heartbeat it read and on `kind = manifest`), which writes the claimant's `spool_host` with the fresh generation and never takes over a legacy record; the successor resumes the restore from the recorded progress (5.3 step 7). A restore record's takeover differs from a backup row's in one write: `claim_restore_dead` sets `heartbeat_at` to the database's clock in the same conditional write as the generation and `spool_host` (PostgreSQL `now()`, SQLite `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, MongoDB `$$NOW`, Cassandra `toTimestamp(now())`), so a second instance's reconciler, whose cycle lands before the successor's first beat (one `heartbeat_interval_secs` after the takeover), finds the record live and is refused. The backup takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale window counts from the owner's last beat; nothing waits on a restore record's old heartbeat after a takeover (the FAILED-target duty ages on `COALESCE(completed_at, heartbeat_at)` and every FAILED write of a resume stamps `completed_at`; the maintenance claim waits on RUNNING, not on the beat). Per backend, the second taker is refused by: PostgreSQL the successor's advisory lock on the target name, then the fresh heartbeat; SQLite the successor's in-process slot (one process serves the file), and the record the startup step had aged is no longer listed as dead once the takeover refreshed its heartbeat; MongoDB the `$expr` staleness term of the takeover's own filter against the refreshed `heartbeat_at`; Cassandra the takeover's serial read, which sees the refreshed `restore_heartbeat_at` and stops before its lightweight transaction. So the two-taker argument of the backup takeover is, for a restore, "a second taker finds the record live"; the belt under it is that every progress write is fenced by the generation (a first taker's next write would match nothing and its job end with no table access) and that a resume's first file is `reapplying`, which covers a partial file on MongoDB and which Cassandra's upserts collapse, so a later change to either mechanism (a progress write without the generation, `reapplying` on a later file) is seen to matter here. The takeover on Cassandra also re-applies the target's `active_restores[table_id] = name` entry on the backup's row when it is absent (one serial read, then `UPDATE ... IF backup_status = 'AVAILABLE'`): the registration writes the target's row and its DDL before the entry, so an owner that died between the two left a resumable record whose backup nothing held in use, and `DeleteBackup` would have succeeded under the resumed load. A backup no longer AVAILABLE keeps no entry; the resumed load fails on the missing prefix and the target is removed. The engine's first confirmed beat of a fresh restore is the return of the one heartbeat the task sends right after `claim_restore` returns and before the reply (5.3 step 4), and of a resumed restore the instant `claim_restore_dead` returned, taken by the reconciler beside the takeover and handed to the job; a takeover's `heartbeat_at` was stamped inside that write, so the instant is at or after the beat it stands for by the write's own duration, and the catalog reads between a takeover and the resumed job's start count against the two-period window rather than widen it. The registration's own `heartbeat_at` stamp is the catalog's clock at some point of the registration, not its end: on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL commits a stale `heartbeat_at`; the job's first beat, sent before the reply, refreshes it. Two readers judge a RUNNING record's owner by the heartbeat alone and need that beat: the reconciler under a maintenance claim (a dead record is failed and its target removed, or failed `target_not_empty`) and `extenddb backup remove-target`; the lock fences only `claim_restore_dead`. A first beat that finds the claim lost, or fails, ends the task before the reply: the record is failed with reason "the claim was lost before the job started" (a write that matches nothing when the claim is another's) and the request is answered `InternalServerError`. What bounds a registration's duration per backend: PostgreSQL holds the `backup_index` row's `FOR UPDATE` lock and the target name's advisory lock through the registration, so `claim_restore_dead` is refused however long it takes; SQLite holds the in-process `restore:/
` slot from before the target exists; MongoDB runs the registration in a transaction bounded by `transactionLifetimeLimitSeconds` (default 60 s); Cassandra's registration writes are lightweight transactions bounded by the coordinator's `write_request_timeout_in_ms` and `cas_contention_timeout_in_ms` (Cassandra defaults 2 s and 1 s), and the manifest registration runs no heartbeat during the target's DDL; a registration whose DDL outlasts the stale window can lose its record to a takeover, and its refusal path then conditions the target's removal on its own generation and removes nothing (the registration answers an error naming the lost claim, and the record's owner finishes the restore). A registration that took two periods does not end the job at its first check, since the two-period rule counts from the first beat's return. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. PostgreSQL's lock rule applies to the resume's takeover alone: the two writes the reconciler makes to a dead RUNNING manifest record without resuming it, FAILED `OWNER_DEAD` (the target removed) or `target_not_empty` (the target left, 5.3 step 7) under a maintenance claim and FAILED `OTHER_STORE`, test the heartbeat alone on every backend, because the operator's command waits on the first and the second is a configuration this design forbids (4(c): instances of one catalog share one root), so neither waits for a stopped owner's session to end: both go through `fail_restore_job`, which conditions on RUNNING alone after `list_dead_restore_jobs` selected the record by heartbeat age, and on PostgreSQL they do not try the owner's advisory lock, so a stopped owner whose session lives keeps its record on the resume path and loses it on these two. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. While the server holds no serving registration (PostgreSQL, MongoDB and Cassandra) it does what the module doc of `crates/server/src/registration.rs` says: it refuses new requests, stops its background workers, the reconciler among them, and starts a new set once it registers again, and the work requests started goes on. Two points that doc leaves to this design: a reconciler cycle running at the stop spawns no resume once its own token has fired (the record is left for the next set's reconciler, which starts from a fresh state), and the backup and restore jobs, resumes included, run on the jobs' token, a child of the server's shutdown token and not of the workers', so a job running when the registration is lost goes on writing the catalog and the store until it ends or the server stops, beside a migration that may already have taken the catalog. Gating the running jobs on the registration (a job stopping while the server is unregistered) is GitHub issue #755. -(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup job checks the claim before its permit and so before its row exists (5.1 step 2; the in-catalog create and restore the same, revision items 29 and 32), the restore job as 5.3 step 4 states, and each refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the command's wait lists the rows again each time it waits, a job whose row lands after the command's claim is seen by the wait, which waits for it; a job that checked before the claim was taken runs to its end, as a job started before the command would. +(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Plain `backup sync` takes it too where it runs beside servers (PostgreSQL, MongoDB, Cassandra; on SQLite the serve lock already excludes purge), without the wait for running jobs: its insert-if-absent orders it against a CreateBackup, and the claim keeps it from adopting a prefix purge has un-rowed and not yet deleted (revision item 34). Every backup job checks the claim before its permit and so before its row exists (5.1 step 2; the in-catalog create and restore the same, revision items 29 and 32), the restore job as 5.3 step 4 states, and each refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the command's wait lists the rows again each time it waits, a job whose row lands after the command's claim is seen by the wait, which waits for it; a job that checked before the claim was taken runs to its end, as a job started before the command would. -(g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and marks it DELETING only with `--prune-missing`; without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size. Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules are the only path from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker), and Describe answers such a row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning: the store a row belongs to is the operator's concern (`backup list` prints every row's store), not the client's. DeleteBackup and Restore check the row's own state first, as for a row of the configured store (a CREATING row is BackupInUseException, a DELETED or FAILED row BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for DeleteBackup a DELETING one (whose deletion only its own store's server can finish), with a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root. +(g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and removes it only with `--prune-missing`, outright, as `--forget-store` removes rows: no DELETING or DELETED tombstone is left and no object is deleted, so on host-local roots a prune on the instance that lacks the bytes leaves them on the other, where a later sync adopts them again (revision item 34); without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size. Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules are the only path from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker), and Describe answers such a row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning: the store a row belongs to is the operator's concern (`backup list` prints every row's store), not the client's. DeleteBackup and Restore check the row's own state first, as for a row of the configured store (a CREATING row is BackupInUseException, a DELETED or FAILED row BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for DeleteBackup a DELETING one (whose deletion only its own store's server can finish), with a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root. (h) Trust boundary of sync: a principal that can write under an account's prefix can forge a backup for that account. Checksums do not help, they are forgeable by the same principal. The deployment guide states that write access to an account's prefix is backup-administrator authority for that account, and gives per-account IAM prefix policies. A server-side MAC over manifests is not in v1; if hostile writers with bucket access are in scope for a deployment, the operator restricts write access rather than relying on ExtendDB to detect forgeries. @@ -136,7 +136,7 @@ The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the 4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the `backup_index` row's lock that DeleteBackup's `start_backup_delete` takes (5.2: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. The task then sends one heartbeat through the claim and takes its return as the instant the heartbeat's two-period rule counts from (`heartbeat.rs`), checks the maintenance claim, and replies: a first beat that finds the claim lost, or fails, runs step 7 with reason "the claim was lost before the job started" and the request returns `InternalServerError` (4(e)); a held maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 29) runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return the `TableDescription` the registration produced, with TableStatus CREATING and its `RestoreSummary`. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle. 5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed` from the spool and `ExtenddbManifest::reader` in place (a filesystem store's file, hashed again as it is read), apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through one `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items)` (section 3) over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB) of data-file line text (the unit the backup side budgets in; a chunk closes at the first item that takes it to the bound, so it holds at most the bound plus one line) through a channel of two chunks, with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise, and `reapplying` true for the first file a resume meets and false after. Each backend writes base rows and index rows together and records progress in the write that applied the items, so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL (a `restore_progress` row per file in the data database, fenced by the base table's primary key and the progress row's own key rather than by the generation, which fences only the record's writes in the catalog) and SQLite (the record's `(file_index, items_applied)` per batch), and by idempotent re-application of the one unrecorded file or batch on MongoDB (the record's `file_index` per file; E11000 suppressed only on the `reapplying` file) and Cassandra (the record's progress per batch under a lightweight transaction; `put_restored_item` is an upsert). The batches: PostgreSQL one data transaction per file, the base rows in multi-row INSERTs closed at 500 rows or 4 MiB of item text (`COPY_BATCH_ITEMS`, `COPY_BATCH_BYTES`), the progress row last in the transaction; SQLite one write transaction per batch, closed at 10,000 items or at the first item that takes the batch's serialized `item_data` text past 32 MiB (`RESTORE_BATCH_ITEMS`, `RESTORE_BATCH_BYTES` in `extenddb_storage::restore_engine`), the record's progress last in the transaction; MongoDB `insert_many(ordered = false)` calls of 1,000 documents (the driver splits a call by the server's message and write-batch bounds; 1,000 is the unit a half-applied file is re-applied in), the record's `file_index` written after the file; Cassandra upserts in batches closed at 10,000 items or 32 MiB of serialized item text, the same two constants, each batch followed by the progress lightweight transaction; an empty file records its progress on every backend: `items_applied = start_at` on PostgreSQL, SQLite and Cassandra, the `file_index` alone on MongoDB, whose record carries no `items_applied` (PR 9a's section). Liveness inside a file: the feeder checks at every chunk boundary that the heartbeat confirmed the claim within two periods and ends the stream as a lost claim otherwise; on MongoDB and Cassandra `bulk_load` reads the record before each write batch (MongoDB one `find_one` on the target's `_id`, `table_id`, `restore_job.status` and `restore_job.owner_generation` before each `insert_many`; Cassandra a lightweight transaction on the record, the heartbeat refreshed `IF` RUNNING under the generation, before each batch of writes, after reading the coordinator's clock as the batch's write timestamp) and ends the file as a lost claim when the record is not RUNNING under its generation, so a writer whose record another instance took over while it stalled lands at most the one batch it was in, and on Cassandra that batch lands `USING TIMESTAMP` the reading taken before the check, below every write and delete the successor made since, so it changes nothing (the PR 9a load paragraph); on Cassandra an unknown outcome of the progress lightweight transaction (a write timeout or failure, a lost connection) is decided by a `LOCAL_SERIAL` re-read of the record: RUNNING under the generation and carrying the values written is applied; RUNNING under the generation with the earlier values is one retry, whose own unknown outcome is re-read the same way, and a second unknown outcome the record does not carry fails the restore; anything else is the lost claim. A stalled owner whose record a successor took over, flipped ACTIVE and whose clients then wrote, lands nothing that outlives the successor's state: on MongoDB the batch's `insert_many` and its progress commit in one transaction whose first write is the claim's fence, so a batch that reaches the server after the takeover is refused whole; on Cassandra the batch's writes carry the timestamp read before its claim check, below every write and delete the successor and its clients made since, so the cells land and lose to the later ones and a key a client deleted stays deleted (the stalled writer's progress write then finds the record not its own and the load stops). The rule on Cassandra needs the ExtendDB hosts' clocks to agree, as the version scheme already does. PostgreSQL's window stays as stated: nothing in its `bulk_load` conditions on the generation; the base table's primary key and the claim's liveness, confirmed before each file, are its fence, the advisory lock holds the record through a stall while the owner's session lives, and the window that remains is a session the database server dropped during a stall longer than the stale window (an `idle_session_timeout`, not a default): the stalled file's transaction can commit after a takeover, and the successor's own load of that file then fails on the duplicate key and the restore is FAILED and its target removed, a lost restore and not wrong data. The restored table has no streams and no TTL, so there is nothing to bypass. 6. (Job.) After all files, and after the count of section 1 matched, the job flips the target to ACTIVE through `finish_restore` (section 3), conditional on the table still being CREATING as a restore target and on the job's `owner_generation` (`NotMatched` otherwise: an error line and no table access); on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE (`Active`, never `Pending`, there). A `finish_restore` that returns an error is a conditional write whose outcome is unknown and is handled as the status writes are: the record is re-read through the claim; DONE under the job's generation is the write landed and the restore complete (one WARN naming the error); RUNNING under it is the write not taken, sent once more at most and its own error re-read the same way; anything else, a second error included, is the DONE write that matched no record, after which the job makes no table access and the reconciler resumes the record once stale. `finish_restore` writes the manifest's `source_table.table_size_bytes` as the restored table's size until the statistics pass rewrites it; `DescribeTable` of a restored table reports the source table's size, not the compressed size of the backup's files. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE (on Cassandra this holds for a resumed restore's re-applied batch because the write path rewrites an unchanged item's index rows instead of deleting them, #424, the PR 9a load paragraph). MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction (`finish_restore` returns `Pending`), and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today. -7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with no further table access and its spool entry left to the sweep, because the claim holder owns the table and may be reading that entry (the entry is keyed by table id, so a same-host successor shares it). DescribeTable then returns ResourceNotFound. Crash mid-restore: once the server's shutdown has begun (the jobs' token is cancelled) the reconciler takes no dead record over and writes no FAILED on the maintenance or other-store arm: the record is left as it is for the next instance's reconciler, since a resume spawned on the way out would stop at its first check and leave the record with a fresh heartbeat, a second stale window for the next instance to wait. Otherwise the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (`claim_restore_dead`, section 4(e)) and resumes from the recorded progress (a dead record met under a maintenance claim is failed through the unclaimed RUNNING-only write before its store is looked at: `target_not_empty` with its target left for `remove-target` (which runs under the same claim) when it recorded no progress and its target holds more items than the backup's row counts, else `OWNER_DEAD` with its target removed; the command's wait is satisfied either way, the record being no longer RUNNING; a record of another store is failed `OTHER_STORE` below, both arms on the heartbeat alone on every backend, 4(e); a resume takes a job permit and is skipped to the next cycle without one; a resume with no recorded progress into a target holding more items than the backup's total (the backup row's `item_count`) is FAILED `target_not_empty` and its target left for `extenddb backup remove-target --account
` (section 4's duty list), which ships with PR 9, the rule for a catalog restored from a dump that names a restore which finished and was written to since; a target holding no more than the total with no recorded progress is a first data file applied in part (on Cassandra, which records a batch's progress after its writes, a crash before the first progress record, the first 10,000 items or 32 MiB, leaves this; MongoDB commits each batch with its progress and leaves nothing unrecorded) and is resumed with the first file re-applied under `reapplying`, every backend's load being idempotent for such a file; the takeover after one instance's fifth of a record (`RESTORE_TAKEOVERS`, counted in the reconciler's state, so per instance and reset by a restart) fails it `owner dead: taken over 5 times by one instance without the restore ending` and removes its target as `OWNER_DEAD` does, so a record whose every resume ends RUNNING again (a progress read or the final count that fails every time, a load that stops with the record RUNNING) is not retaken every stale window for as long as the server runs), reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not (`OTHER_STORE`, the target left to the FAILED-target duty), and resuming nothing while the configured store is unhealthy; on SQLite at once after a restart, by the slot rule; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). The reconciler's sweep of `restore_progress` rows (PostgreSQL) is a positive list: the backend reads the distinct table ids the progress rows name, then the catalog's RUNNING records, and deletes only the rows of ids in the first set that are in neither the second nor the reconciler's own RUNNING list; a target whose first row lands after the first read is never a candidate, and one registered before it is RUNNING at the second read, so a live job's row is never deleted. Shutdown: a restore job stops at its next chunk without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule; it records `RestoreJobs{cancelled}` and no worker metric (section 9), where a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` and leaves the spool entry to the sweep: the reconciler's spool sweep lists the spool directory first and the RUNNING records second, and removes only the entries it listed that no RUNNING record of this host's `spool_host` names, so a restore whose registration commits and whose entry opens between the two reads is never a candidate; an entry its own job removed between the listing and the removal is done, not an error. +7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with no further table access and its spool entry left to the sweep, because the claim holder owns the table and may be reading that entry (the entry is keyed by table id, so a same-host successor shares it). DescribeTable then returns ResourceNotFound. Crash mid-restore: once the server's shutdown has begun (the jobs' token is cancelled) the reconciler takes no dead record over and writes no FAILED on the maintenance or other-store arm: the record is left as it is for the next instance's reconciler, since a resume spawned on the way out would stop at its first check and leave the record with a fresh heartbeat, a second stale window for the next instance to wait. Otherwise the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (`claim_restore_dead`, section 4(e)) and resumes from the recorded progress (a dead record met under a maintenance claim is failed through the unclaimed RUNNING-only write before its store is looked at: `target_not_empty` with its target left for `remove-target` (which runs under the same claim) when it recorded no progress and its target holds more items than the backup's row counts, else `OWNER_DEAD` with its target removed; the command's wait is satisfied either way, the record being no longer RUNNING; a record of another store is failed `OTHER_STORE` below, both arms on the heartbeat alone on every backend, 4(e); a resume takes a job permit and is skipped to the next cycle without one; a resume with no recorded progress into a target holding more items than the backup's total (the backup row's `item_count`) is FAILED `target_not_empty` and its target left for `extenddb backup remove-target --account
` (section 4's duty list), which ships with PR 9, the rule for a catalog restored from a dump that names a restore which finished and was written to since; a target holding no more than the total with no recorded progress is a first data file applied in part (on Cassandra, which records a batch's progress after its writes, a crash before the first progress record, the first 10,000 items or 32 MiB, leaves this; MongoDB commits each batch with its progress and leaves nothing unrecorded) and is resumed with the first file re-applied under `reapplying`, every backend's load being idempotent for such a file; the takeover after one instance's fifth of a record (`RESTORE_TAKEOVERS`, counted in the reconciler's state, so per instance and reset by a restart) fails it `owner dead: taken over 5 times by one instance without the restore ending` and removes its target as `OWNER_DEAD` does, so a record whose every resume ends RUNNING again (a progress read or the final count that fails every time, a load that stops with the record RUNNING) is not retaken every stale window for as long as the server runs), reading the store the job row records if it is the configured store, failing the restore if it is not, with the reason to configure that store in `[backup.store]` and restore again (`OTHER_STORE`, the target left to the FAILED-target duty), and resuming nothing while the configured store is unhealthy; on SQLite at once after a restart, by the slot rule; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). The reconciler's sweep of `restore_progress` rows (PostgreSQL) is a positive list: the backend reads the distinct table ids the progress rows name, then the catalog's RUNNING records, and deletes only the rows of ids in the first set that are in neither the second nor the reconciler's own RUNNING list; a target whose first row lands after the first read is never a candidate, and one registered before it is RUNNING at the second read, so a live job's row is never deleted. Shutdown: a restore job stops at its next chunk without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule; it records `RestoreJobs{cancelled}` and no worker metric (section 9), where a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` and leaves the spool entry to the sweep: the reconciler's spool sweep lists the spool directory first and the RUNNING records second, and removes only the entries it listed that no RUNNING record of this host's `spool_host` names, so a restore whose registration commits and whose entry opens between the two reads is never a candidate; an entry its own job removed between the listing and the removal is done, not an error. Restore overrides. #384 applies every override the operation models as the service applies it (`crates/engine/src/backup/overrides.rs`) and refuses only `VectorIndexOverride`, which the botocore model this project targets does not carry, with a typed ValidationException. The v1 contract keeps exactly that: the new path applies the overrides at step 4 to the manifest's `source_table` before the target is created, through the same `overrides` module, and `VectorIndexOverride` keeps the refusal. The differences and limits docs' override rows stay as they are. @@ -169,7 +169,7 @@ spool_path = "" # restore download area for an S3 store; d On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 29). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there. -Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary or vector indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. +Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary or vector indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. `backup sync`, which is not a job, keeps of the configured root's rows (every status) each row's status, store id and table and the key of each AVAILABLE row of the configured store id, dropping the rest of each row as it is read, and holds the objects of the one prefix the listing is in with their sizes and the prefix of each manifest it listed; every backend's `list_backup_rows` collects its whole result before it yields the first row, so the read of the rows is sync's largest term on a catalog with many rows of the root (tombstones within their retention included): about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of `source_table_details_json`; 610 MiB at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none, and 148 MiB at 50,000 rows whose JSON is `{}`, so nearly all of it is the parsed description), measured on a SQLite debug build on a 16-core x86_64 Linux host under glibc's default arenas, the same peak `backup list` reaches over those rows; the filesystem store builds its listing of the whole root in one `Vec` before it yields the first key (about 72 bytes plus the key per object, the store trait's doc), and the S3 store lists a page of 1,000 keys at a time. Measured on a filesystem store with 100 objects per prefix and 60-byte keys: 45 MiB of peak RSS for an empty root, 66 MiB for 100,000 objects and 98 MiB for 300,000, about 190 bytes per object, nearly all of it the store's listing (revision item 34). The report keeps an entry per prefix for the whole run (it lists every prefix the catalog already has a row for among the skipped ones), so at one object per prefix the per-prefix terms double that: about 350 bytes per prefix, 62 MiB of peak RSS for 50,000 such prefixes and 112 MiB for 200,000 against 45 MiB for an empty root (and about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build of PR 10 with a filesystem root and no rows. On an in-memory SQLite server that cost is paid in the serving process at every start. A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (CreateBackup on PostgreSQL, SQLite and MongoDB, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and the in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. @@ -222,7 +222,7 @@ Which document and which row each PR changes; what each row says is in the ownin | `.github/workflows/matrix-nightly.yml` and `devtools/backup-restore-matrix` (the nightly's script) | PR 9b | | upgrade manual: stop pre-PR-6 servers before `extenddb migrate`; a restore in flight at the stop; the rollback paragraph | PR 6 (and PR 8 for the store-backed backups a restored catalog no longer lists) | | upgrade manual: CreateBackup without `[backup.store]`, `kind = "disabled"`, the SQLite default root, shared store roots and `--adopt-store` | PR 7 (and PR 10 for `--adopt-store`) | -| upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 | +| upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 (`migrate-legacy`: #821, item 34) | | upgrade manual: PR 14's migration refusal, `--force-drop-legacy-backups`, CreateBackup without a store after PR 14 | PR 14 | | upgrade manual: the #618 regression on the off-host path and the way out; the `[backup.store]` paragraph's asynchronous create and the restore of a store-backed backup deferred to PR 9a | PR 8 | | `docs/troubleshooting.md` DeleteTable entry and admin guide `control_plane_delay_seconds` row | #395 (in its diff) | @@ -441,14 +441,14 @@ Kill points (testing summary): every restore kill point, the fenced-out restore ### PR 9b: cross-backend matrix -- PR 9b: cross-backend matrix (shipped in PR 9): the twelve plain cells (PostgreSQL, SQLite and MongoDB sources into PostgreSQL, SQLite, MongoDB and Cassandra targets), the `--duplicate-key` cells into every target (an unordered backup carrying one key in two files fails into every target, into Cassandra by the count alone) and the `--vector` cells (PostgreSQL and SQLite sources restored into PostgreSQL and SQLite with `SearchVectors` answering, refused before any table into MongoDB and Cassandra), run by `devtools/backup-restore-matrix`: a cell takes a backup on the source server, copies its prefix under a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10; the seed stands in for it until then), so the cell proves the restore and not the adoption, then restores and compares the item sets, the descriptions and the index queries; `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs every cell nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do. #384's override tests run against the new path per 5.3; the MongoDB target refuses a vector manifest before any table, an `N` sort-key value beyond 34 significant digits into MongoDB fails the restore with the write path's ValidationException text as its reason and the target gone, and a 38-digit non-key number into MongoDB restores intact (`tests/test_mongodb_backup_restore.py`). Depends on PR 9a. +- PR 9b: cross-backend matrix (shipped in PR 9): the twelve plain cells (PostgreSQL, SQLite and MongoDB sources into PostgreSQL, SQLite, MongoDB and Cassandra targets), the `--duplicate-key` cells into every target (an unordered backup carrying one key in two files fails into every target, into Cassandra by the count alone) and the `--vector` cells (PostgreSQL and SQLite sources restored into PostgreSQL and SQLite with `SearchVectors` answering, refused before any table into MongoDB and Cassandra), run by `devtools/backup-restore-matrix`: a cell takes a backup on the source server, copies its prefix under a root the target catalog owns and gives the target's catalog its `backup_index` row with `extenddb backup sync` (PR 10; until then the cell seeded the row by hand and proved the restore alone, item 34), then restores and compares the item sets, the descriptions and the index queries; `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs every cell nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do. #384's override tests run against the new path per 5.3; the MongoDB target refuses a vector manifest before any table, an `N` sort-key value beyond 34 significant digits into MongoDB fails the restore with the write path's ValidationException text as its reason and the target gone, and a 38-digit non-key number into MongoDB restores intact (`tests/test_mongodb_backup_restore.py`). Depends on PR 9a. ### PR 10: `backup sync`, its options, `remove-target`, `migrate-legacy` -Goal: the catalog-loss recovery and the maintenance commands of section 4. Its line names the interfaces and carries the acceptance: -- PR 10: `extenddb backup sync` per section 4 with store-root scoping, the store id, the maintenance claim with its printed wait bound, tombstone and identity checks, the row check before the manifest read and the insert-if-absent, `--prune-missing`, `--forget-store`, `--forget-backup`, `--move-store` (with `list_backup_rows_by_root` and `rewrite_store`), `--adopt-store`, the SQLite serve lock on every command that writes (`extenddb backup remove-target --account
`, section 4, shipped with PR 9 rather than here, item 33: a crash before the first progress record on MongoDB or Cassandra leaves a `target_not_empty` target that no other command removes) and the dead-owner resolutions such a command runs first, and `extenddb backup migrate-legacy |--all` with its re-read, validation and count checks and its refusals (section 3). Acceptance: drop the catalog rows, run sync, Describe and List and Restore work; a prefix without a manifest, with a bad sidecar, with a mismatched ARN, under `changelog/`, or with a CREATING, DELETING or FAILED row is not adopted; rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes; after a store tree is copied to a new root, `--move-store` re-points its rows and Describe and Restore work against the new root; `--move-store` resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest; `--forget-backup` refuses a CREATING row with a live owner and a backup from which a restore target is CREATING (a RUNNING restore's target, a FAILED restore's target not yet removed, and on MongoDB a DONE restore's target awaiting the backfill), and says so and does nothing for a legacy ARN; a second maintenance command while one holds the claim fails at once; on SQLite every writing command refuses to run beside a server; `migrate-legacy` on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone; with the volume unmounted (an empty root, no id or another id) the store is unhealthy, sync without the option that resolves it refuses, and no row changes; `--adopt-store` of a root another catalog owns makes this catalog the owner, and the other catalog finds the store unhealthy at its next check; with host-local roots on two instances of one catalog no reconciler or sync on either resolves or deletes a row because its own root lacks the bytes; `migrate-legacy` refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason; without `--prune-missing` a row whose manifest is gone is reported and stays AVAILABLE; after `--forget-store` and the store's return, sync adopts its prefixes; a CreateBackup started on another instance during `--move-store` either finishes before the rewrite or is refused; after `--move-store A` stops on a row with one data file missing under the configured root and `--forget-backup` lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied; a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler, which runs those two duties under the claim, has finished the one and failed the other. Depends on PR 6, 8. +Goal: the catalog-loss recovery and the maintenance commands of section 4. As implemented (revision item 34) the section ships as three PRs: #746 plain `backup sync`, `--prune-missing`, `--adopt-store`, the `sqlite-memory` startup sync and `insert_backup_row`, with the acceptance cells those carry; #820 `--move-store`, `--forget-store` and `--forget-backup` with `list_backup_rows_by_root` and `rewrite_store`, and the report of the rows of other roots, with their cells; #821 `migrate-legacy` with its cells; `remove-target` shipped with PR 9 (item 33). Its line names the interfaces and carries the acceptance: +- PR 10: `extenddb backup sync` per section 4 with store-root scoping, the store id, the maintenance claim with its printed wait bound, tombstone and identity checks, the row check before the manifest read and the insert-if-absent, `--prune-missing`, `--forget-store`, `--forget-backup`, `--move-store` (with `list_backup_rows_by_root` and `rewrite_store`), `--adopt-store`, the SQLite serve lock on every command that writes (`extenddb backup remove-target --account
`, section 4, shipped with PR 9 rather than here, item 33: a crash before the first progress record on MongoDB or Cassandra leaves a `target_not_empty` target that no other command removes) and the dead-owner resolutions such a command runs first, and `extenddb backup migrate-legacy |--all` with its re-read, validation and count checks and its refusals (section 3). Acceptance: drop the catalog rows, run sync, Describe and List and Restore work; a prefix without a manifest, with a bad sidecar, with a mismatched ARN, under `changelog/`, or with a CREATING, DELETING or FAILED row is not adopted; rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes; after a store tree is copied to a new root, `--move-store` re-points its rows and Describe and Restore work against the new root; `--move-store` resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest; `--forget-backup` refuses a CREATING row with a live owner and a backup from which a restore target is CREATING (a RUNNING restore's target, a FAILED restore's target not yet removed, and on MongoDB a DONE restore's target awaiting the backfill), and says so and does nothing for a legacy ARN; a second maintenance command while one holds the claim fails at once; on SQLite every writing command refuses to run beside a server; `migrate-legacy` on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone; with the volume unmounted (an empty root, no id or another id) the store is unhealthy, sync without the option that resolves it refuses, and no row changes; `--adopt-store` of a root another catalog owns makes this catalog the owner, and the other catalog finds the store unhealthy at its next check; with host-local roots on two instances of one catalog no reconciler on either deletes a prefix because its own root lacks the bytes, and `--prune-missing` on the instance that lacks them removes the row and deletes nothing: the backup is BackupNotFound on every instance until a sync on the instance that holds the bytes adopts it again (revision item 34); `migrate-legacy` refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason; without `--prune-missing` a row whose manifest is gone is reported and stays AVAILABLE; after `--forget-store` and the store's return, sync adopts its prefixes; a CreateBackup started on another instance during `--move-store` either finishes before the rewrite or is refused; after `--move-store A` stops on a row with one data file missing under the configured root and `--forget-backup` lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied; a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler, which runs those two duties under the claim, has finished the one and failed the other. Depends on PR 6, 8. -Invariants honoured: 4(c) (store identity, scoping, adoption only through `--adopt-store`), 4(f) (every writing command takes the maintenance claim), 4(g) (what sync adopts and never adopts; the opt-ins), 4(d) (the ARN checked against the path and `source_table`), and 4(a)'s rule that no deletion duty acts on the bytes of a row the operator forgot. +Invariants honoured: 4(c) (store identity, scoping, adoption only through `--adopt-store` for a durable catalog and, for an in-memory one, at its startup from another in-memory catalog), 4(f) (every writing command takes the maintenance claim), 4(g) (what sync adopts and never adopts; the opt-ins), 4(d) (the ARN checked against the path and `source_table`), and 4(a)'s rule that no deletion duty acts on the bytes of a row the operator forgot. Interfaces it adds: ```rust fn insert_backup_row(&self, row: &BackupRow) -> BoxFuture>; // sync's unowned insert-if-absent @@ -459,11 +459,11 @@ Interfaces it adds: // listing reads each row's `backups_by_arn` row for the rest of the row fn rewrite_store(&self, keys: &[BackupKey], store: &StoreIdentity) -> BoxFuture>; // `--move-store`, PR 10 ``` -`backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
//` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts (the `backup_index` row PR 9b's matrix seeds by hand is the row this adoption writes); the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. +`backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; the store lists in key order, so sync holds one prefix's objects at a time and the prefix of each manifest it listed, beside what it keeps of the root's rows, whose read is its larger cost (section 7), the reserved `changelog` and `exports/` components skipped; a filesystem store builds its own listing of the root first, section 7), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. A manifest whose ARN is in another region is adopted under this server's region's spelling of the ARN and logged: the row carries the ARN this server issues, so `ListBackups` names this server's region and requests resolve the backup under either spelling, and the manifest keeps its own; the dead-owner probe and the restore accept a manifest whose ARN differs from the row's in the region alone (`BackupArn::names_same_backup`). One of another partition than `aws` is skipped, since every request refuses such an ARN before it reads a row (revision item 34). `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts (the `backup_index` row PR 9b's matrix seeds by hand is the row this adoption writes); the previous owner acts on the root until a later check that can read its catalog finds the store unhealthy, at the earliest one reconciler interval after the cycle in progress ends, unless its deletion fence (section 4(c)) refuses a deletion or a CreateBackup first, so every server of the previous catalog is stopped before the adoption. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too once the previous process has exited (a new in-memory server is started only after the old one has exited; one started beside it adopts the root while the old one still acts on it until a later check or its fence); a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. `migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` and checks that the manifest's item count equals the legacy backup's; only then does it insert the row and delete the legacy rows through the reader's `delete`. It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup_maintenance', RESTORE_LOCK_SEED)` (`MAINTENANCE_LOCK_KEY`), held exclusively on the command's dedicated connection, with the `backup_maintenance` row carrying the holder, the generation and the heartbeat; a job tests the lock with `pg_try_advisory_xact_lock_shared` in one autocommit statement, so the shared lock ends with the statement, and then reads the row's heartbeat age, so a holder whose session ended is still seen held until its heartbeat is older than `stale_job_after_secs`; a command's take tries the exclusive lock a few times over about 10 ms on its connection, so a job's one-statement shared lock does not refuse it. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `LOCAL_SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, and so is the holder of any maintenance claim row, which such a command takes over whatever its heartbeat says; before the wait it runs the reconciler's dead-owner resolutions itself, as the claim's holder (the cycle takes the held branch without reading the claim back through the zero window) (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished) and then waits and removes CREATING rows with a zero stale window, so a job the stop interrupted never makes it wait or refuse whatever its heartbeat says (a RUNNING `manifest` restore, which PR 9a introduces, is the exception revision item 29 states), and a server starting under the serve lock removes a maintenance claim a stopped command left; `backup list` is a read and runs beside a server. -State transitions it owns: a prefix to an AVAILABLE row (adoption); CREATING or DELETING rows of a moved root resolved by `--move-store`; AVAILABLE to DELETING by `--prune-missing`; rows deleted outright by `--forget-store` and `--forget-backup`; a FAILED restore's CREATING target removed by `remove-target`; legacy rows replaced by an AVAILABLE row by `migrate-legacy`. +State transitions it owns: a prefix to an AVAILABLE row (adoption); CREATING or DELETING rows of a moved root resolved by `--move-store`; rows deleted outright by `--prune-missing` (AVAILABLE rows whose manifest is gone), `--forget-store` and `--forget-backup`; a FAILED restore's CREATING target removed by `remove-target`; legacy rows replaced by an AVAILABLE row by `migrate-legacy`. Documentation rows it corrects: upgrade manual: on SQLite the backup CLI commands that write the catalog take the serve lock, so the server is stopped for `backup sync` and its options and for `purge` (section 4); legacy backups are carried forward with `backup migrate-legacy` (above), and one it refuses (an id with no instant, an item the validator refuses, a pre-0.0.4 SQLite definition) is carried forward by restoring it and taking a new backup, which has a new ARN; a store owned by another catalog is adopted only with `backup sync --adopt-store`. ### PR 11: CI and documentation @@ -498,7 +498,7 @@ Phase 3, later (v1.x): - Unit: manifest serde and golden files, key mapping and ARN parsing, store conformance. - Integration, every backend, every PR: round trip with full configuration preservation, composite keys, GSIs and LSIs, vector indexes where supported, legacy fallback, PITR surface. - Consistency: concurrent-writer snapshot test on every backend. -- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with `devtools/backup-kill-points.sh` and its API half `devtools/backup-kill-points.py` (a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store; the clippy job lints the feature so the pause code compiles on every push), not by a pytest cell; the restore kill points are run the same way with `devtools/backup-kill-points.py` (every point of every backend runs in the nightly matrix workflow's four `restore-kill-points-` jobs, one per backend, each built with `extenddb-app/backup-test-hooks` and starting its backend as the matrix job does, the Cassandra job building a plain SQLite binary as the source beside its hooks binary: a server built with `extenddb-app/backup-test-hooks`, whose feature chain pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks` so each backend's own pause point compiles in without the server naming a backend crate, a backup of enough items for two data files, a restore paused at the named point by `EXTENDDB_BACKUP_PAUSE`, `kill -9` by PID, a restart, and the outcome asserted: the exact count after the resume, of the base table and of the GSI `by-g` every item of the fixture carries, or the target gone for a failing restore), on every backend the point applies to (item 33 lists the cells as run on the build before this round, when the first reach of the Cassandra point below ended `target_not_empty`); a Cassandra kill point takes its backup on a SQLite source and seeds the target's row as the matrix does, Cassandra's own CreateBackup being in-catalog (section 3); the Cassandra point that pauses before a batch's progress record (`restore_cassandra_before_progress`) is run at its first reach (a crash before the first progress record leaves the first batch's items with no progress; the resume finds no more items than the backup's total and re-applies the first file with upserts, step 7) and at its second (`--second-reach`, the re-application of at most one batch after a recorded one). MongoDB records each batch's `insert_many` and its progress in one transaction, so a mid-file crash leaves no items the record does not account for and resumes to ACTIVE rather than `target_not_empty`; only a crash before the first batch commits leaves an empty target, which resumes from the start. Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after every file has verified and before the first load (`restore_after_download`; on an S3 store the spool holds every `.verified` marker, the manifest and the sidecar, and no item was written); after a file's transaction commits and before the next file starts (`restore_after_file_commit`); MongoDB after a batch's `insert_many` and its progress commit in one transaction (`restore_mongodb_after_insert_many`, the path a resume continues from the recorded `items_applied`); Cassandra between a batch's last write and its progress write (`restore_cassandra_before_progress`); SQLite mid-batch (`restore_sqlite_mid_batch`); PostgreSQL between the ACTIVE flip and the progress-row deletion (`restore_postgres_before_progress_delete`); MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip (`restore_mongodb_before_worker_flip`), where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7 (`restore_between_failed_and_removal`); a fenced-out restore job that resumes after a successor claimed the row (`restore_fenced_out`: its DONE write matches zero rows, it makes no table access and the successor's table survives with the exact count); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). +- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with `devtools/backup-kill-points.sh` and its API half `devtools/backup-kill-points.py` (a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store; the clippy job lints the feature so the pause code compiles on every push), not by a pytest cell; the restore kill points are run the same way with `devtools/backup-kill-points.py` (every point of every backend runs in the nightly matrix workflow's four `restore-kill-points-` jobs, one per backend, each built with `extenddb-app/backup-test-hooks` and starting its backend as the matrix job does, the Cassandra job building a plain SQLite binary as the source beside its hooks binary: a server built with `extenddb-app/backup-test-hooks`, whose feature chain pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks` so each backend's own pause point compiles in without the server naming a backend crate, a backup of enough items for two data files, a restore paused at the named point by `EXTENDDB_BACKUP_PAUSE`, `kill -9` by PID, a restart, and the outcome asserted: the exact count after the resume, of the base table and of the GSI `by-g` every item of the fixture carries, or the target gone for a failing restore), on every backend the point applies to (item 33 lists the cells as run on the build before this round, when the first reach of the Cassandra point below ended `target_not_empty`); a Cassandra kill point takes its backup on a SQLite source, copies the prefix under the target's root and adopts it with `backup sync`, as the matrix does, Cassandra's own CreateBackup being in-catalog (section 3); the Cassandra point that pauses before a batch's progress record (`restore_cassandra_before_progress`) is run at its first reach (a crash before the first progress record leaves the first batch's items with no progress; the resume finds no more items than the backup's total and re-applies the first file with upserts, step 7) and at its second (`--second-reach`, the re-application of at most one batch after a recorded one). MongoDB records each batch's `insert_many` and its progress in one transaction, so a mid-file crash leaves no items the record does not account for and resumes to ACTIVE rather than `target_not_empty`; only a crash before the first batch commits leaves an empty target, which resumes from the start. Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after every file has verified and before the first load (`restore_after_download`; on an S3 store the spool holds every `.verified` marker, the manifest and the sidecar, and no item was written); after a file's transaction commits and before the next file starts (`restore_after_file_commit`); MongoDB after a batch's `insert_many` and its progress commit in one transaction (`restore_mongodb_after_insert_many`, the path a resume continues from the recorded `items_applied`); Cassandra between a batch's last write and its progress write (`restore_cassandra_before_progress`); SQLite mid-batch (`restore_sqlite_mid_batch`); PostgreSQL between the ACTIVE flip and the progress-row deletion (`restore_postgres_before_progress_delete`); MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip (`restore_mongodb_before_worker_flip`), where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7 (`restore_between_failed_and_removal`); a fenced-out restore job that resumes after a successor claimed the row (`restore_fenced_out`: its DONE write matches zero rows, it makes no table access and the successor's table survives with the exact count); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). - Distribution matrix for the filesystem default path: unprivileged local launch, the npm launcher, the container image with and without a volume, `sqlite-memory`. - Cross-backend matrix: nightly. - Live DynamoDB: Phase 2, measured. @@ -560,7 +560,7 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 3. DeleteBackup deletes the manifest first and tombstones the row as DELETING; FAILED is a tombstone too; sync never adopts a CREATING, DELETING, DELETED or FAILED prefix and only touches rows of the configured store (5.2, section 4). The draft's order could resurrect a half-deleted backup, and a fenced-out job's late manifest could have been adopted over a FAILED row. 4. Restore returns CREATING from the request (table and job row created atomically) and the job verifies every file before loading any; progress is recorded per file in a way a crash cannot split from the data (a data-database row on PostgreSQL, the same transaction on SQLite, idempotent re-application on MongoDB) (5.3). The draft's per-file resume re-applied a file after a crash between commit and checkpoint, which fails on PostgreSQL's plain INSERT and silently picks a winner under UPSERT; an intermediate revision made the request synchronous for the whole download, which contradicted the asynchronous goal. 5. The manifest's self-checksum is replaced by a sidecar over stored bytes (section 1). -6. Per-job claims with fencing replace the assumed single-instance lease, which does not exist on PostgreSQL or MongoDB; no row ever exists unowned (the PostgreSQL lock precedes the row, the MongoDB row is born with its generation), every catalog write is conditional on the generation on every backend, PostgreSQL heartbeats go through the lock connection, and a cancelled job fails rather than waiting an hour in CREATING (5.1, Non-goals). +6. Per-job claims with fencing replace the assumed single-instance lease, which does not exist on PostgreSQL or MongoDB; no CREATING row exists unowned (the PostgreSQL lock precedes the row, the MongoDB row is born with its generation; sync's AVAILABLE rows are unowned, revision item 34), every catalog write is conditional on the generation on every backend, PostgreSQL heartbeats go through the lock connection, and a cancelled job fails rather than waiting an hour in CREATING (5.1, Non-goals). 7. Index population on restore stays per backend (`bulk_load`); the draft's shared backfill worker exists only on MongoDB (5.3 step 6). 8. Failed creations are BackupNotFound, not DELETED (section 4). 9. `item_order`, `key_range`, `md5`, `uncompressed_bytes`, explicit nulls, three-state `vector_indexes`, optional `etag`, `dataFileS3Key` as a full key, partition-aware `BackupArn` (section 1, 4). @@ -591,3 +591,5 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 32. (2026-11-04, after the maintainer's review of PR 8.) The heartbeat's one-period bound is placed by the backend (`JobClaim::heartbeat_within`, a provided method that bounds the whole call; SQLite's override takes the engine write lock first and bounds only the `UPDATE`, so a writer holding that lock past a period, a GSI backfill say, delays the beat and the claim re-check and ends no job; the re-check's confirmation counts as a beat for the pre-put check; after the maintainer's second round, the pre-put check also passes while a beat is in flight and has not failed, since a backup large enough to reach a put while the beat was parked on the lock past two periods stopped at that put, the maintainer's 180,000-item backup beside a GSI build twice at the third or fourth data file, and the reconciler then failed the row and deleted its objects; the safety argument is in 5.1 step 4 and `heartbeat.rs`'s module doc, and the tests are a SQLite cell whose second data file is put under a hold of the lock, `a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it`, and an engine cell with a claim whose beat parks, `a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost`), and `heartbeat.rs`'s SIGSTOP text says that a job with no beat in flight resumes under its own generation once its next beat succeeds. The store-backed create checks the maintenance claim before the permit, as the in-catalog paths do, and inserts no row for a refusal (5.1 step 2 and 4(f) are edited: a FAILED `maintenance` row carried the insert's heartbeat and `backup purge`'s wait counted it as a live job for the stale window). Before every put the job checks the store as the runtime holds it (healthy, and its identity the row's) and on a failure fails the row with the store's reason and makes no store call, the prefix's deletion included (4(c)'s swapped-root case). The open and the insert are selected against the shutdown token. The uploader cancels the feeder's token before it awaits an encoder, so a put that fails mid-file ends the job while the stream is held. A data file whose stored size differs from the encoder's compressed count fails the job (the dead-owner probe refuses that difference). The AVAILABLE write sets `TableSizeBytes` from the stream as it sets `ItemCount` (the differences document names both). `failed_row_retention_secs` is at least `stale_job_after_secs` (a FAILED row removed sooner left a stopped owner's late object with no row to delete it under). A registration lost while serving cancels the jobs' token, which the runtime replaces for the jobs that start under the new registration; the shutdown sequence after the listener is one function the server's tests drive with a job that outlives a short drain. The plain-HTTP listener's drain is bounded as the TLS listener's is. CreateBackup is bound to the in-catalog path on SQLite where the serve lock is not taken (not Unix), since the slot claim is sound only with one process per file. 4(c) states the gap of the different-roots guard before the first AVAILABLE row. The compose files' `stop_grace_period` is 80 s. The kill-point scripts are committed under `devtools/` and the clippy job lints `backup-test-hooks`; a MongoDB job runs the backup suites against a filesystem store. The tests added are the review's: the lock held past two periods on SQLite, the mid-file put failure with the stream held, the unhealthy and renamed store mid-job, the key-ordered stream out of order, `BackupStoreErrors` for the job's `delete_prefix` and the probe's `list`, the re-read under another generation, the two-period sample, the nesting rule through lists and mixed levels, the keepalive literals, the driver's five-row run-ahead, the beat between the takeover's read and its LWT on Cassandra, `pool_size_for_jobs(11, 4)`, and the retention floor. After the third round the clause applies only where the beat parks before its bound (`JobClaim::heartbeat_parks_before_its_bound`, `true` on SQLite only): on PostgreSQL, MongoDB and Cassandra a beat in flight confirms nothing, since a process stopped mid-beat and resumed would otherwise pass the check until its timer raised the loss, and one stopped between beats for the round trip of the beat its missed tick starts, a case a bound of one period from the beat's start (the third round's first fix) did not cover (`a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks`, `a_stop_between_beats_passes_no_check_while_the_resumed_beat_runs`). 33. (2026-11-01, PR 9 as implemented — the design's PR 9a and PR 9b as one PR, the maintainer's decision.) The restore job runs from the store on PostgreSQL, SQLite, MongoDB and Cassandra, same-backend and cross-backend, from a filesystem or an S3 store; the matrix and the nightly are PR 9b's, shipped here. `RestoreTableFromBackup` of a `backup_index` row no longer answers the "in a backup store; this release restores in-catalog backups only" refusal `check_catalog_row_for_restore` carried; a healthy configured store's AVAILABLE row is restored, and the refusal is gone from the engine and the differences, usage and upgrade docs (5.3, section 10, PR 9a). The interface: `RestoreEngine`, a new trait in `StorageEngine`'s bound with `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied`, `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }`, `count_restore_target(target) -> u64`, `read_restore_progress` (default `[]`) and `sweep_restore_progress(running) -> u64` (default 0); `StorageEngine` has no methods of its own, so `bulk_load` is not "a new `StorageEngine` method" (section 3's PR 9a interface list). `BackupCatalog` gains `claim_restore(account_id, input: CreateTableInput, &RestoreRegistrationRow) -> (TableDescription, Box)` (the registration creates the CREATING target and returns the description the request answers; the record is born owned under the backend's generation, `RestoreRegistrationRow.owner_generation` being `UNOWNED_GENERATION` and the backend minting its own), `claim_restore_dead(key, stale_after, spool_host) -> Option>` (the restore analogue of `claim_dead_backup`, writing the claimant's `spool_host`, a fresh generation and a fresh `heartbeat_at` on the database's clock in one conditional write, where the backup takeover leaves the heartbeat: nothing waits on a restore record's old heartbeat after a takeover, and a second taker must find the record live at once, 4(e)), `read_restore_job`, `list_restore_jobs(Option)` and `remove_restore_target`/`fail_restore_job` as the reconciler uses them; `RestoreJobRow` carries `store`, `spool_host`, `file_index`, `items_applied`, `failure_reason`, `completed_at`, so the FAILED-target duty reads a row's reason and the sweeps read its store and `spool_host` (section 3, 4, 5.3, PR 9a, PR 9b). The design's `claim_restore(&RestoreJobRow) -> Box` and `record_progress(&dyn JobClaim, file_index, items_applied)` are gone: there is no `record_progress`; every backend records progress inside `bulk_load`, in the write that applied the items (PostgreSQL a `restore_progress` row in the file's data transaction, SQLite and Cassandra a `tables`-row update per batch, MongoDB a `restore_job.file_index` update per file), and the design's "conditioned on the generation" holds on SQLite, MongoDB and Cassandra but on PostgreSQL the progress row lives in the data database, which holds no job record, so the fence there is the base table's primary key (a file applied twice fails on its first key, a recorded file on the progress row's own key) and the claim's liveness, which the engine confirms before each file and whose loss, the heartbeat's, aborts a file in flight; the generation fences only the record's own writes in the catalog (DONE, FAILED, the takeover) (5.3 step 5, PR 9a). `bulk_load` carries `reapplying` so MongoDB's duplicate suppression is explicit: on a resume the first file the engine meets is `reapplying = true` and every later file false; MongoDB suppresses an all-E11000 `insert_many` only when `reapplying`, else "the backup holds a key twice"; PostgreSQL and SQLite never suppress (their base rows are plain `INSERT`s, a duplicate fails the file whatever the flag, and SQLite has no unrecorded partial write, its progress being in the batch's transaction); Cassandra's `put_restored_item` is an upsert, so a duplicate collapses and `count_restore_target` one short of the manifest's total is the only duplicate check there (5.3 step 6, PR 9a). The restore ends by counting the target against the manifest's item total on every backend, so an unordered backup holding one key in two files and a key-range check that missed are both caught; a key-ordered file runs #394's per-file `KeyRange` check, which ends the file's stream at an item outside its range. The migration for PostgreSQL's `restore_progress` is id 6 (`MigrationId::RestoreProgress`), a `StoreKind::Data` body on PostgreSQL (`data_migrations/0006_restore_progress.sql`), `NotApplicable` on SQLite, and nothing on MongoDB and Cassandra, whose bootstrappers have no `MigrationId` table; this corrects "data migration 005 for `restore_progress`" in the PR 9a section (its acceptance line and its load paragraph) and the PR 6 line on reserving numbers; item 14's "data migration 005 reserved" stands as the history it is (4(i), PR 6, PR 9a). The manifest registration locks the `backup_index` row (the lock `start_backup_delete` takes: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), while the legacy registration keeps the `backups` row; a legacy and a store-backed restore into one name exclude each other (PostgreSQL the advisory `restore_lock_key`, which both paths take; on SQLite the store-backed path's `restore:/
` slot and, for a legacy restore, the `tables` row under the write transaction), so a `DeleteBackup` of a `backup_index` row meets only store-backed registrations and the two orderings resolve one before the other (DeleteBackup wins → the registration is `BackupNotFound` and no target; the registration wins → DeleteBackup is `BackupInUse` and the row stays AVAILABLE) (5.2, 5.3, PR 9a). The restore's ACTIVE flip sets the table and every secondary and vector index of the target in one catalog transaction on PostgreSQL and SQLite (an index of a restore target is created ACTIVE, and `activate_restore_target` sets ACTIVE any `indexes` or `vector_indexes` row that is not), on MongoDB the job writes DONE and `restore_backfill_pending` and the backfill worker flips the table ACTIVE once every index has left CREATING (`finish_restore` returns `Pending` there), and on Cassandra the one lightweight transaction flips the table alone, its index rows created ACTIVE (`Active`, never `Pending`), so a Query on each GSI and `SearchVectors` answer as soon as the target reports ACTIVE (5.3 step 6, PR 9a). The request bounds its wait for the detached registration at `RESTORE_REGISTRATION_TIMEOUT` (60 s) on every backend; past it it answers `InternalServerError` while the task goes on and the target appears through `DescribeTable`, so the differences document's MongoDB-only sentence becomes every backend's (5.3 step 4, PR 9a). The maintenance check runs after the registration on the manifest path (5.3 step 4 as written) and before the permit on the legacy path (item 29), both orders in `restore_table_from_backup` and `restore::start`; the restore's first confirmed beat stands at the instant `claim_restore` returned (the record's `heartbeat_at` was stamped inside the registration, before the return, so the instant is at or after the beat it stands for by the registration's remaining duration, which each backend bounds, 4(e)), where the backup job takes the instant before its claim's insert (5.1 step 4), so a registration that took longer than two heartbeat periods does not end the job at its first check; a held claim on the manifest path leaves a FAILED `maintenance` record whose target the job removes before it answers `LimitExceededException`. Only `StoreError::Transport` is retried on a data-file download, for up to `DOWNLOAD_RETRY_BOUND` (15 min, counted from the job's first download attempt, every file's retries against it) with `min(2^n s, 60 s)` plus up to a second of jitter, so the retry covers S3 stores (a filesystem store's `get` errors are `Io` and fail at once); `NotFound`, a denial, and a digest or size mismatch fail at once naming the file. On a filesystem store "verified in place" is two `get` passes, a hashing pass before any load and a decoding pass through `ExtenddbManifest::reader` that hashes again (so a file replaced between the passes fails in the load) with no spool; on an S3 store the files are spooled as `restore-
/.json.gz` with `.partial` → `sync_all` → rename → `.verified` (the marker holding the file's sha256, compared with the manifest's before the file is trusted; an entry left by a run of this host for a record another host then owned is removed before use) and read through `reader_prehashed`, the spool at `spool_path` (the defaulted `backup-spool/` on file-backed SQLite). A restore the shutdown stops records `RestoreJobs{cancelled}` and no worker metric, keeps its spool and its progress and writes no status (the `SHUTDOWN` reason is a backup's alone); a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` with no catalog or table access; `fail_restore` writes FAILED then runs `remove_restore_target` only when that write matched. A job that learnt the record is another's (the lost claim, the DONE write's `NotMatched`, `fail_restore`'s no-table-access arm) leaves its spool entry to the sweep, since a same-host successor may be reading it (the entry is keyed by table id); every other end but the shutdown's removes it (4(e), 5.3 step 7, PR 9a). The load hands the backend one `bulk_load` per file over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB of data-file line text) through a channel of two chunks; a restore holds the parsed manifest, one decoded line and its parse, the decoded items of four chunk positions and the backend's batch (32 MiB of item text on SQLite and Cassandra, 4 MiB per INSERT on PostgreSQL, 1,000 documents on MongoDB), which is section 7's restore sentence, with the figures measured on this PR's final tree (`docs/design/15-backup-restore-bench-pr9.md`: 848 MiB over idle on SQLite and 778 MiB on PostgreSQL at 24 worst-shape items under the default arenas, 392 and 427 MiB under one arena; 89 and 16 MiB at 1M key-ordered items of 200 bytes, in 26.8 s and 9.5 s, 144 to 164 and 16 MiB for the same items unordered), which replace the earlier sentence, "A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table": the terms stand, the figures are measured, and a restore of large items is of the backup's order. The metrics are `MetricName::{RestoreJobs, RestoreJobDuration, RestoreBytesDownloaded}` with the outcome in the operation dimension, `QuerySource::RestoreJob` for the worker metrics, and `BackupStoreErrors{get}` on every failed `get` of a restore job (the request's and a resume's two document reads and each download attempt the store failed, retried or not; a digest mismatch and the spool's own errors are not the store's); section 9's `restore_jobs_total{outcome}` and `restore_job_duration_seconds` are a converter's spellings, as item 31 said for the backup names (section 9, PR 9a). A backend's refusal of an item in `bulk_load` is not a wire exception: on MongoDB an `N` sort-key value past Decimal128's 34 significant digits is the write path's `ValidationException`, which becomes the restore's FAILED reason, and the client sees the target disappear; a 38-digit non-key number is stored as text and restored intact on every backend. The reconciler resumes a dead store-backed restore from its recorded progress (per file on PostgreSQL and MongoDB, per batch on SQLite and Cassandra) under `claim_restore_dead` once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; it fails a dead manifest record under a maintenance claim before it looks at the record's store (`OWNER_DEAD`, the target removed; `target_not_empty`, the target left for `remove-target`, when it recorded no progress and the target holds items), fails a record of a store other than the configured one or of no store with `OTHER_STORE` (naming `backup sync --move-store`, the target left to the FAILED-target duty), skips a resume without a permit, and FAILs a resume with no recorded progress into a non-empty target as `target_not_empty`, left for `extenddb backup remove-target --account
` (PR 10's command, shipped with PR 9, below); its two sweeps (`restore_progress` rows of tables that are not RUNNING restore targets; spool entries of restores not RUNNING with this host's `spool_host`) run only without the maintenance claim and read `list_restore_jobs(Some(RUNNING))`, a backend that cannot list them logging once and skipping both (section 4, 5.3 step 7, PR 9a). A store-backed backup of a multi-part-key table is written `unordered` and restores on every backend, the target laid out as a natively created table (`bulk_load` keys rows with the backend's own key functions), while a legacy backup of one keeps the reader's refusal; the #384 override tests run through both the legacy and the manifest paths for every case with a definition and assert the same input or the same refusal, the manifest path deriving the SSE specification from the manifest's `sse_description` where the legacy path keeps the specification as written. The cross-backend matrix (PR 9b): a cell copies the source's backup prefix into a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so the cell proves the restore and not the adoption; it runs from the committed `devtools/backup-restore-matrix`, and `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs the twelve plain cells and the duplicate-key and vector cells nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do; the seed stands in for `backup sync --adopt-store` until PR 10 replaces it. The restore kill points are run by hand with the `kill-points.py` posted with PR 9 (outside the repository, as PR 8's was), one mode per backend where it applies: `restore_after_download`, `restore_after_file_commit`, `restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, `restore_mongodb_before_worker_flip`, `restore_between_failed_and_removal`, `restore_fenced_out`, `pg_terminate_backend` and `sigterm`, each a SIGKILL or a signal at the named point with a restart and the outcome asserted (the exact count after a resume, or the target gone for a failing restore). Two points pause before the first progress record of a file or batch (`restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`), so a kill at their first reach leaves written items and no progress and the resume answers `target_not_empty` (the rule of 5.3 step 7, asserted); the re-application path (MongoDB's E11000 suppression of the one unrecorded file, Cassandra's re-applied batch) is run by killing at the point's second reach, after an earlier file's or batch's progress is recorded (the script's `--second-reach`): on MongoDB a 24,000-item backup in three files resumed from file 1 with file 0 recorded and reached ACTIVE with the exact count (file 1's 8,227 items re-applied as duplicates, 7,544 applied by the resume); on Cassandra a one-file backup of 12,000 items killed before the second batch's progress write resumed from item 10,000 and reached ACTIVE with 12,000 (2,000 applied by the resume), and `restore_after_file_commit` resumed from file 1 to the exact count. A Cassandra server's CreateBackup is the in-catalog format (section 3), so a Cassandra kill point takes its backup on a SQLite source, copies the prefix under the Cassandra target's root and seeds the target's `backup_index` row as the matrix does; the single-node Cassandra of the harness needs `replication_factor = 1` in the server's configuration, as `init` writes it, since under the default of 3 every lightweight transaction and quorum read on a keyspace the server creates answers "No nodes available in query plan", which is what had blocked these points. The feature chain for the backends' pause points: `extenddb-app/backup-test-hooks` → `extenddb-server/backup-test-hooks` now pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks`, so a hooks build compiles `extenddb_storage::backup_catalog::restore_pause_point` and each backend's own point (`restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, each a `pub const` beside `restore_pause_point` in `extenddb_storage::backup_catalog`, the one spelling the backends and the kill-point script use) in without naming a backend crate (the server depends on none directly); the engine's five restore points are its `KillPoint` variants (`RestoreAfterDownload`, `RestoreAfterFileCommit`, `RestoreMongodbBeforeWorkerFlip`, `RestoreBetweenFailedAndRemoval`, `RestoreFencedOut`), and that enum is the engine's list, not the whole list. Corrections made before PR 9 merged: (1) the restore takeover writes `heartbeat_at = now()` in the one conditional write with the generation and `spool_host`, on every backend, so a second taker finds the record live at once; the backup takeover keeps leaving it (4(e) states why the rules differ); (2) `LOAD_CHUNK_BYTES` is summed from the decoded line's length (the decoder's `bytes_decoded` across one item), not `item_size_bytes`; section 7's terms are restated in that unit; (3) the download retry bound is per job: `download_all` takes the job's download start once and every file's retries count against it; the module doc, 5.3 step 3 and this item say so; (4) a job that learnt the record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the reconciler's sweep; (5) the `.verified` marker holds the file's sha256 and `is_verified` compares it with the manifest's; `Entry::open` removes an entry whose record's `spool_host` was another host's before use; the resume reads the manifest from the spool only when the record named this host before the takeover (it read the record after the takeover, which had just written this host's name); (6) the decoder checks the heartbeat's pre-write rule (`confirms_the_claim`) at every chunk boundary beside the shutdown token; MongoDB and Cassandra re-read the record per batch; (7) a restore's `claimed_at` is the instant `claim_restore` returned, the record's `heartbeat_at` having been stamped inside the registration. Cassandra's takeover re-applies the `active_restores` entry when absent; the SQLite and Cassandra batch is bounded by `RESTORE_BATCH_BYTES` (32 MiB of item text) beside `RESTORE_BATCH_ITEMS` (10,000), both in `extenddb_storage::restore_engine`; MongoDB and Cassandra read the record before each write batch; `sweep_restore_progress` deletes from a positive list; the backends' pause-point names are `pub const`s beside `restore_pause_point` in `extenddb_storage::backup_catalog`, one spelling each (`RESTORE_SQLITE_MID_BATCH`, `RESTORE_MONGODB_AFTER_INSERT_MANY`, `RESTORE_CASSANDRA_BEFORE_PROGRESS`, `RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE`), and the engine's `KillPoint` is its own list; Cassandra's `count_restore_target` reads the catalog first and answers `TableNotFound` for a key that is not a CREATING restore target with that id, as the other three do; the MongoDB registration stamps the record's heartbeat from the server's clock inside its transaction (the registering process's clock is replaced; the test registers from a process an hour behind); a duplicate primary key fails the file with `data file : holds a primary key the target already has (the backup holds a key twice): `, the engine naming the file by its key once and the backends naming no file; SQLite's startup line for an aged RUNNING record names the reconciler's duty per kind. The reconciler's store check runs first in the cycle, so the manifest arm reads this cycle's health, and the skip line names `not yet checked` for a store whose first check has not recorded; a maintenance-held dead manifest restore is counted once, in `manifest_restores_failed_and_removed`; `RestorableKeySchema::resolve` runs in the request with step 2's refusals; the request's second table-name check and second `check_supported` are gone: the parser's identity rule and version gate run first; the engine's kill points log `job reached kill point `; each backend's own point is named beside `restore_pause_point`; `finish_restore`'s `Pending`, the DONE write's `NotMatched`, the pre-write claim checks of `run`, the download's pre-attempt check and `WorkerErrorCount{RestoreJob}` on a FAILED job have engine tests; the #384 override tests run through both paths for every case with a definition. The wire test `test_create_backup_of_a_vector_table_is_refused` is `test_vector_table_backup_refused_in_catalog_and_restored_from_store`, since it asserts the round trip on a server with a store; the multi-part-key wire cell lives in `tests/test_backup_job.py` on a deployment that enables `allow_multipart_table_keys`, compares the restored item set with the source's `Scan` and makes no `GetItem` by the whole key (the preview's item paths address such a table by its first HASH attribute on every backend: a `GetItem` by the whole key finds nothing on PostgreSQL and the first HASH key's one item on SQLite, restored or not). Corrections made before PR 9 merged, second set: `extenddb backup remove-target --account
` ships with PR 9 rather than PR 10 (the PR 10 section's clause says so): the `target_not_empty` rule was written for a catalog restored from a dump that names a restore which finished and was written to since, but its common trigger on MongoDB and Cassandra, which record progress after the write, is an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra: the kill leaves written items and no progress), and without the command this release had no way to remove such a target (DeleteTable refuses a restore target, the FAILED-target duty skips the reason, DeleteBackup and a second restore of the backup are `BackupInUseException`); the first-reach runs of `restore_mongodb_after_insert_many` and `restore_cassandra_before_progress` are that state, so the PR 9a acceptance line, the PR 9a load paragraph, the testing summary and the differences doc's restore-interruption row name it as the third outcome beside "finishes with the right count" and "gone", where they had named two; the command is the one section 4's duty list describes (the maintenance claim, a FAILED record's target whatever the reason, a RUNNING record's dead target, a legacy record's target, the three refusals, no store needed, the audit line), its tests are the engine's maintenance tests and one wire cell on SQLite seeded into the state at the catalog, and the two first-reach kill points run it and assert the target gone and DeleteBackup allowed. The spool sweep lists the spool before the records (the design's rule, which the shipped order broke); the spooled manifest and sidecar are written as the data files are and a spooled pair that fails verification yields to the store's; the resume's first confirmed beat is the takeover's return, taken beside `claim_restore_dead`; a `finish_restore` error is an unknown outcome re-read through the claim; `BackupStoreErrors{get}` counts every `get` the store failed; the request's and the resume's document reads are bounded (4 KiB, 16 MiB); the restored table's size is the source table's; `RestorePostgresBeforeProgressDelete` is gone from the engine's `KillPoint` (one spelling each holds: the four backend names are the storage crate's constants and nothing else); a registration that fails after the request stopped waiting is logged. The restore's re-applied batch on Cassandra relies on the write path rewriting an unchanged item's GSI/LSI row instead of deleting it (#742, fixed on main by #424), and the `restore_cassandra_before_progress` second-reach run reaches ACTIVE with the GSI count exact where before #742 was corrected it was one batch short. PostgreSQL's `finish_restore` deletes the progress rows after the flip's commit and does not fail the restore if that delete errors (logged and swept). A file's PostgreSQL progress row is a plain INSERT from `start_at == 0` and an upsert from `start_at > 0`. Two concurrent registrations of one backup leave one target on Cassandra (the registration LWT's `active_restores = null` is the fence). What bounds a registration's duration per backend (the advisory/`FOR UPDATE` lock, the slot, `transactionLifetimeLimitSeconds`, the LWT timeouts), so the `claimed_at`-minus-`heartbeat_at` gap can never reach the stale window, replaces the earlier "conservative" in 4(e) and in this item, which had the direction wrong: an instant later than the stamp makes the pre-write check admit a beat older than two periods by the gap, so the gap needs a bound, not a name. The SQLite restore's published memory figures are the no-index shape; a table with indexes costs SQLite's page cache (`cache_size`) plus arena fragmentation, bounded, not the restore's buffers (the record's index table: 143 to 225 MiB at 100k and 165 to 172 MiB at 300k with two GSIs on SQLite, 23 to 34 MiB on PostgreSQL), and the 60,000-item shape of the differences doc's in-catalog timings was measured store-backed on PostgreSQL, SQLite and MongoDB (0.67, 1.85 and 3.74 s to ACTIVE), the row now labelling which path each figure is. The MongoDB CI jobs' `EXTENDDB_TEST_BACKUP_STORE` is `filesystem` from this PR (`devtools/run-mongodb-tests` serves a filesystem root), where it was `none`; the Cassandra job's stays `none`; the runner's `--backup-store none` is the one way to run the legacy LSI fallback cell, the in-catalog restore path's one wire cell on MongoDB, which no CI job runs. The PR 11 line owns the two unmeasured cells (the MongoDB and Cassandra 1M restores, the vector cell) and no longer the nightly schedule, shipped here. The kill-point cells as run on the pushed head's hooks binaries (4c353191), every ACTIVE outcome asserting the base table's and the GSI's `Scan` counts: `restore_after_download`, `restore_after_file_commit`, `restore_between_failed_and_removal` and `sigterm` on all four backends; `restore_sqlite_mid_batch`; `restore_postgres_before_progress_delete`, `restore_fenced_out` and `pg_terminate_backend` on PostgreSQL; `restore_mongodb_after_insert_many` at its first reach (`target_not_empty`, then `remove-target`) and its second (the re-application, 24,000 items in three files); `restore_mongodb_before_worker_flip`; `restore_cassandra_before_progress` at its first reach (`target_not_empty`, then `remove-target`) and its second (12,000 items, the GSI exact); the Cassandra points with a SQLite source. Rerun on the build after the last corrections (the first beat before the reply, the Cassandra re-read and fence, the PostgreSQL vector drop): `pg_terminate_backend` as reworked (the lost claim at the next check 20 s after the termination, the record RUNNING with its `(0, 8228)` progress row, the takeover 45.6 s after the termination and 46.7 s after the last beat, ACTIVE 24,000 on base and GSI), `restore_after_file_commit` and `restore_mongodb_before_worker_flip` on MongoDB (resumed from data file 1 after 0 of its items, 3,771 applied by the resume, DONE then the worker's flip, 12,000 exact; DONE before the kill and the restarted worker's flip after it), `restore_cassandra_before_progress` at its second reach (resumed after 10,000, 2,000 applied, 12,000 on base and GSI) and `restore_between_failed_and_removal` on PostgreSQL with a vector backup (the target gone after the stale window, its data table and `_ddb_vec_` relation gone with it, where the pushed head's build left the relation). The matrix's four `--duplicate-key` and eight `--vector` cells ran on the final binaries. Corrections made before PR 9 merged, third set. The job's first heartbeat is sent right after `claim_restore` returns and before the reply, and the pre-write check counts its two periods from that beat's return; a `Lost` or failed first beat fails the record ("the claim was lost before the job started") and answers `InternalServerError`. `extenddb backup remove-target` decides a RUNNING owner dead through `list_dead_restore_jobs(owner_window)` on the database's clock, and the purge's wait decides a backup row's owner live through `list_backup_rows_older_than` on the heartbeat clock; neither reads a host clock. The reconciler takes nothing over and writes no FAILED on its manifest arm once the server's shutdown has begun. Under a maintenance claim a dead record with no progress over a target that holds items is failed `target_not_empty` through `fail_restore_job`, not `OWNER_DEAD`. The reconciler's `target_not_empty` lines print `extenddb backup remove-target --account
`. The restore job's stop line names the heartbeat timer's reason when the timer raised the loss, and the timer logs the reason at WARN. A DONE write whose answer was lost and whose record re-reads DONE under the job's generation ends with "complete (DONE)", not "ACTIVE". A spool document write that fails and a resume that fails before it runs remove this host's entry. The engine's tests of the 60 s bound, the slot held to the job's end and the re-reads' generation checks are cells of their own. Cassandra's progress lightweight transaction re-reads an unknown outcome (applied, one retry, or the lost claim; a second unknown outcome fails the restore), where the earlier text said the restore fails and the reconciler resumes it, which the FAILED write's removal made false. The Cassandra registration's refusal path conditions the target's removal on its own generation; a record a reconciler took over during the DDL keeps its target. PostgreSQL's `abort_restore` drops a vector index's relation with the data table. The PostgreSQL record's `heartbeat_at` is the registration transaction's start. Tests: `test_an_unknown_progress_write_outcome_is_decided_by_a_re_read`, `test_a_registration_taken_over_during_its_ddl_leaves_the_successors_target`, `test_a_mid_file_resume_records_start_at_in_every_batch_progress`, `remove_restore_target_drops_the_vector_relation_with_the_data_table`. The engine's `KillPoint` has five restore variants, named above. SQLite's takeover clock is `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, the `NOW` constant of its catalog (4(e) names it). `extenddb backup remove-target` refuses a RUNNING record with a live owner by its own text ("stop it or wait for it"); the DONE and no-record refusals are the two "not a restore target" ones. The published 1M figures are the key-ordered shape's and the record says so beside them, with the same items measured as an unordered backup (SQLite 144 to 164 MiB over idle in 66 to 90 s against 87 to 98 MiB in 27 to 56 s key-ordered; PostgreSQL 16 MiB either way), and SQLite's final `COUNT(*)` is the counting connection's page cache, bounded by `cache_size` (32 MiB) and the same size at 1M and at 3M rows (22 to 28 MiB), where the record had called it the one term that grows with the table from a single measurement. The `pg_terminate_backend` cell of `kill-points.py` pauses the job at `restore_after_file_commit`, terminates the claim session while it is paused (psql must answer a row), waits past one heartbeat period, releases, and asserts the lost-claim stop line, the record RUNNING with its progress row, the takeover no earlier than `stale_job_after_secs` after the owner's last beat and the exact counts; the shipped cell had terminated the session after a fixed sleep and asserted the final count alone, which a 2.6 s restore satisfied without the case ever running. The script's `--vector` switch gives the source table a vector index, and the tamper cell on PostgreSQL then asserts the removed target's data table and `_ddb_vec_` relation gone. `.github/workflows/matrix-nightly.yml` installs `postgresql-client` and checks `psql --version` and the two drivers' import before the cells; it has had no run, since `workflow_dispatch` needs the file on the default branch, so its first run after the merge is the proof of the runner image. The upgrade manual counts migrations 1 to 7 as breaking and migration 8 as declared breaking on PostgreSQL for want of the additive-declaration job (on SQLite, MongoDB and Cassandra it does not apply). The body sentences each of these corrects are edited with this item (section 1's duplicate rule and Decimal128 bullets, section 3's PR 9a interface list, 4(b)'s progress members and metric name, 4(e) and 4(i), 5.2, 5.3's preamble and steps 1 and 3 to 7, section 7's restore memory sentence, section 9, section 10's table, the PR 6, PR 8, PR 9a and PR 9b sections, the PR 10 clause naming what the seed stands in for and the command shipped here, the PR 11 line, the testing summary and the review checklist); earlier items keep their wording, as item 22 requires. With the rebase onto #666's head (2026-11-03): the jobs and the reconciler are not gated by the PostgreSQL serving registration, and 4(e) says what they do in the registration's retry window, GitHub issue #755 being the gate; `backup purge`'s own cycle under the SQLite serve lock fails a dead store-backed restore and removes its target rather than resuming it (4(f)), since the command tells the cycle it holds the claim instead of the cycle reading it through a zero window; the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB, as 5.3 step 5 does; and MongoDB's snapshot open waits, at most 2 s, for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a store-backed CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a table that does not exist. With the rebase onto `main` (2026-11-04), after #335, #666 and #681 merged: `restore_progress` was migration 7 (4(i)), declared not applicable in the MongoDB and Cassandra registries #681 added, as on SQLite; 4(e) cites the module doc of `crates/server/src/registration.rs` for what a lost serving registration does, which replaces this item's 2026-11-03 clause on it: the reconciler stops with the server's workers and spawns no resume once its own token has fired, and the running jobs go on, #755 being the gate for them; a restore job joins PR 8's job drain as a backup job does; and `remove-target` releases its serving registration when it exits and is named on MongoDB and Cassandra as on PostgreSQL. With the move onto `main` at e8ca7df3 (2026-11-04), after #424 merged: the Cassandra write path's rule for an unchanged index row (rewritten by its INSERT alone, #742) and its version stamps on every index row are #424's, so this PR no longer changes `sync_indexes` or the GSI worker; the PR 9a load paragraph, 5.3 steps 5 and 6 and this item cite #424 for what a re-applied batch relies on, and a restored item's index rows carry its base row's `version` as #424's write path gives them (`put_restored_item` is that path). After the second review round (2026-11-05): the Cassandra load no longer goes through `put_restored_item`'s lightweight transaction but writes each item's base and index rows in one logged batch `USING TIMESTAMP` a reading taken before the batch's claim check (`put_restored_item_at`), the fence against a batch a stalled writer lands after a takeover (the PR 9a load paragraph); the index rows still carry the base row's `version`, which is now that timestamp. A resume with no recorded progress fails `target_not_empty` only when the target holds more items than the backup's row counts; at or below the total it re-applies the first file with upserts (step 7), so a Cassandra crash before the first progress record resumes to ACTIVE. One instance's sixth takeover of a record fails it (`RESTORE_TAKEOVERS`, step 7). `restore_target_ready` on Cassandra checks each GSI's and LSI's table as well as the data table. The nightly workflow runs every restore kill point of every backend in four jobs (the testing summary); the kill-point script's first-reach cells assert ACTIVE with the exact count and its `--remove-target` flag is gone (the `target_not_empty` state is seeded by `tests/test_backup_job.py`'s `remove-target` cell). With the rebase onto `main` after #671 and #672 merged (2026-11-06), `restore_progress` is migration 8: #819 took 7 for PostgreSQL's `stream_pending`, so the SQL file is `0008_restore_progress.sql`, each backend's arm follows `StreamPending`, and CONTRIBUTING's worked example is migration 9. + +34. (2026-11-04, PR 10 as implemented, #746.) PR 10 is split in three: #746 ships plain `backup sync`, `--prune-missing`, `--adopt-store`, the `sqlite-memory` startup sync and `BackupCatalog::insert_backup_row`; #820 carries `--move-store`, `--forget-store` and `--forget-backup` with `list_backup_rows_by_root` and `rewrite_store`, which rewrite or delete rows of other stores, and the full report of the rows of other roots; #821 carries `migrate-legacy`, whose data path (the frozen reader, `DataFileEncoder`, the item validator) shares nothing with sync but the insert; `remove-target` shipped with PR 9 (item 33). The PR 10 section's line, the PR 9b paragraph (the matrix now adopts with `backup sync`) and the doc-row table's `migrate-legacy` row say so (PR 9b, PR 10, section 10). Neither the Cassandra sweep of orphan `backup_index_by_status` entries nor the MongoDB sweep of the data collections a failed restore-target removal leaves is in any of the three: the code comments that gave them to `backup sync` now say no sweep removes them yet, and the MongoDB error line names the collections for an operator to drop. `insert_backup_row(row) -> bool` inserts the row as given, unowned (`owner_generation` 0), with `completed_at` and `heartbeat_at` on the database's clock, and answers `false` when a row with its key exists: PostgreSQL `INSERT ... ON CONFLICT DO NOTHING` with no conflict target, so both unique indexes (the key and `backup_id` alone) refuse it, on the pool and with no advisory lock; SQLite `INSERT ... ON CONFLICT DO NOTHING` under the write lock and `BEGIN IMMEDIATE`, without a job slot; MongoDB the claim's pipeline upsert filtered on `owner_generation: {$exists: false}` with the row's status, `owner_generation: 0`, and E11000 (the unique `backup_id`) meaning a row exists; Cassandra the `backups_by_arn` lightweight transaction `IF NOT EXISTS`, then the LOCAL_QUORUM batch of the two list rows and the `backup_index_by_status` entry under the row's status (the batch `claim_backup` wrote inline, now shared), an unknown outcome or a `NotMatched` answer re-read at `LOCAL_SERIAL` and settled as inserted when the row read is this one (its entries then written, which completes an insert that stopped between its two writes), as existing when it is another; no row after a lost answer is sent once more, and no row after `NotMatched` answers `false` (a legacy in-catalog backup holds the ARN); a failed batch of the entries is sent once more before the error. The row is AVAILABLE from the transaction on, so a DeleteBackup between the transaction and the batch moves a status entry the batch has not written yet, and the batch then writes an AVAILABLE entry no later write removes; every reader of the status table re-reads the row and skips it, one wasted read each, until a sweep of orphan entries exists. Cassandra has no `backup_id` uniqueness, so its insert-if-absent is per ARN spelling: the same id under another region is another row there, as the trait doc says, and sync closes the gap itself by reading the row under both spellings of the manifest's ARN (this server's region and the manifest's) before the insert and skipping a backup the manifest's spelling, when it is another region's, has a row for, while a row under this server's spelling, the one the insert writes, is left to the insert (which answers `false` for it or, on Cassandra, completes its own torn insert), so through sync no account holds two rows of one backup id under the manifest's account, table and id on any backend (section 3, 4); on Cassandra a row of that id under another table that the rows read did not return (a row of another root, or one whose list entries are missing) is not seen by that read, and within one run the row check keeps a second prefix of an id the run adopted out. Sync reads the configured root's rows first, every status and every store id the catalog records for the root (`read_store_ids`, then `list_backup_rows` per id), keeping of each only the status, store id and table by account and backup id and the key of each AVAILABLE row of the configured store id (the backends collect the rows whole first, so the read peaks at about 22.5 KiB per row of a five-GSI table, section 7), and checks each listed prefix against that set before it reads anything under the prefix; the sync contract's "checks each prefix against the catalog as the listing reaches it" holds with the rows read once up front, and the insert-if-absent then decides against a CreateBackup that landed after the read and against rows of other roots. The listing is consumed as it streams: the store lists in key order, so a prefix is complete once the listing passes it, and sync holds that prefix's objects and the prefix of each manifest it listed; the filesystem store's own listing is the larger cost, measured in section 7. A listing that fails part-way keeps the rows inserted before it, does not read the prefix it was in, and makes no missing-manifest report. An insert that answers `false` is re-read to say which row it met: one that landed after the read, one of another store, account or table holding the backup id, or on Cassandra a row whose list entries are missing (a CreateBackup's claim that stopped between its two writes; sync's own insert that stopped there is completed by the next sync of the prefix). The prefix check is the dead-owner duty's: `reconciler::verify_prefix` reads the manifest and sidecar within their bounds, checks the digest and the reader's acceptance and every listed data file at its size from the listing sync already holds; the dead-owner probe compares the manifest's ARN with the row's apart from the region (a backup adopted from another region's manifest carries this server's region in its row), and sync compares the manifest's account, table and backup id with the path, adopts a manifest of another region under this server's region's spelling of the ARN, logging it (requests resolve it under either spelling), and skips one of another partition than `aws`, since every request refuses such an ARN before it reads a row. `layout.rs` gains `EXPORTS_COMPONENT`, `CHANGELOG_COMPONENT`, `classify_store_key` and `parse_store_prefix`: keys outside every backup prefix are passed over, the reserved components and a third component of another id shape (only the 14-digit shape carries a manifest) are reported as skipped and never read. The adopted row carries the manifest's name, ARN, size and item total and `source_table`, the id's instant as `created_at`, and the configured store's identity (section 4, PR 10). Rows of other roots are reported in part: sync prints the prefixes whose backup id a row of another root holds (the read under the ARN's spellings, or the insert's re-read). The rows of the configured root under another store id are not a case sync meets: a store whose rows carry another id than its file, or more than one id, is unhealthy, and sync refuses an unhealthy store. A listing of every root's rows needs `list_backup_rows_by_root`, which is #820's, so until then 4(g)'s "rows for another store are reported" holds for that one case (4(g)). Every sync takes the maintenance claim where it runs beside servers (PostgreSQL, MongoDB, Cassandra), plain sync with no wait for running jobs: its insert-if-absent orders it against a CreateBackup, and the claim orders it against `purge`, whose `purge_row` removes a dead CREATING row before it deletes the prefix, which can hold a complete manifest (a job that died after its manifest); a sync between the two steps would adopt bytes purge then deletes. The commands' claim (`HeldClaim`) is confirmed by the job's rule: each beat is bounded by one period (`MaintenanceClaim::heartbeat_within`, whose bound on SQLite starts once the beat holds the engine write lock), the claim reads lost once its last confirmed beat is two periods old, and a beat in flight counts only where it parks before its bound (`heartbeat_parks_before_its_bound`, SQLite); before each step that cannot be undone the command sends one beat of its own (`HeldClaim::recheck`) and stops if it finds the claim lost: sync's insert, `--prune-missing`'s removal, `--adopt-store`'s rewrite of the identity file, and purge's removal of a dead CREATING row and the deletion of that row's prefix. On MongoDB and Cassandra, which take a claim over on the heartbeat's age alone, a command stopped (SIGSTOP, a paused VM) past `stale_job_after_secs` can lose its claim to another command; it then stops before its next such step, so sync and purge never act at once while each reads its claim held. What remains is a stop after the re-check confirmed the claim and before the step's own call reached the catalog or the store, or with that call already sent: the call lands when the process resumes. While the claim is held the jobs refuse, as during purge (4(f)), and a sync killed while it holds the claim keeps the jobs of every instance refused until its heartbeat is `stale_job_after_secs` old. The restore's line for a row whose prefix lacks its sidecar says to put the prefix back; only a missing manifest names `--prune-missing`, the one case it removes. On SQLite it opens the catalog as `serve` does, under the serve lock (exit 6 beside a server, the refusal purge gives), and plain sync runs no dead-owner resolution: the lock keeps every server and every other command off the file, which is all the PR 10 rule that every writing command refuses to run beside a server asks, so nothing changes between the row read and the insert. The rows a stopped server left (a CREATING or DELETING backup, a RUNNING store-backed restore) are rows sync skips, and the next server start resolves the backups and resumes the restore; the dead-owner resolutions purge runs would fail that restore (`OWNER_DEAD`, under the claim no resume is spawned) and remove its target. `--prune-missing` and `--adopt-store` take the claim on SQLite too and run those resolutions before their wait, as purge does, so they fail such a restore and remove its target; every command that runs them prints what they did, the failed restores included. `--prune-missing` and `--adopt-store` also wait for running jobs as purge does; every sync installs purge's signal handling, checks it before each prefix and exits 1 "stopped" on a signal, releasing the claim it holds and its serving registration. `--prune-missing` removes each AVAILABLE row whose manifest is gone outright, through `BackupCatalog::forget_available_backup_row` (added here on the four backends: under the row's lock as `start_backup_delete` takes it, the status check and the in-use rule with a delete of the row, and on Cassandra its index entries after it), and deletes nothing in the store. The manifest probe sees only the newest AVAILABLE row's manifest, so a host-local root that holds the newest backup passes it while it lacks older ones; with no tombstone left, no deletion duty on the host that holds those bytes acts on them, but the row is gone for every instance: the backup is BackupNotFound on every instance until a sync on the host that holds the bytes adopts it again, which is the host-local acceptance line as amended here. The objects a removed row's prefix still holds stay until an operator deletes the prefix by hand: `purge` deletes only the prefixes of rows, and every later sync reports the prefix as not a complete backup. `--prune-missing` exits 1 when it leaves a row a later run may remove (a store or catalog call failed, the manifest's `head` included, which keeps the row; or a restore is reading the backup, "run the command again once the restore ends") and 0 when the rows it leaves have their manifest back or were moved by another writer. An unhealthy or `checking` store refuses plain sync and `--prune-missing`, exit 1, naming the problem and changing no row. Sync exits 1 when a prefix was skipped for a store read or catalog call that failed (the run is incomplete; the next run adopts it), and a startup sync that met one stays pending, its skips logged at WARN on every run and its other skips at INFO until a run has listed the whole root and logged its line of counts (a first run cut at the bound or stopped by an error does not count) and at DEBUG after, so a retried sync of a large root does not log a line per prefix each cycle; the command's summary line reads `incomplete` for such a run. So when the newest AVAILABLE row's own manifest is gone the store is unhealthy and neither command runs; the way out in this release is to put the prefix back (#820's `--forget-backup` is the other). The serving registration is named `extenddb backup sync`, and the catalog refusals exit 6 and 3 as the other commands' do (4(f), PR 10). `--adopt-store` is `store_identity::adopt`: with the claim held it rewrites `extenddb-store.json` with a plain `put`, the file's `catalog_id` and `backend` changed and its `store_id` and `created_at_ms` kept, reads it back, and syncs with the adopted identity; it logs an `extenddb::audit` intent line before the put and the audit line after the re-read with its outcome (adopted, or race lost), so a put whose caller stopped waiting is on record too. The re-read detects another adopter's put that landed between this put and the re-read (the store trait has no conditional replace); the command reads the file once more after the sync and fails naming the owner when it changed, which detects one that landed during the sync; a put after both reads is found by this catalog's next check. It refuses when this catalog's rows for the root carry another store id, and on a root this catalog owns it says so and syncs. A later check of the previous owner that can read its catalog finds the file naming another catalog, at the earliest one reconciler interval after its cycle in progress ends; until then it acts on the root, except where its deletion fence stops it. `StoreFence` re-reads the file (one `get` through the store's bounded calls, under its check bound and, on a filesystem root, in a call slot of its own, so a deletion holding the root's call slot does not hold it) right before the deletion of a backup's objects (`finish_backup_delete`, which DeleteBackup and the reconciler's DELETING duty run, one read before its two `delete_prefix` calls, the backup job's failure path, the FAILED-prefix and DELETED-prefix duties, purge, whose one read comes before its row change and its two deletions), before DeleteBackup's DELETING write and before purge's removal of a dead CREATING row or its DELETING write, before the reconciler removes a tombstone, before a CreateBackup's row (at the request) and again in the job right before its first put; when the file names another catalog it refuses, records `ForeignCatalog` so the store is unhealthy at once (and a file naming this catalog under another store id than the store's check settled, a root replaced by another root of the catalog, is refused and recorded as `IdMismatch`; DeleteBackup and CreateBackup answer the unhealthy store's `ValidationException`, and the cycle re-tests the store's health after each store duty and runs none after the refusal, the dead-owner duty included), a root with no file, a file the store refuses to serve and one that does not parse are refused and recorded as the next check would record them (`FileMissingWithRows`, `Denied`, `FileUnreadable`, `Validate`), so CreateBackup answers the unhealthy store's `ValidationException` for them too and a job refused before its first put is FAILED with no object; and a read that fails in transport refuses the call and leaves the health to the next check (CreateBackup answers `InternalServerError`), except at DeleteBackup's read before the DELETING write, where it is a store failure: the request writes DELETING and answers DELETED, as for a deletion the store does not answer (revision item 30), and the read before the deletion refuses it. A refusal before the row is changed leaves it as read, so DescribeBackup does not answer a refused DeleteBackup with DELETED; one at the read before the deletion (an adoption between the two reads) leaves the row DELETING with nothing deleted and DeleteBackup answering DELETED (revision item 30), which this catalog's reconciler finishes only if this catalog owns the root again. The restore deletes nothing in the store, so it has no fence, and neither has `--prune-missing`'s removal of a row, which deletes no object. A write already past the fence's read lands, so every server of the catalog the file names is stopped before an adoption, and a new in-memory server is started only after the old one has exited; the admin guide, the refusal and the command's output say so (4(c)). For `backup sync` a defaulted SQLite root whose file names another catalog while this catalog has no rows for it is configured and unhealthy (`Caller::SyncCommand`) instead of the warning that leaves it unconfigured (item 29), so `--adopt-store` can adopt the root a catalog re-created by `destroy` and `init` finds; the server and purge keep the warning (4(c), section 7, PR 7). The identity file gains `ephemeral_catalog` (`#[serde(default)]`, so the format stays version 1 both ways), written `true` by a server whose SQLite database is in memory (`database_file()` names no file: the state, not the `sqlite-memory` cargo feature) and `false` by every other writer, an adoption included. The adoption of a root an in-memory catalog owned is part of `store_identity::establish` when this catalog is itself in memory and the file carries the marker, and never of `recheck`, so two in-memory servers on one root do not take it from each other each cycle; a root whose file names a durable catalog stays unhealthy for an in-memory server, as 4(c) requires. The startup sync runs on every start of an in-memory catalog with a store configured, not only after an adoption (on a fresh root it is one listing): for a filesystem store in `serve` before the routes serve; for an S3 store, and for a startup sync not finished, in the reconciler's cycle after a healthy check. Every run is bounded at 60 s (`sync::STARTUP_SYNC_TIMEOUT`), in `serve` and in the cycle, where the duties after it wait at most that bound; a run that stops before it listed the whole root, reaches the bound (the rows it inserted stay, but every run lists the root from its first key, so the sync completes only while listing the whole root fits in the bound; past it the sync stays pending and is retried every cycle), or skipped a prefix for a store or catalog error stays pending. On both store kinds the run's listing is collected whole before its first key reaches the run (`BoundedCalls::list` collects the stream into one `Vec` of `ObjectMeta`, one per object of the root, held in memory until the run ends), so a listing that outlasts the bound reaches no prefix; on S3 the abandoned listing's requests end with the dropped future. On a filesystem store a run abandoned at the bound does not stop its listing (the walk has no cancel flag): it runs to its end on the blocking pool, holding the listing it builds and the root's call slot, marked in flight, so no second walk starts beside it: until the walk ends, each cycle's check is refused at once on the mark (the refusal names the caller whose call has not returned, here the startup sync's) and records the store `unreachable`, so CreateBackup, DeleteBackup and restores are refused, the deletions wait, and the sync, which runs only on a healthy store, does not run; the first cycle after the walk ends checks the store healthy and runs the sync again, from the first key. A root too large to list within the bound is never synced by an in-memory server, which an operator resolves with a smaller root or a durable catalog. `serve` answers nothing, `/health` included, until the startup sync before the routes ends, while a signal in that time ends the process at once. Until a run finishes, requests for the earlier backups answer `BackupNotFoundException`, `/health` possibly already healthy; the log line that reports the adopted count ends that window, and for a prefix that line reports as having met a store or catalog error, a later cycle's line does. It takes no claim: the process is the catalog's only user (4(c), PR 7, PR 10). Every `extenddb backup` command refuses an in-memory SQLite database before it opens a catalog, exit 1, with a sentence saying why (a location that does not parse is reported with its parse error instead, as `serve` reports it): it would open a new, empty catalog that ends with the command, and a writing command would hand an adopted root to a catalog nothing can open again; this is the sync contract's "the backup CLI commands do not exist there" (PR 10). The matrix (`devtools/backup-restore-matrix`) copies the source's prefix under the target's root and runs `extenddb backup sync` on the target (on SQLite with the target's server stopped) in place of the hand-written row, so a cell proves the adoption and the restore; `--adopt-store` points the target's store at the source's root, adopts it and checks that the source's server then reports the store owned by another catalog, and the nightly runs that cell from SQLite to PostgreSQL and back (PR 9b). 4(c), 4(e), 4(f), 4(g), decision 6, section 7's memory paragraph, the PR 10 contract, 5.3's failure step (`OTHER_STORE`'s reason, whose text this PR changed) and the testing summary's kill-point line (a Cassandra kill point adopts its copied prefix with `backup sync`) are edited with this item. diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 9c53a53e0..b768e0a76 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -124,7 +124,7 @@ Behaviour stated as measured, and every number below, was observed with release | Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. For a store-backed restore the registration takes the same lock on the `backup_index` row instead of the legacy `backups` row (PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB a `$inc lock_seq` on the `backup_index` document, Cassandra the `active_restores` lightweight transaction on the backup's partition), the same ordering guarantee, so a legacy restore and a store-backed restore of two different backups never contend and a `DeleteBackup` of a store-backed backup meets only store-backed registrations. Two concurrent registrations of one backup leave one restore target on Cassandra as on MongoDB: the registration's lightweight transaction (`IF backup_status = 'AVAILABLE' AND active_restores = null`) is linearized on the backup's partition, so the second finds `active_restores` non-null and is `BackupInUseException`; the in-use pre-read is advisory *(storage-level tests on both)*. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | | Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. A store-backed restore (one run from a `[backup.store]`) is instead resumed, not discarded: its owner's death leaves a RUNNING restore job record with a stale heartbeat and the target `CREATING`, and the reconciler of any instance with the configured store takes the record over under a fresh owner generation once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), stamping a fresh heartbeat in the same write so no second instance takes it too, and resumes the load from its recorded progress (per data file on PostgreSQL, per batch on SQLite, MongoDB and Cassandra), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; the target reaches `ACTIVE` with the backup's full item count, its secondary indexes complete: a resumed Cassandra restore re-applies its last batch as idempotent upserts that rewrite the same base and index rows (the write path rewrites an unchanged item's GSI and LSI rows rather than deleting them, #424). A resume that finds no recorded progress and a target holding more items than the backup's total is `FAILED` with reason `target_not_empty` and the target is left in place: the rule protects a catalog restored from a dump that names a restore which finished and was written to since. A target holding no more than the total with no recorded progress is a first data file applied in part: on Cassandra, which records a batch's progress after its writes, an ordinary crash before the first progress record (the first 10,000 items or 32 MiB) leaves written items and no progress, and the resume re-applies the first file with idempotent upserts and reaches `ACTIVE` with the exact count *(from the code; the storage and engine cells)*. MongoDB records each batch's documents and its progress in one transaction, so a mid-file crash leaves no items the record does not account for and resumes to `ACTIVE`; only a crash before the first batch commits leaves an empty target, which resumes from the start. A record that one instance's reconciler has taken over five times without the restore ending (a resume whose final count or progress read fails every time, or that stops with the record RUNNING) is failed on the next takeover with reason `owner dead: taken over 5 times by one instance without the restore ending` and its target removed, so a record that can never finish does not hold its name and its backup for as long as the server runs; the count is per instance and a restart starts it over *(from the code)*. The target stays `CREATING` with a `FAILED` record: `DeleteTable` on it is `ResourceInUseException`, `DeleteBackup` of its backup and a second restore of that backup are `BackupInUseException`, and the reconciler does not remove it; `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record, after which `DeleteBackup` and a new restore into the name succeed. A legacy (in-catalog) restore is still failed and its target removed. Measured *(binary-level, this release)*: SIGTERM during a store-backed restore left the record RUNNING and the restarted server resumed it to `ACTIVE` with every item (on PostgreSQL after the 45 s stale window, on SQLite at once); a SIGKILL at each restore kill point left the target to finish with the exact count (the base table's and the GSI's) or, for a failing restore, gone, except at the first reach of the Cassandra point that pauses before a batch's first progress record: on Cassandra (killed after the first batch's 10,000 upserts, before their record) the resume of that build answered `target_not_empty` and left the target `CREATING`, and `extenddb backup remove-target` then removed it, after which `DescribeTable` was `ResourceNotFoundException` and `DeleteBackup` succeeded; this release resumes that case with upserts to `ACTIVE` *(from the code; the nightly kill-point job runs it)*. On MongoDB a SIGKILL after a file's `insert_many` leaves the batch's items and its progress committed together, so the resume continues to `ACTIVE` rather than `target_not_empty`. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | | Point-in-time recovery | 35-day window: `UpdateContinuousBackups` enables continuous backups, and `RestoreTableToPointInTime` restores to any second in the window | Not supported; use on-demand backups (`CreateBackup` and `RestoreTableFromBackup`) instead. `DescribeContinuousBackups` reports `ContinuousBackupsStatus: ENABLED` (the service reports this unconditionally) with `PointInTimeRecoveryStatus: DISABLED` and no restorable-time window. `UpdateContinuousBackups` with `PointInTimeRecoveryEnabled: true` returns `ContinuousBackupsUnavailableException`; with `false` it returns the same description. `RestoreTableToPointInTime` checks the source table, then returns `PointInTimeRecoveryUnavailableException` ("Point in time recovery is not enabled for table ''"), the exception the service models on that operation. A table that does not exist is `TableNotFoundException` ("Table not found: ") on all three operations, as on the service. Same on all four backends | -| Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
//`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | +| Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
//`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (`extenddb backup sync` gives a new catalog a row for each backup under the root, and `--adopt-store` first makes it the owner of a root another catalog owned). A prefix copied from a server of another region, whose manifest spells its ARN in that region, is adopted under this server's region: its row carries the ARN this server issues, so `ListBackups` names this server's region (every `ListBackups` answer measured on the service names the endpoint's region) and `DescribeBackup`, `RestoreTableFromBackup` and `DeleteBackup` find it under either region's spelling, as they find the backups this server made and as the service finds a backup under another region's spelling of its ARN (measured; `DescribeBackup` answers with this server's spelling, as it does for this server's own backups, where the service echoes the spelling sent); the manifest keeps the spelling it was written with, and so does the source table's ARN, which `DescribeBackup`'s `SourceTableDetails.TableArn`, `ListBackups`' `BackupSummaries[].TableArn` and the restore's `RestoreSummary.SourceTableArn` answer in the manifest's region (a backup this server made names this server's region in every ARN it answers); the restore and the reconciler accept a manifest whose ARN differs from the row's in the region alone. Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` and `TableSizeBytes` in `SourceTableDetails` are the table's figures as last counted by the stats pass while `CREATING`, which is 0 and 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact figures once `AVAILABLE`: the items written and the sum of their DynamoDB sizes; the service reported 0 and 0 in every fresh-table backup measured). `CREATING` lasts the whole upload (the durations below), and while it lasts `DeleteBackup` and `RestoreTableFromBackup` of the backup answer `BackupInUseException` "Backup is being created: " on all three backends; the 16 service backups measured were `AVAILABLE` 12 to 90 ms after `CreateBackup` answered, so a client that deletes or restores a backup right after creating it polls `DescribeBackup` first here. The backup is the table as it stood when its snapshot opened, which is inside the request but after the request's checks (the table's state, the store's health, the maintenance claim) and after any wait for a job permit, up to 30 s (below): every write acknowledged before the open is in the backup and none acknowledged after it, so a `PutItem` acknowledged while the request waited for a permit is in the backup (measured: a write acknowledged 1.01 s after a `CreateBackup` that waited for a permit was in it, 2 of 2 runs), and `BackupCreationDateTime` is the open's millisecond. Measured on the service: a restored backup contained every write acknowledged before `CreateBackup` was sent and the 18 writes acknowledged in the 0.1 s up to `BackupCreationDateTime`, and none of the 11,754 acknowledged in the 75 s that followed, so the service takes the table at an instant inside the request as well. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). A `CreateBackup` still waiting in its snapshot open or row insert when the server stops writes no row; it is answered `ServiceUnavailable` "The server is shutting down. Retry the request." only after a registration loss, where the jobs' token is cancelled while the listener still serves, since on SIGTERM the listener's 5 s connection drain closes the connection before that token is cancelled (measured: the connection closed 8.2 s after the request was sent). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"), the account checked before the region, so another account's ARN in another region is `AccessDeniedException` (measured on DynamoDB for CreateBackup and ListBackups, twice each); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). Without a store, and on Cassandra, a table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s): the in-catalog format writes no store prefix. With a store, a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it with `ValidationException` before it opens a snapshot or writes a row ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"). Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | | `DeleteBackup` response | The full `DescribeBackup` description with `BackupStatus` `DELETED`, including `SourceTableDetails.ProvisionedThroughput` (a required member) and `SourceTableFeatureDetails` (measured) | The same on all four backends: the engine reads the backup's recorded definition before the delete and fills the description as `DescribeBackup` does, with `BackupStatus` `DELETED`. A backup whose definition this build cannot read or that does not parse is still deleted and described without feature details (logged at warn), which means a damaged `PROVISIONED` backup is described with on-demand `ProvisionedThroughput` `{0, 0}`: ExtendDB's choice, deletability over the throughput member of a row that is being removed, where `DescribeBackup` on the same row is an `InternalServerError`; a failure to reach the database fails the request and deletes nothing. | | A backup maintenance command holding the catalog's claim | Not applicable: the service has no maintenance command | While `extenddb backup purge` holds the catalog-level maintenance claim, `CreateBackup` and `RestoreTableFromBackup` on every instance of the catalog answer HTTP 400 `LimitExceededException` "A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes", after every other check of the request (a request refused on its own terms, a bad table name or override, gets its own error) and before the job permit. The refusal ends when the command releases the claim, or when a command that died goes stale after `[backup] stale_job_after_secs`. | diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index 23649d7a1..39167fb48 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -134,7 +134,7 @@ Source: [AWS DynamoDB Service Quotas](https://docs.aws.amazon.com/amazondynamodb | `ListBackups` page size | 100 | Enforced | `Limit` 1 to 100, default 100; more rows come back through `LastEvaluatedBackupArn` and `ExclusiveStartBackupArn`. Each request reads every backup of the account (or of the table, with `TableName`) and pages in memory, so walking N backups reads about N²/100 rows. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list before paging; a bound before the epoch, or a lower bound above the upper bound ("Time range lower bound must be less than or equal to upper bound"), is `ValidationException` | | Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where CreateBackup writes its backups on PostgreSQL, SQLite and MongoDB (Cassandra, any backend without a store, and SQLite on a platform where the serve lock is not taken (not Unix), write the in-catalog format) and where restores and deletes of store-backed backups read: a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses CreateBackup where it writes to the store and the restore and delete of store-backed backups with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | | Backup data file | Not specified | N/A | A data file holds one item per line as DynamoDB JSON, gzip-compressed, in key order on a table whose key schema is orderable (`item_order` in the manifest); a line is at most 32 MiB (`MAX_LINE_BYTES`, which holds the largest line a 400 KB item can expand to, 27.7 MB); a file closes at the first item that takes its uncompressed bytes to or past `[backup] data_file_target_bytes` (default 64 MiB, minimum 32 MiB, the line cap), so it exceeds the knob by less than one line. The snapshot streams on PostgreSQL and SQLite page the table by key and cut each page at 64 MiB of item text (`SNAPSHOT_FETCH_BYTES`, a constant), so a page holds at most that plus one row; MongoDB's batches are bounded by the server's 16 MiB reply | -| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root and restores into the backend that took it, and into another backend once `backup sync --adopt-store` (a later release) has indexed the backup in the target's catalog; this release restores the backups its own catalog indexes (the differences document's "Where backups live" row). Vector indexes are restored where the target supports them, and a backup of a multi-part-key table (the `limits.allow_multipart_table_keys` preview) restores with its item set intact (the engine's `a_multi_part_key_manifest_is_registered` and `tests/test_backup_job.py`'s multi-part cell, which restores such a backup on a server with the preview enabled and compares the restored item set with the source's); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | +| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root and restores into the backend that took it, and into another backend once `extenddb backup sync` has given the backup a row in the target's catalog (the differences document's "Where backups live" row). Vector indexes are restored where the target supports them, and a backup of a multi-part-key table (the `limits.allow_multipart_table_keys` preview) restores with its item set intact (the engine's `a_multi_part_key_manifest_is_registered` and `tests/test_backup_job.py`'s multi-part cell, which restores such a backup on a server with the preview enabled and compares the restored item set with the source's); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | | Point-in-time recovery | 35-day window | N/A | Not supported: `UpdateContinuousBackups` refuses to enable it (`ContinuousBackupsUnavailableException`) and `RestoreTableToPointInTime` refuses every restore (`PointInTimeRecoveryUnavailableException`); use on-demand backups (see differences-from-dynamodb.md) | | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, `VectorIndexOverride` | Partial | All but `VectorIndexOverride` are applied as the service applies them: an index override list replaces the backup's indexes of that kind (empty restores none, a subset restores those, a GSI entry's throughput is the restored index's), `OnDemandThroughputOverride` is the restored table's on-demand throughput on an on-demand restore. Refused with `ValidationException` before the target is created: an index the backup does not have or a changed index definition, a throughput member contradicting the billing mode, `{0, 0}`, a `PROVISIONED` restore of a GSI without throughput ("Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index "), a GSI entry's `OnDemandThroughput` on a `PROVISIONED` restore (on an on-demand restore it is accepted and not recorded), and `VectorIndexOverride`. See differences-from-dynamodb.md | diff --git a/docs/manuals/03-usage-guide.md b/docs/manuals/03-usage-guide.md index f2b4c96e7..7005f58ed 100755 --- a/docs/manuals/03-usage-guide.md +++ b/docs/manuals/03-usage-guide.md @@ -571,7 +571,7 @@ extenddb reproduces DynamoDB error responses exactly. Common errors: | PartiQL | ✗ (not planned) | | DAX | ✗ (not applicable) | | Global Tables | ✗ (future) | -| On-demand backups (CreateBackup / RestoreTableFromBackup) | ✓: CreateBackup to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB (in-catalog without one and on Cassandra); RestoreTableFromBackup from the store into every backend (a filesystem or S3 store; a backup taken on one backend restores into another once `backup sync --adopt-store` (a later release) has indexed the backup in the target's catalog; this release restores the backups its own catalog indexes), and of in-catalog backups | +| On-demand backups (CreateBackup / RestoreTableFromBackup) | ✓: CreateBackup to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB (in-catalog without one and on Cassandra); RestoreTableFromBackup from the store into every backend (a filesystem or S3 store; a backup taken on one backend restores into another once the target's catalog has a row for it: copy its prefix under the target's root and run `extenddb backup sync`, or adopt the source's root with `extenddb backup sync --adopt-store`, which moves the root to the target's catalog: stop the source's servers first, and they then report the store owned by another catalog), and of in-catalog backups | | Point-in-time recovery | ✗ (`UpdateContinuousBackups` refuses to enable it; see differences-from-dynamodb.md) | | Import/Export (local filesystem) | ✓ (FileSource instead of S3) | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 675763c89..4da1c0d80 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -140,7 +140,7 @@ Backup and restore jobs are bounded per server process by `[backup] max_concurre | Key | Default | Description | |-----|---------|-------------| -| `path` | `extenddb.sqlite` | Database file (`init --sqlite-path` writes it); `:memory:` for an ephemeral database | +| `path` | `extenddb.sqlite` | Database file (`init --sqlite-path` writes it); `:memory:` for an ephemeral database, whose catalog is new at every start: with `[backup.store]` configured the server adopts the root an earlier in-memory catalog owned and syncs its backups into the new catalog before it serves, within 60 s (an S3 store in the reconciler's first cycle; a sync not finished in 60 s, or one that met a store or catalog error on a prefix, runs again in the next cycle). Every run lists the root from its first key, so the startup sync completes only while listing the whole root fits in 60 s; past that it stays pending, is retried every cycle, and the backups beyond what a run reaches are not served. A run holds the whole listing of the root in memory (one entry per object) until it ends, and a listing that outlasts 60 s reaches no prefix. On a filesystem store a run abandoned at the 60 s bound does not stop its listing: the walk of the root runs to its end on the blocking pool, holding the listing it builds, and no second walk starts beside it; until it ends, every check of the store is refused at once ("an abandoned startup sync's store call has not returned") and records the store `unreachable`, so CreateBackup, DeleteBackup and restores are refused and the sync does not run; the first reconciler cycle after the walk ends finds the store healthy and runs the sync again, from the first key. A root too large to list within 60 s is never synced by an in-memory server: give it a smaller root, or a durable catalog. On a filesystem store the server does not answer, `/health` included, until the startup sync ends (a liveness probe with a short timeout restarts such a server on a large root, each start syncing from the first key), and a signal then ends the process at once. Until the log line `startup sync of the in-memory catalog: N backup(s) adopted` the earlier backups answer `BackupNotFoundException`, although `/health` may already report the store healthy; a prefix that line reports as having met a store or catalog error stays so until a later cycle's line. Each run names the prefixes it skipped for a store or catalog error at WARN; until one run has listed the whole root and logged its line of counts, each run also names every other prefix it skipped at INFO (so the reasons reach the log when the first run was cut at the bound or stopped by an error), and a later run of a sync left pending logs those at DEBUG, so a root of many backups with one that fails costs one WARN line and the line of counts per cycle. A server killed inside a DeleteBackup (between the DELETING write and the manifest's deletion), or restarted after a DeleteBackup whose manifest deletion failed, serves that backup again at its next start, since the startup sync adopts the complete prefix; a prefix a stopped server had not finished writing or deleting (a killed CreateBackup, a failed backup whose prefix deletion had not finished, a deletion cut after its manifest) stays under the root with no row to drive its deletion (the next start's sync skips a prefix without its manifest); the startup sync names each such prefix in its log line `backup store startup sync: not adopted: not a complete backup: ...`. Delete such a prefix by hand, since every `extenddb backup` command refuses an in-memory database, but only one that has no `extenddb-manifest.json` and whose backup `ListBackups` does not return: a CreateBackup the running server is still writing has no manifest until its last put either, and its row is in `ListBackups` (status CREATING). Start a new in-memory server only after the old one has exited: one started beside it adopts the root while the old one still acts on it until a later check finds it adopted, its deletion fence (the identity paragraph) excepted. A root that an in-memory server of an earlier development build wrote has an `extenddb-store.json` without `ephemeral_catalog` (no release wrote one), which reads as a durable catalog's: the server finds the root owned by another catalog, its store stays unhealthy, and no command can adopt it. Remove that `extenddb-store.json`; the next start writes a new one (a new store id) and adopts the backups under the root. Every `extenddb backup` command refuses it | | `pool_size` | `10` | Maximum connections. Writes are serialized by the engine whatever the size. A file database raises `pool_size = 1` to 2 and logs it at info (`SQLite pool_size 1 raised to 2 for a file-backed database: ...`), because a backup holds one reader connection for its whole duration while its batches write through another. Use 3 or more so a running backup does not contend with ordinary requests for connections | #### [backup] @@ -171,9 +171,31 @@ When the section is absent, the backend decides: on file-backed SQLite it is a f Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), CreateBackup (where it writes to the store) and the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check's store calls are bounded (5 s for a filesystem root, 30 s for a bucket); the catalog's own reads (the catalog id, the rows' store ids, the newest AVAILABLE row) run beside the calls and not under the bound, so a slow catalog costs the check time and never marks the store unhealthy. A store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned check's call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), a root's calls run one at a time so that calls started before the first reached its bound (two `DeleteBackup` requests together, a `DeleteBackup` beside a check) wait for it under the bound, then have the bound for their own call (a slow call behind another finishes, where one bound over both would abandon it), and park no thread of their own, and `stop` or SIGTERM still ends the server, the runtime giving that thread 5 s before the process exits without it (after the job wait of up to 30 s when a backup job is running; see Stopping). The deletions `DeleteBackup` and the reconciler run against the store go through the same bound and rule: `DeleteBackup` answers the description with `BackupStatus` `DELETED` once its DELETING write has committed, and a deletion the store does not answer within the bound, or that leaves an object under the prefix, leaves the row DELETING for the reconciler and is logged, never an error to the client. A default SQLite root whose startup checks did not finish starts `checking` with its rule kept: a first check that fails while the catalog has no rows for it is the same warning and leaves the store unconfigured. CreateBackup on Cassandra is not refused by an unhealthy store: its in-catalog create touches no store. -The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no AVAILABLE backup for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id, and a file copied from one directory to the other gives both one id), and the manifest probe has nothing to probe: a backup one instance completed under its own directory looks, to the other instance's reconciler, like a dead owner's row with no manifest, which it marks FAILED, and the first instance's restart then deletes the only copy. Point every instance at one shared root before the first backup; copying `extenddb-store.json` between directories defeats the check rather than satisfying it. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. +The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until `extenddb backup sync --adopt-store` makes the new catalog its owner (below), the file is removed or a new root configured (on the default SQLite root, that is the logged warning above for the server, which keeps CreateBackup on the in-catalog path, while `backup sync` handles that root as configured, so `--adopt-store` adopts it). The default root, `backups` beside the database file, is shared by every database file in that directory: `--adopt-store` on it takes the root from whichever catalog wrote its file, so give each database its own `[backup.store]` when several share a directory. A server on an in-memory SQLite database writes the file with `ephemeral_catalog: true`; at its next start it adopts a root whose file names another catalog and carries that marker, and no other, so it never takes a durable catalog's root, and then syncs the root's backups into its new catalog (the `path` row above: until that sync's log line, the earlier backups answer `BackupNotFoundException`). A catalog whose root another catalog adopted (by `--adopt-store`, or a second in-memory server's start) keeps acting on the root until a later check that can read the catalog finds it owned by another catalog, at the earliest one reconciler interval after the cycle in progress ends (a check whose catalog read fails leaves the health as it was), with one exception, its deletion fence: right before it deletes a backup's objects (DeleteBackup, the reconciler's deletions, a failed backup job's cleanup, `purge`), removes a FAILED or DELETED row, before a CreateBackup's row (at the request) and again in the backup job right before its first object, and also before DeleteBackup and `purge` change the row (`purge` reads it once before the row change and its two deletions), a server or `purge` re-reads `extenddb-store.json`, and when the file names another catalog, names this catalog under another store id than the one the store's check settled (a root replaced by another root of this catalog), or the root has none, it deletes nothing and refuses the request. The read records on the store what the next check would record, so the store is unhealthy at once: `owned by another catalog`, `identity mismatch`, `identity file missing` for a root with no file, and, for a file the store refuses to serve or that does not parse, what the check would report (`failed validation` for a file a filesystem root may not read, such as one at mode 000; `permission denied` for an S3 denial; `identity file unreadable` for one that does not parse; `failed validation` too for an S3 server error the SDK's retries did not clear, such as a 503 `SlowDown`, which leaves the store unhealthy until a check passes (the next runs one reconciler interval later)); a CreateBackup refused there answers the unhealthy-store `ValidationException`, and one whose job meets the refusal before its first object is `FAILED` with no object written. The read has a call slot of its own on a filesystem root, so a DeleteBackup's deletion of a prefix of many objects does not hold a CreateBackup's read behind it. Only a read that fails in transport (the store unreachable, the read past the store's 5 s bound) records nothing and leaves the decision to the next check: CreateBackup then answers `InternalServerError`, and DeleteBackup takes it at the read before the row is changed as it takes a store that does not answer: it writes DELETING and answers `DELETED`, the read before the deletion refuses it, and the row stays DELETING for the reconciler, whose deletion runs the fence again. `--prune-missing`'s removal of a row is not fenced: it deletes no object. A refusal at the read before the row is changed (the file names another catalog or another store id) leaves the row as it was, so DescribeBackup still answers AVAILABLE. One at the read before the deletion (an adoption landed between the two) leaves an AVAILABLE row DELETING with nothing deleted (DeleteBackup and DescribeBackup there answer `DELETED`), which this catalog's reconciler finishes only if this catalog owns the root again, and a dead CREATING row that `purge` already removed has its prefix kept. The read narrows the window to the gap between it and the deletion; on S3 that gap is a listing and a `DeleteObjects`, each under the SDK's retries, so seconds when the store throttles. A check that read the file before the adoption and ends after a fence's refusal records the store healthy again until the next cycle's check: each deletion stays fenced in that time, but a restore is accepted. A backup job of the previous owner that fails after the adoption leaves its partial prefix (no manifest, and no row of the new owner) under the root, which no duty or command of the new owner removes. A write that has already passed that read still lands, so stop every server of the catalog a file names before adopting its root. While the catalog has no AVAILABLE backup for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id, and a file copied from one directory to the other gives both one id), and the manifest probe has nothing to probe: a backup one instance completed under its own directory looks, to the other instance's reconciler, like a dead owner's row with no manifest, which it marks FAILED, and the first instance's restart then deletes the only copy. Point every instance at one shared root before the first backup; copying `extenddb-store.json` between directories defeats the check rather than satisfying it. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. -`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge`, and `sync` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup sync`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. + +`extenddb backup sync [--adopt-store] [--prune-missing]` rebuilds the catalog's rows from the configured store: after a lost or re-created catalog, or after a backup's prefix was copied under the root from another deployment or another backend. It reads the catalog's rows for the root (of every status and every store id they carry), then lists the whole root and handles each `/
//` prefix as the listing passes it, checking it against those rows before it reads anything under it. Sync holds one prefix's object list at a time and the prefix of each manifest it saw; a filesystem store builds its listing of the whole root in memory first (about 72 bytes plus the key per object), an S3 store lists a page at a time. The report keeps an entry per prefix for the whole run, and it lists every prefix the catalog already has a row for among the skipped ones, so on a root of small backups the per-prefix term dominates: about 350 bytes per prefix at one object per prefix (62 MiB of peak RSS at 50,000 such prefixes and 112 MiB at 200,000, against 45 MiB for an empty root; about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build with a filesystem root and no rows. Of the catalog's rows it keeps each row's status, store id and table, and the key of each AVAILABLE row of the configured store, but every backend reads the rows whole before sync sees the first, so the read is the larger cost and grows with the rows of the root, tombstones within their retention included: about 22.5 KiB per row of a table with five GSIs (1,452 bytes of source table description per row; 610 MiB of peak RSS at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none; 148 MiB at 50,000 rows whose description is `{}`), measured on a SQLite debug build on a 16-core x86_64 Linux host with glibc's default arenas. `backup list` peaks the same over those rows. A prefix with no row gets an AVAILABLE row under the configured store's identity when its manifest and sidecar verify, every data file the manifest lists is under the prefix at its recorded size, and the manifest's ARN names the prefix's account, table and backup id; an ARN of another region is adopted under this server's region and logged (requests find it under either spelling), and one of another partition than `aws` is skipped, since every request refuses such an ARN. The row carries the manifest's backup name, size, item count and source table, and the backup id's instant as its creation time. It never adopts a prefix with a row of any status (CREATING, DELETING, DELETED, FAILED or AVAILABLE), one whose manifest, sidecar or data files are missing or do not match, one whose manifest names another backup, or anything under `exports/` or a table's `changelog/`, and prints each with its reason. Before the insert it reads the row under both spellings of the ARN (this server's region and the manifest's), and the insert happens only if no row exists: a CreateBackup whose row landed after sync read the rows keeps its row, as does a row of another root, account or table holding the backup id, and sync changes no existing row. It also prints the AVAILABLE rows whose manifest the listing does not hold. It exits 0 once it ran, skipped prefixes included, and 1 when a prefix was skipped because a store read or a catalog call failed (the run is incomplete, and its summary line says `incomplete`; a second run adopts it), when the listing failed part-way (the backups adopted before the failure keep their rows), when a signal stopped it, or when `--prune-missing` left a row it may remove on a later run (a store or catalog call failed, its manifest's `head` included, or a restore is reading the backup: run the command again once the restore ends). A row `--prune-missing` leaves because its manifest is back or another writer moved it is reported, and the exit stays 0: + +``` +=== extenddb backup sync === +Store: filesystem (store id ) + +--- Adopted (an AVAILABLE row inserted for each): + + +--- Skipped (not adopted): + /
//: not a complete backup: no manifest + +--- AVAILABLE rows whose manifest is not in the store (the rows were AVAILABLE when the catalog was read): + (none) + +=== extenddb backup sync complete: 1 backup(s) adopted, 1 prefix(es) skipped, 0 AVAILABLE row(s) whose manifest is gone === +``` + +Sync needs the store healthy: on an unhealthy store it refuses, exit 1, naming the problem and changing no row. That covers an empty root under existing rows, a root whose file carries another store id and a root without the newest AVAILABLE row's manifest, which is what an unmounted volume shows. When the newest AVAILABLE row's own manifest is gone, the store is unhealthy for the same reason, so plain sync and `--prune-missing` both refuse; put the prefix (or its manifest) back and run the command again. On PostgreSQL, MongoDB and Cassandra every sync takes the maintenance claim and heartbeats it, so no `purge` runs between its read of the rows and its insert while the claim is held. Each command that holds the claim reads it lost once no heartbeat confirmed it for two heartbeat periods, and sends one heartbeat of its own right before each step it cannot undo (sync's insert, `--prune-missing`'s removal, `--adopt-store`'s rewrite of the identity file, `purge`'s removal of a dead CREATING row and of that row's prefix), stopping when the claim is lost: on MongoDB and Cassandra a command stopped for `stale_job_after_secs` (a suspended process, a paused VM) can have its claim taken over by another, and then stops before that step. A call the command had already sent when it was stopped still lands when it resumes. Plain sync waits for no job, and while it holds the claim CreateBackup and RestoreTableFromBackup are refused, as during `purge`. A sync killed while it holds the claim (`SIGKILL`, a lost host) leaves it held until its heartbeat is `stale_job_after_secs` old (300 s by default): until then every instance refuses CreateBackup and RestoreTableFromBackup. On PostgreSQL the claim is a session's advisory lock, so the next maintenance command (the sync or `purge` run again) takes a killed command's claim over at once and ends that window; MongoDB and Cassandra take it over only once it is stale. `--prune-missing` and `--adopt-store` also wait for running jobs as `purge` does. A first Ctrl-C or `SIGTERM` stops any sync before its next prefix (or the wait at once), releases the claim it holds and exits 1; the backups adopted before the stop keep their rows. `--prune-missing` removes each AVAILABLE row whose manifest is gone outright, one line per row, and deletes nothing in the store: no DELETING or DELETED row is left for a deletion duty to act on, so on host-local roots (each instance's root at one path holding other bytes) a prune on the instance that lacks a backup's bytes removes its row for every instance of the catalog and leaves the bytes on the other host: the backup answers `BackupNotFoundException` on every instance, the one holding the bytes included, until a sync on that instance adopts them again. A row a restore is reading, or one another writer moved meanwhile, is reported and left (the first exits 1). The objects a removed row's prefix still holds stay until an operator deletes the prefix by hand: `purge` deletes only the prefixes of rows, and every later sync reports the prefix as not a complete backup. `--adopt-store` is the way a durable catalog takes a root whose `extenddb-store.json` names another catalog (plain sync refuses such a root, naming the option; a server on an in-memory database adopts, at startup, a root another in-memory catalog wrote, as the identity paragraph says): under the claim it first runs the check the server would run on the root once this catalog owns it (this catalog's rows for the root against the file's store id, and the manifest of this catalog's newest AVAILABLE backup of the root) and refuses, with nothing written and the file still naming its owner, a root that check fails (a backup of this catalog whose manifest is gone from the root, say); then it logs its intent under `extenddb::audit`, rewrites the file's catalog id and backend, keeping the store id and creation time, reads the file back and logs the outcome (adopted, or the race lost when another adopter's write landed between the two), and syncs. Each store call runs under the store's 5 s bound (the catalog's reads and the claim's heartbeat do not count against it), and a store that stops answering ends the command with exit 1 naming the step that did not finish (the first read, the rewrite, whose outcome the audit log's intent line then leaves open, or the re-read). Every sync, plain or adopting, reads the file once more at the end and fails, exit 1, naming the owner, when another catalog adopted the root during the run: the rows it inserted are then rows of a root this catalog does not own, inert until this catalog adopts the root again. A write that lands after that read is found by this catalog's next check. It refuses when this catalog's rows for the root carry another store id, and on a root this catalog already owns it says so and syncs. Stop every server of the catalog the file names before running it: the previous owner finds the store owned by another catalog only at a later check that can read its catalog, and until then its deletion fence (the identity paragraph) refuses each deletion and CreateBackup once it re-reads the file, but a write already past that read lands. A second maintenance command while one holds the claim fails at once. On SQLite every form takes the serve lock: stop the server first (beside a running server it exits 6). Plain sync there resolves nothing the stopped server left: the lock already keeps every server and command off the file, so a CREATING or DELETING backup and a RUNNING restore the server left are rows sync skips, and the next server start resolves them and resumes the restore. `--prune-missing` and `--adopt-store` take the claim and first resolve what the stopped server left, as `purge` does, which fails a store-backed restore the server left RUNNING and removes its target; each prints what it resolved. The rows of other roots are reported in part: a prefix whose backup id a row of another root holds is skipped naming that root; rows of other roots as such are not listed. + +Every `extenddb backup` command refuses an in-memory SQLite database (`[storage.sqlite] path = ":memory:"`, or no path on a `sqlite-memory` build) before it opens the catalog, exit 1: it would open a new, empty catalog that ends with the command. A server on such a database syncs with its store at every start instead (the `path` row above). `extenddb backup remove-target --account
` removes a restore target that `DeleteTable` refuses (a `CREATING` table) together with its restore record: the target of a `FAILED` restore whatever the reason (`target_not_empty` included: the state a resume leaves when it finds no recorded progress and more items in the target than the backup holds, which a catalog restored from a dump can produce when it names a restore that finished and was written to since; an ordinary crash before the first progress record on Cassandra leaves no more items than the backup holds and is resumed, not failed; until the command runs a `target_not_empty` target keeps its backup in use, so `DeleteBackup` and a second restore of the backup are `BackupInUseException` and the name is held; `owner dead: taken over 5 times by one instance without the restore ending` is the reason of a record one server's reconciler resumed five times without the restore ending, whose target the reconciler removes itself), the target of a `RUNNING` restore whose owner is dead (no heartbeat within `stale_job_after_secs` on the database's clock, the listing the reconciler uses; on SQLite under the serve lock every owner is dead), and a legacy (in-catalog) restore's target the same way. It refuses a `RUNNING` restore with a live owner ("stop it or wait for it"), a `DONE` restore and a table no restore record names, the latter two as "not a restore target", and exits 1. It needs no store (it runs whether or not `[backup.store]` is configured, and creates neither the default root nor its identity file), takes the catalog-level maintenance claim first as `purge` does and releases it before it returns, logs the removal under `extenddb::audit` with the account, the table and the backup's ARN, and prints what it did: diff --git a/docs/manuals/06-developer-test-guide.md b/docs/manuals/06-developer-test-guide.md index c0ec10ee6..22aace5da 100755 --- a/docs/manuals/06-developer-test-guide.md +++ b/docs/manuals/06-developer-test-guide.md @@ -203,24 +203,30 @@ The CI jobs run the suite with `filesystem`. **The cross-backend restore matrix** is `devtools/backup-restore-matrix`: one cell per invocation (`--source --target -`, plus `--duplicate-key` or `--vector`), -each starting a source and a target server from the per-backend release -binaries (`--bin-`, defaulting to +`, plus `--duplicate-key`, `--vector` or +`--adopt-store`), each starting a source and a target server from the +per-backend release binaries (`--bin-`, defaulting to `target/rel-/release/extenddb`) against PostgreSQL at `127.0.0.1:55432`, MongoDB at `127.0.0.1:27117` and Cassandra at `127.0.0.1:9042`; its MongoDB and Cassandra cells need `pymongo` and -`cassandra-driver` (`requirements-matrix.txt`). The nightly workflow -`.github/workflows/matrix-nightly.yml` builds the four binaries, starts the -three services and runs the twelve plain cells, the four duplicate-key -cells and the eight vector cells; `workflow_dispatch` runs it on demand. The -same workflow runs the restore kill points of every backend in four jobs -(`restore-kill-points-`), each building its backend's binary with -`extenddb-app/backup-test-hooks` and starting its backend as the matrix job -does, through `devtools/backup-kill-points.py restore ---bin --work ` (the Cassandra job builds a plain SQLite binary -too, the `--source-bin` whose backup the Cassandra target restores); every -point ends ACTIVE with the exact base and GSI counts, except the -tampered-file point, whose target is removed. +`cassandra-driver` (`requirements-matrix.txt`). A cell copies the source's +backup prefix under the target's root and adopts it with `extenddb backup +sync` on the target (with the target's server stopped when it is SQLite); +`--adopt-store` points the target's store at the source's root instead, +adopts the root with `backup sync --adopt-store` and checks that the +source's server then reports the store owned by another catalog. The +nightly workflow `.github/workflows/matrix-nightly.yml` builds the four +binaries, starts the three services and runs the twelve plain cells, the +four duplicate-key cells, the eight vector cells and two `--adopt-store` +cells (SQLite to PostgreSQL and back); `workflow_dispatch` runs it on +demand. The same workflow runs the restore kill points of every backend in +four jobs (`restore-kill-points-`), each building its backend's +binary with `extenddb-app/backup-test-hooks` and starting its backend as +the matrix job does, through `devtools/backup-kill-points.py restore + --bin --work ` (the Cassandra job builds a +plain SQLite binary too, the `--source-bin` whose backup the Cassandra +target restores); every point ends ACTIVE with the exact base and GSI +counts, except the tampered-file point, whose target is removed. The `run-tests` script automatically: - Performs health check on the target endpoint diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index a10282434..9a9d8ac6a 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -51,7 +51,7 @@ A backup of a table that holds an `N` value the wire did not validate, or an att Migration 6 adds the catalog index of backups kept in a backup store, turns each restore registration into a restore job record, adds the maintenance claim record (`backup_maintenance`) and writes the catalog's random id (`settings.catalog_id`). On SQLite and PostgreSQL the index is the `backup_index` table and the job record is new columns of `table_restores`; on MongoDB the index is the `backup_index` collection and the job record a `restore_job` field of each restore target's `tables` document; on Cassandra the index is new columns of `backups_by_arn` and the `backup_index_by_status` table, and the job record new `restore_*` columns of `tables`. Stop every server built before this migration before running `extenddb migrate --yes`: such a server neither reads the index nor marks the restore registrations it writes, so a restore it ran during the upgrade would be owned by nobody. A restore in flight at that stop is left as a RUNNING job record whose owner is dead; the first reconciler pass of the new server (within `backup.reconcile_interval_secs`, default 60 s) marks it FAILED and removes its CREATING target, so the name is free again and the table has to be restored anew. On Cassandra the backfill reads `tables` with a filtered scan and writes each registration with a lightweight transaction, which takes seconds on a catalog of ordinary size. On Cassandra and MongoDB, which have no transaction around a body, every write of the migration is safe to send again: a `migrate --yes` that fails part way leaves migration 6 unrecorded, `serve` keeps refusing the catalog, and the next `migrate --yes` runs the body again and finishes. -A rollback to the previous binary needs two things: the catalog restored from the pre-upgrade backup (the previous binary refuses a catalog carrying migration 6, exit status 4), and the `[backup]` section removed from the configuration file, which the previous binary refuses as an unknown field before it reaches the catalog (`unknown field 'backup'`). The pre-upgrade backup predates every backup this release wrote to the store, so a restored catalog has no `backup_index` row for them while the store keeps their bytes under its root; the previous binary serves neither (it does not read the store), and a later release's `backup sync --adopt-store` re-adopts them (the store's identity file names the pre-rollback catalog, so the re-upgraded catalog finds the store unhealthy until it is adopted; PR 10). In-catalog backups taken after it are lost with the rows, as before. +A rollback to the previous binary needs two things: the catalog restored from the pre-upgrade backup (the previous binary refuses a catalog carrying migration 6, exit status 4), and the `[backup]` section removed from the configuration file, which the previous binary refuses as an unknown field before it reaches the catalog (`unknown field 'backup'`). The pre-upgrade backup predates every backup this release wrote to the store, so a restored catalog has no `backup_index` row for them while the store keeps their bytes under its root; the previous binary serves neither (it does not read the store). After the upgrade is run again, `extenddb backup sync --adopt-store` gives them rows: the store's identity file names the pre-rollback catalog, so the re-upgraded catalog finds the store owned by another catalog until the command makes it the owner (on SQLite's default root the server instead logs a warning and leaves the store unconfigured, so `/health` has no `backup_store` component until then), and the command then adopts every backup under the root. On SQLite `backup sync`, with or without its options, takes the serve lock as `backup purge` does, so the server is stopped while it runs; on PostgreSQL, MongoDB and Cassandra it runs beside the servers under the maintenance claim. In-catalog backups taken after it are lost with the rows, as before. ## Writing a New Migration diff --git a/requirements-matrix.txt b/requirements-matrix.txt index 3efb146f1..1c0c1801b 100644 --- a/requirements-matrix.txt +++ b/requirements-matrix.txt @@ -1,6 +1,6 @@ # Copyright 2026 ExtendDB contributors # SPDX-License-Identifier: Apache-2.0 -# The drivers devtools/backup-restore-matrix uses to seed and tear down its +# The drivers devtools/backup-restore-matrix uses to set up and tear down its # MongoDB and Cassandra deployments (.github/workflows/matrix-nightly.yml # installs this file beside requirements.txt). Separate from requirements.txt # because nothing else in the test suites talks to those services directly diff --git a/tests/test_backup_job.py b/tests/test_backup_job.py index 0d761c05b..1ba68d83d 100644 --- a/tests/test_backup_job.py +++ b/tests/test_backup_job.py @@ -48,19 +48,71 @@ backup remove-target` removes the target and its record, after which `DescribeTable` is `ResourceNotFoundException` and `DeleteBackup` succeeds. +`extenddb backup sync`, on the SQLite, PostgreSQL and MongoDB deployments +(MongoDB's row edits run `mongosh` in the replica set's container, which +`EXTENDDB_TEST_MONGODB_CONTAINER` names; "beside a server" is PostgreSQL and +MongoDB, whose commands take the maintenance claim): + +- after the catalog loses four rows, sync gives an AVAILABLE row to the one + whole prefix, and DescribeBackup, ListBackups and a restore of it work; a + prefix without its manifest, with a sidecar that does not match, moved + under another backup id, under `changelog/` or `exports/`, or with a + FAILED, DELETING or CREATING row is reported and not adopted, the row + left as it is (on SQLite, where sync takes the serve lock and refuses + beside a server, too: plain sync runs no dead-owner resolution); +- (SQLite) plain sync after a SIGTERM mid-restore leaves the RUNNING record + to the next server start, which resumes the restore to ACTIVE; +- a row whose manifest is gone is reported and stays AVAILABLE, and + `--prune-missing` removes the row and deletes no object; an empty root, a + root holding only the identity file and one with another store id each + make the store unhealthy, and sync and `--prune-missing` refuse with no + row changed and nothing written; so does a missing manifest behind the + newest AVAILABLE row, until the manifest is back; +- a row-less prefix whose manifest the store cannot read is reported and + the run exits 1 as incomplete, its summary line saying so; once it can be + read, the next sync adopts it; +- prefixes whose manifests a server of another region wrote are adopted + under this server's region: ListBackups names it, DescribeBackup answers + either spelling, and a restore and a DeleteBackup by the other region's + spelling work; +- `--adopt-store` on a root whose file names another catalog refuses, with + nothing written, while this catalog's newest backup has no manifest under + it, and adopts the root once the manifest is back; +- an identity file rewritten to name another catalog while a plain sync is + in its prefix loop ends the command with exit 1, naming that catalog; +- (beside a server) while `--prune-missing` holds the maintenance claim, `backup + purge` and `--adopt-store` fail at once, and `--adopt-store` on a root + the catalog owns waits for a running job as `--prune-missing` does; +- (beside a server) on host-local roots, `--prune-missing` on the host that + lacks a backup's bytes removes its row only, the other host keeps the + bytes, and a sync there adopts them again; +- (beside a server) while purge holds the claim a plain sync fails at once; +- (SQLite and PostgreSQL: a second MongoDB catalog needs a replica set of + its own) `--adopt-store` from a second catalog configured with the root makes that + catalog the owner, its server restores the adopted backup, and the first + catalog's server finds the store owned by another catalog; +- an identity file rewritten to name a third catalog while an adopting sync + is in its prefix loop ends the command with exit 1, naming that catalog; +- (PostgreSQL) a DeleteBackup on the first catalog's server, still running + with its last check healthy after the adoption, is refused by its + deletion fence and leaves the bytes, and the adopting catalog's store + stays healthy. + The binary at `EXTENDDB_BINARY` decides which deployment runs: a SQLite build, a PostgreSQL build (with `EXTENDDB_TEST_PG_CONNECTION_STRING`) or a MongoDB build (with `EXTENDDB_TEST_MONGODB_URI`, a replica set of this suite's own: the catalog's database names are fixed, so each test destroys the previous test's catalog and initializes its own); the restore cells run on the SQLite and -PostgreSQL deployments. Excluded from the backend-agnostic suite; run by name. +PostgreSQL deployments. No Cassandra deployment runs this file. Excluded from the backend-agnostic suite; run by name. """ from __future__ import annotations +import hashlib import json import os import re +import shutil import subprocess import threading import time @@ -82,6 +134,7 @@ _client, _Forwarder, _free_port, + _health, _moto_or_skip, _pg_build, _provision, @@ -931,6 +984,45 @@ def test_sigterm_leaves_the_restore_running_and_it_finishes_after_restart(deploy server.stop() +def test_plain_sync_on_sqlite_leaves_a_stopped_servers_restore_to_resume(deployment, tmp_path): + """SQLite: a SIGTERM while a store-backed restore runs leaves its record + RUNNING; plain `backup sync` then runs under the serve lock with no + dead-owner resolution, so it neither fails the restore nor removes its + target, and the next server start resumes the restore to ACTIVE with + every item.""" + d = deployment + if d.backend != "sqlite": + pytest.skip("plain sync runs under the serve lock only on SQLite") + _add_backup_settings(d, RESTORE_BACKUP_SETTINGS) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _backed_up_table(d, ddb, creds, "plainsync") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + status = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + assert status == "CREATING", ( + f"the restore finished ({status}) before the signal was sent; raise EXTENDDB_TEST_RESTORE_ITEMS" + ) + server.proc.terminate() + assert server.wait_exit(timeout=60) == 0, server.log_text()[-4000:] + finally: + server.stop() + res = d.run("backup", "sync") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert "failed the restore" not in out and "removed abandoned restore target" not in out, out + assert "is failed rather than resumed" not in out, out + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + _wait_active(ddb, target, timeout=240) + assert _count(ddb, target) == RESTORE_ITEMS + log = server.log_text() + assert "taken over by this instance and resumed" in log, log[-4000:] + finally: + server.stop() + + def test_a_store_outage_mid_download_leaves_the_restore_running_and_it_completes(request, tmp_path, moto_bucket): """An S3 store (moto) reached through a TCP forwarder, with a spool: the forwarder is cut right after the restore's response, so the job's data @@ -1541,3 +1633,1202 @@ def test_a_multi_part_key_table_restores_with_the_sources_item_set(deployment, t assert f" into {target} " in log and ": ACTIVE, " in log and ": FAILED (" not in log, log[-4000:] finally: server.stop() + + +# --- extenddb backup sync ----------------------------------------------------------- + + +def _sync_cell(d: _Deployment) -> None: + """The sync cells drop and rewrite `backup_index` rows at the catalog: + with SQL on SQLite and PostgreSQL (`_catalog`), and on MongoDB with + `mongosh` in the replica set's container (`_mongosh`), which + `EXTENDDB_TEST_MONGODB_CONTAINER` names.""" + if d.backend == "mongodb" and not _mongodb_container(): + pytest.skip("EXTENDDB_TEST_MONGODB_CONTAINER is unset: the MongoDB row edits run mongosh in it") + + +def _beside(d: _Deployment) -> bool: + """Whether the commands run beside a server, taking the maintenance + claim: every backend but SQLite, whose commands take the serve lock.""" + return d.backend != "sqlite" + + +def _mongodb_container() -> str: + return os.environ.get("EXTENDDB_TEST_MONGODB_CONTAINER", "").strip() + + +def _mongosh(script: str): + """Run `script` with `mongosh` in the replica set's container against + the catalog database (`db`), and return the JSON it prints last.""" + res = subprocess.run( + ["docker", "exec", _mongodb_container(), "mongosh", "--quiet", "--eval", + 'db = db.getSiblingDB("extenddb_catalog");\n' + script], + capture_output=True, text=True, timeout=60, + ) + assert res.returncode == 0, res.stdout + res.stderr + return json.loads(res.stdout.strip().splitlines()[-1]) + + +def _mongodb_row_filter(arn: str) -> str: + return json.dumps({"account_id": ACCOUNT, "backup_id": _backup_id(arn)}) + + +def _catalog(d: _Deployment, statement: str, params: tuple) -> int: + """Run one statement on the deployment's catalog and return its row count. + The statement uses `?` placeholders and `{now}` for the database's clock, + which become `%s` and `now()` on PostgreSQL.""" + if d.backend == "sqlite": + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + n = conn.execute(statement.format(now="strftime('%Y-%m-%dT%H:%M:%fZ', 'now')"), params).rowcount + conn.commit() + finally: + conn.close() + return n + import psycopg2 + + conn = psycopg2.connect(d.catalog) + try: + with conn.cursor() as cur: + cur.execute(statement.format(now="now()").replace("?", "%s"), params) + n = cur.rowcount + conn.commit() + finally: + conn.close() + return n + + +def _backup_id(arn: str) -> str: + return arn.rsplit("/", 1)[1] + + +def _drop_row(d: _Deployment, arn: str) -> None: + """Delete the backup's `backup_index` row: the catalog lost it.""" + if d.backend == "mongodb": + n = _mongosh(f"print(db.backup_index.deleteMany({_mongodb_row_filter(arn)}).deletedCount)") + else: + n = _catalog(d, "DELETE FROM backup_index WHERE account_id = ? AND backup_id = ?", (ACCOUNT, _backup_id(arn))) + assert n == 1, f"{n} backup_index rows match {arn}" + + +def _set_status(d: _Deployment, arn: str, status: str, beat_age_secs: int = 0) -> None: + """Set the backup's row to `status`, its heartbeat `beat_age_secs` ago + and (except for CREATING) its completion now, on the database's clock: + a FAILED row inside its retention, a CREATING row whose owner beat just + now (or, aged, a CREATING row whose job died).""" + if d.backend == "mongodb": + completion = ( + '{$unset: {completed_at: ""}}' if status == "CREATING" else "{$set: {completed_at: now}}" + ) + n = _mongosh( + f"const now = new Date(); const filter = {_mongodb_row_filter(arn)};\n" + f"const n = db.backup_index.updateOne(filter, {{$set: {{status: {json.dumps(status)}, " + f"heartbeat_at: new Date(now.getTime() - {beat_age_secs * 1000})}}}}).matchedCount;\n" + f"db.backup_index.updateOne(filter, {completion});\nprint(n)" + ) + assert n == 1, f"{n} backup_index rows match {arn}" + return + if beat_age_secs: + assert d.backend == "postgres", "an aged heartbeat is written on PostgreSQL and MongoDB" + n = _catalog( + d, + "UPDATE backup_index SET status = ?, completed_at = NULL, " + f"heartbeat_at = {{now}} - interval '{beat_age_secs} seconds' WHERE account_id = ? AND backup_id = ?", + (status, ACCOUNT, _backup_id(arn)), + ) + assert n == 1, f"{n} backup_index rows match {arn}" + return + n = _catalog( + d, + "UPDATE backup_index SET status = ?, heartbeat_at = {now}, " + "completed_at = CASE WHEN ? = 'CREATING' THEN NULL ELSE {now} END " + "WHERE account_id = ? AND backup_id = ?", + (status, status, ACCOUNT, _backup_id(arn)), + ) + assert n == 1, f"{n} backup_index rows match {arn}" + + +def _rows(d: _Deployment) -> dict[str, str]: + """`extenddb backup list`: each row's ARN to its status.""" + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + rows: dict[str, str] = {} + for line in res.stdout.splitlines(): + m = re.match(r"^([A-Z]+)\s+\d{12}\s+\S+\s+\d+\s+\d+\s+(arn:\S+)\s", line) + if m: + rows[m.group(2)] = m.group(1) + return rows + + +def _key_prefix(table: str, arn: str) -> str: + """The prefix as the store's keys, and sync's report, spell it.""" + return f"{ACCOUNT}/{table}/{_backup_id(arn)}/" + + +def _store_id(root: str) -> str: + with open(os.path.join(root, "extenddb-store.json")) as f: + return json.load(f)["store_id"] + + +def _private_file(path: str, data: bytes) -> None: + """Write `path` 0600 with its missing parents 0700, as the store's own + objects are.""" + parent = os.path.dirname(path) + missing = [] + while not os.path.isdir(parent): + missing.append(parent) + parent = os.path.dirname(parent) + for directory in reversed(missing): + os.mkdir(directory, 0o700) + with open(path, "wb") as f: + f.write(data) + os.chmod(path, 0o600) + + +def test_sync_rebuilds_dropped_rows_and_adopts_only_verified_prefixes(deployment, tmp_path): + """The catalog loses the rows of four backups; `extenddb backup sync` + gives an AVAILABLE row to the one whose prefix is whole, after which + DescribeBackup (the manifest's name and item count), ListBackups and a + restore of it work. Not adopted, each reported with its reason and exit + 0: the prefix whose manifest was removed, the prefix whose sidecar no + longer matches, a prefix moved under another backup id (its manifest + names the backup id it was written under), a file under the table's + `changelog/` and one under `exports/`, and the prefixes whose rows are + FAILED, DELETING and CREATING, whose rows sync reads and changes none. + On PostgreSQL plain sync takes the maintenance claim and waits for no + job; on SQLite it runs under the serve lock (it refuses while the + server runs) and runs no dead-owner resolution, so the DELETING and + CREATING rows a stopped server left stay for its next start. A second + sync adopts nothing.""" + _sync_cell(deployment) + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, kept = _small_backed_up_table(ddb, "sync") + others: dict[str, str] = {} + for label in ("no-manifest", "bad-sidecar", "moved", "failed", "deleting", "creating"): + arn = ddb.create_backup(TableName=name, BackupName=label)["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=60) + others[label] = arn + if d.backend == "sqlite": + # The serve lock held is the catalog refusal `serve` gives for + # it: exit 6, before any store or row is read. + for args in (("sync",), ("sync", "--prune-missing")): + beside = d.run("backup", *args) + assert beside.returncode == 6, beside.stdout + beside.stderr + assert "another extenddb process" in beside.stdout + beside.stderr, beside.stdout + beside.stderr + assert "=== extenddb backup sync" not in beside.stdout, beside.stdout + assert server.proc.poll() is None, "the server kept running" + finally: + server.stop() + root = d.root + store_id = _store_id(root) + for arn in (kept, others["no-manifest"], others["bad-sidecar"], others["moved"]): + _drop_row(d, arn) + os.remove(os.path.join(_prefix(d, name, others["no-manifest"]), "extenddb-manifest.json")) + with open(os.path.join(_prefix(d, name, others["bad-sidecar"]), "extenddb-manifest.sha256"), "w") as f: + f.write("0" * 64 + " extenddb-manifest.json\n") + moved_id = _backup_id(others["moved"]) + millis, suffix = moved_id.split("-") + moved_to = f"{int(millis) + 1:014d}-{suffix}" + os.rename(_prefix(d, name, others["moved"]), os.path.join(root, ACCOUNT, name, moved_to)) + _private_file(os.path.join(root, ACCOUNT, name, "changelog", "000001.json"), b"{}\n") + _private_file(os.path.join(root, "exports", "01", "part-0.json"), b"{}\n") + _set_status(d, others["failed"], "FAILED") + _set_status(d, others["deleting"], "DELETING") + _set_status(d, others["creating"], "CREATING") + + res = d.run("backup", "sync") + out = res.stdout + res.stderr + print(f"branch: {d.backend}\n{res.stdout}") + assert res.returncode == 0, out + assert f"Store: filesystem {os.path.realpath(root)} (store id {store_id})" in res.stdout, res.stdout + took_claim = 'Took the maintenance claim as "extenddb backup sync on ' in res.stdout + assert took_claim == _beside(d), res.stdout + assert "Waiting for running jobs" not in res.stdout, res.stdout + adopted = res.stdout.split("--- Adopted", 1)[1].split("--- Skipped", 1)[0] + assert adopted.splitlines()[1:-1] == [f" {kept}"], adopted + assert "1 backup(s) adopted" in res.stdout, res.stdout + skipped = res.stdout.split("--- Skipped (not adopted):", 1)[1].split("--- AVAILABLE rows", 1)[0] + expected = [ + f"{_key_prefix(name, others['no-manifest'])}: not a complete backup: no manifest", + f"{_key_prefix(name, others['bad-sidecar'])}: not a complete backup: the sidecar does not match", + f"{ACCOUNT}/{name}/{moved_to}/: the manifest names {others['moved']}, whose account, table and backup id are " + "not the prefix's", + f"{ACCOUNT}/{name}/changelog/: reserved for the change-log archive", + "exports/: reserved for exports", + f"{_key_prefix(name, others['failed'])}: the catalog has a row for it (FAILED, store id {store_id})", + f"{_key_prefix(name, others['deleting'])}: the catalog has a row for it (DELETING, store id {store_id})", + f"{_key_prefix(name, others['creating'])}: the catalog has a row for it (CREATING, store id {store_id})", + ] + for line in expected: + assert f" {line}" in skipped, f"{line!r} not in\n{skipped}" + assert len([line for line in skipped.splitlines() if line.startswith(" ")]) == len(expected), skipped + rows = _rows(d) + assert rows[kept] == "AVAILABLE", rows + for label in ("no-manifest", "bad-sidecar", "moved"): + assert others[label] not in rows, (label, rows) + assert rows[others["failed"]] == "FAILED", rows + assert rows[others["deleting"]] == "DELETING", rows + assert rows[others["creating"]] == "CREATING", rows + for label in ("deleting", "creating"): + assert os.path.isfile(os.path.join(_prefix(d, name, others[label]), "extenddb-manifest.json")), label + # Nothing under a skipped prefix was changed or removed. + assert os.path.isfile(os.path.join(_prefix(d, name, others["bad-sidecar"]), "extenddb-manifest.json")) + assert os.path.isfile(os.path.join(root, ACCOUNT, name, moved_to, "extenddb-manifest.json")) + assert os.path.isfile(os.path.join(root, ACCOUNT, name, "changelog", "000001.json")) + + again = d.run("backup", "sync") + assert again.returncode == 0, again.stdout + again.stderr + assert "0 backup(s) adopted" in again.stdout, again.stdout + assert f" {_key_prefix(name, kept)}: the catalog has a row for it (AVAILABLE, store id {store_id})" in again.stdout + + for label in ("failed", "deleting", "creating"): + if rows[others[label]] in ("FAILED", "DELETING", "CREATING"): + _set_status(d, others[label], "AVAILABLE") + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + details = ddb.describe_backup(BackupArn=kept)["BackupDescription"] + assert details["BackupDetails"]["BackupStatus"] == "AVAILABLE", details + assert details["BackupDetails"]["BackupName"] == "nightly", details + assert details["SourceTableDetails"]["TableName"] == name, details + assert details["SourceTableDetails"]["ItemCount"] == HAND_BUILT_ITEMS, details + listed = [s["BackupArn"] for s in ddb.list_backups(TableName=name)["BackupSummaries"]] + assert kept in listed, listed + for label in ("no-manifest", "bad-sidecar", "moved"): + with pytest.raises(ClientError) as gone: + ddb.describe_backup(BackupArn=others[label]) + assert gone.value.response["Error"]["Code"] == "BackupNotFoundException", (label, gone.value.response) + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=kept)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + finally: + server.stop() + + +def _with_root_moved_away(d: _Deployment): + """The store volume unmounted: the root renamed away and an empty + directory of the same path in its place; returns the restore callable.""" + away = d.root + ".away" + os.rename(d.root, away) + os.mkdir(d.root, 0o700) + + def restore() -> None: + shutil.rmtree(d.root) + os.rename(away, d.root) + + return away, restore + + +def _tree(root: str) -> list[str]: + out = [] + for dirpath, dirnames, filenames in os.walk(root): + for f in filenames: + path = os.path.join(dirpath, f) + with open(path, "rb") as fh: + out.append(f"{os.path.relpath(path, root)} {hashlib.sha256(fh.read()).hexdigest()}") + return sorted(out) + + +def test_prune_missing_and_the_unhealthy_store_refusals(deployment, tmp_path): + """A row whose manifest is gone (not the newest AVAILABLE one): plain + sync reports it with the `--prune-missing` hint and leaves it AVAILABLE; + `--prune-missing` takes the maintenance claim and removes the row + outright (no tombstone, BackupNotFound), deleting no object: the rest of + its prefix stays. An unhealthy store refuses plain sync and + `--prune-missing`, exit 1, with no row changed and nothing written to the + root, for each shape an unmounted volume gives: an empty root under the + rows, a root holding only the identity file (the newest AVAILABLE row's + manifest absent), and a root whose identity file carries another store + id. Last, the newest AVAILABLE row's manifest + removed: the store is unhealthy (its manifest probe), so sync and + `--prune-missing` both refuse and the row stays AVAILABLE until the + manifest is put back, after which sync runs again. PostgreSQL runs the + commands beside its server; SQLite stops it (the serve lock).""" + _sync_cell(deployment) + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, older = _small_backed_up_table(ddb, "prune") + newest = ddb.create_backup(TableName=name, BackupName="second")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, newest, timeout=60) + if d.backend == "sqlite": + server.stop() + print(f"branch: {d.backend}, the server {'running' if _beside(d) else 'stopped'}") + + os.remove(os.path.join(_prefix(d, name, older), "extenddb-manifest.json")) + res = d.run("backup", "sync") + assert res.returncode == 0, res.stdout + res.stderr + missing = res.stdout.split("--- AVAILABLE rows whose manifest is not in the store (the rows were AVAILABLE when the catalog was read):", 1)[1] + missing = missing.split("=== extenddb backup sync complete", 1)[0] + assert older in missing and newest not in missing, missing + assert "(left AVAILABLE; run with --prune-missing to remove the rows, which deletes no object)" in missing, missing + assert ("Took the maintenance claim" in res.stdout) == _beside(d), res.stdout + assert _rows(d)[older] == "AVAILABLE" + left_behind = _tree(_prefix(d, name, older)) + assert left_behind, "the older prefix keeps its other objects" + + res = d.run("backup", "sync", "--prune-missing") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert 'Took the maintenance claim as "extenddb backup sync on ' in res.stdout, res.stdout + assert f" {older}: removed: the row is gone, and no object was deleted" in res.stdout, res.stdout + assert "(1 removed)" in res.stdout, res.stdout + audit = [ + line for line in out.splitlines() + if "extenddb::audit" in line and "removed an AVAILABLE row whose manifest is not under the root" in line + ] + assert len(audit) == 1 and older in audit[0] and "backup sync --prune-missing" in audit[0], out + rows = _rows(d) + assert older not in rows and rows[newest] == "AVAILABLE", rows + assert _tree(_prefix(d, name, older)) == left_behind, "prune deleted an object" + if _beside(d): + with pytest.raises(ClientError) as gone: + ddb.describe_backup(BackupArn=older) + assert gone.value.response["Error"]["Code"] == "BackupNotFoundException", gone.value.response + + before = _rows(d) + with open(os.path.join(d.root, "extenddb-store.json"), "rb") as f: + identity_file = f.read() + other_id = json.loads(identity_file) + other_id["store_id"] = str(uuid.uuid4()) + shapes = { + "an empty root": None, + "the identity file alone": identity_file, + "another store id": (json.dumps(other_id, indent=2) + "\n").encode(), + } + for shape, content in shapes.items(): + away, put_back = _with_root_moved_away(d) + try: + if content is not None: + _private_file(os.path.join(d.root, "extenddb-store.json"), content) + tree = _tree(d.root) + for args in (("sync",), ("sync", "--prune-missing")): + res = d.run("backup", *args) + out = res.stdout + res.stderr + assert res.returncode == 1, (shape, args, out) + assert f"the filesystem backup store {os.path.realpath(d.root)} is unhealthy" in out, (shape, out) + assert "sync changes no row while the store is unhealthy" in out, (shape, out) + assert "Took the maintenance claim" not in out, (shape, out) + assert _tree(d.root) == tree, f"{shape}: the command wrote to the root" + print(f"{shape} {' '.join(args)}: {out.strip().splitlines()[-1]}") + finally: + put_back() + assert _rows(d) == before, shape + + manifest = os.path.join(_prefix(d, name, newest), "extenddb-manifest.json") + os.rename(manifest, manifest + ".aside") + try: + for args in (("sync",), ("sync", "--prune-missing")): + res = d.run("backup", *args) + out = res.stdout + res.stderr + assert res.returncode == 1 and "is unhealthy" in out and "manifest" in out, out + assert _rows(d) == before + finally: + os.rename(manifest + ".aside", manifest) + res = d.run("backup", "sync") + assert res.returncode == 0, res.stdout + res.stderr + assert "0 backup(s) adopted" in res.stdout and "0 AVAILABLE row(s) whose manifest is gone" in res.stdout, res.stdout + finally: + server.stop() + + +def test_a_second_maintenance_command_fails_at_once(deployment, tmp_path): + """`backup sync --prune-missing` takes the maintenance claim and waits + for a CREATING row whose owner beat just now (the row edited at the + catalog, its heartbeat the database's now); while it waits, `backup + purge --confirm` and `backup sync --adopt-store` each fail within + seconds, exit 1, naming the one-at-a-time rule, and change nothing. The + first command, signalled, stops before any change and releases the + claim. `--adopt-store` on the root this catalog owns then takes the + claim and waits for the same live job ("Waiting for running jobs to + finish"), and a signal stops it before any change; after that a plain + sync and `--prune-missing` run. SQLite has + no second command to refuse: every writing command takes the serve lock + first, which the sync cell shows.""" + _sync_cell(deployment) + d = deployment + if not _beside(d): + pytest.skip("on SQLite the serve lock refuses a second writing command before any claim is read") + import subprocess + + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, busy = _small_backed_up_table(ddb, "claim") + finally: + server.stop() + _set_status(d, busy, "CREATING") + out_path = str(tmp_path / "first.out") + with open(out_path, "wb") as out: + first = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "sync", "--prune-missing"], + cwd=d.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while "Waiting for running jobs to finish" not in open(out_path).read(): + assert first.poll() is None, open(out_path).read() + assert time.monotonic() < deadline, open(out_path).read() + time.sleep(0.2) + assert busy in open(out_path).read() or name in open(out_path).read(), open(out_path).read() + for args in (("purge", "--confirm"), ("sync", "--adopt-store")): + started = time.monotonic() + res = d.run("backup", *args) + took = time.monotonic() - started + out = res.stdout + res.stderr + assert res.returncode == 1, (args, out) + assert "one maintenance command runs at a time" in out, (args, out) + assert took < 15, f"{args} took {took:.1f} s to refuse" + print(f"{' '.join(args)}: refused in {took:.1f} s") + assert first.poll() is None, "the first command stopped while the others ran" + first.send_signal(15) + assert first.wait(timeout=30) == 1 + text = open(out_path).read() + assert "stopped before any change: SIGTERM received while waiting" in text, text + finally: + if first.poll() is None: + first.kill() + first.wait(timeout=10) + assert _rows(d)[busy] == "CREATING" + # `--adopt-store` on a root this catalog owns takes the claim and waits + # for the same live job before its sync, as `--prune-missing` does. + adopt_path = str(tmp_path / "adopt.out") + with open(adopt_path, "wb") as out: + adopting = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "sync", "--adopt-store"], + cwd=d.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while "Waiting for running jobs to finish" not in open(adopt_path).read(): + assert adopting.poll() is None, open(adopt_path).read() + assert time.monotonic() < deadline, open(adopt_path).read() + time.sleep(0.2) + text = open(adopt_path).read() + assert "there is nothing to adopt" in text and 'Took the maintenance claim as "extenddb backup sync on ' in text, text + assert "--- Adopted" not in text, text + adopting.send_signal(15) + assert adopting.wait(timeout=30) == 1 + text = open(adopt_path).read() + assert "stopped before any change: SIGTERM received while waiting" in text, text + finally: + if adopting.poll() is None: + adopting.kill() + adopting.wait(timeout=10) + assert _rows(d)[busy] == "CREATING" + _set_status(d, busy, "AVAILABLE") + res = d.run("backup", "sync", "--prune-missing") + assert res.returncode == 0, res.stdout + res.stderr + assert "Took the maintenance claim" in res.stdout, res.stdout + + +def _adopted_section(stdout: str) -> str: + return stdout.split("--- Adopted", 1)[1].split("--- Skipped", 1)[0] + + +def _rows_present(d: _Deployment, arns: list[str]) -> int: + """How many of `arns` have a `backup_index` row, read at the catalog + (a command holding SQLite's serve lock does not stop the read).""" + ids = tuple(_backup_id(a) for a in arns) + if d.backend == "mongodb": + return _mongosh( + f"print(db.backup_index.countDocuments({{account_id: {json.dumps(ACCOUNT)}, " + f"backup_id: {{$in: {json.dumps(list(ids))}}}}}))" + ) + marks = ", ".join("?" for _ in ids) + statement = f"SELECT count(*) FROM backup_index WHERE account_id = ? AND backup_id IN ({marks})" + if d.backend == "sqlite": + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + return conn.execute(statement, (ACCOUNT, *ids)).fetchone()[0] + finally: + conn.close() + import psycopg2 + + conn = psycopg2.connect(d.catalog) + try: + with conn.cursor() as cur: + cur.execute(statement.replace("?", "%s"), (ACCOUNT, *ids)) + return cur.fetchone()[0] + finally: + conn.close() + + +def test_sigterm_during_the_prefix_loop_stops_sync_and_keeps_its_rows(deployment, tmp_path): + """Two backups whose rows the catalog lost sort first under the root, + followed by many copies of the first under later backup ids (each read + and skipped: its manifest names another backup id). SIGTERM once both + rows are back, while sync is still in the prefix loop: the command exits + 1 saying it stopped, the rows it inserted stay AVAILABLE, no + missing-manifest section is examined, and the next sync runs at once + (on PostgreSQL it takes the claim the first released) and adopts + nothing more.""" + _sync_cell(deployment) + d = deployment + import signal + import subprocess + + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, first = _small_backed_up_table(ddb, "sigterm") + second = ddb.create_backup(TableName=name, BackupName="second")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, second, timeout=60) + finally: + server.stop() + for arn in (first, second): + _drop_row(d, arn) + src = _prefix(d, name, first) + millis, suffix = max(_backup_id(first), _backup_id(second)).split("-") + copies = 10000 + for i in range(copies): + shutil.copytree(src, os.path.join(d.root, ACCOUNT, name, f"{int(millis) + 1 + i:014d}-{suffix}"), + copy_function=os.link) + out_path = str(tmp_path / "sync.out") + with open(out_path, "wb") as out: + sync = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "sync"], + cwd=d.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while _rows_present(d, [first, second]) < 2: + assert sync.poll() is None, open(out_path).read()[-4000:] + assert time.monotonic() < deadline, open(out_path).read()[-4000:] + time.sleep(0.05) + assert sync.poll() is None, "the sync ended before the signal" + sync.send_signal(signal.SIGTERM) + assert sync.wait(timeout=60) == 1, open(out_path).read()[-4000:] + finally: + if sync.poll() is None: + sync.kill() + sync.wait(timeout=10) + text = open(out_path).read() + print("\n".join(line for line in text.splitlines() if not line.startswith(" "))[-3000:]) + assert "stopped: SIGTERM received. 2 backup(s) were adopted before the stop" in text, text[-4000:] + assert " (not examined)" in text, text[-4000:] + assert "=== extenddb backup sync complete" not in text, text[-4000:] + rows = _rows(d) + assert rows[first] == "AVAILABLE" and rows[second] == "AVAILABLE", rows + started = time.monotonic() + res = d.run("backup", "sync") + took = time.monotonic() - started + out = res.stdout + res.stderr + assert res.returncode == 0, out[-4000:] + if _beside(d): + assert "Took the maintenance claim" in res.stdout, res.stdout[:2000] + assert _adopted_section(res.stdout).splitlines()[1:-1] == [" (none)"], res.stdout[:2000] + assert f"0 backup(s) adopted, {copies + 2} prefix(es) skipped" in res.stdout, res.stdout[-2000:] + print(f"the next sync: exit 0 in {took:.1f} s") + + +def test_prune_missing_on_a_host_local_root_deletes_no_object(deployment, tmp_path): + """Host-local roots on two instances of one catalog, one path: backups + K1 < K2 < K3, and the second host's root lacks K2 (its bytes are on the + first host), while it holds the newest, K3, so its store is healthy. + `--prune-missing` there removes K2's row outright and deletes nothing. + Back on the first host, the reconciler's cycles leave K2's prefix whole + (no tombstone names it), DescribeBackup answers BackupNotFound, and a + plain sync beside the server adopts K2 again, after which it is + AVAILABLE. A SQLite catalog has one instance, so the cell is + PostgreSQL's.""" + _sync_cell(deployment) + d = deployment + if not _beside(d): + pytest.skip("host-local roots need two instances of one catalog") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, k1 = _small_backed_up_table(ddb, "hostlocal") + k2 = ddb.create_backup(TableName=name, BackupName="second")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, k2, timeout=60) + k3 = ddb.create_backup(TableName=name, BackupName="third")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, k3, timeout=60) + finally: + server.stop() + k2_tree = _tree(_prefix(d, name, k2)) + host1 = d.root + ".host1" + os.rename(d.root, host1) + try: + shutil.copytree(host1, d.root) + shutil.rmtree(_prefix(d, name, k2)) + res = d.run("backup", "sync", "--prune-missing") + out = res.stdout + res.stderr + print(out) + assert res.returncode == 0, out + assert f" {k2}: removed: the row is gone, and no object was deleted" in res.stdout, out + rows = _rows(d) + assert k2 not in rows and rows[k1] == "AVAILABLE" and rows[k3] == "AVAILABLE", rows + finally: + shutil.rmtree(d.root, ignore_errors=True) + os.rename(host1, d.root) + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + time.sleep(5) # several reconciler cycles (reconcile_interval_secs = 1) + assert _tree(_prefix(d, name, k2)) == k2_tree, "a deletion duty acted on K2's bytes" + with pytest.raises(ClientError) as gone: + ddb.describe_backup(BackupArn=k2) + assert gone.value.response["Error"]["Code"] == "BackupNotFoundException", gone.value.response + res = d.run("backup", "sync") + assert res.returncode == 0, res.stdout + res.stderr + assert _adopted_section(res.stdout).splitlines()[1:-1] == [f" {k2}"], res.stdout + details = ddb.describe_backup(BackupArn=k2)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE" and details["BackupName"] == "second", details + finally: + server.stop() + + +def test_sync_is_refused_while_purge_holds_the_claim(deployment, tmp_path): + """Beside servers every sync takes the maintenance claim, so no sync + runs while purge does. A CREATING row whose job died after its manifest + (its prefix complete, its heartbeat an hour old) and a CREATING row whose + job beat just now: `purge --confirm` takes the claim and waits for the + live job, and while it holds the claim a plain sync fails at once (exit + 1, one maintenance command at a time) and writes nothing. The job ends; + purge removes the dead row first and then fails to delete its prefix + (made read-only), reports it left and exits 1. The prefix is then a + complete backup with no row, which a later sync adopts and the next + purge deletes. SQLite runs both commands under the serve lock.""" + _sync_cell(deployment) + d = deployment + if not _beside(d): + pytest.skip("on SQLite the serve lock orders sync and purge") + import subprocess + + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, dead = _small_backed_up_table(ddb, "gap") + live = ddb.create_backup(TableName=name, BackupName="live")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, live, timeout=60) + finally: + server.stop() + _set_status(d, dead, "CREATING", beat_age_secs=3600) + _set_status(d, live, "CREATING") + dead_prefix = _prefix(d, name, dead) + os.chmod(dead_prefix, 0o500) + out_path = str(tmp_path / "purge.out") + with open(out_path, "wb") as out: + purge = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "purge", "--confirm"], + cwd=d.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while "Waiting for running jobs to finish" not in open(out_path).read(): + assert purge.poll() is None, open(out_path).read() + assert time.monotonic() < deadline, open(out_path).read() + time.sleep(0.2) + before = _rows(d) + started = time.monotonic() + res = d.run("backup", "sync") + took = time.monotonic() - started + out = res.stdout + res.stderr + print(f"sync beside purge: exit {res.returncode} in {took:.1f} s: {out.strip().splitlines()[-1]}") + assert res.returncode == 1, out + assert "one maintenance command runs at a time" in out, out + assert "--- Adopted" not in res.stdout, out + assert took < 15, f"sync took {took:.1f} s to refuse" + assert _rows(d) == before + _set_status(d, live, "AVAILABLE") + assert purge.wait(timeout=90) == 1, open(out_path).read() + text = open(out_path).read() + print(text) + assert f"left: {dead}" in text, text + finally: + if purge.poll() is None: + purge.kill() + purge.wait(timeout=10) + os.chmod(dead_prefix, 0o700) + assert _rows(d) == {}, "purge removed both rows" + assert os.path.isfile(os.path.join(dead_prefix, "extenddb-manifest.json")) + res = d.run("backup", "sync") + assert res.returncode == 0, res.stdout + res.stderr + assert _adopted_section(res.stdout).splitlines()[1:-1] == [f" {dead}"], res.stdout + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 0, res.stdout + res.stderr + assert not os.path.exists(dead_prefix) + + +@pytest.fixture() +def other_catalog(request, tmp_path, deployment): + """A second catalog of the build's backend (its own `init`), whose + `[backup.store]` names the first deployment's root; its databases are + dropped after the test.""" + _sync_cell(deployment) + if deployment.backend == "mongodb": + pytest.skip("a second MongoDB catalog needs a replica set of its own (the catalog database's name is fixed)") + base = os.path.join(os.path.realpath(str(tmp_path)), "other") + os.mkdir(base) + store = ( + "[backup]\nreconcile_interval_secs = 1\n" + f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{deployment.root}"\n' + ) + if deployment.backend == "sqlite": + o = _SqliteDeployment(base) + _append(o.config, store) + yield _Deployment("sqlite", o.config, o.port, deployment.root, o.cert, base, o.db_path) + return + db_name = f"extenddb_test_{uuid.uuid4().hex[:8]}_catalog" + from urllib.parse import urlparse + + parsed = urlparse(lh.PG_CONN) + env = {"db_name": db_name, "pg_host": parsed.hostname or "localhost", "pg_port": str(parsed.port or 5432)} + config = os.path.join(base, "extenddb.toml") + port = _free_port() + try: + result = lh._run_extenddb( + "init", *lh._init_args(env), config=config, + env_override={"EXTENDDB_ADMIN_PASSWORD": ADMIN_PASSWORD}, + ) + assert result.returncode == 0, result.stdout + result.stderr + lh._patch_config_port(config, port) + _append(config, store) + cert = os.path.join(os.path.expanduser("~/.extenddb/tls"), "cert.pem") + yield _Deployment("postgres", config, port, deployment.root, cert, base, f"{lh.PG_CONN}/{db_name}") + finally: + lh._run_extenddb("destroy", "--yes", *lh._pg_args(), config=config, check=False, timeout=30) + lh._drop_database(db_name) + lh._drop_database(db_name[: -len("_catalog")]) + + +def test_adopt_store_makes_this_catalog_the_owner(deployment, other_catalog, tmp_path): + """A second catalog configured with the first deployment's root: plain + sync refuses the root another catalog owns, naming `--adopt-store`, and + leaves the identity file as it was. `--adopt-store` takes the claim, + rewrites the file's owner (the store id and creation time kept), logs + the audit line and adopts the root's backup; the first catalog's + server finds the store owned by another catalog at its next check (on + PostgreSQL the server kept running beside the adoption; on SQLite it was + stopped for the serve lock and the check is its restart's), and its own + sync now refuses. The second catalog's server describes, lists and + restores the adopted backup.""" + _sync_cell(deployment) + d, o = deployment, other_catalog + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "adopt") + if d.backend == "sqlite": + server.stop() + identity = os.path.join(d.root, "extenddb-store.json") + with open(identity) as f: + before = json.load(f) + res = o.run("backup", "sync") + out = res.stdout + res.stderr + assert res.returncode == 1, out + assert "is owned by another catalog" in out and "run with --adopt-store to make this catalog its owner" in out, out + # A configured root: the sentence about the shared default root is + # not the operator's. + assert "The default root beside a SQLite database file" not in out, out + with open(identity) as f: + assert json.load(f) == before, "the refused sync changed the identity file" + + res = o.run("backup", "sync", "--adopt-store") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert 'Took the maintenance claim as "extenddb backup sync on ' in res.stdout, res.stdout + assert f"Adopted the root: extenddb-store.json named catalog {before['catalog_id']} and now names" in res.stdout + assert f" {arn}" in res.stdout and "1 backup(s) adopted" in res.stdout, res.stdout + with open(identity) as f: + after = json.load(f) + assert after["catalog_id"] != before["catalog_id"], (before, after) + assert after["store_id"] == before["store_id"] and after["created_at_ms"] == before["created_at_ms"], (before, after) + assert after["backend"] == o.backend and after.get("ephemeral_catalog") is False, after + assert "extenddb::audit" in out and before["catalog_id"] in out and after["catalog_id"] in out, out + + if d.backend == "sqlite": + server = _Server(d.config, str(tmp_path / "serve-previous-owner.log")) + health = _wait_health( + d.port, + lambda h: h.get("components", {}).get("backup_store", {}).get("reason") == "owned by another catalog", + server=server, + ) + assert health["components"]["backup_store"]["status"] == "unhealthy", health + print(f"branch: {d.backend}: the previous owner's store: {health['components']['backup_store']}") + finally: + server.stop() + res = d.run("backup", "sync") + assert res.returncode == 1 and "is owned by another catalog" in res.stdout + res.stderr, res.stdout + res.stderr + + server, ddb, creds = _started(o, str(tmp_path / "serve-other.log")) + try: + details = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE" and details["BackupName"] == "nightly", details + assert arn in [s["BackupArn"] for s in ddb.list_backups()["BackupSummaries"]] + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + finally: + server.stop() + + +def test_the_previous_owner_deletes_nothing_under_an_adopted_root(deployment, other_catalog, tmp_path): + """The first catalog's server keeps running, its next check an hour + away (`reconcile_interval_secs = 3600`), while a second catalog adopts + its root with `--adopt-store`. Its `/health` still reports the store + healthy, as its last check left it; DeleteBackup of the adopted backup + there is refused as an unhealthy store's, its deletion fence having + re-read the identity file before the row was changed: the backup's + bytes stay whole and DescribeBackup there still answers AVAILABLE; + `/health` + then reports the store owned by another catalog at once. The second + catalog's server finds its store healthy and the backup AVAILABLE, and + restores it. SQLite stops the first server for the serve lock, so the + cell is PostgreSQL's.""" + _sync_cell(deployment) + d, o = deployment, other_catalog + if d.backend != "postgres": + pytest.skip("the previous owner runs beside the adoption on PostgreSQL") + with open(d.config) as f: + text = f.read() + assert text.count("reconcile_interval_secs = 1\n") == 1, text + with open(d.config, "w") as f: + f.write(text.replace("reconcile_interval_secs = 1\n", "reconcile_interval_secs = 3600\n")) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "fence") + before = _tree(_prefix(d, name, arn)) + res = o.run("backup", "sync", "--adopt-store") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert f" {arn}" in res.stdout and "1 backup(s) adopted" in res.stdout, out + assert "Stop every such server before an adoption" in res.stdout, out + health = _health(d.port) + assert health["components"]["backup_store"]["status"] == "healthy", health + with pytest.raises(ClientError) as refused: + ddb.delete_backup(BackupArn=arn) + error = refused.value.response["Error"] + print(f"DeleteBackup on the previous owner: {error}") + assert error["Code"] == "ValidationException", error + assert "unhealthy (owned by another catalog)" in error["Message"], error + assert _tree(_prefix(d, name, arn)) == before, "the previous owner deleted bytes" + details = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE", f"the refused request moved the row: {details}" + assert _rows(d)[arn] == "AVAILABLE" + component = _health(d.port)["components"]["backup_store"] + assert component["status"] == "unhealthy", component + assert component["reason"] == "owned by another catalog", component + log = server.log_text() + assert "refused before the row was changed; nothing was deleted" in log, log[-4000:] + finally: + server.stop() + + server, ddb, creds = _started(o, str(tmp_path / "serve-other.log")) + try: + details = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE", details + target = f"{name}-r" + ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn) + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + finally: + server.stop() + + +def test_a_prefix_the_store_cannot_read_leaves_the_sync_incomplete(deployment, tmp_path): + """A backup whose row the catalog lost, its manifest made unreadable + (mode 000): `extenddb backup sync` reports the prefix as one the store + could not be read for and exits 1 saying the run is incomplete, with no + row inserted. Once the mode is restored, the next sync adopts the + backup and exits 0.""" + _sync_cell(deployment) + d = deployment + if os.geteuid() == 0: + pytest.skip("a mode-000 file is readable by root") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "unreadable") + finally: + server.stop() + _drop_row(d, arn) + manifest = os.path.join(_prefix(d, name, arn), "extenddb-manifest.json") + os.chmod(manifest, 0o000) + try: + res = d.run("backup", "sync") + out = res.stdout + res.stderr + print(f"branch: {d.backend}: {out.strip().splitlines()[-1]}") + assert res.returncode == 1, out + assert f" {_key_prefix(name, arn)}: the store could not be read: " in res.stdout, out + assert "the run is incomplete: 1 prefix(es) were skipped for a store or catalog error" in out, out + assert "=== extenddb backup sync incomplete: 0 backup(s) adopted, 1 prefix(es) skipped," in res.stdout, out + assert "=== extenddb backup sync complete" not in res.stdout, out + assert _adopted_section(res.stdout).splitlines()[1:-1] == [" (none)"], res.stdout + assert arn not in _rows(d) + finally: + os.chmod(manifest, 0o600) + res = d.run("backup", "sync") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert "=== extenddb backup sync complete: 1 backup(s) adopted" in res.stdout, out + assert _adopted_section(res.stdout).splitlines()[1:-1] == [f" {arn}"], res.stdout + assert _rows(d)[arn] == "AVAILABLE" + + +def _respell_manifest(prefix: str, region: str, other: str) -> None: + """Rewrite the prefix's manifest as a server of region `other` wrote + it (its backup ARN and source table ARN in that region), with a sidecar + that matches.""" + path = os.path.join(prefix, "extenddb-manifest.json") + with open(path) as f: + manifest = json.load(f) + for key in ("backup_arn",): + assert f":{region}:" in manifest[key], manifest[key] + manifest[key] = manifest[key].replace(f":{region}:", f":{other}:", 1) + table_arn = manifest["source_table"]["table_arn"] + manifest["source_table"]["table_arn"] = table_arn.replace(f":{region}:", f":{other}:", 1) + data = json.dumps(manifest, indent=2).encode() + b"\n" + with open(path, "wb") as f: + f.write(data) + with open(os.path.join(prefix, "extenddb-manifest.sha256"), "w") as f: + f.write(f"{hashlib.sha256(data).hexdigest()} extenddb-manifest.json\n") + + +def test_sync_adopts_another_regions_prefix_under_this_servers_region(deployment, tmp_path): + """Two backups whose rows the catalog lost, their manifests rewritten as + a server of another region writes them (the prefix copied from such a + server): sync adopts both, exit 0, with a warning naming the manifest's + region, and the rows carry this server's region. ListBackups, with and + without `TableName`, lists them in this server's region; DescribeBackup + answers either spelling; a restore named by the other region's spelling + reaches ACTIVE with every item, and DeleteBackup by that spelling + deletes the backup. The manifests keep their spelling.""" + _sync_cell(deployment) + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, first = _small_backed_up_table(ddb, "region") + second = ddb.create_backup(TableName=name, BackupName="second")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, second, timeout=60) + finally: + server.stop() + region = first.split(":")[3] + other = "us-west-2" if region != "us-west-2" else "eu-west-1" + for arn in (first, second): + _drop_row(d, arn) + _respell_manifest(_prefix(d, name, arn), region, other) + spelled = {arn: arn.replace(f":{region}:", f":{other}:", 1) for arn in (first, second)} + + res = d.run("backup", "sync") + out = res.stdout + res.stderr + print(f"branch: {d.backend}\n{res.stdout}") + assert res.returncode == 0, out + note = f"whose manifest spells its ARN in region {other}, not this server's" + assert sorted(_adopted_section(res.stdout).splitlines()[1:-1]) == sorted( + f" {a} ({note})" for a in (first, second) + ), out + warned = [line for line in out.splitlines() if "WARN" in line and "backup sync: adopted " in line] + assert len(warned) == 2 and all(note in line for line in warned), out + rows = _rows(d) + assert rows[first] == "AVAILABLE" and rows[second] == "AVAILABLE", rows + assert not any(f":{other}:" in arn for arn in rows), rows + for arn in (first, second): + with open(os.path.join(_prefix(d, name, arn), "extenddb-manifest.json")) as f: + assert json.load(f)["backup_arn"] == spelled[arn], "the manifest kept its spelling" + + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + for kwargs in ({}, {"TableName": name}): + listed = sorted(s["BackupArn"] for s in ddb.list_backups(**kwargs)["BackupSummaries"]) + assert set((first, second)) <= set(listed), (kwargs, listed) + assert not any(f":{other}:" in a for a in listed), (kwargs, listed) + for spelling in (first, spelled[first]): + details = ddb.describe_backup(BackupArn=spelling)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE" and details["BackupName"] == "nightly", details + print(f"DescribeBackup {spelling}: BackupArn {details['BackupArn']}") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=spelled[first])["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + deleted = ddb.delete_backup(BackupArn=spelled[second])["BackupDescription"]["BackupDetails"] + assert deleted["BackupStatus"] == "DELETED", deleted + with pytest.raises(ClientError) as gone: + ddb.describe_backup(BackupArn=second) + assert gone.value.response["Error"]["Code"] == "BackupNotFoundException", gone.value.response + finally: + server.stop() + + +def _foreign_identity(d: _Deployment) -> dict: + """Rewrite the root's identity file to name a catalog that does not + exist (the root another catalog owned, a catalog since destroyed), + keeping the store id; returns the file as written.""" + path = os.path.join(d.root, "extenddb-store.json") + with open(path) as f: + file = json.load(f) + file = dict(file, catalog_id=str(uuid.uuid4())) + staged = path + ".staged" + _private_file(staged, (json.dumps(file, indent=2) + "\n").encode()) + os.rename(staged, path) + return file + + +def test_adopt_store_checks_the_root_before_it_writes(deployment, tmp_path): + """A root whose identity file names another catalog over this catalog's + two backups. With the newest backup's manifest gone, `--adopt-store` + refuses, exit 1, saying the root would be unhealthy once this catalog + owns it and that nothing was written: the file is unchanged and no + audit line says a rewrite started. With the manifest back, the same + command adopts the root (the audit lines, the file naming this catalog + with the store id kept), syncs, and plain sync then runs.""" + _sync_cell(deployment) + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, older = _small_backed_up_table(ddb, "adoptcheck") + newest = ddb.create_backup(TableName=name, BackupName="second")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, newest, timeout=60) + finally: + server.stop() + identity = os.path.join(d.root, "extenddb-store.json") + with open(identity) as f: + ours = json.load(f) + foreign = _foreign_identity(d) + with open(identity, "rb") as f: + foreign_bytes = f.read() + manifest = os.path.join(_prefix(d, name, newest), "extenddb-manifest.json") + os.rename(manifest, manifest + ".aside") + try: + res = d.run("backup", "sync", "--adopt-store") + out = res.stdout + res.stderr + print(f"branch: {d.backend}: {out.strip().splitlines()[-1]}") + assert res.returncode == 1, out + assert "the root would be unhealthy once this catalog owns it" in out, out + assert f"nothing was written, and extenddb-store.json still names catalog {foreign['catalog_id']}" in out, out + assert "rewriting extenddb-store.json to name this catalog" not in out, out + with open(identity, "rb") as f: + assert f.read() == foreign_bytes, "the refused adoption wrote the identity file" + assert _rows(d)[newest] == "AVAILABLE" + finally: + os.rename(manifest + ".aside", manifest) + res = d.run("backup", "sync", "--adopt-store") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert f"Adopted the root: extenddb-store.json named catalog {foreign['catalog_id']} and now names" in res.stdout, out + assert "extenddb::audit" in out and "rewriting extenddb-store.json to name this catalog" in out, out + with open(identity) as f: + after = json.load(f) + assert after["catalog_id"] == ours["catalog_id"] and after["store_id"] == ours["store_id"], (ours, after) + res = d.run("backup", "sync") + assert res.returncode == 0, res.stdout + res.stderr + + +def test_a_plain_sync_whose_root_is_adopted_during_it_ends_with_exit_1(deployment, tmp_path): + """A plain sync of a root this catalog owns, holding one backup whose + row the catalog lost and many copies of it under later backup ids (each + read and skipped). Once the backup's row is back and while the sync is + still in the prefix loop, the identity file is rewritten to name + another catalog, as an adopter's put would leave it. The sync's last + read of the file finds it: exit 1, naming the catalog and the store id, + and no `complete` line.""" + _sync_cell(deployment) + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "plainrace") + finally: + server.stop() + _drop_row(d, arn) + src = _prefix(d, name, arn) + millis, suffix = _backup_id(arn).split("-") + copies = 10000 + for i in range(copies): + shutil.copytree(src, os.path.join(d.root, ACCOUNT, name, f"{int(millis) + 1 + i:014d}-{suffix}"), + copy_function=os.link) + with open(os.path.join(d.root, "extenddb-store.json")) as f: + store_id = json.load(f)["store_id"] + out_path = str(tmp_path / "sync.out") + with open(out_path, "wb") as out: + sync = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "sync"], + cwd=d.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while _rows_present(d, [arn]) < 1: + assert sync.poll() is None, open(out_path).read()[-4000:] + assert time.monotonic() < deadline, open(out_path).read()[-4000:] + time.sleep(0.05) + foreign = _foreign_identity(d) + assert sync.poll() is None, "the sync ended before the rewrite" + assert sync.wait(timeout=180) == 1, open(out_path).read()[-4000:] + finally: + if sync.poll() is None: + sync.kill() + sync.wait(timeout=10) + text = open(out_path).read() + print("\n".join(line for line in text.splitlines() if not line.startswith(" "))[-3000:]) + assert ( + f"after the sync, extenddb-store.json names catalog {foreign['catalog_id']} (store id {store_id}): " + "another catalog adopted the root during the sync" + ) in text, text[-4000:] + assert "=== extenddb backup sync complete" not in text, text[-4000:] + + +def test_an_adopter_whose_put_lands_during_the_sync_ends_it_with_exit_1(deployment, other_catalog, tmp_path): + """A second catalog adopts the first one's root with `--adopt-store`, + the root holding one backup and many copies of it under later backup + ids (each read and skipped: its manifest names another backup id). Once + the command has printed its adoption and while its sync is still in the + prefix loop, the identity file is rewritten to name a third catalog + under the same store id, as a second adopter's put would leave it. The + command's last read of the file finds it: exit 1, naming the third + catalog and the store id. The first server is stopped throughout.""" + _sync_cell(deployment) + d, o = deployment, other_catalog + import subprocess + + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "race") + finally: + server.stop() + src = _prefix(d, name, arn) + millis, suffix = _backup_id(arn).split("-") + copies = 10000 + for i in range(copies): + shutil.copytree(src, os.path.join(d.root, ACCOUNT, name, f"{int(millis) + 1 + i:014d}-{suffix}"), + copy_function=os.link) + identity = os.path.join(d.root, "extenddb-store.json") + with open(identity) as f: + before = json.load(f) + out_path = str(tmp_path / "adopt.out") + with open(out_path, "wb") as out: + adopting = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", o.config, "sync", "--adopt-store"], + cwd=o.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while "Adopted the root:" not in open(out_path).read(): + assert adopting.poll() is None, open(out_path).read()[-4000:] + assert time.monotonic() < deadline, open(out_path).read()[-4000:] + time.sleep(0.05) + third = dict(before, catalog_id=str(uuid.uuid4())) + staged = identity + ".staged" + _private_file(staged, (json.dumps(third, indent=2) + "\n").encode()) + os.rename(staged, identity) + assert "--- Adopted" not in open(out_path).read(), "the sync had ended before the rewrite" + assert adopting.poll() is None, "the command ended before the rewrite" + assert adopting.wait(timeout=180) == 1, open(out_path).read()[-4000:] + finally: + if adopting.poll() is None: + adopting.kill() + adopting.wait(timeout=10) + text = open(out_path).read() + print("\n".join(line for line in text.splitlines() if not line.startswith(" "))[-3000:]) + assert ( + f"after the sync, extenddb-store.json names catalog {third['catalog_id']} (store id {before['store_id']}): " + "another catalog adopted the root during the sync" + ) in text, text[-4000:] + assert f" {arn}" in _adopted_section(text), text[-4000:] + assert "=== extenddb backup sync complete" not in text, text[-4000:] diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py index 74085bb0a..1924addd3 100644 --- a/tests/test_backup_store_config.py +++ b/tests/test_backup_store_config.py @@ -29,6 +29,11 @@ shell's directory in daemon mode; - `extenddb backup list` and `backup purge --confirm` on an empty store, and `backup purge` refused while a server holds the SQLite serve lock (exit 6); +- an in-memory SQLite server restarted onto the root it used: the second + start adopts the root the first start's catalog owned and syncs before it + listens, so the first start's backup is described and restored; every + `backup` command refuses the in-memory database; a root whose file names + a durable catalog stays unhealthy; - on PostgreSQL, the backup commands ending with the exit code `serve` gives for a catalog the factory refuses: 6 while a migration holds the serving key, 3 on a catalog behind the binary. @@ -571,6 +576,97 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql finally: server.stop() + def test_an_in_memory_server_restarted_onto_its_root_serves_the_previous_starts_backups( + self, sqlite_deployment, tmp_path + ): + """`sqlite-memory`'s shape with a filesystem store, restarted: the + catalog (and its `catalog_id`) is new at every start, so the first + start writes `extenddb-store.json` with `ephemeral_catalog: true`, + and the second adopts the root that file names (a new `catalog_id`, + the store id and the marker kept) and runs the startup sync before + it listens, so DescribeBackup, ListBackups and a restore of the first + start's backup answer from the first request. Between the two starts + every `extenddb backup` command refuses the in-memory database before + it opens a catalog, and the file is unchanged. A file whose marker is + false (a durable catalog's root) is left alone: the in-memory server + reports the store owned by another catalog.""" + d = sqlite_deployment + env = os.environ.copy() + env["EXTENDDB__STORAGE__SQLITE__PATH"] = ":memory:" + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + root = tmp_path / "memory-root" + root.mkdir(mode=0o700) + identity = root / "extenddb-store.json" + _append(d.config, f'[backup]\nreconcile_interval_secs = 1\n[backup.store]\nkind = "filesystem"\n\n[backup.store.filesystem]\npath = "{root}"\n') + healthy = lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy" # noqa: E731 + + server = _Server(d.config, str(tmp_path / "serve-1.log"), env=env) + try: + _wait_health(d.port, healthy, server=server) + first = json.loads(identity.read_text()) + assert first["ephemeral_catalog"] is True, first + ddb = _client(d.port, _provision(d.port, d.cert, ADMIN_PASSWORD)) + _create_table(ddb, "Music") + for k in ("a", "b"): + ddb.put_item(TableName="Music", Item={"pk": {"S": k}, "v": {"N": "1"}}) + arn = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn) + finally: + server.stop() + + for args in ( + ("backup", "list"), + ("backup", "sync"), + ("backup", "sync", "--adopt-store"), + ("backup", "purge", "--confirm"), + ("backup", "remove-target", "--account", "123456789012", "Music"), + ): + res = d.run(*args, env=env) + out = res.stdout + res.stderr + assert res.returncode == 1, (args, out) + assert "the SQLite database is in memory ([storage.sqlite] path names no file)" in out, (args, out) + assert json.loads(identity.read_text()) == first, "a refused command changed the identity file" + + server = _Server(d.config, str(tmp_path / "serve-2.log"), env=env) + try: + _wait_health(d.port, healthy, server=server) + second = json.loads(identity.read_text()) + assert second["catalog_id"] != first["catalog_id"], (first, second) + assert second["store_id"] == first["store_id"] and second["created_at_ms"] == first["created_at_ms"], (first, second) + assert second["ephemeral_catalog"] is True, second + lines = server.log_text().splitlines() + synced = [i for i, line in enumerate(lines) if "startup sync of the in-memory catalog: 1 backup(s) adopted" in line] + listening = [i for i, line in enumerate(lines) if "extenddb listening on" in line] + assert len(synced) == 1 and listening and synced[0] < listening[0], "\n".join(lines[-40:]) + ddb = _client(d.port, _provision(d.port, d.cert, ADMIN_PASSWORD)) + details = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE" and details["BackupName"] == "nightly", details + assert arn in [s["BackupArn"] for s in ddb.list_backups()["BackupSummaries"]] + restored = ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + assert restored["TableDescription"]["TableStatus"] == "CREATING", restored + deadline = time.monotonic() + 60 + while ddb.describe_table(TableName="Restored")["Table"]["TableStatus"] != "ACTIVE": + assert time.monotonic() < deadline, "the restore did not finish" + time.sleep(0.1) + assert ddb.scan(TableName="Restored", Select="COUNT")["Count"] == 2 + finally: + server.stop() + + durable = dict(second, ephemeral_catalog=False) + identity.write_text(json.dumps(durable, indent=2) + "\n") + os.chmod(identity, 0o600) + before = identity.read_bytes() + server = _Server(d.config, str(tmp_path / "serve-3.log"), env=env) + try: + health = _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert health["components"]["backup_store"] == { + "status": "unhealthy", "kind": "filesystem", "reason": "owned by another catalog", + }, health + assert identity.read_bytes() == before, "an in-memory server rewrote a durable catalog's file" + assert "startup sync of the in-memory catalog" not in server.log_text(), server.log_text()[-3000:] + finally: + server.stop() + def test_default_root_under_a_group_writable_parent_warns_and_the_same_root_explicit_fails( self, sqlite_deployment, tmp_path ): @@ -964,7 +1060,8 @@ def test_a_catalog_re_created_after_destroy_warns_about_the_old_default_root(sel root's identity file names the old catalog. The server starts with the store unconfigured and a warning naming the file (not degraded, not a startup error), CreateBackup takes the in-catalog path with its - deprecation warning, and `destroy`'s output said to remove the file.""" + deprecation warning, and `destroy`'s output named `backup sync + --adopt-store`.""" d = sqlite_deployment server = _Server(d.config, str(tmp_path / "serve.log")) try: @@ -976,7 +1073,7 @@ def test_a_catalog_re_created_after_destroy_warns_about_the_old_default_root(sel old_catalog = json.load(f)["catalog_id"] res = d.run("destroy", "--yes") assert res.returncode == 0, res.stdout + res.stderr - assert "needs that file removed" in res.stdout, res.stdout + assert "backup sync --adopt-store" in res.stdout, res.stdout assert os.path.isfile(identity) env = os.environ.copy() env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD @@ -1001,6 +1098,68 @@ def test_a_catalog_re_created_after_destroy_warns_about_the_old_default_root(sel finally: server.stop() + def test_adopt_store_takes_the_default_root_a_re_created_catalog_finds(self, sqlite_deployment, tmp_path): + """A store-backed backup on the default root, then `destroy` and + `init --overwrite`: the identity file names the old catalog. For + `backup sync` that root is configured and unhealthy (owned by another + catalog), not left unconfigured as the server leaves it: plain sync + exits 1 naming `--adopt-store`, and `--adopt-store` makes the new + catalog the owner, keeps the store id and adopts the backup, which + the restarted server describes and lists.""" + d = sqlite_deployment + _append(d.config, "[backup]\nreconcile_interval_secs = 1\n") + healthy = lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy" # noqa: E731 + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, healthy, server=server) + ddb = _client(d.port, _provision(d.port, d.cert, ADMIN_PASSWORD)) + _create_table(ddb, "Music") + ddb.put_item(TableName="Music", Item={"pk": {"S": "a"}}) + arn = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn) + finally: + server.stop() + identity = os.path.join(d.dir, "backups", "extenddb-store.json") + with open(identity) as f: + before = json.load(f) + res = d.run("destroy", "--yes") + assert res.returncode == 0, res.stdout + res.stderr + env = os.environ.copy() + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + subprocess.run( + [EXTENDDB_BINARY, "init", "--backend", "sqlite", "--config", d.config, "--overwrite"], + cwd=d.dir, capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=60, check=True, env=env, + ) + lh._patch_config_port(d.config, d.port) + _append(d.config, "[backup]\nreconcile_interval_secs = 1\n") + + res = d.run("backup", "sync") + out = res.stdout + res.stderr + assert res.returncode == 1, out + assert "owned by another catalog" in out and "--adopt-store" in out, out + assert "The default root beside a SQLite database file is shared" in out, out + assert "there are no store-backed backups" not in out, out + res = d.run("backup", "sync", "--adopt-store") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert "Adopted the root" in out and "1 backup(s) adopted" in out, out + with open(identity) as f: + after = json.load(f) + assert after["store_id"] == before["store_id"], (before, after) + assert after["created_at_ms"] == before["created_at_ms"], (before, after) + assert after["catalog_id"] != before["catalog_id"], (before, after) + + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + _wait_health(d.port, healthy, server=server) + ddb = _client(d.port, _provision(d.port, d.cert, ADMIN_PASSWORD)) + details = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE", details + listed = [b["BackupArn"] for b in ddb.list_backups()["BackupSummaries"]] + assert listed == [arn], listed + finally: + server.stop() + def test_backup_purge_deletes_each_rows_prefix_and_row_and_leaves_the_rest(self, sqlite_deployment, tmp_path): """Rows of every status seeded in the catalog with objects under their prefixes: purge deletes each row's objects and the row outright; a @@ -1578,6 +1737,7 @@ def command(*args): ("backup", "purge"), ("backup", "purge", "--confirm"), ("backup", "remove-target", "--account", "000000000000", "nosuch"), + ("backup", "sync"), ): res = command(*args) out = res.stdout + res.stderr @@ -1595,6 +1755,7 @@ def command(*args): ("backup", "list"), ("backup", "purge", "--confirm"), ("backup", "remove-target", "--account", "000000000000", "nosuch"), + ("backup", "sync"), ): res = command(*args) out = res.stdout + res.stderr