Skip to content

feat: point-in-time recovery capture, enable, disable and describe on SQLite and PostgreSQL - #934

Draft
eliseong wants to merge 6 commits into
mainfrom
eliseong-pitr-capture-2026-11-09
Draft

feat: point-in-time recovery capture, enable, disable and describe on SQLite and PostgreSQL#934
eliseong wants to merge 6 commits into
mainfrom
eliseong-pitr-capture-2026-11-09

Conversation

@eliseong

@eliseong eliseong commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

The first half of point-in-time recovery on SQLite and PostgreSQL. Behind a new pitr cargo feature, which no shipped build enables:

  • Item history. Every committed PutItem, UpdateItem, DeleteItem and TransactWriteItems write to a table with recovery enabled records the item after the write (or a delete) in the write's own transaction. A write that changes nothing records nothing.
  • UpdateContinuousBackups and DescribeContinuousBackups. Enable, disable and describe answer as the service does: the period checks and their order, the fresh enable at 35 days, the baseline of the items that exist at the enable, the restorable window, and the undocumented PointInTimeRecoveryWindowDays member.
  • Retention. An hourly sweep prunes each table's history to the floor of its window, in batches under a deadline.
  • Restore checks and reconstruct stream. The RestoreTableToPointInTime checks in the service's order and the stream of a table's items at a restore time are in place, but nothing calls them yet. RestoreTableToPointInTime still answers PointInTimeRecoveryUnavailableException. A second PR wires the restore into the store-backed restore job.

Without the feature, and on MongoDB and Cassandra, nothing changes on the wire. Catalog migration 10, point_in_time_recovery, runs on every SQLite and PostgreSQL build, because a migration id cannot be feature-gated. It creates two empty tables, pitr_state and item_history, and does not apply on MongoDB or Cassandra.

The commits each build and pass on their own:

  1. Migration 10, its golden and schema files, and the manuals.
  2. The extenddb_storage::pitr store trait, the SQLite and PostgreSQL stores, the write hooks, the retention sweep, and their shared conformance checks.
  3. The enable, disable and describe handlers, and the pitr feature on the engine, server, app and binary crates.
  4. The restore checks and the reconstruct stream, and InvalidRestoreTimeException.
  5. CI: clippy, unit tests, the PostgreSQL storage tests and the wire tests with the feature on.
  6. PostgreSQL history commit times read from the database clock under the item's row lock.

Why

Part of #446. RestoreTableToPointInTime needs a history of each table's committed writes from the enable on. This PR adds that history and the API that turns it on and off, so the restore itself can land on top of the store-backed restore job.

What the service does, measured on DynamoDB in us-east-1, two samples of each request:

  • Items that exist at the enable are in every restore after it. A disable drops the history, and a re-enable starts a new window with the period back at 35.
  • RecoveryPeriodInDays is 1 to 35, default 35, checked before the table. It is read as an Integer: a fraction is truncated, another JSON type is a SerializationException, and a disable that gives one is refused with "Invalid Request: Cannot specify RecoveryPeriodInDays when disabling point-in-time recovery."
  • An enable on an enabled table keeps the window and changes the period only when one is given.
  • DescribeContinuousBackups reports LatestRestorableDateTime at the enable second for the first minutes, then about 300 s behind the request. The restore ceiling is the request's own time, not the reported Latest, and the refusal names the reported bounds.
  • An enable of a table being created is TableNotFoundException, and of a restore target being loaded ContinuousBackupsUnavailableException "Backups are being enabled for the table: . Please retry later".

Known limitations of this PR

  • With the feature on, an enable answers ENABLED and a restore still answers "Point in time recovery is not enabled". This lasts until the restore wiring lands.
  • For about 11 s after CreateTable, the service refuses an enable with "Backups are being enabled". ExtendDB does not time that window and enables at once.
  • A server built without the feature records no history. If such a server and one built with it share a catalog, the history misses the first server's writes.
  • An enable or disable holds the table's writers while it copies the baseline. On PostgreSQL that took 3.57 s for 300,000 items, on that table only. On SQLite it took 0.31 s for 100,000 items, on every table, because SQLite has one write lock.
  • A history row's commit time comes from one clock per database: the PostgreSQL server's clock, read under the item's row lock, or on SQLite the ExtendDB server's clock under its write lock. A step back of that clock can still put two writes of one key in the wrong order.

Testing done

  • Unit and integration tests, local (macOS): the default workspace, 58 binaries, 4376 pass. The one failure, a_root_that_is_or_contains_the_database_directory_is_refused, also fails on main on macOS (/var resolves to /private/var). The storage, SQLite, engine, core and server crates with the feature, 10 binaries, 2784 pass.
  • PostgreSQL 16 storage tests with EXTENDDB_TEST_PG_REQUIRED=1: without the feature, 28 binaries, 400 pass; with it, 28 binaries, 439 pass. tests/pitr_item_history.rs passed 20 runs in a row.
  • Wire tests: tests/test_pitr_enable.py and tests/test_continuous_backups.py pass against DynamoDB (42 pass, 13 skip) and against a SQLite server built with the feature (54 pass, 1 skip). Against a SQLite server without it, 13 pass and 42 skip.
  • A negative control for each rule of the stores, the write hooks, the writer exclusion, the reconstruct pins, the sweep and the handlers.
  • PostgreSQL put throughput on a table without recovery, release build, within 1.3 percent of a build without the feature (sequential and 8 writers, 6 runs each).
  • Commit-time order: 16 writers on one key for 20 s on PostgreSQL leave no history row with an earlier time than the row before it, on one server and on two servers with clocks 5 s apart (two samples each). Before the last commit, the plain put path gave about 2,100 such rows of 11,800. On the service, two writes of one key 400 ms apart restore to the first at the midpoint and to the second after it (two samples).
  • cargo fmt --check and clippy with -D warnings are clean with the feature on and off, and devtools/check-actions-pinned passes.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace), apart from the macOS failure above that main shares
  • Code is formatted (cargo fmt --check)
  • Clippy is clean
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: none yet. This PR adds catalog migration 10 and the PitrCaptureStore trait. I will add an ADR if the maintainers want one before merge.

Breaking changes

None. Migration 10 adds two empty tables. A build without the feature never reads or writes them.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

@eliseong
feat(storage): add catalog migration 10, point_in_time_recovery
4a05a0c 
A table with point-in-time recovery enabled on the service can be restored
to any second of its window, items written before the enable included. That
needs the table's recovery setting and a history of its committed writes.

Migration 10 creates the two tables that hold them: pitr_state, one row per
table with recovery enabled (the retention in days, the enable instant, the
cutoff of the latest prune, and the time a reconstruct in progress holds
against the retention sweep, with its expiry), and item_history, one row per
committed write (the key as the data table stores it, the commit time in
microseconds, and the item after the write or NULL for a delete). On SQLite
it is a file body in the catalog, and both tables are removed with the
table's catalog row. On PostgreSQL it is a data database body, beside the
item data, so a history row can commit in the write's own transaction;
item_history's sequence has a cache of 1, as stream_pending's has. It does
not apply on MongoDB or Cassandra. The name continuous_backups is not
reused, since migration 5 drops it again under --reconverge.

The MongoDB migration tests count the new id, the SQLite golden test pins
what it creates on a released catalog, and the released-catalog helper of
the resource tag test removes the two tables as it removes migration 9's
columns. docs/schema/sqlite.txt and docs/schema/postgres.txt are regenerated
from fresh installs. The upgrade manual lists migration 10 and states
each migration's declared kind per backend, the guides' sample output reads
migration 10 as this build prints it, and CONTRIBUTING's worked example
becomes migration 11.

Verified: the SQLite migration tests (81 pass); the MongoDB fresh, lock and
released-catalog tests on a mongo:7 replica set (24 pass); the PostgreSQL
migration tests on PostgreSQL 16; a fresh install's schema dumps differ from
the committed ones by exactly the two tables, their indexes, constraints and
sequence; the upgrade manual's kinds against each backend's migration table.
@eliseong
feat(storage): item history on SQLite and PostgreSQL
205f620 
On the service, a restore to a time T holds the latest image of each key
committed at or before T, and a key deleted by T is absent (a restore
between the third and fourth steps of a four-step write timeline returned
the one surviving item, two samples). Items that exist at the enable are in
every restore after it, at their image then; a disable drops the history and
a re-enable starts a new window; the window is RecoveryPeriodInDays, 1 to 35
days, 35 by default (two samples of each).

extenddb_storage::pitr (behind a new pitr feature) is the per-backend store
the engine reads: pitr_state, enable_pitr, disable_pitr, reconstruct_page,
prune_before, pitr_tables and remove_orphaned_history, with the shared
retention sweep and its worker. SQLite and PostgreSQL implement it behind
their own pitr features, and DataEngine::as_pitr returns the store.

- Capture: PutItem, UpdateItem, DeleteItem and TransactWriteItems record the
  mutation in the write's own transaction, after the data write, and only
  for a table with a pitr_state row, read in that transaction. A write that
  changed nothing (a failed condition, a delete of a missing item, a
  REMOVE-only update of a missing item, a condition check) records nothing;
  on PostgreSQL a delete of a missing key holds no row lock, so a recorded
  one could order after a concurrent first put of the key. A transaction's
  members share one commit time. A PostgreSQL put or delete with no
  condition, index or stream stays one autocommit statement, which writes
  the history row in the same statement, so the feature costs a table
  without recovery nothing measurable and adds no create races.
- Enable: records the setting and copies every item into the history at the
  enable instant; an enable on an enabled table changes only a given period.
  Enable and disable exclude the table's writers and each other while they
  run (SQLite's write lock and BEGIN IMMEDIATE, PostgreSQL LOCK TABLE IN
  SHARE ROW EXCLUSIVE MODE), so no write is missing from both the baseline
  and the history, and two enables of one table both succeed. An enable or
  disable of a table a DeleteTable removed is TableNotFound.
- Reconstruct: one page of the rows committed at or before T that no later
  row of the key at or before T supersedes, deletes omitted, by an anti-join
  on (table_id, pk, sk, seq) paged on seq. PostgreSQL keys the sort key as
  text (N plain decimal, B hex, empty without one) so the baseline computes
  the same key from the typed columns. Each page pins its time against the
  sweep for 15 minutes; a page whose state a prune has passed answers
  SnapshotExpired rather than a partial state.
- Retention: an hourly sweep prunes each table to the floor of its window,
  in batches under a deadline, keeping every state from the floor on and
  never passing a live pin. The cutoff of the latest prune is recorded, and
  the floor is never before it, so a period raised after a prune reports
  only a window the history holds. A stored period outside 1 to 35 is an
  error. On
  PostgreSQL it also removes the history and setting of tables the catalog
  no longer has, since the catalog is another database. The worker records
  the PitrRetention metric.

Verified: the shared checks of extenddb_storage::pitr::conformance run on
both backends, with the PostgreSQL cases for concurrent writers, an
uncommitted first put, an enable or disable with a write in flight, two
enables at once, a sweep between reconstruct pages and deleted tables in a
separate catalog database (the PostgreSQL file passed 20 runs in a row); a
negative control for each rule of the stores, the write hooks, the writer
exclusion, the pins and the sweep; release-build put throughput on a table
without recovery within 1.2 percent of a build without the feature
(sequential and 8 writers, 6 runs each).
@eliseong
feat(engine): enable, disable and describe point-in-time recovery
080c596 
On the service, UpdateContinuousBackups answers ENABLED at once.
RecoveryPeriodInDays is 1 to 35, checked before the table on an enable and
on a disable ("Member must have value greater than or equal to 1", "less
than or equal to 35"), and a disable that gives one is refused ("Invalid
Request: Cannot specify RecoveryPeriodInDays when disabling point-in-time
recovery."). The description carries RecoveryPeriodInDays and an
undocumented PointInTimeRecoveryWindowDays equal to it; right after an
enable EarliestRestorableDateTime and LatestRestorableDateTime are both the
enable instant in whole seconds, and LatestRestorableDateTime stays there
for the first minutes and then runs about five minutes behind the request.
A disable answers DISABLED with no window. RecoveryPeriodInDays is read as
an Integer: a fraction is truncated toward zero, a value outside 32 bits is
clamped to it in the message ('2147483647', '-2147483648'), another JSON
type is a SerializationException ("STRING_VALUE cannot be converted to
Integer", "TRUE_VALUE cannot be converted to Integer"), checked before the
TableName constraints; a bad period and a missing PointInTimeRecoveryEnabled
are one message, "2 validation errors detected", the period first. An enable
of a table being created is TableNotFoundException, and of a restore target
being loaded ContinuousBackupsUnavailableException "Backups are being
enabled for the table: <name>. Please retry later" (two samples of each).

ExtendDB refused every enable with ContinuousBackupsUnavailableException.
With the engine's new pitr feature and a backend that has an item-history
store, DescribeContinuousBackups and UpdateContinuousBackups read and write
the store as the service answers: the period read and checked before the
table (its type before the TableName check, through check_member_types,
which the server calls first), the CREATING refusals, the
fresh enable at 35 days, the period changed on an enabled table only when
given, the window from the enable second (or now less the period once that
is later) to now less 300 s and never before it, and the two period members.
RestoreTableToPointInTime answers as before. Without the feature, or on
MongoDB and Cassandra, nothing changes. The server, app and binary crates
gain a pitr feature that turns on the engine's and the selected backend's.

tests/test_pitr_enable.py runs against DynamoDB, and against a server built
with pitr when EXTENDDB_TEST_PITR is set; test_continuous_backups.py skips
there only the cells whose answer the feature changes.

Verified: the two files pass on DynamoDB (42 pass), on a SQLite pitr build
(54 pass, 1 skip) and on a PostgreSQL pitr build, and
test_continuous_backups.py passes on a build without pitr (13 pass); engine
unit tests; negative controls on the window-days member, the floor, the
period's reading and check order, the disable refusal and the CREATING
refusals, each run against a server or the engine.
@eliseong
feat(engine): point-in-time restore checks and reconstruct stream
f5cc587 
On the service, RestoreTableToPointInTime checks the time members' JSON
types first, then the model's member constraints (the overrides' shapes and
enums, a target name under 3 characters), then the source table and its
recovery, then the time, then the
overrides against the source, then the target name. The time's floor is the
reported EarliestRestorableDateTime and its ceiling the request's own time:
a restore to two seconds ago was accepted while the reported
LatestRestorableDateTime was minutes behind, two seconds ahead was refused
with InvalidRestoreTimeException "RestoreDateTime '<T>' must be within
EarliestRestorableDateTime '<E>' and LatestRestorableDateTime '<L>' for table
'<name>'": the requested time to the millisecond, the floor as reported,
and Latest as the request's time less five minutes. RestoreDateTime is
rounded to the millisecond and saturated to the 64-bit millisecond range,
RestoreSummary.RestoreDateTime keeps the milliseconds, and
UseLatestRestorableTime takes string spellings of true and false.
BillingModeOverride PROVISIONED without a throughput override is "Invalid
Request: Must override ProvisionedThroughput if BillingMode is overridden to
PROVISIONED", after the time check, and on a table with a GSI the table rule
comes before the index rule. A taken target name is TableAlreadyExists, a
target still being restored TableInUse "Table is in the process of being
created: <name>" (two samples of each).

Behind the pitr feature, and not yet called: the member checks, the time
check and the RestoreSummary time, the target's CreateTableInput from the
source's description with the overrides (the restore input a backup of the
table would give), the target name check, and the source's items at the
restore time as a stream over reconstruct pages. Adds
InvalidRestoreTimeException. RestoreTableToPointInTime still answers
PointInTimeRecoveryUnavailableException.

Verified: engine unit tests with the measured texts, order and window,
including the target name check; a negative control for the ceiling, the
millisecond rounding, the table rule's order, the target name refusals and
the stream's paging.
@eliseong
ci: build, lint and test the point-in-time recovery features
62889b9 
No job built the pitr features, so the item-history stores, the write hooks
and the continuous-backups path of the engine were neither linted nor
tested. clippy gains a pass with them on; the unit test job runs the storage,
SQLite and engine crates with them; the PostgreSQL storage job runs
tests/pitr_item_history.rs, which fails rather than skips under
EXTENDDB_TEST_PG_REQUIRED; a new build job builds a SQLite and a PostgreSQL
server with pitr once, and a new test job runs the continuous-backups wire
tests against each, with PostgreSQL started only on the PostgreSQL leg and
the server's log printed on failure. actionlint reports nothing new.
@eliseong
eliseong requested a review from amrith as a code owner November 10, 2026 18:45
@eliseong eliseong self-assigned this Nov 10, 2026
@eliseong
fix(storage-postgres): take history commit times under the row lock
74a4e29 
A reconstruct keeps, for each key, the row with the highest seq among the
rows committed at or before the restore time. A put or delete with no
condition, index or stream read the ExtendDB server's clock before its
statement took the item's row lock, so a racing write could take the
lower time and the higher seq. With 16 writers on one key for 20 s,
2,062 and 2,117 of about 11,800 rows had a time earlier than the row
before them, by up to 19 ms; a restore inside such a gap returns a value
committed after the restore time. Two ExtendDB servers whose clocks were
5 s apart gave gaps of 5 s on every path. On the service, two writes of
one key 400 ms apart within one second restore to the first at the
midpoint and to the second after it (two samples).

Every PostgreSQL history row now reads the database server's clock,
clock_timestamp(), after its data write: the autocommit put and delete in
their own statement, the other writes in the history insert, and a
transaction once, at its first recorded member, for all of its members.
Every ExtendDB server of one database reads the same clock. SQLite reads
the server's clock under its write lock and is unchanged (0 of about
11,800 rows, two samples).

Verified: racing_writes_of_one_key_take_their_times_in_seq_order passes
three runs in a row and fails with the earlier clock read (546 and 533 of
about 2,300 rows); the same 16-writer load gives 0 inversions on both
paths, and 0 with one of two servers 5 s ahead (two samples each); the
PostgreSQL storage tests pass with and without the feature (400 and 439);
release-build put throughput on a table without recovery is within 1.3
percent of a build without the feature (sequential and 8 writers, 6 runs
each).
@eliseong
eliseong marked this pull request as draft November 10, 2026 22:12
@hitesh96db hitesh96db mentioned this pull request Nov 11, 2026
Open
5 of 8 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amrith amrith Awaiting requested review from amrith amrith is a code owner
@c33howard c33howard Awaiting requested review from c33howard c33howard is a code owner
@jcshepherd jcshepherd Awaiting requested review from jcshepherd jcshepherd is a code owner
@pdf-amzn pdf-amzn Awaiting requested review from pdf-amzn pdf-amzn is a code owner
@LeeroyHannigan LeeroyHannigan Awaiting requested review from LeeroyHannigan LeeroyHannigan is a code owner
@robinnsc robinnsc Awaiting requested review from robinnsc robinnsc is a code owner
@hitesh96db hitesh96db Awaiting requested review from hitesh96db hitesh96db is a code owner
@navareamol navareamol Awaiting requested review from navareamol navareamol is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@eliseong eliseong

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@eliseong