Repository navigation
fix(auth): enforce dynamodb:LeadingKeys on Query, batch and transaction operations - #418
Merged
Merged
Conversation
…on ops extract_leading_keys returned None for Query, BatchGetItem, BatchWriteItem, TransactGetItems and TransactWriteItems, and the batch/transaction branch of authorize_request passed RequestParams::default(). With the partition key absent, a ForAllValues:StringEquals dynamodb:LeadingKeys condition is vacuously satisfied, so a principal fenced to its own partition could read or write any partition through those operations. Item operations were unaffected. Measured against live DynamoDB and IAM on 2026-10-20: - Query on the base table or an LSI evaluates LeadingKeys against the partition key value in the key condition (an LSI shares the base partition key). - Query on a GSI evaluates against the queried index's partition key value. - Scan sets no LeadingKeys (a ForAllValues allow is vacuous, a ForAnyValue allow is not). - BatchGetItem and BatchWriteItem authorize per table over the set of that table's keys; TransactGetItems and TransactWriteItems authorize per item. - Shapes the engine rejects before returning data (an unparseable, non equality or placeholder-unresolved key condition, a missing key condition, a missing table) keep the key absent, so they return ValidationException or ResourceNotFoundException as the service does rather than AccessDeniedException. A new module crates/server/src/leading_keys.rs extracts the key values. request_helpers.rs routes the single-table and batch/transaction paths through it. condition.rs, the resource ARN derivation and build_resource_arn are unchanged.
…he key absent A table key info lookup error was mapped to None together with a genuine not-found, so the authorization path saw no partition key and left dynamodb:LeadingKeys absent. A ForAllValues:StringEquals LeadingKeys condition is satisfied vacuously when the key is absent, so a lookup failure on an existing table authorized the request and the engine then read the table on its own lookup. Authorization now tells the two apart. A missing table keeps the key absent and the engine returns ResourceNotFoundException, which is what the service returns for a LeadingKeys principal on a missing table. Any other lookup failure returns an internal error before dispatch, so no request is authorized on a key the server could not determine. Removed the fail-open get_optional helper so the auth path reads the three-way get result directly.
… key Live DynamoDB accepts a GSI whose partition key has two HASH attributes and rejects a two-HASH base table. Measured 2026-10-20: a Query on such a GSI evaluates dynamodb:LeadingKeys against the first HASH attribute value only. Under ForAllValues:StringEquals ["alice"] a query of the first attribute set to alice was allowed for any second-attribute value, and the first attribute set to bob was denied; a policy naming the second attribute's value denied even a matching first attribute. The extraction reads the first HASH attribute of the queried key schema, which already matches this behaviour. Added unit tests for the extraction and dual-target wire tests that pin the first-attribute binding, the second attribute staying outside LeadingKeys, and the base partition key being unchanged when the index exists. No extraction change was required.
…tition key The authorization layer and the query engine each resolve the partition key from a KeyConditionExpression or legacy KeyConditions. If an input parsed in the engine but authorization extracted nothing, the request would be authorized with dynamodb:LeadingKeys absent and then served. Both paths build on the same extenddb_core::expression primitives (tokenize, parse_key_condition, resolve_pk_sk, resolve_multipart) and both take the first key-schema element as the partition attribute; key-condition values are placeholders in both. Added a differential unit test that, over a corpus of key-condition shapes (placeholder and name-reference partition keys, nested parentheses, clause order, whitespace, begins_with and BETWEEN sort-key conditions, legacy KeyConditions with EQ and non-EQ operators, LSI and GSI queries, a two-attribute GSI partition key), asserts that every input the engine executes yields the same partition value from authorization. A separate test pins the token-count limit boundary, the one point the two tokenizer entry points could differ, as identical.
A numeric (N) partition key was compared against dynamodb:LeadingKeys as the raw request string. The engine normalizes an N value on write and lookup, so 10, 010, 10.0, 1E1 and +10 all address one stored partition, and the service compares the request's normalized value against the policy value. The raw-string comparison denied a principal its own partition when the client sent a non-canonical spelling, and served a partition under a non-canonical fence value the service treats as matching nothing. Normalize an N value through validate_and_normalize_number in extract_typed_value; leave S and B as sent and leave the policy value as written. The single change covers the item, Query, batch and transaction paths. Add unit tests and dual-target wire cases for both directions on each operation class.
key_info_for_authz mapped every key-info lookup error other than TableNotFound to a generic InternalServerError, so one database fault returned 503 ServiceUnavailable when the engine hit it and 500 InternalServerError when authorization hit it first. Route the error through the engine's storage_err_to_dynamo so both layers return the same wire error for the same fault: a transient or connection fault is ServiceUnavailable, the catalog and backend-internal faults are InternalServerError, and a not-active table is ResourceInUseException. TableNotFound still yields an absent key, and every other fault still fails closed before dispatch so no request is authorized on a key the server could not determine. Expose storage_err_to_dynamo from the engine crate and cover each fault class with a unit test.
…saction Query, BatchGetItem, BatchWriteItem, TransactGetItems and TransactWriteItems now evaluate dynamodb:LeadingKeys, so a fenced principal's cross-partition requests on those operations return AccessDeniedException after upgrade, while own-partition access is unchanged. Add a behaviour-change entry to the upgrade manual and a short operation list to the security model. Both record that a GSI Query binds the index partition key value, that Scan and SearchVectors carry no LeadingKeys so a ForAllValues condition does not fence them, and that a numeric key is compared in normalized form. No catalog migration is required.
…y targets The batch and transaction authorization path parsed the request body twice: it built a full set of targets with an empty key-info map only to read the table names, then built the targets again with the populated map. The fallback onto the first set was dead code, since the second build always returns Some for a batch or transaction operation. Add batch_transact_tables, which walks the body once for the distinct table names, and build the targets a single time from the populated key-info map. Behaviour is unchanged and is covered by the existing unit and wire tests.
…Vectors The module comment claimed every request that reaches the engine carries its LeadingKeys. Scan and SearchVectors reach the engine and return data but carry no LeadingKeys, because neither targets a single partition, so a ForAllValues condition is satisfied vacuously and an operator fences them with ForAnyValue or a Null guard. Reword the comment to state this and to name both operations.
…havior Live measurement of the numeric non-canonical cases on the batch and transaction paths (oracle-ncanon-ops.json) found that the service's write and transaction request handlers allow the spellings 010, +10 and 1.0E1 of the own partition but deny 10.0 and 1E1, while every read handler allows all five. Assert the transaction cases with spellings the service allows on those handlers, and give the batch read one key per request since 1E1 and 10.0 both normalize to the one partition and cannot share a batch. The divergent write-path spellings are recorded in the report under Divergences kept.
…oDB does per operation DynamoDB writes an N partition key into dynamodb:LeadingKeys in one of two forms depending on the operation, and matches the policy value as written: - GetItem, PutItem, UpdateItem, DeleteItem, Query (base table, LSI and GSI, KeyConditionExpression and KeyConditions) and BatchGetItem use the normalized number, so 010, 10.0, 1E1 and +10 are all "10". - BatchWriteItem put and delete requests, TransactGetItems and every TransactWriteItems action use the number as Java's BigDecimal prints it, which keeps the scale the request was written with: 10.0 stays "10.0", 1E1 is "1E+1", 100E-2 is "1.00", and 010, +10 and 1.0E1 are "10". ExtendDB compared the normalized number on every operation, so a principal fenced to ["10"] could write partition 10 spelled 10.0 or 1E1 through BatchWriteItem or a transaction where DynamoDB refuses it, and a fence or Deny written as ["10.0"] matched nothing on those operations. Measured against live DynamoDB on 2026-10-20 over 21 spellings (including 1E-7, 1.23E+4, 0E+5, 1.00E-130 and negative values), with each result repeated 25 times. The unit tests pin every measured spelling and the wire tests pin each operation under allow fences and a Deny statement.
…it list The upgrade audit list still said to write numeric LeadingKeys values in canonical form. It now points at the per-operation forms the entry above describes.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 21, 2026 02:41
This was referenced Oct 21, 2026
robinnsc
reviewed
Oct 21, 2026
| let mut key_info_by_table = HashMap::new(); | ||
| for table in tables { | ||
| if let Some(ki) = | ||
| key_info_for_authz(state.table_key_info_cache.get(account_id, &table).await)? |
Collaborator
There was a problem hiding this comment.
I think there's a window where the new batch and transaction checks can be skipped. Authorization reads key info through table_key_info_cache, which stores a missing table as a negative entry for negative_ttl (5s by default), but the four batch/transaction engine handlers read ctx.storage.table_key_info directly rather than the cache (e.g. batch_write_item.rs:115, batch_get_item.rs:191), so the two can disagree. A table in CREATING reports TableNotFound, and the CREATING to ACTIVE flip in the control-plane worker doesn't invalidate the cache, so if any request touches the table while it's CREATING, then for the rest of that negative TTL authorization sees no table, leaves leading_keys absent, and a ForAllValues fence passes vacuously, while the engine sees the ACTIVE table and runs the request.
I reproduced this on a single-node SQLite build of this branch: with a principal fenced to ["alice"], one BatchWriteItem sent while the table was CREATING was enough for BatchWriteItem, BatchGetItem, TransactGetItems and TransactWriteItems on bob to all succeed for about 4s after the table went ACTIVE, and all of them were denied once the entry expired; a control run with no request during CREATING was denied throughout. Restore and import targets go through the same kind of flip.
One option would be to pass key_info_by_table into the engine, or route those handlers through table_key_info_lookup (which the OperationContext doc comment at engine/src/lib.rs:349 already says they use), so authorization and the engine act on one snapshot. That would also remove the second, uncached key-info fetch these requests now make in the engine after the one here is discarded. Invalidating the entry on status transitions would separately close the related case where single-table ops return ResourceNotFound for an ACTIVE table during the same window.
Collaborator
Author
There was a problem hiding this comment.
Reproduced on SQLite. Fixed in 485d969: authorization now hands the engine every key info it read, including the tables it found missing, and the four batch and transaction handlers resolve those tables from it instead of reading storage again, so the second fetch is gone too. That path now reads storage rather than the cache, so a request sent during CREATING doesn't leave a negative entry it can read later.
test_batch_and_transaction_fenced_once_a_created_table_is_active sends a BatchWriteItem and a GetItem while the table is CREATING, then checks that once it's ACTIVE all four operations are denied on bob and served on alice, and that bob is never served. It fails on 9be0faa.
I left the single-table negative entry as it is. On DynamoDB a GetItem or Query sent right after DescribeTable reports ACTIVE also returns ResourceNotFoundException for 2 to 4 seconds (measured today), and ours clears within the 5 second TTL.
| ) | ||
| .await?; | ||
| } else { | ||
| for target in targets { |
Collaborator
There was a problem hiding this comment.
On main the (action, table) pairs were de-duplicated, so a 100-item TransactWriteItems on one table ran one authorization; now it runs one check_authorization per item, including for principals with no LeadingKeys condition anywhere in their policies. Each check repeats the five authz cache lookups, clones the principal and resource tag maps and runs a full policy evaluation, and with auth.cache.enabled = false each one goes to the catalog, so that request goes from 5 catalog queries to 500. This loop also runs before the engine's 100-item limit in transact_write_items.rs:64, so the only bound on the number of checks is the 16 MiB body limit, which is a couple of hundred thousand small ConditionCheck entries from any principal allowed on the table. Might be worth de-duplicating targets on (action, table, leading_keys), which keeps the per-item semantics since identical triples give the same decision, and rejecting an oversized TransactItems before the targets are built.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 485d969 and b03cf39. The count limits are now checked on the request body before any table is read or any policy is evaluated, using the handlers' own checks and errors: an empty or 101-item transaction, an empty RequestItems, a table with no keys or requests, more than 100 tables or keys in a BatchGetItem, and more than 25 tables or requests in a BatchWriteItem. DynamoDB returns ValidationException for each of these to a principal with no policy, so this order matches the service.
Targets are de-duplicated on (action, table, leading_keys) as you suggested, so 100 ConditionChecks on one partition are one evaluation. A transaction reads at most 400 tables and evaluates at most 400 targets (100 items with four sub-operations each, before the handler rejects an item that has more than one). A BatchGetItem naming 50,000 tables with no keys took 3.3 seconds on PostgreSQL before; it now gets a ValidationException in about 50 ms.
| `ForAllValues:StringEquals dynamodb:LeadingKeys` condition on them was satisfied | ||
| vacuously and a principal fenced to its own partition could read or write any | ||
| partition through them. The item operations (`GetItem`, `PutItem`, `UpdateItem`, | ||
| `DeleteItem`) already enforced the key and are unchanged. |
Collaborator
There was a problem hiding this comment.
Small, but the item operations aren't quite unchanged for numeric keys: main put the raw string into LeadingKeys, so under a fence of ["10"] a GetItem with 010, 10.0 or 1E1 was denied and is now allowed, and under a fence of ["010"] or ["1E1"] the matching spelling was allowed and is now denied, which test_num_noncanonical_fence_matches_nothing pins. It might be worth saying so here, in the Who is affected paragraph and in the Breaking changes section of the description, since a principal with a non-canonical numeric fence on item operations would see a change on upgrade even if it never uses Query, batch or transactions.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 592c75d. What changed, Who is affected and the audit list in the upgrade manual now cover the numeric change on the item operations, with your examples, and the Breaking changes section of the description says the same.
I checked them against DynamoDB today: under ["10"] a GetItem on 010, 10.0, 1E1, +10 and 1.0E1 is allowed and 20 is denied, and under ["010"], ["1E1"] and ["10.0"] every spelling is denied, including the fence's own.
| /// from the placeholder. If this yields a value, authorization must extract the | ||
| /// same one, or a Query the engine serves would reach the evaluator with | ||
| /// LeadingKeys absent (a bypass) or bound to the wrong partition. | ||
| fn engine_partition_value( |
Collaborator
There was a problem hiding this comment.
Since a None from authorization is only safe as long as the engine rejects the same shape, this test carries a lot of the PR's safety argument, but engine_partition_value is a hand-written mirror of query.rs and reuses this module's own names_map, values_map, extract_typed_value and resolve_attr_name, so a bug in those helpers would appear on both sides, and a change to how the engine picks the partition attribute (key_schema[0] at query.rs:275) or desugars KeyConditions (legacy_filter.rs) wouldn't make it fail. Could consider having authorization and the engine call one shared extraction function in core, or, if that's a larger change than this PR wants, driving the comparison through handle_query against a real table so the engine side is the actual engine.
Collaborator
Author
There was a problem hiding this comment.
Replaced in 27cdd8d. The hand-written copy is gone. test_query_shapes_authorize_the_partition_the_engine_reads sends 26 key condition shapes through the server as the table owner, as a principal fenced to the partition each one names, and as one fenced away from it, so the engine side is handle_query itself. Every expected result was measured against DynamoDB. With the legacy KeyConditions extraction removed from authorization, it fails on both legacy shapes.
The token limit test still compares the two sides, now against tokenize_for with ExpressionKind::KeyCondition, which is what parse_key_condition_expr calls.
robinnsc
previously approved these changes
Oct 21, 2026
Authorization read table key info through the key info cache, which keeps a missing table as a negative entry for its TTL (5 seconds by default). The batch and transaction handlers read storage directly. A table that is CREATING reads as missing, and the CREATING to ACTIVE transition does not clear the entry, so one request during CREATING left authorization seeing no table, and evaluating dynamodb:LeadingKeys as absent, for the rest of the TTL, while the engine found the ACTIVE table and ran BatchWriteItem, BatchGetItem, TransactGetItems and TransactWriteItems on any partition. Authorization now returns every key info it read, including the tables it found missing, and the engine resolves those tables from that snapshot (OperationContext::authorized_key_info, replacing pre_fetched_key_info). A table authorization evaluated as missing is missing to the engine for the rest of the request. The batch and transaction path reads each distinct table from storage once, as those handlers did, so it sees the table as soon as it is ACTIVE, and the engine's second read of the same key info is gone. Two more changes bound the work authorization does per request: - The batch and transaction item limits are checked on the request body before any key info is read or any policy is evaluated, with the handlers' own errors (request_limits.rs). DynamoDB returns these ValidationExceptions before authorization, including to a principal with no permissions. - Transaction targets with the same action, table and LeadingKeys are evaluated once.
… the server The unit test that compared authorization with the engine on Query key conditions used a hand-written copy of the engine's parsing built from the same helpers as authorization, so a defect in those helpers appeared on both sides. The previous commit removes it. This one replaces it with a wire test: each key condition shape is sent as the table owner, as a principal fenced to the partition it names and as a principal fenced away from it, and the owner's rows show which partition the engine read. The engine side is the server's own Query path. Two more wire tests cover the fixes in the previous commit. One sends a batch write and a GetItem while a table is CREATING and then checks that the four batch and transaction operations are denied on another partition and served on the principal's own as soon as the table is ACTIVE. The other checks that oversized batch and transaction requests return ValidationException to a principal with no policy and to one whose fence excludes every key. Every expected outcome was measured against DynamoDB. For four rejected Query shapes the service and ExtendDB return different error codes to the principal fenced away from the value, and the test asserts only that no data is returned for those.
Before this branch the item operations wrote a number partition key into dynamodb:LeadingKeys as the request spelled it. They now normalize it, so a fence of ["10"] allows a GetItem on 010, 10.0 or 1E1 that was denied before, and a fence written as ["010"] or ["1E1"] denies the matching spelling it allowed before. The upgrade manual said the item operations were unchanged. It now says so in What changed, Who is affected and the audit list, and records that oversized batch and transaction requests return ValidationException before authorization.
The pre-authorization check counted items but not tables, and a table with no items counted as zero. A BatchGetItem or BatchWriteItem naming thousands of tables with no keys or requests passed it, and authorization read every table and evaluated a policy for each before the first denial: 50,000 tables took 3.3 seconds on PostgreSQL. An empty TransactItems or RequestItems, which produces no target, was evaluated against the table wildcard and denied to a principal without that permission. DynamoDB returns ValidationException for each of these before it evaluates any policy, including to a principal with no permissions (measured 2026-10-21): more than 100 tables in a BatchGetItem, more than 25 in a BatchWriteItem, an empty TransactItems or RequestItems, and a table with no keys or requests. The handlers and the server now run one set of count checks, check_batch_counts and check_transact_count, in one order: an empty container, too many tables, too many items in a table, too many items in total, a table with no items. The messages for a table count over the limit and for a table with no items are the service's. The engine's key info snapshot now treats an entry for another account as an internal error instead of reading the table again through the cache.
A TransactWriteItems item that names several sub-operations is rejected by the handler, after authorization has read the table of each one, so the bound on authorization reads for a transaction is 400 tables, not 100. The comments now say so.
…ement The key info cache module doc said the batch and transaction handlers read through the cache; they now resolve their tables from what authorization read and fall back to storage. The authorized_key_info field doc said the map is empty in developer mode; the single-table entry is recorded before the developer-mode return. The CREATING test now accepts ResourceNotFoundException for up to 15 seconds after the table is ACTIVE. DynamoDB returned it for 2 to 4 seconds after DescribeTable reported ACTIVE (measured 2026-10-21) and then denied the other partition and served the principal's own. The test still fails if the other partition is served at any point. The module header dates the measurements.
…-batch # Conflicts: # crates/server/src/handler.rs # crates/server/src/request_helpers.rs
robinnsc
self-requested a review
October 22, 2026 19:36
robinnsc
approved these changes
Oct 22, 2026
robinnsc
added a commit
that referenced
this pull request
Oct 22, 2026
Brings in #418 (LeadingKeys on Query, batch and transaction operations) and #421 (resource tags only as aws:ResourceTag). authorize_request in request_helpers.rs keeps main's key-info snapshot (AuthorizedKeyInfo, key_info_for_authz, storage reads for batch and transaction tables) and LeadingKeys extraction (leading_keys::single_table_leading_keys, batch_transact_targets), and this branch's resource derivation (authz_targets, table_resource_arn, the DescribeEndpoints, tag-shape, operation-ARN and GetRecords branches, base_params). A TableName given as a table ARN is looked up through key_lookup_name for LeadingKeys and left out of the snapshot handed to the engine, as before. Batch and transaction tables are read under the name the request uses, so a table named by ARN is recorded as missing and the engine answers ResourceNotFound from the same snapshot. The empty-target fallback uses authz_resource::table_wildcard, now pub(crate), in place of build_resource_arn. The upgrade manual keeps both entries, the LeadingKeys entry first. After #421 dynamodb:ResourceTag is absent for every resource, so the index tag inheritance tests in test_authz_resource_arn.py exercise aws:ResourceTag only, and the admin guide no longer says index reads evaluate dynamodb:ResourceTag.
robinnsc
added a commit
that referenced
this pull request
Oct 23, 2026
…; recover abandoned restores
Backup and restore now reproduce the table that was backed up, behave the
same on PostgreSQL, SQLite, MongoDB, and Cassandra, and match DynamoDB on
the points measured against the service.
Engine. The engine resolves a backup into one validated CreateTableInput
before any backend sees it: it runs the multi-part-key and vector-index
refusals, applies the overrides, and runs validate_create_table. Backends
only create the target, register the restore, and copy. The trait gains
backup_restore_source; restore_table_from_backup takes a resolved
RestoreRequest; restore_table_to_point_in_time is removed. Once the request
is authenticated and authorized, the whole restore runs in a task of its
own, so a client that disconnects does not cancel it part-way.
Restore runs after the response on every backend: the target is returned
CREATING and copied by a background task. While it is CREATING, DeleteTable
on it is ResourceInUseException and DeleteBackup on its source is
BackupInUseException. A target whose server stopped mid-copy is removed and
logged: PostgreSQL after 60 s, SQLite at the next start, MongoDB and
Cassandra by a heartbeat sweep at startup and every 15 s.
Wire behaviour: RestoreDateTime is the backup's creation time and
SourceTableArn is set; RestoreSummary appears only while CREATING, on every
table description (TableDescription::prepare_for_response). DescribeBackup
reports SourceTableDetails.ProvisionedThroughput and SourceTableFeatureDetails.
ListBackups pages with Limit and LastEvaluatedBackupArn and carries TableId.
Restoring into an existing name is TableAlreadyExistsException; a malformed
BackupArn is ValidationException, checked before the target name.
SSESpecificationOverride, GlobalSecondaryIndexOverride,
LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as
the service applies them (an empty list restores no indexes of that kind, a
subset restores those, a GSI entry's throughput is the index's); an unknown
or changed index definition, a throughput member contradicting the billing
mode, {0, 0}, and BillingModeOverride=PROVISIONED with a GSI whose
throughput no entry supplies are ValidationException with the service's
text. TableClass is not carried across; the target is STANDARD. ListBackups
filters on BackupType and the time range; its cursor must match the
account, region, and TableName filter (a listed backup's stored ARN in an
earlier region is accepted, so paging passes it), and the time bounds are
checked. A
BackupArn in another partition is a ValidationException. BackupArn and
ExclusiveStartBackupArn shapes are checked as the service checks them
("Invalid Request: BackupArn is not valid" / "Invalid Backup ARN"),
surrounding whitespace is trimmed, and an ARN of the caller's account in
another region resolves (looked up in the server's region first, then as
written, so backups created under an earlier server.region stay
reachable); the 13-digit ids of earlier builds still resolve. BackupName
and TableName are validated as the model says (3 to 255 characters,
[a-zA-Z0-9_.-]+) with the service's "validation error(s) detected" texts;
TableName may be the table's ARN; CreateBackup of a missing or CREATING
table is TableNotFoundException. DescribeBackup propagates a failure to
read the definition as an internal error and degrades only for a version
it does not know; DeleteBackup returns the full description DELETED.
Restoring into a CREATING restore target is TableInUseException and a
second restore of a backup whose restore is CREATING is
BackupInUseException, decided inside each backend's registration
transaction, claim, or lightweight transaction. The key-info cache stores
no negative entry for a name the catalog has in a non-ACTIVE state, so the
first read after ACTIVE succeeds; authorization's key-info snapshot (#418)
records such a name as missing, so the engine does not look it up again. A
PROVISIONED backup whose GSI has no throughput restores with the table's
throughput. CreateTable and UpdateTable refuse GSI throughput below 1 on
PROVISIONED tables and more than 100 projected NonKeyAttributes.
CreateBackup of a table with vector indexes is refused. Backup ids are 14
zero-padded millisecond digits on every backend.
Authorization: backup operations use main's resource authorization (#419).
A BackupArn is trimmed of surrounding ASCII whitespace before it is
canonicalized, as the engine trims it before resolving the backup, so a
Deny on a backup ARN cannot be bypassed with a trailing newline.
PostgreSQL: the copy commits per batch with one multi-row INSERT per index;
the lock session is heartbeated; at most 4 restores per account and
min(16, pool_size / 2) per server, then LimitExceededException after 30 s,
with the whole restore, admission included, in a task the request waits
on; only SQLSTATE 53300 on the lock connection is LimitExceeded. The backup
row is locked FOR UPDATE while a restore registers. serve refuses a
catalog that does not record one of the binary's migration files, and
converges a pre-release 0.0.4 catalog shape with a 5 s lock timeout.
migrate uses the data database name init recorded and accepts --data-db.
SQLite: serve refuses a pre-release 0.0.4 shape; the backup_items rebuild is
named in the migrate preview and logged; the pool_size clamp is logged.
CreateBackup is admitted against the pool (pool_size / 2 - 1 at once, then
LimitExceededException after 30 s). A GSI added by UpdateTable with
throughput on a PROVISIONED table no longer makes DescribeTable fail, and
rows earlier builds wrote are read.
MongoDB: twenty concurrent restores no longer wedge the pool (the
registration hand-over held a checked-out connection while waiting for
another; both futures are polled together now); restores are admitted per
account (4) and per server (min(16, pool / 2)), LimitExceededException
after 30 s, and a registration longer than 60 s answers an internal error
while the restore continues. UpdateTable on a CREATING restore target is
refused. DeleteBackup and the sweep never release a delete claim once the
data drop may have started; a held delete claim answers BackupInUse; a
concurrent restore of one backup retries for 2 s, then BackupInUse; catalog
reads are pinned to the primary with read concern local and writes to
w: majority, and a weaker write concern in the connection string is
refused at startup; heartbeats refresh at once. The test-only protocol
model covers two restores, the delete and its stop points, and the sweep's
delete takeover.
Cassandra: migration V005 (catalog 0.0.4) stores backup definitions and
restore registrations (registered with a lightweight transaction that
requires the backup AVAILABLE and unclaimed); the copy uses the normal put path with GSIs written
synchronously; ordinary CreateTable does not name the V005 columns, so an
unmigrated catalog breaks only CreateBackup, RestoreTableFromBackup, and
DeleteBackup. The restore heartbeat stops on every exit of the copy. On an
unmigrated catalog the backup operations are a ValidationException naming
`extenddb migrate --yes`. UpdateTimeToLive on a table with an asynchronous
GSI is a ValidationException instead of a 500. init writes the
application credentials into the config, created with mode 0600. A
pre-existing defect is fixed: an on-demand GSI omitted
ProvisionedThroughput.
Catalog guards fail closed on an unparseable version. extenddb migrate:
the preview exits 3 when work is pending (main's resource-tag rewrite
included) and 0 when up to date; unrecorded
catalog migration files are pending; admin connection errors are one
plain line; --cassandra-user/--cassandra-pass and
EXTENDDB_CASSANDRA_PASSWORD are accepted.
Docs: differences-from-dynamodb, dynamodb-limits, the admin, upgrade, and
architecture manuals, and the MongoDB design doc are updated for four
backends.
Tests: tests/test_backup_restore_fidelity.py covers the restore surface on
all four backends; devtools/run-cassandra-tests runs its large-backup tests
with 10,000 items, because 60,000 made cdrs-tokio drop its only
connection. tests/test_ttl.py waits for ENABLING to end before disabling.
Signed-off-by: Scott Robinson <robinnsc@amazon.com>
5 of 8 tasks
robinnsc
added a commit
that referenced
this pull request
Oct 23, 2026
…; recover abandoned restores
Backup and restore now reproduce the table that was backed up, behave the
same on PostgreSQL, SQLite, MongoDB, and Cassandra, and match DynamoDB on
the points measured against the service.
Engine. The engine resolves a backup into one validated CreateTableInput
before any backend sees it: it runs the multi-part-key and vector-index
refusals, applies the overrides, and runs validate_create_table. Backends
only create the target, register the restore, and copy. The trait gains
backup_restore_source; restore_table_from_backup takes a resolved
RestoreRequest; restore_table_to_point_in_time is removed. Once the request
is authenticated and authorized, the whole restore runs in a task of its
own, so a client that disconnects does not cancel it part-way.
Restore runs after the response on every backend: the target is returned
CREATING and copied by a background task. While it is CREATING, DeleteTable
on it is ResourceInUseException and DeleteBackup on its source is
BackupInUseException. A target whose server stopped mid-copy is removed and
logged: PostgreSQL after 60 s, SQLite at the next start, MongoDB and
Cassandra by a heartbeat sweep at startup and every 15 s.
Wire behaviour: RestoreDateTime is the backup's creation time and
SourceTableArn is set; RestoreSummary appears only while CREATING, on every
table description (TableDescription::prepare_for_response). DescribeBackup
reports SourceTableDetails.ProvisionedThroughput and SourceTableFeatureDetails.
ListBackups pages with Limit and LastEvaluatedBackupArn and carries TableId.
Restoring into an existing name is TableAlreadyExistsException; a malformed
BackupArn is ValidationException, checked before the target name.
SSESpecificationOverride, GlobalSecondaryIndexOverride,
LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as
the service applies them (an empty list restores no indexes of that kind, a
subset restores those, a GSI entry's throughput is the index's); an unknown
or changed index definition, a throughput member contradicting the billing
mode, {0, 0}, and BillingModeOverride=PROVISIONED with a GSI whose
throughput no entry supplies are ValidationException with the service's
text. TableClass is not carried across; the target is STANDARD. ListBackups
filters on BackupType and the time range; its cursor must match the
account, region, and TableName filter (a listed backup's stored ARN in an
earlier region is accepted, so paging passes it), and the time bounds are
checked. A
BackupArn in another partition is a ValidationException. BackupArn and
ExclusiveStartBackupArn shapes are checked as the service checks them
("Invalid Request: BackupArn is not valid" / "Invalid Backup ARN"),
surrounding whitespace is trimmed, and an ARN of the caller's account in
another region resolves (looked up in the server's region first, then as
written, so backups created under an earlier server.region stay
reachable); the 13-digit ids of earlier builds still resolve. BackupName
and TableName are validated as the model says (3 to 255 characters,
[a-zA-Z0-9_.-]+) with the service's "validation error(s) detected" texts;
TableName may be the table's ARN; CreateBackup of a missing or CREATING
table is TableNotFoundException. DescribeBackup propagates a failure to
read the definition as an internal error and degrades only for a version
it does not know; DeleteBackup returns the full description DELETED.
Restoring into a CREATING restore target is TableInUseException and a
second restore of a backup whose restore is CREATING is
BackupInUseException, decided inside each backend's registration
transaction, claim, or lightweight transaction. The key-info cache stores
no negative entry for a name the catalog has in a non-ACTIVE state, so the
first read after ACTIVE succeeds; authorization's key-info snapshot (#418)
records such a name as missing, so the engine does not look it up again. A
PROVISIONED backup whose GSI has no throughput restores with the table's
throughput. CreateTable and UpdateTable refuse GSI throughput below 1 on
PROVISIONED tables and more than 100 projected NonKeyAttributes.
CreateBackup of a table with vector indexes is refused. Backup ids are 14
zero-padded millisecond digits on every backend.
Authorization: backup operations use main's resource authorization (#419).
A BackupArn is trimmed of surrounding ASCII whitespace before it is
canonicalized, as the engine trims it before resolving the backup, so a
Deny on a backup ARN cannot be bypassed with a trailing newline.
PostgreSQL: the copy commits per batch with one multi-row INSERT per index;
the lock session is heartbeated; at most 4 restores per account and
min(16, pool_size / 2) per server, then LimitExceededException after 30 s,
with the whole restore, admission included, in a task the request waits
on; only SQLSTATE 53300 on the lock connection is LimitExceeded. The backup
row is locked FOR UPDATE while a restore registers. serve refuses a
catalog that does not record one of the binary's migration files, and
converges a pre-release 0.0.4 catalog shape with a 5 s lock timeout.
migrate uses the data database name init recorded and accepts --data-db.
SQLite: serve refuses a pre-release 0.0.4 shape; the backup_items rebuild is
named in the migrate preview and logged; the pool_size clamp is logged.
CreateBackup is admitted against the pool (pool_size / 2 - 1 at once, then
LimitExceededException after 30 s). A GSI added by UpdateTable with
throughput on a PROVISIONED table no longer makes DescribeTable fail, and
rows earlier builds wrote are read.
MongoDB: twenty concurrent restores no longer wedge the pool (the
registration hand-over held a checked-out connection while waiting for
another; both futures are polled together now); restores are admitted per
account (4) and per server (min(16, pool / 2)), LimitExceededException
after 30 s, and a registration longer than 60 s answers an internal error
while the restore continues. UpdateTable on a CREATING restore target is
refused. DeleteBackup and the sweep never release a delete claim once the
data drop may have started; a held delete claim answers BackupInUse; a
concurrent restore of one backup retries for 2 s, then BackupInUse; catalog
reads are pinned to the primary with read concern local and writes to
w: majority, and a weaker write concern in the connection string is
refused at startup; heartbeats refresh at once. The test-only protocol
model covers two restores, the delete and its stop points, and the sweep's
delete takeover.
Cassandra: migration V005 (catalog 0.0.4) stores backup definitions and
restore registrations (registered with a lightweight transaction that
requires the backup AVAILABLE and unclaimed); the copy uses the normal put path with GSIs written
synchronously; ordinary CreateTable does not name the V005 columns, so an
unmigrated catalog breaks only CreateBackup, RestoreTableFromBackup, and
DeleteBackup. The restore heartbeat stops on every exit of the copy. On an
unmigrated catalog the backup operations are a ValidationException naming
`extenddb migrate --yes`. UpdateTimeToLive on a table with an asynchronous
GSI is a ValidationException instead of a 500. init writes the
application credentials into the config, created with mode 0600. A
pre-existing defect is fixed: an on-demand GSI omitted
ProvisionedThroughput.
Catalog guards fail closed on an unparseable version. extenddb migrate:
the preview exits 3 when work is pending (main's resource-tag rewrite
included) and 0 when up to date; unrecorded
catalog migration files are pending; admin connection errors are one
plain line; --cassandra-user/--cassandra-pass and
EXTENDDB_CASSANDRA_PASSWORD are accepted.
Docs: differences-from-dynamodb, dynamodb-limits, the admin, upgrade, and
architecture manuals, and the MongoDB design doc are updated for four
backends.
Tests: tests/test_backup_restore_fidelity.py covers the restore surface on
all four backends; devtools/run-cassandra-tests runs its large-backup tests
with 10,000 items, because 60,000 made cdrs-tokio drop its only
connection. tests/test_ttl.py waits for ENABLING to end before disabling.
Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 23, 2026
…; recover abandoned restores
Backup and restore now reproduce the table that was backed up, behave the
same on PostgreSQL, SQLite, MongoDB, and Cassandra, and match DynamoDB on
the points measured against the service.
Engine. The engine resolves a backup into one validated CreateTableInput
before any backend sees it: it runs the multi-part-key and vector-index
refusals, applies the overrides, and runs validate_create_table. Backends
only create the target, register the restore, and copy. The trait gains
backup_restore_source; restore_table_from_backup takes a resolved
RestoreRequest; restore_table_to_point_in_time is removed. Once the request
is authenticated and authorized, the whole restore runs in a task of its
own, so a client that disconnects does not cancel it part-way.
Restore runs after the response on every backend: the target is returned
CREATING and copied by a background task. While it is CREATING, DeleteTable
on it is ResourceInUseException and DeleteBackup on its source is
BackupInUseException. A target whose server stopped mid-copy is removed and
logged: PostgreSQL after 60 s, SQLite at the next start, MongoDB and
Cassandra by a heartbeat sweep at startup and every 15 s.
Wire behaviour: RestoreDateTime is the backup's creation time and
SourceTableArn is set; RestoreSummary appears only while CREATING, on every
table description (TableDescription::prepare_for_response). DescribeBackup
reports SourceTableDetails.ProvisionedThroughput and
SourceTableFeatureDetails.
ListBackups pages with Limit and LastEvaluatedBackupArn and carries TableId.
Restoring into an existing name is TableAlreadyExistsException; a malformed
BackupArn is ValidationException, checked before the target name.
SSESpecificationOverride, GlobalSecondaryIndexOverride,
LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as
the service applies them (an empty list restores no indexes of that kind, a
subset restores those, a GSI entry's throughput is the index's); an unknown
or changed index definition, a throughput member contradicting the billing
mode, {0, 0}, and BillingModeOverride=PROVISIONED with a GSI whose
throughput no entry supplies are ValidationException with the service's
text. TableClass is not carried across; the target is STANDARD. ListBackups
filters on BackupType and the time range; its cursor must match the
account, region, and TableName filter (a listed backup's stored ARN in an
earlier region is accepted, so paging passes it), and the time bounds are
checked. A BackupArn in another partition is a ValidationException. BackupArn and
ExclusiveStartBackupArn shapes are checked as the service checks them
("Invalid Request: BackupArn is not valid" / "Invalid Backup ARN"),
surrounding whitespace is trimmed, and an ARN of the caller's account in
another region resolves (looked up in the server's region first, then as
written, so backups created under an earlier server.region stay
reachable); the 13-digit ids of earlier builds still resolve. BackupName
and TableName are validated as the model says (3 to 255 characters,
[a-zA-Z0-9_.-]+) with the service's "validation error(s) detected" texts;
TableName may be the table's ARN; CreateBackup of a missing or CREATING
table is TableNotFoundException. DescribeBackup propagates a failure to
read the definition as an internal error and degrades only for a version
it does not know; DeleteBackup returns the full description DELETED.
Restoring into a CREATING restore target is TableInUseException and a
second restore of a backup whose restore is CREATING is
BackupInUseException. Each backend checks the target name before the
backup and before a restore slot is taken; the target insert and the
registration transaction, claim, or lightweight transaction are the
fences. The key-info cache stores
no negative entry for a name the catalog has in a non-ACTIVE state, so the
first read after ACTIVE succeeds; authorization's key-info snapshot (#418)
records such a name as missing, so the engine does not look it up again. A
PROVISIONED backup whose GSI has no throughput restores with the table's
throughput. CreateTable and UpdateTable refuse GSI throughput below 1 on
PROVISIONED tables and more than 100 projected NonKeyAttributes.
CreateBackup of a table with vector indexes is refused. Backup ids are 14
zero-padded millisecond digits on every backend.
Authorization: backup operations use main's resource authorization (#419).
A BackupArn is trimmed of surrounding ASCII whitespace before it is
canonicalized, as the engine trims it before resolving the backup, so a
Deny on a backup ARN cannot be bypassed with a trailing newline.
PostgreSQL: the copy commits per batch with one multi-row INSERT per index;
the lock session is heartbeated; at most 4 restores per account and
min(16, pool_size / 2) per server, then LimitExceededException after 30 s,
with the whole restore, admission included, in a task the request waits
on; only SQLSTATE 53300 on the lock connection is LimitExceeded. The backup
row is locked FOR UPDATE while a restore registers. serve refuses a
catalog that does not record one of the binary's migration files, and
converges a pre-release 0.0.4 catalog shape with a 5 s lock timeout.
migrate uses the data database name init recorded and accepts --data-db.
SQLite: serve refuses a pre-release 0.0.4 shape; the backup_items rebuild is
named in the migrate preview and logged; the pool_size clamp is logged.
CreateBackup is admitted against the pool (pool_size / 2 - 1 at once, then
LimitExceededException after 30 s). A GSI added by UpdateTable with
throughput on a PROVISIONED table no longer makes DescribeTable fail, and
rows earlier builds wrote are read.
MongoDB: twenty concurrent restores no longer wedge the pool (the
registration hand-over held a checked-out connection while waiting for
another; both futures are polled together now); restores are admitted per
account (4) and per server (min(16, pool / 2)), LimitExceededException
after 30 s, and a registration longer than 60 s answers an internal error
while the restore continues. UpdateTable on a CREATING restore target is
refused. DeleteBackup and the sweep never release a delete claim once the
data drop may have started; a held delete claim answers BackupInUse; a
concurrent restore of one backup retries for 2 s, then BackupInUse; catalog
reads are pinned to the primary with read concern local and writes to
w: majority, and a weaker write concern in the connection string is
refused at startup; heartbeats refresh at once. The test-only protocol
model covers two restores, the delete and its stop points, and the sweep's
delete takeover.
Cassandra: migration V005 (catalog 0.0.4) stores backup definitions and
restore registrations (registered with a lightweight transaction that
requires the backup AVAILABLE and unclaimed); the copy uses the normal
put path with GSIs written
synchronously; ordinary CreateTable does not name the V005 columns, so an
unmigrated catalog breaks only CreateBackup, RestoreTableFromBackup, and
DeleteBackup. The restore heartbeat stops on every exit of the copy. On an
unmigrated catalog the backup operations are a ValidationException naming
`extenddb migrate --yes`. UpdateTimeToLive on a table with an asynchronous
GSI is a ValidationException instead of a 500. init writes the
application credentials into the config, created with mode 0600. A
pre-existing defect is fixed: an on-demand GSI omitted
ProvisionedThroughput.
Catalog guards fail closed on an unparseable version. extenddb migrate:
the preview exits 3 when work is pending (main's resource-tag rewrite
included) and 0 when up to date; unrecorded
catalog migration files are pending; admin connection errors are one
plain line; --cassandra-user/--cassandra-pass and
EXTENDDB_CASSANDRA_PASSWORD are accepted.
Docs: differences-from-dynamodb, dynamodb-limits, the admin, upgrade, and
architecture manuals, and the MongoDB design doc are updated for four
backends.
Tests: tests/test_backup_restore_fidelity.py covers the restore surface on
all four backends; devtools/run-cassandra-tests runs its large-backup tests
with 10,000 items, because 60,000 made cdrs-tokio drop its only
connection. tests/test_ttl.py waits for ENABLING to end before disabling.
Signed-off-by: Scott Robinson <robinnsc@amazon.com>
LeeroyHannigan
added a commit
that referenced
this pull request
Oct 23, 2026
Conflicted files and how each was resolved: - crates/engine/src/transact_get_items.rs: main (#418) replaced the inline item-count checks with request_limits::check_transact_count and reads key info through OperationContext::table_key_info_uncached. The branch reorders the handler into the service's phases (projection checks, table lookup, key schema as a first-offender cancellation, duplicates). Kept the branch's phase order and applied both of main's changes inside it: the count check stays first and the key info loop in the table-lookup phase calls table_key_info_uncached. - crates/engine/src/transact_write_items.rs: only the import block conflicted. The branch no longer imports parse_optional_condition; main adds request_limits. Kept build_expression_maps and added request_limits. Main's check_transact_count call and the four table_key_info_uncached lookups in prepare_write_op merged on their own and are kept. Files changed on both sides that merged on their own were read for overlap: crates/core/src/validation/mod.rs (main adds index projection limits, the branch changes key-shape messages; different functions), crates/engine/src/batch_write_item.rs (main replaces the count checks, the branch changes ItemCollectionMetrics; different regions), and docs/differences-from-dynamodb.md, where main's authorization row and the branch's control-plane row both describe a table ARN as TableName; the branch's row now points at the authorization row.
LukeKennedy
pushed a commit
to LukeKennedy/extenddb
that referenced
this pull request
Oct 26, 2026
…; recover abandoned restores (ExtendDB#384) Backup and restore now reproduce the table that was backed up, behave the same on PostgreSQL, SQLite, MongoDB, and Cassandra, and match DynamoDB on the points measured against the service. Engine. The engine resolves a backup into one validated CreateTableInput before any backend sees it: it runs the multi-part-key and vector-index refusals, applies the overrides, and runs validate_create_table. Backends only create the target, register the restore, and copy. The trait gains backup_restore_source; restore_table_from_backup takes a resolved RestoreRequest; restore_table_to_point_in_time is removed. Once the request is authenticated and authorized, the whole restore runs in a task of its own, so a client that disconnects does not cancel it part-way. Restore runs after the response on every backend: the target is returned CREATING and copied by a background task. While it is CREATING, DeleteTable on it is ResourceInUseException and DeleteBackup on its source is BackupInUseException. A target whose server stopped mid-copy is removed and logged: PostgreSQL after 60 s, SQLite at the next start, MongoDB and Cassandra by a heartbeat sweep at startup and every 15 s. Wire behaviour: RestoreDateTime is the backup's creation time and SourceTableArn is set; RestoreSummary appears only while CREATING, on every table description (TableDescription::prepare_for_response). DescribeBackup reports SourceTableDetails.ProvisionedThroughput and SourceTableFeatureDetails. ListBackups pages with Limit and LastEvaluatedBackupArn and carries TableId. Restoring into an existing name is TableAlreadyExistsException; a malformed BackupArn is ValidationException, checked before the target name. SSESpecificationOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as the service applies them (an empty list restores no indexes of that kind, a subset restores those, a GSI entry's throughput is the index's); an unknown or changed index definition, a throughput member contradicting the billing mode, {0, 0}, and BillingModeOverride=PROVISIONED with a GSI whose throughput no entry supplies are ValidationException with the service's text. TableClass is not carried across; the target is STANDARD. ListBackups filters on BackupType and the time range; its cursor must match the account, region, and TableName filter (a listed backup's stored ARN in an earlier region is accepted, so paging passes it), and the time bounds are checked. A BackupArn in another partition is a ValidationException. BackupArn and ExclusiveStartBackupArn shapes are checked as the service checks them ("Invalid Request: BackupArn is not valid" / "Invalid Backup ARN"), surrounding whitespace is trimmed, and an ARN of the caller's account in another region resolves (looked up in the server's region first, then as written, so backups created under an earlier server.region stay reachable); the 13-digit ids of earlier builds still resolve. BackupName and TableName are validated as the model says (3 to 255 characters, [a-zA-Z0-9_.-]+) with the service's "validation error(s) detected" texts; TableName may be the table's ARN; CreateBackup of a missing or CREATING table is TableNotFoundException. DescribeBackup propagates a failure to read the definition as an internal error and degrades only for a version it does not know; DeleteBackup returns the full description DELETED. Restoring into a CREATING restore target is TableInUseException and a second restore of a backup whose restore is CREATING is BackupInUseException. Each backend checks the target name before the backup and before a restore slot is taken; the target insert and the registration transaction, claim, or lightweight transaction are the fences. The key-info cache stores no negative entry for a name the catalog has in a non-ACTIVE state, so the first read after ACTIVE succeeds; authorization's key-info snapshot (ExtendDB#418) records such a name as missing, so the engine does not look it up again. A PROVISIONED backup whose GSI has no throughput restores with the table's throughput. CreateTable and UpdateTable refuse GSI throughput below 1 on PROVISIONED tables and more than 100 projected NonKeyAttributes. CreateBackup of a table with vector indexes is refused. Backup ids are 14 zero-padded millisecond digits on every backend. Authorization: backup operations use main's resource authorization (ExtendDB#419). A BackupArn is trimmed of surrounding ASCII whitespace before it is canonicalized, as the engine trims it before resolving the backup, so a Deny on a backup ARN cannot be bypassed with a trailing newline. PostgreSQL: the copy commits per batch with one multi-row INSERT per index; the lock session is heartbeated; at most 4 restores per account and min(16, pool_size / 2) per server, then LimitExceededException after 30 s, with the whole restore, admission included, in a task the request waits on; only SQLSTATE 53300 on the lock connection is LimitExceeded. The backup row is locked FOR UPDATE while a restore registers. serve refuses a catalog that does not record one of the binary's migration files, and converges a pre-release 0.0.4 catalog shape with a 5 s lock timeout. migrate uses the data database name init recorded and accepts --data-db. SQLite: serve refuses a pre-release 0.0.4 shape; the backup_items rebuild is named in the migrate preview and logged; the pool_size clamp is logged. CreateBackup is admitted against the pool (pool_size / 2 - 1 at once, then LimitExceededException after 30 s). A GSI added by UpdateTable with throughput on a PROVISIONED table no longer makes DescribeTable fail, and rows earlier builds wrote are read. MongoDB: twenty concurrent restores no longer wedge the pool (the registration hand-over held a checked-out connection while waiting for another; both futures are polled together now); restores are admitted per account (4) and per server (min(16, pool / 2)), LimitExceededException after 30 s, and a registration longer than 60 s answers an internal error while the restore continues. UpdateTable on a CREATING restore target is refused. DeleteBackup and the sweep never release a delete claim once the data drop may have started; a held delete claim answers BackupInUse; a concurrent restore of one backup retries for 2 s, then BackupInUse; catalog reads are pinned to the primary with read concern local and writes to w: majority, and a weaker write concern in the connection string is refused at startup; heartbeats refresh at once. The test-only protocol model covers two restores, the delete and its stop points, and the sweep's delete takeover. Cassandra: migration V005 (catalog 0.0.4) stores backup definitions and restore registrations (registered with a lightweight transaction that requires the backup AVAILABLE and unclaimed); the copy uses the normal put path with GSIs written synchronously; ordinary CreateTable does not name the V005 columns, so an unmigrated catalog breaks only CreateBackup, RestoreTableFromBackup, and DeleteBackup. The restore heartbeat stops on every exit of the copy. On an unmigrated catalog the backup operations are a ValidationException naming `extenddb migrate --yes`. UpdateTimeToLive on a table with an asynchronous GSI is a ValidationException instead of a 500. init writes the application credentials into the config, created with mode 0600. A pre-existing defect is fixed: an on-demand GSI omitted ProvisionedThroughput. Catalog guards fail closed on an unparseable version. extenddb migrate: the preview exits 3 when work is pending (main's resource-tag rewrite included) and 0 when up to date; unrecorded catalog migration files are pending; admin connection errors are one plain line; --cassandra-user/--cassandra-pass and EXTENDDB_CASSANDRA_PASSWORD are accepted. Docs: differences-from-dynamodb, dynamodb-limits, the admin, upgrade, and architecture manuals, and the MongoDB design doc are updated for four backends. Tests: tests/test_backup_restore_fidelity.py covers the restore surface on all four backends; devtools/run-cassandra-tests runs its large-backup tests with 10,000 items, because 60,000 made cdrs-tokio drop its only connection. tests/test_ttl.py waits for ENABLING to end before disabling. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
dynamodb:LeadingKeysis now evaluated onQuery,BatchGetItem,BatchWriteItem,TransactGetItemsandTransactWriteItems. Before this change those operations carried no LeadingKeys value, so aForAllValues:StringEquals dynamodb:LeadingKeyscondition on them was satisfied vacuously, and a principal fenced to its own partition could read or write other partitions through them.GetItem,PutItem,UpdateItemandDeleteItemalready enforced the key.How the key is bound, measured against DynamoDB:
Queryon the base table or a local secondary index uses the partition key value in the key condition.Queryon a global secondary index uses the index's partition key value (its first HASH attribute when the index has a multi-attribute partition key).BatchGetItemandBatchWriteItemare authorized once per table with every key that table receives.TransactGetItemsandTransactWriteItemsare authorized once per item; items with the same action, table and key are evaluated once. AForAnyValuecondition therefore passes a batch that mixes the principal's own partition with another one, and denies the same mix in a transaction.ScanandSearchVectorscarry no LeadingKeys, as on DynamoDB.GetItem,PutItem,UpdateItem,DeleteItem,QueryandBatchGetItemuse the normalized number (010,10.0and1E1are all10).BatchWriteItem,TransactGetItemsand everyTransactWriteItemsaction use the number as Java'sBigDecimal.toString()prints it, which keeps the scale the request was written with (10.0stays10.0,1E1is1E+1,010is10). The policy value is matched as written.ResourceNotFoundExceptionare returned before LeadingKeys is evaluated, as on DynamoDB. That includes the count limits of the batch and transaction operations: an empty or 101-item transaction, an emptyRequestItems, a table with no keys or requests, more than 100 tables or keys in aBatchGetItemand more than 25 tables or requests in aBatchWriteItemreturnValidationExceptionbefore any table is read or any policy is evaluated. The handlers and the server run the same count checks.ServiceUnavailableExceptionrather thanInternalServerError.Why
A principal restricted to its own partition with
dynamodb:LeadingKeyscould read every other partition throughQueryandBatchGetItemand write them throughBatchWriteItemand the transaction operations.Testing done
tests/test_fgac_leading_keys.py: 29 wire tests covering each operation, the GSI and LSI cases, a two-attribute GSI partition key,ForAllValuesandForAnyValue, Deny statements, error ordering, the number forms, and three tests added in review:Querykey condition shapes, each sent as the table owner, as a principal fenced to the partition it names and as one fenced away from it. The owner's rows show which partition the engine read; the fenced principal must get the same rows and the other must be denied. Removing the legacyKeyConditionsextraction from authorization fails it.GetItemwhile it is CREATING: once it is ACTIVE, the four batch and transaction operations are denied on another partition and served on the principal's own, and the other partition is never served.ValidationExceptionto a principal with no policy and to one whose fence excludes every key.Every expected response was measured against DynamoDB first. For four rejected
Queryshapes the service and ExtendDB return different error codes to a principal fenced away from the value, and the test asserts only that no data is returned for those. 29 of 29 pass on SQLite and PostgreSQL. Onmain, the enforcement and number cases fail. The CREATING and count limit tests fail on 9be0faa, this PR's head before the review changes.The request-by-request expectations of the number cases were replayed against DynamoDB: 120 checks, 0 differences. 21 spellings, including
1E-7,1.23E+4,0E+5,1.00E-130and negative values, were each repeated 25 times.java_decimal_stringmatches Java'sBigDecimal(String).toString()on 69 inputs, including signs, zero with a scale, 38 significant digits and the exponent boundary where the output switches to scientific notation.Unit tests in
crates/server/src/leading_keys/tests.rsandcrates/engine/src/request_limits.rs, including target de-duplication and the order of the count checks.A
BatchGetItemnaming 50,000 tables with no keys took 3.3 seconds on PostgreSQL on 9be0faa, reading every table before the denial. It now returnsValidationExceptionin about 50 milliseconds.cargo fmt --check,cargo clippyon 1.97.0 with-D warnings,cargo test --workspaceon stable and 1.88, and the existing auth wire files on SQLite.Before review,
devtools/run-tests --pytest --comprehensiveandtests/rustpassed against a SQLite server with this branch and fix(auth): authorize resource-ARN operations against the resource they act on #419 merged together. The combined version will be tested again when the second of the two PRs is rebased.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a. The change makes existing authorization match DynamoDB; the behaviour is recorded in
docs/manuals/07-upgrade-manual.mdanddocs/manuals/10-security-model.md.Breaking changes
dynamodb:LeadingKeyswithForAllValues:StringEquals, and whoseQuery, batch or transaction requests touched partitions outside the fence, getsAccessDeniedExceptionfor those requests after upgrade.["10"], aGetItemon010,10.0or1E1was denied and is now allowed. Under a fence written as["010"]or["1E1"], a request spelled the same way was allowed and is now denied. On batch writes and transactions,10.0and1E1do not match a fence of["10"], as on DynamoDB.ValidationExceptionto every principal. Before this change a principal without permission on the table gotAccessDeniedExceptionfor each of these requests except an emptyTransactItemsorRequestItems.The upgrade manual has an audit list.
Merge order
This PR and #419 both rewrite
authorize_requestincrates/server/src/request_helpers.rs. Either can merge first. I will rebase the second one ontomainand resolve the conflict to the combined version that was tested.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.