Skip to content

fix: populate dynamodb:Attributes for every item request - #732

Open
LeeroyHannigan wants to merge 15 commits into
mainfrom
fix/dynamodb-attributes-key
Open

LeeroyHannigan wants to merge 15 commits into
mainfrom
fix/dynamodb-attributes-key

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

What

dynamodb:Attributes now holds the top-level names of the attributes a request names, on every item, batch and transaction operation. On every item operation other than SearchVectors it also holds the table's key attributes, with the exceptions under "Not covered". Before this change ExtendDB read the key from ProjectionExpression alone, only on single-table requests, and a projected document path went into it whole (secret.a instead of secret).

What the key holds now:

Operation dynamodb:Attributes holds
GetItem The table's key attributes, AttributesToGet, and the attributes in ProjectionExpression
PutItem The table's key attributes, the item's attributes, Expected, and the attributes in ConditionExpression
UpdateItem The Key attributes, the table's key attributes, AttributeUpdates, Expected, the attributes in ConditionExpression, and the target of every SET, REMOVE, ADD and DELETE action, but not the attributes a SET value reads
DeleteItem The Key attributes, the table's key attributes, Expected, and the attributes in ConditionExpression
Query The key attributes of the table and of the queried index, the attributes in KeyConditionExpression and KeyConditions, AttributesToGet, QueryFilter, ExclusiveStartKey, and the attributes in FilterExpression and ProjectionExpression
Scan The table's key attributes, AttributesToGet, ScanFilter, ExclusiveStartKey, and the attributes in FilterExpression and ProjectionExpression
SearchVectors The attributes in ProjectionExpression, as before. The key is absent when there is none
BatchGetItem Per table, as for GetItem: the table's key attributes, AttributesToGet, and the attributes in ProjectionExpression. The names in Keys add nothing
BatchWriteItem Per table, evaluated once: the attributes of every PutRequest item and DeleteRequest key, and the table's key attributes
TransactGetItems, TransactWriteItems Per item, as for the single-item operation the item maps to. A ConditionCheck holds the table's key attributes and the attributes in its ConditionExpression

The code resolves ExpressionAttributeNames placeholders, and a document path contributes its first name (#s.a gives secret when #s is secret). It reads expressions with the engine's tokenizer and parser, under the engine's limits. On every operation other than SearchVectors, an expression the engine accepts gives all its names. An expression the parser refuses gives none, and the engine refuses the same request without reading or writing an item. For a key condition, this code also runs the engine's reserved-word check when enforce_reserved_keywords is on. So a key condition with a bare reserved word gives none. The other expressions skip that check here. A bare reserved word in one of them adds its name, and the engine refuses the request after authorization. "Not covered" lists SearchVectors.

A DeleteItem and a read with no projection add none of the stored item's names to the key. As in DynamoDB, a policy on dynamodb:Attributes does not stop an UpdateItem from copying a denied attribute's value into another attribute through a SET value. It also does not stop ReturnValues ALL_OLD or ReturnValuesOnConditionCheckFailure ALL_OLD from returning the whole stored item.

Changes:

  • New crates/server/src/request_attributes.rs builds the value per operation, per batch table and per transaction item. It replaces extract_attributes in request_helpers.rs.
  • The batch and transaction targets carry the value, and authorization evaluates targets that differ only in it separately. The first commit moved AuthzTarget unchanged into crates/server/src/leading_keys/batch.rs, since leading_keys.rs was at 601 lines.
  • A Scan holds the table's key attributes. A Scan with IndexName does not add the index's key schema, so the index's key attributes are in the key only when the request names them. The ExclusiveStartKey of a later page names them, and DynamoDB counts them too (measured 2026-11-10). A Query of an index adds the index's full key schema, its sort key included.
  • A Query adds the names in KeyConditionExpression and the keys of KeyConditions. On a table that does not exist the key holds them, a non-key name included. On a table that exists, the engine refuses a non-key attribute in KeyConditions. It also refuses one in a KeyConditionExpression on a key schema with several HASH or several RANGE attributes. On a key schema with one HASH attribute and at most one RANGE attribute, it checks a KeyConditionExpression only for an equality on the partition key. So it accepts a clause on another attribute, and the key also holds that name.
  • A BatchGetItem part holds what a GetItem holds, and the names in its Keys add nothing. "Not covered" lists the one exception, a table named by ARN.
  • Comments in the policy evaluator described the key as absent on writes, on AttributesToGet reads and on batch and transaction targets. The no-match answer for a bare negated operator on an absent key stays the same. The comments now say where the key is absent:
    • on a SearchVectors with no ProjectionExpression
    • on a table that does not exist, on a GetItem, transaction Get or BatchGetItem that names no attribute outside its keys
    • on a table that does not exist, on a Query whose key condition the key condition parser refuses and that names none elsewhere
    • on a table that does not exist, on a Scan that names none
  • The security model manual, the troubleshooting guide and the upgrade manual describe the key per operation. The security model manual also says how a PartiQL statement fills the key.

Why

With the key absent, a Deny gated on ForAnyValue:StringEquals over an attribute never applied to a write, an AttributesToGet read, or any batch or transaction request. An Allow gated on ForAllValues:StringEquals granted those requests whatever attributes they named. The security model manual and the troubleshooting guide told operators these controls work.

Measured on DynamoDB on 2026-10-31 and 2026-11-01, with each policy below as a session policy. Every request was sent twice and got the same answer both times. The policies:

  • deny(X): Allow dynamodb:* on the table, and Deny dynamodb:* on it with ForAnyValue:StringEquals {"dynamodb:Attributes": [X]}.
  • allow-put for-all(pk): only Allow dynamodb:PutItem with ForAllValues:StringEquals {"dynamodb:Attributes": ["pk"]}. allow-get for-all(L) is the same for dynamodb:GetItem over the list L. allow-scan for-all(L), allow-query for-all(L), allow-update for-all(L), allow-delete for-all(L) and allow-batch-write for-all(L) are the same for Scan, Query, UpdateItem, DeleteItem and BatchWriteItem.
  • allow-batch-write for-any(name): only Allow dynamodb:BatchWriteItem with ForAnyValue:StringEquals {"dynamodb:Attributes": ["name"]}.
  • allow-scan unless(ssn): only Allow dynamodb:Scan with the bare StringNotEquals {"dynamodb:Attributes": "ssn"}.
  • deny-get-when-absent: Allow dynamodb:* on the table, and Deny dynamodb:GetItem with Null {"dynamodb:Attributes": "true"}. deny-scan-when-absent is the same for dynamodb:Scan.
  • allow-scan when-absent: only Allow dynamodb:Scan with Null {"dynamodb:Attributes": "true"}.

Measured 2026-10-31:

Request Policy DynamoDB ExtendDB before
PutItem {pk, ssn} deny(ssn) AccessDeniedException allowed
UpdateItem SET ssn = :v deny(ssn) AccessDeniedException allowed
PutItem {pk, ssn} allow-put for-all(pk) AccessDeniedException allowed
GetItem AttributesToGet=[ssn] deny(ssn) AccessDeniedException allowed
Scan AttributesToGet=[ssn] deny(ssn) AccessDeniedException allowed
GetItem ProjectionExpression=secret.a deny(secret) AccessDeniedException allowed
GetItem ProjectionExpression=#s.a (#s = secret) deny(secret) AccessDeniedException allowed
BatchGetItem ProjectionExpression=ssn deny(ssn) AccessDeniedException allowed
TransactGetItems Get ProjectionExpression=ssn deny(ssn) AccessDeniedException allowed
BatchWriteItem PutRequest {pk, ssn} deny(ssn) AccessDeniedException allowed
TransactWriteItems Put {pk, ssn} deny(ssn) AccessDeniedException allowed
PutItem {pk, name} deny(ssn) allowed allowed
PutItem {pk} allow-put for-all(pk) allowed allowed
DeleteItem of an item holding ssn deny(ssn) allowed allowed
UpdateItem SET #n = :v (#n = name) deny(ssn) allowed allowed
GetItem ProjectionExpression=ssn deny(ssn) AccessDeniedException AccessDeniedException
GetItem AttributesToGet=[name] deny(ssn) the item's name only the item's name only
GetItem, no projection deny(ssn) the whole item the whole item
Scan Limit=2, no projection deny(ssn) whole items whole items
BatchGetItem, no projection deny(ssn) the whole item the whole item

Measured 2026-11-01. In this table, "authorized" means the request passed authorization, and some of these requests then failed their condition or validation:

Request Policy DynamoDB ExtendDB before
PutItem {pk}, ConditionExpression attribute_not_exists(ssn) deny(ssn) AccessDeniedException authorized
PutItem {pk}, Expected {"ssn": {"Exists": false}} deny(ssn) AccessDeniedException authorized
DeleteItem, ConditionExpression attribute_not_exists(ssn) deny(ssn) AccessDeniedException authorized
UpdateItem REMOVE ssn deny(ssn) AccessDeniedException authorized
UpdateItem ADD ssn :one deny(ssn) AccessDeniedException authorized
UpdateItem DELETE ssn :s deny(ssn) AccessDeniedException authorized
UpdateItem, AttributeUpdates {"ssn": PUT} deny(ssn) AccessDeniedException authorized
Query pk = :p, FilterExpression ssn = :v deny(ssn) AccessDeniedException authorized
Scan, FilterExpression ssn = :v deny(ssn) AccessDeniedException authorized
Scan, ScanFilter {"ssn": NOT_NULL} deny(ssn) AccessDeniedException authorized
TransactWriteItems ConditionCheck attribute_exists(ssn) deny(ssn) AccessDeniedException authorized
UpdateItem SET #n = ssn, SET #n = #s (#s = ssn), if_not_exists(#n, ssn), list_append(ssn, :l), list_append(:l, ssn), ssn + :one, :one + ssn, on an item without ssn deny(ssn) authorized authorized
UpdateItem SET #n = if_not_exists(ssn, :d) deny(ssn) allowed allowed
UpdateItem SET #n = ssn on an item holding ssn deny(ssn) allowed, and name then holds the ssn value, which a later GetItem ProjectionExpression=#n returns the same
Query pk = :p, no filter or projection deny(ssn) the whole item the whole item
Scan, ExclusiveStartKey {pk} only deny(ssn) allowed allowed
GetItem, no projection deny-get-when-absent the whole item AccessDeniedException
GetItem ProjectionExpression=#n deny-get-when-absent name name
GetItem ProjectionExpression=#n allow-get for-all([name]) AccessDeniedException allowed
GetItem ProjectionExpression=#n allow-get for-all([pk, name]) name name
GetItem, no projection allow-get for-all([pk, name]) the whole item the whole item
Scan naming no attribute, and Scan ProjectionExpression=#n allow-scan unless(ssn) AccessDeniedException AccessDeniedException

Also measured 2026-11-01, on the table's key attributes, legacy reads, transaction items and returned items. The ExtendDB column comes from the wire tests run against a build of main on SQLite:

Request Policy DynamoDB ExtendDB before
Scan Limit=1, no projection deny(pk) AccessDeniedException allowed
Scan, ExclusiveStartKey {pk} only deny(pk) AccessDeniedException allowed
Scan with no projection, Scan ProjectionExpression=#n, and Scan Select=COUNT allow-scan for-all([name]) AccessDeniedException allowed
Scan ProjectionExpression=#n allow-scan for-all([pk, name]) allowed allowed
Scan, no projection allow-scan for-all([pk]) whole items whole items
Scan, no projection deny-scan-when-absent whole items AccessDeniedException
Scan ProjectionExpression=#n deny-scan-when-absent allowed allowed
Scan with no projection, and Scan Select=COUNT allow-scan when-absent AccessDeniedException allowed
Query pk = :p, ProjectionExpression=#n allow-query for-all([name]) AccessDeniedException allowed
UpdateItem SET #n = :v allow-update for-all([name]) AccessDeniedException allowed
DeleteItem allow-delete for-all([name]) AccessDeniedException allowed
BatchWriteItem with one DeleteRequest allow-batch-write for-all([name]) AccessDeniedException allowed
Each of the four requests above the same policy over [pk, name] allowed allowed
BatchWriteItem Puts {pk, name} and {pk, ssn} on one table allow-batch-write for-any(name) allowed, both items written AccessDeniedException
BatchWriteItem Put {pk, name} only allow-batch-write for-any(name) allowed AccessDeniedException
BatchWriteItem Put {pk, ssn} only allow-batch-write for-any(name) AccessDeniedException AccessDeniedException
Query with KeyConditions and QueryFilter {"ssn": NOT_NULL} deny(ssn) AccessDeniedException allowed
Query with KeyConditions and AttributesToGet=[ssn] deny(ssn) AccessDeniedException allowed
Query with KeyConditions and AttributesToGet=[name] deny(ssn) the item's name only the item's name only
BatchGetItem AttributesToGet=[ssn] deny(ssn) AccessDeniedException allowed
TransactWriteItems Update SET ssn = :v deny(ssn) AccessDeniedException allowed
TransactWriteItems Delete, ConditionExpression attribute_not_exists(ssn) deny(ssn) AccessDeniedException authorized
TransactWriteItems Delete, no condition deny(ssn) allowed allowed
UpdateItem SET #n = :v, ReturnValues ALL_OLD, on an item holding ssn deny(ssn) the old item, ssn included the same
PutItem {pk} whose condition attribute_not_exists(pk) fails, ReturnValuesOnConditionCheckFailure ALL_OLD deny(ssn) ConditionalCheckFailedException carrying the stored item, ssn included the same
TransactWriteItems ConditionCheck that fails, ReturnValuesOnConditionCheckFailure ALL_OLD deny(ssn) TransactionCanceledException whose reason carries the stored item, ssn included the same

DynamoDB's refusals name dynamodb:BatchGetItem and dynamodb:BatchWriteItem for the batch requests, and dynamodb:GetItem for the TransactGetItems Get. They name dynamodb:PutItem, dynamodb:UpdateItem and dynamodb:DeleteItem for the TransactWriteItems Put, Update and Delete, and dynamodb:ConditionCheckItem for the ConditionCheck. ExtendDB's refusals now name the same actions.

Measured 2026-11-09 by hitesh96db, in two accounts with the same answers in both. The table pr732-c has the partition key pk and the sort key sk. It has a local secondary index lsi on pk and ls, and a global secondary index gsi2 on gk and gs. Its one item holds pk, sk, ls, gk, gs and name. The table pr732-missing does not exist. Each policy also covers the table's indexes, and #n is name. The last column is this PR at 890ee53, as hitesh96db measured it on PostgreSQL:

Request Policy DynamoDB This PR at 890ee53
Scan IndexName=gsi2 ProjectionExpression=#n allow-scan for-all([pk, sk, name]) the item's name AccessDeniedException
Scan IndexName=lsi ProjectionExpression=#n allow-scan for-all([pk, sk, name]) the item's name AccessDeniedException
Scan IndexName=gsi2, no projection allow-scan for-all([pk, sk]) the whole item AccessDeniedException
Scan IndexName=gsi2, no projection deny(gk) the whole item AccessDeniedException
Scan IndexName=gsi2 ProjectionExpression=#n allow-scan for-all([gk, gs, name]) AccessDeniedException AccessDeniedException
Query IndexName=gsi2, gk = :g, ProjectionExpression=#n allow-query for-all([pk, sk, gk, name]) AccessDeniedException AccessDeniedException
The same Query allow-query for-all([pk, sk, gk, gs, name]) the item's name the item's name
Query of pr732-missing, #n = :p deny(name) AccessDeniedException ResourceNotFoundException
Query of pr732-missing, KeyConditions on pk deny(pk) AccessDeniedException ResourceNotFoundException
BatchGetItem of pr732-missing, Keys on pk deny(pk) ResourceNotFoundException AccessDeniedException
GetItem of pr732-missing, Key on pk deny(pk) ResourceNotFoundException ResourceNotFoundException

The head gives the DynamoDB answer on each row, and the wire tests under "Testing done" assert each one. On the same day, hitesh96db re-ran request shapes from the 2026-10-31 and 2026-11-01 tables, and DynamoDB gave the answers those tables show. In one account, hitesh96db also measured ProjectionExpression on Query and Scan, Expected and ConditionExpression on UpdateItem, and Expected on DeleteItem. DynamoDB applies to each the rule measured for the same parameter on the other operations. The 2026-11-09 measurements in two accounts also show that DynamoDB puts a table's sort key in the key.

Measured 2026-11-10 in one account, on a table of the same shape that holds two items. Every request was sent twice and got the same answer both times. Each policy also covers the table's indexes, and #n is name. Page 1 of each Scan set Limit=1, and page 2 sent page 1's LastEvaluatedKey as its ExclusiveStartKey. That start key holds gk, gs, pk and sk on gsi2, ls, pk and sk on lsi, and pk and sk on the table:

Request Policy DynamoDB
Scan IndexName=gsi2 ProjectionExpression=#n, page 1 allow-scan for-all([pk, sk, name]) allowed
The same Scan, page 2 allow-scan for-all([pk, sk, name]) AccessDeniedException
Scan IndexName=lsi ProjectionExpression=#n, page 1 allow-scan for-all([pk, sk, name]) allowed
The same Scan, page 2 allow-scan for-all([pk, sk, name]) AccessDeniedException
Scan IndexName=gsi2, page 1 allow-scan for-all([pk, sk]) allowed
Scan IndexName=gsi2, page 2 allow-scan for-all([pk, sk]) AccessDeniedException
Scan IndexName=lsi, page 2 allow-scan for-all([pk, sk]) AccessDeniedException
Scan of the table, page 2 allow-scan for-all([pk, sk]) allowed
Scan IndexName=gsi2, page 1 deny(gk) allowed
Scan IndexName=gsi2, page 2 deny(gk) AccessDeniedException, explicit deny
Scan IndexName=lsi, page 1 deny(ls) allowed
Scan IndexName=lsi, page 2 deny(ls) AccessDeniedException, explicit deny
Scan IndexName=gsi2 ProjectionExpression=#n, page 1 allow-scan for-all([gk, gs, name]) AccessDeniedException

So DynamoDB adds the names in ExclusiveStartKey to dynamodb:Attributes, and a later page of a Scan with IndexName holds the index's key attributes. An Allow whose ForAllValues list omits them refuses that page, and so does a Deny on one of them. The branch already counted those names, so this needed no code change. The wire tests under "Testing done" assert four page-2 rows: gsi2 and lsi under the first policy, gsi2 under deny(gk), and the table.

Not measured:

  • Whether ExclusiveStartKey adds names of its own on a Query or on a Scan of the table. No request to the service can show it there: a valid start key holds only key attributes, and the key already holds them.
  • The rules on a local secondary index that neither the 2026-11-09 nor the 2026-11-10 table covers, the Query rule among them. For a local secondary index, the two tables cover a Scan under an Allow of the table's keys, alone or with the projected name. They also cover one under a Deny on its sort key. The other Scan rules follow from the key content those rows show, and the Query rule follows the global secondary index measurement.
  • A Query of a table that does not exist, with a key condition that the key condition parser refuses. ExtendDB adds no names from that key condition, so a Deny on one of them does not match. The engine then answers ResourceNotFoundException.

Testing done

The head is 147378a. Runs at 0d578028d or 46d8b051b used earlier versions of the last commit. 0d578028d differs from the head in //! comments of crates/server/src/request_attributes.rs and in comments of its unit tests. It also differs in the docs and in the wire test file, which adds four tests. cargo fmt --check -p extenddb-server passes on the head.

  • Wire tests, each with a measured row's policy as a user policy. There is one for each row of the 2026-10-31, 2026-11-01 and 2026-11-09 tables, and one for each of four rows of the 2026-11-10 table. Each asserts DynamoDB's answer, the action named in a refusal, and that a refused request changes nothing. The files are tests/test_dynamodb_attributes_condition.py (20), tests/test_dynamodb_attributes_expressions.py (40) and tests/test_dynamodb_attributes_key_attributes.py (39). The PutItem with ReturnValues ALL_OLD follows the measured UpdateItem row, and its docstring says so.

    • At the head, all 99 passed on SQLite with the default index_propagation_delay_ms (10 ms) in place. At 0d578028d, the 95 tests of that revision passed with index_propagation_delay_ms at 0. The 35 of the last file also passed at the default delay. These runs used SQLite only.
    • The four tests for the 2026-11-10 table assert its page-2 rows. In each, the table's owner reads page 1 with Limit=1 over the two items. The test then checks the names in the start key that page 1 returns. The tests are:
      • test_for_all_values_allow_of_the_table_keys_refuses_a_later_page_of_an_index_scan, for gsi2 and for lsi, under allow-scan for-all([pk, sk, name])
      • test_a_deny_on_an_index_key_attribute_refuses_a_later_page_of_the_index_scan, for gsi2 under deny(gk)
      • test_for_all_values_allow_of_the_table_keys_alone_admits_a_later_page_of_the_table, under allow-scan for-all([pk, sk])
    • With crates/server/src/request_attributes.rs from the merge commit 6ee143f, 7 of the 11 tests that the last commit adds failed. They are the four Scan rows that DynamoDB allows and the three rows on pr732-missing where 890ee53 differs from DynamoDB. That run used 46d8b051b, the first version of the last commit.
    • At 890ee53, the 84 tests of that revision passed on SQLite and on PostgreSQL. Against a build of main on SQLite, 44 of them failed and 40 passed. The 44 are the 43 tests of rows whose "ExtendDB before" differs from DynamoDB, and one more. That one is the Query on an index without the index key in the list, which main granted.
  • 32 unit tests in crates/server/src/request_attributes/tests.rs cover each operation and the batch and transaction targets. They cover a Scan with no parameters, with Select COUNT, on a table with a sort key, and on a global and a local secondary index. They also cover placeholders, document paths, list indexes, legacy parameters, update targets, SET values, unresolved placeholders and expressions the engine refuses. Five tests that the last commit adds cover key conditions on tables that exist and that do not. They also cover a bare reserved word in a key condition, the Keys of a BatchGetItem and the start key of a later page of an index Scan.

    • At the head and at 0d578028d, all 32 passed.
    • With crates/server/src/request_attributes.rs from 6ee143f, 3 of the 30 tests at 46d8b051b failed: the missing-table key condition test, the BatchGetItem Keys test and the index Scan test.
    • Without the reserved-word check, and with no index_key_schema call in the Query arm, 2 of the 32 failed at 0d578028d.
  • Negative controls, each on a committed tree and restored with git afterwards:

    • With no extraction for PutItem, UpdateItem and DeleteItem, the three write rows that DynamoDB refuses failed (3 of 20), and 8 unit tests failed.
    • With AttributesToGet ignored, the GetItem and Scan AttributesToGet rows failed (2 of 20), and 4 unit tests failed.
    • With no key on the batch and transaction targets, the four batch and transaction refusals failed (4 of 20).
    • With the key holding the paths a SET value reads, the SET value rows and the copy row failed (9 of 49), and 1 unit test failed.
    • With a Scan holding none of the table's key attributes, 7 of 81 wire tests failed on SQLite, and 6 of 260 server unit tests failed. The 7 are the deny(pk) Scan with no projection, the three allow-scan for-all([name]) rows, the deny-scan-when-absent row with no projection and the two allow-scan when-absent rows.

    The first three ran on this PR's first fix commit with the first test file. The fourth ran with the first two files, and the last on the Scan commit.

  • At 890ee53, the 16 auth test files ran with test credentials provisioned as devtools/run-tests does. They are test_auth_integration, test_auth_permissions, test_authz_arn_conditions, test_authz_resource_arn, test_fgac_attributes, test_fgac_leading_keys, the five test_tag_request_* files, test_auth_error_fidelity, test_dev_mode_authorization and the three test_dynamodb_attributes_* files. On SQLite, 418 passed and 5 skipped, and PostgreSQL gave the same counts. The 5 skipped tests are test_dev_mode_authorization.py, which needs a dev-mode build.

  • At 890ee53, on 1.97.0, cargo fmt --all -- --check, cargo clippy --locked --workspace --all-targets -- -D warnings and cargo check --locked --workspace --all-targets gave no warnings. cargo test --locked -p extenddb-server -p extenddb-auth passed on stable (1.99.0) and on 1.88.0, with 260 server and 272 auth unit tests and 3 doc tests.

  • At 0d578028d, cargo fmt --all -- --check passed, and cargo +1.97.0 clippy --locked --all-targets -p extenddb-server -p extenddb-auth -- -D warnings gave no warning. cargo test --locked -p extenddb-auth --lib passed 438 tests, and cargo test --locked -p extenddb-server --lib request_attributes passed 32. cargo test --workspace did not run on this revision.

  • At the head, cargo +1.97.0 clippy -p extenddb-server --all-targets -- -D warnings gave no warning, and cargo test -p extenddb-server --lib request_attributes passed 32 tests. cargo test --workspace did not run on the head.

  • crates/auth changes only in comments.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. A condition key now holds the value DynamoDB gives it. The policy language and the evaluator stay the same.

Breaking changes

Policies that name dynamodb:Attributes can decide differently after this change, as the upgrade manual records under v0.1.15. On a table that exists, every item request other than SearchVectors now carries the key, holding at least the table's key attributes. The exceptions are the batch and transaction parts under "Not covered" that name a table by ARN. So a ForAllValues condition no longer holds vacuously on an absent key, and a ForAnyValue condition can match. Null true stops matching and Null false starts matching. A SearchVectors still holds only the names in its ProjectionExpression. This affects the statements below:

  • A Deny gated on a ForAnyValue:String* operator that matches an attribute's name (StringEquals, StringLike, StringEqualsIgnoreCase) now applies to writes that name the attribute. It also applies to update actions that target it, to AttributesToGet reads and to projections of a path inside it. It applies to conditions, filters and Query key conditions that read it, and to batch and transaction requests.
  • A Deny gated on a negated ForAnyValue:String* operator (StringNotEquals, StringNotLike, StringNotEqualsIgnoreCase) now applies to every item request whose names fall outside its list, the key attributes included. So the list must hold the key attributes of the table. For a Query of an index, or a later page of a Scan of an index, it must also hold the index's key attributes. A list that omits the table's key attributes refuses every GetItem, PutItem, UpdateItem, DeleteItem, Query, Scan, batch and transaction request on the table.
  • An Allow gated on ForAllValues:StringEquals now grants only when the list holds every name the request holds. On every item operation other than SearchVectors those names include the table's key attributes. A list without them stops granting those operations, Scan included. For a Query of an index the list must also hold the index's key attributes, its sort key included (measured 2026-11-09 on a global secondary index). The first page of a Scan of an index needs only the table's key attributes and the names the request gives (measured 2026-11-09). A later page also needs the index's key attributes, which its ExclusiveStartKey carries (measured 2026-11-10).
  • An Allow gated on a ForAnyValue:String* operator now grants every item request its Action covers that holds a name the list matches. Before, it granted only a single-table request whose ProjectionExpression named one. Authorization evaluates a BatchWriteItem once per table, with the union of that table's item names. So over ["name"] such an Allow grants a batch that puts {pk, name} and {pk, ssn} on one table, as on DynamoDB. Over ["pk"], on a table keyed on pk, it grants every item request other than SearchVectors that its Action covers, a read with no projection included.
  • An Allow of PutItem gated on ForAnyValue:StringNotEquals over ["ssn"] granted no PutItem before, and now grants one of {pk, ssn}, because pk is outside the list. An Allow of PutItem gated on ForAllValues:StringNotEquals over ["ssn"] granted every PutItem before, and now refuses one of {pk, ssn}.
  • A statement gated on Null {"dynamodb:Attributes": "true"} now matches no item request except those that leave the key absent. Those are a SearchVectors with no ProjectionExpression, and a part under "Not covered" that names its table by ARN and names no attribute outside its keys. On a table that does not exist, they also include a GetItem, transaction Get or BatchGetItem that names no attribute outside its keys. On such a table they include a Query whose key condition the key condition parser refuses and that names none elsewhere, and a Scan that names none. So such a Deny stops refusing and such an Allow stops granting.
  • A statement gated on Null {"dynamodb:Attributes": "false"} matched only a single-table request with a ProjectionExpression before. It now matches every item request that carries the key. An Allow gated on it, written to require that a read names its attributes, now grants each of these requests that its Action covers. That includes a read with no projection, which returns the whole item. A Deny gated on it now refuses these requests.

Not covered


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

…to a submodule

leading_keys.rs is 601 lines. Move AuthzTarget, batch_transact_targets,
batch_transact_tables and batch_table_keys unchanged into
leading_keys/batch.rs, and re-export the two functions that request
authorization calls.
The dynamodb:Attributes condition key was read from ProjectionExpression
alone, only on single-table requests, and a document path such as secret.a
went into it as written. Writes, AttributesToGet reads and every batch and
transaction request reached the policy evaluator without the key, so a Deny
gated on ForAnyValue:StringEquals over an attribute never applied to them,
and an Allow gated on ForAllValues:StringEquals granted them whatever
attributes they wrote or read.

The key now holds the top-level names a request names: the item or key
attributes, AttributesToGet, the paths in the projection, update, condition
and filter expressions with ExpressionAttributeNames resolved, the legacy
Expected, AttributeUpdates, QueryFilter and ScanFilter maps,
ExclusiveStartKey, and the key attributes of the table and of a queried
index. A batch request carries the names per table and a transaction per
item. A DeleteItem and a read with no projection hold only the names the
request gives, not those of the stored item.
…ansaction requests

One test per request shape measured on DynamoDB on 2026-10-31, under a user
policy of the shape of the session policy each was measured with: a Deny
gated on ForAnyValue:StringEquals over one attribute, and an Allow of
PutItem gated on ForAllValues:StringEquals over pk. Each asserts the
service's answer, the action the refusal names, and that a refused write
stores nothing.
The security model manual lists the names the key holds for each item,
batch and transaction operation, and which entries were measured on
DynamoDB. The troubleshooting entry for an Attributes condition covers a
Deny that does not block a DeleteItem or a read with no projection, and an
allowlist that omits the key attributes. The upgrade manual records the
change for v0.1.15 and the policy statements it affects.
Rename the ExpressionAttributeNames field so it no longer repeats the
struct name, merge the match arms that walk two operands, and replace
for_each with loops. Behaviour is unchanged.
…utes

DynamoDB authorizes an UpdateItem whose only mention of an attribute is in
a SET value. Under a Deny gated on ForAnyValue:StringEquals over ssn, it
authorized SET #n = ssn, SET #n = if_not_exists(ssn, :d),
list_append(ssn, :l) and ssn + :one, and refused REMOVE ssn, ADD ssn :one
and DELETE ssn :s (measured 2026-11-01). The key for an UpdateItem and for
a transaction's Update item now holds the attribute each action names, and
no longer the paths its value reads.
…ns and key attributes

One test per request shape measured on DynamoDB on 2026-11-01: a condition,
filter, Expected, AttributeUpdates or ScanFilter that reads a denied
attribute, a REMOVE, ADD or DELETE of it, and a ConditionCheck on it are
refused; a SET value that reads it is authorized and copies it; a bare
StringNotEquals grants no Scan; a Null condition does not refuse a GetItem
with no projection; and an allowlist without the table's key attribute
refuses a projected GetItem. The new module reuses the fixtures of
test_dynamodb_attributes_condition.py.
The security model manual, the troubleshooting guide and the upgrade
manual now say that the key holds the target of each UpdateExpression
action and not the attributes a SET value reads, and that, as in DynamoDB,
a policy on dynamodb:Attributes does not stop an UpdateItem from copying a
denied attribute's value into another attribute through a SET value. The
security model manual lists the entries measured on DynamoDB on 2026-10-31
and 2026-11-01, and records that a bare StringNotEquals on the key matches
neither a present nor an absent key.
…n Scan

A Scan now puts the key attributes of the table, and with IndexName the key
attributes of that index, in dynamodb:Attributes, as GetItem and Query do.
Before this change a Scan that named no attribute left the key absent.

DynamoDB, measured on 2026-11-01: a Deny over pk refuses a Scan with no
projection. An Allow of Scan alone with ForAllValues:StringEquals over
["name"] refuses a plain, a projected and a Select COUNT Scan; the same Allow
over ["pk", "name"] admits the projected Scan, and over ["pk"] it admits a
plain Scan. A Deny of Scan gated on Null {"dynamodb:Attributes": "true"}
refuses neither a plain nor a projected Scan, and an Allow of Scan alone on
the same Null grants neither a plain nor a Select COUNT Scan.

With the key absent, ExtendDB used to give the opposite answer to each
plain and Select COUNT Scan here except the one under ["pk"]; it also
granted the projected Scan under ["name"], whose key held name alone.

The index key attributes follow the IAM guide's rule that a policy on
dynamodb:Attributes lists the key attributes of the table and of every index
it covers. They were not measured.
Wire tests for rows DynamoDB answered on 2026-11-01 that had none: a Deny
over pk, Allows over ForAllValues lists, and Null in a Deny and an Allow, on
Scan; the table's key attribute on Query, UpdateItem, DeleteItem and a
BatchWriteItem DeleteRequest; QueryFilter, AttributesToGet on Query and
BatchGetItem, and the Update and Delete items of TransactWriteItems under a
Deny on ssn; and ReturnValues ALL_OLD and ReturnValuesOnConditionCheckFailure
ALL_OLD, which return the stored item with the denied attribute in it.

An Allow of BatchWriteItem alone with ForAnyValue:StringEquals over ["name"]
admits a batch that puts {pk, name} and {pk, ssn} on one table and writes
both items, admits {pk, name} alone, and refuses {pk, ssn} alone. The service
evaluates each table's part of the batch once, with the names of all its
items, and ExtendDB does the same.

Two rows were not measured, and their docstrings say so: a PutItem with
ReturnValues ALL_OLD follows the UpdateItem row, and a Query on a global
secondary index follows the IAM guide's rule that a policy on the key lists
the key attributes of the table and of every index it covers.
…hapes

The security model, the upgrade entry, the troubleshooting guide and the
policy evaluator's comments now say that a Scan holds the table's key
attributes. On a table that exists the key is absent only on a
SearchVectors with no ProjectionExpression. On a table that does not exist
it is also absent on a GetItem or transaction Get that names no attribute
outside its Key, a Query that names none outside its key condition and a
Scan that names none, because the extractor reads no Key on a GetItem or a
Get and no key condition on a Query. The rows DynamoDB answered on
2026-11-01 are labelled measured; the index key attributes and the names
of ExclusiveStartKey are labelled unmeasured.

The upgrade entry opens its list with the rule the shapes follow: every
item request other than SearchVectors now carries the key, holding at
least the table's key attributes, so a ForAllValues condition no longer
holds vacuously, a ForAnyValue condition can match, Null true stops
matching and Null false starts matching. Its bullets cover a Deny gated on
a ForAnyValue string operator; a Deny gated on a negated one, which now
needs the key attributes in its list; an Allow gated on ForAllValues; an
Allow gated on ForAnyValue over any listed attribute, with the per-table
BatchWriteItem union as its example; the negated Allow shapes; and Null
with either value.

The security model says that a Deny on the key does not stop ReturnValues
ALL_OLD or ReturnValuesOnConditionCheckFailure ALL_OLD from returning the
whole stored item, as on DynamoDB, and that the IAM guide's guard against
ReturnValues returning attributes is a condition on dynamodb:ReturnValues.
The extractor's module docs say that ReturnValues and
ReturnValuesOnConditionCheckFailure were measured to add no names and that
Select was not measured. The unit test module docs say which unmeasured
rows follow the IAM guide's rule and which follow the rule measured for the
same parameter on another operation.
crates/server/src/lib.rs conflicted on the module list. Main added the
registration module and this branch added request_attributes. The merged
file declares both.

This branch added a LimitsConfig argument to batch_transact_targets, and
main's new cross-account check in authz_resource/cross_account.rs calls
it. That check reads only each target's table and action, so it passes
the default limits, which govern only how expressions are read for
Attributes.
@navareamol navareamol added v1 Needed for the 1.0 release release-blocker Must ship for its milestone's release labels Nov 9, 2026
@navareamol navareamol added this to the 1.0 milestone Nov 9, 2026
@hitesh96db hitesh96db self-assigned this Nov 9, 2026

@hitesh96db hitesh96db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-measured this PR's rules against Amazon DynamoDB and against a build of this PR, 890ee53f on PostgreSQL. Most of it matches, including every row in the description I re-ran, and several of the rows it lists as not measured. One rule is wrong in a way that refuses requests DynamoDB allows: Scan with IndexName (inline, marked Blocker). There are two smaller differences on tables that do not exist, plus one doc note.

Setup

Measured on 2026-11-09 against Amazon DynamoDB in us-east-1. The inline experiments, and the table-key and missing-table rows below, were measured in two accounts with the same answers in both; the remaining rows below were measured in one. Every DynamoDB request was sent at least twice, with the same answer each time. The ExtendDB side ran against 890ee53f, release build, PostgreSQL backend, https://127.0.0.1:18452, with AWS_CA_BUNDLE=~/.extenddb/tls/cert.pem and --endpoint-url https://127.0.0.1:18452 appended to each aws dynamodb command. The commands in the inline comments are the exact commands run. A driver generated both the measurements and these comments from the same definitions, and the results tables are its output, with the caller ARN and the account id replaced by <caller> and <ACCOUNT>.

The caller. Every experiment names a policy, and the caller holds only that policy.

On DynamoDB, a role whose own policy allows dynamodb:* on table/pr732-* and table/pr732-*/index/*, assumed with the experiment's policy as the session policy:

aws iam create-role --role-name pr732-attr-probe --max-session-duration 3600 --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::<ACCOUNT>:role/Admin"},"Action":"sts:AssumeRole"}]}'
aws iam put-role-policy --role-name pr732-attr-probe --policy-name ddb-pr732 --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"]}]}'
aws sts assume-role --role-arn arn:aws:iam::<ACCOUNT>:role/pr732-attr-probe --role-session-name probe-<N> --policy file://<policy>.json --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text

On ExtendDB, a new IAM user per experiment whose only policy is the experiment's:

extenddb manage --user admin --config extenddb.toml create-user --account-id <ACCOUNT> --user-name probe-<N>
extenddb manage --user admin --config extenddb.toml create-access-key --account-id <ACCOUNT> --user-name probe-<N>
extenddb manage --user admin --config extenddb.toml put-user-policy --account-id <ACCOUNT> --user-name probe-<N> --policy-name p --policy-document "$(cat <policy>.json)"

Control. Under control (an Allow of GetItem only), aws dynamodb put-item --region us-east-1 --table-name pr732-c --item '{"pk":{"S":"z"},"sk":{"S":"z"}}' was refused on both endpoints: AccessDeniedException "because no session policy allows the dynamodb:PutItem action" on DynamoDB, and the same with "identity-based policy" on ExtendDB. So the session policy really does restrict the caller.

Table. Created once per endpoint: on DynamoDB with the account's admin credentials, and on ExtendDB by a user allowed dynamodb:* on pr732-*. The table pr732-missing is never created.

aws dynamodb create-table --region us-east-1 --table-name pr732-c --billing-mode PAY_PER_REQUEST --attribute-definitions AttributeName=pk,AttributeType=S AttributeName=sk,AttributeType=S AttributeName=ls,AttributeType=S AttributeName=gk,AttributeType=S AttributeName=gs,AttributeType=S --key-schema AttributeName=pk,KeyType=HASH AttributeName=sk,KeyType=RANGE --local-secondary-indexes '[{"IndexName":"lsi","KeySchema":[{"AttributeName":"pk","KeyType":"HASH"},{"AttributeName":"ls","KeyType":"RANGE"}],"Projection":{"ProjectionType":"ALL"}}]' --global-secondary-indexes '[{"IndexName":"gsi2","KeySchema":[{"AttributeName":"gk","KeyType":"HASH"},{"AttributeName":"gs","KeyType":"RANGE"}],"Projection":{"ProjectionType":"ALL"}}]'
aws dynamodb wait table-exists --region us-east-1 --table-name pr732-c
aws dynamodb put-item --region us-east-1 --table-name pr732-c --item '{"pk":{"S":"p1"},"sk":{"S":"s1"},"ls":{"S":"l1"},"gk":{"S":"g1"},"gs":{"S":"t1"},"name":{"S":"n"}}'

Policies (<ACCOUNT> is the endpoint's account):

  • allow-scan-table-keys: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:Scan","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAllValues:StringEquals":{"dynamodb:Attributes":["pk","sk","name"]}}}]}
  • allow-scan-pk-sk: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:Scan","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAllValues:StringEquals":{"dynamodb:Attributes":["pk","sk"]}}}]}
  • deny-gk: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"]},{"Effect":"Deny","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAnyValue:StringEquals":{"dynamodb:Attributes":["gk"]}}}]}
  • allow-scan-index-keys: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:Scan","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAllValues:StringEquals":{"dynamodb:Attributes":["gk","gs","name"]}}}]}
  • allow-query-no-gs: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:Query","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAllValues:StringEquals":{"dynamodb:Attributes":["pk","sk","gk","name"]}}}]}
  • allow-query-with-gs: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:Query","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAllValues:StringEquals":{"dynamodb:Attributes":["pk","sk","gk","gs","name"]}}}]}
  • deny-name: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"]},{"Effect":"Deny","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAnyValue:StringEquals":{"dynamodb:Attributes":["name"]}}}]}
  • deny-pk: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"]},{"Effect":"Deny","Action":"dynamodb:*","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAnyValue:StringEquals":{"dynamodb:Attributes":["pk"]}}}]}
  • control: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:GetItem","Resource":["arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*","arn:aws:dynamodb:us-east-1:<ACCOUNT>:table/pr732-*/index/*"],"Condition":{"ForAllValues:StringEquals":{"dynamodb:Attributes":["pk"]}}}]}

Reading the results: "allowed" means the request passed authorization and succeeded. Any error other than AccessDeniedException (ResourceNotFoundException, ConditionalCheckFailedException, ValidationException) means it passed authorization and failed afterwards.

Inline

  • [Blocker] request_attributes.rs, Scan arm: a Scan of a GSI or LSI holds the table's key attributes, not the index's (Experiments 1 to 5). Query is right and keeps its rule (6, 7). The same claim is in the docs (comment on 10-security-model.md).
  • [Difference] request_attributes.rs, Query arm: DynamoDB reads the names in the key condition into the key, which shows on a table that does not exist (Experiments 8, 9).
  • [Difference] request_attributes.rs, BatchGetItem: DynamoDB does not add the names in Keys, as for GetItem (Experiments 10, 11).

Re-measured and matching

Beyond the experiments inline, I ran the request shapes below on both endpoints with policies of the description's shapes. Each matched DynamoDB, and each matched what the description says DynamoDB does.

  • PutItem: item attributes, ConditionExpression, Expected, a failing condition with ReturnValuesOnConditionCheckFailure ALL_OLD.
  • UpdateItem:
    • SET, REMOVE, ADD and DELETE targets, a nested SET secret.a and REMOVE lst[5], and AttributeUpdates
    • ConditionExpression and Expected, which the description lists as not measured
    • SET values that read ssn: #n = ssn, if_not_exists(ssn, :v) and list_append(:l, ssn), all authorized under a Deny on ssn
    • ReturnValues ALL_OLD
  • DeleteItem: with and without a condition, and Expected, which the description lists as not measured.
  • GetItem: AttributesToGet, projections of secret.a and #s.a, and no projection; ForAllValues with and without pk; the Null Deny.
  • Query and Scan:
    • FilterExpression, QueryFilter, ScanFilter, AttributesToGet and ExclusiveStartKey
    • ProjectionExpression, which the description lists as not measured
    • Select COUNT, the Null Deny and the Null Allow, and the bare StringNotEquals Allow
    • the table's sort key, which the description lists as not measured: it is in the key
  • BatchGetItem and BatchWriteItem: projections, AttributesToGet, puts and deletes under ForAllValues, and the per-table ForAnyValue union.
  • TransactGetItems and TransactWriteItems: the Get, Put, Update and ConditionCheck forms, a Delete with and without a condition, and a failing ConditionCheck with ALL_OLD.
  • Tables that do not exist: GetItem, Scan and transaction Get answer ResourceNotFoundException on both endpoints. PutItem, UpdateItem, DeleteItem, a BatchWriteItem DeleteRequest and a transaction Delete answer AccessDeniedException on both.

Locally: test_dynamodb_attributes_*.py, test_fgac_attributes.py, test_fgac_leading_keys.py and test_tag_request_allow_conditions.py pass on PostgreSQL (151 passed), and cargo test -p extenddb-server -p extenddb-auth passes.

PartiQL (no change needed here)

This branch predates PartiQL. On a build of this PR merged onto main (SQLite backend), ExecuteStatement, BatchExecuteStatement and ExecuteTransaction matched DynamoDB in all 33 cases I ran. Those covered:

  • SELECT, INSERT, UPDATE (SET and REMOVE) and DELETE
  • ssn read in a WHERE clause, SELECT secret.a, and a SELECT from a GSI, which needs the GSI's key in a ForAllValues list
  • ForAllValues and ForAnyValue Allows

That path fills the key itself (item_authz.rs). The security model manual and the troubleshooting paragraph describe what the key holds "on every item request" without mentioning PartiQL, so a sentence pointing to the PartiQL rules would help readers.

Comment thread crates/server/src/request_attributes.rs Resolved
Comment thread docs/manuals/10-security-model.md Resolved
Comment thread crates/server/src/request_attributes.rs Resolved
Comment thread crates/server/src/request_attributes.rs Resolved
No file conflicted. Main's #830 added a table name check at the top of
authorize_request in crates/server/src/request_helpers.rs, and this
branch replaced extract_attributes there with request_attributes. The
merged function keeps the name check first, before the key info lookup.
It keeps this branch's dynamodb:Attributes value on the single-table
branch and on the batch and transaction branch.
The review of this pull request measured three request shapes on
DynamoDB on 2026-11-09, in two accounts. A further measurement on
2026-11-10, in one account, covered a later page of a Scan. This change
follows those measurements.

A Scan with IndexName no longer adds the index's key schema. On a
global secondary index the service allows a Scan under an Allow whose
ForAllValues list holds only the table's keys. A Deny on that index's
partition key does not refuse it either. On a local secondary index, an
Allow of the table's keys and the projected name grants it. The branch
refused these first-page requests, which carry no ExclusiveStartKey. A
later page's start key holds the table's and the index's key
attributes, and the key holds them. DynamoDB does the same: on
2026-11-10 it refused page 2 of either index under both Allows. A Deny
on the global index's partition key or the local index's sort key
refused it too. Page 2 of the table passed under the Allow of its keys
alone. A Query of an index still adds its full key schema, sort key
included (measured on a global secondary index).

A Query now adds the names in KeyConditionExpression and the keys of
KeyConditions. The expression goes through the engine's limits, its
reserved-word check when the enforce_reserved_keywords setting is on,
and its key-condition parser. A key condition that one of them refuses
adds no names. On a table that exists, the engine refuses a non-key
attribute in KeyConditions. It also refuses one in a
KeyConditionExpression on a key schema with several HASH or several
RANGE attributes. On a key schema with one HASH and at most one RANGE
attribute, it checks a KeyConditionExpression only for an equality on
the partition key. So it accepts a clause on another attribute
(crates/engine/src/query.rs:295-297 and :328-336), and the key also
holds that name. On a missing table the service answers
AccessDeniedException under a Deny on a name in the key condition. The
branch answered ResourceNotFoundException there, which told a denied
caller the table was missing.

BatchGetItem no longer adds the names in Keys. For a table named by its
name it holds what GetItem holds. On a missing table the service
answers ResourceNotFoundException under a Deny on the key attribute,
where the branch answered AccessDeniedException. A RequestItems key
that is a table ARN gets no key info at authorization, so that part
holds no key attributes. A GetItem by the same ARN holds them. A
separate change will close that gap.

The security model manual, troubleshooting guide, v0.1.15 upgrade entry
and evaluator comments on the absent key describe these rules. The
manual also says how the PartiQL path fills the key.

Unit tests cover each arm. Wire tests cover the review's experiments and
four later-page answers.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

@hitesh96db, the branch now holds a merge of main (6ee143f) and a fix commit, 147378a. For each item of your review:

  • Blocker, Scan with IndexName: the Scan arm in crates/server/src/request_attributes.rs no longer adds the index's key schema, and the Query arm keeps it. The unit test is now scan_on_an_index_holds_the_table_key_attributes_and_not_the_index_keys. Experiments 1 to 7 are wire tests in tests/test_dynamodb_attributes_key_attributes.py, on a fixture with your table's keys, indexes and item. A later page of a Scan with IndexName holds the index's key attributes through its ExclusiveStartKey, as measured on 2026-11-10 on a two-item table shaped like yours. DynamoDB refused page 2 of gsi2 and lsi under allow-scan-table-keys and allow-scan-pk-sk, and of gsi2 under deny-gk. Page 2 of the table passed under allow-scan-pk-sk. The branch already counted those names. New wire tests pin four of these answers. For a local secondary index, the docs cite Experiment 2 and the 2026-11-10 rows, and say the other rules follow from that key content.
  • Difference, Query key condition: the Query arm adds the paths in KeyConditionExpression and the keys of KeyConditions. It reads the expression with the engine's key condition parser, under the engine's size and token limits. It also runs the engine's reserved-word check when enforce_reserved_keywords is on. A key condition that one of these refuses adds no names. The evaluator comments in condition.rs, context.rs and multivalued.rs now list a refused key condition on a table that does not exist among the cases with no key. Experiments 8 and 9 are wire tests.
  • Difference, BatchGetItem Keys: the Keys loop is gone, so a BatchGetItem part of a table named by its name holds what a GetItem holds. The BatchGetItem row of 10-security-model.md says so. Experiments 10 and 11 are wire tests.
  • Doc row, 10-security-model.md: the Scan row lists the table's key attributes and not the scanned index's. The paragraph under the table cites your measurements for a Query of an index and for the first page of a Scan of one. docs/troubleshooting.md, the v0.1.15 entry in 07-upgrade-manual.md and the PR description say the same.
  • PartiQL: the security model manual now says the engine authorizes each statement with its dynamodb:PartiQL* action. The engine fills dynamodb:Attributes with the table's key attributes and the names the statement uses. The troubleshooting paragraph names the manual for these rules.
  • One case stays as it is. A BatchGetItem whose RequestItems key is a table ARN, and a transaction Get or ConditionCheck whose TableName is an ARN, get no key attributes at authorization. This PR leaves that to A table ARN passed as TableName returns ResourceNotFoundException #442, which covers a table ARN as TableName.

Tests ran on 147378a unless a line names an earlier version of the fix commit.

  • The 99 tests of the three test_dynamodb_attributes_* files passed on SQLite at the default index_propagation_delay_ms of 10 ms. Four of them are the new later-page tests in test_dynamodb_attributes_key_attributes.py.
  • All 32 request_attributes unit tests passed. The format and clippy checks passed, and the description lists their commands.
  • At 0d578028d, the 438 auth unit tests passed. The head differs from it only in //! comments of request_attributes.rs, test comments, docs and wire tests.
  • Against a server with request_attributes.rs from 6ee143f, the tests of Experiments 1 to 4 and 8 to 10 failed. Those of 5, 6, 7 and 11 passed. That run used 46d8b051b, the first version of the fix commit.

hitesh96db
hitesh96db previously approved these changes Nov 10, 2026

@hitesh96db hitesh96db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-checked 147378a against my review of 890ee53. All of it is addressed:

  • Scan with IndexName (Blocker): the Scan arm no longer adds the index's key schema, and Query keeps it.
  • Query key condition: the Query arm reads KeyConditionExpression and KeyConditions.
  • BatchGetItem Keys: they no longer add names.
  • Docs: the Scan and BatchGetItem rows of 10-security-model.md, troubleshooting.md and the v0.1.15 upgrade entry now match. The PartiQL pointer is in.

Re-test

Release builds of 147378a on PostgreSQL and on SQLite, each a fresh catalog.

  • Wire tests: the three test_dynamodb_attributes_* files, test_fgac_attributes.py, test_fgac_leading_keys.py and test_tag_request_allow_conditions.py: 166 passed on each backend.

  • My review's experiments, re-run: a driver of my own sent the exact aws dynamodb commands and policies from my review. It made a new IAM user per experiment, in a new account. It covered:

    • the control and Experiments 1 to 11
    • a later page (Limit 1, then the first page's LastEvaluatedKey as ExclusiveStartKey) of a Scan of gsi2 and of lsi under allow-scan-table-keys and allow-scan-pk-sk, of gsi2 under deny-gk, and of lsi under a Deny on ls
    • a later page of a Scan of the table under allow-scan-pk-sk

    All 28 cases gave the answer DynamoDB gave: the one in my review for Experiments 1 to 11, and the one in your 2026-11-10 measurement for the later pages. That held on both backends. I did not re-measure the later-page rows on DynamoDB.

  • Rust: cargo test --locked -p extenddb-server -p extenddb-auth passed (877 tests), and cargo fmt --all -- --check and cargo clippy -p extenddb-server -p extenddb-auth --all-targets -- -D warnings on 1.97.0 are clean.

  • Negative control: with request_attributes.rs from 6ee143f, four of the new unit tests fail:

    • scan_on_an_index_holds_the_table_key_attributes_and_not_the_index_keys
    • a_query_of_a_table_that_does_not_exist_holds_its_key_condition_names
    • a_key_condition_with_a_reserved_word_gives_no_names
    • batch_get_item_keys_add_no_names_of_their_own

Outside this PR

  • A table ARN as a RequestItems key (left to #442). Under a Deny on pk, a BatchGetItem of an existing table answers AccessDeniedException when it names the table by its name. It answers ResourceNotFoundException when it names the table by its ARN, because the engine does not resolve the ARN there. Nothing is read today. Once #442 makes the data plane resolve a table ARN, that part holds no key attributes and would pass a Deny on a key attribute. Whatever fixes #442 should fill the key info for an ARN-named part at the same time.
  • A non-key clause in a key condition. On a table keyed on pk and sk, ExtendDB accepts the KeyConditionExpression pk = :p AND #n = :n (with #n for name) and answers Count 0 for an item that matches both clauses. This is #674. As your commit message says, the key then also holds name.

Main at cbc3680 is 13 commits ahead of the last merge. One file
conflicted: crates/server/src/lib.rs, where #355 adds the module
repeated_headers and this branch adds request_attributes beside it.
The merge keeps both, in name order.

docs/manuals/07-upgrade-manual.md, docs/manuals/10-security-model.md
and docs/troubleshooting.md merged without a conflict. Against main,
the merged tree changes the same 18 files with the same line counts as
this branch did against its merge base.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

@hitesh96db Thanks for the approval. The branch conflicted with main, so I pushed 26617aa, a merge of main at cbc3680. The one conflict was in crates/server/src/lib.rs, where #355 adds mod repeated_headers; beside this branch's mod request_attributes;. The merge keeps both. Three docs files merged without a conflict. Against main, the merged tree changes the same 18 files with the same line counts as 147378a did against its merge base. The commit adds no other change. The push dismissed your approval, so the branch needs it again.

@hitesh96db hitesh96db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hitesh96db
hitesh96db added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working release-blocker Must ship for its milestone's release security v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants