[LeeroyHannigan]

What

dynamodb:ResourceTag/<key> is now absent in every policy, as on DynamoDB, which never populates that key. Resource tags evaluate only as aws:ResourceTag/<key>, which is unchanged. Before this change ExtendDB filled dynamodb:ResourceTag/<key> from the table's tags, so a condition on it matched where DynamoDB treats the key as absent.

Stored policies are migrated so that none of them changes meaning:

• extenddb migrate rewrites every dynamodb:ResourceTag/<key> condition key and every ${dynamodb:ResourceTag/...} policy variable to the aws:ResourceTag/ form, in any case, under every operator. It covers inline user, group and role policies, user and role permissions boundaries, and stored session policies. Only Condition blocks are changed. Role trust policies are left alone, because the AssumeRole path never resolved the key.
• Without --yes, migrate lists each policy it would rewrite (account, principal, policy name and key count) and writes nothing. With --yes it rewrites and records that the catalog is migrated. Policy bodies are never printed.
• If one operator block names both aws:ResourceTag/<key> and dynamodb:ResourceTag/<key> with the same value, the two were one condition before this change, and the dynamodb: key is dropped. With different values, migrate lists the policy as NEEDS ATTENTION, exits non-zero and does not record the catalog as migrated.
• A failed policy lookup also makes migrate exit non-zero without recording the catalog as migrated, so the next run finishes the rewrite.
• extenddb serve refuses to start on a catalog that has not been migrated, with an error that names extenddb migrate --yes. A new catalog from init, bootstrap on serve, in-memory or dev mode is recorded as migrated when it is created, on all four backends.

There is no scan at server start. The manuals and design docs now use aws:ResourceTag, and docs/manuals/07-upgrade-manual.md has a v0.1.14 entry with the migrate step and a table of what each statement shape does for a policy that is not rewritten.

Why

With the key resolved, a policy written for DynamoDB behaved differently on ExtendDB. An Allow gated on a positive operator over dynamodb:ResourceTag/<key> granted where DynamoDB grants nothing, and a negated operator on it did not match where DynamoDB matches. Changing the evaluation alone would have changed the meaning of policies already stored with that key, so the change ships with a migration that rewrites them to the form DynamoDB evaluates.

Testing done

• tests/test_resource_tag_migration.py, run by the SQLite integration job and skipped on the PostgreSQL job:
  • On one data directory, it stores a user policy, a group policy with every operator shape and the policy variable form, and a permissions boundary.
  • It runs migrate without --yes and checks the listing and that nothing changed, then runs migrate --yes and checks the decisions after a restart.
  • It checks that a second run changes nothing and that serve refuses a catalog without the migration marker.
  • A second test stores a same-value and a different-value collision. It checks that migrate --yes exits non-zero without the marker and that serve refuses, then fixes the policy, runs migrate again and checks that the same-value Deny still applies.
  • The first test fails on main. The second fails on the previous head of this branch, where migrate --yes recorded the catalog as migrated with a collision left in place.
• tests/test_abac.py: the resource-tag cases use aws:ResourceTag, and new cases pin dynamodb:ResourceTag as absent. Each expected result was measured against DynamoDB.
• Unit tests for the evaluator under bare, IfExists, ForAllValues, ForAnyValue and Null operators, and for the rewrite: every form, mixed case, collisions, text outside Condition blocks, idempotency. An equivalence test evaluates each stored shape on main's semantics before the rewrite and on the new evaluator after it, and requires the same decision. SQLite store tests run the migration end to end.
• An upgrade from main on SQLite and on PostgreSQL: serve refused the old catalog until migrate --yes ran. On SQLite, stored policies gave the same decisions after the migration as on main, in 40 cases across operator forms, policy kinds and tag states. Trust policies were unchanged.
• cargo fmt --check, cargo clippy on 1.97.0 with -D warnings, and cargo test --workspace on stable and 1.88. tests/test_resource_tag_migration.py, tests/test_abac.py, tests/test_auth_integration.py and tests/test_auth_permissions.py pass on SQLite (212) and on PostgreSQL (210, with the 2 SQLite-only tests skipped).

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The ManagementStore trait gains list_session_policies and update_session_policy, with default implementations, for the migration, and the catalog gains one settings row that records the migration. docs/adr/0022-trust-policy-deny-over-allow.md lists the resolved keys.

Breaking changes

• extenddb migrate --yes must run before the new binary serves an existing catalog. On PostgreSQL, pass --pg-user and set EXTENDDB_PG_PASSWORD, as for init.
• A policy that names dynamodb:ResourceTag/<key> and is written after the upgrade, for example from an old template, is not rewritten. Under it, a positive operator no longer matches and a negated operator now matches. The upgrade manual has the table for every operator form.
• A policy that names both forms of one key with different values in one operator block has to be fixed by hand before migrate completes.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

[robinnsc]

These bullets hold for the bare operators, but an absent key evaluates differently under the IfExists, ForAllValues and Null forms (condition.rs:43-123), and for some of them the outcome is the opposite of what's described here. On a SQLite build of this branch, with a table tagged env=prod and D meaning dynamodb:ResourceTag/env:

• Allow GetItem if StringEqualsIfExists D=dev and Allow GetItem if ForAllValues:StringEquals D=dev now allow on every table, where fix(auth): enforce trust-policy Deny and match IAM condition evaluation #420 denied. The same IfExists form in a permissions boundary now lets GetItem through.
• Allow GetItem if Null D=true now allows, and Allow * + Deny if Null D=false no longer blocks.
• Deny if StringEqualsIfExists D=dev now starts blocking, rather than stopping.
• ForAnyValue:StringNotEquals stays false, so a negated operator doesn't always flip.

The ones that matter most are the Allows and boundaries that were written to scope access by tag and now grant everything, since an operator reading this note would expect those to fail closed. Might be worth listing the shapes separately (bare positive, bare negated, IfExists, ForAllValues, ForAnyValue, Null) with the outcome for each, and mirroring that in the Breaking changes section of the description.

[LeeroyHannigan]

Thanks, you're right that the note only held for the bare operators. The PR now rewrites stored policies with extenddb migrate, so a policy that existed before the upgrade keeps its decision under every operator form (ad250e3, 7046522). The upgrade manual has a table of each shape for a policy that isn't rewritten, for example one written later from an old template: bare positive, bare negated, IfExists, ForAllValues, ForAnyValue, Null true and false, and the ${...} variable. I checked each row against the new build with a table tagged env=prod.

[robinnsc]

Condition values go through expand_policy_variables, which calls resolve_key and leaves an unresolvable variable as a literal, so a policy that uses ${dynamodb:ResourceTag/env} as a policy variable also changes meaning, but this only looks at object keys (and ignores_key_name_appearing_as_a_value pins that). On a SQLite build, Allow GetItem if StringEquals aws:PrincipalTag/env = ${dynamodb:ResourceTag/env}, which is the usual principal-tag-equals-resource-tag ABAC pattern, went from allow to deny, and the mirror forms (a StringEquals Deny, a StringNotEquals Allow) open access instead. A policy whose only reference was the variable got no warning on restart. Could consider also matching string values that contain ${dynamodb:resourcetag/ (any case), and adding the variable form to the upgrade note.

[LeeroyHannigan]

migrate now rewrites ${dynamodb:ResourceTag/...} inside condition values as well as the condition keys, in any case (ad250e3). I checked the principal-tag-equals-resource-tag pattern on main and on the migrated build, and it gives the same decisions. One case still differs: on a table without the tag, neither form expands, so the literal left in the comparison changes from ${dynamodb:ResourceTag/<key>} to ${aws:ResourceTag/<key>}. The upgrade manual says so.

[robinnsc]

I think trust policies are a false positive here: they're evaluated with AssumeRoleContext, whose resolve_key never resolved dynamodb:ResourceTag/ on #420 either, so their behaviour doesn't change, and on a SQLite build an AssumeRole gated on StringNotEquals dynamodb:ResourceTag/... succeeded both before and after. The warning tells the operator to rewrite them to aws:ResourceTag/, which won't change anything on that path, and the upgrade note's "Who is affected" list includes them. The opposite case is session policies, which are evaluated with RequestContext and do change (a session-policy Deny on StringEquals dynamodb:ResourceTag/env=prod stopped blocking), but aren't scanned and aren't mentioned in the note. Since sessions can last up to 12 hours, a sentence about re-issuing them might be enough.

Relatedly, the scan only runs at startup, so a policy using the key that's written after the upgrade (for example copied from an old template) has no effect and gives no signal until the next restart. A warning at the put_policy / boundary write points, like the one logged here, might be worth adding alongside the startup scan.

[LeeroyHannigan]

Trust policies aren't touched now: the AssumeRole path never resolved the key, as you found, so the migration leaves them alone. Session policies are stored in iam_sessions on all four backends, and migrate rewrites them along with the identity policies and boundaries. The session token carries no copy of the policy, and the authorization cache is cleared on restart, so a session issued before the upgrade is evaluated with its rewritten policy.

The startup scan is gone. serve now refuses a catalog that hasn't been migrated (0cfb988). I didn't add a warning when a policy is written with the old key after the upgrade. IAM accepts that key and DynamoDB ignores it, and we now do the same. The upgrade manual's table covers what such a policy does.

[robinnsc]

The if let Ok(...) here, and for groups, roles and boundaries, drops the error without logging it, and the account and policy listing errors only log at debug, while the default level is info. So if any lookup fails, the scan is silently incomplete and the operator sees a clean start, which reads as "no affected policies". A couple of concrete paths: on SQLite, list_roles fails the whole call if one trust policy row doesn't parse, which skips every role, role policy and boundary in that account, and a transient error on list_all_accounts at startup skips everything. Might be worth counting failures and emitting one warn-level line such as "resource-tag scan incomplete: N lookups failed" when the count is non-zero.

[LeeroyHannigan]

The scan is gone, and migrate doesn't drop errors. Every failed policy lookup is counted. If the count isn't zero, migrate --yes prints it, exits non-zero and doesn't record the catalog as migrated, so serve keeps refusing and the next run finishes the rewrite (7046522). A stored session policy that doesn't parse can't be evaluated under either form. It's logged with its account, role and session name and skipped (97ef9a9).

[robinnsc]

As far as I can tell this test is skipped in every CI leg: no workflow sets EXTENDDB_SQLITE_BINARY or EXTENDDB_BINARY for the pytest runs (integration.yml builds target/release/extenddb and calls devtools/run-tests, which uses its own shell variable), and the module skips when both are empty. It passes locally when pointed at a binary. Could consider falling back to target/release/extenddb as lifecycle_helpers.py does, or setting the variable in the SQLite job. It would also be worth asserting that the first server start (before the policy is stored) logs no scan line, and checking the count in the summary, so a scan that always warns doesn't pass.

[LeeroyHannigan]

Thanks, that was right. The startup scan and its test are gone in the rebuild. The new test, tests/test_resource_tag_migration.py, takes the binary from lifecycle_helpers.EXTENDDB_BINARY, so it uses target/release/extenddb like the other lifecycle tests and needs no extra variable. It skips only when the binary rejects init --backend sqlite, which is the PostgreSQL leg.

In the SQLite job on 5bc74f3 (run-integration-sqlite, job 119791755900) both tests ran and passed: test_resource_tag_migration_wire and test_resource_tag_migration_collisions.

On your second point, the test checks that the first start of a fresh catalog logs nothing about the migration. After a user policy, a group policy and a boundary with the key are stored, migrate without --yes lists the rewrites and exits non-zero with the stored documents unchanged. After migrate --yes, no stored policy contains dynamodb:ResourceTag, and a second migrate prints No dynamodb:ResourceTag condition keys found. A migrate that always reported rewrites would fail that last check.

[jcshepherd]

Should this tag update be treated as a migration? First, it changes the authorization semantics with only warnings that an end user may or may not see. Second, it seems like fixing it is a fairly mechanical change and preserves the user's intent. Third, we probably want to avoid putting linters, etc., in the path of 'serve', especially when their lifetime is indefinite. If we can't treat it as a migration, can we at least provide tooling that users can use to correct their policies?

[LeeroyHannigan]

@jcshepherd Agreed, it's a migration now. I've rebuilt the PR on main and force-pushed it.

extenddb migrate rewrites dynamodb:ResourceTag/ to aws:ResourceTag/ in stored inline policies, boundaries and session policies. Run without --yes, it lists each policy it would change and writes nothing, which covers the tooling you asked for. Two forms of the same key with the same value are merged. With different values, the policy is listed for a fix by hand and the migration doesn't complete.

serve reads one settings row at startup and refuses a catalog that hasn't been migrated, so there's no scan or linter in the serve path. New catalogs are recorded as migrated when they're created. I tested an upgrade from main on SQLite and PostgreSQL. On SQLite, 40 stored policy shapes gave the same decisions after the migration as before it.