[hitesh96db]

What

A deleted user, group, role or account now leaves nothing behind for one created later with the same name or ID, on all four backends.

Inline policies and permissions boundaries are stored by principal name, with no foreign key to the principal. On MongoDB and Cassandra, no IAM record is linked to its principal or account at all. Before this change, each delete path removed a different subset (read from main at 35edb778):

Delete	SQLite / PostgreSQL left	MongoDB left	Cassandra left
user	inline policies, boundary	boundary, tags	boundary
group	inline policies	inline policies, members	inline policies, members
role	inline policies, boundary	inline policies, boundary, tags, sessions	inline policies, boundary, tags, sessions
account	nothing (foreign keys)	every IAM record of the account	every IAM record of the account

Changes:

• Deletes remove everything the principal owns. SQLite and PostgreSQL delete a principal's inline policies and boundary in the same transaction as its row. MongoDB and Cassandra have no transaction across tables, so they delete what a principal owns first, credentials first, and its own record last. A failure then leaves the principal with less than it had, never a credential or permission without it, and a retry finishes. MongoDB's delete_user now returns errors from these deletes; before, it ignored them.
• Account deletes on MongoDB and Cassandra remove the account's users, groups and roles, and everything they own, before the account record. The "refuse while tables exist" check stays.
• Creates start clean. After its own insert succeeds, a user, group or role create deletes anything stored under its name. A create that is refused as a duplicate touches nothing. On MongoDB and Cassandra, an account create does the same for its ID. Records that an earlier release left behind are cleared when the name or ID is used again, without a catalog migration. If the cleanup fails, MongoDB and Cassandra remove the new record again, so a retry is not refused as a duplicate.
• MongoDB refuses a taken account ID. Its accounts collection is unique on account_name only, so create_account now checks the ID first. Before, a second account with the same ID could be created, and the cleanup on create must never reach a live account.
• A deleted account grants nothing (MongoDB and Cassandra). The access key and session lookups, the console password hash read, the management-API password check and the AssumeRole trust-policy read treat a principal whose account no longer exists as unknown. This covers records that an earlier release's account delete left behind. On Cassandra the account delete now commits at quorum, which the console's quorum reads need to see it.
• A policy or boundary on a missing principal is refused. The management API and the console answer HTTP 404 IAM <user|group|role> not found instead of 204, as IAM answers NoSuchEntity. The check is in the server: the storage methods stay plain data access, and the cleanup on create covers a put that races a delete.
• SQLite: no transaction left open on a pooled connection. Dropping a SQLite transaction only queues its rollback. A BEGIN IMMEDIATE on that connection before the rollback ran failed with attempted to call begin_with at non-zero transaction depth. That already happened on create_user's error paths; the new NotFound and AlreadyExists returns made it frequent. The creates and deletes now commit or roll back explicitly before they return.

I kept cascade rather than IAM's DeleteConflict refusal: ExtendDB already cascades access keys, tags and memberships, and refusing would break existing callers. docs/differences-from-dynamodb.md records the difference.

Why

Closes #693.
Closes #694.

A principal created with a deleted principal's name got its permissions: an inline policy or boundary on every backend, and on MongoDB and Cassandra also its access keys, sessions or group members. On MongoDB and Cassandra, a deleted account's users stayed in the catalog with their password hashes.

On SQLite, on main (35edb778), I reproduced the role, user and group cases with extenddb manage. A recreated role listed the old policy and boundary, and a new session of it was allowed ListTables. A recreated user and a recreated group behaved the same way.

Testing done

• New shared check extenddb_storage::iam_leftovers_conformance (feature test-util). Each backend's suite runs it against a real database. It checks that a delete leaves no policy or boundary, that a create over rows stored under its name starts empty, that a refused duplicate create keeps the live principal's policies and boundary, and that an account created again with the same ID has no principals or policies.
  • SQLite (in-memory) and PostgreSQL 17 (tests/iam_leftovers.rs): pass.
  • Negative controls: the check fails with main's users.rs, groups.rs and roles.rs on both backends. With only the cleanup in SQLite's create_user removed, it fails at "created over rows left under its name".
  • MongoDB (management_store::live_tests, added to devtools/run-mongodb-tests --rust-live) and Cassandra (tests/direct/iam_leftovers.rs): not run locally, because this machine has neither. Both also test that a user and role of a deleted account (only the account record removed) cannot log in or be assumed. The credential stores' new checks have no unit test: MongoDB's hard-codes the extenddb_catalog database, and Cassandra's is private to the crate.
• New wire test tests/test_iam_delete_leftovers.py (9 tests, all backends). For a role, user, group and account, it gives each everything it can own, deletes it, checks that its key or session is refused, creates it again, and checks that it starts empty and denied. It also checks that the account's user can no longer log in through the management API or the console. Two tests check the 404 on a missing principal.
  • On main's SQLite build, 8 of the 9 fail. The account test passes there, as expected: the foreign keys already cascade an account delete on SQLite. Its failure would show on MongoDB and Cassandra, which I could not run.
  • On this branch, the 27 existing pytest files that create, delete or attach to principals, plus the new file, pass on SQLite release builds (668 passed, 8 skipped) and PostgreSQL release builds (666 passed, 10 skipped), on the head with main merged.
• cargo +1.97.0 fmt --all -- --check: clean.
• cargo +1.97.0 clippy --all-targets -- -D warnings on the six changed crates, with both feature sets of clippy.yml: clean. The workspace-wide run stops at an unused variable in crates/core/tests/data_file_decode_peak.rs, which this branch does not touch.
• cargo +1.88.0 check on the changed crates: OK.
• cargo test --lib for storage, storage-sqlite, storage-mongodb, storage-cassandra and server: pass.
• cargo test -p extenddb-storage-postgres --tests against PostgreSQL 17: 395 passed. 4 failed, and they fail the same way on main here: the 3 migrate_data_db_name tests need a password in the connection string, and a_role_that_does_not_own_the_tables_is_told_what_to_run depends on the local superuser setup.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. No schema, trait or wire change. The management API now answers 404 where it answered 204 for a policy or boundary on a missing principal.

Breaking changes

• PUT .../users|groups|roles/{name}/policy/{policy} and PUT .../users|roles/{name}/permissions-boundary on a principal that does not exist answer 404 IAM <type> not found, where they answered 204. A script that attaches before it creates must create first.
• Deleting a principal removes its inline policies and boundary (and, on MongoDB and Cassandra, everything else it owns). A deployment that relied on them coming back when a principal with the same name was created again must attach them again.
• On MongoDB, create-account with an account ID that is already in use answers Account already exists.

The upgrade manual has a v0.1.15 entry. No catalog migration is required.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

[navareamol]

No blocking findings. Two things to settle:

1. CI is red and the branch is behind main. run-tests fails on panicking_refresh_does_not_leak_inflight_flag in crates/cache, which this PR does not touch. It passed 100 of 100 runs on main here, so I think it is a flake. Could you update the branch and rerun it? CodeQL also reports 4 "hard-coded cryptographic value" alerts on the test passwords in iam_leftovers.rs and live_tests.rs. Could you generate the password in the test, or dismiss the alerts with a comment?
2. The description says a deleted principal leaves nothing behind, but the failed-login history in login_attempts is keyed by name and follows it. Measured on SQLite and Cassandra: 6 wrong passwords for a user, delete the user, create the same name again, then the correct password from the same address gets 429 for up to 15 minutes. It is the same on main, so this PR does not cause it. Could you clear that name's login_attempts rows in the delete and the create, or narrow the claim in the description? Filing it as its own issue also works for me.

The wire tests pass on SQLite, PostgreSQL and Cassandra. On main, 8 of the 9 fail on SQLite, as the description says. The two Cassandra direct tests pass too.

I'll approve once 1 is green and 2 is settled.

[navareamol]

If the clear fails and the delete of the new row fails too, the second error is only logged and the create returns the first. The principal then exists with the old rows still under it. A retry of the create gets 409 and skips the clear, so nothing removes them. Is anything else clearing them in that case? I read this from the code and did not inject the failure.

[hitesh96db]

Thanks @navareamol. Both points are addressed in the commits pushed to b61b793e. CI is running on that head now.

1. CI and the branch

• Branch updated (8100d88d): merged main. The one conflict was in the upgrade manual, where this PR and main each added a section in the same place; both are kept.
• panicking_refresh_does_not_leak_inflight_flag (489849bb): it is a timing flake, not caused by this PR. It failed with the same metrics (stale_hits: 2, refresh_success: 0, refresh_skipped_inflight: 1) on main (run 42729580298) and on the merge-queue runs for fix(authz): check a point-in-time restore's target before its source #827 and fix(storage-postgres): store attribute names containing U+0000 in the catalog #833.
  • Cause: the test slept a fixed 50 ms and assumed the spawned refresh that panics had run and been dropped by then. A busy runner misses that window. A 60 ms delay before the panic reproduces it every time.
  • Fix: the test now uses a 60 s hard TTL and waits, with a 10 s deadline, for the panicking refresh to run and then for a refresh to succeed.
  • Checked: it passes with a 60 ms and a 1 s delay injected before the panic, and the cache suite passed 50 of 50 runs. With the RAII guard removed it still fails, at the deadline.
• CodeQL (5ffbe4d8): the Cassandra and MongoDB tests now generate the password per run instead of using a literal, so the 4 alerts should close.

2. login_attempts follows the name (b61b793e)

You were right: failed logins are keyed by the sign-in name account_id/user_name, with no link to the user. I cleared them rather than narrowing the claim:

• a user's delete and its create delete the failed logins under its sign-in name. The create covers failures recorded while no user had the name.
• an account's delete deletes those under every account_id/ name.
• this is on all four backends. SQLite and PostgreSQL do it in the principal's transaction; MongoDB (failed_logins) and Cassandra do it with the rest of what the principal or account owns.
• the account prefix is matched literally, not as a pattern, so another account whose ID starts with the same digits keeps its failures.
• the per-address limit is unchanged.

Tests:

• The shared iam_leftovers_conformance check now covers the user's delete, a create over a name with failures, and the account's delete, including the other account. It passes on SQLite and PostgreSQL locally. Removing any one of the three new SQLite calls makes it fail with the matching message. On MongoDB and Cassandra it runs in CI.
• Three wire tests in test_rate_limiting.py send your sequence (wrong passwords, delete, create the same name, right password), plus the variants of a create over a name with failures and an account created again. They pass against a SQLite build of this branch. Against the same build without the fix, they get 429.
• The upgrade manual, the storage guide and getting-started now list failed logins among what a delete removes.

[navareamol]

Both points are fixed at bb9b025, and CI is green. The failed-login history now goes with the name on delete, create and account delete, and the tests I ran on SQLite, PostgreSQL and Cassandra pass.

One follow-up, not for this PR: if the cleanup in clear_new_principal fails and removing the new row fails too, a retry gets 409 and the old rows stay. Could you file an issue for it? Approving.