diff --git a/README.md b/README.md index 57d0168ac..775c184b3 100755 --- a/README.md +++ b/README.md @@ -1,25 +1,21 @@ # ExtendDB -> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does -> not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room -> implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented -> in [Differences from DynamoDB](docs/differences-from-dynamodb.md). +> **ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented in [Differences from DynamoDB](docs/differences-from-dynamodb.md). -A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works -with DynamoDB works with ExtendDB, unchanged. +ExtendDB is a DynamoDB-compatible API adapter. It speaks the DynamoDB wire protocol, so any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged. ## Use Cases -- **Local development** — run DynamoDB workloads on your laptop with zero cloud dependency -- **CI/CD pipelines** — deterministic integration tests against a DynamoDB-compatible backend -- **Self-hosted deployments** — run DynamoDB workloads on your own infrastructure (on-premises, private cloud, edge) -- **Multi-cloud** — use DynamoDB semantics on any cloud that runs an ExtendDB-supported database backend -- **Air-gapped environments** — DynamoDB functionality with no internet connectivity +- **Local development**: run DynamoDB workloads on your laptop with zero cloud dependency +- **CI/CD pipelines**: deterministic integration tests against a DynamoDB-compatible backend +- **Self-hosted deployments**: run DynamoDB workloads on your own infrastructure (on-premises, private cloud, edge) +- **Multi-cloud**: use DynamoDB semantics on any cloud that runs one of the supported databases +- **Air-gapped environments**: DynamoDB functionality with no internet connectivity ## Features - Full DynamoDB wire protocol: CRUD, Query, Scan, Batch, Transactions, Streams, TTL, Import/Export -- Pluggable storage: PostgreSQL (default), SQLite, MongoDB, and Cassandra +- Pluggable storage: PostgreSQL (default), SQLite, MongoDB and Cassandra, with the database's own backup and replication tools - SigV4 authentication with IAM compatibility: users, groups, roles, policies, permissions boundaries - Web management console for account and credential administration - TLS with automatic self-signed certificate generation (replaceable with CA-signed certs) @@ -39,7 +35,7 @@ cargo build --release # Start ./target/release/extenddb serve --config extenddb.toml -# Use with any AWS SDK (TLS with self-signed cert — trust via AWS_CA_BUNDLE) +# Use with any AWS SDK (TLS with a self-signed certificate; trust it via AWS_CA_BUNDLE) export AWS_CA_BUNDLE=~/.extenddb/tls/cert.pem aws dynamodb list-tables --endpoint-url https://127.0.0.1:18443 --region us-east-1 ``` @@ -117,7 +113,7 @@ key_path = "/etc/extenddb/tls/key.pem" The production images contain ExtendDB only. The storage backend runs as a separate service. See [`docker/README.md`](docker/README.md) for the PostgreSQL image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB image, external database requirements, hardened local Compose stacks, smoke tests, and explicit init/migrate/serve -production lifecycle. The SQLite and Cassandra backends have no dedicated container guide yet. +production lifecycle. The SQLite and Cassandra backends have no dedicated container guide. By default `extenddb serve` daemonizes itself, which doesn't play well with container runtimes and process supervisors that expect PID 1 to stay attached. Pass `--foreground` (alias `--no-daemon`) to keep the process in the foreground and stream logs to stderr instead of syslog: @@ -130,7 +126,7 @@ Use this with Docker, Kubernetes, `systemd Type=simple`, runit, s6, or any other In `--foreground` mode extenddb writes no PID file and never creates `run_dir`, so the container can run with a read-only root filesystem. Two consequences: - `extenddb status` still works (it probes the port) but reports the PID as unknown. -- `extenddb stop` cannot signal the process — there is no PID file to read. Stop it through your supervisor, which delivers `SIGTERM` and triggers the same graceful shutdown. +- `extenddb stop` cannot signal the process, because there is no PID file to read. Stop it through your supervisor, which delivers `SIGTERM` and triggers the same graceful shutdown. Pass `--write-pid-file` to opt back in when you do want `stop` and `status` to work against a foreground server, for example when running it from a shell rather than under a supervisor. The file goes to the usual `run_dir` path, so neither command needs extra arguments, and `run_dir` has to be writable. @@ -144,19 +140,19 @@ extenddb healthcheck --endpoint https://127.0.0.1:18443 # explicit target This is a liveness check, which is what a container `HEALTHCHECK` wants. The `/health` path does not query the storage backend, so it will report healthy even if the storage backend becomes unreachable after startup. That is deliberate, since a liveness probe that failed on a database outage would restart every replica at once. A backend that is -unreachable at startup does stop the server from listening. There is no separate readiness endpoint yet. +unreachable at startup does stop the server from listening. There is no separate readiness endpoint. ### One instance per database Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `.lock` next to the database file and a second server on the same file will refuse to start (`init`, `migrate`, and `destroy` take the same lock while they run). On PostgreSQL, MongoDB and Cassandra nothing prevents a second ExtendDB instance from -running, but running more than one against the same catalog is not supported in this release: every instance runs every +running, but running more than one against the same catalog is not supported: every instance runs every background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key stays valid on the other instance for up to the cache TTL), and `/health` does not check the database. See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations). -To make the generated self-signed certificate valid for the name clients use — an in-cluster service DNS name, for example — pass `--tls-san` to `init` (repeatable): +To make the generated self-signed certificate valid for the name clients use (an in-cluster service DNS name, for example), pass `--tls-san` to `init` (repeatable): ```bash extenddb init --catalog-db extenddb_catalog \ diff --git a/crates/server/src/console/pages/docs_page.rs b/crates/server/src/console/pages/docs_page.rs index 2f19f77e4..91678fdff 100755 --- a/crates/server/src/console/pages/docs_page.rs +++ b/crates/server/src/console/pages/docs_page.rs @@ -35,7 +35,7 @@ pub async fn docs_page(State(state): State>) -> Response { let mut content = String::with_capacity(4096); content.push_str(r#"

Documentation

-

extenddb is not Amazon DynamoDB. It is an independent, clean-room implementation that speaks the DynamoDB wire protocol. It is not affiliated with, endorsed by, or sponsored by Amazon Web Services. “DynamoDB” is a trademark of Amazon.com, Inc.

+

ExtendDB is not Amazon DynamoDB. It is a clean-room implementation that speaks the DynamoDB wire protocol and contains no DynamoDB source code, an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. “DynamoDB” is a trademark of Amazon.com, Inc.

"#); let categories = [ diff --git a/docker/dockerhub-overview-dev.md b/docker/dockerhub-overview-dev.md index a0318079c..545eebdb0 100644 --- a/docker/dockerhub-overview-dev.md +++ b/docker/dockerhub-overview-dev.md @@ -38,6 +38,6 @@ For a durable, TLS-terminated deployment, use [`extenddb/extenddb-postgres`](htt This image is built for local development and CI. It serves plain HTTP and its authorization is open, so whoever holds the credential can do anything. Publish it to loopback only, as the command above does, and do not keep real data in it. -ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does not contain any DynamoDB source code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol; behavioral differences from the service are documented in [Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md). +ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol; behavioral differences from the service are documented in [Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md). More at [extenddb.org](https://extenddb.org) and [github.com/ExtendDB/extenddb](https://github.com/ExtendDB/extenddb). Licensed under Apache-2.0. diff --git a/docker/dockerhub-overview-mongodb.md b/docker/dockerhub-overview-mongodb.md index 772b375e1..e15184468 100644 --- a/docker/dockerhub-overview-mongodb.md +++ b/docker/dockerhub-overview-mongodb.md @@ -93,9 +93,7 @@ The same images and their signatures are mirrored by digest to ## Note -ExtendDB is an independent open source project managed by Amazon Web -Services. It is not Amazon DynamoDB and does not contain any DynamoDB source -code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room +ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol; behavioral differences from the service are documented in [Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md). diff --git a/docker/dockerhub-overview.md b/docker/dockerhub-overview.md index f2a2083fb..668e4ed6f 100644 --- a/docker/dockerhub-overview.md +++ b/docker/dockerhub-overview.md @@ -26,6 +26,6 @@ Every published image is signed with ExtendDB's release key, an ECDSA P-256 key ## Note -ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does not contain any DynamoDB source code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol; behavioral differences from the service are documented in [Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md). +ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol; behavioral differences from the service are documented in [Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md). More at [extenddb.org](https://extenddb.org) and [github.com/ExtendDB/extenddb](https://github.com/ExtendDB/extenddb). Licensed under Apache-2.0. diff --git a/docs/NOTICE.md b/docs/NOTICE.md index 288c4851c..23e4a67da 100755 --- a/docs/NOTICE.md +++ b/docs/NOTICE.md @@ -3,10 +3,10 @@ ## Notice -**ExtendDB (extenddb) is not Amazon DynamoDB.** This software is an independent, clean-room implementation that provides a DynamoDB-compatible API. It does not contain, derive from, or link to any Amazon DynamoDB source code, libraries, or proprietary components. +**ExtendDB is not Amazon DynamoDB.** This software is an independent, clean-room implementation that provides a DynamoDB-compatible API. It does not contain, derive from, or link to any Amazon DynamoDB source code, libraries, or proprietary components. -"DynamoDB", "Amazon DynamoDB", "AWS", and "Amazon Web Services" are trademarks of Amazon.com, Inc. or its affiliates. This project is not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services. +"DynamoDB", "Amazon DynamoDB", "AWS", and "Amazon Web Services" are trademarks of Amazon.com, Inc. or its affiliates. ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. **No warranty.** This software is provided "as is" under the Apache License, Version 2.0, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full terms. -**No guarantee of compatibility.** While extenddb aims to faithfully reproduce DynamoDB behavior, it is a separate implementation with known differences. Applications tested against extenddb should be validated against real DynamoDB before production deployment on AWS. See [Differences from DynamoDB](differences-from-dynamodb.md) for the full list of known behavioral differences. +**No guarantee of compatibility.** While ExtendDB aims to faithfully reproduce DynamoDB behavior, it is a separate implementation with known differences. Applications tested against ExtendDB should be validated against real DynamoDB before production deployment on AWS. See [Differences from DynamoDB](differences-from-dynamodb.md) for the full list of known behavioral differences. diff --git a/docs/adr/0000-template.md b/docs/adr/0000-template.md index b605d0688..5868c0b17 100644 --- a/docs/adr/0000-template.md +++ b/docs/adr/0000-template.md @@ -38,6 +38,6 @@ feedback, dependency constraints, security review. One bullet per reason. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0001-documentation-format.md b/docs/adr/0001-documentation-format.md index 163bd17d5..e7796cfd0 100755 --- a/docs/adr/0001-documentation-format.md +++ b/docs/adr/0001-documentation-format.md @@ -2,12 +2,12 @@ ## Context -extenddb needs a standard format for all documentation: design docs, ADRs, getting-started guides, and troubleshooting references. The two candidates are Markdown and LaTeX. +ExtendDB needs a standard format for all documentation: design docs, ADRs, getting-started guides, and troubleshooting references. The two candidates are Markdown and LaTeX. ## Options Considered -1. **Markdown** — lightweight, renders natively on GitHub/GitLab, editable in any text editor, supported by all Rust documentation tooling (`rustdoc`, `mdbook`). -2. **LaTeX** — typesetting system designed for academic papers and technical documents. Produces high-quality PDF output but requires a TeX distribution to build. +1. **Markdown**: lightweight, renders natively on GitHub/GitLab, editable in any text editor, supported by all Rust documentation tooling (`rustdoc`, `mdbook`). +2. **LaTeX**: typesetting system designed for academic papers and technical documents. Produces high-quality PDF output but requires a TeX distribution to build. ## Decision @@ -15,8 +15,8 @@ Markdown for all documentation. ## Rationale -- All existing extenddb documentation is already Markdown. Switching would require migrating dozens of files. -- Markdown renders in-place on code hosting platforms — no build step, no PDF artifacts to manage. +- All existing ExtendDB documentation is already Markdown. Switching would require migrating dozens of files. +- Markdown renders in-place on code hosting platforms: no build step, no PDF artifacts to manage. - Rust's `rustdoc` uses Markdown natively. Keeping external docs in the same format reduces context switching. - LaTeX adds a build dependency (TeX Live is ~4 GB) with no corresponding benefit for a software project's documentation needs. - Contributors can edit Markdown with zero tooling beyond a text editor. @@ -33,6 +33,6 @@ Markdown for all documentation. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0002-sql-injection-defense.md b/docs/adr/0002-sql-injection-defense.md index 8409f1add..f628218bc 100755 --- a/docs/adr/0002-sql-injection-defense.md +++ b/docs/adr/0002-sql-injection-defense.md @@ -2,11 +2,11 @@ ## Status -Accepted — 2026-04-07. Revised — 2026-04-21 (P30: reflect actual design). +Accepted: 2026-04-07. Revised: 2026-04-21 to reflect the implemented design. ## Context -extenddb stores DynamoDB table metadata and item data in PostgreSQL. User-supplied strings — table names, attribute names, key values, filter expressions — flow from HTTP requests through the engine layer into SQL queries. A SQL injection vulnerability here would allow arbitrary database access. +ExtendDB stores DynamoDB table metadata and item data in PostgreSQL. User-supplied strings (table names, attribute names, key values, filter expressions) flow from HTTP requests through the engine layer into SQL queries. A SQL injection vulnerability here would allow arbitrary database access. ## Decision @@ -14,9 +14,9 @@ extenddb stores DynamoDB table metadata and item data in PostgreSQL. User-suppli **Tier 1: Engine-layer validation.** All user-supplied strings are validated at the engine layer before reaching storage. The storage layer trusts that inputs have been validated. -- Table names: validated against DynamoDB's regex (`[a-zA-Z0-9_.-]+`, 3–255 chars) in `validate_table_name()` +- Table names: validated against DynamoDB's regex (`[a-zA-Z0-9_.-]+`, 3-255 chars) in `validate_table_name()` - Index names: validated against the same character set -- Account IDs: validated in `PostgresEngine::validate_account_id()` — rejects `"`, `\0`, and non-ASCII +- Account IDs: validated in `PostgresEngine::validate_account_id()` (rejects `"`, `\0`, and non-ASCII) - Attribute names: validated in expression parsing - Key values: type-checked via `AttributeValue` deserialization @@ -43,8 +43,8 @@ fn data_table_name(account_id: &str, table_name: &str) -> String { These identifiers are interpolated into SQL strings via `format!`. This is safe because: -1. `account_id` is validated by `validate_account_id()` — rejects `"`, `\0`, non-ASCII -2. `table_name` is validated by `validate_table_name()` — only `[a-zA-Z0-9_.-]` +1. `account_id` is validated by `validate_account_id()`: rejects `"`, `\0`, non-ASCII +2. `table_name` is validated by `validate_table_name()`: only `[a-zA-Z0-9_.-]` 3. The result is double-quoted, preventing interpretation as SQL keywords 4. No character in the validated set can escape a double-quoted identifier @@ -65,8 +65,8 @@ This is **not** the same as raw string interpolation of user input. The validati ## Audit Trail - 2026-04-07: Initial ADR. All SQL used parameterized queries. -- 2026-04-07: `ListTables.ExclusiveStartTableName` gap closed — `validate_table_name_chars()` added. -- 2026-04-21: ADR rewritten to accurately describe the two-tier design. The original ADR claimed "no dynamic SQL" which was misleading — validated identifier interpolation via `format!` is used extensively for per-table storage tables. The defense is validation + quoting, not absence of interpolation. +- 2026-04-07: `ListTables.ExclusiveStartTableName` gap closed: `validate_table_name_chars()` added. +- 2026-04-21: ADR rewritten to accurately describe the two-tier design. The original ADR claimed "no dynamic SQL" which was misleading: validated identifier interpolation via `format!` is used extensively for per-table storage tables. The defense is validation + quoting, not absence of interpolation. --- @@ -75,6 +75,6 @@ This is **not** the same as raw string interpolation of user input. The validati Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0003-catalog-migration-mechanism.md b/docs/adr/0003-catalog-migration-mechanism.md index 5f2a51e19..35c66247a 100644 --- a/docs/adr/0003-catalog-migration-mechanism.md +++ b/docs/adr/0003-catalog-migration-mechanism.md @@ -153,6 +153,6 @@ be recreated and reloaded after `init`. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0004-vector-search-exact-scan.md b/docs/adr/0004-vector-search-exact-scan.md index cda78693b..6af965667 100644 --- a/docs/adr/0004-vector-search-exact-scan.md +++ b/docs/adr/0004-vector-search-exact-scan.md @@ -31,15 +31,15 @@ than preferences: ## Options Considered -1. **`sqlite-vec`** (asg017) — dual Apache-2.0/MIT, index in in-database shadow +1. **`sqlite-vec`** (asg017): dual Apache-2.0/MIT, index in in-database shadow tables, native partition-key columns that pre-filter. The closest fit. -2. **`sqlite-vss`** (asg017) — Faiss-backed, real ANN. -3. **`sqliteai/sqlite-vector`** — SIMD, works on existing table schemas. -4. **`vectorlite`** — real HNSW via hnswlib. -5. **`usearch`'s SQLite extension** — from the usearch project. -6. **`sqlite-muninn`** — HNSW, loadable extension. -7. **libSQL / Turso native vector search** — LM-DiskANN built into a SQLite fork. -8. **A Rust index crate persisted in the database** — `hnsw_rs`, +2. **`sqlite-vss`** (asg017): Faiss-backed, real ANN. +3. **`sqliteai/sqlite-vector`**: SIMD, works on existing table schemas. +4. **`vectorlite`**: real HNSW via hnswlib. +5. **`usearch`'s SQLite extension**: from the usearch project. +6. **`sqlite-muninn`**: HNSW, loadable extension. +7. **libSQL / Turso native vector search**: LM-DiskANN built into a SQLite fork. +8. **A Rust index crate persisted in the database**: `hnsw_rs`, `instant-distance`, `usearch` as a library, `arroy`, `hannoy`. 9. **An exact scan written in Rust**, with vectors stored as ordinary rows. diff --git a/docs/adr/0005-index-build-lifecycle-ownership.md b/docs/adr/0005-index-build-lifecycle-ownership.md index 0a596812a..c370ee933 100644 --- a/docs/adr/0005-index-build-lifecycle-ownership.md +++ b/docs/adr/0005-index-build-lifecycle-ownership.md @@ -135,6 +135,6 @@ lifecycle, which now means any backend logic that bypasses the shared module. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0006-pgvector-storage-and-scoring.md b/docs/adr/0006-pgvector-storage-and-scoring.md index 9e7fc7cea..f0e78c6b9 100644 --- a/docs/adr/0006-pgvector-storage-and-scoring.md +++ b/docs/adr/0006-pgvector-storage-and-scoring.md @@ -115,6 +115,6 @@ class no real embedding produces. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0010-cassandra-ttl-expiration-queue.md b/docs/adr/0010-cassandra-ttl-expiration-queue.md index 9e35d5c56..9c8f3e906 100644 --- a/docs/adr/0010-cassandra-ttl-expiration-queue.md +++ b/docs/adr/0010-cassandra-ttl-expiration-queue.md @@ -15,15 +15,15 @@ A customer calls `UpdateTimeToLive` naming an attribute. We record the attribute The queue lives in the customer's own keyspace and is two tables. `ttl_expirations` holds one row per item that has a timestamp, partitioned by table, generation, the day the item expires, and one of 64 shards derived from the item's key, then ordered by expiration time. `ttl_expiration_buckets` records which of those partitions exist, so a sweep can find the ones that are due without hunting for them. Partitioning this way is what lets a sweep read the queue with complete partition keys and an ordered range, rather than filtering. -While TTL is on, writes keep the queue in step with the items. A Put or Update that changes an item's timestamp deletes the queue row for the old timestamp and inserts one for the new, and it does that in the same logged batch as the item write itself, so the two land together. A Delete removes the queue row the same way. Because a batch is not a transaction — and because the sweeper's own conditional writes to the queue can supersede or erase the batch's queue mutation — a write whose item carries a timestamp also puts a small key-only row into a durable outbox in that same batch. A background pass reads the outbox, quorum-reads the item and table metadata, restores the bucket registration, and makes sure the queue row matches before removing the outbox record. Before the sweeper conditionally destroys any active-generation queue row, it writes a separate non-dischargeable inflight repair marker at quorum. After a definitive true or false LWT response it writes a normal outbox record at quorum and only then removes the marker. An ambiguous error or crash therefore leaves the marker behind; a dedicated repair worker reconciles every marker while its recorded table generation remains active, so even a destroy that lands arbitrarily late is repaired by a later pass. Quorum metadata proving the table or generation retired makes the marker terminal and safe to remove. Transactional writes reconcile after commit rather than inside the batch, using the image captured before the transaction ran. +While TTL is on, writes keep the queue in step with the items. A Put or Update that changes an item's timestamp deletes the queue row for the old timestamp and inserts one for the new, and it does that in the same logged batch as the item write itself, so the two land together. A Delete removes the queue row the same way. Because a batch is not a transaction, and because the sweeper's own conditional writes to the queue can supersede or erase the batch's queue mutation, a write whose item carries a timestamp also puts a small key-only row into a durable outbox in that same batch. A background pass reads the outbox, quorum-reads the item and table metadata, restores the bucket registration, and makes sure the queue row matches before removing the outbox record. Before the sweeper conditionally destroys any active-generation queue row, it writes a separate non-dischargeable inflight repair marker at quorum. After a definitive true or false LWT response it writes a normal outbox record at quorum and only then removes the marker. An ambiguous error or crash therefore leaves the marker behind; a dedicated repair worker reconciles every marker while its recorded table generation remains active, so even a destroy that lands arbitrarily late is repaired by a later pass. Quorum metadata proving the table or generation retired makes the marker terminal and safe to remove. Transactional writes reconcile after commit rather than inside the batch, using the image captured before the transaction ran. Writes on a TTL-enabled table also take a short-lived claim on the item's own row, by setting a `prepared_txn_id` with a conditional update. That claim is what stops a write and an expiration from stepping on each other. If a writer finds the claim already held it waits briefly and re-reads rather than failing, so two writers to the same key still behave as last-writer-wins. Releasing the claim costs nothing extra on the request: the write's own batch clears the claim columns at its pinned timestamp, and a detached, bounded background task performs the exact conditional release to cover the case where the local clock trails the coordinator's. -A worker on every host wakes once a minute and, per table, tries to take a lease. One host wins, and it asks the bucket table which partitions are due, then reads up to a hundred rows whose expiration time has passed. For each one it re-reads the item. If the item is gone, or its timestamp has moved into the future, or the attribute has been removed, there is nothing to expire — the queue row is retired and, where relevant, replaced with one matching the current item. That is the check that stops a renewed item from being deleted by work queued before it was renewed. +A worker on every host wakes once a minute and, per table, tries to take a lease. One host wins, and it asks the bucket table which partitions are due, then reads up to a hundred rows whose expiration time has passed. For each one it re-reads the item. If the item is gone, or its timestamp has moved into the future, or the attribute has been removed, there is nothing to expire: the queue row is retired and, where relevant, replaced with one matching the current item. That is the check that stops a renewed item from being deleted by work queued before it was renewed. -If the item really is expired, the worker moves the queue row through the remaining states, and the states exist so that a crash at any point can be resumed rather than guessed at. It first claims the row, writing into it the exact image of the item it intends to delete, a fixed delete timestamp, and — if the table has a stream — the event ID and sequence number it will use, decided now rather than at write time. It then takes the claim on the item's row. Immediately before effects, it renews the exact generation-bound sweep lease, seals the exact base owner without TTL, and durably moves the queue row to `EFFECTS_APPLYING`. From that point recovery must go forward: it deletes secondary-index rows and writes the deterministic `REMOVE`, advances to `EFFECTS_APPLIED`, then deletes the item conditional on both the owner and recorded image. Finally it deletes the queue row. +If the item really is expired, the worker moves the queue row through the remaining states, and the states exist so that a crash at any point can be resumed rather than guessed at. It first claims the row, writing into it the exact image of the item it intends to delete, a fixed delete timestamp, and, if the table has a stream, the event ID and sequence number it will use, decided now rather than at write time. It then takes the claim on the item's row. Immediately before effects, it renews the exact generation-bound sweep lease, seals the exact base owner without TTL, and durably moves the queue row to `EFFECTS_APPLYING`. From that point recovery must go forward: it deletes secondary-index rows and writes the deterministic `REMOVE`, advances to `EFFECTS_APPLIED`, then deletes the item conditional on both the owner and recorded image. Finally it deletes the queue row. -The ordering matters in one direction in particular. Once `EFFECTS_APPLYING` is durable, effects may already be partially visible, so the exact base owner is non-expiring and work has to be replayed through completion — otherwise a live item can be left with a missing index or published `REMOVE`. Everything before that boundary can be safely walked away from, because nothing outside the queue has changed yet. Retiring a generation follows the same rule: it drops queue rows that were only pending, releases and abandons rows that were merely claimed, and pushes applying or applied rows through to completion. +The ordering matters in one direction in particular. Once `EFFECTS_APPLYING` is durable, effects may already be partially visible, so the exact base owner is non-expiring and work has to be replayed through completion; otherwise a live item can be left with a missing index or published `REMOVE`. Everything before that boundary can be safely walked away from, because nothing outside the queue has changed yet. Retiring a generation follows the same rule: it drops queue rows that were only pending, releases and abandons rows that were merely claimed, and pushes applying or applied rows through to completion. ## Options considered @@ -52,14 +52,14 @@ Use a generation-fenced, durable TTL-specific state machine. * Maintain `ttl_expiration_buckets` and `ttl_expirations` in every account keyspace. Queue partitions use table ID, TTL-enable generation, day bucket, and one of 64 deterministic key shards. Entries are ordered by expiration time and key. * Accept only positive integral DynamoDB `N` values as epoch seconds. Missing, non-numeric, fractional, zero, and negative values are not queued. * Allocate a fresh generation on every enable. Backfill the active generation synchronously and publish `ttl_index_ready` only after reconciliation. Disable, re-enable, cleanup, and sweeps are fenced by the exact attribute and generation. -* Before enabling TTL on a table that is already accepting writes, quiesce writes that began under the disabled generation until enable and synchronous backfill return, plus the write-path configuration cache lifetime (5 seconds) — hosts other than the one issuing the change may serve the prior configuration for up to that long. Once enabled, all ordinary writes enter the exact-claim logged-batch path. Enabling TTL before opening a new table to traffic avoids this operational step. +* Before enabling TTL on a table that is already accepting writes, quiesce writes that began under the disabled generation until enable and synchronous backfill return, plus the write-path configuration cache lifetime (5 seconds); hosts other than the one issuing the change may serve the prior configuration for up to that long. Once enabled, all ordinary writes enter the exact-claim logged-batch path. Enabling TTL before opening a new table to traffic avoids this operational step. * Record ordinary Put/Update reconciliation in a durable key-only `ttl_reconcile_pending` outbox in the same logged batch as the base mutation, whenever the written item carries a valid timestamp. The worker quorum-reads the committed item and authoritative table identity/configuration, restores the bucket registration at quorum before accepting an existing `PENDING` row, and prefers a plain quorum queue read over Paxos when the entry is already present. A conflict with claimed TTL work remains retryable; the outbox is removed at quorum only after reconciliation succeeds. Symmetrically, every conditional destroy of an active-generation queue row first writes a unique `ttl_repair_inflight` marker at `LOCAL_QUORUM`. After a definitive applied true/false response, the destroyer writes a successor normal outbox record at quorum and only then deletes the inflight marker at quorum. Ambiguous errors and crashes retain the non-dischargeable marker, which the dedicated repair worker repeatedly reconciles while its recorded table generation remains active; a late destroy is therefore repaired on a later pass without relying on wall-clock age or skew. Quorum proof that the table or generation retired makes the marker safe to remove. * Reconcile transaction Put/Update/Delete results against the pre-commit image before deleting the recovery ledger, so a transactional write retires the entry the item had before it as well as registering the new one. COMMITTING recovery repeats reconciliation from the persisted ledger payload, which holds only the committed image (see *Known gaps*). * Represent deletion work as `PENDING`, `CLAIMED`, `CLAIM_ABORTING`, `EFFECTS_APPLYING`, and `EFFECTS_APPLIED`. A claim persists the old item image, a stable delete timestamp, a work UUID, and an optional deterministic stream plan. A stale-item or lifecycle abort first wins `CLAIMED` → `CLAIM_ABORTING` on the queue row, then releases the base owner and retires work; a sweep must instead win `CLAIMED` → `EFFECTS_APPLYING` after sealing the exact owner. The mutually exclusive LWTs prevent either side from clearing the other's fence. * Serialize TTL deletion with ordinary and transactional writes through the base row's `prepared_txn_id` LWT field at `LOCAL_QUORUM`/`LOCAL_SERIAL`. Ordinary request claims and worker claims before the must-complete boundary are time-bounded. Immediately before `EFFECTS_APPLYING`, the worker seals its exact owner without TTL so it cannot lose the fence while effects are suspended; durable queue recovery owns completing and releasing that owner. A request's logged batch is stamped with a timestamp pinned immediately after its claim, so a request that resumes after its claim expired loses to any owner that committed in the meantime rather than overwriting it. * Treat contention for that claim as contention, not as a client error. A blocked writer retries against a freshly read image on a jittered backoff, and only reports `TransactionConflictException` once those retries are exhausted, per RFC-0003 §4.3. Ordinary concurrent writes to one key therefore remain last-writer-wins. * Seal the exact base owner without TTL, transition the queue row to `EFFECTS_APPLYING`, then apply synchronous index deletion and the deterministic stream write. Recovery and retired-generation cleanup treat this as must-complete and reapply effects idempotently. Transition to `EFFECTS_APPLIED`, delete the base row with an LWT requiring the exact work UUID and item image, and finally conditionally complete the queue work. -* Read at `LOCAL_QUORUM` wherever an empty or absent result is treated as authoritative and acted on destructively — retiring queue work, dropping an outbox row, or retiring a bucket registration. The default read consistency is `ONE`, at which a lagging replica reads as absent; acting on that would leave an item with no expiration entry and therefore never expiring. Claim and delete LWTs re-verify the exact image and so tolerate a stale read, but decisions about *absence* have nothing to condition on. +* Read at `LOCAL_QUORUM` wherever an empty or absent result is treated as authoritative and acted on destructively: retiring queue work, dropping an outbox row, or retiring a bucket registration. The default read consistency is `ONE`, at which a lagging replica reads as absent; acting on that would leave an item with no expiration entry and therefore never expiring. Claim and delete LWTs re-verify the exact image and so tolerate a stale read, but decisions about *absence* have nothing to condition on. * Persist stream event ID, sequence number, timestamp, region, and view type. Retrying side effects overwrites the same stream row rather than creating a second externally visible `REMOVE` event. * If a stale queue snapshot differs from the current item, conditionally retire that exact work before reconciling the current image. If an old-timestamp delete leaves no item but the permanent work owner survives, recovery conditionally releases that exact UUID before retiring or completing the queue row. * Use a generation-bound table sweep lease and renew it while processing. TTL lifecycle changes and index creation that could invalidate a live sweep take the same lease, retry briefly when a sweep holds it, and report `ResourceInUseException` only after that. @@ -78,7 +78,7 @@ Each phase is idempotent and is entered only from durable state. `work_id` is th | After sealing the owner, before/during effects | `EFFECTS_APPLYING` + non-expiring exact base owner | Re-apply effects. Index deletes are idempotent and the stream write reuses the persisted event identity, then advance to `EFFECTS_APPLIED`. Lifecycle cleanup follows the same must-complete rule. | | After `EFFECTS_APPLIED`, before the base delete | `EFFECTS_APPLIED` | Complete the base delete under the exact owner and image. This is the one phase that must go forward: index rows are already removed, so abandoning it would leave a live item with a missing index. Generation cleanup honours the same rule. | | After the base delete, before completion | Base row gone, `EFFECTS_APPLIED` | Complete the queue row conditionally on the exact owner. | -| Ordinary request suspended past its claim | Claim expired; the resumed batch may still land | The batch is stamped before the suspension, so its cells lose to any newer owner's Paxos cells — except the item cells, which can beat the older image under a sealed owner. `EFFECTS_APPLYING` recovery is therefore fenced on the owner alone, and a changed image is handled by the post-effects path: index rows the replay tombstoned are rebuilt from the live image before the sealed owner is released. | +| Ordinary request suspended past its claim | Claim expired; the resumed batch may still land | The batch is stamped before the suspension, so its cells lose to any newer owner's Paxos cells, except the item cells, which can beat the older image under a sealed owner. `EFFECTS_APPLYING` recovery is therefore fenced on the owner alone, and a changed image is handled by the post-effects path: index rows the replay tombstoned are rebuilt from the live image before the sealed owner is released. | | TTL disabled mid-flight | `ttl_cleanup_generation` set | Drain the generation per the rule above, then remove its `PENDING` rows; retry until empty. | ## Consequences @@ -103,7 +103,7 @@ Each phase is idempotent and is entered only from durable state. `work_id` is th * LWT claims and lifecycle leases add Cassandra coordination cost to TTL-enabled tables. * A destroy whose result remains ambiguous leaves a `ttl_repair_inflight` marker intentionally. A dedicated worker rechecks active-generation markers and safely removes markers only after quorum metadata proves their table or generation retired; operators must monitor persistent marker count/age and investigate active-generation entries. * Cassandra coordinators and ExtendDB hosts must have synchronized clocks, and data-plane requests must have deadlines well below the request claim lifetime. -* Claims fence on the row's `version` column, which every image write bumps (ordinary writes explicitly, transactional commits via server-side increment); rows whose version is null — they predate explicit versioning and have not been rewritten, including rows only ever written transactionally — fall back to `item_data` string equality. The exact base delete keeps owner AND version AND the full image as its condition: version alone is not a unique image identity (two writers can stamp different images with the same successor version under extreme clock divergence, and it restarts across delete/recreate incarnations), and the one statement that destroys data does not rest on a clock assumption. The canonical-encoding unit test therefore remains load-bearing. +* Claims fence on the row's `version` column, which every image write bumps (ordinary writes explicitly, transactional commits via server-side increment); rows whose version is null (they predate explicit versioning and have not been rewritten, including rows only ever written transactionally) fall back to `item_data` string equality. The exact base delete keeps owner AND version AND the full image as its condition: version alone is not a unique image identity (two writers can stamp different images with the same successor version under extreme clock divergence, and it restarts across delete/recreate incarnations), and the one statement that destroys data does not rest on a clock assumption. The canonical-encoding unit test therefore remains load-bearing. ### Operating envelope @@ -112,18 +112,18 @@ This version is suitable for production when the deployment stays inside the sup * enable TTL before opening a new table to writes, or briefly quiesce an existing table while enable/backfill completes; * use no GSI with a nonzero propagation delay on a TTL-enabled table; * serve writes for a given table from one Cassandra datacenter, because all claims and lifecycle LWTs use `LOCAL_QUORUM`/`LOCAL_SERIAL` and are therefore linearizable only within a datacenter; -* stay under roughly 1,000 expirations per table per minute. The sweep processes its batch concurrently across the queue's independent shard partitions (safe because every row transition is conditional on its exact work id), so Cassandra's batch is sized by throughput need — 1,000 per 60-second cycle — rather than by the serial round-trip time that bounds the other backends at 100. The per-table lease means the work still happens exactly once; a table expiring faster than the batch rate accumulates backlog; +* stay under roughly 1,000 expirations per table per minute. The sweep processes its batch concurrently across the queue's independent shard partitions (safe because every row transition is conditional on its exact work id), so Cassandra's batch is sized by throughput need (1,000 per 60-second cycle) rather than by the serial round-trip time that bounds the other backends at 100. The per-table lease means the work still happens exactly once; a table expiring faster than the batch rate accumulates backlog; * size and monitor the expiration backlog, Cassandra LWT latency, and worker health; * keep Cassandra coordinators and ExtendDB hosts time-synchronized, because request batches use explicit timestamps, with request deadlines well below the request claim lifetime; and * accept DynamoDB-style eventual expiration plus the documented brief internal effects-before-delete window. -Within that envelope, item renewal, crash recovery, generation changes, synchronous index cleanup, transaction reconciliation, ordinary write contention, and deterministic stream removal are covered by durable state and real-Cassandra tests. This is not yet a claim of unrestricted DynamoDB parity for every Cassandra table configuration. +Within that envelope, item renewal, crash recovery, generation changes, synchronous index cleanup, transaction reconciliation, ordinary write contention, and deterministic stream removal are covered by durable state and real-Cassandra tests. This is not a claim of unrestricted DynamoDB parity for every Cassandra table configuration. ### Parity with the PostgreSQL backend The bar for this feature is the production readiness of the PostgreSQL TTL implementation. Most of what looks like a Cassandra limitation is in fact the shared TTL design, and it is worth separating the two so that reviewers and operators know which items are Cassandra's to answer. -**Shared with PostgreSQL — same behaviour, same code path or same shared handler:** +**Shared with PostgreSQL (same behaviour, same code path or same shared handler):** | Behaviour | Evidence | | --- | --- | @@ -146,11 +146,11 @@ The bar for this feature is the production readiness of the PostgreSQL TTL imple | Gap | Cause | Fixable here? | | --- | --- | --- | -| Deletion is a crash-recoverable saga, not one atomic transaction, so a brief internal effects-before-delete window exists | Cassandra cannot combine a Paxos condition on the base row with mutations in other partitions. PostgreSQL does base delete, index cleanup, stream record, and async-index enqueue in one `BEGIN`/`COMMIT`. | No — this is the fundamental blocker. It is what the queue, the claims, and the state machine exist to compensate for. | -| TTL cannot be enabled alongside an asynchronously propagated GSI | The async GSI queue has no version-conditional replay fence, so an old TTL delete could overtake a recreated item's insert. PostgreSQL enqueues async GSI cleanup inside the delete transaction, so it has no such race. | No — needs versioned, conditionally applied GSI mutations. This is the one *functional* restriction relative to PostgreSQL. | -| Enabling TTL on a live table needs writes quiesced | Expiry is derived from a durable queue that must be backfilled, and a write that began before enable cannot join the new generation. PostgreSQL's `CREATE INDEX CONCURRENTLY` covers live writes with no transition window. | No — needs a durable background backfill that admits pre-enable writes. | -| Writes on a TTL-enabled table cost an extra catalog read, one request-path LWT, a detached exact-release LWT, and a logged batch | The claim protocol is on the write path; the exact release is bounded background work. PostgreSQL writes touch nothing TTL-related. | Partly — the claim's marginal value is now small enough that removing it from the ordinary write path is a live proposal. | -| Writes for a table must be served from one datacenter | All claims and lifecycle LWTs use `LOCAL_QUORUM`/`LOCAL_SERIAL`. | No — global serial consistency would cost cross-region Paxos on every write. | +| Deletion is a crash-recoverable saga, not one atomic transaction, so a brief internal effects-before-delete window exists | Cassandra cannot combine a Paxos condition on the base row with mutations in other partitions. PostgreSQL does base delete, index cleanup, stream record, and async-index enqueue in one `BEGIN`/`COMMIT`. | No: this is the fundamental blocker. It is what the queue, the claims, and the state machine exist to compensate for. | +| TTL cannot be enabled alongside an asynchronously propagated GSI | The async GSI queue has no version-conditional replay fence, so an old TTL delete could overtake a recreated item's insert. PostgreSQL enqueues async GSI cleanup inside the delete transaction, so it has no such race. | No: needs versioned, conditionally applied GSI mutations. This is the one *functional* restriction relative to PostgreSQL. | +| Enabling TTL on a live table needs writes quiesced | Expiry is derived from a durable queue that must be backfilled, and a write that began before enable cannot join the new generation. PostgreSQL's `CREATE INDEX CONCURRENTLY` covers live writes with no transition window. | No: needs a durable background backfill that admits pre-enable writes. | +| Writes on a TTL-enabled table cost an extra catalog read, one request-path LWT, a detached exact-release LWT, and a logged batch | The claim protocol is on the write path; the exact release is bounded background work. PostgreSQL writes touch nothing TTL-related. | Partly: the claim's marginal value is now small enough that removing it from the ordinary write path is a live proposal. | +| Writes for a table must be served from one datacenter | All claims and lifecycle LWTs use `LOCAL_QUORUM`/`LOCAL_SERIAL`. | No: global serial consistency would cost cross-region Paxos on every write. | | Claims fence on the OCC `version` column (with `item_data` fallback for unversioned legacy rows) | Every write now bumps `version`, unifying the two concurrency mechanisms Joel flagged as parallel systems. | Done in this tranche. | | Transaction recovery reconciles against the ledger-persisted pre-commit image | PREPARE captures each row's prior image; recovery retires the old queue entry as the live path does. | Done in this tranche. | @@ -163,4 +163,4 @@ Deliberately out of scope for this change, in rough priority order: * **Expiration throughput does not scale horizontally.** The sweep lease is per table even though the queue is already sharded 64 ways by key and those shards are disjoint partitions. Leasing per `(table, shard)` would allow up to 64 concurrent workers per table with no change to the claim protocol, because every queue transition is already conditional on the exact work UUID. This is the highest-value follow-up, and it would take Cassandra past the PostgreSQL rate rather than merely matching it. * **TTL alongside asynchronously propagated GSIs**, per the table above. * **Version fence hardening.** The fence now uses the `version` column; remaining follow-up is retiring the `item_data` fallback once no unversioned rows can exist (requires a fleet-wide rewrite pass or an explicit migration), after which the canonical-JSON coupling disappears entirely. -* **Recovery-path test coverage.** The transitions this change introduced — draining a retired generation, retiring a drained bucket registration, an expired worker claim — are reasoned about but not yet covered by tests. +* **Recovery-path test coverage.** The transitions this change introduced (draining a retired generation, retiring a drained bucket registration, an expired worker claim) are reasoned about but not yet covered by tests. diff --git a/docs/adr/0011-cassandra-foreign-key-emulation.md b/docs/adr/0011-cassandra-foreign-key-emulation.md index f9806e240..92923ca3d 100644 --- a/docs/adr/0011-cassandra-foreign-key-emulation.md +++ b/docs/adr/0011-cassandra-foreign-key-emulation.md @@ -12,11 +12,11 @@ Apache Cassandra does not support foreign key constraints. The database will acc ## Options Considered -1. **Application-layer pre-checks** — Before inserting a child entity, execute a SELECT query to verify the parent entity exists. Return `OpError::NotFound` if the parent is missing, matching PostgreSQL FK violation behavior. +1. **Application-layer pre-checks**: Before inserting a child entity, execute a SELECT query to verify the parent entity exists. Return `OpError::NotFound` if the parent is missing, matching PostgreSQL FK violation behavior. -2. **Eventual consistency checks** — Allow writes to proceed without validation, then run periodic background jobs to detect and repair orphaned records. +2. **Eventual consistency checks**: Allow writes to proceed without validation, then run periodic background jobs to detect and repair orphaned records. -3. **Denormalization** — Embed parent entity data in child records to eliminate the need for referential integrity checks. +3. **Denormalization**: Embed parent entity data in child records to eliminate the need for referential integrity checks. ## Decision @@ -48,6 +48,6 @@ Application-layer pre-checks for all child entity creation operations. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0012-cassandra-transaction-atomicity-patterns.md b/docs/adr/0012-cassandra-transaction-atomicity-patterns.md index e4f9e6388..0a4bebe51 100644 --- a/docs/adr/0012-cassandra-transaction-atomicity-patterns.md +++ b/docs/adr/0012-cassandra-transaction-atomicity-patterns.md @@ -12,13 +12,13 @@ Cassandra does not support multi-statement transactions. Logged batches provide ## Options Considered -1. **Check-then-batch pattern** — Execute a SELECT to check for duplicates, then use a logged BATCH for the atomic writes. Accept a small race condition window between check and write. +1. **Check-then-batch pattern**: Execute a SELECT to check for duplicates, then use a logged BATCH for the atomic writes. Accept a small race condition window between check and write. -2. **Sequential LWTs** — Execute each insert with `IF NOT EXISTS` sequentially, rolling back manually on failure. Not truly atomic. +2. **Sequential LWTs**: Execute each insert with `IF NOT EXISTS` sequentially, rolling back manually on failure. Not truly atomic. -3. **Compensation pattern** — Allow writes to proceed without duplicate checks, then clean up on failure. Leaves temporary inconsistent state. +3. **Compensation pattern**: Allow writes to proceed without duplicate checks, then clean up on failure. Leaves temporary inconsistent state. -4. **Single LWT** — Combine both operations into a single denormalized record. Breaks schema compatibility with PostgreSQL. +4. **Single LWT**: Combine both operations into a single denormalized record. Breaks schema compatibility with PostgreSQL. ## Decision @@ -51,6 +51,6 @@ Check-then-batch pattern for operations requiring both duplicate detection and a Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0013-cassandra-keyspace-awareness.md b/docs/adr/0013-cassandra-keyspace-awareness.md index 05d867e03..2bcc7bb27 100644 --- a/docs/adr/0013-cassandra-keyspace-awareness.md +++ b/docs/adr/0013-cassandra-keyspace-awareness.md @@ -16,11 +16,11 @@ Early implementations hardcoded keyspace names like `extenddb_catalog` directly ## Options Considered -1. **Dynamic keyspace construction** — Store `keyspace_prefix` in config, construct keyspace names dynamically via helper methods like `catalog_keyspace()`. Pass keyspace prefix through initialization chain. +1. **Dynamic keyspace construction**: Store `keyspace_prefix` in config, construct keyspace names dynamically via helper methods like `catalog_keyspace()`. Pass keyspace prefix through initialization chain. -2. **USE statement per connection** — Execute `USE extenddb_catalog` on each session, then write queries without keyspace prefix. Mimics PostgreSQL behavior. +2. **USE statement per connection**: Execute `USE extenddb_catalog` on each session, then write queries without keyspace prefix. Mimics PostgreSQL behavior. -3. **Query-time keyspace override** — Allow each query to specify keyspace as a parameter, defaulting to config value. +3. **Query-time keyspace override**: Allow each query to specify keyspace as a parameter, defaulting to config value. ## Decision @@ -59,6 +59,6 @@ Dynamic keyspace construction with JDBC-style connection strings. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0014-cassandra-upsert-semantics.md b/docs/adr/0014-cassandra-upsert-semantics.md index 7e5b00332..e655afb8a 100644 --- a/docs/adr/0014-cassandra-upsert-semantics.md +++ b/docs/adr/0014-cassandra-upsert-semantics.md @@ -14,11 +14,11 @@ Operations like `put_policy` must be idempotent: Calling the operation multiple ## Options Considered -1. **Natural INSERT behavior** — Use plain `INSERT` statements and rely on Cassandra's built-in overwrite semantics. Simpler code, same idempotent result. +1. **Natural INSERT behavior**: Use plain `INSERT` statements and rely on Cassandra's built-in overwrite semantics. Simpler code, same idempotent result. -2. **Explicit UPDATE with conditional INSERT** — Check if row exists via SELECT, then execute UPDATE or INSERT accordingly. Mimics PostgreSQL logic exactly but adds unnecessary complexity. +2. **Explicit UPDATE with conditional INSERT**: Check if row exists via SELECT, then execute UPDATE or INSERT accordingly. Mimics PostgreSQL logic exactly but adds unnecessary complexity. -3. **Lightweight transaction with IF EXISTS** — Use `UPDATE ... IF EXISTS` to make the upsert explicit. Adds LWT overhead for no behavioral benefit. +3. **Lightweight transaction with IF EXISTS**: Use `UPDATE ... IF EXISTS` to make the upsert explicit. Adds LWT overhead for no behavioral benefit. ## Decision @@ -54,6 +54,6 @@ Use natural INSERT behavior for operations requiring UPSERT semantics. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0015-cassandra-account-keyspace-provisioning.md b/docs/adr/0015-cassandra-account-keyspace-provisioning.md index 4720e27ef..afac6cc03 100644 --- a/docs/adr/0015-cassandra-account-keyspace-provisioning.md +++ b/docs/adr/0015-cassandra-account-keyspace-provisioning.md @@ -11,18 +11,18 @@ The Cassandra backend uses a keyspace-per-account isolation model. Each ExtendDB The PostgreSQL backend uses a single database with all accounts sharing the same schema. Account creation in PostgreSQL is a simple catalog INSERT with no storage initialization. This raises the question: when should Cassandra create the account-specific keyspace? Two natural timing points exist: -1. **During account creation** — Create keyspace in `ManagementStore::create_account()` -2. **On first table creation** — Lazy keyspace creation in `TableEngine::create_table()` +1. **During account creation**: Create keyspace in `ManagementStore::create_account()` +2. **On first table creation**: Lazy keyspace creation in `TableEngine::create_table()` Existing code in `create_table_impl` validates that the account keyspace exists and returns an error if it doesn't, requiring an explicit choice. ## Options Considered -1. **Keyspace creation during account creation** — `ManagementStore::create_account_impl()` creates both the catalog entry and the account keyspace in a single operation. +1. **Keyspace creation during account creation**: `ManagementStore::create_account_impl()` creates both the catalog entry and the account keyspace in a single operation. -2. **Lazy keyspace creation on first table** — `TableEngine::create_table_impl()` checks if the account keyspace exists and creates it if missing (idempotent). +2. **Lazy keyspace creation on first table**: `TableEngine::create_table_impl()` checks if the account keyspace exists and creates it if missing (idempotent). -3. **Manual provisioning** — Require operators to explicitly create account keyspaces via separate API or CLI command before creating tables. +3. **Manual provisioning**: Require operators to explicitly create account keyspaces via separate API or CLI command before creating tables. ## Decision @@ -65,6 +65,6 @@ Create account keyspace during account creation (`ManagementStore::create_accoun Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0016-cassandra-index-table-primary-key-structure.md b/docs/adr/0016-cassandra-index-table-primary-key-structure.md index 40f94f024..4067be4f1 100644 --- a/docs/adr/0016-cassandra-index-table-primary-key-structure.md +++ b/docs/adr/0016-cassandra-index-table-primary-key-structure.md @@ -9,24 +9,24 @@ Global Secondary Indexes (GSI) and Local Secondary Indexes (LSI) require separate physical data tables to store indexed items with alternate keys. The PostgreSQL backend uses a PRIMARY KEY constraint of `(pk, base_pk, base_sk_*)` where all columns are part of a composite uniqueness constraint. Cassandra's PRIMARY KEY serves a fundamentally different purpose: it defines both the partition key (for data distribution) and clustering keys (for ordering within a partition). A direct translation of the PostgreSQL schema results in incorrect data distribution and query patterns. The index table stores: -- `pk` — index partition key (computed from index key attributes) -- `sk_*` — index sort keys (e.g., `sk_s`, `sk_n`, `sk_b` for different types) -- `base_pk` — base table partition key (for uniqueness) -- `base_sk_*` — base table sort keys (for uniqueness) -- `item_data` — projected item attributes (JSON) +- `pk`: index partition key (computed from index key attributes) +- `sk_*`: index sort keys (e.g., `sk_s`, `sk_n`, `sk_b` for different types) +- `base_pk`: base table partition key (for uniqueness) +- `base_sk_*`: base table sort keys (for uniqueness) +- `item_data`: projected item attributes (JSON) DynamoDB semantics require that: 1. Queries against an index use the index keys (not base table keys) -2. Index keys are not unique — multiple base items can have the same index keys +2. Index keys are not unique: multiple base items can have the same index keys 3. Each base item appears at most once in the index ## Options Considered -1. **PostgreSQL-style ordering: `PRIMARY KEY (pk, base_pk, base_sk_*)`** — Partition by index PK, cluster by base keys only, with index sort keys as regular columns. +1. **PostgreSQL-style ordering: `PRIMARY KEY (pk, base_pk, base_sk_*)`**: Partition by index PK, cluster by base keys only, with index sort keys as regular columns. -2. **Cassandra-optimized ordering: `PRIMARY KEY ((pk), sk_*, base_pk, base_sk_*)`** — Partition by index PK, cluster first by index sort keys, then by base keys. +2. **Cassandra-optimized ordering: `PRIMARY KEY ((pk), sk_*, base_pk, base_sk_*)`**: Partition by index PK, cluster first by index sort keys, then by base keys. -3. **Composite partition key: `PRIMARY KEY ((pk, base_pk), sk_*, base_sk_*)`** — Include base PK in partition key for uniqueness. +3. **Composite partition key: `PRIMARY KEY ((pk, base_pk), sk_*, base_sk_*)`**: Include base PK in partition key for uniqueness. ## Decision @@ -67,6 +67,6 @@ Cassandra-optimized ordering: `PRIMARY KEY ((pk), sk_*, base_pk, base_sk_*)` Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0017-cassandra-transaction-implementation.md b/docs/adr/0017-cassandra-transaction-implementation.md index fe9026117..18839a027 100644 --- a/docs/adr/0017-cassandra-transaction-implementation.md +++ b/docs/adr/0017-cassandra-transaction-implementation.md @@ -110,7 +110,7 @@ CREATE INDEX ON transaction_ledger (started_at); ```cql SELECT partition_max_delete_timestamp FROM items WHERE pk = ? LIMIT 1; ``` - If `partition_max_delete_timestamp` exists and `txn_timestamp <= partition_max_delete_timestamp`, reject transaction (would violate timestamp ordering—item was deleted at a later timestamp) + If `partition_max_delete_timestamp` exists and `txn_timestamp <= partition_max_delete_timestamp`, reject transaction (would violate timestamp ordering: item was deleted at a later timestamp) 6. If all conditions pass, **execute PREPARE for each item**: - If item exists: `UPDATE items SET prepared_txn_id = ?, prepared_txn_timestamp = ? WHERE pk = ? AND sk = ? IF prepared_txn_id IS NULL` - If item doesn't exist (for Put): `INSERT INTO items (pk, sk, prepared_txn_id, prepared_txn_timestamp, created_to_prepare) VALUES (?, ?, ?, ?, true) IF NOT EXISTS` @@ -427,7 +427,7 @@ Implement transaction coordination logic (two-phase commit, ledger, recovery) in **Rejected**: While appealing for code reuse, this approach has fundamental problems: -1. **Different databases need different mechanisms**: PostgreSQL uses native ACID transactions with `SELECT ... FOR UPDATE` (no ledger, no two-phase commit, no recovery needed). Forcing it to use a ledger would be strictly worse—fighting the database instead of leveraging its strengths. +1. **Different databases need different mechanisms**: PostgreSQL uses native ACID transactions with `SELECT ... FOR UPDATE` (no ledger, no two-phase commit, no recovery needed). Forcing it to use a ledger would be strictly worse, fighting the database instead of leveraging its strengths. 2. **Primitives don't compose uniformly**: The Cassandra implementation uses LWT for conflict detection and STATIC columns for partition-level metadata. Many databases lack these features. Abstracting over such differences creates a lowest-common-denominator that makes all backends worse. @@ -468,6 +468,6 @@ Estimated effort: **3-4 weeks** for complete implementation and testing. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0018-cassandra-stream-implementation.md b/docs/adr/0018-cassandra-stream-implementation.md index bccf28555..00e521617 100644 --- a/docs/adr/0018-cassandra-stream-implementation.md +++ b/docs/adr/0018-cassandra-stream-implementation.md @@ -190,7 +190,7 @@ fn assign_shard(partition_key: &str, table_id: &str) -> String { - CRC32 provides stable, deterministic hashing - Same partition key always routes to same shard (consistent) - Good distribution across shards -- Uses `table_id` (UUID), not `table_name` — prevents shard ID collision if a table is deleted and recreated with the same name (criterion 8.2) +- Uses `table_id` (UUID), not `table_name`: prevents shard ID collision if a table is deleted and recreated with the same name (criterion 8.2) - Matches PostgreSQL implementation ### Atomic Stream + Data Writes @@ -283,7 +283,7 @@ For `StreamViewType` of `OLD_IMAGE` or `NEW_AND_OLD_IMAGES`: ### OLD_IMAGE Capture -For `StreamViewType` of `OLD_IMAGE` or `NEW_AND_OLD_IMAGES`, the pre-mutation item state is required. This is already guaranteed: index maintenance requires a pre-image read on every write path that can affect indexed attributes. Stream record capture piggybacks on this existing read — no additional round-trip is needed in the common case. +For `StreamViewType` of `OLD_IMAGE` or `NEW_AND_OLD_IMAGES`, the pre-mutation item state is required. This is already guaranteed: index maintenance requires a pre-image read on every write path that can affect indexed attributes. Stream record capture piggybacks on this existing read; no additional round-trip is needed in the common case. ### Stream APIs @@ -451,13 +451,13 @@ This refactoring should be done **as part of the Cassandra streams implementatio ## Resolved Questions -1. **Node ID assignment strategy:** Derived as `crc32(format!("{}:{}", hostname, port)) % 9999 + 1`. Hostname alone is insufficient — multiple instances on the same host listening on different ports must have distinct node IDs. Hostname + port is stable across restarts, requires no configuration, and works correctly in containerized environments where the hostname is a pod/container ID and the port is the service binding. Hash collision (two instances mapping to the same node ID) is not a correctness problem — it only creates a theoretical ordering ambiguity that requires matching timestamp + counter on the same shard simultaneously. +1. **Node ID assignment strategy:** Derived as `crc32(format!("{}:{}", hostname, port)) % 9999 + 1`. Hostname alone is insufficient: multiple instances on the same host listening on different ports must have distinct node IDs. Hostname + port is stable across restarts, requires no configuration, and works correctly in containerized environments where the hostname is a pod/container ID and the port is the service binding. Hash collision (two instances mapping to the same node ID) is not a correctness problem; it only creates a theoretical ordering ambiguity that requires matching timestamp + counter on the same shard simultaneously. 2. **HLC state persistence:** Not needed. On restart, `now_ms` will always be greater than any previously generated timestamp (restarts take at least 1ms). Clock-going-backwards is handled by clamping to `last_timestamp_ms` and incrementing the counter, with a WARN log. -3. **TTL configuration:** Global setting (`stream_retention_hours`, default 30). Per-stream configuration adds complexity with no practical benefit — DynamoDB itself uses a fixed 24-hour retention. +3. **TTL configuration:** Global setting (`stream_retention_hours`, default 30). Per-stream configuration adds complexity with no practical benefit; DynamoDB itself uses a fixed 24-hour retention. -4. **Clock skew handling:** Log a WARN if `now_ms < last_timestamp_ms`. Clamp to `last_timestamp_ms` and increment counter. No hard error — the sequence remains valid and ordered. +4. **Clock skew handling:** Log a WARN if `now_ms < last_timestamp_ms`. Clamp to `last_timestamp_ms` and increment counter. No hard error: the sequence remains valid and ordered. ## References diff --git a/docs/adr/0020-cassandra-occ-update-item.md b/docs/adr/0020-cassandra-occ-update-item.md index fd03053b1..ae63a4eb7 100644 --- a/docs/adr/0020-cassandra-occ-update-item.md +++ b/docs/adr/0020-cassandra-occ-update-item.md @@ -6,15 +6,15 @@ ## Context -`update_item_impl` and the conditional path of `put_item_impl` use a read-check-write pattern: read `item_data`, mutate in Rust, write back. This is not atomic. Under concurrent writes to the same item, the last writer wins and intermediate updates are silently lost. This affects all `UpdateItem` calls — not just conditional ones. The root cause is the non-atomic read-modify-write cycle itself. +`update_item_impl` and the conditional path of `put_item_impl` use a read-check-write pattern: read `item_data`, mutate in Rust, write back. This is not atomic. Under concurrent writes to the same item, the last writer wins and intermediate updates are silently lost. This affects all `UpdateItem` calls, not just conditional ones. The root cause is the non-atomic read-modify-write cycle itself. ## Options Considered -1. **Version-based OCC via LWT** — Add a `version bigint` column to each data table. Every successful write increments it. The read-modify-write cycle is made atomic by conditioning the write on `version = AND prepared_txn_id = NULL`. A `[applied] = false` response means another writer won the race; retry from the read. +1. **Version-based OCC via LWT**: Add a `version bigint` column to each data table. Every successful write increments it. The read-modify-write cycle is made atomic by conditioning the write on `version = AND prepared_txn_id = NULL`. A `[applied] = false` response means another writer won the race; retry from the read. -2. **Serialized writes via a per-item lock service** — Acquire an external distributed lock per item key before each write. Rejected: adds infrastructure dependency and defeats Cassandra's distributed architecture. +2. **Serialized writes via a per-item lock service**: Acquire an external distributed lock per item key before each write. Rejected: adds infrastructure dependency and defeats Cassandra's distributed architecture. -3. **Accept last-writer-wins** — Document the behavior and leave it to callers to use conditional expressions. Rejected: DynamoDB's `UpdateItem` is defined to be atomic; silent data loss is not acceptable. +3. **Accept last-writer-wins**: Document the behavior and leave it to callers to use conditional expressions. Rejected: DynamoDB's `UpdateItem` is defined to be atomic; silent data loss is not acceptable. ## Decision @@ -32,9 +32,9 @@ Version-based OCC via LWT (option 1). - **Max retries:** 20 - **Base delay:** 2 ms -- **Backoff:** full jitter — `sleep = random(0, base * 2^min(attempt, 3))`, capped at a 16 ms window to bound tail latency +- **Backoff:** full jitter (`sleep = random(0, base * 2^min(attempt, 3))`), capped at a 16 ms window to bound tail latency - **Retry on:** `[applied] = false` (clean lost race) only -- **Do not retry on:** Cassandra errors (timeout, unavailable) — propagate immediately +- **Do not retry on:** Cassandra errors (timeout, unavailable), which propagate immediately ## Consequences @@ -50,6 +50,6 @@ Version-based OCC via LWT (option 1). Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0021-cassandra-lwt-delete-atomicity-gap.md b/docs/adr/0021-cassandra-lwt-delete-atomicity-gap.md index e8788c564..16f1db2b2 100644 --- a/docs/adr/0021-cassandra-lwt-delete-atomicity-gap.md +++ b/docs/adr/0021-cassandra-lwt-delete-atomicity-gap.md @@ -6,8 +6,8 @@ ## Context -Cassandra's LWT (Lightweight Transaction) mechanism — `INSERT ... IF NOT EXISTS`, -`UPDATE ... IF ...`, `DELETE ... IF EXISTS` — uses Paxos under the hood. A +Cassandra's LWT (Lightweight Transaction) mechanism, meaning `INSERT ... IF NOT EXISTS`, +`UPDATE ... IF ...` and `DELETE ... IF EXISTS`, uses Paxos under the hood. A fundamental Cassandra constraint is that **LWT statements cannot appear inside a logged batch**. Attempting to do so is rejected by the coordinator at runtime. @@ -25,10 +25,10 @@ must therefore use `DELETE ... IF EXISTS`, which cannot be batched. ### Concrete examples -- `delete_user` — must cascade to `iam_users`, `iam_user_tags`, `access_keys`, +- `delete_user`: must cascade to `iam_users`, `iam_user_tags`, `access_keys`, `iam_policies`, and `iam_group_members`. Each table uses LWT for creation. -- `delete_group` — must cascade to `iam_groups` and `iam_group_members`. -- `delete_account` — must cascade to `accounts`, `backups_by_arn`, +- `delete_group`: must cascade to `iam_groups` and `iam_group_members`. +- `delete_account`: must cascade to `accounts`, `backups_by_arn`, `backups_by_table`, `backups_by_account`, and `continuous_backups`. ### Current state (as of this ADR) @@ -84,9 +84,9 @@ backend **must** either: - Implement the worker/saga pattern described above. Adding a plain multi-row delete against LWT-managed rows is not an acceptable -shortcut — it causes `system.paxos` corruption that accumulates across +shortcut; it causes `system.paxos` corruption that accumulates across operations and eventually stalls the entire node (observed: 29-minute test -hangs, GC pauses of 4–11 seconds, `system.paxos` growing without bound). +hangs, GC pauses of 4-11 seconds, `system.paxos` growing without bound). ## Consequences @@ -113,6 +113,6 @@ hangs, GC pauses of 4–11 seconds, `system.paxos` growing without bound). Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/0022-trust-policy-deny-over-allow.md b/docs/adr/0022-trust-policy-deny-over-allow.md index 1489de89c..40eebbd99 100644 --- a/docs/adr/0022-trust-policy-deny-over-allow.md +++ b/docs/adr/0022-trust-policy-deny-over-allow.md @@ -169,6 +169,6 @@ implicitly denied. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/adr/README.md b/docs/adr/README.md index e342f5c68..32be1f9f0 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -30,7 +30,7 @@ decision, write a new ADR. | # | Title | Status | |---|-------|--------| -| [0001](0001-documentation-format.md) | Documentation format — Markdown over LaTeX | Accepted | +| [0001](0001-documentation-format.md) | Documentation format: Markdown over LaTeX | Accepted | | [0002](0002-sql-injection-defense.md) | SQL injection defense | Accepted | | [0003](0003-catalog-migration-mechanism.md) | Adopt sqlx::migrate for PostgreSQL catalog and data schema migrations | Accepted | | [0004](0004-vector-search-exact-scan.md) | Vector search is an exact scan over one row per vector | Accepted | diff --git a/docs/backlog.md b/docs/backlog.md index 7449a1e3e..247bca238 100755 --- a/docs/backlog.md +++ b/docs/backlog.md @@ -103,6 +103,6 @@ Refreshed: v0.0.118 (P115) Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/build-docs.py b/docs/build-docs.py index af6f27722..bb9498923 100755 --- a/docs/build-docs.py +++ b/docs/build-docs.py @@ -251,9 +251,9 @@ def substitute_placeholders(text: str, variables: dict[str, str]) -> str:

License & Disclaimer

Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See the LICENSE file in the project root for the full text.

-

This software is provided “as is” without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. “DynamoDB” is a trademark -of Amazon.com, Inc.

+

This software is provided “as is” without warranty of any kind. ExtendDB is an open source +project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB +and contains no DynamoDB source code. “DynamoDB” is a trademark of Amazon.com, Inc.

""" diff --git a/docs/cassandra-setup.md b/docs/cassandra-setup.md index cedf13518..1d664fb88 100644 --- a/docs/cassandra-setup.md +++ b/docs/cassandra-setup.md @@ -25,7 +25,7 @@ These steps start a single-node Cassandra 4.1 container for local development. 1. Start the container: ```bash - docker run -d --name cassandra -p 9042:9042 \ + docker run -d --name cassandra -p 127.0.0.1:9042:9042 \ -e CASSANDRA_CLUSTER_NAME=extenddb-dev \ cassandra:4.1 ``` @@ -122,13 +122,14 @@ application role with a password, so it fails against an unauthenticated init creates: - The catalog keyspace (`extenddb_catalog` by default) +- The default account's keyspace (`_account_`) - The application role ExtendDB uses at serve time (named `extenddb` by default; override with `--extenddb-user` / `--extenddb-pass`) - Admin user credentials (printed to stdout once; save them) - A self-signed TLS certificate at `~/.extenddb/tls/cert.pem` - The config file `extenddb.toml` -Per-account keyspaces are created later, when accounts are added. +Further accounts get their own keyspace when they are created. > **Save the admin credentials.** They are printed once and cannot be retrieved > later. @@ -224,7 +225,7 @@ topologies, containers, the production checklist, systemd, multi-instance limits), see the [deployment guide](manuals/11-deployment-guide.md). This section covers only what is specific to Cassandra. -> **Not recommended for production yet.** As of October 2026 the Cassandra +> **Not recommended for production.** As of October 2026 the Cassandra > backend is experimental. It has known limitations: some multi-entity > management operations are not atomic, and a few concurrency issues are still > open (see Known Limitations below). Treat a Cassandra deployment as @@ -292,9 +293,12 @@ backend is experimental rather than production-ready. operations use `LOCAL_QUORUM` / `LOCAL_SERIAL`, which are linearizable only within one datacenter. Serve writes for a given table from one Cassandra datacenter. -- **Multi-node ExtendDB needs node-id configuration.** Running more than one - ExtendDB instance against the cluster requires a distinct node ID per - instance (used to derive stream sequence numbers). +- **One instance per catalog.** Running more than one ExtendDB instance against + one catalog is not supported; see the deployment guide's + [Multi-Instance Considerations](manuals/11-deployment-guide.md#multi-instance-considerations). + Stream sequence numbers carry a node component derived from each instance's + bind address and port, so two instances with the same bind address and port + on different hosts would derive the same one. - **TTL throughput.** The TTL sweeper is sized for roughly 1,000 expirations per table per minute; a table expiring faster accumulates backlog. Enable TTL before opening a table to writes, or quiesce it briefly while enable and @@ -315,6 +319,6 @@ rationale are documented in the Cassandra ADRs (`docs/adr/0010` to `0021`). Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/01-requirements.md b/docs/design/01-requirements.md index ef02eee99..eecf7a02a 100755 --- a/docs/design/01-requirements.md +++ b/docs/design/01-requirements.md @@ -1,39 +1,42 @@ --- description: "What ExtendDB must do: the DynamoDB API surface, SDK compatibility, storage independence, and the non-goals." +version: "1.0" +date: "2026-04-03" +status: "Draft" --- # Requirements -**Version:** 1.0 -**Date:** 2026-04-03 -**Status:** Draft - ## 1. Overview -extenddb (ExtendDB) is a standalone, stateless Rust application that provides a 100% DynamoDB-compatible API backed by pluggable storage engines. Any AWS SDK client (Java, Python, Go, Node.js, Rust, .NET, etc.) connects directly with zero code changes. The system runs as its own process, completely independent of any specific database, and supports pluggable authentication providers and pluggable storage backends. +ExtendDB is a standalone, stateless Rust application that provides a DynamoDB-compatible API backed by pluggable storage engines (the [Feature Status](../differences-from-dynamodb.md#feature-status) table on the Differences from DynamoDB page records what is implemented). Any AWS SDK client (Java, Python, Go, Node.js, Rust, .NET, etc.) connects directly with zero code changes. The system runs as its own process, completely independent of any specific database, and supports pluggable authentication providers and pluggable storage backends. ### 1.1 Goals -- **Full DynamoDB API compatibility** — byte-for-byte compatible wire protocol, error responses, and SDK behavior -- **Pluggable storage** — clean trait-based abstraction allowing any database as a backend (PostgreSQL first) -- **Pluggable authentication** — support local SigV4, AWS IAM, Azure AD, and custom providers -- **Standalone & stateless** — single async Rust binary, no embedded database, horizontally scalable -- **Production-ready** — TLS, observability, graceful shutdown, configurable limits, VM and Kubernetes deployment +- **Wire compatibility**: the same wire protocol, error responses and SDK behavior for every implemented operation +- **Pluggable storage**: clean trait-based abstraction allowing any database as a backend (PostgreSQL first) +- **Pluggable authentication**: support local SigV4 and AWS IAM, with the provider interface open to Azure AD and custom providers +- **Standalone & stateless**: single async Rust binary, no embedded database, horizontally scalable +- **Production-ready**: TLS, observability, graceful shutdown, configurable limits, VM and Kubernetes deployment ### 1.2 Non-Goals / Out of Scope +This table records the scope decided when the document was written (April 2026). Where the +implementation has since moved, the [Feature Status](../differences-from-dynamodb.md#feature-status) +table on the Differences from DynamoDB page is the current record. + | Feature | Reason | |---------|--------| -| PartiQL (ExecuteStatement, BatchExecuteStatement, ExecuteTransaction) | Deferred — focus on JSON wire protocol first | -| Backup/Restore (CreateBackup, DeleteBackup, DescribeBackup, ListBackups, RestoreTableFromBackup, DescribeContinuousBackups, UpdateContinuousBackups, RestoreTableToPointInTime) | Excluded — backup is a storage-layer concern, not an API concern | -| Auto-scaling APIs (DescribeTableReplicaAutoScaling, UpdateTableReplicaAutoScaling) | Excluded — auto-scaling is an infrastructure concern | -| Contributor Insights (DescribeContributorInsights, ListContributorInsights, UpdateContributorInsights) | Excluded — analytics feature, not core API | -| Kinesis Streaming Destination (DescribeKinesisStreamingDestination, EnableKinesisStreamingDestination, DisableKinesisStreamingDestination, UpdateKinesisStreamingDestination) | Excluded — AWS-specific integration | -| Resource Policies (GetResourcePolicy, PutResourcePolicy, DeleteResourcePolicy) | Deferred — resource-based policies are a future enhancement | -| IAM Policy Variables (`${aws:PrincipalTag/key}`, `${aws:username}`, etc.) | Deferred — variable substitution in Resource ARNs and Condition values is a future enhancement (see REQ-ABAC-006) | -| Federated Role Assumption (AssumeRoleWithSAML, AssumeRoleWithWebIdentity) | Deferred — v1 supports basic AssumeRole only (see REQ-IDENT-008) | -| Role Chaining (role assumes role) | Deferred — v1 requires CallerArn to be a user; SourceIdentity and TransitiveTagKeys are deferred alongside (see REQ-IDENT-009) | -| Role MaxSessionDuration | Deferred — v1 accepts any DurationSeconds with no per-role cap (see REQ-IDENT-010) | +| PartiQL (ExecuteStatement, BatchExecuteStatement, ExecuteTransaction) | Deferred: focus on JSON wire protocol first | +| Backup/Restore (CreateBackup, DeleteBackup, DescribeBackup, ListBackups, RestoreTableFromBackup, DescribeContinuousBackups, UpdateContinuousBackups, RestoreTableToPointInTime) | Implemented since this document was written: on-demand backup and restore ship, and point-in-time restore is refused (see Feature status) | +| Auto-scaling APIs (DescribeTableReplicaAutoScaling, UpdateTableReplicaAutoScaling) | Excluded: auto-scaling is an infrastructure concern | +| Contributor Insights (DescribeContributorInsights, ListContributorInsights, UpdateContributorInsights) | Excluded: analytics feature, not core API | +| Kinesis Streaming Destination (DescribeKinesisStreamingDestination, EnableKinesisStreamingDestination, DisableKinesisStreamingDestination, UpdateKinesisStreamingDestination) | Excluded: AWS-specific integration | +| Resource Policies (GetResourcePolicy, PutResourcePolicy, DeleteResourcePolicy) | Deferred: resource-based policies are a future enhancement | +| IAM Policy Variables (`${aws:PrincipalTag/key}`, `${aws:username}`, etc.) | Partly implemented since this document was written: variables are expanded in Condition values; substitution in Resource ARNs is not implemented (see REQ-ABAC-006) | +| Federated Role Assumption (AssumeRoleWithSAML, AssumeRoleWithWebIdentity) | Deferred: v1 supports basic AssumeRole only (see REQ-IDENT-008) | +| Role Chaining (role assumes role) | Deferred: v1 requires CallerArn to be a user; SourceIdentity and TransitiveTagKeys are deferred alongside (see REQ-IDENT-009) | +| Role MaxSessionDuration | Deferred: v1 accepts any DurationSeconds with no per-role cap (see REQ-IDENT-010) | ## 2. Wire Protocol Compatibility @@ -134,7 +137,7 @@ Error responses must be byte-for-byte compatible with DynamoDB: | `UnrecognizedClientException` | 403 | Invalid access key or signature mismatch | | `ValidationException` | 400 | Input validation failure (bad table name, missing required field, etc.) | -## 3. Functional Requirements — Operations +## 3. Functional Requirements: Operations ### 3.1 Data Plane Operations (In Scope) @@ -164,8 +167,8 @@ These are the core CRUD operations. All must be fully implemented. - REQ-DATA-008: Scan response size must not exceed 1 MB; return `LastEvaluatedKey` for pagination - REQ-DATA-009: BatchGetItem must support up to 100 items, 16 MB total response; return `UnprocessedKeys` for partial failures - REQ-DATA-010: BatchWriteItem must support up to 25 items (put or delete); return `UnprocessedItems` for partial failures -- REQ-DATA-011: TransactWriteItems must be atomic — all succeed or all fail; support `ClientRequestToken` for idempotency -- REQ-DATA-012: TransactGetItems must be atomic — consistent snapshot of up to 100 items +- REQ-DATA-011: TransactWriteItems must be atomic: all succeed or all fail; support `ClientRequestToken` for idempotency +- REQ-DATA-012: TransactGetItems must be atomic: consistent snapshot of up to 100 items - REQ-DATA-013: All data operations must support legacy `Expected` and `ConditionalOperator` parameters (converted internally to expressions) - REQ-DATA-014: All write operations must trigger stream event capture when streams are enabled on the table - REQ-DATA-015: `ConsistentRead` parameter must be accepted on GetItem, Query, Scan, and BatchGetItem. When a read replica is configured, `ConsistentRead=false` reads must route to the replica; `ConsistentRead=true` reads must route to the primary. When no replica is configured, all reads are strongly consistent. Capacity calculations must reflect the consistency mode regardless of replica configuration (0.5 RCU for eventually consistent, 1.0 RCU for strongly consistent). @@ -212,7 +215,7 @@ These are the core CRUD operations. All must be fully implemented. - REQ-IMPEXP-003: Import/export must work with local filesystem paths (S3 support is a future enhancement) - REQ-IMPEXP-004: Import/export jobs must be tracked with status (IN_PROGRESS, COMPLETED, FAILED, CANCELLED) -### 3.4 DynamoDB Streams Operations (In Scope — Deferred Detail) +### 3.4 DynamoDB Streams Operations (In Scope, Detail Deferred) | # | Operation | Description | |---|-----------|-------------| @@ -228,7 +231,7 @@ These are the core CRUD operations. All must be fully implemented. - REQ-STREAM-004: Stream records must be ordered by sequence number within a shard - REQ-STREAM-005: Detailed shard management and retention design is deferred -### 3.5 Global Tables Operations (In Scope — Deferred Detail) +### 3.5 Global Tables Operations (In Scope, Detail Deferred) | # | Operation | Description | |---|-----------|-------------| @@ -260,47 +263,47 @@ These are the core CRUD operations. All must be fully implemented. ### 4.1.1 Auth Modes -extenddb supports three authentication modes, selectable by configuration. The mode determines how credentials are validated and how authorization decisions are made. +ExtendDB supports two authentication modes, selectable by configuration: Mode 2 (local credential store) and Mode 3 (AWS IAM). Mode 1 (no auth) existed in early builds and has been removed. The mode determines how credentials are validated and how authorization decisions are made. -**Mode 1: No auth** (removed — `auth.provider = "none"` is no longer accepted) +**Mode 1: No auth** (removed: `auth.provider = "none"` is no longer accepted) Previously accepted any request regardless of the Authorization header. This mode has been removed. The server refuses to start with `provider = "none"`. **Mode 2: Local credential store** (`auth.provider = "builtin"`) -extenddb maintains its own credential store and policy documents. SigV4 signatures are validated against locally stored secret keys. Authorization is evaluated against locally stored IAM-style policies. Credentials and policies are managed via the management API or seed configuration. This is the default for production in disconnected or semi-connected environments. +ExtendDB maintains its own credential store and policy documents. SigV4 signatures are validated against locally stored secret keys. Authorization is evaluated against locally stored IAM-style policies. Credentials and policies are managed via the management API or seed configuration. This is the default for production in disconnected or semi-connected environments. **Mode 3: AWS IAM** (`auth.provider = "aws_iam"`) -When extenddb has network connectivity to AWS (e.g., running on EC2), it delegates authentication and authorization to real AWS IAM. This mode has three sub-capabilities: +When ExtendDB has network connectivity to AWS (e.g., running on EC2), it delegates authentication and authorization to real AWS IAM. This mode has three sub-capabilities: -- REQ-AUTH-008: **STS authentication via pre-signed GetCallerIdentity token.** The client generates a pre-signed STS `GetCallerIdentity` URL (signed for the `sts` service) and sends it as an `X-Extenddb-Auth-Token` header alongside the normal DynamoDB request. extenddb calls STS using this pre-signed URL to validate the credential and resolve the caller's IAM ARN, account ID, and user ID. Cache validated identities with configurable TTL to avoid per-request STS calls. This is the same pattern used by EKS (`aws-iam-authenticator`) and Vault's AWS auth method. +- REQ-AUTH-008: **STS authentication via pre-signed GetCallerIdentity token.** The client generates a pre-signed STS `GetCallerIdentity` URL (signed for the `sts` service) and sends it as an `X-Extenddb-Auth-Token` header alongside the normal DynamoDB request. ExtendDB calls STS using this pre-signed URL to validate the credential and resolve the caller's IAM ARN, account ID, and user ID. Cache validated identities with configurable TTL to avoid per-request STS calls. This is the same pattern used by EKS (`aws-iam-authenticator`) and Vault's AWS auth method. - REQ-AUTH-009: **IAM policy retrieval.** Fetch the caller's attached policies, group policies, and permissions boundaries from IAM using `iam:GetUserPolicy`, `iam:ListAttachedUserPolicies`, `iam:GetPolicy`, `iam:GetPolicyVersion`, `iam:ListGroupsForUser`, `iam:GetRolePolicy`, `iam:ListAttachedRolePolicies`, and `iam:GetUserPermissionsBoundary` / `iam:GetRolePermissionsBoundary`. Retrieved policies are evaluated by the same local policy engine used in Mode 2. Cache retrieved policies with configurable TTL. -- REQ-AUTH-010: **Near-seamless SDK experience with thin client wrapper.** A lightweight extenddb client wrapper generates the pre-signed STS token and attaches it as `X-Extenddb-Auth-Token` before each request. The wrapper uses the same AWS credentials and SDK configuration as normal DynamoDB calls. Switching between extenddb and real DynamoDB requires changing `endpoint_url` and adding/removing the wrapper. The wrapper is provided for Python (boto3 plugin), Rust, and Java. -- REQ-AUTH-011: **Graceful degradation.** If STS or IAM is unreachable (network partition, transient failure), extenddb must return a clear error (`ServiceUnavailable` with a message indicating the auth backend is unreachable) rather than silently allowing or denying requests. Cached identities and policies remain valid for their TTL during outages. -- REQ-AUTH-012: **IAM permissions required.** The EC2 instance (or IAM principal) running extenddb must have permissions to call `sts:GetCallerIdentity` (always allowed, no policy needed) and the `iam:Get*`/`iam:List*` actions above. The required IAM policy for the extenddb host is documented in the deployment guide. +- REQ-AUTH-010: **Near-seamless SDK experience with thin client wrapper.** A lightweight ExtendDB client wrapper generates the pre-signed STS token and attaches it as `X-Extenddb-Auth-Token` before each request. The wrapper uses the same AWS credentials and SDK configuration as normal DynamoDB calls. Switching between ExtendDB and real DynamoDB requires changing `endpoint_url` and adding/removing the wrapper. The wrapper is provided for Python (boto3 plugin), Rust, and Java. +- REQ-AUTH-011: **Graceful degradation.** If STS or IAM is unreachable (network partition, transient failure), ExtendDB must return a clear error (`ServiceUnavailable` with a message indicating the auth backend is unreachable) rather than silently allowing or denying requests. Cached identities and policies remain valid for their TTL during outages. +- REQ-AUTH-012: **IAM permissions required.** The EC2 instance (or IAM principal) running ExtendDB must have permissions to call `sts:GetCallerIdentity` (always allowed, no policy needed) and the `iam:Get*`/`iam:List*` actions above. The required IAM policy for the ExtendDB host is documented in the deployment guide. -> **Design note:** Mode 3 enables a powerful workflow: develop locally with Mode 2 (local credentials + policies), then deploy to EC2 with Mode 3 (real IAM) using the same application code and SDK configuration (only `endpoint_url` changes, plus adding the extenddb client wrapper). The same policy documents work in both modes because the policy engine is shared. The pre-signed token approach is a proven pattern used by EKS and Vault — it works because the token is signed for the `sts` service by the client's own credentials, avoiding the SigV4 service-binding problem. +> **Design note:** Mode 3 enables a powerful workflow: develop locally with Mode 2 (local credentials + policies), then deploy to EC2 with Mode 3 (real IAM) using the same application code and SDK configuration (only `endpoint_url` changes, plus adding the ExtendDB client wrapper). The same policy documents work in both modes because the policy engine is shared. The pre-signed token approach is a proven pattern used by EKS and Vault. It works because the token is signed for the `sts` service by the client's own credentials, avoiding the SigV4 service-binding problem. ### 4.2 Identity Model -extenddb implements a local IAM identity model that mirrors the structure of AWS IAM for DynamoDB access control. This enables testing of IAM policies, role-based access, and tag-based access control (ABAC) against extenddb with the same policy documents used in production. +ExtendDB implements a local IAM identity model that mirrors the structure of AWS IAM for DynamoDB access control. This enables testing of IAM policies, role-based access, and tag-based access control (ABAC) against ExtendDB with the same policy documents used in production. - REQ-IDENT-001: Support IAM-style principals: users and roles. A **user** has long-term credentials (access key + secret key). A **role** has a trust policy and is assumed by users to obtain temporary credentials. - REQ-IDENT-002: Support **groups**. A group contains zero or more users. Policies attached to a group apply to all member users. A user can belong to multiple groups. - REQ-IDENT-003: Each principal (user, role) has an ARN: `arn:aws:iam::{account_id}:user/{name}`, `arn:aws:iam::{account_id}:role/{name}`. Groups: `arn:aws:iam::{account_id}:group/{name}`. - REQ-IDENT-004: Support **principal tags** (key-value pairs attached to users and roles). Principal tags are available as `aws:PrincipalTag/{key}` in policy condition evaluation. - REQ-IDENT-005: Support **session tags** passed during role assumption. Session tags override principal tags for the duration of the session and are available as `aws:PrincipalTag/{key}`. -- REQ-IDENT-006: Support **AssumeRole** — a user assumes a role to obtain temporary session credentials (access key, secret key, session token). The session has an expiration time. The role's trust policy controls who can assume it. +- REQ-IDENT-006: Support **AssumeRole**: a user assumes a role to obtain temporary session credentials (access key, secret key, session token). The session has an expiration time. The role's trust policy controls who can assume it. - REQ-IDENT-007: Session credentials include an `aws:PrincipalTag/*` namespace populated from the role's tags, the user's tags, and any session tags passed at assumption time (session tags take precedence). -- REQ-IDENT-008 (Deferred): **Federated role assumption** (`AssumeRoleWithSAML`, `AssumeRoleWithWebIdentity`) is not supported in v1. Applications that use web identity federation (e.g., Lambda functions, EKS pods) to access DynamoDB cannot replicate that auth path against extenddb. Workaround: use basic AssumeRole with equivalent policies. +- REQ-IDENT-008 (Deferred): **Federated role assumption** (`AssumeRoleWithSAML`, `AssumeRoleWithWebIdentity`) is not supported in v1. Applications that use web identity federation (e.g., Lambda functions, EKS pods) to access DynamoDB cannot replicate that auth path against ExtendDB. Workaround: use basic AssumeRole with equivalent policies. - REQ-IDENT-009 (Deferred): **Role chaining** (a role assuming another role) is not supported in v1. `CallerArn` in AssumeRole must reference a user. `SourceIdentity` (tracks the original caller through a chain) and `TransitiveTagKeys` (session tags that persist through chains) are deferred alongside role chaining since they only apply in chained scenarios. -- REQ-IDENT-010 (Deferred): **MaxSessionDuration** per role is not enforced in v1. AWS IAM roles have a configurable maximum session duration (default 1 hour, up to 12 hours) that caps the `DurationSeconds` parameter in AssumeRole. extenddb v1 accepts any `DurationSeconds` value with no per-role validation. +- REQ-IDENT-010 (Deferred): **MaxSessionDuration** per role is not enforced in v1. AWS IAM roles have a configurable maximum session duration (default 1 hour, up to 12 hours) that caps the `DurationSeconds` parameter in AssumeRole. ExtendDB v1 accepts any `DurationSeconds` value with no per-role validation. -### 4.3 Authorization — Identity-Based Policies +### 4.3 Authorization: Identity-Based Policies - REQ-AUTHZ-001: Authorization must evaluate IAM-style policy documents - REQ-AUTHZ-002: Support `Allow` and `Deny` effects with explicit Deny precedence - REQ-AUTHZ-003: Support action matching with wildcards (`dynamodb:*`, `dynamodb:Get*`) - REQ-AUTHZ-004: Support resource matching with wildcards (`arn:aws:dynamodb:*:*:table/Prefix*`) -- REQ-AUTHZ-005: Authorization must be decoupled from authentication — any auth provider can feed into the same policy engine +- REQ-AUTHZ-005: Authorization must be decoupled from authentication: any auth provider can feed into the same policy engine - REQ-AUTHZ-006: Support **identity-based policies** attached to users, groups, and roles. When a request is made: - Collect all policies attached directly to the user - Collect all policies from groups the user belongs to @@ -309,7 +312,7 @@ extenddb implements a local IAM identity model that mirrors the structure of AWS - REQ-AUTHZ-007: Support **permissions boundaries** on users and roles. A permissions boundary is a managed policy that sets the maximum permissions. The effective permissions are the intersection of identity-based policies and the permissions boundary. - REQ-AUTHZ-008: Policy evaluation order follows AWS IAM semantics: (1) explicit Deny in any policy → Deny; (2) if permissions boundary exists, action must be allowed by both the boundary and an identity policy; (3) explicit Allow in any identity policy → Allow; (4) implicit Deny. -### 4.4 Authorization — Tag-Based Access Control (ABAC) +### 4.4 Authorization: Tag-Based Access Control (ABAC) - REQ-ABAC-001: Support **resource tag conditions** in policies via `aws:ResourceTag/{key}`. The policy engine resolves resource tags by querying the tag store for the target table's ARN. - REQ-ABAC-002: Support **principal tag conditions** via `aws:PrincipalTag/{key}`. Principal tags come from the authenticated identity (user tags, role tags, or session tags). @@ -322,17 +325,17 @@ extenddb implements a local IAM identity model that mirrors the structure of AWS - ARN operators: `ArnLike`, `ArnNotLike`, `ArnEquals`, `ArnNotEquals` - Set operators: `ForAllValues:StringEquals`, `ForAnyValue:StringEquals` (and other `ForAllValues`/`ForAnyValue` prefixes with any string/numeric operator) - REQ-ABAC-004: Support the following DynamoDB-specific condition context keys: - - `dynamodb:LeadingKeys` — restricts access based on the partition key value of items being accessed. Used with `ForAllValues:StringEquals` to limit a user to their own items. - - `dynamodb:Attributes` — restricts which attributes can be read or written. Used with `ForAllValues:StringEquals`. - - `dynamodb:Select` — restricts the `Select` parameter in Query/Scan (e.g., force `SPECIFIC_ATTRIBUTES` to prevent full-item reads). - - `dynamodb:ReturnValues` — restricts the `ReturnValues` parameter. - - `dynamodb:ReturnConsumedCapacity` — restricts the `ReturnConsumedCapacity` parameter. - - `dynamodb:FullTableScan` — restricts whether Scan operations are allowed (`Bool` condition). - - `dynamodb:EnclosingOperation` — identifies the parent operation for batch/transact sub-operations. + - `dynamodb:LeadingKeys`: restricts access based on the partition key value of items being accessed. Used with `ForAllValues:StringEquals` to limit a user to their own items. + - `dynamodb:Attributes`: restricts which attributes can be read or written. Used with `ForAllValues:StringEquals`. + - `dynamodb:Select`: restricts the `Select` parameter in Query/Scan (e.g., force `SPECIFIC_ATTRIBUTES` to prevent full-item reads). + - `dynamodb:ReturnValues`: restricts the `ReturnValues` parameter. + - `dynamodb:ReturnConsumedCapacity`: restricts the `ReturnConsumedCapacity` parameter. + - `dynamodb:FullTableScan`: restricts whether Scan operations are allowed (`Bool` condition). + - `dynamodb:EnclosingOperation`: identifies the parent operation for batch/transact sub-operations. - REQ-ABAC-005: The policy engine must build a **request context** for each operation containing all applicable context keys. The request context is populated by the server middleware before policy evaluation. -- REQ-ABAC-006 (Deferred): **Policy variables** (`${aws:PrincipalTag/key}`, `${aws:username}`, etc.) in Resource ARNs and Condition values are not supported in v1. In AWS IAM, policy variables enable a single policy to grant access to resources that match the caller's own tags — e.g., `"Resource": "arn:aws:dynamodb:*:*:table/${aws:PrincipalTag/Team}-*"`. Without policy variables, each team or department requires a separate policy with hardcoded resource ARNs, which reduces the ABAC testing value. This is a high-priority enhancement for a future version. +- REQ-ABAC-006 (Deferred): **Policy variables** (`${aws:PrincipalTag/key}`, `${aws:username}`, etc.) in Resource ARNs and Condition values were not supported in v1; variables in Condition values are expanded since, and substitution in Resource ARNs is not implemented. In AWS IAM, policy variables enable a single policy to grant access to resources that match the caller's own tags, e.g., `"Resource": "arn:aws:dynamodb:*:*:table/${aws:PrincipalTag/Team}-*"`. Without policy variables, each team or department requires a separate policy with hardcoded resource ARNs, which reduces the ABAC testing value. -### 4.5 Authorization — Caching +### 4.5 Authorization: Caching - REQ-CACHE-001: Policy evaluation results must be cached with configurable TTL to avoid repeated database lookups on every request - REQ-CACHE-002: Identity resolution (user → groups → policies, role → policies) must be cached with configurable TTL @@ -362,7 +365,7 @@ All limits must be enforced by default with DynamoDB-compatible values. All limi | Limit | DynamoDB Default | Configurable | Requirement ID | |-------|-----------------|--------------|----------------| -| Table name length | 3–255 characters | Yes | REQ-LIM-020 | +| Table name length | 3-255 characters | Yes | REQ-LIM-020 | | Table name characters | `[a-zA-Z0-9_.-]` | No | REQ-LIM-021 | | Tables per account | 2,500 | Yes | REQ-LIM-022 | | GSIs per table | 20 | Yes | REQ-LIM-023 | @@ -386,13 +389,13 @@ All limits must be enforced by default with DynamoDB-compatible values. All limi | Limit | DynamoDB Default | Configurable | Requirement ID | |-------|-----------------|--------------|----------------| -| BatchGetItem — max items | 100 | Yes | REQ-LIM-040 | -| BatchGetItem — max response size | 16 MB | Yes | REQ-LIM-041 | -| BatchWriteItem — max items | 25 | Yes | REQ-LIM-042 | -| TransactGetItems — max items | 100 | Yes | REQ-LIM-043 | -| TransactWriteItems — max items | 100 | Yes | REQ-LIM-044 | -| Query/Scan — max response size | 1 MB | Yes | REQ-LIM-045 | -| ListTables — max per page | 100 | Yes | REQ-LIM-046 | +| BatchGetItem: max items | 100 | Yes | REQ-LIM-040 | +| BatchGetItem: max response size | 16 MB | Yes | REQ-LIM-041 | +| BatchWriteItem: max items | 25 | Yes | REQ-LIM-042 | +| TransactGetItems: max items | 100 | Yes | REQ-LIM-043 | +| TransactWriteItems: max items | 100 | Yes | REQ-LIM-044 | +| Query/Scan: max response size | 1 MB | Yes | REQ-LIM-045 | +| ListTables: max per page | 100 | Yes | REQ-LIM-046 | | Max request body size | 16 MB | Yes | REQ-LIM-047 | ## 6. Capacity & Throughput Emulation @@ -405,34 +408,34 @@ All limits must be enforced by default with DynamoDB-compatible values. All limi - REQ-CAP-006: Return `ConsumedCapacity` in responses when `ReturnConsumedCapacity` is TOTAL or INDEXES - REQ-CAP-007: Return per-table and per-index capacity breakdown when `ReturnConsumedCapacity` is INDEXES - REQ-CAP-008: Return `ProvisionedThroughputExceededException` when throughput limits are exceeded -- REQ-CAP-009: Throughput tracking must be per-instance (stateless — no cross-instance coordination required) +- REQ-CAP-009: Throughput tracking must be per-instance (stateless: no cross-instance coordination required) ## 7. Catalog & Data Separation Requirements ### 7.1 Catalog Database -The catalog database stores extenddb metadata: table definitions, indexes, tags, and the data database connection info. The data database stores user item data. This follows the PostgreSQL/MySQL pattern of system catalog vs. user databases. +The catalog database stores ExtendDB metadata: table definitions, indexes, tags, and the data database connection info. The data database stores user item data. This follows the PostgreSQL/MySQL pattern of system catalog vs. user databases. -- REQ-CAT-001: extenddb metadata (table definitions, indexes, tags) must reside in a catalog database, separate from user item data +- REQ-CAT-001: ExtendDB metadata (table definitions, indexes, tags) must reside in a catalog database, separate from user item data - REQ-CAT-002: User item data must reside in a data database whose connection info is recorded in the catalog - REQ-CAT-003: The `extenddb.toml` connection string points to the catalog database; the catalog stores the data database location -- REQ-CAT-004: The catalog database name is user-chosen via the connection string — never hardcoded. Multiple independent catalogs can coexist on one PostgreSQL instance +- REQ-CAT-004: The catalog database name is user-chosen via the connection string, never hardcoded. Multiple independent catalogs can coexist on one PostgreSQL instance ### 7.2 Catalog Versioning - REQ-CAT-005: The catalog carries a semver version (`major.minor.patch`). Major = breaking schema changes, minor = additive, patch = non-structural -- REQ-CAT-006: The expected catalog version is a build-time constant compiled into the binary — not configurable at runtime +- REQ-CAT-006: The expected catalog version is a build-time constant compiled into the binary, not configurable at runtime - REQ-CAT-007: On startup, the server validates the catalog version matches the version compiled into the binary. Mismatch → refuse to start with a clear error directing the user to run `extenddb migrate` - REQ-CAT-008: `extenddb --version` prints both binary version and expected catalog version: `extenddb 0.1.0 (catalog 1.0.0)`. Binary version and catalog version are independent -- REQ-CAT-009: Any PR that modifies catalog schema or migration files must bump the catalog version. This is a hard review gate for developer, reviewer, and principal reviewer +- REQ-CAT-009: Any change that modifies catalog schema or migration files must bump the catalog version ### 7.3 Lifecycle Tools - REQ-CAT-010: The server does not run migrations on startup. Migrations are an explicit step via `extenddb init` or `extenddb migrate`. Server startup is read-only against the catalog schema -- REQ-CAT-011: `extenddb init` — subcommand that initializes a new deployment: creates (or connects to) the catalog database, runs initial migrations, creates (or connects to) the data database, records the data database connection in the catalog, generates `extenddb.toml` if needed. Documented in `docs/` -- REQ-CAT-012: `extenddb destroy` — subcommand that tears down a deployment: reads config, connects to catalog, enumerates all user tables, prints what will be destroyed, requires `--yes` to confirm, drops data database, drops catalog database. Documented in `docs/` -- REQ-CAT-013: `extenddb verify` — subcommand that validates a deployment: connects to catalog, checks version, enumerates all tables and indexes in catalog, connects to data database, verifies corresponding storage structures exist. Reports healthy/missing/inconsistent. Documented in `docs/` -- REQ-CAT-014: `extenddb migrate` — subcommand that applies schema migrations to an existing catalog: reads current version, determines required migrations, shows plan, requires `--yes` to confirm, runs migrations, updates catalog version. Documented in `docs/` +- REQ-CAT-011: `extenddb init` is the subcommand that initializes a new deployment. It creates (or connects to) the catalog database, runs initial migrations, creates (or connects to) the data database, records the data database connection in the catalog, generates `extenddb.toml` if needed. Documented in `docs/` +- REQ-CAT-012: `extenddb destroy` is the subcommand that tears down a deployment. It reads config, connects to catalog, enumerates all user tables, prints what will be destroyed, requires `--yes` to confirm, drops data database, drops catalog database. Documented in `docs/` +- REQ-CAT-013: `extenddb verify` is the subcommand that validates a deployment. It connects to catalog, checks version, enumerates all tables and indexes in catalog, connects to data database, verifies corresponding storage structures exist. Reports healthy/missing/inconsistent. Documented in `docs/` +- REQ-CAT-014: `extenddb migrate` is the subcommand that applies schema migrations to an existing catalog. It reads current version, determines required migrations, shows plan, requires `--yes` to confirm, runs migrations, updates catalog version. Documented in `docs/` ## 8. Storage Backend Requirements @@ -441,7 +444,7 @@ The catalog database stores extenddb metadata: table definitions, indexes, tags, - REQ-STOR-001: All data access must go through a `StorageEngine` trait (async, `Send + Sync`) - REQ-STOR-002: The trait must cover: table CRUD, item CRUD, query/scan, batch operations, transaction operations, metadata, stream events, import/export job tracking - REQ-STOR-003: The trait must support transactions with serializable isolation for TransactWriteItems/TransactGetItems -- REQ-STOR-004: The trait must not leak backend-specific types — all inputs and outputs use core DynamoDB types +- REQ-STOR-004: The trait must not leak backend-specific types: all inputs and outputs use core DynamoDB types - REQ-STOR-005: Adding a new backend must not require changes to any existing crate ### 8.2 PostgreSQL Backend (First Implementation) @@ -545,7 +548,7 @@ The system must support all 10 DynamoDB attribute value types: ## 12. SDK Client Configuration Requirements -An unmodified application that works against AWS DynamoDB must work against extenddb by changing only endpoint configuration — no code changes, no SDK patches, no custom plugins. +An unmodified application that works against AWS DynamoDB must work against ExtendDB by changing only endpoint configuration: no code changes, no SDK patches, no custom plugins. ### 12.1 Endpoint Override Mechanism @@ -563,18 +566,18 @@ The service-specific environment variables for DynamoDB and DynamoDB Streams are | DynamoDB | `dynamodb` | `AWS_ENDPOINT_URL_DYNAMODB` | | DynamoDB Streams | `dynamodbstreams` | `AWS_ENDPOINT_URL_DYNAMODB_STREAMS` | -Both must point to the same extenddb address (extenddb serves both services on a single port). +Both must point to the same ExtendDB address (ExtendDB serves both services on a single port). **Requirements:** -- REQ-SDK-001: extenddb serves both DynamoDB and DynamoDB Streams on a single `host:port`. No separate Streams port or path prefix is needed — the `X-Amz-Target` prefix distinguishes the two services. +- REQ-SDK-001: ExtendDB serves both DynamoDB and DynamoDB Streams on a single `host:port`. No separate Streams port or path prefix is needed; the `X-Amz-Target` prefix distinguishes the two services. - REQ-SDK-002: `DescribeEndpoints` must return the server's own `host:port` as the `Address` with a `CachePeriodInMinutes` of 10. This is a no-op for SDK clients that set `endpoint_url` (endpoint discovery is disabled when a custom endpoint is set), but must return a valid response if called directly. - REQ-SDK-003: SigV4 signing name is `dynamodb` for both DynamoDB and DynamoDB Streams requests. The auth layer does not need to distinguish between the two services. ### 12.2 AWS Configuration Files -extenddb requires credentials for SigV4 authentication. The standard `~/.aws/credentials` and `~/.aws/config` files are used by the SDK client, not by extenddb itself. extenddb has its own credential store (see §4). The credentials configured in `~/.aws/credentials` must match a credential pair registered in extenddb's credential store. +ExtendDB requires credentials for SigV4 authentication. The standard `~/.aws/credentials` and `~/.aws/config` files are used by the SDK client, not by ExtendDB itself. ExtendDB has its own credential store (see §4). The credentials configured in `~/.aws/credentials` must match a credential pair registered in ExtendDB's credential store. -**Minimal setup — environment variables only (simplest):** +**Minimal setup, environment variables only (simplest):** ```bash # Point both services at extenddb @@ -587,9 +590,9 @@ export AWS_SECRET_ACCESS_KEY=local-dev-secret export AWS_DEFAULT_REGION=us-east-1 ``` -With this setup, any application using `boto3.client('dynamodb')` or `boto3.client('dynamodbstreams')` — or the equivalent in any other SDK — automatically talks to extenddb with zero code changes. +With this setup, any application using `boto3.client('dynamodb')` or `boto3.client('dynamodbstreams')`, or the equivalent in any other SDK, automatically talks to ExtendDB with zero code changes. -**Alternative — `~/.aws/config` with services section:** +**Alternative: `~/.aws/config` with services section:** ```ini [profile extenddb] @@ -610,34 +613,34 @@ aws_access_key_id = local-dev-key aws_secret_access_key = local-dev-secret ``` -Usage: `export AWS_PROFILE=extenddb` — then all SDK clients in the process use extenddb. +Usage: `export AWS_PROFILE=extenddb`. All SDK clients in the process then use ExtendDB. -**Alternative — global endpoint override (all AWS services go to one place):** +**Alternative: global endpoint override (all AWS services go to one place):** ```bash export AWS_ENDPOINT_URL=https://127.0.0.1:18443 ``` -This sends every AWS SDK call to extenddb. Only useful when the application exclusively uses DynamoDB. +This sends every AWS SDK call to ExtendDB. Only useful when the application exclusively uses DynamoDB. ### 12.3 Endpoint Discovery Behavior DynamoDB SDKs support optional endpoint discovery via the `DescribeEndpoints` operation. The behavior varies by SDK: -- **When `endpoint_url` is set:** Endpoint discovery is completely disabled. The SDK sends all requests to the configured URL. This is the normal extenddb usage path. +- **When `endpoint_url` is set:** Endpoint discovery is completely disabled. The SDK sends all requests to the configured URL. This is the normal ExtendDB usage path. - **When `endpoint_url` is NOT set:** The SDK may call `DescribeEndpoints` to discover the optimal endpoint. This only happens against real AWS. **Requirements:** -- REQ-SDK-004: extenddb must handle `DescribeEndpoints` requests and return a valid response pointing to itself. This ensures correctness if an application explicitly calls `DescribeEndpoints`, even though SDKs skip it when `endpoint_url` is set. -- REQ-SDK-005: Endpoint discovery must not redirect clients away from extenddb. The returned `Address` must always be the server's own listen address. +- REQ-SDK-004: ExtendDB must handle `DescribeEndpoints` requests and return a valid response pointing to itself. This ensures correctness if an application explicitly calls `DescribeEndpoints`, even though SDKs skip it when `endpoint_url` is set. +- REQ-SDK-005: Endpoint discovery must not redirect clients away from ExtendDB. The returned `Address` must always be the server's own listen address. ## 13. Testing Requirements - REQ-TEST-001: A minimal test suite must exist for every phase, covering all implemented operations. No phase exits without a passing test suite -- REQ-TEST-002: Tests are written in Python (boto3) and AWS CLI, covering all Phase 1 operations: CreateTable, DescribeTable, ListTables, DeleteTable, and key error paths -- REQ-TEST-003: Tests must be dual-target — run against both real DynamoDB and extenddb with identical assertions, controlled by an endpoint parameter. No target-specific branching -- REQ-TEST-004: If behavior differs between real DynamoDB and extenddb, that is a bug in extenddb -- REQ-TEST-005: Tests must clean up after themselves — all created resources are deleted on exit, even on failure +- REQ-TEST-002: Tests are written in Python (boto3) and AWS CLI, covering the table operations: CreateTable, DescribeTable, ListTables, DeleteTable, and key error paths +- REQ-TEST-003: Tests must be dual-target: run against both real DynamoDB and ExtendDB with identical assertions, controlled by an endpoint parameter. No target-specific branching +- REQ-TEST-004: If behavior differs between real DynamoDB and ExtendDB, that is a bug in ExtendDB +- REQ-TEST-005: Tests must clean up after themselves: all created resources are deleted on exit, even on failure ## 14. Documentation Requirements @@ -651,20 +654,20 @@ DynamoDB SDKs support optional endpoint discovery via the `DescribeEndpoints` op ### 15.1 Startup Banner -- REQ-LOG-001: On launch, extenddb logs a summary of its effective configuration: binary version, catalog version, bind address, region, account ID, auth provider, catalog database, data database, log level +- REQ-LOG-001: On launch, ExtendDB logs a summary of its effective configuration: binary version, catalog version, bind address, region, account ID, auth provider, catalog database, data database, log level - REQ-LOG-002: Connection strings must redact passwords in log output ### 15.2 Syslog Integration -- REQ-LOG-003: extenddb logs to syslog with ident `extenddb` and facility `daemon`. Read logs with `journalctl -t extenddb` +- REQ-LOG-003: ExtendDB logs to syslog with ident `extenddb` and facility `daemon`. Read logs with `journalctl -t extenddb` - REQ-LOG-004: Default log level is `debug` - REQ-LOG-005: Supported levels: `debug`, `info`, `error`, `critical` - REQ-LOG-006: Log format follows syslog convention: ISO 8601 timestamp, level, message ### 15.3 Log Hygiene -- REQ-LOG-007: Reviewers verify that log statements use appropriate levels. Every `error` and `critical` log message must have a corresponding entry in `docs/troubleshooting.md` -- REQ-LOG-008: Any PR that adds or changes error/critical messages must update the troubleshooting doc +- REQ-LOG-007: Log statements use appropriate levels. Every `error` and `critical` log message must have a corresponding entry in `docs/troubleshooting.md` +- REQ-LOG-008: Any change that adds or changes error/critical messages must update the troubleshooting doc --- @@ -673,6 +676,6 @@ DynamoDB SDKs support optional endpoint discovery via the `DescribeEndpoints` op Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/02-high-level-design.md b/docs/design/02-high-level-design.md index 9c6113c97..bc5926afa 100755 --- a/docs/design/02-high-level-design.md +++ b/docs/design/02-high-level-design.md @@ -1,16 +1,15 @@ --- description: "The Cargo workspace, the request path from HTTP to storage, and how the crates depend on each other." +version: "1.0" +date: "2026-04-03" +status: "Draft" --- # High-level design -**Version:** 1.0 -**Date:** 2026-04-03 -**Status:** Draft - ## 1. Architecture Overview -extenddb (ExtendDB) is a standalone async Rust application structured as a Cargo workspace. It receives DynamoDB wire protocol requests over HTTP, authenticates and authorizes them via a pluggable auth provider, executes the operation logic in a backend-agnostic core, and delegates persistence to a pluggable storage engine. +ExtendDB is a standalone async Rust application structured as a Cargo workspace. It receives DynamoDB wire protocol requests over HTTP, authenticates and authorizes them via a pluggable auth provider, executes the operation logic in a backend-agnostic core, and delegates persistence to a pluggable storage engine. ``` ┌─────────────────────────────────────────────────────────────────────┐ @@ -166,7 +165,7 @@ graph TD STORAGE --> CORE ``` -**Key principle:** `core` depends on nothing in the workspace and has no async runtime — it is pure sync Rust (types, expressions, validation, capacity, errors). `storage` depends only on `core`. `engine` depends on `core` + `storage` and contains the async operation handlers. Backend crates depend on `storage` + `core`. `auth` depends on `core`. `server` depends on `core` + `storage` + `auth` + `engine`. `bin` wires everything together. +**Key principle:** `core` depends on nothing in the workspace and has no async runtime; it is pure sync Rust (types, expressions, validation, capacity, errors). `storage` depends only on `core`. `engine` depends on `core` + `storage` and contains the async operation handlers. Backend crates depend on `storage` + `core`. `auth` depends on `core`. `server` depends on `core` + `storage` + `auth` + `engine`. `bin` wires everything together. ## 3. Request Lifecycle @@ -207,13 +206,13 @@ sequenceDiagram **Decision:** Structure as a Cargo workspace with 7 crates. **Rationale:** -- Compile-time enforcement of dependency boundaries — the storage trait crate physically cannot depend on PostgreSQL -- `core` is pure sync Rust with no async runtime — types, expressions, validation, capacity, errors +- Compile-time enforcement of dependency boundaries: the storage trait crate physically cannot depend on PostgreSQL +- `core` is pure sync Rust with no async runtime: types, expressions, validation, capacity, errors - `engine` contains async operation handlers that depend on `core` + `storage`, keeping the async boundary explicit -- Independent testing — `cargo test -p dynamodb-core` needs no database and no async runtime -- Faster incremental builds — changing the Postgres backend doesn't recompile the expression parser -- Clean extension points — adding a SQLite backend is a new crate, zero changes to existing code -- Lean dependency trees — core has no async runtime, no database drivers, no HTTP framework +- Independent testing: `cargo test -p dynamodb-core` needs no database and no async runtime +- Faster incremental builds: changing the Postgres backend doesn't recompile the expression parser +- Clean extension points: adding a SQLite backend is a new crate, zero changes to existing code +- Lean dependency trees: core has no async runtime, no database drivers, no HTTP framework ### 4.2 axum + tower for HTTP @@ -222,7 +221,7 @@ sequenceDiagram **Rationale:** - axum is the de facto standard for async Rust HTTP servers, built on top of hyper and tower - tower's `Layer` and `Service` traits provide composable middleware that maps directly to the DynamoDB middleware pipeline (auth, capacity, metrics, logging) -- Native async/await with tokio — no blocking threads, efficient connection handling +- Native async/await with tokio: no blocking threads, efficient connection handling - Mature ecosystem with TLS (rustls), compression, and graceful shutdown support - axum's extractor pattern cleanly maps to parsing DynamoDB request headers and bodies @@ -231,22 +230,22 @@ sequenceDiagram **Decision:** Define `StorageEngine` sub-traits as async traits using RPITIT (Return Position Impl Trait In Trait, stable since Rust 1.75), selected at startup via enum dispatch. Define `AuthProvider` and `CredentialStore` using `#[async_trait]` for object-safe dynamic dispatch. **Rationale:** -- **Storage (RPITIT + enum dispatch):** RPITIT avoids the `Box` heap allocation per call that `async_trait` introduces — at high throughput on the data path this matters. Since RPITIT traits are not object-safe, the backend is selected at startup via an enum dispatch wrapper in the `bin` crate (e.g., `enum Storage { Postgres(PostgresEngine) }`) rather than `Arc`. The enum dispatch approach has zero overhead (static dispatch) and the match arms are generated once at startup. -- **Auth (`#[async_trait]` + `Arc`):** Both `AuthProvider` and `CredentialStore` use `#[async_trait]` for object safety. Auth is called once per request and involves crypto operations (SigV4 HMAC-SHA256, policy evaluation) that dwarf the cost of a single `Box` allocation. Using `#[async_trait]` makes `AuthProvider` object-safe, allowing `Arc` in the server crate without enum wrappers. `CredentialStore` uses `#[async_trait]` for the same reason — it allows `Arc` inside `BuiltinAuthProvider`, avoiding a generic parameter that would propagate up through the auth provider. Note: `CredentialStore` (in the `auth` crate) is a separate trait from `CredentialEngine` (in the `storage` crate, which uses RPITIT). The `bin` crate bridges them with a thin adapter. +- **Storage (RPITIT + enum dispatch):** RPITIT avoids the `Box` heap allocation per call that `async_trait` introduces; at high throughput on the data path this matters. Since RPITIT traits are not object-safe, the backend is selected at startup via an enum dispatch wrapper in the `bin` crate (e.g., `enum Storage { Postgres(PostgresEngine) }`) rather than `Arc`. The enum dispatch approach has zero overhead (static dispatch) and the match arms are generated once at startup. +- **Auth (`#[async_trait]` + `Arc`):** Both `AuthProvider` and `CredentialStore` use `#[async_trait]` for object safety. Auth is called once per request and involves crypto operations (SigV4 HMAC-SHA256, policy evaluation) that dwarf the cost of a single `Box` allocation. Using `#[async_trait]` makes `AuthProvider` object-safe, allowing `Arc` in the server crate without enum wrappers. `CredentialStore` uses `#[async_trait]` for the same reason: it allows `Arc` inside `BuiltinAuthProvider`, avoiding a generic parameter that would propagate up through the auth provider. Note: `CredentialStore` (in the `auth` crate) is a separate trait from `CredentialEngine` (in the `storage` crate, which uses RPITIT). The `bin` crate bridges them with a thin adapter. - Avoids monomorphization bloat from generic parameters threaded through the entire codebase -- Clean testing — mock implementations for unit tests, real backends for integration tests +- Clean testing: mock implementations for unit tests, real backends for integration tests ### 4.4 Expression Engine in Core (No Database Dependency) **Decision:** The expression parser and evaluator live in the `core` crate and operate on in-memory `AttributeValue` types. **Rationale:** -- Expressions are DynamoDB-specific logic, not storage-specific — `FilterExpression` is evaluated after items are fetched, `ConditionExpression` is evaluated against the current item before a write +- Expressions are DynamoDB-specific logic, not storage-specific: `FilterExpression` is evaluated after items are fetched, `ConditionExpression` is evaluated against the current item before a write - Keeping expressions in core means every storage backend gets the same expression behavior for free - The storage backend only needs to handle `KeyConditionExpression` translation to its native query language (e.g., SQL WHERE clauses for partition key equality + sort key range) - This matches how DynamoDB actually works: filter expressions are applied after the read, not pushed down to the storage layer -**Important:** While the expression evaluator lives in `core`, `ConditionExpression` evaluation for writes is called by the storage backend inside its transaction (after `SELECT FOR UPDATE`). The storage backend imports `core::expression::evaluate_condition` and calls it within the transaction to prevent TOCTOU races. `FilterExpression` evaluation happens in the core handler after items are returned — this is safe because filters are read-only post-processing. +**Important:** While the expression evaluator lives in `core`, `ConditionExpression` evaluation for writes is called by the storage backend inside its transaction (after `SELECT FOR UPDATE`). The storage backend imports `core::expression::evaluate_condition` and calls it within the transaction to prevent TOCTOU races. `FilterExpression` evaluation happens in the core handler after items are returned; this is safe because filters are read-only post-processing. ### 4.5 Configuration: TOML + Env Vars + CLI @@ -264,7 +263,7 @@ sequenceDiagram **Decision:** The entire stack is async (tokio runtime), including the storage trait. **Rationale:** -- The original pg_dynamodb extension was forced into blocking mode by PostgreSQL SPI constraints. As a standalone process, we have no such limitation +- An earlier design as a PostgreSQL extension (pg_dynamodb) was forced into blocking mode by the extension interface. A standalone process has no such limitation - Async I/O is essential for handling thousands of concurrent connections efficiently - sqlx provides native async PostgreSQL support - The storage trait being async allows backends that use network I/O (e.g., a future DynamoDB-backed backend for testing, or a distributed storage backend) @@ -274,13 +273,13 @@ sequenceDiagram **Decision:** All reads are strongly consistent by default. Eventually consistent reads (`ConsistentRead=false`) are supported by routing to an optional PostgreSQL streaming replica. **Rationale:** -- DynamoDB's eventual consistency is a consequence of reading from storage replicas that haven't received the latest write yet — it's not artificial delays or queues -- Strongly consistent is strictly stronger than eventually consistent — any application that works against eventually consistent DynamoDB also works against a strongly consistent backend -- For migration testing (relational → DynamoDB), teams need to surface bugs where application code assumes strong consistency on default reads. A real read replica provides genuine staleness without synthetic delays -- The read replica is opt-in via `storage.postgres.read_replica_url`. When unset, all reads hit the primary — simple, correct, and matching what DynamoDB Local and other emulators do +- DynamoDB's eventual consistency is a consequence of reading from storage replicas that haven't received the latest write yet; it's not artificial delays or queues +- Strongly consistent is strictly stronger than eventually consistent: any application that works against eventually consistent DynamoDB also works against a strongly consistent backend +- For migration testing (relational → DynamoDB), teams need to surface bugs where application code assumes strong consistency on default reads. A read replica provides the staleness of a replica that has not yet received the write, without synthetic delays +- The read replica is opt-in via `storage.postgres.read_replica_url`. When unset, all reads hit the primary: simple, correct, and matching what DynamoDB Local and other emulators do - Writes always go to the primary. `ConsistentRead=true` always reads from the primary. Only `ConsistentRead=false` reads route to the replica - Capacity calculations always reflect the consistency mode (0.5 RCU for eventually consistent, 1.0 RCU for strongly consistent) regardless of whether a replica is configured -- This requires zero changes to the storage trait, engine, or server — it's entirely contained in the PostgreSQL backend's pool selection logic +- This requires zero changes to the storage trait, engine, or server; it's entirely contained in the PostgreSQL backend's pool selection logic ## 5. Data Flow @@ -311,7 +310,7 @@ sequenceDiagram > **Key invariant:** Condition evaluation, data write, GSI updates, and stream record capture all happen within a single PostgreSQL transaction. This prevents TOCTOU races (another request modifying the item between condition check and write) and ensures atomicity of stream capture. -### 5.1b Write Path (UpdateItem — Additional Detail) +### 5.1b Write Path (UpdateItem: Additional Detail) UpdateItem is more complex than PutItem because the storage backend must also apply the update expression inside the transaction: @@ -359,11 +358,11 @@ UpdateItem is more complex than PutItem because the storage backend must also ap ## 6. Concurrency Model -extenddb handles concurrent requests through three cooperating layers: +ExtendDB handles concurrent requests through three cooperating layers: ### 6.1 Async Runtime (tokio) -The server runs on a tokio multi-thread runtime. Each incoming HTTP request is handled by an independent async task — there is no shared in-memory state on the hot path. Tasks are scheduled cooperatively across OS threads by the tokio work-stealing scheduler. +The server runs on a tokio multi-thread runtime. Each incoming HTTP request is handled by an independent async task; there is no shared in-memory state on the hot path. Tasks are scheduled cooperatively across OS threads by the tokio work-stealing scheduler. ### 6.2 PostgreSQL Connection Pool (sqlx) @@ -401,17 +400,19 @@ There is no in-memory locking (no `Mutex`, `RwLock`, or similar) on the data pat ### 6.4 Implications - **Throughput scales with pool_size.** More connections allow more concurrent transactions. The limit is PostgreSQL's `max_connections` and available CPU/IO. -- **Contention on the same item serializes at the database.** 50 threads incrementing the same counter will queue on the row lock — each update succeeds, but throughput for that single item is bounded by PostgreSQL's single-row transaction rate. +- **Contention on the same item serializes at the database.** 50 threads incrementing the same counter will queue on the row lock; each update succeeds, but throughput for that single item is bounded by PostgreSQL's single-row transaction rate. - **Different items have no contention.** Parallel inserts to different keys proceed fully concurrently up to the pool size. -## 7. High-Level Design Choices — Deferred Components +## 7. Design Choices Recorded for Streams and Global Tables + +This section records the design space as it stood when the document was written (April 2026). DynamoDB Streams has since been implemented; see the [DynamoDB Streams component design](07-component-streams.md). Global Tables remain not implemented. ### 7.1 DynamoDB Streams **Design space:** -- **Option A: Application-layer capture** — The core operation handlers emit stream events after successful writes. The storage trait includes `write_stream_record()`. Stream records are stored in a dedicated table/collection managed by the storage backend. -- **Option B: Storage-layer triggers** — The storage backend uses native CDC (e.g., PostgreSQL LISTEN/NOTIFY or logical replication). More efficient but ties stream behavior to the backend. -- **Option C: Hybrid** — The core handler constructs the stream record (it has access to old/new images). The storage engine persists the stream record in the same transaction as the data write. +- **Option A: Application-layer capture**: The core operation handlers emit stream events after successful writes. The storage trait includes `write_stream_record()`. Stream records are stored in a dedicated table/collection managed by the storage backend. +- **Option B: Storage-layer triggers**: The storage backend uses native CDC (e.g., PostgreSQL LISTEN/NOTIFY or logical replication). More efficient but ties stream behavior to the backend. +- **Option C: Hybrid**: The core handler constructs the stream record (it has access to old/new images). The storage engine persists the stream record in the same transaction as the data write. - **Recommended direction:** Option C (hybrid). The core crate controls what gets captured (portable), while the storage backend ensures atomicity by writing the data and stream record in a single transaction. This guarantees no stream record without a data write and no data write without a stream record (when streams are enabled). **Deferred decisions:** @@ -423,8 +424,8 @@ There is no in-memory locking (no `Mutex`, `RwLock`, or similar) on the data pat ### 7.2 Global Tables **Design space:** -- **Option A: Application-layer replication** — A replication agent reads stream records from one instance and replays them to another. Conflict resolution via last-writer-wins with vector clocks. -- **Option B: Storage-layer replication** — Delegate to the storage backend's native replication (e.g., PostgreSQL logical replication). Simpler but less portable. +- **Option A: Application-layer replication**: A replication agent reads stream records from one instance and replays them to another. Conflict resolution via last-writer-wins with vector clocks. +- **Option B: Storage-layer replication**: Delegate to the storage backend's native replication (e.g., PostgreSQL logical replication). Simpler but less portable. - **Recommended direction:** Option A for portability, built on top of DynamoDB Streams. **Deferred decisions:** @@ -465,6 +466,6 @@ There is no in-memory locking (no `Mutex`, `RwLock`, or similar) on the data pat Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/03-component-core.md b/docs/design/03-component-core.md index 3e2e09b44..b0afc9755 100755 --- a/docs/design/03-component-core.md +++ b/docs/design/03-component-core.md @@ -1,17 +1,16 @@ --- description: "The core crate: DynamoDB types, expression parsing and evaluation, validation and the business logic shared by every backend." +version: "1.0" +date: "2026-04-03" +status: "Draft" +crate: "dynamodb-core" --- # Component design: Core -**Version:** 1.0 -**Date:** 2026-04-03 -**Status:** Draft -**Crate:** `dynamodb-core` - ## 1. Purpose -The `core` crate contains all DynamoDB-specific business logic that is independent of the storage backend, HTTP server, and async runtime. It is pure synchronous Rust — types, expression parsing/evaluation, validation, capacity calculation, and error types. +The `core` crate contains all DynamoDB-specific business logic that is independent of the storage backend, HTTP server, and async runtime. It is pure synchronous Rust: types, expression parsing/evaluation, validation, capacity calculation, and error types. **Dependencies:** serde, serde_json, thiserror, time, uuid, base64, bigdecimal (no tokio, no sqlx, no axum, no async runtime) @@ -129,7 +128,7 @@ impl<'de> Deserialize<'de> for AttributeValue { - `BTreeMap` for Map type and `BTreeSet` for Set types ensure deterministic serialization order (important for CRC32 checksums and test reproducibility). - Number stored as `String` to preserve arbitrary precision (up to 38 digits). Numeric operations in expressions use `bigdecimal::BigDecimal` for comparison and arithmetic, which supports DynamoDB's full 38-digit precision requirement. - Binary values stored as `Vec` internally; base64 encoding/decoding happens only at the serde boundary. -- The `Null` variant carries no data — DynamoDB's `{"NULL": true}` is the only valid form, enforced by the Deserialize impl. `{"NULL": false}` must be rejected with `SerializationException`. +- The `Null` variant carries no data; DynamoDB's `{"NULL": true}` is the only valid form, enforced by the Deserialize impl. `{"NULL": false}` must be rejected with `SerializationException`. ### 3.2 Key Schema Types @@ -274,7 +273,7 @@ pub enum UpdateAction { The evaluator takes an AST and an `EvalContext` (the current item + attribute name/value maps) and returns a boolean (for conditions/filters) or a modified item (for updates). -> **Important:** `ConditionExpression` evaluation for write operations (PutItem, UpdateItem, DeleteItem) must happen inside the storage backend's transaction, not in the core handler after a round-trip. The core crate provides the `evaluate_condition` function, but the storage backend calls it within its transaction after `SELECT FOR UPDATE` to prevent TOCTOU races. `FilterExpression` evaluation (for Query/Scan) happens in the core handler after items are returned from storage — this is safe because filters are read-only. +> **Important:** `ConditionExpression` evaluation for write operations (PutItem, UpdateItem, DeleteItem) must happen inside the storage backend's transaction, not in the core handler after a round-trip. The core crate provides the `evaluate_condition` function, but the storage backend calls it within its transaction after `SELECT FOR UPDATE` to prevent TOCTOU races. `FilterExpression` evaluation (for Query/Scan) happens in the core handler after items are returned from storage; this is safe because filters are read-only. ```rust pub struct EvalContext<'a> { @@ -310,7 +309,7 @@ The `engine` crate depends on `core` (for types, expressions, validation) and `s 1. Deserialize and validate input (using `core::validation`) 2. Parse expressions (using `core::expression`) 3. Call the storage engine (async) -4. Apply post-read processing (filter expressions, projections — using `core::expression`) +4. Apply post-read processing (filter expressions and projections, using `core::expression`) 5. Calculate consumed capacity (using `core::capacity`) 6. Format the response @@ -334,7 +333,7 @@ For `UpdateItem`, the storage backend must execute the following steps inside a 10. COMMIT ``` -Both `evaluate_condition` and `apply_update` are sync functions from `core` — they operate on in-memory data and are called by the storage backend inside its transaction. +Both `evaluate_condition` and `apply_update` are sync functions from `core`; they operate on in-memory data and are called by the storage backend inside its transaction. ## 6. Validation @@ -480,6 +479,6 @@ impl DynamoDbError { Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/04-component-storage.md b/docs/design/04-component-storage.md index 413f15ccc..e64cc6844 100755 --- a/docs/design/04-component-storage.md +++ b/docs/design/04-component-storage.md @@ -1,14 +1,13 @@ --- description: "The storage traits every backend implements and the PostgreSQL implementation of them." +version: "3.0" +date: "2026-05-19" +status: "Active" +crates: "storage (traits), storage-postgres (PostgreSQL backend)" --- # Component design: Storage -**Version:** 3.0 -**Date:** 2026-05-19 -**Status:** Active -**Crates:** `storage` (traits), `storage-postgres` (PostgreSQL backend) - ## 1. Purpose The storage layer provides a trait-based abstraction for all persistent data operations. Traits are defined in the @@ -36,26 +35,26 @@ backends to implement traits incrementally, and allows consumers to depend only ExtendDB defines **13 traits** across three categories: **DynamoDB Data Path** (defined in `storage/src/lib.rs`): -- `TableEngine` — table lifecycle operations -- `DataEngine` — item CRUD, query, scan, batch, transactions -- `MetadataEngine` — TTL, tags, table statistics -- `StreamEngine` — DynamoDB Streams record storage and retrieval -- `WorkerStore` — background worker operations (control plane transitions) -- `BackupEngine` — backup and restore operations +- `TableEngine`: table lifecycle operations +- `DataEngine`: item CRUD, query, scan, batch, transactions +- `MetadataEngine`: TTL, tags, table statistics +- `StreamEngine`: DynamoDB Streams record storage and retrieval +- `WorkerStore`: background worker operations (control plane transitions) +- `BackupEngine`: backup and restore operations **Management and Operational** (defined in `storage/src/management_store.rs` and related modules): -- `ManagementStore` — auth-related CRUD (users, groups, roles, policies, access keys, accounts) -- `AdminStore` — admin user management -- `SettingsStore` — runtime settings (key-value store) -- `MetricsStore` — historical metrics persistence and query -- `RateLimitStore` — login rate limiting and account lockout -- `AuthorizationStore` — policy lookups for authorization decisions +- `ManagementStore`: auth-related CRUD (users, groups, roles, policies, access keys, accounts) +- `AdminStore`: admin user management +- `SettingsStore`: runtime settings (key-value store) +- `MetricsStore`: historical metrics persistence and query +- `RateLimitStore`: login rate limiting and account lockout +- `AuthorizationStore`: policy lookups for authorization decisions **Initialization and Lifecycle** (defined in `storage/src/bootstrapper.rs`): -- `Bootstrapper` — database initialization, migration, verification, destruction +- `Bootstrapper`: database initialization, migration, verification, destruction **Authentication** (defined in `auth/src/lib.rs`): -- `CredentialStore` — access key and session credential lookup for SigV4 verification +- `CredentialStore`: access key and session credential lookup for SigV4 verification ### 2.2 Composite Traits @@ -116,9 +115,9 @@ The `CredentialStore` trait in the `auth` crate uses `#[async_trait]` instead **TableEngine** (table lifecycle): - `create_table`, `delete_table`, `describe_table`, `list_tables`, `update_table` -- `table_key_info` — returns lightweight metadata (key schema, attribute definitions) +- `table_key_info`: returns lightweight metadata (key schema, attribute definitions) for data operations, avoiding the overhead of a full `describe_table` call -- `index_info` — returns GSI/LSI metadata for query operations +- `index_info`: returns GSI/LSI metadata for query operations All table operations are scoped by `account_id` for multi-account isolation. @@ -136,17 +135,17 @@ written atomically with data writes when `stream` is `Some`. **MetadataEngine** (TTL, tags, table statistics): - `describe_ttl`, `update_ttl` - `tag_resource`, `untag_resource`, `list_tags` -- `refresh_table_size` — updates cached table size and item count -- `create_ttl_index`, `find_expired_items_indexed` — TTL cleanup support +- `refresh_table_size`: updates cached table size and item count +- `create_ttl_index`, `find_expired_items_indexed`: TTL cleanup support **StreamEngine** (DynamoDB Streams): -- `write_stream_record` — writes stream record atomically with data write -- `get_stream_records` — retrieves stream records for a shard +- `write_stream_record`: writes stream record atomically with data write +- `get_stream_records`: retrieves stream records for a shard - `describe_stream`, `list_streams` -- `cleanup_expired_stream_records` — removes records older than 24 hours +- `cleanup_expired_stream_records`: removes records older than 24 hours **WorkerStore** (background worker operations): -- `process_control_plane_transitions` — handles table state transitions +- `process_control_plane_transitions`: handles table state transitions (CREATING → ACTIVE, DELETING → deleted) **BackupEngine** (backup and restore): @@ -248,16 +247,16 @@ pub type Item = BTreeMap; ``` `AttributeValue` is an enum representing all supported data types: -- `S(String)` — string -- `N(String)` — number (stored as string to preserve precision) -- `B(Vec)` — binary -- `SS(Vec)` — string set -- `NS(Vec)` — number set -- `BS(Vec>)` — binary set -- `L(Vec)` — list -- `M(BTreeMap)` — map -- `BOOL(bool)` — boolean -- `NULL(bool)` — null (always true) +- `S(String)`: string +- `N(String)`: number (stored as string to preserve precision) +- `B(Vec)`: binary +- `SS(Vec)`: string set +- `NS(Vec)`: number set +- `BS(Vec>)`: binary set +- `L(Vec)`: list +- `M(BTreeMap)`: map +- `BOOL(bool)`: boolean +- `NULL(bool)`: null (always true) Storage backends must preserve the exact type and value of each attribute. @@ -265,23 +264,23 @@ Storage backends must preserve the exact type and value of each attribute. **CreateTableInput**: Specifies table schema for `TableEngine::create_table`: - `table_name: String` -- `key_schema: Vec` — partition key and optional sort key -- `attribute_definitions: Vec` — types for key attributes -- `billing_mode: BillingMode` — PAY_PER_REQUEST or PROVISIONED +- `key_schema: Vec`: partition key and optional sort key +- `attribute_definitions: Vec`: types for key attributes +- `billing_mode: BillingMode`: PAY_PER_REQUEST or PROVISIONED - `global_secondary_indexes: Option>` - `local_secondary_indexes: Option>` - `stream_specification: Option` **TableDescription**: Returned by table operations, includes: - `table_name: String` -- `table_status: TableStatus` — CREATING, ACTIVE, DELETING, UPDATING +- `table_status: TableStatus`: CREATING, ACTIVE, DELETING, UPDATING - `table_arn: String` -- `table_id: String` — backend-specific unique identifier +- `table_id: String`: backend-specific unique identifier - `key_schema: Vec` - `attribute_definitions: Vec` - `table_size_bytes: i64` - `item_count: i64` -- `creation_date_time: f64` — Unix timestamp +- `creation_date_time: f64`: Unix timestamp **TableKeyInfo**: Lightweight metadata for data operations: - `account_id: String` @@ -298,12 +297,12 @@ Storage backends must preserve the exact type and value of each attribute. - Storage backends call `extenddb_core::expression::evaluate()` to evaluate conditions **ExpressionMaps**: Name and value substitutions for expressions: -- `names: HashMap` — maps `#name` placeholders to attribute names -- `values: HashMap` — maps `:value` placeholders to values +- `names: HashMap`: maps `#name` placeholders to attribute names +- `values: HashMap`: maps `:value` placeholders to values **KeyCondition**: Parsed key condition for Query operations: -- `partition_key: (String, AttributeValue)` — partition key name and value -- `sort_key_condition: Option` — optional sort key condition +- `partition_key: (String, AttributeValue)`: partition key name and value +- `sort_key_condition: Option`: optional sort key condition **UpdateAction**: Parsed update expression action: - `SET`, `REMOVE`, `ADD`, `DELETE` operations @@ -313,15 +312,15 @@ Storage backends must preserve the exact type and value of each attribute. **StreamRecord**: Complete stream record for persistence: - `event_id: String` -- `event_name: StreamEventName` — INSERT, MODIFY, REMOVE +- `event_name: StreamEventName`: INSERT, MODIFY, REMOVE - `event_version: String` - `event_source: String` - `aws_region: String` -- `dynamodb: StreamRecordData` — keys, old_image, new_image, size_bytes +- `dynamodb: StreamRecordData`: keys, old_image, new_image, size_bytes **StreamCapture**: Metadata for constructing stream records inside transactions: -- `view_type: StreamViewType` — KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, NEW_AND_OLD_IMAGES -- `user_identity: Option` — set for TTL-originated deletions +- `view_type: StreamViewType`: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, NEW_AND_OLD_IMAGES +- `user_identity: Option`: set for TTL-originated deletions - `region: Arc` Storage backends write stream records atomically with data writes when `stream` is `Some`. @@ -329,15 +328,15 @@ Storage backends write stream records atomically with data writes when `stream` ### Error Types **StorageError**: Errors returned by storage trait methods: -- `TableNotFound(String)` — table does not exist -- `TableAlreadyExists(String)` — table already exists -- `TableNotActive(String)` — table is CREATING, DELETING, or UPDATING -- `ConditionFailed { old_item: Option }` — condition expression evaluated to false -- `TransactionCanceled(Vec)` — transaction failed with per-item reasons -- `IdempotentReplay` — idempotency token matched previous request -- `IdempotentMismatch` — idempotency token exists with different operations -- `IndexNotFound` — secondary index does not exist -- `Internal(String)` — backend-specific error +- `TableNotFound(String)`: table does not exist +- `TableAlreadyExists(String)`: table already exists +- `TableNotActive(String)`: table is CREATING, DELETING, or UPDATING +- `ConditionFailed { old_item: Option }`: condition expression evaluated to false +- `TransactionCanceled(Vec)`: transaction failed with per-item reasons +- `IdempotentReplay`: idempotency token matched previous request +- `IdempotentMismatch`: idempotency token exists with different operations +- `IndexNotFound`: secondary index does not exist +- `Internal(String)`: backend-specific error The `engine` crate maps `StorageError` to wire protocol error responses. @@ -348,22 +347,22 @@ The `engine` crate maps `StorageError` to wire protocol error responses. The PostgreSQL backend uses two categories of tables: **Catalog tables** (metadata, created by migrations): -- `tables` — DynamoDB table metadata (key schema, status, ARN, table_id) -- `indexes` — GSI/LSI metadata (key schema, projection, status, index_id) -- `tags` — resource tags -- `_dynamodb_credentials` — access keys and session tokens -- `_dynamodb_users`, `_dynamodb_roles`, `_dynamodb_groups` — IAM-type entities -- `_dynamodb_group_members` — group membership -- `_dynamodb_principal_tags` — user/role tags -- `_dynamodb_policies` — IAM-type policy documents -- `_dynamodb_sessions` — temporary role session credentials -- `_dynamodb_stream_records` — DynamoDB Streams records -- `_dynamodb_import_jobs`, `_dynamodb_export_jobs` — import/export tracking -- `_dynamodb_idempotency_tokens` — TransactWriteItems idempotency +- `tables`: DynamoDB table metadata (key schema, status, ARN, table_id) +- `indexes`: GSI/LSI metadata (key schema, projection, status, index_id) +- `tags`: resource tags +- `_dynamodb_credentials`: access keys and session tokens +- `_dynamodb_users`, `_dynamodb_roles`, `_dynamodb_groups`: IAM-type entities +- `_dynamodb_group_members`: group membership +- `_dynamodb_principal_tags`: user/role tags +- `_dynamodb_policies`: IAM-type policy documents +- `_dynamodb_sessions`: temporary role session credentials +- `_dynamodb_stream_records`: DynamoDB Streams records +- `_dynamodb_import_jobs`, `_dynamodb_export_jobs`: import/export tracking +- `_dynamodb_idempotency_tokens`: TransactWriteItems idempotency **Data tables** (created dynamically per DynamoDB table): -- `_ddb_` — base table (table_id is a UUID) -- `_ddb_` — GSI/LSI projection tables (index_id is a UUID) +- `_ddb_`: base table (table_id is a UUID) +- `_ddb_`: GSI/LSI projection tables (index_id is a UUID) **Schema files:** - Catalog schema: `crates/storage-postgres/migrations/001_schema.sql` @@ -412,8 +411,8 @@ The PostgreSQL backend uses two categories of tables: - **GSI tables**: GSI tables include base table primary key columns (`base_pk`, `base_sk_*`) as actual SQL columns (not just inside `item_data` JSONB). This - is required because: (1) GSI keys are not unique — two base table items can - project to the same GSI key, so the base table PK is needed for uniqueness; + is required because: (1) GSI keys are not unique (two base table items can + project to the same GSI key), so the base table PK is needed for uniqueness; (2) pagination requires a tiebreaker when GSI keys collide; (3) the base table PK is needed to look up the full item for projections. @@ -483,7 +482,7 @@ parameter is accepted and correctly reflected in capacity calculations (eventual **Read replica mode (`read_replica_url` configured):** Eventually consistent reads (`consistent_read=false`) are routed to a PostgreSQL streaming replica that is naturally a few milliseconds behind the primary. Strongly consistent reads (`consistent_read=true`) always read from the primary. This mirrors how DynamoDB achieves eventual consistency -— via storage node replicas — and surfaces the exact class of bugs that applications may encounter in production +(via storage node replicas) and surfaces the exact class of bugs that applications may encounter in production DynamoDB (e.g., read-after-write without `ConsistentRead=true` returning stale data). **Which operations are affected:** @@ -491,7 +490,7 @@ DynamoDB (e.g., read-after-write without `ConsistentRead=true` returning stale d - `Query`: uses `consistent_read` field (default `false` in DynamoDB) - `Scan`: uses `consistent_read` field (default `false` in DynamoDB) - `BatchGetItem`: uses per-table `consistent_read` field -- `TransactGetItems`: always strongly consistent (DynamoDB spec — serializable isolation) +- `TransactGetItems`: always strongly consistent (DynamoDB spec: serializable isolation) - All write operations: always use the primary pool The `read_pool()` helper method on `PostgresEngine` encapsulates this routing. All read implementations call @@ -613,8 +612,8 @@ All data plane operations (PutItem, GetItem, Query, etc.) must check `table_stat not `ACTIVE`, return `StorageError::TableNotActive` (mapped to `ResourceInUseException`). Control plane operations that modify the table (`UpdateTable`, `DeleteTable`) must: -1. Atomically set `table_status` to `UPDATING` or `DELETING` (using `UPDATE ... WHERE table_status = 'ACTIVE'` — if - zero rows affected, the table is already being modified, return `ResourceInUseException`) +1. Atomically set `table_status` to `UPDATING` or `DELETING` (using `UPDATE ... WHERE table_status = 'ACTIVE'`: if + zero rows are affected, the table is already being modified, so return `ResourceInUseException`) 2. Perform the operation 3. Set `table_status` back to `ACTIVE` (or remove the row for `DeleteTable`) @@ -622,8 +621,8 @@ This prevents concurrent DDL operations on the same table. ### 6.1.1 Async Control Plane Transitions (Phase 1c) -Real DynamoDB control plane operations are not instantaneous — `CreateTable` returns `CREATING` status and the table -transitions to `ACTIVE` asynchronously. extenddb emulates this behavior. +Real DynamoDB control plane operations are not instantaneous: `CreateTable` returns `CREATING` status and the table +transitions to `ACTIVE` asynchronously. ExtendDB emulates this behavior. **Implementation:** @@ -654,13 +653,13 @@ on the `indexes` table when GSI operations are implemented. DynamoDB emulation (Phase 2). - Startup recovery will reset stuck `CREATING` tables to `NOW() + random[5, 20]` instead of instant activation (Phase 2). -- `control_plane_delay_seconds` is a runtime setting (0–300 range), managed +- `control_plane_delay_seconds` is a runtime setting (0-300 range), managed via `extenddb settings set`. It is not a `.toml` config key. **Crash recovery and in-flight operation tracking:** The `status_transition_at` column on the `tables` table serves as the -in-flight operation tracker. When the extenddb server shuts down (cleanly or +in-flight operation tracker. When the ExtendDB server shuts down (cleanly or via crash) while tables have pending transitions, the state is durable in PostgreSQL. On the next startup, `process_control_plane_transitions()` scans for rows where `status_transition_at IS NOT NULL AND @@ -1046,12 +1045,12 @@ pub struct WorkerContext { **Worker classification:** **Backend-agnostic workers** (spawned in `cmd_serve`, use trait methods only): -1. `poll_log_level` — uses `SettingsStore::get_setting` -2. `poll_throttling_enabled` — uses `SettingsStore::get_setting` -3. `metrics_prune_worker` — uses in-memory `MetricsCollector` -4. `metrics_flush_worker` — uses `MetricsStore::insert_metrics` -5. `capacity_warning_worker` — uses in-memory metrics -6. `login_attempt_cleanup_worker` — uses `RateLimitStore::cleanup_old_attempts` +1. `poll_log_level`: uses `SettingsStore::get_setting` +2. `poll_throttling_enabled`: uses `SettingsStore::get_setting` +3. `metrics_prune_worker`: uses in-memory `MetricsCollector` +4. `metrics_flush_worker`: uses `MetricsStore::insert_metrics` +5. `capacity_warning_worker`: uses in-memory metrics +6. `login_attempt_cleanup_worker`: uses `RateLimitStore::cleanup_old_attempts` **Backend-specific workers** (spawned via `RuntimeHooks`, access backend internals): @@ -1398,6 +1397,6 @@ backups). Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/05-component-auth-v3.md b/docs/design/05-component-auth-v3.md index 3ce0ae464..a2234cf5d 100755 --- a/docs/design/05-component-auth-v3.md +++ b/docs/design/05-component-auth-v3.md @@ -1,22 +1,23 @@ --- description: "The third revision of the auth crate: local credential management, policy evaluation and the IAM data model." +version: "3.0" +date: "2026-04-15" +status: "Approved (interactive review session 2026-04-15)" +crate: "extenddb-auth" +supersedes: "05-component-auth.md (v2.0)" --- # Component design: Authentication and authorization (v3) -**Version:** 3.0 -**Date:** 2026-04-15 -**Status:** Approved (interactive review session 2026-04-15) -**Crate:** `extenddb-auth` -**Supersedes:** 05-component-auth.md (v2.0) +This document supersedes the [second revision](05-component-auth.md) of the authentication design. ## 1. Purpose -The `auth` crate provides pluggable authentication and authorization for extenddb. It defines the `AuthProvider` trait, implements the built-in SigV4 provider with local credential management, and contains the IAM policy evaluation engine supporting identity-based access control (IBAC), role-based access control (RBAC), and fine-grained/tag-based access control (ABAC/FGAC). +The `auth` crate provides pluggable authentication and authorization for ExtendDB. It defines the `AuthProvider` trait, implements the built-in SigV4 provider with local credential management, and contains the IAM policy evaluation engine supporting identity-based access control (IBAC), role-based access control (RBAC), and fine-grained/tag-based access control (ABAC/FGAC). The crate depends on `extenddb-core` (for types and errors) and `async_trait` (for object-safe async trait dispatch). It has no HTTP framework or storage dependencies. -> **No caching.** All credential lookups, policy fetches, and identity resolution read directly from Postgres on every request. The latency is sub-millisecond for indexed lookups, and this approach eliminates stale-cache bugs and cache invalidation complexity entirely. Multiple extenddb instances sharing the same catalog see consistent data without coordination. If profiling under production load shows database round-trips are a bottleneck, caching can be added as a transparent layer without changing any interfaces. +> **No caching.** All credential lookups, policy fetches, and identity resolution read directly from Postgres on every request. The latency is sub-millisecond for indexed lookups, and this approach eliminates stale-cache bugs and cache invalidation complexity entirely. Multiple ExtendDB instances sharing the same catalog see consistent data without coordination. If profiling under production load shows database round-trips are a bottleneck, caching can be added as a transparent layer without changing any interfaces. ## 2. Key Design Decisions (from 2026-04-15 review session) @@ -30,7 +31,7 @@ The crate depends on `extenddb-core` (for types and errors) and `async_trait` (f 5. **Multi-account support.** The catalog schema is scoped by `account_id`. Different accounts can have tables with the same name. The `account_id` is resolved from the authenticated identity, not from config. -6. **Auth provider is config-only.** Set in `extenddb.toml` as `auth.provider = "builtin"`. The `"none"` value is no longer accepted — the server refuses to start with it. Cannot be changed at runtime via settings. +6. **Auth provider is config-only.** Set in `extenddb.toml` as `auth.provider = "builtin"`. The `"none"` value is no longer accepted; the server refuses to start with it. Cannot be changed at runtime via settings. 7. **No catalog migration.** `extenddb init` is required. The schema includes auth tables from the start. @@ -58,7 +59,7 @@ JWT-based auth with Azure AD group-to-role mapping. Design unchanged from v2.0 ## 4. Admin User Model -Admin users are organizational administrators who manage accounts, IAM users, roles, groups, and policies. They are separate from IAM users — they do not make DynamoDB requests. +Admin users are organizational administrators who manage accounts, IAM users, roles, groups, and policies. They are separate from IAM users; they do not make DynamoDB requests. ### 4.1 Bootstrap @@ -305,9 +306,9 @@ Per-table Postgres tables change from `_ddb_{table_name}` to `_ddb_{account_id}_ ### 6.3 Indexes, Streams, Tags -- `indexes` FK is on `table_id` (UUID) — no change needed, already account-independent. -- `stream_shards` and `stream_records` reference `table_name` — must add `account_id` or switch to `table_id` FK. -- `tags` keyed by `resource_arn` — ARNs embed account_id, so no change needed. +- `indexes` FK is on `table_id` (UUID): no change needed, already account-independent. +- `stream_shards` and `stream_records` reference `table_name`: must add `account_id` or switch to `table_id` FK. +- `tags` keyed by `resource_arn`: ARNs embed account_id, so no change needed. ### 6.4 Account ID Flow @@ -352,10 +353,10 @@ Unchanged from v2.0 §6. The engine evaluates all three access control patterns 1. Collect applicable policies (user + group policies, or role policies) 2. Collect permissions boundary (if set) 3. Collect session policy (if AssumeRole with inline policy) -4. **Explicit Deny scan** — any Deny statement that matches → DENY -5. **Permissions boundary check** — must find Allow in boundary → else DENY -6. **Session policy check** — must find Allow in session policy → else DENY -7. **Identity policy Allow** — find Allow in identity policies → ALLOW +4. **Explicit Deny scan**: any Deny statement that matches → DENY +5. **Permissions boundary check**: must find Allow in boundary → else DENY +6. **Session policy check**: must find Allow in session policy → else DENY +7. **Identity policy Allow**: find Allow in identity policies → ALLOW 8. **Implicit Deny** → DENY ### 8.2 Condition Operators @@ -364,17 +365,17 @@ Full set from v2.0 §6.5: StringEquals, StringLike, NumericEquals, DateEquals, B ### 8.3 DynamoDB Condition Keys -- `aws:PrincipalTag/*` — from authenticated identity's tags +- `aws:PrincipalTag/*`: from authenticated identity's tags - `aws:ResourceTag/*`, from target table's tags -- `dynamodb:LeadingKeys` — partition key values being accessed -- `dynamodb:Attributes` — attribute names being read/written +- `dynamodb:LeadingKeys`: partition key values being accessed +- `dynamodb:Attributes`: attribute names being read/written - `dynamodb:Select`, `dynamodb:ReturnValues`, `dynamodb:ReturnConsumedCapacity` -- `dynamodb:FullTableScan` — true for Scan operations -- `dynamodb:EnclosingOperation` — parent operation for batch/transact sub-ops +- `dynamodb:FullTableScan`: true for Scan operations +- `dynamodb:EnclosingOperation`: parent operation for batch/transact sub-ops ### 8.4 Management API Policy Evaluation -Management API calls are treated as IAM actions (`iam:CreateUser`, `iam:PutUserPolicy`, etc.) and evaluated against the caller's policies using the same engine. This is how admin vs IAM user permissions are enforced — via default policies, not hard-coded logic. +Management API calls are treated as IAM actions (`iam:CreateUser`, `iam:PutUserPolicy`, etc.) and evaluated against the caller's policies using the same engine. This is how admin vs IAM user permissions are enforced: via default policies, not hard-coded logic. The only exception is the bootstrap admin created by `extenddb init`: if no policies exist for this principal, the system allows the operation. Once the seed policy is in place, normal evaluation takes over. @@ -418,43 +419,43 @@ INSERT INTO settings (key, value) VALUES ('encryption_key', ' **Deferred: Policy Variables.** AWS IAM supports variable substitution in `Resource` ARNs and `Condition` values — e.g., `"Resource": "arn:aws:dynamodb:*:*:table/${aws:PrincipalTag/Team}-*"` grants access to tables matching the caller's own team tag. extenddb v1 does not support policy variables; all Resource and Condition values are treated as literal strings (with wildcard matching where applicable). This means ABAC policies that use variables to generalize across principals must be written as separate per-principal policies. Policy variable support is tracked as REQ-ABAC-006. +> **Deferred: Policy Variables.** AWS IAM supports variable substitution in `Resource` ARNs and `Condition` values, e.g., `"Resource": "arn:aws:dynamodb:*:*:table/${aws:PrincipalTag/Team}-*"` grants access to tables matching the caller's own team tag. ExtendDB v1 does not support policy variables; all Resource and Condition values are treated as literal strings (with wildcard matching where applicable). This means ABAC policies that use variables to generalize across principals must be written as separate per-principal policies. Policy variable support is tracked as REQ-ABAC-006. **JSON deserialization:** IAM policy documents are JSON. Both Mode 2 (management API) and Mode 3 (IAM retrieval) receive policies as JSON strings. Deserialization uses `serde_json` with the following rules: - `Effect`: case-insensitive match to `"Allow"` or `"Deny"`. Any other value is a parse error. -- `Action` / `NotAction`: a single string or array of strings. Mutually exclusive — a statement with both is a parse error. Mapped to `ActionMatch::Actions` or `ActionMatch::NotActions`. +- `Action` / `NotAction`: a single string or array of strings. Mutually exclusive: a statement with both is a parse error. Mapped to `ActionMatch::Actions` or `ActionMatch::NotActions`. - `Resource` / `NotResource`: same pattern as Action. Mapped to `ResourceMatch`. - `Principal` / `NotPrincipal`: same pattern. Only present in trust policies; absent in identity-based policies. Mapped to `Option`. - `Condition`: a nested object `{ "OperatorName": { "key": "value" | ["values"] } }`. The operator name is parsed by splitting on `:` to detect `ForAllValues:` / `ForAnyValue:` prefixes and `IfExists` suffix, then matching the base operator name (e.g., `StringEquals`, `ArnLike`). Unknown operator names are a parse error. @@ -556,11 +559,11 @@ pub trait ConditionContext { condition key as single-valued or multivalued (`ArrayOfString`) in the service authorization reference. For DynamoDB, `dynamodb:Attributes`, `dynamodb:LeadingKeys` and `aws:TagKeys` are multivalued; everything else -extenddb resolves (`dynamodb:Select`, `dynamodb:ReturnValues`, +ExtendDB resolves (`dynamodb:Select`, `dynamodb:ReturnValues`, `aws:RequestTag/`, etc.) is single-valued. A **bare** operator (one without a `ForAllValues:`/`ForAnyValue:` qualifier) -applied to a multivalued key **never matches** — verified against real AWS IAM -(BR-7085), and independent of how many values are present. Only the set-operator +applied to a multivalued key **never matches**. This is verified against AWS IAM +and is independent of how many values are present. Only the set-operator qualifiers evaluate multivalued keys. This is fail-safe: a bare `Deny` on such a key is a no-op (the author must use `ForAnyValue:` to build a denylist), and a bare `Allow` allowlist grants nothing (implicit deny). @@ -754,7 +757,7 @@ pub fn evaluate_condition(condition: &Condition, context: &impl ConditionContext ConditionOperator::IfExists(inner) => { match context_values { None => true, // key absent → condition passes - // bare operator on a multivalued key never matches (BR-7085) + // bare operator on a multivalued key never matches Some(_) if is_multivalued => false, Some(vals) => evaluate_single_value_condition(inner, &vals, &condition.values) } @@ -762,7 +765,7 @@ pub fn evaluate_condition(condition: &Condition, context: &impl ConditionContext other => { match context_values { None => false, // key absent → condition fails (unless IfExists) - // bare operator on a multivalued key never matches (BR-7085) + // bare operator on a multivalued key never matches Some(_) if is_multivalued => false, Some(vals) => evaluate_single_value_condition(other, &vals, &condition.values) } @@ -868,7 +871,7 @@ pub fn arn_match(pattern: &str, value: &str) -> bool; ## 7. Role Assumption (AssumeRole) -extenddb provides a local AssumeRole mechanism for testing role-based access patterns. This is not exposed as an STS API — it's a extenddb management API. +ExtendDB provides a local AssumeRole mechanism for testing role-based access patterns. This is not exposed as an STS API; it's an ExtendDB management API. ### 7.1 AssumeRole Flow @@ -954,12 +957,12 @@ a trust policy are rejected, and those it accepts (a missing `Version`, a single-object `Statement`, numeric condition values, unknown condition keys) are accepted. -> **Note:** Trust policies use `"Action": "sts:AssumeRole"` for compatibility with real IAM policy documents, even though extenddb exposes AssumeRole via `/management/assume-role` rather than an STS endpoint. The policy engine matches this action string during trust policy evaluation regardless of the actual HTTP endpoint used. +> **Note:** Trust policies use `"Action": "sts:AssumeRole"` for compatibility with real IAM policy documents, even though ExtendDB exposes AssumeRole via `/management/assume-role` rather than an STS endpoint. The policy engine matches this action string during trust policy evaluation regardless of the actual HTTP endpoint used. Trust policy conditions are evaluated against an `AssumeRoleContext` (not the DynamoDB `RequestContext`), which contains: -- `aws:PrincipalTag/*` — tags of the calling user (the entity assuming the role) -- `aws:RequestTag/*` — session tags passed in the AssumeRole request -- `sts:ExternalId` — the ExternalId parameter (if provided in the AssumeRole request) +- `aws:PrincipalTag/*`: tags of the calling user (the entity assuming the role) +- `aws:RequestTag/*`: session tags passed in the AssumeRole request +- `sts:ExternalId`: the ExternalId parameter (if provided in the AssumeRole request) ```rust /// Context for trust policy condition evaluation. @@ -988,10 +991,10 @@ impl ConditionContext for AssumeRoleContext { ## 8. Auth Providers -### 8.1 No-Auth Provider (Mode 1) — REMOVED +### 8.1 No-Auth Provider (Mode 1, removed) > **Historical note (P63b):** The `NoopAuthProvider` and `AuthIdentity::Anonymous` -> variant were removed in v0.0.67. Authentication is now mandatory — the server +> variant were removed in v0.0.67. Authentication is now mandatory; the server > refuses to start with `auth.provider = "none"`. The code below is preserved > for historical reference only. @@ -1083,7 +1086,7 @@ impl AuthProvider for BuiltinAuthProvider { ### 8.3 AWS IAM Provider (Mode 3) -When extenddb runs in an environment with AWS connectivity (EC2, ECS, Lambda, corporate network with VPN), the AWS IAM provider delegates authentication to STS and retrieves real IAM policies for local evaluation. This gives full IAM fidelity without maintaining a local credential store. +When ExtendDB runs in an environment with AWS connectivity (EC2, ECS, Lambda, corporate network with VPN), the AWS IAM provider delegates authentication to STS and retrieves real IAM policies for local evaluation. This gives full IAM fidelity without maintaining a local credential store. #### 8.3.1 Architecture @@ -1122,15 +1125,15 @@ Client (with extenddb wrapper) └─────────────────────────────────────────────┘ ``` -This is the same pattern used by EKS (`aws-iam-authenticator`) and Vault's AWS auth method. The client generates a pre-signed STS `GetCallerIdentity` request using its own credentials (signed for the `sts` service), and extenddb calls STS with that pre-signed request to validate the caller's identity. +This is the same pattern used by EKS (`aws-iam-authenticator`) and Vault's AWS auth method. The client generates a pre-signed STS `GetCallerIdentity` request using its own credentials (signed for the `sts` service), and ExtendDB calls STS with that pre-signed request to validate the caller's identity. -> **Why pre-signed tokens instead of header forwarding?** SigV4 signatures are bound to the service name, host, and request body they were generated for. A request signed for `dynamodb` / `localhost:18443` cannot be replayed against `sts` / `sts.amazonaws.com` — STS will reject the signature because the credential scope, signed host, and body hash all mismatch. Pre-signed URLs solve this: the client signs a separate request specifically for STS, and extenddb uses that to authenticate. +> **Why pre-signed tokens instead of header forwarding?** SigV4 signatures are bound to the service name, host, and request body they were generated for. A request signed for `dynamodb` / `localhost:18443` cannot be replayed against `sts` / `sts.amazonaws.com`: STS will reject the signature because the credential scope, signed host, and body hash all mismatch. Pre-signed URLs solve this: the client signs a separate request specifically for STS, and ExtendDB uses that to authenticate. #### 8.3.2 Authentication: Pre-Signed STS GetCallerIdentity STS `GetCallerIdentity` is unique among AWS APIs: it validates the caller's credentials and returns their identity without requiring any IAM permissions on the caller. Any valid AWS credential can call it. -**Client-side token generation (performed by the extenddb wrapper):** +**Client-side token generation (performed by the ExtendDB wrapper):** ``` 1. Using the caller's AWS credentials, generate a pre-signed URL for: @@ -1238,7 +1241,7 @@ struct CachedPolicies { #### 8.3.3 Authorization: IAM Policy Retrieval -After authentication, extenddb needs the caller's policies to make authorization decisions. Rather than trusting IAM blindly (which would mean no fine-grained access control), extenddb fetches the actual policies and evaluates them locally using the same policy engine as Mode 2. +After authentication, ExtendDB needs the caller's policies to make authorization decisions. Rather than trusting IAM blindly (which would mean no fine-grained access control), ExtendDB fetches the actual policies and evaluates them locally using the same policy engine as Mode 2. **Policy retrieval flow for a User principal:** @@ -1312,9 +1315,9 @@ Tags are fetched alongside policies and cached together. These populate aws:PrincipalTag/* for ABAC condition evaluation. ``` -#### 8.3.4 IAM Permissions Required by extenddb Host +#### 8.3.4 IAM Permissions Required by ExtendDB Host -The EC2 instance (or IAM role) running extenddb in Mode 3 needs the following permissions: +The EC2 instance (or IAM role) running ExtendDB in Mode 3 needs the following permissions: ```json { @@ -1347,25 +1350,25 @@ The EC2 instance (or IAM role) running extenddb in Mode 3 needs the following pe } ``` -> `sts:GetCallerIdentity` requires no permissions — any valid AWS credential can call it. +> `sts:GetCallerIdentity` requires no permissions: any valid AWS credential can call it. #### 8.3.5 Known Limitations (Mode 3) -1. **Session policies are not enforceable.** When a caller assumed a role with an inline session policy, that policy is not retrievable from IAM after the fact. extenddb evaluates only the role's attached policies and permissions boundary. This means Mode 3 is slightly more permissive than real DynamoDB for callers using session policies. +1. **Session policies are not enforceable.** When a caller assumed a role with an inline session policy, that policy is not retrievable from IAM after the fact. ExtendDB evaluates only the role's attached policies and permissions boundary. This means Mode 3 is slightly more permissive than real DynamoDB for callers using session policies. 2. **Resource-based policies are not evaluated.** Consistent with the overall deferral of resource-based policies (see Non-Goals table). -3. **Service control policies (SCPs) are not evaluated.** SCPs are an Organizations-level feature. extenddb does not query Organizations APIs. +3. **Service control policies (SCPs) are not evaluated.** SCPs are an Organizations-level feature. ExtendDB does not query Organizations APIs. 4. **Policy variable substitution is not supported.** Consistent with REQ-ABAC-006 deferral. Fetched policies containing `${aws:PrincipalTag/key}` will treat the variable syntax as a literal string. -5. **Cross-account access is not evaluated.** extenddb assumes all callers are in the same account. Cross-account role assumption and resource policies are not supported. +5. **Cross-account access is not evaluated.** ExtendDB assumes all callers are in the same account. Cross-account role assumption and resource policies are not supported. -6. **Latency on cache miss.** Policy retrieval for a user with multiple groups and managed policies can require 10+ IAM API calls. With caching (default 300s TTL), this only happens once per principal per TTL window. First-request latency for a new principal may be 500ms–2s. +6. **Latency on cache miss.** Policy retrieval for a user with multiple groups and managed policies can require 10+ IAM API calls. With caching (default 300s TTL), this only happens once per principal per TTL window. First-request latency for a new principal may be 500ms-2s. 7. **Identity cache eventual consistency.** If a user rotates their access key, the old key's identity remains cached for up to one `identity_cache_ttl_seconds` window. During this window, a pre-signed token generated with the old key (before deletion) that hasn't expired could still produce a cache hit. In practice, the short token expiry (60s) and short default identity cache TTL (60s) limit this window. For security-sensitive deployments, lower the identity cache TTL. -8. **Policy cache eventual consistency.** If an admin detaches or modifies a policy in IAM, extenddb continues using the cached (now-stale) policies for up to `policy_cache_ttl_seconds`. This is analogous to real DynamoDB's IAM propagation delay (typically seconds). This is a conscious design choice — the TTL trades freshness for reduced IAM API call volume. +8. **Policy cache eventual consistency.** If an admin detaches or modifies a policy in IAM, ExtendDB continues using the cached (now-stale) policies for up to `policy_cache_ttl_seconds`. This is analogous to real DynamoDB's IAM propagation delay (typically seconds). This is a conscious design choice: the TTL trades freshness for reduced IAM API call volume. 9. **Client wrapper required.** Unlike Modes 1 and 2, Mode 3 requires a thin client-side wrapper to generate the pre-signed STS token. The wrapper is provided for Python, Rust, and Java. Applications that cannot use the wrapper cannot use Mode 3. @@ -1527,7 +1530,7 @@ impl AuthProvider for AwsIamProvider { } ``` -### 8.4 Azure AD Provider (Sketch — Deferred) +### 8.4 Azure AD Provider (Sketch, Deferred) ```rust pub struct AzureAdProvider { @@ -1614,6 +1617,6 @@ pub fn decrypt_secret(ciphertext: &[u8], key: &[u8; 32]) -> Result, Auth Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/06-component-server.md b/docs/design/06-component-server.md index d5e8fe2cd..58ad15429 100755 --- a/docs/design/06-component-server.md +++ b/docs/design/06-component-server.md @@ -1,14 +1,13 @@ --- description: "The HTTP server: routing, the middleware pipeline, response formatting and how the crates are wired into a service." +version: "1.0" +date: "2026-04-03" +status: "Draft" +crate: "dynamodb-server" --- # Component design: HTTP server -**Version:** 1.0 -**Date:** 2026-04-03 -**Status:** Draft -**Crate:** `dynamodb-server` - ## 1. Purpose The `server` crate provides the HTTP server, request routing, middleware pipeline, and response formatting. It wires together the `core`, `auth`, and `storage` crates into a running service. Built on axum + tower for composable async middleware. @@ -41,9 +40,9 @@ crates/server/src/ `AppState` carries a generic `S: StorageEngine` while `AuthProvider` is erased behind `Arc`. This asymmetry is intentional: -- **Storage uses static dispatch (generic `S`).** The storage engine is on the hot path — every request makes 1-3 storage calls. RPITIT on the storage traits avoids a `Box` heap allocation per call. In absolute terms this saves ~50ns per call against a PostgreSQL round-trip of 0.5-5ms, so it's not measurable today. But this is a database compatibility layer where we control the full stack — there's no reason to leave performance on the table, and the ergonomic cost is small (the generic appears on ~6 function signatures, all in `server` and `engine`; it never leaks into `core`, `auth`, or middleware). +- **Storage uses static dispatch (generic `S`).** The storage engine is on the hot path: every request makes 1-3 storage calls. RPITIT on the storage traits avoids a `Box` heap allocation per call. In absolute terms this saves ~50ns per call against a PostgreSQL round-trip of 0.5-5ms, so it's not measurable today. But this is a database compatibility layer where we control the full stack; there's no reason to leave performance on the table, and the ergonomic cost is small (the generic appears on ~6 function signatures, all in `server` and `engine`; it never leaks into `core`, `auth`, or middleware). - **Auth uses dynamic dispatch (`Arc`).** Auth is called once per request, involves HMAC-SHA256 crypto that dwarfs any vtable cost, and benefits from runtime pluggability (swap providers via config without recompilation). -- **The generic does NOT propagate into middleware.** `RequestIdLayer`, `LoggingLayer`, `MetricsLayer`, `RequestSizeLayer`, `CompressionLayer` — none of these touch storage. Only the main request handler and `OperationContext` carry the generic. +- **The generic does NOT propagate into middleware.** `RequestIdLayer`, `LoggingLayer`, `MetricsLayer`, `RequestSizeLayer`, `CompressionLayer`: none of these touch storage. Only the main request handler and `OperationContext` carry the generic. - **Testing:** Test code needs a concrete `S` type. Use a `MockStorage` struct that implements all storage sub-traits. This is more verbose than `Arc` mocks but straightforward with hand-written mocks or `mockall`. - **If this decision is revisited:** The simplest migration path is to add `#[async_trait]` to the storage traits and replace `Arc` with `Arc`. This removes all generics from the server/engine crates at the cost of one heap allocation per storage call. The change is mechanical and confined to trait definitions + `AppState`. @@ -270,7 +269,7 @@ impl Layer for CompressionLayer { } ``` -The CRC32 header is computed on the uncompressed body (matching DynamoDB behavior — SDKs validate CRC32 after decompression). +The CRC32 header is computed on the uncompressed body (matching DynamoDB behavior; SDKs validate CRC32 after decompression). ## 6. Middleware Pipeline @@ -376,7 +375,7 @@ convert from this JSON externally. ## 8. Management API -The management API provides identity and credential management for extenddb's built-in auth provider. These endpoints are the only way to create users, roles, groups, credentials, and policies — the DynamoDB API surface is not used for identity management. +The management API provides identity and credential management for ExtendDB's built-in auth provider. These endpoints are the only way to create users, roles, groups, credentials, and policies; the DynamoDB API surface is not used for identity management. ### 8.1 Authentication Model @@ -388,11 +387,11 @@ All management routes use `POST /management/` with JSON request/response | Route | Description | Storage method(s) | Cache invalidation | |-------|-------------|-------------------|-------------------| -| `POST /management/create-user` | Create an IAM user | `IdentityEngine::create_user` | — | +| `POST /management/create-user` | Create an IAM user | `IdentityEngine::create_user` | none | | `POST /management/delete-user` | Delete an IAM user | `IdentityEngine::delete_user` | `invalidate_user`, `invalidate_policies` (user ARN + all group ARNs) | -| `POST /management/create-role` | Create an IAM role with trust policy | `IdentityEngine::create_role` | — | +| `POST /management/create-role` | Create an IAM role with trust policy | `IdentityEngine::create_role` | none | | `POST /management/delete-role` | Delete an IAM role | `IdentityEngine::delete_role` | `invalidate_role`, `invalidate_policies` | -| `POST /management/create-group` | Create an IAM group | `IdentityEngine::create_group` | — | +| `POST /management/create-group` | Create an IAM group | `IdentityEngine::create_group` | none | | `POST /management/delete-group` | Delete an IAM group | `IdentityEngine::delete_group` | `invalidate_policies` (group ARN) | | `POST /management/add-user-to-group` | Add a user to a group | `IdentityEngine::add_user_to_group` | `invalidate_user`, `invalidate_policies` (user ARN) | | `POST /management/remove-user-from-group` | Remove a user from a group | `IdentityEngine::remove_user_from_group` | `invalidate_user`, `invalidate_policies` (user ARN) | @@ -400,9 +399,9 @@ All management routes use `POST /management/` with JSON request/response | `POST /management/detach-policy` | Detach a policy from a principal | `IdentityEngine::detach_policy` | `invalidate_policies` (principal ARN) | | `POST /management/set-permissions-boundary` | Set or clear a permissions boundary | `IdentityEngine::set_permissions_boundary` | `invalidate_boundary` | | `POST /management/set-principal-tags` | Set tags on a user or role | `IdentityEngine::set_user_tags` or `set_role_tags` | `invalidate_user` or `invalidate_role` | -| `POST /management/store-credential` | Create an access key for a user | `CredentialEngine::store_credential` | — | +| `POST /management/store-credential` | Create an access key for a user | `CredentialEngine::store_credential` | none | | `POST /management/deactivate-credential` | Deactivate an access key | `CredentialEngine::deactivate_credential` | `invalidate_credential` | -| `POST /management/assume-role` | Assume a role, get temporary credentials | Load role via `IdentityEngine::get_role` → evaluate trust policy via `evaluate_trust_policy` with `AssumeRoleContext` → `SessionEngine::create_session` + `CredentialEngine::store_credential` | — (creates new entries; no stale cache possible) | +| `POST /management/assume-role` | Assume a role, get temporary credentials | Load role via `IdentityEngine::get_role` → evaluate trust policy via `evaluate_trust_policy` with `AssumeRoleContext` → `SessionEngine::create_session` + `CredentialEngine::store_credential` | none (creates new entries; no stale cache possible) | | `POST /management/revoke-session` | Revoke a role session | `SessionEngine::revoke_session` | `invalidate_session`, `invalidate_credential` | ### 8.3 Request/Response Formats @@ -487,7 +486,7 @@ All management routes use `POST /management/` with JSON request/response } ``` -`CallerArn` is required — it identifies the principal assuming the role. It must match the format `arn:aws:iam::{account_id}:user/{user_name}`. If the format is invalid, the handler returns 400 with `{"Error": "ValidationError", "Message": "CallerArn must be a valid IAM user ARN"}`. Role-chaining (a role assuming another role) is deferred to a future version; in v1, `CallerArn` must reference a user. The handler loads the caller's tags via `IdentityEngine`, builds an `AssumeRoleContext` from the caller's tags + `SessionTags` + `ExternalId`, loads the role's trust policy, and evaluates it using `evaluate_trust_policy`. If the trust policy denies the assumption, the handler returns 403 with `{"Error": "AccessDenied", "Message": "Trust policy does not allow this principal to assume the role"}`. This ensures developers can test trust policy conditions (e.g., `sts:ExternalId`, `aws:PrincipalTag/*` restrictions) through the management API. +`CallerArn` is required; it identifies the principal assuming the role. It must match the format `arn:aws:iam::{account_id}:user/{user_name}`. If the format is invalid, the handler returns 400 with `{"Error": "ValidationError", "Message": "CallerArn must be a valid IAM user ARN"}`. Role chaining (a role assuming another role) is not supported; `CallerArn` must reference a user. The handler loads the caller's tags via `IdentityEngine`, builds an `AssumeRoleContext` from the caller's tags + `SessionTags` + `ExternalId`, loads the role's trust policy, and evaluates it using `evaluate_trust_policy`. If the trust policy denies the assumption, the handler returns 403 with `{"Error": "AccessDenied", "Message": "Trust policy does not allow this principal to assume the role"}`. This ensures developers can test trust policy conditions (e.g., `sts:ExternalId`, `aws:PrincipalTag/*` restrictions) through the management API. ### 8.4 Error Responses @@ -591,7 +590,7 @@ impl RateLimiter { ## 11. Throughput Tracking -Throughput tracking enforces DynamoDB's per-partition and per-table capacity limits (REQ-CAP-004, REQ-CAP-005). This is a runtime stateful component — it lives in the server crate, not in `core` (which only contains the pure-math RCU/WCU calculator). +Throughput tracking enforces DynamoDB's per-partition and per-table capacity limits (REQ-CAP-004, REQ-CAP-005). This is a runtime stateful component; it lives in the server crate, not in `core` (which only contains the pure-math RCU/WCU calculator). ```rust use std::sync::Mutex; @@ -627,6 +626,6 @@ The handler calls `core::capacity::calculator` to compute consumed RCU/WCU, then Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/07-component-streams.md b/docs/design/07-component-streams.md index 49a0dc6eb..be2b998cf 100755 --- a/docs/design/07-component-streams.md +++ b/docs/design/07-component-streams.md @@ -1,13 +1,12 @@ --- description: "Change data capture for tables: stream records, shards, iterators and retention." +version: "1.0" +date: "2026-04-03" +status: "Draft: high-level design choices only (detailed design deferred)" --- # Component design: DynamoDB Streams -**Version:** 1.0 -**Date:** 2026-04-03 -**Status:** Draft — High-Level Design Choices Only (Detailed Design Deferred) - ## 1. Purpose DynamoDB Streams provides change data capture (CDC) for DynamoDB tables. When enabled on a table, every write (PutItem, UpdateItem, DeleteItem, BatchWriteItem, TransactWriteItems) generates a stream record containing the item's key and optionally the old/new images. @@ -25,7 +24,7 @@ Four operations, served on the same HTTP endpoint as DynamoDB operations: | `GetShardIterator` | Get an iterator for a shard (TRIM_HORIZON, LATEST, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER) | | `GetRecords` | Read up to 1000 records (or 1 MB) from a shard using an iterator | -**Note:** In real DynamoDB, Streams is a separate service with its own endpoint (`streams.dynamodb..amazonaws.com`) and its own SDK client (`boto3.client('dynamodbstreams')` in Python, `DynamoDbStreamsClient` in Java). extenddb serves both services on a single port. This works because SDK clients accept `endpoint_url` overrides — the user sets both `AWS_ENDPOINT_URL_DYNAMODB` and `AWS_ENDPOINT_URL_DYNAMODB_STREAMS` to the same extenddb address. The `X-Amz-Target` prefix distinguishes the two services: DynamoDB operations use `DynamoDB_20120810.`, Streams operations use `DynamoDBStreams_20120810.`. Both use `dynamodb` as the SigV4 signing name, so authentication is identical. See `01-requirements.md` §11 and `08-component-config.md` §10 for SDK configuration details. +**Note:** In real DynamoDB, Streams is a separate service with its own endpoint (`streams.dynamodb..amazonaws.com`) and its own SDK client (`boto3.client('dynamodbstreams')` in Python, `DynamoDbStreamsClient` in Java). ExtendDB serves both services on a single port. This works because SDK clients accept `endpoint_url` overrides: the user sets both `AWS_ENDPOINT_URL_DYNAMODB` and `AWS_ENDPOINT_URL_DYNAMODB_STREAMS` to the same ExtendDB address. The `X-Amz-Target` prefix distinguishes the two services: DynamoDB operations use `DynamoDB_20120810.`, Streams operations use `DynamoDBStreams_20120810.`. Both use `dynamodb` as the SigV4 signing name, so authentication is identical. See `01-requirements.md` §11 and `08-component-config.md` §10 for SDK configuration details. ## 3. Stream Record Format @@ -171,6 +170,6 @@ GetRecords handler: Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/08-component-config.md b/docs/design/08-component-config.md index dfcf8ab17..50485df46 100755 --- a/docs/design/08-component-config.md +++ b/docs/design/08-component-config.md @@ -1,13 +1,12 @@ --- description: "The configuration system, configurable limits, logging, metrics, health checks and deployment settings." +version: "1.0" +date: "2026-04-03" +status: "Draft" --- # Component design: Configuration and operations -**Version:** 1.0 -**Date:** 2026-04-03 -**Status:** Draft - ## 1. Purpose This document covers the configuration system, configurable limits, logging, metrics, health checks, and deployment considerations. Configuration is handled in the `bin` crate (startup wiring) with types defined in `core` (limits) and `server` (server config). @@ -15,10 +14,10 @@ This document covers the configuration system, configurable limits, logging, met ## 2. Configuration Layering Precedence (highest to lowest): -1. **CLI flags** — `--config`, `--port`, `--version`, `--validate-config` -2. **Environment variables** — `EXTENDDB__SERVER__PORT=18443` -3. **Config file** — TOML format, path specified via `--config` or default `config.toml` -4. **Defaults** — hardcoded in Rust structs via `Default` trait +1. **CLI flags**: `--config`, `--port`, `--version`, `--validate-config` +2. **Environment variables**: `EXTENDDB__SERVER__PORT=18443` +3. **Config file**: TOML format, path specified via `--config` or default `config.toml` +4. **Defaults**: hardcoded in Rust structs via `Default` trait ### 2.1 CLI (clap) @@ -510,11 +509,11 @@ CMD ["--config", "/etc/extenddb/config.toml"] ## 10. SDK Client Configuration -extenddb serves both DynamoDB and DynamoDB Streams on a single port. An unmodified application that works against AWS must work against extenddb by changing only endpoint configuration. +ExtendDB serves both DynamoDB and DynamoDB Streams on a single port. An unmodified application that works against AWS must work against ExtendDB by changing only endpoint configuration. ### 10.1 Recommended Setup: Environment Variables -The simplest approach — set two service-specific endpoint variables and credentials: +The simplest approach is to set two service-specific endpoint variables and credentials: ```bash # Endpoint overrides (both point to the same extenddb instance) @@ -527,19 +526,19 @@ export AWS_SECRET_ACCESS_KEY=local-dev-secret export AWS_DEFAULT_REGION=us-east-1 ``` -This works with every AWS SDK (Python/boto3, Java v2, Rust, Go v2, .NET, Node.js) — all support the `AWS_ENDPOINT_URL_` convention since late 2023. +This works with every AWS SDK (Python/boto3, Java v2, Rust, Go v2, .NET, Node.js); all support the `AWS_ENDPOINT_URL_` convention since late 2023. -**Alternative — single global override** (when the application only uses DynamoDB): +**Alternative: single global override** (when the application only uses DynamoDB): ```bash export AWS_ENDPOINT_URL=https://127.0.0.1:18443 ``` -This sends all AWS SDK calls to extenddb. Simpler, but breaks if the application also calls non-DynamoDB services. +This sends all AWS SDK calls to ExtendDB. Simpler, but breaks if the application also calls non-DynamoDB services. ### 10.2 Alternative Setup: AWS Config File -For teams that prefer file-based configuration or need to switch between extenddb and real DynamoDB: +For teams that prefer file-based configuration or need to switch between ExtendDB and real DynamoDB: ```ini # ~/.aws/config @@ -575,7 +574,7 @@ AWS SDKs resolve endpoints in this order (highest priority first): When any override is set, the SDK disables endpoint discovery (`DescribeEndpoints` is never called automatically). All requests go directly to the configured URL. -extenddb accepts both `DynamoDB_20120810.*` and `DynamoDBStreams_20120810.*` target prefixes on the same port. Both use `dynamodb` as the SigV4 signing name, so authentication is identical. +ExtendDB accepts both `DynamoDB_20120810.*` and `DynamoDBStreams_20120810.*` target prefixes on the same port. Both use `dynamodb` as the SigV4 signing name, so authentication is identical. ### 10.4 DescribeEndpoints Response @@ -596,9 +595,9 @@ The `Address` is derived from the server's configured `bind_addr` and `port`. If ### 10.5 Credential Pairing -The credentials in `~/.aws/credentials` (or env vars) are what the SDK uses to sign requests. extenddb validates these signatures against its own credential store. The access key and secret key must match on both sides: +The credentials in `~/.aws/credentials` (or env vars) are what the SDK uses to sign requests. ExtendDB validates these signatures against its own credential store. The access key and secret key must match on both sides: -1. Register credentials in extenddb's credential store (via admin API or database seeding) +1. Register credentials in ExtendDB's credential store (via admin API or database seeding) 2. Configure the same `aws_access_key_id` / `aws_secret_access_key` in the SDK client's environment For development environments, a default credential pair can be seeded automatically on first startup (configurable via `auth.seed_credentials` in `config.toml`). @@ -610,6 +609,6 @@ For development environments, a default credential pair can be seeded automatica Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/09-testing.md b/docs/design/09-testing.md index 11c20ada4..b356b0a07 100755 --- a/docs/design/09-testing.md +++ b/docs/design/09-testing.md @@ -1,31 +1,30 @@ --- description: "How tests are organized, how golden files work and how multi-language suites check SDK compatibility." +version: "1.0" +date: "2026-04-06" +status: "Draft" --- # Test strategy -**Version:** 1.0 -**Date:** 2026-04-06 -**Status:** Draft - ## 1. Purpose -This document defines extenddb's test strategy: how tests are organized, what reference suites inform coverage, how golden files work, and how multi-language test suites validate SDK compatibility. It is the authoritative reference for anyone writing or running extenddb tests. +This document defines ExtendDB's test strategy: how tests are organized, what reference suites inform coverage, how golden files work, and how multi-language test suites validate SDK compatibility. It is the authoritative reference for anyone writing or running ExtendDB tests. ## 2. Tenets (for testing) -1. **Real DynamoDB is the oracle** — every expected behavior is captured by running against real DynamoDB, never assumed or copied from reference suites. -2. **Reference suites tell us what to test, not how to answer** — we extract scenario names and API coverage from reference suites; we independently discover correct responses via golden files. -3. **Python carries the fidelity burden** — the Python suite is the primary test suite with full golden file coverage. Other languages validate SDK integration. -4. **Coverage grows with the code** — test coverage and coverage maps expand incrementally as phases progress, never front-loaded as prerequisites that gate implementation. -5. **Same code, two targets** — every test runs against both real DynamoDB and extenddb with zero code changes, controlled by environment variables. -6. **Standard operations only** — the test suite validates fidelity mode exclusively. Preview extensions have their own tests and never appear in the fidelity test suite. +1. **Real DynamoDB is the oracle**: every expected behavior is captured by running against real DynamoDB, never assumed or copied from reference suites. +2. **Reference suites tell us what to test, not how to answer**: we extract scenario names and API coverage from reference suites; we independently discover correct responses via golden files. +3. **Python carries the fidelity burden**: the Python suite is the primary test suite with full golden file coverage. Other languages validate SDK integration. +4. **Coverage grows with the code**: test coverage and coverage maps expand incrementally as phases progress, never front-loaded as prerequisites that gate implementation. +5. **Same code, two targets**: every test runs against both real DynamoDB and ExtendDB with zero code changes, controlled by environment variables. +6. **Standard operations only**: the test suite validates fidelity mode exclusively. Preview extensions have their own tests and never appear in the fidelity test suite. ## 3. Reference Suites ### 3.1 Reference Strategy -extenddb uses test suites as a coverage map — a source of *what to test*, not *how to implement*. The test suites are not included here. +ExtendDB uses test suites as a coverage map: a source of *what to test*, not *how to implement*. The test suites are not included here. ### 3.2 Arm's-Length Boundary (MF-1) @@ -60,21 +59,21 @@ tests/reference/ └── coverage-map.md # Test scenario inventory ``` -`check-upstream.sh` compares the stored hash against current HEAD of the reference repo, diffs the file list, and reports new/modified/deleted test files. New files are flagged as high-priority (likely test behaviors extenddb must support). +`check-upstream.sh` compares the stored hash against current HEAD of the reference repo, diffs the file list, and reports new/modified/deleted test files. New files are flagged as high-priority (likely test behaviors ExtendDB must support). When new tests appear upstream, a developer reviews them for new scenarios to add to the coverage map. This is a manual review process, not automated ingestion. ### 3.4 Coverage Map (MF-2) -The coverage map (`tests/reference/coverage-map.md`) tracks which reference suite scenarios have corresponding extenddb tests. Every test method maps to an extenddb test scenario at per-method granularity. +The coverage map (`tests/reference/coverage-map.md`) tracks which reference suite scenarios have corresponding ExtendDB tests. Every test method maps to an ExtendDB test scenario at per-method granularity. -The coverage map is a living document, not a prerequisite — it is updated alongside development and does not block Phase 1 implementation. +The coverage map is a living document, not a prerequisite; it is updated alongside development and does not block Phase 1 implementation. ## 4. Multi-Language Test Suites ### 4.1 Why Multiple Languages -Different SDKs serialize requests differently (field ordering, default values, header formatting), handle error responses differently (some parse `__type`, some parse `Code`), and implement SigV4 with subtle variations. If extenddb only passes Python tests, it may fail with the Java SDK. Real DynamoDB users use all four SDKs — extenddb must work with all of them. +Different SDKs serialize requests differently (field ordering, default values, header formatting), handle error responses differently (some parse `__type`, some parse `Code`), and implement SigV4 with subtle variations. If ExtendDB only passes Python tests, it may fail with the Java SDK. Real DynamoDB users use all four SDKs; ExtendDB must work with all of them. ### 4.2 Language Roles (SF-1) @@ -82,8 +81,8 @@ Different SDKs serialize requests differently (field ordering, default values, h |----------|-----|------|------| | Python | boto3 | Primary test suite. Full fidelity: golden files, error message matching, edge cases. Every phase starts here. | Phase 1+ | | Java | aws-sdk-java-v2 | Elevated integration. Covers more scenarios than Rust/C++ because the PostgreSQL extension suite (our primary reference) is Java. Tests scenarios that have a direct PostgreSQL extension suite counterpart. | Phase 2+ | -| Rust | aws-sdk-rust | SDK integration validation. Core smoke tests proving the Rust SDK works against extenddb. | Phase 2+ | -| C/C++ | aws-sdk-cpp | SDK integration validation. Exercises a fundamentally different SDK architecture. | Phase 7+ (Query/Scan gives a broad enough API surface — CRUD alone is too narrow to exercise C++'s distinct serialization and memory patterns meaningfully) | +| Rust | aws-sdk-rust | SDK integration validation. Core smoke tests proving the Rust SDK works against ExtendDB. | Phase 2+ | +| C/C++ | aws-sdk-cpp | SDK integration validation. Exercises a fundamentally different SDK architecture. | Phase 7+ (Query/Scan gives a broad enough API surface; CRUD alone is too narrow to exercise C++'s distinct serialization and memory patterns meaningfully) | **Phase exit criteria:** - Python tests: mandatory @@ -169,7 +168,7 @@ Each golden file contains: } ``` -The `request` is included so the test harness can replay it. The `response` is the expected result. Each language's test harness deserializes the golden file and compares against its SDK's parsed response. Note: headers like `x-amzn-RequestId` are captured for completeness but are not meaningful for exact comparison — they are validated by format only (see §5.3). +The `request` is included so the test harness can replay it. The `response` is the expected result. Each language's test harness deserializes the golden file and compares against its SDK's parsed response. Note: headers like `x-amzn-RequestId` are captured for completeness but are not meaningful for exact comparison; they are validated by format only (see §5.3). ### 5.2 Capturing Golden Files (N-2) @@ -185,13 +184,13 @@ To capture a new golden file: - Replace stripped values with `"REDACTED"` 4. Save to `tests/golden//.json` 5. Verify the captured response is deterministic (run twice, diff) -6. Commit the golden file — it must never contain real credentials +6. Commit the golden file; it must never contain real credentials The `tests/README.md` documents this workflow. ### 5.3 Field Comparison Rules -Not all fields can be compared exactly between AWS and extenddb. Per-operation YAML files in `tests/comparison_rules/` specify field handling: +Not all fields can be compared exactly between AWS and ExtendDB. Per-operation YAML files in `tests/comparison_rules/` specify field handling: ```yaml # comparison_rules/create_table.yaml @@ -227,7 +226,7 @@ Five comparison categories: | `format_only` | Field must be present and match a format (e.g., UUID regex) but the value is not compared | | `ignore` | Field is skipped entirely during comparison | | `normalize` | Field is transformed before comparison (e.g., timestamps normalized to epoch) | -| `partial` | extenddb's response for this object is a strict subset of the golden file. Fields present in the golden file but absent from extenddb are allowed (extenddb is incomplete). Fields present in extenddb but absent from the golden file are test failures (extenddb must not return data that real DynamoDB does not). Fields present in both are compared according to their own comparison category. This is a transitional mode: as extenddb matures, fields move from `partial` to `exact`. | +| `partial` | ExtendDB's response for this object is a strict subset of the golden file. Fields present in the golden file but absent from ExtendDB are allowed (ExtendDB is incomplete). Fields present in ExtendDB but absent from the golden file are test failures (ExtendDB must not return data that real DynamoDB does not). Fields present in both are compared according to their own comparison category. This is a transitional mode: as ExtendDB matures, fields move from `partial` to `exact`. | The comparison rules live in a top-level shared directory (`tests/comparison_rules/`) so all language suites can read them. The Python test harness reads these rules and applies them during comparison. Other language suites use the same YAML files. @@ -236,9 +235,9 @@ The comparison rules live in a top-level shared directory (`tests/comparison_rul Each language's test harness: - Reads endpoint configuration from environment variables (`AWS_ENDPOINT_URL_DYNAMODB`, `AWS_REGION`, etc.) -- Runs against both real DynamoDB and extenddb with zero code changes -- Handles table creation wait times — both real DynamoDB and extenddb use async control plane transitions. The harness polls `DescribeTable` until `ACTIVE` -- Cleans up test tables using the `EXTENDDB_TABLE_PREFIX` convention — deletes all matching tables at the start of each run (idempotent cleanup) +- Runs against both real DynamoDB and ExtendDB with zero code changes +- Handles table creation wait times: both real DynamoDB and ExtendDB use async control plane transitions. The harness polls `DescribeTable` until `ACTIVE` +- Cleans up test tables using the `EXTENDDB_TABLE_PREFIX` convention: deletes all matching tables at the start of each run (idempotent cleanup) - Reports results in JUnit XML format for CI integration ### 6.1 Test Infrastructure Pattern (Q-7) @@ -248,7 +247,7 @@ All language suites adopt the same architectural pattern independently (not copi - A base class/fixture with table setup and teardown - Assertion helpers for item comparison and attribute value comparison - Endpoint configuration from environment variables -- Table name prefix (`EXTENDDB_TABLE_PREFIX`, defaults to `extenddb_test_`) for cleanup isolation — configurable to support parallel test runs +- Table name prefix (`EXTENDDB_TABLE_PREFIX`, defaults to `extenddb_test_`) for cleanup isolation; configurable to support parallel test runs This pattern is standard for DynamoDB test suites and proven by the PostgreSQL extension suite's architecture. @@ -256,7 +255,7 @@ This pattern is standard for DynamoDB test suites and proven by the PostgreSQL e | Variable | Purpose | Default | |----------|---------|---------| -| `AWS_ENDPOINT_URL_DYNAMODB` | extenddb endpoint override | (none — uses real DynamoDB) | +| `AWS_ENDPOINT_URL_DYNAMODB` | ExtendDB endpoint override | (none: uses real DynamoDB) | | `AWS_REGION` | Region for requests | `us-east-1` | | `AWS_ACCESS_KEY_ID` | Credentials | (from AWS config) | | `AWS_SECRET_ACCESS_KEY` | Credentials | (from AWS config) | @@ -269,12 +268,12 @@ This pattern is standard for DynamoDB test suites and proven by the PostgreSQL e | Phase | Python | Java | Rust | C++ | |-------|--------|------|------|-----| -| 1 (Server + table CRUD) | Full: table lifecycle, error paths, raw HTTP edge cases, health/metrics | — | — | — | -| 2 (PutItem + GetItem) | Full: all attribute types, size limits, ReturnValues | Smoke: basic put/get | Smoke: basic put/get | — | -| 3 (SigV4) | Full: valid/invalid credentials, clock skew, missing headers | Smoke: SigV4 with Java SDK | Smoke: SigV4 with Rust SDK | — | -| 4 (Policy engine) | Full: policy enforcement, ABAC, management API | — | — | — | -| 5 (DeleteItem + UpdateItem) | Full: CRUD + expressions | Java scenarios from PostgreSQL ext suite | — | — | -| 6 (Full expressions) | Full: all operators, functions, Expected | — | — | — | +| 1 (Server + table CRUD) | Full: table lifecycle, error paths, raw HTTP edge cases, health/metrics | none | none | none | +| 2 (PutItem + GetItem) | Full: all attribute types, size limits, ReturnValues | Smoke: basic put/get | Smoke: basic put/get | none | +| 3 (SigV4) | Full: valid/invalid credentials, clock skew, missing headers | Smoke: SigV4 with Java SDK | Smoke: SigV4 with Rust SDK | none | +| 4 (Policy engine) | Full: policy enforcement, ABAC, management API | none | none | none | +| 5 (DeleteItem + UpdateItem) | Full: CRUD + expressions | Java scenarios from PostgreSQL ext suite | none | none | +| 6 (Full expressions) | Full: all operators, functions, Expected | none | none | none | | 7 (Query + Scan) | Full: key conditions, pagination, 1MB limit | Java scenarios from PostgreSQL ext suite | Smoke: basic query/scan | Smoke: basic CRUD + query | | 8+ | Python mandatory, Java for PostgreSQL ext counterparts, Rust/C++ extended smoke | | | | @@ -319,6 +318,6 @@ Phase exit criteria are defined in the implementation plan. This document adds t Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/10-dependency-licenses.md b/docs/design/10-dependency-licenses.md index 91da31ad8..f1cfbbc56 100755 --- a/docs/design/10-dependency-licenses.md +++ b/docs/design/10-dependency-licenses.md @@ -441,6 +441,6 @@ This file is a human-controlled policy input and must not be modified by agents. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/design/11-high-availability.md b/docs/design/11-high-availability.md index b51069590..289c2dcb2 100755 --- a/docs/design/11-high-availability.md +++ b/docs/design/11-high-availability.md @@ -161,7 +161,7 @@ The "No Caching Rule" applies to data-path state: items, query results, throttle - Self-induced changes (admin API mutations) propagate **instantly** within a single instance via write-through invalidation hooks. Off-instance changes (multi-frontend, or direct DB writes) are bounded by the configured TTL. - The kill switch (`auth.cache.enabled = false`) restores the original "no caching" behavior without restart-triggered code changes. -**Rationale (still valid for the data path):** Multiple frontends sharing a catalog would have stale caches. For data, that's intolerable. For IAM, bounded staleness is the same trade-off real AWS IAM exposes. Cross-frontend invalidation fanout for the IAM cache is designed in [Appendix B of `12-auth-authz-cache.md`](12-auth-authz-cache.md#appendix-b--multi-instance-cache-invalidation-deferred) but not yet implemented; deployments running multiple frontends should treat `auth.cache.ttl_seconds` as the worst-case lag for off-instance IAM mutations. +**Rationale (still valid for the data path):** Multiple frontends sharing a catalog would have stale caches. For data, that's intolerable. For IAM, bounded staleness is the same trade-off real AWS IAM exposes. Cross-frontend invalidation fanout for the IAM cache is designed in [Appendix B of `12-auth-authz-cache.md`](12-auth-authz-cache.md#appendix-b-multi-instance-cache-invalidation-deferred) but not yet implemented; deployments running multiple frontends should treat `auth.cache.ttl_seconds` as the worst-case lag for off-instance IAM mutations. ### D2: Consistency Routing Lives in the Storage Adapter diff --git a/docs/design/12-auth-authz-cache.md b/docs/design/12-auth-authz-cache.md index 039aa8054..8e29f4fd6 100644 --- a/docs/design/12-auth-authz-cache.md +++ b/docs/design/12-auth-authz-cache.md @@ -1,19 +1,16 @@ --- description: "The in-process cache of IAM data (credentials, policies, tags, table key information) and its invalidation." +status: "Implemented for a single instance; the multi-instance fanout in Appendix B is designed and deferred" --- # Component design: Authorization cache -> Status: implemented (single-instance). Multi-instance fanout in -> Appendix B is designed but deferred — implementation is gated on a -> separate review. -> Author: assisted, reviewed by repository owner -> Companion document to: [05-component-auth.md](05-component-auth.md), [06-component-server.md](06-component-server.md) -> -> Note: this cache is the deliberate revision of the "No In-Process State" -> rule in the [high availability design draft](https://github.com/ExtendDB/extenddb/blob/main/docs/design/11-high-availability.md) §D1. The -> revision is scoped to IAM data only (credentials, policies, tags, table -> key info). The data path remains uncached. +Companion to the [authentication design](05-component-auth.md) and the [HTTP server design](06-component-server.md). + +> Note: this cache is a deliberate exception to the rule that the server +> holds no in-process state. The exception is scoped to IAM data only +> (credentials, policies, tags, table key info). The data path remains +> uncached. ## 1. Motivation @@ -29,7 +26,7 @@ Each DynamoDB request issues 6+ catalog queries before dispatch can begin: | 6 | Resource tags | `fetch_resource_tags` | | 7 | TableKeyInfo (item-level ops only) | `request_helpers.rs::authorize_request` (serial, outside the `try_join!`) | -Queries 2–6 fan out concurrently via `tokio::try_join!` (`crates/server/src/authorization.rs:82`), but they still each consume a connection from the catalog pool. With a 1024-connection catalog pool (post-PR #46), the pool itself is no longer the limit — but the **per-request DB roundtrips and JSON-parse cost** are now the dominant overhead before dispatch. +Queries 2-6 fan out concurrently via `tokio::try_join!` (`crates/server/src/authorization.rs:82`), but they still each consume a connection from the catalog pool. With a 1024-connection catalog pool (post-PR #46), the pool itself is no longer the limit, but the **per-request DB roundtrips and JSON-parse cost** are now the dominant overhead before dispatch. In addition, every call to `fetch_policies` reparses policy JSON via `PolicyDocument::from_json` (`crates/server/src/authorization.rs:189`). Caching parsed `PolicyDocument` values eliminates that cost, not just the DB roundtrip. @@ -81,7 +78,7 @@ Considered: fetch a cheap `updated_at` first, only refetch full data if changed. ## 5. Negative caching -Negative results (`Ok(None)` — key not found) are cached for a **short** TTL (default: 5 seconds). Reasons: +Negative results (`Ok(None)`, key not found) are cached for a **short** TTL (default: 5 seconds). Reasons: - Stops attackers from hammering the catalog with random `AKIA...` access keys. - 5s ensures newly-created keys become usable quickly without manual cache flush. @@ -120,11 +117,11 @@ A cross-process invalidation channel is **not** added in this iteration. Multi-n Write-through hooks (§6) cover every IAM mutation we own. They do **not** cover off-instance changes on multi-node deployments before TTL kicks in, bugs in the write-through plumbing, or test scenarios that need a deterministic flush. For those cases, an admin-authenticated **manual invalidation API** complements the automatic hooks. -Both the management API and the web console expose this surface; the CLI (`extenddb manage cache invalidate ...`) is a thin client over the management API. The console route and the management API route call the same handler logic, so behavior is identical regardless of how it's invoked. All three paths use the existing admin authentication mechanism (Basic auth for the API, session-based auth for the console — neither introduces a new auth path). +Both the management API and the web console expose this surface; the CLI (`extenddb manage cache invalidate ...`) is a thin client over the management API. The console route and the management API route call the same handler logic, so behavior is identical regardless of how it's invoked. All three paths use the existing admin authentication mechanism (Basic auth for the API, session-based auth for the console; neither introduces a new auth path). ### HTTP endpoint -`POST /management/cache/invalidate` — admin-only, JSON body, returns 200 with an audit-friendly response or 400 on malformed scope. +`POST /management/cache/invalidate`: admin-only, JSON body, returns 200 with an audit-friendly response or 400 on malformed scope. ```json { "scope": "", "selectors": { ... } } @@ -143,7 +140,7 @@ Both the management API and the web console expose this surface; the CLI (`exten Composite scopes (`user`, `role`) match what an operator most often actually wants when they reach for this tool ("forget everything about user X"). The narrow scopes are still exposed for surgical use. `all` is the broadest hammer and requires `--yes` from the CLI / a confirmation token from the console. -`account` sweeps the authz and credential caches but not `table_key_info` (the `TableKeyInfo` cache is keyed by `(account, table)` and the registry's `invalidate_account` does not currently fan out across it). To clear cached table-key-info for an account, either call `scope: table_key_info` per table or use `scope: all`. This is a deliberate scoping choice — the table-key-info cache rarely diverges per account in practice, so the narrower `invalidate_account` semantics keep the most common path cheap. +`account` sweeps the authz and credential caches but not `table_key_info` (the `TableKeyInfo` cache is keyed by `(account, table)` and the registry's `invalidate_account` does not currently fan out across it). To clear cached table-key-info for an account, either call `scope: table_key_info` per table or use `scope: all`. This is a deliberate scoping choice: the table-key-info cache rarely diverges per account in practice, so the narrower `invalidate_account` semantics keep the most common path cheap. ### Response shape @@ -151,7 +148,7 @@ Composite scopes (`user`, `role`) match what an operator most often actually wan { "scope": "user", "invalidated": ["user_policies", "user_group_policies", "user_boundary", "user_tags", "principal_credentials"] } ``` -The `invalidated` array tells operators which subcaches were touched — useful for confirming that composite scopes did what was expected, and for the console UI's success banner. +The `invalidated` array tells operators which subcaches were touched, which is useful for confirming that composite scopes did what was expected and for the console UI's success banner. ### Audit trail @@ -165,7 +162,7 @@ Cache invalidation counters on each subcache (exposed via `/management/auth-cach ### Behavior when caches are disabled -When `auth.cache.enabled = false`, the cache layer is in pass-through mode and there is nothing to invalidate. The endpoint still returns 200 — the `AuthCacheRegistry` invalidate methods are no-ops in this state, and the operator's contract ("cache-driven staleness for X is now cleared") is trivially satisfied. +When `auth.cache.enabled = false`, the cache layer is in pass-through mode and there is nothing to invalidate. The endpoint still returns 200: the `AuthCacheRegistry` invalidate methods are no-ops in this state, and the operator's contract ("cache-driven staleness for X is now cleared") is trivially satisfied. ### CLI @@ -201,13 +198,13 @@ max_entries = 10000 # per-cache LRU size limit Default rationale: -- **60 s hard TTL** mirrors AWS IAM's "policy propagation" expectation (~30 s–2 min). Long enough to absorb the perf bottleneck (60 s of caching → 99.99%+ hit rate per hot key under any meaningful load), short enough to bound the blast radius of off-instance credential revocation. +- **60 s hard TTL** mirrors AWS IAM's "policy propagation" expectation (~30 s-2 min). Long enough to absorb the perf bottleneck (60 s of caching → 99.99%+ hit rate per hot key under any meaningful load), short enough to bound the blast radius of off-instance credential revocation. - **30 s soft TTL** = TTL / 2 keeps refreshes from clustering at the boundary. - **5 s negative TTL** is short enough that newly-issued keys work quickly, long enough to absorb a dictionary attack against `lookup_credential`. - **10 000 entries per cache** is a generous default for typical deployments; under memory pressure the LRU evicts cold entries. - **`enabled = false`** is a kill switch for incident response. -The cache is configured statically (`extenddb.toml`). Runtime tunability is not added — cache TTL changes are rare and a server restart is acceptable. Adding it to the existing `settings` poller table is an option for a follow-up if operationally needed. +The cache is configured statically (`extenddb.toml`). Runtime tunability is not added: cache TTL changes are rare and a server restart is acceptable. Adding it to the existing `settings` poller table is an option for a follow-up if operationally needed. ## 8. Architecture: decorator pattern, one cache per data shape @@ -268,7 +265,7 @@ Caching is layered as **wrapper types** over the existing trait implementations, └────────────────────────────────────┘ ``` -The auth cache stores **parsed `PolicyDocument`s** wrapped in `Arc` — multiple cache entries (e.g. a shared inline policy attached to several users) share the parsed document via `Arc::clone`. The `check_authorization` path is updated to call a new method on the cached wrapper that returns parsed documents, eliminating the `from_json` step from the request hot path. +The auth cache stores **parsed `PolicyDocument`s** wrapped in `Arc`; multiple cache entries (e.g. a shared inline policy attached to several users) share the parsed document via `Arc::clone`. The `check_authorization` path is updated to call a new method on the cached wrapper that returns parsed documents, eliminating the `from_json` step from the request hot path. ## 9. Library choice: `moka` @@ -277,12 +274,12 @@ The implementation uses the `moka` crate (`features = ["future"]`): - Async-aware (`moka::future::Cache`) integrates cleanly with `tokio`. - Built-in time-based expiration with O(1) eviction. - Bounded with LRU/W-TinyLFU eviction policy under memory pressure. -- **Single-flight** via `entry().or_insert_with(future)` — concurrent misses for the same key share one load future. This is critical: prevents thundering herd at cache cold-start. -- Internal sharding — no global lock contention. +- **Single-flight** via `entry().or_insert_with(future)`: concurrent misses for the same key share one load future. This is critical: prevents thundering herd at cache cold-start. +- Internal sharding: no global lock contention. The SWR layer is a thin (~80 line) wrapper on top of `moka` that adds the soft-TTL / spawn-refresh logic. We do **not** hand-roll the cache itself; that would be 200+ lines of subtle concurrency code with the same external behavior. -The `moka` `future` feature uses `tokio` internals consistent with the rest of the codebase. License is MIT/Apache-2.0 — compatible with our Apache-2.0 license. +The `moka` `future` feature uses `tokio` internals consistent with the rest of the codebase. License is MIT/Apache-2.0, compatible with our Apache-2.0 license. ## 10. Cache key shapes and value types @@ -355,7 +352,7 @@ auth, admin user). The response shape is: | Field | Description | |-------|-------------| | `hits` | Fresh cache hits (younger than `soft_ttl`). | -| `stale_hits` | Hits between `soft_ttl` and `ttl` — served immediately, refresh spawned. | +| `stale_hits` | Hits between `soft_ttl` and `ttl`: served immediately, refresh spawned. | | `misses` | Hard misses (caller waited for the loader). | | `negative_hits` | Cached `None` returned (negative cache). | | `refresh_success` | Background refresh wrote a fresh entry. | @@ -372,7 +369,7 @@ gauges/counters under `/metrics`; until then, scrape the JSON endpoint. ## 12. Error semantics - DB errors during a **request-blocking load** propagate to the caller as today (`InternalServerError`). -- DB errors during a **background refresh** are logged and counted in `auth_cache_refresh_failure`. The stale entry is **not** evicted — the next request continues serving it (until the hard TTL passes, at which point it transitions back to a request-blocking load). +- DB errors during a **background refresh** are logged and counted in `auth_cache_refresh_failure`. The stale entry is **not** evicted; the next request continues serving it (until the hard TTL passes, at which point it transitions back to a request-blocking load). - Decrypt failures during a credential load return `InternalServerError` and are not cached. ## 13. Test strategy @@ -405,9 +402,9 @@ gauges/counters under `/metrics`; until then, scrape the JSON endpoint. | Phase | Scope | Risk | |-------|-------|------| -| 1 | `SwrCache` primitive + `moka` dep | low — pure library code, no behavior change | -| 2 | `CachedCredentialStore` + invalidation hooks for access-key mutations | medium — touches auth path | -| 3 | `CachedAuthorizationStore` (with parsed `PolicyDocument` cache) + invalidation hooks for IAM mutations | medium — touches authorization path | +| 1 | `SwrCache` primitive + `moka` dep | low: pure library code, no behavior change | +| 2 | `CachedCredentialStore` + invalidation hooks for access-key mutations | medium: touches auth path | +| 3 | `CachedAuthorizationStore` (with parsed `PolicyDocument` cache) + invalidation hooks for IAM mutations | medium: touches authorization path | | 4 | `CachedTableKeyInfoStore` + invalidation hooks on `CreateTable`/`UpdateTable`/`DeleteTable` | low | | 5 | Documentation, config sample, `init` template, admin guide, metrics console row | trivial | @@ -420,8 +417,8 @@ Each phase is a separate PR. After each phase the test suite runs green and the > **Propagation timing for invalidations**: > > - **Single-key invalidations** (e.g. `DeleteAccessKey`, `PutUserPolicy`) bump an internal epoch counter and drop the moka entry immediately. The next request blocks on a fresh load. -> - **Fanout invalidations** (e.g. `DeleteAccount`, `DeleteRole`'s session sweep, `DeleteGroup`'s member fanout) use moka's `invalidate_entries_if`, which is **asynchronous** — predicate evaluation runs on moka's internal worker after the management call returns. There is a small window (~ms) between the API responding 200/204 and the matching entries actually being evicted from the underlying store. The accompanying epoch bump ensures any in-flight refresh on those keys is dropped on completion, but a request that reads the slot during this window can still see the pre-invalidation value. Operators should treat fanout invalidation as "eventually consistent within a few ms"; for hard cutover (e.g. revoking a compromised key), prefer the single-key invalidations (`DeleteAccessKey`) over the cascade ones. -> - **Off-instance changes** (a separate process modifying the catalog directly, or a different instance in a multi-node deployment) take up to `auth.cache.ttl_seconds` to propagate. The cross-process invalidation channel described in Appendix B is not yet implemented. +> - **Fanout invalidations** (e.g. `DeleteAccount`, `DeleteRole`'s session sweep, `DeleteGroup`'s member fanout) use moka's `invalidate_entries_if`, which is **asynchronous**: predicate evaluation runs on moka's internal worker after the management call returns. There is a small window (~ms) between the API responding 200/204 and the matching entries actually being evicted from the underlying store. The accompanying epoch bump ensures any in-flight refresh on those keys is dropped on completion, but a request that reads the slot during this window can still see the pre-invalidation value. Operators should treat fanout invalidation as "eventually consistent within a few ms"; for hard cutover (e.g. revoking a compromised key), prefer the single-key invalidations (`DeleteAccessKey`) over the cascade ones. +> - **Off-instance changes** (a separate process modifying the catalog directly, or a different instance in a multi-node deployment) take up to `auth.cache.ttl_seconds` to propagate. The cross-process invalidation channel described in Appendix B is not implemented. > > Self-induced single-key changes via the admin API or the web console propagate instantly on the local instance via write-through invalidation; both paths share the same `AuthCacheRegistry`. > @@ -429,7 +426,7 @@ Each phase is a separate PR. After each phase the test suite runs green and the --- -## Appendix A — alternatives considered +## Appendix A: alternatives considered - **No cache, just larger pool.** Insufficient. The catalog roundtrip itself is ~1 ms on a co-located DB; per-request 6× roundtrips × 1 ms = 6 ms of pre-dispatch latency that is impossible to remove without caching. No pool size fixes this. - **Verdict cache (cache `Allow`/`Deny` for `(principal, action, resource)`).** Considered. Rejected: condition expressions can depend on per-request context (`aws:CurrentTime`, request IP, leading keys, attribute names) that varies per call. Caching the verdict requires either invalidating on every condition-relevant input change (impractical) or only caching for a subset of verdicts (complexity not worth it). Caching the inputs and re-evaluating in CPU per request gives the same throughput at lower complexity and zero correctness risk. @@ -440,7 +437,7 @@ Each phase is a separate PR. After each phase the test suite runs green and the --- -## Appendix B — Multi-instance cache invalidation (deferred) +## Appendix B: Multi-instance cache invalidation (deferred) > **Status:** designed, **not implemented**. Single-instance correctness is the focus of the initial cache PR; multi-instance fanout requires additional engineering review (failure modes, schema, ordering guarantees) before implementation. > @@ -448,7 +445,7 @@ Each phase is a separate PR. After each phase the test suite runs green and the ### B.1 Why the current state is not enough for multi-instance deployments -In a multi-instance deployment (multiple extenddb processes sharing a catalog DB), each instance's in-memory cache is local. Self-induced changes propagate instantly **on the local instance** via the existing write-through hooks. Off-instance changes — an admin-API mutation on instance A, where instance B is serving the same principal — only become visible on instance B when its cache entry expires (`auth.cache.ttl_seconds`, default 60 s). +In a multi-instance deployment (multiple ExtendDB processes sharing a catalog DB), each instance's in-memory cache is local. Self-induced changes propagate instantly **on the local instance** via the existing write-through hooks. Off-instance changes (an admin-API mutation on instance A, where instance B is serving the same principal) only become visible on instance B when its cache entry expires (`auth.cache.ttl_seconds`, default 60 s). For an auth/authz cache, "60 seconds of stale policy" can mean a deleted access key still authenticates, or a Deny statement still being shadowed by a stale Allow. That window is acceptable for soft state but not for IAM revocation. @@ -532,7 +529,7 @@ If the catch-up scan exceeds the cleanup window (B.4), assume too much state has | Scenario | Behavior | |----------|----------| | Listener connection drops, reconnects within the cleanup window | Catch-up via table scan. No data loss. | -| Listener can't drain fast enough → NOTIFY queue overflow | Same — catch-up via table scan on next iteration. | +| Listener can't drain fast enough → NOTIFY queue overflow | Same: catch-up via table scan on next iteration. | | Network partition between instance and DB | NOTIFY paused; catch-up on heal. | | Instance offline > cleanup window | `invalidate_all` on every cache. Equivalent to a fresh start; correct but cold. | | Total LISTEN channel failure | Falls back gracefully to TTL-bounded propagation. No worse than today. | @@ -542,7 +539,7 @@ If the catch-up scan exceeds the cleanup window (B.4), assume too much state has ### B.4 Latency expectations - Within-DC: typical NOTIFY roundtrip < 50 ms. -- Across-region (rare; multi-region should usually use one extenddb cluster per region): bounded by the catalog DB's replication lag. +- Across-region (rare; multi-region should usually use one ExtendDB cluster per region): bounded by the catalog DB's replication lag. - Worst case (listener disconnected): catch-up window equals the cleanup window (default 4 min). Operators tune by adjusting the cleanup interval. ### B.5 What this does **not** solve @@ -564,7 +561,7 @@ enabled = false ### B.7 Implementation cost estimate -- ~400–500 LOC, self-contained, in `crates/storage-postgres/src/cache_fanout.rs`. +- ~400-500 LOC, self-contained, in `crates/storage-postgres/src/cache_fanout.rs`. - One new migration. - ~10 lines per IAM mutation handler to insert the row and trigger NOTIFY. - Behind the `auth.cache.fanout.enabled` flag (default off) so single-instance deployments are unaffected. diff --git a/docs/design/13-storage-mongodb.md b/docs/design/13-storage-mongodb.md index 35c4f55f2..b1a1844ea 100644 --- a/docs/design/13-storage-mongodb.md +++ b/docs/design/13-storage-mongodb.md @@ -29,7 +29,7 @@ for those targets, but neither provides snapshot's single point-in-time view. Local may also observe data that is later rolled back after failover. **Read preference:** `primary` only. `MongoEngine::new` rejects connection strings -that request `secondary`, `secondaryPreferred`, `primaryPreferred`, or `nearest` — +that request `secondary`, `secondaryPreferred`, `primaryPreferred`, or `nearest`. DynamoDB's `ConsistentRead=true` contract requires linearizable reads, which only Primary provides. Silently routing reads to replicas would return stale data with no signal to the caller. @@ -51,7 +51,7 @@ creation. ## 3. Catalog Database Collections Collections created by `run_catalog_migrations` in `bootstrapper.rs`. Two -additional collections (`iam_group_members`, `backup_items` — the latter is +additional collections (`iam_group_members`, `backup_items`; the latter is no longer used) are auto-created on first insert and not in the migration list. ### 3.1 `accounts` @@ -253,7 +253,7 @@ Compound index on `(principal, attempted_at)`. } ``` Index on `(account_id, table_name)`. The physical backup collection lives in -`extenddb_data` as `_backup_{backup_id}` — see §4.5. +`extenddb_data` as `_backup_{backup_id}`; see §4.5. ### 3.17 `continuous_backups` ```json @@ -296,19 +296,19 @@ names. Fields: -- `_id` — netstring-encoded composite key `:,:,`. +- `_id`: netstring-encoded composite key `:,:,`. Netstring framing gives an unambiguous boundary between `pk` and `sk` regardless of their contents. A naive `{pk}#{sk}` delimiter collides on `pk="a#b",sk="c"` vs `pk="a",sk="b#c"`. PK-only tables use raw `pk` text as `_id`. -- `pk` — partition key text (matches `composite_pk_to_text` from +- `pk`: partition key text (matches `composite_pk_to_text` from `extenddb_storage::util`). -- `sk_s` / `sk_n` / `sk_b` — typed sort key, absent when the schema has +- `sk_s` / `sk_n` / `sk_b`: typed sort key, absent when the schema has no sort key. See §5.2 for the type-specific encoding. -- `_v` — OCC version counter used by `UpdateItem`'s versioned filter +- `_v`: OCC version counter used by `UpdateItem`'s versioned filter guard. Absent on freshly-inserted rows (treated as 0); bumped by every update, including the native fast path. -- `item_data` — full DynamoDB item serialized as BSON via the +- `item_data`: full DynamoDB item serialized as BSON via the `AttributeValue` JSON representation. Non-key attribute values retain their DynamoDB type tags (`{"S": "hello"}`, `{"N": "42"}`, ...). @@ -336,14 +336,14 @@ with the base table's key attributes as first-class fields: Two reasons for the extra base-key material: -1. **Unique identity per base item.** GSI keys are non-unique — multiple +1. **Unique identity per base item.** GSI keys are non-unique. Multiple base items can share `(index_pk, index_sk)`. If `_id` were only derived from index keys, two base items with the same GSI-key values would upsert to the same document; one would silently overwrite the other. Including the base keys in `_id` and the entry-delete filter (`index_entry_filter`) makes each entry addressable independently. 2. **Compound pagination cursor.** Index queries and scans sort and - paginate on `(index_sk?, base_pk, base_sk?)` — see §5.3. + paginate on `(index_sk?, base_pk, base_sk?)`; see §5.3. Having the base-key components as fields (not buried under `item_data..S`) lets the sort and cursor filters use per-field indexes. @@ -364,9 +364,9 @@ Simple collation is used whenever the tuple contains a string component. } ``` -- TTL index on `created_at` with `expireAfterSeconds = 24 * 3600` — +- TTL index on `created_at` with `expireAfterSeconds = 24 * 3600`: primary retention enforcement. -- Compound index on `(shard_id, sequence_number)` — powers `GetRecords` +- Compound index on `(shard_id, sequence_number)`: powers `GetRecords` (`shard_id` equality + `sequence_number > cursor` range with ascending sort). Without it, every consumer poll runs a full-collection scan. @@ -423,7 +423,7 @@ The unique index closes a race window: two concurrent `TransactWriteItems` calls with the same token can both miss the other's uncommitted insert; without the constraint, both would commit and the operation would execute twice. With it, the second inserter fails `E11000` and the write path resolves -the winner by re-reading (still subject to the age filter — if the winner has +the winner by re-reading (still subject to the age filter; if the winner has just expired, the retry does a fresh insert). ### 4.7 `_backup_{backup_id}` @@ -438,8 +438,8 @@ drops the collection. ### 5.1 Session-scoped conditional writes -`PutItem`, `DeleteItem`, and `UpdateItem` — when they carry a -`ConditionExpression`, a `StreamCapture`, or write to a table with GSIs — +`PutItem`, `DeleteItem`, and `UpdateItem` (when they carry a +`ConditionExpression`, a `StreamCapture`, or write to a table with GSIs) run inside a MongoDB client session bound to a multi-document transaction with the configured `transaction_read_concern` (default `"snapshot"`) and majority write concern. Within the session: @@ -451,12 +451,12 @@ majority write concern. Within the session: 3. Write (`find_one_and_replace` / `delete_one` / versioned `replace_one`). 4. Synchronize GSIs (`sync_indexes_in_session`). 5. Emit any stream record (`write_stream_inline_in_session`), including - per-shard sequence-number `$inc` — also in the same session. + per-shard sequence-number `$inc`, also in the same session. 6. Commit. In snapshot mode, all steps observe one point-in-time view, so a concurrent -conflicting writer manifests as a WriteConflict at commit — which the caller -retries — rather than as a stale-read anomaly. Majority and local retain the +conflicting writer manifests as a WriteConflict at commit, which the caller +retries, rather than as a stale-read anomaly. Majority and local retain the transaction's atomic commit but weaken those snapshot semantics; reads within the transaction are not guaranteed to share one point-in-time view. Local may also read data that is later rolled back after failover. The pre-image loaded @@ -467,7 +467,7 @@ Update-as-insert (the pre-image was `None`) emits an `INSERT` stream event with no `OldImage`, not a `MODIFY` with a fabricated key-only stub. `UpdateItem` also always fetches the pre-image regardless of the caller's -`ReturnValues` setting — the pre-image is required to compute correct +`ReturnValues` setting: the pre-image is required to compute correct GSI deltas when the update changes or removes an indexed attribute, and skipping it leaves stale entries in index collections forever. @@ -496,7 +496,7 @@ that a static analyzer certifies as safe. The path is gated on: - `pushdown::is_pushable(cond, maps) == Pushable::Yes`. Under those guards, single-document `find_one_and_replace` / -`find_one_and_delete` provides atomicity — no session, no GSI sync, no +`find_one_and_delete` provides atomicity: no session, no GSI sync, no stream write. The compiled filter is merged with the primary-key filter under `$and`. @@ -511,24 +511,24 @@ The **analyzer** (`pushdown.rs::is_pushable`) is the load-bearing correctness boundary. It certifies a whole-condition subset that is provably in agreement with `evaluate_condition`: -- Existence functions (`attribute_exists`, `attribute_not_exists`) — always +- Existence functions (`attribute_exists`, `attribute_not_exists`): always pushable. -- `attribute_type(path, :t)` — pushable when `:t` resolves to a placeholder +- `attribute_type(path, :t)`: pushable when `:t` resolves to a placeholder whose value is one of the 10 valid DDB type tags (`S`, `N`, `B`, `BOOL`, `NULL`, `L`, `M`, `SS`, `NS`, `BS`). Without the whitelist a malicious `:t` could produce a `$`-prefixed pseudo-field. -- `begins_with(path, :S)` — string-only. -- `contains(path, :S)` — string-only. -- `path :S` for any comparator — string operands are stored +- `begins_with(path, :S)`: string-only. +- `contains(path, :S)`: string-only. +- `path :S` for any comparator: string operands are stored verbatim, lex order matches wire order. -- `path = :B` / `path <> :B` — binary equality only. Ordering +- `path = :B` / `path <> :B`: binary equality only. Ordering comparators on binary are refused because the compiler stores B as base64 strings inside `item_data`, and base64 lex order diverges from bytewise order across mismatched lengths. -- `path = / <> :BOOL` and `path = / <> :NULL` — value-only. -- `AND` / `OR` — pushable iff both children are pushable (all-or-nothing; +- `path = / <> :BOOL` and `path = / <> :NULL`: value-only. +- `AND` / `OR`: pushable iff both children are pushable (all-or-nothing; cherry-picking would confuse composition semantics). -- `NOT attribute_exists(path)` / `NOT attribute_not_exists(path)` — the +- `NOT attribute_exists(path)` / `NOT attribute_not_exists(path)`: the only pushable `NOT` forms. Anywhere else, MongoDB's `$nor` semantics on missing paths diverge from DDB's three-valued logic. @@ -577,7 +577,7 @@ single `$gt` / `$lt` sort-key comparison. Naively inserting - Existing equality (`sk = X`) → wrap both under `$and`. **Index Query and Scan cursors.** Index-key values are non-unique, so -pagination cannot rely on `(pk, sk)` alone — items with duplicate index +pagination cannot rely on `(pk, sk)` alone; items with duplicate index keys would form an unstable page boundary. Instead, index queries paginate on the compound tuple `(index_sk?, base_pk, base_sk?)` expressed as a lexicographic `$or`: @@ -599,10 +599,10 @@ in-segment items are accumulated or the cursor exhausts. It does not impose a server-side hard limit. `Parallel Scan` filters items in the application via `crc32(pk) % TotalSegments == Segment`. A hard `(limit + 1) * TotalSegments` limit combined with post-fetch filtering -silently drops items under hot-key skew — an entire limit window can +silently drops items under hot-key skew: an entire limit window can land in one segment, terminating the scan with the others empty. MongoDB batches under the hood (~101 docs), so lazy iteration is -efficient even without a hard limit — at most one extra network batch +efficient even without a hard limit: at most one extra network batch beyond what is returned. Implementation: `data_engine.rs::query_impl`, `scan_impl`, @@ -648,7 +648,7 @@ runs every 5 seconds: `index_status: "ACTIVE"` and unset `backfill_cursor`. Live writes during the backfill window continue to hit `sync_indexes_in_session`, -which writes to CREATING indexes too — index-catalog membership, not +which writes to CREATING indexes too: index-catalog membership, not status, is what gates the write path. All writes are upserts on the same `_id` shape, so a base item touched by both paths converges regardless of interleaving. @@ -666,7 +666,7 @@ top-level `ValidationException`. Stream records are written inline during data operations, using the same storage model as the PostgreSQL backend. This design gives ExtendDB full -control over sequence numbers, shard assignment, and retention — all of +control over sequence numbers, shard assignment, and retention, all of which the DynamoDB Streams API contract tightly specifies. Native MongoDB Change Streams are not used. @@ -689,7 +689,7 @@ as the data write. **Session-scoped sequence numbers.** `next_sequence_number_in_session` does `find_one_and_update` with `$inc` on the per-shard counter -document — inside the write session. Without this, a fast writer B +document, inside the write session. Without this, a fast writer B could draw seq=6 and commit before a slow writer A (which drew seq=5) commits; a consumer polling between B's commit and A's commit sees seq=6 and advances past it, so when A finally commits, seq=5 lands @@ -701,7 +701,7 @@ retries. **Per-shard counters.** Counter documents are keyed by `_id: "stream_seq:"`. A single global counter would couple the sequence spaces of unrelated shards, so a writer pushing records -into shard B would advance the counter shard A reads back — producing +into shard B would advance the counter shard A reads back, producing non-contiguous sequence numbers on shard A's `GetRecords` pages. **Event names.** `event_name_ddb_str` emits DynamoDB wire casing @@ -718,7 +718,7 @@ with ascending sort, backed by the compound index `(shard_id, sequence_number)`. **Non-session `write_stream_record`.** The `StreamEngine::write_stream_record` -trait method is a stub that returns an explicit error — the mongo backend +trait method is a stub that returns an explicit error: the mongo backend has no callers for it, and enrolling in the wrong or no session would let a stream record commit while its base-table write rolls back. @@ -780,12 +780,12 @@ the existing-doc branch. Two TTL surfaces: -**Storage-native TTL indexes** — configured at bootstrap: -- `idempotency_tokens.created_at` — 540s (§4.6 for the rationale). -- `stream_records.created_at` — 24 h. -- `iam_sessions.expires_at` — `expireAfterSeconds: 0`. +**Storage-native TTL indexes**, configured at bootstrap: +- `idempotency_tokens.created_at`: 540s (§4.6 for the rationale). +- `stream_records.created_at`: 24 h. +- `iam_sessions.expires_at`: `expireAfterSeconds: 0`. -**Application-level DynamoDB TTL** — user-configured `TimeToLive` +**Application-level DynamoDB TTL**: user-configured `TimeToLive` attribute per table. MongoDB's native TTL runs at the storage engine and cannot emit ExtendDB stream records with the required `Service` user identity, so the backend maintains its own worker. @@ -840,7 +840,7 @@ construction. `read_catalog_version` returns the `settings.catalog_version` value; `expected_catalog_version` returns the compiled-in `0.0.2`. The bin -layer compares them on `extenddb serve` startup — same pattern as the +layer compares them on `extenddb serve` startup, same pattern as the PostgreSQL backend. ## 6. Crate Structure @@ -951,20 +951,20 @@ layer redacts the password from it for display via the databases lazily on first write, so we explicitly create `schema_history` in the catalog db and `idempotency_tokens` in the data db). -2. `run_catalog_migrations` — create the 17 catalog collections and +2. `run_catalog_migrations`: create the 17 catalog collections and their indexes; seed `settings.catalog_version = 0.0.2`; record the migration in `schema_history`. -3. `create_data_db` — create `idempotency_tokens` (TTL + unique index), +3. `create_data_db`: create `idempotency_tokens` (TTL + unique index), `stream_shards` (unique on shard_id), `stream_records` (TTL + compound query index). -4. `bootstrap_encryption_key` — generate a random 256-bit key, - base64-encode, insert into `settings` (idempotent — E11000 races are +4. `bootstrap_encryption_key`: generate a random 256-bit key, + base64-encode, insert into `settings` (idempotent; E11000 races are silently ignored). -5. `bootstrap_default_account` — insert `default` account if none +5. `bootstrap_default_account`: insert `default` account if none exists. -6. `bootstrap_admin_user` — bcrypt-hash the password, insert into +6. `bootstrap_admin_user`: bcrypt-hash the password, insert into `admin_users`. -7. `record_data_connection` — record `data_database_name` and +7. `record_data_connection`: record `data_database_name` and `data_connection_string` in `settings`. **`extenddb destroy`:** drop both databases. @@ -973,23 +973,23 @@ layer redacts the password from it for display via the Six `inventory::submit!` blocks in `lib.rs`: -- `BackendRegistration` — factory for `MongoBootstrapper`, called by +- `BackendRegistration`: factory for `MongoBootstrapper`, called by `extenddb init`. -- `OperationsEngineRegistration` — CLI operations (connection parsing, +- `OperationsEngineRegistration`: CLI operations (connection parsing, redaction, identifier validation, sensitive-key detection). -- `StorageConfigRegistration` — TOML deserializer for +- `StorageConfigRegistration`: TOML deserializer for `[storage.mongodb]`. -- `SettingsStoreRegistration` — factory for `MongoCatalogStore` acting +- `SettingsStoreRegistration`: factory for `MongoCatalogStore` acting as `SettingsStore`. -- `DiagnosticsStoreRegistration` — factory for `MongoCatalogStore` +- `DiagnosticsStoreRegistration`: factory for `MongoCatalogStore` acting as `DiagnosticsStore`. -- `ServerComponentsRegistration` — factory called by `extenddb serve` +- `ServerComponentsRegistration`: factory called by `extenddb serve` that constructs `MongoEngine`, `MongoCatalogStore`, `MongoCredentialStore`, and `MongoRuntimeHooks` (which spawns the TTL, stream-cleanup, and GSI-backfill workers). No changes to `crates/server/`, `crates/engine/`, or `crates/auth/` -are required — everything flows through the plugin registration +are required; everything flows through the plugin registration system. ## 11. Dependencies @@ -1030,46 +1030,46 @@ proptest = "1" # pushdown parity harness The backend implements every trait in `extenddb-storage`: **Data plane** -- `TableEngine` — Create/Delete/Describe/List/UpdateTable, including +- `TableEngine`: Create/Delete/Describe/List/UpdateTable, including GSI create with async backfill, LSI create, and idempotent stream-enable on UpdateTable. -- `DataEngine` — PutItem, GetItem, DeleteItem, UpdateItem, Query, +- `DataEngine`: PutItem, GetItem, DeleteItem, UpdateItem, Query, Scan (including parallel scan), TransactGetItems, TransactWriteItems, BatchGetItem, BatchWriteItem. Condition expressions run session- scoped with an analyzer-gated pushdown fast path for a certified subset. -- `StreamEngine` — session-scoped per-shard sequence numbers, four +- `StreamEngine`: session-scoped per-shard sequence numbers, four shards per stream, CRC32 pk routing, TRIM_HORIZON / LATEST / AT_SEQUENCE_NUMBER / AFTER_SEQUENCE_NUMBER iterators, 24h retention. -- `MetadataEngine` — TTL lifecycle, tags, table-size tracking. -- `BackupEngine` — CreateBackup, RestoreTableFromBackup, DeleteBackup +- `MetadataEngine`: TTL lifecycle, tags, table-size tracking. +- `BackupEngine`: CreateBackup, RestoreTableFromBackup, DeleteBackup via server-side `$out` aggregation. **Control plane and catalog** -- `Bootstrapper` — init, destroy, migrate, verify. Creates the +- `Bootstrapper`: init, destroy, migrate, verify. Creates the catalog and data databases, seeds encryption key and admin user, applies index schema. -- `WorkerStore` — `process_control_plane_transitions` flips tables +- `WorkerStore`: `process_control_plane_transitions` flips tables from `CREATING` to `ACTIVE` once their `status_transition_at` passes. `create_table_impl` (and restore) write `CREATING` with a scheduled transition when `control_plane_delay_seconds` > 0, or `ACTIVE` directly when it is 0. `delete_table_impl` remains inline (no `DELETING` transient state). - `ManagementStore`, `AdminStore`, `SettingsStore`, `MetricsStore`, - `RateLimitStore` — the catalog trait surface. -- `AuthorizationStore` — user/group/role/permissions-boundary/session + `RateLimitStore`: the catalog trait surface. +- `AuthorizationStore`: user/group/role/permissions-boundary/session policy lookup for IAM evaluation. -- `MongoCredentialStore` — SigV4 credential resolution with +- `MongoCredentialStore`: SigV4 credential resolution with AES-GCM-decrypted secret keys. -**Background workers** — spawned from `MongoRuntimeHooks::spawn_workers`: -- `ttl_cleanup_worker` — sweep expired items every 60 s, emit +**Background workers**, spawned from `MongoRuntimeHooks::spawn_workers`: +- `ttl_cleanup_worker`: sweep expired items every 60 s, emit service-attributed stream records for the deletes. -- `stream_record_cleanup_worker` — hourly defense-in-depth for the +- `stream_record_cleanup_worker`: hourly defense-in-depth for the 24 h retention TTL index. -- `gsi_backfill_worker` — drain `indexes` rows in `CREATING` state, +- `gsi_backfill_worker`: drain `indexes` rows in `CREATING` state, scan the base collection with a persistent cursor, flip to ACTIVE. -- `control_plane_worker` — flip `tables` rows from `CREATING` to +- `control_plane_worker`: flip `tables` rows from `CREATING` to `ACTIVE` once their scheduled `status_transition_at` passes. ## 13. Testing Strategy @@ -1077,8 +1077,8 @@ The backend implements every trait in `extenddb-storage`: - **Unit tests:** netstring `_id` encoding, hex sort-key ordering, condition compiler, shard-id derivation, next_string_prefix, Decimal128 rejection, index-doc key disambiguation. -- **Property tests:** `tests/pushdown_parity.rs` — random items and - expressions checked for agreement between the compiled BSON filter +- **Property tests:** `tests/pushdown_parity.rs` checks random items and + expressions for agreement between the compiled BSON filter and `evaluate_condition`. - **Integration tests:** The dual-target `tests/rust/` SDK suite (the same AWS-SDK wire tests the PostgreSQL backend runs) executed against a @@ -1131,11 +1131,11 @@ The backend implements every trait in `extenddb-storage`: ## 16. Performance Characteristics -**Hot path — single-item conditional write.** One MongoDB transaction +**Hot path: single-item conditional write.** One MongoDB transaction session covers pre-image read, condition eval, base write, GSI sync, stream insert (per-shard counter `$inc` + document insert). On a local replica set, session overhead is ~sub-ms over a raw driver call. The -session is what buys the DDB atomicity contract — it is the +session is what buys the DDB atomicity contract; it is the compatibility, not overhead. The pushdown fast path collapses this to a single `find_one_and_*` for the certified subset on tables with no GSIs / streams. diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index ebaf3318c..aa0c1238c 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -1,6 +1,11 @@ -# Design Spec: Off-Host Backup and Restore for ExtendDB - -Status: Draft for maintainer review. Written against main `abf8082f` (which carries PR 0 = #384 as `9d13202`, the Cassandra backend, #338, and the PostgreSQL GSI build, #401) and the open series PR 2 = #396 (this document), PR 3 = #393 (supersedes #336, head `aef17ddd`), PR 4 = #394 (supersedes #337, head `3f7107dc`), PR 5 = #395 (stacked on #394, head `069c4b17`); PR 1 is #335 rebased. The revision log at the end holds the dates of each revision. Target: v1.0. +--- +description: "Off-host backups: one portable format the engine writes to a local directory or an S3 bucket, and the backup and restore operations built on it." +status: "Draft" +target: "v1.0" +base: "main `abf8082f` (which carries PR 0 = #384 as `9d13202`, the Cassandra backend, #338, and the PostgreSQL GSI build, #401) and the open series PR 2 = #396 (this document), PR 3 = #393 (supersedes #336, head `aef17ddd`), PR 4 = #394 (supersedes #337, head `3f7107dc`), PR 5 = #395 (stacked on #394, head `069c4b17`); PR 1 is #335 rebased" +--- + +# Design spec: Off-host backup and restore for ExtendDB ## Summary diff --git a/docs/design/README.md b/docs/design/README.md index 1f41f48ad..f6c519280 100755 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -35,6 +35,6 @@ This directory contains the authoritative design documents for ExtendDB (extendd Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 94183748f..9320f4f64 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -8,11 +8,42 @@ This document lists all known behavioral differences between ExtendDB and the Amazon DynamoDB service. Use it to understand what works identically and what requires adaptation when switching between ExtendDB and the service. +## Feature Status + +This table is the one record of which DynamoDB features ExtendDB implements. The sections that follow +give the behavioral detail for each area. + +| Feature | Status | +|---------|--------| +| CreateTable / DeleteTable / DescribeTable / ListTables / UpdateTable | ✓ | +| PutItem / GetItem / UpdateItem / DeleteItem | ✓ | +| Query / Scan (with filter, projection, pagination) | ✓ | +| BatchWriteItem / BatchGetItem | ✓ | +| TransactWriteItems / TransactGetItems | ✓ | +| Global Secondary Indexes (GSI) | ✓ | +| Vector indexes and SearchVectors | ✓ (PostgreSQL needs pgvector; refused on MongoDB and Cassandra) | +| Local Secondary Indexes (LSI) | ✓ | +| DynamoDB Streams | ✓ (Cassandra: see Storage and Infrastructure) | +| ConditionExpression / FilterExpression / UpdateExpression / ProjectionExpression | ✓ | +| KeyConditionExpression | ✓ | +| ReturnValues / ReturnConsumedCapacity | ✓ | +| TTL (DescribeTimeToLive / UpdateTimeToLive) | ✓ | +| Tagging (TagResource / UntagResource / ListTagsOfResource) | ✓ | +| SigV4 Authentication | ✓ | +| IAM Policy Evaluation | ✓ | +| Legacy API (Expected, AttributeUpdates, AttributesToGet) | ✓ | +| PartiQL (ExecuteStatement, BatchExecuteStatement, ExecuteTransaction) | ✗ Not implemented (see Operations Not Implemented) | +| DAX | ✗ (not applicable) | +| Global Tables | ✗ Not implemented (see Operations Not Implemented) | +| On-demand backups (CreateBackup, RestoreTableFromBackup, DescribeBackup, ListBackups, DeleteBackup) | ✓ (see Backup and Restore below) | +| Point-in-time recovery | ✗ Not supported (see Backup and Restore) | +| Import/Export (local filesystem) | ✓ (FileSource instead of S3) | + ## Storage and Infrastructure | Area | DynamoDB | ExtendDB | |------|----------|------| -| Storage backend | Proprietary distributed storage | PostgreSQL (default), SQLite, MongoDB, or Cassandra, selected by mutually exclusive Cargo features at build time; one backend per binary | +| Storage backend | Proprietary distributed storage | PostgreSQL (default), SQLite, MongoDB, or Cassandra (experimental), selected by mutually exclusive Cargo features at build time; one backend per binary | | Global Tables | CreateGlobalTable, replication | Not implemented (returns UnknownOperationException) | | DAX (Accelerator) | In-memory caching layer | Not applicable | | PartiQL | ExecuteStatement, BatchExecuteStatement | Not implemented (returns UnknownOperationException) | @@ -21,6 +52,7 @@ adaptation when switching between ExtendDB and the service. | `TransactGetItems` overlapping an in-flight `TransactWriteItems` on the same item | May cancel the read with `TransactionCanceledException` and reason `TransactionConflict` | Never cancels for this reason on PostgreSQL: the read runs in one `REPEATABLE READ` snapshot taken at its first read, so it sees either all of the write or none of it and returns the pre-write state while the write is in flight. That is a legal serial order (the read ordered before the write) and the all-or-nothing guarantee holds, but a client that retries on `TransactionConflict` to observe the newest state will not get that signal here. SQLite behaves the same (one WAL snapshot); MongoDB with the default `snapshot` read concern likewise, and the row above covers the other read concerns. | | Transaction read concern (MongoDB backend only) | No user-configurable equivalent | `snapshot` is the default, fidelity-preserving mode. With `majority` or `local`, `TransactGetItems` is not guaranteed a single point-in-time snapshot and transaction condition reads have weaker isolation. With `local`, a condition can be evaluated against data that is later rolled back after failover. | | Contended `TransactWriteItems` | A transaction that conflicts with another in-flight transaction on the same item is canceled with a `TransactionConflict` reason ("Transaction is ongoing for the item") | **PostgreSQL** queues the second transaction on the first one's row locks and **SQLite** runs one writer at a time, so both normally commit and no `TransactionConflict` is returned. PostgreSQL locks each transaction's items in table and key order, so two `TransactWriteItems` cannot deadlock on each other. A PutItem, UpdateItem, DeleteItem or BatchWriteItem of an item that a `TransactWriteItems` has locked also waits for it, where the Amazon DynamoDB developer guide lists that request as a conflict. On PostgreSQL this includes an item that the transaction checks or deletes while it is missing: the transaction holds the key until it commits or rolls back, so a request that creates the item waits. Many transactions that check one hot missing item queue less fairly than on an existing item, because every waiter wakes at each commit and only one proceeds. Each such check or delete of a missing item also inserts and deletes its key, so it writes WAL and leaves a dead row and index entry for autovacuum, even though it changes no item. When PostgreSQL still aborts a transaction (a deadlock with another lock holder, or a serialization failure under a stricter `default_transaction_isolation`), the answer is a `TransactionCanceledException` with a `TransactionConflict` reason, not a 500. **MongoDB** retries write conflicts internally and cancels with `TransactionConflict` on every item only after sustained contention. A client that relies on `TransactionConflict` to back off does not see it under ordinary contention. | +| Stream record for an item write (Cassandra backend only) | Every item-level change appears in the stream exactly once | PutItem, UpdateItem and a conditional DeleteItem on a table without TTL, and every item a TransactWriteItems writes, commit the item first with a lightweight transaction and write its stream record in a following logged batch, so a failure between the two leaves the item changed with no stream record. On a TTL-enabled table, PutItem, UpdateItem and DeleteItem write the item and its stream record in one logged batch, and an unconditional DeleteItem does so on any table. BatchWriteItem writes each item through the PutItem and DeleteItem paths. | ## Authentication and Authorization (AWS IAM/STS auth surface used by DynamoDB) @@ -32,7 +64,7 @@ adaptation when switching between ExtendDB and the service. | Role chaining | Supported | Not implemented | | SourceIdentity, TransitiveTagKeys | Supported | Not implemented | | Resource policies | Supported | Not implemented (deferred) | -| Table ARN as `TableName` | Accepted; the request acts on that table | Authorized against that table, then the request fails with `ResourceNotFoundException` | +| Table ARN as `TableName` | Accepted; the request acts on that table | Not accepted: the request is authorized against the named table and then refused with `ResourceNotFoundException` | | AccessDenied message for an ARN outside this deployment (caller with no policy) | Names the ARN as supplied; an ARN in another account gets `Access is denied` | Names the ARN rewritten to this deployment's partition, service, region and account; an ARN in another account gets the full `is not authorized to perform` message. The decision is the same. | ## Import and Export @@ -66,7 +98,7 @@ adaptation when switching between ExtendDB and the service. | `DescribeTimeToLive` transitional states | `ENABLING` and `DISABLING` are reported while a change settles | Matches on Cassandra (all four states; `ENABLED` means the queue covers every item) and on PostgreSQL for enable (`ENABLING` until the concurrent index build publishes; its disable is synchronous so `DISABLING` never appears). During `DISABLING`, `AttributeName` is absent, unlike DynamoDB. Updates are rejected while a transition is settling, as in DynamoDB. | | Concurrent same-key writes on a TTL-enabled table | Unaffected by TTL | Serialized through a base-row claim. Contention is retried internally against a re-read image, so writes stay last-writer-wins; only sustained contention surfaces `TransactionConflictException`. | | TTL-enabled write cost | Unaffected by TTL | A claim LWT, with the base write becoming a logged batch (measured: roughly 2× put latency on a single-node bench). The TTL configuration is served from a 5-second cache, the remaining catalog read is overlapped with the index read, and the release is folded into the batch with an exact conditional release off the request path. Non-TTL tables keep the existing fast paths. | -| TTL on a table with an asynchronously propagated GSI (Cassandra backend only) | Supported | `UpdateTimeToLive` enabling TTL on such a table is a `ValidationException` naming the table, the first asynchronous GSI and its propagation delay. Before, it was an `InternalServerError`, which SDKs retry. A GSI is asynchronous unless its delay, or `index_propagation_delay_ms` when it sets none, is 0. Checked against Cassandra with a storage-level test and on a served build at the default delay (`.review` logs cited in the PR); the wire suite runs Cassandra at delay 0, where TTL is enabled | +| TTL on a table with an asynchronously propagated GSI (Cassandra backend only) | Supported | `UpdateTimeToLive` enabling TTL on such a table is a `ValidationException` naming the table, the first asynchronous GSI and its propagation delay. A GSI is asynchronous unless its delay, or `index_propagation_delay_ms` when it sets none, is 0. Checked against Cassandra with a storage-level test and on a running server at the default delay; the wire suite runs Cassandra at delay 0, where TTL is enabled | | Cassandra TTL with asynchronous GSIs | Supported | Not currently supported. The Cassandra backend rejects TTL enable when any GSI has a nonzero effective propagation delay, and rejects creating such a GSI on a TTL-enabled table; base tables, LSIs, and synchronous GSIs are supported. See [ADR-0010](adr/0010-cassandra-ttl-expiration-queue.md). | | Enabling Cassandra TTL on a live table | No application write quiescence required | Writes that began while TTL was disabled must be quiesced until enable and synchronous backfill complete. Prefer enabling TTL before opening a new table to traffic. Once enabled, writes use the durable exact-claim path. | | TTL expiration throughput | Managed by the service | Cassandra: about 1,000 items per table per minute: the sweep processes its batch concurrently across the queue's shard partitions under one per-table lease. Other backends: about 100 per table per minute (a serial 100-item batch per 60-second cycle, with unleased hosts contending for the same rows). A table expiring faster than its backend's rate accumulates backlog. | @@ -88,13 +120,13 @@ adaptation when switching between ExtendDB and the service. | Area | DynamoDB | ExtendDB | |------|----------|------| | GSI update propagation | Eventually consistent (milliseconds to seconds) | Per-GSI propagation delay. System default: `index_propagation_delay_ms` setting (default 10ms). Each GSI can override with its own `propagation_delay_ms` (stored in catalog). A value of 0 means synchronous (future sync GSI feature). | -| Vector index update propagation | Eventually consistent, the same model as a GSI | Matches DynamoDB. Maintenance is queued on the same propagation queue as async GSIs, so a search immediately after a write may not see it. Governed by the same `index_propagation_delay_ms` setting; unlike a GSI there is no per-index override. A value of 0 applies maintenance inline in the write's own transaction, which is stricter than the service and exists so a test can assert steady state without waiting. Inline additionally requires the table's propagation queue to be empty: while rows queued during an index build are still draining, a new write queues behind them instead of overtaking them, so a brief search-after-write window exists even at 0 until the queue drains. That window is ordering, not data loss; the write is applied, in order, by the queue worker. That zero behaves differently while an index is still building, and the two backends differ: **PostgreSQL** applies inline only to an index that is already `ACTIVE`, and defers a write to a building index whatever the delay says, because the write must not reach the index ahead of the backfill's older snapshot of the same item; **SQLite** does not check the status on this path, so with a zero delay a write during a backfill is applied inline and bypasses the hold that keeps the two ordered. The SQLite behaviour is tracked as a defect (F-20), not intended, and it is reachable only with the zero delay, which is a test setting. | +| Vector index update propagation | Eventually consistent, the same model as a GSI | Matches DynamoDB. Maintenance is queued on the same propagation queue as async GSIs, so a search immediately after a write may not see it. Governed by the same `index_propagation_delay_ms` setting; unlike a GSI there is no per-index override. A value of 0 applies maintenance inline in the write's own transaction, which is stricter than the service and exists so a test can assert steady state without waiting. Inline additionally requires the table's propagation queue to be empty: while rows queued during an index build are still draining, a new write queues behind them instead of overtaking them, so a brief search-after-write window exists even at 0 until the queue drains. That window is ordering, not data loss; the write is applied, in order, by the queue worker. That zero behaves differently while an index is still building, and the two backends differ: **PostgreSQL** applies inline only to an index that is already `ACTIVE`, and defers a write to a building index whatever the delay says, because the write must not reach the index ahead of the backfill's older snapshot of the same item; **SQLite** does not check the status on this path, so with a zero delay a write during a backfill is applied inline and bypasses the hold that keeps the two ordered. The SQLite behaviour is a known defect, not intended, and it is reachable only with the zero delay, which is a test setting. | | `SearchVectors` score at extreme magnitudes (PostgreSQL backend only) | Returns the true distance as a number for any vector of finite components | The score is bounded to a finite value instead, and the bound is not a measured service answer. pgvector accumulates distances in single precision, so magnitudes far below `f32::MAX` overflow inside the extension: Euclidean above about 9.2e18, dot product above about 1.8e19, and cosine at both ends, above about 1.8e19 and below about 3.7e-23, which is where a component's single-precision square rounds to zero. A non-finite score cannot be serialised as JSON at all, so the result is bounded in SQL: `1e308` for an overflowed distance, `-1e308` for an overflowed negated inner product, which the score contract negates so a client sees `1e308` in `Score`, and 1.0 for a cosine that comes back NaN. Ranking is unaffected at the overflow end, because each bound sits at the end its metric overflows towards, so the farthest row stays farthest and the most similar stays most similar; two rows that both overflow tie, and the tie breaks on the base key. At the underflow end cosine loses resolution rather than being bounded, and which side is tiny decides how much. For a tiny **query** vector, its norm underflows while the inner product usually does not, so the quotient is an infinity that pgvector clamps and the reported distance collapses to one of 0, 1 or 2 following the sign of the inner product, with the 1.0 substitute firing only when the vectors are exactly orthogonal and the quotient is therefore 0/0 (measured). For a tiny **stored** vector it is worse: the stored norm is computed in single precision and reaches zero, so the zero-vector guard fires and that row reports 1.0 at every angle, parallel included. A corpus of tiny embeddings therefore loses ranking altogether rather than losing resolution. For a tiny query vector, by contrast, ranking still separates nearer-than-orthogonal from farther and loses resolution only within each half. The SQLite backend owns its own arithmetic, computes in double precision, and reports the true value, which is why this row is scoped to PostgreSQL. | | Vector index deletion window | `UpdateTable` Delete leaves the index in `DELETING` long enough to observe, then removes it | No observable `DELETING` window on either backend: the catalog row is removed inside the `UpdateTable` transaction, so a `DescribeTable` immediately afterwards already omits the index. The index's data table is dropped after that commit, in a separate transaction, and the two backends handle a failure there differently. **PostgreSQL** treats it as best effort, because the data table lives in a different database entirely: a failure is logged and skipped rather than failing the request. **SQLite** propagates it, so a failed drop returns an error from an `UpdateTable` whose catalog change has already committed, which is the more surprising outcome of the two: the index is gone from the catalog and the caller saw a failure. So an operator debugging a leftover `_ddb_vec_*` table should look for that warning rather than assume the delete was incomplete. | | Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Not supported. On PostgreSQL and SQLite, the two backends with vector indexes, `CreateBackup` of such a table is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend", and no backup is created, because a restore cannot carry the index data across. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | -| `SearchVectors` endpoint | Served only on `search-dynamodb..amazonaws.com`. The standard `dynamodb..amazonaws.com` endpoint answers the same request with HTTP 400 `UnknownOperationException` ("This operation is not supported by this endpoint"); every control-plane and item operation stays on the standard endpoint. Signing is unchanged either way (service name `dynamodb`, target prefix `DynamoDB_20120810`) | Served on the same endpoint as every other operation, so a client is pointed at one ExtendDB endpoint for all of them. Two consequences worth knowing: an SDK that resolves a separate search hostname from its endpoint ruleset needs its endpoint overridden to reach ExtendDB, and ExtendDB does **not** reproduce the service's refusal, so a test asserting `UnknownOperationException` for vector search on the base endpoint passes against Amazon DynamoDB and fails here. | +| `SearchVectors` endpoint | Served only on `search-dynamodb..amazonaws.com`. The standard `dynamodb..amazonaws.com` endpoint answers the same request with HTTP 400 `UnknownOperationException` ("This operation is not supported by this endpoint"); every control-plane and item operation stays on the standard endpoint. Signing is unchanged either way (service name `dynamodb`, target prefix `DynamoDB_20120810`) | Served on the same endpoint as every other operation, so a client is pointed at one ExtendDB endpoint for all of them. Two consequences: an SDK that resolves a separate search hostname from its endpoint ruleset needs its endpoint overridden to reach ExtendDB, and ExtendDB does **not** reproduce the service's refusal, so a test asserting `UnknownOperationException` for vector search on the base endpoint passes against Amazon DynamoDB and fails here. | | `SearchVectors` result order for equal distances | Measured unstable: three identical searches returned tied rows in three different orders, and a top-k that truncates a tie group keeps an arbitrary subset of it | Deterministic on both backends, by different means. **PostgreSQL** sorts explicitly on the base table's full primary key after the score, so the order is a property of the query. **SQLite** issues no `ORDER BY` and resolves ties by scan order through a stable top-k, so its order is a property of the plan rather than something the query guarantees. Either way a client re-issuing an identical search sees the same order, which is stricter than the service rather than divergent in outcome. Do not rely on the two backends agreeing on which subset of a truncated tie group they keep. | -| Vector indexes on the MongoDB backend | Vector indexes and `SearchVectors` are available | Not supported at all. That backend provides no vector search implementation, so the engine's capability gate refuses `CreateTable` and `UpdateTable` carrying `VectorIndexes` and every `SearchVectors` request, with the same capability message a PostgreSQL deployment without pgvector returns. Every other vector row in this document describes the PostgreSQL and SQLite backends. | +| Vector indexes on the MongoDB and Cassandra backends | Vector indexes and `SearchVectors` are available | Not supported at all. Neither backend provides a vector search implementation, so the engine's capability gate refuses `CreateTable` and `UpdateTable` carrying `VectorIndexes` with `ValidationException` ("Vector indexes are not supported by this storage backend") and every `SearchVectors` request with `ValidationException` ("SearchVectors is not supported by this storage backend"), the same messages a PostgreSQL deployment without pgvector returns. Every other vector row in this document describes the PostgreSQL and SQLite backends. | | Multi-part base table keys | Not supported | Preview extension (opt-in via `enable_multipart_keys` setting). Standard single/composite keys work identically. | | GSI throughput on a `PROVISIONED` table | Every GSI needs `ProvisionedThroughput` of at least 1 read and 1 write unit | A GSI `ProvisionedThroughput` below 1 (including 0/0) is `ValidationException` on `CreateTable` of a `PROVISIONED` table and on `UpdateTable`'s GSI `Create` ("ReadCapacityUnits and WriteCapacityUnits must both be greater than or equal to 1 for index: "; the service's text was not captured). A GSI with no `ProvisionedThroughput` is still accepted on a `PROVISIONED` table, and `UpdateTable` to `PROVISIONED` leaves an on-demand table's GSIs without one; see the `BillingModeOverride` row under Backup and Restore for how a backup of such a table restores | | Projected attributes across all indexes | At most 100 | Enforced on `CreateTable` (and so on `RestoreTableFromBackup`, which validates its target the same way) and on `UpdateTable`'s GSI `Create`: every `NonKeyAttributes` entry of every GSI and LSI is counted, so an attribute projected into two indexes counts twice. Over 100 is `ValidationException` ("Number of projected attributes in all indexes exceeds limit of 100, projected: "; the service's text was not captured) | @@ -190,6 +222,6 @@ and account management. DynamoDB uses the AWS Management Console. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index 18d6160e0..823ac996c 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -194,6 +194,6 @@ The following unenforced limits are tracked in the repository's [technical debt Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/first-requests.md b/docs/first-requests.md index f44b59a67..56966aa0e 100644 --- a/docs/first-requests.md +++ b/docs/first-requests.md @@ -343,6 +343,6 @@ aws dynamodb delete-table --table-name StreamTable Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/install-from-source.md b/docs/install-from-source.md index c96f14810..98e8a04e0 100644 --- a/docs/install-from-source.md +++ b/docs/install-from-source.md @@ -552,6 +552,6 @@ docker exec extenddb-cassandra cqlsh -u cassandra -p cassandra \ Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/local-dev-docker-compose.md b/docs/local-dev-docker-compose.md index 768d091e4..192f84cee 100644 --- a/docs/local-dev-docker-compose.md +++ b/docs/local-dev-docker-compose.md @@ -173,6 +173,6 @@ aws ecr-public get-login-password --region us-east-1 | \ Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index 000c57c25..9af89f8d6 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -8,7 +8,7 @@ description: "The parts of an ExtendDB server, how a request flows through them, ## Overview -ExtendDB (ExtendDB) is a standalone DynamoDB-compatible API server written in Rust. It receives DynamoDB wire protocol +ExtendDB is a standalone DynamoDB-compatible API server written in Rust. It receives DynamoDB wire protocol requests over HTTP/HTTPS, authenticates and authorizes them via SigV4 and a local IAM policy engine, executes operation logic in a backend-agnostic engine, and delegates persistence to a pluggable storage backend (PostgreSQL, SQLite, MongoDB, or Apache Cassandra). @@ -189,7 +189,7 @@ expect. 3. Fork to background via `daemonize`, writing a PID file to `run_dir` (skipped entirely in `--foreground` mode, so no run directory is needed and the root filesystem can be read-only) -4. Initialize logging — syslog when daemonized, stderr in `--foreground` +4. Initialize logging: syslog when daemonized, stderr in `--foreground` 5. Connect to the storage backend 6. Verify catalog version matches binary expectation 7. Start axum server on the pre-bound socket @@ -241,7 +241,7 @@ The IAM policy engine evaluates: - Permissions boundaries - Explicit Deny takes precedence over Allow -Unparseable policies fail closed — a corrupted Deny policy results in access denied, not silent skip. +Unparseable policies fail closed: a corrupted Deny policy results in access denied, not silent skip. ### Transport @@ -260,7 +260,7 @@ All user-supplied strings are validated at the engine layer before reaching stor ## Web Console -The management web console is a server-rendered HTML interface mounted at `/console/*`. It runs inside the `server` crate — no separate process, no external dependencies. All HTML is generated with Rust string formatting. +The management web console is a server-rendered HTML interface mounted at `/console/*`. It runs inside the `server` crate: no separate process, no external dependencies. All HTML is generated with Rust string formatting. ### Features @@ -278,7 +278,7 @@ The management web console is a server-rendered HTML interface mounted at `/cons ### Architecture -The console uses the `ManagementStore` trait for all data access — it never queries the database directly. Write operations route through shared functions in `management::ops`, ensuring validation and business logic are defined once. This means every action available in the console is also available through the management REST API. +The console uses the `ManagementStore` trait for all data access; it never queries the database directly. Write operations route through shared functions in `management::ops`, ensuring validation and business logic are defined once. This means every action available in the console is also available through the management REST API. ### Authentication Model @@ -331,6 +331,6 @@ The storage backend provides durability, replication, and backup. Use that backe Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/02-design-guide.md b/docs/manuals/02-design-guide.md index 509c3d79b..7b055dfa6 100755 --- a/docs/manuals/02-design-guide.md +++ b/docs/manuals/02-design-guide.md @@ -14,8 +14,8 @@ ExtendDB manages two kinds of stored state in every storage backend. The examples below name the PostgreSQL databases; other backends map the same two roles onto their own structures (see backend-specific documentation). -- **Catalog** (e.g., the `extenddb_catalog` PostgreSQL database): All metadata — table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. -- **Data** (e.g., the `extenddb_catalog_data` PostgreSQL database): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. +- **Catalog** (e.g., the `extenddb_catalog` PostgreSQL database): All metadata, meaning table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. +- **Data** (e.g., the `extenddb` PostgreSQL database): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. On PostgreSQL, the data database connection string is stored in the catalog's `settings` table under the key `data_database_url`. This allows the catalog and data databases to live on different PostgreSQL instances. @@ -80,7 +80,7 @@ Condition evaluation happens inside the storage transaction (after `SELECT FOR U ### FilterExpression -Applied after reads (Query/Scan) to exclude non-matching items. Same syntax as ConditionExpression. Filter expressions do not reduce consumed capacity — all scanned items count toward RCU. +Applied after reads (Query/Scan) to exclude non-matching items. Same syntax as ConditionExpression. Filter expressions do not reduce consumed capacity; all scanned items count toward RCU. ### UpdateExpression @@ -120,11 +120,11 @@ Full SigV4 signature verification: After authentication, the authorization layer evaluates IAM policies using a 5-phase algorithm: -1. **Explicit Deny** — scan all policies (identity, permissions boundary, session). Any matching Deny → access denied. -2. **Permissions Boundary** — if set, must contain a matching Allow → else denied. -3. **Session Policy** — if set (AssumeRole), must contain a matching Allow → else denied. -4. **Identity Allow** — scan identity policies (user, group, role). Any matching Allow → access granted. -5. **Implicit Deny** — no matching Allow → access denied. +1. **Explicit Deny**: scan all policies (identity, permissions boundary, session). Any matching Deny → access denied. +2. **Permissions Boundary**: if set, must contain a matching Allow → else denied. +3. **Session Policy**: if set (AssumeRole), must contain a matching Allow → else denied. +4. **Identity Allow**: scan identity policies (user, group, role). Any matching Allow → access granted. +5. **Implicit Deny**: no matching Allow → access denied. Policy conditions support all IAM condition operators: `StringEquals`, `StringNotEquals`, `StringEqualsIgnoreCase`, `StringLike`, `StringNotLike`, `NumericEquals`, `NumericNotEquals`, `NumericLessThan`, `NumericLessThanEquals`, `NumericGreaterThan`, `NumericGreaterThanEquals`, `DateEquals`, `DateNotEquals`, `DateLessThan`, `DateLessThanEquals`, `DateGreaterThan`, `DateGreaterThanEquals`, `Bool`, `Null`, `ArnEquals`, `ArnNotEquals`, `ArnLike`, `ArnNotLike`, plus `ForAllValues`, `ForAnyValue`, and `IfExists` modifiers. Supported condition keys include `aws:PrincipalTag/*`, `aws:ResourceTag/*`, `dynamodb:LeadingKeys`, `dynamodb:Attributes`, `dynamodb:Select`, `dynamodb:ReturnValues`, `dynamodb:ReturnConsumedCapacity`, `dynamodb:FullTableScan`, and `dynamodb:EnclosingOperation`. @@ -132,7 +132,7 @@ Policy conditions support all IAM condition operators: `StringEquals`, `StringNo Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `encryption_keys` table. Each access key record stores the encrypted secret and a unique nonce. -Credential lookups (access key → encrypted secret) read directly from the database on every request — there is no in-process cache for credentials. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below). +Credential lookups (access key → encrypted secret) read directly from the database on every request; there is no in-process cache for credentials. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below). ## DynamoDB Streams Internals @@ -202,7 +202,7 @@ All cached values are operational tuning knobs where a briefly-stale value does Catalog state is never cached because correctness requires every request to see the current state: -- **Table metadata** (key schema, attribute definitions, status, billing mode): A stale cache could serve the wrong key schema after a table is deleted and recreated with the same name but different schema. The new table has a different `table_id`, different key schema, and different indexes — stale cache serves wrong schema, writes corrupt data, reads return garbage. +- **Table metadata** (key schema, attribute definitions, status, billing mode): A stale cache could serve the wrong key schema after a table is deleted and recreated with the same name but different schema. The new table has a different `table_id`, different key schema, and different indexes: stale cache serves wrong schema, writes corrupt data, reads return garbage. - **IAM policies and credentials**: A revoked Deny policy still cached as absent creates a security gap. A deleted access key still cached as valid allows unauthorized access. - **Tags**: Tag-based authorization (`aws:ResourceTag/*`) requires current tag values. - **GSI definitions**: Stale GSI metadata could route writes to wrong index tables. @@ -235,6 +235,6 @@ Caching of operational settings is currently unconditional. If issues arise (e.g Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/03-usage-guide.md b/docs/manuals/03-usage-guide.md index d980ab046..537daef08 100755 --- a/docs/manuals/03-usage-guide.md +++ b/docs/manuals/03-usage-guide.md @@ -551,33 +551,11 @@ ExtendDB returns the DynamoDB error types and, where measured against the servic | `UnrecognizedClientException` | 403 | Invalid access key | | `InternalServerError` | 500 | Unexpected server error | -## Supported Features Summary - -| Feature | Status | -|---------|--------| -| CreateTable / DeleteTable / DescribeTable / ListTables / UpdateTable | ✓ | -| PutItem / GetItem / UpdateItem / DeleteItem | ✓ | -| Query / Scan (with filter, projection, pagination) | ✓ | -| BatchWriteItem / BatchGetItem | ✓ | -| TransactWriteItems / TransactGetItems | ✓ | -| Global Secondary Indexes (GSI) | ✓ | -| Vector indexes and SearchVectors | ✓ (PostgreSQL requires the pgvector extension) | -| Local Secondary Indexes (LSI) | ✓ | -| DynamoDB Streams | ✓ | -| ConditionExpression / FilterExpression / UpdateExpression / ProjectionExpression | ✓ | -| KeyConditionExpression | ✓ | -| ReturnValues / ReturnConsumedCapacity | ✓ | -| TTL (DescribeTimeToLive / UpdateTimeToLive) | ✓ | -| Tagging (TagResource / UntagResource / ListTagsOfResource) | ✓ | -| SigV4 Authentication | ✓ | -| IAM Policy Evaluation | ✓ | -| Legacy API (Expected, AttributeUpdates, AttributesToGet) | ✓ | -| PartiQL (ExecuteStatement, BatchExecuteStatement, ExecuteTransaction) | ✗ (not implemented; returns UnknownOperationException) | -| DAX | ✗ (not applicable) | -| Global Tables | ✗ (not implemented; returns UnknownOperationException) | -| On-demand backups (CreateBackup, RestoreTableFromBackup, DescribeBackup, ListBackups, DeleteBackup) | ✓ (see the backup rows on the differences page) | -| Point-in-time recovery | ✗ (RestoreTableToPointInTime is refused with ValidationException) | -| Import/Export (local filesystem) | ✓ (FileSource instead of S3) | +## Supported features + +The status of every feature area (implemented, not implemented, or implemented with a difference) is +recorded in one place: the [Feature Status](../differences-from-dynamodb.md#feature-status) table on +the Differences from DynamoDB page. That page also lists every measured behavioral difference by area. --- @@ -586,6 +564,6 @@ ExtendDB returns the DynamoDB error types and, where measured against the servic Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 3446b6805..cdd805ac7 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -80,9 +80,9 @@ failed on a database outage would restart every replica at once and prolong the outage. A database that is unreachable at startup stops the server from listening at all, so that case is still caught. -There is no readiness endpoint yet, so nothing will currently drain traffic from -a replica whose backend is gone. Until one exists, gate on `/health` for restarts -only, and rely on client retries for backend failures. +There is no readiness endpoint, so nothing drains traffic from a replica whose +backend is gone. Gate on `/health` for restarts only, and rely on client retries +for backend failures. ## Configuration Reference @@ -1246,6 +1246,6 @@ psql -f data_backup.sql extenddb_catalog_data Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/06-developer-test-guide.md b/docs/manuals/06-developer-test-guide.md index cb8a68e82..d67d57adc 100755 --- a/docs/manuals/06-developer-test-guide.md +++ b/docs/manuals/06-developer-test-guide.md @@ -437,6 +437,6 @@ Output goes to `pdfs/` (gitignored). Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 696349d56..8a3a4791e 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -763,6 +763,6 @@ No catalog migration is required. Before upgrading a deployment that restricts t Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/10-security-model.md b/docs/manuals/10-security-model.md index d9ab59e53..ef05d694d 100755 --- a/docs/manuals/10-security-model.md +++ b/docs/manuals/10-security-model.md @@ -269,6 +269,6 @@ Encryption keys are stored in the catalog database. A catalog backup includes th Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index 6a5607e57..81a25960a 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -20,7 +20,7 @@ This guide covers deploying ExtendDB in various environments beyond local develo ExtendDB is a single Rust binary that connects to the storage backend it was built with (PostgreSQL by default; see [Storage backends](../storage-backends.md)). All durable state lives in the backend; ExtendDB keeps only short-lived credential, policy, and table-metadata caches in process. This means: -- Running more than one ExtendDB instance against one catalog is not supported in this release (see [Multi-Instance Considerations](#multi-instance-considerations)) +- Running more than one ExtendDB instance against one catalog is not supported (see [Multi-Instance Considerations](#multi-instance-considerations)) - The backend's own HA, backup, and replication tools provide durability - ExtendDB can run anywhere the storage backend is reachable @@ -126,10 +126,9 @@ want: one that failed on a database outage would restart every replica at once. A backend that is unreachable at startup does stop the server from listening, so container start-up failures are still caught. -There is no readiness endpoint yet, so do not expect a Kubernetes -`readinessProbe` on `/health` to remove a replica from service when its database -becomes unreachable. Until a readiness endpoint that probes the storage layer -exists, treat backend failures as something clients retry through. +There is no readiness endpoint, so do not expect a Kubernetes `readinessProbe` +on `/health` to remove a replica from service when its database becomes +unreachable. Treat backend failures as something clients retry through. For Kubernetes, run `extenddb init` as an init container or a one-time Job, then deploy ExtendDB as a Deployment with the generated `extenddb.toml` mounted as a ConfigMap or Secret. @@ -267,7 +266,7 @@ sudo systemctl start extenddb ## Multi-Instance Considerations -Run one ExtendDB instance per catalog. Multiple instances behind a load balancer are not supported in this release: +Run one ExtendDB instance per catalog. Multiple instances behind a load balancer are not supported: - Credentials, policies, and table metadata are cached in each process (see the cache TTLs in the admin guide). Nothing invalidates a cache on another instance, so a key revoked or a policy changed through one instance stays in force on the others until the TTL expires. - Every instance runs every background worker (TTL expiry, GSI backfill, stream retention, control-plane transitions). The workers are safe to run twice, but the work is duplicated. @@ -341,6 +340,6 @@ ORDER BY mean_exec_time DESC LIMIT 10; Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/manuals/13-event-ticketing-demo.md b/docs/manuals/13-event-ticketing-demo.md index 21247dc1b..603ed6e2a 100755 --- a/docs/manuals/13-event-ticketing-demo.md +++ b/docs/manuals/13-event-ticketing-demo.md @@ -5,7 +5,7 @@ description: "A worked application on ExtendDB: accounts and IAM users, tables, # Event ticketing demo This guide walks you through building a complete event ticketing system on -ExtendDB (extenddb). You will create tables, load data, purchase tickets +ExtendDB. You will create tables, load data, purchase tickets with transactions, observe TTL-based reservation expiry, consume change events via DynamoDB Streams, enforce IAM policies, and verify operational metrics. @@ -677,7 +677,7 @@ If your `extenddb.toml` doesn't have an `[export]` section, add one: paths = ["/tmp/extenddb-exports"] ``` -Then restart extenddb: +Then restart ExtendDB: ```bash ./target/release/extenddb stop --config extenddb.toml diff --git a/docs/manuals/14-writing-migrations.md b/docs/manuals/14-writing-migrations.md index 0b477969a..46c7bab16 100644 --- a/docs/manuals/14-writing-migrations.md +++ b/docs/manuals/14-writing-migrations.md @@ -96,6 +96,6 @@ The consolidated schema file is what fresh installs get. Add your new column, ta Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/rfcs/0000-mongodb-backend.md b/docs/rfcs/0000-mongodb-backend.md index 59d73cacd..9d3b36ce0 100644 --- a/docs/rfcs/0000-mongodb-backend.md +++ b/docs/rfcs/0000-mongodb-backend.md @@ -19,7 +19,7 @@ ExtendDB's core premise is DynamoDB API compatibility over multiple storage back MongoDB is a natural fit as an additional database target: data model alignment; high read/write throughput through horizontal scalability; infrastructure fit. -DynamoDB and MongoDB share the same data model approach — documents stored as schema-less JSON-like data. MongoDB's document model maps directly to the approach taken by DynamoDB with each item stored as a MongoDB BSON document with no impedance mismatch at the data model level. Unlike relational databases, the translation from JSON to BSON is direct without complicated relational mapping techniques required. +DynamoDB and MongoDB share the same data model approach: documents stored as schema-less JSON-like data. MongoDB's document model maps directly to the approach taken by DynamoDB with each item stored as a MongoDB BSON document with no impedance mismatch at the data model level. Unlike relational databases, the translation from JSON to BSON is direct without complicated relational mapping techniques required. Customers evaluating DynamoDB and MongoDB often consider scalability as a key requirement. ExtendDB's deployment approach requiring high write throughput is matched by MongoDB's replica set model via horizontal scaling. High read and write throughput across multiple nodes is a core tenet of MongoDB and aligns naturally with the scalability requirement for an ExtendDB customer. @@ -50,23 +50,23 @@ This RFC does not propose: The backend lives at `crates/storage-mongodb/` in the main ExtendDB repository, following the mono-repo structure prescribed by RFC-0002. It is selected at build time via a `mongodb` Cargo feature flag on the `extenddb` binary crate. -Feature flag definition: `crates/bin/Cargo.toml` — `[features]` section defines `mongodb = ["extenddb-storage-mongodb"]` with the crate as an optional dependency. The `postgres` feature remains the default. Both features can be enabled together to compile a binary supporting both backends. +Feature flag definition: the `[features]` section of `crates/bin/Cargo.toml` defines `mongodb = ["extenddb-storage-mongodb"]` with the crate as an optional dependency. The `postgres` feature remains the default. Both features can be enabled together to compile a binary supporting both backends. ### Plugin registration The backend registers itself with ExtendDB's `inventory`-based plugin system without modifying the server, engine, or auth layers. Six `inventory::submit!` calls in `lib.rs` register the backend for: operations engine (CLI commands), bootstrapping (`extenddb init`), config parsing, settings store access, diagnostics store access, and server component construction (`extenddb serve`). -All registration blocks live in `crates/storage-mongodb/src/lib.rs`. The `ServerComponentsRegistration` block is the critical one — it is the factory function called when `extenddb serve --backend mongodb` is run. No changes are required in `crates/server/`, `crates/engine/`, or `crates/auth/`. +All registration blocks live in `crates/storage-mongodb/src/lib.rs`. The `ServerComponentsRegistration` block is the critical one; it is the factory function called when `extenddb serve --backend mongodb` is run. No changes are required in `crates/server/`, `crates/engine/`, or `crates/auth/`. ### Database layout The backend uses two MongoDB databases: -**`extenddb_catalog`** — metadata and management. Created on `extenddb init`. Contains collections for table definitions (`tables`), index metadata (`indexes`), accounts, tags, admin users, IAM users, groups, roles, access keys, IAM sessions, policies, permissions boundaries, settings, metrics, login attempts, backup metadata, continuous backup state, and schema migration history. +**`extenddb_catalog`**: metadata and management. Created on `extenddb init`. Contains collections for table definitions (`tables`), index metadata (`indexes`), accounts, tags, admin users, IAM users, groups, roles, access keys, IAM sessions, policies, permissions boundaries, settings, metrics, login attempts, backup metadata, continuous backup state, and schema migration history. -**`extenddb_data`** — item data. One MongoDB collection per DynamoDB table, named `_ddb_{table_id}`. One additional collection per GSI/LSI, named `_ddb_{index_id}`. Shared collections: `stream_records` and `stream_shards` for DynamoDB Streams, `counters` for per-shard sequence-number counters, `idempotency_tokens` for transaction deduplication, and one `_backup_{backup_id}` collection per user-created backup. +**`extenddb_data`**: item data. One MongoDB collection per DynamoDB table, named `_ddb_{table_id}`. One additional collection per GSI/LSI, named `_ddb_{index_id}`. Shared collections: `stream_records` and `stream_shards` for DynamoDB Streams, `counters` for per-shard sequence-number counters, `idempotency_tokens` for transaction deduplication, and one `_backup_{backup_id}` collection per user-created backup. -Catalog collection creation and index setup: `crates/storage-mongodb/src/bootstrapper.rs` — `run_catalog_migrations()`. Data-database setup (`idempotency_tokens`, `stream_shards`, `stream_records` and their indexes): `create_data_db()` in the same file. Collection naming: `data/mod.rs` — `data_collection_name()`, shared between base-table and index collections. +Catalog collection creation and index setup: `run_catalog_migrations()` in `crates/storage-mongodb/src/bootstrapper.rs`. Data-database setup (`idempotency_tokens`, `stream_shards`, `stream_records` and their indexes): `create_data_db()` in the same file. Collection naming: `data_collection_name()` in `data/mod.rs`, shared between base-table and index collections. ### Document structure for DynamoDB items @@ -98,13 +98,13 @@ The full item is stored in `item_data` using DynamoDB's own type-tagged format ( ### Condition expression evaluation -Conditional writes (`ConditionExpression` on PutItem, DeleteItem, UpdateItem) run the condition read, evaluation, and write inside a MongoDB client session bound to a multi-document transaction. Within the session, the backend reads the current item, evaluates the DynamoDB condition in Rust against the loaded item (`extenddb_core::expression::evaluate_condition`), and issues the write on the same session. Read and write share the transaction's snapshot isolation, so a concurrent writer's changes either become visible to the condition (in which case the caller sees the same outcome as if the writes were serial) or trigger a WriteConflict at commit (which the write path retries — see Write conflict handling). +Conditional writes (`ConditionExpression` on PutItem, DeleteItem, UpdateItem) run the condition read, evaluation, and write inside a MongoDB client session bound to a multi-document transaction. Within the session, the backend reads the current item, evaluates the DynamoDB condition in Rust against the loaded item (`extenddb_core::expression::evaluate_condition`), and issues the write on the same session. Read and write share the transaction's snapshot isolation, so a concurrent writer's changes either become visible to the condition (in which case the caller sees the same outcome as if the writes were serial) or trigger a WriteConflict at commit (which the write path retries; see Write conflict handling). This delivers DynamoDB's atomicity contract for conditional writes and matches `ReturnValuesOnConditionCheckFailure = ALL_OLD` semantics naturally: the item loaded to evaluate the condition is reused directly in the failure response. An optional filter-pushdown fast path (`pushdown.rs` + `condition.rs`) skips the session for a restricted subset of conditions on tables that have no GSIs and no stream capture. A compile-time analyzer (`is_pushable`) certifies that a condition's compiled MongoDB filter agrees with `evaluate_condition` on every item; when it says yes, the backend collapses read + check + write into a single `find_one_and_replace` / `find_one_and_delete` or replace with a merged key+condition filter. The analyzer is the correctness boundary: the compiler in `condition.rs` covers a broader syntax (numeric compare, sets, `IN`, `BETWEEN`, `size`, arbitrary `NOT`) than the analyzer certifies, and only the analyzer-approved subset ever reaches production. Anything else falls through to the session-scoped path, which is always authoritative. -Session-scoped condition path: `data_engine.rs` — `put_item_impl`, `delete_item_impl`, `update_item_impl`, and each `OwnedTransactWriteOp` arm in `execute_transact_write_op_in_session`. Pushdown fast path: `delete_item_pushdown` and `update_item_pushdown` in the same file, gated on `is_pushable(cond, maps) == Yes && stream.is_none() && gsi_cache_get_fresh(table_id) == Some(false)`. +Session-scoped condition path: `put_item_impl`, `delete_item_impl`, `update_item_impl` in `data_engine.rs`, and each `OwnedTransactWriteOp` arm in `execute_transact_write_op_in_session`. Pushdown fast path: `delete_item_pushdown` and `update_item_pushdown` in the same file, gated on `is_pushable(cond, maps) == Yes && stream.is_none() && gsi_cache_get_fresh(table_id) == Some(false)`. ### Query and Scan @@ -112,11 +112,11 @@ Session-scoped condition path: `data_engine.rs` — `put_item_impl`, `delete_ite Pagination via `ExclusiveStartKey` **merges** the resume bound into the existing sort-key predicate rather than replacing it. Naively inserting `{sk: {$gt: cursor}}` drops the caller's original `BETWEEN` / `begins_with` bound and returns items outside it on page 2 and beyond. The merge covers three cases: no existing sk predicate (insert), existing operator map (merge into it), and existing equality (fall back to `$and`). See `query_impl` in `data_engine.rs`. -**Index queries** paginate over a compound tuple `(index_sk?, base_pk, base_sk?)`. Index-key values are non-unique — duplicates fall through to the base-key tie-breaker. The cursor is expressed as a lexicographic `$or` of the form `(a > A) OR (a == A AND b > B) OR (a == A AND b == B AND c > C)` (reversed for descending). Sort direction is applied to the same compound tuple so ordering is deterministic across groups of items sharing index keys. `LastEvaluatedKey` carries both index-key and base-key components so the next page's `ExclusiveStartKey` resolves the compound cursor. +**Index queries** paginate over a compound tuple `(index_sk?, base_pk, base_sk?)`. Index-key values are non-unique; duplicates fall through to the base-key tie-breaker. The cursor is expressed as a lexicographic `$or` of the form `(a > A) OR (a == A AND b > B) OR (a == A AND b == B AND c > C)` (reversed for descending). Sort direction is applied to the same compound tuple so ordering is deterministic across groups of items sharing index keys. `LastEvaluatedKey` carries both index-key and base-key components so the next page's `ExclusiveStartKey` resolves the compound cursor. **Scan** performs a full collection scan with lazy cursor iteration. Base-table scans paginate on `_id` (unique after netstring encoding). Index scans paginate on the same compound cursor as index Query. Filter expressions are evaluated after retrieval. -**Parallel scan** (`Segment` / `TotalSegments`) filters items in the application via `crc32(pk) % TotalSegments == Segment`. Each segment scans the full collection. The scan loop streams the cursor and terminates when either `limit + 1` in-segment items are accumulated or the cursor exhausts, without imposing a server-side hard limit — a hard `limit * total_segments` cap combined with post-fetch segment filtering silently drops items under any hot-key skew. Pre-bucketing documents at write time would avoid the per-segment full scan but adds overhead to every write for a feature that is rarely used in practice. +**Parallel scan** (`Segment` / `TotalSegments`) filters items in the application via `crc32(pk) % TotalSegments == Segment`. Each segment scans the full collection. The scan loop streams the cursor and terminates when either `limit + 1` in-segment items are accumulated or the cursor exhausts, without imposing a server-side hard limit: a hard `limit * total_segments` cap combined with post-fetch segment filtering silently drops items under any hot-key skew. Pre-bucketing documents at write time would avoid the per-segment full scan but adds overhead to every write for a feature that is rarely used in practice. ### Global and Local Secondary Indexes @@ -124,7 +124,7 @@ Each secondary index has its own MongoDB collection with a compound index on `(p A `DashMap` in-memory cache on `MongoEngine` short-circuits the catalog lookup for tables known to have no indexes. Cache entries carry an insertion timestamp and expire after 60 seconds (`GSI_CACHE_TTL`), so out-of-band GSI changes on another ExtendDB instance converge within the TTL window. -**Async GSI backfill.** `UpdateTable`'s GSI-create path writes the catalog document with `index_status: "CREATING"` and pre-creates the mongo collection and its query index; a background `gsi_backfill_worker` (in `ttl_worker.rs`) discovers `CREATING` rows, iterates the base collection in batches keyed by a persistent `backfill_cursor` field on the index document, upserts projected items into the index collection via `backfill_gsi_batch`, and flips the status to `ACTIVE` when the base is fully scanned. The cursor is persisted between batches so a mid-backfill server restart resumes where it left off. Live writes during the backfill window continue to route through `sync_indexes_in_session`, which writes to CREATING indexes too — all writes are upserts on the same `_id` shape, so a base item touched by both paths converges regardless of interleaving. +**Async GSI backfill.** `UpdateTable`'s GSI-create path writes the catalog document with `index_status: "CREATING"` and pre-creates the mongo collection and its query index; a background `gsi_backfill_worker` (in `ttl_worker.rs`) discovers `CREATING` rows, iterates the base collection in batches keyed by a persistent `backfill_cursor` field on the index document, upserts projected items into the index collection via `backfill_gsi_batch`, and flips the status to `ACTIVE` when the base is fully scanned. The cursor is persisted between batches so a mid-backfill server restart resumes where it left off. Live writes during the backfill window continue to route through `sync_indexes_in_session`, which writes to CREATING indexes too: all writes are upserts on the same `_id` shape, so a base item touched by both paths converges regardless of interleaving. Before every put and update, `validate_index_keys_for_item` rejects wrong-type or empty index-key attributes as a top-level `ValidationException` (or a per-item `CancellationReason` inside `TransactWriteItems`). Without this check, a mismatched-type index-key attribute would be silently dropped from the index doc, leaving the row un-locatable for subsequent deletes. @@ -132,30 +132,30 @@ Before every put and update, `validate_index_keys_for_item` rejects wrong-type o `TransactWriteItems` runs all operations inside a single MongoDB multi-document ACID transaction with snapshot read concern and majority write concern. Each operation's condition evaluation, base-row write, GSI synchronization, and stream record insert happens on the same `ClientSession` (`sync_indexes_in_session` + `write_stream_inline_in_session` are called inline from each `OwnedTransactWriteOp` arm). Without this, a transactional write to a GSI-bearing or streams-enabled table would commit the base row while silently dropping its dependent side effects. -Idempotency tokens live in the `idempotency_tokens` collection in `extenddb_data`. A unique compound index on `(account_id, token)` catches races between concurrent transacts under snapshot isolation — an inserter that races through the pre-check gets an `E11000` on insert, resolved by re-reading the winner and returning `IdempotentReplay` (fingerprints match) or `IdempotentMismatch` (fingerprints differ). Retention is enforced by a 540-second MongoDB TTL index plus a data-plane age filter (`created_at` within 600 000 ms) so worst-case retention stays ≤10 minutes regardless of the TTL monitor's ~60s cadence. +Idempotency tokens live in the `idempotency_tokens` collection in `extenddb_data`. A unique compound index on `(account_id, token)` catches races between concurrent transacts under snapshot isolation: an inserter that races through the pre-check gets an `E11000` on insert, resolved by re-reading the winner and returning `IdempotentReplay` (fingerprints match) or `IdempotentMismatch` (fingerprints differ). Retention is enforced by a 540-second MongoDB TTL index plus a data-plane age filter (`created_at` within 600 000 ms) so worst-case retention stays ≤10 minutes regardless of the TTL monitor's ~60s cadence. `TransactGetItems` performs a consistent snapshot read using a `ClientSession` with snapshot read concern. -Transaction implementation: `data_engine.rs` — `transact_write_items_impl`, `transact_get_items_impl`, `execute_transact_write_op_in_session`. +Transaction implementation: `transact_write_items_impl`, `transact_get_items_impl` and `execute_transact_write_op_in_session` in `data_engine.rs`. ### Write conflict handling **Session-scoped writes** (PutItem, DeleteItem, UpdateItem, TransactWriteItems) detect transient MongoDB conflicts via `is_transient_write_conflict`, which returns true for any of: the `TransientTransactionError` label, the `UnknownTransactionCommitResult` label, or a raw `WriteConflict` (code 112). Conflicts trigger a retry loop with jittered exponential backoff (`backoff_sleep`, base 50 µs) up to `TRANSIENT_RETRY_ATTEMPTS` (50). Exhausted retries on single-item operations return a `StorageError::Internal`; on `TransactWriteItems`, they surface as a `TransactionCanceled` with a synthetic per-op `TransactionConflict` cancellation reason so wire consumers see the DDB-canonical error string instead of a bare HTTP 500. -**UpdateItem** additionally uses an optimistic-concurrency version guard on top of the transaction. A `_v` counter is stored on each document. The write path reads the current `_v` under the transaction snapshot, applies the update expression in memory, sets `_v = current_version + 1`, then executes `replace_one` filtered on both the primary key AND the expected `_v`. If `matched_count == 0`, a concurrent writer committed a higher version between the snapshot read and the replace; the attempt aborts and the outer retry loop re-reads. The version guard doubles up when a native-fast-path update (unconditional, no stream, no GSI) is possible — that path uses a single `find_one_and_update` with `$inc: {_v: 1}` outside a transaction, and always bumps the counter so a concurrent session-scoped update against a stale snapshot fails its versioned filter and retries. +**UpdateItem** additionally uses an optimistic-concurrency version guard on top of the transaction. A `_v` counter is stored on each document. The write path reads the current `_v` under the transaction snapshot, applies the update expression in memory, sets `_v = current_version + 1`, then executes `replace_one` filtered on both the primary key AND the expected `_v`. If `matched_count == 0`, a concurrent writer committed a higher version between the snapshot read and the replace; the attempt aborts and the outer retry loop re-reads. The version guard doubles up when a native-fast-path update (unconditional, no stream, no GSI) is possible: that path uses a single `find_one_and_update` with `$inc: {_v: 1}` outside a transaction, and always bumps the counter so a concurrent session-scoped update against a stale snapshot fails its versioned filter and retries. **PutItem** with an existence guard on a new document maps duplicate-key errors (`E11000`) to `ConditionFailed` after re-reading the winner. This is the runtime signature of a conditional-put race the transaction snapshot didn't see. ### DynamoDB Streams -DynamoDB Streams are implemented using explicit stream record storage in MongoDB collections, not MongoDB's native Change Streams feature. The explicit approach maintains behavioral parity with the PostgreSQL backend and retains full application control over sequence-number generation, shard assignment, and record retention — all of which the DynamoDB Streams API contract tightly specifies. +DynamoDB Streams are implemented using explicit stream record storage in MongoDB collections, not MongoDB's native Change Streams feature. The explicit approach maintains behavioral parity with the PostgreSQL backend and retains full application control over sequence-number generation, shard assignment, and record retention, all of which the DynamoDB Streams API contract tightly specifies. Each stream-enabled table is assigned 4 shards at creation time. Shard identifiers embed the table's globally-unique `table_id` UUID rather than the caller-visible `table_name` (`build_shard_id` in `stream_engine.rs`): `shardId-{table_id}-{i:012}`. Table names are only unique per-account, so a name-derived shard_id would let one account's `GetRecords` read another's records on same-named tables. `table_id` is per-instance, so a `DeleteTable + CreateTable` sequence produces fresh shard_ids; leftover stream records from the deleted table are cleaned up in `delete_table_impl` (`cleanup_stream_state_for_table`). A unique index on `stream_shards.shard_id` rules out duplicate insertions structurally. On each data write with streams enabled, `write_stream_inline_in_session` runs inside the same session as the base write: 1. Resolve the shard for the item's partition key by reading the table's shard set under the session (`assign_shard_in_session`) and hashing the pk with CRC32. -2. Draw the next sequence number by `$inc`-ing the per-shard counter at `_id: "stream_seq:"` in the `counters` collection — also under the session. +2. Draw the next sequence number by `$inc`-ing the per-shard counter at `_id: "stream_seq:"` in the `counters` collection, also under the session. 3. Insert the stream record into `stream_records`. Per-shard counters preserve DynamoDB Streams' contract that sequence numbers are strictly monotonic within a shard and independent across shards. A single global counter would couple unrelated shards' sequence spaces. Session-scoped assignment closes an ordering hole: without it, a fast writer B can draw seq=6 and commit before a slow writer A (which drew seq=5) commits, and a consumer polling between B's commit and A's commit would advance past seq=6 and never see seq=5. With the counter increment inside the write transaction, two writers racing on the same shard conflict at commit time and the loser retries. @@ -170,13 +170,13 @@ The `StreamEngine::write_stream_record` trait method is not used on this backend When TTL is enabled on a table, the backend creates a sparse MongoDB index on `item_data.{ttl_attribute}.N` for ordinary attribute names and marks `ttl_index_ready: true` on the table doc. This flag means that the table's TTL cleanup path is ready. Dotted attribute names use the literal-field expression path instead of a physical index, but are also marked ready. A background TTL worker (spawned at server startup by `MongoRuntimeHooks::spawn_workers`) sweeps expired items every 60 seconds in batches of 100 per table. Each deletion goes through `DataEngine::delete_item` with a condition expression re-checking expiry, preventing races with concurrent writes. TTL deletions carry `UserIdentity { type: "Service", principalId: "dynamodb.amazonaws.com" }` on their stream records, matching DynamoDB's TTL stream record format. -TTL index creation: `metadata_engine.rs` — `create_ttl_index`. Background worker and stream/GSI companions: `ttl_worker.rs` — `ttl_cleanup_worker`, `stream_record_cleanup_worker`, `gsi_backfill_worker`. Worker spawn: `lib.rs` — `MongoRuntimeHooks::spawn_workers`. +TTL index creation: `create_ttl_index` in `metadata_engine.rs`. Background worker and stream/GSI companions: `ttl_cleanup_worker`, `stream_record_cleanup_worker` and `gsi_backfill_worker` in `ttl_worker.rs`. Worker spawn: `MongoRuntimeHooks::spawn_workers` in `lib.rs`. ### Control plane state transitions -`CreateTable` (and `RestoreTableFromBackup`) write the catalog row and create the data collection with its indexes before returning. When `control_plane_delay_seconds` > 0 (the default is 0.25) the row is written with `TableStatus: CREATING` and a `status_transition_at` timestamp, and the returned `TableDescription` carries `TableStatus: CREATING`; a background `control_plane_worker` (`ttl_worker.rs`) flips the row to `ACTIVE` once the transition time passes. During the window, data-plane operations against the table return `ResourceNotFoundException`, matching DynamoDB and the PostgreSQL backend. When `control_plane_delay_seconds` is 0, the row is written `ACTIVE` directly and the worker has nothing to do. `DeleteTable` remains inline: it removes the catalog row, drops the data + index collections, deletes tags, and cleans up stream shards / records / counters via `cleanup_stream_state_for_table` (`table_engine.rs`), all before returning — there is no transient `DELETING` state. +`CreateTable` (and `RestoreTableFromBackup`) write the catalog row and create the data collection with its indexes before returning. When `control_plane_delay_seconds` > 0 (the default is 0.25) the row is written with `TableStatus: CREATING` and a `status_transition_at` timestamp, and the returned `TableDescription` carries `TableStatus: CREATING`; a background `control_plane_worker` (`ttl_worker.rs`) flips the row to `ACTIVE` once the transition time passes. During the window, data-plane operations against the table return `ResourceNotFoundException`, matching DynamoDB and the PostgreSQL backend. When `control_plane_delay_seconds` is 0, the row is written `ACTIVE` directly and the worker has nothing to do. `DeleteTable` remains inline: it removes the catalog row, drops the data + index collections, deletes tags, and cleans up stream shards / records / counters via `cleanup_stream_state_for_table` (`table_engine.rs`), all before returning; there is no transient `DELETING` state. -GSI creation on `UpdateTable` is the one control-plane operation that does need asynchronous work — a background worker drains index rows in `CREATING` state, backfills the base collection, and flips the row to `ACTIVE`. See the Global and Local Secondary Indexes section for the state machine. +GSI creation on `UpdateTable` is the one control-plane operation that does need asynchronous work: a background worker drains index rows in `CREATING` state, backfills the base collection, and flips the row to `ACTIVE`. See the Global and Local Secondary Indexes section for the state machine. ### Authentication and authorization @@ -184,7 +184,7 @@ ExtendDB's mandatory SigV4 authentication is fully supported. Access-key secrets IAM policy evaluation fetches user-attached policies, group-attached policies (via `iam_groups.members` → `iam_policies` join), role policies, permissions boundaries, and session policies from the catalog. -`MongoEngine::new` rejects connection strings that specify a non-primary read preference (`secondary`, `secondaryPreferred`, `nearest`, `primaryPreferred`). DynamoDB's `ConsistentRead=true` requires linearizable reads; only MongoDB's Primary read preference provides that. A connection string that routes reads to a replica would silently return stale data — a fidelity violation the caller has no way to detect. The check fails at engine construction so misconfiguration surfaces at `extenddb serve` startup, not at request time. +`MongoEngine::new` rejects connection strings that specify a non-primary read preference (`secondary`, `secondaryPreferred`, `nearest`, `primaryPreferred`). DynamoDB's `ConsistentRead=true` requires linearizable reads; only MongoDB's Primary read preference provides that. A connection string that routes reads to a replica would silently return stale data, a fidelity violation the caller has no way to detect. The check fails at engine construction so misconfiguration surfaces at `extenddb serve` startup, not at request time. Encryption key bootstrap: `bootstrapper.rs::bootstrap_encryption_key`. Admin password hashing: same file, `bootstrap_admin_user`. Access-key decryption: `credential_store.rs`. IAM policy fetching: `authorization_store.rs`. @@ -225,7 +225,7 @@ Configuration struct: `crates/storage-mongodb/src/config.rs`. Sample configurati | Crate modified | Change | |---|---| -| `crates/storage-mongodb/` | New crate — full backend implementation | +| `crates/storage-mongodb/` | New crate: full backend implementation | | `crates/bin/Cargo.toml` | Added `mongodb` optional feature flag | | `crates/bin/src/main.rs` | Added `#[cfg(feature = "mongodb")] extern crate` | | `crates/bin/src/cmd_serve.rs` | Generalized the supported-backend gate from a hard-coded `"postgres"` check to a compile-time list built from enabled features | @@ -255,17 +255,17 @@ No changes to `crates/engine/`, `crates/server/`, `crates/storage/` (trait defin ### Performance characteristics -**Single-item conditional writes.** One transaction session covers the pre-image read, condition evaluation, base write, GSI synchronization, and stream record insert. On a local replica set this adds ~sub-millisecond of session-start/commit overhead over a raw driver call. The session wrap is what gives DynamoDB's atomicity contract on conditional writes — it is the compatibility, not overhead. The analyzer-gated pushdown fast path collapses this to a single `find_one_and_*` call for the narrow case of certified pushable conditions on tables with no GSIs and no streams. +**Single-item conditional writes.** One transaction session covers the pre-image read, condition evaluation, base write, GSI synchronization, and stream record insert. On a local replica set this adds ~sub-millisecond of session-start/commit overhead over a raw driver call. The session wrap is what gives DynamoDB's atomicity contract on conditional writes; it is the compatibility, not overhead. The analyzer-gated pushdown fast path collapses this to a single `find_one_and_*` call for the narrow case of certified pushable conditions on tables with no GSIs and no streams. **Unconditional single-item updates on GSI-free / stream-free tables.** A native-fast-path `find_one_and_update` runs outside any transaction. It always includes `$inc: {_v: 1}` so a concurrent slow-path update cannot pass its versioned filter against a stale snapshot. -**GSI write overhead.** For tables with no GSIs, the `gsi_cache` short-circuits to zero overhead (no catalog query, no I/O) — refreshed at most once per `GSI_CACHE_TTL` window per table. For tables with GSIs, one catalog query fetches the index definitions (cached for subsequent writes) and one upsert or delete runs per index collection per write, all within the base write's session. +**GSI write overhead.** For tables with no GSIs, the `gsi_cache` short-circuits to zero overhead (no catalog query, no I/O), refreshed at most once per `GSI_CACHE_TTL` window per table. For tables with GSIs, one catalog query fetches the index definitions (cached for subsequent writes) and one upsert or delete runs per index collection per write, all within the base write's session. **Stream write overhead.** When streams are enabled, each write adds one atomic per-shard counter `$inc` and one document insert into `stream_records`, both within the base write's session. **Query and Scan.** Direct index lookups on `(pk, sk_?)` for base tables; compound `(pk, sk_?, base_pk, base_sk_?)` lookups for index queries. `GetRecords` uses the compound `(shard_id, sequence_number)` index. -**TransactWriteItems.** Multi-collection ACID transaction; up to 100 operations per the DDB spec. Uncommon in practice — most workloads are single-item operations. +**TransactWriteItems.** Multi-collection ACID transaction; up to 100 operations per the DDB spec. Uncommon in practice: most workloads are single-item operations. ### Testing @@ -273,27 +273,27 @@ Testing is organized in three layers. **Unit tests** cover pure logic without a live MongoDB instance: netstring composite `_id` encoding, hex sort-key ordering, condition filter compilation, pushdown-analyzer decisions, sequence-number formatting, stream shard-id derivation. Property tests (`crates/storage-mongodb/tests/pushdown_parity.rs`) exercise the parity between the pushdown compiler and the in-Rust `evaluate_condition` reference over randomly generated items and expressions. -**Integration tests** run the dual-target `tests/rust/` suite — the same AWS-SDK wire-conformance tests the PostgreSQL backend runs — against a MongoDB-backed ExtendDB server on a single-node replica set (`mongod --replSet rs0`), covering the full table lifecycle, all item operations (conditional and unconditional), query and scan pagination (base and index), transactions, TTL worker behavior, stream record writes and consumer pagination, GSI propagation and async backfill, backup and restore, and catalog operations. They execute via `devtools/run-mongodb-tests -- --rust --rust-integration`, which stands up the replica set, serves ExtendDB against it, and delegates to `devtools/run-tests --backend mongodb`. +**Integration tests** run the dual-target `tests/rust/` suite, the same AWS-SDK wire-conformance tests the PostgreSQL backend runs, against a MongoDB-backed ExtendDB server on a single-node replica set (`mongod --replSet rs0`), covering the full table lifecycle, all item operations (conditional and unconditional), query and scan pagination (base and index), transactions, TTL worker behavior, stream record writes and consumer pagination, GSI propagation and async backfill, backup and restore, and catalog operations. They execute via `devtools/run-mongodb-tests -- --rust --rust-integration`, which stands up the replica set, serves ExtendDB against it, and delegates to `devtools/run-tests --backend mongodb`. -**End-to-end tests** run the existing ExtendDB pytest suite (`tests/`) unchanged against a MongoDB-backed ExtendDB server. The pytest suite speaks the DynamoDB wire protocol and has no backend awareness — a passing run against MongoDB is equivalent to a passing run against PostgreSQL. This is the conformance test baseline required by RFC-0002. +**End-to-end tests** run the existing ExtendDB pytest suite (`tests/`) unchanged against a MongoDB-backed ExtendDB server. The pytest suite speaks the DynamoDB wire protocol and has no backend awareness; a passing run against MongoDB is equivalent to a passing run against PostgreSQL. This is the conformance test baseline required by RFC-0002. -CI (`.github/workflows/integration-mongodb.yml`) runs the pytest and rust-integration suites as two parallel jobs. Each builds ExtendDB with `--no-default-features --features mongodb` (the backend features are mutually exclusive, so the default `postgres` feature must be disabled) and delegates to `devtools/run-mongodb-tests`, which bootstraps a single-node MongoDB 7.0 replica set (`rs.initiate` + wait-for-PRIMARY — a step GitHub `services:` cannot express), serves ExtendDB against it, provisions credentials, and runs the suite via `devtools/run-tests --backend mongodb`. Backend-crate unit and property tests run in the standard `cargo test` workflow. +CI (`.github/workflows/integration-mongodb.yml`) runs the pytest and rust-integration suites as two parallel jobs. Each builds ExtendDB with `--no-default-features --features mongodb` (the backend features are mutually exclusive, so the default `postgres` feature must be disabled) and delegates to `devtools/run-mongodb-tests`, which bootstraps a single-node MongoDB 7.0 replica set (`rs.initiate` + wait-for-PRIMARY, a step GitHub `services:` cannot express), serves ExtendDB against it, provisions credentials, and runs the suite via `devtools/run-tests --backend mongodb`. Backend-crate unit and property tests run in the standard `cargo test` workflow. ## Drawbacks **Replica set requirement.** MongoDB must be run as a replica set for multi-document transactions. Users who run standalone MongoDB will receive a runtime error on transactional operations. This is a MongoDB architectural constraint, not an ExtendDB limitation, and is documented in setup guides. -**TTL throughput at scale.** DynamoDB TTL deletions must emit stream records with a specific service identity. MongoDB's native TTL indexes operate at the storage-engine level with no awareness of ExtendDB's stream system, so this implementation uses an application-level background worker that owns the full deletion lifecycle. The worker runs every 60 seconds and processes 100 expired items per table per pass. This is sufficient for ExtendDB's target deployment contexts. At very high sustained expiration rates the worker will fall behind, and the backlog will grow. This is a known scale limitation, not a correctness issue — DynamoDB's own contract only guarantees expiration within 48 hours, not immediately (see `docs/differences-from-dynamodb.md`, TTL row). +**TTL throughput at scale.** DynamoDB TTL deletions must emit stream records with a specific service identity. MongoDB's native TTL indexes operate at the storage-engine level with no awareness of ExtendDB's stream system, so this implementation uses an application-level background worker that owns the full deletion lifecycle. The worker runs every 60 seconds and processes 100 expired items per table per pass. This is sufficient for ExtendDB's target deployment contexts. At very high sustained expiration rates the worker will fall behind, and the backlog will grow. This is a known scale limitation, not a correctness issue: DynamoDB's own contract only guarantees expiration within 48 hours, not immediately (see `docs/differences-from-dynamodb.md`, TTL row). ## Alternatives ### Use MongoDB Change Streams for DynamoDB Streams -MongoDB has a native change-data-capture feature (Change Streams) that could back DynamoDB Streams. The implementation instead adopted the explicit `stream_records` collection approach used by the PostgreSQL backend, which gives ExtendDB full control over sequence-number generation, shard assignment, record retention, and iterator behavior — all of which the DynamoDB Streams API contract tightly specifies. Reviewers are invited to weigh in on whether a comparative evaluation of the Change Streams approach should be documented before acceptance. +MongoDB has a native change-data-capture feature (Change Streams) that could back DynamoDB Streams. The implementation instead adopted the explicit `stream_records` collection approach used by the PostgreSQL backend, which gives ExtendDB full control over sequence-number generation, shard assignment, record retention, and iterator behavior, all of which the DynamoDB Streams API contract tightly specifies. Reviewers are invited to weigh in on whether a comparative evaluation of the Change Streams approach should be documented before acceptance. ## Prior art -**MongoDB document model and DynamoDB.** MongoDB's flexible document model has been noted as a natural fit for DynamoDB-style workloads in multiple independent analyses. Amazon DocumentDB (MongoDB-compatible) demonstrates AWS's own recognition of this overlap. The key difference in this implementation is that ExtendDB provides the full DynamoDB API layer — clients using the AWS SDK do not need to know they are talking to MongoDB. +**MongoDB document model and DynamoDB.** MongoDB's flexible document model has been noted as a natural fit for DynamoDB-style workloads in multiple independent analyses. Amazon DocumentDB (MongoDB-compatible) demonstrates AWS's own recognition of this overlap. The key difference in this implementation is that ExtendDB provides the full DynamoDB API layer; clients using the AWS SDK do not need to know they are talking to MongoDB. **Condition pushdown pattern.** Compiling application-level filter expressions into storage-native query operators is a well-established pattern in query engines (Apache Arrow DataFusion, Spark, Presto all implement predicate pushdown). The pushdown-analyzer / compiler split in this implementation applies the same principle at the storage backend level, with the analyzer serving as the correctness boundary between the two. @@ -304,6 +304,6 @@ MongoDB has a native change-data-capture feature (Change Streams) that could bac Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/rfcs/0000-template.md b/docs/rfcs/0000-template.md index ad9d6905d..0c18b6ea5 100644 --- a/docs/rfcs/0000-template.md +++ b/docs/rfcs/0000-template.md @@ -53,6 +53,6 @@ projects. What did they get right or wrong? Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/rfcs/0002-backend-plugin-policy.md b/docs/rfcs/0002-backend-plugin-policy.md index 0fc93d192..f61a85672 100644 --- a/docs/rfcs/0002-backend-plugin-policy.md +++ b/docs/rfcs/0002-backend-plugin-policy.md @@ -318,7 +318,7 @@ Fork entire ExtendDB repo for each backend. Each fork contains frontend + one ba ## Unresolved questions -1. **Security review process.** Backend implementations handle database credentials, execute queries, and manage connections—all surfaces with security implications (SQL injection, connection security, credential leakage). Should we formalize security review as part of the PR process? If so, what does that look like? Do we need security-focused reviewers, automated scanning tools, or documented security patterns? This wasn't part of the original discussion but seems important for production deployments. +1. **Security review process.** Backend implementations handle database credentials, execute queries, and manage connections, all surfaces with security implications (SQL injection, connection security, credential leakage). Should we formalize security review as part of the PR process? If so, what does that look like? Do we need security-focused reviewers, automated scanning tools, or documented security patterns? This wasn't part of the original discussion but seems important for production deployments. 2. **Conformance test ownership.** Should conformance tests live in the main `extenddb` repo, or in a separate `extenddb-conformance` repo used by all backends? Leaning toward main repo for simplicity, but open to feedback. @@ -349,6 +349,6 @@ Fork entire ExtendDB repo for each backend. Each fork contains frontend + one ba Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/rfcs/0003-backend-acceptance-criteria.md b/docs/rfcs/0003-backend-acceptance-criteria.md index b70b5f2d3..2580d0ae9 100644 --- a/docs/rfcs/0003-backend-acceptance-criteria.md +++ b/docs/rfcs/0003-backend-acceptance-criteria.md @@ -170,6 +170,6 @@ The following scenarios expose the most common failure modes and should be teste Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is -a trademark of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/rfcs/README.md b/docs/rfcs/README.md index b102288d9..f2af85594 100644 --- a/docs/rfcs/README.md +++ b/docs/rfcs/README.md @@ -26,7 +26,7 @@ you whether an RFC, an ADR, or neither is needed. ## Process 1. **Draft.** Copy `0000-template.md` to `docs/rfcs/0000-my-proposal.md` (keep `0000` - until accepted). Open a PR. The PR is the discussion thread for now — + until accepted). Open a PR. The PR is the discussion thread for now; if conversations grow long enough that the PR becomes hard to follow, a maintainer may move the broad design debate to GitHub Discussions and reserve the PR for line-level comments. diff --git a/docs/rfcs/draft-cassandra-backend.md b/docs/rfcs/draft-cassandra-backend.md index 80259095f..7b6a54002 100644 --- a/docs/rfcs/draft-cassandra-backend.md +++ b/docs/rfcs/draft-cassandra-backend.md @@ -17,7 +17,7 @@ architecture with no single write bottleneck: a deployment profile the reference The non-trivial design work is in conforming to DynamoDB semantics on a database that provides neither multi-row ACID transactions nor native sequence primitives. This document focuses on those gaps, the decisions made to close them, and -the places where full parity with Amazon DynamoDB is not yet achieved. +the places where full parity with Amazon DynamoDB is not achieved. ## Motivation @@ -27,11 +27,11 @@ need linear, horizontal write scaling, geographic distribution with active-activ with no single point of failure often already operate Cassandra, and will want DynamoDB API compatibility without adopting a new infrastructure dependency. -Cassandra is a natural fit for this role. Its data model — partition key for distribution, clustering key for ordering -within a partition — maps directly onto DynamoDB's data architecture. Both systems are designed for high write throughput +Cassandra is a natural fit for this role. Its data model (partition key for distribution, clustering key for ordering +within a partition) maps directly onto DynamoDB's data architecture. Both systems are designed for high write throughput with tunable consistency, both treat non-key attributes as schemaless, and both support native multi-datacenter replication. The design challenges addressed in this RFC arise not from a mismatch between the two systems but from the specific -DynamoDB features — transactions, TTL with Streams integration, eventually consistent GSIs — that require capabilities +DynamoDB features (transactions, TTL with Streams integration, eventually consistent GSIs) that require capabilities Cassandra does not provide natively. The Cassandra backend is not a replacement for the PostgreSQL backend. It targets a different operational profile: @@ -57,11 +57,11 @@ and server component construction. The backend uses Cassandra keyspaces to independently manage cross-account 'catalog' data, and individual account-owned data. -**`{prefix}_catalog`** — system-wide metadata, created during `extenddb init`. Holds accounts, IAM entities (users, +**`{prefix}_catalog`**: system-wide metadata, created during `extenddb init`. Holds accounts, IAM entities (users, groups, roles, policies, access keys), table definitions, stream shard metadata, idempotency tokens, settings, and schema migration history. -**`{prefix}_account_{account_id}`** — one keyspace per ExtendDB account, created when the account is created. Holds item +**`{prefix}_account_{account_id}`**: one keyspace per ExtendDB account, created when the account is created. Holds item tables (one per DynamoDB table), secondary index tables, stream records, transaction ledger, TTL expiration queues, and backup payloads. Account deletion drops the entire keyspace, atomically removing all the account's data. @@ -80,7 +80,7 @@ linearizable within a datacenter. Background workers use `LOCAL_ONE` where event ### Catalog integrity -The IAM catalog — which has parent-child relationships between accounts, users, roles, policies, and access keys — relies +The IAM catalog, which has parent-child relationships between accounts, users, roles, policies, and access keys, relies on referential integrity and transactional mutations across entities to maintain consistency. Cassandra does not support referential integrity constraints, and Cassandra's logged batches (atomic batch updates) impose significant constraints on LWT statements (`IF NOT EXISTS`, `IF EXISTS`) within a batch. Duplicate detection and multi-row atomic writes cannot @@ -118,7 +118,7 @@ return the last committed value and ignore the marker entirely, matching DynamoD non-transactional reads. **Durable ledger.** Before touching any item, a ledger entry is written to `transaction_ledger` in the account keyspace -containing the complete write set. The ledger lives in the account keyspace — not the catalog — because transactions are +containing the complete write set. The ledger lives in the account keyspace, not the catalog, because transactions are scoped to a single account's data, and co-locating the ledger with the data it describes means it is dropped atomically with the account. The ledger is written in two phases: key information before PREPARE begins, full computed write data after all claims succeed and before transitioning to COMMITTING. A transaction in COMMITTING state therefore always has @@ -209,7 +209,7 @@ history are not restored; this is a known gap. Backup creation and restore execute synchronously within the API request. This is consistent with the current behavior of all ExtendDB backends, but differs from Amazon DynamoDB, where these are asynchronous operations. The same applies -to `ExportTableToPointInTime` and `ImportTable`. Moving these to background jobs is planned future work. +to `ExportTableToPointInTime` and `ImportTable`. ## 4. Behavioral Differences from DynamoDB @@ -252,9 +252,7 @@ per table with no change to the claim protocol. This is the highest-value follow and the API call blocks for its duration. Large tables will need a durable, checkpointed background backfill before this path is suitable at scale. -**Backup and restore are synchronous.** Moving `CreateBackup`, `RestoreTableFromBackup`, `ExportTableToPointInTime`, and -`ImportTable` to background jobs with status polling is planned but not yet implemented. This applies to all current -ExtendDB backends. +**Backup and restore are synchronous.** This applies to all current ExtendDB backends. **Restore does not recover GSI/LSI definitions, tags, TTL configuration, or stream history.** @@ -264,5 +262,5 @@ the dependency on stable JSON encoding. **Transaction recovery does not reconcile pre-commit images.** The transaction ledger does not persist the pre-commit item image, so a transaction that crashes between COMMIT and TTL queue reconciliation can leave the item's previous -expiration entry in the queue until it comes due. The entry is inert — the worker revalidates before deleting — but it +expiration entry in the queue until it comes due. The entry is inert (the worker revalidates before deleting) but it is queue garbage. diff --git a/docs/storage-backends.md b/docs/storage-backends.md index 221f5899d..9888a1ec0 100644 --- a/docs/storage-backends.md +++ b/docs/storage-backends.md @@ -12,7 +12,7 @@ ExtendDB stores tables, items, indexes and its own catalog in a database you run | PostgreSQL (default) | `postgres` | `extenddb/extenddb-postgres` | PostgreSQL 14 or newer; the pgvector extension on the data database for vector indexes | [Docker Compose](/local-dev-docker-compose/), [from source](/install-from-source/#install-postgresql), [deployment](/manuals/deployment-guide/) | | MongoDB | `mongodb` | `extenddb/extenddb-mongodb` | MongoDB 7.0 or newer as a replica set, including for a single node | [from source](/install-from-source/#install-mongodb), [MongoDB container](https://github.com/ExtendDB/extenddb/blob/main/docker/README-mongodb.md) | | SQLite | `sqlite` | `extenddb/extenddb-dev` (built in developer mode, see below) | None; a file, or `:memory:` | [development image](/dev-image/), [from source](/install-from-source/#initialize-on-sqlite) | -| Cassandra | `cassandra` | None | An Apache Cassandra cluster | [Cassandra setup](/cassandra-setup/), [from source](/install-from-source/#install-cassandra), [upgrade manual](/manuals/upgrade-manual/), [backend RFC (draft)](/rfcs/draft-cassandra-backend/) | +| Cassandra (experimental) | `cassandra` | None | An Apache Cassandra 4.1 or newer cluster | [Cassandra setup](/cassandra-setup/), [from source](/install-from-source/#install-cassandra), [upgrade manual](/manuals/upgrade-manual/), [backend RFC (draft)](/rfcs/draft-cassandra-backend/) | ## Selecting a backend @@ -42,9 +42,9 @@ The DynamoDB API surface is the same on every backend. Where a backend's behavio - PostgreSQL: vector indexes and `SearchVectors` need the pgvector extension on the data database; without it every vector operation is refused and nothing else is affected. - MongoDB: partition and sort key numbers carry 34 significant digits (BSON Decimal128) against the service's 38; larger precision is refused with a `ValidationException`. - SQLite: one server per database file, enforced with a lock file next to the database. The `extenddb/extenddb-dev` image compiles this backend in developer mode: plain HTTP, open authorization, a seeded credential, for local development and CI only ([development image](/dev-image/)). -- Cassandra: backups copy the live table and are not point-in-time; see the backup rows on the differences page. +- Cassandra: the backend is experimental. It does not implement vector indexes or `SearchVectors`, so every vector operation is refused. PutItem, UpdateItem and a conditional DeleteItem on a table without TTL, and the writes of a TransactWriteItems, commit the item before its stream record is written, so a failure between the two leaves the item changed with no stream record. Backups copy the live table and are not point-in-time. The vector, stream and backup rows on the differences page give the detail. -Running more than one server against one PostgreSQL or MongoDB catalog is not supported in this release; the [deployment guide](/manuals/deployment-guide/#multi-instance-considerations) explains why. +Running more than one server against one PostgreSQL, MongoDB or Cassandra catalog is not supported; the [deployment guide](/manuals/deployment-guide/#multi-instance-considerations) explains why. ## Developer notes diff --git a/docs/technical-debt.md b/docs/technical-debt.md index f267feeec..e6fea8265 100755 --- a/docs/technical-debt.md +++ b/docs/technical-debt.md @@ -149,6 +149,6 @@ Resolved (split in P94–P96): Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/todo-index.md b/docs/todo-index.md index f1b67225c..5fbfac167 100755 --- a/docs/todo-index.md +++ b/docs/todo-index.md @@ -28,6 +28,6 @@ Regenerated: v0.1.6 (BR-7085) Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index e084dbbb6..901975fcc 100755 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -967,6 +967,6 @@ even after the extension exists. Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) for the full text. -This software is provided "as is" without warranty of any kind. ExtendDB is not -affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark -of Amazon.com, Inc. +This software is provided "as is" without warranty of any kind. ExtendDB is maintained by +engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. diff --git a/packaging/npm/README.md b/packaging/npm/README.md index cbacb5e8e..edcb770ac 100644 --- a/packaging/npm/README.md +++ b/packaging/npm/README.md @@ -93,9 +93,7 @@ startup; whoever holds it can do anything. Do not expose it beyond localhost and do not keep real data in it. For a durable, TLS-terminated deployment, use [`extenddb/extenddb-postgres`](https://hub.docker.com/r/extenddb/extenddb-postgres). -ExtendDB is an independent open source project managed by Amazon Web -Services. It is not Amazon DynamoDB and does not contain any DynamoDB source -code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room +ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol. More at [extenddb.org](https://extenddb.org) and diff --git a/website/docusaurus.config.ts b/website/docusaurus.config.ts index b8001256b..e5e288a0b 100644 --- a/website/docusaurus.config.ts +++ b/website/docusaurus.config.ts @@ -196,7 +196,7 @@ const config: Config = { ], copyright: `Copyright ${new Date().getUTCFullYear()} ExtendDB contributors. Licensed under the Apache License, Version 2.0. ` + - 'ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code. ' + + 'ExtendDB is an open source project under the Apache License 2.0, maintained by engineers at Amazon Web Services. It is not Amazon DynamoDB and contains no DynamoDB source code. ' + '"DynamoDB" is a trademark of Amazon.com, Inc.', }, prism: { diff --git a/website/sidebars.ts b/website/sidebars.ts index 2ddd8ddd7..74e0a9e4f 100644 --- a/website/sidebars.ts +++ b/website/sidebars.ts @@ -172,7 +172,7 @@ const sidebars: SidebarsConfig = { type: 'generated-index', title: 'Component designs', slug: '/design', - description: 'The requirements and the design of each component: core, storage, authentication, server, streams, configuration and testing.', + description: 'The requirements and the design of each component (core, storage, authentication, server, streams, configuration and testing), written as the system was built. Where behaviour has since changed, the Differences from DynamoDB page is the current record.', }, items: [ {type: 'doc', id: 'design/requirements', label: 'Requirements'}, @@ -180,8 +180,8 @@ const sidebars: SidebarsConfig = { {type: 'doc', id: 'design/component-core', label: 'Core'}, {type: 'doc', id: 'design/component-storage', label: 'Storage'}, {type: 'doc', id: 'design/storage-mongodb', label: 'Storage: MongoDB'}, - {type: 'doc', id: 'design/component-auth', label: 'Authentication and authorization'}, - {type: 'doc', id: 'design/component-auth-v3', label: 'Authentication v3'}, + {type: 'doc', id: 'design/component-auth', label: 'Authentication and authorization (v2)'}, + {type: 'doc', id: 'design/component-auth-v3', label: 'Authentication and authorization (v3)'}, {type: 'doc', id: 'design/auth-authz-cache', label: 'Authorization cache'}, {type: 'doc', id: 'design/component-server', label: 'Server'}, {type: 'doc', id: 'design/component-streams', label: 'Streams'}, diff --git a/website/src/pages/index.tsx b/website/src/pages/index.tsx index 7e73de7f2..894bc3f95 100644 --- a/website/src/pages/index.tsx +++ b/website/src/pages/index.tsx @@ -91,10 +91,10 @@ function QuickStarts(): ReactNode {
- For local development and CI + For local development and continuous integration

One container over SQLite, plain HTTP, a seeded example credential. Nothing to initialize and no - certificate to trust. Bound to the loopback interface; not for anything reachable from other + certificate to trust. It listens only on the local machine and is not for anything reachable from other machines.

{devImage}