Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,23 @@ with DynamoDB works with ExtendDB, unchanged.
- **Multi-cloud** — use DynamoDB semantics on any cloud that runs an ExtendDB-supported database backend
- **Air-gapped environments** — DynamoDB functionality with no internet connectivity

## Scope

ExtendDB is a server that implements the Amazon DynamoDB API on a database that you provide: PostgreSQL, SQLite, MongoDB or Cassandra. An application connects to it through the endpoint setting of an AWS SDK or the AWS CLI. The application signs each request with an access key stored in ExtendDB.

ExtendDB 1.0 is for:

- Local development and continuous integration (CI) test runs.
- Self-hosted deployments: on premises, in a private or public cloud, at the edge, or on a network with no internet access.

ExtendDB 1.0 is not:

- Amazon DynamoDB: ExtendDB is a separate implementation of the API. [Differences from DynamoDB](docs/differences-from-dynamodb.md) lists the known differences and the operations that ExtendDB does not implement, for example global tables.
- A managed service: you provide the database, and the database's own tools handle replication, failover and database backups.
- A multi-server deployment: run one ExtendDB server for each catalog. In 1.0, more than one server on one catalog is not supported.

[Storage backends](#storage-backends) says what 1.0 promises for each backend. The [versioning policy](docs/versioning.md) says what a release can change.

## Features

- Full DynamoDB wire protocol: CRUD, Query, Scan, Batch, Transactions, Streams, TTL, Import/Export
Expand All @@ -27,6 +44,27 @@ with DynamoDB works with ExtendDB, unchanged.
- JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions
- Daemon mode with syslog logging, plus `--foreground` for container and supervisor environments

## Storage backends

An ExtendDB binary contains one storage backend, which the build selects. The tier of a backend says what the 1.x releases promise for it:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm reluctant to introduce tiers unless we have a process/policy for how a backend moves from one tier to another. I also thought that last week we'd converged on "a backend is production (or stable) if the ExtendDB CI is reliably green for it".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three tiers are the ones #662 asks for, so I kept them. 9d136d2 and 4e44a88 add the rule for moving between them, built on that definition. Both tiers for production, stable and beta, now say that the ExtendDB CI is reliably green for the backend. A backend moves up one tier in a minor release. Before that, the maintainers agree in an issue that its CI has been reliably green and that its known limits fit the higher tier. It moves down only through the deprecation steps of the versioning policy.


- Stable: for production. The [versioning policy](docs/versioning.md) applies in full. The ExtendDB CI is reliably green for the backend.
- Beta: for production within the known limits in the table. The ExtendDB CI is reliably green for the backend. The versioning policy applies with one exception. To remove a known limit, a minor release can change the behavior that the table lists, and its release notes name the change.
- Experimental: for evaluation. The backend is not for production use. The versioning policy does not apply. A release can change the backend's behavior or configuration with no deprecation period.

A backend moves up one tier in a minor release. Before that release, the maintainers agree in an issue that the ExtendDB CI has been reliably green for the backend. They also agree that its known limits fit the higher tier. The release notes name the move. A backend moves down a tier only through the [deprecation steps](docs/versioning.md#deprecation) of the versioning policy.

| Backend | Tier | Topology | Data across upgrades | API coverage | CI | Known limits |
|---|---|---|---|---|---|---|
| PostgreSQL | Stable | PostgreSQL 14 or later, on the same host, on another host or as a managed service. | `extenddb migrate --yes` migrates the catalog in place. CI upgrades catalogs that 0.1.x releases created. | The operations in [Supported Operations](#supported-operations). Vector search needs the pgvector extension. | `integration`, required, on PostgreSQL 16 | Time to Live (TTL) removes about 100 expired items per table per minute. |
| SQLite | Stable (single node) | SQLite runs in the server process. A lock file allows one server for each database file. There is no production image, so build from source. | As PostgreSQL. `migrate --yes` copies the database file before its first change. | The operations in Supported Operations. | `integration`, required | One write transaction at a time. TTL as PostgreSQL. |
| MongoDB | Beta | MongoDB 7.0 or later, as a replica set, also for one node. | As PostgreSQL. | No vector search. On a global secondary index (GSI) with more than one RANGE attribute, `Query` takes a key condition on the first one only. | `mongodb-integration`, required, on MongoDB 7 | A number in a key attribute has at most 34 significant digits. The database names are fixed, so one replica set holds one catalog. TTL as PostgreSQL. |
| Cassandra | Experimental | Cassandra 4.1 or later, with `PasswordAuthenticator`. Each table takes its writes in one datacenter. | `migrate --yes` migrates the catalog. No rollback procedure is tested. | No vector search. CreateBackup keeps each backup in the catalog, with or without a backup store. The `Query` limit of MongoDB on a GSI with more than one RANGE attribute. | `cassandra-integration`, required, on Cassandra 4.1 | Some management operations are not atomic. A few known race conditions are open. TTL removes about 1,000 expired items per table per minute. See [Known Limitations](docs/cassandra-setup.md#known-limitations). |

On every backend, run one ExtendDB server for each catalog. Only SQLite enforces this. [Differences from DynamoDB](docs/differences-from-dynamodb.md) lists the differences of each backend from the service.

The `@extenddb/dev` npm package and the `extenddb/extenddb-dev` image run a dev-mode SQLite build, for local development and CI. Dev mode uses plain HTTP and seeds an example key that can do everything, so keep a dev-mode server on loopback. At startup it migrates a catalog that is behind it, after it copies the database file.

## Quick Start

```bash
Expand Down Expand Up @@ -283,6 +321,7 @@ docs/
| [Deployment Guide](docs/manuals/11-deployment-guide.md) | Self-hosted, multi-cloud, air-gapped deployments |
| [Backup and Restore](docs/manuals/14-backup-and-restore.md) | Backup strategy, recovery figures, restore drill, catalog-loss recovery |
| [Differences from DynamoDB](docs/differences-from-dynamodb.md) | Behavioral differences and unsupported operations |
| [Versioning and Support Policy](docs/versioning.md) | What each release can change, and how long a release gets fixes |
| [Troubleshooting](docs/troubleshooting.md) | Common errors and solutions |
| [Verifying Container Images](docs/verifying-images.md) | Build provenance and SBOM attestations for the published images |

Expand Down
7 changes: 4 additions & 3 deletions crates/app/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ rust-version.workspace = true
license.workspace = true

[features]
# Developer mode: plain HTTP on loopback, open authorization (SigV4 still
# enforced), seeded dev credential. A build-time profile for local/CI only;
# the thin bin enforces that it is paired with a dev/CI backend.
# Developer mode: plain HTTP on loopback, open authorization (SigV4
# authentication still enforced), seeded dev credential. A build-time profile
# for local development and CI. The thin bin enforces that it is paired with the
# `sqlite` backend.
dev-mode = ["extenddb-config/dev-mode"]
mongodb-test-hooks = ["extenddb-server/mongodb-test-hooks"]
backup-test-hooks = ["extenddb-server/backup-test-hooks"]
Expand Down
10 changes: 6 additions & 4 deletions crates/bin/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@ mongodb-test-hooks = [
"extenddb-storage-mongodb/test-hooks",
"extenddb-app/mongodb-test-hooks",
]
# Developer mode: plain HTTP on loopback, open authorization (SigV4 still
# enforced), seeded dev credential. A build-time profile for local/CI only; it
# requires a dev backend (`sqlite`/`sqlite-memory`) and fails to build with a
# production backend like `postgres` (enforced at compile time in `main.rs`).
# Developer mode: plain HTTP on loopback, open authorization (SigV4
# authentication still enforced), seeded dev credential. A build-time profile
# for local development and CI. It compiles only with `sqlite` or
# `sqlite-memory`. A build of any other backend with it, `postgres` among them,
# fails (`main.rs` checks this at compile time). A production `sqlite` build
# leaves it off.
dev-mode = ["extenddb-app/dev-mode"]

[dependencies]
Expand Down
28 changes: 14 additions & 14 deletions crates/bin/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,11 @@
//!
//! In-tree backends are selected by mutually exclusive Cargo features:
//! `postgres` (the default production backend), `mongodb` (production, built with
//! `--no-default-features --features mongodb`), `cassandra`, and `sqlite`/`sqlite-memory`
//! (the dev/CI backend). Exactly one must be enabled: [`set_backend`] installs one
//! backend per process, so a build with more than one would be ambiguous and is
//! rejected at compile time.
//! `--no-default-features --features mongodb`), `cassandra`, and `sqlite`/`sqlite-memory`.
//! `sqlite` is for production on a single node, and `sqlite-memory` keeps the database
//! in memory for development and CI. Exactly one must be enabled. [`set_backend`]
//! installs one backend per process, so a build with more than one would be ambiguous
//! and is rejected at compile time.

// Exactly one backend feature must be enabled.
#[cfg(any(
Expand Down Expand Up @@ -42,18 +43,17 @@ compile_error!(
);

// Developer mode relaxes the security posture (plain HTTP on loopback, open
// authorization). It is a dev/CI-only profile and must be built only with a
// dev/CI-suitable backend. Rather than denying each production backend by name
// (every backend is a production backend unless proven otherwise, so a deny-list
// would have to grow with each new one), require a known dev backend: dev-mode
// compiles only when `sqlite` is enabled. `sqlite-memory` enables `sqlite`, so it
// is covered too; postgres, mongodb — or any future production backend — fail the
// build, so there is no path by which a production deployment can serve in dev mode.
// authorization) for development and CI. It compiles only when the build enables
// `sqlite`. `sqlite-memory` enables `sqlite`, so dev mode also compiles with
// `sqlite-memory`. The check names the one backend that dev mode allows, because a
// deny-list of the other backends would have to grow with each new backend. A build
// of any other backend with `dev-mode` fails. A production `sqlite` build leaves
// `dev-mode` off.
#[cfg(all(feature = "dev-mode", not(feature = "sqlite")))]
compile_error!(
"the `dev-mode` feature requires a dev/CI backend such as `sqlite`; it must \
not be built with a production backend like `postgres` or `mongodb` (build \
with `--no-default-features --features sqlite-memory,dev-mode`)"
"the `dev-mode` feature requires the `sqlite` backend and does not build with \
`postgres`, `mongodb` or `cassandra` (build with \
`--no-default-features --features sqlite-memory,dev-mode`)"
);

fn main() -> anyhow::Result<()> {
Expand Down
Loading
Loading