From 253d1673dae21d582d6699e7286a2d1759ee656d Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Mon, 27 Jul 2026 16:27:22 +0000 Subject: [PATCH 1/3] docs: state the catalog trust boundary for the credential encryption key The security model noted that the encryption key lives in the catalog database and that a catalog backup includes it, but left the consequence for the reader to infer and did not mention that no rotation path exists. - Add the catalog database contents as an explicit trust boundary: the key and the ciphertext it protects share one trust domain, so catalog read access yields every stored access key secret. - Scope that boundary to direct database access: the management API and console redact `encryption_key` from settings responses. - Record that there is no key rotation path; re-keying means re-creating the affected access keys. - Replace the "use a KMS-backed encryption layer at the PostgreSQL level" guidance with what extenddb does and does not do, and what protection below extenddb can and cannot achieve. - State that a catalog dump is credential-equivalent and should not be copied to less-protected environments. --- docs/manuals/10-security-model.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/manuals/10-security-model.md b/docs/manuals/10-security-model.md index bb7a1c75d..359eabb52 100755 --- a/docs/manuals/10-security-model.md +++ b/docs/manuals/10-security-model.md @@ -19,12 +19,13 @@ This document describes the security architecture of extenddb, including the thr 1. **Client ↔ extenddb**: Untrusted. All input is validated. SigV4 signatures are verified. IAM policies are evaluated. 2. **extenddb ↔ PostgreSQL**: Trusted network. The PostgreSQL connection string contains credentials. Use TLS for the PostgreSQL connection in production (`sslmode=require` in the connection string). 3. **Admin ↔ Management API/Console**: Authenticated via admin credentials or IAM user credentials. CSRF tokens protect the web console. +4. **Catalog database contents**: Credential-equivalent. The AES-256-GCM key that encrypts access key secrets lives in the catalog `settings` table alongside the ciphertext it protects, so read access to the catalog database yields every stored access key secret. Treat catalog read access, and any catalog backup, as equivalent to holding those credentials. See [Credential Storage](#credential-storage). ### Out of Scope - **PostgreSQL security**: extenddb relies on PostgreSQL access controls and network security. Securing the PostgreSQL instance (firewall rules, TLS, authentication) is the operator's responsibility. - **Operating system security**: File permissions on `extenddb.toml`, TLS keys, and the PID file are the operator's responsibility. -- **Key management**: Access key secrets are encrypted with a locally generated AES key stored in the catalog database. For HSM-grade key management, use a KMS-backed encryption layer at the PostgreSQL level. +- **Key management**: extenddb generates the credential encryption key locally and stores it unwrapped in the catalog database. It does not wrap the key with an external KMS or HSM, derive it from an operator passphrase, or rotate it. Deployments that require any of those must add protection below extenddb, for example an encrypted volume, a managed database with a customer-managed KMS key, or a PostgreSQL distribution that adds transparent data encryption. That protects the key at rest but not from a principal that can query the catalog. ## Authentication @@ -54,7 +55,10 @@ extenddb uses SigV4 signature verification with a local credential store and IAM #### Credential Storage - Secret keys encrypted with AES-256-GCM using a per-catalog encryption key -- Encryption key generated during `extenddb init` and stored in the catalog database +- Encryption key generated during `extenddb init` and stored unwrapped in the catalog `settings` table, under the key `encryption_key` +- The encryption key and the ciphertext it protects therefore share one trust domain: anyone who can read the catalog database can decrypt every stored access key secret. Restrict catalog database access to the extenddb service account, and protect catalog backups to the same standard as the credentials themselves. +- The management API and web console redact `encryption_key` from settings responses. The key is readable only with catalog database access: a database client, or `extenddb settings get encryption_key --show-secret` on a host that holds the catalog connection string. +- extenddb provides no key rotation path. Re-keying requires re-creating the affected access keys. - Console passwords hashed with bcrypt (cost factor 12) - No in-process credential cache — every request reads directly from PostgreSQL @@ -252,7 +256,7 @@ pg_dump extenddb_catalog > catalog_backup.sql pg_dump extenddb_catalog_data > data_backup.sql ``` -Encryption keys are stored in the catalog database. A catalog backup includes the encryption key needed to decrypt access key secrets. +The encryption key is stored in the catalog database, so a catalog dump contains both the key and the encrypted access key secrets. A catalog backup is credential-equivalent: store it encrypted, restrict who can read it, and do not copy it to less-protected environments (for example a developer machine or a shared bucket) to reproduce an issue. --- From 754cdee5a65bcc6336a1426d6909967f01204ed5 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 09:54:30 +0000 Subject: [PATCH 2/3] docs: name where catalog credentials can be read, fix stale key notes - 10-security-model.md: the catalog trust boundary names the PostgreSQL data database login held in settings; a Server host boundary covers extenddb.toml and the server's memory; the rotation bullet says what a manual rotation breaks and why the server must restart; and the Operational Security bullet no longer says secrets cannot be retrieved. - 02-design-guide.md: the key is a settings row, not an encryption_keys table. - 08-component-config.md: drop the auth.encryption_key setting and EXTENDDB__AUTH__ENCRYPTION_KEY, which AuthConfig does not have. --- docs/design/08-component-config.md | 10 ++-------- docs/manuals/02-design-guide.md | 5 ++--- docs/manuals/10-security-model.md | 7 ++++--- 3 files changed, 8 insertions(+), 14 deletions(-) diff --git a/docs/design/08-component-config.md b/docs/design/08-component-config.md index 8e7e59f15..5bcddb5d7 100755 --- a/docs/design/08-component-config.md +++ b/docs/design/08-component-config.md @@ -52,7 +52,6 @@ Environment variables use double-underscore (`__`) as a nesting separator, prefi | `storage.postgres.read_replica_url` | `EXTENDDB__STORAGE__POSTGRES__READ_REPLICA_URL` | | `storage.sqlite.path` | `EXTENDDB__STORAGE__SQLITE__PATH` | | `auth.provider` | `EXTENDDB__AUTH__PROVIDER` | -| `auth.encryption_key` | `EXTENDDB__AUTH__ENCRYPTION_KEY` | | `auth.aws_iam.region` | `EXTENDDB__AUTH__AWS_IAM__REGION` | | `auth.aws_iam.identity_cache_ttl_seconds` | `EXTENDDB__AUTH__AWS_IAM__IDENTITY_CACHE_TTL_SECONDS` | | `auth.aws_iam.policy_cache_ttl_seconds` | `EXTENDDB__AUTH__AWS_IAM__POLICY_CACHE_TTL_SECONDS` | @@ -121,7 +120,8 @@ statement_timeout_secs = 30 [auth] provider = "builtin" # "none" | "builtin" | "aws_iam" | future: "azure_ad" region = "us-east-1" # Region for SigV4 validation -# encryption_key: Set via EXTENDDB__AUTH__ENCRYPTION_KEY env var (never in config files) +# The credential encryption key is not a setting: `extenddb init` generates it +# and stores it in the catalog `settings` table (see docs/manuals/10-security-model.md). credential_cache_ttl_secs = 300 policy_cache_ttl_secs = 300 timestamp_skew_secs = 300 @@ -429,7 +429,6 @@ Type=simple ExecStart=/usr/local/bin/extenddb --config /etc/extenddb/config.toml Restart=always RestartSec=5 -Environment=EXTENDDB__AUTH__ENCRYPTION_KEY= [Install] WantedBy=multi-user.target @@ -459,11 +458,6 @@ spec: secretKeyRef: name: db-credentials key: connection-string - - name: EXTENDDB__AUTH__ENCRYPTION_KEY - valueFrom: - secretKeyRef: - name: auth-secrets - key: encryption-key livenessProbe: httpGet: path: /health diff --git a/docs/manuals/02-design-guide.md b/docs/manuals/02-design-guide.md index 26fac44cb..69ef3923c 100755 --- a/docs/manuals/02-design-guide.md +++ b/docs/manuals/02-design-guide.md @@ -23,7 +23,7 @@ On PostgreSQL, the data database connection string is stored in the catalog's `s | `tables` | DynamoDB table metadata. Composite PK: `(account_id, table_name)`. Stores key schema, attribute definitions, billing mode, stream spec, status, ARN, and TTL config. | | `indexes` | GSI/LSI metadata. FK on `table_id` (UUID) with CASCADE delete. | | `tags` | Resource tags. PK: `(resource_arn, tag_key)`. | -| `settings` | Key-value store for catalog version, data DB URL, and runtime settings. | +| `settings` | Key-value store for catalog version, data DB URL, the AES-256-GCM key that encrypts access key secrets (`encryption_key`), and runtime settings. | | `schema_history` | Migration tracking. Records which SQL files have been applied. | | `admin_users` | Admin credentials (bcrypt-hashed passwords). | | `iam_users` | IAM users scoped to accounts. Optional console password. | @@ -37,7 +37,6 @@ On PostgreSQL, the data database connection string is stored in the catalog's `s | `iam_user_tags` | Tags on IAM users. | | `iam_role_tags` | Tags on IAM roles. | | `permissions_boundaries` | Permissions boundaries for users and roles. | -| `encryption_keys` | AES-256-GCM key used to encrypt access key secrets. | | `stream_shards` | Stream shard metadata (parent shard, sequence range). | | `stream_records` | Stream change records with 24-hour retention. | @@ -126,7 +125,7 @@ Policy conditions support all IAM condition operators: `StringEquals`, `StringNo ### Credential Storage -Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `encryption_keys` table. Each access key record stores the encrypted secret and a unique nonce. +Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `settings` table under the key `encryption_key`. Each access key record stores the encrypted secret and a unique nonce. Credential lookups (access key → encrypted secret) read directly from the database on every request — there is no in-process cache for credentials. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below). diff --git a/docs/manuals/10-security-model.md b/docs/manuals/10-security-model.md index 359eabb52..4cc0fe57f 100755 --- a/docs/manuals/10-security-model.md +++ b/docs/manuals/10-security-model.md @@ -19,7 +19,8 @@ This document describes the security architecture of extenddb, including the thr 1. **Client ↔ extenddb**: Untrusted. All input is validated. SigV4 signatures are verified. IAM policies are evaluated. 2. **extenddb ↔ PostgreSQL**: Trusted network. The PostgreSQL connection string contains credentials. Use TLS for the PostgreSQL connection in production (`sslmode=require` in the connection string). 3. **Admin ↔ Management API/Console**: Authenticated via admin credentials or IAM user credentials. CSRF tokens protect the web console. -4. **Catalog database contents**: Credential-equivalent. The AES-256-GCM key that encrypts access key secrets lives in the catalog `settings` table alongside the ciphertext it protects, so read access to the catalog database yields every stored access key secret. Treat catalog read access, and any catalog backup, as equivalent to holding those credentials. See [Credential Storage](#credential-storage). +4. **Catalog database contents**: Credential-equivalent. The AES-256-GCM key that encrypts access key secrets lives in the catalog `settings` table alongside the ciphertext it protects, so read access to the catalog database yields every stored access key secret. On PostgreSQL the same table holds `data_database_connection_string`, the login for the data database; the console, the management API and the `settings` CLI mask it, but a reader of the catalog database gets it in full. Treat catalog read access, and any catalog backup, as equivalent to holding those credentials. See [Credential Storage](#credential-storage). +5. **Server host**: Credential-equivalent as well. `extenddb.toml` holds the catalog connection string, and a running server keeps the encryption key and recently used decrypted secrets in process memory, so read access to the config file, a core dump of the server, or a read of its memory yields the same credentials. ### Out of Scope @@ -58,7 +59,7 @@ extenddb uses SigV4 signature verification with a local credential store and IAM - Encryption key generated during `extenddb init` and stored unwrapped in the catalog `settings` table, under the key `encryption_key` - The encryption key and the ciphertext it protects therefore share one trust domain: anyone who can read the catalog database can decrypt every stored access key secret. Restrict catalog database access to the extenddb service account, and protect catalog backups to the same standard as the credentials themselves. - The management API and web console redact `encryption_key` from settings responses. The key is readable only with catalog database access: a database client, or `extenddb settings get encryption_key --show-secret` on a host that holds the catalog connection string. -- extenddb provides no key rotation path. Re-keying requires re-creating the affected access keys. +- extenddb has no command to rotate the encryption key. Rotating it by hand makes every stored access key secret and session credential unreadable: re-create all access keys under the new key, then restart the server, which keeps the old key and decrypted secrets in memory. - Console passwords hashed with bcrypt (cost factor 12) - No in-process credential cache — every request reads directly from PostgreSQL @@ -233,7 +234,7 @@ Import/export file paths are validated: - `extenddb init` prints admin credentials once; they are not stored in plaintext - The `--password` flag accepts passwords via environment variable (`EXTENDDB_ADMIN_PASSWORD`) to avoid process listing exposure -- Access key secrets are shown once on creation and cannot be retrieved afterward +- Access key secrets are shown once on creation, and no API or console page returns them again; a reader of the catalog database can still decrypt them (see [Credential Storage](#credential-storage)) ### Logging From 9c2a4b80121eadc02dc474f2e2f32269484a0337 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 10 Nov 2026 22:29:58 +0000 Subject: [PATCH 3/3] docs: give the order and the client impact of a manual key rotation The server reads the encryption key at startup and keeps it in memory. The server encrypts an access key created before the restart with the old key. So the steps are: replace the key, restart every server, then delete every access key and create a new one. The new key is 32 random bytes in standard base64. The server checks only that the key is present, so a value of any other form makes every signed request fail. From the restart on, a request signed with an old access key or an unexpired session credential fails with InternalServerError. Each client needs its new key, and an admin calls AssumeRole again for each client that uses session credentials. The web console and extenddb manage sign in with a password, so they still work for the new keys. extenddb settings set refuses encryption_key, so the operator replaces the key in the settings table by hand. The design guide no longer calls the key immutable. The configuration design no longer says that the key is not a setting. Signed-off-by: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> --- docs/design/08-component-config.md | 2 +- docs/manuals/02-design-guide.md | 4 ++-- docs/manuals/10-security-model.md | 6 +++++- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/design/08-component-config.md b/docs/design/08-component-config.md index 5bcddb5d7..bb0e1967f 100755 --- a/docs/design/08-component-config.md +++ b/docs/design/08-component-config.md @@ -120,7 +120,7 @@ statement_timeout_secs = 30 [auth] provider = "builtin" # "none" | "builtin" | "aws_iam" | future: "azure_ad" region = "us-east-1" # Region for SigV4 validation -# The credential encryption key is not a setting: `extenddb init` generates it +# The credential encryption key is not read from this file: `extenddb init` generates it # and stores it in the catalog `settings` table (see docs/manuals/10-security-model.md). credential_cache_ttl_secs = 300 policy_cache_ttl_secs = 300 diff --git a/docs/manuals/02-design-guide.md b/docs/manuals/02-design-guide.md index 0e5a6f94e..af924870f 100755 --- a/docs/manuals/02-design-guide.md +++ b/docs/manuals/02-design-guide.md @@ -128,7 +128,7 @@ Policy conditions support all IAM condition operators: `StringEquals`, `StringNo Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `settings` table under the key `encryption_key`. Each access key record stores the encrypted secret and a unique nonce. -Credential lookups (access key ID → decrypted secret) go through each server's in-memory credential cache; a miss reads the database. See Credential and Policy Caching in the Security Model. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below). +Credential lookups (access key ID → decrypted secret) go through each server's in-memory credential cache; a miss reads the database. See Credential and Policy Caching in the Security Model. The server reads the encryption key that decrypts secrets once, at startup. extenddb never changes the key. A change made by hand in the `settings` table takes effect at the next restart. See Credential Storage in the Security Model, and Caching Design below. ## DynamoDB Streams Internals @@ -190,7 +190,7 @@ extenddb caches a small set of operational settings in memory to avoid per-reque | Setting | Mechanism | Refresh | Justification | |---------|-----------|---------|---------------| | `index_propagation_delay_ms` | `AtomicU64` | Background poller every 30s | Write-path hot path; briefly-stale value only affects GSI propagation timing | -| `encryption_key` | `Arc` loaded at startup | Never (immutable after `extenddb init`) | Decryption key for access key secrets; generated once, never changes | +| `encryption_key` | `Arc` loaded at startup | Never: read at startup, so a change made by hand takes a restart | Decryption key for access key secrets. extenddb generates it once and never changes it | | `log_level` / `log_destination` | Tracing filter reload | Background poller every 30s | Observability tuning; stale value only delays log level changes | | `throttling_enabled` | `AtomicBool` | Background poller every 30s | Capacity management toggle; briefly-stale is safe | diff --git a/docs/manuals/10-security-model.md b/docs/manuals/10-security-model.md index a53c6f572..b69ea2b23 100755 --- a/docs/manuals/10-security-model.md +++ b/docs/manuals/10-security-model.md @@ -63,7 +63,11 @@ Request bodies are parsed before authentication (the service does the same), so - Encryption key generated during `extenddb init` and stored unwrapped in the catalog `settings` table, under the key `encryption_key` - The encryption key and the ciphertext it protects therefore share one trust domain: anyone who can read the catalog database can decrypt every stored access key secret. Restrict catalog database access to the extenddb service account, and protect catalog backups to the same standard as the credentials themselves. - The management API and web console redact `encryption_key` from settings responses. The key is readable only with catalog database access: a database client, or `extenddb settings get encryption_key --show-secret` on a host that holds the catalog connection string. -- extenddb has no command to rotate the encryption key. Rotating it by hand makes every stored access key secret and session credential unreadable: re-create all access keys under the new key, then restart the server, which keeps the old key and decrypted secrets in memory. +- extenddb has no command to rotate the encryption key, and `extenddb settings set` refuses `encryption_key`. Replacing the key by hand in the catalog `settings` table makes every stored access key secret and session credential unreadable. The server reads the key at startup and keeps it in memory, so a rotation takes these steps in order: + 1. Replace the value of `encryption_key` in the catalog `settings` table with 32 random bytes in standard base64, for example the output of `openssl rand -base64 32`. On MongoDB the key is the `value` field of the `settings` document whose `_id` is `encryption_key`. The server checks only that the key is present. A value of any other form makes every signed request, CreateAccessKey call and AssumeRole call fail. + 2. Restart every server that serves the catalog. From the restart on, each request signed with an old access key or an unexpired session credential fails with `InternalServerError` (HTTP 500). Once step 3 deletes an old key, a request signed with it fails with `UnrecognizedClientException`. + 3. Delete every access key and create a new one. For each client that uses AssumeRole session credentials, call AssumeRole again with `extenddb manage assume-role`. The web console and `extenddb manage` sign in with a password, so they still work for this step. + 4. Give each client its new credentials: an access key ID and secret, or the new session's access key ID, secret and session token. Until then, each of its requests fails. - Console passwords hashed with bcrypt (cost factor 12) - Credentials and the inputs to authorization are cached in each server's memory; see Credential and Policy Caching below