Skip to content

docs: state the catalog trust boundary for the credential encryption key #228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LeeroyHannigan wants to merge 3 commits into
base: main
Choose a base branch
from
+13 −15
Open

docs: state the catalog trust boundary for the credential encryption key #228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
253d167
docs: state the catalog trust boundary for the credential encryption key
LeeroyHannigan Jul 27, 2026
cac6eee
Merge branch 'main' into docs/catalog-trust-boundary
LeeroyHannigan Nov 3, 2026
754cdee
docs: name where catalog credentials can be read, fix stale key notes
LeeroyHannigan Nov 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 2 additions & 8 deletions docs/design/08-component-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ Environment variables use double-underscore (`__`) as a nesting separator, prefi
| `storage.postgres.read_replica_url` | `EXTENDDB__STORAGE__POSTGRES__READ_REPLICA_URL` |
| `storage.sqlite.path` | `EXTENDDB__STORAGE__SQLITE__PATH` |
| `auth.provider` | `EXTENDDB__AUTH__PROVIDER` |
| `auth.encryption_key` | `EXTENDDB__AUTH__ENCRYPTION_KEY` |
| `auth.aws_iam.region` | `EXTENDDB__AUTH__AWS_IAM__REGION` |
| `auth.aws_iam.identity_cache_ttl_seconds` | `EXTENDDB__AUTH__AWS_IAM__IDENTITY_CACHE_TTL_SECONDS` |
| `auth.aws_iam.policy_cache_ttl_seconds` | `EXTENDDB__AUTH__AWS_IAM__POLICY_CACHE_TTL_SECONDS` |
Expand Down Expand Up @@ -121,7 +120,8 @@ statement_timeout_secs = 30
[auth]
provider = "builtin" # "none" | "builtin" | "aws_iam" | future: "azure_ad"
region = "us-east-1" # Region for SigV4 validation
# encryption_key: Set via EXTENDDB__AUTH__ENCRYPTION_KEY env var (never in config files)
# The credential encryption key is not a setting: `extenddb init` generates it
# and stores it in the catalog `settings` table (see docs/manuals/10-security-model.md).
credential_cache_ttl_secs = 300
policy_cache_ttl_secs = 300
timestamp_skew_secs = 300
Expand Down Expand Up @@ -429,7 +429,6 @@ Type=simple
ExecStart=/usr/local/bin/extenddb --config /etc/extenddb/config.toml
Restart=always
RestartSec=5
Environment=EXTENDDB__AUTH__ENCRYPTION_KEY=<secret>

[Install]
WantedBy=multi-user.target
Expand Down Expand Up @@ -459,11 +458,6 @@ spec:
secretKeyRef:
name: db-credentials
key: connection-string
- name: EXTENDDB__AUTH__ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: auth-secrets
key: encryption-key
livenessProbe:
httpGet:
path: /health
Expand Down
5 changes: 2 additions & 3 deletions docs/manuals/02-design-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ On PostgreSQL, the data database connection string is stored in the catalog's `s
| `tables` | DynamoDB table metadata. Composite PK: `(account_id, table_name)`. Stores key schema, attribute definitions, billing mode, stream spec, status, ARN, and TTL config. |
| `indexes` | GSI/LSI metadata. FK on `table_id` (UUID) with CASCADE delete. |
| `tags` | Resource tags. PK: `(resource_arn, tag_key)`. |
| `settings` | Key-value store for catalog version, data DB URL, and runtime settings. |
| `settings` | Key-value store for catalog version, data DB URL, the AES-256-GCM key that encrypts access key secrets (`encryption_key`), and runtime settings. |
| `schema_history` | Migration tracking. Records which SQL files have been applied. |
| `admin_users` | Admin credentials (bcrypt-hashed passwords). |
| `iam_users` | IAM users scoped to accounts. Optional console password. |
Expand All @@ -37,7 +37,6 @@ On PostgreSQL, the data database connection string is stored in the catalog's `s
| `iam_user_tags` | Tags on IAM users. |
| `iam_role_tags` | Tags on IAM roles. |
| `permissions_boundaries` | Permissions boundaries for users and roles. |
| `encryption_keys` | AES-256-GCM key used to encrypt access key secrets. |
| `stream_shards` | Stream shard metadata (parent shard, sequence range). |
| `stream_records` | Stream change records with 24-hour retention. |

Expand Down Expand Up @@ -126,7 +125,7 @@ Policy conditions support all IAM condition operators: `StringEquals`, `StringNo

### Credential Storage

Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `encryption_keys` table. Each access key record stores the encrypted secret and a unique nonce.
Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `settings` table under the key `encryption_key`. Each access key record stores the encrypted secret and a unique nonce.

Credential lookups (access key → encrypted secret) read directly from the database on every request — there is no in-process cache for credentials. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below).

Expand Down
13 changes: 9 additions & 4 deletions docs/manuals/10-security-model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,14 @@ This document describes the security architecture of extenddb, including the thr
1. **Client ↔ extenddb**: Untrusted. All input is validated. SigV4 signatures are verified. IAM policies are evaluated.
2. **extenddb ↔ PostgreSQL**: Trusted network. The PostgreSQL connection string contains credentials. Use TLS for the PostgreSQL connection in production (`sslmode=require` in the connection string).
3. **Admin ↔ Management API/Console**: Authenticated via admin credentials or IAM user credentials. CSRF tokens protect the web console.
4. **Catalog database contents**: Credential-equivalent. The AES-256-GCM key that encrypts access key secrets lives in the catalog `settings` table alongside the ciphertext it protects, so read access to the catalog database yields every stored access key secret. On PostgreSQL the same table holds `data_database_connection_string`, the login for the data database; the console, the management API and the `settings` CLI mask it, but a reader of the catalog database gets it in full. Treat catalog read access, and any catalog backup, as equivalent to holding those credentials. See [Credential Storage](#credential-storage).
5. **Server host**: Credential-equivalent as well. `extenddb.toml` holds the catalog connection string, and a running server keeps the encryption key and recently used decrypted secrets in process memory, so read access to the config file, a core dump of the server, or a read of its memory yields the same credentials.

### Out of Scope

- **PostgreSQL security**: extenddb relies on PostgreSQL access controls and network security. Securing the PostgreSQL instance (firewall rules, TLS, authentication) is the operator's responsibility.
- **Operating system security**: File permissions on `extenddb.toml`, TLS keys, and the PID file are the operator's responsibility.
- **Key management**: Access key secrets are encrypted with a locally generated AES key stored in the catalog database. For HSM-grade key management, use a KMS-backed encryption layer at the PostgreSQL level.
- **Key management**: extenddb generates the credential encryption key locally and stores it unwrapped in the catalog database. It does not wrap the key with an external KMS or HSM, derive it from an operator passphrase, or rotate it. Deployments that require any of those must add protection below extenddb, for example an encrypted volume, a managed database with a customer-managed KMS key, or a PostgreSQL distribution that adds transparent data encryption. That protects the key at rest but not from a principal that can query the catalog.

## Authentication

Expand Down Expand Up @@ -54,7 +56,10 @@ extenddb uses SigV4 signature verification with a local credential store and IAM
#### Credential Storage

- Secret keys encrypted with AES-256-GCM using a per-catalog encryption key
- Encryption key generated during `extenddb init` and stored in the catalog database
- Encryption key generated during `extenddb init` and stored unwrapped in the catalog `settings` table, under the key `encryption_key`
- The encryption key and the ciphertext it protects therefore share one trust domain: anyone who can read the catalog database can decrypt every stored access key secret. Restrict catalog database access to the extenddb service account, and protect catalog backups to the same standard as the credentials themselves.
- The management API and web console redact `encryption_key` from settings responses. The key is readable only with catalog database access: a database client, or `extenddb settings get encryption_key --show-secret` on a host that holds the catalog connection string.
- extenddb has no command to rotate the encryption key. Rotating it by hand makes every stored access key secret and session credential unreadable: re-create all access keys under the new key, then restart the server, which keeps the old key and decrypted secrets in memory.

@jcshepherd jcshepherd Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my suggested wording got the order wrong and also omits a major difficulty with this.

The working order is: replace the encryption key, restart every server (which still is using the old encryption key it has in memory), then re-create the access keys.

However, that means that after the restart every stored access key and session credential will fail to decrypt, so every signed request fails. The user will have to not only recreate all access keys needed by clients, but also distribute the new keys and secrets to each client to enable that client to successfully log in. Major headache people should be aware of before attempting a rotation.

Could the bullet say that plainly? For example:

"Extenddb has no command to rotate the encryption key. Replacing it by hand invalidates every access key and session credential. Replace the key, restart every server, then re-create all access keys and give each client its new credentials. After the restart, each client's requests will until the client has its new credentials."

- Console passwords hashed with bcrypt (cost factor 12)
- No in-process credential cache — every request reads directly from PostgreSQL

@jcshepherd jcshepherd Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hasn't been true for a while: unencrypted credentials are in-memory with a running extenddb server.

@LeeroyHannigan LeeroyHannigan Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. #786 rewrites this bullet with a section on the credential and policy caches, so I left this line to it, to keep the two PRs from conflicting. The new Server host item in Trust Boundaries says the running server keeps the key and recently used decrypted secrets in memory.


Expand Down Expand Up @@ -229,7 +234,7 @@ Import/export file paths are validated:

- `extenddb init` prints admin credentials once; they are not stored in plaintext
- The `--password` flag accepts passwords via environment variable (`EXTENDDB_ADMIN_PASSWORD`) to avoid process listing exposure
- Access key secrets are shown once on creation and cannot be retrieved afterward
- Access key secrets are shown once on creation, and no API or console page returns them again; a reader of the catalog database can still decrypt them (see [Credential Storage](#credential-storage))

### Logging

Expand All @@ -252,7 +257,7 @@ pg_dump extenddb_catalog > catalog_backup.sql
pg_dump extenddb_catalog_data > data_backup.sql
```

Encryption keys are stored in the catalog database. A catalog backup includes the encryption key needed to decrypt access key secrets.
The encryption key is stored in the catalog database, so a catalog dump contains both the key and the encrypted access key secrets. A catalog backup is credential-equivalent: store it encrypted, restrict who can read it, and do not copy it to less-protected environments (for example a developer machine or a shared bucket) to reproduce an issue.

---

Expand Down
Loading