Rotate the credential encryption key without invalidating access keys #941

What

ExtendDB encrypts each stored access key secret with one AES-256-GCM key, encryption_key, in the catalog settings table. There is no way to rotate that key without breaking every credential.

• No command replaces the key. extenddb settings set encryption_key and the management API refuse it, because it is not a known setting.
• An operator can only replace it by hand in the catalog database.
• Every stored secret is encrypted with the old key, so after a server restart none of them decrypts. Every access key and session credential stops working.
• The operator must then re-create every access key. Each one gets a new access key ID and secret, so every client needs new credentials.

So a rotation is a full credential reset, with an outage for every signed request until each client has its new credentials. Rotating a key after a suspected leak, or on a schedule, is a routine security task, so this makes ExtendDB hard to run safely.

Code read on main at 4ee13f9 (PostgreSQL):

• The server reads encryption_key once at startup and keeps it (crates/storage-postgres/src/lib.rs:949-964).
• create_access_key encrypts with the kept key (crates/storage-postgres/src/management_store/access_keys.rs:22).
• extenddb manage create-access-key goes through the running server (crates/app/src/manage_http.rs:290).

I did not check the other three backends' startup code. Each one stores the key in its catalog settings and writes it only when none exists.

What a rotation should do

Clients should not notice. Access key IDs and secrets stay the same.

1. Generate a new key.
2. For each stored secret (access keys, and the encrypted secret of each IAM session): decrypt it with the old key, and encrypt it with the new key.
3. Store the new key, and keep the old key until every server uses the new one.
4. Servers decrypt with either key while the rotation runs, and encrypt only with the new key.
5. Remove the old key once every row and every server has moved.

Open questions:

• Should each stored secret record which key encrypted it (a key ID or version), so a server knows which key to use? This also makes a partial rotation safe to resume.
• How does each running server learn of the new key: a restart, a reload signal, or a poll of the catalog?
• Should the decrypted secrets that servers keep in their credential cache be flushed?
• Should this also allow a key from outside the catalog (for example a KMS-wrapped key), so the key no longer sits beside its ciphertext?

Related

• docs: state the catalog trust boundary for the credential encryption key #228 documents the current behavior.
• PostgreSQL: the data database login is stored in the catalog settings table #789: the PostgreSQL data database login is stored in the catalog.