Skip to content

CNTRLPLANE-2121: Add KMS key rotation section #2000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tjungblu wants to merge 2 commits into
base: master
Choose a base branch
from
Open

CNTRLPLANE-2121: Add KMS key rotation section #2000

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0a7ffc5
CNTRLPLANE-2121: Add KMS key rotation section
tjungblu May 7, 2026
d5c7067
revised after todays meeting
tjungblu May 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions enhancements/kube-apiserver/kms-encryption-foundations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ approvers:
api-approvers:
- "@JoelSpeed"
creation-date: 2025-12-03
last-updated: 2026-04-23
last-updated: 2026-05-06
tracking-link:
- "https://redhat.atlassian.net/browse/CNTRLPLANE-243"
see-also:
Expand Down Expand Up @@ -415,9 +415,17 @@ Without this protocol a race can occur: preflight passes for config A, the key-c

Each controller writing its own condition prevents this. If the config changes mid-flight, the key-controller posts a new hash and the preflight controller sees the mismatch and re-runs the check.

#### Variation: KMS Key Rotation
#### KMS Key Rotation

When a KMS plugin rotates its `key_id` (KEK), this triggers neither a new encryption key secret nor a new revision. The mechanism for detecting and handling `key_id` rotation is under evaluation and not covered in this enhancement.
The KMS plugin returns `Status.key_id`, containing a handle for the active Transit identity. Typically combining address information, transit key version and an identifier for the backing key material, so callers can detect that Transit has moved forward without parsing internals.
Copy link
Copy Markdown
Contributor

@csrwng csrwng May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: pls reword this to make it more kms v2 generic. Vault transit is just an instance of a provider.

for instance:

The KMS plugin returns Status.key_id, an opaque string that identifies the currently active KEK in the external KMS (e.g. Vault Transit). This string typically encodes enough information — the transit engine address, key name, and key version in the case of Vault transit — for the operator to detect that the KEK has rotated without needing to parse KMS-specific details.


We want to keep using the existing encryption controllers and to avoid issuing extra kube-apiserver static pod revisions whenever Vault rotates outside OpenShift. The obvious approach, minting a new encryption key secret for every rotation and running the usual multi-secret migration, would reuse migration machinery at the cost of revision churn.

Instead we keep a single encryption key secret per operand and update it in place when the keyId rotates. That breaks the historical one-to-one mapping between encryption key secret name and what the migration state machine treats as the write key, because the secret name no longer advances on an external rotation event.
Copy link
Copy Markdown
Contributor

@csrwng csrwng May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That breaks the historical one-to-one mapping between encryption key secret name and what the migration state machine treats as the write key

What are the implications of this?


We still want migrationController to drive storage migration. For KMS v2 we therefore repurpose internal `WriteKey` so it tracks the KMS plugin `key_id` string rather than equating write identity solely to encryption key secret `metadata.name`. When `key_id` advances, migrationController still observes a write-key transition and can run the existing list and re-encrypt path without requiring a new `encryption-key-kube-apiserver-<n>` resource for that case alone.
Copy link
Copy Markdown
Contributor

@csrwng csrwng May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

repurpose internal WriteKey so it tracks

The internal WriteKey is not described anywhere else in the design. Is it a field in an internal structure?


We proprose to add a RotationController. Plugin monitors publish KMS v2 Status `key_id` and health into operator status, scoped per KMS plugin, per API server, and per control-plane instance. RotationController waits until every monitor reports the same new `key_id`, then patches the existing encryption key secret with an annotation holding that value (exact annotation key TBD, for example `encryption.apiserver.operator.openshift.io/kms-key-id`). That annotation is the operator’s durable record of which `key_id` the cluster has committed to. If monitors disagree, the operator should remain progressing and must not advance the annotation.
Copy link
Copy Markdown
Contributor

@csrwng csrwng May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are plugin monitors part of the new controller? It's not clear what they are from the text.

nit: s/scoped per KMS plugin, per API server, and per control-plane instance/scoped per KMS plugin, per API server, and per control-plane node/
(it's not clear that instance refers to a master node)


### KMS Plugin Lifecycle Management (Tech Preview v2)

Expand Down