Rekey unseal keys too, and stop losing the key that unseals vault-unseal

[sethbergman]

rotate-keys.sh covered the recovery-key path only, because the local cluster is auto-unsealed and its shares are recovery keys. The unseal-key path had no coverage at all.

--unseal-keys runs the same ceremony against sys/rekey instead of sys/rekey-recovery-key, exercised against vault-unseal — the Shamir-sealed Vault underneath the profile. Which kind of key a Vault has depends only on how it's sealed, and migrate-seal.sh turns each into the other without changing their values, so this is one ceremony against two endpoints rather than two scripts.

Flag	Endpoint	CLI
--recovery-keys	sys/rekey-recovery-key/*	vault operator rekey -target=recovery
--unseal-keys	sys/rekey/*	vault operator rekey

The CLI calls the second one barrier and makes it the default target — so vault operator rekey with no arguments, run against an auto-unsealed cluster, addresses a set of keys that cluster does not use.

The prerequisite turned out to be a bug

Rekeying needs a quorum of the current shares, and vault-unseal had none. bootstrap-dev-cluster.sh held its unseal key and root token in shell variables and never wrote them down.

That made a single docker compose restart vault-unseal unrecoverable — which is what a Docker Desktop restart or a host reboot does. And it doesn't fail gracefully. vault-unseal comes back sealed, and a cluster node restarted afterwards doesn't come back sealed either; it fails to start:

error parsing Seal configuration: ... 503  * Vault is sealed

with no key anywhere to fix it. The only route back was make destroy — losing the cluster because the container providing its seal was restarted.

This is the same shape as the recovery keys the bootstrap used to discard, fixed in v0.15, in the one place where it costs more. docker/dev/.unseal-keys.json is written 0600 and gitignored now, and the suite restarts vault-unseal on every run to prove the kept keys open it and that a cluster node auto-unseals against it afterwards.

One stale share proves nothing

The suite rekeys vault-unseal twice — 1-of-1 to 5-of-3, then again — so a full quorum of the superseded generation exists to test with.

Vault accepts unseal shares and only validates the combination once the threshold is reached, so a single stale share returns success and 1/3 progress. An assertion built on one old share would pass whether or not the rekey did anything — and would leave that progress behind to break the next unseal. I know because I wrote exactly that assertion first and watched it lie to me.

Testing

tests/key-rotation goes from 28 assertions to 46. Three mutations, all watched to fail.

#	Mutation	Caught by
U1	vault-unseal's keys are not kept, as before this change	the bootstrap kept vault-unseal's own keys; and the kept keys open it again (10 in total)
U2	--unseal-keys addresses the recovery endpoint	the rekey failing outright (5)
U3	Unseal mode reads recovery_keys_b64	the rekey failing outright (5)

U3 is recorded for what it actually broke rather than what it was aimed at: it missed both assertions about handing the wrong key file to the wrong mode, because the run was refused anyway for a different reason and the refusal happened to satisfy them. The table records observations, not intentions.

Still uncovered

A rekey of a Shamir-sealed cluster — several nodes, each needing the new shares — as opposed to the single-node vault-unseal. And migration between two cloud KMS providers, which needs a cloud account.

🤖 Generated with Claude Code