Rekey unseal keys too, and stop losing the key that unseals vault-unseal

.gitignore

@@ -39,6 +39,12 @@ docker/dev/tls/
 # were live minutes ago.
 docker/dev/.recovery-keys.json*
 
+# vault-unseal's own unseal key and root token. It is the root of
+# trust for the local profile — the Transit key every node unseals
+# against — so this file is the most sensitive one the bootstrap
+# writes, and the one whose loss costs the whole cluster.
+docker/dev/.unseal-keys.json*
+
 # Vault Agent credentials, written by scripts/bootstrap-agent.sh.
 # role_id is not a secret; secret_id is, and Agent deletes it after
 # reading it anyway. Neither belongs in the repository.

.gitleaks.toml

@@ -0,0 +1,42 @@
+# gitleaks configuration.
+#
+# Until this file existed gitleaks ran on its defaults, which is what you
+# want almost all of the time — this exists to record one accepted
+# finding, not to relax the ruleset. `useDefault` keeps every default
+# rule; nothing below removes one.
+#
+# The same rule applies here as in .trivyignore.yaml: an allowlist entry
+# states what it permits and why. A suppression without a justification
+# is indistinguishable from never having run the scanner, and gitleaks
+# scans full history, so anything muted here stays muted for every commit
+# that follows.
+
+title = "vault-reference-platform"
+
+[extend]
+useDefault = true
+
+[allowlist]
+description = """
+The names of two JSON fields, which are not secrets.
+
+Vault returns key shares under `unseal_keys_b64` (Shamir) or
+`recovery_keys_b64` (auto-unsealed), and scripts/rotate-keys.sh selects
+between them with a variable:
+
+    KEY_FIELD="unseal_keys_b64"
+
+generic-api-key reads that as KEY = <high-entropy value> and reports it,
+with an entropy of 3.5 for a string that is documented API vocabulary and
+appears verbatim in Vault's own documentation.
+
+The allowlist is anchored to those two exact strings rather than to the
+file or the rule, so a real credential on the same line, in the same
+file, or assigned to the same variable is still reported. Shares
+themselves are ~44 characters of base64 and match nothing below.
+"""
+regexTarget = "secret"
+regexes = [
+  '''^unseal_keys_b64$''',
+  '''^recovery_keys_b64$''',
+]

CLAUDE.md

@@ -122,11 +122,15 @@ Log output goes to stderr; the root token is the only thing on stdout,
 so `ROOT_TOKEN=$(./scripts/bootstrap-dev-cluster.sh)` works. Preserve
 that split when editing.
 
-The recovery keys go to `docker/dev/.recovery-keys.json` (0600,
-gitignored) rather than stdout. With a seal stanza `operator init`
-returns those instead of unseal keys, and they are what `generate-root`
-and `rekey` need — the script used to discard them, which made revoking
-the root token a one-way door.
+Two key files go to `docker/dev/` (0600, gitignored) rather than stdout:
+`.recovery-keys.json` for the cluster, and `.unseal-keys.json` for
+`vault-unseal` itself — that one is the root of trust for the profile,
+and without it a restart of that single container ends the cluster.
+
+With a seal stanza `operator init` returns recovery keys instead of
+unseal keys, and they are what `generate-root` and `rekey` need. The
+script used to discard both sets, which made revoking the root token a
+one-way door and a restart of vault-unseal an unrecoverable one.
 
 ## The four profiles
 
@@ -383,7 +387,13 @@ CI enforces several invariants worth knowing before you push:
   go in `.trivyignore.yaml` *with the reason* — an unjustified
   suppression is indistinguishable from never having run the scanner.
 - **gitleaks scans full history** (`fetch-depth: 0`). A secret committed
-  and later removed is still leaked.
+  and later removed is still leaked. Accepted findings go in
+  `.gitleaks.toml`, with the reason, on the same terms as
+  `.trivyignore.yaml` — and anchored to the exact string rather than to a
+  file or a rule, because anything muted there stays muted for every
+  commit after it. The one entry today is two JSON field names,
+  `unseal_keys_b64` and `recovery_keys_b64`, which `generic-api-key`
+  reads as high-entropy values assigned to something called KEY.
 - **Every alert rule needs a severity route of its own** and every
   freshness alert needs a paired `absent()` alert; the alerting and
   alert-routing suites fail if a new one arrives without its partner. A

docs/roadmap.md

@@ -373,11 +373,26 @@ not a property the architecture has to have on the day it is stood up.
   will not auto-unseal even with a working seal stanza, which makes the
   obvious way to verify the migration the thing that breaks it.
 
-  What remains needs something this repository does not have. Migrating
-  between two *cloud* KMS providers — the case where an organisation
-  changes cloud — has the same shape and no coverage, and neither does a
-  Shamir **rekey**: the local profile is Transit-sealed, so its shares
-  are recovery keys and `rotate-keys.sh` exercises that path only.
+  The Shamir rekey came off this list too. `rotate-keys.sh
+  --unseal-keys` runs the same ceremony against `sys/rekey` rather than
+  `sys/rekey-recovery-key`, and `tests/key-rotation` exercises it against
+  `vault-unseal` — rekeying it from the 1-of-1 the bootstrap creates to
+  5-of-3, then again, and requiring that a full quorum of the superseded
+  generation no longer opens it. One share proves nothing there: Vault
+  accepts shares and only validates the combination at the threshold, so
+  a lone stale share returns success.
+
+  That work needed a prior fix, the same shape as the recovery keys in
+  v0.15. `vault-unseal` is the root of trust for the whole local profile
+  and its unseal key lived only in a shell variable, so a restart of that
+  one container ended the cluster — it came back sealed with nobody
+  holding the key, and cluster nodes restarted afterwards failed to start
+  rather than coming back sealed. The keys are kept now, and the suite
+  restarts `vault-unseal` on every run to prove it.
+
+  What remains needs something this repository does not have: migrating
+  between two *cloud* KMS providers, the case where an organisation
+  changes cloud, which has the same shape and no coverage.
 - **Restore verification at a real cloud destination.** The mechanism
   came off this list in v0.16. `tests/restore-from-object-store` puts a
   real snapshot of a real cluster through an S3 API, reads the object

docs/security.md

@@ -314,6 +314,57 @@ shares are dead, and if the new ones were not captured, nobody can
 generate a root token or unseal by recovery again — discovered in the
 emergency where you needed them.
 
+### Recovery keys and unseal keys are the same ceremony
+
+Which kind a Vault has depends only on how it is sealed. An auto-unsealed
+cluster has recovery keys; a Shamir-sealed one has unseal keys;
+`scripts/migrate-seal.sh` turns each into the other without changing
+their values. So one script covers both, against two endpoints:
+
+| Flag | Endpoint | CLI |
+|---|---|---|
+| `--recovery-keys` | `sys/rekey-recovery-key/*` | `vault operator rekey -target=recovery` |
+| `--unseal-keys` | `sys/rekey/*` | `vault operator rekey` |
+
+The CLI calls the second one *barrier* and makes it the default target,
+which is worth knowing: `vault operator rekey` with no arguments, run
+against an auto-unsealed cluster, addresses a set of keys that cluster
+does not use.
+
+```bash
+export VAULT_ADDR=https://127.0.0.1:8300      # vault-unseal
+export VAULT_TOKEN=$(jq -r .root_token docker/dev/.unseal-keys.json)
+
+./scripts/rotate-keys.sh --unseal-keys \
+    --keys-file docker/dev/.unseal-keys.json --shares 5 --threshold 3
+```
+
+### The local root of trust keeps its own key now
+
+`vault-unseal` is a Shamir-sealed Vault holding the Transit key every
+cluster node auto-unseals against. Its unseal key used to live in a shell
+variable inside `bootstrap-dev-cluster.sh` and nowhere else.
+
+That made a single `docker compose restart vault-unseal` unrecoverable,
+which is not hypothetical — it is what a Docker Desktop restart or a host
+reboot does. And the failure is not graceful. `vault-unseal` comes back
+sealed, and a cluster node restarted after that does not come back
+sealed; it fails to start:
+
+```text
+error parsing Seal configuration: ... 503  * Vault is sealed
+```
+
+with no key anywhere to fix it. The only way back was `make destroy`.
+
+The bootstrap writes `docker/dev/.unseal-keys.json` (0600, gitignored)
+now, holding the shares and the root token, and `tests/key-rotation`
+restarts `vault-unseal` on every run to prove the kept keys open it and
+that a cluster node auto-unseals against it afterwards.
+
+It is also what makes a Shamir rekey testable at all: rekeying needs a
+quorum of the current shares, and there were none.
+
 ### Why the rekey is scripted rather than documented
 
 Vault has a verification phase for exactly this risk: the new shares are

scripts/bootstrap-dev-cluster.sh

@@ -257,6 +257,34 @@ INIT_JSON="$(compose exec -T vault-unseal vault operator init -key-shares=1 -key
 UNSEAL_KEY="$(jq -r '.unseal_keys_b64[0]' <<< "$INIT_JSON")"
 UNSEAL_ROOT_TOKEN="$(jq -r '.root_token' <<< "$INIT_JSON")"
 
+# Kept, for the same reason the cluster's recovery keys are kept a few
+# hundred lines below, and with more at stake.
+#
+# vault-unseal is the root of trust for the whole local profile: it holds
+# the Transit key every cluster node auto-unseals against. Until now its
+# unseal key lived in this shell variable and nowhere else, which made a
+# single `docker compose restart vault-unseal` unrecoverable — and that
+# is not a hypothetical, it is what a Docker Desktop restart or a host
+# reboot does.
+#
+# The failure is not graceful. vault-unseal comes back sealed, and a
+# cluster node restarted after that does not come back sealed, it fails
+# to start at all:
+#
+#     error parsing Seal configuration: ... 503  * Vault is sealed
+#
+# with no key anywhere to fix it. The only route back was `make destroy`,
+# which is to say: losing everything in the cluster because the container
+# providing its seal was restarted.
+UNSEAL_KEYS_FILE="${COMPOSE_DIR}/.unseal-keys.json"
+rm -f "$UNSEAL_KEYS_FILE"
+( umask 077; jq '{unseal_keys_b64, unseal_keys_shares: .unseal_shares,
+                  unseal_threshold: .unseal_threshold, root_token}' \
+    <<< "$INIT_JSON" > "$UNSEAL_KEYS_FILE" )
+chmod 0600 "$UNSEAL_KEYS_FILE"
+log "vault-unseal keys written to docker/dev/.unseal-keys.json (mode 0600)."
+log "  Without them a restart of vault-unseal takes the cluster with it."
+
 log "Unsealing vault-unseal..."
 compose exec -T vault-unseal vault operator unseal "$UNSEAL_KEY" >/dev/null