Skip to content

Issue the cloud cluster's bootstrap certificates after the apply - #93

Merged
sethbergman merged 2 commits into
mainfrom
feat/cloud-bootstrap-certs
Sep 16, 2026
Merged

sethbergman merged 2 commits into
mainfrom
feat/cloud-bootstrap-certs

Conversation

@sethbergman

Copy link
Copy Markdown
Owner

Stacked on #92, which is stacked on #91. Base retargets as they merge.

Third and last of the blockers found while staging the first real AWS
apply.

The gap

The vault role copies files/tls/<inventory_hostname>.{crt,key} to each
node. On a cloud profile the inventory is dynamic and a host is named by
its instance id, so those filenames are not knowable until after the
apply. There was nothing to pre-generate and commit, and a first real
apply would have stalled here with a cluster running and no way to
configure it.

scripts/generate-cloud-certs.sh reads the hosts from the same inventory
the playbook will use, so it slots between terraform-to-ansible.sh and
ansible-playbook.

SAN on each leaf Who checks it
IP:<private ip> the vault role, verifying the certificate it just delivered
DNS:<instance id> the common name the PKI role later renews it under
DNS:<cluster>.vault.internal every follower, verifying whichever node is leader
DNS:localhost, IP:127.0.0.1 the node curling its own API

Deliberately the same set issue-node-cert.sh and the vault_pki role
issue on renewal: let them diverge and a node that has renewed stops
satisfying a check a node that has not still passes.

The load balancer's name is not among them — AWS generates it and it
cannot be known before the apply, so --extra-san takes it. Without one,
clients dialling terraform output vault_addr get a name mismatch
against a certificate that is otherwise correct.

Writing it turned up why this had never worked

The role verified a delivered certificate with:

openssl x509 -checkhost {{ ansible_default_ipv4.address }}

-checkhost does hostname matching. Against a certificate carrying
IP:10.0.2.15 it reports does NOT match, because it is looking for a
DNS SAN spelled like an address:

$ openssl x509 -in c.pem -noout -checkhost 10.0.2.15
Hostname 10.0.2.15 does NOT match certificate
$ openssl x509 -in c.pem -noout -checkip 10.0.2.15
IP 10.0.2.15 does match certificate

So the check demanded a certificate nothing here issues —
issue-node-cert.sh and the vault_pki role both put the address in
--ip-sans, which is what -checkip reads. A correctly issued
certificate failed it, and the only way to pass was to issue a wrong one.

It had never run: the local profile is Docker Compose and does not use
this role, and neither cloud profile has been applied.

Why the tests live in tests/pki

They read the flag out of the role rather than naming it:

CHECK_FLAG="$(grep -oE '\-check(host|ip)' "$ROLE_TASKS" | head -1)"

A test that picked its own flag would have agreed with the generator and
missed exactly this. Thirteen assertions, suite 75 → 89 — including that
another node's address fails the same check, that a host with no private
address is skipped rather than half-issued, that an empty inventory is an
error rather than a CA with no leaves, and that a second run refuses
without --force (a new CA delivered to some nodes and not others leaves
a cluster that cannot form).

Also

ansible/files/tls/ is now gitignored. It was not, and this is the
first thing that writes private keys there — including a CA key kept
rather than discarded, because an autoscaling group produces replacement
nodes without asking and each one needs a certificate.

Verified

shellcheck clean over scripts/*.sh and every harness. pki 89,
ansible 80, preflight-static 30, cloud-preflight 48, docs-index
and lint green — run in a Linux checkout, not through /mnt/c, where
CRLF makes tests/pki report a false failure on an unrelated assertion.

None of it has run against a real account. The tests assert what the
generator produces and that the role's own check accepts it; what they
cannot show is a node completing a Raft join with the result.

🤖 Generated with Claude Code

sethbergman and others added 2 commits September 15, 2026 21:17
The vault role copies files/tls/<inventory_hostname>.{crt,key} to each
node. On a cloud profile the inventory is dynamic and a host is named by
its instance id, so those filenames are not knowable until after the
apply: there was nothing to pre-generate, and a first real apply would
have stalled with a cluster running and no way to configure it.

generate-cloud-certs.sh reads the hosts from the same inventory the
playbook will use, so it runs between terraform-to-ansible.sh and
ansible-playbook. Each leaf carries the private IP, the instance id, the
cluster servername and localhost -- deliberately the same set
issue-node-cert.sh and the vault_pki role issue on renewal, because if
they diverge a node that has renewed stops satisfying a check a node that
has not still passes.

The load balancer's name is not among them. It is AWS-generated and
unknowable before the apply, so --extra-san takes it; without one,
clients dialling `terraform output vault_addr` get a name mismatch
against a certificate that is otherwise correct.

Writing it turned up why this had never worked. The role verified a
delivered certificate with

    openssl x509 -checkhost {{ ansible_default_ipv4.address }}

and -checkhost does hostname matching. Against a certificate carrying
IP:10.0.2.15 it reports "does NOT match", because it is looking for a DNS
SAN spelled like an address. So the check demanded a certificate nothing
here issues -- issue-node-cert.sh and the vault_pki role both put the
address in --ip-sans, which is what -checkip reads. A correctly issued
certificate failed, and the only way to pass was to issue a wrong one.

It had never run: the local profile is Docker Compose and does not use
this role, and neither cloud profile has been applied.

The tests take the flag out of the role rather than naming it, which is
the point of putting them in tests/pki rather than beside the generator.
A test that picked its own flag would have agreed with the generator and
missed exactly this.

ansible/files/tls/ is now gitignored. It was not, and this is the first
thing that writes private keys there -- including a CA key kept rather
than discarded, because an autoscaling group produces replacement nodes
without asking and each one needs a certificate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sethbergman
sethbergman changed the base branch from feat/ansible-over-ssm to main September 16, 2026 02:40
@sethbergman
sethbergman merged commit 3b6ce19 into main Sep 16, 2026
38 checks passed
@sethbergman
sethbergman deleted the feat/cloud-bootstrap-certs branch September 16, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant