Stop three AWS nodes collapsing into one inventory host - #91
Merged
Merged
Conversation
ansible/inventory/aws.yml preferred tag:Name, and an autoscaling group tags every instance it launches identically -- a launch template has no per-instance interpolation, so compute.tf gives all three nodes Name=<cluster>-vault. aws_ec2's `hostnames` is a list of preferences: _add_hosts takes [_get_preferred_hostname(...)] when allow_duplicated_hosts is unset, and that breaks on the first entry that resolves. All three resolved to the same string, and an Ansible inventory is keyed by host name -- add_host() returns the host that already exists and set_variable() overwrites its vars, ansible_host included. So site.yml configured whichever instance the paginator returned last, and exited 0. Snapshots, audit devices, PKI certificates and hardening on one machine out of three, reported as success. It would also have broken the per-node TLS design, which names certificates after inventory_hostname and would have found one name for three hosts. Name them by instance id instead. It is unique, and it is the identity the node already carries: user-data.sh.tftpl derives Raft's node_id from the same value, so a host in the inventory and a voter in `raft list-peers` can be matched by eye. The filters are untouched -- the tag that couples discovery to configuration management is still the one auto_join uses. Azure needed nothing: azure_rm defaults to the VM name, which is per instance and is what its cloud-init uses for node_id. The assertions go in tests/preflight-static, which exists for strings one layer produces and another consumes strictly. Nothing else could see this: terraform validate reads one file, the mocks read the configuration's shape, and tests/ansible checks the inventory parses as YAML. They pin which values distinguish an instance rather than forbidding tag:Name, because tag:Name is not the only constant an ASG propagates -- VaultCluster is another and it is in that same file. Verified by mutating to tag:VaultCluster, the neighbouring change rather than the named one, plus five others: tag:Name restored, private-ip-address (unique but not the node_id, which fails the agreement half only), the hostnames list deleted, and each side of the node_id derivation broken on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 16, 2026
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while staging the first real AWS apply, which is exactly where it
would otherwise have been found — after spending money.
The bug
ansible/inventory/aws.ymlpreferredtag:Name. An autoscaling grouptags every instance it launches identically — a launch template has no
per-instance interpolation — so
compute.tf:80and:154give all three nodesName=<cluster>-vault.aws_ec2'shostnamesis a list of preferences. Withallow_duplicated_hostsunset (it was),_add_hoststakes[_get_preferred_hostname(...)], which breaks on the first entry thatresolves. All three instances resolved to the same string — and an
Ansible inventory is keyed by host name, so
add_host()returns the hostthat already exists and
set_variable()overwrites its vars,ansible_hostincluded.site.ymlconfigured whichever instance the paginator returned last,and exited 0. Snapshots, audit devices, PKI certificates and hardening
on one machine out of three, reported as success. It would also have
broken the per-node TLS design, which names certificates after
inventory_hostnameand would have found one name for three hosts.The fix
Name hosts by instance id. It is unique, and it is the identity the node
already carries:
user-data.sh.tftplderives Raft'snode_idfrom thesame value, so a host in the inventory and a voter in
vault operator raft list-peerscan be matched by eye.The
filtersare untouched — the tag that couples discovery toconfiguration management is still the one
auto_joinuses. Azure needednothing:
azure_rmdefaults to the VM name, which is per instance and iswhat its cloud-init uses for
node_idtoo.The assertions
They go in
tests/preflight-static, which exists for strings one layerproduces and another consumes strictly. Nothing else could see this:
terraform validatereads one file, the mocked providers read theconfiguration's shape, and
tests/ansiblechecks this file parses asYAML.
They pin which values distinguish an instance rather than forbidding
tag:Name, becausetag:Nameis not the only constant an ASGpropagates —
VaultClusteris another, and it is in that same file.tag:VaultCluster— the neighbouring constant, not the named onetag:Namerestored as first preferenceprivate-ip-address— unique, but not thenode_idhostnameslist deletedlocal-hostnameinstead ofinstance-idnode_idstops usingINSTANCE_ID, imds line untouchedThe third and sixth are the ones that matter: they show the two
assertions discriminate independently rather than both riding on one
string.
🤖 Generated with Claude Code