Skip to content

Stop three AWS nodes collapsing into one inventory host - #91

Merged
sethbergman merged 1 commit into
mainfrom
fix/aws-inventory-host-collision
Sep 16, 2026
Merged

sethbergman merged 1 commit into
mainfrom
fix/aws-inventory-host-collision

Conversation

@sethbergman

Copy link
Copy Markdown
Owner

Found while staging the first real AWS apply, which is exactly where it
would otherwise have been found — after spending money.

The bug

ansible/inventory/aws.yml preferred tag:Name. An autoscaling group
tags every instance it launches identically — a launch template has no
per-instance interpolation — so
compute.tf:80 and
:154 give all three nodes
Name=<cluster>-vault.

aws_ec2's hostnames is a list of preferences. With
allow_duplicated_hosts unset (it was), _add_hosts takes
[_get_preferred_hostname(...)], which breaks on the first entry that
resolves. All three instances resolved to the same string — and an
Ansible inventory is keyed by host name, so add_host() returns the host
that already exists and set_variable() overwrites its vars,
ansible_host included.

site.yml configured whichever instance the paginator returned last,
and exited 0.
Snapshots, audit devices, PKI certificates and hardening
on one machine out of three, reported as success. It would also have
broken the per-node TLS design, which names certificates after
inventory_hostname and would have found one name for three hosts.

The fix

Name hosts by instance id. It is unique, and it is the identity the node
already carries: user-data.sh.tftpl derives Raft's node_id from the
same value, so a host in the inventory and a voter in vault operator raft list-peers can be matched by eye.

The filters are untouched — the tag that couples discovery to
configuration management is still the one auto_join uses. Azure needed
nothing: azure_rm defaults to the VM name, which is per instance and is
what its cloud-init uses for node_id too.

The assertions

They go in tests/preflight-static, which exists for strings one layer
produces and another consumes strictly. Nothing else could see this:
terraform validate reads one file, the mocked providers read the
configuration's shape, and tests/ansible checks this file parses as
YAML.

They pin which values distinguish an instance rather than forbidding
tag:Name, because tag:Name is not the only constant an ASG
propagates — VaultCluster is another, and it is in that same file.

Mutation Result
tag:VaultCluster — the neighbouring constant, not the named one uniqueness + agreement both fail
tag:Name restored as first preference uniqueness + agreement both fail
private-ip-address — unique, but not the node_id agreement fails alone
hostnames list deleted "declares how it names a host" fails
user-data reads local-hostname instead of instance-id agreement fails
node_id stops using INSTANCE_ID, imds line untouched node_id-source fails alone

The third and sixth are the ones that matter: they show the two
assertions discriminate independently rather than both riding on one
string.

🤖 Generated with Claude Code

ansible/inventory/aws.yml preferred tag:Name, and an autoscaling group
tags every instance it launches identically -- a launch template has no
per-instance interpolation, so compute.tf gives all three nodes
Name=<cluster>-vault.

aws_ec2's `hostnames` is a list of preferences: _add_hosts takes
[_get_preferred_hostname(...)] when allow_duplicated_hosts is unset, and
that breaks on the first entry that resolves. All three resolved to the
same string, and an Ansible inventory is keyed by host name --
add_host() returns the host that already exists and set_variable()
overwrites its vars, ansible_host included.

So site.yml configured whichever instance the paginator returned last,
and exited 0. Snapshots, audit devices, PKI certificates and hardening
on one machine out of three, reported as success. It would also have
broken the per-node TLS design, which names certificates after
inventory_hostname and would have found one name for three hosts.

Name them by instance id instead. It is unique, and it is the identity
the node already carries: user-data.sh.tftpl derives Raft's node_id from
the same value, so a host in the inventory and a voter in `raft
list-peers` can be matched by eye. The filters are untouched -- the tag
that couples discovery to configuration management is still the one
auto_join uses. Azure needed nothing: azure_rm defaults to the VM name,
which is per instance and is what its cloud-init uses for node_id.

The assertions go in tests/preflight-static, which exists for strings
one layer produces and another consumes strictly. Nothing else could see
this: terraform validate reads one file, the mocks read the
configuration's shape, and tests/ansible checks the inventory parses as
YAML.

They pin which values distinguish an instance rather than forbidding
tag:Name, because tag:Name is not the only constant an ASG propagates --
VaultCluster is another and it is in that same file. Verified by
mutating to tag:VaultCluster, the neighbouring change rather than the
named one, plus five others: tag:Name restored, private-ip-address
(unique but not the node_id, which fails the agreement half only), the
hostnames list deleted, and each side of the node_id derivation broken
on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sethbergman
sethbergman merged commit 98065ec into main Sep 16, 2026
38 checks passed
@sethbergman
sethbergman deleted the fix/aws-inventory-host-collision branch September 16, 2026 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant