Reach the AWS nodes by tunnelling SSH through Session Manager

[sethbergman]

Replaces #92, which GitHub auto-closed when its base branch
(fix/aws-inventory-host-collision, now merged as #91) was deleted. A
closed PR whose base is gone cannot be reopened or retargeted. Same head
branch, same commits, plus the ansible-lint fix below.

Second of the three blockers found while staging the first real AWS
apply.

The gap

The nodes sit in private subnets with no public address, and
security.tf opens port 22 from nowhere. There was nothing outside the
VPC to SSH to. docs/deployment.md said reaching them "needs SSM, a
bastion, or a VPN" and the repository shipped none of the three —
ansible.cfg is two lines and sets no connection arguments. The
documented apply sequence ends in cd ansible && ansible-playbook, which
could not have worked.

Why Session Manager

It changes neither fact about the network: the agent holds an outbound
connection to the SSM service, and AWS-StartSSHSession carries an
ordinary SSH session down it. No inbound rule, no public address, no
bastion to patch and pay for, and the session is recorded against the
caller's IAM identity rather than against whoever holds a key.

It also needs no new Terraform — iam.tf already attaches
AmazonSSMManagedInstanceCore.

Two consequences in the inventory:

• ansible_host is the instance id. --target names an instance to
  SSM, and ProxyCommand's %h is whatever ansible_host holds. The
  private IP is not routable from where the playbook runs, so it would
  only look correct.
• ansible_user is set at all, for the first time. Nothing defined
  it, so Ansible was connecting as the local username.

Host keys are accept-new, not no. A host here is an instance id, so a
replaced node is a new name with a new key and is correctly unknown,
while a changed key under an id already seen is the case worth refusing.
Still trust-on-first-use, and the doc says what verifying properly would
take rather than implying this is it.

ssh_key_name said something untrue

Its description claimed that leaving it empty "disables SSH entirely and
uses SSM Session Manager instead". That reads as a supported arrangement
and was not one: SSM replaces the network path, not the authentication,
and what answers at the far end of the tunnel is still sshd reading
authorized_keys.

Corrected rather than made required — aws ssm start-session with no
document does give a shell, which is the one case where empty is
deliberate. It is not enough for Ansible, and the description now says so.

Testing it needed a new shape

compose values are Jinja expressions. A literal written bare —
ansible_user: ec2-user — is an undefined variable, which composes to
nothing rather than erroring: Ansible drops the setting silently, and
the file is valid YAML either way, which is all tests/ansible checked.

eval-compose.py renders them as aws_ec2 would, against a synthetic
instance, and the suite asserts the results. Suite 71 → 80.

Mutation	What goes red
ansible_user: ec2-user, unquoted	every compose assertion — the render raises, which is the point
ansible_host: private_ip_address	the SSM target assertion alone
drop --document-name AWS-StartSSHSession	the SSH-document assertion alone
StrictHostKeyChecking=no	both host-key assertions
UserKnownHostsFile=/dev/null added	the exclusion assertion alone

One is worth repeating. Under the unquoted-literal mutation the
exclusion assertion — that nothing throws the host-key check away —
stayed green, because a render that raised left no string to search.
Its paired positive assertion went red, and that pairing is the only
reason the mutation was caught. An exclusion with nothing to exclude
passes.

Pre-flight

preflight-cloud.sh now checks for session-manager-plugin. The AWS CLI
execs it, and without it every connection fails naming the plugin rather
than the thing you were doing. A warning, not a failure — running the
playbook from inside the VPC is legitimate, which matches how the
existing ssh_key_name check is graded. Suite 43 → 48, verified by
removing the --cloud gate and by turning the warning into a failure.

Since #92

ansible-lint's yaml[line-length] caps a line at 160 and the
expression was 183. Jinja concatenates adjacent string literals the way
Python does, so it folds across three lines under >-, with every space
that must survive kept inside the quotes. Verified by rendering: the
value is byte-identical to the single-line form.

The DR restore drill also failed on #92, on a Docker Hub 502 pulling
hashicorp/vault:1.17.2. Upstream and unrelated.

What this does not do

Azure is deliberately untouched. Its nodes are equally private, but
adding an untested tunnel to a profile that has never been applied would
make the gap harder to see rather than smaller.

None of this has run against a real account. The tests assert the values
it produces, not that the tunnel opens.

🤖 Generated with Claude Code