Reach the AWS nodes by tunnelling SSH through Session Manager

[sethbergman]

Stacked on #91 — it depends on ansible_host being the instance id.
Base retargets to main when that merges.

Second of the three blockers found while staging the first real AWS
apply.

The gap

The nodes sit in private subnets with no public address, and
security.tf opens port 22 from nowhere. There was nothing outside the
VPC to SSH to. docs/deployment.md said reaching them "needs SSM, a
bastion, or a VPN" and the repository shipped none of the three —
ansible.cfg is two lines and sets no connection arguments. The
documented apply sequence ends in cd ansible && ansible-playbook,
which could not have worked.

Why Session Manager

It changes neither fact about the network: the agent holds an outbound
connection to the SSM service, and AWS-StartSSHSession carries an
ordinary SSH session down it. No inbound rule, no public address, no
bastion to patch and pay for, and the session is recorded against the
caller's IAM identity rather than against whoever holds a key.

It also needs no new Terraform — iam.tf already attaches
AmazonSSMManagedInstanceCore.

Two consequences in the inventory:

• ansible_host is the instance id. --target names an instance to
  SSM, and ProxyCommand's %h is whatever ansible_host holds. The
  private IP is not routable from where the playbook runs, so it would
  only look correct.
• ansible_user is set at all, for the first time. Nothing defined
  it, so Ansible was connecting as the local username.

Host keys are accept-new, not no. A host here is an instance id, so a
replaced node is a new name with a new key and is correctly unknown,
while a changed key under an id already seen is the case worth refusing.
It is still trust-on-first-use, and the doc says what verifying properly
would take rather than implying this is it.

ssh_key_name said something untrue

Its description claimed that leaving it empty "disables SSH entirely and
uses SSM Session Manager instead". That reads as a supported arrangement
and was not one: SSM replaces the network path, not the authentication,
and what answers at the far end of the tunnel is still sshd reading
authorized_keys.

Corrected rather than made required — aws ssm start-session with no
document does give a shell, which is the one case where empty is
deliberate. It is not enough for Ansible, and the description now says
so.

Testing it needed a new shape

compose values are Jinja expressions. A literal written bare —
ansible_user: ec2-user — is an undefined variable, which composes to
nothing rather than erroring: Ansible drops the setting silently, and
the file is valid YAML either way, which is all tests/ansible checked.

eval-compose.py renders them as aws_ec2 would, against a synthetic
instance, and the suite asserts the results. Suite goes 71 → 80.

Mutation	What goes red
ansible_user: ec2-user, unquoted	every compose assertion — the render raises, which is the point
ansible_host: private_ip_address	the SSM target assertion alone
drop --document-name AWS-StartSSHSession	the SSH-document assertion alone
StrictHostKeyChecking=no	both host-key assertions
UserKnownHostsFile=/dev/null added	the exclusion assertion alone

One is worth repeating. Under the unquoted-literal mutation the
exclusion assertion — that nothing throws the host-key check away —
stayed green, because a render that raised left no string to search.
Its paired positive assertion went red, and that pairing is the only
reason the mutation was caught. An exclusion with nothing to exclude
passes.

Pre-flight

preflight-cloud.sh now checks for session-manager-plugin. The AWS CLI
execs it, and without it every connection fails naming the plugin rather
than the thing you were doing. A warning, not a failure — running the
playbook from inside the VPC is legitimate and needs none of this, which
matches how the existing ssh_key_name check is graded. Suite goes 43 →
48, verified by removing the --cloud gate (Azure's assertion catches
it) and by turning the warning into a failure.

What this does not do

Azure is deliberately untouched. Its nodes are equally private, but
adding an untested tunnel to a profile that has never been applied would
make the gap harder to see rather than smaller.

And none of this has run against a real account. It is configuration with
reasoning attached; the tests assert the values it produces, not that the
tunnel opens.

🤖 Generated with Claude Code