From 1a316e3409eed964a50783ace1efd666dac99e94 Mon Sep 17 00:00:00 2001 From: Seth Bergman Date: Tue, 15 Sep 2026 21:53:32 -0500 Subject: [PATCH] Record what staging the AWS apply found before it happened Three defects, none of which any test here could have caught, all found by writing out the sequence someone would run and asking of each step what it needed that nothing provided. They belong in "What 'done' means here" because they are a different answer to the same question as the five entries above it. Those are about a test that existed and was weaker than it looked, or a claim nobody checked. These are about code no test could reach: the cloud profiles have never been applied, so a whole layer of the repository has only ever been read. The entry keeps three things that are easy to lose. tests/preflight-static exists for exactly the seam the first defect sat in, and missed it -- the suite was written around the two strings someone had thought of, and host naming is the same shape and was not one of them. A suite aimed at a class of defect still only covers the instances somebody enumerated. The emulated apply could not have seen it either, because moto has no Ansible in it. The roadmap already says a profile that applies is not a cluster that works; this is that sentence with a worked example. And three known defects becoming zero known defects is a statement about what has been looked at, not about what is there. The entry says so, because the alternative reading -- that the profile is now closer to proven -- is exactly the overstatement this file exists to refuse. Co-Authored-By: Claude Opus 5 --- docs/roadmap.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index ff3fe2b..57d3ca5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -589,5 +589,63 @@ the fix is different. The table above was right the whole time; what drifted was a hand-written summary of it. A weak assertion gets widened. An unchecked claim gets tied to whatever is already correct. +A sixth came from none of the above. Nothing was asserted weakly and +nothing had drifted; three defects were found by writing down the +sequence someone would actually run, in order, and asking of each step +what it needed that nothing provided. + +The occasion was staging the first real AWS apply — the preparation for +blocker 1, which is meant to be so complete that the session spends its +money on questions only a real account can answer. All the preparation +described above already existed: `preflight-cloud.sh`, `teardown-cloud.sh` +and a ten-item verification checklist in +[cloud-apply.md](cloud-apply.md). None of it caught these, because all of +it is about what to check once a cluster is up. None of it asks whether +the path to getting there is continuous. + +| What | Why nothing saw it | +|---|---| +| An autoscaling group tags every instance identically, and `inventory/aws.yml` preferred `tag:Name`. An Ansible inventory is keyed by host name, so three nodes collapsed into one and `site.yml` configured a single node and exited 0 | Terraform sets the tags and is tested; the inventory filters on them and was tested as valid YAML. Neither asks whether the inventory can tell two instances apart | +| Nothing could reach the nodes at all. Private subnets, no inbound 22, and the documented sequence ends in `ansible-playbook`. `docs/deployment.md` said reaching them "needs SSM, a bastion, or a VPN" and the repository shipped none of the three. `ansible_user` was never set either | Both sides were tested. `security.tf` was tested for what it refuses, the playbook for what it renders, and nothing tested that one could reach the other | +| The vault role verified a delivered certificate with `openssl -checkhost` against an IP address, which reports "does NOT match" for every certificate this repository issues — `issue-node-cert.sh` and the `vault_pki` role both put the address in `--ip-sans`. A correct certificate failed; only a wrong one passed | It had never executed. The local profile is Docker Compose and does not use this role, and neither cloud profile has been applied | + +The first is the one worth dwelling on, because +[`tests/preflight-static`](../tests/preflight-static/run-tests.sh) exists +for precisely this seam and did not catch it. That suite was written +around two strings produced by one layer and consumed strictly by +another — an `auto_join` selector and a `leader_tls_servername` — and it +checks those two. Host naming is the same shape and nobody had thought of +it. A suite aimed at a class of defect still only covers the instances +someone wrote down. + +It was also invisible to `tests/cloud-apply-emulated`, which applies the +whole AWS profile against a real implementation of the AWS API. The +emulator has no Ansible in it. That is the distinction that section +already draws, arriving as a concrete example: a profile that applies is +not a cluster that configures. + +All three now have assertions, and the second and third needed code +before they could have any — `ansible/inventory/aws.yml` tunnels SSH +through Session Manager, and `scripts/generate-cloud-certs.sh` issues the +bootstrap material after the apply, because the filenames follow +instance ids that do not exist until then. + +Two things this does not mean. It does not mean the profile works: three +known defects became zero known defects, which is a statement about what +has been looked at and not about what is there. And it does not mean the +apply is cheaper than the blocker list claims — every item on that list +is still a question about runtime behaviour, and none of these three was. +What it means is that the session will now fail on those questions rather +than on a missing SSH key, which is the entire purpose of the +preparation. + +The cost asymmetry is the part worth keeping. Finding these took an +afternoon and no money. Two of the three would have surfaced during the +apply disguised as something else: a cluster that came up healthy and was +one-third configured, and a Raft join failure that reads like a network +problem. The question that found them is a variant of the one above, +asked of a procedure instead of an assertion: **what does this step need +that nothing here provides?** + None of this changes what the table above claims. It changes how much the word "tested" in it is worth, which seemed worth writing down.