fix(systemtest): the Hetzner lane picks its location at run time (ankra-ews9qk)

[Myothas]

What & why

This is a follow-up to #510. The live run 38573723784, dispatched right after #510 merged, showed two things:

• The new reporting works. Both lanes now say exactly why they failed.
• The fsn1 default was itself refused: Server type 'cpx32' is not available in location 'fsn1'.

Hetzner's shared-vCPU stock moves by the minute. On 2026-10-14 cpx32 left nbg1, then fsn1, then came back to nbg1, all within the hour. No fixed location keeps the lane creatable.

So the lane now picks its location at run time:

• Default: HETZNER_LOCATION defaults to empty.
• Candidates: the lane takes the first of HETZNER_LOCATION_CANDIDATES (fsn1 nbg1 hel1) that offers every type it uses right now: control plane, worker, bastion and the resize type. It reads them from ankra cluster hetzner server-types --available-only with the lane's own credential.
• Nothing fits: the lane fails with "no Hetzner location in '' offers right now".
• Pinning: setting HETZNER_LOCATION still pins it.

The lane keeps the cheap shared-vCPU types (cpx22 / cpx32) instead of moving to dedicated ccx.

Test plan

• TestHetznerLanePicksALocationThatOffersItsTypesNow covers:
  • the empty default;
  • hetzner_types_from_table on the CLI's table: header skipped, unavailable rows dropped;
  • pick_hetzner_location with stubbed availability: skips a location missing a type, keeps the candidate order, fails when none fits.
• Run live against Hetzner just now: fsn1 lacked cpx32, and the picker chose nbg1.
• The pre-commit gate passed: full go test ./... and golangci-lint run with 0 issues.
• A lifecycle run will be dispatched after merge.

Docs checklist

• No user-facing command/flag changes — no docs needed. This changes only the system test.
• Command/flag changes — the CLI reference regenerates automatically on release (tools/gendocs); ensure Short/Long/Example on new commands are written for docs readers
• Broader behaviour changes — docs updated in ankra-docs (link the PR):

Bead: ankra-ews9qk

🤖 Generated with Claude Code

[ankra-platform]

Ankra AI review

Verdict: looks good.

The lane now probes each candidate location at run time and, importantly, distinguishes 'read and lacking' from 'could not read' - an unreadable location is never treated as a negative answer, and the all-unreadable case is reported as a credential/API problem rather than a stock problem. The Go test exercises both the table parser and the picker against a stubbed reader, including the failure-wording cases. Only a minor note about the picked location being reused for the rest of the run.

Findings

• [low] Picked location is pinned for the rest of the run (systemtest/lifecycle_systemtest.sh:1965)
  Assigning HETZNER_LOCATION here mutates the global, so any later hetzner lane in the same script run reuses this pick without re-probing. That is probably desirable (a mid-run location switch would be surprising), but if the intent was per-lane freshness, the assignment should be scoped or re-picked per lane. Worth a one-line comment either way.

---

Reviewed commit f20d958. This review is read-only and advisory.

[ankra-platform]

[high] Failed server-types read is indistinguishable from 'location offers nothing'

available="$(hetzner_available_types "$location" || true)" swallows any failure of the CLI call (bad credential, network error, changed table format that makes awk print nothing). An unobserved location then looks exactly like an observed location missing a type, and if every read fails the lane dies with 'no Hetzner location ... offers ... right now', which is UNKNOWN being reported as a negative answer. Capture the exit status separately: on read failure, log it and either fail immediately with a 'could not list server types' message or skip the candidate while recording that it was never observed.

_{Ankra AI review. Read-only and advisory.}

[ankra-platform]

[medium] awk parser assumes a fixed two-line preamble in the table output

NR > 2 skips exactly two lines, assuming the CLI always prints the 'Server types for location X:' banner plus a header row. The diff does not show ank_out's output format; if the banner is omitted (or a second header/wrapping line appears), the parser silently drops a real data row or keeps a header, again producing an empty/wrong answer that looks like 'type not offered'. Match on the row shape instead (e.g. require $NF == "yes" and $1 to look like a type name) rather than counting lines.

_{Ankra AI review. Read-only and advisory.}

[ankra-platform]

[low] Empty HETZNER_*_TYPE variables silently make every location incomplete

If any of HETZNER_CP_TYPE / HETZNER_WORKER_TYPE / HETZNER_BASTION_TYPE / HETZNER_BIGGER_TYPE is unset or empty, grep -qx "$needed" never matches and every candidate is rejected, surfacing as the misleading 'no location offers the types' failure. The diff does not show where these are defaulted; if they can be empty, validate them up front and fail with a configuration error instead.

_{Ankra AI review. Read-only and advisory.}

[Myothas]

Review addressed in the latest commit: [high] a failed or empty server-types read now marks the location unread, never lacking; the failure says 'could not read: ' alongside 'lacking', and 'could not list Hetzner server types for any of ...' when nothing was readable (credential/API, not stock); hetzner_available_types returns the CLI's own failure. [medium] the parser matches the row shape (type-like name first, yes last), tested with and without the banner and header. [low] empty HETZNER_*_TYPE is a configuration error. All pinned in TestHetznerLanePicksALocationThatOffersItsTypesNow.

[ankra-platform]

[low] Picked location is pinned for the rest of the run

Assigning HETZNER_LOCATION here mutates the global, so any later hetzner lane in the same script run reuses this pick without re-probing. That is probably desirable (a mid-run location switch would be surprising), but if the intent was per-lane freshness, the assignment should be scoped or re-picked per lane. Worth a one-line comment either way.

_{Ankra AI review. Read-only and advisory.}