PS-11298: Add nightly x86_64 gcc-16 build on Hetzner

[nogueiraanderson]

Feature

• Add a nightly x86_64 gcc-16 build to builds.yml as sibling jobs beside the arm64 build: Hetzner cpx62 primary, AWS c7i.4xlarge + amd64 AMI fallback, gcc-16 via COMPILER_VER=16.

Why

• Azure Pipelines has no Ubuntu 26.04 and only an experimental gcc-16 beta. This gives early x86_64 coverage of stable gcc 16.1 on 26.04 and a safeguard if Azure degrades. Scoped to 8.4 and 9.7 (Ubuntu 26.04, where g++-16 is in the stock archive).

Notes

• Nightly-gated (workflow_dispatch + RelWithDebInfo). The arm64 build path and env defaults are unchanged. The only arm64 edit is a Name-tag filter on its EC2-teardown safety net so the two chains never reap each other when both fall back to AWS in one run.

Tickets

• PS-11298 (this), PS-11078 (parent), PS-11219 (arm64 fallback)
• Companion: cd-platform PR 255 (x86_64 added to the EC2 fallback IAM role)

[inikep]

Reviewed the x86_64 chain by diffing it against the arm64 chain with arch tokens normalized away. The arch-sensitive values all check out — RUNNER_SHA256 048024cd…d271 is the correct actions-runner-linux-x64-2.334.0.tar.gz digest, the SSM path is the amd64 one, cpx62 is a real CPX Gen2 type at 16 vCPU / 32 GB (genuine cax41 / c7g.4xlarge parity), and the Name-tag filter added to delete-runner-aws is safe because the tag is applied at RunInstances via --tag-specifications, so the "cancelled mid-launch" discovery case still works. Run 27974395052 is green end to end (cpx62 → gcc 16.0.1 → binlog_nogtid, 70/70, clean teardown).

Two things I'd like addressed:

1. orphan-sweep.yml doesn't know about ps-amd64-, so leaked x86_64 VMs are never reaped

The sweeper is prefix-scoped to the arm64 naming:

• .github/workflows/orphan-sweep.yml:58 — select(.name | test("^(ps-arm64-|cap-probe-)"))
• .github/workflows/orphan-sweep.yml:111 — select(.name | test("^ps-arm64-"))

A ps-amd64-<run>-<attempt> cpx62 that survives delete-runner-hetzner-x86_64 (cancelled run, hcloud API failure, job-level failure before teardown) will run indefinitely, and orphaned ps-amd64-* runner registrations are never unregistered. The new job's own debug summary asserts the opposite:

echo "orphan-sweep.yml will reap this VM after 6h regardless."

which is not true today for ps-amd64- names.

Two regexes fix it — ^(ps-arm64-|ps-amd64-|cap-probe-) and ^ps-(arm64|amd64)-. Please land that either in this PR or ahead of it, so there's no window where the nightly can leak an unreaped server.

Not affected: the capacity probes (cap-probe-x86-… still matches the cap-probe- prefix) and the EC2 sweep (tag-based, so iit-billing-tag + github_repository already covers x86_64 instances).

2. ~900 duplicated lines, and the two copies have already drifted

builds.yml is now ~2100 lines carrying two near-identical copies of pick-target, create-runner-hetzner, create-runner-aws (including the ~120-line userData heredoc), and both delete-runner-* jobs. Every future change — runner version + SHA bump, apt package list, backoff curve, cmake flags, metadata-options — has to be applied twice, and a miss on one side is invisible in review.

Drift is already present in this diff: the arm64 userData still says "Canonical's Ubuntu 24.04 ARM64 AMI" while the x86_64 copy says 26.04; the arm64 RunInstances error still hardcodes c7g.4xlarge while the copy uses $AWS_INSTANCE_TYPE_X86_64. Those are harmless today, but they're exactly the shape of the bug that won't be harmless next time (e.g. bumping RUNNER_VERSION in one chain and leaving the other pinned to a deprecated runner, which fails with the "cannot receive messages" mode the comment at line 566 already warns about).

Preferred fix is a workflow_call reusable workflow (or a matrix over arch) parameterized on: server-type list, instance type, AMI SSM path, runner tarball arch + SHA, runner-name prefix, COMPILER_VER, ccache key suffix, and the checkout ref expression. That collapses this PR to roughly the ~100 lines of real delta and makes the trust-split invariant reviewable in one place instead of two.

If splitting it out is too invasive right now, I'd at least want the duplication acknowledged as a follow-up ticket, plus a short "keep in sync with the arm64 chain" marker on each block that must be edited in pairs.

---

Also worth a quick check before merge: in both dispatch runs so far, arm64 pick-target failed at ~17m (force_provider=hetzner with arm64 capacity exhausted), so the arm64 chain never launched alongside the x86_64 one. The isolation deltas that motivate the arm64 edit — Name-tag scoping, distinct runner names, distinct ccache keys — haven't actually been exercised concurrently yet. One force_provider=aws + RelWithDebInfo dispatch would put both chains on EC2 in the same run and confirm they don't reap each other.