chapel: Wave 3 — cross-node gasnet/ofi over real NIC (cluster runner required)

[hyperpolymath]

Context

PR #85 landed the Wave 1 chapel-ci scaffolding (six strict gates,
detachable-harness ADR, four silent-loss fixes in Imaging.chpl,
`--resume` hard-fail in static mode). Wave 1 chapel-e2e exercises
`mass-panic --numLocales=1` against a synthetic 2-repo corpus.

True multi-locale execution (`-nl 2` and higher) requires Chapel
built with `CHPL_COMM=gasnet` (or `ofi` / `ucx`). The stock
`chapel-2.8.0-1.ubuntu22.amd64.deb` ships with `CHPL_COMM=none`
and rejects `-nl >1` with:

```
error: Only 1 locale may be used for CHPL_COMM layer 'none'
```

Wave 2 scope

1. CI install path for multilocale Chapel — either:

   • A `chapel-multilocale-2.8.0` .deb if upstream publishes one, or
   • A build step that rebuilds Chapel from source with
     `CHPL_COMM=gasnet` + `CHPL_LAUNCHER=…` flags
     (slow — caches required to keep CI under a few minutes), or
   • Self-hosted runners with multilocale Chapel preinstalled.

2. `chapel-multilocale` CI job — add to `.github/workflows/chapel-ci.yml`,
   bumping the gate count from 6 to 7. Exercises `-nl 2` against the
   same synthetic 2-repo corpus as `chapel-e2e` so the cross-locale
   code path runs even on a single GH runner (locales oversubscribed).
   Add to the Base ruleset's `required_status_checks` once green
   (ruleset id 14968583).

3. Non-trivial corpus benchmark — once a multilocale runner is
   available, exercise a ~50-repo cluster scan to verify the queue
   scheduler's resumability claims and produce numbers for the
   "~5–15% slower" assertion in `chapel/README.md` that's currently
   labelled "estimate pending Wave 2 benchmark".

Acceptance

• `chapel-multilocale` job green on PR + main ≥1 merge cycle
• Added to Base ruleset required_status_checks (gate count 7)
• README perf claim either backed by measured numbers or
  revised based on findings

References

• ADR: `docs/adr/0001-chapel-distributed-scanner.md` (Wave 2 deferred items list)
• Wave 1 PR: feat(chapel): wire MassPanic CI + cross-rust contract checks (refs #33) #85 (merged 2026-05-30)
• Ruleset closure pattern: see `feedback_ruleset_required_checks_close_in_same_cycle` in memory

[hyperpolymath]

Adaptive update 2026-06-01 — option 1 confirmed unavailable

Audited upstream chapel-lang/chapel release 2.8.0 (latest). Asset list:

```
chapel-2.8.0-1.{debian12,debian13,ubuntu22,ubuntu24}.{amd64,arm64}.deb
chapel-2.8.0-1.{el10,fc42,fc43}.{aarch64,x86_64}.rpm
chapel-2.8.0-1.{amzn2023}.{aarch64,x86_64}.rpm (chapel-minimal-* variant)
chapel-2.8.0-20260309195041_d334648.hpecrayex.{aarch64,x86_64}.rpm
chapel-2.8.0.tar.gz
```

No chapel-multilocale-* package exists in upstream releases. The only multilocale-capable artefacts are:

1. The HPE Cray EX RPMs (*.hpecrayex.*) — not deployable on standard GitHub runners.
2. The chapel-2.8.0.tar.gz source — requires the source-build path (option 2 in the issue body).

→ Option 1 in the issue body ("chapel-multilocale-2.8.0 .deb if upstream publishes one") is conclusively unavailable as of upstream 2.8.0 release.

Remaining viable paths are option 2 (source build with CHPL_COMM=gasnet + ccache, slow) or option 3 (self-hosted runner with multilocale Chapel preinstalled). Decision is yours; no estate-side precedent yet for either in CI.

[hyperpolymath]

Wave 2 status — MERGED

Wave 2 (single-host oversubscribed gasnet+smp via source-built Chapel) landed via #99 at 2026-06-02 04:27Z, after six cold-cache CI iterations debugging four Chapel-2.8.0 sharp edges:

1. MANPATH: unbound variable in setchplenv.bash under set -u — fixed by export MANPATH=${MANPATH:-} before sourcing.
2. CHPL_LLVM required for source build — fixed with export CHPL_LLVM=none (gnu C backend is enough for multilocale comms).
3. chpl --about removed in 2.8.0 — replaced with printchplenv.
4. printchplenv --simple doesn't emit clean KEY=value — switched to runtime-layout existence check (find -type d -name comm-gasnet).

A fifth edge surfaced in #100: cached CHPL_UNWIND=system vs cache-hit-inferred bundled mismatch — fixed by pinning CHPL_UNWIND=system explicitly + always-installing libunwind-dev + cache-gen v1→v2.

All four edges documented in the wiki at Chapel-Metalayer for the next chapel-from-source step to crib from.

Wave 3 status — still cluster-runner-gated

Wave 3 (cross-node gasnet/ofi over a real NIC) is not exercisable on default GH runners — needs a beefier or self-hosted runner. ROADMAP records this as a separate [ ] under the 'Multi-locale Chapel orchestration' parent item.

Keeping this issue open as the Wave 3 tracker.