Skip to content

test: add airgap.daily build tag for daily airgap CI#770

Open
floatingman wants to merge 1 commit into
rancher:mainfrom
floatingman:feature/airgap-daily-build-tag
Open

test: add airgap.daily build tag for daily airgap CI#770
floatingman wants to merge 1 commit into
rancher:mainfrom
floatingman:feature/airgap-daily-build-tag

Conversation

@floatingman

Copy link
Copy Markdown
Contributor

Summary

Introduces a new `airgap.daily` build tag that defines a curated set of `pit.daily` suites to run daily against an airgap cluster. Each suite was validated against a live airgap RKE2 cluster (Rancher v2.14.2, custom ansible-provisioned nodes) — suites either passed and were tagged `|| airgap.daily`, or were verified unsuitable for airgap and excluded with `&& !airgap.daily`.

The tag doubles as both the CI gate and a visual audit trail of which suites work in which environment.

Matrix (all 15 `pit.daily` suites)

Suite Verdict Change
`validation/workloads` ✅ 13/13 subtests pass `
`networking/connectivity` (×2) ❌ requires node-driver SSH (custom airgap nodes carry none; no external IPs) `&& !airgap.daily`
`fleet/public_gitrepo` ❌ clones public GitHub (no egress) `&& !airgap.daily`
`fleet/upgrade` ❌ node SSH + public GitHub `&& !airgap.daily`
`upgrade/workload` ❌ upgrade-flow test (needs real cluster upgrade between pre/post phases) `&& !airgap.daily`
`charts/{alerting,cis,istio,logging,monitoring,neuvector}`, `charts/appco/istio`, `longhorn/{longhorn,chartinstall}` ❌ blocked by catalog-apps watch instability + external-node-IP deps (see follow-ups) `&& !airgap.daily`

Verification

  • `-tags=airgap.daily` → only `TestWorkloadTestSuite` compiles.
  • `-tags=pit.daily` → all 15 target suites still compile/run (nothing lost).
  • 15 files changed, one build-constraint line each.

Run the daily airgap job

```bash
CATTLE_TEST_CONFIG=$(pwd)/cattle-config.yaml go test -tags=airgap.daily ./validation/workloads/...
```
(Scoped to the `workloads` package because two non-`pit.daily` files with negative-only build constraints also compile under `airgap.daily`.)

Confirmed airgap-environment constraints on this cluster

  • ✅ Image pulls work (docker.io mirror in front)
  • ✅ Chart installs + ingress work
  • ❌ No public-GitHub egress
  • ❌ Custom ansible nodes carry no node-driver SSH credentials and no external IPs

Follow-ups (to re-enable the chart/longhorn suites)

  • rancher/rancher: catalog app watch (`catalog.cattle.io.apps`) intermittently resets with HTTP/2 `INTERNAL_ERROR` / returns a collection instead of a stream (reproduced on CI). Root cause of the chart-suite flakiness.
  • rancher/tests: normalize watch-retry resilience across chart install helpers (monitoring retries, alerting does not); monitoring test's external-node-IP dependency (`monitoring_test.go` webhook NodePort); monitoring teardown bug (empty pod name in log streaming).

`cattle-config.yaml` is intentionally not committed (contains credentials).

Introduce a new `airgap.daily` build tag to define a curated set of
pit.daily suites that run daily against an airgap cluster. Each suite
was validated against a live airgap RKE2 cluster (Rancher v2.14.2,
custom ansible-provisioned nodes).

Included (cleared, || airgap.daily):
- validation/workloads        TestWorkloadTestSuite (13/13 pass)

Excluded (&& !airgap.daily) — verified unsuitable for airgap:
- networking/connectivity     requires node-driver SSH (custom airgap
                              nodes carry none; no external IPs)
- fleet/public_gitrepo        clones public GitHub (no egress)
- fleet/upgrade               node SSH + public GitHub
- upgrade/workload            upgrade-flow test (needs real cluster
                              upgrade between pre/post phases)
- charts/{alerting,cis,istio,logging,monitoring,neuvector},
  charts/appco/istio, longhorn/{longhorn,chartinstall}
                              blocked by catalog-apps watch instability
                              (reproduced on CI) + external-node-IP deps

Under -tags=airgap.daily only TestWorkloadTestSuite compiles; under
-tags=pit.daily all 15 suites remain intact.
@floatingman
floatingman requested a review from a team as a code owner July 17, 2026 21:38
@floatingman floatingman self-assigned this Jul 17, 2026
@floatingman floatingman added the team/pit-crew slack notifier for pit crew label Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team/pit-crew slack notifier for pit crew

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant