Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .github/workflows/negctl-adhoc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,24 @@ jobs:
negctl-adhoc:
name: negctl adhoc — prove the named defects in one isolated job (real /dev/vms)
runs-on: ubuntu-latest
timeout-minutes: 60
# The adhoc path runs the FULL ~124-suite pristine control UNSHARDED, plus one
# full-suite boot per named defect. Under TCG contention a single 124-suite
# boot can take ~2700s (measured 89/124 in 1800s), so worst case ≈ 250s build
# + 2700 pristine + 2700 defect ≈ 94m. 100m gives margin. This does NOT touch
# the sharded CI negctl gate (kernel-executive-facility-negative-controls-shard,
# 22 shards @ 50m, NEGCTL_WALL unset ⇒ still 1800): that gate stays tight.
timeout-minutes: 100
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Inject each named defect and assert its exact red set against a real /dev/vms
env:
CONTAINER_ENGINE: docker
# Adhoc-only per-boot wall raise (vms-926 lineage): the full unsharded
# pristine + defect boots need >1800s under contention. Ceiling only —
# an uncontended boot still finishes in ~250s and never reaches it.
NEGCTL_WALL: "2700"
# Advisory emission only; not uploaded or compared (matches ci.yml, vms-49f).
FACILITY_NEGCTL_RECORD_OUT: ${{ runner.temp }}/facility_negctl_observed.tsv
run: |
Expand Down
25 changes: 21 additions & 4 deletions tests/qemu/run_facility_negctl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -250,10 +250,27 @@ run_harness() {
# (init.sh's tail: test_syssvc_* from ~procnam on + all test_imgact_*)
# never executes and reports rc=MISSING -- a mass, intermittent
# false-red that looks like a build/staging drop but is a timeout.
# Give the pristine boot the SAME 1800s budget the defect runs have.
"$ENGINE" run --rm -e KE_WALL_TIMEOUT=1800 "$BASE_TAG" >"$OUTFILE.raw" 2>&1
# Give the pristine boot the SAME budget the defect runs have. The wall
# is env-configurable via NEGCTL_WALL (default 1800, unchanged for the
# sharded CI negctl gate, which runs ~7 defects/shard under a 50m job
# budget and must stay tight). The ADHOC path (negctl-adhoc.yml) sets
# NEGCTL_WALL=2700 + a 100m job budget: it runs the FULL ~124-suite
# pristine UNSHARDED, and measured 89/124 suites in 1800s (~20s/suite)
# means 124 needs ~2510s -- so 1800 intermittently overruns under TCG
# contention and 2700 gives headroom. Raising a per-boot CEILING never
# slows an uncontended boot (which still finishes in ~250s and never
# reaches the wall); it only lets a starved boot complete instead of
# SIGTERMing the run-order tail into a false-red rc=MISSING mass.
"$ENGINE" run --rm -e KE_WALL_TIMEOUT="${NEGCTL_WALL:-1800}" "$BASE_TAG" >"$OUTFILE.raw" 2>&1
else
"$ENGINE" run --rm -e "FACILITY_DEFECT=$_defect" "$BASE_TAG" \
# Defect run: same env-configurable wall. inject_and_run.sh honours an
# inherited KE_WALL_TIMEOUT (its own default is also 1800), so passing it
# here keeps the sharded gate at 1800 (NEGCTL_WALL unset) while the adhoc
# gets 2700 -- pristine and each defect run boot the full suite set at the
# same cost, so both walls must move together or the flake just relocates
# from the pristine control to the first defect run.
"$ENGINE" run --rm -e "FACILITY_DEFECT=$_defect" \
-e KE_WALL_TIMEOUT="${NEGCTL_WALL:-1800}" "$BASE_TAG" \
/inject_and_run.sh >"$OUTFILE.raw" 2>&1
fi
_rc=$?
Expand Down Expand Up @@ -446,7 +463,7 @@ if [ "$BASE_RC" -eq 124 ]; then
# full suite finished, so the guest was SIGTERM'd mid-run and every suite
# AFTER the wall reports rc=MISSING below. Name it as a TIMEOUT, not a
# build/staging drop or a real red -- the mass rc=MISSING is the SYMPTOM.
bad "the PRISTINE boot hit the wall-clock TIMEOUT (rc=124) before completing -- the guest was killed mid-run; the rc=MISSING suites below are the run-order TAIL that never got to execute, NOT a build/staging drop. The pristine boot now gets KE_WALL_TIMEOUT=1800; if this still fires, the full ~$N_EXPECTED-suite boot needs a larger budget or the runner is badly starved."
bad "the PRISTINE boot hit the wall-clock TIMEOUT (rc=124) before completing -- the guest was killed mid-run; the rc=MISSING suites below are the run-order TAIL that never got to execute, NOT a build/staging drop. The pristine boot gets KE_WALL_TIMEOUT=${NEGCTL_WALL:-1800}s (raise NEGCTL_WALL, as the adhoc path does to 2700); if this still fires, the full ~$N_EXPECTED-suite boot needs a larger budget or the runner is badly starved."
fi
if [ "$BASE_RC" -ne 0 ]; then
bad "the PRISTINE harness exited $BASE_RC; every negative control below would be meaningless"
Expand Down
Loading