Repository navigation
tests: run-engine and session tests read the box's real load and time out on a busy machine #1525
Description
What happened
Several run-engine and session tests start real run seats, which pass through the machine gate (task.max_load, default 1.5 per core). On a busy box they wait machine busy · load per core at or above task.max_load 1.5, then time out. They are green in CI and alone, and red when another suite is running beside them. CLAUDE.md treats that as a bug, not a known flake.
Seen on 2026-09-25 on an 8-core box at load 11–24:
internal/sessionTestUnderOneModelEveryRunSeatRidesTheConversationsModel/the_launch_model:timed out waiting for the run to endondev@691ade547at load 12, and 11 s alone. Added in task room, landing and --one-model: real file names, one note sentence, every seat on one model, no build caches #1517.internal/runTestSupervisorLaunchesEveryReadyLeafAtOnceWithNoSlotBoundandTestAnUnboundedRunEndedEarlyStillDrainsEveryWorker: fail even alone at load about 11.- Also timing out only under load:
TestPlandbCliWorkerOwnDone,TestARunCorrectedPastItsRoundLimitStopsWithoutMergingOrSayingDone,TestBeltRunCarriesMachineGateAndShowsItsHold,TestPlandbCliLoopThroughBash,TestACarriedOverDesignRaisesItsCardAndSavesOnYes,TestTheDelegationDoorsCrossFromTheEnginesProfile(session);TestHostedTasksToolStopEndsTheRunAndReportsWhatHappened(remote).
Replication
Deterministic:
stress-ng --cpu $(( $(nproc) * 2 )) --timeout 600s & # or any load above 1.5 per core
go test -count=1 -run 'TestUnderOneModelEveryRunSeatRidesTheConversationsModel' ./internal/session
go test -count=1 -run 'TestSupervisorLaunchesEveryReadyLeafAtOnceWithNoSlotBound' ./internal/runBoth fail today. Without the load, both pass.
The fix
A test that is not about the machine gate runs with the gate out of its way, through its fixture profile or an injected load reader, never the box's real /proc/loadavg. A test that IS about the gate injects the load it needs.
Acceptance
- Unit: each named test passes with
stress-ngholding the box above 1.5 per core. - A law test, or a single fixture helper, so that a new run-engine test cannot read the real load by accident.
Activity
santoshkumarradha commentedon Sep 28, 2026
Confirmed another instance of this issue in the three
cmd/codeafrun-engine model-seat tests at635723f03.beltRunEnvdisables the machine load/memory gates in the fixture profile, but each test replaced the profile JSON with model-only rows, restoringtask.max_load=1.5. The original combined run reached its 60-second deadline without starting any provider call.A deterministic assertion immediately after that setup reproduces all three failures (
got "1.5", want 0) without synthetic host load. The bounded test-only correction applies model rows through the existing settings API, preserving the fixture limits and asserting both remain zero. Original model/exit/output assertions and timeouts are unchanged.The three complete model-seat workflows and
TestDoRunHonoursProfileMachineFlooreach passed three repetitions (18.241 seconds total). The latter retains coverage that an explicitly configured real machine gate still blocks. This correction is pending aggregate integration; it does not claim the other tests listed in this issue are resolved.AbirAbbas commentedon Sep 29, 2026
More evidence from 2026-09-29 (dev
461a43fb6, an 8-core WSL laptop with nothing else running):make test-report PKGS=./internal/sessionon an untouched checkout failedTestARefutedNodesRepairRoundRunsOnTheCarefulModel,TestAuditVerifiesAChangeThatPassesItsTest,TestRefutedWorkIsRepairedInPlaceAndLandsWhenItHolds; threemake pr-readyruns of a branch that only changes one prompt paragraph failed a different set each time —task N never finishedafter 30–130 s (TestADoneLandingWithARewrittenReportStillDeliversItsAnswer,TestAnOrdinaryQuestionLeavesTheAssignmentExactlyWhereItWas,TestTickingAnItemMovesTheRowToTheNextItem,TestThresholdsStopANodeThatIsNotGettingAnywhere,TestAStalledCheckIsAbandonedAndTheSecondCallAnswers, …). The package's own eight shards drive the load average to ~8, and the shards run 244–291 s against ~160 s when quieter. Every named test passes alone, repeatedly. CI'stouched packagespassed the same branch (#1682). So on a laptop the session suite is not a usable local merge gate today, even with no other suite beside it.