Repository navigation
fix(controller): investigate repeated listener exits on empty runner completion events #158
Description
Problem
Worker 01 repeatedly exits its scale-set listener on the log message job completed for unknown runner "". It is not reliably serving CI. The same recovery incident includes exhausted Docker network pools and disk usage above the installer admission threshold.
Recorded evidence
The operator's authorized recovery thread recorded these observations on 2026-10-29 at approximately 17:55 to 18:10 UTC:
- Both hosts still ran engine312, previously identified as
312d22d712d5a5e87bdb14796bf2c40f33d53f92. - Worker 01 used 84% of its Docker filesystem; installation requires below 80%.
- Controller logs reported Docker network-pool exhaustion and repeated listener exits on
job completed for unknown runner "". - The installed daily cleanup does not reclaim images or build cache, and network cleanup defers while workloads remain running.
- Active jobs, uncertain resources, volumes, and recovery images have been preserved. These recorded facts are not a fresh host measurement made when creating this issue.
The exact source of the empty runner identity and its relationship to retention or network exhaustion remain unproven. Do not assume these symptoms share one cause.
Scope and related issues
This issue owns diagnosis and repair of the empty-identity completion event and repeated listener exit/recovery behavior.
- feat(maintenance): bound Docker image and build-cache growth before upgrades are blocked #111 already owns bounded image/build-cache retention and upgrade disk headroom.
- feat: Add Docker default-address-pools policy, subnet-headroom health gate, and controller circuit breaker for managed fleet lifecycle #81 already owns subnet headroom and abandoned-resource reconciliation.
- Investigate Docker build and image-operation stalls exceeding consumer deadlines #139 owns Docker build/image-operation latency.
- Detect live runner-group admission drift before reporting consumer readiness #142 owns runner-group admission observations.
- PR fix(fleet): unblock image validation and safe upgrade preparation #157 fixes a separate validation image-test client expiry and is not evidence that this listener defect is fixed.
Acceptance
- Trace the recorded completion event through the installed listener/controller code and safe timestamped logs. Determine whether upstream event data, local identity tracking, or lifecycle ordering causes the empty identity.
- Add a focused regression for the established cause and preserve duplicate-event handling, job ownership, capacity accounting, and reconciliation.
- Prevent a malformed, late, or duplicate completion event from causing repeated listener failure or loss of trustworthy job state. Unknown work must remain explicitly unknown until reconciled.
- Verify representative jobs can start and complete on worker 01, the listener remains available afterward, and runner cleanup and later reconciliation succeed.
- Record the installed engine and configuration identities and link operational recovery evidence for feat(maintenance): bound Docker image and build-cache growth before upgrades are blocked #111/feat: Add Docker default-address-pools policy, subnet-headroom health gate, and controller circuit breaker for managed fleet lifecycle #81 separately. A running process or passing fixture suite alone does not prove a recovered host.
- Update the affected controller/recovery documentation.
The owner has separately authorized the existing recovery thread to SSH into worker 01 and clean up its disk. This issue does not authorize deleting active or uncertain resources, volumes, or recovery images, and does not expand GitHub access or runner capacity.
Activity
Nickfost commentedon Oct 29, 2026
Authorized recovery investigation, 2026-10-29 UTC.
Fresh host reads at 20:14:48 found installed runtime
312d22d712d5a5e87bdb14796bf2c40f33d53f92, controller restart count 191 and zero managed runner containers. A separate read at 20:31:57 retained repeatedjob completed for unknown runner ""exits, including one immediately after session initialization at 20:31:24. These are host observations, not fixture results.The installed source has no special handling for an unassigned completion.
HandleJobCompletedlooks up the supplied runner name, fails the owned-prior-runner check for an empty name, and returns the unknown-runner error. That error escapeslistener.Run; main returns a listener error and exits with status 1. Docker's existing restart policy restarts the controller. This establishes the exit mechanism, but the old log does not retain the completion's result, runner ID, assignment time, job ID and request ID, so it cannot establish which upstream empty-name event was received.Merged PR #148, commit
a6fd61c1cc4877dd903abef2877dda957b974007, already adds a narrowly validated unassigned-cancellation case and logs those diagnostic fields. It is present in the intended engine ancestry but absent from the installed runtime. Its regression checks that a valid cancellation does not interrupt the rest of an SDK batch or change unrelated busy-runner accounting; contradictory blank events still fail. PR #157's validation-client correction is a separate change and is not live-host proof of this repair.Next evidence is the actual completion diagnostics and representative jobs after the reviewed engine rollout, followed by runner cleanup and later reconciliation. The source guard alone does not close this issue. Disk recovery is being recorded under #111 and subnet recovery under #81; no common cause for all three symptoms is asserted. Consumer tests and configured runner capacity have not been changed.
Nickfost commentedon Oct 29, 2026
Independent installed/current-source comparison, 2026-10-29 UTC.
I compared installed-source identity 312d22d with current main 48f5d47 and pinned actions/scaleset v0.4.0 source. This confirms the recovery owner's exit-path diagnosis. Engine312 treats a blank completion name as an unknown runner; the handler error exits listener.Run, then main exits without completing or shutting down surviving runners. The existing restart policy can repeat the failure. The SDK implementation acknowledges a batch before invoking completion callbacks, so a callback failure can also prevent processing later events in that already acknowledged batch.
The SDK README documents canceled assignments that were not acquired by a runner, followed by job requeue. That explains one valid class of empty-identity completion. It does not establish the incident event's fields. The old host log omits result, runner ID, assignment time and job/request identities, so the original completion remains unclassified. No common cause with disk or network exhaustion is inferred.
Merged PR148, a6fd61c, already handles only an explicit canceled result with blank name, zero runner ID, unset assignment time, nonempty job ID and positive request ID. It changes no managed-runner accounting and preserves the rest of the SDK batch in that valid case. Contradictory blank or unrelated unknown events still fail; they are not treated as completed work. I have not duplicated or broadened that behavior without evidence.
Candidate 94dedf9 adds operational diagnosis guidance to CONTROLLER-DESIGN.md while addressing separate cleanup gaps under #81/#111. Full source verification, review and merge remain pending. The recovery owner retains PR157 validation and host rollout. After the supported engine change, acceptance still needs actual completion diagnostics, host-attributed representative starts and completions, sustained listener availability, runner cleanup and later reconciliation. A passing fixture or PR157's separate validation-client repair cannot establish those outcomes. No host mutation or listener restart was performed in this implementation thread.
fix(maintenance): keep storage failures out of network recovery
Nickfost commentedon Oct 29, 2026
Fresh recovery observation, 2026-10-29 22:02UTC. The affected worker remains on engine312. After scoped disk/network recovery, its listener continued processing jobs. RestartCount stayed204 from21:23 through22:02UTC; the retained last700 log lines contained46 runner starts,44 job starts and46 job completions, with no ERROR rows. Representative successful jobs were separately joined to GitHub runner attribution. This is a bounded recovery interval, not proof the repeated-exit defect is fixed.
Current main48f5d47 already contains merged PR148's narrow unassigned-cancellation handling, which engine312 lacks. The prepared branch adds diagnosis guidance but no new completion-accounting change. Preserve the distinction between storage admission, subnet exhaustion and empty-runner callback failure. After PR157 passes and the merged engine is deployed, acceptance still needs actual completion fields, stable listener behavior, runner teardown, reconciliation and recovery evidence. Full validation run40962536216 is now executing on the supplied self-hosted VM; neither host has been upgraded by this update.
Nickfost commentedon Oct 30, 2026
Listener-status evidence correction for later comparisons:01's controller was recreated at2026-10-30T18:48:35Z by protected checkpoint restoration after interrupted incumbent maintenance. At19:06Z it was running on the original312 image with RestartCount0 because it is a new container. That reset is not listener-fix or engine-upgrade evidence. Both hosts still select/run312. 02's unchanged controller had RestartCount378 and last start18:39:20Z at the same read. Those exits still require their completion-event classification; I am not assuming the symptoms share a cause. No840 runtime has been deployed. The maintenance recovery preserved the nine legacy workloads and all volumes; its disk/cache evidence is recorded separately under#111.