Skip to content

fix(controller): handle unassigned canceled completions - #148

Merged
Nickfost merged 1 commit into
mainfrom
t3/handle-unassigned-cancellation
Oct 24, 2026
Merged

Nickfost merged 1 commit into
mainfrom
t3/handle-unassigned-cancellation

Conversation

@Nickfost

Copy link
Copy Markdown
Member

The pinned SDK can report a canceled scale-set assignment before a runner acquires it. The controller currently rejects its empty runner name, which terminates listener processing after the SDK has acknowledged the batch and can prevent later callbacks from running.

Handle only an explicit canceled completion with no runner name, zero runner ID, unset runner-assignment time, a nonempty job ID and a positive runner-request ID. It does not change runner state or access Docker. Other blank, contradictory, unrelated named or Docker-error cases retain their existing failures. Completion metadata is logged before handling so an unclassified event retains the evidence needed to diagnose it. An empty name alone does not establish cancellation.

Validation: the new handler, diagnostic and real pinned-SDK batch regressions failed before the correction and pass afterward. Existing named-runner rejection coverage now also exercises canceled events. Local documentation links and the trusted committed-secret scan pass. Complete controller race checks and exact-head hosted validation are required before merge or adoption; focused checks are not full-suite or host evidence. See the PR checks and validation comments for their results.

Documentation: docs/CONTROLLER-DESIGN.md describes the exact conservative exception, diagnostics and SDK acknowledgment ordering. This runtime-only correction changes no schema, capability, initializer, desired-state field, example or operator procedure, so no standalone configuration-template change is needed. Engine availability, adopter promotion, reproducible image identities and host acceptance remain separate.

The reported historical empty-name event did not retain its result or runner ID. This patch covers the documented SDK case; it does not prove that historical event's cause or authorize a retry or runtime substitution. The Docker removal race is a separate issue.

@Nickfost
Nickfost marked this pull request as ready for review October 24, 2026 09:18
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T09:53:24.770079Z 6baf72a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copy link
Copy Markdown
Member Author

Validation update for exact head 6baf72a:

  • The new handler/diagnostic/pinned-SDK batch regressions failed against unchanged code and passed with the correction.
  • Changed-path checks also pass with Go1.26.3 and the race detector, including retained unrelated-runner and Docker-error rejection.
  • Independent exact-diff source review found no actionable findings. Documentation links and the trusted committed-secret scan pass.
  • A complete local controller race run FAILED in unchanged TestDockerSocketProxyReleasesSpoolBeforeBuildResponseCompletes at its two-second HTTP deadline. An isolated run of that same fixture also FAILED on exact base456c92e02c8266ee0d45e81b562cd8b20ec49f11. The local full-suite failure is retained; it is not relabeled a pass or excluded from coverage.
  • The required exact-head hosted full workflow is https://github.com/RandomDevelopment/ci-fleet/actions/runs/40081185025. It must finish successfully before merge/adoption. No full-validation or host-success claim is made here.

The conservative zero-ID/time and job/request-identity conditions are deliberate local restrictions; the SDK README does not guarantee every legitimate cancellation has that shape. The observed historical empty-name event remains unclassified. This patch neither silently accepts other blanks nor proves that historical incident fixed.

Copy link
Copy Markdown
Member Author

Complete controller race checks PASS on unchanged exact head6baf72a8429bfc6072f65c9b79f77359a52c3678 with Go1.27.2:

GOTOOLCHAIN=local go test -race ./... -count=1
ok github.com/RandomDevelopment/ci-fleet/controller 74.542s

The full Go1.26.3 run and its isolated proxy fixture failed as already recorded, with the same isolated failure reproduced on base456c92e. Those results remain retained. Changed-path race checks pass on Go1.26.3, and the complete controller package passes on Go1.27.2.

Independent exact-source review, local documentation links and committed-secret scanning pass. The separate eight-stage hosted full run and ordinary exact-head review remain required before merge/adoption. These controller checks alone do not authorize runtime promotion or host activation. No source changes were made after6baf72a.

Copy link
Copy Markdown
Member Author

@codex review

One exact-head standalone-receipt follow-up for the completed review of6baf72a8429bfc6072f65c9b79f77359a52c3678, base456c92e02c8266ee0d45e81b562cd8b20ec49f11, treef0aaa72634cd87d7320f6913953e922b11054b65.

Summary6382459371 completed clean at09:19:49Z with both finding flags false, but there is still no standalone receipt or formal review at readback. The full hosted run40081185025 has now completed SUCCESS on this exact head. No other receipt follow-up exists.

Please independently confirm the exact source and provide the missing standalone receipt if a clean result is warranted. Use the plain standard no-major-issues opening and Reviewed commit binding without an added suffix. Report actual findings normally; do not infer approval from the summary or this request.

The source is unchanged. Preserve the narrow cancellation restrictions and retained invalid-event/Docker failures. The historical empty-name event remains unclassified. No new tests, input edits, parser change, trust expansion, runtime substitution or host action is requested. This is one coordinated follow-up, not a request to repeat other reviews.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 6baf72a842

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Nickfost

Copy link
Copy Markdown
Member Author

The owner explicitly accepted standalone receipt 6382774661 from bot ID 199175422 as source-review evidence for a gated public merge of exact head 6baf72a8429bfc6072f65c9b79f77359a52c3678, base 456c92e02c8266ee0d45e81b562cd8b20ec49f11, tree f0aaa72634cd87d7320f6913953e922b11054b65.

The unedited receipt's abbreviation resolves authoritatively to that full head. Its body SHA-256 is 8ea1cf0676097b45e4ac33410a351a02ac41bc898adfda7e97f5276e5769e8df. The unchanged recognizer rejects the new version-marker header. The initial completed summary had false/false findings; the latest completed summary has null/null fields. Those null values are not treated as a clean-summary verdict. This is an explicit decision for this single review record. It grants no parser change or acceptance of future records. A changed source or receipt needs a new decision or normally recognized evidence.

Hosted run 40081185025 attempt 1 completed successfully. Both jobs passed, the complete driver exited successfully, and all eight full stages appeared in order, including container build and cleanup. Actual checkout b4143eb651664685b07f51e2d6288e3c2748fc13 has the exact base/head parents and reviewed tree. This is hosted validation evidence, not an exported signed bundle. Independent exact-source review found no actionable findings. Fresh readback found no review threads or adverse reviews.

The owner decision leaves all runtime, image, private-validation and host-admission gates unchanged. The historical empty-name event remains unclassified. No host retry or runtime substitution follows from this public merge. Documentation impact is already covered by docs/CONTROLLER-DESIGN.md; this runtime-only correction needs no standalone template configuration change.

@Nickfost
Nickfost merged commit a6fd61c into main Oct 24, 2026
2 checks passed
@Nickfost
Nickfost deleted the t3/handle-unassigned-cancellation branch October 24, 2026 17:04
@Nickfost

Copy link
Copy Markdown
Member Author

PR148 was SHA-bound squash merged at 2026-10-24T17:04:20Z. Actual merged commit is a6fd61c1cc4877dd903abef2877dda957b974007. Its tree is f0aaa72634cd87d7320f6913953e922b11054b65, exactly the reviewed tree, and its sole parent is the validated base 456c92e02c8266ee0d45e81b562cd8b20ec49f11. Main was independently read back at the merged commit.

The main-push run is 40114451217, attempt 1. Its current status is in_progress and conclusion is None. It has not yet established successful merged-main full validation. The passed PR run is separate evidence.

The owner-selected source-review decision is recorded in 6387045218. Parser behavior is unchanged. No private input, runtime pin, image policy or host was changed. 02 remains drained under the last verified return state; this merged commit does not authorize admission. Actual merged-main success and the separate runtime, image, private-validation, recovery and host gates remain required.

Documentation impact remains covered by docs/CONTROLLER-DESIGN.md. No template configuration or operator-procedure change was required.

@Nickfost

Copy link
Copy Markdown
Member Author

Merged-main validation review for exact a6fd61c1cc4877dd903abef2877dda957b974007 is complete. Run 40114451217 attempt 1 and both jobs report success. Both checkout logs identify the merged commit. Its tree matches the reviewed source, and its sole parent is validated base 456c92e02c8266ee0d45e81b562cd8b20ec49f11. Producer App 15368, workflow 312031923 and workflow blob 7b3775bbccca6ca3add2da03b7e627b3268ddb99 match the preserved workflow.

The full driver completed, followed by the convention gate with CONVENTION_RESULT=success and post-job cleanup. Seven push-applicable stages ran in order: commits, history-secrets, shellcheck, documentation, tests-and-fixtures, embedded-template, and container-build-and-cleanup. The separate PR run had eight stages because pr-title runs only for PR events.

Evidence discrepancy is retained. The job-step API still labels the suite in progress and later steps pending despite completed successful jobs and full logs proving those commands finished. No successful per-step API verdict is claimed. The logs are hosted execution evidence; raw stage reports, detailed cleanup inventory and a signed validation bundle were not uploaded.

The owner's exact source-review acceptance in 6387045218 remains limited to that record and public merge. No parser, runtime pin, image policy, private input or host changed. Runtime and host adoption remain separate. No new implementation or documentation finding emerged from this evidence audit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant