Skip to content

fix(controller): wait for overlapping runner removal - #145

Merged
Nickfost merged 1 commit into
mainfrom
t3/next-steps-1
Oct 18, 2026
Merged

Nickfost merged 1 commit into
mainfrom
t3/next-steps-1

Conversation

@Nickfost

Copy link
Copy Markdown
Member

When job completion overlaps the runner exit watcher, Docker can report that container removal is already in progress. That error currently aborts the scale-set listener and restarts the controller.

Wait up to 30 seconds for Docker's removed condition after this specific conflict. An absent container also succeeds. Other conflicts, Docker failures, failed wait responses, cancellation and timeout remain errors.

Validation:

  • The deterministic completion/watcher overlap regression fails on the original code with the reported listener error and passes with this fix. It verifies removal, released runner state/capacity and processing of an unrelated completion in the same SDK batch.
  • Full controller tests passed with go test -race -count=1 -timeout=3m ./....
  • go vet, controller binary build, formatting, exact-commit secret scan, commit/title policy and documentation links passed.
  • Full engine/container acceptance has not run locally because Docker is unavailable. Hosted full validation remains required before merge.

Documentation impact: updates docs/ARCHITECTURE.md with bounded overlapping-removal behavior and failure handling. No historical release notes change.

Template assessment: no standalone or embedded template change is needed. This internal controller cleanup fix changes no schema, desired-state field, initializer, capability evidence or operator adoption procedure. Engine promotion and live-host acceptance remain separate.

Fixes #143.

Wait up to 30 seconds for Docker to confirm removal when completion and exit cleanup overlap. Keep unrelated Docker failures visible and cover listener batch continuation, missing containers, cancellation and failed waits.

Fixes #143
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T18:15:26.801568Z 6595ba8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Nickfost
Nickfost merged commit eea0770 into main Oct 18, 2026
2 checks passed
@Nickfost
Nickfost deleted the t3/next-steps-1 branch October 18, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runner removal in progress aborts the listener and restarts the controller

1 participant