Skip to content

fix(local): retain sandbox folders after incomplete teardown - #561

Merged
earakely-scale merged 3 commits into
scaleapi:mainfrom
morluto:codex/local-cleanup-verification
Nov 9, 2026
Merged

earakely-scale merged 3 commits into
scaleapi:mainfrom
morluto:codex/local-cleanup-verification

Conversation

@morluto

@morluto morluto commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Local sandbox teardown can report success when Docker is unavailable or rejects container removal. The run then removes the sandbox work folder, losing the identity needed to retry. This complements #555 and addresses the teardown failure reported in #546.

Check the sandbox's named container and labeled containers and networks, report failed queries or removals, and report marked Linux processes that survive termination. Still attempt the other cleanup stages, including Compose, before returning an error. Existing run teardown retains the work folder and reports the sandbox as still up when termination fails. Image removal remains best effort because images can be shared.

With no Docker installed, nothing of a sandbox can have been started with it, so teardown finishes. With Docker unreachable, teardown fails only for a sandbox whose folder records its own container, compose stack or closed network; one that ran only commands on this machine finishes with a warning naming the cleanup command. Process discovery on macOS remains limited: /proc-based host-process cleanup is Linux-only.

How it was tested

  • .venv/bin/ruff check . passed.
  • The local sandbox, task teardown, sandbox teardown step, retry, and environment-close unit suites passed: 142 passed, 3 skipped. The regressions cover failed container listing, container removal and network listing, retained folders and retry, plus continued Compose cleanup.
  • With Docker Desktop on macOS, a real local sandbox's container and network were removed. In a separate real-container probe, a simulated Docker listing refusal surfaced as an error and retained the folder and container; retry against Docker removed both.
  • A full run_bundle SIGKILL integration run and real Linux-host process cleanup were not run. The existing macOS host-process limitation is unchanged.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues remain.

Summary

Local teardown now reports failed Docker queries and removals, plus Linux processes that survive termination. Existing teardown callers keep the sandbox folder when termination fails.

  • Local sandbox teardown keeps failed cleanup visible for a later retry.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Stop marked processes] --> B[Check Docker]
  B -->|Reachable| C[Attempt container, Compose and labeled cleanup]
  B -->|Unreachable with recorded objects| D[Record failure]
  B -->|Missing, or unreachable without recorded objects| E[Finish Docker checks]
  C --> F{Any cleanup failures?}
  D --> F
  E --> F
  F -->|Yes| G[Raise error and retain folder]
  F -->|No| H[Caller removes folder]
Loading

Reviews (5) · Last reviewed commit: "fix(local): a sandbox that made no Docke..." · Reviewed by Greptile

@morluto
morluto requested a review from a team as a code owner November 9, 2026 14:47
@earakely-scale

Copy link
Copy Markdown
Collaborator

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from the sibling PR rather than from this diff: #555 (open, also answering #546) ships a documented, supported recipe for cleaning up a hard-killed run's local sandboxes, and it calls the method this PR changes.

That recipe appears in local_sandbox_ids()'s docstring (provider.py:630-633 on that branch), in run_bundle's docstring (src/agent_env/bundle/run.py:243-244), in .env.example, and it's what its new integration test runs at tst/integration/cli/run_bundle_test.py:272-274:

for sandbox_id in local_sandbox_ids():
    await LocalSandboxProvider().terminate_sandbox(sandbox_id)
    remove_local_work_dir(sandbox_id)

LocalSandboxProvider.terminate_sandbox is just await _found(sandbox_id).terminate() with no try/except, so the new raise RuntimeError("Local sandbox teardown incomplete: …") at the end of terminate() (provider.py:364 on this branch) propagates straight into that loop. terminate() can already raise today on a failed docker compose down, but this PR widens it to the cases a supervisor is most likely to meet — Docker missing or unreachable, a failed docker ps listing, marked processes that survive — which today log and return. One sandbox in that state would then abort the loop, leaving the remaining sandboxes of the killed run up.

Might be worth the two PRs agreeing on whether that documented loop should catch per id, and whether it should still call remove_local_work_dir after a raised terminate (retaining the folder for retry being the point of this change). Whichever lands second probably wants to update the snippet in those three docstrings too.


Generated by Claude Code

earakely-scale and others added 2 commits November 9, 2026 09:22
…Docker

With no Docker installed, nothing of a sandbox can have been started with it, so there's nothing to remove. With Docker
unreachable, teardown now fails only for a sandbox whose folder records its own container, compose stack or closed
network; one that ran only commands on this machine finishes with the old warning naming the cleanup command. The
repeated stderr limit is named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator

Thanks, this is the right direction: a teardown that couldn't confirm removal should keep the folder for a retry, and it makes #555's recipe exact. I pushed one change (9588e12, merged with main first, no force-push), for one case the strict version regressed.

The case: a local sandbox that never used Docker, such as the shipped hello example's VM-mode sandbox, failed its teardown whenever Docker wasn't installed or its daemon was down, and left its folder each run. On a Linux box with the PR as it was:

main before 9588e12 after
Docker running torn down torn down torn down
Daemon unreachable (as when Docker Desktop is stopped) torn down, warning "couldn't tear down", folder left torn down, warning
No docker installed torn down "couldn't tear down", folder left torn down

The change: teardown first asks Docker whether it's reachable (docker version):

  • reachable: your strict container, compose and labeled-object removal runs unchanged;
  • not installed: nothing Docker-made can exist, so there's nothing to remove;
  • unreachable: it fails only if the sandbox's folder records Docker objects of its own (its container, a compose stack or a closed network), which keeps the folder for a retry. Otherwise it logs main's warning, with the docker rm -f $(docker ps -aq --filter label=…) command, and finishes.

Also: the repeated stderr limit is named (_ERROR_TAIL_CHARS, Greptile's comment), and the AGENTS.md line says the above.

Tests, on a Linux box:

  • Unit: test_teardown_without_docker_installed_finishes, and test_teardown_with_docker_unreachable_fails_only_for_a_sandbox_that_made_docker_objects across four cases (no record, its container, its compose stack, its closed network). tst/unit/providers/sandbox_providers, tst/unit/task and tst/unit/task_step: 4,409 passed.
  • run_bundle_test.py, which includes feat(local): a supervisor can terminate the sandboxes of a run it killed #555's SIGKILL cleanup test: 10 passed.
  • A real container-mode sandbox torn down with DOCKER_HOST pointing at a missing socket: the teardown failed and kept both its folder and its nginx container. A retry once Docker was reachable removed them.
  • The hello bundle in all three Docker states above.

One follow-up for #555's recipe: terminate_sandbox now raises on an incomplete teardown, so a supervisor's loop should catch each sandbox's error and carry on to the rest, keeping that sandbox's id for a retry. Your helper already retains unresolved ids that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants