Skip to content

Report an early exit 124 or 137 in the Docker backend instead of a timeout - #108

Open
willtong-aq wants to merge 4 commits into
ArtificialAnalysis:mainfrom
willtong-aq:fix/sigkill-not-timeout
Open

willtong-aq wants to merge 4 commits into
ArtificialAnalysis:mainfrom
willtong-aq:fix/sigkill-not-timeout

Conversation

@willtong-aq

@willtong-aq willtong-aq commented Nov 13, 2026 •

Copy link
Copy Markdown

Fixes #107.

Summary

DockerCodeExecToolProvider runs commands under timeout --kill-after=5s <N>s and mapped every exit 124 or 137 to error_kind="timeout". Both can occur before the limit: a 137 usually means an outside SIGKILL, for example the kernel out-of-memory killer, and a 124 can come from a nested timeout. The model was told "Command timed out after N seconds" without the real exit code. format_result also dropped stdout from error results.

Changes

  • Docker backend: exit 124 or 137 is a timeout only when the elapsed time is at least the limit. An earlier 124 or 137 keeps the real exit code, stdout and stderr, adds a note to stderr and sets no error_kind. The elapsed time is measured in the worker thread around exec_run.
  • base.py format_result: the error branch also writes <stdout> when stdout is not empty. An error result with empty stdout formats exactly as before.
  • Tests cover the limit boundary, early 124 and 137, and format_result.
  • CHANGELOG.md: an Unreleased / Fixed entry.

Before and after

echo allocating; python3 -c "x = b'a' * (1024 ** 3)", timeout=60, container limited to 256 MiB.

Before:

<shell_results>
<error_kind>timeout</error_kind>
<details>Command timed out after 60 seconds</details>
</shell_results>

After:

<shell_results>
<exit_code>137</exit_code>
<stdout>allocating
</stdout>
<stderr>Exit code 137 after 0.33 seconds, before the 60 second limit. This usually means the process was killed with SIGKILL, possibly by the out-of-memory killer.</stderr>
</shell_results>

Validation

Against a real python:3.12-slim container limited to 256 MiB and 1 CPU:

  • out-of-memory command: exit 137 after about 0.3 s, the note above, no "timed out"
  • timeout 3 sleep 60 with timeout=60: exit 124 after 3.1 s, note "The limit did not stop this command."
  • sleep 60 with timeout=5, and a command that ignores TERM with timeout=3: still reported as timeouts
  • exit 0 and exit 1: output unchanged

ruff format --check, ruff check and ty check pass. uv run --extra docker pytest tests/test_docker_execution.py passes locally (17 passed).

Limits

  • The note is hedged: an exit code does not show who sent SIGKILL, and Docker's State.OOMKilled is container-wide.
  • Docker API overhead counts towards the elapsed time, so an early exit very close to the limit can still read as a timeout.
  • Local and E2B backends do not map 124 or 137 to a timeout and are unchanged.

The Docker backend runs each command under coreutils timeout(1) with
--kill-after=5s and mapped both exit 124 and exit 137 to
error_kind="timeout". timeout(1) exits 137 when the child dies by
SIGKILL (or exits 137 itself), and it sends SIGKILL itself only after
the limit. A 137 before the limit usually means a SIGKILL from
elsewhere, for example the kernel out-of-memory killer, so the model
was told "Command timed out after N seconds" for a command that was
killed in under a second.

Exit 137 is now a timeout only when the elapsed time is at least the
timeout. The elapsed time is measured in the worker thread around
exec_run, so the wait for a thread is not counted. An earlier 137
returns the real exit code, stdout and stderr plus a note: "Exit code
137 after X seconds, before the N second limit. This usually means the
process was killed with SIGKILL, possibly by the out-of-memory killer."
It sets no error_kind.

format_result now also writes stdout in the error branch when it is not
empty, so partial output of a timed-out command reaches the model. An
error result with empty stdout formats as before.
Exit 124 was always reported as a timeout. A command can exit 124
before the limit by itself, for example when it runs its own nested
`timeout 3 sleep 60` under a 60 second limit. The model was then told
"Command timed out after 60 seconds" after 3 seconds, and the real exit
code was hidden.

Exit 124 is now a timeout only when the elapsed time is at least the
timeout, the same rule as exit 137. An earlier 124 returns the real exit
code, stdout and stderr plus a note: "Exit code 124 after X seconds,
before the N second limit. The limit did not stop this command." It sets
no error_kind.
The early exit 124 or 137 branch now appends its note to stderr and
reaches the existing final return, instead of building its own result
and writing a log line. The tests use a plain perf_counter patch, and
the format_result test is a module-level function. The output is
unchanged.
@willtong-aq willtong-aq changed the title Report an early exit 137 in the Docker backend instead of a timeout Report an early exit 124 or 137 in the Docker backend instead of a timeout Nov 13, 2026
The perf_counter patch held two values, so the chown exec during
provider teardown exhausted it. The error was hidden by the provider's
generic exception handler. The patch now repeats the last value.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker backend reports an early exit 137 (e.g. out-of-memory kill) as "Command timed out"

1 participant