Skip to content

SWE-bench eval_limit=500 stalls: all runtime pods stuck in pending, no progress for 45+ minutes #580

Closed
Closed
SWE-bench eval_limit=500 stalls: all runtime pods stuck in pending, no progress for 45+ minutes#580

Description

@simonrosenberg
simonrosenberg
opened on Mar 27, 2026

Summary

SWE-bench ACP-claude eval with eval_limit=500 stalls completely — runtime pods never leave pending status, the orchestrator hangs with no log output for 45+ minutes, and 0/500 instances complete.

This affects ALL benchmarks using SDK >= v1.15.0 (commit b99f15f), not just SWE-bench. Confirmed broken: swebench, swebenchmultimodal, commit0. Only GAIA (binary build) is unaffected.

Details

Triggered run:

  • K8s job: eval-23622798964-claude-4-6
  • SDK run: 23622781040
  • Eval run: 23622798964
  • CID: 2A6EC49A
  • Params: benchmark=swebench, model=claude-4.6-opus, agent_type=acp-claude, eval_limit=500, benchmarks_branch=fix/copy-python-runtime-for-eval-images

Other affected runs (same root cause):

Job Benchmark Branch Status
eval-23622889791-claude-son swebenchmultimodal fix/swebenchmultimodal-api-timeout All 5 instances CreateContainerError
eval-23624504367-claude-son swebenchmultimodal fix/swebenchmultimodal-api-timeout All 5 instances CreateContainerError
commit0 (deleted) commit0 unknown All instances failed, job deleted

Concurrent GAIA run (same SDK) is progressing fine — eval-23622799654-claude-4-6 uses b99f15f-gaia-binary images (different build path, not affected).

Observed Behavior

  1. Job starts, image build succeeds, 8 instances are launched at 00:01:28 UTC
  2. All 8 instances immediately fail with Runtime not yet ready (status: pending) — runtime_id=None, session_id=None
  3. Retries also fail with same error (32 total failures across 8 instances, all on retry 2-3)
  4. Last log line at 00:21:52 UTC — no output for 45+ minutes after that
  5. Progress bar stuck at 0/500
  6. runtime-pods namespace shows no pods at all: No resources found in runtime-pods namespace
  7. Orchestrator pod itself is healthy (Running, 1/1 READY)

Root Cause

SDK v1.15.0 broke all source-minimal eval images. PR OpenHands/software-agent-sdk#2567 changed the agent-server Dockerfile builder from --managed-python (self-contained venv) to --python-preference only-system (venv symlinks to system Python). The eval-base images don't have Python 3.13 at /usr/local/bin/, so the symlink is broken.

Every source-minimal runtime pod fails with:

CreateContainerError: exec: "/agent-server/.venv/bin/python":
stat /agent-server/.venv/bin/python: no such file or directory

Pod status logger confirms: 10 out of 13 runtime pods are in CreateContainerError (all source-minimal), while 3 GAIA pods (-binary build) run fine.

PR #578 fix is not sufficient. The Dockerfile.agent-layer changes are present on both branches (fix/copy-python-runtime-for-eval-images and fix/swebenchmultimodal-api-timeout), builds succeed, but pods still crash. The validation only tested 1 instance (sympy__sympy-23824). All other instances remain broken, likely because:

  1. Stale images cached in GHCR from pre-fix builds (tag b99f15f-* already existed), or
  2. Python 3.13 is not at /usr/local/bin/python3.13 in the builder (the assumed COPY path may be wrong)

Contributing factors

  • Cluster resource exhaustion: 4 concurrent evals saturate the 23-node runtime cluster (17/23 insufficient CPU, 19/23 insufficient memory)
  • Warm runtime pool broken: warm_runtimes.py crashes with relation "runtimes" does not exist — all sessions cold-start
  • Orchestrator hangs silently after retries exhaust instead of failing fast

Suggested Fix

Step 1: Verify the Python path in the builder image

docker run --rm ghcr.io/openhands/eval-builder:b99f15f ls -la /usr/local/bin/python*
docker run --rm ghcr.io/openhands/eval-builder:b99f15f readlink -f /agent-server/.venv/bin/python

If Python is not at /usr/local/bin/python3.13, update the COPY paths in Dockerfile.agent-layer to match the actual location.

Step 2: Force-rebuild images (purge GHCR cache)

The b99f15f-* image tags in GHCR are stale (built before the Dockerfile fix). Either:

  • Delete the stale tags from ghcr.io/openhands/eval-agent-server and re-trigger builds, or
  • Use a new SDK commit so the image tags are fresh (e.g. abc1234-sweb.eval.* has never been built, guaranteeing no cache hit)

Step 3: Merge the corrected fix to main

Once verified, merge PR #578 (with correct COPY paths) to main so ALL benchmark branches inherit the fix. Currently every feature branch needs the fix independently.

Step 4: Address cluster saturation (separate issue)

  • Add autoscaling or resource limits for concurrent eval jobs
  • Run the missing database migration for the runtimes table to restore the warm runtime pool
  • Add a circuit-breaker in the orchestrator to fail fast instead of hanging silently after retries exhaust

Timeline (UTC, 2026-03-27)

Time Event
00:00:29 SWE-bench eval starts, 10 workers
00:01:15 prune_db.py crashes: relation "runtimes" does not exist
00:01:28 8 instances dispatched to runtime-api
00:01:32 warm_runtimes.py crashes: same DB error
00:11:32 All 8 instances timeout: Runtime not yet ready (status: pending), runtime_id=None
00:21:52 Retry round 2 also fails; orchestrator goes silent after this

Activity

  1. simonrosenberg commented on Mar 27, 2026

    @simonrosenberg
    simonrosenberg
    on Mar 27, 2026
    MemberAuthor

    Root Cause Analysis

    Investigation traced the failure through K8s pod status logs, runtime-api logs, Datadog, and the evaluation job logs. There are two compounding root causes, plus an amplifying factor.

    Root Cause 1: Broken SWE-bench eval images (CreateContainerError)

    SDK v1.15.0 (PR OpenHands/software-agent-sdk#2567) changed the agent-server Dockerfile builder from --managed-python to --python-preference only-system, making the venv's bin/python a symlink to /usr/local/bin/python3.13 instead of a self-contained managed Python. The eval-base images don't have Python 3.13 at that path, so the symlink is broken.

    Every SWE-bench runtime pod fails immediately with:

    CreateContainerError: exec: "/agent-server/.venv/bin/python":
stat /agent-server/.venv/bin/python: no such file or directory

    Pod status logger snapshot confirms 6 out of 10 active runtime pods stuck in CreateContainerError, all using b99f15f-sweb.eval.x86_64.*-source-minimal images. GAIA pods (b99f15f-gaia-binary) are unaffected and running fine.

    Fix: PR #578 correctly addresses this by copying the Python 3.13 runtime from the builder stage.

    Root Cause 2: Runtime cluster resource exhaustion (Unschedulable)

    The 23-node eval-runtime cluster is fully saturated by concurrent eval workloads. Pod status logger shows:

    0/23 nodes available: 17 Insufficient cpu, 19 Insufficient memory,
3 node(s) didn't match Pod's node affinity/selector,
8 Insufficient ephemeral-storage

    At the time of this run, 4 eval jobs were running simultaneously:

    • eval-23622798964-claude-4-6 — swebench, eval_limit=500 (this job)
    • eval-23622799654-claude-4-6 — GAIA, 165 instances
    • eval-23622889791-claude-son
    • eval-23624504367-claude-son

    This is why the validation run in #578 (eval_limit=1, idle cluster) succeeded but this 500-instance run failed — the fix is correct but the validation didn't test at scale.

    Additionally, transient PVC provisioning races cause pods to initially fail with persistentvolumeclaim "runtime-*" not found before eventually being scheduled.

    Amplifying Factor: Warm runtime pool is broken

    The runtime-api's warm_runtimes.py and prune_db.py are both crashing with:

    relation "runtimes" does not exist

    All 3 runtime-api pods log Found 0 warm runtimes in the database and No warm runtimes found for API key mamoodi-eval for every session. This forces all sessions to cold-start, adding scheduling pressure to an already saturated cluster.

    Why GAIA works but SWE-bench doesn't

    GAIA SWE-bench
    Image b99f15f-gaia-binary b99f15f-sweb.eval.x86_64.*-source-minimal
    Python venv Present (different build) Missing (broken symlink)
    Pod status Running CreateContainerError
    eval_limit 165 500 (higher cluster pressure)

    Timeline (UTC, 2026-03-27)

    Time Event
    00:00:29 SWE-bench eval starts, 10 workers
    00:01:15 prune_db.py crashes: relation "runtimes" does not exist
    00:01:28 8 instances dispatched to runtime-api
    00:01:32 warm_runtimes.py crashes: same DB error
    00:11:32 All 8 instances timeout: Runtime not yet ready (status: pending), runtime_id=None
    00:21:52 Retry round 2 also fails; orchestrator goes silent after this

    Recommendations

    1. Merge fix: copy Python 3.13 runtime into eval images for venv compatibility #578 — fixes the CreateContainerError for eval images
    2. Run the missing database migration for the runtimes table to restore the warm runtime pool
    3. Limit concurrent eval jobs or add cluster autoscaling — 4 concurrent evals with 500+ total instances overwhelm 23 nodes
    4. Add a timeout/fail-fast in the orchestrator — currently hangs silently after retries exhaust instead of reporting failure

  2. simonrosenberg commented on Apr 2, 2026

    @simonrosenberg
    simonrosenberg
    on Apr 2, 2026
    MemberAuthor

    The core issue was fixed by OpenHands/software-agent-sdk#2656

  3. simonrosenberg
    closed this as completedon Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions