Move agent lazy initialization from send_message() to server startup #2588

Problem

The agent-server's /health endpoint returns "OK" unconditionally (just checks the process is alive), but the actual agent initialization — plugin loading, ACP subprocess startup, protocol handshake, session creation — happens lazily on the first send_message() call via _ensure_agent_ready().

This creates a timing mismatch: Kubernetes marks the pod as ready (health check passes), the client connects and sends a message, but the server blocks for 1-5+ minutes doing initialization while the HTTP response hangs. If the client has a shorter timeout (e.g. 60s default api_timeout), the request fails with httpx.ReadTimeout.

Call chain on first send_message()

POST /api/conversations/{id}/events
  → event_service.send_message()                    # event_service.py:308-312
    → run_in_executor(conversation.send_message)     # blocks HTTP response
      → _ensure_agent_ready()                        # local_conversation.py:435-481
        → _ensure_plugins_loaded()                   # network I/O for plugins
        → agent.init_state()                         
          → ACPAgent._start_acp_server()             # acp_agent.py:568-677
            → subprocess_exec(claude-code-acp, ...)  # spawns ACP subprocess
            → conn.initialize()                      # protocol handshake
            → conn.new_session(cwd=working_dir)      # session creation (slow for large repos)
            → conn.set_session_mode()                # permission setup

Impact

In swebenchmultimodal evaluation runs on 2026-03-26, this caused 100% failure rates for large-image instances even at resource_factor=8:

Eval job references:

• eval-23598243168-claude-son (claude-sonnet-4-5, acp-claude) — SDK run 23598214617
• eval-23600690440-claude-4-6 (claude-opus-4-6, acp-claude) — SDK run 23600660578

Datadog log query to reproduce:

kube_job:eval-23598243168-claude-son "failed after 3 retries" "read operation timed out"

Proposed fix

1. Move _ensure_agent_ready() (or its heavy parts) to run during server startup in api_lifespan(), before mark_initialization_complete() is called
2. The /ready endpoint should only return 200 after the conversation is actually ready to handle messages, not just after the process starts
3. Consider adding a /warm-up endpoint that pre-initializes the agent for a given configuration, so the readiness probe can gate on actual readiness

Workaround

Increasing api_timeout on the client side (OpenHands/benchmarks#579). This is a band-aid — the server should be ready before accepting traffic.

Related code

• /health endpoint: openhands/agent_server/server_details_router.py:73-76 — always returns "OK"
• Lazy init: openhands/sdk/conversation/impl/local_conversation.py:435-481 — _ensure_agent_ready()
• ACP startup: openhands/sdk/agent/acp_agent.py:568-677 — _start_acp_server()
• Server lifespan: openhands/agent_server/api.py:48-142 — mark_initialization_complete() called before conversations are ready