From a11a17b3df6160333bfb8e151c594f28868616b5 Mon Sep 17 00:00:00 2001 From: enyst Date: Sun, 15 Nov 2026 09:17:55 +0000 Subject: [PATCH 1/2] docs(verify): map maintenance 2026-11-15 (delta pass to main@2aae48b6) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013ArwwJSRAiRLzhbUrepjY4 --- .../feature-map/F02-auth-and-sessions.md | 21 +++ .../feature-map/F07-session-socket.md | 28 +-- .../F10-fork-navigate-condense-ask.md | 21 +-- .../references/feature-map/F19-mcp.md | 173 ++++++++++++++++-- .../feature-map/F30-openai-gateway.md | 47 ++--- .../feature-map/F31-persistence-and-limits.md | 28 +-- .../references/feature-map/README.md | 8 +- .../skills/verify-sdk/programs/S01/loop.py | 59 ++++++ .../verify-sdk/programs/S01/step_errors.json | 1 - .../verify-sdk/programs/S02/fake_provider.py | 16 +- .../verify-sdk/programs/S02/llm_calls.py | 135 +++++++++++++- .../programs/S04/pause_during_startup.py | 85 +++++++++ .../skills/verify-sdk/programs/S04/views.py | 132 +++++++++++++ .../verify-sdk/programs/S05/llm_failures.py | 99 ++++++++++ .../verify-sdk/programs/S20/retry_log.py | 93 ++++++++++ .../references/feature-map/README.md | 8 +- .../references/feature-map/S01-agent-loop.md | 25 ++- .../references/feature-map/S02-llm-core.md | 42 +++-- .../feature-map/S04-local-conversation.md | 30 ++- .../references/feature-map/S05-events.md | 26 ++- .../feature-map/S13-security-confirmation.md | 2 +- .../references/feature-map/S15-browser.md | 2 +- .../feature-map/S20-observability.md | 22 ++- 23 files changed, 985 insertions(+), 118 deletions(-) create mode 100644 .agents/skills/verify-sdk/programs/S04/pause_during_startup.py create mode 100644 .agents/skills/verify-sdk/programs/S05/llm_failures.py create mode 100644 .agents/skills/verify-sdk/programs/S20/retry_log.py diff --git a/.agents/skills/verify-agent-server/references/feature-map/F02-auth-and-sessions.md b/.agents/skills/verify-agent-server/references/feature-map/F02-auth-and-sessions.md index 7df70e738b0..66b44af5c93 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F02-auth-and-sessions.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F02-auth-and-sessions.md @@ -26,6 +26,7 @@ Routes: `POST /api/auth/workspace-session`, `DELETE /api/auth/workspace-session` - `F02.ws-legacy-key`: the deprecated `session_api_key` query parameter and `X-Session-API-Key` header still open a socket, and a redundant auth frame sent after them is ignored. - `F02.ws-legacy-reject`: a wrong legacy key rejects the handshake itself (HTTP 403, no close frame), even when a correct first frame would follow. - `F02.workspace-session-mint`: `POST /api/auth/workspace-session` answers 204 with an HttpOnly `oh_workspace_session_key` cookie on `Path=/api/conversations`, `SameSite=none; Secure; Partitioned` on a loopback host, and 401 with no cookie without a valid header (the cookie itself cannot mint another). +- `F02.workspace-session-plain-http`: over plain HTTP to a non-loopback host (no `X-Forwarded-Proto: https`) the workspace cookie is minted `SameSite=lax` without `Secure` or `Partitioned`, so browsers store it; the same host over HTTPS gets `SameSite=none; Secure; Partitioned`. - `F02.workspace-cookie-access`: the minted cookie alone (a cookie jar, no header) loads workspace files, also after a restart; a bogus cookie or no credential gets 401. - `F02.workspace-session-delete`: `DELETE /api/auth/workspace-session` answers 204 with a `Max-Age=0` cookie on the same path, which drops the cookie from the jar so workspace files are 401 again; 401 without a header. - `F02.multiple-keys`: every configured key (`OH_SESSION_API_KEYS_0`, `_1`, ...) authenticates HTTP and WebSocket clients; an unlisted key does not. @@ -518,6 +519,25 @@ Preconditions: `http://[::1]:PORT`. `request.url.hostname` returns the IPv6 address unbracketed (`::1`), so `_request_is_secure_context` strips a `:port` suffix only from a bracketed host or a host with a single colon. +- **Plain HTTP to a non-loopback host (`F02.workspace-session-plain-http`).** + Mint as a browser at a LAN address over plain HTTP would (Canvas served by + a VM, a container or another machine), then as the same host behind a + TLS-terminating proxy. + ```sh + control-agent-server api POST /api/auth/workspace-session --header 'Host: 10.244.8.163:8000' --expect 204 --save F02.workspace-session-plain-http/lan-http \ + --check-header set-cookie contains 'SameSite=lax' --check-header set-cookie contains HttpOnly \ + --check-header set-cookie not-contains Secure --check-header set-cookie not-contains Partitioned + control-agent-server api POST /api/auth/workspace-session --header 'Host: 10.244.8.163:8000' --header 'X-Forwarded-Proto: https' --expect 204 \ + --save F02.workspace-session-plain-http/lan-https --check-header set-cookie contains 'SameSite=none' \ + --check-header set-cookie contains Secure --check-header set-cookie contains Partitioned + ``` + Over plain HTTP to a non-loopback host the cookie is `SameSite=lax`, + without `Secure` or `Partitioned`: browsers refuse to store a + `SameSite=none` cookie that is not `Secure`, while a `Lax` cookie is stored + and sent on the same-site requests Canvas makes when it shares the + server's origin, so its file previews load. Cross-site embedding on plain + HTTP stays unsupported. The same host behind HTTPS (`X-Forwarded-Proto: + https`) gets `SameSite=none; Secure; Partitioned`, like loopback. ## Gotchas @@ -558,3 +578,4 @@ Preconditions: - With keys configured and no `--host`, the server binds `0.0.0.0`; without keys, `127.0.0.1` (`F02.default-bind`). The launcher always passes `--host 127.0.0.1`, so only `F02.default-bind` sees the default. +- Agent Canvas bundles Agent Server 1.62.0 by default, which mints the cookie `SameSite=none` on plain HTTP too, so Canvas file previews on a plain-HTTP non-loopback origin get 401 until Canvas pins a release that has `F02.workspace-session-plain-http` ([OpenHands/OpenHands#15567](https://github.com/OpenHands/OpenHands/issues/15567), [#15519](https://github.com/OpenHands/OpenHands/issues/15519)). diff --git a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md index 2ee43e1bcc5..0f857937ad7 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md @@ -26,7 +26,7 @@ Routes: `WS /sockets/session/{conversation_id}` - `F07.sync-first`: the first server frame is `{"type": "sync"}`; `from_seq` echoes `after_seq` (absent when live-only) and `through_seq` is the last on-disk `seq` (absent for an empty log). - `F07.sync-first-during-run`: while a run holds the conversation's state lock, `sync` is still the first frame; an `error` reply to an invalid frame sent right after connecting comes after it. - `F07.snapshot-transient`: every connection gets one `transient` frame with a `full_state` `ConversationStateUpdateEvent` and no `seq`, after `sync` (and after the replay when one ran); it is never persisted. -- `F07.snapshot-compact-stats`: the `full_state` snapshot carries each LLM's aggregate stats (`accumulated_cost`, `accumulated_token_usage`) but not the per-call `costs`, `response_latencies` and `token_usages` lists, so it stays small however long the conversation runs. +- `F07.snapshot-compact-stats`: the `full_state` snapshot carries each LLM's aggregate stats (`accumulated_cost`, `accumulated_token_usage`) but not the per-call `costs`, `response_latencies` and `token_usages` lists, so it stays small however long the conversation runs; `GET /api/conversations/{id}` returns the same aggregates with empty per-call lists. - `F07.full-replay`: `after_seq=-1` (or any value below it) replays every persisted event as `durable` frames with `seq` 0..`through_seq`, in order, once each, with the same ids REST returns. - `F07.seq-on-disk`: a durable frame's `seq` is the event's on-disk index, the `NNNNN` in `events/event-NNNNN-.json`. - `F07.resume-cursor`: `after_seq=k` replays exactly the events with `seq > k`; `after_seq=through_seq` replays nothing. @@ -448,10 +448,10 @@ Preconditions: ``` The replay carries the durable reply (its id is `ITEM`) and no progress frame of any kind. -- **Compact snapshot stats (`F07.snapshot-compact-stats`), known bug #6741.** +- **Compact snapshot stats (`F07.snapshot-compact-stats`).** Reconnect live-only to the streamed conversation, which has made one model call, and read its snapshot's stats for the agent's LLM (usage id - `default`). + `default`), then the same conversation's REST `ConversationInfo`. Requires: `F07.progress-frames` ```sh @@ -463,17 +463,22 @@ Preconditions: --check frames.2.frame.event.value.stats.usage_to_metrics.default.accumulated_cost exists \ --check frames.2.frame.event.value.stats.usage_to_metrics.default.response_latencies missing \ --check frames.2.frame.event.value.stats.usage_to_metrics.default.token_usages missing \ - --check frames.2.frame.event.value.stats.usage_to_metrics.default.costs missing # bug: checks.0.ok eq true && checks.1.actual contains latency + --check frames.2.frame.event.value.stats.usage_to_metrics.default.costs missing + control-agent-server api GET "/api/conversations/$SID" --expect 200 --save "F07.snapshot-compact-stats/info-$SID" \ + --check stats.usage_to_metrics.default.accumulated_token_usage.prompt_tokens gt 0 \ + --check stats.usage_to_metrics.default.costs len-eq 0 \ + --check stats.usage_to_metrics.default.response_latencies len-eq 0 \ + --check stats.usage_to_metrics.default.token_usages len-eq 0 ``` Expected: the snapshot keeps the aggregates (prompt tokens above 0, an `accumulated_cost`) and drops the per-call lists, as the incremental - `stats` updates do (`use_snapshot`). Today the `full_state` snapshot is - dumped without that context, so it carries one `costs`, - `response_latencies` and `token_usages` entry per model call (about - 2.6 KB each); past roughly 1,550 calls the snapshot exceeds the 4 MiB - frame cap and every connection closes with 1013 `frame_too_large` - before the client sees anything (#6741). The legacy events socket's - snapshot (F06) is built the same way. + `stats` updates do (`use_snapshot`), so its size does not grow with the + number of model calls and a long conversation's snapshot stays under the + 4 MiB frame cap. `GET /api/conversations/{id}` (and search) keeps the same + `stats` schema with the aggregates and empty per-call lists. The + conversation's `base_state.json` still keeps every per-call entry; only + the wire payloads are compact. The legacy events socket's snapshot (F06) + is built the same way. - **Interrupted stream (`F07.item-aborted`).** Start a streamed deepseek-flash answer that takes tens of seconds (counting to 3000), wait until its `delta` frames flow on a background capture, then interrupt the @@ -786,3 +791,4 @@ Preconditions: POSTs above use `--quiet` to keep the map-run transcript small (`--expect` and `--check` still see the bodies); `--max-chars` only trims non-JSON response bodies. +- Agent Canvas bundles Agent Server 1.62.0 by default, whose `full_state` snapshot still carries the per-call lists, so a very long conversation keeps closing with 1013 `frame_too_large` in Canvas until it pins a release that has `F07.snapshot-compact-stats` ([OpenHands/OpenHands#18935](https://github.com/OpenHands/OpenHands/issues/18935)). diff --git a/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md b/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md index 787ebd137a4..5f3e1e8de4c 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md @@ -556,7 +556,7 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key A second `Condensation` is recorded and ask_agent still names the codename from the reloaded history (the condenser keeps the first two events, the system prompt and the user's message); `A` stays `finished`. -- **Ask usage on GET and after a restart (`F10.ask-agent-usage-saved`), known bug #6844.** +- **Ask usage on GET and after a restart (`F10.ask-agent-usage-saved`).** A conversation of its own on an `llm-stub` with fixed usage numbers: the run's reply reports 11 prompt and 7 completion tokens, the ask's reply 5 and 3. The bullet restarts the run, so it comes last. @@ -575,7 +575,7 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key control-agent-server api GET "/api/conversations/$U" --max-chars 300 \ --check stats.usage_to_metrics.default.accumulated_token_usage.completion_tokens eq 7 \ --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.completion_tokens eq 3 \ - --save F10.ask-agent-usage-saved/get # bug: status eq 200 && checks.0.ok eq true && checks.1.actual eq null + --save F10.ask-agent-usage-saved/get control-agent-server restart control-agent-server api GET "/api/conversations/$U" --max-chars 300 \ --check stats.usage_to_metrics.default.accumulated_token_usage.completion_tokens eq 7 \ @@ -584,12 +584,9 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key ``` The run's usage is on GET (7 completion tokens under `default`), the ask answers, and the `stats` event the ask recorded reports 5 and 3 tokens - under `ask-agent-llm`. Expected: GET shows the same `ask-agent-llm` entry - right away and after a restart - ([#6844](https://github.com/OpenHands/software-agent-sdk/issues/6844), fix in [#6845](https://github.com/OpenHands/software-agent-sdk/pull/6845)). - Today GET still lists only `default`: the ask updates the live stats - without saving the conversation's state, GET reads the saved state, and a - restart right after the ask loses the entry. + under `ask-agent-llm`. GET shows the same `ask-agent-llm` entry right + away, next to `default`, and still after a restart: the ask saves its + usage with the conversation's state. ## Gotchas @@ -629,11 +626,9 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key Each call records and pushes one `stats` `ConversationStateUpdateEvent` whose `usage_to_metrics` includes an `ask-agent-llm` entry (`F10.ask-agent` checks the total count), so a client that counts events - around an ask sees one more. GET's `stats` does not show the - `ask-agent-llm` entry until the conversation's state is next saved (a - message, a run), and a restart before that loses it - (`F10.ask-agent-usage-saved`, - [#6844](https://github.com/OpenHands/software-agent-sdk/issues/6844)). It passes the agent's tools to + around an ask sees one more. The same entry is saved with the + conversation's state, so GET and a restarted server show it + (`F10.ask-agent-usage-saved`). It passes the agent's tools to the model and returns the first text block, so on a conversation with no messages yet DeepSeek's answer came back as an empty `response` (200). - `fork` and `navigate` return `ConversationInfo` with diff --git a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md index ebc7acb34e8..6ff756f51a6 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md @@ -22,6 +22,7 @@ Launch: `--webhook-sink` Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, `GET /api/mcp/oauth/status/{job_id}`, `POST /api/mcp/oauth/callback/{job_id}`, +`GET /api/mcp/oauth/redirect`, `POST /api/settings/mcp/{settings_key}`, `PATCH /api/settings/mcp/{settings_key}`, `DELETE /api/settings/mcp/{settings_key}` @@ -48,7 +49,11 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, - `F19.oauth-callback-validation`: on a live job, an https URL, a foreign host, another port or another path is 400 and the job keeps waiting; the redirect URL with `127.0.0.1` instead of `localhost` completes it. - `F19.oauth-consent-denied`: submitting the redirect a provider sends when the user denies consent (`error=access_denied`) answers `ok: false` and fails the job without tokens. - `F19.oauth-consent-timeout`: a consent that is not completed within the job's `timeout` fails the job with `error_kind: timeout`. -- `F19.oauth-public-redirect`: on a server that knows its external URL (`OH_WEB_URL`), OAuth start registers a redirect URI on that public origin, path prefix included, while a server without one keeps the loopback callback (known bug #6828: always `http://localhost:/callback`). +- `F19.oauth-public-redirect`: on a server that knows its external URL (`OH_WEB_URL`), OAuth start registers `{web_url}/api/mcp/oauth/redirect` as the redirect URI, path prefix included, while a server without one keeps the loopback callback `http://localhost:/callback`. +- `F19.oauth-redirect-route`: with `callback_base_url` set to the server's own base URL (and no `web_url`), the provider redirects the browser to `/api/mcp/oauth/redirect`, which answers 200 "Authorization complete" without a session key; the job then succeeds with an encrypted `oauth_state`, and a second delivery of the same redirect is 409. +- `F19.oauth-redirect-unknown-state`: the redirect route answers 400 "Authorization failed" for a forged, empty or missing `state` and leaves the waiting job untouched, while the job's real `state` with a provider error is recognized (400 naming the error) and fails the job. +- `F19.oauth-callback-base-url-validation`: a `callback_base_url` with a query, a fragment, userinfo, no host or a non-http(s) scheme is 422; a plain https base with a path prefix is accepted. +- `F19.oauth-web-url-precedence`: on a server with `OH_WEB_URL`, a `callback_base_url` in the start body is ignored for the `web_url` redirect URI, and the redirect delivered to that server's route completes the job. - `F19.oauth-callback-no-redirect`: the callback route forwards the submitted query only to the job's own callback address and does not follow a redirect it gets from there (known bug #6808: the redirect is followed). - `F19.oauth-inline-writeback`: an OAuth server passed inline on a conversation's agent (not in settings) whose access token has expired is refreshed when the conversation connects, keeps its tools, and the rotated tokens are posted with the run's session key to the webhook's `/mcp-oauth-state`, not stored in settings. - `F19.oauth-stored-state`: the returned `oauth_state` used as `auth.state` connects without consent (test, and start without `authorization_url`), and stored through the settings route it is redacted, encrypted on disk and decrypted to the real token. @@ -74,6 +79,16 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, `POST /api/mcp/oauth/callback/{job_id}` with the redirect URL the browser landed on. The client then saves `oauth_state` under the server's `auth.state` through the settings route. +- REST, browser redirect (no session key, not in the OpenAPI document): + `GET /api/mcp/oauth/redirect?code=...&state=...`. When the server has + `web_url` (`OH_WEB_URL`), start registers `{web_url}/api/mcp/oauth/redirect` + as the provider's redirect URI (also for a pre-registered client); without + it, `callback_base_url` in the start body (the server's base URL as the + browser reaches it) does the same for a dynamically registered client. The + route finds the pending job by the OAuth `state`, answers an HTML page, and + the job exchanges the code; the client polls the status route. Without + either, the redirect URI is the probe's loopback listener and the client + submits the redirect URL to the callback route. - Conversations: `POST /api/conversations` with `agent_settings` from `GET /api/settings` (what `conversation start` and Agent Canvas do) carries `mcp_config`; the server connects to the enabled servers when the agent @@ -257,6 +272,8 @@ Preconditions: test "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$OAUTH_PORT/mcp")" = 401 test "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$NONE_PORT/mcp")" != 000 OAUTH_SRV=$(jq -nc --arg url "http://127.0.0.1:$OAUTH_PORT/mcp" '{type: "http", url: $url, auth: {strategy: "oauth2", authentication: {type: "oauth", client_auth_method: "none", scopes: ["mail.read"]}}}') + QS_OF='import json, sys, urllib.parse as u; print(json.dumps({k: v[0] for k, v in u.parse_qs(u.urlsplit(sys.argv[1]).query).items()}))' + "$PY" -c "$QS_OF" 'http://x.invalid/a?redirect_uri=http%3A%2F%2Fy&state=s' | control-agent-server assert --check redirect_uri eq http://y --check state eq s ``` - **Auth (`F19.auth`).** Every route without a key, or with a wrong one. @@ -625,7 +642,7 @@ Preconditions: The job waits in `authorizing`, then, between 4 and 15 seconds after start (about four), is `failed` with `error_kind: timeout`; the user would have to start over. -- **Redirect URI on the public origin (`F19.oauth-public-redirect`), known bug #6828.** +- **Redirect URI on the public origin (`F19.oauth-public-redirect`).** A browser that is not on the agent-server's host (a container that publishes only the API port, a reverse proxy) cannot reach `http://localhost:/callback`. First this run, which has no external @@ -633,10 +650,9 @@ Preconditions: `redirect_uri` the provider was asked to return to out of each `authorization_url`. ```sh - REDIRECT_OF='import json, sys, urllib.parse as u; print(json.dumps({"redirect_uri": u.parse_qs(u.urlsplit(sys.argv[1]).query)["redirect_uri"][0]}))' PSTART=$(jq -nc --argjson s "$OAUTH_SRV" '{server: $s, timeout: 5}') LAU=$(control-agent-server api POST /api/mcp/oauth/start --json "$PSTART" --expect 200 --check ok eq true --field authorization_url) - "$PY" -c "$REDIRECT_OF" "$LAU" | control-agent-server assert --check redirect_uri matches '^http://localhost:[0-9]+/callback$' + "$PY" -c "$QS_OF" "$LAU" | control-agent-server assert --check redirect_uri matches '^http://localhost:[0-9]+/callback$' stop_public() { control-agent-server stop --run "$PUB" >/dev/null } @@ -645,17 +661,137 @@ Preconditions: control-agent-server config --run "$PUB" | jq -e '.server_env.OH_WEB_URL == "https://qa-f19.example/qa-f19"' PAU=$(control-agent-server api POST /api/mcp/oauth/start --run "$PUB" --json "$PSTART" --expect 200 --check ok eq true \ --check authorization_url matches "^http://127.0.0.1:$OAUTH_PORT/authorize\\?" --save F19.oauth-public-redirect/start --field authorization_url) - "$PY" -c "$REDIRECT_OF" "$PAU" | control-agent-server assert --check redirect_uri matches '^https://qa-f19\.example/qa-f19/' # bug: value.redirect_uri matches '^http://localhost:[0-9]+/callback$' + "$PY" -c "$QS_OF" "$PAU" | control-agent-server assert --check redirect_uri eq https://qa-f19.example/qa-f19/api/mcp/oauth/redirect control-agent-server stop --run "$PUB" trap - EXIT ``` Without an external URL the redirect URI is the probe's loopback listener, `http://localhost:/callback`, which a browser on the same machine - reaches. With `OH_WEB_URL=https://qa-f19.example/qa-f19` it must start with - that origin and prefix, so the provider sends the browser back to a route - the agent-server serves; the bug: `web_url` is not consulted and the - redirect URI is the loopback one again, so sign-in never completes for a - remote browser. Both jobs end on their 5-second consent timeout. + reaches. With `OH_WEB_URL=https://qa-f19.example/qa-f19` it is exactly + `https://qa-f19.example/qa-f19/api/mcp/oauth/redirect`: the public origin, + its path prefix and the agent-server's redirect route, so the provider + sends the browser back to a route the agent-server serves + (`F19.oauth-web-url-precedence` delivers one). Both jobs end on their + 5-second consent timeout. +- **Redirect to the agent-server's own route (`F19.oauth-redirect-route`).** + The browser-facing flow without `web_url`: the start body names this run's + base URL as `callback_base_url` (with a trailing slash, which is dropped), + and the recipe plays the browser in two hops: it reads the provider's + `Location` without following it, then requests that URL with no session + key, as a browser would. + ```sh + CBR="$RUN_URL/api/mcp/oauth/redirect" + JP=$(control-agent-server api POST /api/mcp/oauth/start --expect 200 --save F19.oauth-redirect-route/start \ + --json "$(jq -nc --argjson s "$OAUTH_SRV" --arg cb "$RUN_URL/" '{server: $s, timeout: 60, callback_base_url: $cb, tool_call: {name: "read_subject", arguments: {subject: "pub"}}}')" \ + --check ok eq true --check authorization_url matches "^http://127.0.0.1:$OAUTH_PORT/authorize\\?" --field job_id) + control-agent-server api GET "/api/mcp/oauth/status/$JP" --check status eq authorizing --check callback_ready eq true --until-ok 20 --save F19.oauth-redirect-route/authorizing + PAU=$(control-agent-server api GET "/api/mcp/oauth/status/$JP" --field authorization_url) + "$PY" -c "$QS_OF" "$PAU" | control-agent-server assert --check redirect_uri eq "$CBR" --check state exists + PCB=$(curl -s --noproxy '*' -o /dev/null -w '%{redirect_url}' "$PAU") + test "${PCB%%\?*}" = "$CBR" + PQ=$("$PY" -c "$QS_OF" "$PCB") + PCODE=$(echo "$PQ" | jq -r .code) + PSTATE=$(echo "$PQ" | jq -r .state) + control-agent-server api GET /api/mcp/oauth/redirect --auth none --query "code=$PCODE" --query "state=$PSTATE" --expect 200 \ + --check . contains '

Authorization complete

' --check-header content-type contains text/html --save F19.oauth-redirect-route/redirect + control-agent-server api GET "/api/mcp/oauth/status/$JP" --until-ok 30 --save F19.oauth-redirect-route/succeeded \ + --check status eq succeeded --check tools contains read_subject --check tool_result.text eq QA_HTTP_SUBJECT:pub \ + --check oauth_state.tokens.access_token matches '^gAAAA' --check oauth_state.client_info.client_id exists + control-agent-server api GET /api/mcp/oauth/redirect --auth none --query "code=$PCODE" --query "state=$PSTATE" --expect 409 \ + --check . contains 'This sign-in was already completed.' --save F19.oauth-redirect-route/replay + control-agent-server api GET "/api/mcp/oauth/status/$JP" --check status eq succeeded + ``` + The authorization URL asks the provider to return to + `/api/mcp/oauth/redirect`, and the provider's `Location` points + there with `code` and `state`. That request, without any key, answers 200 + with an HTML page titled "Authorization complete"; the job then exchanges + the code itself and is `succeeded` with `read_subject`, its result and + Fernet-encrypted tokens. Delivering the same redirect again answers 409 + "This sign-in was already completed." and the job is unchanged. +- **Unknown sign-ins on the redirect route (`F19.oauth-redirect-unknown-state`).** + A job waiting on the redirect route, then requests that do not carry its + `state`; the control is the provider's denial with the real `state`. + ```sh + JX=$(control-agent-server api POST /api/mcp/oauth/start --expect 200 --check ok eq true --field job_id \ + --json "$(jq -nc --argjson s "$OAUTH_SRV" --arg cb "$RUN_URL" '{server: $s, timeout: 60, callback_base_url: $cb}')") + control-agent-server api GET "/api/mcp/oauth/status/$JX" --check callback_ready eq true --until-ok 20 + XQ=$("$PY" -c "$QS_OF" "$(curl -s --noproxy '*' -o /dev/null -w '%{redirect_url}' "$(control-agent-server api GET "/api/mcp/oauth/status/$JX" --field authorization_url)")") + XCODE=$(echo "$XQ" | jq -r .code) + XSTATE=$(echo "$XQ" | jq -r .state) + test -n "$XCODE" && test -n "$XSTATE" && test "$XSTATE" != qa-f19-forged + control-agent-server api GET /api/mcp/oauth/redirect --auth none --query "code=$XCODE" --query state=qa-f19-forged --expect 400 \ + --check . contains '

Authorization failed

' --check . contains 'This sign-in is unknown or has expired.' --save F19.oauth-redirect-unknown-state/forged-state + control-agent-server api GET /api/mcp/oauth/redirect --auth none --query "code=$XCODE" --query state= --expect 400 \ + --check . contains 'This sign-in is unknown or has expired.' + control-agent-server api GET /api/mcp/oauth/redirect --auth none --query "code=$XCODE" --expect 400 \ + --check . contains 'This sign-in is unknown or has expired.' --save F19.oauth-redirect-unknown-state/no-state + control-agent-server api GET "/api/mcp/oauth/status/$JX" --check status eq authorizing --check oauth_state missing --save F19.oauth-redirect-unknown-state/still-waiting + control-agent-server api GET /api/mcp/oauth/redirect --auth none --query error=access_denied --query 'error_description=QA denied' --query "state=$XSTATE" \ + --expect 400 --check . contains '

Authorization failed

' --check . contains 'The provider reported: QA denied' --save F19.oauth-redirect-unknown-state/denied + control-agent-server api GET "/api/mcp/oauth/status/$JX" --until-ok 15 --save F19.oauth-redirect-unknown-state/failed \ + --check status eq failed --check ok eq false --check error eq 'Client failed to connect: OAuth authorization failed: QA denied' \ + --check error_kind eq connection --check oauth_state missing --check tools missing + ``` + A forged, empty or missing `state` answers 400 with the "Authorization + failed" page ("This sign-in is unknown or has expired."), and the job keeps + waiting with no tokens. The same route recognizes the job's real `state`: + a provider error there answers 400 "The provider reported: QA denied" and + fails the job without tokens (`Client failed to connect: OAuth authorization + failed: QA denied`, `error_kind: connection`). +- **Callback base URL validation (`F19.oauth-callback-base-url-validation`).** + ```sh + cbstart() { jq -nc --argjson s "$OAUTH_SRV" --arg cb "$1" '{server: $s, timeout: 3, callback_base_url: $cb}'; } + control-agent-server api POST /api/mcp/oauth/start --json "$(cbstart "$RUN_URL/?qa=1")" --expect 422 \ + --check detail.0.loc contains callback_base_url --check detail.0.msg contains 'callback_base_url must be a plain http(s) base URL' --save F19.oauth-callback-base-url-validation/query + control-agent-server api POST /api/mcp/oauth/start --json "$(cbstart "$RUN_URL/#qa")" --expect 422 \ + --check detail.0.msg contains 'callback_base_url must be a plain http(s) base URL' --save F19.oauth-callback-base-url-validation/fragment + control-agent-server api POST /api/mcp/oauth/start --json "$(cbstart 'http://qa:pw@127.0.0.1:9')" --expect 422 \ + --check detail.0.msg contains 'callback_base_url must be a plain http(s) base URL' --save F19.oauth-callback-base-url-validation/userinfo + control-agent-server api POST /api/mcp/oauth/start --json "$(cbstart 'ftp://127.0.0.1:9')" --expect 422 \ + --check detail.0.msg contains 'callback_base_url must be a plain http(s) base URL' --save F19.oauth-callback-base-url-validation/scheme + control-agent-server api POST /api/mcp/oauth/start --json "$(cbstart 'http:///qa-f19')" --expect 422 \ + --check detail.0.msg contains 'callback_base_url must be a plain http(s) base URL' + CAU=$(control-agent-server api POST /api/mcp/oauth/start --json "$(cbstart 'https://qa-f19.example/qa-f19/')" --expect 200 \ + --check ok eq true --save F19.oauth-callback-base-url-validation/accepted --field authorization_url) + "$PY" -c "$QS_OF" "$CAU" | control-agent-server assert --check redirect_uri eq https://qa-f19.example/qa-f19/api/mcp/oauth/redirect + ``` + Query, fragment, userinfo, a missing host and an `ftp` scheme are each 422 + on `callback_base_url` before any job starts; a plain https base with a + path prefix and a trailing slash is accepted and becomes + `https://qa-f19.example/qa-f19/api/mcp/oauth/redirect` (that job ends on + its 3-second consent timeout). +- **`web_url` wins over the request (`F19.oauth-web-url-precedence`).** A + second server with `OH_WEB_URL`; the start body also names this run's URL + as `callback_base_url`. Translated: no reverse proxy serves + `https://qa-f19.example`, so the recipe plays it and delivers the + provider's redirect to the second server's own route. + ```sh + stop_public_cb() { + control-agent-server stop --run "$PUBC" >/dev/null + } + PUBC=$(control-agent-server launch --new --name f19-public-cb --env OH_WEB_URL=https://qa-f19.example/qa-f19 --print-run) + trap stop_public_cb EXIT + WJ=$(control-agent-server api POST /api/mcp/oauth/start --run "$PUBC" --expect 200 --check ok eq true --save F19.oauth-web-url-precedence/start --field job_id \ + --json "$(jq -nc --argjson s "$OAUTH_SRV" --arg cb "$RUN_URL" '{server: $s, timeout: 60, callback_base_url: $cb, tool_call: {name: "read_subject", arguments: {subject: "web"}}}')") + control-agent-server api GET "/api/mcp/oauth/status/$WJ" --run "$PUBC" --check callback_ready eq true --until-ok 20 + WAU=$(control-agent-server api GET "/api/mcp/oauth/status/$WJ" --run "$PUBC" --field authorization_url) + "$PY" -c "$QS_OF" "$WAU" | control-agent-server assert --check redirect_uri eq https://qa-f19.example/qa-f19/api/mcp/oauth/redirect + WCB=$(curl -s --noproxy '*' -o /dev/null -w '%{redirect_url}' "$WAU") + test "${WCB%%\?*}" = https://qa-f19.example/qa-f19/api/mcp/oauth/redirect + WQ=$("$PY" -c "$QS_OF" "$WCB") + control-agent-server api GET /api/mcp/oauth/redirect --run "$PUBC" --auth none --query "code=$(echo "$WQ" | jq -r .code)" --query "state=$(echo "$WQ" | jq -r .state)" \ + --expect 200 --check . contains '

Authorization complete

' --save F19.oauth-web-url-precedence/redirect + control-agent-server api GET "/api/mcp/oauth/status/$WJ" --run "$PUBC" --until-ok 30 --save F19.oauth-web-url-precedence/succeeded \ + --check status eq succeeded --check tool_result.text eq QA_HTTP_SUBJECT:web --check oauth_state.tokens.access_token matches '^gAAAA' + control-agent-server stop --run "$PUBC" + trap - EXIT + ``` + The redirect URI is the `web_url` one, not `/api/mcp/oauth/redirect` + from the body, and the provider's `Location` points at it. The second + server's route takes the redirect without a key (200 "Authorization + complete") and its job succeeds with `QA_HTTP_SUBJECT:web` and encrypted + tokens. The evidence of the second server's calls is saved under that + run's `evidence/F19.oauth-web-url-precedence/`. - **Callback forwarding stops at the listener (`F19.oauth-callback-no-redirect`), known bug #6808.** Translated: the probe's own callback listener never redirects, so a stand-in takes its place. The job's `tool_call` keeps the job alive for 20 @@ -960,11 +1096,17 @@ Preconditions: a submitted URL to the freed port, to whatever listens there (`F19.oauth-callback-no-redirect`), or answers 409 `OAuth callback server is not listening`. -- The redirect URI is always FastMCP's loopback listener, - `http://localhost:/callback`, whatever `OH_WEB_URL` says - (`F19.oauth-public-redirect`); a client whose browser is elsewhere has to - copy the failed redirect URL from the browser and submit it to the callback - route. +- The redirect URI is `{web_url}/api/mcp/oauth/redirect` when the server + has `OH_WEB_URL` (whatever the start body says), else + `{callback_base_url}/api/mcp/oauth/redirect` for a dynamically registered + client, else FastMCP's loopback listener + `http://localhost:/callback`. Only the loopback form goes + through the callback route; the redirect route is reached by the browser + itself, needs no session key and identifies the job by `state` alone, so a + recipe extracts `code` and `state` from the provider's `Location` and + passes them as `--query` with `--auth none`. It answers HTML (400 for an + unknown, empty or missing `state` or a provider error, 409 for a second + delivery), so assert with `--check . contains`. - `oauth_state` is Fernet-encrypted when the server has `OH_SECRET_KEY` (every launched run does) and plaintext otherwise; posting it unchanged into `auth.state` of the settings route is the intended way to persist it. @@ -988,3 +1130,4 @@ Preconditions: `control-agent-server stop` ends them, and each also exits after 25 minutes or 60 seconds after the run's `/alive` stops answering; their logs are under `fixtures/f19/`. +- Agent Canvas sends `callback_base_url` on OAuth start ([OpenHands/OpenHands#19744](https://github.com/OpenHands/OpenHands/pull/19744)), but bundles Agent Server 1.62.0 by default, which ignores the field and keeps the loopback redirect; the public redirect (`F19.oauth-redirect-route`) reaches Canvas users once Canvas pins a release that has it. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md b/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md index c0853c1e331..478f78b8a25 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md @@ -479,18 +479,21 @@ control-agent-server api GET /api/settings --check conversation_settings.confirm --check choices.0.message.content eq QA_F30_U2 --raw-out "$F30/usage-2.json" --save F30.chat-usage-per-request/second-turn control-agent-server sink read --name qa-f30-usage --contains QA_F30_USAGE_2 --expect-min 1 --expect-max 1 control-agent-server sink read --name qa-f30-usage --expect-min 2 --expect-max 2 - control-agent-server api GET "/api/conversations/$UCID" --check stats.usage_to_metrics.default.token_usages len-eq 2 \ - --check stats.usage_to_metrics.default.token_usages.1.prompt_tokens eq 3 \ - --check stats.usage_to_metrics.default.token_usages.1.completion_tokens eq 2 \ + control-agent-server api GET "/api/conversations/$UCID" \ --check stats.usage_to_metrics.default.accumulated_token_usage.prompt_tokens eq 13 \ --check stats.usage_to_metrics.default.accumulated_token_usage.completion_tokens eq 7 --quiet + control-agent-server state cat base_state.json --conversation "$UCID" --check stats.usage_to_metrics.default.token_usages len-eq 2 \ + --check stats.usage_to_metrics.default.token_usages.1.prompt_tokens eq 3 \ + --check stats.usage_to_metrics.default.token_usages.1.completion_tokens eq 2 control-agent-server state cat fixtures/qa-f30/usage-2.json --check usage.prompt_tokens eq 3 \ --check usage.completion_tokens eq 2 --check usage.total_tokens eq 5 ``` The stub got exactly two model calls, one per request, so the second - request's own usage is the stub's 3/2/5: the conversation's stats hold one - `token_usages` entry per model call (10/5, then 3/2) and a running total - of 13/7 (the positive control: the server has the per-call numbers). + request's own usage is the stub's 3/2/5: the conversation's saved state + (`base_state.json`) holds one `token_usages` entry per model call (10/5, + then 3/2) and GET reports the running total of 13/7 (the positive + control: the server has the per-call numbers; GET's `stats` carries only + the aggregates, see `F07.snapshot-compact-stats`). The second response reports 3/2/5, not the conversation's accumulated 13/7/20: the gateway snapshots the combined metrics before the turn and reports the difference, so an OpenAI client that sums `usage` per request @@ -629,7 +632,7 @@ control-agent-server api GET /api/settings --check conversation_settings.confirm call. The direct run then finishes with its own answer; after that the identical request is 200 with the new turn's answer and the same id, and its message is stored. -- **Continue a paused conversation (`F30.chat-reuse-paused`), known bug #6842.** +- **Continue a paused conversation (`F30.chat-reuse-paused`).** A conversation started through the native API whose first model call hangs is interrupted, so it rests in `paused` with no run active; then a chat completion names it in the header. The request's output is kept in a @@ -652,18 +655,16 @@ control-agent-server api GET /api/settings --check conversation_settings.confirm control-agent-server api GET "/api/conversations/$PCID/agent_final_response" --check response eq QA_F30_PAUSED_OK --quiet control-agent-server sink read --name qa-f30-paused --expect-min 2 --expect-max 2 control-agent-server assert --json "$F30/paused-turn.json" --check status eq 200 \ - --check response.body.choices.0.message.content eq QA_F30_PAUSED_OK --check checks.0.ok eq true # bug: value.status eq 409 && value.response.body.detail contains 'ended with status: paused' + --check response.body.choices.0.message.content eq QA_F30_PAUSED_OK --check checks.0.ok eq true ``` - The turn is saved and run either way: the conversation finishes, holds - exactly one `QA_F30_PAUSED_TURN` user message, its final response is the - stub's `QA_F30_PAUSED_OK`, and the stub got exactly two calls. Expected: - the request answers 200 with that answer and the conversation's id in - `X-OpenHands-ServerConversation-ID` - ([#6842](https://github.com/OpenHands/software-agent-sdk/issues/6842), fix in [#6843](https://github.com/OpenHands/software-agent-sdk/pull/6843)). - Today it answers 409 `Agent run ended with status: paused` within a - fraction of a second, without the header: the gateway reads the resting - `paused` status as the end of the turn it has just started, so a client - that retries the 409 adds a second turn. + The request answers 200 with the turn's own answer, the stub's + `QA_F30_PAUSED_OK`, and the conversation's id in + `X-OpenHands-ServerConversation-ID`; the conversation finishes, holds + exactly one `QA_F30_PAUSED_TURN` user message, and the stub got exactly + two calls. The status the earlier run left behind (`paused`, or `error`) + does not end the wait while the turn's own run is alive; only that run's + own end decides the answer (409 if it pauses or waits for confirmation, + 500 if it ends in `error`). - **The agent runs its own tools (`F30.chat-agent-tools`).** Ask the real model to use the terminal, and pass an OpenAI client tool with `tool_choice: required`. @@ -1108,11 +1109,11 @@ control-agent-server api GET /api/settings --check conversation_settings.confirm conversation whose direct run is active is 409 `Conversation run in progress; message not saved` (`F30.chat-reuse-direct-run-409`). Neither saves the message, so the request can be sent again later. -- A 409 or 500 on a continued conversation does not always mean nothing - ran: on a conversation resting in `paused` or `error` the turn is saved - and runs although the request fails (`F30.chat-reuse-paused`, - [#6842](https://github.com/OpenHands/software-agent-sdk/issues/6842)). - Read the conversation before retrying. +- A turn that continues a conversation resting in `paused` or `error` is + judged by its own run, not by the status the earlier run left behind + (`F30.chat-reuse-paused`). A 409 or 500 then means the turn's own run + paused, waits for confirmation or failed; its message is saved either way, + so read the conversation before retrying. Docker gateway turns for one conversation wait in order inside the container. A turn that cannot start before its deadline returns 504 without saving its message. A new turn waits for a prior run that remains active after a 504. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md b/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md index ac3fe2617d9..e53874ea4e3 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md @@ -24,7 +24,7 @@ Routes: none - `F31.lease-renewed`: while the server lives, the lease's `expires_at` moves forward within 15 s, with the same owner and generation. - `F31.lease-released-graceful`: a graceful restart deletes the lease of every loaded conversation, catalog reads do not claim it again, and the next load claims generation 1 under the new pid. - `F31.lease-dead-owner-takeover`: after a SIGKILL the lease still names the dead pid; the restarted server takes it over on first load with generation 2 and logs the takeover. -- `F31.lease-zombie-owner-takeover`: a restarted server serves its own conversation at once and takes the lease over as generation 2 also when the SIGKILLed owner's pid is still in the process table as a zombie (killed, not reaped) (known bug #6826: 404 until the lease expires). +- `F31.lease-zombie-owner-takeover`: a restarted server serves its own conversation at once and takes the lease over as generation 2 also when the SIGKILLed owner's pid is still in the process table as a zombie (killed, not reaped); the lease records the owner's process start time (`owner_started_at`). - `F31.crash-tool-call`: a conversation SIGKILLed during a tool call is loaded again at startup, comes back `error` with one `AgentErrorEvent` (`recovery_reason: server_restart`) answering the interrupted call and no observation, and the resumed run hands that error to the model. - `F31.state-survives-restarts`: a conversation's title, tags and events, the settings, a stored secret and the LLM profiles read back unchanged after a graceful restart and after a SIGKILL. - `F31.no-plaintext-at-rest`: after a real model run, the provider key, a stored secret and a conversation secret appear nowhere in the run's persisted files or the server log, while the server still hands out the provider key and the stored secret and keeps the conversation secret encrypted in its registry. @@ -189,7 +189,7 @@ Preconditions: first load on the restarted server takes it over at once (same host, pid gone) as generation 2 and logs `Taking over conversation lease ... from dead owner`. -- **Zombie owner takeover (`F31.lease-zombie-owner-takeover`), known bug #6826.** +- **Zombie owner takeover (`F31.lease-zombie-owner-takeover`).** The same crash, seen where nothing reaps the dead server (a container whose PID 1 does not): its pid stays in the process table as a zombie. To get that state on any machine, a helper makes itself a child subreaper @@ -211,7 +211,8 @@ Preconditions: ZPID=$(jq -r .pid "$Z/run.json") ZC=$(control-agent-server conversation start --run "$Z" --tools none --no-autotitle --prompt 'qa-f31-zombie-marker' --no-run --print-id) ZN=$(control-agent-server api GET "/api/conversations/$ZC/events/count" --run "$Z" --expect 200 --field .) - control-agent-server state cat owner_lease.json --run "$Z" --conversation "$ZC" --check owner_pid eq "$ZPID" --check generation eq 1 + control-agent-server state cat owner_lease.json --run "$Z" --conversation "$ZC" --check owner_pid eq "$ZPID" --check generation eq 1 \ + --check owner_started_at exists control-agent-server restart --hard --run "$Z" test "$(ps -o state= -p "$ZPID")" = Z ZPID2=$(jq -r .pid "$Z/run.json") @@ -219,7 +220,7 @@ Preconditions: control-agent-server state cat owner_lease.json --run "$Z" --conversation "$ZC" --check owner_pid eq "$ZPID" --check generation eq 1 \ --check expires_at gt "$(date +%s)" control-agent-server api GET "/api/conversations/$ZC" --run "$Z" --expect 200 --check id eq "$ZC" --quiet - control-agent-server api GET "/api/conversations/$ZC/events/count" --run "$Z" --expect 200 --check . eq "$ZN" --save F31.lease-zombie-owner-takeover/first-load # bug: status eq 404 && response.body.detail contains 'Conversation not found' + control-agent-server api GET "/api/conversations/$ZC/events/count" --run "$Z" --expect 200 --check . eq "$ZN" --save F31.lease-zombie-owner-takeover/first-load control-agent-server state cat owner_lease.json --run "$Z" --conversation "$ZC" --check owner_pid eq "$ZPID2" --check generation eq 2 control-agent-server stop --run "$Z" control-agent-server helper stop qa-f31-zombie-parent @@ -231,10 +232,8 @@ Preconditions: read) and must load it on the first request with the events it had before the crash, taking the lease over as generation 2, as it does when the pid is gone - (`F31.lease-dead-owner-takeover`). The bug: a zombie still answers the - liveness probe, so the lease is honoured and the load answers 404 - `Conversation not found: ` until the lease expires, up to 45 s after - the crash. Evidence lands in `Z`'s `evidence/`. + (`F31.lease-dead-owner-takeover`): a pid in state `Z` (or `X`) counts as + dead. Evidence lands in `Z`'s `evidence/`. - **Crash during a tool call (`F31.crash-tool-call`).** `T`'s model asks for `terminal` `sleep 300`; the server is SIGKILLed while it runs, then the stub is rewritten to call `finish` and the run resumes. @@ -873,12 +872,13 @@ Preconditions: dead pid. A frozen or hung owner (pid alive), an owner on another host, or a lease file from an older server without `owner_host`/`owner_pid` is taken over only after `expires_at`. A TTL below the 15 s renewal interval expires - between renewals. The pid counts as alive while it is a zombie (a killed - server nobody reaped, as under a PID 1 that does not reap), and a lease - that names the restarted server's own pid (a pid reused in a container) is - never treated as dead; both wait for `expires_at` - ([#6826](https://github.com/OpenHands/software-agent-sdk/issues/6826), - `F31.lease-zombie-owner-takeover`). + between renewals. A zombie pid (a killed server nobody reaped, as under a + PID 1 that does not reap) counts as dead (`F31.lease-zombie-owner-takeover`). + On Linux the lease also records the owner's process start time + (`owner_started_at`, from `/proc//stat`), so a live pid whose start + time differs (a pid reused by a restarted server in a container) counts as + dead too; a lease without that field, or a host without `/proc`, falls back + to the pid check alone. - An owner that lost its lease keeps the conversation loaded and answers every write with 409 until idle eviction or a restart closes it, and its reads serve the stale in-memory state; each attempt counts as activity, diff --git a/.agents/skills/verify-agent-server/references/feature-map/README.md b/.agents/skills/verify-agent-server/references/feature-map/README.md index 580e4d9dd96..6f5b501341f 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/README.md +++ b/.agents/skills/verify-agent-server/references/feature-map/README.md @@ -8,9 +8,9 @@ consumers). Read this index before driving the server, then use the matching family file as the recipe. -Maintenance baseline: main@3c66af1d14d0a7fbe74b8dc4218412ec7a877b79 (2026-11-14). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. +Maintenance baseline: main@2aae48b67a57fa852ebca946ac64ee8b8500e889 (2026-11-15). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. -The map has **1002 sub-features**. +The map has **1007 sub-features**. ## Baseline preconditions @@ -66,7 +66,7 @@ them honest: [../maintenance.md](../maintenance.md). | ID | Family | What it covers | Entry points | Needs | Sub-features | |---|---|---|---|---|---| | F01 | [Server status, info and contract](F01-server-status.md) | liveness, health, readiness, server info and its `/` alias, uptime and idle time, OpenAPI document, the static-files mode, DEBUG tracebacks, tool preloading | `GET /alive`, `/health`, `/ready`, `/server_info`, `/`, `/openapi.json` | baseline | 11 | -| F02 | [Authentication and workspace sessions](F02-auth-and-sessions.md) | session keys on HTTP and on WebSockets (first frame, deprecated query and header forms, close codes), several and rotated keys, no-auth and legacy modes, the default bind address, the workspace session cookie, CORS | `X-Session-API-Key`, `POST`/`DELETE /api/auth/workspace-session`, WebSocket auth on every socket | recipes: tmux | 21 | +| F02 | [Authentication and workspace sessions](F02-auth-and-sessions.md) | session keys on HTTP and on WebSockets (first frame, deprecated query and header forms, close codes), several and rotated keys, no-auth and legacy modes, the default bind address, the workspace session cookie, CORS | `X-Session-API-Key`, `POST`/`DELETE /api/auth/workspace-session`, WebSocket auth on every socket | recipes: tmux | 22 | | F03 | [Deferred init (warm-pool dormant mode)](F03-deferred-init.md) | warm-pool dormant mode: the gate on `/api`, `/v1`, `/app-backends` and every socket (checked before auth), init key auth, validation and redaction, rollback, one-shot init, and every field the init body delivers (keys, secret key, paths, worktree root, run limit, CORS, webhooks, env, web URL) | `GET`/`POST /api/init`, `X-Init-API-Key` | git; launch `--no-auth --deferred-init` | 26 | | F04 | [Conversation lifecycle and catalog](F04-conversation-lifecycle.md) | create from settings, an agent profile or an inline agent; client ids, parents, worktrees, start-time secrets, 422 redaction; get, batch get, search, count; PATCH; final response; DELETE; restart | `POST`/`GET`/`PATCH`/`DELETE /api/conversations[/{id}]`, `/search`, `/count`, `/agent_final_response`, `/children` | llm, git | 36 | | F05 | [Run, pause, interrupt and status transitions](F05-run-pause-interrupt.md) | run, pause and interrupt during model and tool calls; status transitions on every view; stuck detection and max_iterations stops; 409 and 429; graceful and SIGKILL restarts | `POST /api/conversations/{id}/run`, `/pause`, `/interrupt` | llm, tmux; launch `--config-json '{"max_concurrent_runs": 1}'` | 32 | @@ -83,7 +83,7 @@ them honest: [../maintenance.md](../maintenance.md). | F16 | [Workspace file serving and VS Code endpoints](F16-workspace-serving-and-vscode.md) | the per-conversation static workspace server (index, content types, ranges, traversal, cookie auth) and the VS Code status and URL routes | `GET /api/conversations/{id}/workspace[/{path}]`, `/api/vscode/*`, `/api/conversations/{id}/vscode/*` | llm, git | 27 | | F17 | [Workspace registry and parents](F17-workspaces-registry.md) | saved workspaces and parents: add, list, delete, validation, persistence, concurrent writers, corrupted files | `GET`/`POST`/`DELETE /api/workspaces`, `/api/workspaces/parents` | git, node | 24 | | F18 | [Server settings, settings schemas and the secrets store](F18-settings-and-secrets.md) | persisted settings with redacted, plaintext and encrypted exposure, sparse PATCH, settings schemas, the named secrets store, encryption at rest, cipher-key changes | `GET`/`PATCH /api/settings`, `/api/settings/agent-schema`, `/conversation-schema`, `/api/settings/secrets[/{name}]` | git, node | 32 | -| F19 | [MCP server settings, connection test and OAuth](F19-mcp.md) | MCP server settings CRUD, connection tests (stdio and HTTP), install-time OAuth jobs and token refresh, a model-driven MCP tool call | `/api/settings/mcp/{key}`, `POST /api/mcp/test`, `/api/mcp/oauth/*` | llm; launch `--webhook-sink` | 30 | +| F19 | [MCP server settings, connection test and OAuth](F19-mcp.md) | MCP server settings CRUD, connection tests (stdio and HTTP), install-time OAuth jobs and token refresh, a model-driven MCP tool call | `/api/settings/mcp/{key}`, `POST /api/mcp/test`, `/api/mcp/oauth/*` | llm; launch `--webhook-sink` | 34 | | F20 | [LLM profiles](F20-llm-profiles.md) | LLM profiles: save, read, validate against real and stubbed providers, activate, rename, delete, name rules, limits, store contention, restart | `/api/profiles[/{name}]`, `/validate`, `/activate`, `/rename` | llm, node | 37 | | F21 | [Agent profiles](F21-agent-profiles.md) | agent profiles (OpenHands and ACP kinds): CRUD, default seed, activation, rename, delete, materialize, conversations started from a profile, store contention | `/api/agent-profiles[/{name}]`, `/rename`, `/{id}/activate`, `/materialize` | llm | 32 | | F22 | [Meta-profiles (model router)](F22-meta-profiles.md) | meta-profiles (model router): save, read, activate, delete, settings reflection, routing in a conversation | `/api/meta-profiles[/{name}]`, `/activate` | llm, node | 24 | diff --git a/.agents/skills/verify-sdk/programs/S01/loop.py b/.agents/skills/verify-sdk/programs/S01/loop.py index 522e2b4e29e..6677cbe24d9 100644 --- a/.agents/skills/verify-sdk/programs/S01/loop.py +++ b/.agents/skills/verify-sdk/programs/S01/loop.py @@ -7,6 +7,9 @@ budget max_budget_per_run with a priced model thresholds custom stuck_detection_thresholds reasoning-only a reply with only reasoning gets a nudge + content-free a real LLM's reply with no text, tool call or reasoning is + retried inside the call, not nudged; reasoning-only still + nudged; exhausted retries end the run in ERROR parallel tool_concurrency_limit, order, a failing call, finish truncation locks resource keys, undeclared tools, declared-safe tools tool-exception an exception inside a tool executor @@ -225,6 +228,62 @@ def completion(self, *args, **kwargs): } conversation.close() +elif mode == "content-free": + # A real LLM against the local endpoint: TestLLM bypasses the response + # validation that decides a reply is content-free. + def provider_run(script: list[dict], num_retries: int) -> dict: + with FakeProvider(script) as server: + llm = LLM( + model="openai/gpt-4o-mini", + base_url=server.url, + api_key=SecretStr("sk-test"), + usage_id="agent", + num_retries=num_retries, + temperature=0.0, + retry_min_wait=0, + retry_max_wait=0, + retry_multiplier=0.1, + ) + conversation = conversation_for(Agent(llm=llm, tools=[])) + conversation.send_message("Answer.") + try: + conversation.run() + raised = None + except Exception as exc: + raised = type(exc).__name__ + events = conversation.state.events + messages = [e for e in events if isinstance(e, MessageEvent)] + errors = [e for e in events if isinstance(e, ConversationErrorEvent)] + result = { + "status": conversation.state.execution_status.value, + "raised": raised, + "requests": len(server.requests), + "temperatures": [r["body"].get("temperature") for r in server.requests], + "nudges": sum(m.source == "environment" for m in messages), + "agent_messages": [ + "".join(getattr(c, "text", "") for c in m.llm_message.content) + for m in messages + if m.source == "agent" + ], + "error_codes": [e.code for e in errors], + "error_classifications": [ + e.classification.model_dump(mode="json") + if e.classification + else None + for e in errors + ], + } + conversation.close() + return result + + out["empty_then_answer"] = provider_run( + [{"text": ""}, {"text": ""}, {"text": "The answer is ready."}], 5 + ) + out["reasoning_only"] = provider_run( + [{"reasoning": "I am only thinking."}, {"finish": "done"}], 5 + ) + out["exhausted"] = provider_run([{"text": ""}] * 3, 2) + elif mode == "parallel": calls = [sleeper_call(f"s{i}", 1.0) for i in range(4)] sequential, seq_secs = run_sleepers(calls, 1) diff --git a/.agents/skills/verify-sdk/programs/S01/step_errors.json b/.agents/skills/verify-sdk/programs/S01/step_errors.json index 1265fa71c49..47a18e1d5bc 100644 --- a/.agents/skills/verify-sdk/programs/S01/step_errors.json +++ b/.agents/skills/verify-sdk/programs/S01/step_errors.json @@ -3,6 +3,5 @@ {"tool": "terminal", "args": {"bogus": 1}}, {"tool": "task_tracker", "args": {"command": "plan", "task_list": "[{\"title\": \"repaired\", \"status\": \"todo\"}]"}}, {"error": "LLMContentPolicyViolationError", "message": "Output blocked by content filtering policy"}, - {"text": ""}, {"finish": "survived every error"} ] diff --git a/.agents/skills/verify-sdk/programs/S02/fake_provider.py b/.agents/skills/verify-sdk/programs/S02/fake_provider.py index b27158db1ed..6c51ceabbdf 100644 --- a/.agents/skills/verify-sdk/programs/S02/fake_provider.py +++ b/.agents/skills/verify-sdk/programs/S02/fake_provider.py @@ -20,6 +20,8 @@ {"finish": "message"} the finish tool {"status": 429, "message": "..."} an HTTP error {"empty": true} a reply with no choices + {"text": "", "finish_reason": "length"} the choice's finish_reason + {"raw": "...", "raw_status": 200} a non-JSON body {"delay": 2.0, ...} wait before answering {"stall": 2.0, ...} (stream) pause after two events {"thinking": "...", "redacted": "..."} Anthropic thinking blocks @@ -122,6 +124,14 @@ def do_POST(self) -> None: return if step.get("delay"): time.sleep(float(step["delay"])) + if "raw" in step: + data = str(step["raw"]).encode() + self.send_response(int(step.get("raw_status", 200))) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + return if "status" in step: self._send( int(step["status"]), @@ -249,7 +259,9 @@ def chat_reply(self, step: dict[str, Any]) -> dict[str, Any]: { "index": 0, "message": message, - "finish_reason": "tool_calls" if calls else "stop", + "finish_reason": step.get( + "finish_reason", "tool_calls" if calls else "stop" + ), } ], } @@ -314,7 +326,7 @@ def chunk(delta: dict[str, Any], finish: str | None = None) -> dict[str, Any]: ), ) ) - last = chunk({}, "tool_calls" if calls else "stop") + last = chunk({}, step.get("finish_reason", "tool_calls" if calls else "stop")) last["usage"] = { "prompt_tokens": usage["prompt_tokens"], "completion_tokens": usage["completion_tokens"], diff --git a/.agents/skills/verify-sdk/programs/S02/llm_calls.py b/.agents/skills/verify-sdk/programs/S02/llm_calls.py index ec4f9acc886..b02f5128b52 100644 --- a/.agents/skills/verify-sdk/programs/S02/llm_calls.py +++ b/.agents/skills/verify-sdk/programs/S02/llm_calls.py @@ -2,10 +2,14 @@ against a local endpoint that answers from a script. One scenario per mode. Usage: llm_calls.py MODE - retries transient errors retried, others raised at once, listener - no-response empty replies retried with a temperature bump, then raised + retries transient errors retried, others raised at once, listener; + a 2xx body that is not JSON retried + no-response empty and content-free replies retried with a temperature + bump, then raised; a length-capped empty reply and a + reasoning-only reply returned errors provider errors mapped to the SDK's typed exceptions - timeouts per-attempt timeout and async stream idle timeout + timeouts per-attempt timeout and async stream idle timeout; the + timeout retry budget and the final error's summary streaming sync/async streaming callbacks, and stream without callback prompted text replies parsed back into tool calls (non-native) cache-too-small a prompt-cache rejection retried without cache markers @@ -60,6 +64,20 @@ def attempt(fn): return None, type(exc).__name__, round(time.monotonic() - started, 1) +def attempt_message(fn): + """Run fn; return (result, error type name, error text, seconds).""" + started = time.monotonic() + try: + return fn(), None, None, round(time.monotonic() - started, 1) + except Exception as exc: + return ( + None, + type(exc).__name__, + str(exc), + round(time.monotonic() - started, 1), + ) + + def chat_llm(server: FakeProvider, **kw) -> LLM: return LLM( model=kw.pop("model", "openai/gpt-4o-mini"), @@ -110,6 +128,24 @@ def chat_llm(server: FakeProvider, **kw) -> LLM: lambda: chat_llm(s, num_retries=3, **FAST).completion(MESSAGES) ) out["budget_denial"] = {"error": err, "requests": len(s.requests)} + # A 2xx reply whose body is not JSON: LiteLLM's OpenAI-compatible handler + # (`hosted_vllm/`) raises APIError with the 2xx status. A 4xx body that is + # not JSON is not retried. + for name, status in (("malformed_2xx", 200), ("malformed_4xx", 400)): + with FakeProvider( + [{"raw": "upstream glitch", "raw_status": status}] + + [{"text": "hi"}] + ) as s: + r, err, _ = attempt( + lambda: chat_llm( + s, model="hosted_vllm/verify-model", num_retries=3, **FAST + ).completion(MESSAGES) + ) + out[name] = { + "text": text_of(r) if r else None, + "error": err, + "requests": len(s.requests), + } elif mode == "no-response": with FakeProvider([{"empty": True}, {"empty": True}, {"text": "hi"}]) as s: @@ -130,6 +166,38 @@ def chat_llm(server: FakeProvider, **kw) -> LLM: llm = chat_llm(s, num_retries=2, **FAST) r, err, _ = attempt(lambda: asyncio.run(llm.acompletion(MESSAGES))) out["chat_async"] = {"text": text_of(r) if r else None, "error": err} + # A choice with empty text, no tool call and no reasoning, plain and + # streamed. + for name, stream in (("content_free", False), ("content_free_stream", True)): + with FakeProvider([{"text": ""}, {"text": "hi"}]) as s: + llm = chat_llm(s, num_retries=3, temperature=0.0, stream=stream, **FAST) + kwargs = {"on_token": lambda _c: None} if stream else {} + r, err, _ = attempt(lambda: llm.completion(MESSAGES, **kwargs)) + out[name] = { + "text": text_of(r) if r else None, + "error": err, + "requests": len(s.requests), + "temperatures": [q["body"].get("temperature") for q in s.requests], + } + with FakeProvider([{"text": "", "finish_reason": "length"}, {"text": "hi"}]) as s: + r, err, _ = attempt( + lambda: chat_llm(s, num_retries=3, **FAST).completion(MESSAGES) + ) + out["length_capped"] = { + "text": text_of(r) if r else None, + "error": err, + "requests": len(s.requests), + } + with FakeProvider([{"reasoning": "only thinking"}, {"text": "hi"}]) as s: + r, err, _ = attempt( + lambda: chat_llm(s, num_retries=3, **FAST).completion(MESSAGES) + ) + out["reasoning_only"] = { + "text": text_of(r) if r else None, + "reasoning": r.message.reasoning_content if r else None, + "error": err, + "requests": len(s.requests), + } # On the Responses path a no-response is a stream that ends without its # completed event. incomplete = {"text": "lost", "incomplete": True} @@ -224,6 +292,67 @@ async def on_token(chunk) -> None: lambda: asyncio.run(llm.acompletion(MESSAGES, on_token=on_token)) ) out["stream_idle"] = {"error": err, "seconds": secs} + + # Every attempt reaches the timeout: retrying stops after two timeouts' + # worth of time, whatever num_retries allows. + def summary(text): + text = text or "" + return { + "names_limit": "within the LLM timeout of 1s" in text, + "names_attempts": "gave up after 2 attempts" in text, + "names_settings": "higher timeout in the LLM settings" in text, + } + + never = {"delay": 30, "text": "never sent"} + with FakeProvider([never] * 6) as s: + _, err, text, secs = attempt_message( + lambda: chat_llm(s, num_retries=5, timeout=1, **FAST).completion(MESSAGES) + ) + out["hard"] = { + "error": err, + "seconds": secs, + "requests": len(s.requests), + "message": text, + **summary(text), + } + with FakeProvider([never] * 6) as s: + llm = chat_llm( + s, num_retries=5, timeout=1, stream=True, stream_idle_timeout=None, **FAST + ) + + async def ignore(chunk) -> None: + pass + + _, err, text, secs = attempt_message( + lambda: asyncio.run(llm.acompletion(MESSAGES, on_token=ignore)) + ) + out["hard_async"] = { + "error": err, + "seconds": secs, + "requests": len(s.requests), + "message": text, + "hard_timeout": "LLM hard timeout after 1 seconds" in (text or ""), + **summary(text), + } + # Timeouts far below `timeout` (a stream idle timeout of 0.3 s against a + # 30 s limit) keep every attempt. + with FakeProvider([{"stall": 5, "text": "stalled"}] * 4) as s: + llm = chat_llm( + s, num_retries=3, timeout=30, stream=True, stream_idle_timeout=0.3, **FAST + ) + + async def ignore(chunk) -> None: + pass + + _, err, text, secs = attempt_message( + lambda: asyncio.run(llm.acompletion(MESSAGES, on_token=ignore)) + ) + out["quick"] = { + "error": err, + "seconds": secs, + "requests": len(s.requests), + "message": text, + } # The same stall inside an async agent run ends the conversation in ERROR. from openhands.sdk import Agent, Conversation diff --git a/.agents/skills/verify-sdk/programs/S04/pause_during_startup.py b/.agents/skills/verify-sdk/programs/S04/pause_during_startup.py new file mode 100644 index 00000000000..d0376d6b212 --- /dev/null +++ b/.agents/skills/verify-sdk/programs/S04/pause_during_startup.py @@ -0,0 +1,85 @@ +"""pause() while run() is still making the agent ready (issue #6848). + +A conversation restored from `persistence_dir` makes its agent ready on the +first `run()`: here that starts a stdio MCP server (`S09/mcp_server.py`) that +takes 2 s to come up. `run()` goes to a thread, `pause()` follows 0.5 s later, +while the server is still starting. The model is a scripted `TestLLM` with one +`finish` step; the program reports the status around the pause, the LLM calls +made after it, and what a later `run()` does. + +Usage: pause_during_startup.py +""" + +import sys +import threading +import time +import uuid +from pathlib import Path + +from sdk_verify import emit, make_llm, run_paths + +from openhands.sdk import Agent, Conversation +from openhands.sdk.event import PauseEvent +from openhands.sdk.mcp import MCPServer + + +paths = run_paths() +SERVER = str(Path(__file__).parent.parent / "S09" / "mcp_server.py") +START_SECONDS = 2.0 +PAUSE_AFTER_SECONDS = 0.5 +cid = uuid.uuid4() + + +def open_conversation(llm, mcp_config): + agent = Agent(llm=llm, tools=[], mcp_config=mcp_config) + return Conversation( + agent=agent, + workspace=str(paths["workspace"]), + persistence_dir=str(paths["conversations"]), + conversation_id=cid, + visualizer=None, + delete_on_close=False, + ) + + +# The first conversation only stores the message; it has no MCP server, so +# only the restored one waits for a server to start. +first = open_conversation(make_llm(script=[]), {}) +first.send_message("Say done.") +first.close() + +llm = make_llm(script=[{"finish": "done"}]) +slow_server = MCPServer( + command=sys.executable, + args=[SERVER, "stdio", "--slow-start", str(START_SECONDS)], +) +conversation = open_conversation(llm, {"slow": slow_server}) +out = {"ok": True, "status_after_restore": conversation.state.execution_status.value} + +runner = threading.Thread(target=conversation.run) +started = time.monotonic() +runner.start() +time.sleep(PAUSE_AFTER_SECONDS) +# Holding the state's (reentrant, FIFO) lock across pause() reads the status +# pause() left before the run thread can take the lock back. +with conversation.state: + conversation.pause() + calls_at_pause = llm.call_count + out["pause_returned_after_s"] = round(time.monotonic() - started, 2) + out["status_after_pause"] = conversation.state.execution_status.value +runner.join(timeout=60) +out |= { + "run_returned_after_s": round(time.monotonic() - started, 2), + "status_after_run": conversation.state.execution_status.value, + "llm_calls_after_pause": llm.call_count - calls_at_pause, + "pause_events": sum(isinstance(e, PauseEvent) for e in conversation.state.events), +} + +conversation.run() +out |= { + "status_after_resume": conversation.state.execution_status.value, + "llm_calls_total": llm.call_count, + "script_left": llm.remaining_responses, +} +conversation.close() +emit(out) diff --git a/.agents/skills/verify-sdk/programs/S04/views.py b/.agents/skills/verify-sdk/programs/S04/views.py index e3ae8d69d30..8e464218537 100644 --- a/.agents/skills/verify-sdk/programs/S04/views.py +++ b/.agents/skills/verify-sdk/programs/S04/views.py @@ -6,6 +6,8 @@ Usage: views.py MODE visualizer default Rich output, None, a class, an instance, a custom one fork fork(): copy, independence, metrics, agent, title, tags, reload + fork-secrets fork() of a cipher-backed conversation: the api key and the + conversation secrets on the fork's disk, a reload, a step branch parent ids, fork(from_event_id=...), navigate_to(...) goal run_goal with a judge LLM: complete, capped, unparsable verdict final get_agent_final_response for finish, text, nothing, both @@ -225,6 +227,136 @@ def priced(srv, model="openai/gpt-4o-mini") -> LLM: out["fork_events"] = fork_events reloaded.close() +elif mode == "fork-secrets": + from openhands.sdk.conversation import LocalConversation + from openhands.sdk.utils.cipher import Cipher + + API_KEY = "sk-fork-verify-123" + SECRETS = {"VERIFY_TOKEN": "tok-fork-123", "LATER_TOKEN": "later-token-4567"} + cipher = Cipher("verify-fork-cipher-key") + + def on_disk(raw, plain): + """How a secret field sits in base_state.json.""" + if raw == plain: + return "plaintext" + if isinstance(raw, str): + try: + decrypted = cipher.decrypt(raw) + except Exception: + decrypted = None + if decrypted is not None and decrypted.get_secret_value() == plain: + return "encrypted" + return f"other: {str(raw)[:40]}" + + def base_state(conv): + return json.loads((BASE / conv.id.hex / "base_state.json").read_text()) + + def disk_view(conv): + state = base_state(conv) + sources = state.get("secret_registry", {}).get("secret_sources", {}) + return { + "api_key": on_disk(state["agent"]["llm"].get("api_key"), API_KEY), + **{ + name: on_disk((sources.get(name) or {}).get("value"), value) + for name, value in SECRETS.items() + }, + } + + with FakeProvider( + [ + term( + 'echo "v=$VERIFY_TOKEN l=$LATER_TOKEN";' + ' printf %s "$VERIFY_TOKEN" | wc -c; printf %s "$LATER_TOKEN" | wc -c', + summary="s", + security_risk="LOW", + ), + {"finish": "fork done"}, + ] + ) as server: + llm = LLM( + model="openai/gpt-4o-mini", + base_url=server.url, + api_key=SecretStr(API_KEY), + usage_id="agent", + num_retries=1, + ) + source = LocalConversation( + agent=Agent(llm=llm, tools=[Tool(name=TerminalTool.name)]), + workspace=WORKSPACE, + persistence_dir=str(BASE), + cipher=cipher, + secrets={"VERIFY_TOKEN": SECRETS["VERIFY_TOKEN"]}, + visualizer=None, + delete_on_close=False, + ) + source.update_secrets({"LATER_TOKEN": SECRETS["LATER_TOKEN"]}) + source.send_message("Echo the tokens, then finish.") + fork = source.fork() + out["source_on_disk"] = disk_view(source) + out["fork_on_disk"] = disk_view(fork) + out["fork_registry_has"] = sorted(fork.state.secret_registry.secret_sources) + fork_id = fork.id + fork.close() + source.close() + + # Reopen the fork from disk alone (agent=None), as a restarted server does. + reloaded = LocalConversation( + agent=None, + workspace=WORKSPACE, + persistence_dir=str(BASE), + conversation_id=fork_id, + cipher=cipher, + visualizer=None, + delete_on_close=False, + ) + key = reloaded.agent.llm.api_key + out["reloaded_api_key_is_real"] = ( + key is not None and key.get_secret_value() == API_KEY + ) + out["reloaded_registry"] = { + name: reloaded.state.secret_registry.get_secret_value(name) == value + for name, value in SECRETS.items() + } + reloaded.run() + observations = [ + e + for e in reloaded.state.events + if isinstance(e, ObservationEvent) and e.tool_name == TerminalTool.name + ] + text = observations[-1].observation.text if observations else "" + out |= { + "reloaded_status": reloaded.state.execution_status.value, + "requests": len(server.requests), + "authorization": [ + r["headers"].get("authorization") == f"Bearer {API_KEY}" + for r in server.requests + ], + "command_output": [line.strip() for line in text.splitlines()[:3]], + "values_in_output": any(v in text for v in SECRETS.values()), + "fork_on_disk_after_run": disk_view(reloaded), + } + reloaded.close() + + # A conversation without a cipher still forks with its secrets redacted. + plain = LocalConversation( + agent=Agent( + llm=LLM(model="openai/gpt-4o-mini", api_key=SecretStr(API_KEY)), + tools=[], + ), + workspace=WORKSPACE, + persistence_dir=str(BASE), + visualizer=None, + ) + plain_fork = plain.fork() + raw = base_state(plain_fork)["agent"]["llm"].get("api_key") + out["no_cipher_fork_api_key"] = ( + "redacted" + if isinstance(raw, str) and set(raw) == {"*"} + else on_disk(raw, API_KEY) + ) + plain_fork.close() + plain.close() + elif mode == "branch": conversation = conversation_for( [ diff --git a/.agents/skills/verify-sdk/programs/S05/llm_failures.py b/.agents/skills/verify-sdk/programs/S05/llm_failures.py new file mode 100644 index 00000000000..d6c6bef2b1d --- /dev/null +++ b/.agents/skills/verify-sdk/programs/S05/llm_failures.py @@ -0,0 +1,99 @@ +"""Run conversations whose real `LLM` keeps failing against the local +endpoint, and report the `ConversationErrorEvent` each run ends with and its +classification. One scenario per mode. + +Usage: llm_failures.py MODE + exhausted every attempt reaches the hard timeout (`timeout=1`, + `num_retries=5`); every attempt is content-free; plus the + classification of a single quick timeout's text + quick every attempt hits a stream idle timeout far below `timeout` + (`stream_idle_timeout=0.3`, `timeout=30`, `num_retries=3`) +""" + +import asyncio +import sys + +from pydantic import SecretStr +from S02.fake_provider import FakeProvider +from sdk_verify import emit, run_paths + +from openhands.sdk import LLM, Agent, Conversation +from openhands.sdk.event.conversation_error import ConversationErrorEvent +from openhands.sdk.event.error_classification import classify_error + + +mode = sys.argv[1] +out: dict = {"ok": True, "mode": mode} +FAST = {"retry_min_wait": 0, "retry_max_wait": 0, "retry_multiplier": 0.1} + + +def failing_run(script: list[dict], streamed: bool = False, **llm_kwargs) -> dict: + with FakeProvider(script) as server: + llm = LLM( + model="openai/gpt-4o-mini", + base_url=server.url, + api_key=SecretStr("sk-test"), + usage_id="agent", + stream=streamed, + **FAST, + **llm_kwargs, + ) + conversation = Conversation( + agent=Agent(llm=llm, tools=[]), + workspace=str(run_paths()["workspace"]), + visualizer=None, + token_callbacks=[lambda _chunk: None] if streamed else None, + ) + conversation.send_message("Answer.") + try: + if streamed: + asyncio.run(conversation.arun()) + else: + conversation.run() + raised = None + except Exception as exc: + raised = type(exc).__name__ + errors = [ + e + for e in conversation.state.events + if isinstance(e, ConversationErrorEvent) + ] + result = { + "status": conversation.state.execution_status.value, + "raised": raised, + "requests": len(server.requests), + "errors": len(errors), + "code": errors[0].code if errors else None, + "detail": errors[0].detail if errors else None, + # The classification as the serialized event carries it. + "classification": ( + errors[0].model_dump(mode="json").get("classification") + if errors + else None + ), + } + conversation.close() + return result + + +if mode == "exhausted": + never = {"delay": 30, "text": "never sent"} + out["hard_timeout"] = failing_run([never] * 6, timeout=1, num_retries=5) + out["content_free"] = failing_run([{"text": ""}] * 3, num_retries=2) + single = classify_error("LLMTimeoutError", "litellm.Timeout: Connection timed out.") + out["single_quick_timeout"] = single.model_dump(mode="json") + +elif mode == "quick": + stall = {"stall": 5, "text": "stalled"} + out["quick_timeout"] = failing_run( + [stall] * 4, + streamed=True, + timeout=30, + stream_idle_timeout=0.3, + num_retries=3, + ) + +else: + raise SystemExit(f"unknown mode {mode}") + +emit(out) diff --git a/.agents/skills/verify-sdk/programs/S20/retry_log.py b/.agents/skills/verify-sdk/programs/S20/retry_log.py new file mode 100644 index 00000000000..2e1ac16abf1 --- /dev/null +++ b/.agents/skills/verify-sdk/programs/S20/retry_log.py @@ -0,0 +1,93 @@ +"""Call a real `LLM` against the local endpoint while failing requests are +retried, and report the SDK's retry log lines, as a program's log handler +sees them. Every provider error body carries a fake secret marker. + +Cases: a 503 then a reply (`num_retries=3`); a 400 (not retried); a 429 that +never clears (`num_retries=2`); a 2xx body that is not JSON, through the +OpenAI-compatible `hosted_vllm/` route, then a reply. +""" + +import logging +import re + +from pydantic import SecretStr +from S02.fake_provider import FakeProvider +from sdk_verify import emit + +from openhands.sdk import LLM +from openhands.sdk.llm import Message, TextContent + + +MARKER = "sk-leak-RETRYLOG-7731" +MESSAGES = [Message(role="user", content=[TextContent(text="Say hi.")])] +FAST = {"retry_min_wait": 0, "retry_max_wait": 0, "retry_multiplier": 0.1} +PREFIX = "LLM request failed:" + + +class Capture(logging.Handler): + def __init__(self) -> None: + super().__init__(level=logging.DEBUG) + self.records: list[logging.LogRecord] = [] + + def emit(self, record: logging.LogRecord) -> None: + self.records.append(record) + + +capture = Capture() +logging.getLogger().addHandler(capture) + + +def full_text(record: logging.LogRecord) -> str: + text = record.getMessage() + if record.exc_info: + text += logging.Formatter().formatException(record.exc_info) + return text + + +def run_case(script: list[dict], model: str = "openai/gpt-4o-mini", **kw) -> dict: + start = len(capture.records) + with FakeProvider(script) as server: + llm = LLM( + model=model, + base_url=server.url, + api_key=SecretStr("sk-test"), + usage_id="direct", + **FAST, + **kw, + ) + try: + llm.completion(MESSAGES) + error = None + except Exception as exc: + error = type(exc).__name__ + requests = len(server.requests) + records = capture.records[start:] + lines = [r for r in records if r.getMessage().startswith(PREFIX)] + return { + "error": error, + "requests": requests, + "lines": len(lines), + "levels": [r.levelname for r in lines], + "fields": [dict(re.findall(r"(\w+)=(\S+)", r.getMessage())) for r in lines], + "marker_in_logs": any(MARKER in full_text(r) for r in records), + } + + +secret_body = f"upstream said: Authorization: Bearer {MARKER}" +out: dict = {"ok": True} +out["transient"] = run_case( + [{"status": 503, "message": secret_body}, {"text": "hi"}], num_retries=3 +) +out["non_retryable"] = run_case( + [{"status": 400, "message": f"Invalid value {MARKER}"}], num_retries=3 +) +out["exhausted"] = run_case( + [{"status": 429, "message": f"Rate limit reached {MARKER}"}] * 2, num_retries=2 +) +out["malformed_2xx"] = run_case( + [{"raw": f"{secret_body}"}, {"text": "hi"}], + model="hosted_vllm/verify-model", + num_retries=3, +) +out["marker_in_any_log"] = any(MARKER in full_text(r) for r in capture.records) +emit(out) diff --git a/.agents/skills/verify-sdk/references/feature-map/README.md b/.agents/skills/verify-sdk/references/feature-map/README.md index 37725d5e792..fb206cc4216 100644 --- a/.agents/skills/verify-sdk/references/feature-map/README.md +++ b/.agents/skills/verify-sdk/references/feature-map/README.md @@ -10,9 +10,9 @@ agent-server map and follows its layout. Family IDs here are `Snn`, so they never collide with that map's `Fnn`. -Maintenance baseline: main@3c66af1d14d0a7fbe74b8dc4218412ec7a877b79 (2026-11-14). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. +Maintenance baseline: main@2aae48b67a57fa852ebca946ac64ee8b8500e889 (2026-11-15). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. -The map has **476 sub-features**, **31 of them untested** (no unit or cross +The map has **479 sub-features**, **32 of them untested** (no unit or cross test exercises the behavior; most of those have an example script). ## What each sub-feature links @@ -74,7 +74,7 @@ an interactive login are labeled **blocked**. | S01 | [Agent step loop](S01-agent-loop.md) | step handling (tool calls, text, empty replies), iteration and budget limits, stuck detector, parallel tools, messages sent mid-run, image input, `ask_agent`, structured output, recovery from bad tool calls and context overflow | `Conversation(max_iteration_per_run=, stuck_detection=, max_budget_per_run=)`, `Agent(tool_concurrency_limit=)`, `ask_agent()`, `response_schema` | llm, tmux, network | 24 | 1 | | S02 | [LLM core](S02-llm-core.md) | completion, responses and `generate`; streaming; reasoning and thinking blocks; native and prompted tool calling; metrics, stats and pricing; retries and error mapping; caching, token counts, image inlining, completion logs | `LLM(...)`, `llm.completion`/`responses`, `Conversation(token_callbacks=)`, `llm.metrics` | llm, network | 28 | 0 | | S03 | [LLM registry, routing, fallback, profiles and model switching](S03-llm-routing-profiles.md) | `LLMRegistry`, multimodal and random routers, `FallbackStrategy`, `LLMProfileStore`, `switch_profile`/`switch_llm`, the switch-LLM tool, `route_task_to_model`, subscription login | `LLMRegistry`, `MultimodalRouter`, `FallbackStrategy`, `LLMProfileStore`, `LLM.subscription_login()` | llm, network | 26 | 3 | -| S04 | [Local conversation lifecycle](S04-local-conversation.md) | factory and ids, persistence and resume, callbacks, close, pause, `arun` and interrupt, secrets, visualizers, fork and branches, goal loop, final response, stats, title and tags | `Conversation(persistence_dir=, callbacks=, visualizer=, secrets=)`, `run`/`pause`/`arun`/`interrupt`/`fork`/`run_goal` | llm, tmux, network, uvx | 27 | 1 | +| S04 | [Local conversation lifecycle](S04-local-conversation.md) | factory and ids, persistence and resume, callbacks, close, pause, `arun` and interrupt, secrets, visualizers, fork and branches, goal loop, final response, stats, title and tags | `Conversation(persistence_dir=, callbacks=, visualizer=, secrets=)`, `run`/`pause`/`arun`/`interrupt`/`fork`/`run_goal` | llm, tmux, network, uvx | 29 | 2 | | S05 | [Events and the event log](S05-events.md) | event model, JSON round trip, conversion to LLM messages and OpenAI formats, unmatched actions, the `EventLog`, error classification, completion and token events | `Event.model_validate_json`, `events_to_messages`, `conversation.state.events` | llm, tmux | 18 | 1 | | S06 | [Agent context, system prompt, condensers and memory](S06-context-condensers.md) | `AgentContext`, system prompt sections and vendor files, persona, persistent memory, the condenser `View`, `LLMSummarizingCondenser`, `condense()`, overflow condensation, no-op and pipeline condensers | `Agent(agent_context=, system_prompt=, condenser=)`, `AgentContext(load_memory=)`, `conversation.condense()` | llm, tmux | 24 | 1 | | S07 | [Skills](S07-skills.md) | skill sources (programmatic, Markdown, AGENTS.md, SKILL.md, user, project, public, installed), keyword, task and path triggers, ``, `invoke_skill`, inline commands, disabling, install and marketplace | `AgentContext(skills=, load_*_skills=, disabled_skills=)`, `.agents/skills/`, `install_skill` | llm, git, network | 24 | 0 | @@ -90,7 +90,7 @@ an interactive login are labeled **blocked**. | S17 | [Custom tools, client-defined tools and the tool registry](S17-custom-and-client-tools.md) | custom tool classes and tool sets, registration and naming, tool schemas, injected fields, catalog seal, module qualnames, client tools, `execute_tool` | `register_tool`, `Tool(name, params)`, `Conversation(client_tools=)`, `execute_tool` | llm | 24 | 1 | | S18 | [Workspaces and sandbox backends](S18-workspace-backends.md) | the `Workspace` factory, `LocalWorkspace`, `RemoteWorkspace`, and the Docker, API-remote, Cloud, Apptainer and agent-sandbox backends (start, health, commands, files, pause, cleanup) | `Workspace(...)`, `LocalWorkspace`, `openhands.workspace.*Workspace` | docker, apptainer, cloud, k8s, network, llm, git | 24 | 3 | | S19 | [Agent settings, persisted schema migrations and agent profiles](S19-settings.md) | `OpenHandsAgentSettings`, ACP settings, schema migrations v0 to v8, settings diffs and export, the persisted-settings compatibility gate, agent profiles and their store | `openhands.sdk.settings`, `openhands.sdk.profiles`, `~/.openhands/agent-profiles/` | llm, tmux, network | 25 | 0 | -| S20 | [Observability, tracing and logging](S20-observability.md) | enabling tracing, Laminar and OTLP backends, conversation, step, tool and LLM spans, sub-agent trace links, automation helpers, the logger | `LMNR_*`/`OTEL_*` env vars, `Conversation(observability_*=)`, `get_logger` | llm, network | 21 | 5 | +| S20 | [Observability, tracing and logging](S20-observability.md) | enabling tracing, Laminar and OTLP backends, conversation, step, tool and LLM spans, sub-agent trace links, automation helpers, the logger | `LMNR_*`/`OTEL_*` env vars, `Conversation(observability_*=)`, `get_logger` | llm, network | 22 | 5 | | S21 | [ACP agent (in-process)](S21-acp-agent.md) | construction, provider registry, handshake, turns and tool-call events, permission auto-approval, timeouts and cancel, session resume, model switching, usage, MCP forwarding, secrets, cleanup | `ACPAgent(acp_command=...)`, `switch_acp_model`, `ACPAgentSettings` | node, network, llm | 27 | 2 | ## Not mapped diff --git a/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md b/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md index c68484a0477..666688be93f 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md +++ b/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md @@ -1,6 +1,6 @@ # Agent step loop -A Python program builds `Agent(llm=..., tools=[...])`, wraps it in `Conversation(agent=..., workspace=...)`, calls `send_message(...)` and then `run()` (or `await arun()`). `run()` repeatedly calls `Agent.step()`: each step builds the LLM message list from the event view (condensing first if the condenser asks), calls `LLM.generate`, and dispatches the reply. Tool calls become `ActionEvent`s that are executed (sequentially by default, or concurrently up to `Agent.tool_concurrency_limit`) and answered by `ObservationEvent`s; a plain text reply or a `FinishAction` sets the status to `FINISHED`; a reasoning-only or empty reply gets a corrective nudge and the loop continues. The loop stops on `FINISHED`, `PAUSED`, `WAITING_FOR_CONFIRMATION`, `STUCK` (the stuck detector saw a repeating pattern), or `ERROR` with a `ConversationErrorEvent` (`MaxIterationsReached` after `max_iteration_per_run` steps, `MaxBudgetReached` after `max_budget_per_run` dollars). Messages sent from another thread while `run()` is busy are picked up by the next step. Images go in as `ImageContent` inside a `Message`; a non-vision model gets a capability message instead of a call. `conversation.ask_agent(question)` asks a side question that leaves no trace in the conversation. A tool spec with `params={"response_schema": Model}` turns that tool's arguments (typically `finish`) into typed structured output. +A Python program builds `Agent(llm=..., tools=[...])`, wraps it in `Conversation(agent=..., workspace=...)`, calls `send_message(...)` and then `run()` (or `await arun()`). `run()` repeatedly calls `Agent.step()`: each step builds the LLM message list from the event view (condensing first if the condenser asks), calls `LLM.generate`, and dispatches the reply. Tool calls become `ActionEvent`s that are executed (sequentially by default, or concurrently up to `Agent.tool_concurrency_limit`) and answered by `ObservationEvent`s; a plain text reply or a `FinishAction` sets the status to `FINISHED`; a reasoning-only reply gets a corrective nudge and the loop continues, while a reply with no text, no tool call and no reasoning never reaches the agent: the LLM call retries it. The loop stops on `FINISHED`, `PAUSED`, `WAITING_FOR_CONFIRMATION`, `STUCK` (the stuck detector saw a repeating pattern), or `ERROR` with a `ConversationErrorEvent` (`MaxIterationsReached` after `max_iteration_per_run` steps, `MaxBudgetReached` after `max_budget_per_run` dollars). Messages sent from another thread while `run()` is busy are picked up by the next step. Images go in as `ImageContent` inside a `Message`; a non-vision model gets a capability message instead of a call. `conversation.ask_agent(question)` asks a side question that leaves no trace in the conversation. A tool spec with `params={"response_schema": Model}` turns that tool's arguments (typically `finish`) into typed structured output. Source: `openhands-sdk/openhands/sdk/agent/agent.py`, `openhands-sdk/openhands/sdk/agent/base.py`, `openhands-sdk/openhands/sdk/agent/response_dispatch.py`, `openhands-sdk/openhands/sdk/agent/parallel_executor.py`, `openhands-sdk/openhands/sdk/agent/stream_context.py`, `openhands-sdk/openhands/sdk/agent/utils.py`, `openhands-sdk/openhands/sdk/agent/critic_mixin.py`, `openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py`, `openhands-sdk/openhands/sdk/conversation/stuck_detector.py`, `openhands-sdk/openhands/sdk/conversation/types.py`, `openhands-sdk/openhands/sdk/tool/builtins/finish.py`, `openhands-sdk/openhands/sdk/tool/tool.py` @@ -12,7 +12,7 @@ Examples: `examples/01_standalone_sdk/01_hello_world.py`, `examples/01_standalon - `S01.run-to-finish`: `send_message` + `run()` steps until the agent finishes; status moves IDLE to RUNNING to FINISHED, a second `run()` on a finished conversation returns at once, and a new message moves FINISHED back to IDLE. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py`. Example: `examples/01_standalone_sdk/01_hello_world.py`. Real-LLM: `tests/integration/tests/t01_fix_simple_typo.py`, `tests/integration/tests/t02_add_bash_hello.py`. - `S01.response-dispatch`: each LLM reply is classified as tool calls, content, reasoning-only or empty; tool calls execute, content sets FINISHED, and the batch's `ActionEvent`s are emitted consecutively. Unit: `tests/sdk/agent/test_response_dispatch.py`. Example: none. Real-LLM: none. -- `S01.empty-reply-nudge`: a reasoning-only or completely empty reply injects a corrective user nudge and the loop continues instead of finishing; the nudge does not reset the stuck-detection window. Unit: `tests/sdk/agent/test_reasoning_only_responses.py`, `tests/sdk/agent/test_tool_call_recovery.py`, `tests/sdk/agent/test_response_dispatch.py::test_corrective_nudge_does_not_reset_stuck_detection_window`. Example: none. Real-LLM: none. +- `S01.empty-reply-nudge`: a reasoning-only reply (reasoning or thinking, no text, no tool call) injects a corrective user nudge and the loop continues instead of finishing; the nudge does not reset the stuck-detection window. A content-free reply (no text, no tool call, no reasoning) is not an agent turn: the LLM call retries it (`S02.no-response-retry`), so no empty agent message or nudge is recorded and the run goes on with the next reply; when every attempt is content-free the run ends `ERROR` with one retryable `ConversationErrorEvent` (code `LLMNoResponseError`), not `STUCK`. Unit: `tests/sdk/agent/test_reasoning_only_responses.py`, `tests/sdk/agent/test_tool_call_recovery.py`, `tests/sdk/conversation/local/test_empty_response_retry.py`, `tests/sdk/agent/test_response_dispatch.py::test_corrective_nudge_does_not_reset_stuck_detection_window`. Example: none. Real-LLM: none. - `S01.max-iterations`: after `max_iteration_per_run` steps (default 500) the run stops with status ERROR and a `ConversationErrorEvent(code="MaxIterationsReached")`, unless the last step finished, which keeps FINISHED. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_error_on_max_iterations`, `::test_execution_status_finished_on_final_iteration`. Example: `examples/01_standalone_sdk/28_ask_agent_example.py` (`max_iteration_per_run=5`). Real-LLM: none. - `S01.max-budget`: with `max_budget_per_run` set, the run stops with ERROR and `MaxBudgetReached` once the combined cost of all the conversation's LLMs reaches it; a finished run stays FINISHED. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_error_on_max_budget`, `::test_finished_preserved_even_when_over_budget`. Example: none. Real-LLM: none. - `S01.stuck-detect`: with `stuck_detection=True` (default) the detector flags repeated action/observation pairs (4), repeated action/error pairs (3), agent monologue (3) and alternating patterns (6) in the events since the last user message, and sets status STUCK. Unit: `tests/cross/test_stuck_detector.py`. Example: `examples/01_standalone_sdk/20_stuck_detector.py`. Real-LLM: none. @@ -78,7 +78,7 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` --check kinds.2 eq ObservationEvent --check kinds.3 eq ObservationEvent \ --check events.0.tool eq file_editor --check events.1.tool eq terminal ``` -- **Recovering inside a step (`S01.empty-reply-nudge`, `S01.malformed-tool-call`, `S01.unknown-tool`, `S01.content-policy`).** One scripted run meets an unknown tool, arguments that fail validation, a stringified JSON list (repaired), a content-filter block and an empty reply, then finishes. +- **Recovering inside a step (`S01.malformed-tool-call`, `S01.unknown-tool`, `S01.content-policy`).** One scripted run meets an unknown tool, arguments that fail validation, a stringified JSON list (repaired) and a content-filter block, then finishes. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) CID=$(control-sdk conversation new --script S01/step_errors.json --field id) @@ -91,15 +91,27 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` --check events.0.tool eq task_tracker --check events.0.is_error eq false --check events.0.text contains "1 item" control-sdk conversation events "$CID" --kind MessageEvent --source user \ --check events.1.text contains "blocked by the model's content filter" - control-sdk conversation events "$CID" --kind MessageEvent --source environment \ - --check count eq 1 --check events.0.text contains "did not include a function call" ``` -- **A reasoning-only reply (`S01.empty-reply-nudge`).** A reply with `reasoning_content` and no text or tool call gets the same nudge. +- **A reasoning-only reply (`S01.empty-reply-nudge`).** A scripted reply with `reasoning_content` and no text or tool call gets the corrective nudge. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) control-sdk program S01/loop --arg reasoning-only \ --check status eq finished --check llm_calls eq 2 --check nudges eq 1 --check nudge_text contains "did not include a function call" ``` +- **A content-free reply from a real model (`S01.empty-reply-nudge`).** A real `LLM` (`openai/gpt-4o-mini`, `temperature=0`) against `programs/S02/fake_provider.py`: two replies with empty text and no tool call or reasoning, then an answer (`num_retries=5`); a reasoning-only reply, then `finish`; and only content-free replies (`num_retries=2`). + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S01/loop --arg content-free \ + --check empty_then_answer.status eq finished --check empty_then_answer.requests eq 3 \ + --check empty_then_answer.temperatures.1 eq 1.0 --check empty_then_answer.nudges eq 0 \ + --check empty_then_answer.agent_messages len-eq 1 --check empty_then_answer.agent_messages.0 eq "The answer is ready." \ + --check reasoning_only.status eq finished --check reasoning_only.requests eq 2 --check reasoning_only.nudges eq 1 \ + --check exhausted.status eq error --check exhausted.raised eq ConversationRunError --check exhausted.requests eq 2 \ + --check exhausted.nudges eq 0 --check exhausted.agent_messages len-eq 0 \ + --check exhausted.error_codes len-eq 1 --check exhausted.error_codes.0 eq LLMNoResponseError \ + --check exhausted.error_classifications.0.kind eq transient --check exhausted.error_classifications.0.retryable eq true + ``` + The two empty replies are retried inside one LLM call (the temperature bumped from 0 to 1.0), so the conversation holds one agent message and no nudge. The reasoning-only reply is a turn of its own and gets the nudge. When both attempts are empty the run ends `ERROR` (`run()` raises `ConversationRunError`) with one transient, retryable `LLMNoResponseError` event, no empty agent message and no `STUCK`. - **The iteration cap (`S01.max-iterations`).** `max_iteration_per_run=2` with three steps of work, and with the finish on step two. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -280,3 +292,4 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` - `ConversationStateUpdateEvent`s are not delivered to `callbacks` in a local run; poll `conversation.state.execution_status` (from another thread) to watch status changes. - Once a run has ended STUCK, calling `run()` again is re-detected as STUCK straight away with no LLM call; only a new user message lets the agent continue. - `ConversationErrorEvent` is not exported from `openhands.sdk.event`; import it from `openhands.sdk.event.conversation_error`. +- `TestLLM` skips the LLM's response validation, so a scripted reply with empty text and no tool call still reaches the agent and gets the corrective nudge; only a real `LLM` (here against the local endpoint) shows a content-free reply being retried. diff --git a/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md b/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md index 7b75edff78c..a6dee0b3fa8 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md +++ b/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md @@ -14,10 +14,10 @@ Examples: `examples/01_standalone_sdk/13_get_llm_metrics.py`, `examples/01_stand - `S02.api-mode`: `generate()` / `agenerate()` dispatch to Chat Completions or Responses by `api_mode` (`auto`, `chat`, `responses`); `uses_responses_api()` reports the choice. Unit: `tests/sdk/llm/test_llm.py::test_generate_dispatches_to_configured_api`, `::test_agenerate_dispatches_to_configured_api`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py`. Real-LLM: none. - `S02.streaming`: with `stream=True` and an `on_token` callback, content, reasoning and tool-call chunks arrive as `ModelResponseStream` deltas and the final message (tool calls and cache-read tokens included) is reassembled; sync and async generators are accepted. Unit: `tests/sdk/llm/test_llm_completion.py::test_llm_completion_streaming_with_callback`, `::test_llm_completion_streaming_with_tools`, `::test_llm_streaming_preserves_cache_read_tokens`, `tests/sdk/event/test_streaming.py`, `tests/sdk/agent/test_stream_context.py`. Example: `examples/01_standalone_sdk/29_llm_streaming.py`. Real-LLM: none. - `S02.stream-no-callback`: `stream=True` without a callback degrades to a normal, non-streaming result instead of failing. Unit: `tests/sdk/llm/test_llm_completion.py::test_llm_streaming_without_callback_degrades`. Example: none. Real-LLM: none. -- `S02.timeouts`: `timeout` (default 300 s, `None` disables) is passed to LiteLLM and enforced per attempt; `stream_idle_timeout` (default 300 s) aborts and retries a stalled async stream, and an exhausted stream timeout puts the conversation in ERROR. Unit: `tests/sdk/llm/test_llm_timeout.py`. Example: none. Real-LLM: none. +- `S02.timeouts`: `timeout` (default 300 s, `None` disables) is passed to LiteLLM and enforced per attempt; `stream_idle_timeout` (default 300 s) aborts and retries a stalled async stream, and an exhausted stream timeout puts the conversation in ERROR. Timeouts stop being retried once the call has used twice `timeout` of wall-clock time, so attempts that each run into the full timeout end after two, whatever `num_retries` allows, while timeouts far shorter than `timeout` (a stream idle timeout, a connect timeout) keep every attempt. The final `LLMTimeoutError` of a call that ran out of attempts says that the model did not return a complete response within the LLM timeout, with the limit, the number of attempts and the total seconds, and points to a faster model, a shorter prompt or a higher timeout in the LLM settings. Unit: `tests/sdk/llm/test_llm_timeout.py` (`test_hard_timeouts_stop_retrying_after_the_timeout_budget`, `test_quick_timeouts_keep_every_retry`, `test_non_timeout_errors_are_reraised_unchanged` for the budget and the summary). Example: none. Real-LLM: none. - `S02.responses-reasoning`: on the Responses path `reasoning_effort`, `reasoning_summary` and `enable_encrypted_reasoning` (default on) are sent only to reasoning models; output items, reasoning summaries and encrypted reasoning are parsed into the `Message` and replayed on the next turn; streamed Responses events are reconstructed with or without a callback. Unit: `tests/sdk/llm/test_responses_parsing_and_kwargs.py`, `tests/sdk/llm/test_responses_serialization.py`, `tests/sdk/llm/test_llm.py::test_enable_encrypted_reasoning_respects_flag_and_defaults_true`. Example: `examples/01_standalone_sdk/23_responses_reasoning.py`. Real-LLM: none. - `S02.anthropic-thinking`: Anthropic thinking and redacted-thinking blocks (with or without signature) are kept on assistant messages and shown on `MessageEvent`/`ActionEvent`; the blocks of a tool-call reply are sent back unchanged, signature included, on the next request; `extended_thinking_budget` is clamped below `max_tokens`. Unit: `tests/sdk/llm/test_thinking_blocks.py`, `tests/sdk/llm/test_chat_options.py::test_extended_thinking_budget_clamped_below_max_tokens`, `tests/sdk/llm/test_joint_budget_clamp.py`. Example: `examples/01_standalone_sdk/22_anthropic_thinking.py`. Real-LLM: `tests/integration/tests/c01_thinking_block_condenser.py`. -- `S02.thinking-text-reply`: a reply without tool calls (text, or thinking only) keeps its thinking blocks on the `MessageEvent` with the provider's signature when no registered secret was masked in them, and they are sent back unchanged on the next request; a block whose text was masked has no signature (`S13.mask-thinking`). Unit: `tests/sdk/agent/test_response_dispatch.py::test_thinking_blocks_mask_registered_secret` (the masked case only). Example: none. Real-LLM: none. +- `S02.thinking-text-reply`: a reply without tool calls (text, or thinking only) keeps its thinking blocks on the `MessageEvent` with the provider's signature when no registered secret was masked in them, and they are sent back unchanged on the next request; a block whose text was masked has no signature (`S13.mask-thinking`). Unit: `tests/sdk/agent/test_response_dispatch.py::test_thinking_blocks_mask_registered_secret`, `tests/sdk/agent/test_response_dispatch.py::test_thinking_block_untouched_by_masking_keeps_signature`, `tests/sdk/agent/test_response_dispatch.py::test_only_the_thinking_block_holding_a_secret_loses_its_signature`. Example: none. Real-LLM: none. - `S02.reasoning-content`: chat-path `reasoning_content` (DeepSeek-style) is captured on the `Message` and carried on the `ActionEvent`. Unit: `tests/sdk/llm/test_reasoning_content.py`. Example: `examples/01_standalone_sdk/29_llm_streaming.py` (prints streamed reasoning). Real-LLM: none. - `S02.reasoning-options`: per model family the chat options send `reasoning_effort` or adaptive thinking and strip `temperature`/`top_p` where the model rejects them (GPT-5, Claude Opus 4.5 and Sonnet 4.6+, Kimi), keep them for non-reasoning models, rename Azure `max_completion_tokens`, and add `prompt_cache_key` and OpenRouter headers where supported. Unit: `tests/sdk/llm/test_chat_options.py`. Example: none. Real-LLM: none. - `S02.model-features`: the feature table answers per model whether it supports reasoning effort, extended or adaptive thinking, sampling params, prompt cache (and key, retention), stop words, Responses API, vision and inline images, and whether to force string content; proxy prefixes are stripped before lookup. Each capability is decided by the first source that states it: the caller's `capability_overrides`, then what the endpoint's model info declares, then the SDK's own table, then discovery (LiteLLM's registry and name heuristics, which never outrank the SDK's table); `ModelFeatures.sources` names the source per capability (`caller`, `endpoint`, `sdk`, `litellm`, `discovery`). Unit: `tests/sdk/llm/test_model_features.py`, `tests/sdk/llm/test_model_info_proxy_lookup.py`, `tests/sdk/llm/test_model_canonical_name_resolution.py`. Example: none. Real-LLM: none. @@ -26,8 +26,8 @@ Examples: `examples/01_standalone_sdk/13_get_llm_metrics.py`, `examples/01_stand - `S02.metrics`: `llm.metrics` accumulates `accumulated_cost`, per-call `costs`, `token_usages` (prompt, completion, cache read, cache write, reasoning, context window), `response_latencies` and a cache hit rate; metrics merge, diff, deep-copy, reset and restore. Unit: `tests/sdk/llm/test_llm_metrics.py`, `tests/sdk/llm/test_llm_completion.py::test_llm_cost_tracking`, `::test_llm_token_usage_tracking`, `::test_llm_latency_tracking`, `tests/sdk/llm/test_llm.py::test_llm_reset_metrics`. Example: `examples/01_standalone_sdk/13_get_llm_metrics.py`. Real-LLM: none. - `S02.conversation-stats`: `conversation.conversation_stats` keeps one `Metrics` per `usage_id` (agent, condenser, any LLM added to `conversation.llm_registry`) and `get_combined_metrics()` sums them; restored metrics keep tracking after a resume. Unit: `tests/sdk/conversation/test_conversation_stats.py`, `tests/sdk/llm/test_llm.py::test_conversation_stats_restore_then_track`. Example: `examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py`. Real-LLM: none. - `S02.pricing-override`: `input_cost_per_token` / `output_cost_per_token` override LiteLLM pricing for cost calculation. Unit: `tests/sdk/llm/test_llm_pricing_passthrough.py`, `tests/sdk/llm/test_llm_span_cost.py`. Example: none. Real-LLM: none. -- `S02.retries`: rate-limit, connection and other transient errors are retried up to `num_retries` (default 5) with backoff from `retry_min_wait`/`retry_max_wait`/`retry_multiplier`; non-retryable errors raise at once; a retry listener sees each attempt. Unit: `tests/sdk/llm/test_api_connection_error_retry.py`, `tests/sdk/llm/test_llm.py::test_llm_retry_on_rate_limit`, `tests/sdk/llm/test_llm_retry_telemetry.py`, `tests/sdk/llm/test_auth_refresh_retry.py`. Example: none. Real-LLM: none. -- `S02.no-response-retry`: an empty provider response is retried with a temperature bump and finally raises `LLMNoResponseError`, on both chat and Responses paths, sync and async. Unit: `tests/sdk/llm/test_llm_no_response_retry.py`. Example: none. Real-LLM: none. +- `S02.retries`: rate-limit, connection and other transient errors are retried up to `num_retries` (default 5) with backoff from `retry_min_wait`/`retry_max_wait`/`retry_multiplier`; non-retryable errors raise at once; a retry listener sees each attempt. A LiteLLM `APIError` for a 2xx response whose body could not be parsed as JSON counts as transient and is retried; the same unparseable body with a 4xx status is not. Repeated timeouts are bounded separately (`S02.timeouts`), and each failed attempt is logged (`S20.llm-retry-log`). Unit: `tests/sdk/llm/test_api_connection_error_retry.py`, `tests/sdk/llm/test_llm.py::test_llm_retry_on_rate_limit`, `tests/sdk/llm/test_llm_retry_telemetry.py`, `tests/sdk/llm/test_auth_refresh_retry.py`, `tests/sdk/llm/test_retry_diagnostics.py::test_malformed_provider_response_uses_existing_retry_budget`, `::test_unrelated_api_errors_are_not_transient`, `::test_malformed_provider_response_recovers_on_async_retry`. Example: none. Real-LLM: none. +- `S02.no-response-retry`: an empty provider response is retried with a temperature bump and finally raises `LLMNoResponseError`, on both chat and Responses paths, sync and async. On the chat path that covers a reply with no choices and a reply whose only choice has no text, no tool calls and no reasoning, streamed or not; a choice with reasoning or thinking only is returned as it is, and so is a content-free choice whose `finish_reason` is `length` (the output-token cap; a retry would give the same answer). Unit: `tests/sdk/llm/test_llm_no_response_retry.py`. Example: none. Real-LLM: none. - `S02.error-mapping`: LiteLLM and HTTP errors map to `LLMContextWindowExceedError`, `LLMMalformedConversationHistoryError`, `LLMAuthenticationError`, `LLMRateLimitError`, `LLMTimeoutError`, `LLMServiceUnavailableError`, `LLMBadRequestError`, `LLMContentPolicyViolationError` and budget denials. Unit: `tests/sdk/llm/test_exception_mapping.py`, `tests/sdk/llm/test_exception_classifier.py`, `tests/sdk/llm/test_exception.py`, `tests/sdk/llm/test_budget_denial_http.py`. Example: none. Real-LLM: `tests/integration/tests/a01_unmatched_tool_use.py` (provider responses to malformed histories, run through `tests/integration/api_compliance/run_compliance.py`). - `S02.prompt-caching`: with `caching_prompt=True` (default) cache markers or keys are added for supporting models, and a "prompt too small to cache" rejection is retried without caching. Unit: `tests/sdk/llm/test_prompt_caching_cross_conversation.py`, `tests/sdk/llm/test_llm_completion.py::test_completion_retries_without_caching_on_prompt_cache_too_small`. Example: none. Real-LLM: none. - `S02.token-count`: `llm.get_token_count(messages)` counts tokens with LiteLLM or a chat-template tokenizer (`custom_tokenizer`), including tool definitions, with fallbacks when the tokenizer is unavailable. Unit: `tests/sdk/llm/test_llm.py::test_llm_token_counting`, `::test_llm_token_counting_includes_tools`. Example: none. Real-LLM: none. @@ -35,7 +35,7 @@ Examples: `examples/01_standalone_sdk/13_get_llm_metrics.py`, `examples/01_stand - `S02.message-formatting`: persisted events reload with `Event.model_validate_json`, `LLMConvertibleEvent.events_to_messages` merges them into `Message`s, and `llm.format_messages_for_llm` / `format_messages_for_responses` produce the provider payload (string serializer, vision, cache markers). Unit: `tests/sdk/event/test_events_to_messages.py`, `tests/sdk/llm/test_message.py`, `tests/sdk/llm/test_message_serialization.py`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py`. Real-LLM: none. - `S02.image-inline`: with `inline_image_urls` (auto for providers that reject URLs, such as Moonshot Kimi) http image URLs are fetched and sent as base64 `data:` URLs, bounded by `OH_INLINE_IMAGE_MAX_MB`, `OH_INLINE_IMAGE_FETCH_TIMEOUT_S`, `OH_INLINE_IMAGE_CACHE_BYTES`. Unit: `tests/sdk/llm/test_image_inline.py`. Example: none. Real-LLM: none. - `S02.image-request-guard`: `completion`, `acompletion`, `responses` and `aresponses` raise `LLMBadRequestError` before any HTTP request when the latest user message carries an image and `vision_is_active()` is false (a model without vision, or `disable_vision=True`); `get_token_count` and `format_messages_for_llm` still work on those messages; an image only in an older message does not block the call and is left out of the request; `capability_overrides={"supports_vision": True}` sends the image. Unit: `tests/sdk/llm/test_capability_wire.py::test_disabled_vision_rejects_request_but_not_token_count`, `::test_image_reaches_http_boundary`, `tests/sdk/agent/test_non_multimodal_image_input.py::test_llm_image_guard_scoped_to_latest_user_message`. Example: none. Real-LLM: none. -- `S02.image-adapter-loss`: on the native `deepseek/` route, when LiteLLM's adapter drops an image from the request body the call is refused as `LLMBadRequestError` before anything is sent and is not retried; an adapter that delivers every image sends the request as usual. Unit: `tests/sdk/llm/test_capability_wire.py::test_native_deepseek_adapter_must_deliver_or_reject_images`. Example: none. Real-LLM: none. +- `S02.image-adapter-loss`: on the native `deepseek/` route, when LiteLLM's adapter drops an image from the request body the call is refused as `LLMBadRequestError` before anything is sent and is not retried; an adapter that delivers every image sends the request as usual. Unit: `tests/sdk/llm/test_capability_wire.py::test_native_deepseek_adapter_must_deliver_or_reject_images`, `tests/sdk/llm/test_image_delivery.py::test_image_loss_surfaces_as_bad_request_through_litellm_wrapping`, `tests/sdk/llm/test_image_delivery.py::test_provider_failure_with_images_delivered_is_left_alone`. Example: none. Real-LLM: none. - `S02.log-completions`: `log_completions_to_file=True` writes each request and response as JSON under `log_completions_folder`. `log_completions` is a deprecated alias for the same flag (deprecated in v1.63.0, removal scheduled for v1.68.0): either name sets both, passing the two with different values raises `ValueError`, and a dump carries both keys. Unit: `tests/sdk/llm/test_llm_log_completions_integration.py`, `tests/sdk/llm/test_completion_logging_aliases.py`. Example: none. Real-LLM: none. - `S02.serialization`: an `LLM` dumps with `api_key` and other secrets masked, round-trips through JSON, `model_copy` keeps configuration while refreshing provider info, and `LLM.load_from_env(prefix="LLM_")` builds an `LLM` from `LLM_*` variables, casting types. Unit: `tests/sdk/llm/test_llm_serialization.py`, `tests/sdk/llm/test_llm.py::test_llm_model_copy_preserves_configuration`, `tests/sdk/test_settings.py::test_llm_load_from_env_rehydrates_subscription_runtime` (the only `load_from_env` test; subscription case). Example: none. Real-LLM: none. @@ -114,14 +114,20 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check chat_cache_write eq 600 --check responses_cache_read eq 200 --check responses_cache_write eq 600 ``` Expected: `TokenUsage.cache_write_tokens` is 600 on both paths. Known failure (reproduced 2026-11-12, [#6691](https://github.com/OpenHands/software-agent-sdk/issues/6691)): the Responses path records 0 (`normalize_usage` reads only `cached_tokens` from `input_tokens_details`); the chat path records 600. -- **Timeouts (`S02.timeouts`).** A reply slower than `timeout=1`, a stream that stalls longer than `stream_idle_timeout=1` on `acompletion`, and the same stall inside `await conversation.arun()`. +- **Timeouts (`S02.timeouts`).** A reply slower than `timeout=1`, a stream that stalls longer than `stream_idle_timeout=1` on `acompletion`, and the same stall inside `await conversation.arun()`. Then an endpoint that never answers, with `timeout=1` and `num_retries=5`, on `completion` and on a streamed `acompletion`; and a stream that stalls on every attempt with `stream_idle_timeout=0.3`, `timeout=30` and `num_retries=3`. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) control-sdk program S02/llm_calls --arg timeouts \ --check timeout.error eq LLMTimeoutError --check timeout.seconds lt 3 \ --check stream_idle.error eq LLMTimeoutError --check stream_idle.seconds lt 3 \ - --check conversation.status eq error --check conversation.error_events.0 eq LLMTimeoutError + --check conversation.status eq error --check conversation.error_events.0 eq LLMTimeoutError \ + --check hard.error eq LLMTimeoutError --check hard.requests eq 2 --check hard.seconds lt 4 \ + --check hard.names_limit eq true --check hard.names_attempts eq true --check hard.names_settings eq true \ + --check hard_async.error eq LLMTimeoutError --check hard_async.requests eq 2 --check hard_async.hard_timeout eq true \ + --check hard_async.names_limit eq true --check hard_async.names_attempts eq true --check hard_async.names_settings eq true \ + --check quick.error eq LLMTimeoutError --check quick.requests eq 3 ``` + The never-answering endpoint gets two requests of the five allowed (about 2 s), and the error reads like `... Request timed out. The model did not return a complete response within the LLM timeout of 1s (gave up after 2 attempts, 2s in total). Try a faster model, a shorter prompt (fewer tools or MCP servers), or a higher timeout in the LLM settings.`; on the async path it starts `LLM hard timeout after 1 seconds`. The 0.3 s stalls are far inside the 2 x 30 s budget, so all three attempts run. - **Anthropic thinking (`S02.anthropic-thinking`, `S02.prompt-caching`).** `anthropic/claude-sonnet-4-5-20250929` with `extended_thinking_budget=200000` and `max_output_tokens=4096`; the first reply carries a signed thinking block. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -140,7 +146,7 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check message_thinking.0 eq "considering the answer" \ --check message_signatures.0 eq sig-1 --check thinking_sent_back.0 eq sig-1 ``` - Expected: the `MessageEvent` holds the thinking block as the provider sent it, signature `sig-1`, and the second request carries it back in the assistant turn. Known failure (reproduced 2026-11-14, [#6837](https://github.com/OpenHands/software-agent-sdk/issues/6837), fix in [#6839](https://github.com/OpenHands/software-agent-sdk/pull/6839)): the thinking text is on the `MessageEvent`, but its signature is null although nothing was masked (`message_signatures` is `[null]`), and the unsigned block is left out of the second request (`thinking_sent_back` is empty). `_mask_secrets` clears the signature of every thinking block on a reply without tool calls, whether or not a secret matched. + Expected: the `MessageEvent` holds the thinking block as the provider sent it, signature `sig-1`, and the second request carries it back in the assistant turn. Only a block whose text held a registered secret loses its signature. - **Reasoning options and model features (`S02.reasoning-options`, `S02.model-features`).** The feature table for several models, and the chat parameters each model's request actually carried. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -176,24 +182,30 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check parsed.unknown_tool.error eq FunctionCallValidationError \ --check parsed.unknown_parameter.error eq FunctionCallValidationError ``` -- **Retries (`S02.retries`).** A 429 then a 503 then success; a 400; a 429 that never clears; a budget denial. +- **Retries (`S02.retries`).** A 429 then a 503 then success; a 400; a 429 that never clears; a budget denial; an HTTP 200 and an HTTP 400 whose body is not JSON (`hosted_vllm/`, LiteLLM's OpenAI-compatible handler), each followed by a good reply. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) control-sdk program S02/llm_calls --arg retries \ --check transient.text eq hi --check transient.requests eq 3 --check transient.listener len-eq 2 \ --check non_retryable.error eq LLMBadRequestError --check non_retryable.requests eq 1 \ --check exhausted.error eq LLMRateLimitError --check exhausted.requests eq 3 \ - --check budget_denial.requests eq 1 + --check budget_denial.requests eq 1 \ + --check malformed_2xx.text eq hi --check malformed_2xx.requests eq 2 \ + --check malformed_4xx.error eq LLMBadRequestError --check malformed_4xx.requests eq 1 ``` - `num_retries=3` means three attempts in all; the retry listener sees each retry; a bad request and a budget denial are not retried. -- **No-response retry (`S02.no-response-retry`).** Empty replies on the chat path (a reply with no `choices`) and on the Responses path (a stream that ends without its completed event), with `temperature=0`. + `num_retries=3` means three attempts in all; the retry listener sees each retry; a bad request and a budget denial are not retried. The unparseable 200 body is retried and the next reply returned; the unparseable 400 body is raised at once. +- **No-response retry (`S02.no-response-retry`).** Empty replies on the chat path (a reply with no `choices`; a choice with empty text and no tool call or reasoning, plain and streamed) and on the Responses path (a stream that ends without its completed event), with `temperature=0`; a content-free choice with `finish_reason: "length"`, and a choice with only `reasoning_content`, each followed by a good reply. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) control-sdk program S02/llm_calls --arg no-response \ --check responses.text eq hi --check responses.requests eq 2 --check responses.temperatures.1 eq 1.0 \ --check responses_async_exhausted.error eq LLMNoResponseError --check responses_async_exhausted.requests eq 2 \ --check chat.text eq hi --check chat.temperatures.1 eq 1.0 \ - --check chat_exhausted.error eq LLMNoResponseError + --check chat_exhausted.error eq LLMNoResponseError \ + --check content_free.text eq hi --check content_free.requests eq 2 --check content_free.temperatures.1 eq 1.0 \ + --check content_free_stream.text eq hi --check content_free_stream.requests eq 2 --check content_free_stream.temperatures.1 eq 1.0 \ + --check length_capped.error missing --check length_capped.text eq "" --check length_capped.requests eq 1 \ + --check reasoning_only.reasoning eq "only thinking" --check reasoning_only.requests eq 1 ``` Known failure (reproduced 2026-11-12, [#6657](https://github.com/OpenHands/software-agent-sdk/issues/6657)): the chat path behaves as documented, but on the Responses path a stream that ends without its completed event does not reach the SDK's `LLMNoResponseError` check. LiteLLM raises `APIConnectionError` first, the SDK retries it as a transient error without the temperature bump (`responses.temperatures` is `[0.0, 0.0]`), and an exhausted async call ends in `LLMServiceUnavailableError`. The unit tests mock the LiteLLM call, so they do not meet it. - **Error mapping (`S02.error-mapping`).** Each provider error body, as an OpenAI-compatible endpoint sends it, and a dead endpoint. @@ -275,7 +287,7 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check tool_image.delivered_or_refused_once eq true \ --check tool_image.error eq LLMBadRequestError --check tool_image.retries eq 0 --check tool_image.requests eq 0 ``` - Expected: with the installed LiteLLM (1.105.1) the adapter drops both images, so each call is refused once as `LLMBadRequestError` and the endpoint receives nothing. Known failure (reproduced 2026-11-14, [#6840](https://github.com/OpenHands/software-agent-sdk/issues/6840), fix in [#6841](https://github.com/OpenHands/software-agent-sdk/pull/6841)): nothing is sent (`requests` is 0), but the refusal is raised inside LiteLLM's HTTP client, LiteLLM wraps it as `InternalServerError`, and the SDK treats that as transient: each call is retried (`retries` is 2 with `num_retries=3`, so `delivered_or_refused_once` is false) and ends in `LLMServiceUnavailableError`. With the default retry settings that is minutes of backoff for a request that can never be sent. + Expected: with the installed LiteLLM (1.105.1) the adapter drops both images, so each call is refused once as `LLMBadRequestError` and the endpoint receives nothing; the refusal is not retried, also when LiteLLM wraps it as a provider error, while a provider failure on a request whose images were all delivered keeps its usual mapping and retries. - **The completion-log flag and its alias (`S02.log-completions`).** ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -355,7 +367,7 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog - `openrouter_site_url`/`openrouter_app_name` have defaults, so `HTTP-Referer` and `X-Title` headers go to every provider, not only OpenRouter. - `num_retries` is the total number of attempts, not extra retries; the temperature bump on an empty response applies only when the temperature is exactly 0. - LiteLLM picks the exception class from the error body's `type`, not only the HTTP status: a 503 or 401 whose body says `invalid_request_error` surfaces as a bad-request error. Fake servers must set a type that matches the status. -- A chat reply with `"choices": []` is raised by LiteLLM as `InternalServerError` (no 'choices'), so the SDK treats it as a service error, not as an empty response, and does not bump the temperature. +- `TestLLM` skips the response validation that decides a choice is content-free; a scripted empty reply is returned to the agent as it is. Prove `S02.no-response-retry` with a real `LLM` against the local endpoint. - The `log_completions` alias warns (`DeprecatedWarning`) only once the installed SDK version reaches 1.63.0, the release its deprecation names; an older build accepts it silently. The conflict error is a pydantic `ValidationError`, which is a `ValueError`. - The image guard looks only at the latest user message. Images in older turns or in tool results never raise; for a model without vision they are dropped from the request. - The output budget default (at most 16384) applies to every model, so a model that advertises 64000 output tokens still gets 16384 per request until `max_output_tokens` is set. diff --git a/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md b/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md index 7ff7656204a..916b4898833 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md +++ b/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md @@ -21,6 +21,7 @@ Examples: `examples/01_standalone_sdk/06_interactive_terminal_w_reasoning.py`, ` - `S04.run-error`: an exception inside `run()` surfaces as `ConversationRunError` that names the conversation id, the persistence dir and the issue URL, and emits one `ConversationErrorEvent` (no duplicate when the agent already emitted one). Unit: `tests/sdk/conversation/local/test_run_exception_includes_conversation_id.py`. Example: none. Real-LLM: none. - `S04.close`: `close()` closes tool executors (terminal sessions), is idempotent, unregisters its atexit hook, and still closes the executors with `delete_on_close=False` (persisted files stay either way); `__del__` also closes. Unit: `tests/sdk/conversation/test_atexit_cleanup.py`, `tests/tools/terminal/test_conversation_cleanup.py`, `tests/sdk/conversation/local/test_conversation_core.py::test_close_propagates_agent_failure_and_allows_retry`. Example: none. Real-LLM: none. - `S04.pause`: `pause()` from another thread moves RUNNING or IDLE to PAUSED, emits exactly one `PauseEvent`, takes effect between steps (not mid-LLM-call), and a later `run()` resumes. Unit: `tests/sdk/conversation/local/test_conversation_pause_functionality.py`, `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_transitions_to_running_from_paused`. Example: `examples/01_standalone_sdk/09_pause_example.py`. Real-LLM: `tests/integration/tests/c04_token_condenser.py` and `c05_size_condenser.py` call `pause()`. +- `S04.pause-during-startup`: a `pause()` from another thread that arrives after `run()` was called but while `run()` is still making the agent ready (plugins, MCP servers; the first `run()` of a conversation restored from `persistence_dir`) keeps the status PAUSED, makes no LLM call, and the next `run()` resumes; a conversation already PAUSED when `run()` is called is resumed by it. Unit: `tests/sdk/conversation/local/test_conversation_pause_functionality.py` (a pause before `run()` only; the startup window is **untested**). Example: none. Real-LLM: none. - `S04.async-callback`: `AsyncCallbackWrapper(coro, loop)` lets a synchronous `run()` in an executor thread deliver events to a coroutine on the main loop. Unit: `tests/sdk/utils/test_async_utils.py`. Example: `examples/01_standalone_sdk/11_async.py`. Real-LLM: none. - `S04.arun-interrupt`: `await conversation.arun()` uses the async LLM path; `interrupt()` (safe from a signal handler or another thread) cancels it at once, even mid-LLM-call, sets PAUSED, emits an `InterruptEvent`, does not raise `CancelledError`, and the conversation can be resumed; with no `arun()` in flight it falls back to `pause()`. Unit: `tests/sdk/conversation/test_interrupt.py`. Example: `examples/01_standalone_sdk/50_async_cancellation.py`. Real-LLM: none. - `S04.arun-pause-persisted`: with a `persistence_dir`, a `pause()` made from another thread while `arun()` waits on the model returns at once and is in `base_state.json` (`execution_status: paused`) at once, before the model call returns; `arun()` then stops PAUSED after that call and a later `run()` finishes. Unit: `tests/sdk/agent/test_astep_releases_state_lock.py::test_pause_during_llm_call_is_persisted_at_once`. Example: none. Real-LLM: none. @@ -31,6 +32,7 @@ Examples: `examples/01_standalone_sdk/06_interactive_terminal_w_reasoning.py`, ` - `S04.visualizer-default`: by default a `DefaultConversationVisualizer` prints each event as a Rich panel; `visualizer=None` turns it off; a class or instance can be passed. Unit: `tests/sdk/conversation/local/test_conversation_visualize_param.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. - `S04.visualizer-custom`: a `ConversationVisualizerBase` subclass passed as `visualizer=` receives every event through `on_event`; each event type has its own `visualize` rendering. Unit: `tests/sdk/conversation/test_visualizer.py`. Example: `examples/01_standalone_sdk/26_custom_visualizer.py`. Real-LLM: none. - `S04.fork`: `fork()` deep-copies the conversation under a new id (or `conversation_id=`), starts IDLE, leaves the source untouched, resets metrics unless `reset_metrics=False`, accepts `agent=` and `tags=` (`title=` is deprecated and ignored locally; only `RemoteConversation` stores it), shares the workspace, and persists so the fork survives a reload. Unit: `tests/sdk/conversation/local/test_fork.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py`. Real-LLM: none. +- `S04.fork-keeps-secrets`: `fork()` of a `LocalConversation(..., cipher=...)` with a `persistence_dir` creates the fork with the source's cipher and copies the source's conversation secrets (`secrets=`, `update_secrets`), so the fork's `base_state.json` holds the LLM api key and every secret encrypted (not redacted); `LocalConversation(agent=None, conversation_id=fork_id, cipher=...)` reopens the fork with the real key and secrets, and it runs. Without a cipher the fork's key stays redacted on disk. Since [#5765](https://github.com/OpenHands/software-agent-sdk/pull/5765) ([#5465](https://github.com/OpenHands/software-agent-sdk/issues/5465)). Unit: `tests/sdk/conversation/local/test_fork.py::test_fork_keeps_source_cipher_for_persisted_secrets`, `::test_fork_without_cipher_still_redacts_secrets`, `tests/agent_server/test_conversation_service.py::TestConversationTreeForkAndNavigate::test_cipher_backed_fork_resumes_with_credentials`. Example: none. Real-LLM: none. - `S04.fork-branch`: events carry parent ids forming a tree; `fork(from_event_id=...)` copies only that branch, and `navigate_to(event_id)` moves HEAD so the next event starts a sibling branch (`None` empties the active branch). Unit: `tests/sdk/conversation/local/test_conversation_tree.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py`. Real-LLM: none. - `S04.goal-loop`: `run_goal(conversation, objective, judge_llm, max_iterations=N)` sends the objective, runs, asks the judge LLM for a verdict, re-prompts with the missing items, and returns a `GoalOutcome` with status `"complete"` or `"capped"`, the iteration count and the verdict. Unit: `tests/sdk/conversation/goal/`. Example: `examples/01_standalone_sdk/54_goal_completion_loop.py`. Real-LLM: none. - `S04.interactive-terminal`: with `Tool(name=TerminalTool.name, params={"no_change_timeout_seconds": 3})` the agent can start `python3`, feed it input and exit; an `LLMConvertibleEvent` callback collects the raw LLM messages. Unit: `tests/tools/terminal/test_terminal_session.py::test_python_interactive_input`, `::test_interactive_command`. Example: `examples/01_standalone_sdk/06_interactive_terminal_w_reasoning.py`. Real-LLM: `tests/integration/tests/t07_interactive_commands.py`. @@ -42,10 +44,10 @@ Examples: `examples/01_standalone_sdk/06_interactive_terminal_w_reasoning.py`, ` - `Conversation(agent, workspace=..., persistence_dir=None, conversation_id=None, callbacks=None, visualizer=DefaultConversationVisualizer, secrets=None, delete_on_close=True, tags=None, ...)` in `openhands.sdk`: `S04.factory`, `S04.conversation-id`, `S04.persist-events`, `S04.resume`, `S04.resume-agent-compat`, `S04.callbacks`, `S04.visualizer-*`, `S04.secrets-constructor`, `S04.title-and-tags`. The step-loop arguments (`max_iteration_per_run`, `stuck_detection*`, `max_budget_per_run`) belong to S01. - On disk: `//base_state.json` and `.../events/event-NNNNN-.json` (`persistence_const.py`): `S04.persist-events`, `S04.resume`, `S04.ask-agent-usage-saved` (`stats.usage_to_metrics` in `base_state.json`). `ConversationState.create(..., file_store=...)` / `LocalConversation(..., file_store=...)`: `S04.custom-file-store`. -- `conversation.run()`, `conversation.pause()`, `await conversation.arun()`, `conversation.interrupt()`, `conversation.cancel_token`: `S04.pause`, `S04.arun-interrupt`, `S04.arun-pause-persisted`, `S04.cancel-token`. `openhands.sdk.utils.async_utils.AsyncCallbackWrapper`: `S04.async-callback`. +- `conversation.run()`, `conversation.pause()`, `await conversation.arun()`, `conversation.interrupt()`, `conversation.cancel_token`: `S04.pause`, `S04.pause-during-startup`, `S04.arun-interrupt`, `S04.arun-pause-persisted`, `S04.cancel-token`. `openhands.sdk.utils.async_utils.AsyncCallbackWrapper`: `S04.async-callback`. - `conversation.update_secrets({...})`, `openhands.sdk.secret.SecretSource` (subclass with `get_value()`), `StaticSecret`, `LookupSecret`: `S04.secrets-*`. - `openhands.sdk.conversation.visualizer.ConversationVisualizerBase`, `DefaultConversationVisualizer`: `S04.visualizer-*`. -- `conversation.fork(conversation_id=, agent=, title=, tags=, reset_metrics=True, from_event_id=)`, `conversation.navigate_to(event_id)`: `S04.fork`, `S04.fork-branch`. +- `conversation.fork(conversation_id=, agent=, title=, tags=, reset_metrics=True, from_event_id=)`, `conversation.navigate_to(event_id)`: `S04.fork`, `S04.fork-branch`; `LocalConversation(..., cipher=Cipher(key))` (`openhands.sdk.utils.cipher`; the `Conversation` factory does not take a cipher): `S04.fork-keeps-secrets`. - `openhands.sdk.conversation.goal.run_goal`, `GoalController`, `judge_goal`: `S04.goal-loop`. - `openhands.sdk.conversation.response_utils.get_agent_final_response`: `S04.final-response`. `conversation.conversation_stats` / `conversation.state.stats`: `S04.stats`. `conversation.generate_title()`: `S04.title-and-tags`. `conversation.close()`: `S04.close`. - Env: `LLM_API_KEY` (required), `LLM_MODEL` (default `gpt-5.5`; `anthropic/claude-sonnet-4-5-20250929` in example 50), `LLM_BASE_URL`; example 50 also reads `AUTO_CANCEL_SECONDS` (default 8). @@ -111,7 +113,7 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` --check disk_after_ask.agent.cost eq 0.14 --check disk_after_ask.ask-agent-llm.cost eq 0.14 \ --check reloaded.ask-agent-llm.cost eq 0.14 --check reloaded_combined_cost eq 0.28 ``` - Expected: the ask's 0.14 is under `ask-agent-llm` in memory, in `base_state.json` and in the reloaded conversation, whose combined cost is 0.28. Known failure (reproduced 2026-11-14, [#6844](https://github.com/OpenHands/software-agent-sdk/issues/6844), fix in [#6845](https://github.com/OpenHands/software-agent-sdk/pull/6845)): the usage is recorded in memory only. `base_state.json` right after the ask still holds just `agent` (0.14), so the reloaded conversation has no `ask-agent-llm` entry and a combined cost of 0.14; the ask's usage reaches the file only when a later state change saves it. + Expected: the ask's 0.14 is under `ask-agent-llm` in memory, in `base_state.json` and in the reloaded conversation, whose combined cost is 0.28. - **Which agent changes a resume accepts (`S04.resume-agent-compat`).** Removing a tool, removing a default tool and changing the agent class fail; adding a tool and changing the LLM work. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -168,6 +170,15 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` --check status eq paused --check pause_returned_after_s gt 0.5 --check llm_calls eq 1 \ --check actions eq 1 --check observations eq 1 --check pause_events eq 1 --check status_after_resume eq finished ``` +- **A pause while `run()` is starting (`S04.pause-during-startup`).** A conversation restored from `persistence_dir` whose agent has a stdio MCP server (`S09/mcp_server.py --slow-start 2`) runs in a thread; `pause()` follows 0.5 s later, while `run()` is still starting the server. The status read under the state lock right after `pause()`, the status when `run()` returns, the LLM calls made after the pause, then a second `run()`. + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S04/pause_during_startup \ + --check status_after_restore eq idle --check status_after_pause eq paused --check pause_events eq 1 \ + --check status_after_run eq paused --check llm_calls_after_pause eq 0 \ + --check status_after_resume eq finished --check llm_calls_total eq 1 --check script_left eq 0 + ``` + Expected: `run()` returns PAUSED without calling the model, and the second `run()` makes the one call and finishes. Known failure (reproduced 2026-11-15, [#6848](https://github.com/OpenHands/software-agent-sdk/issues/6848), fix in [#6849](https://github.com/OpenHands/software-agent-sdk/pull/6849)): `run()` makes the agent ready (`_ensure_agent_ready()`) while the status is still IDLE, so `pause()` moves IDLE to PAUSED (it waits for the server, about 3.5 s, then returns PAUSED with one `PauseEvent`), and `run()` then treats PAUSED as a resume: it moves to RUNNING, calls the model once after the pause and ends FINISHED (`status_after_run` finished, `llm_calls_after_pause` 1); the second `run()` has nothing left to do. - **An async program (`S04.async-callback`).** `run()` in an executor thread, events delivered to a coroutine on the main loop through `AsyncCallbackWrapper`. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -235,6 +246,19 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` --check fork_request_had_source_history eq true --check same_workspace eq true \ --check fork_on_disk eq true --check reloaded_fork_events eq 9 --check reloaded_fork_title missing ``` +- **A cipher-backed fork keeps its secrets (`S04.fork-keeps-secrets`).** `LocalConversation(cipher=Cipher(...), persistence_dir=..., secrets={"VERIFY_TOKEN": ...})` on a priced `LLM` with an api key (pointed at `S02/fake_provider.py`), plus `update_secrets({"LATER_TOKEN": ...})`, then `fork()`. The program classifies each field of the fork's `base_state.json` as `encrypted` (decrypts with the cipher to the value), `plaintext` or something else; closes both, reopens the fork with `LocalConversation(agent=None, conversation_id=fork_id, cipher=...)` and runs it: the model's terminal command echoes both secrets and prints their lengths (12 and 16). A cipher-less fork is the control. + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S04/views --arg fork-secrets \ + --check fork_on_disk.api_key eq encrypted --check fork_on_disk.VERIFY_TOKEN eq encrypted \ + --check fork_on_disk.LATER_TOKEN eq encrypted --check fork_registry_has len-eq 2 \ + --check reloaded_api_key_is_real eq true --check reloaded_registry.VERIFY_TOKEN eq true \ + --check reloaded_registry.LATER_TOKEN eq true --check reloaded_status eq finished \ + --check requests eq 2 --check authorization.0 eq true --check authorization.1 eq true \ + --check command_output.0 eq "v= l=" \ + --check command_output.1 eq 12 --check command_output.2 eq 16 --check values_in_output eq false \ + --check fork_on_disk_after_run.api_key eq encrypted --check no_cipher_fork_api_key eq redacted + ``` - **Branches (`S04.fork-branch`).** Each event's `parent_id` is the one before; `fork(from_event_id=first_answer)` copies three events; `navigate_to(first_answer)` then a new message starts a sibling branch; `navigate_to(None)` empties the active branch but keeps the log. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) diff --git a/.agents/skills/verify-sdk/references/feature-map/S05-events.md b/.agents/skills/verify-sdk/references/feature-map/S05-events.md index 73188dc3d07..397c83883da 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S05-events.md +++ b/.agents/skills/verify-sdk/references/feature-map/S05-events.md @@ -23,7 +23,7 @@ Examples: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py` - `S05.event-cache`: deserialized events are cached, so repeated iteration does not re-read and re-parse files. Unit: `tests/sdk/conversation/test_event_store.py::test_event_cache_eliminates_repeated_deserialization`, `::test_event_cache_survives_across_multiple_iterations`. Example: none. Real-LLM: none. - `S05.event-tree`: events carry `parent_id`; `path_to_root` walks a branch, detects cycles, and `append` rejects a parent that does not exist. Unit: `tests/sdk/conversation/test_event_tree.py`. Example: none. Real-LLM: none. (Branch navigation from the conversation API is `S04.fork-branch`.) - `S05.visualize`: each event has a Rich `visualize` rendering: action summaries, truncated system-prompt tool parameters, without mutating the event. Unit: `tests/sdk/event/test_action_event_summary.py`, `tests/sdk/event/test_system_prompt_event_visualize.py`, `tests/sdk/conversation/test_visualizer.py`. Example: `examples/01_standalone_sdk/26_custom_visualizer.py`. Real-LLM: none. -- `S05.error-classification`: `ConversationErrorEvent` and `AgentErrorEvent` carry a `FailureKind` classification derived from the error code before the detail text; a sensitive detail is classified without being serialized. Unit: `tests/sdk/event/test_error_classification.py`. Example: none. Real-LLM: none. +- `S05.error-classification`: `ConversationErrorEvent` and `AgentErrorEvent` carry a `FailureKind` classification derived from the error code before the detail text; a sensitive detail is classified without being serialized. An `LLMTimeoutError` whose detail is a hard timeout (`LLM hard timeout after ...`) or carries the exhausted-timeout summary (`did not return a complete response within the LLM timeout`, `S02.timeouts`) is `config`, not retryable, `user_action=settings`; a timeout that did not use the whole LLM timeout (`Connection timed out.`, `Request timed out.`) stays `transient`, retryable, `user_action=retry`, and so does an `LLMNoResponseError` from a run whose replies were all content-free. Unit: `tests/sdk/event/test_error_classification.py` (`test_exhausted_llm_timeout_points_to_settings_not_retry` and the timeout cases for the timeouts), `tests/sdk/llm/test_llm_timeout.py::test_hard_timeouts_stop_retrying_after_the_timeout_budget`, `tests/sdk/conversation/local/test_empty_response_retry.py::test_exhausted_empty_completions_end_as_retryable_error`. Example: none. Real-LLM: none. - `S05.streaming-delta`: `StreamingDeltaEvent` holds a `content` and `reasoning_content` token delta, dumps with its `kind` and round-trips through JSON; it is transient (published by the agent server for WebSocket clients, never appended to the event log). Unit: `tests/sdk/event/test_streaming.py`. Example: none. Real-LLM: none. - `S05.completion-log-event`: `LLMCompletionLogEvent` carries a completion log (filename, JSON data, model, usage id) as an event; the agent server emits it for remote conversations. Unit: `tests/sdk/event/test_llm_completion_log_event.py`. Example: none. Real-LLM: none. - `S05.token-event`: with `litellm_extra_body={"return_token_ids": True}` (vLLM), the agent emits a `TokenEvent` with prompt and response token ids. Unit: **none (untested)**. Example: none. Real-LLM: none. @@ -78,6 +78,30 @@ Preconditions: --check auth_kind eq auth --check rate_limit_kind eq rate_limit --check code_beats_detail eq internal \ --check secret_in_classification eq false ``` +- **LLM failures that end a run (`S05.error-classification`).** A conversation whose real `LLM` (`openai/gpt-4o-mini`) talks to `programs/S02/fake_provider.py`: an endpoint that never answers (`timeout=1`, `num_retries=5`), and an endpoint whose every reply is content-free (`num_retries=2`); then `classify_error` on the text of one quick timeout. + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S05/llm_failures --arg exhausted \ + --check hard_timeout.status eq error --check hard_timeout.requests eq 2 --check hard_timeout.errors eq 1 \ + --check hard_timeout.code eq LLMTimeoutError --check hard_timeout.detail contains "within the LLM timeout of 1s" \ + --check hard_timeout.classification.kind eq config --check hard_timeout.classification.retryable eq false \ + --check hard_timeout.classification.user_action eq settings \ + --check content_free.status eq error --check content_free.errors eq 1 --check content_free.code eq LLMNoResponseError \ + --check content_free.classification.kind eq transient --check content_free.classification.retryable eq true \ + --check content_free.classification.user_action eq retry \ + --check single_quick_timeout.kind eq transient --check single_quick_timeout.retryable eq true + ``` + The timed-out run's `ConversationErrorEvent` points the user at the LLM settings rather than at another identical attempt; the content-free run's event stays a retryable transient failure. +- **A run of quick timeouts (`S05.error-classification`).** The same conversation with a streamed `LLM` whose every stream stalls past `stream_idle_timeout=0.3`, far inside `timeout=30` (`num_retries=3`, `await conversation.arun()`). + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S05/llm_failures --arg quick \ + --check quick_timeout.status eq error --check quick_timeout.requests eq 3 --check quick_timeout.errors eq 1 \ + --check quick_timeout.code eq LLMTimeoutError \ + --check quick_timeout.classification.kind eq transient --check quick_timeout.classification.retryable eq true \ + --check quick_timeout.classification.user_action eq retry + ``` + Expected: the stalls never come near the LLM timeout, so the event stays `transient`, retryable, `user_action=retry`, like any timeout that did not use the whole LLM timeout. Known failure (reproduced 2026-11-15, [#6864](https://github.com/OpenHands/software-agent-sdk/issues/6864)): all three attempts run, but the final `LLMTimeoutError` reads `LLM stream idle timeout after 0.3 seconds. The model did not return a complete response within the LLM timeout of 30s (gave up after 3 attempts, 1s in total). ... or a higher timeout in the LLM settings.`, and the event is classified `config`, not retryable, `user_action=settings`: every timeout that runs out of attempts gets the hard-timeout summary, and the classifier keys on that summary. - **Concurrent appends (`S05.event-log-append`).** Four threads append 40 events to a finished conversation's log, as two parts of one program would. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) diff --git a/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md b/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md index 5cd448bedb0..2ed0f2a8a06 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md +++ b/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md @@ -201,7 +201,7 @@ Preconditions: --check thinking.arun.persisted_masked_thinking eq true ``` The value appears nowhere in the conversation's event files. The masked block has no signature, because the provider's signature would not match the masked text. -- **A real model labels its own actions (`S13.llm-analyzer`, `S13.confirm-wait`, `S13.confirm-approve-reject`).** Live. The model is asked to run `ls` and then `rm -rf scratch`; `ConfirmRisky()` stops, and the program rejects until the agent finishes. A model that declines to propose the deletion at all (DeepSeek Flash did on 2026-11-03: it only listed and read `scratch/`) leaves the confirmation path unexercised, and the program exits 3. +- **A real model labels its own actions (`S13.llm-analyzer`, `S13.confirm-wait`, `S13.confirm-approve-reject`).** Live. The model is asked to run `ls` and then `rm -rf scratch`; `ConfirmRisky()` stops, and the program rejects until the agent finishes. A model that declines to propose the deletion at all (DeepSeek Flash did on 2026-11-03: it only listed and read `scratch/`) leaves the confirmation path unexercised, and the program exits 3. The risk label is the model's own judgment: DeepSeek Flash labels `rm -rf scratch` `MEDIUM` in many runs, which `ConfirmRisky()` lets through (`stops` 0, `scratch_kept` false), so this bullet can fail on any revision without a product change; read the `risks` it printed before suspecting the analyzer. ```sh export SDK_VERIFY_RUN=$(control-sdk new --llm deepseek-flash --field run) control-sdk program S13/live_risk \ diff --git a/.agents/skills/verify-sdk/references/feature-map/S15-browser.md b/.agents/skills/verify-sdk/references/feature-map/S15-browser.md index 31b2424b091..f2d0680bda0 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S15-browser.md +++ b/.agents/skills/verify-sdk/references/feature-map/S15-browser.md @@ -155,7 +155,7 @@ Preconditions: ## Gotchas -- The live bullets run on `deepseek/deepseek-flash`, a native `deepseek/` route whose adapter drops images. A step whose history carries an image (a screenshot in a browser observation) is refused before it is sent, and until [#6840](https://github.com/OpenHands/software-agent-sdk/issues/6840) is fixed the refusal is retried and ends the run with `LLMServiceUnavailableError` (`The installed LiteLLM adapter discarded image content`). Whether a run meets it depends on which browser tools the model picks, so `S15.navigate-and-read` and `S15.interact` can fail on one run and pass on the next. +- The live bullets run on `deepseek/deepseek-flash`, a native `deepseek/` route whose adapter drops images. A step whose history carries an image (a screenshot in a browser observation) is refused before it is sent, at once and without retries, as `LLMBadRequestError` (`The installed LiteLLM adapter discarded image content`; `S02.image-adapter-loss`). Whether a run meets it depends on which browser tools the model picks, so `S15.navigate-and-read` and `S15.interact` can fail on one run and pass on the next. - `examples/01_standalone_sdk/15_browser_use.py` is in `_EXCLUDED_EXAMPLES` of `tests/examples/test_examples.py`, so CI never runs it; example 38 does run on the `test-examples` label and exits 1 if the browser cannot initialize. - Recordings are written to `.agent_tmp/browser_observations/` relative to the process working directory, not under `persistence_dir`, although example 38's comments say "persistence directory". Folders from earlier runs stay there; the example reports only the newest `recording-*` folder by name. - Recording loads rrweb from unpkg.com (`RecordingConfig.cdn_url`, 10 s load timeout); if the CDN is unreachable, `browser_start_recording` fails with an error message instead of recording. diff --git a/.agents/skills/verify-sdk/references/feature-map/S20-observability.md b/.agents/skills/verify-sdk/references/feature-map/S20-observability.md index 98c5ea71feb..2ee0834280e 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S20-observability.md +++ b/.agents/skills/verify-sdk/references/feature-map/S20-observability.md @@ -1,6 +1,6 @@ # Observability, tracing and logging -A Python program gets OpenTelemetry traces of its conversations by setting environment variables before it imports the SDK: `LMNR_PROJECT_API_KEY` sends them to Laminar, and `OTEL_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` send them to any OTLP collector through Laminar's exporter. With none of these set (and no user-side `Laminar.initialize()`), every `@observe` decorator is a pass-through and `lmnr` is not even imported. When enabled, each `Conversation` owns a long-lived root span named `conversation` whose trace session id is the conversation id, with `user_id`, `observability_metadata` (also as span attributes), `observability_tags`, `conversation.tags.*` attributes and an optional named child span; `conversation.send_message` / `run` / `arun`, `agent.step`, tool calls (one TOOL span per call with `tool_call_id` metadata), MCP calls, condenser and hook calls, and LLM calls (an `LLM` span carrying the request and response messages, token totals, cost and cache buckets) nest under it, across threads and parallel tool workers. Sub-agent conversations start their own trace linked back to the parent. The `openhands.sdk.automation` helpers turn the generic `OPENHANDS_OBSERVABILITY_*` env vars an automation runner sets into `Conversation(...)` kwargs and HTTP headers. Separately, `get_logger(__name__)` returns standard loggers under a root configured on import from `LOG_*` env vars (Rich console, or JSON in CI or with `LOG_JSON`, optional rotating file), with LiteLLM, OpenAI, httpx and libtmux loggers quieted. +A Python program gets OpenTelemetry traces of its conversations by setting environment variables before it imports the SDK: `LMNR_PROJECT_API_KEY` sends them to Laminar, and `OTEL_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` send them to any OTLP collector through Laminar's exporter. With none of these set (and no user-side `Laminar.initialize()`), every `@observe` decorator is a pass-through and `lmnr` is not even imported. When enabled, each `Conversation` owns a long-lived root span named `conversation` whose trace session id is the conversation id, with `user_id`, `observability_metadata` (also as span attributes), `observability_tags`, `conversation.tags.*` attributes and an optional named child span; `conversation.send_message` / `run` / `arun`, `agent.step`, tool calls (one TOOL span per call with `tool_call_id` metadata), MCP calls, condenser and hook calls, and LLM calls (an `LLM` span carrying the request and response messages, token totals, cost and cache buckets) nest under it, across threads and parallel tool workers. Sub-agent conversations start their own trace linked back to the parent. The `openhands.sdk.automation` helpers turn the generic `OPENHANDS_OBSERVABILITY_*` env vars an automation runner sets into `Conversation(...)` kwargs and HTTP headers. Separately, `get_logger(__name__)` returns standard loggers under a root configured on import from `LOG_*` env vars (Rich console, or JSON in CI or with `LOG_JSON`, optional rotating file), with LiteLLM, OpenAI, httpx and libtmux loggers quieted. Every failed attempt of an LLM call logs one `LLM request failed:` warning that says what failed and what the retry policy decided, without the provider's message. Source: `openhands-sdk/openhands/sdk/observability/`, `openhands-sdk/openhands/sdk/automation/`, `openhands-sdk/openhands/sdk/logger/`, `openhands-sdk/openhands/sdk/conversation/base.py`, `openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py`, `openhands-sdk/openhands/sdk/agent/agent.py`, `openhands-sdk/openhands/sdk/agent/parallel_executor.py`, `openhands-sdk/openhands/sdk/llm/` @@ -30,6 +30,7 @@ Examples: `examples/01_standalone_sdk/27_observability_laminar.py` - `S20.get-env`: observability reads env vars straight from `os.environ`, never through python-dotenv's `find_dotenv()`. Unit: `tests/sdk/observability/test_utils.py`. Example: none. Real-LLM: none. - `S20.logger-quiet-thirdparty`: importing the SDK sets LiteLLM and OpenAI loggers to ERROR (DEBUG only after a confirmed `DEBUG_LLM`), and httpx, httpcore and libtmux to WARNING, so LiteLLM INFO lines do not show. Unit: `tests/sdk/logger/test_litellm_log_suppression.py`. Example: none. Real-LLM: none. - `S20.logger-config`: `get_logger(name)` returns a propagating stdlib logger; `setup_logging(...)` (auto-run on import unless `LOG_AUTO_CONFIG=false`) honors `LOG_LEVEL`, `DEBUG`, `LOG_JSON`, `CI`/`GITHUB_ACTIONS` (JSON console), `LOG_RICH_TRACEBACKS`, and `LOG_TO_FILE` with `LOG_DIR`, `LOG_ROTATE_WHEN`, `LOG_BACKUP_COUNT` (`/app.log`), and does not add a second stream handler. Unit: **none (untested)**. Example: none. Real-LLM: none. +- `S20.llm-retry-log`: each failed LLM attempt logs one WARNING `LLM request failed: exception_type= status_code= attempt=N max_attempts=M retry_eligible= will_retry=`, for retried, exhausted and non-retryable failures alike; the line never carries the exception message, so a provider error body (and any credential in it) stays out of the logs. Unit: `tests/sdk/llm/test_retry_diagnostics.py` (`test_failure_logging_explains_retry_decision`, `test_failure_logging_handles_optional_status_safely`, `test_async_retry_logs_failure_and_preserves_success`, `test_long_retry_after_logs_no_retry`). Example: none. Real-LLM: none. - `S20.rolling-log-view`: `rolling_log_view(logger, max_lines=...)` shows a live last-N-lines panel on a TTY, plain lines in CI, and one buffered JSON record on exit in JSON mode. Unit: **none (untested)**. Example: none. Real-LLM: none. ## How to get to it (agent POV) @@ -39,6 +40,7 @@ Examples: `examples/01_standalone_sdk/27_observability_laminar.py` - `openhands.sdk.observability`: `observe(name=..., span_type="TOOL", ...)` to trace a user function, `maybe_init_laminar()`, `init_laminar_for_external()`, `OPERATION_METADATA_KEY`. - `OPENHANDS_OBSERVABILITY_METADATA` (JSON object), `OPENHANDS_OBSERVABILITY_TAGS`, `OPENHANDS_OBSERVABILITY_SPAN_NAME`, `OPENHANDS_OBSERVABILITY_PARENT_SPAN_CONTEXT`, and `openhands.sdk.automation`: `automation_observability_context`, `automation_conversation_kwargs`, `automation_conversation_tags`, `automation_observability_headers` (`S20.automation-*`, `S20.env-conversation-defaults`). - `openhands.sdk.logger` (also `openhands.sdk.get_logger`): `get_logger`, `setup_logging`, `rolling_log_view`; env `LOG_LEVEL`, `DEBUG`, `LOG_JSON`, `LOG_TO_FILE`, `LOG_DIR`, `LOG_ROTATE_WHEN`, `LOG_BACKUP_COUNT`, `LOG_RICH_TRACEBACKS`, `LOG_AUTO_CONFIG`, `DEBUG_LLM` (`S20.logger-*`, `S20.rolling-log-view`). +- A `logging.Handler` on the root logger (or `openhands.sdk.llm.utils.retry_mixin`) receives the retry lines (`S20.llm-retry-log`). ## Driving it @@ -162,6 +164,24 @@ Preconditions: --check warning_level.console_has_info eq false --check no_auto_config.handlers len-eq 0 ``` Default: a Rich console handler; `LOG_JSON=1` a JSON stream handler and, with `LOG_TO_FILE`, a JSON `/app.log`; `CI=true` JSON on the console; `DEBUG=true` lowers the root to DEBUG; `LOG_AUTO_CONFIG=false` adds no handler. In JSON mode `rolling_log_view` emits one record with the header and the last N lines on exit. +- **Retry log lines (`S20.llm-retry-log`).** A real `LLM` against `programs/S02/fake_provider.py`, with a handler on the root logger; every error body carries the fake secret `sk-leak-RETRYLOG-7731`. A 503 then a reply (`num_retries=3`); a 400; a 429 that never clears (`num_retries=2`); and, on `hosted_vllm/`, an HTTP 200 whose body is not JSON, then a reply. + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S20/retry_log \ + --check transient.lines eq 1 --check transient.levels.0 eq WARNING \ + --check transient.fields.0.exception_type eq litellm.exceptions.ServiceUnavailableError \ + --check transient.fields.0.status_code eq 503 --check transient.fields.0.attempt eq 1 \ + --check transient.fields.0.max_attempts eq 3 --check transient.fields.0.retry_eligible eq True \ + --check transient.fields.0.will_retry eq True \ + --check non_retryable.lines eq 1 --check non_retryable.fields.0.exception_type eq litellm.exceptions.BadRequestError \ + --check non_retryable.fields.0.retry_eligible eq False --check non_retryable.fields.0.will_retry eq False \ + --check exhausted.lines eq 2 --check exhausted.fields.0.will_retry eq True \ + --check exhausted.fields.1.attempt eq 2 --check exhausted.fields.1.retry_eligible eq True --check exhausted.fields.1.will_retry eq False \ + --check malformed_2xx.error missing --check malformed_2xx.fields.0.exception_type eq litellm.exceptions.APIError \ + --check malformed_2xx.fields.0.status_code eq 200 --check malformed_2xx.fields.0.will_retry eq True \ + --check marker_in_any_log eq false + ``` + One line per failed attempt, naming the LiteLLM exception class with its module and the HTTP status; the last attempt of the exhausted call says it is eligible but will not retry. The fake secret is in no log record, exception text included. - **Live: a real model traced to the local collector (`S20.root-span`, `S20.llm-span`, `S20.llm-span-content`, `S20.step-and-tool-spans`).** ```sh export SDK_VERIFY_RUN=$(control-sdk new --llm deepseek-flash --field run) From ede5eb7d1db8b14aece16ef9677e4203876b1670 Mon Sep 17 00:00:00 2001 From: enyst Date: Sun, 15 Nov 2026 09:44:10 +0000 Subject: [PATCH 2/2] docs(verify-sdk): link the #6864 fix PR from S05 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013ArwwJSRAiRLzhbUrepjY4 --- .agents/skills/verify-sdk/references/feature-map/S05-events.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/verify-sdk/references/feature-map/S05-events.md b/.agents/skills/verify-sdk/references/feature-map/S05-events.md index 397c83883da..2998039d09e 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S05-events.md +++ b/.agents/skills/verify-sdk/references/feature-map/S05-events.md @@ -101,7 +101,7 @@ Preconditions: --check quick_timeout.classification.kind eq transient --check quick_timeout.classification.retryable eq true \ --check quick_timeout.classification.user_action eq retry ``` - Expected: the stalls never come near the LLM timeout, so the event stays `transient`, retryable, `user_action=retry`, like any timeout that did not use the whole LLM timeout. Known failure (reproduced 2026-11-15, [#6864](https://github.com/OpenHands/software-agent-sdk/issues/6864)): all three attempts run, but the final `LLMTimeoutError` reads `LLM stream idle timeout after 0.3 seconds. The model did not return a complete response within the LLM timeout of 30s (gave up after 3 attempts, 1s in total). ... or a higher timeout in the LLM settings.`, and the event is classified `config`, not retryable, `user_action=settings`: every timeout that runs out of attempts gets the hard-timeout summary, and the classifier keys on that summary. + Expected: the stalls never come near the LLM timeout, so the event stays `transient`, retryable, `user_action=retry`, like any timeout that did not use the whole LLM timeout. Known failure (reproduced 2026-11-15, [#6864](https://github.com/OpenHands/software-agent-sdk/issues/6864), fix in [#6866](https://github.com/OpenHands/software-agent-sdk/pull/6866)): all three attempts run, but the final `LLMTimeoutError` reads `LLM stream idle timeout after 0.3 seconds. The model did not return a complete response within the LLM timeout of 30s (gave up after 3 attempts, 1s in total). ... or a higher timeout in the LLM settings.`, and the event is classified `config`, not retryable, `user_action=settings`: every timeout that runs out of attempts gets the hard-timeout summary, and the classifier keys on that summary. - **Concurrent appends (`S05.event-log-append`).** Four threads append 40 events to a finished conversation's log, as two parts of one program would. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run)