diff --git a/.agents/skills/verify-agent-server/references/daily.md b/.agents/skills/verify-agent-server/references/daily.md index 13f04c73f3..df908d1527 100644 --- a/.agents/skills/verify-agent-server/references/daily.md +++ b/.agents/skills/verify-agent-server/references/daily.md @@ -85,6 +85,14 @@ nothing in `BASE..TARGET` touched is not driven: it waits for the weekly pass. it is reproduced and gets its note, so bugs filed outside a pass reach the map. Model-backed recipes run only inside the LLM budget; without a key they are `blocked` with the missing prerequisite. +3. **Twins.** For every family step 2 drove, `control-agent-server map links + --file Fnn` lists its SDK twins (`Same promise:` links). When verify-sdk's + map is in the checkout, replay each twin in-process too (`sdk-map run + --file Snn --only Snn.x`) unless its family is replayed anyway. Put each + pair side by side: a pair whose results disagree is a finding, usually at + the seam between the SDK and the server (serialization, the async path, + persistence, restart), unless the difference is already a known bug on + one side. Drive each bullet once. ## Reading the result diff --git a/.agents/skills/verify-agent-server/references/feature-map/F04-conversation-lifecycle.md b/.agents/skills/verify-agent-server/references/feature-map/F04-conversation-lifecycle.md index 0ef3a7d84d..5d45e71ad5 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F04-conversation-lifecycle.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F04-conversation-lifecycle.md @@ -23,7 +23,7 @@ Routes: `GET /api/conversations/search`, `GET /api/conversations/count`, ## Sub-features -- `F04.create-settings`: a create from the saved settings (`agent_settings` plus `secrets_encrypted`, the Agent Canvas path) returns 201 `idle`, adds one to the count, and persists the agent only in `base_state.json`, never in `meta.json`. +- `F04.create-settings`: a create from the saved settings (`agent_settings` plus `secrets_encrypted`, the Agent Canvas path) returns 201 `idle`, adds one to the count, and persists the agent only in `base_state.json`, never in `meta.json`. Same promise: `S19.settings-create-agent`. - `F04.create-inline-agent`: an inline `agent` without `kind` is accepted as `Agent`; the API key is never echoed back and the response trims inline skills. - `F04.create-agent-profile`: `agent_profile_id` resolves a stored Agent Profile and its LLM profile and records `launched_agent_profile`. - `F04.create-validation`: a missing agent source, two agent sources, a missing workspace, a bad tag key, `max_iterations` 0 and invalid `agent_settings` are 422; an unknown `agent_profile_id` is 404; a profile with a dangling LLM reference is 422 `unresolved_profile_references`; no key is 401; nothing is created. @@ -33,14 +33,14 @@ Routes: `GET /api/conversations/search`, `GET /api/conversations/count`, - `F04.create-initial-sender`: an `initial_message` with `sender_conversation_id` naming another conversation saves the first user message with `sender: "conversation:"` and a first text part naming the sender (`[Sent by [](conversation:<hex>), another conversation, not by the user. ...]`); an unknown sender or the new conversation's own id is 422 and creates nothing. - `F04.create-parent`: `parent_conversation_id` links a child that the parent lists in `sub_conversation_ids`; an unknown parent, a different workspace and a self-parent are 422. - `F04.create-worktree`: `worktree: true` inside a git repository moves the conversation into its own worktree on branch `openhands/<id>`. -- `F04.create-observability`: the `X-OpenHands-Observability-*` headers are validated (422 for non-object metadata) and stored in `meta.json`. -- `F04.create-observability-precedence`: body `observability_span_name`, `observability_tags` and `observability_parent_span_context` win over their headers, header metadata merges under the body's (body keys win), the parent-span header applies when the body has none, and an over-long or malformed span-name header, non-scalar header metadata and an empty body tag are 422. -- `F04.create-secrets`: start-time `StaticSecret` values appear masked in the create response and are encrypted at rest. +- `F04.create-observability`: the `X-OpenHands-Observability-*` headers are validated (422 for non-object metadata) and stored in `meta.json`. Same promise: `S20.conversation-args`, `S20.automation-headers`. +- `F04.create-observability-precedence`: body `observability_span_name`, `observability_tags` and `observability_parent_span_context` win over their headers, header metadata merges under the body's (body keys win), the parent-span header applies when the body has none, and an over-long or malformed span-name header, non-scalar header metadata and an empty body tag are 422. Same promise: `S20.conversation-args`. +- `F04.create-secrets`: start-time `StaticSecret` values appear masked in the create response and are encrypted at rest. Same promise: `S04.secrets-constructor`. - `F04.create-secrets-get`: a start-time secret is listed by `GET /api/conversations/{id}` right after the create. - `F04.create-secrets-plain`: a plain-string secret value in the create body is rejected with 422, never a 500. - `F04.run-real`: an initial message runs on DeepSeek flash to `finished`; the events socket replays the run, the final response is non-empty and autotitle names the conversation. -- `F04.final-response`: `agent_final_response` is empty before the agent answers, carries the exact `finish` message afterwards, and is 404 for an unknown id. -- `F04.title-llm-profile`: with `title_llm_profile` naming a saved LLM profile, autotitle calls that profile and not the agent's LLM; an unknown profile name falls back to the agent's LLM and never fails the create. +- `F04.final-response`: `agent_final_response` is empty before the agent answers, carries the exact `finish` message afterwards, and is 404 for an unknown id. Same promise: `S04.final-response`. +- `F04.title-llm-profile`: with `title_llm_profile` naming a saved LLM profile, autotitle calls that profile and not the agent's LLM; an unknown profile name falls back to the agent's LLM and never fails the create. Same promise: `S04.title-and-tags`. - `F04.get`: `GET /api/conversations/{id}` returns `runtime_info`, trims skills unless `include_skills=true`, accepts the 32-hex id form, and answers 404, 422 and 401. - `F04.batch-get`: `GET /api/conversations?ids=..` returns a list aligned with the ids, `null` for unknown ones, and 422 without or with malformed ids. - `F04.batch-get-limit`: 100 ids in one batch are refused with a client error, not a 500. @@ -49,14 +49,14 @@ Routes: `GET /api/conversations/search`, `GET /api/conversations/count`, - `F04.count`: `GET /api/conversations/count` equals the number of conversations search lists. - `F04.search-at-scale`: with 2,000 conversations, search pages through every one exactly once in both orders, count is 2,000, the last page answers within 4× the first page's p95 (first page under 2 s), 50 listings grow the server's RSS by under 300 MB, and all 2,000 are listed again after a restart. - `F04.patch-title`: PATCH `title` is stripped, stored in `meta.json`, bumps `updated_at` and moves the conversation to the top of `UPDATED_AT_DESC`. -- `F04.patch-tags`: PATCH `tags` replaces the whole map in `meta.json` and `base_state.json` and pushes a `tags` state update on the events socket. +- `F04.patch-tags`: PATCH `tags` replaces the whole map in `meta.json` and `base_state.json` and pushes a `tags` state update on the events socket. Same promise: `S04.title-and-tags`. - `F04.patch-tags-while-running`: while a run holds the conversation's state for a tool step, a tags PATCH waits for it, but GET, the batch get and search already show the new tags (from `meta.json`) although `base_state.json` still has the old ones; once the step ends the PATCH answers 200 and both files agree. - `F04.patch-errors`: an empty or 201-character title, a bad tag key and a malformed id are 422; no key is 401; the stored values are untouched. - `F04.patch-whitespace-title`: a whitespace-only title is rejected instead of erasing the stored title. - `F04.patch-unknown-id`: PATCH on an unknown id answers the 404 the OpenAPI document declares, not 200 `{"success": false}`. - `F04.delete`: DELETE closes the conversation's open sockets with 4004, stops a running conversation, removes its state, keeps the workspace, removes a worktree conversation's worktree but keeps its branch, orphans children, and later reads are 404 (socket close 4004); a malformed id is 422 and no key is 401. - `F04.delete-unknown-id`: DELETE on an unknown or already deleted id answers the 404 the OpenAPI document declares, not 400. -- `F04.restart-persist`: after a restart the catalog, statuses, titles, tags, final responses and event replay are unchanged, and deleted conversations stay deleted. +- `F04.restart-persist`: after a restart the catalog, statuses, titles, tags, final responses and event replay are unchanged, and deleted conversations stay deleted. Same promise: `S04.persist-events`, `S04.resume`. - `F04.read-no-hydrate`: after a restart, GET, the batch get, search and count read a cold conversation without loading it (no `owner_lease.json`), while `agent_final_response` loads it and claims the lease. - `F04.child-start`: `POST .../children` starts a real-model task with fresh history, inherited configuration and a parent link, using a shared directory, an isolated git worktree, or the shared fallback for a non-git workspace. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F05-run-pause-interrupt.md b/.agents/skills/verify-agent-server/references/feature-map/F05-run-pause-interrupt.md index d5bfb30afe..27ac3b4770 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F05-run-pause-interrupt.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F05-run-pause-interrupt.md @@ -23,33 +23,33 @@ Routes: `POST /api/conversations/{conversation_id}/run`, ## Sub-features -- `F05.run-queued`: a message appended with `run: false` leaves the conversation `idle`; `POST /run` returns 200 at once and the agent runs on DeepSeek flash through `running` to `finished`, both transitions pushed on the events socket. -- `F05.run-finished-noop`: `POST /run` on a `finished` conversation without new input returns 200 and the run ends at once: no `execution_status` change on the socket, still `finished`, no new events (no model call). -- `F05.run-new-input`: a new user message moves a `finished` conversation back to `idle`, and `POST /run` runs it to `finished` again. -- `F05.pause-idle`: pausing an idle conversation returns 200, reports `paused` on GET, the status search and count and the events socket, and persists one `PauseEvent`. -- `F05.pause-repeat`: pausing a `paused` conversation again returns 200 and adds no second `PauseEvent`. -- `F05.run-resume-paused`: `POST /run` resumes a `paused` conversation through `running` to `finished`. -- `F05.interrupt-idle`: interrupt with nothing running falls back to pause: `paused`, one `PauseEvent`, no `InterruptEvent`. -- `F05.run-error`: a run whose model provider is unreachable ends in `error` with a `ConversationErrorEvent` (also pushed on the events socket) and is listed by `status=error`. +- `F05.run-queued`: a message appended with `run: false` leaves the conversation `idle`; `POST /run` returns 200 at once and the agent runs on DeepSeek flash through `running` to `finished`, both transitions pushed on the events socket. Same promise: `S01.run-to-finish`. +- `F05.run-finished-noop`: `POST /run` on a `finished` conversation without new input returns 200 and the run ends at once: no `execution_status` change on the socket, still `finished`, no new events (no model call). Same promise: `S01.run-to-finish`. +- `F05.run-new-input`: a new user message moves a `finished` conversation back to `idle`, and `POST /run` runs it to `finished` again. Same promise: `S01.run-to-finish`. +- `F05.pause-idle`: pausing an idle conversation returns 200, reports `paused` on GET, the status search and count and the events socket, and persists one `PauseEvent`. Same promise: `S04.pause`. +- `F05.pause-repeat`: pausing a `paused` conversation again returns 200 and adds no second `PauseEvent`. Same promise: `S04.pause`. +- `F05.run-resume-paused`: `POST /run` resumes a `paused` conversation through `running` to `finished`. Same promise: `S04.pause`. +- `F05.interrupt-idle`: interrupt with nothing running falls back to pause: `paused`, one `PauseEvent`, no `InterruptEvent`. Same promise: `S04.arun-interrupt`. +- `F05.run-error`: a run whose model provider is unreachable ends in `error` with a `ConversationErrorEvent` (also pushed on the events socket) and is listed by `status=error`. Same promise: `S04.run-error`. - `F05.pause-terminal`: pause and interrupt on a `finished` or `error` conversation return 200 and change neither the status nor the event log. - `F05.run-error-responsive`: a run that fails does not stall the server: other requests keep answering while the error is handled. - `F05.unknown-id-run`: `POST /run` on an unknown id returns 404. - `F05.unknown-id-pause`: `POST /pause` and `POST /interrupt` on an unknown id return the documented 404. - `F05.control-errors`: all three routes answer 401 without or with a wrong session key and 422 for a malformed id, and the idle conversation they targeted is untouched (still `idle`, no `PauseEvent`). - `F05.run-conflict`: `POST /run` while a run is active returns 409 `Conversation already running...`, and the active run keeps going without a second model call. -- `F05.pause-running`: pause during a long model call returns 200 at once; the events socket, status search and event log say `paused`, and `/run` stays 409 while the step is still in flight. +- `F05.pause-running`: pause during a long model call returns 200 at once; the events socket, status search and event log say `paused`, and `/run` stays 409 while the step is still in flight. Same promise: `S04.pause`. - `F05.pause-running-get`: after a pause during a long model call, `GET /api/conversations/{id}` and `base_state.json` report `paused` like every other view. - `F05.pause-running-release`: interrupting a paused conversation whose step is still in flight cancels it (`InterruptEvent`), GET catches up to `paused`, and the run slot is free again (a create that needs the slot succeeds). -- `F05.interrupt-running`: interrupt cancels a hanging model call well within the server's 5 s wait: 200, `paused`, an `InterruptEvent` and no `PauseEvent`, pushed on the events socket. -- `F05.run-after-interrupt`: after an interrupt, `POST /run` is accepted at once and the resumed run reaches `finished`. -- `F05.pause-tool-step`: pause while a tool call runs lets that step finish (its observation is recorded), then stops the loop before the next model call: `paused`, one `PauseEvent`, one model request. -- `F05.interrupt-tool-call`: interrupt while a tool call runs returns at once: `paused`, an `InterruptEvent`, an `AgentErrorEvent` for the unfinished call (same `tool_call_id`) and no observation; the resumed run sends that error to the model as the tool result and finishes. +- `F05.interrupt-running`: interrupt cancels a hanging model call well within the server's 5 s wait: 200, `paused`, an `InterruptEvent` and no `PauseEvent`, pushed on the events socket. Same promise: `S04.arun-interrupt`. +- `F05.run-after-interrupt`: after an interrupt, `POST /run` is accepted at once and the resumed run reaches `finished`. Same promise: `S04.arun-interrupt`. +- `F05.pause-tool-step`: pause while a tool call runs lets that step finish (its observation is recorded), then stops the loop before the next model call: `paused`, one `PauseEvent`, one model request. Same promise: `S04.pause`. +- `F05.interrupt-tool-call`: interrupt while a tool call runs returns at once: `paused`, an `InterruptEvent`, an `AgentErrorEvent` for the unfinished call (same `tool_call_id`) and no observation; the resumed run sends that error to the model as the tool result and finishes. Same promise: `S04.cancel-token`. - `F05.run-capacity`: with `max_concurrent_runs` 1, `POST /run` on a second conversation returns 429 while the first holds the slot, leaves it `idle`, and succeeds once the first is interrupted. -- `F05.stuck-detected`: with `stuck_detection` on (the default), a run whose model repeats the same tool call ends `stuck` after four identical action/observation pairs (four model calls), with no `ConversationErrorEvent`; GET, `status=stuck` search and count and the events socket say `stuck`. -- `F05.stuck-rerun`: `POST /run` on a `stuck` conversation with no new input returns 200 and resumes it with a fresh stuck-detection window: the model is called again, and a model that keeps looping ends `stuck` again after four more identical action/observation pairs (as the SDK does, `S01.stuck-recover`). -- `F05.stuck-new-input`: a new user message moves a `stuck` conversation to `idle`, and `POST /run` runs it again to `finished`. -- `F05.max-iterations-stop`: a conversation created with `max_iterations: 6` and `stuck_detection: false` whose model never finishes makes exactly six model calls (past the stuck threshold) and ends `error` with exactly one `ConversationErrorEvent` `code: MaxIterationsReached` naming the limit, on REST and the events socket, never `stuck`. -- `F05.max-iterations-resume`: `POST /run` after `MaxIterationsReached` is accepted and gets a fresh budget of six model calls (a second `MaxIterationsReached`), and once the model calls `finish` the next run ends `finished`. +- `F05.stuck-detected`: with `stuck_detection` on (the default), a run whose model repeats the same tool call ends `stuck` after four identical action/observation pairs (four model calls), with no `ConversationErrorEvent`; GET, `status=stuck` search and count and the events socket say `stuck`. Same promise: `S01.stuck-detect`. +- `F05.stuck-rerun`: `POST /run` on a `stuck` conversation with no new input returns 200 and resumes it with a fresh stuck-detection window: the model is called again, and a model that keeps looping ends `stuck` again after four more identical action/observation pairs (as the SDK does, `S01.stuck-recover`). Same promise: `S01.stuck-recover`. +- `F05.stuck-new-input`: a new user message moves a `stuck` conversation to `idle`, and `POST /run` runs it again to `finished`. Same promise: `S01.stuck-recover`. +- `F05.max-iterations-stop`: a conversation created with `max_iterations: 6` and `stuck_detection: false` whose model never finishes makes exactly six model calls (past the stuck threshold) and ends `error` with exactly one `ConversationErrorEvent` `code: MaxIterationsReached` naming the limit, on REST and the events socket, never `stuck`. Same promise: `S01.max-iterations`. +- `F05.max-iterations-resume`: `POST /run` after `MaxIterationsReached` is accepted and gets a fresh budget of six model calls (a second `MaxIterationsReached`), and once the model calls `finish` the next run ends `finished`. Same promise: `S01.max-iterations`. - `F05.restart-running-graceful`: a conversation running at a graceful restart comes back `paused`, with a `PauseEvent` and an `InterruptEvent`. - `F05.restart-keeps-status`: `finished`, `paused` and `error` statuses survive a restart unchanged. - `F05.restart-hard-running`: a conversation running when the server is SIGKILLed comes back `error` on the next start, and `/run` resumes it. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md b/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md index 0fb9f45665..a14b0c3822 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md @@ -41,13 +41,13 @@ Routes: `GET /api/conversations/{conversation_id}/events/search`, - `F06.batch-post`: `POST .../events/batch` with the same JSON array returns byte for byte what the GET batch returns (events in request order, `null` for an unknown id, `[]` for `[]`), appends nothing, and answers 422 for no body or a non-array body, 404 for an unknown conversation and 401 without a key. - `F06.not-found-auth`: every event route returns 404 for an unknown conversation, 422 for a malformed conversation id and 401 without a valid session key. - `F06.send-validation`: a missing body, a message with an unknown role, string content or an unknown content type is rejected with 422 and nothing is appended. -- `F06.send-image`: a message with a text part and an `image` part (a `data:` URL) is persisted with its `image_urls` intact and returned the same by search, get-one and the socket replay; an `image` part without `image_urls` is a 422 and appends nothing. +- `F06.send-image`: a message with a text part and an `image` part (a `data:` URL) is persisted with its `image_urls` intact and returned the same by search, get-one and the socket replay; an `image` part without `image_urls` is a 422 and appends nothing. Same promise: `S01.image-input`. - `F06.send-non-user-role`: a message with a schema-valid non-user role (`assistant`) is rejected as a client error (4xx), not a 500, and leaves no event behind (not even the system prompt of an empty log). - `F06.send-run`: `POST .../events` with `run: true` drives a real agent turn to `finished`, persisting the kinds of a normal run (user message, `execution_status` running and finished, `ActionEvent`/`ObservationEvent` pairs, `stats`). - `F06.ws-live`: a socket opened before the run receives each durable event live, durable frame IDs match the persisted log, and a transient closing `full_state` snapshot follows the run. -- `F06.ws-streaming-deltas`: a streamed real-model reply emits transient `StreamingDeltaEvent` frames live; REST history and socket replay omit them. +- `F06.ws-streaming-deltas`: a streamed real-model reply emits transient `StreamingDeltaEvent` frames live; REST history and socket replay omit them. Same promise: `S02.streaming`, `S05.streaming-delta`. - `F06.search-timestamp-order`: after a tool-using run, `sort_order=TIMESTAMP` lists events in timestamp order, so a `timestamp__gte` or `resend_mode=since` cut at a listed event keeps every event listed after it. -- `F06.send-resets-finished`: a message appended to a `finished` conversation sets it back to `idle` with a persisted `execution_status` update. +- `F06.send-resets-finished`: a message appended to a `finished` conversation sets it back to `idle` with a persisted `execution_status` update. Same promise: `S01.run-to-finish`. - `F06.ws-snapshot`: the first frame after authentication is a transient `full_state` `ConversationStateUpdateEvent` that never appears in search; without `resend_mode` nothing else is replayed. - `F06.ws-handshake`: the socket accepts first-frame, deprecated query and deprecated header keys; a wrong first-frame key, a first frame that is not an auth frame, or no auth frame within 10 s closes with 4001, a wrong deprecated query or header key is refused at the upgrade (HTTP 403), and an unknown conversation closes with 4004. - `F06.ws-delete-close`: deleting a conversation closes the sockets that follow it. @@ -55,11 +55,11 @@ Routes: `GET /api/conversations/{conversation_id}/events/search`, - `F06.ws-replay-since`: `resend_mode=since` with `after_timestamp` replays exactly the events at or after that timestamp; without `after_timestamp` it replays nothing. - `F06.ws-bad-frame`: an invalid inbound frame returns a `ServerErrorEvent` (`ValidationError`, `AssertionError`, `JSONDecodeError` for text that is not JSON), appends nothing and leaves the socket open; a redundant auth frame is ignored. - `F06.ws-send`: a user message frame sent on the socket is persisted and runs the agent to `finished`, streaming the run on the same socket. -- `F06.restart-persist`: a socket open during a server restart is closed with 1012; event ids, order and count survive the restart, the socket replays the same log, and new messages append after it. +- `F06.restart-persist`: a socket open during a server restart is closed with 1012; event ids, order and count survive the restart, the socket replays the same log, and new messages append after it. Same promise: `S04.resume`. - `F06.send-run-limit`: with every run slot taken, `POST .../events` with `run: true` returns 429 `Message saved...` and the message is persisted; a socket send gets a `ConversationRunLimitExceeded` `ServerErrorEvent`. -- `F06.send-while-running`: `POST .../events` with `run: true` on a conversation that is already running returns 200 (not the 409 of `POST /run`), persists the message, keeps the conversation `running` and starts no second run. +- `F06.send-while-running`: `POST .../events` with `run: true` on a conversation that is already running returns 200 (not the 409 of `POST /run`), persists the message, keeps the conversation `running` and starts no second run. Same promise: `S01.message-while-running`. -- `F06.send-sender-conversation`: `POST .../events` with `sender_conversation_id` naming another existing conversation saves the user `MessageEvent` with `sender: "conversation:<sender id hex>"` and a first text part `[Sent by [<sender title>](conversation:<hex>), another conversation, not by the user. ...]` before the posted parts, the same in search, get-one and the socket replay; a message without the field has `sender: null` and no such line. +- `F06.send-sender-conversation`: `POST .../events` with `sender_conversation_id` naming another existing conversation saves the user `MessageEvent` with `sender: "conversation:<sender id hex>"` and a first text part `[Sent by [<sender title>](conversation:<hex>), another conversation, not by the user. ...]` before the posted parts, the same in search, get-one and the socket replay; a message without the field has `sender: null` and no such line. Same promise: `S04.conversation-sender`. - `F06.send-sender-errors`: a `sender_conversation_id` naming an unknown conversation (422 `Sender conversation not found`), the receiving conversation itself (422 `A conversation cannot message itself`) or a non-UUID (422 at `sender_conversation_id`) is refused and nothing is appended. - `F06.send-during-tool-call`: `POST .../events` to a conversation whose agent is inside a long tool call saves the message and answers at once, before the tool call ends. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md index b1a46de071..705dc809ed 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md @@ -29,7 +29,7 @@ Routes: `WS /sockets/session/{conversation_id}` - `F07.snapshot-compact-stats`: the `full_state` snapshot carries each LLM's aggregate stats (`accumulated_cost`, `accumulated_token_usage`) but not the per-call `costs`, `response_latencies` and `token_usages` lists, so it stays small however long the conversation runs; `GET /api/conversations/{id}` returns the same aggregates with empty per-call lists. - `F07.snapshot-no-skill-bodies`: the `full_state` snapshot and live `agent` state updates carry the agent with `agent_context.skills` emptied, so skill bodies never ride the socket live, while `GET /api/conversations/{id}?include_skills=true` and the stored event keep them. - `F07.full-replay`: `after_seq=-1` (or any value below it) replays every persisted event as `durable` frames with `seq` 0..`through_seq`, in order, once each, with the same ids REST returns. -- `F07.seq-on-disk`: a durable frame's `seq` is the event's on-disk index, the `NNNNN` in `events/event-NNNNN-<event id>.json`. +- `F07.seq-on-disk`: a durable frame's `seq` is the event's on-disk index, the `NNNNN` in `events/event-NNNNN-<event id>.json`. Same promise: `S04.persist-events`, `S05.event-log-files`. - `F07.resume-cursor`: `after_seq=k` replays exactly the events with `seq > k`; `after_seq=through_seq` replays nothing. - `F07.live-durable`: on a live-only connection, an event appended later (a REST message with `run: false`) arrives as a `durable` frame with the next `seq`. - `F07.live-order`: live `durable` frames on one connection arrive in ascending `seq` order. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md b/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md index 7c58e0ce6d..eb00bb4699 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md @@ -24,28 +24,28 @@ Routes: `POST /api/conversations/{conversation_id}/events/respond_to_confirmatio ## Sub-features -- `F08.policy-set`: `POST .../confirmation_policy` switches between `AlwaysConfirm`, `ConfirmRisky` (with `threshold` and `confirm_unknown`) and `NeverConfirm`; GET, `base_state.json` and a `confirmation_policy` state update on the events socket reflect each change at once. +- `F08.policy-set`: `POST .../confirmation_policy` switches between `AlwaysConfirm`, `ConfirmRisky` (with `threshold` and `confirm_unknown`) and `NeverConfirm`; GET, `base_state.json` and a `confirmation_policy` state update on the events socket reflect each change at once. Same promise: `S13.policies`. - `F08.policy-errors`: an unknown policy kind, a `threshold` of `UNKNOWN` or an unknown level, and a missing or empty `policy` are 422; an unknown conversation is 404; no key is 401; the stored policy is unchanged. -- `F08.always-confirm-wait`: after a runtime switch to `AlwaysConfirm`, the agent's next terminal action parks the conversation in `waiting_for_confirmation` (pushed on the socket) with the `ActionEvent` recorded, no observation, and the command not run. -- `F08.confirm-accept`: `accept: true` runs the pending command (observation and file on disk) and the run continues to `finished`. -- `F08.confirm-reject`: `accept: false` records one `UserRejectObservation` with the given reason (REST and socket) and returns to `idle` without running the command; a second reject is a no-op, and a later accept only resumes the agent. +- `F08.always-confirm-wait`: after a runtime switch to `AlwaysConfirm`, the agent's next terminal action parks the conversation in `waiting_for_confirmation` (pushed on the socket) with the `ActionEvent` recorded, no observation, and the command not run. Same promise: `S13.confirm-wait`. +- `F08.confirm-accept`: `accept: true` runs the pending command (observation and file on disk) and the run continues to `finished`. Same promise: `S13.confirm-approve-reject`. +- `F08.confirm-reject`: `accept: false` records one `UserRejectObservation` with the given reason (REST and socket) and returns to `idle` without running the command; a second reject is a no-op, and a later accept only resumes the agent. Same promise: `S13.confirm-approve-reject`. - `F08.confirm-errors`: a missing or non-boolean `accept` and a malformed id are 422; an unknown conversation is 404; no key is 401. - `F08.confirm-capacity`: with the server's run capacity full, accept is 429 and leaves the action pending and unrun; reject still works; accept on a running conversation is a 200 no-op. - `F08.confirm-real`: DeepSeek flash under `AlwaysConfirm` stops before its terminal command and finishes the task once its actions are accepted. - `F08.analyzer-set-clear`: `POST .../security_analyzer` sets `PatternSecurityAnalyzer` or `LLMSecurityAnalyzer` and clears it with `null`; GET, `base_state.json` and the socket reflect each change. - `F08.analyzer-errors`: a missing `security_analyzer` field and an unknown kind are 422; an unknown conversation is 404; no key is 401; the stored analyzer is unchanged. -- `F08.confirm-risky`: under `ConfirmRisky` with `PatternSecurityAnalyzer`, a low-risk command runs unasked and `rm -rf` waits for confirmation; once the analyzer is cleared, an unrated command waits too. -- `F08.secrets-update`: `POST .../secrets` accepts plain strings, `{"value": ...}` and `StaticSecret` objects (200); once the state is saved, GET lists the names with masked values and `base_state.json` holds them encrypted, never in plaintext. +- `F08.confirm-risky`: under `ConfirmRisky` with `PatternSecurityAnalyzer`, a low-risk command runs unasked and `rm -rf` waits for confirmation; once the analyzer is cleared, an unrated command waits too. Same promise: `S13.pattern-analyzer`. +- `F08.secrets-update`: `POST .../secrets` accepts plain strings, `{"value": ...}` and `StaticSecret` objects (200); once the state is saved, GET lists the names with masked values and `base_state.json` holds them encrypted, never in plaintext. Same promise: `S04.secrets-update`. - `F08.secrets-visible`: a secret posted at runtime is listed by GET right away, without waiting for another state change. - `F08.secrets-errors`: a missing or non-object `secrets`, an unknown secret kind and a `LookupSecret` without `url` are 422; an unknown conversation is 404; no key is 401; nothing is stored. - `F08.secrets-bad-value`: a number, `null` or a boolean as a secret value is refused with 422, not a 500. -- `F08.secrets-terminal`: a terminal command that names a runtime secret receives its value, while the observation and the next request to the model show `<secret-hidden>` and the plaintext is stored nowhere. -- `F08.secrets-task-delegate`: a runtime secret of the conversation reaches the terminal of a sub-agent its task tool delegates to; the delegate's observation and its next model request show `<secret-hidden>`, and the plaintext is stored nowhere. +- `F08.secrets-terminal`: a terminal command that names a runtime secret receives its value, while the observation and the next request to the model show `<secret-hidden>` and the plaintext is stored nowhere. Same promise: `S04.secrets-masking`, `S13.secret-injection`, `S13.mask-output`, `S14.terminal-secret-masking`. +- `F08.secrets-task-delegate`: a runtime secret of the conversation reaches the terminal of a sub-agent its task tool delegates to; the delegate's observation and its next model request show `<secret-hidden>`, and the plaintext is stored nowhere. Same promise: `S11.task-secrets`. - `F08.secrets-agent-message`: a registered secret value in the model's text reply is stored and returned as `<secret-hidden>` in the agent's `MessageEvent`, and the plaintext is stored nowhere. - `F08.secrets-git-provider`: a `GITHUB_TOKEN` runtime secret is what the agent's `git` uses for `https://github.com`, without the command naming the secret. -- `F08.secrets-override`: a runtime secret replaces a start-time secret of the same name for the next command. -- `F08.secrets-lookup`: a runtime `LookupSecret` is fetched from its URL with its headers only when a command names it; the command gets the fetched text, the observation and the model see `<secret-hidden>`, and GET hides the `Authorization` header that `base_state.json` keeps encrypted. -- `F08.secrets-lookup-self`: a runtime `LookupSecret` whose URL is the server's own `/api/settings/secrets/<name>` (hostless, made absolute from `OH_INTERNAL_SERVER_URL`, or absolute on the run's address) is answered in process from the secrets store with no HTTP request and no session key; a name the store lacks falls through to HTTP (401 without a key) and the command gets no value. +- `F08.secrets-override`: a runtime secret replaces a start-time secret of the same name for the next command. Same promise: `S04.secrets-update`, `S13.secret-registry`. +- `F08.secrets-lookup`: a runtime `LookupSecret` is fetched from its URL with its headers only when a command names it; the command gets the fetched text, the observation and the model see `<secret-hidden>`, and GET hides the `Authorization` header that `base_state.json` keeps encrypted. Same promise: `S13.secret-sources`. +- `F08.secrets-lookup-self`: a runtime `LookupSecret` whose URL is the server's own `/api/settings/secrets/<name>` (hostless, made absolute from `OH_INTERNAL_SERVER_URL`, or absolute on the run's address) is answered in process from the secrets store with no HTTP request and no session key; a name the store lacks falls through to HTTP (401 without a key) and the command gets no value. Same promise: `S13.secret-sources`. - `F08.secrets-profile-scope`: for a conversation launched from an Agent Profile with `secret_refs`, names outside the list are silently dropped (200). - `F08.secrets-no-cipher`: on a server with no cipher (no `OH_SECRET_KEY`, no session key) a runtime secret reaches commands while the conversation lives, is persisted only as the redaction placeholder (the server warns at startup), and is empty after a restart although GET still lists the name. - `F08.secrets-restart`: a runtime secret that reached `base_state.json` is still injected into commands after a server restart. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F09-model-switching-plugins.md b/.agents/skills/verify-agent-server/references/feature-map/F09-model-switching-plugins.md index 2284244f25..3ab71319fa 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F09-model-switching-plugins.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F09-model-switching-plugins.md @@ -34,53 +34,53 @@ Routes: `POST /api/conversations/{conversation_id}/switch_profile`, ## Sub-features - `F09.auth-required`: all five routes answer 401 without a valid session key and change nothing. -- `F09.switch-llm`: `switch_llm` with a fresh `usage_id` answers 200 `{"success": true}`; `GET` and `base_state.json` show the new model and `usage_id`, the condenser LLM that mirrored the old one follows, and the key is never echoed or stored in plaintext. +- `F09.switch-llm`: `switch_llm` with a fresh `usage_id` answers 200 `{"success": true}`; `GET` and `base_state.json` show the new model and `usage_id`, the condenser LLM that mirrored the old one follows, and the key is never echoed or stored in plaintext. Same promise: `S03.switch-llm-object`. - `F09.switch-llm-ws`: a switch pushes a `ConversationStateUpdateEvent` with key `agent` and the new model on the events socket, with the key masked. - `F09.switch-llm-errors`: an unknown conversation is 404, a body without `llm` or without `llm.model` is 422, and a dangling `provider_connection_id` is 422; the installed LLM is unchanged. - `F09.switch-llm-encrypted-key`: mid-conversation, a switch whose `api_key` is the `gAAAAA` token from an `encrypted` profile read is decrypted before install; the next turn runs on DeepSeek and its usage is recorded under the new `usage_id`. - `F09.switch-llm-provider-connection`: a switch that references a saved provider connection instead of a key runs the next turn with the connection's key; `GET` shows the `provider_connection_id` and no key. -- `F09.switch-llm-registered-usage-id`: after the agent initialized, a switch whose `usage_id` is already registered (the `default` a saved profile's config carries) with a different config installs the posted LLM, never a silent 200 no-op. +- `F09.switch-llm-registered-usage-id`: after the agent initialized, a switch whose `usage_id` is already registered (the `default` a saved profile's config carries) with a different config installs the posted LLM, never a silent 200 no-op. Same promise: `S03.switch-llm-object`. - `F09.switch-llm-before-first-run`: a switch before the conversation's first run is used by that run, and the run's token usage is recorded under the switched LLM's `usage_id`. - `F09.switch-llm-mid-run`: a `switch_llm` while a turn is running (the agent is inside a 20 s tool call) answers 200, the socket pushes the new `agent`, and the same run's next LLM call already uses the switched LLM (usage under both `usage_id`s) and the run finishes. - `F09.switch-llm-mid-run-get`: right after a mid-run switch answers 200, `GET` and `base_state.json` show the switched LLM, as the socket already does. -- `F09.switch-profile`: `switch_profile` to the saved `deepseek-pro` profile answers 200, installs it under `usage_id` `profile:deepseek-pro`, and the next turn runs on `deepseek/deepseek-v4-pro` with its usage recorded under that id. -- `F09.switch-profile-errors`: an unknown profile is 404 `Profile '<name>' not found`, an invalid name and a corrupted profile file are 400, a missing `profile_name` is 422 and an unknown conversation 404; the installed LLM is unchanged. +- `F09.switch-profile`: `switch_profile` to the saved `deepseek-pro` profile answers 200, installs it under `usage_id` `profile:deepseek-pro`, and the next turn runs on `deepseek/deepseek-v4-pro` with its usage recorded under that id. Same promise: `S03.switch-profile`. +- `F09.switch-profile-errors`: an unknown profile is 404 `Profile '<name>' not found`, an invalid name and a corrupted profile file are 400, a missing `profile_name` is 422 and an unknown conversation 404; the installed LLM is unchanged. Same promise: `S03.switch-profile`. - `F09.switch-profile-edited`: switching to a profile again after it was edited installs the edited config, not the copy cached at the first switch. - `F09.acp-model-native`: `switch_acp_model` on a regular (non-ACP) conversation is 400 `switch_acp_model is only supported for ACP conversations.`; a missing `model` is 422 and an unknown conversation 404. -- `F09.acp-model-deferred`: on an ACP conversation that has not run yet, `switch_acp_model` answers 200 without starting the subprocess and persists the model as `agent.acp_model`, `agent_state.acp_current_model_id` and `current_model_id`. -- `F09.acp-model-blank`: a blank or whitespace-only ACP model is refused before the first run, as it is on a live session, and the stored model is kept. +- `F09.acp-model-deferred`: on an ACP conversation that has not run yet, `switch_acp_model` answers 200 without starting the subprocess and persists the model as `agent.acp_model`, `agent_state.acp_current_model_id` and `current_model_id`. Same promise: `S21.model-switch`. +- `F09.acp-model-blank`: a blank or whitespace-only ACP model is refused before the first run, as it is on a live session, and the stored model is kept. Same promise: `S21.model-switch`. - `F09.switch-llm-acp-refused`: `switch_llm` and `switch_profile` on an ACP conversation are refused, as `switch_acp_model` is on a regular one, and leave the ACP agent's placeholder LLM (`usage_id` `acp-managed`) alone. -- `F09.acp-model-live`: on an ACP conversation with a live session, `switch_acp_model` switches the subprocess's model in place (same ACP session) and `current_model_id`, `acp_model` and `base_state.json` follow; a blank model is 400, a model the provider does not offer is refused and changes nothing, and the next turn runs on the switched model in the same session. -- `F09.acp-model-live-unknown`: a live switch to a model id the provider does not offer (not in `available_models`) is a client error (400, as for a blank model), not a 500. +- `F09.acp-model-live`: on an ACP conversation with a live session, `switch_acp_model` switches the subprocess's model in place (same ACP session) and `current_model_id`, `acp_model` and `base_state.json` follow; a blank model is 400, a model the provider does not offer is refused and changes nothing, and the next turn runs on the switched model in the same session. Same promise: `S21.model-switch`. +- `F09.acp-model-live-unknown`: a live switch to a model id the provider does not offer (not in `available_models`) is a client error (400, as for a blank model), not a 500. Same promise: `S21.model-switch`. - `F09.acp-model-live-timeout`: a live switch whose `session/set_model` round-trip outlasts `acp_prompt_timeout` is 504 with `detail` `Internal Server Error` and leaves the model unchanged. - `F09.acp-effort-native`: `switch_acp_effort` on a regular (non-ACP) conversation is 400 `switch_acp_effort is only supported for ACP conversations.`; a missing `effort` is 422 and an unknown conversation 404. -- `F09.acp-effort-deferred`: on an ACP conversation that has not run yet, `switch_acp_effort` answers 200 without starting the subprocess and persists the level as `agent.acp_effort`, `agent_state.acp_current_effort` and `current_effort` (no level list exists before a session, so any non-blank level is stored); a blank level is 400. +- `F09.acp-effort-deferred`: on an ACP conversation that has not run yet, `switch_acp_effort` answers 200 without starting the subprocess and persists the level as `agent.acp_effort`, `agent_state.acp_current_effort` and `current_effort` (no level list exists before a session, so any non-blank level is stored); a blank level is 400. Same promise: `S21.effort-switch`. - `F09.acp-model-id-bare`: on a live ACP session `current_model_id` is the bare model id: a legacy `<model>/<effort>` `acp_model` (`sonnet/high`) still starts the session, `current_model_id` is `sonnet`, and an explicit `acp_effort` wins over the suffix in `current_effort`. -- `F09.acp-effort-live`: on a live ACP session, `switch_acp_effort` to a level in `available_efforts` answers 200 and switches the session in place (same `acp_session_id`); `current_effort`, `agent.acp_effort`, `agent_state.acp_current_effort` and `base_state.json` follow, and the next turn finishes in the same session. -- `F09.acp-effort-unoffered`: on a live session, a level the current model does not offer is 400 naming the offered levels, a blank level is 400, and `current_effort` and `acp_effort` are unchanged. -- `F09.acp-effort-model-switch`: after `switch_acp_model` to another model (`haiku`) and back (`sonnet`), the pinned `acp_effort` is kept and applied on each model that offers it (`current_effort` follows, `available_efforts` lists that model's levels), in the same session. +- `F09.acp-effort-live`: on a live ACP session, `switch_acp_effort` to a level in `available_efforts` answers 200 and switches the session in place (same `acp_session_id`); `current_effort`, `agent.acp_effort`, `agent_state.acp_current_effort` and `base_state.json` follow, and the next turn finishes in the same session. Same promise: `S21.effort-switch`. +- `F09.acp-effort-unoffered`: on a live session, a level the current model does not offer is 400 naming the offered levels, a blank level is 400, and `current_effort` and `acp_effort` are unchanged. Same promise: `S21.effort-switch`. +- `F09.acp-effort-model-switch`: after `switch_acp_model` to another model (`haiku`) and back (`sonnet`), the pinned `acp_effort` is kept and applied on each model that offers it (`current_effort` follows, `available_efforts` lists that model's levels), in the same session. Same promise: `S21.effort`. - `F09.acp-effort-live-timeout`: a live `switch_acp_effort` whose round-trip outlasts `acp_prompt_timeout` is 504 with `detail` `Internal Server Error` and leaves the effort unchanged. - `F09.acp-effort-codex`: on a Codex (`codex`) ACP conversation signed in with a ChatGPT subscription, `available_efforts` lists the levels the current model offers and `switch_acp_effort` switches among them live (blocked: needs a ChatGPT login). - `F09.ts-client`: the TypeScript client's `ConversationClient.switchLLM`, `switchProfile` and `switchAcpModel` and `ConversationManager.switchProfile` reach the routes: each switch resolves and reads back through `getConversation`, an unknown profile rejects with 404, `switchAcpModel` on a regular conversation with 400, a call without a key with 401, and `switchAcpModel` on an ACP conversation that has not run resolves and stores the model. - `F09.ts-client-acp-docs`: the TypeScript client's `switchAcpModel` docstrings describe what the server does before an ACP conversation's first run (200, the model is stored for the first session), not a 409. - `F09.ts-client-acp-effort`: the TypeScript client's `ConversationClient.switchAcpEffort` stores the level on an ACP conversation that has not run (`getConversation` shows `acp_effort` and `current_effort`), and rejects with 400 on a regular conversation and with 401 without a key. -- `F09.load-plugin`: `load_plugin` with `<plugin>@<marketplace>` from a local marketplace registered on the agent's `agent_context` answers 200 and merges the plugin's skill, its command and its hooks into the live agent, visible in `GET` and `base_state.json`; the socket pushes the `agent` and `hook_config` updates, the `agent` one without skill bodies (`skills: []`). -- `F09.load-plugin-hook-runs`: after a runtime load, the next user turn runs the plugin's `UserPromptSubmit` hook (a `HookExecutionEvent` with its output) and finishes. -- `F09.load-plugin-errors`: no registered marketplaces and no agent context are 400, a registered marketplace whose source is missing is 400, an unknown marketplace or plugin is 404, empty and malformed refs are 400, a missing `plugin_ref` is 422 and an unknown conversation 404; a bare plugin name resolves across the registered marketplaces. -- `F09.load-plugin-twice`: loading the same plugin twice leaves one copy of its hooks, so they still run once per message. -- `F09.load-plugin-sdk`: the Python SDK's `RemoteConversation.load_plugin(ref)`, on a conversation reattached with `RemoteConversation.attach`, raises `HTTPStatusError` 404 for an unknown plugin and then loads the plugin, whose skill and hook `GET` shows. +- `F09.load-plugin`: `load_plugin` with `<plugin>@<marketplace>` from a local marketplace registered on the agent's `agent_context` answers 200 and merges the plugin's skill, its command and its hooks into the live agent, visible in `GET` and `base_state.json`; the socket pushes the `agent` and `hook_config` updates, the `agent` one without skill bodies (`skills: []`). Same promise: `S08.load-plugin-runtime`. +- `F09.load-plugin-hook-runs`: after a runtime load, the next user turn runs the plugin's `UserPromptSubmit` hook (a `HookExecutionEvent` with its output) and finishes. Same promise: `S10.plugin-hooks-merge`. +- `F09.load-plugin-errors`: no registered marketplaces and no agent context are 400, a registered marketplace whose source is missing is 400, an unknown marketplace or plugin is 404, empty and malformed refs are 400, a missing `plugin_ref` is 422 and an unknown conversation 404; a bare plugin name resolves across the registered marketplaces. Same promise: `S08.marketplace-registry`, `S08.load-plugin-runtime`. +- `F09.load-plugin-twice`: loading the same plugin twice leaves one copy of its hooks, so they still run once per message. Same promise: `S08.load-plugin-runtime`, `S10.plugin-hooks-reload-once`. +- `F09.load-plugin-sdk`: the Python SDK's `RemoteConversation.load_plugin(ref)`, on a conversation reattached with `RemoteConversation.attach`, raises `HTTPStatusError` 404 for an unknown plugin and then loads the plugin, whose skill and hook `GET` shows. Same promise: `S08.load-plugin-runtime`. - `F09.restart-persist`: after a restart the switched LLM, the switched profile (which the next turn runs on), the deferred ACP model and the loaded plugin's skills and hooks are still in place. -- `F09.model-router-status`: `GET .../model_router` on a plain-LLM conversation is 200 `{"enabled": false, "selection": null, "last_route_name": null, "routes": []}`; on a conversation created with a `MultimodalRouter` LLM it reports `enabled: true`, `selection.mode` `auto`, no `last_route_name` and each route's `name`, `model` and `label`; an unknown conversation is 404 and no key 401. -- `F09.model-router-add-route`: `POST .../model_router/routes` with a saved profile wraps a plain LLM in a `base_router` (the old LLM becomes route `default`, or `base_route_name`), adds the profile as a route named `route_name` (default: the profile name) with `usage_id` `profile:<name>`, extends an existing router keeping its type, stays in `auto` mode and answers the new status; `GET`, `base_state.json` (keys encrypted) and the socket's `agent` update show the routes, and no plaintext key is stored or pushed. +- `F09.model-router-status`: `GET .../model_router` on a plain-LLM conversation is 200 `{"enabled": false, "selection": null, "last_route_name": null, "routes": []}`; on a conversation created with a `MultimodalRouter` LLM it reports `enabled: true`, `selection.mode` `auto`, no `last_route_name` and each route's `name`, `model` and `label`; an unknown conversation is 404 and no key 401. Same promise: `S03.model-router-selection`. +- `F09.model-router-add-route`: `POST .../model_router/routes` with a saved profile wraps a plain LLM in a `base_router` (the old LLM becomes route `default`, or `base_route_name`), adds the profile as a route named `route_name` (default: the profile name) with `usage_id` `profile:<name>`, extends an existing router keeping its type, stays in `auto` mode and answers the new status; `GET`, `base_state.json` (keys encrypted) and the socket's `agent` update show the routes, and no plaintext key is stored or pushed. Same promise: `S03.model-router-add-route`. - `F09.model-router-add-route-errors`: an unknown profile is 404 `Profile '<name>' not found`, a route name already in use 400 `Model router route already exists: <name>`, an invalid profile name 400, a missing `profile_name` 422, an unknown conversation 404 and no key 401; the routes are unchanged. -- `F09.model-router-select`: `PATCH .../model_router` with `{"selection": {"mode": "manual", "manual_route_name": ...}}` pins that route and `{"mode": "auto"}` unpins it, keeping the router and its routes; the answer, `GET .../model_router`, `GET` (`agent.llm.selection`), `base_state.json` and the socket's `agent` update all show the selection. +- `F09.model-router-select`: `PATCH .../model_router` with `{"selection": {"mode": "manual", "manual_route_name": ...}}` pins that route and `{"mode": "auto"}` unpins it, keeping the router and its routes; the answer, `GET .../model_router`, `GET` (`agent.llm.selection`), `base_state.json` and the socket's `agent` update all show the selection. Same promise: `S03.model-router-selection`. - `F09.model-router-select-errors`: an unknown route is 400 naming the available routes, `manual` without a route, `auto` with one, an unknown `mode` and a missing `selection` are 422, a conversation whose LLM is not a router is 400 `Current agent LLM is not a RouterLLM.`, an unknown conversation 404 and no key 401; the selection is unchanged. -- `F09.model-router-switch-replaces`: `switch_profile` and `switch_llm` on a router conversation install the plain LLM in place of the router (routes and pin gone, `model_router` reports `enabled: false`, `PATCH` is then 400), and a later route add wraps the switched LLM in a new router. +- `F09.model-router-switch-replaces`: `switch_profile` and `switch_llm` on a router conversation install the plain LLM in place of the router (routes and pin gone, `model_router` reports `enabled: false`, `PATCH` is then 400), and a later route add wraps the switched LLM in a new router. Same promise: `S03.model-router-switch-replaces`. - `F09.model-router-acp-refused`: on an ACP conversation `GET .../model_router` reports `enabled: false`, and adding a route or changing the selection is 400 pointing at `switch_acp_model`, leaving the `acp-managed` placeholder LLM alone. - `F09.model-router-sdk`: the Python SDK's `RemoteConversation.get_model_router()`, `add_model_router_route(profile, route_name=, base_route_name=)`, `update_model_router_selection(selection)`, `switch_routed_llm(route)` and `resume_auto_llm_routing()` reach the routes and return `ModelRouterStatus`; a refused route or profile raises `httpx.HTTPStatusError` 400 or 404. - `F09.ts-client-model-router`: the TypeScript client's `getModelRouter`, `addModelRouterRoute`, `updateModelRouter`, `switchModelRouter` and `resumeAutoModelRouting` on `ConversationClient` and `ConversationManager` reach the routes and resolve with the status; an unknown route rejects with 400, an unknown profile 404, a duplicate route 400 and a keyless client 401. - `F09.model-router-run`: after a first turn, a route added from the `deepseek-flash` profile and pinned serves the next turn (its tokens under `profile:deepseek-flash`, `last_route_name` the route), and after a return to `auto` the next turn runs on the first route (`default`) again. -- `F09.model-router-before-first-run`: a route added and pinned before the conversation's first message serves the first run, and that run's tokens are recorded under the route's `usage_id` `profile:deepseek-flash`. +- `F09.model-router-before-first-run`: a route added and pinned before the conversation's first message serves the first run, and that run's tokens are recorded under the route's `usage_id` `profile:deepseek-flash`. Same promise: `S03.model-router-before-first-run`. - `F09.model-router-restart`: after a restart the routes and the pinned selection are still in place and the selection can be changed again; conversations whose router a switch replaced stay plain. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md b/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md index 5f3e1e8de4..153672c3ff 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md @@ -22,30 +22,30 @@ Routes: `POST /api/conversations/{conversation_id}/ask_agent`, ## Sub-features -- `F10.fork-whole`: `POST /fork` with no body answers 201 with a new `idle` conversation that names its source in `forked_from_conversation_id`, keeps the source's HEAD and working directory, and holds a copy of every event (same ids) in its own persistence directory; the source is unchanged. -- `F10.fork-named`: a fork request's `id`, `title` and `tags` become the fork's id, title and tags on the response, on GET and in `meta.json`; the title does not add a tag. +- `F10.fork-whole`: `POST /fork` with no body answers 201 with a new `idle` conversation that names its source in `forked_from_conversation_id`, keeps the source's HEAD and working directory, and holds a copy of every event (same ids) in its own persistence directory; the source is unchanged. Same promise: `S04.fork`. +- `F10.fork-named`: a fork request's `id`, `title` and `tags` become the fork's id, title and tags on the response, on GET and in `meta.json`; the title does not add a tag. Same promise: `S04.fork`. - `F10.fork-tags-consistent`: when a fork request gives no tags, the fork's tags on GET and in `meta.json` are both empty; the source's tags are not inherited. -- `F10.fork-from-event`: `from_event_id` copies only the branch up to and including that event, sets the fork's HEAD and `forked_from_event_id` to it, and leaves later events out. -- `F10.fork-metrics`: by default the fork's `stats` start empty; `reset_metrics: false` copies the source's accumulated cost and token usage. -- `F10.fork-independent`: messages appended to the fork never reach the source and vice versa; the fork's events socket replays the copied history plus its own events. +- `F10.fork-from-event`: `from_event_id` copies only the branch up to and including that event, sets the fork's HEAD and `forked_from_event_id` to it, and leaves later events out. Same promise: `S04.fork-branch`. +- `F10.fork-metrics`: by default the fork's `stats` start empty; `reset_metrics: false` copies the source's accumulated cost and token usage. Same promise: `S04.fork`. +- `F10.fork-independent`: messages appended to the fork never reach the source and vice versa; the fork's events socket replays the copied history plus its own events. Same promise: `S04.fork`. - `F10.fork-errors`: an unknown source is 404, an unknown `from_event_id` is 404, a taken `id` is 409, a bad tag key or a title over 200 characters is 422, a malformed id is 422 and a missing key is 401; none of them creates a conversation. - `F10.fork-keeps-credentials`: the fork's `base_state.json` keeps the agent's (encrypted) model keys, like the source's. -- `F10.fork-run-memory`: a fork of a finished DeepSeek conversation runs to `finished` with the source's history in context (it recalls a codename only the source was told). -- `F10.navigate-event`: `POST /navigate` with an earlier `event_id` answers 200 with `leaf_event_id` set to it; GET and `base_state.json` agree and no event is deleted. -- `F10.navigate-branch`: after navigating, the next appended event's `parent_id` is the new HEAD (a sibling of the abandoned branch), pushed live on the events socket; the HEAD move itself is not broadcast. -- `F10.navigate-empty`: `{}` or `{"event_id": null}` selects the empty tree (`leaf_event_id` null, `head_is_empty` true on disk); the next appended event becomes a new root (`parent_id` `__root__`). +- `F10.fork-run-memory`: a fork of a finished DeepSeek conversation runs to `finished` with the source's history in context (it recalls a codename only the source was told). Same promise: `S04.fork`. +- `F10.navigate-event`: `POST /navigate` with an earlier `event_id` answers 200 with `leaf_event_id` set to it; GET and `base_state.json` agree and no event is deleted. Same promise: `S04.fork-branch`. +- `F10.navigate-branch`: after navigating, the next appended event's `parent_id` is the new HEAD (a sibling of the abandoned branch), pushed live on the events socket; the HEAD move itself is not broadcast. Same promise: `S04.fork-branch`, `S05.event-tree`. +- `F10.navigate-empty`: `{}` or `{"event_id": null}` selects the empty tree (`leaf_event_id` null, `head_is_empty` true on disk); the next appended event becomes a new root (`parent_id` `__root__`). Same promise: `S04.fork-branch`. - `F10.navigate-errors`: an unknown `event_id` is 404 `Unknown event_id: ...` and an unknown conversation 404, a missing body 422, a missing key 401; none of them moves HEAD. - `F10.navigate-final-response`: after navigating back before the agent's answer, `agent_final_response` reports the active branch (empty), not the abandoned answer. -- `F10.condense-llm`: on a finished DeepSeek conversation `POST /condense` answers 200 `{"success": true}` after recording one `CondensationRequest` and one `Condensation` with a summary, pushed live on the events socket; the status stays `finished`. -- `F10.condense-no-condenser`: an agent without a request-handling condenser (condenser disabled, or `no_op`) gets an error naming the cause, and no `CondensationRequest` or other event is recorded. +- `F10.condense-llm`: on a finished DeepSeek conversation `POST /condense` answers 200 `{"success": true}` after recording one `CondensationRequest` and one `Condensation` with a summary, pushed live on the events socket; the status stays `finished`. Same promise: `S06.manual-condense`. +- `F10.condense-no-condenser`: an agent without a request-handling condenser (condenser disabled, or `no_op`) gets an error naming the cause, and no `CondensationRequest` or other event is recorded. Same promise: `S06.manual-condense`. - `F10.condense-status`: that refusal is a client error (4xx), not a 500. -- `F10.condense-llm-failure`: when the condenser's model is unreachable, condense answers 500 with an `error_id`, leaves the `CondensationRequest` pending and records no `Condensation`. +- `F10.condense-llm-failure`: when the condenser's model is unreachable, condense answers 500 with an `error_id`, leaves the `CondensationRequest` pending and records no `Condensation`. Same promise: `S06.manual-condense`. - `F10.condense-errors`: an unknown id is 404 `Conversation not found`, a malformed id 422, a missing key 401 (and records nothing). -- `F10.ask-agent`: `POST /ask_agent` answers a question about the conversation with DeepSeek (200 `{"response": ...}`), records no message or action and pushes none on the socket, and leaves the status unchanged; the only event it adds is one `stats` `ConversationStateUpdateEvent` that reports the call's usage under `ask-agent-llm`, in the event log and on the socket. +- `F10.ask-agent`: `POST /ask_agent` answers a question about the conversation with DeepSeek (200 `{"response": ...}`), records no message or action and pushes none on the socket, and leaves the status unchanged; the only event it adds is one `stats` `ConversationStateUpdateEvent` that reports the call's usage under `ask-agent-llm`, in the event log and on the socket. Same promise: `S01.ask-agent`. - `F10.ask-agent-errors`: a missing `question` is 422, a missing key 401, and an unreachable model 500 with an `error_id`. -- `F10.ask-agent-running`: while a run waits on its model, `ask_agent` answers at once (it does not wait for the step), records nothing but its `stats` update (usage under `ask-agent-llm`) and leaves the status `running`. +- `F10.ask-agent-running`: while a run waits on its model, `ask_agent` answers at once (it does not wait for the step), records nothing but its `stats` update (usage under `ask-agent-llm`) and leaves the status `running`. Same promise: `S01.ask-agent`. - `F10.ask-agent-unknown`: an unknown conversation id is 404, as the route documents. -- `F10.ask-agent-usage-saved`: the usage of an ask is part of the conversation's `stats` on `GET /api/conversations/{id}` right after the call (an `ask-agent-llm` entry, as in the `stats` event the ask recorded) and is still there after a restart. +- `F10.ask-agent-usage-saved`: the usage of an ask is part of the conversation's `stats` on `GET /api/conversations/{id}` right after the call (an `ask-agent-llm` entry, as in the `stats` event the ask recorded) and is still there after a restart. Same promise: `S04.ask-agent-usage-saved`. - `F10.restart-persist`: after a restart, forks keep their lineage, title, tags, HEAD and events, a fork id that names a persisted but not yet loaded conversation is still 409, and a navigated empty HEAD stays empty (the next message is still a new root). - `F10.restart-cold-calls`: after a restart, condense and ask_agent work on a conversation whose agent has not been initialized yet, and ask_agent still sees the history (the codename). diff --git a/.agents/skills/verify-agent-server/references/feature-map/F11-goals.md b/.agents/skills/verify-agent-server/references/feature-map/F11-goals.md index 725fe314f8..f4fc576ffa 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F11-goals.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F11-goals.md @@ -31,11 +31,11 @@ Routes: `POST /api/conversations/{conversation_id}/goal`, - `F11.goal-auth`: all three routes return 401 without a session key or with a wrong one, while the right key is accepted. - `F11.goal-stop-noop`: `/goal/stop` with no active loop returns 200 and records nothing. - `F11.goal-resume-none`: `/goal/resume` on a conversation that never had a goal returns 400 `no_resumable_goal` and records nothing. -- `F11.goal-complete`: on DeepSeek flash a small objective runs to a goal status `complete` (`active: false`, a verdict with `complete: true`), the agent's file is in the workspace and the conversation is `finished`. +- `F11.goal-complete`: on DeepSeek flash a small objective runs to a goal status `complete` (`active: false`, a verdict with `complete: true`), the agent's file is in the workspace and the conversation is `finished`. Same promise: `S04.goal-loop`. - `F11.goal-ws`: the events socket pushes the goal status as `ConversationStateUpdateEvent` frames with `key: "goal"`: `running` at the start, then the terminal status. - `F11.goal-in-place`: the loop creates no conversation; the objective is a user message and the agent's tool work is in the same conversation's event log. -- `F11.goal-followup`: an incomplete verdict records a `running` status with `iteration: 1` and that verdict, then posts a follow-up user message carrying the judge's `missing` text and runs another round. -- `F11.goal-capped`: when no verdict says complete, the loop stops after `max_iterations` rounds with `status: capped`, `iteration` equal to the cap and the last verdict, after exactly one agent and one judge call per round. +- `F11.goal-followup`: an incomplete verdict records a `running` status with `iteration: 1` and that verdict, then posts a follow-up user message carrying the judge's `missing` text and runs another round. Same promise: `S04.goal-loop`. +- `F11.goal-capped`: when no verdict says complete, the loop stops after `max_iterations` rounds with `status: capped`, `iteration` equal to the cap and the last verdict, after exactly one agent and one judge call per round. Same promise: `S04.goal-loop`. - `F11.goal-resume-finished`: `/goal/resume` after a `complete` or `capped` goal returns 400 `no_resumable_goal`. - `F11.goal-busy`: while a goal round is in flight, a second `/goal`, `/goal/resume` and `/run` return 409 and add no goal status; an omitted `max_iterations` defaults to 10. - `F11.goal-stop`: `/goal/stop` on an active loop returns 200 at once and records `interrupted` (`active: false`) with the same objective and iteration, but cancels only the loop: the in-flight round keeps the conversation `running` until it is interrupted. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F12-runtime-and-credentials.md b/.agents/skills/verify-agent-server/references/feature-map/F12-runtime-and-credentials.md index b204e8d50d..c27c735c11 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F12-runtime-and-credentials.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F12-runtime-and-credentials.md @@ -50,9 +50,9 @@ Routes: `GET /api/conversations/{conversation_id}/runtime`, - `F12.sandbox-pause-websocket`: an events socket or a session socket that was open before the pause is either closed by the server or keeps receiving the conversation's new events; it is never left open and silent. - `F12.binding-unsupported-secret`: only file credentials can be bound: any other secret name gets 404 without the source being contacted. - `F12.binding-validation`: a body with extra keys, no `url`, an empty or over-long (4097+ characters) `url`, an empty or over-long (8193+ characters) header value or more than 16 headers, or a non-UUID id, gets 422 without the source being contacted, also for an unsupported secret name (the body is validated first); no key gets 401. -- `F12.binding-pending`: a binding for a conversation id the server does not know answers 204 after exactly one probe `GET` that carries the given headers; the id stays unknown (404). -- `F12.binding-source-statuses`: the source's answer maps to the activation status: 401/403 to 403, 400/404/422 to 422, 409 to 409, 501 to 501, a 200 that is not `{"value": str, "version": non-empty str}` to 422, and a URL that is not http(s) to 422, each after a single probe (none for the bad URL). -- `F12.binding-source-unavailable`: a source answering 5xx is probed three times with backoff (at least 0.75 s), then the activation fails with 502; an unreachable source also gives 502. +- `F12.binding-pending`: a binding for a conversation id the server does not know answers 204 after exactly one probe `GET` that carries the given headers; the id stays unknown (404). Same promise: `S19.credential-binding`. +- `F12.binding-source-statuses`: the source's answer maps to the activation status: 401/403 to 403, 400/404/422 to 422, 409 to 409, 501 to 501, a 200 that is not `{"value": str, "version": non-empty str}` to 422, and a URL that is not http(s) to 422, each after a single probe (none for the bad URL). Same promise: `S19.credential-binding`. +- `F12.binding-source-unavailable`: a source answering 5xx is probed three times with backoff (at least 0.75 s), then the activation fails with 502; an unreachable source also gives 502. Same promise: `S19.credential-binding`. - `F12.binding-native-too-late`: binding a loaded regular (non-ACP) conversation is refused with 409 `Credential binding activation arrived after ACP initialization` and records nothing. - `F12.binding-native-cold`: a binding sent while a regular (non-ACP) conversation is not loaded never makes that conversation demand a credential binding after the next pause. - `F12.binding-live-acp`: binding a loaded ACP Codex conversation that has not run yet answers 204, records `required_runtime_credential_bindings: ["CODEX_AUTH_JSON"]` in `meta.json` and removes the stored `CODEX_AUTH_JSON` secret. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F13-bash.md b/.agents/skills/verify-agent-server/references/feature-map/F13-bash.md index ff86ed6fc1..89fba51129 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F13-bash.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F13-bash.md @@ -42,13 +42,13 @@ Routes: `GET /api/conversations/{runtime_conversation_id}/bash/bash_events/searc - `F13.get-events`: `GET /api/bash/bash_events/{event_id}` returns one event by its hex id (404 when unknown), and `GET /api/bash/bash_events/` with a JSON array body returns events in input order with `null` for missing ids (422 without a body). - `F13.search`: search filters by `kind__eq`, `command_id__eq` (outputs only), a UTC `timestamp__gte`/`timestamp__lt` window, sorts with `sort_order`, pages with `limit`/`next_page_id`, and rejects bad parameters with 422. - `F13.timestamp-offset`: a `timestamp__gte` with a non-UTC offset selects the same instant as its UTC spelling. -- `F13.output-chunks`: output over 1 MiB is stored as ordered `BashOutput` chunks of 1048576 characters plus a final chunk with the exit code; `order__gt` skips seen chunks, and the SDK's `RemoteWorkspace.execute_command` reassembles them. +- `F13.output-chunks`: output over 1 MiB is stored as ordered `BashOutput` chunks of 1048576 characters plus a final chunk with the exit code; `order__gt` skips seen chunks, and the SDK's `RemoteWorkspace.execute_command` reassembles them. Same promise: `S18.remote-command-exec`. - `F13.order-gt-paging`: `order__gt` filters before `limit` applies, so a poller asking for outputs after order N with a small page gets them. - `F13.execute-large-output`: `execute_bash_command` returns the complete stdout of a command whose output is over 1 MiB. - `F13.timeout`: a command that outlives its `timeout` is killed with its process group and reports `exit_code` -1. - `F13.stop`: stopping a running command by id terminates only that command (final `exit_code` -15, process gone) while another command finishes normally. - `F13.stop-idempotent`: stopping an already finished command (hex or dashed id) returns 200, an unknown id 404 and a non-UUID id 422. -- `F13.scoped-cwd`: conversation-scoped commands run in the conversation workspace by default, accept an absolute cwd inside it (also through the SDK's `runtime_conversation_id`), and reject a cwd outside it, an escaping `..` path or a relative path with 422. +- `F13.scoped-cwd`: conversation-scoped commands run in the conversation workspace by default, accept an absolute cwd inside it (also through the SDK's `runtime_conversation_id`), and reject a cwd outside it, an escaping `..` path or a relative path with 422. Same promise: `S18.runtime-scope`. - `F13.scoped-routes`: the conversation-scoped execute, start, stop, get, batch get and search routes work on the conversation's own history, and the command's files land in the workspace. - `F13.scoped-isolation`: scoped events are invisible to the global search, the global socket, the global stop route and other conversations, and are stored under the conversation's directory. - `F13.scoped-unknown`: every conversation-scoped bash route answers 404 `Conversation not found` for an unknown conversation id. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F14-files.md b/.agents/skills/verify-agent-server/references/feature-map/F14-files.md index 8981b8f317..1e43f1bd6e 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F14-files.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F14-files.md @@ -32,7 +32,7 @@ Routes: `GET /api/file/home`, `GET /api/file/search_subdirs`, - `F14.upload-download`: a multipart upload creates missing parent directories, a download returns byte-identical bytes as an `application/octet-stream` attachment named after the basename, and a second upload to the same path replaces the content. - `F14.upload-errors`: an upload without the `file` part or without `path` is 422, and a relative `path` is 400 `Path must be absolute`. - `F14.upload-onto-directory`: uploading onto an existing directory, or to a path whose parent is a file, is a client error (400), as `create_directory` answers for the same collisions. -- `F14.upload-size`: the SDK's `RemoteWorkspace.file_upload` and the TypeScript `RemoteWorkspace.fileUpload` report the uploaded `file_size`, as `LocalWorkspace.file_upload` does; the upload response carries it as `{"success": true, "file_size": N}`. +- `F14.upload-size`: the SDK's `RemoteWorkspace.file_upload` and the TypeScript `RemoteWorkspace.fileUpload` report the uploaded `file_size`, as `LocalWorkspace.file_upload` does; the upload response carries it as `{"success": true, "file_size": N}`. Same promise: `S18.remote-files-git`. - `F14.upload-atomic`: an overwrite that fails mid-write leaves the previous file intact; today the target is truncated before streaming and keeps a partial file. - `F14.download-errors`: downloading a missing file is 404 `File not found`, a directory 400 `Path is not a file`, a relative path 400. - `F14.mkdir`: `create_directory` creates missing parents, is idempotent and keeps existing contents, the new directory is listed by `search_subdirs`, and a path that is or runs through a file is 400. @@ -56,7 +56,7 @@ Routes: `GET /api/file/home`, `GET /api/file/search_subdirs`, - `F14.scoped-path-guard`: scoped routes reject paths outside the workspace (another conversation's workspace, `..` traversal, a symlink escape, a sibling with the same prefix, a relative path) with 422 and write nothing, while the global route serves the same path. - `F14.scoped-conversation-errors`: a scoped route for an unknown conversation is 404 and does nothing; a malformed conversation id is 422. - `F14.scoped-trajectory-archive`: the scoped trajectory route serves only its own conversation (another or malformed id is 422), and the scoped archive packs the workspace, including the agent's file, but refuses a path outside it with 422. -- `F14.sdk-workspace`: the SDK's `RemoteWorkspace.file_upload`/`file_download` round-trip bytes through the global routes and, with `runtime_conversation_id`, through the scoped routes; a scoped upload outside the workspace returns `success=False` with the 422 and writes nothing. +- `F14.sdk-workspace`: the SDK's `RemoteWorkspace.file_upload`/`file_download` round-trip bytes through the global routes and, with `runtime_conversation_id`, through the scoped routes; a scoped upload outside the workspace returns `success=False` with the 422 and writes nothing. Same promise: `S18.remote-files-git`. - `F14.restart`: after a restart, uploaded files, the scoped download and the trajectory are still served. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F15-git.md b/.agents/skills/verify-agent-server/references/feature-map/F15-git.md index 22f0b33634..0e9944fb31 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F15-git.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F15-git.md @@ -53,7 +53,7 @@ Routes: `GET /api/conversations/{runtime_conversation_id}/git/changes`, - `F15.scoped`: the four conversation-scoped routes on a conversation whose workspace is the fixture repository return exactly what the global routes return. - `F15.scoped-guard`: scoped routes refuse a `path` outside the workspace (another directory, a same-prefix sibling, `..`, a symlink escape, a relative path, a second `path` parameter pointing outside) with 422 while the global route serves it; an unknown conversation is 404, a malformed id 422, and there is no scoped repository search. - `F15.scoped-restart`: after a restart the scoped routes still resolve the conversation's workspace and return the same changes. -- `F15.sdk-workspace`: the Python SDK's `RemoteWorkspace.git_changes` and `git_diff` join relative paths to `working_dir` and return the same results plain and scoped (`runtime_conversation_id`), and the scoped workspace refuses a path outside it with 422. +- `F15.sdk-workspace`: the Python SDK's `RemoteWorkspace.git_changes` and `git_diff` join relative paths to `working_dir` and return the same results plain and scoped (`runtime_conversation_id`), and the scoped workspace refuses a path outside it with 422. Same promise: `S18.remote-files-git`. - `F15.ts-client`: the TypeScript client's `RemoteWorkspace.gitChanges` (with `ref`) and `gitDiff` return the same results plain and with `conversationId`, the scoped one rejects a path outside the workspace, and `GitClient.searchRepositories` returns the `missing_token` page. - `F15.agent-changes`: after a real model run writes a file in a repository workspace, the scoped `changes` lists it as `ADDED` and the scoped `diff` shows the written text, and the event stream has the file editor's observation. - `F15.repo-search-token`: without a stored token, GitHub search is 200 `{"items": [], "next_page_id": null, "missing_token": true}`; a secret named `github_token` or `GH_TOKEN` is picked up (a malformed `page_id` is then validated and refused with 400), and deleting it restores `missing_token`. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md b/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md index 64e8a70078..be65e49e27 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md @@ -28,25 +28,25 @@ Routes: `GET /api/settings/agent-schema`, `GET /api/settings/conversation-schema ## Sub-features - `F18.get-defaults`: on a fresh persistence directory `GET /api/settings` returns OpenHands defaults (no API key, `llm_api_key_is_set: false`, every pointer null, empty `misc_settings`) without creating `settings.json`. -- `F18.agent-schema`: `GET /api/settings/agent-schema` returns the `AgentSettings` form schema with shared, `openhands` and `acp` sections, secret fields flagged, and every OpenHands field key present in `agent_settings`. -- `F18.conversation-schema`: `GET /api/settings/conversation-schema` returns the `ConversationSettings` schema whose ten field keys and defaults match `conversation_settings` on a fresh server. +- `F18.agent-schema`: `GET /api/settings/agent-schema` returns the `AgentSettings` form schema with shared, `openhands` and `acp` sections, secret fields flagged, and every OpenHands field key present in `agent_settings`. Same promise: `S19.schema-export`. +- `F18.conversation-schema`: `GET /api/settings/conversation-schema` returns the `ConversationSettings` schema whose ten field keys and defaults match `conversation_settings` on a fresh server. Same promise: `S19.schema-export`. - `F18.auth-required`: with a session key configured, every settings and secrets route answers 401 to a missing or wrong `X-Session-API-Key`, whatever `X-Expose-Secrets` says. - `F18.patch-llm`: a PATCH of `agent_settings_diff.llm` stores the model and key, flips `llm_api_key_is_set`, creates `settings.json`, and answers redacted even when `X-Expose-Secrets: plaintext` is sent. - `F18.expose-modes`: GET redacts secrets as `**********` by default, returns the raw value with `plaintext`, a Fernet token with `encrypted` (and the legacy `true`), and 400 for any other value. - `F18.roundtrip`: posting back the encrypted token keeps the real key; posting back the redacted placeholder clears it. -- `F18.agent-diff-merge`: an `agent_settings_diff` merges sparsely (siblings such as the key survive) and a `null` unsets a field (RFC 7386). -- `F18.conversation-diff`: a `conversation_settings_diff` updates only the named fields; a top-level `null` or an out-of-range value is 422 and changes nothing. +- `F18.agent-diff-merge`: an `agent_settings_diff` merges sparsely (siblings such as the key survive) and a `null` unsets a field (RFC 7386). Same promise: `S19.apply-diff`. +- `F18.conversation-diff`: a `conversation_settings_diff` updates only the named fields; a top-level `null` or an out-of-range value is 422 and changes nothing. Same promise: `S19.conversation-settings`. - `F18.misc-settings-merge`: `misc_settings_diff` deep-merges nested objects, replaces lists, removes a nested key on `null`, and stores a top-level `null` as `null`. - `F18.patch-validation`: an empty or unknown-keys-only PATCH is 400, an invalid value is 422 `Settings validation failed` without echoing the posted secret, malformed pointers are 422, and nothing is stored. - `F18.active-profile-pointer`: PATCH `active_profile` applies that LLM profile to `agent_settings.llm`, an unknown profile is 404, an explicit `llm` diff wins over the profile, and `null` clears only the pointer. -- `F18.agent-kind-switch`: switching `agent_kind` to `acp` rebuilds from a fresh ACP base (no `llm`, no key); switching back gives fresh OpenHands defaults. The server-wide `mcp_config`, `agent_context.load_memory` and `verification.critic_api_key` survive both switches. +- `F18.agent-kind-switch`: switching `agent_kind` to `acp` rebuilds from a fresh ACP base (no `llm`, no key); switching back gives fresh OpenHands defaults. The server-wide `mcp_config`, `agent_context.load_memory` and `verification.critic_api_key` survive both switches. Same promise: `S19.apply-diff`. - `F18.secret-put-list-get`: `PUT /api/settings/secrets` answers `{name, description}`, the list shows names and descriptions but never values, GET by name returns the raw value as `text/plain`, and a second PUT overwrites value and description. - `F18.secret-delete`: DELETE answers `{"deleted": true}`; afterwards GET and a second DELETE are 404 `Secret not found` and the list omits the name. - `F18.secret-name-validation`: names that do not match `^[a-zA-Z][a-zA-Z0-9_]{0,63}$` are 422 on GET, PUT and DELETE (64 characters are accepted, 65 are not); a PUT without `value` is 422; an unknown valid name is 404. -- `F18.encrypted-at-rest`: `settings.json` and `secrets.json` hold Fernet tokens, never the raw values, with mode `0600`. +- `F18.encrypted-at-rest`: `settings.json` and `secrets.json` hold Fernet tokens, never the raw values, with mode `0600`. Same promise: `S19.mcp-config-secrets`. - `F18.secret-scope-agent-profile`: `?agent_profile_id=` limits the list to that agent profile's `secret_refs` (`[]` lists nothing, no `secret_refs` lists everything); an unknown id is 404 `Agent profile not found`. -- `F18.sdk-remote-workspace`: the SDK's `RemoteWorkspace.get_secrets()` returns `LookupSecret`s whose URLs resolve to the stored values, and `get_llm()` returns the stored model and key. -- `F18.sdk-clone-repos`: with the server secret `GITHUB_TOKEN` stored, the SDK's `RemoteWorkspace.clone_repos()` reads it from `GET /api/settings/secrets/GITHUB_TOKEN` and clones with it (a local git remote reachable only through the token-bearing URL), a provider whose token secret is missing (`GITLAB_TOKEN`) clones anonymously, and `get_repos_context()` names both clones. +- `F18.sdk-remote-workspace`: the SDK's `RemoteWorkspace.get_secrets()` returns `LookupSecret`s whose URLs resolve to the stored values, and `get_llm()` returns the stored model and key. Same promise: `S18.remote-settings`. +- `F18.sdk-clone-repos`: with the server secret `GITHUB_TOKEN` stored, the SDK's `RemoteWorkspace.clone_repos()` reads it from `GET /api/settings/secrets/GITHUB_TOKEN` and clones with it (a local git remote reachable only through the token-bearing URL), a provider whose token secret is missing (`GITLAB_TOKEN`) clones anonymously, and `get_repos_context()` names both clones. Same promise: `S18.repos-and-skills`. - `F18.ts-client`: the TypeScript `SettingsClient` reads both schemas, reads settings redacted, `plaintext` and `encrypted`, applies a sparse `updateSettings` that a second read shows, upserts, lists (also with `agentProfileId`), reads as text and deletes a secret, and rejects an unknown secret (404), a bad name (422) and a wrong key (401) with an `HttpError`. - `F18.persist-restart`: settings (plaintext key, model, conversation settings) and secrets read back unchanged after a restart with the same cipher key. - `F18.experimental-features`: `GET /api/settings/experimental` lists the experimental features the user can switch (`conversation_reactions`, `ask_user`, `conversation_mentions` and `message_edits`, all off) without writing `settings.json`, and `PATCH /api/settings/experimental/{key}` answers 404 for a key the server does not know and stores nothing (`experimental_features` stays `{}`). diff --git a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md index c78782be5b..6c52cfd5bb 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md @@ -34,20 +34,20 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, - `F19.auth`: every F19 route answers 401 without or with a wrong session key and stores or spawns nothing. - `F19.settings-create`: `POST /api/settings/mcp/{settings_key}` stores one server (201, the whole redacted `SettingsResponse`), `GET /api/settings` shows it redacted, `X-Expose-Secrets: plaintext` returns the credentials, the file on disk holds them encrypted, and creating an existing key is 409 without changing it. -- `F19.settings-validation`: creating a stdio server without `command`, a remote server without `url`, an unknown field or transport, or `auth` next to an `Authorization` header is 422 and stores nothing. +- `F19.settings-validation`: creating a stdio server without `command`, a remote server without `url`, an unknown field or transport, or `auth` next to an `Authorization` header is 422 and stores nothing. Same promise: `S09.config-model`. - `F19.settings-empty-server`: a server body with neither `command` nor `url` (`{}`) is rejected with 422 instead of being stored. - `F19.settings-patch`: `PATCH` merges sparsely (`enabled: false` keeps url and headers, a `null` header entry deletes only that entry, `enabled: null` restores `true`); an unknown key is 404, an unknown field or a merge that leaves an http server without `url` is 422. - `F19.settings-delete`: `DELETE` removes one server and leaves the others and their credentials intact; deleting a missing key is 404. -- `F19.test-stdio`: `POST /api/mcp/test` spawns a stdio server, lists its tools, runs a requested `tool_call` and returns its text, and no stdio child survives the response. -- `F19.test-tool-errors`: a tool that raises, an unadvertised tool name and a tool that outlives the timeout are reported in `tool_result` with `is_error: true` while `ok` stays `true`. -- `F19.test-connect-failures`: a missing executable, a closed port and a command that exits at once answer 200 with `ok: false`, an `error` and an `error_kind`. -- `F19.test-timeout`: a stdio server that never completes the handshake yields `ok: false`, `error_kind: timeout` close to the requested `timeout`, and its process is killed. -- `F19.test-validation`: a missing server, an empty command, an unknown transport, a `timeout` outside (0, 120], top-level `authentication` and `auth` next to an `Authorization` header are 422. -- `F19.test-encrypted-env`: `env` values encrypted by the settings API are decrypted before the stdio server starts, and the child does not inherit the agent server's own keys. -- `F19.test-remote-bearer`: a streamable-HTTP server behind a bearer token connects with `auth.strategy: bearer`, an `Authorization` header (also as `type: shttp`) or `auth.strategy: header`, and a wrong token answers `ok: false` naming the 401 (known bug #6687: the status is lost). +- `F19.test-stdio`: `POST /api/mcp/test` spawns a stdio server, lists its tools, runs a requested `tool_call` and returns its text, and no stdio child survives the response. Same promise: `S09.stdio-server`. +- `F19.test-tool-errors`: a tool that raises, an unadvertised tool name and a tool that outlives the timeout are reported in `tool_result` with `is_error: true` while `ok` stays `true`. Same promise: `S09.observation`. +- `F19.test-connect-failures`: a missing executable, a closed port and a command that exits at once answer 200 with `ok: false`, an `error` and an `error_kind`. Same promise: `S09.listing-timeout`. +- `F19.test-timeout`: a stdio server that never completes the handshake yields `ok: false`, `error_kind: timeout` close to the requested `timeout`, and its process is killed. Same promise: `S09.listing-timeout`. +- `F19.test-validation`: a missing server, an empty command, an unknown transport, a `timeout` outside (0, 120], top-level `authentication` and `auth` next to an `Authorization` header are 422. Same promise: `S09.config-model`. +- `F19.test-encrypted-env`: `env` values encrypted by the settings API are decrypted before the stdio server starts, and the child does not inherit the agent server's own keys. Same promise: `S09.config-secrets`. +- `F19.test-remote-bearer`: a streamable-HTTP server behind a bearer token connects with `auth.strategy: bearer`, an `Authorization` header (also as `type: shttp`) or `auth.strategy: header`, and a wrong token answers `ok: false` naming the 401 (known bug #6687: the status is lost). Same promise: `S09.http-sse-server`, `S09.static-auth`. - `F19.oauth-start-errors`: OAuth start is 400 for a stdio or non-oauth2 server, reports `ok: false` with a failed job for an unreachable server (whose callback is then 409), and finishes without consent for a server that needs no OAuth. - `F19.oauth-unknown-job`: status and callback for an unknown job id are 404, and a callback without `callback_url` is 422. -- `F19.oauth-flow`: against a local OAuth-protected MCP server, start returns an `authorization_url`, status reports `authorizing` and `callback_ready`, and submitting the browser's redirect URL completes the job with tools, the tool result and an encrypted `oauth_state`. +- `F19.oauth-flow`: against a local OAuth-protected MCP server, start returns an `authorization_url`, status reports `authorizing` and `callback_ready`, and submitting the browser's redirect URL completes the job with tools, the tool result and an encrypted `oauth_state`. Same promise: `S09.oauth`. - `F19.oauth-callback-replay`: submitting a callback again after the job finished answers 200 with the job's status, not 500, and the job stays `succeeded`. - `F19.oauth-callback-bad-port`: a callback URL with a port outside 0-65535 is rejected with 400, not 500. - `F19.oauth-callback-validation`: on a live job, an https URL, a foreign host, another port or another path is 400 and the job keeps waiting; the redirect URL with `127.0.0.1` instead of `localhost` completes it. @@ -59,19 +59,19 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, - `F19.oauth-callback-base-url-validation`: a `callback_base_url` with a query, a fragment, userinfo, no host or a non-http(s) scheme is 422; a plain https base with a path prefix is accepted. - `F19.oauth-web-url-precedence`: on a server with `OH_WEB_URL`, a `callback_base_url` in the start body is ignored for the `web_url` redirect URI, and the redirect delivered to that server's route completes the job. - `F19.oauth-callback-no-redirect`: the callback route forwards the submitted query only to the job's own callback address and does not follow a redirect it gets from there (known bug #6808: the redirect is followed). -- `F19.oauth-inline-writeback`: an OAuth server passed inline on a conversation's agent (not in settings) whose access token has expired is refreshed when the conversation connects, keeps its tools, and the rotated tokens are posted with the run's session key to the webhook's `/mcp-oauth-state`, not stored in settings. +- `F19.oauth-inline-writeback`: an OAuth server passed inline on a conversation's agent (not in settings) whose access token has expired is refreshed when the conversation connects, keeps its tools, and the rotated tokens are posted with the run's session key to the webhook's `/mcp-oauth-state`, not stored in settings. Same promise: `S09.oauth`. - `F19.oauth-stored-state`: the returned `oauth_state` used as `auth.state` connects without consent (test, and start without `authorization_url`), and stored through the settings route it is redacted, encrypted on disk and decrypted to the real token. -- `F19.disabled-server`: a conversation started from the settings exposes the tools of enabled servers (including the OAuth server from its stored tokens) and neither exposes nor spawns a disabled server. -- `F19.conversation-tool`: a model-driven conversation calls the configured stdio server's `qa_echo` tool, the `ObservationEvent` (REST and WebSocket) carries `QA_MCP_ECHO:hi`, and deleting the conversation stops the stdio child; an embedded resource's text and a structured-only result reach the observation (known bug #6993: both are dropped), and a single server that advertises `terminal` next to the built-in terminal gets it as `<server>_terminal` while both run (known bug #6992: a server that joins after the startup budget loses all its tools, so its other tools are `not found`; one ready within the budget makes the first message 500 `Duplicate tool names found: {'terminal'}`). -- `F19.oauth-refresh-writeback`: when a stored OAuth server's access token has expired, the next conversation refreshes it, keeps the server's tools, and writes the rotated token and new expiry back into the settings (encrypted on disk) without posting them to the webhook. -- `F19.conversation-secrets-at-rest`: a conversation started from the settings persists its agent's MCP credentials (a disabled server's `env`, OAuth tokens) encrypted, and no MCP credential this family used appears in plaintext under the conversations directory or in the server log. +- `F19.disabled-server`: a conversation started from the settings exposes the tools of enabled servers (including the OAuth server from its stored tokens) and neither exposes nor spawns a disabled server. Same promise: `S09.enabled-flag`. +- `F19.conversation-tool`: a model-driven conversation calls the configured stdio server's `qa_echo` tool, the `ObservationEvent` (REST and WebSocket) carries `QA_MCP_ECHO:hi`, and deleting the conversation stops the stdio child; an embedded resource's text and a structured-only result reach the observation (known bug #6993: both are dropped), and a single server that advertises `terminal` next to the built-in terminal gets it as `<server>_terminal` while both run (known bug #6992: a server that joins after the startup budget loses all its tools, so its other tools are `not found`; one ready within the budget makes the first message 500 `Duplicate tool names found: {'terminal'}`). Same promise: `S09.observation`. +- `F19.oauth-refresh-writeback`: when a stored OAuth server's access token has expired, the next conversation refreshes it, keeps the server's tools, and writes the rotated token and new expiry back into the settings (encrypted on disk) without posting them to the webhook. Same promise: `S09.oauth`. +- `F19.conversation-secrets-at-rest`: a conversation started from the settings persists its agent's MCP credentials (a disabled server's `env`, OAuth tokens) encrypted, and no MCP credential this family used appears in plaintext under the conversations directory or in the server log. Same promise: `S09.config-secrets`, `S19.mcp-config-secrets`. - `F19.restart`: stored MCP servers and their credentials survive a restart (plaintext GET and the SDK's `RemoteWorkspace.get_mcp_config()`), while in-memory OAuth jobs are gone (404). -- `F19.conversation-mcp-add`: `POST /api/conversations/{conversation_id}/mcp` on a conversation whose agent is initialized starts the server before it answers 200 `{"success": true}`, `GET /api/conversations/{conversation_id}` and `base_state.json` show it in `agent.mcp_config`, the model's next request offers its tools as `<server>_<tool>`, a call returns the tool's result, deleting the conversation stops the child, and `/server_info` advertises `conversation_mcp_install_v1` and `conversation_configuration_additions_v1`. -- `F19.conversation-mcp-add-before-init`: on a conversation without a first message the route only records the server (no process, fast answer), with `env` values encrypted by the settings API decrypted for the child and stored encrypted in `base_state.json`; the first message connects it like any configured server. +- `F19.conversation-mcp-add`: `POST /api/conversations/{conversation_id}/mcp` on a conversation whose agent is initialized starts the server before it answers 200 `{"success": true}`, `GET /api/conversations/{conversation_id}` and `base_state.json` show it in `agent.mcp_config`, the model's next request offers its tools as `<server>_<tool>`, a call returns the tool's result, deleting the conversation stops the child, and `/server_info` advertises `conversation_mcp_install_v1` and `conversation_configuration_additions_v1`. Same promise: `S09.runtime-add-servers`. +- `F19.conversation-mcp-add-before-init`: on a conversation without a first message the route only records the server (no process, fast answer), with `env` values encrypted by the settings API decrypted for the child and stored encrypted in `base_state.json`; the first message connects it like any configured server. Same promise: `S09.runtime-add-servers`. - `F19.conversation-mcp-add-errors`: an unknown conversation is 404, a server name the agent already has and an SSE server are 400 with the reason, a server without `command`/`url`, an unknown field and a non-UUID id are 422, none of them changes the agent's `mcp_config`, and a disabled server is stored without being spawned. - `F19.conversation-mcp-add-start-failure`: a server that cannot start (a command that exits at once) is refused with the route's 400 naming the server and the cause (`Could not start MCP server <name>: Connection closed`, since [#6928](https://github.com/OpenHands/software-agent-sdk/pull/6928)) and is not added. -- `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (known bug [#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927), closed 2026-11-18 with no fix merged; fix proposed in [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929): with one server the tools come back unprefixed, so `qa-f19-rs_qa_echo` is `Tool ... not found`). -- `F19.conversation-mcp-slow-start`: a conversation whose stdio server delays its handshake by 8 s is created within a few seconds, the server is spawned, and the agent's first model request waits for it and offers its tools; the agent's call returns its result. +- `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (known bug [#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927), closed 2026-11-18 with no fix merged; fix proposed in [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929): with one server the tools come back unprefixed, so `qa-f19-rs_qa_echo` is `Tool ... not found`). Same promise: `S09.runtime-add-resume`. +- `F19.conversation-mcp-slow-start`: a conversation whose stdio server delays its handshake by 8 s is created within a few seconds, the server is spawned, and the agent's first model request waits for it and offers its tools; the agent's call returns its result. Same promise: `S09.startup-budget`. - `F19.conversation-mcp-late-failure`: a server that fails after the startup budget costs only its own tools: the conversation starts at once, the first step waits for the failure, the other server's tools are offered, the run finishes and the log names the failed server. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F20-llm-profiles.md b/.agents/skills/verify-agent-server/references/feature-map/F20-llm-profiles.md index 8863fcee72..c4b64db617 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F20-llm-profiles.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F20-llm-profiles.md @@ -25,39 +25,39 @@ Routes: `GET /api/profiles`, `GET /api/profiles/{name}`, ## Sub-features - `F20.auth-required`: every profile route answers 401 without a valid `X-Session-API-Key`, and nothing changes. -- `F20.save`: `POST /api/profiles/{name}` with a DeepSeek model and key answers 201 `Profile '<name>' saved`; the list shows the summary `{name, model, base_url, provider_connection_id, provider_connection_broken, api_key_set: true}` and the active profile, and `GET` by name returns the config. -- `F20.key-encrypted-at-rest`: the profile file `home/.openhands/profiles/<name>.json` is mode `600` with `schema_version` 1 and a Fernet `gAAAAA` token as `api_key`; the plaintext key appears nowhere under the server's home. -- `F20.get-secrets`: `GET /api/profiles/{name}` returns `config.api_key: null` with `api_key_set: true` by default, the raw key with `X-Expose-Secrets: plaintext`, a `gAAAAA` token with `encrypted` (or the legacy `true`), and 400 for any other header value. +- `F20.save`: `POST /api/profiles/{name}` with a DeepSeek model and key answers 201 `Profile '<name>' saved`; the list shows the summary `{name, model, base_url, provider_connection_id, provider_connection_broken, api_key_set: true}` and the active profile, and `GET` by name returns the config. Same promise: `S03.profile-save-load`. +- `F20.key-encrypted-at-rest`: the profile file `home/.openhands/profiles/<name>.json` is mode `600` with `schema_version` 1 and a Fernet `gAAAAA` token as `api_key`; the plaintext key appears nowhere under the server's home. Same promise: `S03.profile-secrets`. +- `F20.get-secrets`: `GET /api/profiles/{name}` returns `config.api_key: null` with `api_key_set: true` by default, the raw key with `X-Expose-Secrets: plaintext`, a `gAAAAA` token with `encrypted` (or the legacy `true`), and 400 for any other header value. Same promise: `S03.profile-secrets`. - `F20.save-encrypted-roundtrip`: posting back the `gAAAAA` token from an `encrypted` read stores the original key (the server decrypts it before re-encrypting), not the token. -- `F20.save-without-secrets`: `include_secrets: false` stores the placeholder `**********` instead of the key; both the list and `GET` report `api_key_set: false` and plaintext reads return `null`. +- `F20.save-without-secrets`: `include_secrets: false` stores the placeholder `**********` instead of the key; both the list and `GET` report `api_key_set: false` and plaintext reads return `null`. Same promise: `S03.profile-secrets`. - `F20.not-found`: `GET`, `activate` and `rename` of an unknown name answer 404 `Profile '<name>' not found`, also for a same-name rename. -- `F20.name-rules`: names must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`: a leading `-` is 422 on all six named routes, 64 characters are accepted and 65 are 422; a save or validate body without `llm`, or a save without `llm.model`, is 422 and saves nothing. -- `F20.corrupt-files`: a profile file that is not valid JSON (or not an object) is left out of the list and answers 400 `Failed to load profile ...` on `GET` and `activate`, saving over it repairs it, and an unreadable `settings.json` makes activation 409 without touching the file. -- `F20.store-busy`: while another process holds the profile store's lock, a save waits out the 30-second lock timeout and answers 503 `Profile store is busy. Please retry.` without saving; once the lock is free the same save is 201. +- `F20.name-rules`: names must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`: a leading `-` is 422 on all six named routes, 64 characters are accepted and 65 are 422; a save or validate body without `llm`, or a save without `llm.model`, is 422 and saves nothing. Same promise: `S03.profile-save-load`. +- `F20.corrupt-files`: a profile file that is not valid JSON (or not an object) is left out of the list and answers 400 `Failed to load profile ...` on `GET` and `activate`, saving over it repairs it, and an unreadable `settings.json` makes activation 409 without touching the file. Same promise: `S03.profile-save-load`. +- `F20.store-busy`: while another process holds the profile store's lock, a save waits out the 30-second lock timeout and answers 503 `Profile store is busy. Please retry.` without saving; once the lock is free the same save is 201. Same promise: `S03.profile-concurrency-limits`. - `F20.store-busy-responsive`: while a profile request waits for the store lock, the server keeps answering other requests (`GET /alive` within 3 seconds). -- `F20.profile-limit`: with 50 profiles stored, creating another is 409 `Profile limit reached (50)` while overwriting an existing one still answers 201. +- `F20.profile-limit`: with 50 profiles stored, creating another is 409 `Profile limit reached (50)` while overwriting an existing one still answers 201. Same promise: `S03.profile-concurrency-limits`. - `F20.validate-live`: `validate` with a working DeepSeek flash config answers `{valid: true, error: null}`, also when the key is the `gAAAAA` token from an `encrypted` read; the draft is not saved. - `F20.validate-rejects`: `validate` answers 200 `valid: false` with a typed error for a model the provider does not serve (`LLMBadRequestError`) and for an invalid DeepSeek key, without echoing the key. -- `F20.validate-bad-key-type`: an invalid DeepSeek key (provider HTTP 401) is reported as `LLMAuthenticationError`, like any other provider's 401. -- `F20.validate-provider-errors`: with retries off, a provider 401 is `valid: false` `LLMAuthenticationError`, a 500 or an unreachable `base_url` is `valid: false` `LLMServiceUnavailableError`, and a transient 429 or a per-attempt timeout is `valid: true`; the timeout verdict arrives after one attempt of the posted `timeout`, long before the hung provider would answer; the provider receives a system message `Reply with one token.`, a user `ping` and `max_tokens: 1`. -- `F20.validate-quota-exhausted`: a provider 429 whose JSON `error.type` and `error.code` are `insufficient_quota` fails validation with `LLMRateLimitError`, even when the human-readable message does not repeat the machine code; an ordinary 429 remains transient. +- `F20.validate-bad-key-type`: an invalid DeepSeek key (provider HTTP 401) is reported as `LLMAuthenticationError`, like any other provider's 401. Same promise: `S02.error-mapping`. +- `F20.validate-provider-errors`: with retries off, a provider 401 is `valid: false` `LLMAuthenticationError`, a 500 or an unreachable `base_url` is `valid: false` `LLMServiceUnavailableError`, and a transient 429 or a per-attempt timeout is `valid: true`; the timeout verdict arrives after one attempt of the posted `timeout`, long before the hung provider would answer; the provider receives a system message `Reply with one token.`, a user `ping` and `max_tokens: 1`. Same promise: `S02.error-mapping`. +- `F20.validate-quota-exhausted`: a provider 429 whose JSON `error.type` and `error.code` are `insufficient_quota` fails validation with `LLMRateLimitError`, even when the human-readable message does not repeat the machine code; an ordinary 429 remains transient. Same promise: `S02.error-mapping`. - `F20.validate-subscription-login`: `validate` of an `auth_type: subscription` config on a server with no stored OpenAI subscription login answers 200 `valid: false` (`ValueError`, `OpenAI subscription login is required`) at once, not a 500. - `F20.validate-unreachable-bounded`: `validate` of a config whose `base_url` refuses connections answers `valid: false` within 30 seconds with the default retry policy. - `F20.validate-hang-bounded`: `validate` of a config whose provider never answers returns a verdict within 30 seconds with the default timeout and retry policy. - `F20.activate`: `POST /api/profiles/{name}/activate` answers 200 `llm_applied: true`; `GET /api/settings` then has the profile's LLM (model, key, options) in `agent_settings.llm` and `active_profile` set, the list reports the same `active_profile`, and `settings.json` holds the key encrypted. - `F20.activate-conversation`: a conversation started from the saved settings after an activation runs on the activated profile's model and options and finishes. - `F20.activate-acp`: with ACP agent settings, activation only moves the pointer: 200 `llm_applied: false`, `active_profile` set, no `agent_settings.llm`. -- `F20.sdk-get-llm`: the SDK's `RemoteWorkspace.get_llm(profile_name=...)` returns the stored model, options and plaintext key with `usage_id` `profile:<name>`, `get_llm()` follows `active_profile`, keyword overrides win, and an unknown name raises `FileNotFoundError`. +- `F20.sdk-get-llm`: the SDK's `RemoteWorkspace.get_llm(profile_name=...)` returns the stored model, options and plaintext key with `usage_id` `profile:<name>`, `get_llm()` follows `active_profile`, keyword overrides win, and an unknown name raises `FileNotFoundError`. Same promise: `S18.remote-settings`. - `F20.ts-client`: the TypeScript `ProfilesClient` lists, saves, reads (hidden, `plaintext`, `encrypted`), validates, renames, activates and deletes a profile, and surfaces 404 and 422 as `HttpError`s. - `F20.activate-dangling-provider`: a profile linked to a provider connection that does not exist is saved without its inline key and `base_url`, is listed as `provider_connection_broken: true`, reads fine, and its activation is 422 without moving the active pointer. -- `F20.activate-linked-provider`: a profile linked to an existing provider connection is saved without its inline key, is listed and read with `api_key_set: true` (the connection's key) and a `null` key, and its activation writes the connection's key into `agent_settings.llm`. -- `F20.sdk-get-llm-linked`: an SDK conversation whose LLM comes from `RemoteWorkspace.get_llm(profile_name=<linked profile>)` runs on the provider connection's key and finishes. +- `F20.activate-linked-provider`: a profile linked to an existing provider connection is saved without its inline key, is listed and read with `api_key_set: true` (the connection's key) and a `null` key, and its activation writes the connection's key into `agent_settings.llm`. Same promise: `S03.provider-connections`. +- `F20.sdk-get-llm-linked`: an SDK conversation whose LLM comes from `RemoteWorkspace.get_llm(profile_name=<linked profile>)` runs on the provider connection's key and finishes. Same promise: `S03.provider-connections`. - `F20.no-cipher`: on a server with neither `OH_SECRET_KEY` nor a session key (no cipher), a saved key is stored in plaintext in a `600` file, plaintext reads and activation work, and `X-Expose-Secrets: encrypted` answers 503. - `F20.persist-restart`: profiles, their encrypted keys, options and the active pointer survive a restart with the same secret key. -- `F20.rename`: `POST /api/profiles/{name}/rename` moves the file and keeps the key; the old name is 404, `active_profile` follows the rename in the list and in settings, and agent profiles whose `llm_profile_ref` cited the old name now cite the new one. +- `F20.rename`: `POST /api/profiles/{name}/rename` moves the file and keeps the key; the old name is 404, `active_profile` follows the rename in the list and in settings, and agent profiles whose `llm_profile_ref` cited the old name now cite the new one. Same promise: `S03.profile-save-load`, `S19.profile-refs`. - `F20.rename-errors`: renaming onto an existing name is 409 and changes nothing, a same-name rename is 200 `unchanged (same name)`, and an invalid or missing `new_name` is 422. -- `F20.delete-guarded`: deleting a profile that an agent profile references is 409 naming the referrer, and the profile stays. -- `F20.delete`: `DELETE /api/profiles/{name}` removes the file and the list entry and clears `active_profile` in both views, leaves the applied `agent_settings.llm` in place, and answers 200 again for a name that is already gone. +- `F20.delete-guarded`: deleting a profile that an agent profile references is 409 naming the referrer, and the profile stays. Same promise: `S19.profile-refs`. +- `F20.delete`: `DELETE /api/profiles/{name}` removes the file and the list entry and clears `active_profile` in both views, leaves the applied `agent_settings.llm` in place, and answers 200 again for a name that is already gone. Same promise: `S03.profile-save-load`. - `F20.json-alias-guard`: a name with a `.json` suffix addresses the same profile (`GET /api/profiles/<name>.json` reads `<name>`), and deleting through that alias is refused with 409 while an agent profile references `<name>`, as it is for the plain name. - `F20.secret-key-previous-keys`: after a rotation that keeps the old key in `OH_PREVIOUS_SECRET_KEYS`, a profile saved under the old key still reads its key, and once re-saved it reads without the old key. - `F20.secret-key-rotation`: after the server's secret key changes and the old key is lost (not in `OH_PREVIOUS_SECRET_KEYS`), activating or re-saving a profile whose stored key can no longer be decrypted is refused (409) instead of silently applying a keyless LLM or overwriting the key. @@ -124,7 +124,7 @@ Preconditions: - Bodies that carry the real key are piped from the shell builtin `printf` into `api ... --stdin`, so the key never appears on a command line; the CLI redacts it from output and evidence. -- `F20.store-busy` and `F20.store-busy-responsive` hold the store lock for +- `F20.store-busy` and `F20.store-busy-responsive` hold the store lock for Same promise: `S03.profile-concurrency-limits`. about 34 seconds each; `F20.validate-hang-bounded` waits about 15 seconds for the server's verdict and hard-restarts the run so no server-side request outlives it; `F20.no-cipher` launches and stops a second server; diff --git a/.agents/skills/verify-agent-server/references/feature-map/F21-agent-profiles.md b/.agents/skills/verify-agent-server/references/feature-map/F21-agent-profiles.md index 461dae47f1..7642d67d8c 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F21-agent-profiles.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F21-agent-profiles.md @@ -29,28 +29,28 @@ Routes: `GET /api/agent-profiles`, `GET /api/agent-profiles/{name}`, ## Sub-features - `F21.auth-required`: every agent-profile route answers 401 without a valid `X-Session-API-Key`, and nothing is written (not even the lazy seed). -- `F21.lazy-seed`: the first list on an empty store seeds one `default` OpenHands profile (revision 0, `llm_profile_ref` = the active LLM profile), points `active_agent_profile_id` at it, and later lists reuse it; a persisted missing or dangling pointer is repaired to the same default. +- `F21.lazy-seed`: the first list on an empty store seeds one `default` OpenHands profile (revision 0, `llm_profile_ref` = the active LLM profile), points `active_agent_profile_id` at it, and later lists reuse it; a persisted missing or dangling pointer is repaired to the same default. Same promise: `S19.profile-seed`. - `F21.seed-no-ghost-llm`: on a never-configured server the seeded `default` keeps a soft `llm_profile_ref` `default` without writing a keyless `default` LLM profile (materialize reports it unresolved); once the live LLM has a real key, the next seed mirrors it into a keyed `default` LLM profile. -- `F21.seed-acp`: when the live `agent_settings` are ACP, the first list seeds an ACP `default` profile with the live backend and model, no LLM reference, and no LLM profile written. -- `F21.save-create`: `POST /api/agent-profiles/{name}` answers 201 and stores the profile under the path name with a server-minted id (a client `id` and body `name` are ignored) at revision 0; GET, the list summary and the file agree. -- `F21.save-overwrite`: saving an existing name replaces the whole profile (no merge), keeps its `id` and sets `revision` to the previous one plus one, ignoring a client `revision`. -- `F21.acp-kind`: an ACP profile (`agent_kind` `acp`, `acp_server`) saves with its ACP defaults and lists with `llm_profile_ref` null. -- `F21.save-validation`: a missing `llm_profile_ref`, a field of the other kind or an unknown field, an invalid `acp_server` or `agent_kind`, a too-new `schema_version` and a non-object body are 422 with a reason, and nothing is stored. +- `F21.seed-acp`: when the live `agent_settings` are ACP, the first list seeds an ACP `default` profile with the live backend and model, no LLM reference, and no LLM profile written. Same promise: `S19.profile-seed`. +- `F21.save-create`: `POST /api/agent-profiles/{name}` answers 201 and stores the profile under the path name with a server-minted id (a client `id` and body `name` are ignored) at revision 0; GET, the list summary and the file agree. Same promise: `S19.profile-store`. +- `F21.save-overwrite`: saving an existing name replaces the whole profile (no merge), keeps its `id` and sets `revision` to the previous one plus one, ignoring a client `revision`. Same promise: `S19.profile-store`. +- `F21.acp-kind`: an ACP profile (`agent_kind` `acp`, `acp_server`) saves with its ACP defaults and lists with `llm_profile_ref` null. Same promise: `S19.profile-model`. +- `F21.save-validation`: a missing `llm_profile_ref`, a field of the other kind or an unknown field, an invalid `acp_server` or `agent_kind`, a too-new `schema_version` and a non-object body are 422 with a reason, and nothing is stored. Same promise: `S19.profile-model`. - `F21.name-rules`: names must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$` on every name route and in `new_name`; 64 characters are accepted, 65, a leading dot or dash and a space are 422; activate's id is at most 128 characters (422). - `F21.not-found`: GET, rename and stored materialize of an unknown name are 404 `Agent profile '<name>' not found`, activate of an unknown id or of a name is 404, and DELETE of an unknown name is 200. - `F21.activate`: `POST /api/agent-profiles/{profile_id}/activate` sets only `active_agent_profile_id` (`agent_settings_applied` false): the list and settings show the id and `agent_settings` is unchanged. - `F21.activate-corrupt-settings`: with an unparsable `settings.json`, activate answers 500 `Failed to activate agent profile` (where LLM and meta-profile activation answer 409) and leaves the file as it was; the same activation works again once the file is restored. -- `F21.rename`: rename moves the profile to the new name with the same `id` and `revision`, keeps the active pointer, frees the old name (404), answers 200 "unchanged" for the same name and 409 when the target exists. -- `F21.delete`: DELETE removes the profile file; deleting another profile leaves the pointer alone, deleting the active one selects the existing default, or reseeds it when absent. +- `F21.rename`: rename moves the profile to the new name with the same `id` and `revision`, keeps the active pointer, frees the old name (404), answers 200 "unchanged" for the same name and 409 when the target exists. Same promise: `S19.profile-store`. +- `F21.delete`: DELETE removes the profile file; deleting another profile leaves the pointer alone, deleting the active one selects the existing default, or reseeds it when absent. Same promise: `S19.profile-store`. - `F21.delete-active-acp-reset`: deleting the active profile while `agent_settings` are ACP resets them to OpenHands defaults, keeps the server's `mcp_config`, and selects the default profile. -- `F21.materialize-valid`: materializing a stored profile whose references resolve answers `valid` true with `llm_profile_resolved`, `llm_api_key_set`, the resolved MCP keys and skills, the secret allow-list and `resolved_settings` with the key masked (the raw key appears nowhere), and persists nothing. -- `F21.materialize-dangling`: dangling LLM, MCP and meta-profile references and a tool the server cannot run give 200 `valid` false with every problem in `errors`, the dangling names in their fields, and no `resolved_settings`. -- `F21.materialize-draft`: a draft body is previewed under the path name without being saved: an ACP draft reports its credential secret names, `disabled_skills` removes a skill from `resolved_skills`, `mcp_server_refs` null exposes every configured MCP server and `[]` none, and an invalid draft or an extra top-level key is 422. -- `F21.limit`: the store holds at most 50 profiles: creating a 51st is 409 and writes nothing, overwriting one at the cap still works. -- `F21.corrupt-file`: a profile file that is not valid JSON is skipped by the list, GET, stored materialize and rename of it are 400 with the parse error, and DELETE still removes it. -- `F21.conversation-start`: a conversation started with `agent_profile_id` runs the profile's LLM profile and tool selection (not the live settings'), records `launched_agent_profile` with the id and the revision it launched with, and its stats and events name the profile's model after a run. +- `F21.materialize-valid`: materializing a stored profile whose references resolve answers `valid` true with `llm_profile_resolved`, `llm_api_key_set`, the resolved MCP keys and skills, the secret allow-list and `resolved_settings` with the key masked (the raw key appears nowhere), and persists nothing. Same promise: `S19.profile-resolve`. +- `F21.materialize-dangling`: dangling LLM, MCP and meta-profile references and a tool the server cannot run give 200 `valid` false with every problem in `errors`, the dangling names in their fields, and no `resolved_settings`. Same promise: `S19.profile-resolve`. +- `F21.materialize-draft`: a draft body is previewed under the path name without being saved: an ACP draft reports its credential secret names, `disabled_skills` removes a skill from `resolved_skills`, `mcp_server_refs` null exposes every configured MCP server and `[]` none, and an invalid draft or an extra top-level key is 422. Same promise: `S07.disabled-skills`. +- `F21.limit`: the store holds at most 50 profiles: creating a 51st is 409 and writes nothing, overwriting one at the cap still works. Same promise: `S19.profile-store`. +- `F21.corrupt-file`: a profile file that is not valid JSON is skipped by the list, GET, stored materialize and rename of it are 400 with the parse error, and DELETE still removes it. Same promise: `S19.profile-store`. +- `F21.conversation-start`: a conversation started with `agent_profile_id` runs the profile's LLM profile and tool selection (not the live settings'), records `launched_agent_profile` with the id and the revision it launched with, and its stats and events name the profile's model after a run. Same promise: `S19.profile-resolve`. - `F21.secret-scope`: a profile's `secret_refs` keep only the allowed names of the secrets a conversation is started with, without an error, and the dropped name is not persisted with the conversation; a profile without `secret_refs` keeps them all. -- `F21.launch-additions`: a conversation started with `agent_profile_id` and `agent_launch_additions.system_message_suffix_append` gets the text appended to the profile's `system_message_suffix` in its agent and its `SystemPromptEvent`, the additions are not stored in `meta.json`, and the stored profile keeps its revision; an unknown key or a value over 32768 characters is 422 and creates nothing. +- `F21.launch-additions`: a conversation started with `agent_profile_id` and `agent_launch_additions.system_message_suffix_append` gets the text appended to the profile's `system_message_suffix` in its agent and its `SystemPromptEvent`, the additions are not stored in `meta.json`, and the stored profile keeps its revision; an unknown key or a value over 32768 characters is 422 and creates nothing. Same promise: `S06.context-suffixes`. - `F21.ts-client`: the TypeScript `AgentProfilesClient` lists the seed, saves, reads, renames, activates (by id) and deletes a profile, previews a draft and the stored profile with `materializeAgentProfile`, `AgentServerClient.agentProfiles` reads the same store, and an unknown or invalid name rejects with a 404 or 422 `HttpError`. - `F21.restart`: profiles, their ids and revisions, and the active pointer survive a restart. - `F21.acp-skill-sourcing`: materializing an ACP profile injects no skills under the default `acp_skill_sourcing` `native`, and the discovered catalog (with `qa-f21-skill`) once the server runs with `OH_ACP_SKILL_SOURCING=openhands_managed` (what the shipped image sets); an OpenHands profile gets the catalog either way. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F22-meta-profiles.md b/.agents/skills/verify-agent-server/references/feature-map/F22-meta-profiles.md index f65dcfe9ba..7e44526cc2 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F22-meta-profiles.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F22-meta-profiles.md @@ -27,19 +27,19 @@ Routes: `GET /api/meta-profiles`, `GET /api/meta-profiles/{name}`, ## Sub-features - `F22.auth-required`: every meta-profile route answers 401 without a valid `X-Session-API-Key`, and nothing is stored. -- `F22.save-classes`: `POST /api/meta-profiles/{name}` with `classifier_model` and `classes` answers 201 `Meta-profile '<name>' saved`; the list shows `{name, classifier_model, num_classes}`, `GET` returns the full config, and the file `meta-profiles/<name>.json` is mode `600`. -- `F22.save-direct`: a direct-routing body (`prompt_template` with `{{ instance_text }}`, `model_table`, no `classes`) is saved; the list reports `num_classes: 0` and `GET` returns the template and table. -- `F22.validation`: a template without `{{ instance_text }}`, a template together with `classes`, a missing `classifier_model` and a class without `model` are 422 and save nothing. +- `F22.save-classes`: `POST /api/meta-profiles/{name}` with `classifier_model` and `classes` answers 201 `Meta-profile '<name>' saved`; the list shows `{name, classifier_model, num_classes}`, `GET` returns the full config, and the file `meta-profiles/<name>.json` is mode `600`. Same promise: `S03.meta-profile-store`. +- `F22.save-direct`: a direct-routing body (`prompt_template` with `{{ instance_text }}`, `model_table`, no `classes`) is saved; the list reports `num_classes: 0` and `GET` returns the template and table. Same promise: `S03.meta-profile-store`. +- `F22.validation`: a template without `{{ instance_text }}`, a template together with `classes`, a missing `classifier_model` and a class without `model` are 422 and save nothing. Same promise: `S03.meta-profile-store`. - `F22.empty-router`: a body with neither `classes` nor `prompt_template` (a router that can never route) is refused with 422. - `F22.overwrite`: saving an existing name replaces the whole config (here classes mode becomes direct mode) without adding a list entry. - `F22.name-rules`: names must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`: a leading `-` is 422 on all four named routes, 64 characters are accepted and 65 are 422. - `F22.not-found`: `GET` and `activate` of an unknown name answer 404 `Meta-profile '<name>' not found` and the active pointer stays; `DELETE` of an unknown name is 200 (idempotent). - `F22.activate`: `activate` answers 200 `Meta-profile '<name>' activated` with `applied: true`; the list's `active_meta_profile` and `GET /api/settings` (`active_meta_profile`, `agent_settings.active_meta_profile`, `enable_classify_and_switch_llm_tool: true`, inline `meta_profile`) follow, the agent's LLM is unchanged, and `settings.json` stores it. -- `F22.route-classes`: a conversation started from the settings carries the `ClassifyAndSwitchLLMTool` spec; when the agent calls `route_task_to_model`, the classifier profile picks the class, the observation names the class, profile and model, the conversation switches to `deepseek-pro`, both LLMs' usage is metered, and the events socket shows the switch live. -- `F22.route-direct`: with a direct-routing meta-profile active, the classifier's JSON answer switches the conversation to the named profile (`chosen_class` `model: deepseek-pro`). +- `F22.route-classes`: a conversation started from the settings carries the `ClassifyAndSwitchLLMTool` spec; when the agent calls `route_task_to_model`, the classifier profile picks the class, the observation names the class, profile and model, the conversation switches to `deepseek-pro`, both LLMs' usage is metered, and the events socket shows the switch live. Same promise: `S03.route-task-to-model`. +- `F22.route-direct`: with a direct-routing meta-profile active, the classifier's JSON answer switches the conversation to the named profile (`chosen_class` `model: deepseek-pro`). Same promise: `S03.route-task-to-model`. - `F22.route-missing-profile`: a meta-profile whose `classifier_model` names no saved LLM profile is accepted at save time; overwriting the active one takes effect without re-activation (the store wins over the inline copy), and the tool then answers an error observation naming the missing profile while the conversation keeps its LLM. - `F22.persist-restart`: meta-profiles, the active pointer, the settings' routing flag and a routed conversation's switched LLM survive a restart. -- `F22.delete`: deleting an inactive meta-profile removes the file and the list entry, keeps the active pointer, and a repeated delete is 200. +- `F22.delete`: deleting an inactive meta-profile removes the file and the list entry, keeps the active pointer, and a repeated delete is 200. Same promise: `S03.meta-profile-store`. - `F22.delete-active-clears`: deleting the active meta-profile clears `active_meta_profile` in the list and settings, disables the routing tool and drops the inline copy; new conversations no longer get the tool. - `F22.acp-clears-pointer`: switching the settings' agent kind to ACP clears the active pointer; back on OpenHands settings the meta-profile can be activated again. - `F22.activate-acp`: on ACP agent settings, where no routing tool can be attached, activation answers 200 `applied: false` with `Meta-profile '<name>' not activated: ...`, and `active_meta_profile` stays `null` in the list and in settings (as LLM-profile activation reports `llm_applied: false`). @@ -47,7 +47,7 @@ Routes: `GET /api/meta-profiles`, `GET /api/meta-profiles/{name}`, - `F22.corrupt-settings`: with an unreadable `settings.json`, activation is refused with 409 (as LLM-profile activation does) and the file is left untouched. - `F22.store-busy`: while another process holds the meta-profile store's lock, a save and a delete each wait out the 30-second lock timeout and answer 503 `Meta-profile store is busy. Please retry.` without changing anything; once the lock is free the same save is 201. - `F22.store-busy-responsive`: while a meta-profile save waits for the store lock, the server keeps answering other requests (`GET /alive` within 3 seconds). -- `F22.limit`: with 50 meta-profiles stored, creating another is 409 `Meta-profile limit reached (50)` while overwriting an existing one still answers 201. +- `F22.limit`: with 50 meta-profiles stored, creating another is 409 `Meta-profile limit reached (50)` while overwriting an existing one still answers 201. Same promise: `S03.meta-profile-store`. - `F22.ts-client`: the TypeScript `MetaProfilesClient` saves, reads, lists, activates and deletes a meta-profile and surfaces 404 and 422 as `HttpError`s. - `F22.persistence-dir`: with `OH_PERSISTENCE_DIR=~/<dir>`, meta-profiles are stored under the expanded directory next to `settings.json`, like every other store. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F23-llm-catalog-and-connections.md b/.agents/skills/verify-agent-server/references/feature-map/F23-llm-catalog-and-connections.md index abd45caa6e..8cf944073d 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F23-llm-catalog-and-connections.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F23-llm-catalog-and-connections.md @@ -28,8 +28,8 @@ Routes: `GET /api/llm/providers`, `GET /api/llm/models`, ## Sub-features -- `F23.acp-discovery-errors`: `POST /api/acp/models` requires a session key, validates `agent_settings`, and reports a missing local executable as HTTP 200 with `error.code: ACPSpawnError`, no models and no new conversation; `refresh: true` reports the same failure; `ACPClient.discoverModels` exposes the same structured errors and rejects unauthorized calls. -- `F23.acp-discovery-live`: the real Claude Code ACP adapter reports its default and offered models through REST and TypeScript `ACPClient.discoverModels`; a repeated request and `refresh: true` return the same catalog without persisting any conversation or probe directory. +- `F23.acp-discovery-errors`: `POST /api/acp/models` requires a session key, validates `agent_settings`, and reports a missing local executable as HTTP 200 with `error.code: ACPSpawnError`, no models and no new conversation; `refresh: true` reports the same failure; `ACPClient.discoverModels` exposes the same structured errors and rejects unauthorized calls. Same promise: `S21.model-discovery`. +- `F23.acp-discovery-live`: the real Claude Code ACP adapter reports its default and offered models through REST and TypeScript `ACPClient.discoverModels`; a repeated request and `refresh: true` return the same catalog without persisting any conversation or probe directory. Same promise: `S21.model-discovery`. - `F23.auth-required`: all seven routes answer 401 `{"detail": "Unauthorized"}` without a valid `X-Session-API-Key`, and nothing is created. - `F23.providers`: `GET /api/llm/providers` returns LiteLLM's provider names, sorted and unique, including `deepseek`, `openai` and `anthropic`. - `F23.models`: `GET /api/llm/models` returns every LiteLLM model, sorted and unique, with no `bedrock`-prefixed id, including `deepseek/deepseek-flash`. @@ -42,7 +42,7 @@ Routes: `GET /api/llm/providers`, `GET /api/llm/models`, - `F23.no-cipher`: on a server with neither `OH_SECRET_KEY` nor a session key (no cipher), a connection's key is stored in plaintext in the `600` file, the list still never returns it, and activating a linked profile applies it. - `F23.conn-create-validation`: a missing or empty `api_key` or `display_name`, an empty `provider`, a 129-character `display_name`, and any extra field (`extra_headers`, a client-chosen `id`) are 422 and create nothing; an empty `api_key` reports `value_error` `The secret must be a real value, not blank or redacted`. - `F23.conn-create-blank-key`: a whitespace-only or redacted-placeholder (`**********`) `api_key` is refused like an empty one instead of creating a keyless connection. -- `F23.linked-run`: a profile saved with `provider_connection_id` stores no inline key; activating it writes the connection's key into `agent_settings.llm`, and a conversation started from those settings runs on the connection's DeepSeek key and finishes. +- `F23.linked-run`: a profile saved with `provider_connection_id` stores no inline key; activating it writes the connection's key into `agent_settings.llm`, and a conversation started from those settings runs on the connection's DeepSeek key and finishes. Same promise: `S03.provider-connections`. - `F23.conn-update`: `PATCH` renames a connection and sets or clears (`null`) its `base_url`, keeping `id`, `created_at` and the key; the list shows the change, and the next activation of a linked profile applies the connection's `base_url` (and clears it). - `F23.conn-update-validation`: an empty body, `api_key: null` ("cannot be cleared"), a `null` `display_name` or `provider`, an empty `display_name` and extra fields are 422, an unknown id is 404, and none of them changes the stored connections. - `F23.conn-patch-blank-key`: a `PATCH` whose `api_key` is empty, whitespace-only or the redacted placeholder never replaces the stored key: the next activation of a linked profile still applies the previous key. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F24-openai-subscription.md b/.agents/skills/verify-agent-server/references/feature-map/F24-openai-subscription.md index 51d13b2e72..77d4c4330b 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F24-openai-subscription.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F24-openai-subscription.md @@ -33,7 +33,7 @@ Routes: `GET /api/llm/subscription/openai/models`, - `F24.status-invalid-file`: a credentials file that is not valid JSON or misses fields reads as `connected: false`, and status deletes it. - `F24.device-poll-invalid`: a poll without a body, without `device_code` or with a non-string one answers 422. - `F24.device-poll-unknown`: polling a token the server never issued answers 404 `Subscription device login not found or expired`. -- `F24.device-start`: `device/start` answers 200 with an opaque 43-character `device_code`, a `user_code`, `verification_uri` `https://auth.openai.com/codex/device`, `verification_uri_complete` null, `expires_at` 15 minutes ahead and `interval_seconds` of at least 1, and stores nothing on disk. +- `F24.device-start`: `device/start` answers 200 with an opaque 43-character `device_code`, a `user_code`, `verification_uri` `https://auth.openai.com/codex/device`, `verification_uri_complete` null, `expires_at` 15 minutes ahead and `interval_seconds` of at least 1, and stores nothing on disk. Same promise: `S03.subscription-login`. - `F24.device-poll-pending`: polling a started login before anyone approves it answers 200 `connected: false`, the token stays usable for the next poll, and no credentials are written. - `F24.logout-cancels-pending`: after logout, polling a token issued before it answers 404. - `F24.pending-lost-on-restart`: pending logins live in server memory, so after a restart a token that was pending answers 404. @@ -42,7 +42,7 @@ Routes: `GET /api/llm/subscription/openai/models`, - `F24.device-poll-outage`: a poll whose call to OpenAI fails (the network drops after `device/start`) answers a 5xx but keeps the login pending: once OpenAI is reachable again, the same token polls 200 `connected: false`. - `F24.status-offline`: with an expired login and no route to `auth.openai.com`, status answers 200 `connected: false`, as it does for a rejected refresh. - `F24.device-start-offline`: with no route to `auth.openai.com`, `device/start` and a poll of a login started before the outage answer a gateway error (502, 503 or 504), not an unhandled 500. -- `F24.login-complete`: after a human approves the user code with a ChatGPT account, a poll answers `connected: true` with `expires_at`, writes `auth/openai_oauth.json` with mode 600, consumes the token (the next poll is 404), and status shows the login. +- `F24.login-complete`: after a human approves the user code with a ChatGPT account, a poll answers `connected: true` with `expires_at`, writes `auth/openai_oauth.json` with mode 600, consumes the token (the next poll is 404), and status shows the login. Same promise: `S03.subscription-login`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F25-skills.md b/.agents/skills/verify-agent-server/references/feature-map/F25-skills.md index 3046b4e98c..d1cae8e188 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F25-skills.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F25-skills.md @@ -30,41 +30,41 @@ Routes: `POST /api/skills`, `GET /api/skills/catalog`, `POST /api/skills/sync`, ## Sub-features - `F25.auth`: every skills route answers 401 without a valid session key, and nothing is installed. -- `F25.public-repo`: against the real public extensions repository (network), sync clones it into the public cache and `POST /api/skills` with `load_public` lists skills loaded from that clone. +- `F25.public-repo`: against the real public extensions repository (network), sync clones it into the public cache and `POST /api/skills` with `load_public` lists skills loaded from that clone. Same promise: `S07.public-skills`. - `F25.marketplace-public`: against the real public extensions repository, `GET /api/skills/marketplace` lists its installable entries, discovered from `.plugin/marketplace.json` (the repository has no `marketplaces/default.json`). - `F25.profile-catalog`: `GET /api/skills/catalog` lists the public and user skills available to profile launches with their `origin`, excluding project-only skills. -- `F25.catalog-project`: `POST /api/skills` offline with a `project_dir` lists the project's AgentSkills (`agentskills`), legacy trigger skill (`knowledge`) and `AGENTS.md` (`repo`), counts them in `sources.project`, and silently skips a SKILL.md with an invalid name. -- `F25.catalog-user-precedence`: user skills from `~/.agents/skills` appear only with `load_user`, and a project skill overrides a user skill of the same name. +- `F25.catalog-project`: `POST /api/skills` offline with a `project_dir` lists the project's AgentSkills (`agentskills`), legacy trigger skill (`knowledge`) and `AGENTS.md` (`repo`), counts them in `sources.project`, and silently skips a SKILL.md with an invalid name. Same promise: `S07.project-skills`. +- `F25.catalog-user-precedence`: user skills from `~/.agents/skills` appear only with `load_user`, and a project skill overrides a user skill of the same name. Same promise: `S07.user-skills`, `S07.merge-precedence`. - `F25.catalog-server-sources`: `sandbox_config` URLs named `WORKER_*` become one always-on `work_hosts` skill, and `org_configs` repositories are cloned from a git URL (an unreachable one contributes nothing). - `F25.catalog-validation`: a wrong type, an incomplete `org_configs` entry or a traversing marketplace `repo_path` is 422. -- `F25.install-local`: installing a local skill directory copies it into the installed store, records metadata (`resolved_ref` null), and the skill appears in the installed list, the single view, `.installed.json` and the user tier of `POST /api/skills`. -- `F25.enable-disable`: `PATCH {"enabled": false}` keeps the files but hides the skill from the catalog, `{"enabled": true}` restores it, and a body without `enabled` is 422. +- `F25.install-local`: installing a local skill directory copies it into the installed store, records metadata (`resolved_ref` null), and the skill appears in the installed list, the single view, `.installed.json` and the user tier of `POST /api/skills`. Same promise: `S07.installed-lifecycle`. +- `F25.enable-disable`: `PATCH {"enabled": false}` keeps the files but hides the skill from the catalog, `{"enabled": true}` restores it, and a body without `enabled` is 422. Same promise: `S07.installed-lifecycle`. - `F25.install-conflict`: re-installing an installed skill is 409 and changes nothing; `force: true` reinstalls it and keeps the disabled flag. - `F25.install-errors`: an unfetchable source (missing path, unparsable source, escaping `repo_path`) is 400, an invalid skill (no SKILL.md, name mismatch, invalid directory name) is 422, an empty source is 422, and nothing is installed. - `F25.install-bad-frontmatter`: a SKILL.md with malformed YAML frontmatter is rejected as an invalid skill (422) and nothing is installed. - `F25.install-git`: a `file://` git source with `repo_path` installs offline and records the source, `repo_path` and the resolved commit SHA. -- `F25.sdk-load-skills`: `RemoteWorkspace.load_skills_from_agent_server()` turns the merged catalog of its `working_dir` into SDK `Skill` objects (project, user and enabled installed skills, keyword triggers kept) and an `AgentContext` that does not reload public skills. +- `F25.sdk-load-skills`: `RemoteWorkspace.load_skills_from_agent_server()` turns the merged catalog of its `working_dir` into SDK `Skill` objects (project, user and enabled installed skills, keyword triggers kept) and an `AgentContext` that does not reload public skills. Same promise: `S18.repos-and-skills`. - `F25.install-github`: the working GitHub form (`github:OpenHands/extensions` plus `repo_path`) installs a public skill from github.com at a full commit SHA, and uninstalling it removes it. - `F25.install-documented-sources`: every GitHub source form the OpenAPI description of `InstallSkillRequest.source` advertises installs, including the `github:owner/repo/<path>` shorthand. -- `F25.refresh`: refresh re-fetches from the recorded source: a new commit in the git source updates `resolved_ref` and the installed files, an edited local source updates the description, and the enabled flag is preserved. +- `F25.refresh`: refresh re-fetches from the recorded source: a new commit in the git source updates `resolved_ref` and the installed files, an edited local source updates the description, and the enabled flag is preserved. Same promise: `S07.installed-lifecycle`. - `F25.install-ref`: installing a git source with `ref` pins that commit (`resolved_ref`, the installed files, `requested_ref` in `.installed.json`); refresh moves it to the source's latest commit and drops `requested_ref`. - `F25.install-unknown-ref`: a `ref` the git source does not have is 400 and leaves the installation unchanged for both a first clone and a cached source. - `F25.refresh-unfetchable`: refreshing a skill whose recorded source no longer exists is a client error (400) and keeps the installation. -- `F25.name-errors`: an unknown skill name is 404 and a name outside `^[a-z0-9]+(-[a-z0-9]+)*$` is 422 on get, enable/disable, uninstall and refresh. +- `F25.name-errors`: an unknown skill name is 404 and a name outside `^[a-z0-9]+(-[a-z0-9]+)*$` is 422 on get, enable/disable, uninstall and refresh. Same promise: `S08.installation-manager`. - `F25.self-heal`: a skill directory copied into the store by hand is invisible to the single view until the list call discovers it with source `local`; a tracked directory removed by hand is pruned from the metadata by the list call. - `F25.sync`: `POST /api/skills/sync` updates the cached public repository from its origin and clears the 60 s public-skills cache, so a skill committed upstream is listed immediately (translated: a local git origin stands in for GitHub). - `F25.sync-offline`: when the public repository cannot be fetched, sync reports `status: error` and the stale clone stays in place for offline loading. -- `F25.catalog-registered-marketplace`: a registered marketplace with `auto_load` adds its standalone skills and plugin skills (and plugin commands) to the catalog, a name list selects some, and `auto_load: false` or `load_public: false` adds none. -- `F25.conversation-activation`: in a real DeepSeek conversation that loads user skills, a trigger word activates an enabled installed skill: `activated_knowledge_skills`, the user `MessageEvent` (REST and WebSocket replay) and the system prompt name it. -- `F25.invoke-skill`: an enabled installed AgentSkill without triggers is reachable only through the auto-attached `invoke_skill` tool; when the agent calls it, the observation carries the skill body and the conversation lists the skill in `invoked_skills`, not in `activated_knowledge_skills`. +- `F25.catalog-registered-marketplace`: a registered marketplace with `auto_load` adds its standalone skills and plugin skills (and plugin commands) to the catalog, a name list selects some, and `auto_load: false` or `load_public: false` adds none. Same promise: `S08.marketplace-auto-load`. +- `F25.conversation-activation`: in a real DeepSeek conversation that loads user skills, a trigger word activates an enabled installed skill: `activated_knowledge_skills`, the user `MessageEvent` (REST and WebSocket replay) and the system prompt name it. Same promise: `S06.context-skills`, `S07.keyword-trigger`. +- `F25.invoke-skill`: an enabled installed AgentSkill without triggers is reachable only through the auto-attached `invoke_skill` tool; when the agent calls it, the observation carries the skill body and the conversation lists the skill in `invoked_skills`, not in `activated_knowledge_skills`. Same promise: `S07.invoke-skill-autoattach`, `S07.invoke-skill-run`. - `F25.restart-persist`: installed skills, their enabled flags and resolved commits survive a server restart and keep their catalog visibility. - `F25.catalog-server-marketplaces`: a `registered_marketplaces` entry in the server config contributes without a request entry, a request entry of the same name replaces it, and one of another name adds to it. -- `F25.marketplace-catalog`: the catalog lists the public repository's entries (name, description, absolute source) with `installed: false`, and installing an entry's `source` flips it to `installed: true` immediately (translated: a local stand-in with `marketplaces/default.json`). +- `F25.marketplace-catalog`: the catalog lists the public repository's entries (name, description, absolute source) with `installed: false`, and installing an entry's `source` flips it to `installed: true` immediately (translated: a local stand-in with `marketplaces/default.json`). Same promise: `S07.marketplace-install`. - `F25.ts-client`: the TypeScript `SkillsClient` installs a skill from a git source, lists, reads, disables and re-enables it (`toggleSkill`, with `getSkills` following the flag), refreshes, syncs, reads the marketplace and uninstalls it; an unknown skill is a 404 `HttpError` and a client without a key gets 401. - `F25.marketplace-stale-cache`: after a sync brings a new entry into the public repository, the catalog lists it: a successful sync clears the catalog's 5-minute cache, which never keeps an empty result. -- `F25.uninstall`: uninstall deletes the directory and the metadata entry, the single view and a second uninstall are 404, the catalog drops the skill and the marketplace entry reads `installed: false` again. +- `F25.uninstall`: uninstall deletes the directory and the metadata entry, the single view and a second uninstall are 404, the catalog drops the skill and the marketplace entry reads `installed: false` again. Same promise: `S07.installed-lifecycle`. - `F25.public-catalog-disabled`: with `OH_PUBLIC_CATALOG_ENABLED=false` the marketplace catalog answers `disabled: true` and no entries although the public clone lists some (the default server answers `disabled: false` with them), while public skills still load and sync still succeeds. -- `F25.public-skills-disabled`: with `OH_PUBLIC_SKILLS_ENABLED=false` neither `POST /api/skills`, the profile catalog nor a conversation with `load_public_skills` gets public skills (nothing is cloned, and a present clone is not read), while user, project and registered-marketplace skills still load and the marketplace catalog stays enabled. +- `F25.public-skills-disabled`: with `OH_PUBLIC_SKILLS_ENABLED=false` neither `POST /api/skills`, the profile catalog nor a conversation with `load_public_skills` gets public skills (nothing is cloned, and a present clone is not read), while user, project and registered-marketplace skills still load and the marketplace catalog stays enabled. Same promise: `S07.public-skills-switch`. - `F25.public-extensions-disabled`: with both switches off, sync answers `status: error` with `Public extensions are disabled by instance policy` and leaves the clone untouched, the marketplace is `disabled: true` and empty, and public skills do not load. - `F25.load-registered-marketplaces`: `load_registered_marketplaces: true` loads auto-load registered marketplaces with `load_public: false`, `false` leaves them out with `load_public: true`, and leaving it out follows `load_public`. - `F25.long-description`: a skill description over 1024 characters is cut to 1024 in the project catalog, the installed view and the profile catalog, with a notice that names `invoke_skill(name=...)` and not the `SKILL.md` path. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F26-plugins.md b/.agents/skills/verify-agent-server/references/feature-map/F26-plugins.md index b347aabb0d..8111ee7e5a 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F26-plugins.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F26-plugins.md @@ -25,28 +25,28 @@ Routes: `POST /api/plugins`, `POST /api/plugins/fetch`, `POST /api/plugins/insta ## Sub-features - `F26.auth`: every plugins route answers 401 without a valid session key and installs nothing. -- `F26.catalog-project`: `POST /api/plugins` with a `project_dir` lists the plugins under its `.agents/plugins` and `.openhands/plugins` (also from the enclosing git root) with their skills, command skills and files; `load_project: false` or no `project_dir` lists none, and a request without a JSON body is 422. -- `F26.catalog-user-precedence`: `load_user` lists the plugins under `~/.agents/plugins` and `$OH_PERSISTENCE_DIR/plugins`; a project plugin wins over a user plugin of the same name, and a user-directory plugin wins over an enabled installed plugin of the same name. -- `F26.fetch-source`: strict fetch pins and describes a git plugin without installing it, reuses the pinned cache, and rejects an unknown ref, escaping subpath or missing secret with a typed 422 error. -- `F26.install-local`: installing a local Claude Code layout plugin answers 200 with its metadata, skills and files, copies it into the store, records it in `.installed.json`, and the ambient catalog lists it (never the store itself). -- `F26.install-agent-plugins`: a root `plugin.json` with the Agent Plugins `$schema` installs with its `skills/` and `dev.openhands/commands`; an object without `$schema` falls back to the Claude Code layout and infers metadata from its directory. -- `F26.install-git`: installing from a `file://` git source with `repo_path` records the source, `repo_path` and the resolved commit, and copies only that subdirectory. +- `F26.catalog-project`: `POST /api/plugins` with a `project_dir` lists the plugins under its `.agents/plugins` and `.openhands/plugins` (also from the enclosing git root) with their skills, command skills and files; `load_project: false` or no `project_dir` lists none, and a request without a JSON body is 422. Same promise: `S08.ambient-plugins`. +- `F26.catalog-user-precedence`: `load_user` lists the plugins under `~/.agents/plugins` and `$OH_PERSISTENCE_DIR/plugins`; a project plugin wins over a user plugin of the same name, and a user-directory plugin wins over an enabled installed plugin of the same name. Same promise: `S08.ambient-plugins`. +- `F26.fetch-source`: strict fetch pins and describes a git plugin without installing it, reuses the pinned cache, and rejects an unknown ref, escaping subpath or missing secret with a typed 422 error. Same promise: `S08.fetch-cache`. +- `F26.install-local`: installing a local Claude Code layout plugin answers 200 with its metadata, skills and files, copies it into the store, records it in `.installed.json`, and the ambient catalog lists it (never the store itself). Same promise: `S08.claude-code-layout`, `S08.installed-plugins`. +- `F26.install-agent-plugins`: a root `plugin.json` with the Agent Plugins `$schema` installs with its `skills/` and `dev.openhands/commands`; an object without `$schema` falls back to the Claude Code layout and infers metadata from its directory. Same promise: `S08.agent-plugins-layout`. +- `F26.install-git`: installing from a `file://` git source with `repo_path` records the source, `repo_path` and the resolved commit, and copies only that subdirectory. Same promise: `S08.source-parsing`. - `F26.install-repo-root-name`: a git source installed without `repo_path` and without a manifest is named after the repository, not after the server's cache directory. -- `F26.install-conflict`: a second install of the same plugin is 409 and changes nothing; `force: true` reinstalls and keeps the enabled flag. +- `F26.install-conflict`: a second install of the same plugin is 409 and changes nothing; `force: true` reinstalls and keeps the enabled flag. Same promise: `S08.installed-plugins`, `S08.installation-manager`. - `F26.install-errors`: unfetchable sources are 400, invalid plugins 422, malformed bodies 422, and none of them leaves anything in the store. -- `F26.install-github`: the GitHub shorthand `github:owner/repo` with a `repo_path` installs that subdirectory of the public repository with its resolved commit. +- `F26.install-github`: the GitHub shorthand `github:owner/repo` with a `repo_path` installs that subdirectory of the public repository with its resolved commit. Same promise: `S08.source-parsing`. - `F26.install-documented-source`: the GitHub source form that the OpenAPI description of `InstallPluginRequest.source` advertises installs the plugin. -- `F26.enable-disable`: `PATCH` with `enabled` toggles the flag, which the single view, the list and `.installed.json` show; a disabled plugin leaves the ambient catalog and comes back when re-enabled. -- `F26.name-errors`: an unknown plugin is 404 and a name outside `^[a-z0-9]+(-[a-z0-9]+)*$` is 422 on get, enable/disable, uninstall and refresh. +- `F26.enable-disable`: `PATCH` with `enabled` toggles the flag, which the single view, the list and `.installed.json` show; a disabled plugin leaves the ambient catalog and comes back when re-enabled. Same promise: `S08.installed-plugins`. +- `F26.name-errors`: an unknown plugin is 404 and a name outside `^[a-z0-9]+(-[a-z0-9]+)*$` is 422 on get, enable/disable, uninstall and refresh. Same promise: `S08.installation-manager`. - `F26.refresh`: refresh re-fetches from the recorded source (a new commit, an edited local manifest), drops a pinned ref, and keeps the enabled flag. - `F26.self-heal`: a directory copied into the store is unknown to the single view until a listing adopts it with source `local`; listing also ignores a directory whose manifest name differs and drops a record whose directory vanished. - `F26.refresh-unfetchable`: refreshing a plugin whose source vanished, whose source now holds an invalid manifest, or one adopted with source `local`, is a client error and keeps the installation. -- `F26.uninstall`: uninstall removes the directory and the record; the single view, the list and the ambient catalog no longer show it, and a second uninstall is 404. -- `F26.conversation-ambient`: a new conversation loads an enabled installed plugin on its first message: its `UserPromptSubmit` hook runs (a `HookExecutionEvent`, also on the events socket), its command skill activates, and its skills show in the conversation. +- `F26.uninstall`: uninstall removes the directory and the record; the single view, the list and the ambient catalog no longer show it, and a second uninstall is 404. Same promise: `S08.installed-plugins`. +- `F26.conversation-ambient`: a new conversation loads an enabled installed plugin on its first message: its `UserPromptSubmit` hook runs (a `HookExecutionEvent`, also on the events socket), its command skill activates, and its skills show in the conversation. Same promise: `S08.commands-as-skills`, `S08.ambient-plugins`, `S10.plugin-hooks-merge`. - `F26.conversation-disabled`: a conversation started after the plugin was disabled loads neither its hook nor its skills, while the user-directory plugins still load. - `F26.conversation-resume-disabled`: after a restart, an existing conversation drops a since-disabled plugin entirely: its command, skills and hook, and the MCP server its `.mcp.json` added. - `F26.restart-persist`: installed plugins, their records and their enabled flags survive a restart. -- `F26.marketplace-local`: against a local stand-in for the extensions repository, `GET /api/plugins/marketplace` lists only `./plugins/` entries (local contents for local entries, attach coordinates for git entries) with a fresh `installed` flag; a repository without a manifest gives an empty catalog that is not cached. +- `F26.marketplace-local`: against a local stand-in for the extensions repository, `GET /api/plugins/marketplace` lists only `./plugins/` entries (local contents for local entries, attach coordinates for git entries) with a fresh `installed` flag; a repository without a manifest gives an empty catalog that is not cached. Same promise: `S08.marketplace-manifest`. - `F26.marketplace-cache`: a non-empty catalog is served from a per-process cache (a new upstream entry appears only after a restart; the 5-minute TTL itself is not waited out), while `installed` stays fresh. - `F26.ts-client`: every `PluginsClient` method of the TypeScript client works against the server, and errors surface as `HttpError` with the status and detail. - `F26.marketplace-disabled`: with `OH_PUBLIC_CATALOG_ENABLED=false` the marketplace answers `disabled: true` and no plugins although the clone lists some (the default server answers `disabled: false` with them); installing from an explicit source still works, and an unrecognised value keeps the catalog on. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F27-hooks-subagents-tools.md b/.agents/skills/verify-agent-server/references/feature-map/F27-hooks-subagents-tools.md index 0bf5a34510..0890c5d4ba 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F27-hooks-subagents-tools.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F27-hooks-subagents-tools.md @@ -31,34 +31,34 @@ Routes: `GET /api/tools/`, `GET /api/tools/catalog`, `POST /api/sub-agents`, - `F27.validation`: a missing body or a wrongly typed field is 422 on `POST /api/hooks` and `POST /api/sub-agents`. - `F27.tools-list`: `GET /api/tools/` lists the registered tool names, including the default tools, `browser_tool_set`, the task and planning tools and registrations under a class name (`ClassifyAndSwitchLLMTool`). - `F27.tools-redirect`: `GET /api/tools` (no trailing slash) answers 307 to `/api/tools/`, and `/api/tools/catalog/` answers 307 to `/api/tools/catalog`; the CLI does not follow redirects. -- `F27.catalog`: `GET /api/tools/catalog` describes each offered tool with `user_selectable`, `usable`, `description` and `in_default_set`: the default tools are selectable default-set entries, internal tools are not selectable, and built-ins appear under their tool names. +- `F27.catalog`: `GET /api/tools/catalog` describes each offered tool with `user_selectable`, `usable`, `description` and `in_default_set`: the default tools are selectable default-set entries, internal tools are not selectable, and built-ins appear under their tool names. Same promise: `S14.preset-default`, `S17.usability`, `S17.catalog`. - `F27.child-conversation-tool`: the registered internal tool is usable, non-selectable and outside the default set; explicitly attaching it lets a real model start a child whose observation, parent link and shared workspace agree. -- `F27.hooks-project`: the project fixture's `hooks.json` (legacy `{"hooks": {...}}` wrapper with PascalCase keys) comes back as snake_case `HookConfig` with defaults filled in. -- `F27.hooks-formats`: a snake_case `hooks.json` keeps its matcher, timeout and `async` flag, a PascalCase key next to it is normalized, a `prompt` hook keeps its prompt, and a matcher left out becomes `*`. -- `F27.hooks-absent`: no `project_dir`, a misspelled field (`projectDir`), a missing directory, a directory without `.openhands/hooks.json`, a `hooks.json` only under `.agents/`, and an empty `{}` file all return 200 with `hook_config: null`. -- `F27.hooks-invalid`: unparsable JSON, an unknown event (PascalCase or snake_case), both spellings of one event, and a command hook without a command return 200 with `hook_config: null` and log `Failed to load hooks`. -- `F27.hooks-conversation`: the config `POST /api/hooks` returns, passed as `hook_config` to a DeepSeek conversation in that project, runs: a `HookExecutionEvent` with the hook's stdout appears over REST and the WebSocket replay, and the conversation keeps its `hook_config` across a restart; a command hook sees the conversation's secrets in its environment, as the agent's terminal does (known bug #6966: the hook's environment has none). +- `F27.hooks-project`: the project fixture's `hooks.json` (legacy `{"hooks": {...}}` wrapper with PascalCase keys) comes back as snake_case `HookConfig` with defaults filled in. Same promise: `S10.config-format`, `S10.config-discovery`. +- `F27.hooks-formats`: a snake_case `hooks.json` keeps its matcher, timeout and `async` flag, a PascalCase key next to it is normalized, a `prompt` hook keeps its prompt, and a matcher left out becomes `*`. Same promise: `S10.config-format`. +- `F27.hooks-absent`: no `project_dir`, a misspelled field (`projectDir`), a missing directory, a directory without `.openhands/hooks.json`, a `hooks.json` only under `.agents/`, and an empty `{}` file all return 200 with `hook_config: null`. Same promise: `S10.config-discovery`. +- `F27.hooks-invalid`: unparsable JSON, an unknown event (PascalCase or snake_case), both spellings of one event, and a command hook without a command return 200 with `hook_config: null` and log `Failed to load hooks`. Same promise: `S10.definition-validation`. +- `F27.hooks-conversation`: the config `POST /api/hooks` returns, passed as `hook_config` to a DeepSeek conversation in that project, runs: a `HookExecutionEvent` with the hook's stdout appears over REST and the WebSocket replay, and the conversation keeps its `hook_config` across a restart; a command hook sees the conversation's secrets in its environment, as the agent's terminal does (known bug #6966: the hook's environment has none). Same promise: `S10.command-exec`, `S10.execution-events`. - `F27.hooks-not-automatic`: a conversation in the same project started without `hook_config` runs no hook, although the project has `hooks.json` and the agent used the terminal. -- `F27.hooks-block`: a `PreToolUse` hook that exits 2 stops the agent's terminal call (a `HookExecutionEvent` with `blocked: true`, a `UserRejectObservation` from the hook with its stderr as the reason, no terminal observation, no file) and records the action in `blocked_actions` until the agent consumes it; a `UserPromptSubmit` hook that exits 2 records the queued message in `blocked_messages`, and running finishes at once without a model call. -- `F27.subagents-builtin`: without project and user agents (switched off, or a `project_dir` that does not exist) the catalog is exactly the four built-ins (`bash-runner`, `code-explorer`, `general-purpose`, `web-researcher`) with `level: builtin` and `is_builtin: true`, `mcp_servers` mirrors `mcp_config`, and `load_builtin: false` empties it. -- `F27.subagents-project`: project agents from `.agents/agents` and `.openhands/agents` are listed with `level: project`, their `source` file, and every frontmatter field (tools, skills, color, permission mode, iteration limit, `<example>` triggers, unknown keys in `metadata`, the body as `system_prompt`). -- `F27.subagents-user`: user agents from `~/.agents/agents` and `$OH_PERSISTENCE_DIR/agents` are listed with `level: user` only while `load_user` is on, and `load_project: false` hides the project's agents. -- `F27.subagents-precedence`: names are unique and the first source wins: a project agent shadows the built-in and the user agent of the same name, and `.agents/agents` shadows `.openhands/agents`. -- `F27.subagents-invalid`: definitions that do not validate (bad `permission_mode`, `max_iteration_per_run: 0`) are skipped with a logged warning, and `README.md`, subdirectories and non-`.md` files are ignored, while the request still answers 200. +- `F27.hooks-block`: a `PreToolUse` hook that exits 2 stops the agent's terminal call (a `HookExecutionEvent` with `blocked: true`, a `UserRejectObservation` from the hook with its stderr as the reason, no terminal observation, no file) and records the action in `blocked_actions` until the agent consumes it; a `UserPromptSubmit` hook that exits 2 records the queued message in `blocked_messages`, and running finishes at once without a model call. Same promise: `S10.exit-codes`, `S10.pre-tool-use-block`. +- `F27.subagents-builtin`: without project and user agents (switched off, or a `project_dir` that does not exist) the catalog is exactly the four built-ins (`bash-runner`, `code-explorer`, `general-purpose`, `web-researcher`) with `level: builtin` and `is_builtin: true`, `mcp_servers` mirrors `mcp_config`, and `load_builtin: false` empties it. Same promise: `S11.builtin-agents`. +- `F27.subagents-project`: project agents from `.agents/agents` and `.openhands/agents` are listed with `level: project`, their `source` file, and every frontmatter field (tools, skills, color, permission mode, iteration limit, `<example>` triggers, unknown keys in `metadata`, the body as `system_prompt`). Same promise: `S11.markdown-schema`, `S11.file-discovery`. +- `F27.subagents-user`: user agents from `~/.agents/agents` and `$OH_PERSISTENCE_DIR/agents` are listed with `level: user` only while `load_user` is on, and `load_project: false` hides the project's agents. Same promise: `S11.file-discovery`. +- `F27.subagents-precedence`: names are unique and the first source wins: a project agent shadows the built-in and the user agent of the same name, and `.agents/agents` shadows `.openhands/agents`. Same promise: `S11.file-precedence`. +- `F27.subagents-invalid`: definitions that do not validate (bad `permission_mode`, `max_iteration_per_run: 0`) are skipped with a logged warning, and `README.md`, subdirectories and non-`.md` files are ignored, while the request still answers 200. Same promise: `S11.markdown-schema`. - `F27.subagents-bad-skill`: a project agent whose `skills` names a skill that does not exist does not break conversations in that workspace: the catalog lists it (it validates frontmatter only), and the conversation skips it with a logged warning and accepts every message. -- `F27.catalog-sealed`: a client tool registered by a conversation (`client_tools`) is part of that conversation's agent but never enters the catalog, which is sealed at startup. -- `F27.client-tool-call`: when the agent calls a client tool, the events socket delivers a `ClientAction_<name>` `ActionEvent` with the model's arguments, the server answers it with the `Tool call dispatched to client.` acknowledgment and the run finishes; the tool is registered again when the conversation resumes after a restart; a duplicate name in one request, a name that collides with `terminal` and a reused name with a different schema are each 422 and create nothing, while the same name with the same schema is accepted. -- `F27.agent-definitions`: `agent_definitions` in the create request make that sub-agent available to the conversation's task tool, which delegates to it by name (the delegate runs the command, the `TaskObservation` names it); the definition is stored with the conversation and is not listed by `POST /api/sub-agents`. -- `F27.custom-tool-modules`: with `OH_EXTRA_PYTHON_PATH` naming a directory, `tool_module_qualnames` imports a module from it at create, the agent runs its tool, the conversation reports the mapping and `GET /api/tools/` lists the tool (the sealed catalog does not); after a restart the tool is gone until the conversation is loaded again, and a new message runs it again; an unimportable module is a logged warning and the create still succeeds. -- `F27.tool-schema-keywords`: the tool `parameters` a custom tool module's `Action` produces in the model request keep its `pattern`, `const` and `additionalProperties: false`; today they are dropped (#6766). +- `F27.catalog-sealed`: a client tool registered by a conversation (`client_tools`) is part of that conversation's agent but never enters the catalog, which is sealed at startup. Same promise: `S17.catalog`. +- `F27.client-tool-call`: when the agent calls a client tool, the events socket delivers a `ClientAction_<name>` `ActionEvent` with the model's arguments, the server answers it with the `Tool call dispatched to client.` acknowledgment and the run finishes; the tool is registered again when the conversation resumes after a restart; a duplicate name in one request, a name that collides with `terminal` and a reused name with a different schema are each 422 and create nothing, while the same name with the same schema is accepted. Same promise: `S17.client-tool-ack`, `S17.client-tools-conversation`. +- `F27.agent-definitions`: `agent_definitions` in the create request make that sub-agent available to the conversation's task tool, which delegates to it by name (the delegate runs the command, the `TaskObservation` names it); the definition is stored with the conversation and is not listed by `POST /api/sub-agents`. Same promise: `S11.definition-factory`, `S11.task-run`. +- `F27.custom-tool-modules`: with `OH_EXTRA_PYTHON_PATH` naming a directory, `tool_module_qualnames` imports a module from it at create, the agent runs its tool, the conversation reports the mapping and `GET /api/tools/` lists the tool (the sealed catalog does not); after a restart the tool is gone until the conversation is loaded again, and a new message runs it again; an unimportable module is a logged warning and the create still succeeds. Same promise: `S17.register-tool`, `S17.module-qualnames`. +- `F27.tool-schema-keywords`: the tool `parameters` a custom tool module's `Action` produces in the model request keep its `pattern`, `const` and `additionalProperties: false`; today they are dropped (#6766). Same promise: `S17.schema-constraints`. - `F27.import-modules`: a server started with `--import-modules` (and `OH_EXTRA_PYTHON_PATH`) lists the module's tool in `GET /api/tools/` and offers it in the catalog before any conversation exists; a module that cannot be imported stops the server before it binds its port. - `F27.custom-tool-missing-on-resume`: when a conversation's tool module can no longer be imported after a restart, the server drops the tool with its warning, keeps the conversation readable, and answers a new message without a 500; today the message is 500 (`ToolDefinition ... is not registered`). -- `F27.browser-playwright-path`: the browser probe finds a Playwright Chromium in `$PLAYWRIGHT_BROWSERS_PATH` as well as in standard paths, `~/.cache/ms-playwright` and `PATH`, and reports `browser_tool_set` usable. -- `F27.browser-probe`: without a Chromium the server can find, the catalog reports `browser_tool_set` unusable and `/server_info` `usable_tools` leaves it out; with a Playwright Chromium in the server's `~/.cache/ms-playwright` and a restart, the catalog reports it usable and `usable_tools` lists it. -- `F27.browser-disabled-launch`: a conversation started from the saved settings (default tools) gets `browser_tool_set` while the server finds Chromium, and leaves it out after a restart with `enable_browser: false`; a conversation created before keeps its stored tools. +- `F27.browser-playwright-path`: the browser probe finds a Playwright Chromium in `$PLAYWRIGHT_BROWSERS_PATH` as well as in standard paths, `~/.cache/ms-playwright` and `PATH`, and reports `browser_tool_set` usable. Same promise: `S15.chromium-detection`. +- `F27.browser-probe`: without a Chromium the server can find, the catalog reports `browser_tool_set` unusable and `/server_info` `usable_tools` leaves it out; with a Playwright Chromium in the server's `~/.cache/ms-playwright` and a restart, the catalog reports it usable and `usable_tools` lists it. Same promise: `S15.chromium-detection`, `S17.usability`. +- `F27.browser-disabled-launch`: a conversation started from the saved settings (default tools) gets `browser_tool_set` while the server finds Chromium, and leaves it out after a restart with `enable_browser: false`; a conversation created before keeps its stored tools. Same promise: `S15.preset-browser`. - `F27.browser-disabled`: with `enable_browser: false` the catalog reports `browser_tool_set` unusable although Chromium is present, while `GET /api/tools/` still lists it. - `F27.server-info-browser-disabled`: with `enable_browser: false`, `/server_info` `usable_tools` agrees with the catalog and leaves out `browser_tool_set`, although Chromium is present. -- `F27.subagents-browser-disabled`: with `enable_browser: false`, the sub-agents catalog leaves out the browser-only `web-researcher` and keeps the other built-ins. +- `F27.subagents-browser-disabled`: with `enable_browser: false`, the sub-agents catalog leaves out the browser-only `web-researcher` and keeps the other built-ins. Same promise: `S11.builtin-agents`, `S15.preset-browser`. - `F27.catalog-docker-runtime`: with `conversation_runtime: docker` the catalog probes nothing in the server process: `browser_tool_set` is usable for the stock image although the server finds no Chromium, and unusable for a `-minimal` image tag or with `conversation_image_has_browser: false`; `/server_info` `usable_tools` follows it for the browser. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md b/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md index d47e5490ed..5ccb79152f 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F30-openai-gateway.md @@ -28,7 +28,7 @@ Routes: `GET /v1/models`, `POST /v1/chat/completions`, `POST /v1/responses`, `PO - `F30.chat-basic`: `POST /v1/chat/completions` on a real model returns a `chat.completion` whose single choice carries the agent's final answer with `finish_reason` `stop`, echoes the model, reports usage, and names a new finished conversation in `X-OpenHands-ServerConversation-ID`. - `F30.inner-gateway-run`: the inner gateway route runs a new user turn on an existing conversation and returns its final text and token use. - `F30.active-agent-profile`: new gateway conversations keep the active Agent Profile's identity, tools and system suffix, override its LLM only for the launch, append gateway instructions without changing the stored profile, reject oversized instructions with 400, and fall back to default after active-profile deletion. -- `F30.chat-system-suffix`: `system` and `developer` messages become the new conversation's `agent_context.system_message_suffix` (in the system prompt) and never appear as user messages. +- `F30.chat-system-suffix`: `system` and `developer` messages become the new conversation's `agent_context.system_message_suffix` (in the system prompt) and never appear as user messages. Same promise: `S06.context-suffixes`. - `F30.chat-latest-user-only`: a new conversation receives only the last user message of `messages[]`; earlier user and assistant turns are dropped and stored nowhere. - `F30.chat-stream`: `stream: true` answers `text/event-stream` with the conversation header and the frames role, whole content, `finish_reason` `stop`, usage (only with `stream_options.include_usage`) and `[DONE]`; a failing streaming request is a plain JSON error, not an SSE frame. - `F30.chat-reuse-header`: sending `X-OpenHands-ServerConversation-ID` back appends the new user message to that conversation, reruns it (live on its events socket) and returns the same id. @@ -42,7 +42,7 @@ Routes: `GET /v1/models`, `POST /v1/chat/completions`, `POST /v1/responses`, `PO - `F30.chat-reuse-paused`: a chat completion that continues a conversation resting in `paused` (after an interrupt) answers 200 with its own turn's answer and the conversation header, and the turn's user message is stored once. - `F30.chat-agent-tools`: the agent runs its own tools in the server workspace (a file appears there), the answer carries no `tool_calls`, and the client's `tools`/`tool_choice` are ignored. - `F30.chat-input-errors`: bad chat input fails before any conversation exists: 400 without a user message or with a bad content part, 404 for a model without the `openhands_` prefix or an unknown profile, 400 for an invalid profile name, 422 for schema errors. -- `F30.observability-headers`: `X-OpenHands-Observability-*` headers are stored on the new conversation (chat and responses); invalid values are 422 before anything runs. +- `F30.observability-headers`: `X-OpenHands-Observability-*` headers are stored on the new conversation (chat and responses); invalid values are 422 before anything runs. Same promise: `S20.automation-headers`. - `F30.chat-run-error-500`: a run that ends in `error` answers 500 with the rewritten body (`exception` `500: Agent run ended with status: error`), without the conversation header, and leaves the conversation in `error`. - `F30.chat-confirmation-409`: with confirmation mode saved in the settings, a new conversation that stops `waiting_for_confirmation` answers 409 at once. - `F30.chat-reuse-confirmation`: a continued conversation that stops `waiting_for_confirmation` also answers 409 within seconds, like a new one. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md b/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md index e53874ea4e..3d5ade27e2 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F31-persistence-and-limits.md @@ -25,7 +25,7 @@ Routes: none - `F31.lease-released-graceful`: a graceful restart deletes the lease of every loaded conversation, catalog reads do not claim it again, and the next load claims generation 1 under the new pid. - `F31.lease-dead-owner-takeover`: after a SIGKILL the lease still names the dead pid; the restarted server takes it over on first load with generation 2 and logs the takeover. - `F31.lease-zombie-owner-takeover`: a restarted server serves its own conversation at once and takes the lease over as generation 2 also when the SIGKILLed owner's pid is still in the process table as a zombie (killed, not reaped); the lease records the owner's process start time (`owner_started_at`). -- `F31.crash-tool-call`: a conversation SIGKILLed during a tool call is loaded again at startup, comes back `error` with one `AgentErrorEvent` (`recovery_reason: server_restart`) answering the interrupted call and no observation, and the resumed run hands that error to the model. +- `F31.crash-tool-call`: a conversation SIGKILLed during a tool call is loaded again at startup, comes back `error` with one `AgentErrorEvent` (`recovery_reason: server_restart`) answering the interrupted call and no observation, and the resumed run hands that error to the model. Same promise: `S05.unmatched-actions`. - `F31.state-survives-restarts`: a conversation's title, tags and events, the settings, a stored secret and the LLM profiles read back unchanged after a graceful restart and after a SIGKILL. - `F31.no-plaintext-at-rest`: after a real model run, the provider key, a stored secret and a conversation secret appear nowhere in the run's persisted files or the server log, while the server still hands out the provider key and the stored secret and keeps the conversation secret encrypted in its registry. - `F31.second-instance-catalog`: a second server sharing the conversations directory lists, counts and reads conversations that existed when it started, without claiming their leases. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F32-webhooks-and-telemetry.md b/.agents/skills/verify-agent-server/references/feature-map/F32-webhooks-and-telemetry.md index fd34c62f80..2d09fdf86d 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F32-webhooks-and-telemetry.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F32-webhooks-and-telemetry.md @@ -30,7 +30,7 @@ Routes: none - `F32.events-post`: a run's events reach `POST {base_url}/events/<conversation id hex, no dashes>` as JSON arrays, through the agent's reply and the `finished` status update. - `F32.events-session-key`: every webhook request carries `Content-Type: application/json` and the server's first session key as `X-Session-API-Key`. - `F32.events-full-state`: each attach of a conversation (its creation, and its first use after a restart) posts a transient `ConversationStateUpdateEvent` with key `full_state`. -- `F32.events-no-deltas`: a streaming run's `StreamingDeltaEvent`s reach the events WebSocket but never a webhook. +- `F32.events-no-deltas`: a streaming run's `StreamingDeltaEvent`s reach the events WebSocket but never a webhook. Same promise: `S05.streaming-delta`. - `F32.conversation-created`: creating a conversation POSTs its `ConversationInfo` to `{base_url}/conversations`. - `F32.conversation-lifecycle`: pausing, interrupting, updating and deleting a conversation each POST its info again (`paused`, `paused`, the new title, `deleting`). - `F32.custom-headers`: a webhook's `headers` and a `base_url` path prefix apply to both event and conversation posts. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F33-sdk-remote-client.md b/.agents/skills/verify-agent-server/references/feature-map/F33-sdk-remote-client.md index a24585f7cf..b913156d24 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F33-sdk-remote-client.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F33-sdk-remote-client.md @@ -23,49 +23,49 @@ Routes: none ## Sub-features -- `F33.workspace-exec`: `Workspace(host=...)` returns a `RemoteWorkspace` whose `execute_command` returns the command's exit code, stdout and stderr, runs in the given `cwd`, gives up after `timeout` with exit `-1` and `timeout_occurred`; `alive` and `get_server_info()` match the server. -- `F33.workspace-bad-key`: with a wrong `api_key`, `execute_command` and `file_upload` report a 401 failure instead of raising, creating a conversation raises a 401 `HTTPStatusError`, and the server records no command, file or conversation. -- `F33.workspace-background`: `start_command` returns a command id at once, `get_command_output` shows no exit code while it runs, `stop_command` ends it (no later output), and stopping an unknown id is a no-op. -- `F33.workspace-files`: `file_upload` (from a path or from bytes) and `file_download` round-trip a binary file byte for byte; a missing remote file or a relative destination reports `success: false` with the server's 404 or 400 and leaves no local file. -- `F33.workspace-git`: `git_changes` (a path relative to `working_dir`) and `git_diff` return what the server's git routes return for a repository with a modified and an untracked file. -- `F33.workspace-settings`: `get_llm()` resolves the server's active LLM profile with a usable key, `get_llm(profile_name=...)` a named one (unknown names raise `FileNotFoundError`), `get_secrets()` returns `LookupSecret` references without values, and `get_mcp_config()` is empty on a fresh server. -- `F33.async-workspace`: `AsyncRemoteWorkspace` runs a command, starts, reads and stops a background command, uploads and downloads a file, reads git changes, a git diff and server info, and closes its client on context exit. -- `F33.workspace-runtime-scope`: a `RemoteWorkspace(runtime_conversation_id=...)` routes through `/api/conversations/{id}/...`: commands default to the conversation's workspace and land in its own bash history, paths outside that workspace are refused with 422, and an unknown conversation gives 404. -- `F33.workspace-runtime-lifecycle`: synchronous and asynchronous `get_runtime_session_key()` obtain a distinct worker key that authenticates a real child command; `release_runtime()` removes its container while conversation history stays readable, and reprovision permits the second SDK variant to repeat the lifecycle. -- `F33.conversation-create`: `Conversation(agent, workspace=RemoteWorkspace)` returns a `RemoteConversation` that creates the server conversation with the agent, working directory, tags and `max_iteration_per_run`; `send_message` stores the message without running and the agent's key is not echoed back. -- `F33.conversation-create-request`: `RemoteConversation.create(workspace, StartConversationRequest(...))` creates the conversation with the requested id from server-built `agent_settings`, tags and iteration limit, and runs its `initial_message`; repeating the same request returns the same conversation and adds no conversation or message. +- `F33.workspace-exec`: `Workspace(host=...)` returns a `RemoteWorkspace` whose `execute_command` returns the command's exit code, stdout and stderr, runs in the given `cwd`, gives up after `timeout` with exit `-1` and `timeout_occurred`; `alive` and `get_server_info()` match the server. Same promise: `S18.workspace-factory`, `S18.remote-command-exec`. +- `F33.workspace-bad-key`: with a wrong `api_key`, `execute_command` and `file_upload` report a 401 failure instead of raising, creating a conversation raises a 401 `HTTPStatusError`, and the server records no command, file or conversation. Same promise: `S18.remote-client`. +- `F33.workspace-background`: `start_command` returns a command id at once, `get_command_output` shows no exit code while it runs, `stop_command` ends it (no later output), and stopping an unknown id is a no-op. Same promise: `S18.remote-command-exec`. +- `F33.workspace-files`: `file_upload` (from a path or from bytes) and `file_download` round-trip a binary file byte for byte; a missing remote file or a relative destination reports `success: false` with the server's 404 or 400 and leaves no local file. Same promise: `S18.remote-files-git`. +- `F33.workspace-git`: `git_changes` (a path relative to `working_dir`) and `git_diff` return what the server's git routes return for a repository with a modified and an untracked file. Same promise: `S18.remote-files-git`. +- `F33.workspace-settings`: `get_llm()` resolves the server's active LLM profile with a usable key, `get_llm(profile_name=...)` a named one (unknown names raise `FileNotFoundError`), `get_secrets()` returns `LookupSecret` references without values, and `get_mcp_config()` is empty on a fresh server. Same promise: `S18.remote-settings`. +- `F33.async-workspace`: `AsyncRemoteWorkspace` runs a command, starts, reads and stops a background command, uploads and downloads a file, reads git changes, a git diff and server info, and closes its client on context exit. Same promise: `S18.async-remote`. +- `F33.workspace-runtime-scope`: a `RemoteWorkspace(runtime_conversation_id=...)` routes through `/api/conversations/{id}/...`: commands default to the conversation's workspace and land in its own bash history, paths outside that workspace are refused with 422, and an unknown conversation gives 404. Same promise: `S18.runtime-scope`. +- `F33.workspace-runtime-lifecycle`: synchronous and asynchronous `get_runtime_session_key()` obtain a distinct worker key that authenticates a real child command; `release_runtime()` removes its container while conversation history stays readable, and reprovision permits the second SDK variant to repeat the lifecycle. Same promise: `S18.runtime-scope`. +- `F33.conversation-create`: `Conversation(agent, workspace=RemoteWorkspace)` returns a `RemoteConversation` that creates the server conversation with the agent, working directory, tags and `max_iteration_per_run`; `send_message` stores the message without running and the agent's key is not echoed back. Same promise: `S04.factory`, `S18.conversation-binding`. +- `F33.conversation-create-request`: `RemoteConversation.create(workspace, StartConversationRequest(...))` creates the conversation with the requested id from server-built `agent_settings`, tags and iteration limit, and runs its `initial_message`; repeating the same request returns the same conversation and adds no conversation or message. Same promise: `S04.conversation-id`. - `F33.conversation-create-profile`: `RemoteConversation.create(workspace, StartConversationRequest(agent_profile_id=...))` builds the agent from a saved agent profile (its LLM profile, tool selection and system message suffix), records the launched profile and runs the `initial_message` to `finished`; `get_secrets(agent_profile_id=...)` returns only the profile's `secret_refs`. - `F33.conversation-run`: `send_message` plus a blocking `run()` returns once the agent finished; the server reports `finished` and the file the agent wrote exists. -- `F33.conversation-callbacks`: callbacks receive the run's events live over the events WebSocket, and after `run()` the callback counts and `state.events` match the server's persisted events kind by kind. -- `F33.conversation-stats`: `conversation_stats` and `state.execution_status` read back the server's numbers (accumulated cost and prompt tokens of the agent's LLM). -- `F33.conversation-stop-hook`: with a `Stop` hook that refuses the first stop (exit 2), a blocking `run()` returns only after the extra agent turn the hook forced: the server is `finished` when `run()` returns, and `state.events` already holds both stop `HookExecutionEvent`s (blocked, then allowed) and the hook's feedback, matching the server kind by kind. -- `F33.client-tools`: `Conversation(client_tools=[ClientToolSpec(...)])` delivers the model's call to the callbacks as a typed `ClientAction_<name>` `ActionEvent`, the server acknowledges it with a `ClientToolObservation` and finishes, and a second program that attaches without `client_tools` replays those events. -- `F33.conversation-streaming`: with a `stream=True` agent LLM, callbacks receive the run's `StreamingDeltaEvent` frames live and the server stores none of them. -- `F33.conversation-token-callbacks`: `Conversation(token_callbacks=[...])` on a remote workspace calls the token callbacks with the run's streamed deltas. -- `F33.conversation-streaming-state`: streamed deltas stay out of `state.events`, which keeps matching the server's stored events. -- `F33.completion-logs`: an agent LLM with `log_completions_to_file=True` gets one log file per model call in its `log_completions_folder`, written by the client from the `LLMCompletionLogEvent`s the server streams and stores (the server writes no file). -- `F33.conversation-attach`: a second program reattaches by id with `RemoteConversation.attach()` or `Conversation(conversation_id=...)` and replays the same history without creating anything; `attach()` on an unknown id raises 404 while `Conversation(conversation_id=<new id>)` creates that id. +- `F33.conversation-callbacks`: callbacks receive the run's events live over the events WebSocket, and after `run()` the callback counts and `state.events` match the server's persisted events kind by kind. Same promise: `S04.callbacks`. +- `F33.conversation-stats`: `conversation_stats` and `state.execution_status` read back the server's numbers (accumulated cost and prompt tokens of the agent's LLM). Same promise: `S02.conversation-stats`, `S04.stats`. +- `F33.conversation-stop-hook`: with a `Stop` hook that refuses the first stop (exit 2), a blocking `run()` returns only after the extra agent turn the hook forced: the server is `finished` when `run()` returns, and `state.events` already holds both stop `HookExecutionEvent`s (blocked, then allowed) and the hook's feedback, matching the server kind by kind. Same promise: `S10.stop-hook`. +- `F33.client-tools`: `Conversation(client_tools=[ClientToolSpec(...)])` delivers the model's call to the callbacks as a typed `ClientAction_<name>` `ActionEvent`, the server acknowledges it with a `ClientToolObservation` and finishes, and a second program that attaches without `client_tools` replays those events. Same promise: `S17.client-tool-ack`, `S17.client-tools-conversation`. +- `F33.conversation-streaming`: with a `stream=True` agent LLM, callbacks receive the run's `StreamingDeltaEvent` frames live and the server stores none of them. Same promise: `S02.streaming`. +- `F33.conversation-token-callbacks`: `Conversation(token_callbacks=[...])` on a remote workspace calls the token callbacks with the run's streamed deltas. Same promise: `S02.streaming`. +- `F33.conversation-streaming-state`: streamed deltas stay out of `state.events`, which keeps matching the server's stored events. Same promise: `S05.streaming-delta`. +- `F33.completion-logs`: an agent LLM with `log_completions_to_file=True` gets one log file per model call in its `log_completions_folder`, written by the client from the `LLMCompletionLogEvent`s the server streams and stores (the server writes no file). Same promise: `S02.log-completions`, `S05.completion-log-event`. +- `F33.conversation-attach`: a second program reattaches by id with `RemoteConversation.attach()` or `Conversation(conversation_id=...)` and replays the same history without creating anything; `attach()` on an unknown id raises 404 while `Conversation(conversation_id=<new id>)` creates that id. Same promise: `S04.resume`. - `F33.conversation-attach-large`: attaching to a conversation with more than 100 stored events pages through the whole history: `state.events` holds the server's events in the server's order (plus the one unpersisted socket snapshot), and the first reads of `state.confirmation_policy`, `state.security_analyzer`, `state.hook_config` and `state.agent` equal the server's. - `F33.conversation-state-reread`: reading `state.confirmation_policy`, `state.security_analyzer`, `state.agent` and `is_confirmation_mode_active` a second time gives the same answer as the first. -- `F33.conversation-ask-title`: `ask_agent()` answers and `generate_title()` returns a title within `max_length` without adding any event but `stats` state updates (the ask's usage is reported under `ask-agent-llm`), and `set_title()` changes the server's title. -- `F33.conversation-fork`: `fork(title=..., tags=...)` returns a `RemoteConversation` for a new server conversation with the same history, the title and the tags; `fork(agent=...)` is refused client-side. -- `F33.conversation-fork-branch`: `fork(from_event_id=..., reset_metrics=False)` returns a fork whose history ends at that event (its HEAD) with the source's stats, and `navigate_to(None)` and `navigate_to(event_id)` move the fork's HEAD without changing its events. -- `F33.conversation-plugins-tag`: `Conversation(plugins=[PluginSource(...)])` with inline URL credentials stores the `plugins` tag with the credentials masked, and the token reaches no server state. -- `F33.conversation-confirmation`: `set_confirmation_policy(AlwaysConfirm())` makes `run()` return at `waiting_for_confirmation`; `reject_pending_actions()` records a `UserRejectObservation`, never runs the action and leaves the conversation `idle`. -- `F33.conversation-pause`: `pause()` from another thread during a blocking `run()` lets the tool step in flight finish, makes `run()` return with `paused` and a `PauseEvent`, and a later `run()` resumes to `finished`. -- `F33.conversation-interrupt`: `interrupt()` from another thread while a blocking `run()` waits on a model call makes `run()` return at once without raising, at `paused` with one `InterruptEvent` and no `PauseEvent`, and a later `run()` reaches `finished`. +- `F33.conversation-ask-title`: `ask_agent()` answers and `generate_title()` returns a title within `max_length` without adding any event but `stats` state updates (the ask's usage is reported under `ask-agent-llm`), and `set_title()` changes the server's title. Same promise: `S04.title-and-tags`. +- `F33.conversation-fork`: `fork(title=..., tags=...)` returns a `RemoteConversation` for a new server conversation with the same history, the title and the tags; `fork(agent=...)` is refused client-side. Same promise: `S04.fork`. +- `F33.conversation-fork-branch`: `fork(from_event_id=..., reset_metrics=False)` returns a fork whose history ends at that event (its HEAD) with the source's stats, and `navigate_to(None)` and `navigate_to(event_id)` move the fork's HEAD without changing its events. Same promise: `S04.fork-branch`. +- `F33.conversation-plugins-tag`: `Conversation(plugins=[PluginSource(...)])` with inline URL credentials stores the `plugins` tag with the credentials masked, and the token reaches no server state. Same promise: `S08.conversation-attach`. +- `F33.conversation-confirmation`: `set_confirmation_policy(AlwaysConfirm())` makes `run()` return at `waiting_for_confirmation`; `reject_pending_actions()` records a `UserRejectObservation`, never runs the action and leaves the conversation `idle`. Same promise: `S13.confirm-wait`, `S13.confirm-approve-reject`. +- `F33.conversation-pause`: `pause()` from another thread during a blocking `run()` lets the tool step in flight finish, makes `run()` return with `paused` and a `PauseEvent`, and a later `run()` resumes to `finished`. Same promise: `S04.pause`. +- `F33.conversation-interrupt`: `interrupt()` from another thread while a blocking `run()` waits on a model call makes `run()` return at once without raising, at `paused` with one `InterruptEvent` and no `PauseEvent`, and a later `run()` reaches `finished`. Same promise: `S04.arun-interrupt`. - `F33.conversation-run-timeout`: `run(timeout=...)` raises `ConversationRunError` wrapping a `TimeoutError` on time while the server keeps running, and a second `run()` joins the run in flight (the server's 409 is not an error) and returns at `finished`. - `F33.conversation-run-join-long-tool`: a second `run()` also joins a run whose current tool call lasts longer than 30 s, instead of raising. -- `F33.conversation-stuck`: when the server's stuck detector stops a looping agent, `run()` raises `ConversationRunError` ("got stuck") at once and the server reports `stuck`. -- `F33.conversation-secrets`: `update_secrets()` with a plain value and with a `LookupSecret` from `get_secrets()` makes both available to the agent's commands, masked in observations and not stored in plaintext. -- `F33.conversation-run-error`: when the model provider rejects the key, `run()` raises `ConversationRunError` carrying the server's `ConversationErrorEvent` code, and the server reports `error`. +- `F33.conversation-stuck`: when the server's stuck detector stops a looping agent, `run()` raises `ConversationRunError` ("got stuck") at once and the server reports `stuck`. Same promise: `S01.stuck-detect`. +- `F33.conversation-secrets`: `update_secrets()` with a plain value and with a `LookupSecret` from `get_secrets()` makes both available to the agent's commands, masked in observations and not stored in plaintext. Same promise: `S04.secrets-update`. +- `F33.conversation-run-error`: when the model provider rejects the key, `run()` raises `ConversationRunError` carrying the server's `ConversationErrorEvent` code, and the server reports `error`. Same promise: `S04.run-error`. - `F33.ws-unavailable`: when the events socket cannot connect, `Conversation(...)` raises `WebSocketConnectionError` after `OPENHANDS_REMOTE_WS_READY_TIMEOUT` seconds (the conversation it created stays on the server); with `OPENHANDS_REMOTE_WS_READY_REQUIRED=false` it continues, and `run()` returns at `finished` through the REST fallback about 30 s after the server finished, with `state.events` matching the server. - `F33.conversation-reconnect`: a live `RemoteConversation` survives a server restart: its events socket reconnects, a new message runs to `finished`, and `state.events` again matches the server. - `F33.conversation-reconnect-callbacks`: an event the server records while a live `RemoteConversation`'s events socket is down reaches its callbacks once the socket is back, not only `state.events`. -- `F33.conversation-restart`: after a restart a new program reattaches to a finished conversation, sees the same history, and continues it to `finished`. -- `F33.custom-tool-module`: a tool registered in the client process is sent as `tool_module_qualnames`, imported by a server started with `OH_EXTRA_PYTHON_PATH`, and runs on the server when the agent calls it; a client-registered module the server cannot import is only a logged warning; reattaching from a process without the module raises `ImportError` naming the tool. +- `F33.conversation-restart`: after a restart a new program reattaches to a finished conversation, sees the same history, and continues it to `finished`. Same promise: `S04.resume`. +- `F33.custom-tool-module`: a tool registered in the client process is sent as `tool_module_qualnames`, imported by a server started with `OH_EXTRA_PYTHON_PATH`, and runs on the server when the agent calls it; a client-registered module the server cannot import is only a logged warning; reattaching from a process without the module raises `ImportError` naming the tool. Same promise: `S17.register-tool`, `S17.module-qualnames`. - `F33.custom-tool-module-unimportable`: a conversation whose agent names a tool from a module the server cannot import is rejected at create with a 422 naming the tool and the module, never a 500 on a later message. -- `F33.conversation-close`: `close()` keeps the server conversation when `delete_on_close=False` and deletes it when the `Conversation(...)` factory's default `delete_on_close=True` applies. +- `F33.conversation-close`: `close()` keeps the server conversation when `delete_on_close=False` and deletes it when the `Conversation(...)` factory's default `delete_on_close=True` applies. Same promise: `S04.close`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md b/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md index be7b4c3bb2..9c25fb62b9 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md @@ -41,7 +41,7 @@ Routes: `POST /api/conversations/{conversation_id}/reactions` - `F37.flag-gates`: with the flag off, `POST .../reactions` is 409 and a new conversation has no `react`/`unreact`; with it on, a new conversation started without naming them has both tools. Turning it off again refuses the route at once (409 `Conversation reactions are turned off`) while that conversation keeps its tools. - `F37.rest`: `POST .../reactions` with `{emoji, target_event_id}` saves a user `ReactionEvent` on that message and returns it; `removed: true` saves the removal; the conversation stays `idle`. - `F37.rest-validation`: an emoji outside the set is 422, an unknown target is 400, and leaving out the target with no agent message yet is 400; nothing is saved. -- `F37.tool`: an agent with `react` reacts to the user's latest message when asked to react first, and leaves the reaction on after it answers: the log has the tool's `ObservationEvent` followed by one agent `ReactionEvent` whose target is that message, and no removal. +- `F37.tool`: an agent with `react` reacts to the user's latest message when asked to react first, and leaves the reaction on after it answers: the log has the tool's `ObservationEvent` followed by one agent `ReactionEvent` whose target is that message, and no removal. Same promise: `S16.reactions`. - `F37.rest-while-confirming`: a reaction posted while a tool call waits for confirmation answers 200 but is held back: it joins the log right after the call's result once the call is accepted, never between the call and its result, and the run finishes. - `F37.rest-default`: after the agent has answered, a reaction without `target_event_id` goes on its answer: the latest agent message or `finish`. - `F37.agent-sees-user-reactions`: a user's reaction on the agent's `finish` keeps the answer's text in `target_text_ending`, and the next model request carries it as `[The user reacted ❤️ to this message:]` plus that text, while the agent's own `react` reaction never appears as a reaction line. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F39-message-edits.md b/.agents/skills/verify-agent-server/references/feature-map/F39-message-edits.md index cc7600049a..33a36a9839 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F39-message-edits.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F39-message-edits.md @@ -32,8 +32,8 @@ Routes: none of its own (the flag's routes are F18's). - `F39.flag-gates`: `GET /api/settings/experimental` lists `message_edits` with `enabled: false`; a conversation started while it is off has no `edit_message`, and one started after `PATCH .../message_edits` `{"enabled": true}` has it without naming it. - `F39.tool`: a real model that asked the user a question, then got the answer, edits the question: one agent `MessageEditEvent` targets its earlier message, its `text` strikes the question with `~~` and carries an `[Edit:` note, and the stored message is unchanged. -- `F39.agent-sees`: with an `llm-stub` recorder, the requests after an edit carry the message as first written plus the `edit_message` call and its `Edited.` result, and nothing else for the edit. -- `F39.miss`: an `old_str` the agent never wrote answers an error observation (`old_str is not in any of your messages`) and records no `MessageEditEvent`. +- `F39.agent-sees`: with an `llm-stub` recorder, the requests after an edit carry the message as first written plus the `edit_message` call and its `Edited.` result, and nothing else for the edit. Same promise: `S16.message-edit-llm-view`. +- `F39.miss`: an `old_str` the agent never wrote answers an error observation (`old_str is not in any of your messages`) and records no `MessageEditEvent`. Same promise: `S16.message-edit-tool`. - `F39.reaction-quotes-edit`: with `conversation_reactions` also on, a user's reaction on the edited message keeps the edited text in `target_text_ending`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-agent-server/references/feature-map/F40-ask-user.md b/.agents/skills/verify-agent-server/references/feature-map/F40-ask-user.md index ce19eaea35..9cdbde4d4a 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F40-ask-user.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F40-ask-user.md @@ -39,13 +39,13 @@ Routes: `POST /api/conversations/{conversation_id}/ask_user/answer` ## Sub-features - `F40.flag-gates`: `GET /api/settings/experimental` lists `ask_user` ("Decision cards") off on a fresh server; with it off the answer route is 409 `Decision cards are turned off` and a new conversation has no `ask_user` tool; `PATCH .../ask_user` `{"enabled": true}` turns it on and a new conversation started without naming the tool has exactly `ask_user`. -- `F40.question-event`: on DeepSeek flash, an agent asked to let the user pick calls `ask_user`: the log and the socket carry an `ActionEvent` (`tool_name: ask_user`, an `AskUserAction` with the question and two options) and its `AskUserObservation` (`is_error: false`, `Your turn ends here`), and the conversation is `finished` with no `finish` call. -- `F40.answer-errors`: an `ask_id` that is not the pending question, an option past the last one and an answer with neither option nor words are 400; a negative `option_index` or a missing `ask_id` is 422; an unknown conversation is 404 and a missing key 401; switching the flag off refuses a valid answer at once (409) while the conversation keeps its tool; none of these saves anything and the question stays pending. +- `F40.question-event`: on DeepSeek flash, an agent asked to let the user pick calls `ask_user`: the log and the socket carry an `ActionEvent` (`tool_name: ask_user`, an `AskUserAction` with the question and two options) and its `AskUserObservation` (`is_error: false`, `Your turn ends here`), and the conversation is `finished` with no `finish` call. Same promise: `S16.ask-user-tool`. +- `F40.answer-errors`: an `ask_id` that is not the pending question, an option past the last one and an answer with neither option nor words are 400; a negative `option_index` or a missing `ask_id` is 422; an unknown conversation is 404 and a missing key 401; switching the flag off refuses a valid answer at once (409) while the conversation keeps its tool; none of these saves anything and the question stays pending. Same promise: `S16.ask-user-answer`. - `F40.answer`: `POST .../ask_user/answer` `{ask_id, option_index, text}` returns the saved `AskUserAnswerEvent` (`source: user`, `option_label`, the words, `parent_id` the question's observation); the user message `[Answer to your question: "<question>"]` / `Option 2: <label>` / words follows it, the agent runs to `finished` and answers after it, the socket pushes both events, and a second answer to the same question is 400. -- `F40.chat-reply`: a chat message answers a pending question in words: after `POST .../events` with the user's reply the agent runs, and the card can no longer be answered (400), with no `AskUserAnswerEvent`. -- `F40.stop-hook-waits`: with a Stop hook that denies the first completion, a turn that ends on `ask_user` stays `finished` with no hook run and no second model turn, while an ordinary turn on the same hook is denied once and continues. -- `F40.goal-asked`: a `/goal` round that ends on the agent's question records goal status `interrupted` (`verdict: null`, iteration 0) without judging or sending a follow-up; the objective message carries `sender: goal`; `/goal` and `/goal/resume` are 400 `pending_ask` while the question waits and add no goal status; after the user answers on the card, `/goal/resume` runs and the judge records `complete`. -- `F40.goal-abandoned-branch`: after navigating back to a user message before an unanswered question, a new `/goal` runs its round and is judged (`complete` or `capped`, with a verdict) rather than stopping on the abandoned question. +- `F40.chat-reply`: a chat message answers a pending question in words: after `POST .../events` with the user's reply the agent runs, and the card can no longer be answered (400), with no `AskUserAnswerEvent`. Same promise: `S16.ask-user-chat-reply`. +- `F40.stop-hook-waits`: with a Stop hook that denies the first completion, a turn that ends on `ask_user` stays `finished` with no hook run and no second model turn, while an ordinary turn on the same hook is denied once and continues. Same promise: `S16.ask-user-holds-completion`. +- `F40.goal-asked`: a `/goal` round that ends on the agent's question records goal status `interrupted` (`verdict: null`, iteration 0) without judging or sending a follow-up; the objective message carries `sender: goal`; `/goal` and `/goal/resume` are 400 `pending_ask` while the question waits and add no goal status; after the user answers on the card, `/goal/resume` runs and the judge records `complete`. Same promise: `S04.goal-asked`. +- `F40.goal-abandoned-branch`: after navigating back to a user message before an unanswered question, a new `/goal` runs its round and is judged (`complete` or `capped`, with a verdict) rather than stopping on the abandoned question. Same promise: `S04.goal-asked-branch`. - `F40.ts-client`: the TypeScript `ConversationClient.answerAskUser` answers a pending question with `optionIndex`, returning an event that `isAskUserAnswerEvent` accepts, and a second answer rejects with a 400 `HttpError` (`That question is not waiting for an answer.`). - `F40.runtime-env`: with `OH_RUNTIME_EXPERIMENTAL_FEATURES='{"ask_user": true}'` and the stored flag off, new conversations get `ask_user` and the answer route no longer answers 409 (a stale `ask_id` is 400); with `{}` new conversations do not get the tool. diff --git a/.agents/skills/verify-agent-server/references/feature-map/README.md b/.agents/skills/verify-agent-server/references/feature-map/README.md index 7fcaa36b8f..41ecfe64f4 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/README.md +++ b/.agents/skills/verify-agent-server/references/feature-map/README.md @@ -61,6 +61,29 @@ sections in this order: `Sub-features`, `How to get to it (agent POV)`, How to write and prove new entries: [../mapping.md](../mapping.md). How to keep them honest: [../maintenance.md](../maintenance.md). +## Twin links with the SDK map + +The in-process SDK map +([`verify-sdk`](../../../verify-sdk/references/feature-map/README.md)) +proves many of the same promises without a server: a server conversation runs +the SDK's `LocalConversation`, and family F33 drives the server through the +SDK's own remote client. When a bullet here proves the same promise as an SDK +bullet, the two are twins and link each other at the end of their lines: + +- here: `` Same promise: `S04.pause`. `` +- there: `` Server: `F05.pause-idle`, `F33.conversation-pause`. `` + +Link only the same promise (a pass driving both bullets expects the same +answer), not a related feature. Server-only contract (status codes, auth, +stores, sockets, webhooks) has no twin. `map check` (and `sdk-map check`) +validate both directions: every named ID exists in the other map and links +back. `map links [--file Fnn] [--id Fnn.x]` lists the pairs. Twins that +disagree, one passing and the other failing, usually point at the seam +between the SDK and the server (serialization, the async path, persistence, +restart). When a pass adds, splits, renames or removes a linked bullet, it +updates the twin's link in the same commit; see +[mapping.md](../mapping.md#3-write-each-entry). + ## Families | ID | Family | What it covers | Entry points | Needs | Sub-features | diff --git a/.agents/skills/verify-agent-server/references/mapping.md b/.agents/skills/verify-agent-server/references/mapping.md index 4fffd2420c..6c1fff8999 100644 --- a/.agents/skills/verify-agent-server/references/mapping.md +++ b/.agents/skills/verify-agent-server/references/mapping.md @@ -114,6 +114,10 @@ Then exactly four H2 sections, in this order: 1. `## Sub-features`: one bullet per ID: `` - `F03.pause`: pausing a running conversation stops it and reports `paused`. `` + When the SDK map has a bullet for the same promise, end the line with + `` Same promise: `S04.pause`. `` and add `` Server: `F03.pause`. `` to that + SDK bullet (see the index's + [Twin links](feature-map/README.md#twin-links-with-the-sdk-map)). 2. `## How to get to it (agent POV)`: every entry point, as a consumer meets it: the REST route, the WebSocket, the SDK method (`RemoteConversation.pause()`), the TypeScript client method, the config @@ -366,10 +370,13 @@ control-agent-server map check --file F03 # one entry while others are being w control-agent-server map check # whole map: structure, IDs, index, commands parse, routes exist control-agent-server map coverage # routes no family owns or drives control-agent-server map ids # every ID with its file, for the index table +control-agent-server map links # SDK/server twin pairs (`Same promise:` / `Server:`) ``` `map check` parses every `control-agent-server ...` command in the map with the -real argument parser (so a renamed flag fails the check), verifies every route +real argument parser (so a renamed flag fails the check), checks that every +`Same promise:` and SDK `Server:` link names an existing bullet that links +back, verifies every route on a `Routes:` line and every `api METHOD PATH` call against the app's route table, and checks the index counts. Whoever owns the index ([feature-map/README.md](feature-map/README.md)) adds the new file to the diff --git a/.agents/skills/verify-agent-server/scripts/control_agent_server.py b/.agents/skills/verify-agent-server/scripts/control_agent_server.py index 8715685ac6..cb50809e97 100644 --- a/.agents/skills/verify-agent-server/scripts/control_agent_server.py +++ b/.agents/skills/verify-agent-server/scripts/control_agent_server.py @@ -4122,6 +4122,139 @@ def parse_index() -> list[dict[str, Any]]: return rows +# -------------------------------------------------------------------------- +# Twin links between the SDK map and the agent-server map +# -------------------------------------------------------------------------- + +# An SDK bullet whose promise an agent-server bullet also proves (through the +# server's routes, sockets or the SDK's remote client) ends with +# `Server: `Fnn.x`, ...`; each of those bullets ends with +# `Same promise: `Snn.y`, ...` naming it back. Both map tools check both +# directions, so a link is never one-sided. The other map is the sibling +# skill's, found next to this one under .agents/skills/. +TWIN_LINKS = { + "S": ("verify-sdk", "Server", "F"), + "F": ("verify-agent-server", "Same promise", "S"), +} + + +def twin_map_dir(side: str) -> Path: + skills = FEATURE_MAP_DIR.parents[2] + return skills / TWIN_LINKS[side][0] / "references" / "feature-map" + + +TWIN_SUB_FEATURE_RE = re.compile(r"^- `([SF]\d{2}\.[a-z0-9][a-z0-9-]*)`: ") + + +def read_twin_links(side: str) -> dict[str, dict[str, Any]] | None: + """Every sub-feature of one map with the twin IDs its link names. + + None when that map is not in this checkout. + """ + _, label, other = TWIN_LINKS[side] + directory = twin_map_dir(side) + if not directory.is_dir(): + return None + one = rf"`{other}\d{{2}}\.[a-z0-9-]+`" + link_re = re.compile(rf" {label}: ({one}(?:, {one})*)\.$") + rows: dict[str, dict[str, Any]] = {} + for path in sorted(directory.glob(f"{side}[0-9][0-9]-*.md")): + section = None + for number, line in enumerate(path.read_text().splitlines(), 1): + if line.startswith("## "): + section = line[3:].strip() + continue + match = ( + TWIN_SUB_FEATURE_RE.match(line) if section == "Sub-features" else None + ) + if not match or not match.group(1).startswith(side): + continue + row: dict[str, Any] = { + "file": path.name, + "line": number, + "links": [], + "error": None, + } + if f" {label}:" in line: + found = link_re.search(line) + if found: + row["links"] = re.findall(r"`([^`]+)`", found.group(1)) + else: + row["error"] = ( + f"`{label}:` must end the line as " + f"`{label}: `{other}nn.id`, `{other}nn.id`.`" + ) + rows[match.group(1)] = row + return rows + + +def twin_link_problems(side: str, ids: set[str] | None = None) -> dict[str, list[str]]: + """Link problems keyed by file name, checked from both maps. + + `side` is this tool's map ("S" or "F"); `ids` limits the check to links + that touch those sub-features of it. + """ + other = TWIN_LINKS[side][2] + maps = {side: read_twin_links(side), other: read_twin_links(other)} + if maps[side] is None or maps[other] is None: + return {} + problems: dict[str, list[str]] = {} + + def add(row: dict[str, Any], message: str) -> None: + problems.setdefault(row["file"], []).append(message) + + for here_side, there_side in ((side, other), (other, side)): + here, there = maps[here_side], maps[there_side] + label, back = TWIN_LINKS[here_side][1], TWIN_LINKS[there_side][1] + for twin_id, row in here.items(): + where = f"line {row['line']}: {twin_id}" + if ids is None or twin_id in ids: + if row["error"]: + add(row, f"{where}: {row['error']}") + if len(set(row["links"])) != len(row["links"]): + add(row, f"{where}: `{label}:` names an ID twice") + for other_id in row["links"]: + if ids is not None and twin_id not in ids and other_id not in ids: + continue + if other_id not in there: + add( + row, + f"{where}: `{label}:` names {other_id}, not in the other map", + ) + elif twin_id not in there[other_id]["links"]: + at = f"{there[other_id]['file']} line {there[other_id]['line']}" + add( + row, + f"{where}: `{label}:` names {other_id}, whose `{back}:`" + f" does not name it back ({at})", + ) + return problems + + +def twin_pairs(side: str, ids: set[str] | None = None) -> list[dict[str, str]]: + """The SDK/server pairs that touch `ids` of this tool's map (all when None).""" + sdk = read_twin_links("S") or {} + pairs = [] + for sdk_id, row in sdk.items(): + for server_id in row["links"]: + mine = sdk_id if side == "S" else server_id + if ids is None or mine in ids: + pairs.append({"sdk": sdk_id, "server": server_id}) + return pairs + + +def cmd_map_links(args: argparse.Namespace) -> dict[str, Any]: + ids = ( + {row["id"] for row in collect_ids(family_files(args.file))} + if args.file + else None + ) + if args.id: + ids = (ids or set()) | set(args.id) + pairs = twin_pairs("F", ids) + return {"ok": True, "count": len(pairs), "pairs": pairs} + + def cmd_map_check(args: argparse.Namespace) -> dict[str, Any]: files = family_files(args.file) routes = None if args.no_routes else route_templates() @@ -4138,6 +4271,9 @@ def cmd_map_check(args: argparse.Namespace) -> dict[str, Any]: f"duplicate ID {row['id']} (also in {seen[row['id']]})" ) seen[row["id"]] = row["file"] + checked = {row["id"] for row in collect_ids(files)} if args.file else None + for name, found in twin_link_problems("F", checked).items(): + problems.setdefault(name, []).extend(found) counts: dict[str, int] = {} for row in all_ids: counts[row["file"]] = counts.get(row["file"], 0) + 1 @@ -6515,6 +6651,16 @@ def build_parser() -> argparse.ArgumentParser: ) mi.set_defaults(func=cmd_map_ids) mi.add_argument("--file") + ml = mp_sub.add_parser( + "links", + help="SDK/agent-server twin pairs (`Same promise:` and `Server:` links)", + formatter_class=Formatter, + epilog=f"Examples:\n {CLI} map links\n {CLI} map links --file F05\n" + f" {CLI} map links --id F05.pause-idle", + ) + ml.set_defaults(func=cmd_map_links) + ml.add_argument("--file") + ml.add_argument("--id", action="append", help="a sub-feature ID (repeatable)") mr = mp_sub.add_parser( "routes", help="every HTTP and WebSocket route of the app", diff --git a/.agents/skills/verify-sdk/SKILL.md b/.agents/skills/verify-sdk/SKILL.md index 6a07c44be0..52b86d5a1b 100644 --- a/.agents/skills/verify-sdk/SKILL.md +++ b/.agents/skills/verify-sdk/SKILL.md @@ -7,7 +7,8 @@ description: > of what the OpenHands SDK does in-process (agent loop, LLMs, conversations, events, condensers, skills, plugins, MCP, hooks, sub-agents, critic, security, tools, workspaces, settings, observability, ACP), the control-sdk - driver its recipes use, and sdk-map (check, owners, baseline, run, report). + driver its recipes use, and sdk-map (check, links, owners, baseline, run, + report). triggers: - /verify-sdk - sdk feature map @@ -47,8 +48,9 @@ and report `blocked`. ## Doctor `sdk-map check` is the read-only health check: it validates the map's -structure and IDs, the index counts, the baseline line, and that every path -the map cites still exists. Run it first and whenever a recipe surprises you. +structure and IDs, the index counts, the baseline line, that every path +the map cites still exists, and that every `Server:` link to the agent-server +map names a bullet that links back (`sdk-map links` lists the pairs). Run it first and whenever a recipe surprises you. For the environment, `uv run python -c "import openhands.sdk, openhands.tools, openhands.workspace"` must succeed. diff --git a/.agents/skills/verify-sdk/references/daily.md b/.agents/skills/verify-sdk/references/daily.md index 53799e3e64..1f527ff68f 100644 --- a/.agents/skills/verify-sdk/references/daily.md +++ b/.agents/skills/verify-sdk/references/daily.md @@ -59,6 +59,14 @@ the weekly pass. and its note updated, and every open `bug` issue opened since `BASE` that names a mapped feature is reproduced and gets its note. Model-backed blocks run only inside the budget; without a key they are `blocked`. +3. **Twins.** For every family step 2 replayed, `sdk-map links --file Snn` + lists its agent-server twins (`Server:` links). Drive each through the + server too (`control-agent-server map run --file Fnn --only Fnn.x --fresh + --record`) unless its family is driven anyway. Put each pair side by side: + a pair whose results disagree is a finding, usually at the seam between the + SDK and the server (serialization, the async path, persistence, restart), + unless the difference is already a known bug on one side. Drive each + bullet once. ## Reading the result diff --git a/.agents/skills/verify-sdk/references/feature-map/README.md b/.agents/skills/verify-sdk/references/feature-map/README.md index 0dd0b1a8c7..08c3fb163e 100644 --- a/.agents/skills/verify-sdk/references/feature-map/README.md +++ b/.agents/skills/verify-sdk/references/feature-map/README.md @@ -30,6 +30,10 @@ Every sub-feature bullet ends with three links: b01-b05 behavior, c01-c05 condenser, a01-a08 API compliance), which run only on labels or dispatch. +A bullet whose promise the agent server also keeps ends with a fourth link, +`Server:`, naming its twins in the agent-server map (see +[Twin links](#twin-links-with-the-agent-server-map)). + So a row with a `Unit:` link is checked on every PR, and a row with only an `Example:` or `Real-LLM:` link is checked only when someone adds a label. The map's own recipes are a third, independent proof: programs that use the SDK @@ -71,6 +75,33 @@ gone, or a `control-sdk` call does not parse or names a missing program. Bullets that need Docker, Apptainer, a cloud account, a Kubernetes cluster or an interactive login are labeled **blocked**. +## Twin links with the agent-server map + +Many SDK promises are also kept by the agent server: a server conversation +runs the same `LocalConversation`, and the SDK's `RemoteWorkspace` and +`RemoteConversation` drive the server from a Python program. When an +agent-server bullet proves the same promise through the server (a route, a +socket, or the SDK's remote client in F33), the two bullets are twins and +link each other at the end of their lines: + +- here: `` Server: `F05.pause-idle`, `F33.conversation-pause`. `` +- there: `` Same promise: `S04.pause`. `` + +A twin is the same promise seen from another surface, not a related feature: +link only when a pass that drives both bullets would expect the same answer. +One SDK bullet may have several twins and one server bullet may serve several +SDK bullets. A bullet that runs inside the server but that no agent-server +bullet drives has no link; a bullet that only exists in-process (local +workspaces, visualizers, logging, LLM internals) has none either. + +`sdk-map check` and `control-agent-server map check` both validate the links +in both directions: every named ID exists in the other map and names this +bullet back, and the link ends the line in exactly that form. `sdk-map links +[--file Snn] [--id Snn.x]` lists the pairs; a daily pass uses it to drive the +twins of the bullets a change touched (see [daily.md](../daily.md)). When a +pass adds, splits, renames or removes a linked bullet, it updates the twin's +link in the same commit. + ## Families | ID | Family | What it covers | Entry points | Needs | Sub-features | Untested | diff --git a/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md b/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md index 666688be93..051dfae03a 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md +++ b/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md @@ -10,23 +10,23 @@ Examples: `examples/01_standalone_sdk/01_hello_world.py`, `examples/01_standalon ## Sub-features -- `S01.run-to-finish`: `send_message` + `run()` steps until the agent finishes; status moves IDLE to RUNNING to FINISHED, a second `run()` on a finished conversation returns at once, and a new message moves FINISHED back to IDLE. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py`. Example: `examples/01_standalone_sdk/01_hello_world.py`. Real-LLM: `tests/integration/tests/t01_fix_simple_typo.py`, `tests/integration/tests/t02_add_bash_hello.py`. +- `S01.run-to-finish`: `send_message` + `run()` steps until the agent finishes; status moves IDLE to RUNNING to FINISHED, a second `run()` on a finished conversation returns at once, and a new message moves FINISHED back to IDLE. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py`. Example: `examples/01_standalone_sdk/01_hello_world.py`. Real-LLM: `tests/integration/tests/t01_fix_simple_typo.py`, `tests/integration/tests/t02_add_bash_hello.py`. Server: `F05.run-queued`, `F05.run-finished-noop`, `F05.run-new-input`, `F06.send-resets-finished`. - `S01.response-dispatch`: each LLM reply is classified as tool calls, content, reasoning-only or empty; tool calls execute, content sets FINISHED, and the batch's `ActionEvent`s are emitted consecutively. Unit: `tests/sdk/agent/test_response_dispatch.py`. Example: none. Real-LLM: none. - `S01.empty-reply-nudge`: a reasoning-only reply (reasoning or thinking, no text, no tool call) injects a corrective user nudge and the loop continues instead of finishing; the nudge does not reset the stuck-detection window. A content-free reply (no text, no tool call, no reasoning) is not an agent turn: the LLM call retries it (`S02.no-response-retry`), so no empty agent message or nudge is recorded and the run goes on with the next reply; when every attempt is content-free the run ends `ERROR` with one retryable `ConversationErrorEvent` (code `LLMNoResponseError`), not `STUCK`. Unit: `tests/sdk/agent/test_reasoning_only_responses.py`, `tests/sdk/agent/test_tool_call_recovery.py`, `tests/sdk/conversation/local/test_empty_response_retry.py`, `tests/sdk/agent/test_response_dispatch.py::test_corrective_nudge_does_not_reset_stuck_detection_window`. Example: none. Real-LLM: none. -- `S01.max-iterations`: after `max_iteration_per_run` steps (default 500) the run stops with status ERROR and a `ConversationErrorEvent(code="MaxIterationsReached")`, unless the last step finished, which keeps FINISHED. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_error_on_max_iterations`, `::test_execution_status_finished_on_final_iteration`. Example: `examples/01_standalone_sdk/28_ask_agent_example.py` (`max_iteration_per_run=5`). Real-LLM: none. +- `S01.max-iterations`: after `max_iteration_per_run` steps (default 500) the run stops with status ERROR and a `ConversationErrorEvent(code="MaxIterationsReached")`, unless the last step finished, which keeps FINISHED. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_error_on_max_iterations`, `::test_execution_status_finished_on_final_iteration`. Example: `examples/01_standalone_sdk/28_ask_agent_example.py` (`max_iteration_per_run=5`). Real-LLM: none. Server: `F05.max-iterations-stop`, `F05.max-iterations-resume`. - `S01.max-budget`: with `max_budget_per_run` set, the run stops with ERROR and `MaxBudgetReached` once the combined cost of all the conversation's LLMs reaches it; a finished run stays FINISHED. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_error_on_max_budget`, `::test_finished_preserved_even_when_over_budget`. Example: none. Real-LLM: none. -- `S01.stuck-detect`: with `stuck_detection=True` (default) the detector flags repeated action/observation pairs (4), repeated action/error pairs (3), agent monologue (3) and alternating patterns (6) in the events since the last user message, and sets status STUCK. Unit: `tests/cross/test_stuck_detector.py`. Example: `examples/01_standalone_sdk/20_stuck_detector.py`. Real-LLM: none. +- `S01.stuck-detect`: with `stuck_detection=True` (default) the detector flags repeated action/observation pairs (4), repeated action/error pairs (3), agent monologue (3) and alternating patterns (6) in the events since the last user message, and sets status STUCK. Unit: `tests/cross/test_stuck_detector.py`. Example: `examples/01_standalone_sdk/20_stuck_detector.py`. Real-LLM: none. Server: `F05.stuck-detected`, `F33.conversation-stuck`. - `S01.stuck-thresholds`: `stuck_detection_thresholds` (a `StuckDetectionThresholds` or a dict) overrides the four thresholds. Unit: `tests/cross/test_stuck_detector_config.py`. Example: none. Real-LLM: none. - `S01.stuck-nudge`: a repeating action-error streak first gets one environment nudge message; the conversation goes STUCK only if the streak continues, and recovers if the model self-corrects. Unit: `tests/sdk/conversation/local/test_stuck_detector_nudge.py`. Example: none. Real-LLM: none. -- `S01.stuck-recover`: `run()` on a STUCK conversation and `send_message` after STUCK both resume the loop. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_run_recovers_from_stuck`, `::test_send_message_resets_stuck_to_idle`. Example: none. Real-LLM: none. +- `S01.stuck-recover`: `run()` on a STUCK conversation and `send_message` after STUCK both resume the loop. Unit: `tests/sdk/conversation/local/test_agent_status_transition.py::test_run_recovers_from_stuck`, `::test_send_message_resets_stuck_to_idle`. Example: none. Real-LLM: none. Server: `F05.stuck-rerun`, `F05.stuck-new-input`. - `S01.parallel-tools`: `Agent(tool_concurrency_limit=N)` (default 1, must be >= 1) runs up to N tool calls of one step concurrently, is faster than sequential, keeps results in call order, and keeps the other results when one call errors. Unit: `tests/sdk/agent/test_parallel_execution_integration.py`, `tests/sdk/agent/test_parallel_executor.py`. Example: `examples/01_standalone_sdk/45_parallel_tool_execution.py`. Real-LLM: none. - `S01.parallel-resource-locks`: concurrent tools that declare the same resource keys serialize, tools that declare no keys share a per-tool mutex, and an empty key list skips locking. Unit: `tests/sdk/agent/test_parallel_executor_locking.py`. Example: none. Real-LLM: none. - `S01.finish-truncates-batch`: tool calls after a `finish` call in the same batch are dropped, and a call blocked by a hook produces a rejection in place. Unit: `tests/sdk/agent/test_action_batch.py`, `tests/sdk/agent/test_parallel_execution_integration.py::test_finish_tool_truncates_subsequent_tools`. Example: none. Real-LLM: none. -- `S01.message-while-running`: `send_message` from another thread while `run()` or `arun()` is mid-step is absorbed by the next step instead of being lost or ending the run; a message during a confirmation wait supersedes the pending action. Unit: `tests/sdk/agent/test_message_during_streaming_arun.py`, `tests/sdk/agent/test_message_while_finishing.py`. Example: `examples/01_standalone_sdk/18_send_message_while_processing.py`. Real-LLM: none. +- `S01.message-while-running`: `send_message` from another thread while `run()` or `arun()` is mid-step is absorbed by the next step instead of being lost or ending the run; a message during a confirmation wait supersedes the pending action. Unit: `tests/sdk/agent/test_message_during_streaming_arun.py`, `tests/sdk/agent/test_message_while_finishing.py`. Example: `examples/01_standalone_sdk/18_send_message_while_processing.py`. Real-LLM: none. Server: `F06.send-while-running`. - `S01.async-step`: `arun()`/`astep()` follow the same control flow and release the state lock while the LLM call is in flight, so other threads can read state and send messages. Unit: `tests/sdk/agent/test_astep_releases_state_lock.py`. Example: none. Real-LLM: none. -- `S01.image-input`: a user `Message` with `ImageContent(image_urls=[...])` (http URL or base64 `data:` URL) reaches a vision model; with more than 20 images to an Anthropic model, base64 images are downscaled to 2000x2000 px. Unit: `tests/sdk/llm/test_vision_support.py`, `tests/sdk/llm/test_llm_image_resizing.py`. Example: `examples/01_standalone_sdk/17_image_input.py`. Real-LLM: none (`tests/integration/tests/t08_image_file_viewing.py` covers images read by a tool, not user input). +- `S01.image-input`: a user `Message` with `ImageContent(image_urls=[...])` (http URL or base64 `data:` URL) reaches a vision model; with more than 20 images to an Anthropic model, base64 images are downscaled to 2000x2000 px. Unit: `tests/sdk/llm/test_vision_support.py`, `tests/sdk/llm/test_llm_image_resizing.py`. Example: `examples/01_standalone_sdk/17_image_input.py`. Real-LLM: none (`tests/integration/tests/t08_image_file_viewing.py` covers images read by a tool, not user input). Server: `F06.send-image`. - `S01.image-non-vision`: an image in the latest user message to a non-vision model produces an agent message saying the model cannot see images and the run finishes without an LLM call; if the vision inspect tool (`inspect_image_with_vision`, added when a saved vision-capable LLM profile exists) is available, every image in the conversation, in user messages and tool observations alike, is replaced by a reference numbered in conversation order that the agent can inspect with that tool; without the tool, images only in older messages or tool observations do not stop the run. Unit: `tests/sdk/agent/test_non_multimodal_image_input.py`. Example: none. Real-LLM: none. -- `S01.ask-agent`: `conversation.ask_agent(question)` returns a direct LLM answer built from the current history, adds no events and leaves status unchanged, works while `run()` is in flight, drops unanswered tool calls from the context, and raises on context-window errors. Unit: `tests/sdk/conversation/test_ask_agent.py`. Example: `examples/01_standalone_sdk/28_ask_agent_example.py`. Real-LLM: none. +- `S01.ask-agent`: `conversation.ask_agent(question)` returns a direct LLM answer built from the current history, adds no events and leaves status unchanged, works while `run()` is in flight, drops unanswered tool calls from the context, and raises on context-window errors. Unit: `tests/sdk/conversation/test_ask_agent.py`. Example: `examples/01_standalone_sdk/28_ask_agent_example.py`. Real-LLM: none. Server: `F10.ask-agent`, `F10.ask-agent-running`. - `S01.structured-output`: `Tool(name=..., params={"response_schema": Model})` extends that tool's action schema with the model's fields, validates the LLM's arguments against it, and `tool.parse_last_response(events)` returns the latest typed instance. Invalid schemas (object-level constraints, dynamic fields, field collisions, reserved names, toolsets) are rejected. Unit: `tests/sdk/tool/test_response_schema.py`. Example: `examples/01_standalone_sdk/56_structured_output.py`. Real-LLM: none. - `S01.malformed-tool-call`: malformed tool arguments are repaired when possible (control characters, stringified JSON); a native tool call that still fails validation becomes an `AgentErrorEvent` naming the validation error (a malformed prompted-mode `<function=...>` reply becomes a user `MessageEvent` instead) and the loop continues. Unit: `tests/sdk/agent/test_fix_malformed_tool_arguments.py`, `tests/sdk/agent/test_tool_validation_error_message.py`. Example: none. Real-LLM: none. - `S01.unknown-tool`: a call to a tool the agent does not have yields an `AgentErrorEvent` the model sees, not a crash. Unit: `tests/sdk/agent/test_nonexistent_tool_handling.py`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md b/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md index 93818d44e5..8146a2637d 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md +++ b/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md @@ -12,7 +12,7 @@ Examples: `examples/01_standalone_sdk/13_get_llm_metrics.py`, `examples/01_stand - `S02.completion`: `llm.completion(messages, tools)` calls LiteLLM and returns an `LLMResponse` with the converted `Message`, tool calls and a metrics snapshot; custom params (temperature, top_p, max_output_tokens, seed, extra headers, `litellm_extra_body`) are forwarded. Unit: `tests/sdk/llm/test_llm_completion.py`, `tests/sdk/llm/test_llm_litellm_extra_body.py`. Example: `examples/01_standalone_sdk/13_get_llm_metrics.py`. Real-LLM: `tests/integration/tests/t01_fix_simple_typo.py`. - `S02.api-mode`: `generate()` / `agenerate()` dispatch to Chat Completions or Responses by `api_mode` (`auto`, `chat`, `responses`); `uses_responses_api()` reports the choice. Unit: `tests/sdk/llm/test_llm.py::test_generate_dispatches_to_configured_api`, `::test_agenerate_dispatches_to_configured_api`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py`. Real-LLM: none. -- `S02.streaming`: with `stream=True` and an `on_token` callback, content, reasoning and tool-call chunks arrive as `ModelResponseStream` deltas and the final message (tool calls and cache-read tokens included) is reassembled; sync and async generators are accepted. Unit: `tests/sdk/llm/test_llm_completion.py::test_llm_completion_streaming_with_callback`, `::test_llm_completion_streaming_with_tools`, `::test_llm_streaming_preserves_cache_read_tokens`, `tests/sdk/event/test_streaming.py`, `tests/sdk/agent/test_stream_context.py`. Example: `examples/01_standalone_sdk/29_llm_streaming.py`. Real-LLM: none. +- `S02.streaming`: with `stream=True` and an `on_token` callback, content, reasoning and tool-call chunks arrive as `ModelResponseStream` deltas and the final message (tool calls and cache-read tokens included) is reassembled; sync and async generators are accepted. Unit: `tests/sdk/llm/test_llm_completion.py::test_llm_completion_streaming_with_callback`, `::test_llm_completion_streaming_with_tools`, `::test_llm_streaming_preserves_cache_read_tokens`, `tests/sdk/event/test_streaming.py`, `tests/sdk/agent/test_stream_context.py`. Example: `examples/01_standalone_sdk/29_llm_streaming.py`. Real-LLM: none. Server: `F06.ws-streaming-deltas`, `F33.conversation-streaming`, `F33.conversation-token-callbacks`. - `S02.stream-no-callback`: `stream=True` without a callback degrades to a normal, non-streaming result instead of failing. Unit: `tests/sdk/llm/test_llm_completion.py::test_llm_streaming_without_callback_degrades`. Example: none. Real-LLM: none. - `S02.timeouts`: `timeout` (default 300 s, `None` disables) is passed to LiteLLM and enforced per attempt; `stream_idle_timeout` (default 300 s) aborts and retries a stalled async stream, and an exhausted stream timeout puts the conversation in ERROR. Timeouts stop being retried once the call has used twice `timeout` of wall-clock time, so attempts that each run into the full timeout end after two, whatever `num_retries` allows, while timeouts far shorter than `timeout` (a stream idle timeout, a connect timeout) keep every attempt. The final `LLMTimeoutError` of a call that ran out of attempts says that the model did not return a complete response within the LLM timeout, with the limit, the number of attempts and the total seconds, and points to a faster model, a shorter prompt or a higher timeout in the LLM settings. Unit: `tests/sdk/llm/test_llm_timeout.py` (`test_hard_timeouts_stop_retrying_after_the_timeout_budget`, `test_quick_timeouts_keep_every_retry`, `test_non_timeout_errors_are_reraised_unchanged` for the budget and the summary). Example: none. Real-LLM: none. - `S02.responses-reasoning`: on the Responses path `reasoning_effort`, `reasoning_summary` and `enable_encrypted_reasoning` (default on) are sent only to reasoning models; output items, reasoning summaries and encrypted reasoning are parsed into the `Message` and replayed on the next turn; streamed Responses events are reconstructed with or without a callback. Unit: `tests/sdk/llm/test_responses_parsing_and_kwargs.py`, `tests/sdk/llm/test_responses_serialization.py`, `tests/sdk/llm/test_llm.py::test_enable_encrypted_reasoning_respects_flag_and_defaults_true`. Example: `examples/01_standalone_sdk/23_responses_reasoning.py`. Real-LLM: none. @@ -24,11 +24,11 @@ Examples: `examples/01_standalone_sdk/13_get_llm_metrics.py`, `examples/01_stand - `S02.native-tool-calling`: `native_tool_calling` defaults to true and tools are sent as provider tool definitions; setting it false switches to the prompted path. Unit: `tests/sdk/llm/test_llm.py::test_llm_function_calling_enabled_by_default`, `::test_llm_function_calling_can_be_disabled`, `tests/sdk/llm/test_llm_completion.py::test_llm_completion_with_tools`. Example: none. Real-LLM: none. - `S02.prompted-tool-calling`: on the prompted path tools are described in the system prompt with in-context examples, the model's text is parsed back into `MessageToolCall`s, and conversion errors raise `FunctionCallConversionError` / `FunctionCallValidationError`. Unit: `tests/sdk/llm/test_llm_fncall_converter.py`, `tests/sdk/llm/test_llm_completion.py::test_llm_completion_non_function_call_mode`, `::test_llm_completion_function_call_vs_non_function_call_mode`. Example: none. Real-LLM: none. - `S02.metrics`: `llm.metrics` accumulates `accumulated_cost`, per-call `costs`, `token_usages` (prompt, completion, cache read, cache write, reasoning, context window), `response_latencies` and a cache hit rate; metrics merge, diff, deep-copy, reset and restore. Unit: `tests/sdk/llm/test_llm_metrics.py`, `tests/sdk/llm/test_llm_completion.py::test_llm_cost_tracking`, `::test_llm_token_usage_tracking`, `::test_llm_latency_tracking`, `tests/sdk/llm/test_llm.py::test_llm_reset_metrics`. Example: `examples/01_standalone_sdk/13_get_llm_metrics.py`. Real-LLM: none. -- `S02.conversation-stats`: `conversation.conversation_stats` keeps one `Metrics` per `usage_id` (agent, condenser, any LLM added to `conversation.llm_registry`) and `get_combined_metrics()` sums them; restored metrics keep tracking after a resume. Unit: `tests/sdk/conversation/test_conversation_stats.py`, `tests/sdk/llm/test_llm.py::test_conversation_stats_restore_then_track`. Example: `examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py`. Real-LLM: none. +- `S02.conversation-stats`: `conversation.conversation_stats` keeps one `Metrics` per `usage_id` (agent, condenser, any LLM added to `conversation.llm_registry`) and `get_combined_metrics()` sums them; restored metrics keep tracking after a resume. Unit: `tests/sdk/conversation/test_conversation_stats.py`, `tests/sdk/llm/test_llm.py::test_conversation_stats_restore_then_track`. Example: `examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py`. Real-LLM: none. Server: `F33.conversation-stats`. - `S02.pricing-override`: `input_cost_per_token` / `output_cost_per_token` override LiteLLM pricing for cost calculation. Unit: `tests/sdk/llm/test_llm_pricing_passthrough.py`, `tests/sdk/llm/test_llm_span_cost.py`. Example: none. Real-LLM: none. - `S02.retries`: rate-limit, connection and other transient errors are retried up to `num_retries` (default 5) with backoff from `retry_min_wait`/`retry_max_wait`/`retry_multiplier`; non-retryable errors raise at once; a retry listener sees each attempt. A LiteLLM `APIError` for a 2xx response whose body could not be parsed as JSON counts as transient and is retried; the same unparseable body with a 4xx status is not. Repeated timeouts are bounded separately (`S02.timeouts`), and each failed attempt is logged (`S20.llm-retry-log`). Unit: `tests/sdk/llm/test_api_connection_error_retry.py`, `tests/sdk/llm/test_llm.py::test_llm_retry_on_rate_limit`, `tests/sdk/llm/test_llm_retry_telemetry.py`, `tests/sdk/llm/test_auth_refresh_retry.py`, `tests/sdk/llm/test_retry_diagnostics.py::test_malformed_provider_response_uses_existing_retry_budget`, `::test_unrelated_api_errors_are_not_transient`, `::test_malformed_provider_response_recovers_on_async_retry`. Example: none. Real-LLM: none. - `S02.no-response-retry`: an empty provider response is retried with a temperature bump and finally raises `LLMNoResponseError`, on both chat and Responses paths, sync and async. On the chat path that covers a reply with no choices and a reply whose only choice has no text, no tool calls and no reasoning, streamed or not; a choice with reasoning or thinking only is returned as it is, and so is a content-free choice whose `finish_reason` is `length` (the output-token cap; a retry would give the same answer). Unit: `tests/sdk/llm/test_llm_no_response_retry.py`. Example: none. Real-LLM: none. -- `S02.error-mapping`: LiteLLM and HTTP errors map to `LLMContextWindowExceedError`, `LLMMalformedConversationHistoryError`, `LLMAuthenticationError`, `LLMRateLimitError`, `LLMTimeoutError`, `LLMServiceUnavailableError`, `LLMBadRequestError`, `LLMContentPolicyViolationError` and budget denials. Unit: `tests/sdk/llm/test_exception_mapping.py`, `tests/sdk/llm/test_exception_classifier.py`, `tests/sdk/llm/test_exception.py`, `tests/sdk/llm/test_budget_denial_http.py`. Example: none. Real-LLM: `tests/integration/tests/a01_unmatched_tool_use.py` (provider responses to malformed histories, run through `tests/integration/api_compliance/run_compliance.py`). +- `S02.error-mapping`: LiteLLM and HTTP errors map to `LLMContextWindowExceedError`, `LLMMalformedConversationHistoryError`, `LLMAuthenticationError`, `LLMRateLimitError`, `LLMTimeoutError`, `LLMServiceUnavailableError`, `LLMBadRequestError`, `LLMContentPolicyViolationError` and budget denials. Unit: `tests/sdk/llm/test_exception_mapping.py`, `tests/sdk/llm/test_exception_classifier.py`, `tests/sdk/llm/test_exception.py`, `tests/sdk/llm/test_budget_denial_http.py`. Example: none. Real-LLM: `tests/integration/tests/a01_unmatched_tool_use.py` (provider responses to malformed histories, run through `tests/integration/api_compliance/run_compliance.py`). Server: `F20.validate-bad-key-type`, `F20.validate-provider-errors`, `F20.validate-quota-exhausted`. - `S02.prompt-caching`: with `caching_prompt=True` (default) cache markers or keys are added for supporting models, and a "prompt too small to cache" rejection is retried without caching. A normal turn and `ask_agent` share the tool schemas and conversation prefix on Anthropic. Unit: `tests/sdk/llm/test_prompt_caching_cross_conversation.py`, `tests/sdk/llm/test_llm_completion.py::test_completion_retries_without_caching_on_prompt_cache_too_small`, `tests/sdk/conversation/test_ask_agent.py::test_ask_agent_reuses_normal_tool_schemas`. Example: none. Real-LLM: none. - `S02.token-count`: `llm.get_token_count(messages)` counts tokens with LiteLLM or a chat-template tokenizer (`custom_tokenizer`), including tool definitions, with fallbacks when the tokenizer is unavailable. Unit: `tests/sdk/llm/test_llm.py::test_llm_token_counting`, `::test_llm_token_counting_includes_tools`. Example: none. Real-LLM: none. - `S02.context-window-guard`: a model whose context window is below the SDK minimum raises `LLMContextWindowTooSmallError` unless `ALLOW_SHORT_CONTEXT_WINDOWS` is set. Without an explicit `max_output_tokens` the output budget a request asks for (`effective_max_output_tokens`) is the model's advertised output limit capped at 16384, and 16384 when none is advertised; an explicit `max_output_tokens` is used as given. Unit: `tests/sdk/llm/test_llm.py::test_llm_raises_error_on_small_context_window`, `::test_llm_respects_allow_short_context_windows_env_var`, `::test_advertised_output_limit_is_not_a_request_default`, `::test_explicit_max_output_tokens_not_overridden`, `::test_default_manual_thinking_budget_is_bounded`. Example: none. Real-LLM: none. @@ -36,7 +36,7 @@ Examples: `examples/01_standalone_sdk/13_get_llm_metrics.py`, `examples/01_stand - `S02.image-inline`: with `inline_image_urls` (auto for providers that reject URLs, such as Moonshot Kimi) http image URLs are fetched and sent as base64 `data:` URLs, bounded by `OH_INLINE_IMAGE_MAX_MB`, `OH_INLINE_IMAGE_FETCH_TIMEOUT_S`, `OH_INLINE_IMAGE_CACHE_BYTES`. Unit: `tests/sdk/llm/test_image_inline.py`. Example: none. Real-LLM: none. - `S02.image-request-guard`: `completion`, `acompletion`, `responses` and `aresponses` raise `LLMBadRequestError` before any HTTP request when the latest user message carries an image and `vision_is_active()` is false (a model without vision, or `disable_vision=True`); `get_token_count` and `format_messages_for_llm` still work on those messages; an image only in an older message does not block the call and is left out of the request; `capability_overrides={"supports_vision": True}` sends the image. Unit: `tests/sdk/llm/test_capability_wire.py::test_disabled_vision_rejects_request_but_not_token_count`, `::test_image_reaches_http_boundary`, `tests/sdk/agent/test_non_multimodal_image_input.py::test_llm_image_guard_scoped_to_latest_user_message`. Example: none. Real-LLM: none. - `S02.image-adapter-loss`: on the native `deepseek/` route, when LiteLLM's adapter drops an image from the request body the call is refused as `LLMBadRequestError` before anything is sent and is not retried; an adapter that delivers every image sends the request as usual. Unit: `tests/sdk/llm/test_capability_wire.py::test_native_deepseek_adapter_must_deliver_or_reject_images`, `tests/sdk/llm/test_image_delivery.py::test_image_loss_surfaces_as_bad_request_through_litellm_wrapping`, `tests/sdk/llm/test_image_delivery.py::test_provider_failure_with_images_delivered_is_left_alone`. Example: none. Real-LLM: none. -- `S02.log-completions`: `log_completions_to_file=True` writes each request and response as JSON under `log_completions_folder`. `log_completions` is a deprecated alias for the same flag (deprecated in v1.63.0, removal scheduled for v1.68.0): either name sets both, passing the two with different values raises `ValueError`, and a dump carries both keys. Unit: `tests/sdk/llm/test_llm_log_completions_integration.py`, `tests/sdk/llm/test_completion_logging_aliases.py`. Example: none. Real-LLM: none. +- `S02.log-completions`: `log_completions_to_file=True` writes each request and response as JSON under `log_completions_folder`. `log_completions` is a deprecated alias for the same flag (deprecated in v1.63.0, removal scheduled for v1.68.0): either name sets both, passing the two with different values raises `ValueError`, and a dump carries both keys. Unit: `tests/sdk/llm/test_llm_log_completions_integration.py`, `tests/sdk/llm/test_completion_logging_aliases.py`. Example: none. Real-LLM: none. Server: `F33.completion-logs`. - `S02.serialization`: an `LLM` dumps with `api_key` and other secrets masked, round-trips through JSON, `model_copy` keeps configuration while refreshing provider info, and `LLM.load_from_env(prefix="LLM_")` builds an `LLM` from `LLM_*` variables, casting types. Unit: `tests/sdk/llm/test_llm_serialization.py`, `tests/sdk/llm/test_llm.py::test_llm_model_copy_preserves_configuration`, `tests/sdk/test_settings.py::test_llm_load_from_env_rehydrates_subscription_runtime` (the only `load_from_env` test; subscription case). Example: none. Real-LLM: none. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md b/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md index acbe2f0d99..6db6be49b9 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md +++ b/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md @@ -19,26 +19,26 @@ Examples: `examples/01_standalone_sdk/05_use_llm_registry.py`, `examples/01_stan - `S03.multimodal-router-text`: text-only turns go to `secondary`, and turns whose token count exceeds the secondary's `effective_max_input_tokens` go to `primary`. Unit: **none (untested)**. Example: `examples/01_standalone_sdk/19_llm_routing.py` (text turns before and after the image turn). Real-LLM: none. - `S03.router-validation`: `MultimodalRouter` without a `primary` or `secondary` key and any `RouterLLM` with empty `llms_for_routing` fail validation. Unit: **none (untested)**. Example: none. Real-LLM: none. - `S03.random-router`: `RandomRouter` picks a random entry of `llms_for_routing` per call. Unit: **none (untested)**. Example: none. Real-LLM: none. -- `S03.model-router-selection`: on a conversation whose agent LLM is a `RouterLLM`, `router.status()` reports `enabled`, the `selection` (`auto`, no route) and every route's `name`, `model` and `label`; an `auto` turn goes to the first route and sets `last_route_name`; `conversation.switch_routed_llm(route)` pins a route (the agent keeps the router, `state.agent` follows, `base_state.json` stores the selection) and the next turn goes there; an unknown route raises `ValueError` and keeps the selection; `resume_auto_llm_routing()` returns to the first route; `ModelRouterSelection` refuses `manual` without a route and `auto` with one; a plain-LLM conversation raises `ValueError: Current agent LLM is not a RouterLLM.`; a resumed conversation runs on the router the program passes, not the stored selection. Unit: `tests/sdk/llm/test_router_llm.py`, `tests/sdk/conversation/test_switch_model.py::test_switch_routed_llm_updates_agent_and_state`, `::test_resume_auto_llm_routing_updates_agent_and_state`, `::test_switch_routed_llm_survives_state_roundtrip`, `::test_switch_routed_llm_rejects_non_router`. Example: none. Real-LLM: none. -- `S03.model-router-add-route`: `conversation.add_model_router_route(profile, route_name=None, base_route_name="default")` wraps a plain agent LLM in a `RouterLLM` (the old LLM becomes route `base_route_name`) or extends an existing router (keeping its class, e.g. `MultimodalRouter`), registers the profile's LLM as `profile:<name>`, returns the new `ModelRouterStatus` and stays in `auto` mode; once selected the route serves the turn and its cost is recorded under `profile:<name>`; a duplicate route name raises `ValueError`, a missing profile `FileNotFoundError`, and a later edit of the saved profile does not change the route. Unit: `tests/sdk/conversation/test_switch_model.py::test_add_model_router_route_wraps_plain_llm`, `::test_add_model_router_route_extends_existing_router`, `::test_add_model_router_route_rejects_duplicate_name`, `::test_add_model_router_route_rejects_missing_profile`, `::test_added_model_router_route_survives_profile_change`. Example: none. Real-LLM: none. -- `S03.model-router-switch-replaces`: `conversation.switch_profile(name)` and `switch_llm(llm)` on a router conversation install the plain LLM in place of the router (status `enabled: false`, route selection then raises `ValueError`), and the next turn runs on it; the router's own `switch_llm(route)` / `with_manual_route`, `resume_auto_routing()` and `with_route` return changed copies and leave the original alone, refusing an unknown or duplicate route with `ValueError`. Unit: `tests/sdk/llm/test_router_llm.py::test_router_manual_switch_returns_copy_and_routes_to_key`, `::test_router_with_route_rejects_duplicate_key`. Example: none. Real-LLM: none. -- `S03.model-router-before-first-run`: a route added with `add_model_router_route` and selected before the conversation's first message serves the first turn, and that turn's cost is recorded in `conversation_stats` under the route's `profile:<name>`. Unit: **none (untested)**. Example: none. Real-LLM: none. +- `S03.model-router-selection`: on a conversation whose agent LLM is a `RouterLLM`, `router.status()` reports `enabled`, the `selection` (`auto`, no route) and every route's `name`, `model` and `label`; an `auto` turn goes to the first route and sets `last_route_name`; `conversation.switch_routed_llm(route)` pins a route (the agent keeps the router, `state.agent` follows, `base_state.json` stores the selection) and the next turn goes there; an unknown route raises `ValueError` and keeps the selection; `resume_auto_llm_routing()` returns to the first route; `ModelRouterSelection` refuses `manual` without a route and `auto` with one; a plain-LLM conversation raises `ValueError: Current agent LLM is not a RouterLLM.`; a resumed conversation runs on the router the program passes, not the stored selection. Unit: `tests/sdk/llm/test_router_llm.py`, `tests/sdk/conversation/test_switch_model.py::test_switch_routed_llm_updates_agent_and_state`, `::test_resume_auto_llm_routing_updates_agent_and_state`, `::test_switch_routed_llm_survives_state_roundtrip`, `::test_switch_routed_llm_rejects_non_router`. Example: none. Real-LLM: none. Server: `F09.model-router-select`, `F09.model-router-status`. +- `S03.model-router-add-route`: `conversation.add_model_router_route(profile, route_name=None, base_route_name="default")` wraps a plain agent LLM in a `RouterLLM` (the old LLM becomes route `base_route_name`) or extends an existing router (keeping its class, e.g. `MultimodalRouter`), registers the profile's LLM as `profile:<name>`, returns the new `ModelRouterStatus` and stays in `auto` mode; once selected the route serves the turn and its cost is recorded under `profile:<name>`; a duplicate route name raises `ValueError`, a missing profile `FileNotFoundError`, and a later edit of the saved profile does not change the route. Unit: `tests/sdk/conversation/test_switch_model.py::test_add_model_router_route_wraps_plain_llm`, `::test_add_model_router_route_extends_existing_router`, `::test_add_model_router_route_rejects_duplicate_name`, `::test_add_model_router_route_rejects_missing_profile`, `::test_added_model_router_route_survives_profile_change`. Example: none. Real-LLM: none. Server: `F09.model-router-add-route`. +- `S03.model-router-switch-replaces`: `conversation.switch_profile(name)` and `switch_llm(llm)` on a router conversation install the plain LLM in place of the router (status `enabled: false`, route selection then raises `ValueError`), and the next turn runs on it; the router's own `switch_llm(route)` / `with_manual_route`, `resume_auto_routing()` and `with_route` return changed copies and leave the original alone, refusing an unknown or duplicate route with `ValueError`. Unit: `tests/sdk/llm/test_router_llm.py::test_router_manual_switch_returns_copy_and_routes_to_key`, `::test_router_with_route_rejects_duplicate_key`. Example: none. Real-LLM: none. Server: `F09.model-router-switch-replaces`. +- `S03.model-router-before-first-run`: a route added with `add_model_router_route` and selected before the conversation's first message serves the first turn, and that turn's cost is recorded in `conversation_stats` under the route's `profile:<name>`. Unit: **none (untested)**. Example: none. Real-LLM: none. Server: `F09.model-router-before-first-run`. - `S03.fallback-transient`: after the primary exhausts its retries on a transient error (rate limit, timeout, connection, service unavailable, transient HTTP status) the fallback profiles are tried in order; the first success is returned; if all fail the primary's error is raised; each new call starts on the primary again. Chat and Responses, sync and async. Unit: `tests/sdk/llm/test_llm_fallback.py`. Example: `examples/01_standalone_sdk/39_llm_fallback.py`. Real-LLM: none. - `S03.fallback-skip`: non-transient errors (bad request, auth, context window) and budget denials do not fall back; quota exhaustion fails over without retrying the primary. Unit: `tests/sdk/llm/test_llm_fallback.py::test_non_transient_error_skips_fallback`, `::test_budget_denial_stops_retries_and_fallbacks`, `::test_quota_exhaustion_fails_over_without_retries`. Example: none. Real-LLM: none. - `S03.fallback-metrics`: cost and token usage of a fallback call are recorded in the caller's usage session (the primary's `llm.metrics`, and the primary's `usage_id` in `conversation_stats` when the call belongs to a conversation), and each cost entry's `model` names the model that served the call, so a fallback's entry carries the fallback model; caller kwargs and the conversation call context are forwarded to the fallback. Unit: `tests/sdk/llm/test_llm_fallback.py::test_metrics_merged_from_fallback`, `::test_fallback_forwards_caller_kwargs`, `::test_completion_fallback_receives_call_context`, `tests/sdk/llm/test_llm_fallback_sessions.py::test_fallback_preserves_caller_accounting`. Example: `examples/01_standalone_sdk/39_llm_fallback.py`. Real-LLM: none. - `S03.fallback-profile-resolution`: `fallback_llms` names are loaded from `LLMProfileStore(profile_store_dir)` lazily on first fallback. Unit: `tests/sdk/llm/test_llm_fallback.py::test_fallback_profiles_resolved_via_store`. Example: `examples/01_standalone_sdk/39_llm_fallback.py`. Real-LLM: none. -- `S03.profile-save-load`: `LLMProfileStore(base_dir).save(name, llm)` writes `<name>.json` atomically with a schema version, `load(name)` returns the `LLM`, `list()`, `list_summaries()`, `delete()`, `rename()` work; invalid names, missing and corrupted profiles raise with the available names listed. Unit: `tests/sdk/llm/test_llm_profile_store.py`. Example: `examples/01_standalone_sdk/37_llm_profile_store/main.py`. Real-LLM: none. -- `S03.profile-secrets`: `save` masks the API key unless `include_secrets=True`; with a `cipher` secrets are encrypted at rest and decrypted on `load`; `list_summaries` reports whether a key is set without exposing it. Unit: `tests/sdk/llm/test_llm_profile_store.py::test_save_without_secrets`, `::test_save_with_secrets`, `tests/sdk/conversation/test_switch_model.py::test_switch_profile_decrypts_with_cipher`. Example: `examples/01_standalone_sdk/37_llm_profile_store/main.py`. Real-LLM: none. -- `S03.profile-concurrency-limits`: concurrent saves and reads are serialized by a file lock, and `max_profiles` blocks new names over the limit but allows overwrites. Unit: `tests/sdk/llm/test_llm_profile_store.py::test_concurrent_saves`, `::test_concurrent_reads_and_writes`, `::test_save_with_max_profiles_blocks_over_limit`. Example: none. Real-LLM: none. +- `S03.profile-save-load`: `LLMProfileStore(base_dir).save(name, llm)` writes `<name>.json` atomically with a schema version, `load(name)` returns the `LLM`, `list()`, `list_summaries()`, `delete()`, `rename()` work; invalid names, missing and corrupted profiles raise with the available names listed. Unit: `tests/sdk/llm/test_llm_profile_store.py`. Example: `examples/01_standalone_sdk/37_llm_profile_store/main.py`. Real-LLM: none. Server: `F20.save`, `F20.rename`, `F20.delete`, `F20.name-rules`, `F20.corrupt-files`. +- `S03.profile-secrets`: `save` masks the API key unless `include_secrets=True`; with a `cipher` secrets are encrypted at rest and decrypted on `load`; `list_summaries` reports whether a key is set without exposing it. Unit: `tests/sdk/llm/test_llm_profile_store.py::test_save_without_secrets`, `::test_save_with_secrets`, `tests/sdk/conversation/test_switch_model.py::test_switch_profile_decrypts_with_cipher`. Example: `examples/01_standalone_sdk/37_llm_profile_store/main.py`. Real-LLM: none. Server: `F20.key-encrypted-at-rest`, `F20.get-secrets`, `F20.save-without-secrets`. +- `S03.profile-concurrency-limits`: concurrent saves and reads are serialized by a file lock, and `max_profiles` blocks new names over the limit but allows overwrites. Unit: `tests/sdk/llm/test_llm_profile_store.py::test_concurrent_saves`, `::test_concurrent_reads_and_writes`, `::test_save_with_max_profiles_blocks_over_limit`. Example: none. Real-LLM: none. Server: `F20.store-busy`, `F20.profile-limit`. - `S03.profile-migration`: legacy OpenHands-proxy profiles are migrated on load, third-party LiteLLM-proxy profiles are kept, and a profile with a newer schema version is rejected. Unit: `tests/sdk/llm/test_llm_profile_store.py::test_load_migrates_legacy_openhands_proxy_profile`, `::test_load_rejects_newer_profile_schema_version`. Example: none. Real-LLM: none. -- `S03.provider-connections`: a profile can reference a saved provider connection (`provider_connection_id`) whose credentials are resolved at load time from a sibling `provider-connections` store. Unit: `tests/sdk/llm/test_provider_connection_store.py`, `tests/sdk/llm/test_llm_profile_store.py::test_default_provider_store_is_sibling_of_base_dir`. Example: none. Real-LLM: none. -- `S03.switch-profile`: `conversation.switch_profile(name)` loads the profile (cached in the registry as `profile:<name>`), swaps the agent's LLM, updates persisted state, keeps the prompt cache key, and raises for a missing profile; later messages use the new model and both models' costs appear in the stats. Unit: `tests/sdk/conversation/test_switch_model.py`. Example: `examples/01_standalone_sdk/44_model_switching_in_convo.py`. Real-LLM: none. -- `S03.switch-llm-object`: `conversation.switch_llm(llm)` swaps to an `LLM` object without touching the profile store, installs the posted object even when its `usage_id` is already registered (#5779), refreshes an LLM condenser that shared the agent's credentials while leaving an independently configured one alone, and `ask_agent` follows the switch. Unit: `tests/sdk/conversation/test_switch_model.py::test_switch_llm_swaps_when_store_empty`, `::test_switch_llm_refreshes_llm_condenser_credentials`, `::test_switch_llm_preserves_independent_condenser_profile`, `tests/sdk/conversation/test_ask_agent.py::test_ask_agent_llm_refreshes_after_switch_llm`. Example: none. Real-LLM: none. +- `S03.provider-connections`: a profile can reference a saved provider connection (`provider_connection_id`) whose credentials are resolved at load time from a sibling `provider-connections` store. Unit: `tests/sdk/llm/test_provider_connection_store.py`, `tests/sdk/llm/test_llm_profile_store.py::test_default_provider_store_is_sibling_of_base_dir`. Example: none. Real-LLM: none. Server: `F23.linked-run`, `F20.activate-linked-provider`, `F20.sdk-get-llm-linked`. +- `S03.switch-profile`: `conversation.switch_profile(name)` loads the profile (cached in the registry as `profile:<name>`), swaps the agent's LLM, updates persisted state, keeps the prompt cache key, and raises for a missing profile; later messages use the new model and both models' costs appear in the stats. Unit: `tests/sdk/conversation/test_switch_model.py`. Example: `examples/01_standalone_sdk/44_model_switching_in_convo.py`. Real-LLM: none. Server: `F09.switch-profile`, `F09.switch-profile-errors`. +- `S03.switch-llm-object`: `conversation.switch_llm(llm)` swaps to an `LLM` object without touching the profile store, installs the posted object even when its `usage_id` is already registered (#5779), refreshes an LLM condenser that shared the agent's credentials while leaving an independently configured one alone, and `ask_agent` follows the switch. Unit: `tests/sdk/conversation/test_switch_model.py::test_switch_llm_swaps_when_store_empty`, `::test_switch_llm_refreshes_llm_condenser_credentials`, `::test_switch_llm_preserves_independent_condenser_profile`, `tests/sdk/conversation/test_ask_agent.py::test_ask_agent_llm_refreshes_after_switch_llm`. Example: none. Real-LLM: none. Server: `F09.switch-llm`, `F09.switch-llm-registered-usage-id`. - `S03.switch-keeps-usage`: usage stays with the role (`usage_id`) in `conversation_stats` across `switch_llm`: the costs recorded before the switch remain, and what the condenser and `ask_agent` spend on the new model is added under their own `usage_id`s (`condenser`, `ask-agent-llm`), each cost entry naming the model that served it. Unit: `tests/sdk/conversation/local/test_llm_session_isolation.py::test_condenser_usage_is_counted_after_switch_llm`, `::test_ask_agent_usage_is_counted_after_every_switch`, `::test_stats_callback_keeps_firing_after_switch_llm`. Example: none. Real-LLM: none. - `S03.switch-llm-tool`: with `SwitchLLMTool` among the agent's tools (`include_default_tools=["FinishTool", "SwitchLLMTool"]`, or `OpenHandsAgentSettings.create_agent()`, which adds it when `tools` is left unset), the model calls `switch_llm(profile_name=...)`; the tool description lists saved profiles, the conversation switches, and a missing profile or load error comes back as an error observation. The tool is exempt from `filter_tools_regex` and does not deadlock under `arun()`. Unit: `tests/sdk/tool/test_switch_llm.py`, `tests/sdk/conversation/test_switch_model.py::test_switch_llm_tool_during_arun_does_not_deadlock`. Example: `examples/01_standalone_sdk/49_switch_llm_tool.py`. Real-LLM: none. -- `S03.route-task-to-model`: with `OpenHandsAgentSettings(enable_classify_and_switch_llm_tool=True, meta_profile=..., active_meta_profile=...)` the agent gets `route_task_to_model`; calling it sends recent conversation text to the classifier profile, which returns a class index (structured `classes`) or a model name (`prompt_template` with `{{ instance_text }}` and `{{ model_table }}`), and the conversation switches to that profile. Classifier cost lands in the conversation stats under one usage bucket; unknown classes, missing classifier or target profiles produce error observations. Unit: `tests/sdk/tool/test_classify_and_switch_llm.py`. Example: `examples/01_standalone_sdk/59_route_task_to_model.py`. Real-LLM: none. -- `S03.meta-profile-store`: `MetaProfileStore` saves, lists, loads and deletes meta-profiles under `OH_PERSISTENCE_DIR` (or `~/.openhands`), validates direct-prompt templates (requires `{{ instance_text }}`, no `classes`), and enforces `max_profiles`. Unit: `tests/sdk/llm/test_meta_profile_store.py`. Example: none. Real-LLM: none. -- `S03.subscription-login`: `LLM.subscription_login(vendor="openai", model=..., auth_method="browser"|"device_code", force_login=False, open_browser=True)` runs an OAuth PKCE browser flow or a device-code flow, caches credentials under `~/.openhands/auth/`, refreshes expired tokens, asks for consent once, and returns an `LLM` pointed at the Codex Responses endpoint; models outside the supported set are rejected. Unit: `tests/sdk/llm/auth/test_openai.py`, `tests/sdk/llm/auth/test_credentials.py`. Example: `examples/01_standalone_sdk/35_subscription_login.py`. Real-LLM: none. +- `S03.route-task-to-model`: with `OpenHandsAgentSettings(enable_classify_and_switch_llm_tool=True, meta_profile=..., active_meta_profile=...)` the agent gets `route_task_to_model`; calling it sends recent conversation text to the classifier profile, which returns a class index (structured `classes`) or a model name (`prompt_template` with `{{ instance_text }}` and `{{ model_table }}`), and the conversation switches to that profile. Classifier cost lands in the conversation stats under one usage bucket; unknown classes, missing classifier or target profiles produce error observations. Unit: `tests/sdk/tool/test_classify_and_switch_llm.py`. Example: `examples/01_standalone_sdk/59_route_task_to_model.py`. Real-LLM: none. Server: `F22.route-classes`, `F22.route-direct`. +- `S03.meta-profile-store`: `MetaProfileStore` saves, lists, loads and deletes meta-profiles under `OH_PERSISTENCE_DIR` (or `~/.openhands`), validates direct-prompt templates (requires `{{ instance_text }}`, no `classes`), and enforces `max_profiles`. Unit: `tests/sdk/llm/test_meta_profile_store.py`. Example: none. Real-LLM: none. Server: `F22.save-classes`, `F22.save-direct`, `F22.validation`, `F22.delete`, `F22.limit`. +- `S03.subscription-login`: `LLM.subscription_login(vendor="openai", model=..., auth_method="browser"|"device_code", force_login=False, open_browser=True)` runs an OAuth PKCE browser flow or a device-code flow, caches credentials under `~/.openhands/auth/`, refreshes expired tokens, asks for consent once, and returns an `LLM` pointed at the Codex Responses endpoint; models outside the supported set are rejected. Unit: `tests/sdk/llm/auth/test_openai.py`, `tests/sdk/llm/auth/test_credentials.py`. Example: `examples/01_standalone_sdk/35_subscription_login.py`. Real-LLM: none. Server: `F24.device-start`, `F24.login-complete`. - `S03.subscription-requests`: a subscription `LLM` omits parameters the Codex endpoint rejects or mishandles (`prompt_cache_retention`, `include`, `reasoning`, `temperature`, `max_output_tokens`), adds no temperature on retry, keeps `is_subscription` across serialization, and rehydrates from `LLM.load_from_env`. Unit: `tests/sdk/llm/test_subscription_mode.py`, `tests/sdk/test_settings.py::test_llm_load_from_env_rehydrates_subscription_runtime`. Example: none. Real-LLM: none. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md b/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md index 54982621d5..aff7f0a9a3 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md +++ b/.agents/skills/verify-sdk/references/feature-map/S04-local-conversation.md @@ -10,38 +10,38 @@ Examples: `examples/01_standalone_sdk/06_interactive_terminal_w_reasoning.py`, ` ## Sub-features -- `S04.factory`: `Conversation(...)` returns a `LocalConversation` for a path or `LocalWorkspace` and a `RemoteConversation` for a `RemoteWorkspace`; `persistence_dir` is only accepted on the local branch. Unit: `tests/sdk/conversation/test_conversation_factory.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: every `tests/integration/tests/t*.py` builds a `LocalConversation` through `tests/integration/base.py`. -- `S04.conversation-id`: each conversation gets a unique UUID, a passed `conversation_id` is kept and persisted, and the id pins the LLM prompt-cache key. Unit: `tests/sdk/conversation/local/test_conversation_id.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. -- `S04.persist-events`: with `persistence_dir`, every event is written as `events/event-NNNNN-<id>.json` and base state as `base_state.json` under `<persistence_dir>/<id.hex>/`, incrementally; on load, an empty, invalid or missing event file makes the reload fail with an error naming the file or event. Unit: `tests/sdk/conversation/local/test_state_serialization.py`, `tests/sdk/conversation/local/test_conversation_core.py`, `tests/cross/test_persistence_dir_call_sites.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. -- `S04.resume`: a new `Conversation` with the same `persistence_dir` and `conversation_id` reloads the events and state and continues the dialogue with full memory. Unit: `tests/sdk/conversation/local/test_state_serialization.py::test_conversation_state_persistence_save_load`, `tests/cross/test_conversation_restore_behavior.py::test_conversation_restore_lifecycle_happy_path`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. -- `S04.ask-agent-usage-saved`: with a `persistence_dir`, the usage of a `conversation.ask_agent(...)` call (`conversation_stats.usage_to_metrics["ask-agent-llm"]`) is written to `base_state.json` when the call returns, so a conversation reloaded from that directory counts it. Unit: **none (untested)**. Example: none. Real-LLM: none. +- `S04.factory`: `Conversation(...)` returns a `LocalConversation` for a path or `LocalWorkspace` and a `RemoteConversation` for a `RemoteWorkspace`; `persistence_dir` is only accepted on the local branch. Unit: `tests/sdk/conversation/test_conversation_factory.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: every `tests/integration/tests/t*.py` builds a `LocalConversation` through `tests/integration/base.py`. Server: `F33.conversation-create`. +- `S04.conversation-id`: each conversation gets a unique UUID, a passed `conversation_id` is kept and persisted, and the id pins the LLM prompt-cache key. Unit: `tests/sdk/conversation/local/test_conversation_id.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. Server: `F33.conversation-create-request`. +- `S04.persist-events`: with `persistence_dir`, every event is written as `events/event-NNNNN-<id>.json` and base state as `base_state.json` under `<persistence_dir>/<id.hex>/`, incrementally; on load, an empty, invalid or missing event file makes the reload fail with an error naming the file or event. Unit: `tests/sdk/conversation/local/test_state_serialization.py`, `tests/sdk/conversation/local/test_conversation_core.py`, `tests/cross/test_persistence_dir_call_sites.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. Server: `F07.seq-on-disk`, `F04.restart-persist`. +- `S04.resume`: a new `Conversation` with the same `persistence_dir` and `conversation_id` reloads the events and state and continues the dialogue with full memory. Unit: `tests/sdk/conversation/local/test_state_serialization.py::test_conversation_state_persistence_save_load`, `tests/cross/test_conversation_restore_behavior.py::test_conversation_restore_lifecycle_happy_path`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. Server: `F04.restart-persist`, `F06.restart-persist`, `F33.conversation-attach`, `F33.conversation-restart`. +- `S04.ask-agent-usage-saved`: with a `persistence_dir`, the usage of a `conversation.ask_agent(...)` call (`conversation_stats.usage_to_metrics["ask-agent-llm"]`) is written to `base_state.json` when the call returns, so a conversation reloaded from that directory counts it. Unit: **none (untested)**. Example: none. Real-LLM: none. Server: `F10.ask-agent-usage-saved`. - `S04.resume-agent-compat`: on resume, removing tools, removing default tools or changing the agent class fails; adding tools and changing the LLM, condenser or skills succeeds; security risk and summaries survive. Unit: `tests/cross/test_conversation_restore_behavior.py`. Example: none. Real-LLM: none. - `S04.custom-file-store`: an injected `FileStore` takes precedence over `persistence_dir` for where state is written, while `persistence_dir` is still reported. Unit: `tests/sdk/conversation/local/test_custom_file_store.py`. Example: none. Real-LLM: none. -- `S04.callbacks`: `callbacks=[...]` receive every event after the default callback has persisted it; a persistence failure blocks downstream callbacks; the visualizer runs in a fixed order relative to user callbacks. Unit: `tests/sdk/conversation/local/test_conversation_default_callback.py`, `tests/sdk/conversation/local/test_conversation_visualize_param.py::test_conversation_callback_order`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. -- `S04.run-error`: an exception inside `run()` surfaces as `ConversationRunError` that names the conversation id, the persistence dir and the issue URL, and emits one `ConversationErrorEvent` (no duplicate when the agent already emitted one). Unit: `tests/sdk/conversation/local/test_run_exception_includes_conversation_id.py`. Example: none. Real-LLM: none. -- `S04.close`: `close()` closes tool executors (terminal sessions), is idempotent, unregisters its atexit hook, and still closes the executors with `delete_on_close=False` (persisted files stay either way); `__del__` also closes. Unit: `tests/sdk/conversation/test_atexit_cleanup.py`, `tests/tools/terminal/test_conversation_cleanup.py`, `tests/sdk/conversation/local/test_conversation_core.py::test_close_propagates_agent_failure_and_allows_retry`. Example: none. Real-LLM: none. -- `S04.pause`: `pause()` from another thread moves RUNNING or IDLE to PAUSED, emits exactly one `PauseEvent`, takes effect between steps (not mid-LLM-call), and a later `run()` resumes. Unit: `tests/sdk/conversation/local/test_conversation_pause_functionality.py`, `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_transitions_to_running_from_paused`. Example: `examples/01_standalone_sdk/09_pause_example.py`. Real-LLM: `tests/integration/tests/c04_token_condenser.py` and `c05_size_condenser.py` call `pause()`. +- `S04.callbacks`: `callbacks=[...]` receive every event after the default callback has persisted it; a persistence failure blocks downstream callbacks; the visualizer runs in a fixed order relative to user callbacks. Unit: `tests/sdk/conversation/local/test_conversation_default_callback.py`, `tests/sdk/conversation/local/test_conversation_visualize_param.py::test_conversation_callback_order`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. Server: `F33.conversation-callbacks`. +- `S04.run-error`: an exception inside `run()` surfaces as `ConversationRunError` that names the conversation id, the persistence dir and the issue URL, and emits one `ConversationErrorEvent` (no duplicate when the agent already emitted one). Unit: `tests/sdk/conversation/local/test_run_exception_includes_conversation_id.py`. Example: none. Real-LLM: none. Server: `F05.run-error`, `F33.conversation-run-error`. +- `S04.close`: `close()` closes tool executors (terminal sessions), is idempotent, unregisters its atexit hook, and still closes the executors with `delete_on_close=False` (persisted files stay either way); `__del__` also closes. Unit: `tests/sdk/conversation/test_atexit_cleanup.py`, `tests/tools/terminal/test_conversation_cleanup.py`, `tests/sdk/conversation/local/test_conversation_core.py::test_close_propagates_agent_failure_and_allows_retry`. Example: none. Real-LLM: none. Server: `F33.conversation-close`. +- `S04.pause`: `pause()` from another thread moves RUNNING or IDLE to PAUSED, emits exactly one `PauseEvent`, takes effect between steps (not mid-LLM-call), and a later `run()` resumes. Unit: `tests/sdk/conversation/local/test_conversation_pause_functionality.py`, `tests/sdk/conversation/local/test_agent_status_transition.py::test_execution_status_transitions_to_running_from_paused`. Example: `examples/01_standalone_sdk/09_pause_example.py`. Real-LLM: `tests/integration/tests/c04_token_condenser.py` and `c05_size_condenser.py` call `pause()`. Server: `F05.pause-idle`, `F05.pause-repeat`, `F05.pause-running`, `F05.pause-tool-step`, `F05.run-resume-paused`, `F33.conversation-pause`. - `S04.pause-during-startup`: a `pause()` from another thread that arrives after `run()` was called but while `run()` is still making the agent ready (plugins, MCP servers; the first `run()` of a conversation restored from `persistence_dir`) keeps the status PAUSED, makes no LLM call, and the next `run()` resumes; a conversation already PAUSED when `run()` is called is resumed by it. Unit: `tests/sdk/conversation/local/test_conversation_pause_functionality.py` (a pause before `run()` only; the startup window is **untested**). Example: none. Real-LLM: none. - `S04.async-callback`: `AsyncCallbackWrapper(coro, loop)` lets a synchronous `run()` in an executor thread deliver events to a coroutine on the main loop. Unit: `tests/sdk/utils/test_async_utils.py`. Example: `examples/01_standalone_sdk/11_async.py`. Real-LLM: none. -- `S04.arun-interrupt`: `await conversation.arun()` uses the async LLM path; `interrupt()` (safe from a signal handler or another thread) cancels it at once, even mid-LLM-call, sets PAUSED, emits an `InterruptEvent`, does not raise `CancelledError`, and the conversation can be resumed; with no `arun()` in flight it falls back to `pause()`. Unit: `tests/sdk/conversation/test_interrupt.py`. Example: `examples/01_standalone_sdk/50_async_cancellation.py`. Real-LLM: none. +- `S04.arun-interrupt`: `await conversation.arun()` uses the async LLM path; `interrupt()` (safe from a signal handler or another thread) cancels it at once, even mid-LLM-call, sets PAUSED, emits an `InterruptEvent`, does not raise `CancelledError`, and the conversation can be resumed; with no `arun()` in flight it falls back to `pause()`. Unit: `tests/sdk/conversation/test_interrupt.py`. Example: `examples/01_standalone_sdk/50_async_cancellation.py`. Real-LLM: none. Server: `F05.interrupt-running`, `F05.interrupt-idle`, `F05.run-after-interrupt`, `F33.conversation-interrupt`. - `S04.arun-pause-persisted`: with a `persistence_dir`, a `pause()` made from another thread while `arun()` waits on the model returns at once and is in `base_state.json` (`execution_status: paused`) at once, before the model call returns; `arun()` then stops PAUSED after that call and a later `run()` finishes. Unit: `tests/sdk/agent/test_astep_releases_state_lock.py::test_pause_during_llm_call_is_persisted_at_once`. Example: none. Real-LLM: none. -- `S04.cancel-token`: each `run()`/`arun()` gets a fresh `CancellationToken` that `interrupt()` sets before cancelling, so pending parallel tool calls are skipped. Unit: `tests/sdk/conversation/test_interrupt.py::test_interrupt_sets_cancel_token`, `::test_execute_batch_skips_all_on_pre_cancelled_token`. Example: `examples/01_standalone_sdk/50_async_cancellation.py`. Real-LLM: none. -- `S04.secrets-update`: `update_secrets({...})` accepts plain strings and `SecretSource` objects (resolved on first use, which is the first command output the conversation masks, then cached), overwrites existing keys, and exports referenced keys as environment variables for terminal commands. Unit: `tests/sdk/conversation/test_secrets_manager.py`, `tests/sdk/conversation/test_secret_source.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. -- `S04.secrets-constructor`: `Conversation(secrets={...})` seeds the same registry at construction, for static and callable values. Unit: `tests/sdk/conversation/test_conversation_secrets_constructor.py`. Example: none. Real-LLM: none. -- `S04.secrets-masking`: secret values echoed by a command are replaced by `<secret-hidden>` in observations and never reach the persisted events. Unit: `tests/tools/terminal/test_secrets_masking.py`, `tests/cross/test_agent_secrets_integration.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. +- `S04.cancel-token`: each `run()`/`arun()` gets a fresh `CancellationToken` that `interrupt()` sets before cancelling, so pending parallel tool calls are skipped. Unit: `tests/sdk/conversation/test_interrupt.py::test_interrupt_sets_cancel_token`, `::test_execute_batch_skips_all_on_pre_cancelled_token`. Example: `examples/01_standalone_sdk/50_async_cancellation.py`. Real-LLM: none. Server: `F05.interrupt-tool-call`. +- `S04.secrets-update`: `update_secrets({...})` accepts plain strings and `SecretSource` objects (resolved on first use, which is the first command output the conversation masks, then cached), overwrites existing keys, and exports referenced keys as environment variables for terminal commands. Unit: `tests/sdk/conversation/test_secrets_manager.py`, `tests/sdk/conversation/test_secret_source.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-update`, `F08.secrets-override`, `F33.conversation-secrets`. +- `S04.secrets-constructor`: `Conversation(secrets={...})` seeds the same registry at construction, for static and callable values. Unit: `tests/sdk/conversation/test_conversation_secrets_constructor.py`. Example: none. Real-LLM: none. Server: `F04.create-secrets`. +- `S04.secrets-masking`: secret values echoed by a command are replaced by `<secret-hidden>` in observations and never reach the persisted events. Unit: `tests/tools/terminal/test_secrets_masking.py`, `tests/cross/test_agent_secrets_integration.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-terminal`. - `S04.visualizer-default`: by default a `DefaultConversationVisualizer` prints each event as a Rich panel; `visualizer=None` turns it off; a class or instance can be passed. Unit: `tests/sdk/conversation/local/test_conversation_visualize_param.py`. Example: `examples/01_standalone_sdk/10_persistence.py`. Real-LLM: none. - `S04.visualizer-custom`: a `ConversationVisualizerBase` subclass passed as `visualizer=` receives every event through `on_event`; each event type has its own `visualize` rendering. Unit: `tests/sdk/conversation/test_visualizer.py`. Example: `examples/01_standalone_sdk/26_custom_visualizer.py`. Real-LLM: none. -- `S04.fork`: `fork()` deep-copies the conversation under a new id (or `conversation_id=`), starts IDLE, leaves the source untouched, resets metrics unless `reset_metrics=False`, accepts `agent=` and `tags=` (`title=` is deprecated and ignored locally; only `RemoteConversation` stores it), shares the workspace, and persists so the fork survives a reload. Unit: `tests/sdk/conversation/local/test_fork.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py`. Real-LLM: none. +- `S04.fork`: `fork()` deep-copies the conversation under a new id (or `conversation_id=`), starts IDLE, leaves the source untouched, resets metrics unless `reset_metrics=False`, accepts `agent=` and `tags=` (`title=` is deprecated and ignored locally; only `RemoteConversation` stores it), shares the workspace, and persists so the fork survives a reload. Unit: `tests/sdk/conversation/local/test_fork.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py`. Real-LLM: none. Server: `F10.fork-whole`, `F10.fork-named`, `F10.fork-metrics`, `F10.fork-independent`, `F10.fork-run-memory`, `F33.conversation-fork`. - `S04.fork-keeps-secrets`: `fork()` of a `LocalConversation(..., cipher=...)` with a `persistence_dir` creates the fork with the source's cipher and copies the source's conversation secrets (`secrets=`, `update_secrets`), so the fork's `base_state.json` holds the LLM api key and every secret encrypted (not redacted); `LocalConversation(agent=None, conversation_id=fork_id, cipher=...)` reopens the fork with the real key and secrets, and it runs. Without a cipher the fork's key stays redacted on disk. Since [#5765](https://github.com/OpenHands/software-agent-sdk/pull/5765) ([#5465](https://github.com/OpenHands/software-agent-sdk/issues/5465)). Unit: `tests/sdk/conversation/local/test_fork.py::test_fork_keeps_source_cipher_for_persisted_secrets`, `::test_fork_without_cipher_still_redacts_secrets`, `tests/agent_server/test_conversation_service.py::TestConversationTreeForkAndNavigate::test_cipher_backed_fork_resumes_with_credentials`. Example: none. Real-LLM: none. -- `S04.fork-branch`: events carry parent ids forming a tree; `fork(from_event_id=...)` copies only that branch, and `navigate_to(event_id)` moves HEAD so the next event starts a sibling branch (`None` empties the active branch). Unit: `tests/sdk/conversation/local/test_conversation_tree.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py`. Real-LLM: none. -- `S04.goal-loop`: `run_goal(conversation, objective, judge_llm, max_iterations=N)` sends the objective, runs, asks the judge LLM for a verdict, re-prompts with the missing items, and returns a `GoalOutcome` with status `"complete"` or `"capped"`, the iteration count and the verdict. Unit: `tests/sdk/conversation/goal/`. Example: `examples/01_standalone_sdk/54_goal_completion_loop.py`. Real-LLM: none. -- `S04.goal-asked`: when a round ends on the agent's `ask_user` question, `run_goal` returns `GoalAsked(ask_id, question)` for it without calling the judge, and the objective it sent is a user message with `sender="goal"`, which does not answer the question; called again while the question is pending it returns the same `GoalAsked` at once, sending nothing and calling neither model; once the user has answered (`conversation.answer_ask(...)` plus the returned message) it runs and the judge can say complete. Unit: `tests/tools/ask_user/test_ask_user.py::test_a_goal_stops_on_a_question_without_judging`. Example: none. Real-LLM: none. -- `S04.goal-asked-branch`: only a question on the active branch stops a goal: after `navigate_to(...)` back to the user message before an unanswered question, `run_goal` with a new objective sends it (`sender="goal"`), runs the agent and asks the judge. Unit: **none (untested)**. Example: none. Real-LLM: none. +- `S04.fork-branch`: events carry parent ids forming a tree; `fork(from_event_id=...)` copies only that branch, and `navigate_to(event_id)` moves HEAD so the next event starts a sibling branch (`None` empties the active branch). Unit: `tests/sdk/conversation/local/test_conversation_tree.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py`. Real-LLM: none. Server: `F10.fork-from-event`, `F10.navigate-event`, `F10.navigate-branch`, `F10.navigate-empty`, `F33.conversation-fork-branch`. +- `S04.goal-loop`: `run_goal(conversation, objective, judge_llm, max_iterations=N)` sends the objective, runs, asks the judge LLM for a verdict, re-prompts with the missing items, and returns a `GoalOutcome` with status `"complete"` or `"capped"`, the iteration count and the verdict. Unit: `tests/sdk/conversation/goal/`. Example: `examples/01_standalone_sdk/54_goal_completion_loop.py`. Real-LLM: none. Server: `F11.goal-complete`, `F11.goal-followup`, `F11.goal-capped`. +- `S04.goal-asked`: when a round ends on the agent's `ask_user` question, `run_goal` returns `GoalAsked(ask_id, question)` for it without calling the judge, and the objective it sent is a user message with `sender="goal"`, which does not answer the question; called again while the question is pending it returns the same `GoalAsked` at once, sending nothing and calling neither model; once the user has answered (`conversation.answer_ask(...)` plus the returned message) it runs and the judge can say complete. Unit: `tests/tools/ask_user/test_ask_user.py::test_a_goal_stops_on_a_question_without_judging`. Example: none. Real-LLM: none. Server: `F40.goal-asked`. +- `S04.goal-asked-branch`: only a question on the active branch stops a goal: after `navigate_to(...)` back to the user message before an unanswered question, `run_goal` with a new objective sends it (`sender="goal"`), runs the agent and asks the judge. Unit: **none (untested)**. Example: none. Real-LLM: none. Server: `F40.goal-abandoned-branch`. - `S04.interactive-terminal`: with `Tool(name=TerminalTool.name, params={"no_change_timeout_seconds": 3})` the agent can start `python3`, feed it input and exit; an `LLMConvertibleEvent` callback collects the raw LLM messages. Unit: `tests/tools/terminal/test_terminal_session.py::test_python_interactive_input`, `::test_interactive_command`. Example: `examples/01_standalone_sdk/06_interactive_terminal_w_reasoning.py`. Real-LLM: `tests/integration/tests/t07_interactive_commands.py`. -- `S04.final-response`: `get_agent_final_response(conversation.state.events)` returns the text of whichever came last of the agent's `finish` call and its plain-text `MessageEvent`, else `""`. Unit: `tests/sdk/conversation/test_agent_final_response.py`. Example: none. Real-LLM: `tests/integration/tests/t08_image_file_viewing.py`. -- `S04.stats`: `conversation.conversation_stats` keeps `Metrics` per LLM `usage_id`, combines them with `get_combined_metrics()`, restores them on resume, and stats update events carry a snapshot rather than the full history. Unit: `tests/sdk/conversation/test_conversation_stats.py`, `tests/sdk/conversation/test_stats_update_event_snapshot.py`. Example: `examples/01_standalone_sdk/11_async.py` (prints cost). Real-LLM: none. -- `S04.title-and-tags`: `generate_title(llm=None, max_length=50)` asks an LLM (the agent's by default) for a title from the first user message, falls back to truncation on error, and strips reasoning text; `tags={...}` keys must be lowercase alphanumeric. Unit: `tests/sdk/conversation/test_generate_title.py`, `tests/sdk/conversation/test_tags.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py` (`title=`). Real-LLM: none. -- `S04.conversation-sender`: `conversation_sender(conversation_id)` (`openhands.sdk.conversation.types`) is `CONVERSATION_SENDER_PREFIX` (`"conversation:"`) plus the id's hex; `send_message(text, sender=...)` records it on the saved user `MessageEvent`, a plain `send_message` records no sender, and both survive a reload from `persistence_dir`. Unit: **none (untested)** in the SDK; `tests/agent_server/test_event_router.py` covers the server's use of it. Example: none. Real-LLM: none. +- `S04.final-response`: `get_agent_final_response(conversation.state.events)` returns the text of whichever came last of the agent's `finish` call and its plain-text `MessageEvent`, else `""`. Unit: `tests/sdk/conversation/test_agent_final_response.py`. Example: none. Real-LLM: `tests/integration/tests/t08_image_file_viewing.py`. Server: `F04.final-response`. +- `S04.stats`: `conversation.conversation_stats` keeps `Metrics` per LLM `usage_id`, combines them with `get_combined_metrics()`, restores them on resume, and stats update events carry a snapshot rather than the full history. Unit: `tests/sdk/conversation/test_conversation_stats.py`, `tests/sdk/conversation/test_stats_update_event_snapshot.py`. Example: `examples/01_standalone_sdk/11_async.py` (prints cost). Real-LLM: none. Server: `F33.conversation-stats`. +- `S04.title-and-tags`: `generate_title(llm=None, max_length=50)` asks an LLM (the agent's by default) for a title from the first user message, falls back to truncation on error, and strips reasoning text; `tags={...}` keys must be lowercase alphanumeric. Unit: `tests/sdk/conversation/test_generate_title.py`, `tests/sdk/conversation/test_tags.py`. Example: `examples/01_standalone_sdk/48_conversation_fork.py` (`title=`). Real-LLM: none. Server: `F33.conversation-ask-title`, `F04.title-llm-profile`, `F04.patch-tags`. +- `S04.conversation-sender`: `conversation_sender(conversation_id)` (`openhands.sdk.conversation.types`) is `CONVERSATION_SENDER_PREFIX` (`"conversation:"`) plus the id's hex; `send_message(text, sender=...)` records it on the saved user `MessageEvent`, a plain `send_message` records no sender, and both survive a reload from `persistence_dir`. Unit: **none (untested)** in the SDK; `tests/agent_server/test_event_router.py` covers the server's use of it. Example: none. Real-LLM: none. Server: `F06.send-sender-conversation`. - `S04.resolution-verdict`: `verdict_from_judgments(judgments)` (`openhands.sdk.conversation.resolution`) turns the seven yes/no judgments of `RESOLUTION_QUESTIONS` into a `ResolutionVerdict`: `owed` `nothing` (resolved) only when `handled` and `task_done` are both at least 0.85 and every hand-off stays below 0.2; otherwise the strongest hand-off at or above 0.5 (`answer`, `review`, `decide`, `check`), else `unfinished` when the work is not done, else `unclear`; `probability` is `min(handled, task_done, 1 - strongest hand-off)`, `needs_user` is true only for a question, review, choice or check (not a reported problem), and judgments are clamped to 0..1, a non-numeric one counting as 0. Unit: `tests/sdk/conversation/test_resolution.py`. Example: none. Real-LLM: none. - `S04.resolution-judge`: `resolution_judge(fallback_llm, LLMProfileStore(...), cipher=None)` loads the saved LLM profile `resolution_judge`, first saving a copy of the fallback LLM under that name (with its key, encrypted on disk when a cipher is given) when the store has none; an existing profile wins over a later fallback. The returned `LLMDecisionModel` runs on that profile with a 60 s timeout and no LLM-level retries (`usage_id` `resolution_judge`), and `await judge_resolution(judge, ResolutionState(...))` sends the turn and every question in one chat request and returns the verdict with the profile's model; a reply that is not JSON raises `ValueError` after one request. Unit: `tests/sdk/conversation/test_resolution.py`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S05-events.md b/.agents/skills/verify-sdk/references/feature-map/S05-events.md index ddd37a05c4..4cbf058ada 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S05-events.md +++ b/.agents/skills/verify-sdk/references/feature-map/S05-events.md @@ -16,16 +16,16 @@ Examples: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py` - `S05.events-to-messages`: `events_to_messages` merges `ActionEvent`s sharing an `llm_response_id` into one assistant message (keeping the first event's reasoning and Responses reasoning item), merges consecutive plain user messages (not ones with tool calls, a tool call id or a name), and merges a condensation summary into the adjacent user message. Unit: `tests/sdk/event/test_events_to_messages.py`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py`. Real-LLM: `tests/integration/tests/a01_unmatched_tool_use.py` through `a08_parallel_wrong_order.py` (API compliance of the produced history). - `S05.system-prompt-blocks`: a `SystemPromptEvent` with static and dynamic context becomes system content blocks in a fixed order, so the static block stays cacheable. Unit: `tests/sdk/event/test_dynamic_context_message_sequence.py`. Example: none. Real-LLM: none. - `S05.openai-format`: the `Message` list from saved events formats into Chat Completions messages (`llm.format_messages_for_llm`) or Responses `instructions` plus input items (`llm.format_messages_for_responses`). Unit: `tests/sdk/llm/test_responses_serialization.py`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py`. Real-LLM: none. -- `S05.unmatched-actions`: `ConversationState.get_unmatched_actions(events)` lists actions with no observation, rejection or matching `AgentErrorEvent`, ignores non-executable actions, and so prevents re-running a finished action after a crash. Unit: `tests/sdk/conversation/test_get_unmatched_actions.py`. Example: none. Real-LLM: none. +- `S05.unmatched-actions`: `ConversationState.get_unmatched_actions(events)` lists actions with no observation, rejection or matching `AgentErrorEvent`, ignores non-executable actions, and so prevents re-running a finished action after a crash. Unit: `tests/sdk/conversation/test_get_unmatched_actions.py`. Example: none. Real-LLM: none. Server: `F31.crash-tool-call`. - `S05.event-log-append`: `EventLog.append` assigns consecutive indexes, rejects duplicate ids, supports negative indexing and `get_index`/`get_id`, serializes concurrent appends from threads, and does not rescan the directory on every append. Unit: `tests/sdk/conversation/test_event_store.py`. Example: none. Real-LLM: none. -- `S05.event-log-files`: the file name is `event-NNNNN-<event_id>.json` (5-digit minimum, wider past 99999), writer and reader agree on it, and a cold reload of more than 100k events works. Unit: `tests/sdk/conversation/test_event_store.py::test_event_filename_writer_reader_agree`, `::test_event_log_cold_reload_past_100k_events`, `::test_event_log_large_index_formatting`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py` (reads `events/event-*.json`). Real-LLM: none. +- `S05.event-log-files`: the file name is `event-NNNNN-<event_id>.json` (5-digit minimum, wider past 99999), writer and reader agree on it, and a cold reload of more than 100k events works. Unit: `tests/sdk/conversation/test_event_store.py::test_event_filename_writer_reader_agree`, `::test_event_log_cold_reload_past_100k_events`, `::test_event_log_large_index_formatting`. Example: `examples/01_standalone_sdk/36_event_json_to_openai_messages.py` (reads `events/event-*.json`). Real-LLM: none. Server: `F07.seq-on-disk`. - `S05.event-log-recovery`: missing files, corrupted JSON, index gaps, stale indexes, sidecar files and file-store exceptions are tolerated or reported without losing the rest of the log; another writer's appends are picked up. Unit: `tests/sdk/conversation/test_event_store.py`, `tests/sdk/conversation/local/test_state_serialization.py::test_conversation_state_corrupted_event_handling`. Example: none. Real-LLM: none. - `S05.event-cache`: deserialized events are cached, so repeated iteration does not re-read and re-parse files. Unit: `tests/sdk/conversation/test_event_store.py::test_event_cache_eliminates_repeated_deserialization`, `::test_event_cache_survives_across_multiple_iterations`. Example: none. Real-LLM: none. -- `S05.event-tree`: events carry `parent_id`; `path_to_root` walks a branch, detects cycles, and `append` rejects a parent that does not exist. Unit: `tests/sdk/conversation/test_event_tree.py`. Example: none. Real-LLM: none. (Branch navigation from the conversation API is `S04.fork-branch`.) +- `S05.event-tree`: events carry `parent_id`; `path_to_root` walks a branch, detects cycles, and `append` rejects a parent that does not exist. Unit: `tests/sdk/conversation/test_event_tree.py`. Example: none. Real-LLM: none. (Branch navigation from the conversation API is `S04.fork-branch`.) Server: `F10.navigate-branch`. - `S05.visualize`: each event has a Rich `visualize` rendering: action summaries, truncated system-prompt tool parameters, without mutating the event. Unit: `tests/sdk/event/test_action_event_summary.py`, `tests/sdk/event/test_system_prompt_event_visualize.py`, `tests/sdk/conversation/test_visualizer.py`. Example: `examples/01_standalone_sdk/26_custom_visualizer.py`. Real-LLM: none. - `S05.error-classification`: `ConversationErrorEvent` and `AgentErrorEvent` carry a `FailureKind` classification derived from the error code before the detail text; a sensitive detail is classified without being serialized. An `LLMTimeoutError` whose detail is a hard timeout (`LLM hard timeout after ...`) or carries the exhausted-timeout summary (`did not return a complete response within the LLM timeout`, `S02.timeouts`) is `config`, not retryable, `user_action=settings`; a timeout that did not use the whole LLM timeout (`Connection timed out.`, `Request timed out.`) stays `transient`, retryable, `user_action=retry`, and so does an `LLMNoResponseError` from a run whose replies were all content-free. Unit: `tests/sdk/event/test_error_classification.py` (`test_exhausted_llm_timeout_points_to_settings_not_retry` and the timeout cases for the timeouts), `tests/sdk/llm/test_llm_timeout.py::test_hard_timeouts_stop_retrying_after_the_timeout_budget`, `tests/sdk/conversation/local/test_empty_response_retry.py::test_exhausted_empty_completions_end_as_retryable_error`. Example: none. Real-LLM: none. -- `S05.streaming-delta`: `StreamingDeltaEvent` holds a `content` and `reasoning_content` token delta, dumps with its `kind` and round-trips through JSON; it is transient (published by the agent server for WebSocket clients, never appended to the event log). Unit: `tests/sdk/event/test_streaming.py`. Example: none. Real-LLM: none. -- `S05.completion-log-event`: `LLMCompletionLogEvent` carries a completion log (filename, JSON data, model, usage id) as an event; the agent server emits it for remote conversations. Unit: `tests/sdk/event/test_llm_completion_log_event.py`. Example: none. Real-LLM: none. +- `S05.streaming-delta`: `StreamingDeltaEvent` holds a `content` and `reasoning_content` token delta, dumps with its `kind` and round-trips through JSON; it is transient (published by the agent server for WebSocket clients, never appended to the event log). Unit: `tests/sdk/event/test_streaming.py`. Example: none. Real-LLM: none. Server: `F06.ws-streaming-deltas`, `F32.events-no-deltas`, `F33.conversation-streaming-state`. +- `S05.completion-log-event`: `LLMCompletionLogEvent` carries a completion log (filename, JSON data, model, usage id) as an event; the agent server emits it for remote conversations. Unit: `tests/sdk/event/test_llm_completion_log_event.py`. Example: none. Real-LLM: none. Server: `F33.completion-logs`. - `S05.token-event`: with `litellm_extra_body={"return_token_ids": True}` (vLLM), the agent emits a `TokenEvent` with prompt and response token ids. Unit: **none (untested)**. Example: none. Real-LLM: none. - `S05.resume-transcript`: `render_resume_transcript(events)` renders user and assistant messages, tool uses with status and I/O, and ACP tool calls into one text transcript (for re-bootstrapping an ACP session), or `None` when nothing relevant exists. Unit: `tests/sdk/event/test_resume_transcript.py`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md b/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md index aff352b3d5..65b3a3efa6 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md +++ b/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md @@ -10,8 +10,8 @@ Examples: `examples/01_standalone_sdk/14_context_condenser.py`, `examples/01_sta ## Sub-features -- `S06.context-suffixes`: `AgentContext(system_message_suffix=..., user_message_suffix=...)` appends text to the system prompt and to user messages; empty values add nothing. Unit: `tests/sdk/context/test_agent_context.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: none. -- `S06.context-skills`: repo skills (no triggers) go into the system prompt and triggered skills into the user message whose text matches a trigger (case-insensitive), with already-activated skills skipped; `disabled_skills` drops named skills and duplicate names are rejected. Skill loading and discovery are family S07. Unit: `tests/sdk/context/test_agent_context.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. +- `S06.context-suffixes`: `AgentContext(system_message_suffix=..., user_message_suffix=...)` appends text to the system prompt and to user messages; empty values add nothing. Unit: `tests/sdk/context/test_agent_context.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: none. Server: `F30.chat-system-suffix`, `F21.launch-additions`. +- `S06.context-skills`: repo skills (no triggers) go into the system prompt and triggered skills into the user message whose text matches a trigger (case-insensitive), with already-activated skills skipped; `disabled_skills` drops named skills and duplicate names are rejected. Skill loading and discovery are family S07. Unit: `tests/sdk/context/test_agent_context.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. Server: `F25.conversation-activation`. - `S06.context-secrets`: secret names and descriptions (never values) from `AgentContext(secrets=...)` and the conversation's secret registry are listed in the system prompt; raw string secrets are redacted on serialization and round-trip through a cipher. Unit: `tests/sdk/context/test_agent_context.py::test_get_system_message_suffix_with_secrets_only`, `tests/sdk/context/test_agent_context_serialization.py`, `tests/sdk/context/prompts/test_prompt_snapshot.py::test_dynamic_context_with_secret_registry`. Example: none. Real-LLM: none. - `S06.context-datetime`: `current_datetime` (a `datetime` or a string; defaults to now) is rendered in the dynamic part of the system prompt. Unit: `tests/sdk/context/test_agent_context.py::test_get_system_message_suffix_with_datetime_only`, `::test_agent_context_default_datetime`. Example: none. Real-LLM: none. - `S06.system-prompt-registry`: the built-in prompt is assembled from registered sections (default and planning presets), its rendering for fixed inputs matches checked-in snapshots, and the static block has no dynamic content. Unit: `tests/sdk/context/prompts/test_prompt_snapshot.py`, `tests/sdk/context/prompts/test_registry.py`, `tests/sdk/context/prompts/test_default_registry.py`, `tests/sdk/context/prompts/test_planning_registry.py`, `tests/sdk/context/prompts/test_section.py`. Example: none. Real-LLM: none. @@ -29,7 +29,7 @@ Examples: `examples/01_standalone_sdk/14_context_condenser.py`, `examples/01_sta - `S06.soft-vs-hard`: a soft requirement (size) with no valid range to forget leaves the view unchanged and the run continues; a hard requirement (explicit request, token pressure) with no valid range triggers a hard context reset. Unit: `tests/sdk/context/condenser/test_llm_summarizing_condenser.py::test_condense_with_soft_requirement_and_no_condensation_available`, `::test_condense_with_hard_requirement_and_no_condensation_available`, `tests/sdk/context/condenser/test_rolling_condenser.py`. Example: none. Real-LLM: `tests/integration/tests/c03_delayed_condensation.py`. - `S06.hard-reset`: when summarizing fails (LLM error or invalid range) the condenser retries a hard context reset up to `hard_context_reset_max_retries` (5), shrinking event text by `hard_context_reset_context_scaling` (0.8) each time, and keeps the leading system prompt. Unit: `tests/sdk/context/condenser/test_llm_summarizing_condenser.py::test_llm_error_triggers_hard_context_reset`, `::test_hard_context_reset_preserves_leading_system_prompt`, `tests/sdk/context/condenser/test_rolling_condenser.py::test_hard_context_reset_condensation_is_returned`. Example: none. Real-LLM: `tests/integration/tests/c02_hard_context_reset.py`. - `S06.summary-llm-streaming`: the summarizing LLM is a separate `LLM` (own `usage_id` for cost tracking); streaming is switched off for summaries unless the provider requires it, and `extra_body` is not passed. Unit: `tests/sdk/context/condenser/test_llm_summarizing_condenser.py::test_summarization_disables_streaming_when_llm_streams`, `::test_get_condensation_does_not_pass_extra_body`. Example: `examples/01_standalone_sdk/14_context_condenser.py`. Real-LLM: none. -- `S06.manual-condense`: `conversation.condense()` adds a `CondensationRequest` and runs one step to process it; it raises `ValueError` with setup instructions when there is no condenser or the condenser does not handle requests. Unit: `tests/sdk/conversation/test_condense.py`. Example: none. Real-LLM: none. +- `S06.manual-condense`: `conversation.condense()` adds a `CondensationRequest` and runs one step to process it; it raises `ValueError` with setup instructions when there is no condenser or the condenser does not handle requests. Unit: `tests/sdk/conversation/test_condense.py`. Example: none. Real-LLM: none. Server: `F10.condense-llm`, `F10.condense-no-condenser`, `F10.condense-llm-failure`. - `S06.context-overflow-request`: a context-window or malformed-history error from the LLM becomes a `CondensationRequest` when the condenser handles requests, and is raised (with a warning) otherwise. Unit: `tests/sdk/agent/test_agent_context_window_condensation.py`. Example: none. Real-LLM: `tests/integration/tests/c02_hard_context_reset.py`. (Also listed as `S01.context-overflow`.) - `S06.condenser-resume`: the state's cached `View` applies condensations incrementally and matches a full rebuild, and the resume path rebuilds it from the reloaded `Condensation` events. Unit: `tests/sdk/conversation/test_state_view_cache.py`. Example: `examples/01_standalone_sdk/14_context_condenser.py`. Real-LLM: none. - `S06.noop-condenser`: `NoOpCondenser` returns the view unchanged, with or without an agent LLM. Unit: `tests/sdk/context/condenser/test_no_op_condenser.py`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S07-skills.md b/.agents/skills/verify-sdk/references/feature-map/S07-skills.md index ffa1b0884f..ed350884e8 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S07-skills.md +++ b/.agents/skills/verify-sdk/references/feature-map/S07-skills.md @@ -11,7 +11,7 @@ Examples: `examples/01_standalone_sdk/03_activate_skill.py`, `examples/05_skills ## Sub-features - `S07.repo-skill-always-on`: a skill with `trigger=None` in legacy format (programmatic, plain `.md`, or third-party file) is rendered into the system prompt's repo context on every turn. Unit: `tests/sdk/context/test_agent_context.py` (`test_get_system_message_suffix_with_repo_skills`, `test_mixed_skill_types`). Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: none. -- `S07.keyword-trigger`: a `KeywordTrigger` skill is injected into the user message (`skill_knowledge_info.j2`) when a keyword matches as a whole word, case-insensitively; the first matching keyword in list order is reported. Unit: `tests/sdk/context/test_agent_context.py` (`test_get_user_message_suffix_with_single_trigger`, `test_case_insensitive_trigger_matching`), `tests/sdk/skills/test_skill_utils.py::test_match_trigger_whole_word`. Example: `examples/01_standalone_sdk/03_activate_skill.py` (`flarglebargle`). Real-LLM: none. +- `S07.keyword-trigger`: a `KeywordTrigger` skill is injected into the user message (`skill_knowledge_info.j2`) when a keyword matches as a whole word, case-insensitively; the first matching keyword in list order is reported. Unit: `tests/sdk/context/test_agent_context.py` (`test_get_user_message_suffix_with_single_trigger`, `test_case_insensitive_trigger_matching`), `tests/sdk/skills/test_skill_utils.py::test_match_trigger_whole_word`. Example: `examples/01_standalone_sdk/03_activate_skill.py` (`flarglebargle`). Real-LLM: none. Server: `F25.conversation-activation`. - `S07.trigger-once`: a triggered skill is recorded in `state.activated_knowledge_skills` and is not injected again later in the same conversation; a message with no match stays plain. Unit: `tests/sdk/conversation/local/test_conversation_send_message.py` (`test_send_message_activates_skill_on_trigger_match`, `test_send_message_skips_already_activated_skill`, `test_send_message_without_trigger_match_leaves_event_plain`). Example: none. Real-LLM: none. - `S07.task-trigger`: a `TaskTrigger` skill with `inputs` gets a prompt appended asking the user for missing `${variables}`. Unit: `tests/sdk/skills/test_task_skill.py`, `tests/sdk/skills/test_skill_info.py`. Example: none. Real-LLM: none. - `S07.message-suffixes`: `system_message_suffix` is appended to the system prompt and `user_message_suffix` to every user message, also when no skill triggers. Unit: `tests/sdk/context/test_agent_context.py` (`test_get_system_message_suffix_with_custom_suffix`, `test_get_user_message_suffix_empty_query_with_suffix`). Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: none. @@ -19,21 +19,21 @@ Examples: `examples/01_standalone_sdk/03_activate_skill.py`, `examples/05_skills - `S07.resources`: `discover_skill_resources(dir)` and `Skill.resources` list files under `scripts/`, `references/`, `assets/` with the skill root. Unit: `tests/sdk/skills/test_resource_directories.py`. Example: `examples/05_skills_and_plugins/01_loading_agentskills/main.py`. Real-LLM: none. - `S07.available-skills-prompt`: AgentSkills skills (and triggered legacy skills) are listed in `<available_skills>` with name and description (XML-escaped, descriptions over 1024 chars truncated with a pointer to `invoke_skill`); the skill body is not in the prompt. Unit: `tests/sdk/skills/test_validation_prompt.py`, `tests/sdk/skills/test_validation_improvements.py`, `tests/sdk/context/test_agent_context.py::TestAgentContext::test_agentskills_format_progressive_disclosure`, `tests/sdk/context/prompts/test_default_registry.py::test_available_skills_section`. Example: `examples/05_skills_and_plugins/01_loading_agentskills/main.py`. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. - `S07.agentskills-trigger`: an AgentSkills skill with `triggers:` in its frontmatter is both listed and auto-injected on a keyword match. Unit: `tests/sdk/context/test_agent_context.py::TestAgentContext::test_agentskills_format_progressive_disclosure`. Example: `examples/05_skills_and_plugins/01_loading_agentskills/main.py` (`encrypt`). Real-LLM: none. -- `S07.invoke-skill-autoattach`: the `invoke_skill` tool is attached to the agent iff some loaded AgentSkills skill does not set `disable_model_invocation`, whatever `include_default_tools` says. Unit: `tests/sdk/tool/test_invoke_skill.py::test_agent_auto_attaches_invoke_skill_tool`. Example: `examples/05_skills_and_plugins/01_loading_agentskills/main.py`. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. -- `S07.invoke-skill-run`: `invoke_skill(name)` trims the name, renders the skill body (including inline commands), appends a footer with the skill's path (relative inside the workspace, absolute outside, none for programmatic skills), records the name once in `state.invoked_skills`, and returns an error observation without state change for an unknown name. Unit: `tests/sdk/tool/test_invoke_skill.py`. Example: none. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. +- `S07.invoke-skill-autoattach`: the `invoke_skill` tool is attached to the agent iff some loaded AgentSkills skill does not set `disable_model_invocation`, whatever `include_default_tools` says. Unit: `tests/sdk/tool/test_invoke_skill.py::test_agent_auto_attaches_invoke_skill_tool`. Example: `examples/05_skills_and_plugins/01_loading_agentskills/main.py`. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. Server: `F25.invoke-skill`. +- `S07.invoke-skill-run`: `invoke_skill(name)` trims the name, renders the skill body (including inline commands), appends a footer with the skill's path (relative inside the workspace, absolute outside, none for programmatic skills), records the name once in `state.invoked_skills`, and returns an error observation without state change for an unknown name. Unit: `tests/sdk/tool/test_invoke_skill.py`. Example: none. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. Server: `F25.invoke-skill`. - `S07.disable-model-invocation`: `disable_model_invocation: true` hides a skill from `<available_skills>`, makes `invoke_skill` refuse it, and keeps its triggers working. Unit: `tests/sdk/context/test_agent_context.py::TestAgentContext::test_disable_model_invocation_hides_skill_but_preserves_triggers`, `tests/sdk/tool/test_invoke_skill.py::test_disable_model_invocation_rejects_direct_invocation`. Example: none. Real-LLM: none. - `S07.inline-commands`: `` !`cmd` `` in skill content runs in the workspace at render time (10 s timeout, 50 KB cap) and is replaced by its output; fenced and inline code, unclosed fences and `` \!`cmd` `` stay literal. Unit: `tests/sdk/skills/test_skill_commands.py`. Example: none. Real-LLM: none. - `S07.path-rules`: a skill with `paths:` frontmatter (a `PathTrigger` rule), and every nested `AGENTS.md`-style file below the workspace, is kept out of the catalog and the repo context and is appended once to the observation of the first tool call that touches a matching file inside the workspace. Unit: `tests/sdk/skills/test_path_trigger.py`, `tests/sdk/conversation/test_path_rules_injection.py`. Example: none. Real-LLM: none. -- `S07.project-skills`: `load_project_skills(work_dir)` reads third-party files, then `.agents/skills`, `.openhands/skills`, `.openhands/microagents` in that order (first wins), for the work dir and then the enclosing git root; one bad skill does not stop the others. Unit: `tests/sdk/skills/test_load_project_skills.py`. Example: none. Real-LLM: none. +- `S07.project-skills`: `load_project_skills(work_dir)` reads third-party files, then `.agents/skills`, `.openhands/skills`, `.openhands/microagents` in that order (first wins), for the work dir and then the enclosing git root; one bad skill does not stop the others. Unit: `tests/sdk/skills/test_load_project_skills.py`. Example: none. Real-LLM: none. Server: `F25.catalog-project`. - `S07.load-project-flag`: `AgentContext(load_project_skills=True)` is resolved by `LocalConversation` on the first `send_message()`/`run()` from `workspace.working_dir`; project skills override same-named explicit skills, `disabled_skills` still applies, and a load failure does not stop the conversation. Unit: `tests/sdk/conversation/test_repo_root_project_skills.py`. Example: none. Real-LLM: none. -- `S07.user-skills`: `AgentContext(load_user_skills=True)` merges `~/.agents/skills`, `$OH_PERSISTENCE_DIR`-or-`~/.openhands` `skills/` and `microagents/`, then enabled installed skills; explicit skills win on a name conflict. Unit: `tests/sdk/skills/test_load_user_skills.py`. Example: none. Real-LLM: none. -- `S07.public-skills`: `AgentContext(load_public_skills=True)` (or `load_public_skills()`) clones `https://github.com/OpenHands/extensions` into the skills cache, keeps only skills listed in `marketplace_path` (default `marketplaces/default.json`, `None` for all), caches the result in-process (60 s for a branch, forever for a tag or SHA), honors `EXTENSIONS_REF`, and falls back to the cached clone when the update fails. Unit: `tests/sdk/skills/test_load_public_skills.py`, `tests/sdk/skills/test_extensions_ref.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: none. -- `S07.public-skills-switch`: `OH_PUBLIC_SKILLS_ENABLED` set to a false value (`false`, `off`, `0`, `no`) makes `load_public_skills()` for the default repository, and therefore `AgentContext(load_public_skills=True)`, return no public skills without cloning or reading anything; it is read on each call, unset, empty or unrecognised values (with a warning) keep public skills on, explicit `AgentContext(skills=[...])` are kept, and a `load_public_skills(repo_url=<another repository>)` still loads. `EXTENSIONS_PATH` names a local copy of the default repository that is read in place instead of cloned. Unit: `tests/sdk/skills/test_load_public_skills.py`, `tests/agent_server/test_public_extension_policy.py` (the agent server's side). Example: none. Real-LLM: none. -- `S07.merge-precedence`: `load_available_skills` layers public, then user, then project (later wins); `merge_skills_by_name` keeps the primary list's order; duplicate names inside `AgentContext.skills` raise. Unit: `tests/sdk/context/test_agent_context.py::TestAgentContext::test_skill_validation_duplicate_names`, `tests/sdk/conversation/test_repo_root_project_skills.py::test_load_project_skills_flag_merges_with_project_precedence`. Example: none. Real-LLM: none. -- `S07.disabled-skills`: `AgentContext(disabled_skills=[...])` drops those names after every source is loaded; an absent name is a no-op. Unit: `tests/sdk/context/test_agent_context.py::TestAgentContext::test_disabled_skills_drops_named_skill`, `tests/sdk/conversation/test_repo_root_project_skills.py::test_disabled_skills_drops_project_skill_at_lazy_merge`. Example: none. Real-LLM: none. +- `S07.user-skills`: `AgentContext(load_user_skills=True)` merges `~/.agents/skills`, `$OH_PERSISTENCE_DIR`-or-`~/.openhands` `skills/` and `microagents/`, then enabled installed skills; explicit skills win on a name conflict. Unit: `tests/sdk/skills/test_load_user_skills.py`. Example: none. Real-LLM: none. Server: `F25.catalog-user-precedence`. +- `S07.public-skills`: `AgentContext(load_public_skills=True)` (or `load_public_skills()`) clones `https://github.com/OpenHands/extensions` into the skills cache, keeps only skills listed in `marketplace_path` (default `marketplaces/default.json`, `None` for all), caches the result in-process (60 s for a branch, forever for a tag or SHA), honors `EXTENSIONS_REF`, and falls back to the cached clone when the update fails. Unit: `tests/sdk/skills/test_load_public_skills.py`, `tests/sdk/skills/test_extensions_ref.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: none. Server: `F25.public-repo`. +- `S07.public-skills-switch`: `OH_PUBLIC_SKILLS_ENABLED` set to a false value (`false`, `off`, `0`, `no`) makes `load_public_skills()` for the default repository, and therefore `AgentContext(load_public_skills=True)`, return no public skills without cloning or reading anything; it is read on each call, unset, empty or unrecognised values (with a warning) keep public skills on, explicit `AgentContext(skills=[...])` are kept, and a `load_public_skills(repo_url=<another repository>)` still loads. `EXTENSIONS_PATH` names a local copy of the default repository that is read in place instead of cloned. Unit: `tests/sdk/skills/test_load_public_skills.py`, `tests/agent_server/test_public_extension_policy.py` (the agent server's side). Example: none. Real-LLM: none. Server: `F25.public-skills-disabled`. +- `S07.merge-precedence`: `load_available_skills` layers public, then user, then project (later wins); `merge_skills_by_name` keeps the primary list's order; duplicate names inside `AgentContext.skills` raise. Unit: `tests/sdk/context/test_agent_context.py::TestAgentContext::test_skill_validation_duplicate_names`, `tests/sdk/conversation/test_repo_root_project_skills.py::test_load_project_skills_flag_merges_with_project_precedence`. Example: none. Real-LLM: none. Server: `F25.catalog-user-precedence`. +- `S07.disabled-skills`: `AgentContext(disabled_skills=[...])` drops those names after every source is loaded; an absent name is a no-op. Unit: `tests/sdk/context/test_agent_context.py::TestAgentContext::test_disabled_skills_drops_named_skill`, `tests/sdk/conversation/test_repo_root_project_skills.py::test_disabled_skills_drops_project_skill_at_lazy_merge`. Example: none. Real-LLM: none. Server: `F21.materialize-draft`. - `S07.skill-mcp-json`: an AgentSkills directory's `.mcp.json` (legacy skills: frontmatter `mcp_tools`) is parsed into `Skill.mcp_tools`, with `${VAR}` expanded from secrets, then env, then defaults; invalid JSON raises. Unit: `tests/sdk/skills/test_mcp_json.py`, `tests/sdk/skills/test_mcp_config_expansion.py`. Example: none. Real-LLM: none. -- `S07.installed-lifecycle`: `install_skill(source)` copies a skill into the install store and records it in `.installed.json`; `list_installed_skills`, `load_installed_skills` (enabled only), `disable_skill`/`enable_skill`, `update_skill` and `uninstall_skill` act on that store; an existing name needs `force=True`. Unit: `tests/sdk/skills/test_installed_skills.py`. Example: `examples/05_skills_and_plugins/03_managing_installed_skills/main.py`. Real-LLM: none. -- `S07.marketplace-install`: `install_skills_from_marketplace(dir)` installs the standalone skills and plugin skills a `marketplace.json` lists, from local paths and GitHub `blob` URLs, skipping a missing source. Unit: `tests/sdk/skills/test_installed_skills.py::TestInstallSkillsFromMarketplace`. Example: `examples/05_skills_and_plugins/04_mixed_marketplace_skills/main.py`. Real-LLM: none. +- `S07.installed-lifecycle`: `install_skill(source)` copies a skill into the install store and records it in `.installed.json`; `list_installed_skills`, `load_installed_skills` (enabled only), `disable_skill`/`enable_skill`, `update_skill` and `uninstall_skill` act on that store; an existing name needs `force=True`. Unit: `tests/sdk/skills/test_installed_skills.py`. Example: `examples/05_skills_and_plugins/03_managing_installed_skills/main.py`. Real-LLM: none. Server: `F25.install-local`, `F25.enable-disable`, `F25.refresh`, `F25.uninstall`. +- `S07.marketplace-install`: `install_skills_from_marketplace(dir)` installs the standalone skills and plugin skills a `marketplace.json` lists, from local paths and GitHub `blob` URLs, skipping a missing source. Unit: `tests/sdk/skills/test_installed_skills.py::TestInstallSkillsFromMarketplace`. Example: `examples/05_skills_and_plugins/04_mixed_marketplace_skills/main.py`. Real-LLM: none. Server: `F25.marketplace-catalog`. - `S07.serialization`: skills of every trigger type round-trip through JSON, and `mcp_tools` secrets are redacted unless a cipher is given. Unit: `tests/sdk/skills/test_skill_serialization.py`, `tests/sdk/context/test_agent_context_serialization.py`. Example: none. Real-LLM: none. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S08-plugins-marketplace.md b/.agents/skills/verify-sdk/references/feature-map/S08-plugins-marketplace.md index f45b63a826..e0dc4c04c4 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S08-plugins-marketplace.md +++ b/.agents/skills/verify-sdk/references/feature-map/S08-plugins-marketplace.md @@ -10,25 +10,25 @@ Examples: `examples/05_skills_and_plugins/02_loading_plugins/main.py`, `examples ## Sub-features -- `S08.claude-code-layout`: `Plugin.load(dir)` reads `.plugin/plugin.json` or `.claude-plugin/plugin.json` (author as string or object; no manifest means the name comes from the directory), `skills/` (or a root `SKILL.md` for a single-skill plugin), `commands/`, `agents/`, `hooks/hooks.json` and `.mcp.json`; one bad skill is skipped without dropping the rest. Unit: `tests/sdk/plugin/test_plugin_loading.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (`example_plugins/code-quality`). Real-LLM: none. -- `S08.agent-plugins-layout`: a directory with a root `plugin.json` is loaded as an Agent Plugins v1.0.0 package (detected ahead of the Claude Code fallback): the manifest must validate against the vendored schema version, `skills/` and `mcp.json` are portable components, and hooks, commands, agents and `entry_command` come only from the `dev.openhands/` extension directory and namespace; a fatal violation rejects the whole package. Unit: `tests/sdk/plugin/test_agent_plugins_format.py`, `tests/sdk/plugin/test_agent_plugins_conformance.py`. Example: none. Real-LLM: `tests/integration/tests/t10_agent_plugins_skill.py`. +- `S08.claude-code-layout`: `Plugin.load(dir)` reads `.plugin/plugin.json` or `.claude-plugin/plugin.json` (author as string or object; no manifest means the name comes from the directory), `skills/` (or a root `SKILL.md` for a single-skill plugin), `commands/`, `agents/`, `hooks/hooks.json` and `.mcp.json`; one bad skill is skipped without dropping the rest. Unit: `tests/sdk/plugin/test_plugin_loading.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (`example_plugins/code-quality`). Real-LLM: none. Server: `F26.install-local`. +- `S08.agent-plugins-layout`: a directory with a root `plugin.json` is loaded as an Agent Plugins v1.0.0 package (detected ahead of the Claude Code fallback): the manifest must validate against the vendored schema version, `skills/` and `mcp.json` are portable components, and hooks, commands, agents and `entry_command` come only from the `dev.openhands/` extension directory and namespace; a fatal violation rejects the whole package. Unit: `tests/sdk/plugin/test_agent_plugins_format.py`, `tests/sdk/plugin/test_agent_plugins_conformance.py`. Example: none. Real-LLM: `tests/integration/tests/t10_agent_plugins_skill.py`. Server: `F26.install-agent-plugins`. - `S08.agent-plugins-mcp`: an Agent Plugins `mcp.json` expands only `${PLUGIN_ROOT}` and `${PLUGIN_DATA}` (single pass), resolves plugin-relative commands and `cwd` inside the package, requires `https` for non-loopback URLs, rejects user info and fragments, drops client-generated headers, and skips an invalid entry while its siblings load; an invalid document disables MCP only. Unit: `tests/sdk/plugin/test_agent_plugins_mcp.py`. Example: none. Real-LLM: none. - `S08.path-containment`: manifests, skills, agents, commands and MCP files that resolve outside the plugin root (symlinks included) are denied while the rest of the plugin loads. Unit: `tests/sdk/plugin/test_path_containment.py`. Example: none. Real-LLM: none. -- `S08.commands-as-skills`: each `commands/<name>.md` becomes a keyword-triggered skill `/<plugin>:<name>` with its description and `$ARGUMENTS` hint, and `entry_command` exposes `Plugin.entry_slash_command`. Unit: `tests/sdk/plugin/test_plugin_loading.py` (`test_command_to_skill_conversion`, `test_get_all_skills_with_commands`, `test_load_plugin_with_entry_command`), `tests/sdk/conversation/test_local_conversation_agent_plugins.py::TestSkills::test_the_entry_command_arrives_as_a_skill`. Example: none. Real-LLM: none. +- `S08.commands-as-skills`: each `commands/<name>.md` becomes a keyword-triggered skill `/<plugin>:<name>` with its description and `$ARGUMENTS` hint, and `entry_command` exposes `Plugin.entry_slash_command`. Unit: `tests/sdk/plugin/test_plugin_loading.py` (`test_command_to_skill_conversion`, `test_get_all_skills_with_commands`, `test_load_plugin_with_entry_command`), `tests/sdk/conversation/test_local_conversation_agent_plugins.py::TestSkills::test_the_entry_command_arrives_as_a_skill`. Example: none. Real-LLM: none. Server: `F26.conversation-ambient`. - `S08.merge-semantics`: `Plugin.add_skills_to(ctx, max_skills=)` and `add_mcp_config_to(mcp)` return new objects in which plugin entries override same-named ones (with a warning) and `max_skills` raises when exceeded; `load_plugins(specs, agent)` applies this in order (last plugin wins) and concatenates hooks via `HookConfig.merge`. Unit: `tests/sdk/plugin/test_plugin_merging.py`, `tests/sdk/plugin/test_plugin_loader.py`. Example: none. Real-LLM: none. -- `S08.conversation-attach`: `Conversation(plugins=[...])` (also through the `Conversation` factory) loads plugins lazily on the first `send_message()`/`run()`, puts their skills in `conversation.agent.agent_context.skills`, starts their MCP servers, and runs their hooks after the explicit `hook_config`. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_create_conversation_with_plugins`, `test_plugins_not_loaded_until_needed`, `test_conversation_with_multiple_plugins`, `test_plugin_hooks_combined_with_explicit_hooks`, `test_plugin_mcp_config_are_initialized`), `::TestConversationFactoryPlugins`, `tests/sdk/conversation/test_local_conversation_agent_plugins.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (demo 1). Real-LLM: `tests/integration/tests/t10_agent_plugins_skill.py`. +- `S08.conversation-attach`: `Conversation(plugins=[...])` (also through the `Conversation` factory) loads plugins lazily on the first `send_message()`/`run()`, puts their skills in `conversation.agent.agent_context.skills`, starts their MCP servers, and runs their hooks after the explicit `hook_config`. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_create_conversation_with_plugins`, `test_plugins_not_loaded_until_needed`, `test_conversation_with_multiple_plugins`, `test_plugin_hooks_combined_with_explicit_hooks`, `test_plugin_mcp_config_are_initialized`), `::TestConversationFactoryPlugins`, `tests/sdk/conversation/test_local_conversation_agent_plugins.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (demo 1). Real-LLM: `tests/integration/tests/t10_agent_plugins_skill.py`. Server: `F33.conversation-plugins-tag`. - `S08.plugin-agents`: agent definitions from a plugin's `agents/` are registered as sub-agent types for the task tool (details in S11). Unit: `tests/sdk/conversation/test_local_conversation_agent_plugins.py::TestClientExtension::test_agents_are_registered`. Example: none. Real-LLM: none. - `S08.resolved-refs`: `conversation.resolved_plugins` lists each explicitly attached plugin with its resolved commit SHA (`None` for a local path), built from the original spec so `${VAR}` placeholders and inline credentials are not stored. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins::test_create_conversation_with_plugins`, `tests/sdk/git/test_url_redaction.py::TestResolvedPluginSourceCredentialRedaction`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (prints `Resolved plugin refs`). Real-LLM: none. - `S08.secret-expansion`: `${VAR}` in a plugin source, ref and Claude Code `.mcp.json` is expanded from the conversation's secrets (never from the host env), defaults apply only when no secret exists, Agent Plugins package values are left literal, and load logs never print credentials. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestPluginMcpSecretsExpansion`, `::TestPluginSourceSecretExpansion`, `::TestAgentPluginsMCPExpansion`. Example: none. Real-LLM: none. -- `S08.fetch-cache`: `fetch_plugin(source, ref=, repo_path=, update=)` clones git sources into `~/.openhands/cache/plugins/` (keyed by source), updates or reuses the cache, checks out a ref, returns `repo_path` inside the clone, and raises `PluginFetchError` on git or path errors; local paths are returned as-is. Unit: `tests/sdk/plugin/test_plugin_fetch.py`, `tests/sdk/plugin/test_plugin_fetch_integration.py`, `tests/sdk/extensions/test_fetch.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py`. Real-LLM: none. -- `S08.source-parsing`: `github:owner/repo`, https, ssh and `git://` URLs (GitHub, GitLab, Bitbucket, self-hosted) and local paths are classified; GitHub `blob`/`tree` URLs split into repo, branch and subpath; `repo_path` must be relative without `..`. Unit: `tests/sdk/extensions/test_fetch.py`, `tests/sdk/plugin/test_source.py`, `tests/sdk/plugin/test_plugin_loader.py::TestPluginSource`. Example: none. Real-LLM: none. -- `S08.installed-plugins`: `install_plugin(source, ref=, installed_dir=, force=)` copies a fetched plugin into `~/.openhands/plugins/installed/` and records it in `.installed.json`; `list_installed_plugins`, `load_installed_plugins` (enabled only), `disable_plugin`/`enable_plugin`, `get_installed_plugin`, `update_plugin` and `uninstall_plugin` act on that store. Unit: `tests/sdk/plugin/test_installed_plugins.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (demos 2 to 6). Real-LLM: none. -- `S08.installation-manager`: the generic `InstallationManager[T]` validates names (no path traversal), records requested and resolved refs, keeps the enabled flag across `force` reinstalls, re-clones on `update`, and migrates legacy `plugins`/`skills` keys of `.installed.json` into `extensions`. Unit: `tests/sdk/extensions/installation/`. Example: none. Real-LLM: none. -- `S08.ambient-plugins`: `load_user_plugins()` (`~/.agents/plugins`, then `~/.openhands/plugins`, then enabled installed plugins) and `load_project_plugins(work_dir)` (work dir and git root, `.agents/plugins` then `.openhands/plugins`) are merged into every `LocalConversation`, project over user; an explicitly attached plugin of the same name wins, and ambient plugins are not listed in `resolved_plugins`. The conversation state records the skill and MCP server names ambient plugins contributed (`ambient_plugin_skills`, `ambient_plugin_mcp_servers`), so a resumed conversation drops a since-disabled or removed plugin's skills and MCP servers before re-discovery; a failed discovery keeps the previous contributions. Unit: `tests/sdk/plugin/test_discovery.py`, `tests/sdk/conversation/test_local_conversation_plugins.py::TestAmbientPluginAutoLoad`. Example: none. Real-LLM: none. -- `S08.marketplace-manifest`: `Marketplace.load(dir)` reads `.plugin/marketplace.json` or `.claude-plugin/marketplace.json` with `name`, `owner`, `metadata`, `plugins` (string source or `{source: github|url, repo, url, ref, path}`) and `skills`, and `resolve_plugin_source(entry)` turns an entry into `(source, ref, repo_path)` relative to the marketplace. Unit: `tests/sdk/marketplace/test_marketplace.py`. Example: `examples/05_skills_and_plugins/04_mixed_marketplace_skills/main.py` (default mode prints the manifest). Real-LLM: none. -- `S08.marketplace-registry`: `MarketplaceRegistry(registrations)` rejects duplicate names, fetches and caches each manifest lazily (keeping the resolved ref), and `resolve_plugin("name@market")` or `resolve_plugin("name")` returns a `PluginSource`, raising `PluginNotFoundError`, `AmbiguousPluginError` or `MarketplaceNotFoundError`. Unit: `tests/sdk/marketplace/test_marketplace_registry.py`. Example: none. Real-LLM: none. -- `S08.marketplace-auto-load`: `AgentContext(registered_marketplaces=[MarketplaceRegistration(name, source, ref, repo_path, auto_load=True | [names])])` loads the chosen plugins and standalone skills when the conversation starts; a plugin wins over a standalone skill of the same name, explicit `plugins=` win over auto-loaded ones, a marketplace that fails to fetch is skipped, and `auto_load=False` only registers it. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_auto_load_marketplace_plugins`, `test_auto_load_marketplace_standalone_skills`, `test_plugin_wins_over_standalone_on_name_collision`, `test_auto_load_marketplace_continues_after_fetch_failure`, `test_registered_only_marketplace_does_not_auto_load`, `test_explicit_plugins_override_auto_loaded_marketplace_plugins`), `tests/sdk/skills/test_installed_skills.py::TestLoadMarketplaceStandaloneSkills`. Example: none. Real-LLM: none. -- `S08.load-plugin-runtime`: `conversation.load_plugin("name@market")` resolves through the registered marketplaces, adds the plugin's skills, MCP tools and hooks to a running conversation without rebuilding existing tools, and raises when no marketplace is registered. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_load_plugin_from_registered_marketplace`, `test_load_plugin_adds_runtime_tools_without_reinitializing_existing_tools`, `test_load_plugin_merges_runtime_hooks_and_restarts_processor`, `test_load_plugin_requires_registered_marketplaces`). Example: none. Real-LLM: none. +- `S08.fetch-cache`: `fetch_plugin(source, ref=, repo_path=, update=)` clones git sources into `~/.openhands/cache/plugins/` (keyed by source), updates or reuses the cache, checks out a ref, returns `repo_path` inside the clone, and raises `PluginFetchError` on git or path errors; local paths are returned as-is. Unit: `tests/sdk/plugin/test_plugin_fetch.py`, `tests/sdk/plugin/test_plugin_fetch_integration.py`, `tests/sdk/extensions/test_fetch.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py`. Real-LLM: none. Server: `F26.fetch-source`. +- `S08.source-parsing`: `github:owner/repo`, https, ssh and `git://` URLs (GitHub, GitLab, Bitbucket, self-hosted) and local paths are classified; GitHub `blob`/`tree` URLs split into repo, branch and subpath; `repo_path` must be relative without `..`. Unit: `tests/sdk/extensions/test_fetch.py`, `tests/sdk/plugin/test_source.py`, `tests/sdk/plugin/test_plugin_loader.py::TestPluginSource`. Example: none. Real-LLM: none. Server: `F26.install-github`, `F26.install-git`. +- `S08.installed-plugins`: `install_plugin(source, ref=, installed_dir=, force=)` copies a fetched plugin into `~/.openhands/plugins/installed/` and records it in `.installed.json`; `list_installed_plugins`, `load_installed_plugins` (enabled only), `disable_plugin`/`enable_plugin`, `get_installed_plugin`, `update_plugin` and `uninstall_plugin` act on that store. Unit: `tests/sdk/plugin/test_installed_plugins.py`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py` (demos 2 to 6). Real-LLM: none. Server: `F26.install-local`, `F26.install-conflict`, `F26.enable-disable`, `F26.uninstall`. +- `S08.installation-manager`: the generic `InstallationManager[T]` validates names (no path traversal), records requested and resolved refs, keeps the enabled flag across `force` reinstalls, re-clones on `update`, and migrates legacy `plugins`/`skills` keys of `.installed.json` into `extensions`. Unit: `tests/sdk/extensions/installation/`. Example: none. Real-LLM: none. Server: `F26.name-errors`, `F25.name-errors`, `F26.install-conflict`. +- `S08.ambient-plugins`: `load_user_plugins()` (`~/.agents/plugins`, then `~/.openhands/plugins`, then enabled installed plugins) and `load_project_plugins(work_dir)` (work dir and git root, `.agents/plugins` then `.openhands/plugins`) are merged into every `LocalConversation`, project over user; an explicitly attached plugin of the same name wins, and ambient plugins are not listed in `resolved_plugins`. The conversation state records the skill and MCP server names ambient plugins contributed (`ambient_plugin_skills`, `ambient_plugin_mcp_servers`), so a resumed conversation drops a since-disabled or removed plugin's skills and MCP servers before re-discovery; a failed discovery keeps the previous contributions. Unit: `tests/sdk/plugin/test_discovery.py`, `tests/sdk/conversation/test_local_conversation_plugins.py::TestAmbientPluginAutoLoad`. Example: none. Real-LLM: none. Server: `F26.conversation-ambient`, `F26.catalog-project`, `F26.catalog-user-precedence`. +- `S08.marketplace-manifest`: `Marketplace.load(dir)` reads `.plugin/marketplace.json` or `.claude-plugin/marketplace.json` with `name`, `owner`, `metadata`, `plugins` (string source or `{source: github|url, repo, url, ref, path}`) and `skills`, and `resolve_plugin_source(entry)` turns an entry into `(source, ref, repo_path)` relative to the marketplace. Unit: `tests/sdk/marketplace/test_marketplace.py`. Example: `examples/05_skills_and_plugins/04_mixed_marketplace_skills/main.py` (default mode prints the manifest). Real-LLM: none. Server: `F26.marketplace-local`. +- `S08.marketplace-registry`: `MarketplaceRegistry(registrations)` rejects duplicate names, fetches and caches each manifest lazily (keeping the resolved ref), and `resolve_plugin("name@market")` or `resolve_plugin("name")` returns a `PluginSource`, raising `PluginNotFoundError`, `AmbiguousPluginError` or `MarketplaceNotFoundError`. Unit: `tests/sdk/marketplace/test_marketplace_registry.py`. Example: none. Real-LLM: none. Server: `F09.load-plugin-errors`. +- `S08.marketplace-auto-load`: `AgentContext(registered_marketplaces=[MarketplaceRegistration(name, source, ref, repo_path, auto_load=True | [names])])` loads the chosen plugins and standalone skills when the conversation starts; a plugin wins over a standalone skill of the same name, explicit `plugins=` win over auto-loaded ones, a marketplace that fails to fetch is skipped, and `auto_load=False` only registers it. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_auto_load_marketplace_plugins`, `test_auto_load_marketplace_standalone_skills`, `test_plugin_wins_over_standalone_on_name_collision`, `test_auto_load_marketplace_continues_after_fetch_failure`, `test_registered_only_marketplace_does_not_auto_load`, `test_explicit_plugins_override_auto_loaded_marketplace_plugins`), `tests/sdk/skills/test_installed_skills.py::TestLoadMarketplaceStandaloneSkills`. Example: none. Real-LLM: none. Server: `F25.catalog-registered-marketplace`. +- `S08.load-plugin-runtime`: `conversation.load_plugin("name@market")` resolves through the registered marketplaces, adds the plugin's skills, MCP tools and hooks to a running conversation without rebuilding existing tools, and raises when no marketplace is registered. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_load_plugin_from_registered_marketplace`, `test_load_plugin_adds_runtime_tools_without_reinitializing_existing_tools`, `test_load_plugin_merges_runtime_hooks_and_restarts_processor`, `test_load_plugin_requires_registered_marketplaces`). Example: none. Real-LLM: none. Server: `F09.load-plugin`, `F09.load-plugin-twice`, `F09.load-plugin-errors`, `F09.load-plugin-sdk`. - `S08.marketplace-import-path`: marketplace classes import only from `openhands.sdk.marketplace`; the old `openhands.sdk.plugin` names raise. Unit: `tests/sdk/marketplace/test_deprecation.py`. Example: none. Real-LLM: none. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md b/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md index 64020958ed..6d1a5a0718 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md +++ b/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md @@ -10,18 +10,18 @@ Examples: `examples/01_standalone_sdk/07_mcp_integration.py`, `examples/01_stand ## Sub-features -- `S09.config-model`: `MCPServer` forbids unknown fields, requires `command` for stdio and `url` for remote transports, refuses `auth` together with an explicit `Authorization` header, and `coerce_mcp_config` also accepts the external `{"mcpServers": {...}}` shape while `create_mcp_tools` rejects it. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_rejects_external_config_shapes`). Example: none. Real-LLM: none. -- `S09.config-secrets`: `env`, `headers` and credential values are `SecretStr`, redacted in dumps unless a cipher is in context, round-trip through encrypted JSON, and never appear in persisted conversation state or state-update events. Unit: `tests/sdk/mcp/test_mcp_config_secrets.py`, `tests/sdk/conversation/test_mcp_secrets_serialization_leak.py`. Example: none. Real-LLM: none. -- `S09.stdio-server`: a `command`/`args` server is started as a subprocess and its tools are listed with their input schemas. Unit: `tests/sdk/mcp/test_create_mcp_tool.py::test_create_mcp_tools_stdio_server` (spawns `tests/sdk/mcp/stdio_test_server.py`). Example: `examples/01_standalone_sdk/07_mcp_integration.py`. Real-LLM: none. -- `S09.http-sse-server`: `url` servers connect over streamable HTTP or SSE (transport inferred from a `/sse` URL when omitted), tools from several servers carry `<server>_` prefixes, and calls return the server's result. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_http_server`, `test_create_mcp_tools_sse_server`, `test_create_mcp_tools_mixed_servers`, `test_create_mcp_tools_transport_inferred_from_url`, `test_execute_http_tool`, `test_execute_sse_tool`). Example: none. Real-LLM: none. -- `S09.static-auth`: `auth` strategies `none`, `bearer`, `api_key` (custom `header_name`), `basic` and `header` turn into the right HTTP headers. Unit: `tests/sdk/mcp/test_create_mcp_tool.py::test_mcp_auth_credentials_to_http_headers`. Example: none. Real-LLM: none. -- `S09.oauth`: `auth=MCPOAuthAuthCredential(strategy="oauth2", authentication=MCPOAuthAuthentication(...))` builds a FastMCP OAuth client with the given scopes, client id/secret or metadata URL and an optional token store or factory; an expired token is refreshed against the token endpoint discovered from server metadata, falling back to `<origin>/token`, and a valid token is sent without discovery. Unit: `tests/sdk/mcp/test_oauth.py`, `tests/sdk/mcp/test_create_mcp_tool.py` (`test_prepare_mcp_config_*oauth*`). Example: `examples/01_standalone_sdk/08_mcp_with_oauth.py`. Real-LLM: none. -- `S09.enabled-flag`: `MCPServer(enabled=False)` keeps a server configured but never connects to it; if every server is disabled `create_mcp_tools` raises and the conversation skips MCP entirely. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_skips_disabled_servers`, `test_create_mcp_tools_all_servers_disabled`), `tests/sdk/conversation/test_local_conversation_mcp.py::test_disabling_every_server_skips_the_mcp_connection`. Example: none. Real-LLM: none. -- `S09.listing-timeout`: tool listing that exceeds `timeout` (30 s in a conversation) raises `MCPTimeoutError` naming the servers and closes the client; an unreachable server raises. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_timeout_error_message`, `test_create_mcp_tools_connection_to_nonexistent_server`). Example: none. Real-LLM: none. +- `S09.config-model`: `MCPServer` forbids unknown fields, requires `command` for stdio and `url` for remote transports, refuses `auth` together with an explicit `Authorization` header, and `coerce_mcp_config` also accepts the external `{"mcpServers": {...}}` shape while `create_mcp_tools` rejects it. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_rejects_external_config_shapes`). Example: none. Real-LLM: none. Server: `F19.settings-validation`, `F19.test-validation`. +- `S09.config-secrets`: `env`, `headers` and credential values are `SecretStr`, redacted in dumps unless a cipher is in context, round-trip through encrypted JSON, and never appear in persisted conversation state or state-update events. Unit: `tests/sdk/mcp/test_mcp_config_secrets.py`, `tests/sdk/conversation/test_mcp_secrets_serialization_leak.py`. Example: none. Real-LLM: none. Server: `F19.test-encrypted-env`, `F19.conversation-secrets-at-rest`. +- `S09.stdio-server`: a `command`/`args` server is started as a subprocess and its tools are listed with their input schemas. Unit: `tests/sdk/mcp/test_create_mcp_tool.py::test_create_mcp_tools_stdio_server` (spawns `tests/sdk/mcp/stdio_test_server.py`). Example: `examples/01_standalone_sdk/07_mcp_integration.py`. Real-LLM: none. Server: `F19.test-stdio`. +- `S09.http-sse-server`: `url` servers connect over streamable HTTP or SSE (transport inferred from a `/sse` URL when omitted), tools from several servers carry `<server>_` prefixes, and calls return the server's result. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_http_server`, `test_create_mcp_tools_sse_server`, `test_create_mcp_tools_mixed_servers`, `test_create_mcp_tools_transport_inferred_from_url`, `test_execute_http_tool`, `test_execute_sse_tool`). Example: none. Real-LLM: none. Server: `F19.test-remote-bearer`. +- `S09.static-auth`: `auth` strategies `none`, `bearer`, `api_key` (custom `header_name`), `basic` and `header` turn into the right HTTP headers. Unit: `tests/sdk/mcp/test_create_mcp_tool.py::test_mcp_auth_credentials_to_http_headers`. Example: none. Real-LLM: none. Server: `F19.test-remote-bearer`. +- `S09.oauth`: `auth=MCPOAuthAuthCredential(strategy="oauth2", authentication=MCPOAuthAuthentication(...))` builds a FastMCP OAuth client with the given scopes, client id/secret or metadata URL and an optional token store or factory; an expired token is refreshed against the token endpoint discovered from server metadata, falling back to `<origin>/token`, and a valid token is sent without discovery. Unit: `tests/sdk/mcp/test_oauth.py`, `tests/sdk/mcp/test_create_mcp_tool.py` (`test_prepare_mcp_config_*oauth*`). Example: `examples/01_standalone_sdk/08_mcp_with_oauth.py`. Real-LLM: none. Server: `F19.oauth-flow`, `F19.oauth-refresh-writeback`, `F19.oauth-inline-writeback`. +- `S09.enabled-flag`: `MCPServer(enabled=False)` keeps a server configured but never connects to it; if every server is disabled `create_mcp_tools` raises and the conversation skips MCP entirely. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_skips_disabled_servers`, `test_create_mcp_tools_all_servers_disabled`), `tests/sdk/conversation/test_local_conversation_mcp.py::test_disabling_every_server_skips_the_mcp_connection`. Example: none. Real-LLM: none. Server: `F19.disabled-server`. +- `S09.listing-timeout`: tool listing that exceeds `timeout` (30 s in a conversation) raises `MCPTimeoutError` naming the servers and closes the client; an unreachable server raises. Unit: `tests/sdk/mcp/test_create_mcp_tool.py` (`test_create_mcp_tools_timeout_error_message`, `test_create_mcp_tools_connection_to_nonexistent_server`). Example: none. Real-LLM: none. Server: `F19.test-timeout`, `F19.test-connect-failures`. - `S09.startup-failure-tolerated`: when MCP startup fails inside a `LocalConversation`, a warning is logged and the conversation runs with its other tools; a single server that advertises a built-in tool's name (`terminal`) gets that tool as `<server>_terminal` while the built-in and the server's other tools stay (known failure #6992: initialization raises `ValueError: Duplicate tool names found: {'terminal'}`). Unit: `tests/sdk/conversation/test_local_conversation_mcp.py::test_mcp_startup_failure_does_not_abort_conversation`. Example: none. Real-LLM: none. - `S09.schema-conversion`: an MCP `inputSchema` becomes the tool's OpenAI and Responses parameters with nested objects, arrays, required lists and descriptions preserved, plus `summary` and (with a security analyzer) `security_risk`; MCP annotations are kept. Unit: `tests/sdk/mcp/test_mcp_nested_schema.py`, `tests/sdk/mcp/test_mcp_security_risk.py`, `tests/sdk/mcp/test_mcp_tool.py`. Example: none. Real-LLM: none. - `S09.argument-validation`: `action_from_arguments` validates the model's arguments against the schema, drops `None` values and the OpenHands `kind` discriminator before the call, keeps a schema property literally named `kind`, and raises on invalid input; `security_risk` and `summary` are removed by the agent before it calls `action_from_arguments`, which rejects them as extra inputs when given directly. Unit: `tests/sdk/mcp/test_mcp_tool_validation.py`, `tests/sdk/mcp/test_mcp_tool_kind_field.py`, `tests/sdk/mcp/test_mcp_action_serialization.py`. Example: none. Real-LLM: none. -- `S09.observation`: a `CallToolResult` becomes an `MCPToolObservation` with text and image content, `isError` mapped to `is_error`, structured JSON rendered as text, and the text of an `EmbeddedResource` block and a result that has only `structuredContent` kept (known failure #6993: both are dropped, leaving only `[Tool '<name>' executed.]`); an exception or timeout (300 s per call) becomes an error observation. Unit: `tests/sdk/mcp/test_mcp_tool.py`, `tests/sdk/mcp/test_mcp_observation.py`. Example: none. Real-LLM: none. +- `S09.observation`: a `CallToolResult` becomes an `MCPToolObservation` with text and image content, `isError` mapped to `is_error`, structured JSON rendered as text, and the text of an `EmbeddedResource` block and a result that has only `structuredContent` kept (known failure #6993: both are dropped, leaving only `[Tool '<name>' executed.]`); an exception or timeout (300 s per call) becomes an error observation. Unit: `tests/sdk/mcp/test_mcp_tool.py`, `tests/sdk/mcp/test_mcp_observation.py`. Example: none. Real-LLM: none. Server: `F19.conversation-tool`, `F19.test-tool-errors`. - `S09.secret-refs-in-calls`: `$VAR`, `${VAR}` and `${VAR:-default}` in a call's arguments are expanded from the conversation's secret registry (never from the host environment), and secret values in the result text are masked. Unit: `tests/sdk/mcp/test_mcp_secret_expansion.py`. Example: none. Real-LLM: none. - `S09.session-reuse`: all calls go through one client connection (one subprocess for a stdio server), so what the server keeps per process persists across calls and tools, and closing the client releases the connection. What the server keeps per MCP session (counters, auth tokens keyed by session id) persists only on a handshake connection (`S09.protocol-negotiation`), where every call carries the same session id; on a `server/discover` connection there is no session and the server sees a different session id on every call. Unit: `tests/sdk/mcp/test_mcp_session_persistence.py`, `tests/sdk/mcp/test_stateful_mcp.py`. Example: none. Real-LLM: none. - `S09.protocol-negotiation`: `MCPClient` connects with FastMCP `mode="auto"`: it probes `server/discover` and uses the sessionless protocol when the server answers it (stdio and streamable HTTP to a current FastMCP server; `client.initialize_result` is `None`), and falls back to the `initialize` handshake when the server does not (`client.initialize_result` is set); SSE always uses the handshake. Unit: `tests/sdk/mcp/test_create_mcp_tool.py::test_create_mcp_tools_http_server` (`mode == "auto"`, `initialize_result is None`), `tests/sdk/mcp/test_stateful_mcp.py` (SSE keeps a session). Example: none. Real-LLM: none. @@ -31,9 +31,9 @@ Examples: `examples/01_standalone_sdk/07_mcp_integration.py`, `examples/01_stand - `S09.config-var-expansion`: `${VAR}` and `${VAR:-default}` placeholders in `Agent.mcp_config` (and Claude Code plugin `.mcp.json`) are expanded from conversation secrets before servers start. Unit: `tests/sdk/skills/test_mcp_config_expansion.py`, `tests/sdk/conversation/test_local_conversation_plugins.py::TestPluginMcpSecretsExpansion`. Example: none. Real-LLM: none. - `S09.tool-serialization`: `MCPToolDefinition` is frozen, serializes polymorphically with `kind`, is written in the MCP wire spelling (`inputSchema`) and loads either spelling, so conversations persisted under older `mcp` majors resume. Unit: `tests/sdk/mcp/test_mcp_tool_serialization.py`, `tests/sdk/mcp/test_mcp_tool_immutability.py`, `tests/sdk/mcp/test_mcp_wire_compat.py`. Example: none. Real-LLM: none. -- `S09.runtime-add-servers`: `LocalConversation.add_mcp_servers({name: MCPServer})` on an initialized agent connects each enabled server before it returns and registers its tools as `<server>_<tool>` (whatever the server count), adds the servers to `agent.mcp_config` and the persisted state, and the next LLM call offers them; before the first message it only records them and they connect with the agent like configured servers (bare names for a single server). A duplicate name and `transport="sse"` raise `ValueError`, a server that cannot start raises `MCPServerStartError` (an `MCPError` from `openhands.sdk.mcp`, with `server_name` and the client's connection `MCPError` as `__cause__`; message `Could not start MCP server <name>: <reason>`) after closing the servers it already started, nothing refused is added, and a disabled server is recorded without connecting. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_add_mcp_servers_registers_modern_runtime_tools`, `test_add_mcp_servers_rejects_sse_transport`, `test_add_mcp_servers_accepts_running_conversation`). Example: none. Real-LLM: none. -- `S09.runtime-add-resume`: a conversation reopened from its persistence directory reconnects the servers added at runtime and the agent can call their tools by the names it was given (known failure: with one server they come back unprefixed, so `verify_echo` is `Tool ... not found`). Unit: **none (untested)**. Example: none. Real-LLM: none. -- `S09.startup-budget`: `LocalConversation(mcp_startup_budget=seconds)` lets agent initialization (the first `send_message`) return after the budget while slower servers keep starting; their tools join the agent when ready, the first step of `run()` waits for them (up to 30 s each), and one that fails late is logged and skipped. Without a budget (the default; the agent server passes 1.0) initialization waits for every server. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_startup_budget_starts_the_agent_before_a_slow_mcp_server`, `test_late_mcp_tools_are_closed_when_the_conversation_closed_first`, `test_first_step_waits_for_mcp_servers_that_missed_the_budget`). Example: none. Real-LLM: none. +- `S09.runtime-add-servers`: `LocalConversation.add_mcp_servers({name: MCPServer})` on an initialized agent connects each enabled server before it returns and registers its tools as `<server>_<tool>` (whatever the server count), adds the servers to `agent.mcp_config` and the persisted state, and the next LLM call offers them; before the first message it only records them and they connect with the agent like configured servers (bare names for a single server). A duplicate name and `transport="sse"` raise `ValueError`, a server that cannot start raises `MCPServerStartError` (an `MCPError` from `openhands.sdk.mcp`, with `server_name` and the client's connection `MCPError` as `__cause__`; message `Could not start MCP server <name>: <reason>`) after closing the servers it already started, nothing refused is added, and a disabled server is recorded without connecting. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_add_mcp_servers_registers_modern_runtime_tools`, `test_add_mcp_servers_rejects_sse_transport`, `test_add_mcp_servers_accepts_running_conversation`). Example: none. Real-LLM: none. Server: `F19.conversation-mcp-add`, `F19.conversation-mcp-add-before-init`. +- `S09.runtime-add-resume`: a conversation reopened from its persistence directory reconnects the servers added at runtime and the agent can call their tools by the names it was given (known failure: with one server they come back unprefixed, so `verify_echo` is `Tool ... not found`). Unit: **none (untested)**. Example: none. Real-LLM: none. Server: `F19.conversation-mcp-add-restart`. +- `S09.startup-budget`: `LocalConversation(mcp_startup_budget=seconds)` lets agent initialization (the first `send_message`) return after the budget while slower servers keep starting; their tools join the agent when ready, the first step of `run()` waits for them (up to 30 s each), and one that fails late is logged and skipped. Without a budget (the default; the agent server passes 1.0) initialization waits for every server. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_startup_budget_starts_the_agent_before_a_slow_mcp_server`, `test_late_mcp_tools_are_closed_when_the_conversation_closed_first`, `test_first_step_waits_for_mcp_servers_that_missed_the_budget`). Example: none. Real-LLM: none. Server: `F19.conversation-mcp-slow-start`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S10-hooks.md b/.agents/skills/verify-sdk/references/feature-map/S10-hooks.md index 037f1b0c15..77d8a1696d 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S10-hooks.md +++ b/.agents/skills/verify-sdk/references/feature-map/S10-hooks.md @@ -10,27 +10,27 @@ Examples: `examples/01_standalone_sdk/33_hooks/main.py`, `examples/01_standalone ## Sub-features -- `S10.config-format`: `HookConfig` accepts typed snake_case fields, PascalCase keys (`PreToolUse`) and the legacy `{"hooks": {...}}` wrapper, rejects unknown event names and both spellings of one event, forbids extra fields, round-trips through JSON (with `async` as alias), and `HookConfig.merge` concatenates matchers per event. Unit: `tests/sdk/hooks/test_config.py::TestHookConfig`, `::TestAsyncHooks`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (typed config; `from_dict` shown in comments). Real-LLM: none. -- `S10.definition-validation`: `type="command"` needs `command`; `prompt` needs `prompt` and forbids `command` and `async`; `agent` forbids `command` and `async`; `display_command` labels command-less hooks as `<type>-hook:<name>`. Unit: `tests/sdk/hooks/test_config.py` (`test_command_hook_requires_command`, `test_hook_definition_validation_errors`, `test_display_command`, `test_agent_hook_rejects_async`, `test_prompt_hook_rejects_async`). Example: none. Real-LLM: none. -- `S10.config-discovery`: `HookConfig.load(path)` reads one file; `HookConfig.load(working_dir=...)` looks for `<dir>/.openhands/hooks.json`, then `~/.openhands/hooks.json`; a missing file gives an empty config. Unit: `tests/sdk/hooks/test_config.py::TestHookConfig` (`test_load_from_json_file`, `test_load_missing_file_returns_empty`, `test_load_discovers_config_in_working_dir`). Example: none. Real-LLM: none. +- `S10.config-format`: `HookConfig` accepts typed snake_case fields, PascalCase keys (`PreToolUse`) and the legacy `{"hooks": {...}}` wrapper, rejects unknown event names and both spellings of one event, forbids extra fields, round-trips through JSON (with `async` as alias), and `HookConfig.merge` concatenates matchers per event. Unit: `tests/sdk/hooks/test_config.py::TestHookConfig`, `::TestAsyncHooks`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (typed config; `from_dict` shown in comments). Real-LLM: none. Server: `F27.hooks-formats`, `F27.hooks-project`. +- `S10.definition-validation`: `type="command"` needs `command`; `prompt` needs `prompt` and forbids `command` and `async`; `agent` forbids `command` and `async`; `display_command` labels command-less hooks as `<type>-hook:<name>`. Unit: `tests/sdk/hooks/test_config.py` (`test_command_hook_requires_command`, `test_hook_definition_validation_errors`, `test_display_command`, `test_agent_hook_rejects_async`, `test_prompt_hook_rejects_async`). Example: none. Real-LLM: none. Server: `F27.hooks-invalid`. +- `S10.config-discovery`: `HookConfig.load(path)` reads one file; `HookConfig.load(working_dir=...)` looks for `<dir>/.openhands/hooks.json`, then `~/.openhands/hooks.json`; a missing file gives an empty config. Unit: `tests/sdk/hooks/test_config.py::TestHookConfig` (`test_load_from_json_file`, `test_load_missing_file_returns_empty`, `test_load_discovers_config_in_working_dir`). Example: none. Real-LLM: none. Server: `F27.hooks-project`, `F27.hooks-absent`. - `S10.matcher`: `*` or an empty matcher matches every tool (and events without a tool), `/regex/` and strings with regex metacharacters are full-matched, anything else must equal the tool name; an invalid regex falls back to an exact match. Unit: `tests/sdk/hooks/test_config.py::TestHookMatcher`, `::TestHookConfig::test_get_hooks_filters_by_tool_name`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (`matcher="terminal"`). Real-LLM: none. -- `S10.command-exec`: a command hook runs through the shell in the workspace with the event JSON on stdin, `OPENHANDS_PROJECT_DIR`, `OPENHANDS_SESSION_ID`, `OPENHANDS_EVENT_TYPE`, `OPENHANDS_TOOL_NAME` in a sanitized environment that also carries the conversation's secrets (known failure #6966: it carries none), captures stdout and stderr, and reports a timeout or a missing command as an error result. Unit: `tests/sdk/hooks/test_executor.py::TestHookExecutor`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (`log_tools.sh`). Real-LLM: none. -- `S10.exit-codes`: exit 0 is success, exit 2 blocks, any other non-zero exit is a non-blocking failure; JSON stdout `decision: deny` or `continue: false` also blocks, a non-string `decision` is ignored; `execute_all` stops at the first block. Unit: `tests/sdk/hooks/test_executor.py::TestHookExecutor` (`test_execute_blocking_exit_code`, `test_execute_json_output_decision`, `test_non_string_decision_is_ignored_rather_than_fatal`, `test_execute_all_stops_on_block`). Example: `examples/01_standalone_sdk/33_hooks/main.py` (`block_dangerous.sh`). Real-LLM: none. +- `S10.command-exec`: a command hook runs through the shell in the workspace with the event JSON on stdin, `OPENHANDS_PROJECT_DIR`, `OPENHANDS_SESSION_ID`, `OPENHANDS_EVENT_TYPE`, `OPENHANDS_TOOL_NAME` in a sanitized environment that also carries the conversation's secrets (known failure #6966: it carries none), captures stdout and stderr, and reports a timeout or a missing command as an error result. Unit: `tests/sdk/hooks/test_executor.py::TestHookExecutor`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (`log_tools.sh`). Real-LLM: none. Server: `F27.hooks-conversation`. +- `S10.exit-codes`: exit 0 is success, exit 2 blocks, any other non-zero exit is a non-blocking failure; JSON stdout `decision: deny` or `continue: false` also blocks, a non-string `decision` is ignored; `execute_all` stops at the first block. Unit: `tests/sdk/hooks/test_executor.py::TestHookExecutor` (`test_execute_blocking_exit_code`, `test_execute_json_output_decision`, `test_non_string_decision_is_ignored_rather_than_fatal`, `test_execute_all_stops_on_block`). Example: `examples/01_standalone_sdk/33_hooks/main.py` (`block_dangerous.sh`). Real-LLM: none. Server: `F27.hooks-block`. - `S10.async-hooks`: `async: true` command hooks start in their own process group, get stdin, return at once with `async_started`, are killed after their timeout or at `SessionEnd`; an async `PreToolUse` hook logs that it cannot block. Unit: `tests/sdk/hooks/test_executor.py::TestAsyncHookExecution`, `::TestAsyncProcessManager`, `tests/sdk/hooks/test_manager.py::TestAsyncHookManager`. Example: `examples/05_skills_and_plugins/02_loading_plugins/example_plugins/code-quality/hooks/hooks.json` (plugin hook). Real-LLM: none. -- `S10.pre-tool-use-block`: a blocking `PreToolUse` hook records the action in `state.blocked_actions`; the agent then emits a `UserRejectObservation` (`rejection_source="hook"`, the hook's reason) instead of running the tool, and other actions in the same batch still run. Unit: `tests/sdk/hooks/test_integration.py::TestHookEventProcessorBlocking`, `tests/sdk/agent/test_action_batch.py` (`test_prepare_with_blocked`, `test_emit_blocked_produces_rejection`), `tests/sdk/hooks/test_manager.py::TestHookManager::test_run_pre_tool_use_blocks_when_hook_denies`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (demo 2). Real-LLM: none. +- `S10.pre-tool-use-block`: a blocking `PreToolUse` hook records the action in `state.blocked_actions`; the agent then emits a `UserRejectObservation` (`rejection_source="hook"`, the hook's reason) instead of running the tool, and other actions in the same batch still run. Unit: `tests/sdk/hooks/test_integration.py::TestHookEventProcessorBlocking`, `tests/sdk/agent/test_action_batch.py` (`test_prepare_with_blocked`, `test_emit_blocked_produces_rejection`), `tests/sdk/hooks/test_manager.py::TestHookManager::test_run_pre_tool_use_blocks_when_hook_denies`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (demo 2). Real-LLM: none. Server: `F27.hooks-block`. - `S10.post-tool-use`: `PostToolUse` hooks get `tool_name`, `tool_input` and the structured `tool_response` of the matching action, looked up from the event log; an error only logs. Unit: `tests/sdk/hooks/test_integration.py::TestPostToolUseActionLookup`, `tests/sdk/hooks/test_manager.py::TestHookManager::test_run_post_tool_use`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (demo 1). Real-LLM: none. - `S10.user-prompt-context`: `additionalContext` from `UserPromptSubmit` hooks is appended to the message's `extended_content`, reaches the LLM message, and is persisted in the stored event. Unit: `tests/sdk/hooks/test_integration.py::TestAdditionalContextInjection`, `::TestLocalConversationHookCallbackWiring`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (demo 3, `inject_git_context.sh`). Real-LLM: none. - `S10.user-prompt-block`: a blocking `UserPromptSubmit` hook records the message in `state.blocked_messages`; the next agent step pops it and finishes the run without calling the LLM. Unit: `tests/sdk/hooks/test_integration.py::TestUserPromptSubmitBlocking`, `tests/sdk/test_agent_step_bounded_scan.py`. Example: none. Real-LLM: none. - `S10.blocked-state-persistence`: `blocked_actions` and `blocked_messages` survive a persisted conversation's resume, and removals persist too. Unit: `tests/sdk/hooks/test_integration.py::TestBlockedStatePersistence`, `tests/sdk/conversation/local/test_state_serialization.py`. Example: none. Real-LLM: none. -- `S10.stop-hook`: when the agent finishes, `Stop` hooks run; a deny (exit 2 or `decision: deny`) injects the hooks' `additionalContext` (or the reason) as an environment message and sets the conversation back to running; a hook error allows the stop. Unit: `tests/sdk/hooks/test_integration.py::TestStopHookIntegration`, `::TestStopHookConversationIntegration::test_stop_hook_denial_injects_feedback_and_continues`, `tests/sdk/hooks/test_manager.py::TestHookManager::test_run_stop_blocked_means_continue`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (`require_summary.sh`), `examples/01_standalone_sdk/51_agent_hooks/main.py` (quality reviewer). Real-LLM: none. +- `S10.stop-hook`: when the agent finishes, `Stop` hooks run; a deny (exit 2 or `decision: deny`) injects the hooks' `additionalContext` (or the reason) as an environment message and sets the conversation back to running; a hook error allows the stop. Unit: `tests/sdk/hooks/test_integration.py::TestStopHookIntegration`, `::TestStopHookConversationIntegration::test_stop_hook_denial_injects_feedback_and_continues`, `tests/sdk/hooks/test_manager.py::TestHookManager::test_run_stop_blocked_means_continue`. Example: `examples/01_standalone_sdk/33_hooks/main.py` (`require_summary.sh`), `examples/01_standalone_sdk/51_agent_hooks/main.py` (quality reviewer). Real-LLM: none. Server: `F33.conversation-stop-hook`. - `S10.session-start`: `SessionStart` hooks run once when the conversation first initializes (first `send_message()`/`run()`), and emit a `HookExecutionEvent`. Unit: `tests/sdk/hooks/test_integration.py::TestHookExecutionEventEmission::test_hook_execution_event_emitted_for_session_start`, `tests/sdk/hooks/test_manager.py::TestHookManager::test_run_session_start`. Example: none. Real-LLM: none. - `S10.session-end-on-close`: `conversation.close()` runs `SessionEnd` hooks, then kills leftover async hook processes. Unit: **none (untested)** at the conversation level; `tests/sdk/hooks/test_manager.py::TestAsyncHookManager::test_session_end_runs_hooks_before_cleanup` covers only `HookManager.run_session_end`. Example: none. Real-LLM: none. -- `S10.execution-events`: each hook run emits a `HookExecutionEvent` (`hook_event_type`, `hook_command`, `tool_name`, `success`, `blocked`, `exit_code`, `stdout`, `stderr`, `reason`, `additional_context`, `error`, `action_id` or `message_id`, `hook_input`) unless `emit_hook_events=False`. Unit: `tests/sdk/hooks/test_integration.py::TestHookExecutionEventEmission`. Example: `examples/01_standalone_sdk/51_agent_hooks/main.py` (prints them). Real-LLM: none. +- `S10.execution-events`: each hook run emits a `HookExecutionEvent` (`hook_event_type`, `hook_command`, `tool_name`, `success`, `blocked`, `exit_code`, `stdout`, `stderr`, `reason`, `additional_context`, `error`, `action_id` or `message_id`, `hook_input`) unless `emit_hook_events=False`. Unit: `tests/sdk/hooks/test_integration.py::TestHookExecutionEventEmission`. Example: `examples/01_standalone_sdk/51_agent_hooks/main.py` (prints them). Real-LLM: none. Server: `F27.hooks-conversation`. - `S10.prompt-hook`: a `type="prompt"` hook sends the policy and the event as separate messages to a non-streaming copy of the conversation's current LLM (`usage_id` `prompt-hook:<name>`, the hook's timeout), parses the first JSON object of the reply, falls open on a bad answer or failure, and merges its cost into the parent's stats. Unit: `tests/sdk/hooks/test_executor.py::TestPromptHookExecution`, `tests/sdk/hooks/test_manager.py::TestHookManager::test_pre_tool_use_prompt_deny_blocks`. Example: `examples/01_standalone_sdk/57_prompt_hooks/main.py`. Real-LLM: none. - `S10.agent-hook`: a `type="agent"` hook runs a `LocalConversation` with the hook's `system_prompt`, the listed `tools` plus `FinishTool`, `max_iterations`, no hooks of its own, and a sub-visualizer; the final response is parsed like a prompt hook (fenced or prose-wrapped JSON accepted), failures fall open, and metrics land under `agent-hook:<name>` in the parent's stats. Unit: `tests/sdk/hooks/test_executor.py::TestAgentHookExecution`, `tests/sdk/hooks/test_manager.py::TestHookManager::test_stop_prompt_deny_keeps_agent_running`. Example: `examples/01_standalone_sdk/51_agent_hooks/main.py`. Real-LLM: none. - `S10.llm-follows-switch`: prompt and agent hooks read the conversation's LLM at call time, so `switch_llm()`/`switch_profile()` changes the model they use. Unit: `tests/sdk/hooks/test_executor.py::TestAgentHookExecution::test_llm_getter_is_resolved_live`, `::TestPromptHookExecution::test_llm_getter_is_resolved_live`. Example: none. Real-LLM: none. -- `S10.plugin-hooks-merge`: hooks from attached, ambient or runtime-loaded plugins run after the explicit `hook_config`, and the merged config is stored in `conversation.state.hook_config`; loading a plugin later restarts the hook processor only when the load adds a hook matcher. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_plugin_hooks_combined_with_explicit_hooks`, `test_load_plugin_merges_runtime_hooks_and_restarts_processor`), `tests/sdk/plugin/test_plugin_loader.py::TestHookConfigMerge`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py`. Real-LLM: none. -- `S10.plugin-hooks-reload-once`: loading a plugin that is already loaded (by bare name or by full `name@marketplace` ref) leaves one copy of its hook matchers in `conversation.state.hook_config`, does not restart the hook processor or run `SessionStart` again, and each of its hooks fires once per event. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins::test_load_plugin_binds_plugin_root_to_its_hooks` (`load_twice`). Example: none. Real-LLM: none. +- `S10.plugin-hooks-merge`: hooks from attached, ambient or runtime-loaded plugins run after the explicit `hook_config`, and the merged config is stored in `conversation.state.hook_config`; loading a plugin later restarts the hook processor only when the load adds a hook matcher. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins` (`test_plugin_hooks_combined_with_explicit_hooks`, `test_load_plugin_merges_runtime_hooks_and_restarts_processor`), `tests/sdk/plugin/test_plugin_loader.py::TestHookConfigMerge`. Example: `examples/05_skills_and_plugins/02_loading_plugins/main.py`. Real-LLM: none. Server: `F26.conversation-ambient`, `F09.load-plugin-hook-runs`. +- `S10.plugin-hooks-reload-once`: loading a plugin that is already loaded (by bare name or by full `name@marketplace` ref) leaves one copy of its hook matchers in `conversation.state.hook_config`, does not restart the hook processor or run `SessionStart` again, and each of its hooks fires once per event. Unit: `tests/sdk/conversation/test_local_conversation_plugins.py::TestLocalConversationPlugins::test_load_plugin_binds_plugin_root_to_its_hooks` (`load_twice`). Example: none. Real-LLM: none. Server: `F09.load-plugin-twice`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S11-subagents-delegation.md b/.agents/skills/verify-sdk/references/feature-map/S11-subagents-delegation.md index 3cbe761db8..22364baf7b 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S11-subagents-delegation.md +++ b/.agents/skills/verify-sdk/references/feature-map/S11-subagents-delegation.md @@ -12,16 +12,16 @@ Examples: `examples/01_standalone_sdk/25_agent_delegation.py`, `examples/01_stan - `S11.register-programmatic`: `register_agent(name, factory_func, description)` adds a type that `get_agent_factory(name)` returns; a duplicate name raises, and an unknown name raises `ValueError("Unknown agent ...")`. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_register_and_retrieve_custom_agent_factory`, `::test_unknown_agent_type_raises_value_error`. Example: `examples/01_standalone_sdk/25_agent_delegation.py`. Real-LLM: none. - `S11.register-if-absent`: `register_agent_if_absent` keeps an existing registration and returns False, so plugin, file and built-in sources never overwrite programmatic ones. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_register_agent_if_absent_existing`, `::test_register_file_agents_skips_programmatic`. Example: none. Real-LLM: none. -- `S11.markdown-schema`: `AgentDefinition.load` parses YAML frontmatter (`name`, `description`, `tools`, `skills`, `model`, `color`, `max_iteration_per_run`, `max_budget_per_run`, `hooks`, `profile_store_dir`, `mcp_config`, `permission_mode`, `condenser`); the body becomes the system prompt, `<example>` tags become `when_to_use_examples`, unknown keys land in `metadata`, invalid values raise. Unit: `tests/sdk/subagent/test_subagent_schema.py`. Example: none. Real-LLM: none. -- `S11.file-discovery`: `load_project_agents` / `load_user_agents` scan only top-level `*.md` in `.agents/agents/` then `.openhands/agents/`, skip `README.md`, and `.agents` wins over `.openhands` for one name; `discover_agents` reports level and source without touching the registry. Unit: `tests/sdk/subagent/test_subagent_loader.py`, `tests/sdk/subagent/test_discover_agents.py`. Example: none. Real-LLM: none. -- `S11.file-precedence`: `register_file_agents(work_dir)` registers project agents before user agents, so project wins by name. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_register_file_agents_project_priority`. Example: none. Real-LLM: none. +- `S11.markdown-schema`: `AgentDefinition.load` parses YAML frontmatter (`name`, `description`, `tools`, `skills`, `model`, `color`, `max_iteration_per_run`, `max_budget_per_run`, `hooks`, `profile_store_dir`, `mcp_config`, `permission_mode`, `condenser`); the body becomes the system prompt, `<example>` tags become `when_to_use_examples`, unknown keys land in `metadata`, invalid values raise. Unit: `tests/sdk/subagent/test_subagent_schema.py`. Example: none. Real-LLM: none. Server: `F27.subagents-project`, `F27.subagents-invalid`. +- `S11.file-discovery`: `load_project_agents` / `load_user_agents` scan only top-level `*.md` in `.agents/agents/` then `.openhands/agents/`, skip `README.md`, and `.agents` wins over `.openhands` for one name; `discover_agents` reports level and source without touching the registry. Unit: `tests/sdk/subagent/test_subagent_loader.py`, `tests/sdk/subagent/test_discover_agents.py`. Example: none. Real-LLM: none. Server: `F27.subagents-project`, `F27.subagents-user`. +- `S11.file-precedence`: `register_file_agents(work_dir)` registers project agents before user agents, so project wins by name. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_register_file_agents_project_priority`. Example: none. Real-LLM: none. Server: `F27.subagents-precedence`. - `S11.conversation-autoregister`: the first `send_message()`/`run()` of a `LocalConversation` calls `register_file_agents(workspace.working_dir)`, so Markdown agents in the workspace become task types without any explicit call. Unit: **none (untested)** (only `register_file_agents` itself and plugin agents, `tests/sdk/conversation/test_local_conversation_agent_plugins.py::TestClientExtension::test_agents_are_registered`, are tested). Example: none. Real-LLM: none. - `S11.plugin-agents`: `register_plugin_agents` registers `Plugin.agents` with their hooks, skipping existing names. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_register_plugin_agents`, `::test_register_plugin_agents_with_hooks`. Example: none. Real-LLM: none. -- `S11.definition-factory`: `agent_definition_to_factory(defn)` builds an `Agent` with the definition's tools (unknown tool raises), skills (project over user, unknown raises), system prompt as `system_message_suffix`, MCP config, and a default summarizing condenser unless `condenser: none`. Unit: `tests/sdk/subagent/test_subagent_registry.py` (`test_agent_definition_to_factory_*`, `test_factory_*condenser*`). Example: `examples/01_standalone_sdk/42_file_based_subagents.py`. Real-LLM: none. +- `S11.definition-factory`: `agent_definition_to_factory(defn)` builds an `Agent` with the definition's tools (unknown tool raises), skills (project over user, unknown raises), system prompt as `system_message_suffix`, MCP config, and a default summarizing condenser unless `condenser: none`. Unit: `tests/sdk/subagent/test_subagent_registry.py` (`test_agent_definition_to_factory_*`, `test_factory_*condenser*`). Example: `examples/01_standalone_sdk/42_file_based_subagents.py`. Real-LLM: none. Server: `F27.agent-definitions`. - `S11.definition-model`: `model: inherit` reuses the parent LLM; another value loads that LLM profile from `profile_store_dir` or the default store, and a missing profile raises. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_agent_definition_to_factory_model_profile`, `::test_agent_definition_to_factory_model_profile_not_found`, `::test_agent_definition_to_factory_profile_store_dir`. Example: none. Real-LLM: none. -- `S11.builtin-agents`: `register_builtins_agents(enable_browser=True)` registers the four built-in Markdown agents as fallbacks, leaves out `web-researcher` when `enable_browser=False`, and keeps deprecated names (`default`, `default cli mode`, `explore`, `bash`) working with a `DeprecatedWarning`. Unit: `tests/tools/test_builtin_agents.py`. Example: `examples/01_standalone_sdk/25_agent_delegation.py`. Real-LLM: none. +- `S11.builtin-agents`: `register_builtins_agents(enable_browser=True)` registers the four built-in Markdown agents as fallbacks, leaves out `web-researcher` when `enable_browser=False`, and keeps deprecated names (`default`, `default cli mode`, `explore`, `bash`) working with a `DeprecatedWarning`. Unit: `tests/tools/test_builtin_agents.py`. Example: `examples/01_standalone_sdk/25_agent_delegation.py`. Real-LLM: none. Server: `F27.subagents-builtin`, `F27.subagents-browser-disabled`. - `S11.factory-info`: the `task` tool description lists the registered types (`get_factory_info`) and includes usage examples only for registered built-ins. Unit: `tests/sdk/subagent/test_subagent_registry.py::test_factory_info`, `tests/tools/task/test_task_tool_set.py::TestTaskToolExamples`. Example: none. Real-LLM: none. -- `S11.task-run`: a `task` call starts a fresh sub-conversation, sends the prompt (with the parent visualizer name as sender), runs it to completion and returns its final response with `status` `completed`; ids are sequential `task_00000001`, ... and unique under concurrency. Unit: `tests/tools/task/test_task_tool_set.py::TestTaskToolSetIntegration::test_basic_task_delegation_and_result`, `tests/tools/task/test_task_manager_thread_safety.py`. Example: `examples/01_standalone_sdk/41_task_tool_set.py`. Real-LLM: none. +- `S11.task-run`: a `task` call starts a fresh sub-conversation, sends the prompt (with the parent visualizer name as sender), runs it to completion and returns its final response with `status` `completed`; ids are sequential `task_00000001`, ... and unique under concurrency. Unit: `tests/tools/task/test_task_tool_set.py::TestTaskToolSetIntegration::test_basic_task_delegation_and_result`, `tests/tools/task/test_task_manager_thread_safety.py`. Example: `examples/01_standalone_sdk/41_task_tool_set.py`. Real-LLM: none. Server: `F27.agent-definitions`. - `S11.task-resume`: `task(resume="task_...")` reopens the same sub-conversation id from the persistence dir, so the sub-agent keeps its earlier context; an unknown id returns an error observation listing the known ids. Unit: `tests/tools/task/test_task_tool_set.py::TestTaskToolSetIntegration::test_task_resume_across_turns`, `::test_resume_nonexistent_task_returns_error`, `tests/tools/task/test_task_manager.py::TestTaskManager::test_resume_after_evict`. Example: `examples/01_standalone_sdk/41_task_tool_set.py`. Real-LLM: none. - `S11.task-errors`: an unknown `subagent_type`, an exception in the sub-agent, or a non-FINISHED stop (iteration limit, stuck, paused) comes back as an error observation with the last `ConversationErrorEvent` detail and any partial output, never as an empty success. Unit: `tests/tools/task/test_task_tool_set.py::TestTaskToolSetIntegration::test_unknown_agent_type_returns_error_observation`, `::test_sub_agent_exception_returns_error_observation`, `tests/tools/task/test_task_manager.py::TestRunTask::test_non_finished_status_surfaced_as_error`, `::TestRunErrorSurfacing`. Example: none. Real-LLM: none. - `S11.task-limits`: the sub-conversation's `max_iteration_per_run` and `max_budget_per_run` come from the definition, else from the parent conversation. Unit: `tests/tools/task/test_task_manager.py::TestTaskManager::test_create_task_prefers_factory_max_iteration_over_parent`, `::TestTaskManagerBudget`. Example: none. Real-LLM: none. @@ -29,7 +29,7 @@ Examples: `examples/01_standalone_sdk/25_agent_delegation.py`, `examples/01_stan - `S11.task-persistence`: with a persisting parent, sub-conversations are stored under `<parent persistence_dir>/subagents/` and kept on close; without one they go to a temporary `openhands_tasks_*` dir that `close()` deletes. Unit: `tests/tools/task/test_task_manager.py::TestTaskManagerPersistence`. Example: none. Real-LLM: none. - `S11.task-hooks`: a definition's `hooks` become the sub-conversation's `hook_config`, on create and on resume. Unit: `tests/tools/task/test_task_manager.py::TestTaskManagerHooks`. Example: none. Real-LLM: none. - `S11.task-observability`: sub-conversations share the parent's prompt cache key and carry delegate trace metadata (`is_delegate`, `task_id`, `subagent_type`, `parent_session_id`, parent span link) and the `delegate` tag. Unit: `tests/tools/task/test_task_manager.py::TestTaskManager::test_sub_agents_inherit_parent_prompt_cache_key`, `::test_marks_conversation_as_delegate_with_linking_metadata`, `::test_delegate_metadata_includes_parent_span_link`. Example: none. Real-LLM: none. -- `S11.task-secrets`: secrets the parent conversation holds (`conversation.update_secrets(...)`) resolve in a task sub-agent's terminal, on create and on resume, and stay masked in what comes back. Unit: `tests/tools/task/test_secret_inheritance.py`. Example: none. Real-LLM: none. +- `S11.task-secrets`: secrets the parent conversation holds (`conversation.update_secrets(...)`) resolve in a task sub-agent's terminal, on create and on resume, and stay masked in what comes back. Unit: `tests/tools/task/test_secret_inheritance.py`. Example: none. Real-LLM: none. Server: `F08.secrets-task-delegate`. - `S11.task-confirmation`: a definition's `permission_mode` sets the sub-conversation's confirmation policy, omission inherits the parent's; when the sub-agent waits for confirmation, `TaskToolSet.create(confirmation_handler=...)` decides; without a handler the actions are approved and the parent's security analyzer is not passed on (a known failure, [#6999](https://github.com/OpenHands/software-agent-sdk/issues/6999)). Unit: **none (untested)**. Example: none. Real-LLM: none. - `S11.sub-agent-scope`: `SubAgentScope(tools=True, mcp_servers=True)` (the `sub_agent_scope` tool param) hides and refuses sub-agents whose tools or MCP servers the parent lacks, compares built-in tool names canonically, propagates to nested delegation tools, and says so when no sub-agent fits. Unit: `tests/sdk/subagent/test_subagent_scope.py`, `tests/tools/task/test_sub_agent_scope.py`. Example: none. Real-LLM: none. - `S11.delegate-executor`: the older `DelegateExecutor` (`spawn` named sub-agents, then `delegate` a dict of tasks run in parallel threads) closes replaced and spawned sub-agents, rolls back a partial spawn, merges metrics, and inherits the parent's persistence dir. Unit: `tests/tools/delegate/test_delegation.py`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md b/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md index 2ed0f2a8a0..d19690b95b 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md +++ b/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md @@ -11,23 +11,23 @@ Examples: `examples/01_standalone_sdk/04_confirmation_mode_example.py`, `example ## Sub-features - `S13.risk-levels`: `SecurityRisk` orders `LOW < MEDIUM < HIGH`, `is_riskier` is reflexive, comparing `UNKNOWN` raises, `max` works on concrete risks. Unit: `tests/sdk/security/test_security_risk.py`. Example: none. Real-LLM: none. -- `S13.policies`: `NeverConfirm` never and `AlwaysConfirm` always asks; `ConfirmRisky` asks at or above `threshold` (default `HIGH`, `UNKNOWN` not allowed as threshold) and for `UNKNOWN` when `confirm_unknown=True` (default); policies round-trip polymorphically. Unit: `tests/sdk/security/test_confirmation_policy.py`, `tests/sdk/security/defense_in_depth/test_pattern.py` (`ConfirmRisky().should_confirm`), `tests/sdk/agent/test_security_analysis_event.py`. Example: `examples/01_standalone_sdk/47_defense_in_depth_security.py`. Real-LLM: none. -- `S13.confirm-wait`: with a confirming policy, `run()` stops in `WAITING_FOR_CONFIRMATION` before executing; `ConversationState.get_unmatched_actions(events)` returns the pending actions; a message-only response does not wait; a lone `FinishAction` or `ThinkAction` never waits; pausing keeps the waiting status; `is_confirmation_mode_active` reflects analyzer plus non-Never policy. Unit: `tests/sdk/conversation/local/test_confirmation_mode.py`. Example: `examples/01_standalone_sdk/04_confirmation_mode_example.py`. Real-LLM: none. -- `S13.confirm-approve-reject`: calling `run()` again executes the pending actions; `reject_pending_actions(reason)` emits a `UserRejectObservation` per action and the agent continues. Unit: `tests/sdk/conversation/local/test_confirmation_mode.py::TestConfirmationMode::test_action_then_confirm_or_reject`. Example: `examples/01_standalone_sdk/04_confirmation_mode_example.py`. Real-LLM: none. +- `S13.policies`: `NeverConfirm` never and `AlwaysConfirm` always asks; `ConfirmRisky` asks at or above `threshold` (default `HIGH`, `UNKNOWN` not allowed as threshold) and for `UNKNOWN` when `confirm_unknown=True` (default); policies round-trip polymorphically. Unit: `tests/sdk/security/test_confirmation_policy.py`, `tests/sdk/security/defense_in_depth/test_pattern.py` (`ConfirmRisky().should_confirm`), `tests/sdk/agent/test_security_analysis_event.py`. Example: `examples/01_standalone_sdk/47_defense_in_depth_security.py`. Real-LLM: none. Server: `F08.policy-set`. +- `S13.confirm-wait`: with a confirming policy, `run()` stops in `WAITING_FOR_CONFIRMATION` before executing; `ConversationState.get_unmatched_actions(events)` returns the pending actions; a message-only response does not wait; a lone `FinishAction` or `ThinkAction` never waits; pausing keeps the waiting status; `is_confirmation_mode_active` reflects analyzer plus non-Never policy. Unit: `tests/sdk/conversation/local/test_confirmation_mode.py`. Example: `examples/01_standalone_sdk/04_confirmation_mode_example.py`. Real-LLM: none. Server: `F08.always-confirm-wait`, `F33.conversation-confirmation`. +- `S13.confirm-approve-reject`: calling `run()` again executes the pending actions; `reject_pending_actions(reason)` emits a `UserRejectObservation` per action and the agent continues. Unit: `tests/sdk/conversation/local/test_confirmation_mode.py::TestConfirmationMode::test_action_then_confirm_or_reject`. Example: `examples/01_standalone_sdk/04_confirmation_mode_example.py`. Real-LLM: none. Server: `F08.confirm-accept`, `F08.confirm-reject`, `F33.conversation-confirmation`. - `S13.risk-extraction`: the agent pops `security_risk` from tool arguments; read-only tools and conversations without an analyzer get `UNKNOWN`; an invalid label raises. Unit: `tests/sdk/agent/test_extract_security_risk.py`, `tests/sdk/agent/test_security_policy_integration.py::test_security_risk_param_ignored_when_no_analyzer`. Example: none. Real-LLM: none. - `S13.llm-analyzer`: `LLMSecurityAnalyzer` returns the risk the actor LLM put on the `ActionEvent`; combined with `ConfirmRisky` only HIGH (and UNKNOWN) actions stop. Unit: `tests/sdk/security/test_llm_security_analyzer.py`. Example: `examples/01_standalone_sdk/16_llm_security_analyzer.py`. Real-LLM: none. - `S13.analysis-event`: each confirmation check with an analyzer emits one `SecurityAnalysisEvent` (analyzer and policy class names, `risks` and `details` keyed by action id) before the wait, even under `NeverConfirm`; an analyzer exception rates the action `HIGH` with the error as detail; secrets in it are masked for callbacks and persistence. Unit: `tests/sdk/agent/test_security_analysis_event.py`, `tests/sdk/security/test_security_analyzer.py`. Example: none. Real-LLM: none. - `S13.security-policy-prompt`: the default risk policy is in the system message; `security_policy_filename` swaps in a custom file inserted verbatim (not Jinja-rendered); `None` or an empty file disables it without leaking the default; risk-assessment instructions stay with or without an analyzer. Unit: `tests/sdk/agent/test_security_policy_integration.py`. Example: `examples/01_standalone_sdk/32_configurable_security_policy.py`. Real-LLM: none. -- `S13.pattern-analyzer`: `PatternSecurityAnalyzer` normalizes Unicode evasions (fullwidth, zero-width, bidi, controls), scans tool name and arguments for destructive, code-execution and fetch-to-exec signatures and all fields (including thought and summary) for injection; reasoning alone does not trip exec patterns; input is capped with arguments first. Unit: `tests/sdk/security/defense_in_depth/test_pattern.py`, `tests/sdk/security/defense_in_depth/test_field_cap.py`, `tests/sdk/security/defense_in_depth/test_adversarial.py`. Example: `examples/01_standalone_sdk/47_defense_in_depth_security.py`. Real-LLM: none. +- `S13.pattern-analyzer`: `PatternSecurityAnalyzer` normalizes Unicode evasions (fullwidth, zero-width, bidi, controls), scans tool name and arguments for destructive, code-execution and fetch-to-exec signatures and all fields (including thought and summary) for injection; reasoning alone does not trip exec patterns; input is capped with arguments first. Unit: `tests/sdk/security/defense_in_depth/test_pattern.py`, `tests/sdk/security/defense_in_depth/test_field_cap.py`, `tests/sdk/security/defense_in_depth/test_adversarial.py`. Example: `examples/01_standalone_sdk/47_defense_in_depth_security.py`. Real-LLM: none. Server: `F08.confirm-risky`. - `S13.policy-rails`: `PolicyRailSecurityAnalyzer` returns HIGH for fetch piped to a shell, raw-disk `dd`/`mkfs`, and catastrophic `rm` of `/` or home, LOW for safe commands, using the tree-sitter shell AST (quoted, path-qualified and `sh -c` nested verbs resolved; depth bound and parse errors map to UNKNOWN). Unit: `tests/sdk/security/defense_in_depth/test_policy_rails.py`, `tests/sdk/security/defense_in_depth/test_shell_semantics.py`, `tests/sdk/security/defense_in_depth/test_shell_parser_bypasses.py`, `tests/sdk/security/test_shell_ast.py`, `tests/sdk/security/test_shell_parser.py`, `tests/sdk/security/test_shell_parser_node_shapes.py`. Example: `examples/01_standalone_sdk/47_defense_in_depth_security.py`. Real-LLM: none. - `S13.ensemble`: `EnsembleSecurityAnalyzer(analyzers=[...])` takes the maximum concrete risk, fails closed to HIGH on a child exception, ignores UNKNOWN next to a concrete verdict unless `propagate_unknown=True`, rejects an empty list, and serializes with nested children. Unit: `tests/sdk/security/defense_in_depth/test_ensemble.py`, `tests/sdk/security/defense_in_depth/test_serialization.py`. Example: `examples/01_standalone_sdk/47_defense_in_depth_security.py`. Real-LLM: none. - `S13.toolshield`: `ToolShieldLLMSecurityAnalyzer(llm=..., history_window, safety_experiences)` asks a separate guardrail LLM about the action plus recent history, wraps untrusted fields in escaped XML, accepts only one standalone risk label (smuggled or conflicting labels give UNKNOWN); helpers detect active MCP tools and load ToolShield safety experiences. Unit: `tests/sdk/security/test_toolshield_llm_analyzer.py` (7 tests skip without the `toolshield` extra). Example: none. Real-LLM: none. - `S13.grayswan`: `GraySwanAnalyzer` reads `GRAYSWAN_API_KEY` / `GRAYSWAN_POLICY_ID`, sends recent events as OpenAI messages to the Gray Swan API, maps the violation score with `low_threshold`/`medium_threshold` (indirect prompt injection escalates to HIGH), and returns UNKNOWN on error, timeout or no key. Unit: `tests/sdk/security/grayswan/test_grayswan_analyzer.py`, `tests/sdk/security/grayswan/test_grayswan_utils.py`. Example: none. Real-LLM: none. - `S13.settings-security`: `ConversationSettings(confirmation_mode=True, security_analyzer="llm")` builds `ConfirmRisky` plus `LLMSecurityAnalyzer` (no analyzer gives `AlwaysConfirm`; off gives `NeverConfirm`). Unit: `tests/sdk/test_settings.py::test_conversation_settings_create_request`. Example: none. Real-LLM: none. -- `S13.secret-sources`: `StaticSecret` and `LookupSecret` (URL plus headers, relative URLs resolved against the current server, `register_local_secret_resolver` short-circuits HTTP and falls back when it declines or raises) redact values and sensitive headers on dump unless exposed or encrypted with a cipher. Unit: `tests/sdk/conversation/test_secret_source.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. -- `S13.secret-registry`: `SecretRegistry.update_secrets` adds or overwrites; `find_secrets_in_text` matches names case-insensitively; `get_secrets_as_env_vars(command)` resolves only referenced names (a failing source is skipped); resolved values are tracked for masking. Unit: `tests/sdk/conversation/test_secrets_manager.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. -- `S13.secret-injection`: secrets from `Conversation(secrets=...)`, `update_secrets(...)` and `AgentContext.secrets` (request wins on collision) reach the terminal as env vars and appear by name in the dynamic system context. Unit: `tests/sdk/conversation/test_conversation_secrets_constructor.py`, `tests/cross/test_agent_secrets_integration.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. -- `S13.mask-output`: `mask_secrets_in_output` replaces every tracked value with `<secret-hidden>`, survives a serialization round trip, tolerates and backs off failing sources; the streaming mask catches a value split across chunks. Unit: `tests/sdk/conversation/test_secrets_manager.py` (`test_mask_secrets_*`, `test_compile_stream_mask_*`), `tests/cross/test_agent_secrets_integration.py::test_mask_secrets`, `::test_masking_persists`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. +- `S13.secret-sources`: `StaticSecret` and `LookupSecret` (URL plus headers, relative URLs resolved against the current server, `register_local_secret_resolver` short-circuits HTTP and falls back when it declines or raises) redact values and sensitive headers on dump unless exposed or encrypted with a cipher. Unit: `tests/sdk/conversation/test_secret_source.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-lookup`, `F08.secrets-lookup-self`. +- `S13.secret-registry`: `SecretRegistry.update_secrets` adds or overwrites; `find_secrets_in_text` matches names case-insensitively; `get_secrets_as_env_vars(command)` resolves only referenced names (a failing source is skipped); resolved values are tracked for masking. Unit: `tests/sdk/conversation/test_secrets_manager.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-override`. +- `S13.secret-injection`: secrets from `Conversation(secrets=...)`, `update_secrets(...)` and `AgentContext.secrets` (request wins on collision) reach the terminal as env vars and appear by name in the dynamic system context. Unit: `tests/sdk/conversation/test_conversation_secrets_constructor.py`, `tests/cross/test_agent_secrets_integration.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-terminal`. +- `S13.mask-output`: `mask_secrets_in_output` replaces every tracked value with `<secret-hidden>`, survives a serialization round trip, tolerates and backs off failing sources; the streaming mask catches a value split across chunks. Unit: `tests/sdk/conversation/test_secrets_manager.py` (`test_mask_secrets_*`, `test_compile_stream_mask_*`), `tests/cross/test_agent_secrets_integration.py::test_mask_secrets`, `::test_masking_persists`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-terminal`. - `S13.mask-tool-output`: every tool observation goes through `ToolDefinition.__call__`, which masks it with `mask_secrets_in_model`: file editor views, grep's echoed pattern, glob paths and nested apply-patch content are masked; no tool bypasses it; without a conversation nothing is masked. Unit: `tests/tools/test_tool_output_secret_masking.py`. Example: none. Real-LLM: none. - `S13.mask-models`: `mask_secrets_in_model` walks nested models, lists, sets, dicts and namedtuples, leaves non-strings, `SecretStr` and enum members alone, and keeps base64 image data (data URLs, browser screenshots) intact while masking regular URLs. Unit: `tests/sdk/conversation/test_secret_registry_model_masking.py`, `tests/cross/test_secret_masking_tool_payloads.py`. Example: none. Real-LLM: none. - `S13.mask-async`: in `arun()`, agent messages and tool thoughts are masked the same as in sync, including `LookupSecret`s resolved over loopback and secrets registered during a lookup, without blocking the loop. Unit: `tests/sdk/agent/test_async_secret_masking.py`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S14-core-tools.md b/.agents/skills/verify-sdk/references/feature-map/S14-core-tools.md index e65fd1bf07..41b5eea17a 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S14-core-tools.md +++ b/.agents/skills/verify-sdk/references/feature-map/S14-core-tools.md @@ -20,7 +20,7 @@ Examples: `examples/04_llm_specific_tools/01_gpt5_apply_patch_preset.py`, `examp - `S14.terminal-output-truncation`: long output is truncated for the LLM with metadata kept, and the full output is written under the conversation's observation persistence directory (`full_output_save_dir`). Unit: `tests/tools/terminal/test_observation_truncation.py`, `tests/tools/terminal/test_terminal_session.py::test_session_truncates_large_command_output`, `::test_truncation_preserves_metadata_in_llm_content`. Example: none. Real-LLM: none. - `S14.terminal-metadata`: the PS1 marker carries cwd, exit code, username and Python interpreter into `CmdOutputMetadata`, and survives corrupted or interleaved prompts. Unit: `tests/tools/terminal/test_terminal_ps1_metadata.py`, `tests/tools/terminal/test_ps1_corruption.py`, `tests/tools/terminal/test_terminal_parsing.py`. Example: none. Real-LLM: none. - `S14.terminal-input-encoding`: heredocs are sent in chunks and complete correctly, terminal query sequences (such as the DSR cursor-position request and OSC queries) are filtered from output, other escape sequences pass through, and literal arguments that would be mangled by the shell are guarded. Unit: `tests/tools/terminal/test_heredoc_chunked_send.py`, `tests/tools/terminal/test_escape_filter.py`, `tests/tools/terminal/test_literal_arg_guard.py`. Example: none. Real-LLM: none. -- `S14.terminal-secret-masking`: secret values registered on the conversation are masked in terminal output and in tmux debug logs. Unit: `tests/tools/terminal/test_secrets_masking.py`, `tests/tools/terminal/test_tmux_log_redaction.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. +- `S14.terminal-secret-masking`: secret values registered on the conversation are masked in terminal output and in tmux debug logs. Unit: `tests/tools/terminal/test_secrets_masking.py`, `tests/tools/terminal/test_tmux_log_redaction.py`. Example: `examples/01_standalone_sdk/12_custom_secrets.py`. Real-LLM: none. Server: `F08.secrets-terminal`. - `S14.terminal-cleanup`: closing the conversation closes the terminal session (and tmux panes), and executor shutdown tolerates interpreter teardown. Unit: `tests/tools/terminal/test_conversation_cleanup.py`, `tests/tools/terminal/test_shutdown_handling.py`. Example: none. Real-LLM: none. - `S14.terminal-windows`: the PowerShell backend runs commands, sends keys and handles Ctrl+C on Windows. Unit: `tests/tools/terminal/test_windows_terminal.py`, `tests/tools/terminal/test_windows_ctrl_c.py` (both skip off Windows), `tests/tools/terminal/test_windows_terminal_send_keys.py` (runs anywhere). Example: none. Real-LLM: none. - `S14.terminal-observation-wait-cap`: `Tool(name="terminal", params={"max_observation_wait_seconds": N})` (also `TerminalExecutor(max_observation_wait_seconds=N)`) caps how long one call waits for output, even under a larger `timeout`: after about N seconds the call returns the still-running observation (`exit_code` -1, not an error, the "timed out after N seconds" note) without killing the command, and each empty `is_input` poll waits at most N seconds again until the command's output and exit code arrive. The cap holds on tmux and subprocess and across a reset; 0, a negative, NaN or infinite value raises `ValueError` when the tool is created; `None` (the default) keeps the old waits. Unit: `tests/tools/terminal/test_observation_wait.py`. Example: none. Real-LLM: none. @@ -34,7 +34,7 @@ Examples: `examples/04_llm_specific_tools/01_gpt5_apply_patch_preset.py`, `examp - `S14.grep`: `grep` searches file contents by regex with an `include` filter, returns matching files (capped), uses ripgrep and falls back to Python with the same results. Unit: `tests/tools/grep/test_grep_tool.py`, `tests/tools/grep/test_grep_executor.py`, `tests/tools/grep/test_consistency.py`. Example: `examples/01_standalone_sdk/24_planning_agent_workflow.py`. Real-LLM: none. - `S14.tool-output-masking`: file editor views, the grep pattern, glob paths and apply_patch commit content are masked through one chokepoint when the conversation has secrets; without a conversation nothing is masked. Unit: `tests/tools/test_tool_output_secret_masking.py`. Example: none. Real-LLM: none. - `S14.shared-working-dir`: terminal, file editor and other tools take their working directory from the conversation workspace and need no `params` for it. Unit: `tests/tools/test_working_dir_standardization.py`, `tests/cross/test_registry_directories.py`. Example: none. Real-LLM: none. -- `S14.preset-default`: `get_default_tools(enable_browser=..., enable_sub_agents=...)` returns `terminal`, `file_editor`, `task_tracker` (plus `browser_tool_set` and `task_tool_set` when asked); `get_default_agent(llm, cli_mode=...)` adds the default condenser, drops the browser in CLI mode and, with `finish_tool_response_schema`, registers a structured finish tool; the SDK's `default_tool_specs()` matches it. Unit: `tests/tools/test_preset_default.py`, `tests/sdk/tool/test_defaults.py`, `tests/cross/test_default_tool_names.py`, `tests/integration/test_tool_presets.py::test_get_tools_for_preset_default`. Example: `examples/01_standalone_sdk/24_planning_agent_workflow.py`. Real-LLM: every `tests/integration/tests/t*.py` with the default preset. +- `S14.preset-default`: `get_default_tools(enable_browser=..., enable_sub_agents=...)` returns `terminal`, `file_editor`, `task_tracker` (plus `browser_tool_set` and `task_tool_set` when asked); `get_default_agent(llm, cli_mode=...)` adds the default condenser, drops the browser in CLI mode and, with `finish_tool_response_schema`, registers a structured finish tool; the SDK's `default_tool_specs()` matches it. Unit: `tests/tools/test_preset_default.py`, `tests/sdk/tool/test_defaults.py`, `tests/cross/test_default_tool_names.py`, `tests/integration/test_tool_presets.py::test_get_tools_for_preset_default`. Example: `examples/01_standalone_sdk/24_planning_agent_workflow.py`. Real-LLM: every `tests/integration/tests/t*.py` with the default preset. Server: `F27.catalog`. - `S14.preset-gpt5-gemini`: `get_gpt5_tools()` swaps `file_editor` for `apply_patch`, `get_gemini_tools()` swaps it for the four Gemini tools; `get_gpt5_agent(llm)` and `get_gemini_agent(llm)` wrap them with a condenser. Unit: `tests/integration/test_tool_presets.py` (tool lists only; the `get_*_agent` builders are not unit-tested). Example: `examples/04_llm_specific_tools/01_gpt5_apply_patch_preset.py`, `examples/04_llm_specific_tools/02_gemini_file_tools.py`. Real-LLM: `tests/integration/run_infer.py --tool-preset gpt5` / `--tool-preset gemini`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S15-browser.md b/.agents/skills/verify-sdk/references/feature-map/S15-browser.md index d4e085ccab..989d13e945 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S15-browser.md +++ b/.agents/skills/verify-sdk/references/feature-map/S15-browser.md @@ -13,7 +13,7 @@ Examples: `examples/01_standalone_sdk/15_browser_use.py`, `examples/01_standalon - `S15.toolset-create`: `BrowserToolSet.create(conv_state)` returns the 14 browser tools, all bound to one executor, each with the browser prompt guidance and an MCP-convertible schema; the set takes no LLM-facing parameters and is registered on import. Unit: `tests/tools/browser_use/test_browser_toolset.py` (`test_browser_toolset_create_includes_all_browser_tools`, `test_browser_toolset_create_tools_have_shared_executor`, `test_browser_toolset_create_tools_can_generate_mcp_schema`), `tests/cross/test_automatic_registration.py::test_browser_tool_automatic_registration`. Example: `examples/01_standalone_sdk/15_browser_use.py`. Real-LLM: `tests/integration/tests/t05_simple_browsing.py`. - `S15.shared-executor`: repeated `create()` calls (parent and sub-agents) reuse one executor; construction is serialized without holding the shared lock; a second call's executor config is ignored with a warning; a closing executor is waited for before a new one is built. Unit: `tests/tools/browser_use/test_browser_toolset.py` (`test_browser_toolset_create_multiple_calls_share_executor`, `test_browser_toolset_shared_executor_survives_multiple_subagents`, `test_browser_toolset_warns_when_config_ignored`, `test_browser_toolset_create_waits_for_shared_executor_close`). Example: none. Real-LLM: none. - `S15.degrade-on-failure`: when the executor fails to start, `create()` logs a warning and returns no tools, tool resolution for the rest of the agent succeeds, and the system prompt drops the browser guidance; a profile pinned to the browser still resolves on a browserless runtime. Unit: `tests/tools/browser_use/test_browser_toolset.py` (`test_browser_toolset_create_degrades_when_executor_fails`, `test_resolve_tool_survives_browser_executor_failure`, `test_no_browser_guidance_when_browser_fails_to_start`, `test_migrated_profile_with_pinned_browser_resolves_on_browserless_runtime`). Example: none. Real-LLM: none. -- `S15.chromium-detection`: `BrowserToolExecutor.check_chromium_available()` (cached) finds `google-chrome`, `chrome`, `chromium`, `chromium-browser` or `microsoft-edge` on `PATH`, standard Linux, macOS and Windows install paths (Edge on Windows), or Playwright's browsers (`$PLAYWRIGHT_BROWSERS_PATH`, then the HOME cache), in that order after the system paths, and returns `None` otherwise; `BrowserToolSet.is_usable()` and `list_usable_tools()` follow it; a missing binary raises with install instructions (`uvx playwright install chromium`). Unit: `tests/tools/browser_use/test_chromium_detection.py`, `tests/cross/test_automatic_registration.py::test_browser_tool_usable_listing_respects_chromium_availability`. Example: none. Real-LLM: none. +- `S15.chromium-detection`: `BrowserToolExecutor.check_chromium_available()` (cached) finds `google-chrome`, `chrome`, `chromium`, `chromium-browser` or `microsoft-edge` on `PATH`, standard Linux, macOS and Windows install paths (Edge on Windows), or Playwright's browsers (`$PLAYWRIGHT_BROWSERS_PATH`, then the HOME cache), in that order after the system paths, and returns `None` otherwise; `BrowserToolSet.is_usable()` and `list_usable_tools()` follow it; a missing binary raises with install instructions (`uvx playwright install chromium`). Unit: `tests/tools/browser_use/test_chromium_detection.py`, `tests/cross/test_automatic_registration.py::test_browser_tool_usable_listing_respects_chromium_availability`. Example: none. Real-LLM: none. Server: `F27.browser-probe`, `F27.browser-playwright-path`. - `S15.executor-config`: `headless`, `allowed_domains`, `session_timeout_minutes`, `init_timeout_seconds`, `action_timeout_seconds` (must be positive), `inject_scripts` and extra keyword arguments reach the browser-use server; each executor gets its own profile directory unless `user_data_dir` is given; initialization is lazy and idempotent and times out cleanly. Unit: `tests/tools/browser_use/test_browser_executor.py` (`test_browser_executor_config_passing`, `test_browser_executor_rejects_non_positive_action_timeout`, `test_browser_executor_initialization_lazy`, `test_browser_executor_initialization_idempotent`), `tests/tools/browser_use/test_browser_initialization.py`. Example: none. Real-LLM: none. - `S15.action-routing`: each action type is routed to its server method, an unsupported action and a raised exception come back as error observations, and a slow action is cut off at `action_timeout_seconds`. Unit: `tests/tools/browser_use/test_browser_executor.py` (`test_browser_executor_action_routing_*`, `test_browser_executor_unsupported_action_handling`, `test_browser_executor_error_wrapping`, `test_browser_executor_timeout_wrapping_live_service`). Example: none. Real-LLM: none. - `S15.session-recovery`: consecutive timeouts reset the browser session after the third; the last try before a reset uses a fixed 30-second timeout (`DEGRADED_TIMEOUT_SECONDS`, shorter than the 300-second default but longer than a smaller `action_timeout_seconds`); a success resets the counter; ordinary action errors do not count. The reset kills the local browser's process tree when its 5-second cleanup leaves it running and drops the server's session, so the next action starts a new browser. Unit: `tests/tools/browser_use/test_browser_executor.py` (`test_issue_2412_consecutive_failures_reset_session`, `test_issue_2412_success_resets_failure_counter`, `test_issue_2412_action_errors_do_not_trigger_reset`, `test_issue_2412_degraded_timeout_after_failures`, `test_reset_kills_hung_browser_and_forgets_its_session`). Example: none. Real-LLM: none. @@ -26,7 +26,7 @@ Examples: `examples/01_standalone_sdk/15_browser_use.py`, `examples/01_standalon - `S15.recording-storage`: events flush periodically (`flush_interval_seconds`, default 5) into numbered, timestamped JSON chunks; each recording gets its own `recording-<timestamp>` subfolder; concurrent flushes do not corrupt the buffer; the reported file count matches the files on disk. Unit: `tests/tools/browser_use/test_recording_flush.py`. Example: `examples/01_standalone_sdk/38_browser_session_recording.py`. Real-LLM: none. - `S15.cleanup`: `executor.close()` (and `__del__`) closes all browser sessions, falls back to `close_browser`, kills a local browser process tree its cleanup left running (a process it cannot kill is logged as a warning), always stops the async executor, releases the shared executor, and serializes concurrent closes; `conversation.close()` triggers it. Unit: `tests/tools/browser_use/test_browser_cleanup.py`, `tests/tools/browser_use/test_browser_executor.py::test_kill_processes_reports_a_process_it_cannot_kill`, `tests/tools/browser_use/test_browser_executor_e2e.py::TestBrowserExecutorE2E::test_executor_initialization_and_cleanup` (skipped without Chromium). Example: `examples/01_standalone_sdk/38_browser_session_recording.py`. Real-LLM: none. - `S15.prompt-guidance`: the browser guidance renders once in the system prompt, before external-service guidance, and survives a custom system prompt and a persona. Unit: `tests/tools/browser_use/test_browser_toolset.py` (`test_browser_guidance_renders_once_before_external_services`, `test_custom_system_prompt_keeps_browser_guidance`, `test_persona_keeps_browser_guidance`). Example: none. Real-LLM: none. -- `S15.preset-browser`: `get_default_tools(enable_browser=True)` appends `browser_tool_set` before the sub-agent tool, `get_default_agent(llm, cli_mode=True)` leaves it out, and `register_builtins_agents(enable_browser=False)` skips the browser-only `web-researcher` sub-agent. Unit: `tests/sdk/tool/test_defaults.py::test_explicit_browser_appends_before_sub_agents`, `tests/cross/test_default_tool_names.py::test_browser_name_matches_browser_tool_set`, `tests/tools/test_builtin_agents.py::test_register_builtins_agents_skips_web_researcher_without_browser`. Example: `examples/01_standalone_sdk/15_browser_use.py`. Real-LLM: none. +- `S15.preset-browser`: `get_default_tools(enable_browser=True)` appends `browser_tool_set` before the sub-agent tool, `get_default_agent(llm, cli_mode=True)` leaves it out, and `register_builtins_agents(enable_browser=False)` skips the browser-only `web-researcher` sub-agent. Unit: `tests/sdk/tool/test_defaults.py::test_explicit_browser_appends_before_sub_agents`, `tests/cross/test_default_tool_names.py::test_browser_name_matches_browser_tool_set`, `tests/tools/test_builtin_agents.py::test_register_builtins_agents_skips_web_researcher_without_browser`. Example: `examples/01_standalone_sdk/15_browser_use.py`. Real-LLM: none. Server: `F27.browser-disabled-launch`, `F27.subagents-browser-disabled`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S16-planning-workflow-tools.md b/.agents/skills/verify-sdk/references/feature-map/S16-planning-workflow-tools.md index 3c927eef69..31473bd322 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S16-planning-workflow-tools.md +++ b/.agents/skills/verify-sdk/references/feature-map/S16-planning-workflow-tools.md @@ -30,16 +30,16 @@ Examples: `examples/01_standalone_sdk/24_planning_agent_workflow.py`, `examples/ - `S16.workflow-pipeline`: `wf.pipeline(items, stage1, stage2, ...)` runs sync or async stages per item with no barrier between stages, keeps item order, and drops a failed item to `None`; it needs at least one stage. Unit: `tests/tools/workflow/test_workflow_tool.py` (`test_pipeline_*`). Example: none. Real-LLM: none. - `S16.workflow-errors-and-output`: the script can catch common exceptions and `ExceptionGroup`s; uncaught errors come back formatted (with group details); the input `wf.reduce_agent` sends its sub-agent is cut to 12,000 characters, by whole trailing elements ("items omitted to fit") or characters; the script's own return value reaches the LLM whole. Unit: `tests/tools/workflow/test_workflow_tool.py` (`test_workflow_script_can_catch_*`, `test_format_exception_includes_exception_group_details`, `test_format_value_*`). Example: none. Real-LLM: none. - `S16.workflow-lifecycle`: closing the context closes the task manager once, and `run_agent` after close raises. Unit: `tests/tools/workflow/test_workflow_tool.py` (`test_workflow_context_close_propagates_to_manager`, `test_workflow_context_close_is_idempotent`, `test_run_agent_raises_after_close`). Example: none. Real-LLM: none. -- `S16.reactions`: `Tool(name=ReactTool.name)` / `Tool(name=UnreactTool.name)` (`react`, `unreact`) put an emoji on, or take it off, the user's latest message or the `message_id` given; each call returns `Reacted :<emoji>:.` / `Removed :<emoji>:.` and the agent records one agent `ReactionEvent` right after the observation; an emoji outside the six is rejected as a tool validation error. `conversation.react(emoji)` records a user `ReactionEvent` on the agent's latest answer (`finish` or message) with its text ending, raises `ValueError` for an unknown emoji, leaves the run status alone, and the agent sees it as one `[The user reacted ...]` line on its next turn, while its own reactions stay out of what it sees. Unit: `tests/tools/reaction/test_reaction_tools.py`. Example: none. Real-LLM: none. -- `S16.message-edit-tool`: `Tool(name=EditMessageTool.name)` (`edit_message`, no params) takes `old_str` and `new_str`; on success the observation reads `Edited. The message now reads:` plus the whole new text. `old_str` must occur exactly once across the agent's messages and `finish` answers as they read after earlier edits; text that is only in a user message, missing, empty or unchanged is an error observation and records nothing. Two edits of the same message in one response: the first stands and the later one becomes an error observation (`Not edited: an earlier call in this response already edits this message`) quoting the message as the first edit left it. Unit: `tests/tools/message_edit/test_message_edit.py`. Example: none. Real-LLM: none. +- `S16.reactions`: `Tool(name=ReactTool.name)` / `Tool(name=UnreactTool.name)` (`react`, `unreact`) put an emoji on, or take it off, the user's latest message or the `message_id` given; each call returns `Reacted :<emoji>:.` / `Removed :<emoji>:.` and the agent records one agent `ReactionEvent` right after the observation; an emoji outside the six is rejected as a tool validation error. `conversation.react(emoji)` records a user `ReactionEvent` on the agent's latest answer (`finish` or message) with its text ending, raises `ValueError` for an unknown emoji, leaves the run status alone, and the agent sees it as one `[The user reacted ...]` line on its next turn, while its own reactions stay out of what it sees. Unit: `tests/tools/reaction/test_reaction_tools.py`. Example: none. Real-LLM: none. Server: `F37.tool`. +- `S16.message-edit-tool`: `Tool(name=EditMessageTool.name)` (`edit_message`, no params) takes `old_str` and `new_str`; on success the observation reads `Edited. The message now reads:` plus the whole new text. `old_str` must occur exactly once across the agent's messages and `finish` answers as they read after earlier edits; text that is only in a user message, missing, empty or unchanged is an error observation and records nothing. Two edits of the same message in one response: the first stands and the later one becomes an error observation (`Not edited: an earlier call in this response already edits this message`) quoting the message as the first edit left it. Unit: `tests/tools/message_edit/test_message_edit.py`. Example: none. Real-LLM: none. Server: `F39.miss`. - `S16.message-edit-event`: each successful `edit_message` call records one agent `MessageEditEvent(target_event_id, text)` right after its `ObservationEvent`; `message_text(state.walk_back(until=message.id), message)` (`openhands.sdk.event.message_edit`) returns the latest edit's text, a user `conversation.react(...)` on an edited message quotes the edited text, and the events round-trip through JSON and survive reloading the persisted conversation. Unit: `tests/tools/message_edit/test_message_edit.py` (`test_agent_records_the_edit_after_the_observation`, `test_edit_round_trips_as_an_event`, `test_a_users_reaction_quotes_the_edited_text`). Example: none. Real-LLM: none. -- `S16.message-edit-llm-view`: `MessageEditEvent` is not an `LLMConvertibleEvent` and is not in `conversation.state.view`; the LLM's next call, like `LLMConvertibleEvent.events_to_messages(...)`, still holds the assistant message as first written, and sees the edits only as its own `edit_message` calls and their results. Unit: `tests/tools/message_edit/test_message_edit.py::test_the_agent_sees_its_call_but_not_the_edit_event`. Example: none. Real-LLM: none. +- `S16.message-edit-llm-view`: `MessageEditEvent` is not an `LLMConvertibleEvent` and is not in `conversation.state.view`; the LLM's next call, like `LLMConvertibleEvent.events_to_messages(...)`, still holds the assistant message as first written, and sees the edits only as its own `edit_message` calls and their results. Unit: `tests/tools/message_edit/test_message_edit.py::test_the_agent_sees_its_call_but_not_the_edit_event`. Example: none. Real-LLM: none. Server: `F39.agent-sees`. - `S16.conversation-mention-link`: `conversation_mention(title, conversation_id)` returns `[Title](conversation:<32-hex id>)` for a UUID or a dashed id string; the title is flattened to one line without brackets, parentheses, quotes, backticks or backslashes, cut to 80 characters, and a blank title reads `untitled conversation`. Unit: `tests/sdk/conversation/test_conversation_mention.py`. Example: none. Real-LLM: none. - `S16.conversation-mention-tools`: `Tool(name=FindConversationsTool.name)` and `Tool(name=ReadConversationTool.name)` (`find_conversations`, `read_conversation`) resolve without params (params raise `ValueError`), are read-only, and add the mention-link guidance to the system prompt. In a local conversation they call the agent server at `OH_INTERNAL_SERVER_URL`: with no server there, each call is an error observation (`Could not list conversations: ...`, `Could not read conversation <hex>: ...`), and `read_conversation` with text that holds no mention or id is an error before any request. Unit: `tests/tools/conversation_mentions/test_conversation_mention_tools.py`. Example: none. Real-LLM: none. -- `S16.ask-user-tool`: `Tool(name=AskUserTool.name)` (`ask_user`, no params) records the question as an `ActionEvent` holding an `AskUserAction(question, options=[AskUserOption(label, consequence)], recommended)`; its observation reads `The question is shown to the user. Your turn ends here; ...`, the run ends `finished` after that one LLM call with no `finish`, and `find_pending_ask` returns the call's id, labels and recommendation, also after the persisted conversation is reloaded. One option or a `recommended` index past the options is an `AgentErrorEvent` (tool validation); a blank label is an error observation (`Each option needs a nonblank label. ...`); neither leaves a question pending nor ends the turn. A second `ask_user` in the same response gets an error observation (`Not shown: this response already ends the turn ...`) and only the first is pending; with `AlwaysConfirm` the question needs no confirmation. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_options_are_checked`, `test_asking_ends_the_turn_and_leaves_the_question_pending`, `test_an_invalid_question_does_not_end_the_turn`, `test_only_the_first_question_in_a_response_is_asked`, `test_asking_needs_no_confirmation_even_when_every_action_does`, `test_the_question_survives_a_restart`). Example: none. Real-LLM: none. -- `S16.ask-user-answer`: `conversation.answer_ask(ask_id, option_index=None, text=None)` raises `ValueError` for an id that is not the pending question (`That question is not waiting for an answer.`), an option that does not exist (`The question has 2 options; there is no option 2.`) and an answer with neither an option nor words, recording nothing; otherwise it records a user `AskUserAnswerEvent(ask_id, option_index, option_label, text)` (words trimmed, parent the call's observation) and returns the message `[Answer to your question: "<question>"]`, `Option <n>: <label>`, then the words, which keeps two equal labels apart. The question stays pending until the program sends that message; the next `run()` answers, and a second tap is refused. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_a_tap_records_the_answer_and_the_message_the_agent_reads`, `test_equal_labels_preserve_which_option_the_user_selected`, `test_an_answer_is_refused_for_another_question_or_option`, `test_a_second_tap_is_refused`, `test_a_tap_whose_message_was_lost_can_be_sent_again`, `test_the_answer_event_round_trips`). Example: none. Real-LLM: none. -- `S16.ask-user-chat-reply`: an ordinary user message answers the question in words and closes it; a message relayed with a `sender` (from another conversation) does not. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_a_chat_reply_answers_the_question`, `test_a_chat_reply_then_runs_cleanly`, `test_a_message_from_another_conversation_does_not_answer`, `test_the_user_s_own_message_is_what_closes_it`). Example: none. Real-LLM: none. -- `S16.ask-user-holds-completion`: a pending question is not consumed by completion logic: a Stop hook that denies completion does not run, and the agent gets no extra turn, after the agent asks (with `run()` and `arun()`), while after the user's answer the same hook runs on the next completion; `ask_user` and `finish` in one response with iterative refinement on end the turn on the question without a refinement message. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_stop_hooks_wait_for_an_answer_but_still_check_completion`, `test_a_question_takes_precedence_over_finish_refinement`). Example: none. Real-LLM: none. +- `S16.ask-user-tool`: `Tool(name=AskUserTool.name)` (`ask_user`, no params) records the question as an `ActionEvent` holding an `AskUserAction(question, options=[AskUserOption(label, consequence)], recommended)`; its observation reads `The question is shown to the user. Your turn ends here; ...`, the run ends `finished` after that one LLM call with no `finish`, and `find_pending_ask` returns the call's id, labels and recommendation, also after the persisted conversation is reloaded. One option or a `recommended` index past the options is an `AgentErrorEvent` (tool validation); a blank label is an error observation (`Each option needs a nonblank label. ...`); neither leaves a question pending nor ends the turn. A second `ask_user` in the same response gets an error observation (`Not shown: this response already ends the turn ...`) and only the first is pending; with `AlwaysConfirm` the question needs no confirmation. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_options_are_checked`, `test_asking_ends_the_turn_and_leaves_the_question_pending`, `test_an_invalid_question_does_not_end_the_turn`, `test_only_the_first_question_in_a_response_is_asked`, `test_asking_needs_no_confirmation_even_when_every_action_does`, `test_the_question_survives_a_restart`). Example: none. Real-LLM: none. Server: `F40.question-event`. +- `S16.ask-user-answer`: `conversation.answer_ask(ask_id, option_index=None, text=None)` raises `ValueError` for an id that is not the pending question (`That question is not waiting for an answer.`), an option that does not exist (`The question has 2 options; there is no option 2.`) and an answer with neither an option nor words, recording nothing; otherwise it records a user `AskUserAnswerEvent(ask_id, option_index, option_label, text)` (words trimmed, parent the call's observation) and returns the message `[Answer to your question: "<question>"]`, `Option <n>: <label>`, then the words, which keeps two equal labels apart. The question stays pending until the program sends that message; the next `run()` answers, and a second tap is refused. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_a_tap_records_the_answer_and_the_message_the_agent_reads`, `test_equal_labels_preserve_which_option_the_user_selected`, `test_an_answer_is_refused_for_another_question_or_option`, `test_a_second_tap_is_refused`, `test_a_tap_whose_message_was_lost_can_be_sent_again`, `test_the_answer_event_round_trips`). Example: none. Real-LLM: none. Server: `F40.answer-errors`. +- `S16.ask-user-chat-reply`: an ordinary user message answers the question in words and closes it; a message relayed with a `sender` (from another conversation) does not. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_a_chat_reply_answers_the_question`, `test_a_chat_reply_then_runs_cleanly`, `test_a_message_from_another_conversation_does_not_answer`, `test_the_user_s_own_message_is_what_closes_it`). Example: none. Real-LLM: none. Server: `F40.chat-reply`. +- `S16.ask-user-holds-completion`: a pending question is not consumed by completion logic: a Stop hook that denies completion does not run, and the agent gets no extra turn, after the agent asks (with `run()` and `arun()`), while after the user's answer the same hook runs on the next completion; `ask_user` and `finish` in one response with iterative refinement on end the turn on the question without a refinement message. Unit: `tests/tools/ask_user/test_ask_user.py` (`test_stop_hooks_wait_for_an_answer_but_still_check_completion`, `test_a_question_takes_precedence_over_finish_refinement`). Example: none. Real-LLM: none. Server: `F40.stop-hook-waits`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S17-custom-and-client-tools.md b/.agents/skills/verify-sdk/references/feature-map/S17-custom-and-client-tools.md index 8beb6ccb0a..4e277bce6d 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S17-custom-and-client-tools.md +++ b/.agents/skills/verify-sdk/references/feature-map/S17-custom-and-client-tools.md @@ -12,27 +12,27 @@ Examples: `examples/01_standalone_sdk/02_custom_tools.py`, `examples/01_standalo - `S17.custom-tool-definition`: a `ToolDefinition` built from an `Action`, an `Observation` and an executor validates actions, calls the executor, checks the returned observation type, and exposes `as_executable()`; calling a tool without an executor raises. Unit: `tests/sdk/tool/test_tool_definition.py`. Example: `examples/01_standalone_sdk/02_custom_tools.py` (`GrepTool`). Real-LLM: none. - `S17.tool-set`: one registered name whose `create()` returns several tools sharing state (`BashAndGrepToolSet` returns a terminal and a grep tool on one `TerminalExecutor`); every returned tool is added to the agent. Unit: `tests/tools/browser_use/test_browser_toolset.py::test_browser_toolset_create_tools_have_shared_executor` (the built-in instance of the pattern). Example: `examples/01_standalone_sdk/02_custom_tools.py`. Real-LLM: none. -- `S17.register-tool`: `register_tool(name, factory)` accepts a `ToolDefinition` subclass (resolved through `create(**params)`) or a ready instance with an executor (params rejected, same object returned); a bare callable is rejected, an empty name raises `ValueError`, and a duplicate name logs a warning and replaces the old entry. Unit: `tests/sdk/tool/test_registry.py` (`test_register_tool_rejects_callable_factory`, `test_register_tool_instance_rejects_params`, `test_register_tool_instance_returns_same_object`, `test_register_tool_type_uses_create_params`, `test_tool_registration_typed_inspection`). Example: `examples/01_standalone_sdk/02_custom_tools.py`. Real-LLM: none. +- `S17.register-tool`: `register_tool(name, factory)` accepts a `ToolDefinition` subclass (resolved through `create(**params)`) or a ready instance with an executor (params rejected, same object returned); a bare callable is rejected, an empty name raises `ValueError`, and a duplicate name logs a warning and replaces the old entry. Unit: `tests/sdk/tool/test_registry.py` (`test_register_tool_rejects_callable_factory`, `test_register_tool_instance_rejects_params`, `test_register_tool_instance_returns_same_object`, `test_register_tool_type_uses_create_params`, `test_tool_registration_typed_inspection`). Example: `examples/01_standalone_sdk/02_custom_tools.py`. Real-LLM: none. Server: `F27.custom-tool-modules`, `F33.custom-tool-module`. - `S17.resolve-tool`: `resolve_tool(Tool(name=..., params=...), conv_state)` passes `params` to `create()` with the conversation state (working dir, persistence dir), falls back to built-ins under their snake_case names, applies a `response_schema` param to a single tool, and raises `KeyError` for an unregistered name. Unit: `tests/cross/test_registry_directories.py`, `tests/sdk/tool/test_registry.py::test_builtin_resolves_under_its_snake_case_name`, `tests/sdk/tool/test_response_schema.py::test_builtin_finish_tool_spec_resolves_without_registry_entry` (the unregistered-name `KeyError` has no test). Example: none. Real-LLM: none. - `S17.auto-naming`: `ToolDefinition.name` is derived from the class name (`FileEditorTool` to `file_editor`) unless the class sets `name`. Unit: `tests/cross/test_automatic_naming.py`. Example: `examples/01_standalone_sdk/02_custom_tools.py` (`BashAndGrepToolSet.name`). Real-LLM: none. - `S17.auto-registration`: importing a built-in tool module (or `openhands.tools`) registers its tools, after which specs resolve. Unit: `tests/cross/test_automatic_registration.py`, `tests/tools/test_init.py`. Example: none. Real-LLM: none. - `S17.tool-spec`: `Tool(name, params)` copies its `params` (later changes to the caller's dict do not reach it), round-trips through JSON, and rejects an empty name; it is not frozen (attributes can be reassigned). Unit: `tests/sdk/tool/test_tool.py`. Example: `examples/01_standalone_sdk/02_custom_tools.py`. Real-LLM: none. - `S17.agent-tool-init`: the agent turns its specs into tools at conversation start, adds the built-in `finish` (and `think` only when `include_default_tools` opts into it), rejects duplicate tool names, and does not let a runtime tool replace an existing one; a custom tool may replace `finish`. Unit: `tests/sdk/agent/test_agent_tool_init.py`. Example: `examples/01_standalone_sdk/02_custom_tools.py`. Real-LLM: none. - `S17.llm-schemas`: `to_openai_tool()`, `to_responses_tool()` (with `strict` and parameters) and `to_mcp_tool()` render the action schema without the internal `kind` field, keep required fields and annotations, and expand nested and circular `$ref`s without recursion errors. Unit: `tests/sdk/tool/test_mcp_schema.py`, `tests/sdk/tool/test_to_responses_tool.py`, `tests/sdk/tool/test_tool_definition.py` (`test_to_mcp_tool_*`, `test_mcp_tool_schema_required_fields`). Example: none. Real-LLM: none. -- `S17.schema-constraints`: the tool schema sent to the model keeps the JSON Schema constraints the `Action` model declares: a field's `pattern`, a one-value `Literal` (`const`), and `additionalProperties: false` from `extra="forbid"`. Unit: **none (untested)**. Example: none. Real-LLM: none. +- `S17.schema-constraints`: the tool schema sent to the model keeps the JSON Schema constraints the `Action` model declares: a field's `pattern`, a one-value `Literal` (`const`), and `additionalProperties: false` from `extra="forbid"`. Unit: **none (untested)**. Example: none. Real-LLM: none. Server: `F27.tool-schema-keywords`. - `S17.injected-fields`: a `security_risk` parameter is added to tools that are not read-only (optional, ahead of content parameters, gated by the security setting) and a `summary` parameter is always added, while an MCP tool's own description is kept. Unit: `tests/sdk/tool/test_tool_definition.py` (`test_security_risk_only_added_for_non_readonly_tools`, `test_security_risk_is_optional_field_in_schema`, `test_security_risk_precedes_content_params_in_schema`), `tests/sdk/tool/test_to_responses_tool_security.py`, `tests/sdk/tool/test_to_responses_tool_summary.py`. Example: none. Real-LLM: none. - `S17.schema-serialization`: `Action`/`Observation` subclasses are frozen and serialize as a discriminated union by `kind`, including nested polymorphic fields; tool definitions are immutable and `set_executor` returns a copy. Unit: `tests/sdk/tool/test_tool_serialization.py`, `tests/sdk/tool/test_schema_immutability.py`, `tests/sdk/tool/test_tool_immutability.py`. Example: none. Real-LLM: none. - `S17.py-type`: `py_type()` maps JSON-schema types (string, integer, number, boolean, object, arrays, nullable unions) to Python types for dynamically built actions. Unit: `tests/sdk/tool/test_py_type.py`. Example: none. Real-LLM: none. - `S17.output-coercion`: odd executor results (an observation with `None` content and other shapes) still become a valid tool result message. Unit: `tests/sdk/tool/test_tool_call_output_coercion.py`. Example: none. Real-LLM: none. - `S17.annotations-and-resources`: `ToolAnnotations` (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) travel with the tool, and `declared_resources(action)` returns `DeclaredResources` keys that the parallel executor locks on. Unit: `tests/sdk/tool/test_tool_definition.py::test_tool_creation_with_annotations`, `tests/sdk/agent/test_parallel_executor_locking.py`. Example: `examples/01_standalone_sdk/45_parallel_tool_execution.py`. Real-LLM: none. -- `S17.usability`: a tool class's `is_usable()` decides `is_tool_usable(name)`, `is_tool_available(name)` and `list_usable_tools()`; a checker that raises counts as unusable. Unit: `tests/sdk/tool/test_registry.py::test_register_tool_type_respects_is_usable`, `tests/sdk/tool/test_defaults.py::test_is_tool_usable_contract`. Example: none. Real-LLM: none. -- `S17.catalog`: `list_tool_catalog(check_usable=...)` lists offered tools with `user_selectable`, `usable`, `description` and `in_default_set`, offers built-ins once under their snake_case names, hides the profile router, and after `seal_tool_catalog()` ignores later registrations until `unseal_tool_catalog()`. Unit: `tests/sdk/tool/test_registry.py` (`test_catalog_*`, `test_sealed_catalog_*`, `test_unsealed_catalog_tracks_registrations_again`). Example: none. Real-LLM: none. -- `S17.module-qualnames`: `get_tool_module_qualnames()` maps each registered tool name to the module that defined it (a copy), which a remote conversation forwards so the server imports the same modules. Unit: `tests/cross/test_registry_qualnames.py`. Example: `examples/02_remote_agent_server/06_custom_tool/main.py` (server side). Real-LLM: none. +- `S17.usability`: a tool class's `is_usable()` decides `is_tool_usable(name)`, `is_tool_available(name)` and `list_usable_tools()`; a checker that raises counts as unusable. Unit: `tests/sdk/tool/test_registry.py::test_register_tool_type_respects_is_usable`, `tests/sdk/tool/test_defaults.py::test_is_tool_usable_contract`. Example: none. Real-LLM: none. Server: `F27.catalog`, `F27.browser-probe`. +- `S17.catalog`: `list_tool_catalog(check_usable=...)` lists offered tools with `user_selectable`, `usable`, `description` and `in_default_set`, offers built-ins once under their snake_case names, hides the profile router, and after `seal_tool_catalog()` ignores later registrations until `unseal_tool_catalog()`. Unit: `tests/sdk/tool/test_registry.py` (`test_catalog_*`, `test_sealed_catalog_*`, `test_unsealed_catalog_tracks_registrations_again`). Example: none. Real-LLM: none. Server: `F27.catalog`, `F27.catalog-sealed`. +- `S17.module-qualnames`: `get_tool_module_qualnames()` maps each registered tool name to the module that defined it (a copy), which a remote conversation forwards so the server imports the same modules. Unit: `tests/cross/test_registry_qualnames.py`. Example: `examples/02_remote_agent_server/06_custom_tool/main.py` (server side). Real-LLM: none. Server: `F27.custom-tool-modules`, `F33.custom-tool-module`. - `S17.client-tool-spec`: `ClientToolSpec(name, description, parameters, annotations)` round-trips through JSON and requires an object `parameters` schema. Unit: `tests/sdk/tool/test_client_tool.py` (`test_spec_*`). Example: `examples/01_standalone_sdk/53_client_defined_tools.py`. Real-LLM: none. -- `S17.client-tool-ack`: a `ClientTool` built from a spec gets a dynamic `ClientAction_<name>` action type, its executor returns a `ClientToolObservation` acknowledgment without doing anything, and its `ActionEvent` round-trips through JSON (including an argument named `kind`). Unit: `tests/sdk/tool/test_client_tool.py` (`test_executor_returns_acknowledgment`, `test_from_spec_*`, `test_client_tool_callable`, `test_client_tool_kind_argument_*`, `test_create_*`). Example: `examples/01_standalone_sdk/53_client_defined_tools.py`. Real-LLM: none. +- `S17.client-tool-ack`: a `ClientTool` built from a spec gets a dynamic `ClientAction_<name>` action type, its executor returns a `ClientToolObservation` acknowledgment without doing anything, and its `ActionEvent` round-trips through JSON (including an argument named `kind`). Unit: `tests/sdk/tool/test_client_tool.py` (`test_executor_returns_acknowledgment`, `test_from_spec_*`, `test_client_tool_callable`, `test_client_tool_kind_argument_*`, `test_create_*`). Example: `examples/01_standalone_sdk/53_client_defined_tools.py`. Real-LLM: none. Server: `F27.client-tool-call`, `F33.client-tools`. - `S17.client-tool-schema`: the OpenAI schema keeps the client's constraints (enums, required), `security_risk` is injected when annotations are unset, and the MCP view uses the original schema. Unit: `tests/sdk/tool/test_client_tool.py` (`test_to_openai_tool`, `test_openai_schema_preserves_client_constraints`, `test_security_risk_injected_when_annotations_unset`, `test_mcp_tool_uses_original_schema`). Example: none. Real-LLM: none. - `S17.client-tool-registration`: `register_client_tools(specs)` returns `Tool` specs carrying the schema in `params`, reuses the action type for an identical spec, raises `ClientToolSchemaConflictError` for the same name with a different schema, and `ClientToolRegistrationError` for duplicate names in one call or a built-in tool name. Unit: `tests/sdk/tool/test_client_tool.py` (`test_same_spec_reuses_action_type`, `test_same_name_different_schema_conflicts`, `test_register_client_tools_*`), `tests/cross/test_client_tool_registration.py`. Example: none. Real-LLM: none. -- `S17.client-tools-conversation`: `Conversation(..., client_tools=SPECS)` registers the specs and injects them into the agent; a persisted conversation resumes with its client tools without passing them again; a client tool named like a built-in the agent does not attach still resumes, one named like an attached built-in is rejected. Unit: `tests/sdk/conversation/local/test_client_tools_persistence.py`. Example: `examples/01_standalone_sdk/53_client_defined_tools.py`. Real-LLM: none. +- `S17.client-tools-conversation`: `Conversation(..., client_tools=SPECS)` registers the specs and injects them into the agent; a persisted conversation resumes with its client tools without passing them again; a client tool named like a built-in the agent does not attach still resumes, one named like an attached built-in is rejected. Unit: `tests/sdk/conversation/local/test_client_tools_persistence.py`. Example: `examples/01_standalone_sdk/53_client_defined_tools.py`. Real-LLM: none. Server: `F27.client-tool-call`, `F33.client-tools`. - `S17.execute-tool`: `conversation.execute_tool(name, action)` runs a resolved tool outside the agent loop and returns its observation. Unit: `tests/sdk/tool/test_switch_llm.py`, `tests/tools/ask_oracle/test_ask_oracle.py`. Example: `examples/01_standalone_sdk/38_browser_session_recording.py`. Real-LLM: none. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S18-workspace-backends.md b/.agents/skills/verify-sdk/references/feature-map/S18-workspace-backends.md index 6655929aad..ed8b295bd1 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S18-workspace-backends.md +++ b/.agents/skills/verify-sdk/references/feature-map/S18-workspace-backends.md @@ -10,19 +10,19 @@ Examples: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server ## Sub-features -- `S18.workspace-factory`: `Workspace(working_dir=...)` returns a `LocalWorkspace` and `Workspace(host=..., api_key=...)` a `RemoteWorkspace`; `working_dir` accepts `str` or `Path`. Unit: **none (untested)**; nothing calls the `Workspace` factory. Example: none. Real-LLM: none. +- `S18.workspace-factory`: `Workspace(working_dir=...)` returns a `LocalWorkspace` and `Workspace(host=..., api_key=...)` a `RemoteWorkspace`; `working_dir` accepts `str` or `Path`. Unit: **none (untested)**; nothing calls the `Workspace` factory. Example: none. Real-LLM: none. Server: `F33.workspace-exec`. - `S18.local-ops`: `LocalWorkspace.execute_command(cmd, cwd=None, timeout=...)` returns a `CommandResult` (`exit_code`, `stdout`, `stderr`, `timeout_occurred`), `file_upload`/`file_download` copy files (creating parent directories) and return a `FileOperationResult`, and `git_changes`/`git_diff` report the repository state. Unit: **none (untested)** for the workspace methods; the git helpers they call are covered by `tests/sdk/git/test_git_changes.py` and `tests/sdk/git/test_git_diff.py`. Example: none. Real-LLM: none. - `S18.local-pause-resume`: `LocalWorkspace.pause()` and `resume()` do nothing and do not raise. Unit: `tests/workspace/test_workspace_pause_resume.py` (`test_local_workspace_pause_is_noop`, `test_local_workspace_resume_is_noop`). Example: none. Real-LLM: none. -- `S18.conversation-binding`: `Conversation(workspace=<path or LocalWorkspace>)` builds a `LocalConversation`, `Conversation(workspace=<RemoteWorkspace subclass>)` a `RemoteConversation`, with parameters forwarded to each. Unit: `tests/sdk/conversation/test_conversation_factory.py`. Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py`. Real-LLM: none. +- `S18.conversation-binding`: `Conversation(workspace=<path or LocalWorkspace>)` builds a `LocalConversation`, `Conversation(workspace=<RemoteWorkspace subclass>)` a `RemoteConversation`, with parameters forwarded to each. Unit: `tests/sdk/conversation/test_conversation_factory.py`. Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py`. Real-LLM: none. Server: `F33.conversation-create`. - `S18.completion-callback`: on context exit any workspace POSTs `{status, run_id, conversation_id, cost, error}` to `AUTOMATION_CALLBACK_URL` (Bearer `AUTOMATION_CALLBACK_API_KEY`, `AUTOMATION_RUN_ID`), with the cost registered through `register_cost` and the conversation id from `register_conversation`; it is skipped without the URL and swallows errors. Unit: `tests/sdk/workspace/test_local_workspace.py`, `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_send_completion_callback_*`, `test_register_cost_stores_cost`), `tests/workspace/test_api_remote_workspace.py::test_api_remote_workspace_exit_sends_callback`, `tests/workspace/test_cloud_workspace.py` (`test_callback_*`). Example: none. Real-LLM: none. -- `S18.remote-client`: `RemoteWorkspace` normalizes `host`, creates its `httpx` client lazily with `base_url`, sends `X-Session-API-Key` when `api_key` is set plus observability headers, and can `reset_client()`. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py`, `tests/sdk/workspace/remote/test_remote_workspace_mixin.py`, `tests/sdk/workspace/remote/test_client_base_url.py`. Example: `examples/02_remote_agent_server/01_convo_with_local_agent_server.py`. Real-LLM: none. -- `S18.remote-command-exec`: `execute_command` starts `start_bash_command`, polls `bash_events/search` filtered by its command id until an exit code or the timeout, decodes base64 output, and never repeats an output chunk across polls; concurrent commands do not mix output. Unit: `tests/sdk/workspace/remote/test_remote_workspace_mixin.py` (`test_execute_command_generator_*`), `tests/sdk/workspace/remote/test_polling_duplicates_output.py`, `tests/sdk/workspace/remote/test_multiple_commands_isolation.py`. Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py` (`workspace.execute_command(...)`). Real-LLM: none. -- `S18.remote-files-git`: `file_upload`, `file_download`, `git_changes` and `git_diff` go through the server's file and git routes and accept `Path` arguments. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_file_upload`, `test_file_download`, `test_file_operations_with_path_objects`), `tests/sdk/workspace/remote/test_runtime_scope.py`. Example: none. Real-LLM: none. -- `S18.runtime-scope`: a workspace bound to a conversation sends command, file and git calls to `/api/conversations/{id}/...` with its runtime session key, can `release_runtime()`, closes its async client even when the body fails, and rejects a missing runtime credential. Unit: `tests/sdk/workspace/remote/test_runtime_scope.py`. Example: none. Real-LLM: none. +- `S18.remote-client`: `RemoteWorkspace` normalizes `host`, creates its `httpx` client lazily with `base_url`, sends `X-Session-API-Key` when `api_key` is set plus observability headers, and can `reset_client()`. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py`, `tests/sdk/workspace/remote/test_remote_workspace_mixin.py`, `tests/sdk/workspace/remote/test_client_base_url.py`. Example: `examples/02_remote_agent_server/01_convo_with_local_agent_server.py`. Real-LLM: none. Server: `F33.workspace-bad-key`. +- `S18.remote-command-exec`: `execute_command` starts `start_bash_command`, polls `bash_events/search` filtered by its command id until an exit code or the timeout, decodes base64 output, and never repeats an output chunk across polls; concurrent commands do not mix output. Unit: `tests/sdk/workspace/remote/test_remote_workspace_mixin.py` (`test_execute_command_generator_*`), `tests/sdk/workspace/remote/test_polling_duplicates_output.py`, `tests/sdk/workspace/remote/test_multiple_commands_isolation.py`. Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py` (`workspace.execute_command(...)`). Real-LLM: none. Server: `F33.workspace-exec`, `F33.workspace-background`, `F13.output-chunks`. +- `S18.remote-files-git`: `file_upload`, `file_download`, `git_changes` and `git_diff` go through the server's file and git routes and accept `Path` arguments. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_file_upload`, `test_file_download`, `test_file_operations_with_path_objects`), `tests/sdk/workspace/remote/test_runtime_scope.py`. Example: none. Real-LLM: none. Server: `F33.workspace-files`, `F33.workspace-git`, `F14.sdk-workspace`, `F14.upload-size`, `F15.sdk-workspace`. +- `S18.runtime-scope`: a workspace bound to a conversation sends command, file and git calls to `/api/conversations/{id}/...` with its runtime session key, can `release_runtime()`, closes its async client even when the body fails, and rejects a missing runtime credential. Unit: `tests/sdk/workspace/remote/test_runtime_scope.py`. Example: none. Real-LLM: none. Server: `F33.workspace-runtime-scope`, `F33.workspace-runtime-lifecycle`, `F13.scoped-cwd`. - `S18.remote-alive`: `alive` is true on a 2xx from the server's health route and false on server errors, client errors, connection errors and timeouts. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_alive_*`). Example: none. Real-LLM: none. -- `S18.async-remote`: `AsyncRemoteWorkspace` offers the same command, file and git operations as coroutines under `async with`. Unit: `tests/sdk/workspace/remote/test_async_remote_workspace.py`, `tests/sdk/workspace/remote/test_runtime_scope.py`. Example: none. Real-LLM: none. -- `S18.remote-settings`: `workspace.get_llm(profile_name=None, **overrides)` builds an `LLM` from the server's active profile (or a named one, or legacy settings), `get_secrets(names=None)` returns `LookupSecret` references, `get_mcp_config()` returns MCP servers, and `_get_secret_value` returns `None` on 404. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_get_llm_*`, `test_get_secrets_*`, `test_get_mcp_config_*`, `test_get_secret_value_*`). Example: `examples/02_remote_agent_server/13_workspace_get_llm.py`. Real-LLM: none. -- `S18.repos-and-skills`: `clone_repos(repos, target_dir=...)` accepts URLs, `owner/repo` with a provider, or dicts (`RepoSource`), clones with provider tokens and returns `RepoMapping`s; `get_repos_context()` renders them for the prompt; `load_skills_from_agent_server()` fetches skills and falls back to public skills. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_clone_repos_*`, `test_get_repos_context_*`, `test_load_skills_from_agent_server_*`), `tests/workspace/test_cloud_workspace_repos.py`. Example: none. Real-LLM: none. +- `S18.async-remote`: `AsyncRemoteWorkspace` offers the same command, file and git operations as coroutines under `async with`. Unit: `tests/sdk/workspace/remote/test_async_remote_workspace.py`, `tests/sdk/workspace/remote/test_runtime_scope.py`. Example: none. Real-LLM: none. Server: `F33.async-workspace`. +- `S18.remote-settings`: `workspace.get_llm(profile_name=None, **overrides)` builds an `LLM` from the server's active profile (or a named one, or legacy settings), `get_secrets(names=None)` returns `LookupSecret` references, `get_mcp_config()` returns MCP servers, and `_get_secret_value` returns `None` on 404. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_get_llm_*`, `test_get_secrets_*`, `test_get_mcp_config_*`, `test_get_secret_value_*`). Example: `examples/02_remote_agent_server/13_workspace_get_llm.py`. Real-LLM: none. Server: `F33.workspace-settings`, `F20.sdk-get-llm`, `F18.sdk-remote-workspace`. +- `S18.repos-and-skills`: `clone_repos(repos, target_dir=...)` accepts URLs, `owner/repo` with a provider, or dicts (`RepoSource`), clones with provider tokens and returns `RepoMapping`s; `get_repos_context()` renders them for the prompt; `load_skills_from_agent_server()` fetches skills and falls back to public skills. Unit: `tests/sdk/workspace/remote/test_remote_workspace.py` (`test_clone_repos_*`, `test_get_repos_context_*`, `test_load_skills_from_agent_server_*`), `tests/workspace/test_cloud_workspace_repos.py`. Example: none. Real-LLM: none. Server: `F18.sdk-clone-repos`, `F25.sdk-load-skills`. - `S18.docker-start`: `DockerWorkspace(server_image=..., host_port=..., forward_env=[...], volumes=[...], network=..., platform=..., enable_gpu=..., extra_ports=..., health_check_timeout=120)` runs the container, publishes ports on 127.0.0.1, forwards `SESSION_API_KEY`/`OH_SESSION_API_KEYS_0`, streams logs, and waits for health; the removed `mount_dir` argument fails loudly. Unit: `tests/workspace/test_docker_workspace.py`. Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py`. Real-LLM: none. - `S18.docker-dev-build`: `DockerDevWorkspace(base_image=..., target="binary"|"source")` builds the agent-server image from a base image before starting; importing `DockerWorkspace` does not import the build code. Unit: `tests/workspace/test_docker_workspace.py` (`test_docker_dev_workspace_*`, `test_docker_workspace_no_build_import`). Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py` (commented alternative). Real-LLM: none. - `S18.docker-lifecycle`: `pause()`/`resume()` run `docker pause`/`docker unpause` (resume waits for health again) and raise without a container; `cleanup()` (on exit) stops and removes the container and deletes the image only with `cleanup_image=True`. Unit: `tests/workspace/test_workspace_pause_resume.py` (`test_docker_workspace_*`), `tests/workspace/test_docker_workspace.py` (`test_cleanup_without_image_deletion`, `test_cleanup_with_image_deletion`, `test_docker_workspace_resume_uses_health_check_timeout`). Example: `examples/02_remote_agent_server/02_convo_with_docker_sandboxed_server.py`. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S19-settings.md b/.agents/skills/verify-sdk/references/feature-map/S19-settings.md index ba01344800..595d5b53b4 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S19-settings.md +++ b/.agents/skills/verify-sdk/references/feature-map/S19-settings.md @@ -10,32 +10,32 @@ Examples: `examples/01_standalone_sdk/46_agent_settings.py` ## Sub-features -- `S19.settings-create-agent`: `OpenHandsAgentSettings(...).create_agent()` returns an `Agent` with the settings' LLM, tools, MCP config and agent context. Unit: `tests/sdk/test_settings.py::test_llm_create_agent_uses_settings_llm_and_tools`. Example: `examples/01_standalone_sdk/46_agent_settings.py`. Real-LLM: none. +- `S19.settings-create-agent`: `OpenHandsAgentSettings(...).create_agent()` returns an `Agent` with the settings' LLM, tools, MCP config and agent context. Unit: `tests/sdk/test_settings.py::test_llm_create_agent_uses_settings_llm_and_tools`. Example: `examples/01_standalone_sdk/46_agent_settings.py`. Real-LLM: none. Server: `F04.create-settings`. - `S19.settings-roundtrip`: settings dump to JSON and `model_validate` restores them, preserving the LLM model, condenser fields, disabled MCP servers and secrets held by a validated instance. Unit: `tests/sdk/test_settings.py::test_llm_roundtrip_preserves_llm_model`, `::test_disabled_mcp_server_survives_settings_round_trip`. Example: `examples/01_standalone_sdk/46_agent_settings.py`. Real-LLM: none. - `S19.settings-tools`: `tools=None` gives the standard tool set, `tools=[]` gives a bare agent, duplicates collapse, built-ins are matched by class name and attached once, and a registered tool named like a built-in is not replaced. Unit: `tests/sdk/test_settings.py::test_create_agent_defaults_tools_when_none`, `::test_create_agent_empty_tools_stays_bare`, `::test_create_agent_collapses_duplicate_tools`, `::test_builtin_named_twice_is_attached_once`. Example: `examples/01_standalone_sdk/46_agent_settings.py` (terminal-only variant). Real-LLM: none. - `S19.retired-switches`: the retired booleans `enable_sub_agents` and `enable_switch_llm_tool` (`RETIRED_TOOL_SWITCHES` in `openhands-sdk/openhands/sdk/tool/defaults.py`) are refused as input since schema v9 (`Removed agent settings: ...` from code, request payloads and diffs); persisted v6 payloads still fold them into `tools` through the v7 to v8 step, v8 payloads drop them, and they are never serialized again. Unit: `tests/sdk/test_settings.py::test_removed_switches_are_rejected_by_settings_and_diffs`, `::test_retired_tool_switches_are_not_serialized`, `::test_v6_settings_fold_the_retired_switches`, `::test_unversioned_settings_get_no_retired_switch_defaults`. Example: none. Real-LLM: none. - `S19.condenser-settings`: `LLMSummarizingCondenserSettings(enabled=True, max_size=...)` builds an `LLMSummarizingCondenser` whose fields match the condenser's, inheriting `max_tokens` from the LLM unless set; `enabled=False` gives no condenser; `NoOpCondenserSettings` builds a `NoOpCondenser`; a legacy condenser payload defaults to the summarizing variant. Unit: `tests/sdk/test_settings.py::test_llm_create_agent_builds_condenser_when_enabled`, `::test_llm_summarizing_condenser_settings_match_condenser_fields`, `::test_llm_create_agent_builds_no_op_condenser_variant`, `::test_llm_create_agent_no_condenser_when_disabled`. Example: `examples/01_standalone_sdk/46_agent_settings.py`. Real-LLM: none. - `S19.verification-settings`: `VerificationSettings(critic_enabled=True, ...)` builds a critic (with iterative refinement when configured) only when a critic API key or the LLM key is available; the key round-trips. Unit: `tests/sdk/test_settings.py::test_llm_create_agent_builds_critic_when_enabled`, `::test_llm_create_agent_no_critic_without_api_key`, `::test_llm_create_agent_critic_with_iterative_refinement`. Example: none. Real-LLM: none. -- `S19.mcp-config-secrets`: `mcp_config` is a typed server map; env, headers and bearer auth are redacted in dumps, encrypted with a cipher when one is supplied, legacy plaintext decrypts, and `create_agent()` gets the real values. Unit: `tests/sdk/test_settings.py::test_openhands_agent_settings_mcp_config_redacts_env_and_headers`, `::test_mcp_config_encrypts_env_and_headers_with_cipher`, `::test_openhands_agent_settings_create_agent_keeps_real_mcp_secrets`. Example: none. Real-LLM: none. +- `S19.mcp-config-secrets`: `mcp_config` is a typed server map; env, headers and bearer auth are redacted in dumps, encrypted with a cipher when one is supplied, legacy plaintext decrypts, and `create_agent()` gets the real values. Unit: `tests/sdk/test_settings.py::test_openhands_agent_settings_mcp_config_redacts_env_and_headers`, `::test_mcp_config_encrypts_env_and_headers_with_cipher`, `::test_openhands_agent_settings_create_agent_keeps_real_mcp_secrets`. Example: none. Real-LLM: none. Server: `F19.conversation-secrets-at-rest`, `F18.encrypted-at-rest`. - `S19.tool-concurrency-persona`: `tool_concurrency_limit` defaults to 1 and rejects invalid values; `persona` is applied to the built agent's prompt, and without it the built-in prompt is used. Unit: `tests/sdk/test_settings.py::test_tool_concurrency_limit_invalid_values_rejected`, `::test_create_agent_applies_persona`. Example: none. Real-LLM: none. - `S19.subscription-llm-settings`: subscription-auth LLM fields round-trip through settings and are rehydrated into a working runtime LLM on load. Unit: `tests/sdk/test_settings.py::test_llm_subscription_fields_roundtrip`, `::test_llm_from_persisted_rehydrates_subscription_runtime`. Example: none. Real-LLM: none. - `S19.acp-settings`: `ACPAgentSettings(acp_server=...)` resolves the default command for known servers (rewriting `npx` to a pinned binary when present), requires an explicit command for a custom server, refuses an `llm` (`ACPAgentSettings.llm was removed; use acp_model instead.`; a persisted v8 ACP payload drops it on migration), and maps the provider's API-key env var. Unit: `tests/sdk/test_settings.py::test_acp_create_agent_uses_server_default_command`, `::test_acp_custom_server_requires_explicit_command`, `::test_acp_settings_llm_is_removed`. Example: none. Real-LLM: none. - `S19.acp-providers-catalog`: the ACP provider registry (`ACP_PROVIDERS`, detection by agent name or command, file-secret specs) and the install catalog of pinned npm packages with a rendered Docker install plan. Unit: `tests/sdk/settings/test_acp_providers.py`, `tests/sdk/settings/test_acp_install_catalog.py`. Example: none. Real-LLM: none. - `S19.validate-dispatch`: `validate_agent_settings` dispatches on `agent_kind` (`openhands` or `acp`), defaults to `openhands` when it is missing, and canonicalizes the legacy `llm` kind so it never returns `LLMAgentSettings`. Unit: `tests/sdk/test_settings.py::test_validate_agent_settings_dispatches_on_agent_kind`, `::test_validate_agent_settings_defaults_to_openhands_when_discriminator_missing`, `tests/sdk/test_apply_agent_settings_diff.py::test_validate_never_returns_llm_subclass`. Example: none. Real-LLM: none. - `S19.schema-migrations`: persisted agent settings from v0 to v9 migrate step by step to v10 (legacy proxy LLM, `modify_params`, MCP auth shapes and `type` to `transport`, ACP LLM removal, retired switches folded at v8 and dropped at v9, MCP servers with neither `command` nor `url` dropped at v10); a newer, negative or non-integer `schema_version` is rejected; `from_persisted` and `model_validate` agree on current payloads. Unit: `tests/sdk/test_settings.py::test_validate_agent_settings_migrates_v0_llm_payload`, `::test_validate_agent_settings_rejects_newer_schema_version`, `::test_openhands_agent_settings_from_persisted_migrates_legacy_payloads`. Example: none. Real-LLM: none. -- `S19.conversation-settings`: `ConversationSettings` (schema v1) round-trips, migrates a v0 payload, rejects newer versions, validates observability metadata, accepts both agent variants, and `create_request(...)` builds a start-conversation request with confirmation policy and security analyzer. Unit: `tests/sdk/test_settings.py::test_conversation_settings_model_dump_roundtrip`, `::test_conversation_settings_from_persisted_migrates_v0_payload`, `::test_conversation_settings_create_request`. Example: none. Real-LLM: none. -- `S19.apply-diff`: `apply_agent_settings_diff(base, diff)` deep-merges within a variant, treats `None` as unset (merge patch), replaces the base with a fresh variant when `agent_kind` switches (dropping `persona` for ACP), and keeps secrets from the base. Unit: `tests/sdk/test_apply_agent_settings_diff.py`. Example: none. Real-LLM: none. -- `S19.schema-export`: `export_agent_settings_schema()` and `export_settings_schema(Model)` emit variant-tagged sections with field metadata (label, prominence), stay backward compatible, and work on both variants. Unit: `tests/sdk/test_settings.py::test_export_agent_settings_schema_emits_variant_tagged_sections`, `::test_export_settings_schema_remains_backward_compatible`, `::test_llm_agent_settings_export_schema_groups_sections`. Example: none. Real-LLM: none. +- `S19.conversation-settings`: `ConversationSettings` (schema v1) round-trips, migrates a v0 payload, rejects newer versions, validates observability metadata, accepts both agent variants, and `create_request(...)` builds a start-conversation request with confirmation policy and security analyzer. Unit: `tests/sdk/test_settings.py::test_conversation_settings_model_dump_roundtrip`, `::test_conversation_settings_from_persisted_migrates_v0_payload`, `::test_conversation_settings_create_request`. Example: none. Real-LLM: none. Server: `F18.conversation-diff`. +- `S19.apply-diff`: `apply_agent_settings_diff(base, diff)` deep-merges within a variant, treats `None` as unset (merge patch), replaces the base with a fresh variant when `agent_kind` switches (dropping `persona` for ACP), and keeps secrets from the base. Unit: `tests/sdk/test_apply_agent_settings_diff.py`. Example: none. Real-LLM: none. Server: `F18.agent-diff-merge`, `F18.agent-kind-switch`. +- `S19.schema-export`: `export_agent_settings_schema()` and `export_settings_schema(Model)` emit variant-tagged sections with field metadata (label, prominence), stay backward compatible, and work on both variants. Unit: `tests/sdk/test_settings.py::test_export_agent_settings_schema_emits_variant_tagged_sections`, `::test_export_settings_schema_remains_backward_compatible`, `::test_llm_agent_settings_export_schema_groups_sections`. Example: none. Real-LLM: none. Server: `F18.agent-schema`, `F18.conversation-schema`. - `S19.persisted-baselines`: every `tests/sdk/persisted_settings_baselines/vN/*.json` (agent settings, agent profiles, conversation settings, agent-server persisted settings) loads with the current code and keeps its sentinel values; a shape change without a version bump, or a removed profile field, is rejected; every schema version must have a fixture. Unit: `tests/cross/test_check_persisted_settings_compat.py`. Example: none. Real-LLM: none. - `S19.pypi-baseline-gate`: the CI script generates payloads with the previous PyPI release of `openhands-sdk` and `openhands-agent-server` (via `uv`, with a release-date cutoff) and validates them against the current code. Unit: `tests/cross/test_check_persisted_settings_compat.py::test_generate_baseline_payloads_uses_uv_with_release_cutoff`, `::test_get_pypi_baseline_version_prefers_current_or_previous` (mocked). Example: none. Real-LLM: none. -- `S19.profile-model`: `OpenHandsAgentProfile` requires `llm_profile_ref`, `ACPAgentProfile` requires a known ACP server and rejects LLM refs and persona; both carry a UUID `id`, `name`, `revision`, `mcp_server_refs` (null means all, `[]` means none), `secret_refs`, and never persist secrets. Unit: `tests/sdk/profiles/test_agent_profile.py`, `tests/sdk/profiles/test_secret_refs.py`. Example: none. Real-LLM: none. +- `S19.profile-model`: `OpenHandsAgentProfile` requires `llm_profile_ref`, `ACPAgentProfile` requires a known ACP server and rejects LLM refs and persona; both carry a UUID `id`, `name`, `revision`, `mcp_server_refs` (null means all, `[]` means none), `secret_refs`, and never persist secrets. Unit: `tests/sdk/profiles/test_agent_profile.py`, `tests/sdk/profiles/test_secret_refs.py`. Example: none. Real-LLM: none. Server: `F21.save-validation`, `F21.acp-kind`. - `S19.profile-migrations`: profiles at schema v1 to v3 migrate to v4 (empty or default tools, embedded skills to `disabled_skills`, v2 switches to tools, v3 retired switch inputs dropped); newer, negative or non-integer versions are rejected. Unit: `tests/sdk/profiles/test_agent_profile.py::test_v1_untouched_default_migrates_empty_tools_to_null`, `::test_rejects_newer_schema_version`. Example: none. Real-LLM: none. -- `S19.profile-store`: `AgentProfileStore(base_dir=None)` saves atomically with the schema version and id, lists, loads (showing available names on a miss), renames (keeping the id), deletes, lists summaries skipping corrupted files, enforces `max_profiles`, and is safe under concurrent access through a file lock. Unit: `tests/sdk/profiles/test_agent_profile_store.py`. Example: none. Real-LLM: none. -- `S19.profile-resolve`: since 1.60 a launch has two stages in `openhands.sdk.launch`. `resolve(profile_or_id, LaunchStores(llm_profiles=..., mcp_config=..., skills=...))` runs where the stores live and returns a reference-free `ResolvedLaunch` with validated settings: it loads the LLM profile, filters MCP servers, applies `disabled_skills`, scopes tools and delegation, and keeps persona and verification; every dangling reference (LLM profile, MCP servers, meta-profile) is raised together as `UnresolvedProfileReferences`. `finalize(resolved, LaunchRuntime(...))` builds the agent where it runs; `preview_launch(profile, stores, runtime)` reports the same verdict without raising, with secrets redacted. The legacy `resolve_agent_profile` / `resolve_agent_profile_dry_run` were removed. Unit: `tests/sdk/launch/test_resolve_legacy_parity.py`. Example: none. Real-LLM: none. -- `S19.profile-refs`: renaming an LLM profile cascades to every agent profile that cites it, atomically; deleting a referenced LLM profile raises `ProfileReferenced` naming the referrers. Unit: `tests/sdk/profiles/test_profile_refs.py`, `tests/sdk/profiles/test_store_protocol_hoists.py`. Example: none. Real-LLM: none. -- `S19.profile-seed`: `build_seed_profile(settings)` derives the default profile from current settings (persona, explicit tools, ACP branch, no disabled skills); `save_profile_preserving_identity` keeps the id and bumps `revision`. Unit: `tests/sdk/profiles/test_store_protocol_hoists.py`. Example: none. Real-LLM: none. +- `S19.profile-store`: `AgentProfileStore(base_dir=None)` saves atomically with the schema version and id, lists, loads (showing available names on a miss), renames (keeping the id), deletes, lists summaries skipping corrupted files, enforces `max_profiles`, and is safe under concurrent access through a file lock. Unit: `tests/sdk/profiles/test_agent_profile_store.py`. Example: none. Real-LLM: none. Server: `F21.save-create`, `F21.save-overwrite`, `F21.rename`, `F21.delete`, `F21.limit`, `F21.corrupt-file`. +- `S19.profile-resolve`: since 1.60 a launch has two stages in `openhands.sdk.launch`. `resolve(profile_or_id, LaunchStores(llm_profiles=..., mcp_config=..., skills=...))` runs where the stores live and returns a reference-free `ResolvedLaunch` with validated settings: it loads the LLM profile, filters MCP servers, applies `disabled_skills`, scopes tools and delegation, and keeps persona and verification; every dangling reference (LLM profile, MCP servers, meta-profile) is raised together as `UnresolvedProfileReferences`. `finalize(resolved, LaunchRuntime(...))` builds the agent where it runs; `preview_launch(profile, stores, runtime)` reports the same verdict without raising, with secrets redacted. The legacy `resolve_agent_profile` / `resolve_agent_profile_dry_run` were removed. Unit: `tests/sdk/launch/test_resolve_legacy_parity.py`. Example: none. Real-LLM: none. Server: `F21.materialize-valid`, `F21.materialize-dangling`, `F21.conversation-start`. +- `S19.profile-refs`: renaming an LLM profile cascades to every agent profile that cites it, atomically; deleting a referenced LLM profile raises `ProfileReferenced` naming the referrers. Unit: `tests/sdk/profiles/test_profile_refs.py`, `tests/sdk/profiles/test_store_protocol_hoists.py`. Example: none. Real-LLM: none. Server: `F20.rename`, `F20.delete-guarded`. +- `S19.profile-seed`: `build_seed_profile(settings)` derives the default profile from current settings (persona, explicit tools, ACP branch, no disabled skills); `save_profile_preserving_identity` keeps the id and bumps `revision`. Unit: `tests/sdk/profiles/test_store_protocol_hoists.py`. Example: none. Real-LLM: none. Server: `F21.lazy-seed`, `F21.seed-acp`. - `S19.profile-sandbox`: both profile variants take an optional `sandbox=ProfileSandboxSettings(spec_id=...)` (exported from `openhands.sdk.profiles`): a sandbox spec id the host that provisions sandboxes reads; it round-trips through `validate_agent_profile` and `AgentProfileStore`, a stored profile without it loads with `sandbox` `None`, an empty `spec_id`, a missing `spec_id`, an extra key or a non-object is refused, and `resolve()` gives exactly the same `AgentSettings` with or without it. Unit: `tests/sdk/profiles/test_agent_profile.py::test_sandbox_defaults_to_none_and_older_payloads_load`, `::test_sandbox_round_trips_on_both_variants`, `::test_sandbox_rejects_empty_or_unknown_fields`, `tests/sdk/launch/test_resolve.py::test_sandbox_metadata_does_not_change_the_resolved_settings`. Example: none. Real-LLM: none. -- `S19.credential-binding`: `HttpVersionedCredentialBinding(url, headers).load()` GETs `{value, version}`, `replace(expected_version, value)` PUTs and returns the new version; 404 and 400/422 raise `CredentialNeedsReauthentication`, 409 `CredentialConflict`, 401/403 `CredentialAuthorizationRejected`, 501 `CredentialBindingUnsupported`, invalid URLs `CredentialInvalidResponse`; `reauthorize()` swaps headers for the same URL. Unit: `tests/sdk/test_credential.py`. Example: none. Real-LLM: none. +- `S19.credential-binding`: `HttpVersionedCredentialBinding(url, headers).load()` GETs `{value, version}`, `replace(expected_version, value)` PUTs and returns the new version; 404 and 400/422 raise `CredentialNeedsReauthentication`, 409 `CredentialConflict`, 401/403 `CredentialAuthorizationRejected`, 501 `CredentialBindingUnsupported`, invalid URLs `CredentialInvalidResponse`; `reauthorize()` swaps headers for the same URL. Unit: `tests/sdk/test_credential.py`. Example: none. Real-LLM: none. Server: `F12.binding-pending`, `F12.binding-source-statuses`, `F12.binding-source-unavailable`. ## How to get to it (agent POV) diff --git a/.agents/skills/verify-sdk/references/feature-map/S20-observability.md b/.agents/skills/verify-sdk/references/feature-map/S20-observability.md index 2ee0834280..3a4ba1f487 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S20-observability.md +++ b/.agents/skills/verify-sdk/references/feature-map/S20-observability.md @@ -15,7 +15,7 @@ Examples: `examples/01_standalone_sdk/27_observability_laminar.py` - `S20.laminar-init`: `maybe_init_laminar()` (run when `openhands.sdk.agent` is imported) calls `Laminar.initialize` once, passing `LMNR_BASE_URL` (unless empty), `LMNR_HTTP_PORT`/`LMNR_GRPC_PORT`, `LMNR_FORCE_HTTP` and `LMNR_INSTRUMENTS` (invalid names ignored with a warning), and skips when already initialized. Unit: `tests/sdk/observability/test_laminar.py::test_lmnr_base_url_passed_to_laminar`, `::test_lmnr_base_url_not_passed_when_empty`, `::test_maybe_init_laminar_skips_when_already_initialized`, `::test_lmnr_force_http_passed_to_laminar`, `::test_lmnr_instruments_passed_to_laminar`. Example: `examples/01_standalone_sdk/27_observability_laminar.py`. Real-LLM: none. - `S20.otel-backend`: without `LMNR_PROJECT_API_KEY` but with an `OTEL_*` endpoint, Laminar initializes against the generic OTLP backend with browser-session, Patchright and Playwright instruments disabled. Unit: **none (untested)**. Example: none. Real-LLM: none. - `S20.root-span`: each conversation starts a `conversation` root span (session id = conversation id, `user_id`, metadata, tags, attributes) that `@observe` methods re-attach in any thread or task, ends exactly once on `close()`, and does nothing when disabled; two concurrent conversations keep separate traces. Unit: `tests/sdk/observability/test_laminar.py` (`test_root_span_*`, `test_observe_calls_use_span_with_owner_root_span_*`, `test_two_concurrent_conversations_do_not_collide`), `tests/sdk/conversation/test_base_span_management.py`, `tests/sdk/conversation/local/test_span_double_ending.py`. Example: `examples/01_standalone_sdk/27_observability_laminar.py`. Real-LLM: none. -- `S20.conversation-args`: `Conversation(user_id=..., observability_metadata=..., observability_tags=..., observability_span_name=..., observability_parent_span_context=..., tags=...)` reach the root span: metadata and `conversation.tags.<key>` become attributes, a non-default span name becomes a child span, a parent context makes the trace a child of the caller's. Unit: `tests/sdk/conversation/test_base_span_management.py::test_base_conversation_passes_observability_metadata_and_tag_attributes`, `::test_base_conversation_uses_custom_observability_span_name_as_child_span`, `tests/sdk/conversation/test_conversation_factory.py::test_conversation_factory_forwards_local_observability_span_name`. Example: none. Real-LLM: none. +- `S20.conversation-args`: `Conversation(user_id=..., observability_metadata=..., observability_tags=..., observability_span_name=..., observability_parent_span_context=..., tags=...)` reach the root span: metadata and `conversation.tags.<key>` become attributes, a non-default span name becomes a child span, a parent context makes the trace a child of the caller's. Unit: `tests/sdk/conversation/test_base_span_management.py::test_base_conversation_passes_observability_metadata_and_tag_attributes`, `::test_base_conversation_uses_custom_observability_span_name_as_child_span`, `tests/sdk/conversation/test_conversation_factory.py::test_conversation_factory_forwards_local_observability_span_name`. Example: none. Real-LLM: none. Server: `F04.create-observability`, `F04.create-observability-precedence`. - `S20.env-conversation-defaults`: a local conversation also merges `OPENHANDS_OBSERVABILITY_METADATA` (JSON), uses `OPENHANDS_OBSERVABILITY_SPAN_NAME` when no span name is passed, and `OPENHANDS_OBSERVABILITY_PARENT_SPAN_CONTEXT` or `LMNR_SPAN_CONTEXT` as parent. Unit: **none (untested)** at the conversation level (the env readers are tested through `tests/sdk/test_automation_observability.py`). Example: none. Real-LLM: none. - `S20.step-and-tool-spans`: `agent.step`/`agent.astep` spans contain one TOOL span per declared tool call, including rejected and cancelled calls, whose input is the action only and whose metadata carries `tool_call_id`. Unit: `tests/sdk/agent/test_tool_span_input.py`, `tests/sdk/agent/test_tool_span_tool_call_id_metadata.py`, `tests/sdk/observability/test_laminar.py::test_async_agent_and_conversation_paths_are_observed`. Example: none. Real-LLM: none. - `S20.parallel-span-context`: parallel tool workers copy the context so their spans nest under the dispatcher, with and without `lmnr`, and overlapping tasks get fresh contexts. Unit: `tests/sdk/agent/test_parallel_tool_span_context.py`. Example: none. Real-LLM: none. @@ -26,7 +26,7 @@ Examples: `examples/01_standalone_sdk/27_observability_laminar.py` - `S20.external-entry`: `init_laminar_for_external()` initializes Laminar and returns the current span context for a webhook-style caller to pass on. Unit: **none (untested)**. Example: none. Real-LLM: none. - `S20.automation-context`: `automation_observability_context()` collects `OPENHANDS_OBSERVABILITY_METADATA`, `_TAGS` (comma list, deduplicated), `_SPAN_NAME` and `_PARENT_SPAN_CONTEXT` and merges caller additions. Unit: `tests/sdk/test_automation_observability.py::test_automation_observability_context_collects_generic_env`. Example: none. Real-LLM: none. - `S20.automation-kwargs`: `automation_conversation_kwargs()` returns `Conversation(...)` kwargs with product `tags` built from `automation.*` metadata and the observability fields; an explicit span name overrides the env one. Unit: `tests/sdk/test_automation_observability.py::test_automation_conversation_tags_from_metadata`, `::test_automation_conversation_kwargs_include_product_and_observability_tags`, `::test_automation_conversation_kwargs_uses_env_span_name`, `::test_automation_conversation_kwargs_explicit_span_name_overrides_env`. Example: none. Real-LLM: none. -- `S20.automation-headers`: `automation_observability_headers()` / `observability_headers_from_env()` build `X-OpenHands-Observability-Metadata`, `-Tags`, `-Span-Name`, `-Parent-Span-Context` plus OTel propagation headers for direct agent-server calls; remote workspaces and remote conversations send them. Unit: `tests/sdk/test_automation_observability.py::test_automation_observability_headers_for_direct_api_calls`, `tests/sdk/workspace/remote/test_remote_workspace_mixin.py::test_headers_property_includes_observability_context`, `tests/sdk/workspace/remote/test_remote_workspace.py::test_send_completion_callback_includes_observability_headers`, `tests/sdk/conversation/remote/test_remote_conversation.py` (`-k observability`). Example: none. Real-LLM: none. +- `S20.automation-headers`: `automation_observability_headers()` / `observability_headers_from_env()` build `X-OpenHands-Observability-Metadata`, `-Tags`, `-Span-Name`, `-Parent-Span-Context` plus OTel propagation headers for direct agent-server calls; remote workspaces and remote conversations send them. Unit: `tests/sdk/test_automation_observability.py::test_automation_observability_headers_for_direct_api_calls`, `tests/sdk/workspace/remote/test_remote_workspace_mixin.py::test_headers_property_includes_observability_context`, `tests/sdk/workspace/remote/test_remote_workspace.py::test_send_completion_callback_includes_observability_headers`, `tests/sdk/conversation/remote/test_remote_conversation.py` (`-k observability`). Example: none. Real-LLM: none. Server: `F30.observability-headers`, `F04.create-observability`. - `S20.get-env`: observability reads env vars straight from `os.environ`, never through python-dotenv's `find_dotenv()`. Unit: `tests/sdk/observability/test_utils.py`. Example: none. Real-LLM: none. - `S20.logger-quiet-thirdparty`: importing the SDK sets LiteLLM and OpenAI loggers to ERROR (DEBUG only after a confirmed `DEBUG_LLM`), and httpx, httpcore and libtmux to WARNING, so LiteLLM INFO lines do not show. Unit: `tests/sdk/logger/test_litellm_log_suppression.py`. Example: none. Real-LLM: none. - `S20.logger-config`: `get_logger(name)` returns a propagating stdlib logger; `setup_logging(...)` (auto-run on import unless `LOG_AUTO_CONFIG=false`) honors `LOG_LEVEL`, `DEBUG`, `LOG_JSON`, `CI`/`GITHUB_ACTIONS` (JSON console), `LOG_RICH_TRACEBACKS`, and `LOG_TO_FILE` with `LOG_DIR`, `LOG_ROTATE_WHEN`, `LOG_BACKUP_COUNT` (`<LOG_DIR>/app.log`), and does not add a second stream handler. Unit: **none (untested)**. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md b/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md index e88d8b5c50..948fb7a8ea 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md +++ b/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md @@ -24,10 +24,10 @@ Examples: `examples/01_standalone_sdk/40_acp_agent_example.py` - `S21.cancel-pause`: cancelling `arun()` mid-prompt sends `session/cancel`, drains the prompt, emits failed tool calls, pauses without a turn error, and restarts the session on the next turn if the drain times out. Unit: `tests/sdk/agent/test_acp_agent.py::TestACPAgentAstep` (`test_astep_*cancel*`, `test_cleanup_interruption_*`). Example: none. Real-LLM: none. - `S21.session-resume`: the ACP session id is persisted in `state.agent_state` and used with `load_session` when the conversation is reloaded; `acp_resume_session_id` overrides it; a failed load falls back to a new session; session ids are redacted in logs and serialization. Unit: `tests/sdk/agent/test_acp_agent.py::TestACPSessionIdPersistence`, `::TestHasLiveACPSession`. Example: none. Real-LLM: none. - `S21.model-selection`: `acp_model` is applied through `set_config_option` (codex, claude) or `set_session_model`, reapplied on resume, without silent fallback; `current_model_id` and `available_models` come from the server's session response, and `current_model_id` is the bare model id (a legacy `<model>/<effort>` id for a provider with an effort option is split into the model and the effort). Unit: `tests/sdk/agent/test_acp_agent.py::TestMaybeSetSessionModel`, `::TestReapplySessionModelOnResume`, `::TestConfigOptionModelMechanism`, `::TestApplyAcpModel`, `::TestApplyAcpModelNoFallback`, `::TestExtractSessionModels`, `::TestACPAgentCurrentModelIdProperty`, `::TestACPAgentAvailableModelsProperty`, `tests/sdk/agent/test_acp_contracts.py`. Example: none. Real-LLM: none. -- `S21.model-switch`: `agent.set_acp_model(model)` switches a live session (an invalid model or unsupported provider raises `ValueError`); `Conversation.switch_acp_model(...)` defers before the session starts, persists the authoritative model and refreshes `current_model_id`. Unit: `tests/sdk/agent/test_acp_agent.py::TestSetACPModel`, `::TestACPAgentSupportsRuntimeModelSwitch`, `tests/sdk/conversation/test_switch_model.py` (`-k acp`). Example: none. Real-LLM: none. -- `S21.model-discovery`: `discover_acp_models(settings, secrets=...)` starts the server in a throwaway session and reports its default and offered models (config options or the legacy models block), the default model's `current_effort` and `available_efforts`, and `efforts_by_model` (the levels each model offers), passing secrets and reporting auth and spawn failures. Unit: `tests/sdk/agent/test_acp_model_discovery.py` (runs `tests/sdk/agent/fake_acp_models_server.py` as a real subprocess). Example: none. Real-LLM: none. -- `S21.effort`: `ACPAgent(acp_effort=...)` sets the reasoning effort after the model through the session config option the server tags `thought_level`, and `agent.current_effort` / `available_efforts` report what the session runs and offers for its current model; a level the model does not offer is skipped with a warning (the turn still runs at the server's default), and an explicit `acp_effort` wins over a legacy `<model>/<effort>` suffix. Unit: `tests/sdk/agent/test_acp_model_discovery.py::test_session_start_applies_the_requested_effort`, `::test_effort_is_skipped_for_a_model_that_offers_none`, `tests/sdk/agent/test_acp_contracts.py::TestSessionEffortContract`, `tests/sdk/agent/test_acp_agent.py::TestSplitEffortSuffix`. Example: none. Real-LLM: none. -- `S21.effort-switch`: `Conversation.switch_acp_effort(effort)` before the first run only persists `acp_effort` (no subprocess starts) and the first session applies it; on a live session it switches the effort in place (same session) and persists `acp_effort` and the `acp_current_effort` hint in `base_state.json`; a level the model does not offer, a blank level, a model that offers no effort and a non-ACP conversation raise `ValueError` and change nothing; after `switch_acp_model` the effort follows the new model's options, and switching back re-applies the pinned effort. Unit: `tests/sdk/conversation/test_switch_model.py::test_switch_acp_effort_before_session_persists`, `::test_switch_acp_effort_live_applies_and_persists`, `::test_switch_acp_effort_rejected_keeps_persisted_state`, `::test_switch_acp_effort_rejects_non_acp_agent`, `tests/sdk/agent/test_acp_model_discovery.py::test_switches_effort_at_runtime`, `::test_model_switch_keeps_the_pinned_effort`. Example: none. Real-LLM: none. +- `S21.model-switch`: `agent.set_acp_model(model)` switches a live session (an invalid model or unsupported provider raises `ValueError`); `Conversation.switch_acp_model(...)` defers before the session starts, persists the authoritative model and refreshes `current_model_id`. Unit: `tests/sdk/agent/test_acp_agent.py::TestSetACPModel`, `::TestACPAgentSupportsRuntimeModelSwitch`, `tests/sdk/conversation/test_switch_model.py` (`-k acp`). Example: none. Real-LLM: none. Server: `F09.acp-model-deferred`, `F09.acp-model-live`, `F09.acp-model-blank`, `F09.acp-model-live-unknown`. +- `S21.model-discovery`: `discover_acp_models(settings, secrets=...)` starts the server in a throwaway session and reports its default and offered models (config options or the legacy models block), the default model's `current_effort` and `available_efforts`, and `efforts_by_model` (the levels each model offers), passing secrets and reporting auth and spawn failures. Unit: `tests/sdk/agent/test_acp_model_discovery.py` (runs `tests/sdk/agent/fake_acp_models_server.py` as a real subprocess). Example: none. Real-LLM: none. Server: `F23.acp-discovery-live`, `F23.acp-discovery-errors`. +- `S21.effort`: `ACPAgent(acp_effort=...)` sets the reasoning effort after the model through the session config option the server tags `thought_level`, and `agent.current_effort` / `available_efforts` report what the session runs and offers for its current model; a level the model does not offer is skipped with a warning (the turn still runs at the server's default), and an explicit `acp_effort` wins over a legacy `<model>/<effort>` suffix. Unit: `tests/sdk/agent/test_acp_model_discovery.py::test_session_start_applies_the_requested_effort`, `::test_effort_is_skipped_for_a_model_that_offers_none`, `tests/sdk/agent/test_acp_contracts.py::TestSessionEffortContract`, `tests/sdk/agent/test_acp_agent.py::TestSplitEffortSuffix`. Example: none. Real-LLM: none. Server: `F09.acp-effort-model-switch`. +- `S21.effort-switch`: `Conversation.switch_acp_effort(effort)` before the first run only persists `acp_effort` (no subprocess starts) and the first session applies it; on a live session it switches the effort in place (same session) and persists `acp_effort` and the `acp_current_effort` hint in `base_state.json`; a level the model does not offer, a blank level, a model that offers no effort and a non-ACP conversation raise `ValueError` and change nothing; after `switch_acp_model` the effort follows the new model's options, and switching back re-applies the pinned effort. Unit: `tests/sdk/conversation/test_switch_model.py::test_switch_acp_effort_before_session_persists`, `::test_switch_acp_effort_live_applies_and_persists`, `::test_switch_acp_effort_rejected_keeps_persisted_state`, `::test_switch_acp_effort_rejects_non_acp_agent`, `tests/sdk/agent/test_acp_model_discovery.py::test_switches_effort_at_runtime`, `::test_model_switch_keeps_the_pinned_effort`. Example: none. Real-LLM: none. Server: `F09.acp-effort-deferred`, `F09.acp-effort-live`, `F09.acp-effort-unoffered`. - `S21.ask-agent`: `conversation.ask_agent(question)` forks the ACP session, prompts the fork, records its token usage and cleans the fork state, leaving the main session untouched; it raises before the session exists. Unit: `tests/sdk/agent/test_acp_agent.py::TestACPAgentAskAgent`, `::TestClientForkTextRouting`. Example: `examples/01_standalone_sdk/40_acp_agent_example.py`. Real-LLM: none. - `S21.usage-cost`: token usage and cost from `usage_update` notifications (or estimated from tokens) are recorded on the sentinel LLM's metrics per turn, incrementally, with latency and context window; partial metrics are kept on a usage timeout. Unit: `tests/sdk/agent/test_acp_agent.py::TestACPAgentTelemetry`, `::TestExtractTokenUsage`, `::TestEstimateCostFromTokens`. Example: `examples/01_standalone_sdk/40_acp_agent_example.py` (prints `EXAMPLE_COST`). Real-LLM: none. - `S21.mcp-forwarding`: the agent's `mcp_config` is converted to ACP stdio or HTTP/SSE servers (with headers) and passed to `new_session` and `load_session`; none gives an empty list. Unit: `tests/sdk/agent/test_acp_agent.py::TestMcpConfigToAcpServers`, `::TestACPMcpForwarding`. Example: none. Real-LLM: none. diff --git a/.agents/skills/verify-sdk/scripts/sdk_map.py b/.agents/skills/verify-sdk/scripts/sdk_map.py index 1ef01e9b52..badf696e6c 100755 --- a/.agents/skills/verify-sdk/scripts/sdk_map.py +++ b/.agents/skills/verify-sdk/scripts/sdk_map.py @@ -349,6 +349,9 @@ def cmd_check(args: argparse.Namespace) -> dict[str, Any]: f"duplicate ID {row['id']} (also in {seen[row['id']]})" ) seen[row["id"]] = row["file"] + checked = {row["id"] for row in collect_ids(files)} if args.file else None + for name, found in twin_link_problems("S", checked).items(): + problems.setdefault(name, []).extend(found) if not args.file and index_path().exists(): index = problems.setdefault("README.md", []) text = index_path().read_text() @@ -420,6 +423,139 @@ def cmd_ids(args: argparse.Namespace) -> dict[str, Any]: return {"ok": True, "count": len(rows), "ids": [row["id"] for row in rows]} +# -------------------------------------------------------------------------- +# Twin links between the SDK map and the agent-server map +# -------------------------------------------------------------------------- + +# An SDK bullet whose promise an agent-server bullet also proves (through the +# server's routes, sockets or the SDK's remote client) ends with +# `Server: `Fnn.x`, ...`; each of those bullets ends with +# `Same promise: `Snn.y`, ...` naming it back. Both map tools check both +# directions, so a link is never one-sided. The other map is the sibling +# skill's, found next to this one under .agents/skills/. +TWIN_LINKS = { + "S": ("verify-sdk", "Server", "F"), + "F": ("verify-agent-server", "Same promise", "S"), +} + + +def twin_map_dir(side: str) -> Path: + skills = FEATURE_MAP_DIR.parents[2] + return skills / TWIN_LINKS[side][0] / "references" / "feature-map" + + +TWIN_SUB_FEATURE_RE = re.compile(r"^- `([SF]\d{2}\.[a-z0-9][a-z0-9-]*)`: ") + + +def read_twin_links(side: str) -> dict[str, dict[str, Any]] | None: + """Every sub-feature of one map with the twin IDs its link names. + + None when that map is not in this checkout. + """ + _, label, other = TWIN_LINKS[side] + directory = twin_map_dir(side) + if not directory.is_dir(): + return None + one = rf"`{other}\d{{2}}\.[a-z0-9-]+`" + link_re = re.compile(rf" {label}: ({one}(?:, {one})*)\.$") + rows: dict[str, dict[str, Any]] = {} + for path in sorted(directory.glob(f"{side}[0-9][0-9]-*.md")): + section = None + for number, line in enumerate(path.read_text().splitlines(), 1): + if line.startswith("## "): + section = line[3:].strip() + continue + match = ( + TWIN_SUB_FEATURE_RE.match(line) if section == "Sub-features" else None + ) + if not match or not match.group(1).startswith(side): + continue + row: dict[str, Any] = { + "file": path.name, + "line": number, + "links": [], + "error": None, + } + if f" {label}:" in line: + found = link_re.search(line) + if found: + row["links"] = re.findall(r"`([^`]+)`", found.group(1)) + else: + row["error"] = ( + f"`{label}:` must end the line as " + f"`{label}: `{other}nn.id`, `{other}nn.id`.`" + ) + rows[match.group(1)] = row + return rows + + +def twin_link_problems(side: str, ids: set[str] | None = None) -> dict[str, list[str]]: + """Link problems keyed by file name, checked from both maps. + + `side` is this tool's map ("S" or "F"); `ids` limits the check to links + that touch those sub-features of it. + """ + other = TWIN_LINKS[side][2] + maps = {side: read_twin_links(side), other: read_twin_links(other)} + if maps[side] is None or maps[other] is None: + return {} + problems: dict[str, list[str]] = {} + + def add(row: dict[str, Any], message: str) -> None: + problems.setdefault(row["file"], []).append(message) + + for here_side, there_side in ((side, other), (other, side)): + here, there = maps[here_side], maps[there_side] + label, back = TWIN_LINKS[here_side][1], TWIN_LINKS[there_side][1] + for twin_id, row in here.items(): + where = f"line {row['line']}: {twin_id}" + if ids is None or twin_id in ids: + if row["error"]: + add(row, f"{where}: {row['error']}") + if len(set(row["links"])) != len(row["links"]): + add(row, f"{where}: `{label}:` names an ID twice") + for other_id in row["links"]: + if ids is not None and twin_id not in ids and other_id not in ids: + continue + if other_id not in there: + add( + row, + f"{where}: `{label}:` names {other_id}, not in the other map", + ) + elif twin_id not in there[other_id]["links"]: + at = f"{there[other_id]['file']} line {there[other_id]['line']}" + add( + row, + f"{where}: `{label}:` names {other_id}, whose `{back}:`" + f" does not name it back ({at})", + ) + return problems + + +def twin_pairs(side: str, ids: set[str] | None = None) -> list[dict[str, str]]: + """The SDK/server pairs that touch `ids` of this tool's map (all when None).""" + sdk = read_twin_links("S") or {} + pairs = [] + for sdk_id, row in sdk.items(): + for server_id in row["links"]: + mine = sdk_id if side == "S" else server_id + if ids is None or mine in ids: + pairs.append({"sdk": sdk_id, "server": server_id}) + return pairs + + +def cmd_links(args: argparse.Namespace) -> dict[str, Any]: + ids = ( + {row["id"] for row in collect_ids(family_files(args.file))} + if args.file + else None + ) + if args.id: + ids = (ids or set()) | set(args.id) + pairs = twin_pairs("S", ids) + return {"ok": True, "count": len(pairs), "pairs": pairs} + + # -------------------------------------------------------------------------- # owners, baseline # -------------------------------------------------------------------------- @@ -1011,6 +1147,19 @@ def build_parser() -> argparse.ArgumentParser: p.set_defaults(func=cmd_ids) p.add_argument("--file") + p = sub.add_parser( + "links", + help="SDK/agent-server twin pairs (`Server:` and `Same promise:` links)", + formatter_class=fmt, + epilog=f"""Examples: + {CLI} links + {CLI} links --file S04 + {CLI} links --id S04.pause --id S04.fork""", + ) + p.set_defaults(func=cmd_links) + p.add_argument("--file") + p.add_argument("--id", action="append", help="a sub-feature ID (repeatable)") + p = sub.add_parser( "owners", help="map changed source paths to families", diff --git a/tests/cross/test_verify_agent_server_skill.py b/tests/cross/test_verify_agent_server_skill.py index 0f31bbfb1f..2201727a54 100644 --- a/tests/cross/test_verify_agent_server_skill.py +++ b/tests/cross/test_verify_agent_server_skill.py @@ -1252,3 +1252,37 @@ def save(run, name, model, key, base_url, activate, validate): assert saved == [ ("deepseek-flash", "litellm_proxy/deepseek-v4.1-flash", "https://proxy.example") ] + + +def test_map_check_validates_twin_links_with_the_sdk_map(tmp_path, monkeypatch, capsys): + """`Same promise:` and the SDK map's `Server:` must name each other.""" + skills = tmp_path / "skills" + server = skills / "verify-agent-server" / "references" / "feature-map" + sdk = skills / "verify-sdk" / "references" / "feature-map" + server.mkdir(parents=True) + sdk.mkdir(parents=True) + (server / "F98-demo.md").write_text( + "# Demo\n\n## Sub-features\n\n" + "- `F98.pause`: pauses. Same promise: `S98.pause`.\n" + "- `F98.run`: runs. Same promise: `S98.gone`.\n" + ) + (sdk / "S98-demo.md").write_text( + "# Demo\n\n## Sub-features\n\n" + "- `S98.pause`: pauses. Server: `F98.pause`, `F98.run`.\n" + ) + monkeypatch.setattr(cas, "FEATURE_MAP_DIR", server) + assert cas.twin_link_problems("F") == { + "F98-demo.md": [ + "line 6: F98.run: `Same promise:` names S98.gone, not in the other map" + ], + "S98-demo.md": [ + "line 5: S98.pause: `Server:` names F98.run, whose `Same promise:`" + " does not name it back (F98-demo.md line 6)" + ], + } + code, out = _run(capsys, "map", "links", "--file", "F98") + assert code == 0 + assert json.loads(out)["pairs"] == [ + {"sdk": "S98.pause", "server": "F98.pause"}, + {"sdk": "S98.pause", "server": "F98.run"}, + ] diff --git a/tests/cross/test_verify_sdk_skill.py b/tests/cross/test_verify_sdk_skill.py index 84ebce8dab..75ac7ff725 100644 --- a/tests/cross/test_verify_sdk_skill.py +++ b/tests/cross/test_verify_sdk_skill.py @@ -446,3 +446,66 @@ def test_every_command_documents_examples(capsys): with pytest.raises(SystemExit): parser.parse_args([name, "--help"]) assert "Examples:" in capsys.readouterr().out, name + + +def _twin_maps(tmp_path, sdk_line: str, server_line: str) -> Path: + """A skills tree with one SDK family and one agent-server family.""" + skills = tmp_path / "skills" + sdk = skills / "verify-sdk" / "references" / "feature-map" + server = skills / "verify-agent-server" / "references" / "feature-map" + sdk.mkdir(parents=True) + server.mkdir(parents=True) + (sdk / "S98-demo.md").write_text( + f"# Demo\n\n## Sub-features\n\n{sdk_line}\n- `S98.alone`: x.\n" + ) + (server / "F98-demo.md").write_text( + f"# Demo\n\n## Sub-features\n\n{server_line}\n- `F98.other`: y.\n" + ) + return sdk + + +def test_twin_links_are_checked_both_ways(tmp_path, monkeypatch): + sdk = _twin_maps( + tmp_path, + "- `S98.pause`: pauses. Unit: none. Server: `F98.pause`, `F98.other`.", + "- `F98.pause`: pauses. Same promise: `S98.pause`, `S98.alone`.", + ) + monkeypatch.setattr(sdk_map, "FEATURE_MAP_DIR", sdk) + problems = sdk_map.twin_link_problems("S") + assert problems == { + "S98-demo.md": [ + "line 5: S98.pause: `Server:` names F98.other, whose `Same promise:`" + " does not name it back (F98-demo.md line 6)" + ], + "F98-demo.md": [ + "line 5: F98.pause: `Same promise:` names S98.alone, whose `Server:`" + " does not name it back (S98-demo.md line 6)" + ], + } + # Limited to the IDs of one family, only links that touch them count. + assert sdk_map.twin_link_problems("S", {"S98.alone"}) == { + "F98-demo.md": [problems["F98-demo.md"][0]] + } + assert sdk_map.twin_pairs("S") == [ + {"sdk": "S98.pause", "server": "F98.pause"}, + {"sdk": "S98.pause", "server": "F98.other"}, + ] + + +def test_twin_links_must_name_existing_ids_in_the_exact_form(tmp_path, monkeypatch): + sdk = _twin_maps( + tmp_path, + "- `S98.pause`: pauses. Server: `F98.gone`.", + "- `F98.pause`: pauses. Same promise: S98.pause", + ) + monkeypatch.setattr(sdk_map, "FEATURE_MAP_DIR", sdk) + problems = sdk_map.twin_link_problems("S") + assert problems["S98-demo.md"] == [ + "line 5: S98.pause: `Server:` names F98.gone, not in the other map" + ] + assert "must end the line" in problems["F98-demo.md"][0] + + +def test_twin_links_are_skipped_without_the_other_map(demo_map): + assert sdk_map.twin_link_problems("S") == {} + assert sdk_map.twin_pairs("S") == []