diff --git a/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md b/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md index fef6731215..86cada1982 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md @@ -50,6 +50,8 @@ Routes: `GET /api/settings/agent-schema`, `GET /api/settings/conversation-schema - `F18.ts-client`: the TypeScript `SettingsClient` reads both schemas, reads settings redacted, `plaintext` and `encrypted`, applies a sparse `updateSettings` that a second read shows, upserts, lists (also with `agentProfileId`), reads as text and deletes a secret, and rejects an unknown secret (404), a bad name (422) and a wrong key (401) with an `HttpError`. - `F18.persist-restart`: settings (plaintext key, model, conversation settings) and secrets read back unchanged after a restart with the same cipher key. - `F18.experimental-features`: `GET /api/settings/experimental` lists the experimental features the user can switch (`conversation_reactions`, `ask_user`, `conversation_mentions`, `message_edits` `factories` and `projects`, all off) without writing `settings.json`, and `PATCH /api/settings/experimental/{key}` answers 404 for a key the server does not know and stores nothing (`experimental_features` stays `{}`). +- `F18.experimental-switch`: `PATCH /api/settings/experimental/{key}` `{"enabled": true}` turns each of the six features on (the response and the list agree), `settings.json` `experimental_features` holds all six, they stay on after a restart, and `{"enabled": false}` stores each one off; with only `factories` on, a new conversation gets no tools from it. +- `F18.experimental-file-changes`: a route that follows a switch at once (`GET /api/projects`) follows an edit of `settings.json` by another process on the next request, in both directions; an unparsable `settings.json` counts as every feature off (404) at once, and restoring the file turns it back on. - `F18.upgrade-from-release`: settings, an LLM key and a secret saved by the latest released server read back identical after the same state is restarted on this checkout (`restart --checkout`), the upgrade does not rewrite either file, and the experimental routes answer. - `F18.corrupted-files`: with an unparsable `settings.json`, GET silently returns defaults (every experimental feature off) and PATCH, of the settings or of an experimental feature, is 409 without touching the file; with an unparsable `secrets.json` the list is empty, GET by name 404, PUT and DELETE 500; restoring the files restores the values. - `F18.no-cipher`: without any cipher key, `X-Expose-Secrets: encrypted` is 503, `plaintext` still works, both files store plaintext, and the secrets routes are open without a key. @@ -76,9 +78,13 @@ Routes: `GET /api/settings/agent-schema`, `GET /api/settings/conversation-schema `PATCH /api/settings/experimental/{key}` `{"enabled": bool}` (404 for an unknown key, 409 when `settings.json` cannot be read); the switches are stored in `settings.json` `experimental_features` (what each one does is - mapped by its own family: `conversation_reactions` in F37, - `conversation_mentions` in F38, `message_edits` in F39, `projects` in F36; `ask_user` is not - mapped yet). + mapped by its own family: `conversation_reactions` in F37, `ask_user` in + F40, `conversation_mentions` in F38, `message_edits` in F39, `projects` in + F36; `factories` is read by clients only and gives the agent no tools or + routes). A route that follows a switch at once (`/api/projects`, the + reaction and `ask_user` answer routes) reads it through a cache keyed on + the file's modification time and size, so a save by this server or an + edit by another process shows on the next request. - Neighbours used as arrange steps and second views, owned by other families: `POST/DELETE /api/profiles/{name}` and `GET /api/profiles` (LLM profiles), `POST/GET/DELETE /api/agent-profiles/{name}` and @@ -703,6 +709,65 @@ Preconditions: 404, and neither the reads nor the refused switches create or change `settings.json`; once other bullets saved it, its `experimental_features` is the empty object every save writes. +- **Switch every feature (`F18.experimental-switch`).** Turn each one on, + restart, and turn each one off again. + Requires: `F18.experimental-features` + ```sh + S="$AGENT_SERVER_VERIFY_RUN/home/.openhands/settings.json" + QA_FLAGS='conversation_reactions ask_user conversation_mentions message_edits factories projects' + control-agent-server api PATCH /api/settings/experimental/factories --json '{"enabled": true}' --expect 200 \ + --check key eq factories --check label eq Factories --check enabled eq true --save F18.experimental-switch/factories + FC=$(control-agent-server conversation start --no-run --placeholder-agent --tools none --no-autotitle --print-id) + control-agent-server api GET "/api/conversations/$FC" --check agent.tools len-eq 0 + for K in $QA_FLAGS; do + control-agent-server api PATCH "/api/settings/experimental/$K" --json '{"enabled": true}' --expect 200 \ + --check key eq "$K" --check enabled eq true --quiet + done + control-agent-server api GET /api/settings/experimental --expect 200 --check . len-eq 6 --check . not-contains '"enabled": false' \ + --save F18.experimental-switch/all-on + jq -e '.experimental_features == {"conversation_reactions": true, "ask_user": true, "conversation_mentions": true, "message_edits": true, "factories": true, "projects": true}' "$S" + control-agent-server restart + control-agent-server api GET /api/settings/experimental --expect 200 --check . not-contains '"enabled": false' \ + --save F18.experimental-switch/after-restart + control-agent-server api GET /api/projects --expect 200 + for K in $QA_FLAGS; do + control-agent-server api PATCH "/api/settings/experimental/$K" --json '{"enabled": false}' --expect 200 \ + --check key eq "$K" --check enabled eq false --quiet + done + control-agent-server api GET /api/settings/experimental --expect 200 --check . not-contains '"enabled": true' \ + --save F18.experimental-switch/all-off + jq -e '[.experimental_features[]] == [false, false, false, false, false, false]' "$S" + control-agent-server api GET /api/projects --expect 404 + ``` + `factories` alone adds no tool to a new conversation. Each switch is + saved under its key, survives a restart, and is stored as `false` once + turned off (the key stays in the file). `/api/projects` (F36) follows the + `projects` switch on the same requests. +- **Edits on disk (`F18.experimental-file-changes`).** Another process + rewrites `settings.json` while the server runs. + Requires: `F18.experimental-switch` + ```sh + S="$AGENT_SERVER_VERIFY_RUN/home/.openhands/settings.json" + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": true}' --expect 200 + control-agent-server api GET /api/projects --expect 200 + jq '.experimental_features.projects = false' "$S" > "$F18/qa-flags.json" + cat "$F18/qa-flags.json" > "$S" + control-agent-server api GET /api/projects --expect 404 --check detail eq 'Projects are turned off' --save F18.experimental-file-changes/edited-off + control-agent-server api GET /api/settings/experimental --check 5.key eq projects --check 5.enabled eq false + jq '.experimental_features.projects = true' "$S" > "$F18/qa-flags.json" + cat "$F18/qa-flags.json" > "$S" + control-agent-server api GET /api/projects --expect 200 --save F18.experimental-file-changes/edited-on + cp "$S" "$F18/qa-flags.bak" + printf '{"qa-f18-flags' > "$S" + control-agent-server api GET /api/projects --expect 404 --check detail eq 'Projects are turned off' --save F18.experimental-file-changes/corrupted + cp "$F18/qa-flags.bak" "$S" + control-agent-server api GET /api/projects --expect 200 --save F18.experimental-file-changes/restored + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": false}' --expect 200 + control-agent-server api GET /api/projects --expect 404 + ``` + No stale switch is served: each edit (one byte shorter or longer, a new + modification time) shows on the next request, an unreadable file reads as + off, and the restored file as on. `projects` ends off. - **Upgrade from the latest release (`F18.upgrade-from-release`).** A second run starts on the newest release tag, saves settings as a user of that release would, and is restarted on this checkout over the same state. @@ -1075,6 +1140,9 @@ Preconditions: token back adds `--unsafe-unredacted` to its `--field` read (`TOKEN=$(... --field ... --unsafe-unredacted)`) and keeps it out of evidence. +- The switch cache is keyed on `settings.json`'s modification time and + size only: an edit that keeps the size and puts the old modification time + back (`touch -r`) is not seen until the next save or restart. - `experimental_features` lives inside `settings.json` schema v7 without a version bump. A server older than the experimental routes ignores the key and drops it on its next save, so restarting on such a release diff --git a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md index 545eebb4ff..1dd9a16154 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md @@ -69,7 +69,7 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, - `F19.conversation-mcp-add`: `POST /api/conversations/{conversation_id}/mcp` on a conversation whose agent is initialized starts the server before it answers 200 `{"success": true}`, `GET /api/conversations/{conversation_id}` and `base_state.json` show it in `agent.mcp_config`, the model's next request offers its tools as `_`, a call returns the tool's result, deleting the conversation stops the child, and `/server_info` advertises `conversation_mcp_install_v1` and `conversation_configuration_additions_v1`. - `F19.conversation-mcp-add-before-init`: on a conversation without a first message the route only records the server (no process, fast answer), with `env` values encrypted by the settings API decrypted for the child and stored encrypted in `base_state.json`; the first message connects it like any configured server. - `F19.conversation-mcp-add-errors`: an unknown conversation is 404, a server name the agent already has and an SSE server are 400 with the reason, a server without `command`/`url`, an unknown field and a non-UUID id are 422, none of them changes the agent's `mcp_config`, and a disabled server is stored without being spawned. -- `F19.conversation-mcp-add-start-failure`: a server that cannot start (a command that exits at once) is refused with the route's 400 naming the server and the cause (`Could not start MCP server : Connection closed`, since [#6928](https://github.com/OpenHands/software-agent-sdk/pull/6928)) and is not added. +- `F19.conversation-mcp-add-start-failure`: a server that cannot start (a command that exits at once) is refused with the route's 400 naming the server and the cause (`Could not start MCP server : Connection closed`) and is not added. - `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (`qa-f19-rs_qa_echo`, prefixed even as the agent's only server). - `F19.conversation-mcp-slow-start`: a conversation whose stdio server delays its handshake by 8 s is created within a few seconds, the server is spawned, and the agent's first model request waits for it and offers its tools; the agent's call returns its result. - `F19.conversation-mcp-late-failure`: a server that fails after the startup budget costs only its own tools: the conversation starts at once, the first step waits for the failure, the other server's tools are offered, the run finishes and the log names the failed server. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F34-typescript-client.md b/.agents/skills/verify-agent-server/references/feature-map/F34-typescript-client.md index 1150bf5727..1214682199 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F34-typescript-client.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F34-typescript-client.md @@ -1622,11 +1622,13 @@ Preconditions: jq -e '.paths["/api/conversations"].post and .paths["/api/bash/execute_bash_command"].post' "$A/openapi.json" > /dev/null for f in package.json src .prettierrc; do ln -sfn "$PWD/clients/typescript/$f" "$A/$f"; done rm -f "$A/endpoint-audit.config.json" - jq '.allowClientOnly |= map(select(.endpoints | index("GET /alive")))' clients/typescript/endpoint-audit.config.json > "$A/endpoint-audit.config.json" + jq '.allowClientOnly |= map(select((.endpoints | index("GET /alive")) or (.endpoints | index("GET /api/projects"))))' \ + clients/typescript/endpoint-audit.config.json > "$A/endpoint-audit.config.json" control-agent-server exec --cwd "$A" --env AGENT_SERVER_OPENAPI_PATH="$A/openapi.json" --timeout 120 \ --expect-output 'endpoint audit passed' --save F34.endpoint-audit/audit -- node "$PWD/clients/typescript/scripts/endpoint-audit.mjs" jq -e '.clientOnly == [] and .client > 100' "$A/.audit/endpoint-audit.json" - jq -e '[.allowedClientOnly[].endpoint] == ["GET /", "GET /alive", "GET /health", "GET /ready", "GET /server_info"]' "$A/.audit/endpoint-audit.json" + jq -e '[.allowedClientOnly[].endpoint | select(test(" /api/projects($|/)") | not)] == ["GET /", "GET /alive", "GET /health", "GET /ready", "GET /server_info"]' "$A/.audit/endpoint-audit.json" + jq -e '.allowedClientOnly | map(.endpoint) | index("GET /api/projects") != null and index("POST /api/projects") != null' "$A/.audit/endpoint-audit.json" jq -r '.serverOnly[]' "$A/.audit/endpoint-audit.json" > "$A/server-only.txt" if grep -E '^(GET|POST) /api/(init|file/archive|git/commits|conversations/\{\}/(events|load_plugin))' "$A/server-only.txt"; then false; fi jq -e '[.explicitlyCoveredServerOperations[].endpoint] | index("GET /api/conversations/{}/events") != null' "$A/.audit/endpoint-audit.json" @@ -1639,11 +1641,14 @@ Preconditions: `.github/scripts/export_agent_server_openapi.py` writes the same public contract CI uses for unreleased versions. The audit runs with a copy of `endpoint-audit.config.json` whose `allowClientOnly` keeps only the five - operational routes the public contract omits (F01 proves them), so its - client-ahead exceptions (meta-profiles, profile validation) are checked - too; the report-only gate never fails on client-only calls, which is why - the recipe asserts `clientOnly == []` on the JSON itself. It reports no - client-only call (every handwritten call exists on this checkout); + operational routes the public contract omits (F01 proves them) and the + `ProjectsClient` routes, which the server leaves out of its OpenAPI + document on purpose (`include_in_schema=False` while Projects is + experimental; F36 drives them), so its other client-ahead exceptions + (meta-profiles, profile validation) are checked too; the report-only + gate never fails on client-only calls, which is why the recipe asserts + `clientOnly == []` on the JSON itself. It reports no client-only call + (every handwritten call exists on this checkout); `server-only.txt` lists the operations without a literal client call (42 on this checkout: the conversation-scoped bash, file, git and VS Code aliases the client reaches by rewriting paths at runtime, the canvas diff --git a/.agents/skills/verify-agent-server/references/feature-map/F36-projects.md b/.agents/skills/verify-agent-server/references/feature-map/F36-projects.md index 5d4853aa80..bfc49e31e1 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F36-projects.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F36-projects.md @@ -42,16 +42,21 @@ Routes: `GET /api/projects/{project_id}/artifacts`, `GET /api/projects/{project_id}/artifacts/{name}`, `DELETE /api/projects/{project_id}/artifacts/{name}`, `GET /api/projects/{project_id}/memory`, -`GET`/`PUT`/`DELETE /api/projects/{project_id}/memory/{name}`, +`GET /api/projects/{project_id}/memory/{name}`, +`PUT /api/projects/{project_id}/memory/{name}`, +`DELETE /api/projects/{project_id}/memory/{name}`, `GET /api/projects`, `POST /api/projects`, `GET /api/projects/{project_id}`, `PATCH /api/projects/{project_id}`, `DELETE /api/projects/{project_id}`, `GET /api/projects/{project_id}/conversations`, `DELETE /api/projects/{project_id}/conversations/{conversation_id}`, `POST /api/projects/{project_id}/conversations/{conversation_id}/resolve`, `POST /api/projects/{project_id}/conversations/{conversation_id}/reopen`, -`GET /api/projects/{project_id}/conversations/{conversation_id}/messages`, -`POST /api/conversations/{conversation_id}/events`, -`POST /api/conversations/{conversation_id}/children`, `POST /api/conversations`, `POST /api/conversations/{conversation_id}/fork` (server-owned tags) +`GET /api/projects/{project_id}/conversations/{conversation_id}/messages` + +Projects also changes conversation routes that other families own: the +sender and confirmation guard of `POST .../events` (F06), the coordinator's +workers on `POST .../children` (F04), and the server-owned tags on a start +(F04) and a fork (F10). ## Sub-features @@ -63,23 +68,30 @@ Routes: `GET /api/projects/{project_id}/artifacts`, - `F36.coordinator`: creating a project with `coordinator` starts a new, idle coordinator conversation tagged `project=` with the client's other tags, the coordinator prompt (naming its own id) and the coordinator tools; the conversations endpoint reports it as `coordinator`, apart from the other conversations. A project never adopts an existing conversation. - `F36.coordinator-missing`: a coordinator with an unknown `agent_profile_id` is 404 and creates no project. - `F36.start-member`: only the coordinator starts conversations in its project: its `POST .../children` starts a member tagged `project=`, and a member's own `POST .../children` is 409. -- `F36.client-tags`: a client cannot start or fork a conversation whose `project` tag names an existing project: 422, and nothing is created. The other project keys (`turn`, `ask`, `resolved`, `resolvenote`), or a `project` tag naming no project, are ordinary client tags. +- `F36.client-tags`: a client cannot start or fork a conversation whose `project` tag names an existing project: 422, and nothing is created. The other project keys (`turn`, `ask`, `resolved`, `resolvenote`, `prs`), or a `project` tag naming no project, are ordinary client tags. - `F36.attention`: conversations are grouped `needs_you` (waiting for confirmation), then `working`, `idle` (finished or never run), newest-updated first within a group; non-members are left out. -- `F36.remove-conversation`: `DELETE .../conversations/{conversation_id}` clears the `project` tag and the board tags (`resolved`, `resolvenote`, `turn`, `ask`), and no other tag; a second delete, or a delete through another project, is 404. +- `F36.remove-conversation`: `DELETE .../conversations/{conversation_id}` clears the `project` tag and the board tags (`resolved`, `resolvenote`, `turn`, `ask`, and `prs` per `F36.pull-requests`), and no other tag; a second delete, or a delete through another project, is 404. - `F36.remove-coordinator`: removing the coordinator from the project is 409. - `F36.message-rest`: `POST /api/conversations/{id}/events` saves the message and starts a run. Without `sender_conversation_id` the message has no `sender`; a member sending to a conversation outside its project is 422. - `F36.turn`: when a member's run finishes, its `turn` tag says whether the agent's last message asked something (`asking`, attention `asking`) or not (`finished`), and an `asking` turn saves the question, in one line of at most 200 characters, as the `ask` tag (reported as `ask`); a new user message clears both. -- `F36.resolved`: a `resolved` tag (`user` or `auto`) puts a member in the `resolved` group, reported as `resolved`; a new user message clears it. Children do not copy `resolved` or `turn`. `POST .../resolve` and `.../reopen` set and clear it (204, 404 for a non-member), and a client's tag `PATCH` cannot change `project`, `turn`, `ask`, `resolved` or `resolvenote`. +- `F36.resolved`: a `resolved` tag (`user` or `auto`) puts a member in the `resolved` group, reported as `resolved`; a new user message clears it. Children do not copy `resolved` or `turn`. `POST .../resolve` and `.../reopen` set and clear it (204, 404 for a non-member), and a client's tag `PATCH` cannot change `project`, `turn`, `ask`, `resolved`, `resolvenote` or `prs`. +- `F36.auto-resolve`: the resolution judge, on the `resolution_judge` LLM profile the first judged turn creates from the conversation's LLM, resolves a member's plain answer (`resolved=auto`, `resolvenote` like `1.00 nothing`, shown as `resolve_note`) and makes a hand-off `turn=asking` with the reply's last paragraph as `ask`; over that question the member's `{"by": "agent"}` resolve is 409 and a coordinator's message leaves it asking. +- `F36.needs-you`: a turn that leaves part of the task undone, with no question, is `finished` and not resolved; once the rest is done the judge resolves it. +- `F36.judge-failed`: when the resolution judge fails (its `resolution_judge` profile cannot reach a model), a member's finished turn is `turn=asking` with the reply's last paragraph as `ask`, listed as `asking`, and is not resolved. +- `F36.pull-requests`: when a member's agent runs `gh pr create` and the command prints a GitHub pull request URL, the server adds `owner/repo#number` to its `prs` tag, oldest first, kept across turns; `GET .../conversations` reports it as `pull_requests` (`{owner, repo, number}`). A URL printed by another command is not recorded, and a client's tag `PATCH` cannot change `prs`. - `F36.self-resolve`: a member with `resolve_conversation` resolves its own conversation when nothing is left, such as after the user's "merged": `POST .../resolve` with `{"by": "agent"}` answers 202 at once (the agent is inside its own tool call, which holds the state lock) and sets `resolved=auto` and `resolvenote=agent`, never over the user's `resolved=user`, and the resolution judge does not run on that turn. -- `F36.artifacts-rest`: a project's artifacts can be uploaded (multipart), listed with `source: user`, downloaded byte for byte and deleted; unsafe names are 400. +- `F36.artifacts-rest`: a project's artifacts can be uploaded (multipart), listed with `source: user`, downloaded byte for byte and deleted; unsafe names are 400. `POST .../artifacts/from-conversation` copies a member's workspace file as `source: agent` with that conversation's id; a path outside its workspace is 400 and a conversation outside the project 404. - `F36.artifacts-tool`: a member with `write_artifact` saves a file from its workspace; the list shows it with `source: agent` and the conversation's id, and a path outside the workspace is refused. With `list_artifacts` and `read_artifact` it sees the project's files and reads one: small UTF-8 text inline, anything else copied into its workspace (`.agent_tmp/artifacts/` unless it names a path inside the workspace). - `F36.memory-rest`: `PUT/GET/DELETE /api/projects/{id}/memory/{name}` keep `MEMORY.md` and lowercase `.md` topic files under `projects//memory/`; `GET .../memory` lists them, index first. A write or delete with a stale `base_revision` (or a create over an existing file) is 409 with the current file; other names are 400. - `F36.memory-tool`: an agent with `read_project_memory` and `update_project_memory` reads `MEMORY.md` and adds a line to it with the revision it read; a child a project member starts gets `MEMORY.md` in its first message, marked as data. - `F36.persist-restart`: projects and membership survive a server restart. +- `F36.flag-off`: turning `projects` off makes every `/api/projects` route 404 `Projects are turned off` at once, without a restart, and writes nothing; the projects on disk and the members' `project` tags stay, and turning it on again serves them (200) at once. +- `F36.runtime-env`: `OH_RUNTIME_EXPERIMENTAL_FEATURES` replaces the stored switch for the routes: `{"projects": false}` makes them 404 while the stored switch (and the list) says on, `{"projects": true}` serves them while it says off, and an unparsable value counts as every switch off. +- `F36.flag-off-tools`: while `projects` is off a coordinator's project tool answers `Projects is turned off`, its child is an ordinary child that keeps the `project` tag and wakes no coordinator, and after turning it on again that child is on the board. - `F36.delete`: `DELETE /api/projects/{id}` removes the record, then the conversations endpoint is 404, and member conversations lose the `project` tag and the board's tags (their other tags stay). - `F36.corrupted-file`: an unparsable or future-schema `projects.json` makes reads and writes 409 and is never rewritten. - `F36.tool-start-and-list`: a coordinator with `start_child_conversation` and `list_workers` starts a child that joins the project, and its list call reports that child with its attention group. The child's system prompt names the coordinator, and its first message has `sender: "conversation:"`. -- `F36.wake`: when a member finishes a run the coordinator started (its task or follow-up message), the coordinator gets one user message with the member's final message, `sender: "conversation:"` and an "Update from" header, and runs on it. Each message the user writes in a member, and the member's final reply to it, reaches the coordinator as a copy for its awareness (`[Message from the user in ...`, `[Reply in ...`, sender the member) without running it (`F36.turn`). +- `F36.wake`: when a member finishes a run the coordinator started (its task or follow-up message), the coordinator gets one user message with the member's final message, `sender: "conversation:"` and an "Update from" header, and runs on it. The coordinator's own turns get a `turn` tag but are never judged or resolved. Each message the user writes in a member, and the member's final reply to it, reaches the coordinator as a copy for its awareness (`[Message from the user in ...`, `[Reply in ...`, sender the member) without running it (`F36.turn`). - `F36.read`: `GET .../conversations/{conversation_id}/messages` lists a member's latest messages (with senders) and `finish`, oldest first; the `read_worker` tool reads it. - `F36.tool-message`: `message_worker` delivers a follow-up to a finished conversation and requests a run: the conversation gets a second user message, sent by the coordinator (`sender` and a header line saying it is not the user), and runs again. - `F36.mention`: tools and headers name conversations as mentions, `[Title](conversation:)`: `list_workers` lists each conversation that way, and a coordinator asked which conversation handles a task answers with that mention. @@ -329,14 +341,19 @@ Preconditions: Requires: `F36.start-member`, `F36.create` ```sh control-agent-server api PATCH "/api/conversations/$M1" --json '{"tags": {"qa": "f36"}}' --expect 200 + control-agent-server api GET "/api/conversations/$M1" --check tags.project eq "$P2" --check tags.turn exists --until-ok 30 control-agent-server api DELETE "/api/projects/$P1/conversations/$M1" --expect 404 \ --check detail eq 'Conversation not in project' --save F36.remove-conversation/other-project control-agent-server api DELETE "/api/projects/$P2/conversations/$M1" --expect 200 --check deleted eq true --save F36.remove-conversation/delete control-agent-server api DELETE "/api/projects/$P2/conversations/$M1" --expect 404 - control-agent-server api GET "/api/conversations/$M1" --check tags not-contains project --check tags.qa eq f36 - ``` - Only the owning project can remove it, only once, and only the `project` - tag goes. + control-agent-server api GET "/api/conversations/$M1" --check tags not-contains project --check tags.qa eq f36 \ + --check tags.turn missing --check tags.ask missing --check tags.resolved missing --check tags.resolvenote missing \ + --save F36.remove-conversation/after + ``` + Only the owning project can remove it, only once, and the `project` tag + and the board's tags (here `turn`, and `resolved` when the judge resolved + it) go while the client's own tag stays. `F36.pull-requests` shows `prs` + cleared the same way. - **Coordinator membership (`F36.remove-coordinator`).** Requires: `F36.coordinator` ```sh @@ -404,6 +421,13 @@ Preconditions: control-agent-server api POST "/api/projects/$P2/conversations/$W2/resolve" --expect 204 control-agent-server api GET "/api/projects/$P2/conversations" --check conversations contains '"resolved": "user"' \ --check conversations contains '"attention": "resolved"' --save F36.resolved/conversations + control-agent-server api POST "/api/projects/$P2/conversations/$W2/reopen" --expect 204 --save F36.resolved/reopen + control-agent-server api GET "/api/conversations/$W2" --check tags.project eq "$P2" --check tags.resolved missing \ + --check tags.resolvenote missing --save F36.resolved/reopened-by-route + control-agent-server api POST "/api/projects/$P2/conversations/$W2/reopen" --expect 204 + control-agent-server api POST "/api/projects/$P2/conversations/$W1/reopen" --expect 404 + control-agent-server api POST "/api/projects/$P2/conversations/$W2/resolve" --expect 204 + control-agent-server api GET "/api/conversations/$W2" --check tags.resolved eq user control-agent-server api POST "/api/conversations/$W2/events" --json '{"role": "user", "content": [{"type": "text", "text": "Reply with the single word reopened and finish."}], "run": true}' --expect 200 control-agent-server conversation wait "$W2" --until finished --timeout 180 control-agent-server api GET "/api/conversations/$W2" --check tags.resolved ne user --check tags.turn eq finished --until-ok 10 \ @@ -411,8 +435,9 @@ Preconditions: control-agent-server conversation events "$W2" --kinds MessageEvent --contains '"sender": "conversation:' --expect-count 1 ``` A client's tag `PATCH` keeps the membership and the `turn` the server set. - The user's resolve shows as `resolved: "user"`; a new message reopens the - conversation. The reply to that message is judged again, so a trivial turn + The user's resolve shows as `resolved: "user"`; `reopen` clears it (again + 204 when nothing is left to clear, 404 for `W1`, which is not a member), + and after a second resolve a new message reopens the conversation. The reply to that message is judged again, so a trivial turn may come back as `resolved: "auto"`, never `user`. - **Resolved automatically (`F36.auto-resolve`).** Requires: `F36.message-rest` @@ -462,6 +487,67 @@ Preconditions: With half the task left and no question, the turn is `finished` and not resolved (Idle). Once the second haiku is written, the judge reads the first message as context and resolves. Verified 2026-11-10 with deepseek-flash. +- **The judge fails (`F36.judge-failed`).** + Requires: `F36.auto-resolve` + Needs: `llm` + ```sh + control-agent-server api POST /api/profiles/resolution_judge \ + --json '{"llm": {"model": "openai/qa-f36-judge", "api_key": "qa-f36-judge-key", "base_url": "http://127.0.0.1:9/v1", "num_retries": 0}}' --expect 201 + JF=$(control-agent-server api POST "/api/conversations/$CAT/children" \ + --json '{"title": "qa-f36-judge-down", "task": "What is 17 times 23? Answer in one sentence and finish. Do not use tools.", "isolation": "shared"}' \ + --expect 201 --field conversation_id) + control-agent-server conversation wait "$JF" --until finished --timeout 180 + control-agent-server api GET "/api/conversations/$JF" --check tags.turn eq asking --check tags.ask matches '391' \ + --check tags.resolved missing --until-ok 120 --save F36.judge-failed/asking + control-agent-server api GET "/api/projects/$P2/conversations" --check conversations contains qa-f36-judge-down \ + --check conversations contains '"attention": "asking"' --save F36.judge-failed/board + control-agent-server logs --grep 'Resolution judge failed' --expect-min 1 + control-agent-server api DELETE /api/profiles/resolution_judge --expect 200 + ``` + The same plain answer that `F36.auto-resolve` resolves waits on the user + when nobody could judge it: `turn=asking`, the reply's last paragraph as + `ask`, no `resolved`. The recipe deletes the broken judge profile, so the + next judged turn copies the conversation's LLM again. +- **Pull requests a member opened (`F36.pull-requests`).** Translated: a + stub `gh` in the shared workspace prints a pull request URL, as the real + `gh pr create` does; no GitHub account is used. + Requires: `F36.coordinator` + Needs: `llm`, `tmux` + ```sh + mkdir -p "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f36-cat/qa-bin" + cat > "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f36-cat/qa-bin/gh" <<'EOF' + #!/bin/sh + n=$(cat "$0.count" 2>/dev/null || echo 40); n=$((n + 1)); echo "$n" > "$0.count" + echo "https://github.com/qa-f36/qa-repo/pull/$n" + EOF + chmod +x "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f36-cat/qa-bin/gh" + PRW=$(control-agent-server api POST "/api/conversations/$CAT/children" \ + --json '{"title": "qa-f36-prs", "task": "Run these two shell commands with the terminal tool, one at a time, exactly as written: first `echo https://github.com/qa-f36/qa-repo/pull/99`, then `qa-bin/gh pr create --title qa --body qa`. Then reply with the URL the second one printed and finish.", "isolation": "shared"}' \ + --expect 201 --field conversation_id) + control-agent-server conversation wait "$PRW" --until finished --timeout 300 + control-agent-server conversation events "$PRW" --kinds ActionEvent --contains 'gh pr create' --expect-min 1 + control-agent-server api GET "/api/conversations/$PRW" --check tags.prs eq 'qa-f36/qa-repo#41' --until-ok 30 --save F36.pull-requests/tag + control-agent-server api GET "/api/projects/$P2/conversations" --field conversations \ + | jq -e '.[] | select(.title == "qa-f36-prs") | .pull_requests == [{"owner": "qa-f36", "repo": "qa-repo", "number": 41}]' + control-agent-server api GET "/api/projects/$P2/conversations" --check conversations contains '"number": 41' --save F36.pull-requests/board + control-agent-server conversation wait "$CAT" --until idle,finished --timeout 300 + control-agent-server conversation send "$PRW" --wait --timeout 300 \ + --text 'Run `qa-bin/gh pr create --title qa2 --body qa2` once with the terminal tool, then reply with the URL it printed and finish.' + control-agent-server api GET "/api/conversations/$PRW" --check tags.prs eq 'qa-f36/qa-repo#41,qa-f36/qa-repo#42' --until-ok 30 \ + --save F36.pull-requests/second + control-agent-server api PATCH "/api/conversations/$PRW" --json '{"tags": {"prs": "qa-f36/qa-other#1", "qa": "f36"}}' --expect 200 + control-agent-server api GET "/api/conversations/$PRW" --check tags.prs eq 'qa-f36/qa-repo#41,qa-f36/qa-repo#42' --check tags.qa eq f36 + control-agent-server conversation wait "$CAT" --until idle,finished --timeout 300 + control-agent-server api DELETE "/api/projects/$P2/conversations/$PRW" --expect 200 --check deleted eq true + control-agent-server api GET "/api/conversations/$PRW" --check tags.prs missing --check tags.project missing --check tags.turn missing \ + --check tags.qa eq f36 --save F36.pull-requests/removed + ``` + The `echo` of a URL is not a pull request and is not recorded; the stub's + `gh pr create` adds `qa-f36/qa-repo#41`, the board lists it as + `pull_requests`, and the next turn's `gh pr create` appends `#42` to it. + A client's tag `PATCH` keeps the server's value, and removing the + conversation from the project clears `prs` with the board's other tags. + Needs `jq`. - **The worker resolves itself (`F36.self-resolve`).** Requires: `F36.auto-resolve` Needs: `llm` @@ -482,7 +568,7 @@ Preconditions: `resolve_conversation` and ends its turn, and the note is `agent`, not a judge's probability. An agent's resolve never replaces the user's. - **Artifacts over REST (`F36.artifacts-rest`).** - Requires: `F36.coordinator` + Requires: `F36.coordinator`, `F36.unknown-project` ```sh printf '# qa-f36 notes\n' > "$AGENT_SERVER_VERIFY_RUN/qa-f36-notes.md" control-agent-server api POST "/api/projects/$P2/artifacts" --file "file=$AGENT_SERVER_VERIFY_RUN/qa-f36-notes.md" \ @@ -493,8 +579,24 @@ Preconditions: control-agent-server api POST "/api/projects/$P2/artifacts" --file "file=$AGENT_SERVER_VERIFY_RUN/qa-f36-notes.md" --form name=.hidden --expect 400 control-agent-server api DELETE "/api/projects/$P2/artifacts/qa-f36-notes.md" --expect 200 --check deleted eq true control-agent-server api GET "/api/projects/$P2/artifacts" --check artifacts len-eq 0 + printf 'qa f36 from cat\n' > "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f36-cat/qa-f36-cat-notes.txt" + control-agent-server api POST "/api/projects/$P2/artifacts/from-conversation" \ + --json "{\"conversation_id\": \"$CAT\", \"path\": \"qa-f36-cat-notes.txt\", \"name\": \"qa-f36-copied.txt\"}" \ + --expect 201 --check name eq qa-f36-copied.txt --check source eq agent --check size eq 16 \ + --check conversation_id eq "$(printf %s "$CAT" | tr -d -)" --save F36.artifacts-rest/from-conversation + control-agent-server api GET "/api/projects/$P2/artifacts/qa-f36-copied.txt" --expect 200 --raw-out "$AGENT_SERVER_VERIFY_RUN/qa-f36-copied.txt" + cmp "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f36-cat/qa-f36-cat-notes.txt" "$AGENT_SERVER_VERIFY_RUN/qa-f36-copied.txt" + control-agent-server api POST "/api/projects/$P2/artifacts/from-conversation" \ + --json "{\"conversation_id\": \"$CAT\", \"path\": \"/etc/hostname\"}" --expect 400 + control-agent-server api POST "/api/projects/$P2/artifacts/from-conversation" \ + --json "{\"conversation_id\": \"$W1\", \"path\": \"qa-f36-cat-notes.txt\"}" --expect 404 --check detail eq 'Conversation not in project' + control-agent-server api DELETE "/api/projects/$P2/artifacts/qa-f36-copied.txt" --expect 200 + control-agent-server api GET "/api/projects/$P2/artifacts" --check artifacts len-eq 0 ``` - The download matches the upload byte for byte; a dot name is refused. + The download matches the upload byte for byte; a dot name is refused. A + file in the coordinator's workspace is copied in as an `agent` artifact + with its conversation's id; a path outside the workspace is 400 and `W1`, + which is not a member, is 404. The recipe deletes both artifacts again. - **Model-backed: an agent saves an artifact (`F36.artifacts-tool`).** Requires: `F36.artifacts-rest` Needs: `llm` @@ -516,6 +618,9 @@ Preconditions: - **Project memory over REST (`F36.memory-rest`).** Requires: `F36.coordinator` ```sh + for N in $(control-agent-server api GET "/api/projects/$P2/memory" --expect 200 --field files | jq -r '.[].name'); do + control-agent-server api DELETE "/api/projects/$P2/memory/$N" --expect 200 --check deleted eq true + done control-agent-server api GET "/api/projects/$P2/memory" --expect 200 --check files len-eq 0 REV=$(control-agent-server api PUT "/api/projects/$P2/memory/MEMORY.md" --json '{"content": "- The user prefers short answers.\n"}' \ --expect 200 --check revision matches '^[0-9a-f]{16}$' --field revision) @@ -528,8 +633,10 @@ Preconditions: control-agent-server api DELETE "/api/projects/$P2/memory/deploy.md" --expect 200 --check deleted eq true control-agent-server api GET "/api/projects/$P2/memory/MEMORY.md" --check revision eq "$REV" --check content contains 'short answers' ``` - The index is created, a blind overwrite is refused with the current file, - and a topic file is added, listed after the index and deleted. + The coordinator may already have written to the project's memory on its + own (earlier bullets give it turns), so the recipe first deletes what is + there. The index is created, a blind overwrite is refused with the current + file, and a topic file is added, listed after the index and deleted. - **Model-backed: an agent keeps project memory (`F36.memory-tool`).** Requires: `F36.memory-rest` Needs: `llm` @@ -547,10 +654,17 @@ Preconditions: --contains 'Project memory: MEMORY.md as it was when this conversation started' --expect-count 1 \ --save F36.memory-tool/child control-agent-server conversation events "$MEMCHILD" --kinds SystemPromptEvent --contains 'Deploys go through staging.' --expect-count 1 + control-agent-server conversation wait "$MEMCHILD" --until finished --timeout 300 + control-agent-server api GET "/api/conversations/$CAT/events/search" --query kind=MessageEvent --query 'body=Update from' \ + --query sort_order=TIMESTAMP_DESC --check items contains "conversation:${MEM//-/}" \ + --check items contains "conversation:${MEMCHILD//-/}" --until-ok 120 + control-agent-server conversation wait "$CAT" --until idle,finished --timeout 300 ``` The member merges its line into the index it read, and the next conversation the coordinator starts sees the index, with that line, in the - `` block of its system prompt. + `` block of its system prompt. Both members' updates reach the + coordinator, and the recipe waits for its run on them, so a later restart + does not leave an update unanswered. - **Restart (`F36.persist-restart`).** Requires: `F36.create`, `F36.coordinator` ```sh @@ -559,6 +673,50 @@ Preconditions: control-agent-server api GET "/api/projects/$P2/conversations" --check coordinator.title eq qa-f36-cat --save F36.persist-restart/conversations ``` Records and membership are read back from disk. +- **Projects turned off and on (`F36.flag-off`).** + Requires: `F36.create`, `F36.coordinator`, `F36.memory-rest` + ```sh + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": false}' --expect 200 --check enabled eq false + control-agent-server api GET /api/projects --expect 404 --check detail eq 'Projects are turned off' --save F36.flag-off/list + control-agent-server api GET "/api/projects/$P2" --expect 404 --check detail eq 'Projects are turned off' + control-agent-server api GET "/api/projects/$P2/conversations" --expect 404 --check detail eq 'Projects are turned off' + control-agent-server api GET "/api/projects/$P2/memory/MEMORY.md" --expect 404 --check detail eq 'Projects are turned off' + control-agent-server api GET "/api/projects/$P2/artifacts" --expect 404 --check detail eq 'Projects are turned off' + control-agent-server api POST /api/projects --json '{"name": "qa-f36-while-off"}' --expect 404 --save F36.flag-off/create + control-agent-server state cat "$PJ" --check projects len-eq 2 --not-contains qa-f36-while-off + control-agent-server api GET "/api/conversations/$CAT" --check tags.project eq "$P2" + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": true}' --expect 200 --check enabled eq true + control-agent-server api GET /api/projects --expect 200 --check projects len-eq 2 --check projects.1.id eq "$P2" --save F36.flag-off/on-again + control-agent-server api GET "/api/projects/$P2/conversations" --expect 200 --check coordinator.title eq qa-f36-cat + control-agent-server api GET "/api/projects/$P2/memory/MEMORY.md" --expect 200 --check content contains 'short answers' + ``` + Both switches take effect on the next request, with no restart. While off + every route, a create included, is 404 and nothing is written; the + projects, their memory and the coordinator's `project` tag are kept and + served again once it is on. +- **The host's switch wins (`F36.runtime-env`).** A Docker runtime's + server gets the user's switches from its host in + `OH_RUNTIME_EXPERIMENTAL_FEATURES`; a restart with `--env` stands in for + that host. + Requires: `F36.flag-off` + ```sh + control-agent-server restart --env 'OH_RUNTIME_EXPERIMENTAL_FEATURES={"projects": false}' + control-agent-server api GET /api/settings/experimental --check 5.key eq projects --check 5.enabled eq true + control-agent-server api GET /api/projects --expect 404 --check detail eq 'Projects are turned off' --save F36.runtime-env/host-off + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": false}' --expect 200 + control-agent-server restart --reset-config --env 'OH_RUNTIME_EXPERIMENTAL_FEATURES={"projects": true}' + control-agent-server api GET /api/settings/experimental --check 5.key eq projects --check 5.enabled eq false + control-agent-server api GET /api/projects --expect 200 --check projects len-eq 2 --save F36.runtime-env/host-on + control-agent-server restart --reset-config --env 'OH_RUNTIME_EXPERIMENTAL_FEATURES=qa-not-json' + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": true}' --expect 200 + control-agent-server api GET /api/projects --expect 404 --check detail eq 'Projects are turned off' --save F36.runtime-env/host-unparsable + control-agent-server restart --reset-config + control-agent-server api GET /api/projects --expect 200 --check projects len-eq 2 + ``` + With the variable set, the routes follow the host's value and ignore the + stored switch, which `GET /api/settings/experimental` still lists (it + shows the stored switch, not the host's). Without it, the stored switch + (on) applies again. - **Delete a project (`F36.delete`).** Requires: `F36.create`, `F36.coordinator` ```sh @@ -605,9 +763,12 @@ Preconditions: --expect-count 1 --save F36.wake/update control-agent-server conversation events "$CAT" --kinds MessageEvent --contains "\"sender\": \"conversation:$CHILD_HEX\"" --expect-count 1 control-agent-server conversation wait "$CAT" --until finished --timeout 300 + control-agent-server api GET "/api/conversations/$CAT" --check tags.turn matches '^(finished|asking)$' --check tags.resolved missing \ + --check tags.resolvenote missing --until-ok 30 --save F36.wake/coordinator-tags ``` When the child finished, its final message reached the coordinator as one - update (sender: the child), and the coordinator ran on it. + update (sender: the child), and the coordinator ran on it. Its own turn + is recorded in `turn` but never resolved. - **Model-backed: read a conversation (`F36.read`).** Requires: `F36.wake` Needs: `llm` @@ -642,13 +803,15 @@ Preconditions: control-agent-server conversation events "$CAT" --kinds ObservationEvent --contains "[qa-f36-child](conversation:$CHILD_HEX)" --expect-min 1 \ --save F36.mention/list control-agent-server conversation send "$CAT" --wait --timeout 300 --text 'Which conversation did you start for the qa-f36-child task? Answer in one sentence that names it for me. Do nothing else.' - control-agent-server conversation events "$CAT" --kinds ActionEvent,MessageEvent --show 1 --full --save F36.mention/answer \ - | jq -r '.events[-1] | select(.source == "agent") | (.llm_message.content[0].text // .action.message // "")' \ + control-agent-server conversation events "$CAT" --kinds ActionEvent,MessageEvent --show 5 --full --save F36.mention/answer \ + | jq -r '[.events[] | select(.source == "agent")][-1] | (.llm_message.content[0].text // .action.message // "")' \ | grep -Fi -e "](conversation:$CHILD_HEX)" -e "](conversation:$CHILD)" ``` The list observation names the child as a mention, and the coordinator's last message, its answer, carries the same link although the prompt does not - say how to write it (the routing guidance does). Canvas accepts the target + say how to write it (the routing guidance does). The recipe reads the last + agent event among the last five, since the child's update can land right + after the answer. Canvas accepts the target UUID with or without dashes, case-insensitively. Needs `jq`. - **Model-backed: no approval by message (`F36.tool-message-refuses-confirmation`).** Requires: `F36.attention`, `F36.tool-start-and-list` @@ -686,6 +849,30 @@ Preconditions: agent saved with it: the coordinator and worker keep their prompt, their `` block and their tools, and each uses a project tool on its next turn. There is no Projects-specific restore code. Needs `jq`. +- **Model-backed: project tools while Projects is off (`F36.flag-off-tools`).** + Requires: `F36.restore` + Needs: `llm` + ```sh + control-agent-server conversation wait "$CAT" --until idle,finished --timeout 300 + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": false}' --expect 200 + control-agent-server conversation send "$CAT" --wait --timeout 300 \ + --text 'This is the user, writing to you directly (not a copy from another conversation): call your list_workers tool now, exactly once, even if you called it before, and tell me what it returned.' + control-agent-server conversation events "$CAT" --kinds ObservationEvent --contains 'Projects is turned off' --expect-min 1 \ + --save F36.flag-off-tools/refused + OFFCHILD=$(control-agent-server api POST "/api/conversations/$CAT/children" \ + --json '{"title": "qa-f36-off-child", "task": "Reply with the single word ok and finish. Do not use tools.", "isolation": "shared"}' \ + --expect 201 --check parent_conversation_id eq "$CAT" --field conversation_id) + control-agent-server conversation wait "$OFFCHILD" --until finished --timeout 180 + control-agent-server api GET "/api/conversations/$OFFCHILD" --check tags.project eq "$P2" \ + --save F36.flag-off-tools/child + OFF_HEX=$(printf %s "$OFFCHILD" | tr -d -) + control-agent-server conversation events "$CAT" --kinds MessageEvent --contains "conversation:$OFF_HEX" --expect-count 0 + control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": true}' --expect 200 + control-agent-server api GET "/api/projects/$P2/conversations" --check conversations contains qa-f36-off-child --save F36.flag-off-tools/board + ``` + The coordinator keeps its tools, and `list_workers` answers that Projects + is turned off. Its child starts as any child does, copying its parent's + tags (`project` included), and sends the coordinator no update; it joins the board when Projects is on again. - **Corrupted file (`F36.corrupted-file`).** Run last: it replaces the file. ```sh printf '{not json' > "$PJ_FILE" @@ -712,10 +899,24 @@ Preconditions: conversation with `project` (or another server-owned tag), and no route adds an existing conversation; making a conversation the coordinator is the one way an existing conversation joins. +- These guards hold only while `projects` is on. With it off, a client can + start a conversation tagged `project=` and `PATCH` + `project`, `prs`, `resolved` and the other board tags onto any + conversation; once `projects` is back on, those conversations are members + with that board state (a faked `resolved=user` and `pull_requests`), and a + client can no longer remove the tags. Whether that is intended needs a + product decision + ([#7054](https://github.com/OpenHands/software-agent-sdk/issues/7054), open). - Each turn the coordinator starts in a member (its task, its messages) wakes the coordinator when it ends, so the coordinator runs after every member recipe here. Wait for it (`conversation wait "$CAT" --until idle,finished`) before checking its status. +- A graceful restart while the coordinator runs on a member's update pauses + that run, as for any running conversation (`F05.restart-running-graceful`), + and nothing runs it again: the coordinator stays `paused` with the update + unanswered, because startup catch-up counts a saved update as delivered + ([#7058](https://github.com/OpenHands/software-agent-sdk/issues/7058), open). + Recipes wait for the coordinator's run before a restart. - A question for the user (`turn=asking`) stays open until the user writes in the conversation, or answers the coordinator's `ask_user` card made for it (`for_conversation`), which passes their answer on. Other messages from the diff --git a/.agents/skills/verify-agent-server/references/feature-map/README.md b/.agents/skills/verify-agent-server/references/feature-map/README.md index 3335952fa2..38bc81ba72 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/README.md +++ b/.agents/skills/verify-agent-server/references/feature-map/README.md @@ -8,9 +8,9 @@ consumers). Read this index before driving the server, then use the matching family file as the recipe. -Maintenance baseline: main@0275e714df62d3dd66eb51d518dd1da953ec2c16 (2026-11-17). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. +Maintenance baseline: main@3a893a1b26de7ec02cc0dd2427906eea6aea3fc7 (2026-11-21). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. -The map has **1098 sub-features**. +The map has **1107 sub-features**. ## Baseline preconditions @@ -82,7 +82,7 @@ them honest: [../maintenance.md](../maintenance.md). | F15 | [Git changes, diffs, commits and repository search](F15-git.md) | working-tree changes and diffs, explicit refs, nested repositories, commit history and per-commit changes, scoped variants, GitHub repository search | `/api/git/*`, `/api/conversations/{id}/git/*`, `GET /api/git/repositories/search` | llm, git, node | 36 | | F16 | [Workspace file serving and VS Code endpoints](F16-workspace-serving-and-vscode.md) | the per-conversation static workspace server (index, content types, ranges, traversal, cookie auth) and the VS Code status and URL routes | `GET /api/conversations/{id}/workspace[/{path}]`, `/api/vscode/*`, `/api/conversations/{id}/vscode/*` | llm, git | 27 | | F17 | [Workspace registry and parents](F17-workspaces-registry.md) | saved workspaces and parents: add, list, delete, validation, persistence, concurrent writers, corrupted files | `GET`/`POST`/`DELETE /api/workspaces`, `/api/workspaces/parents` | git, node | 24 | -| F18 | [Server settings, settings schemas and the secrets store](F18-settings-and-secrets.md) | persisted settings with redacted, plaintext and encrypted exposure, sparse PATCH, settings schemas, the named secrets store, encryption at rest, cipher-key changes | `GET`/`PATCH /api/settings`, `/api/settings/agent-schema`, `/conversation-schema`, `/api/settings/secrets[/{name}]` | git, node | 32 | +| F18 | [Server settings, settings schemas and the secrets store](F18-settings-and-secrets.md) | persisted settings with redacted, plaintext and encrypted exposure, sparse PATCH, settings schemas, the named secrets store, encryption at rest, cipher-key changes, the experimental feature switches (list, switch, persist, edits on disk) | `GET`/`PATCH /api/settings`, `/api/settings/agent-schema`, `/conversation-schema`, `/api/settings/secrets[/{name}]`, `GET /api/settings/experimental`, `PATCH /api/settings/experimental/{key}` | git, node | 34 | | F19 | [MCP server settings, connection test and OAuth](F19-mcp.md) | MCP server settings CRUD, connection tests (stdio and HTTP), install-time OAuth jobs and token refresh, a model-driven MCP tool call, MCP servers added to a running conversation, conversations that start before a slow MCP server is ready | `/api/settings/mcp/{key}`, `POST /api/mcp/test`, `/api/mcp/oauth/*`, `POST /api/conversations/{id}/mcp` | llm; launch `--webhook-sink` | 41 | | F20 | [LLM profiles](F20-llm-profiles.md) | LLM profiles: save, read, validate against real and stubbed providers, activate, rename, delete, name rules, limits, store contention, restart | `/api/profiles[/{name}]`, `/validate`, `/activate`, `/rename` | llm, node | 37 | | F21 | [Agent profiles](F21-agent-profiles.md) | agent profiles (OpenHands and ACP kinds): CRUD, default seed, activation, rename, delete, materialize, conversations started from a profile, store contention | `/api/agent-profiles[/{name}]`, `/rename`, `/{id}/activate`, `/materialize` | llm | 33 | @@ -99,7 +99,7 @@ them honest: [../maintenance.md](../maintenance.md). | F32 | [Event webhooks and product telemetry](F32-webhooks-and-telemetry.md) | event webhooks (event batches, conversation updates, headers, buffering, flush, limits, retries) and product telemetry (off by default, exporter lifecycle, sanitized payloads, consent) | behavior only: `webhooks` / `OH_WEBHOOKS_*`, `OH_TELEMETRY_*` | llm; launch `--webhook-sink` | 30 | | F33 | [Python SDK remote client (RemoteConversation, RemoteWorkspace)](F33-sdk-remote-client.md) | the Python SDK in this checkout against the server: RemoteWorkspace commands, files, git and settings readers; RemoteConversation create, run, callbacks, streaming, reattach, fork, confirmation, pause, interrupt, stop hooks, client tools, completion logs | behavior only: `Workspace(host=...)`, `RemoteWorkspace`, `RemoteConversation` | llm, tmux, git | 43 | | F34 | [TypeScript client (@openhands/typescript-client)](F34-typescript-client.md) | the TypeScript client against this checkout: server and version gate, workspace, bash, conversation, state, events and socket clients, hooks and MCP clients, typed batch-events, loadPlugin, git commits, archive and deferred-init clients, endpoint audit and drift | behavior only: `@openhands/typescript-client` (`clients/typescript`) | llm, node, uvx, git, tmux, network | 38 | -| F36 | [Projects and coordinator tools](F36-projects.md) | project records, membership through the `project` conversation tag (only the coordinator starts members), conversations grouped by attention, confirmation-safe messages, coordinator `list_workers` and `message_worker` tools, asks, project memory, restart, corrupted file | `GET`/`POST /api/projects`, `GET`/`PATCH`/`DELETE /api/projects/{id}`, `GET /api/projects/{id}/conversations`, `DELETE /api/projects/{id}/conversations/{conversation_id}`, `POST /api/conversations/{id}/children` (coordinator only), `GET .../conversations/{conversation_id}/messages`, `POST /api/conversations/{id}/events` (sender, confirmation guard) | llm; recipes: tmux | 30 | +| F36 | [Projects and coordinator tools](F36-projects.md) | project records, membership through the `project` conversation tag (only the coordinator starts members), conversations grouped by attention, confirmation-safe messages, coordinator `list_workers` and `message_worker` tools, asks, a failed resolution judge, pull requests a member opened (`prs`), artifacts, project memory, restart, the `projects` switch (stored and runtime env), corrupted file | `GET`/`POST /api/projects`, `GET`/`PATCH`/`DELETE /api/projects/{id}`, `GET /api/projects/{id}/conversations`, `DELETE /api/projects/{id}/conversations/{conversation_id}`, `.../resolve`, `.../reopen`, `/api/projects/{id}/artifacts[/from-conversation|/{name}]`, `/api/projects/{id}/memory[/{name}]`, `POST /api/conversations/{id}/children` (coordinator only), `GET .../conversations/{conversation_id}/messages`, `POST /api/conversations/{id}/events` (sender, confirmation guard) | llm; recipes: tmux | 37 | | F37 | [Emoji reactions and feature flags](F37-reactions.md) | the `conversation_reactions` feature flag (stored and runtime env), the user's reactions over REST and the TypeScript client, validation, active-branch targets, the agent's `react`/`unreact` tools, what the model sees | `POST /api/conversations/{id}/reactions` (the flag's routes are F18's) | llm | 12 | | F38 | [Conversation mentions](F38-conversation-mentions.md) | the `conversation_mentions` feature flag, the agent's read-only `find_conversations` and `read_conversation` tools, mention links `[Title](conversation:)` in its answers | behavior only: agent tools (the flag's routes are F18's) | llm | 5 | | F39 | [Message edits](F39-message-edits.md) | the `message_edits` feature flag, the agent's `edit_message` tool (`str_replace` on its own earlier messages), the `MessageEditEvent` clients show in place, what the model sees, reactions on an edited message | behavior only: `edit_message` tool, `MessageEditEvent` (the flag's routes are F18's) | llm | 5 | diff --git a/.agents/skills/verify-sdk/programs/S03/switching.py b/.agents/skills/verify-sdk/programs/S03/switching.py index aeecb143ef..0e4fd50886 100644 --- a/.agents/skills/verify-sdk/programs/S03/switching.py +++ b/.agents/skills/verify-sdk/programs/S03/switching.py @@ -12,6 +12,7 @@ tool the switch_llm tool (sync run, or `tool-async` for arun) route route_task_to_model with structured classes and errors route-direct route_task_to_model with a direct prompt template + route-router route_task_to_model on a RouterLLM agent: pins a route """ import asyncio @@ -445,6 +446,83 @@ def stats(conversation) -> dict: ) conversation.close() +elif mode == "route-router": + # The agent LLM is a RouterLLM: the classifier picks one of the router's + # route names (or a unique underlying model) and the tool pins that route, + # keeping the router as the conversation LLM. + from openhands.sdk.llm import RouterLLM + from openhands.sdk.llm.meta_profile_store import MetaProfile, MetaProfileClass + + route = {"tool": "route_task_to_model", "args": {}} + + def router_run(reply: str) -> dict: + with ( + FakeProvider([route, {"finish": "gave up"}]) as planner, + FakeProvider([{"text": reply}]) as classifier, + FakeProvider([{"finish": "done on strong"}]) as strong, + ): + store.save( + "classifier", + fake_llm(classifier, "classifier", "openai/gpt-4o-mini"), + include_secrets=True, + ) + router = RouterLLM( + usage_id="router", + llms_for_routing={ + "planner": fake_llm(planner, "route-planner", "openai/gpt-4o-mini"), + "strong": fake_llm(strong, "route-strong", "openai/gpt-4.1"), + }, + ) + meta = MetaProfile( + classifier_model="classifier", + classes=[ + MetaProfileClass(description="hard debugging", model="strong") + ], + ) + agent = Agent( + llm=router, + tools=[ + Tool( + name="ClassifyAndSwitchLLMTool", + params={ + "active_meta_profile": "router", + "meta_profile": meta.model_dump(mode="json"), + }, + ) + ], + ) + conversation = Conversation( + agent=agent, workspace=WORKSPACE, visualizer=None + ) + conversation.send_message("Debug this hard failure on the right route.") + conversation.run() + prompt = ( + json.dumps(classifier.requests[0]["body"]) + if classifier.requests + else "" + ) + llm = conversation.agent.llm + row = { + "status": conversation.state.execution_status.value, + "agent_llm": type(llm).__name__, + "selection": llm.status().selection.model_dump(mode="json") + if isinstance(llm, RouterLLM) + else None, + "requests": { + "planner": len(planner.requests), + "classifier": len(classifier.requests), + "strong": len(strong.requests), + }, + "classifier_saw_routes": "- planner" in prompt and "- strong" in prompt, + "observation": (observations(conversation) or [None])[0], + } + conversation.close() + return row + + out["by_route"] = router_run('{"route": "strong", "reason": "hard"}') + out["by_model"] = router_run('{"model": "openai/gpt-4.1"}') + out["unknown"] = router_run('{"route": "nonexistent"}') + else: raise SystemExit(f"unknown mode {mode}") diff --git a/.agents/skills/verify-sdk/programs/S06/prompt_variants.py b/.agents/skills/verify-sdk/programs/S06/prompt_variants.py index d419b1a01d..1a015d9332 100644 --- a/.agents/skills/verify-sdk/programs/S06/prompt_variants.py +++ b/.agents/skills/verify-sdk/programs/S06/prompt_variants.py @@ -2,7 +2,7 @@ choose it? Each variant builds an Agent the way a user would (model name, inline prompt, -template file, persona, planning preset, project vendor files), runs one +template file, persona, planning and project presets, project vendor files), runs one scripted turn, and reads the SystemPromptEvent the conversation saved. """ @@ -78,6 +78,18 @@ def error_of(fn): out["planning_differs"] = planning != static_a out["planning_tags"] = tags(planning) +# Project presets: a coordinator and a worker, each with a project context. +project_ctx = AgentContext(project_context="Project Apollo: ship the launcher.") +for name in ("coordinator", "worker"): + static, dynamic = system_prompt( + system_prompt_filename=f"system_prompt_project_{name}.j2", + agent_context=project_ctx, + ) + out[f"project_{name}_tags"] = tags(static) + out[f"project_{name}_context"] = ( + "\nProject Apollo: ship the launcher.\n" in dynamic + ) + # Model-specific sections. for name, model, kwargs, agent_kwargs in [ ("plain", "custom-made-model", {}, {}), diff --git a/.agents/skills/verify-sdk/references/feature-map/README.md b/.agents/skills/verify-sdk/references/feature-map/README.md index 21b2663520..525b89c118 100644 --- a/.agents/skills/verify-sdk/references/feature-map/README.md +++ b/.agents/skills/verify-sdk/references/feature-map/README.md @@ -10,7 +10,7 @@ agent-server map and follows its layout. Family IDs here are `Snn`, so they never collide with that map's `Fnn`. -Maintenance baseline: main@722e5d403cf75d4e032b79296cfaf178f8678a93 (2026-11-20). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. +Maintenance baseline: main@3a893a1b26de7ec02cc0dd2427906eea6aea3fc7 (2026-11-21). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. The map has **504 sub-features**, **35 of them untested** (no unit or cross test exercises the behavior; most of those have an example script). @@ -119,6 +119,15 @@ Left out of this map, with the reason: `start_child_conversation` tool posts to the agent server's child-launch route and is attached only by a host; owned by the agent-server map's F27 (the mention tools reuse its URL and key helpers, see S16). +- `openhands-tools/openhands/tools/project/`: the project tools + (`list_workers`, `message_worker`, `read_worker`, the artifact and project + memory tools, `resolve_conversation`) call the agent server's projects API + and are attached only by the server to a project's coordinator and workers; + owned by the agent-server map's F36. +- `openhands-sdk/openhands/sdk/errors.py`: transport-neutral domain error + categories (`NotFound`, `Conflict`, `Invalid`, `TooLarge`, ...) that stores + raise and the agent server maps to HTTP statuses; an SDK program meets them + only as the store errors S03 and S19 check. - `openhands-sdk/pyproject.toml`, `openhands-tools/pyproject.toml`: package metadata and dependency pins, with no SDK behavior of their own; every run's `uv sync --dev` exercises them, and a dependency bump calls for the full diff --git a/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md b/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md index acbe2f0d99..19ad69279c 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md +++ b/.agents/skills/verify-sdk/references/feature-map/S03-llm-routing-profiles.md @@ -1,6 +1,6 @@ # LLM registry, routing, fallback, profiles and model switching -A Python program that uses more than one model meets this family. `LLMRegistry` holds `LLM`s by `usage_id`, gives each one independent metrics, and notifies subscribers; every `LocalConversation` has its own `conversation.llm_registry`, into which it registers every LLM the agent reaches (the agent's, the condenser's, router children), and `conversation_stats` tracks them all, per conversation: an `LLM` object used by two conversations counts each one's usage in that conversation's stats and the total in `llm.metrics`. A `RouterLLM` (`MultimodalRouter`, `RandomRouter`) is itself an `LLM` that picks one of `llms_for_routing` per call; `MultimodalRouter` sends a request with images, or one too large for the secondary model, to `primary` and everything else to `secondary`. The router owns its route table and its selection: in `auto` mode it routes by its policy (the base `RouterLLM` uses its first route) and records the route it chose as `last_route_name`, in `manual` mode every call goes to the pinned route; `router.status()` (a `ModelRouterStatus`) reports both, and on a conversation whose agent LLM is a router `conversation.switch_routed_llm(route)`, `resume_auto_llm_routing()` and `update_model_router_selection(ModelRouterSelection(...))` change the selection without replacing the router, while `conversation.add_model_router_route(profile)` adds a saved profile as a route (wrapping a plain LLM in a `RouterLLM` first). `LLM(fallback_strategy=FallbackStrategy(fallback_llms=[...]))` retries a call that failed with a transient error on saved profiles, in order, and records their cost under the caller's usage. `LLMProfileStore` saves and loads named `LLM` configurations as JSON (default `~/.openhands/profiles`, secrets masked unless asked for). A running conversation can change model with `conversation.switch_profile(name)` or `conversation.switch_llm(llm)`, the agent can do it itself with the `switch_llm` tool, or a classifier LLM can pick a profile through the `route_task_to_model` tool configured by a meta-profile. `LLM.subscription_login(vendor="openai", model=...)` returns an `LLM` backed by a ChatGPT Plus/Pro subscription through OAuth instead of an API key. +A Python program that uses more than one model meets this family. `LLMRegistry` holds `LLM`s by `usage_id`, gives each one independent metrics, and notifies subscribers; every `LocalConversation` has its own `conversation.llm_registry`, into which it registers every LLM the agent reaches (the agent's, the condenser's, router children), and `conversation_stats` tracks them all, per conversation: an `LLM` object used by two conversations counts each one's usage in that conversation's stats and the total in `llm.metrics`. A `RouterLLM` (`MultimodalRouter`, `RandomRouter`) is itself an `LLM` that picks one of `llms_for_routing` per call; `MultimodalRouter` sends a request with images, or one too large for the secondary model, to `primary` and everything else to `secondary`. The router owns its route table and its selection: in `auto` mode it routes by its policy (the base `RouterLLM` uses its first route) and records the route it chose as `last_route_name`, in `manual` mode every call goes to the pinned route; `router.status()` (a `ModelRouterStatus`) reports both, and on a conversation whose agent LLM is a router `conversation.switch_routed_llm(route)`, `resume_auto_llm_routing()` and `update_model_router_selection(ModelRouterSelection(...))` change the selection without replacing the router, while `conversation.add_model_router_route(profile)` adds a saved profile as a route (wrapping a plain LLM in a `RouterLLM` first). `LLM(fallback_strategy=FallbackStrategy(fallback_llms=[...]))` retries a call that failed with a transient error on saved profiles, in order, and records their cost under the caller's usage. `LLMProfileStore` saves and loads named `LLM` configurations as JSON (default `~/.openhands/profiles`, secrets masked unless asked for). A running conversation can change model with `conversation.switch_profile(name)` or `conversation.switch_llm(llm)`, the agent can do it itself with the `switch_llm` tool, or a classifier LLM can pick a profile (or, when the agent LLM is a router, a route) through the `route_task_to_model` tool configured by a meta-profile. `LLM.subscription_login(vendor="openai", model=...)` returns an `LLM` backed by a ChatGPT Plus/Pro subscription through OAuth instead of an API key. Source: `openhands-sdk/openhands/sdk/llm/llm_registry.py`, `openhands-sdk/openhands/sdk/llm/router/`, `openhands-sdk/openhands/sdk/llm/fallback_strategy.py`, `openhands-sdk/openhands/sdk/llm/llm_profile_store.py`, `openhands-sdk/openhands/sdk/llm/meta_profile_store.py`, `openhands-sdk/openhands/sdk/llm/provider_connection_store.py`, `openhands-sdk/openhands/sdk/llm/auth/`, `openhands-sdk/openhands/sdk/llm/llm.py`, `openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py`, `openhands-sdk/openhands/sdk/conversation/conversation_stats.py`, `openhands-sdk/openhands/sdk/tool/builtins/switch_llm.py`, `openhands-sdk/openhands/sdk/tool/builtins/classify_and_switch_llm.py`, `openhands-sdk/openhands/sdk/settings/model.py` @@ -36,7 +36,7 @@ Examples: `examples/01_standalone_sdk/05_use_llm_registry.py`, `examples/01_stan - `S03.switch-llm-object`: `conversation.switch_llm(llm)` swaps to an `LLM` object without touching the profile store, installs the posted object even when its `usage_id` is already registered (#5779), refreshes an LLM condenser that shared the agent's credentials while leaving an independently configured one alone, and `ask_agent` follows the switch. Unit: `tests/sdk/conversation/test_switch_model.py::test_switch_llm_swaps_when_store_empty`, `::test_switch_llm_refreshes_llm_condenser_credentials`, `::test_switch_llm_preserves_independent_condenser_profile`, `tests/sdk/conversation/test_ask_agent.py::test_ask_agent_llm_refreshes_after_switch_llm`. Example: none. Real-LLM: none. - `S03.switch-keeps-usage`: usage stays with the role (`usage_id`) in `conversation_stats` across `switch_llm`: the costs recorded before the switch remain, and what the condenser and `ask_agent` spend on the new model is added under their own `usage_id`s (`condenser`, `ask-agent-llm`), each cost entry naming the model that served it. Unit: `tests/sdk/conversation/local/test_llm_session_isolation.py::test_condenser_usage_is_counted_after_switch_llm`, `::test_ask_agent_usage_is_counted_after_every_switch`, `::test_stats_callback_keeps_firing_after_switch_llm`. Example: none. Real-LLM: none. - `S03.switch-llm-tool`: with `SwitchLLMTool` among the agent's tools (`include_default_tools=["FinishTool", "SwitchLLMTool"]`, or `OpenHandsAgentSettings.create_agent()`, which adds it when `tools` is left unset), the model calls `switch_llm(profile_name=...)`; the tool description lists saved profiles, the conversation switches, and a missing profile or load error comes back as an error observation. The tool is exempt from `filter_tools_regex` and does not deadlock under `arun()`. Unit: `tests/sdk/tool/test_switch_llm.py`, `tests/sdk/conversation/test_switch_model.py::test_switch_llm_tool_during_arun_does_not_deadlock`. Example: `examples/01_standalone_sdk/49_switch_llm_tool.py`. Real-LLM: none. -- `S03.route-task-to-model`: with `OpenHandsAgentSettings(enable_classify_and_switch_llm_tool=True, meta_profile=..., active_meta_profile=...)` the agent gets `route_task_to_model`; calling it sends recent conversation text to the classifier profile, which returns a class index (structured `classes`) or a model name (`prompt_template` with `{{ instance_text }}` and `{{ model_table }}`), and the conversation switches to that profile. Classifier cost lands in the conversation stats under one usage bucket; unknown classes, missing classifier or target profiles produce error observations. Unit: `tests/sdk/tool/test_classify_and_switch_llm.py`. Example: `examples/01_standalone_sdk/59_route_task_to_model.py`. Real-LLM: none. +- `S03.route-task-to-model`: with `OpenHandsAgentSettings(enable_classify_and_switch_llm_tool=True, meta_profile=..., active_meta_profile=...)` the agent gets `route_task_to_model`; calling it sends recent conversation text to the classifier profile, which returns a class index (structured `classes`) or a model name (`prompt_template` with `{{ instance_text }}` and `{{ model_table }}`), and the conversation switches to that profile. When the agent LLM is a `RouterLLM`, the classifier is shown the router's route names (with each route's label, model and matching class description) instead of profiles, may answer a route name or a model that only one route uses, and the tool pins that route with `switch_routed_llm`, keeping the router as the agent LLM; an answer that names no route is an error observation and leaves the selection unchanged. Classifier cost lands in the conversation stats under one usage bucket; unknown classes, missing classifier or target profiles produce error observations. Unit: `tests/sdk/tool/test_classify_and_switch_llm.py`. Example: `examples/01_standalone_sdk/59_route_task_to_model.py`. Real-LLM: none. - `S03.meta-profile-store`: `MetaProfileStore` saves, lists, loads and deletes meta-profiles under `OH_PERSISTENCE_DIR` (or `~/.openhands`), validates direct-prompt templates (requires `{{ instance_text }}`, no `classes`), and enforces `max_profiles`. Unit: `tests/sdk/llm/test_meta_profile_store.py`. Example: none. Real-LLM: none. - `S03.subscription-login`: `LLM.subscription_login(vendor="openai", model=..., auth_method="browser"|"device_code", force_login=False, open_browser=True)` runs an OAuth PKCE browser flow or a device-code flow, caches credentials under `~/.openhands/auth/`, refreshes expired tokens, asks for consent once, and returns an `LLM` pointed at the Codex Responses endpoint; models outside the supported set are rejected. Unit: `tests/sdk/llm/auth/test_openai.py`, `tests/sdk/llm/auth/test_credentials.py`. Example: `examples/01_standalone_sdk/35_subscription_login.py`. Real-LLM: none. - `S03.subscription-requests`: a subscription `LLM` omits parameters the Codex endpoint rejects or mishandles (`prompt_cache_retention`, `include`, `reasoning`, `temperature`, `max_output_tokens`), adds no temperature on retry, keeps `is_subscription` across serialization, and rehydrates from `LLM.load_from_env`. Unit: `tests/sdk/llm/test_subscription_mode.py`, `tests/sdk/test_settings.py::test_llm_load_from_env_rehydrates_subscription_runtime`. Example: none. Real-LLM: none. @@ -295,6 +295,19 @@ Every model in the scripted bullets is a real `LLM` answering from its own local --check status eq finished --check active_model eq openai/gpt-4.1-nano --check requests.cheap eq 1 \ --check stats contains classifier:classifier --check bad_class.error eq true ``` +- **`route_task_to_model` on a router conversation (`S03.route-task-to-model`).** The agent LLM is a base `RouterLLM` with routes `planner` (first, so `auto` sends the first turn there) and `strong` (`openai/gpt-4.1`), and the tool comes from `Tool(name="ClassifyAndSwitchLLMTool", params={...})`; the classifier answers `{"route": "strong"}`, then the model `openai/gpt-4.1`, then a route that does not exist. + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S03/switching --arg route-router \ + --check by_route.status eq finished --check by_route.agent_llm eq RouterLLM \ + --check by_route.selection.mode eq manual --check by_route.selection.manual_route_name eq strong \ + --check by_route.requests.strong eq 1 --check by_route.classifier_saw_routes eq true \ + --check by_route.observation.text contains "switched to RouterLLM route 'strong'" \ + --check by_model.agent_llm eq RouterLLM --check by_model.selection.manual_route_name eq strong \ + --check by_model.requests.strong eq 1 \ + --check unknown.observation.error eq true --check unknown.observation.text contains "did not resolve to an available RouterLLM route" \ + --check unknown.selection.mode eq auto --check unknown.requests.strong eq 0 + ``` - **Meta-profiles (`S03.meta-profile-store`).** ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) diff --git a/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md b/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md index aff352b3d5..3d42bfe32a 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md +++ b/.agents/skills/verify-sdk/references/feature-map/S06-context-condensers.md @@ -14,7 +14,7 @@ Examples: `examples/01_standalone_sdk/14_context_condenser.py`, `examples/01_sta - `S06.context-skills`: repo skills (no triggers) go into the system prompt and triggered skills into the user message whose text matches a trigger (case-insensitive), with already-activated skills skipped; `disabled_skills` drops named skills and duplicate names are rejected. Skill loading and discovery are family S07. Unit: `tests/sdk/context/test_agent_context.py`. Example: `examples/01_standalone_sdk/03_activate_skill.py`. Real-LLM: `tests/integration/tests/t09_invoke_skill.py`. - `S06.context-secrets`: secret names and descriptions (never values) from `AgentContext(secrets=...)` and the conversation's secret registry are listed in the system prompt; raw string secrets are redacted on serialization and round-trip through a cipher. Unit: `tests/sdk/context/test_agent_context.py::test_get_system_message_suffix_with_secrets_only`, `tests/sdk/context/test_agent_context_serialization.py`, `tests/sdk/context/prompts/test_prompt_snapshot.py::test_dynamic_context_with_secret_registry`. Example: none. Real-LLM: none. - `S06.context-datetime`: `current_datetime` (a `datetime` or a string; defaults to now) is rendered in the dynamic part of the system prompt. Unit: `tests/sdk/context/test_agent_context.py::test_get_system_message_suffix_with_datetime_only`, `::test_agent_context_default_datetime`. Example: none. Real-LLM: none. -- `S06.system-prompt-registry`: the built-in prompt is assembled from registered sections (default and planning presets), its rendering for fixed inputs matches checked-in snapshots, and the static block has no dynamic content. Unit: `tests/sdk/context/prompts/test_prompt_snapshot.py`, `tests/sdk/context/prompts/test_registry.py`, `tests/sdk/context/prompts/test_default_registry.py`, `tests/sdk/context/prompts/test_planning_registry.py`, `tests/sdk/context/prompts/test_section.py`. Example: none. Real-LLM: none. +- `S06.system-prompt-registry`: the built-in prompt is assembled from registered sections (default, planning, and the `project_coordinator` and `project_worker` presets that `system_prompt_filename="system_prompt_project_coordinator.j2"` / `"system_prompt_project_worker.j2"` select; `AgentContext.project_context`, which the agent server sets for a project's members, renders as a `` block in the dynamic context), its rendering for fixed inputs matches checked-in snapshots, and the static block has no dynamic content. Unit: `tests/sdk/context/prompts/test_prompt_snapshot.py`, `tests/sdk/context/prompts/test_registry.py`, `tests/sdk/context/prompts/test_default_registry.py`, `tests/sdk/context/prompts/test_planning_registry.py`, `tests/sdk/context/prompts/test_project_registry.py`, `tests/sdk/context/prompts/test_section.py`. Example: none. Real-LLM: none. - `S06.system-prompt-model-specific`: GPT-5 and GPT-5 Codex models get their own prompt section (detected from the canonical model name or a variant override); Claude and Gemini vendor files are included only for their models. Unit: `tests/sdk/context/test_prompt_model_spec.py`, `tests/sdk/context/test_agent_context_model_specific.py`. Example: none. Real-LLM: none. - `S06.system-prompt-custom`: `Agent(system_prompt=...)` replaces the built-in prompt; `system_prompt_filename` accepts a relative or absolute template path (missing files raise); `system_prompt_kwargs` feed the template. Unit: `tests/sdk/context/test_prompt_absolute_path.py`, `tests/sdk/context/prompts/test_persona.py::test_inline_system_prompt_wins_over_persona`. Example: none. Real-LLM: none. - `S06.persona-soul`: a persona replaces the persona sections and keeps capability and dynamic sections; the identity line comes from `~/.openhands/SOUL.md` when present, else a built-in default. Unit: `tests/sdk/context/prompts/test_persona.py`, `tests/sdk/context/prompts/test_prompt_snapshot.py::test_soul_default_snapshot`, `::test_soul_custom_snapshot`. Example: none. Real-LLM: none. @@ -38,7 +38,7 @@ Examples: `examples/01_standalone_sdk/14_context_condenser.py`, `examples/01_sta ## How to get to it (agent POV) - `Agent(llm=..., tools=..., agent_context=AgentContext(...))` with `skills`, `system_message_suffix`, `user_message_suffix`, `secrets`, `current_datetime`, `disabled_skills`, `load_user_skills`, `load_public_skills`, `load_project_skills`, `load_memory`: `S06.context-*`, `S06.memory-*`. -- `Agent(system_prompt=..., system_prompt_filename="system_prompt.j2" | "system_prompt_planning.j2" | path, system_prompt_kwargs={...})`: `S06.system-prompt-*`. Files: `~/.openhands/SOUL.md` (`S06.persona-soul`), `~/.openhands/memory/MEMORY.md` and `/.openhands/memory/MEMORY.md` (`S06.memory-load`). `OH_PERSISTENCE_DIR` replaces `~/.openhands`. +- `Agent(system_prompt=..., system_prompt_filename="system_prompt.j2" | "system_prompt_planning.j2" | "system_prompt_project_coordinator.j2" | "system_prompt_project_worker.j2" | path, system_prompt_kwargs={...})`: `S06.system-prompt-*`. Files: `~/.openhands/SOUL.md` (`S06.persona-soul`), `~/.openhands/memory/MEMORY.md` and `/.openhands/memory/MEMORY.md` (`S06.memory-load`). `OH_PERSISTENCE_DIR` replaces `~/.openhands`. - `Agent(condenser=LLMSummarizingCondenser(llm=llm.model_copy(update={"usage_id": "condenser"}), max_size=..., max_tokens=..., keep_first=..., minimum_progress=..., hard_context_reset_max_retries=..., hard_context_reset_context_scaling=...))`: `S06.summarizing-*`, `S06.soft-vs-hard`, `S06.hard-reset`, `S06.summary-llm-streaming`. - `from openhands.sdk.context.condenser import NoOpCondenser, PipelineCondenser, LLMSummarizingCondenser, CondenserBase, RollingCondenser`: `S06.noop-condenser`, `S06.pipeline-condenser`; subclass `RollingCondenser` for a custom condenser. - `conversation.condense()`: `S06.manual-condense`. Events: `Condensation(forgotten_event_ids, summary, summary_offset)`, `CondensationRequest`, `CondensationSummaryEvent` in `openhands.sdk.event`: `S06.view`. @@ -82,6 +82,10 @@ The programs under `programs/S06/` record every scripted LLM call (`recorder.py` --check static_same_across_contexts eq true --check static_has_dynamic_values eq false \ --check dynamic_has_values eq true --check soul_default eq true \ --check planning_differs eq true --check planning_tags contains PLAN_STRUCTURE \ + --check project_coordinator_tags contains ROUTING --check project_coordinator_tags contains BRIEFS \ + --check project_coordinator_tags not-contains CODE_QUALITY --check project_coordinator_context eq true \ + --check project_worker_tags contains WHO_IS_SPEAKING --check project_worker_tags contains YOUR_REPORT \ + --check project_worker_tags contains CODE_QUALITY --check project_worker_context eq true \ --check plain_gpt5_section eq false --check gpt5_gpt5_section eq true --check gpt5_preamble eq true \ --check codex_gpt5_section eq true --check codex_preamble eq false \ --check canonical_gpt5_section eq true --check override_preamble eq true \ diff --git a/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md b/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md index 6cef6f6420..1173f9bb0b 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md +++ b/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md @@ -32,7 +32,7 @@ Examples: `examples/01_standalone_sdk/07_mcp_integration.py`, `examples/01_stand - `S09.tool-serialization`: `MCPToolDefinition` is frozen, serializes polymorphically with `kind`, is written in the MCP wire spelling (`inputSchema`) and loads either spelling, so conversations persisted under older `mcp` majors resume. Unit: `tests/sdk/mcp/test_mcp_tool_serialization.py`, `tests/sdk/mcp/test_mcp_tool_immutability.py`, `tests/sdk/mcp/test_mcp_wire_compat.py`. Example: none. Real-LLM: none. - `S09.runtime-add-servers`: `LocalConversation.add_mcp_servers({name: MCPServer})` on an initialized agent connects each enabled server before it returns and registers its tools as `_` (whatever the server count), adds the servers to `agent.mcp_config` and the persisted state, and the next LLM call offers them; before the first message it only records them and they connect with the agent like configured servers (bare names for a single server). A duplicate name and `transport="sse"` raise `ValueError`, a server that cannot start raises `MCPServerStartError` (an `MCPError` from `openhands.sdk.mcp`, with `server_name` and the client's connection `MCPError` as `__cause__`; message `Could not start MCP server : `) after closing the servers it already started, nothing refused is added, and a disabled server is recorded without connecting. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_add_mcp_servers_registers_modern_runtime_tools`, `test_add_mcp_servers_rejects_sse_transport`, `test_add_mcp_servers_accepts_running_conversation`). Example: none. Real-LLM: none. -- `S09.runtime-add-resume`: a conversation reopened from its persistence directory reconnects the servers added at runtime and the agent can call their tools by the names it was given: `ConversationState.runtime_mcp_servers` records the servers added after the conversation started, which keep `_` on every (re)start, while the other servers follow the one-or-many prefix rule counted without them. Since [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929) ([#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927)). Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_runtime_added_mcp_tools_keep_their_names_after_reopen`, `test_mcp_server_added_before_a_reopened_agent_starts_keeps_old_names`, `test_reopened_single_configured_mcp_server_stays_unprefixed`). Example: none. Real-LLM: none. +- `S09.runtime-add-resume`: a conversation reopened from its persistence directory reconnects the servers added at runtime and the agent can call their tools by the names it was given: `ConversationState.runtime_mcp_servers` records the servers added after the conversation started, which keep `_` on every (re)start, while the other servers follow the one-or-many prefix rule counted without them. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_runtime_added_mcp_tools_keep_their_names_after_reopen`, `test_mcp_server_added_before_a_reopened_agent_starts_keeps_old_names`, `test_reopened_single_configured_mcp_server_stays_unprefixed`). Example: none. Real-LLM: none. - `S09.startup-budget`: `LocalConversation(mcp_startup_budget=seconds)` lets agent initialization (the first `send_message`) return after the budget while slower servers keep starting; their tools join the agent when ready, the first step of `run()` waits for them (up to 30 s each), and one that fails late is logged and skipped. Without a budget (the default; the agent server passes 1.0) initialization waits for every server. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_startup_budget_starts_the_agent_before_a_slow_mcp_server`, `test_late_mcp_tools_are_closed_when_the_conversation_closed_first`, `test_first_step_waits_for_mcp_servers_that_missed_the_budget`). Example: none. Real-LLM: none. ## How to get to it (agent POV)