diff --git a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md index b1a46de0712..d1433dcfe05 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md @@ -742,6 +742,7 @@ Preconditions: ## Gotchas +- The 4 MiB frame cap of `F07.frame-too-large` applies to live delivery too, not only to replay: a socket open while a 5 MiB event is appended receives the events before it and then closes with 1013 `frame_too_large`, so later events never arrive on that socket; reconnecting with `after_seq` past the big event resumes. A client that does not reconnect past it loses the live stream ([#6934](https://github.com/OpenHands/software-agent-sdk/issues/6934), open). - `F07.reject-non-user`, `F07.error-frame`, `F07.error-non-json` and `F07.auth-frame-ignored` read the `error` frames at fixed positions after `sync` and the snapshot. While #6745 lasts, an `error` can overtake `sync` when the server is slow to subscribe, on an idle machine as well as under load (for `F07.auth-frame-ignored` as `frames.3.frame.type` `error`, for `F07.error-non-json` as an `error` frame ahead of `sync`), and the bullet then fails on its first position check. A fail of one of these four whose transcript shows that order is #6745, not a new defect; another attempt often passes. `F07.sync-first-during-run` reproduces that order deterministically. - Neither the SDK nor the TypeScript client uses this socket yet, so a regression here breaks no shipped consumer's tests; only these recipes and diff --git a/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md b/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md index 7c58e0ce6de..f31252e8617 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md @@ -783,6 +783,7 @@ KID=$(control-agent-server api POST /api/conversations \ ## Gotchas +- Rotating a secret with `POST /api/conversations/{id}/secrets` (`F08.secrets-override`) does not keep both values masked: after the update, output that prints the new value shows it in cleartext, and once the new value has been exported the old value is no longer masked either; both reach the persisted events. `F08.secrets-override` asserts only that the next command gets the new value. The SDK twin is `S13.secret-registry` (its rotated-secret bullet carries the same known failure) ([#7002](https://github.com/OpenHands/software-agent-sdk/issues/7002), open; fix proposed in [#7003](https://github.com/OpenHands/software-agent-sdk/pull/7003)). - `respond_to_confirmation` with `accept: true` is exactly a run request: it needs a run slot (429 when full), it is a no-op on a running conversation, and with nothing pending it simply resumes the agent. `accept: false` with diff --git a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md index c78782be5b3..545eebb4ffe 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F19-mcp.md @@ -70,7 +70,7 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`, - `F19.conversation-mcp-add-before-init`: on a conversation without a first message the route only records the server (no process, fast answer), with `env` values encrypted by the settings API decrypted for the child and stored encrypted in `base_state.json`; the first message connects it like any configured server. - `F19.conversation-mcp-add-errors`: an unknown conversation is 404, a server name the agent already has and an SSE server are 400 with the reason, a server without `command`/`url`, an unknown field and a non-UUID id are 422, none of them changes the agent's `mcp_config`, and a disabled server is stored without being spawned. - `F19.conversation-mcp-add-start-failure`: a server that cannot start (a command that exits at once) is refused with the route's 400 naming the server and the cause (`Could not start MCP server : Connection closed`, since [#6928](https://github.com/OpenHands/software-agent-sdk/pull/6928)) and is not added. -- `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (known bug [#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927), closed 2026-11-18 with no fix merged; fix proposed in [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929): with one server the tools come back unprefixed, so `qa-f19-rs_qa_echo` is `Tool ... not found`). +- `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (`qa-f19-rs_qa_echo`, prefixed even as the agent's only server). - `F19.conversation-mcp-slow-start`: a conversation whose stdio server delays its handshake by 8 s is created within a few seconds, the server is spawned, and the agent's first model request waits for it and offers its tools; the agent's call returns its result. - `F19.conversation-mcp-late-failure`: a server that fails after the startup budget costs only its own tools: the conversation starts at once, the first step waits for the failure, the other server's tools are offered, the run finishes and the log names the failed server. @@ -1263,7 +1263,7 @@ Preconditions: ``` The 400's `detail` names the server and the cause, and the server is not added (`agent.mcp_config` keeps no `qa-f19-dies`). -- **Added servers after a restart (`F19.conversation-mcp-add-restart`), known bug #6927.** +- **Added servers after a restart (`F19.conversation-mcp-add-restart`).** The stub calls the added tool, finishes, and after a restart and a new message calls the same tool again. ```sh @@ -1279,12 +1279,13 @@ Preconditions: control-agent-server restart control-agent-server api GET "/api/conversations/$RCID" --check agent.mcp_config.qa-f19-rs.command eq "$PY" --save F19.conversation-mcp-add-restart/after-restart control-agent-server conversation send "$RCID" --text 'Again.' --wait --until finished --timeout 90 - control-agent-server conversation events "$RCID" --kinds ObservationEvent --contains QA_MCP_ECHO:two --expect-kind ObservationEvent --save F19.conversation-mcp-add-restart/second-call # bug: matching eq 0 && kinds.AgentErrorEvent ge 1 + control-agent-server conversation events "$RCID" --kinds ObservationEvent --contains QA_MCP_ECHO:two --expect-kind ObservationEvent --save F19.conversation-mcp-add-restart/second-call ``` The server survives the restart in the conversation's agent and is - reconnected on the next message, but as the agent's only server its tools - come back bare (`qa_echo`), so the second call by the name the agent was - given ends in an `AgentErrorEvent` `Tool 'qa-f19-rs_qa_echo' not found`. + reconnected on the next message under the name the agent was given: the + conversation state records it as added at runtime (`runtime_mcp_servers`), + so its tools stay `qa-f19-rs_qa_echo` although it is the agent's only + server, and the second call returns `QA_MCP_ECHO:two`. - **A slow server does not hold up the start (`F19.conversation-mcp-slow-start`).** The F19 tools fixture with a handshake delayed by 8 s, inline on the conversation's agent; the stub reads one variable through it. @@ -1357,10 +1358,13 @@ Preconditions: request (`sink read --name --contains '"name": ""'`), not the first event. Tools added through `POST /api/conversations/{id}/mcp` are not in it either. -- Tools of a server added through the conversation route are always named - `_` while the agent is running, but a restart (or an add - before the first message) names them like configured servers: bare when it - is the agent's only server (`F19.conversation-mcp-add-restart`). +- Tools of a server added through the conversation route once the + conversation has started (its agent is initialized or it has events) are + named `_`, and keep that name across a restart + (`F19.conversation-mcp-add-restart`); the other servers are counted without + it for the two-or-more prefix rule. An add to a brand-new conversation + before it has started is named like a configured server: bare when it is + the agent's only server. - The conversation MCP route runs the connection inside the request: an initialized conversation answers only after the server has started (or failed, up to 30 s). Explicit `transport: sse` is refused, but a `/sse` URL diff --git a/.agents/skills/verify-sdk/programs/S13/secret_rotation.py b/.agents/skills/verify-sdk/programs/S13/secret_rotation.py new file mode 100644 index 00000000000..c8340c3e823 --- /dev/null +++ b/.agents/skills/verify-sdk/programs/S13/secret_rotation.py @@ -0,0 +1,100 @@ +"""Usage: secret_rotation.py + +A secret rotated mid-conversation with `update_secrets()`, and a program's own +`SecretSource` that returns a new value on each lookup. + +`DEPLOY_TOKEN` starts as `dep-v1-s13` and `OAUTH_TOKEN` (a rotating source) +answers `oauth-v-s13`. A scripted agent cats files that hold the values +without naming the secrets (as a token in a git remote or a log would be): + + run 1 cat v1.txt (dep-v1-s13 oauth-v1-s13) + update_secrets({"DEPLOY_TOKEN": "dep-v2-s13"}) + run 2 cat v2.txt (dep-v2-s13, never named yet) + echo $DEPLOY_TOKEN $OAUTH_TOKEN | wc -c (exports both: v2 and oauth-v2) + cat v1.txt (the previous values again) + +Each `*_leaks` field lists the cleartext values found in that observation; +`persisted_leaks` lists every value found in the persisted events. +""" + +from pathlib import Path + +from sdk_verify import emit, make_llm, run_paths + +from openhands.sdk import Agent, Conversation, Tool +from openhands.sdk.event import ObservationEvent +from openhands.sdk.secret import SecretSource +from openhands.tools.terminal import TerminalTool + + +paths = run_paths() +ws = paths["workspace"] +VALUES = ["dep-v1-s13", "dep-v2-s13", "oauth-v1-s13", "oauth-v2-s13"] + + +class Rotating(SecretSource): + calls: int = 0 + + def get_value(self) -> str | None: + self.calls += 1 + return f"oauth-v{self.calls}-s13" + + +(ws / "v1.txt").write_text("remote has dep-v1-s13 log has oauth-v1-s13\n") +(ws / "v2.txt").write_text("config has dep-v2-s13\n") + +script = [ + {"tool": "terminal", "args": {"command": "cat v1.txt"}}, + {"finish": "one"}, + {"tool": "terminal", "args": {"command": "cat v2.txt"}}, + { + "tool": "terminal", + "args": {"command": "echo $DEPLOY_TOKEN $OAUTH_TOKEN | wc -c"}, + }, + {"tool": "terminal", "args": {"command": "cat v1.txt"}}, + {"finish": "two"}, +] +rotating = Rotating() +c = Conversation( + agent=Agent(llm=make_llm(script), tools=[Tool(name=TerminalTool.name)]), + workspace=str(ws), + persistence_dir=str(paths["conversations"]), + visualizer=None, + secrets={"DEPLOY_TOKEN": "dep-v1-s13", "OAUTH_TOKEN": rotating}, +) +c.send_message("first") +c.run() +c.update_secrets({"DEPLOY_TOKEN": "dep-v2-s13"}) +c.send_message("second") +c.run() + +texts = [ + "".join(getattr(x, "text", "") for x in e.observation.to_llm_content) + for e in c.state.events + if isinstance(e, ObservationEvent) and e.tool_name == "terminal" +] + + +def leaks(text: str) -> list[str]: + return [v for v in VALUES if v in text] + + +persisted = "" +for f in sorted(Path(paths["conversations"]).rglob("events/*.json")): + persisted += f.read_text() + +emit( + { + "status": c.state.execution_status.value, + "observations": len(texts), + "before_rotation": texts[0] if texts else None, + "before_rotation_leaks": leaks(texts[0]) if texts else None, + "new_value": texts[1] if len(texts) > 1 else None, + "new_value_leaks": leaks(texts[1]) if len(texts) > 1 else None, + "export_wc": texts[2].split("\n")[0] if len(texts) > 2 else None, + "previous_values": texts[3] if len(texts) > 3 else None, + "previous_values_leaks": leaks(texts[3]) if len(texts) > 3 else None, + "source_lookups": rotating.calls, + "persisted_leaks": leaks(persisted), + } +) diff --git a/.agents/skills/verify-sdk/references/feature-map/README.md b/.agents/skills/verify-sdk/references/feature-map/README.md index 0dd0b1a8c75..21b26635203 100644 --- a/.agents/skills/verify-sdk/references/feature-map/README.md +++ b/.agents/skills/verify-sdk/references/feature-map/README.md @@ -10,9 +10,9 @@ agent-server map and follows its layout. Family IDs here are `Snn`, so they never collide with that map's `Fnn`. -Maintenance baseline: main@141423921fdaeb08cd5f53464a69e860da70b66f (2026-11-19). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. +Maintenance baseline: main@722e5d403cf75d4e032b79296cfaf178f8678a93 (2026-11-20). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. -The map has **504 sub-features**, **36 of them untested** (no unit or cross +The map has **504 sub-features**, **35 of them untested** (no unit or cross test exercises the behavior; most of those have an example script). ## What each sub-feature links @@ -83,7 +83,7 @@ an interactive login are labeled **blocked**. | S06 | [Agent context, system prompt, condensers and memory](S06-context-condensers.md) | `AgentContext`, system prompt sections and vendor files, persona, persistent memory, the condenser `View`, `LLMSummarizingCondenser`, `condense()`, overflow condensation, no-op and pipeline condensers | `Agent(agent_context=, system_prompt=, condenser=)`, `AgentContext(load_memory=)`, `conversation.condense()` | llm, tmux | 24 | 1 | | S07 | [Skills](S07-skills.md) | skill sources (programmatic, Markdown, AGENTS.md, SKILL.md, user, project, public, installed), keyword, task and path triggers, ``, `invoke_skill`, inline commands, disabling, install and marketplace, the `OH_PUBLIC_SKILLS_ENABLED` switch | `AgentContext(skills=, load_*_skills=, disabled_skills=)`, `.agents/skills/`, `install_skill`, `OH_PUBLIC_SKILLS_ENABLED` | llm, git, network | 25 | 0 | | S08 | [Plugins, marketplaces and the extension store](S08-plugins-marketplace.md) | both plugin layouts, path containment, commands as skills, merging into the agent, `resolved_plugins`, fetch cache, installed and ambient plugins, marketplace registry, `load_plugin()` | `Conversation(plugins=[PluginSource(...)])`, `Plugin.load`, `conversation.load_plugin()` | llm, git, network, uvx | 20 | 0 | -| S09 | [MCP client and tools](S09-mcp.md) | stdio, HTTP and SSE servers, auth and OAuth, enabled flag, timeouts, failing servers, schema conversion, observations, secrets in calls, reconnection, `list_changed`, tool filtering, servers added to a running conversation, the startup budget | `Agent(mcp_config=, filter_tools_regex=)`, `create_mcp_tools`, `MCPOAuthAuthentication`, `LocalConversation.add_mcp_servers`, `LocalConversation(mcp_startup_budget=)` | llm, uvx, node, network | 23 | 1 | +| S09 | [MCP client and tools](S09-mcp.md) | stdio, HTTP and SSE servers, auth and OAuth, enabled flag, timeouts, failing servers, schema conversion, observations, secrets in calls, reconnection, `list_changed`, tool filtering, servers added to a running conversation, the startup budget | `Agent(mcp_config=, filter_tools_regex=)`, `create_mcp_tools`, `MCPOAuthAuthentication`, `LocalConversation.add_mcp_servers`, `LocalConversation(mcp_startup_budget=)` | llm, uvx, node, network | 23 | 0 | | S10 | [Hooks](S10-hooks.md) | config format and discovery, matchers, command hooks, blocking tool calls and prompts, context injection, Stop, SessionStart and SessionEnd, `HookExecutionEvent`, prompt and agent hooks, plugin hooks | `Conversation(hook_config=HookConfig(...))`, `HookDefinition(type=)`, `HookManager` | llm, git | 21 | 1 | | S11 | [Sub-agents and delegation](S11-subagents-delegation.md) | the `task` tool lifecycle, agent registry, Markdown agent files and `AgentDefinition`, built-in agents, `SubAgentScope`, the legacy delegate executor | `Tool(name=TaskToolSet.name)`, `register_agent`, `.agents/agents/*.md` | llm | 24 | 2 | | S12 | [Critic and critic-driven refinement](S12-critic.md) | `CriticResult`, rule critics, evaluation modes, iterative refinement, `APIBasedCritic`, critic from settings | `Agent(critic=APIBasedCritic(...))`, `event.critic_result` | llm, network | 15 | 4 | diff --git a/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md b/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md index 666688be93f..45ac86637b0 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md +++ b/.agents/skills/verify-sdk/references/feature-map/S01-agent-loop.md @@ -237,6 +237,7 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` --check finish_schema_fields contains facts --check finish_schema_fields contains message \ --check rejected.reserved_name eq ValueError --check rejected.field_collision eq ValueError ``` + Expected: both rejected terminal schemas raise `ValueError`, and the executors the registry already built for them are closed, so no tmux server from the program is left running after it exits (`ps -eo args | grep "[t]mux.*"` is empty). - **Context overflow (`S01.context-overflow`).** A context-window error with an `LLMSummarizingCondenser` (emits a `CondensationRequest` and the run continues) and without one (the run ends in ERROR). ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -278,7 +279,6 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)` ## Gotchas -- A rejected `response_schema` on a terminal tool raises `ValueError` after the terminal has already started its tmux server, and nothing closes it: `close()` leaves the server running after the program exits ([#6748](https://github.com/OpenHands/software-agent-sdk/issues/6748)). `S01.structured-output` passes but leaves two such servers per run; end them with `TMUX_TMPDIR= tmux -L kill-server`. - `01_hello_world.py` is in `_EXCLUDED_EXAMPLES` in `tests/examples/test_examples.py`, so the `test-examples` label never runs it; the other seven examples here do run there (`pytest tests/examples/test_examples.py --run-examples`). - All examples read `LLM_API_KEY`, `LLM_MODEL`, `LLM_BASE_URL`; `56_structured_output.py` defaults `LLM_MODEL` to `anthropic/claude-sonnet-4-5-20250929`, the rest to `gpt-5.5`. `17_image_input.py` asserts `llm.vision_is_active()` and fails fast on a text-only model; it also fetches an image from python.org. - Examples run with the repo root as workspace and write files there (`FACTS.txt`, `image_report.md`, the document from example 18). diff --git a/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md b/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md index 1ffd143267c..ad167778c3c 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md +++ b/.agents/skills/verify-sdk/references/feature-map/S02-llm-core.md @@ -106,7 +106,7 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check final_text eq "the answer" --check final_summary.0 eq "plan the reply" --check content_text eq "the answer" \ --check summary_as_reasoning eq "plan the reply" ``` - Expected: the summary reaches the callback as `delta.reasoning_content`, as chat-path reasoning does, and the answer as `delta.content`; the final message carries both. Known failure (reproduced 2026-11-17, [#6695](https://github.com/OpenHands/software-agent-sdk/issues/6695)): no summary delta reaches the callback at all (`summary_as_reasoning` is empty; every delta is the message's `content`, chunk id `msg_1`). With the installed LiteLLM (1.105.1) a `response.reasoning_summary_text.delta` event is parsed as a `GenericEvent`, never the `ReasoningSummaryTextDeltaEvent` that `LLM._process_stream_event` looks for, and that branch would emit it as `delta.content` rather than `delta.reasoning_content`. The final message still has the summary. + Expected: the summary reaches the callback as `delta.reasoning_content`, as chat-path reasoning does, and the answer as `delta.content`; the final message carries both. Known failure (reproduced 2026-11-20, [#6695](https://github.com/OpenHands/software-agent-sdk/issues/6695)): no summary delta reaches the callback at all (`summary_as_reasoning` is empty; every delta is the message's `content`, chunk id `msg_1`). With the installed LiteLLM (1.108.3) a `response.reasoning_summary_text.delta` event is parsed as a `GenericEvent`, never the `ReasoningSummaryTextDeltaEvent` that `LLM._process_stream_event` looks for, and that branch would emit it as `delta.content` rather than `delta.reasoning_content`. The final message still has the summary. - **Cache-write tokens on the Responses path (`S02.metrics`).** The same program: the Responses reply reports `input_tokens_details.cache_write_tokens: 600`, and a Chat Completions reply reports `prompt_tokens_details.cache_creation_tokens: 600`. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -297,7 +297,7 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check tool_image.delivered_or_refused_once eq true \ --check tool_image.error eq LLMBadRequestError --check tool_image.retries eq 0 --check tool_image.requests eq 0 ``` - Expected: with the installed LiteLLM (1.105.1) the adapter drops both images, so each call is refused once as `LLMBadRequestError` and the endpoint receives nothing; the refusal is not retried, also when LiteLLM wraps it as a provider error, while a provider failure on a request whose images were all delivered keeps its usual mapping and retries. + Expected: with the installed LiteLLM (1.108.3) the adapter drops both images, so each call is refused once as `LLMBadRequestError` and the endpoint receives nothing; the refusal is not retried, also when LiteLLM wraps it as a provider error, while a provider failure on a request whose images were all delivered keeps its usual mapping and retries. - **The completion-log flag and its alias (`S02.log-completions`).** ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) @@ -361,7 +361,7 @@ The scripted bullets point a real `LLM` (`openai/...`, `anthropic/...`) at `prog --check results.duplicate_tool_call_id.error_type eq LLMMalformedConversationHistoryError \ --check results.parallel_wrong_order.error_type eq LLMMalformedConversationHistoryError ``` - Expected: DeepSeek rejects all eight, each as `LLMMalformedConversationHistoryError`. Before #6658, the three whose provider message is "Messages with role 'tool' must be a response to a preceding message with 'tool_calls'" (unmatched tool result, duplicate id, parallel results in the wrong order) came back as the generic `LLMBadRequestError`; now the classifier knows that message. + Expected: DeepSeek rejects all eight, each as `LLMMalformedConversationHistoryError`, including the three whose provider message is "Messages with role 'tool' must be a response to a preceding message with 'tool_calls'" (unmatched tool result, duplicate id, parallel results in the wrong order). - **Examples 22 and 23 and the thinking-block condenser test (`S02.anthropic-thinking`, `S02.responses-reasoning`), blocked.** Example 22 and `c01_thinking_block_condenser` need an Anthropic extended-thinking model and key; example 23 hardcodes `openhands/gpt-5-mini-2025-08-07` and needs an OpenHands provider key. The run's DeepSeek key reaches neither. ## Gotchas diff --git a/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md b/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md index 64020958edc..6cef6f64204 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md +++ b/.agents/skills/verify-sdk/references/feature-map/S09-mcp.md @@ -32,7 +32,7 @@ Examples: `examples/01_standalone_sdk/07_mcp_integration.py`, `examples/01_stand - `S09.tool-serialization`: `MCPToolDefinition` is frozen, serializes polymorphically with `kind`, is written in the MCP wire spelling (`inputSchema`) and loads either spelling, so conversations persisted under older `mcp` majors resume. Unit: `tests/sdk/mcp/test_mcp_tool_serialization.py`, `tests/sdk/mcp/test_mcp_tool_immutability.py`, `tests/sdk/mcp/test_mcp_wire_compat.py`. Example: none. Real-LLM: none. - `S09.runtime-add-servers`: `LocalConversation.add_mcp_servers({name: MCPServer})` on an initialized agent connects each enabled server before it returns and registers its tools as `_` (whatever the server count), adds the servers to `agent.mcp_config` and the persisted state, and the next LLM call offers them; before the first message it only records them and they connect with the agent like configured servers (bare names for a single server). A duplicate name and `transport="sse"` raise `ValueError`, a server that cannot start raises `MCPServerStartError` (an `MCPError` from `openhands.sdk.mcp`, with `server_name` and the client's connection `MCPError` as `__cause__`; message `Could not start MCP server : `) after closing the servers it already started, nothing refused is added, and a disabled server is recorded without connecting. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_add_mcp_servers_registers_modern_runtime_tools`, `test_add_mcp_servers_rejects_sse_transport`, `test_add_mcp_servers_accepts_running_conversation`). Example: none. Real-LLM: none. -- `S09.runtime-add-resume`: a conversation reopened from its persistence directory reconnects the servers added at runtime and the agent can call their tools by the names it was given (known failure: with one server they come back unprefixed, so `verify_echo` is `Tool ... not found`). Unit: **none (untested)**. Example: none. Real-LLM: none. +- `S09.runtime-add-resume`: a conversation reopened from its persistence directory reconnects the servers added at runtime and the agent can call their tools by the names it was given: `ConversationState.runtime_mcp_servers` records the servers added after the conversation started, which keep `_` on every (re)start, while the other servers follow the one-or-many prefix rule counted without them. Since [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929) ([#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927)). Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_runtime_added_mcp_tools_keep_their_names_after_reopen`, `test_mcp_server_added_before_a_reopened_agent_starts_keeps_old_names`, `test_reopened_single_configured_mcp_server_stays_unprefixed`). Example: none. Real-LLM: none. - `S09.startup-budget`: `LocalConversation(mcp_startup_budget=seconds)` lets agent initialization (the first `send_message`) return after the budget while slower servers keep starting; their tools join the agent when ready, the first step of `run()` waits for them (up to 30 s each), and one that fails late is logged and skipped. Without a budget (the default; the agent server passes 1.0) initialization waits for every server. Unit: `tests/sdk/conversation/test_local_conversation_mcp.py` (`test_startup_budget_starts_the_agent_before_a_slow_mcp_server`, `test_late_mcp_tools_are_closed_when_the_conversation_closed_first`, `test_first_step_waits_for_mcp_servers_that_missed_the_budget`). Example: none. Real-LLM: none. ## How to get to it (agent POV) @@ -249,7 +249,7 @@ Every server here is `programs/S09/mcp_server.py`, a small FastMCP server the pr --check early_first_call_tools contains echo --check early_first_call_tools not-contains solo_echo --check early_texts.0 contains "echo: early" ``` The add on a ready agent connects at once (tools listed right after the call, prefixed although `verify` is the only server); the duplicate and the SSE server raise `ValueError`, a server that exits at startup raises `MCPServerStartError` naming it (the agent server turns both into its 400, F19), and all three leave `mcp_config` with only `verify` and the disabled `off`; the server added before the first message is connected with the agent under bare names. -- **Reopening a conversation with added servers (`S09.runtime-add-resume`).** Known failure (reproduced 2026-11-17 and 2026-11-19, [#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927), closed 2026-11-18 with no fix merged; fix proposed in [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929), open): the reopened conversation offers the tools as `echo`, ..., so the second `verify_echo` call is an `AgentErrorEvent` `Tool 'verify_echo' not found` and `resumed_second_call_ok` is `false`. +- **Reopening a conversation with added servers (`S09.runtime-add-resume`).** `programs/S09/runtime.py resume`: `verify` is added to a running conversation (the only server) and its `verify_echo` called; the conversation is closed and reopened from its persistence directory with the same agent configuration, and the model calls `verify_echo` again. Expected: the reopened conversation offers the tools as `verify_echo`, ... and the second call returns `echo: two`. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) control-sdk program S09/runtime --arg resume \ diff --git a/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md b/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md index 2ed0f2a8a06..a6af1733b40 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md +++ b/.agents/skills/verify-sdk/references/feature-map/S13-security-confirmation.md @@ -170,6 +170,14 @@ Preconditions: --check models.https_url eq "https://img.example/?k=" --check models.text eq "" ``` `token.txt` holds the request's value, not the `AgentContext` one, so the request wins. The values reach the terminal, but every observation shows ``. The failing `BROKEN` lookup is skipped, and masking again inside the back-off window sends no new request. +- **A secret rotated mid-conversation (`S13.secret-registry`, `S13.mask-output`).** `programs/S13/secret_rotation.py`: `Conversation(secrets={"DEPLOY_TOKEN": "dep-v1-s13", "OAUTH_TOKEN": -s13>})`; a scripted agent cats a file holding both first values, then `update_secrets({"DEPLOY_TOKEN": "dep-v2-s13"})` and a second run cats a file holding the new value (never named by a command), exports both names (`echo $DEPLOY_TOKEN $OAUTH_TOKEN | wc -c`), and cats the first file again. + ```sh + export SDK_VERIFY_RUN=$(control-sdk new --field run) + control-sdk program S13/secret_rotation --check status eq finished --check observations eq 4 \ + --check before_rotation_leaks len-eq 0 --check export_wc eq 24 --check source_lookups eq 2 \ + --check new_value_leaks len-eq 0 --check previous_values_leaks len-eq 0 --check persisted_leaks len-eq 0 + ``` + Expected: every value the registry ever resolved for a name stays masked: the updated value is masked before any command names it, and after the export (which resolves `dep-v2-s13` and `oauth-v2-s13`) the earlier values are still masked, in the observations and in the persisted events. Known failure (reproduced 2026-11-20, [#7002](https://github.com/OpenHands/software-agent-sdk/issues/7002)): the first run is masked, but `new_value_leaks` is `["dep-v2-s13"]` (`update_secrets` leaves the old value cached in `_exported_values`, so `resolve_uncached_values` never resolves the new one), `previous_values_leaks` is `["dep-v1-s13", "oauth-v1-s13"]` (the export overwrites the single cached value per name), and `persisted_leaks` is `["dep-v1-s13", "dep-v2-s13", "oauth-v1-s13"]`. Fix proposed in [#7003](https://github.com/OpenHands/software-agent-sdk/pull/7003). - **Masking in `arun()` with a lookup the running loop serves (`S13.mask-async`).** The event loop that runs `arun()` also answers the `LookupSecret`, and while answering it registers `NESTED` on the conversation. In the first case the lookup first happens while an agent message is being masked; in the second, while a terminal observation is being masked. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) diff --git a/.agents/skills/verify-sdk/references/feature-map/S15-browser.md b/.agents/skills/verify-sdk/references/feature-map/S15-browser.md index d4e085ccabe..cb5c4fd28c2 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S15-browser.md +++ b/.agents/skills/verify-sdk/references/feature-map/S15-browser.md @@ -77,7 +77,7 @@ Preconditions: control-sdk program S15/close_tab --timeout 300 \ --check setup_ok eq true --check all_closed eq true --check stale_lists eq 0 ``` - Expected: once `browser_close_tab` has answered, `browser_list_tabs` lists only the remaining tab. Known failure (reproduced 2026-11-17, [#6785](https://github.com/OpenHands/software-agent-sdk/issues/6785)): the close answers when Chrome acknowledges `Target.closeTarget`, but browser-use's tab list drops the tab only when `Target.detachedFromTarget` is handled 10–30 ms later, so the immediate list still shows it (`stale_lists` 2 to 5 of 5; closing the focused tab usually hides the lag). Fix: [#6786](https://github.com/OpenHands/software-agent-sdk/pull/6786). + Expected: once `browser_close_tab` has answered, `browser_list_tabs` lists only the remaining tab (`stale_lists` 0 of 5). The close waits, up to 3 s, until browser-use's tab list no longer holds the closed target, which it drops only when `Target.detachedFromTarget` is handled a few milliseconds after Chrome acknowledges `Target.closeTarget`; past 3 s it logs `Closed tab ... is still listed` and returns. - **No Chromium (`S15.degrade-on-failure`, `S15.chromium-detection`, `S15.prompt-guidance`).** The run's `HOME` has no Playwright cache, `PLAYWRIGHT_BROWSERS_PATH` is unset, and no Chromium is on `PATH` or in the system paths (on a machine with a system Chrome the program exits 3, since it cannot hide that one). A scripted agent with the terminal and the browser tool set still runs its terminal command. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run) diff --git a/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md b/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md index a0bfe67845c..3e4802b5877 100644 --- a/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md +++ b/.agents/skills/verify-sdk/references/feature-map/S21-acp-agent.md @@ -104,7 +104,7 @@ Preconditions: --check ask_prompt_tokens_delta eq 100 \ --check ask_cost_delta eq 0.05 --check ask_warnings len-eq 0 --check ask_seconds lt 1.5 ``` - Expected: the fork's cost (0.05) is billed, no warning, and `ask_agent` returns as soon as the fork answers. Before #6646, the client bridge routed every fork-session update to a branch that kept only message chunks, so the `usage_update` was dropped and each call waited out the 2 s usage timeout. + Expected: the fork's cost (0.05) is billed, no warning, and `ask_agent` returns as soon as the fork answers: the client bridge records the fork session's `usage_update` as well as its message chunks, so the call does not wait out the 2 s usage timeout. - **Resume a persisted session (`S21.session-resume`, `S21.model-selection`).** A conversation with `persistence_dir` runs a turn and closes. The program then reopens it by id with a new `ACPAgent` (a new subprocess). A third agent carries `acp_resume_session_id` with no saved state, and a fourth an id the server has forgotten. ```sh export SDK_VERIFY_RUN=$(control-sdk new --field run)