Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -742,6 +742,7 @@ Preconditions:

## Gotchas

- The 4 MiB frame cap of `F07.frame-too-large` applies to live delivery too, not only to replay: a socket open while a 5 MiB event is appended receives the events before it and then closes with 1013 `frame_too_large`, so later events never arrive on that socket; reconnecting with `after_seq` past the big event resumes. A client that does not reconnect past it loses the live stream ([#6934](https://github.com/OpenHands/software-agent-sdk/issues/6934), open).
- `F07.reject-non-user`, `F07.error-frame`, `F07.error-non-json` and `F07.auth-frame-ignored` read the `error` frames at fixed positions after `sync` and the snapshot. While #6745 lasts, an `error` can overtake `sync` when the server is slow to subscribe, on an idle machine as well as under load (for `F07.auth-frame-ignored` as `frames.3.frame.type` `error`, for `F07.error-non-json` as an `error` frame ahead of `sync`), and the bullet then fails on its first position check. A fail of one of these four whose transcript shows that order is #6745, not a new defect; another attempt often passes. `F07.sync-first-during-run` reproduces that order deterministically.
- Neither the SDK nor the TypeScript client uses this socket yet, so a
regression here breaks no shipped consumer's tests; only these recipes and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -783,6 +783,7 @@ KID=$(control-agent-server api POST /api/conversations \

## Gotchas

- Rotating a secret with `POST /api/conversations/{id}/secrets` (`F08.secrets-override`) does not keep both values masked: after the update, output that prints the new value shows it in cleartext, and once the new value has been exported the old value is no longer masked either; both reach the persisted events. `F08.secrets-override` asserts only that the next command gets the new value. The SDK twin is `S13.secret-registry` (its rotated-secret bullet carries the same known failure) ([#7002](https://github.com/OpenHands/software-agent-sdk/issues/7002), open; fix proposed in [#7003](https://github.com/OpenHands/software-agent-sdk/pull/7003)).
- `respond_to_confirmation` with `accept: true` is exactly a run request: it
needs a run slot (429 when full), it is a no-op on a running conversation,
and with nothing pending it simply resumes the agent. `accept: false` with
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Routes: `POST /api/mcp/test`, `POST /api/mcp/oauth/start`,
- `F19.conversation-mcp-add-before-init`: on a conversation without a first message the route only records the server (no process, fast answer), with `env` values encrypted by the settings API decrypted for the child and stored encrypted in `base_state.json`; the first message connects it like any configured server.
- `F19.conversation-mcp-add-errors`: an unknown conversation is 404, a server name the agent already has and an SSE server are 400 with the reason, a server without `command`/`url`, an unknown field and a non-UUID id are 422, none of them changes the agent's `mcp_config`, and a disabled server is stored without being spawned.
- `F19.conversation-mcp-add-start-failure`: a server that cannot start (a command that exits at once) is refused with the route's 400 naming the server and the cause (`Could not start MCP server <name>: Connection closed`, since [#6928](https://github.com/OpenHands/software-agent-sdk/pull/6928)) and is not added.
- `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (known bug [#6927](https://github.com/OpenHands/software-agent-sdk/issues/6927), closed 2026-11-18 with no fix merged; fix proposed in [#6929](https://github.com/OpenHands/software-agent-sdk/pull/6929): with one server the tools come back unprefixed, so `qa-f19-rs_qa_echo` is `Tool ... not found`).
- `F19.conversation-mcp-add-restart`: a server added at runtime is still in the conversation's agent after a restart, and the tools the agent already called keep working under the same names (`qa-f19-rs_qa_echo`, prefixed even as the agent's only server).
- `F19.conversation-mcp-slow-start`: a conversation whose stdio server delays its handshake by 8 s is created within a few seconds, the server is spawned, and the agent's first model request waits for it and offers its tools; the agent's call returns its result.
- `F19.conversation-mcp-late-failure`: a server that fails after the startup budget costs only its own tools: the conversation starts at once, the first step waits for the failure, the other server's tools are offered, the run finishes and the log names the failed server.

Expand Down Expand Up @@ -1263,7 +1263,7 @@ Preconditions:
```
The 400's `detail` names the server and the cause, and the server is not
added (`agent.mcp_config` keeps no `qa-f19-dies`).
- **Added servers after a restart (`F19.conversation-mcp-add-restart`), known bug #6927.**
- **Added servers after a restart (`F19.conversation-mcp-add-restart`).**
The stub calls the added tool, finishes, and after a restart and a new
message calls the same tool again.
```sh
Expand All @@ -1279,12 +1279,13 @@ Preconditions:
control-agent-server restart
control-agent-server api GET "/api/conversations/$RCID" --check agent.mcp_config.qa-f19-rs.command eq "$PY" --save F19.conversation-mcp-add-restart/after-restart
control-agent-server conversation send "$RCID" --text 'Again.' --wait --until finished --timeout 90
control-agent-server conversation events "$RCID" --kinds ObservationEvent --contains QA_MCP_ECHO:two --expect-kind ObservationEvent --save F19.conversation-mcp-add-restart/second-call # bug: matching eq 0 && kinds.AgentErrorEvent ge 1
control-agent-server conversation events "$RCID" --kinds ObservationEvent --contains QA_MCP_ECHO:two --expect-kind ObservationEvent --save F19.conversation-mcp-add-restart/second-call
```
The server survives the restart in the conversation's agent and is
reconnected on the next message, but as the agent's only server its tools
come back bare (`qa_echo`), so the second call by the name the agent was
given ends in an `AgentErrorEvent` `Tool 'qa-f19-rs_qa_echo' not found`.
reconnected on the next message under the name the agent was given: the
conversation state records it as added at runtime (`runtime_mcp_servers`),
so its tools stay `qa-f19-rs_qa_echo` although it is the agent's only
server, and the second call returns `QA_MCP_ECHO:two`.
- **A slow server does not hold up the start (`F19.conversation-mcp-slow-start`).**
The F19 tools fixture with a handshake delayed by 8 s, inline on the
conversation's agent; the stub reads one variable through it.
Expand Down Expand Up @@ -1357,10 +1358,13 @@ Preconditions:
request (`sink read --name <stub> --contains '"name": "<tool>"'`), not the
first event. Tools added through `POST /api/conversations/{id}/mcp` are not
in it either.
- Tools of a server added through the conversation route are always named
`<server>_<tool>` while the agent is running, but a restart (or an add
before the first message) names them like configured servers: bare when it
is the agent's only server (`F19.conversation-mcp-add-restart`).
- Tools of a server added through the conversation route once the
conversation has started (its agent is initialized or it has events) are
named `<server>_<tool>`, and keep that name across a restart
(`F19.conversation-mcp-add-restart`); the other servers are counted without
it for the two-or-more prefix rule. An add to a brand-new conversation
before it has started is named like a configured server: bare when it is
the agent's only server.
- The conversation MCP route runs the connection inside the request: an
initialized conversation answers only after the server has started (or
failed, up to 30 s). Explicit `transport: sse` is refused, but a `/sse` URL
Expand Down
100 changes: 100 additions & 0 deletions .agents/skills/verify-sdk/programs/S13/secret_rotation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
"""Usage: secret_rotation.py

A secret rotated mid-conversation with `update_secrets()`, and a program's own
`SecretSource` that returns a new value on each lookup.

`DEPLOY_TOKEN` starts as `dep-v1-s13` and `OAUTH_TOKEN` (a rotating source)
answers `oauth-v<n>-s13`. A scripted agent cats files that hold the values
without naming the secrets (as a token in a git remote or a log would be):

run 1 cat v1.txt (dep-v1-s13 oauth-v1-s13)
update_secrets({"DEPLOY_TOKEN": "dep-v2-s13"})
run 2 cat v2.txt (dep-v2-s13, never named yet)
echo $DEPLOY_TOKEN $OAUTH_TOKEN | wc -c (exports both: v2 and oauth-v2)
cat v1.txt (the previous values again)

Each `*_leaks` field lists the cleartext values found in that observation;
`persisted_leaks` lists every value found in the persisted events.
"""

from pathlib import Path

from sdk_verify import emit, make_llm, run_paths

from openhands.sdk import Agent, Conversation, Tool
from openhands.sdk.event import ObservationEvent
from openhands.sdk.secret import SecretSource
from openhands.tools.terminal import TerminalTool


paths = run_paths()
ws = paths["workspace"]
VALUES = ["dep-v1-s13", "dep-v2-s13", "oauth-v1-s13", "oauth-v2-s13"]


class Rotating(SecretSource):
calls: int = 0

def get_value(self) -> str | None:
self.calls += 1
return f"oauth-v{self.calls}-s13"


(ws / "v1.txt").write_text("remote has dep-v1-s13 log has oauth-v1-s13\n")
(ws / "v2.txt").write_text("config has dep-v2-s13\n")

script = [
{"tool": "terminal", "args": {"command": "cat v1.txt"}},
{"finish": "one"},
{"tool": "terminal", "args": {"command": "cat v2.txt"}},
{
"tool": "terminal",
"args": {"command": "echo $DEPLOY_TOKEN $OAUTH_TOKEN | wc -c"},
},
{"tool": "terminal", "args": {"command": "cat v1.txt"}},
{"finish": "two"},
]
rotating = Rotating()
c = Conversation(
agent=Agent(llm=make_llm(script), tools=[Tool(name=TerminalTool.name)]),
workspace=str(ws),
persistence_dir=str(paths["conversations"]),
visualizer=None,
secrets={"DEPLOY_TOKEN": "dep-v1-s13", "OAUTH_TOKEN": rotating},
)
c.send_message("first")
c.run()
c.update_secrets({"DEPLOY_TOKEN": "dep-v2-s13"})
c.send_message("second")
c.run()

texts = [
"".join(getattr(x, "text", "") for x in e.observation.to_llm_content)
for e in c.state.events
if isinstance(e, ObservationEvent) and e.tool_name == "terminal"
]


def leaks(text: str) -> list[str]:
return [v for v in VALUES if v in text]


persisted = ""
for f in sorted(Path(paths["conversations"]).rglob("events/*.json")):
persisted += f.read_text()

emit(
{
"status": c.state.execution_status.value,
"observations": len(texts),
"before_rotation": texts[0] if texts else None,
"before_rotation_leaks": leaks(texts[0]) if texts else None,
"new_value": texts[1] if len(texts) > 1 else None,
"new_value_leaks": leaks(texts[1]) if len(texts) > 1 else None,
"export_wc": texts[2].split("\n")[0] if len(texts) > 2 else None,
"previous_values": texts[3] if len(texts) > 3 else None,
"previous_values_leaks": leaks(texts[3]) if len(texts) > 3 else None,
"source_lookups": rotating.calls,
"persisted_leaks": leaks(persisted),
}
)
6 changes: 3 additions & 3 deletions .agents/skills/verify-sdk/references/feature-map/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ agent-server map
and follows its layout. Family IDs here are `Snn`, so they never collide with
that map's `Fnn`.

Maintenance baseline: main@141423921fdaeb08cd5f53464a69e860da70b66f (2026-11-19). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it.
Maintenance baseline: main@722e5d403cf75d4e032b79296cfaf178f8678a93 (2026-11-20). The next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it.

The map has **504 sub-features**, **36 of them untested** (no unit or cross
The map has **504 sub-features**, **35 of them untested** (no unit or cross
test exercises the behavior; most of those have an example script).

## What each sub-feature links
Expand Down Expand Up @@ -83,7 +83,7 @@ an interactive login are labeled **blocked**.
| S06 | [Agent context, system prompt, condensers and memory](S06-context-condensers.md) | `AgentContext`, system prompt sections and vendor files, persona, persistent memory, the condenser `View`, `LLMSummarizingCondenser`, `condense()`, overflow condensation, no-op and pipeline condensers | `Agent(agent_context=, system_prompt=, condenser=)`, `AgentContext(load_memory=)`, `conversation.condense()` | llm, tmux | 24 | 1 |
| S07 | [Skills](S07-skills.md) | skill sources (programmatic, Markdown, AGENTS.md, SKILL.md, user, project, public, installed), keyword, task and path triggers, `<available_skills>`, `invoke_skill`, inline commands, disabling, install and marketplace, the `OH_PUBLIC_SKILLS_ENABLED` switch | `AgentContext(skills=, load_*_skills=, disabled_skills=)`, `.agents/skills/`, `install_skill`, `OH_PUBLIC_SKILLS_ENABLED` | llm, git, network | 25 | 0 |
| S08 | [Plugins, marketplaces and the extension store](S08-plugins-marketplace.md) | both plugin layouts, path containment, commands as skills, merging into the agent, `resolved_plugins`, fetch cache, installed and ambient plugins, marketplace registry, `load_plugin()` | `Conversation(plugins=[PluginSource(...)])`, `Plugin.load`, `conversation.load_plugin()` | llm, git, network, uvx | 20 | 0 |
| S09 | [MCP client and tools](S09-mcp.md) | stdio, HTTP and SSE servers, auth and OAuth, enabled flag, timeouts, failing servers, schema conversion, observations, secrets in calls, reconnection, `list_changed`, tool filtering, servers added to a running conversation, the startup budget | `Agent(mcp_config=, filter_tools_regex=)`, `create_mcp_tools`, `MCPOAuthAuthentication`, `LocalConversation.add_mcp_servers`, `LocalConversation(mcp_startup_budget=)` | llm, uvx, node, network | 23 | 1 |
| S09 | [MCP client and tools](S09-mcp.md) | stdio, HTTP and SSE servers, auth and OAuth, enabled flag, timeouts, failing servers, schema conversion, observations, secrets in calls, reconnection, `list_changed`, tool filtering, servers added to a running conversation, the startup budget | `Agent(mcp_config=, filter_tools_regex=)`, `create_mcp_tools`, `MCPOAuthAuthentication`, `LocalConversation.add_mcp_servers`, `LocalConversation(mcp_startup_budget=)` | llm, uvx, node, network | 23 | 0 |
| S10 | [Hooks](S10-hooks.md) | config format and discovery, matchers, command hooks, blocking tool calls and prompts, context injection, Stop, SessionStart and SessionEnd, `HookExecutionEvent`, prompt and agent hooks, plugin hooks | `Conversation(hook_config=HookConfig(...))`, `HookDefinition(type=)`, `HookManager` | llm, git | 21 | 1 |
| S11 | [Sub-agents and delegation](S11-subagents-delegation.md) | the `task` tool lifecycle, agent registry, Markdown agent files and `AgentDefinition`, built-in agents, `SubAgentScope`, the legacy delegate executor | `Tool(name=TaskToolSet.name)`, `register_agent`, `.agents/agents/*.md` | llm | 24 | 2 |
| S12 | [Critic and critic-driven refinement](S12-critic.md) | `CriticResult`, rule critics, evaluation modes, iterative refinement, `APIBasedCritic`, critic from settings | `Agent(critic=APIBasedCritic(...))`, `event.critic_result` | llm, network | 15 | 4 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,7 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)`
--check finish_schema_fields contains facts --check finish_schema_fields contains message \
--check rejected.reserved_name eq ValueError --check rejected.field_collision eq ValueError
```
Expected: both rejected terminal schemas raise `ValueError`, and the executors the registry already built for them are closed, so no tmux server from the program is left running after it exits (`ps -eo args | grep "[t]mux.*<run workspace>"` is empty).
- **Context overflow (`S01.context-overflow`).** A context-window error with an `LLMSummarizingCondenser` (emits a `CondensationRequest` and the run continues) and without one (the run ends in ERROR).
```sh
export SDK_VERIFY_RUN=$(control-sdk new --field run)
Expand Down Expand Up @@ -278,7 +279,6 @@ Scripted bullets use the SDK's `TestLLM` (`--script`, or `make_llm(script=...)`

## Gotchas

- A rejected `response_schema` on a terminal tool raises `ValueError` after the terminal has already started its tmux server, and nothing closes it: `close()` leaves the server running after the program exits ([#6748](https://github.com/OpenHands/software-agent-sdk/issues/6748)). `S01.structured-output` passes but leaves two such servers per run; end them with `TMUX_TMPDIR=<run tmux dir> tmux -L <socket> kill-server`.
- `01_hello_world.py` is in `_EXCLUDED_EXAMPLES` in `tests/examples/test_examples.py`, so the `test-examples` label never runs it; the other seven examples here do run there (`pytest tests/examples/test_examples.py --run-examples`).
- All examples read `LLM_API_KEY`, `LLM_MODEL`, `LLM_BASE_URL`; `56_structured_output.py` defaults `LLM_MODEL` to `anthropic/claude-sonnet-4-5-20250929`, the rest to `gpt-5.5`. `17_image_input.py` asserts `llm.vision_is_active()` and fails fast on a text-only model; it also fetches an image from python.org.
- Examples run with the repo root as workspace and write files there (`FACTS.txt`, `image_report.md`, the document from example 18).
Expand Down
Loading
Loading