diff --git a/.agents/skills/verify-agent-server/references/feature-map/F01-server-status.md b/.agents/skills/verify-agent-server/references/feature-map/F01-server-status.md index e267ac64e64..1c409f88a0b 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F01-server-status.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F01-server-status.md @@ -294,5 +294,5 @@ Preconditions: leaks file paths and source lines in 5xx bodies, so it is never for production. Unhandled exceptions get the same `traceback` field next to their `error_id`; the map's unhandled 500s either need a conversation or - are known bugs, so this family drives the deliberate `HTTPException` 500, + are known bugs, so this family drives a deliberate `OperationFailed` 500, the cheapest stable 5xx. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F03-deferred-init.md b/.agents/skills/verify-agent-server/references/feature-map/F03-deferred-init.md index e9560e416f4..5da35c0f7f8 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F03-deferred-init.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F03-deferred-init.md @@ -48,6 +48,7 @@ Routes: `GET /api/init`, `POST /api/init` - `F03.init-delivered-webhooks`: `webhooks` delivered at init (on a pod that booted with none) receive the new conversation's `ConversationInfo` at `{base_url}/conversations`, with the delivered headers and the delivered session key. - `F03.init-enforces-session-keys`: after init, `/api/*` rejects a missing or wrong key with 401 and accepts the delivered session key with 200, on a pod that booted without keys. - `F03.init-applies-env`: `env` entries delivered at init reach `os.environ` and therefore the bash subprocesses, and the delivered `bash_events_dir` receives the events. +- `F03.init-failure-restores-env`: `env` entries sent only in a failed init are taken back with it: after the successful retry bash subprocesses do not see them, while the retry's own entries are there. - `F03.init-keys-gate-bash-socket`: after init, `/sockets/bash-events` closes with 4001 for a client without the delivered key and runs commands for one with it. - `F03.init-opens-events-socket-and-v1`: after init, the conversation events socket that was refused while dormant opens and replays the conversation's state with the delivered key (4001 without it), and `GET /v1/models` answers 401 without a key and 200 with the delivered key as a Bearer token. - `F03.init-web-url-root-path`: a delivered `web_url` with a path becomes the root path: prefixed URLs serve and the OpenAPI document advertises the prefix. @@ -248,7 +249,7 @@ Preconditions: - **Failure rollback (`F03.init-failure-rollback`).** A failing init returns 500 and returns to dormant with the error recorded. ```sh control-agent-server api POST /api/init --auth none --header "X-Init-API-Key: $SK" \ - --json '{"conversations_path":"/proc/oh-cannot-create-f03"}' --expect 500 \ + --json '{"conversations_path":"/proc/oh-cannot-create-f03","env":{"QA_F03_FAILED_ONLY":"qa-failed-attempt"}}' --expect 500 \ --check exception contains oh-cannot-create-f03 --save F03.init-failure-rollback/failure control-agent-server api GET /api/init --auth none --check state eq dormant --check error contains oh-cannot-create-f03 control-agent-server api GET /api/settings --auth none --expect 503 @@ -257,7 +258,9 @@ Preconditions: rolls the state back to `dormant`, keeps the gate closed, and stores `: ` under `error` (on Linux `FileNotFoundError: [Errno 2] ... '/proc/oh-cannot-create-f03'`), so an - orchestrator can retry. The successful retry is the next bullet. + orchestrator can retry. The body also carries an `env` entry that no later + init sends (`F03.init-failure-restores-env` reads it back). The successful + retry is the next bullet. - **Successful init (`F03.init-success`).** A valid body flips the pod to `ready`; conversations land in the delivered directory. The body delivers every field the later bullets read back. ```sh BODY="$(python3 -c "import json,os,sys;r=os.environ['AGENT_SERVER_VERIFY_RUN']+'/fixtures/';print(json.dumps({'session_api_keys':['qa-user-key'],'secret_key':'qa-f03-delivered-secret','conversations_path':r+'conv','bash_events_dir':r+'bash','conversation_worktree_root':r+'qa-f03-worktrees','webhooks':[{'base_url':sys.argv[1],'headers':{'X-QA-F03':'delivered-hook'},'flush_delay':1}],'web_url':'http://qa-f03.invalid/rt/qa-f03','allow_cors_origins':['https://qa-f03.example'],'max_concurrent_runs':1,'env':{'PROBE_VAR':'qa-hello'}}))" "$F03_HOOKS")" @@ -306,6 +309,18 @@ Preconditions: ``` The command prints `PROBE=qa-hello` (the init-delivered env), and the delivered bash-events directory now holds the command and output events. +- **A failed init takes its env back (`F03.init-failure-restores-env`).** The failed attempt carried `QA_F03_FAILED_ONLY`; the successful one carried only `PROBE_VAR`. + + Requires: `F03.init-failure-rollback`, `F03.init-success` + ```sh + control-agent-server api POST /api/bash/execute_bash_command --auth none --header 'X-Session-API-Key: qa-user-key' \ + --json '{"command":"echo \"PROBE=$PROBE_VAR FAILED_ONLY=[${QA_F03_FAILED_ONLY-unset}]\""}' --expect 200 \ + --check stdout contains 'PROBE=qa-hello' --check stdout contains 'FAILED_ONLY=[unset]' --save F03.init-failure-restores-env/env + ``` + One command prints both: `PROBE=qa-hello`, the entry of the successful + init (the positive control), and `FAILED_ONLY=[unset]`, because the failed + attempt's entry was removed from the server's environment when that attempt + was rolled back. - **Delivered keys gate the bash socket (`F03.init-keys-gate-bash-socket`).** After init the socket authenticates with the delivered keys. ```sh SOCK_PROBE="$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f03-ws-after-init.txt" @@ -578,11 +593,10 @@ Preconditions: the init body sets `secret_key`, or if the dormant config had none and session keys are delivered, the bootstrap key changes and a retry with the old key is 401 instead of 400. -- A failed init rolls back only the state: `env` entries it applied stay in - `os.environ` (observed: a variable sent only in a failed attempt is visible - to bash after the successful retry), and the bash service it entered before - failing is not exited. `InitRequest.env` documents the `os.environ.update`, - not a rollback; resend the full env on retry. +- A failed init is all-or-nothing: the state returns to `dormant`, the + services it started are stopped, and the `env` entries it applied are put + back to their previous values (`F03.init-failure-restores-env`). Nothing + of a failed attempt carries over, so resend the full env on retry. - `OH_DEFERRED_INIT` must be `1`/`true` (case-insensitive); `yes` parses as False. - An init-delivered `allow_cors_origins` is a no-op diff --git a/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md b/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md index 66e63c49c3f..e11580c181b 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F06-conversation-events.md @@ -803,7 +803,10 @@ Preconditions: recipes that stop at `finished` re-read the status over REST. - `POST .../events` with `run: true` on a `running` conversation is a 200, not the 409 of `POST /run`: the message joins the running loop. A client - that wants to know whether a new run started must watch the socket. + that wants to know whether a new run started must watch the socket. The + exception is a conversation whose current turn was started by the OpenAI + gateway: there the POST is 409 `Gateway run in progress; message not + saved` and nothing is stored (`F30.direct-event-during-turn-409`). - The body is required (no body is a 422), but every field has a default: `{}` is a valid request and appends a user `MessageEvent` with empty `content`. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md index 240b77aa4b1..2ee43e1bcc5 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F07-session-socket.md @@ -701,7 +701,7 @@ Preconditions: ## Gotchas -- `F07.reject-non-user`, `F07.error-frame` and `F07.auth-frame-ignored` read the `error` frames at fixed indexes after `sync` and the snapshot. While #6745 lasts, an `error` can overtake `sync` when the server is slow to subscribe (seen under load, for `F07.auth-frame-ignored` as `frames.3.frame.type` `error`), and the bullet then fails on its first index check; rerun it alone with `--only`. `F07.sync-first-during-run` reproduces that order deterministically. +- `F07.reject-non-user`, `F07.error-frame`, `F07.error-non-json` and `F07.auth-frame-ignored` read the `error` frames at fixed positions after `sync` and the snapshot. While #6745 lasts, an `error` can overtake `sync` when the server is slow to subscribe, on an idle machine as well as under load (for `F07.auth-frame-ignored` as `frames.3.frame.type` `error`, for `F07.error-non-json` as an `error` frame ahead of `sync`), and the bullet then fails on its first position check. A fail of one of these four whose transcript shows that order is #6745, not a new defect; another attempt often passes. `F07.sync-first-during-run` reproduces that order deterministically. - Neither the SDK nor the TypeScript client uses this socket yet, so a regression here breaks no shipped consumer's tests; only these recipes and `tests/agent_server/test_session_socket.py` notice. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md b/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md index a31c1b37c95..7c58e0ce6de 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md @@ -40,6 +40,9 @@ Routes: `POST /api/conversations/{conversation_id}/events/respond_to_confirmatio - `F08.secrets-errors`: a missing or non-object `secrets`, an unknown secret kind and a `LookupSecret` without `url` are 422; an unknown conversation is 404; no key is 401; nothing is stored. - `F08.secrets-bad-value`: a number, `null` or a boolean as a secret value is refused with 422, not a 500. - `F08.secrets-terminal`: a terminal command that names a runtime secret receives its value, while the observation and the next request to the model show `` and the plaintext is stored nowhere. +- `F08.secrets-task-delegate`: a runtime secret of the conversation reaches the terminal of a sub-agent its task tool delegates to; the delegate's observation and its next model request show ``, and the plaintext is stored nowhere. +- `F08.secrets-agent-message`: a registered secret value in the model's text reply is stored and returned as `` in the agent's `MessageEvent`, and the plaintext is stored nowhere. +- `F08.secrets-git-provider`: a `GITHUB_TOKEN` runtime secret is what the agent's `git` uses for `https://github.com`, without the command naming the secret. - `F08.secrets-override`: a runtime secret replaces a start-time secret of the same name for the next command. - `F08.secrets-lookup`: a runtime `LookupSecret` is fetched from its URL with its headers only when a command names it; the command gets the fetched text, the observation and the model see ``, and GET hides the `Authorization` header that `base_state.json` keeps encrypted. - `F08.secrets-lookup-self`: a runtime `LookupSecret` whose URL is the server's own `/api/settings/secrets/` (hostless, made absolute from `OH_INTERNAL_SERVER_URL`, or absolute on the run's address) is answered in process from the secrets store with no HTTP request and no session key; a name the store lacks falls through to HTTP (401 without a key) and the command gets no value. @@ -112,7 +115,14 @@ Preconditions: `$ACCEPT_AGENT` (write `qa-f08-accepted.txt`, finish), `$REJECT_AGENT` (`rm -rf qa-f08-victim`, finish), `$RISKY_AGENT` (a low-risk `echo`, `rm -rf qa-f08-victim`, an `echo`, finish), `$SECRET_AGENT` (write - `$QA_F08_TOKEN` to a file and echo it, finish, twice), `$ROTATE_AGENT` + `$QA_F08_TOKEN` to a file and echo it, finish, twice), `$GITPROV_AGENT` + (ask `git credential fill` for `https://github.com` credentials without + naming a secret, write `$GITHUB_TOKEN` to a file, finish), `$DELEGATE_AGENT` (the task tool + instead of the terminal: delegate to the sub-agent `qa-f08-delegate`, + whose model calls are the next steps of the same stub, write + `$QA_F08_DELEGATED` to a file and echo it, finish the delegate, finish), + `$MESSAGE_AGENT` (a text reply that contains a secret's value), + `$ROTATE_AGENT` (write `$QA_F08_ROT` to a file, finish), `$LOOKUP_AGENT` (write `$QA_F08_LOOKUP` to a file and echo it, finish) and `$SELF_AGENT` (write `$QA_F08_HOSTLESS`, `$QA_F08_ABSOLUTE` and `$QA_F08_ABSENT` to three files @@ -129,7 +139,8 @@ Preconditions: control-agent-server llm preset deepseek W="$AGENT_SERVER_VERIFY_RUN/fixtures/qa-f08" mkdir -p "$W/knobs" "$W/accept" "$W/reject/qa-f08-victim" "$W/risky/qa-f08-victim" "$W/secret" "$W/rotate" \ - "$W/lookup" "$W/self" "$W/scoped" "$W/nocipher" "$W/real" + "$W/lookup" "$W/self" "$W/scoped" "$W/nocipher" "$W/real" "$W/gitprov" \ + "$W/delegate" "$W/message" NOPE=$(python3 -c 'import uuid; print(uuid.uuid4())') OFFLINE_AGENT='{"llm": {"model": "openai/qa-offline", "api_key": "qa-offline-key", "base_url": "http://127.0.0.1:9/v1", "num_retries": 0}, "tools": []}' MSG='{"role": "user", "content": [{"type": "text", "text": "Run the command."}]}' @@ -144,6 +155,14 @@ RISKY_AGENT=$(stub_agent qa-f08-risky --step 'tool:terminal:{"command":"echo qa- --step 'tool:terminal:{"command":"rm -rf qa-f08-victim"}' --step 'tool:terminal:{"command":"echo qa-unrated > qa-f08-unrated.txt"}' --step "$FINISH") SECRET_STEP='tool:terminal:{"command":"printf %s \"$QA_F08_TOKEN\" > qa-f08-token.txt; echo \"token=$QA_F08_TOKEN\""}' SECRET_AGENT=$(stub_agent qa-f08-secret --step "$SECRET_STEP" --step "$FINISH" --step "$SECRET_STEP" --step "$FINISH") +GITPROV_AGENT=$(stub_agent qa-f08-gitprov \ + --step 'tool:terminal:{"command":"printf \"protocol=https\\nhost=github.com\\n\\n\" | GIT_TERMINAL_PROMPT=0 git credential fill > qa-f08-cred.txt 2>&1; echo cred-done"}' \ + --step 'tool:terminal:{"command":"printf %s \"$GITHUB_TOKEN\" > qa-f08-named.txt"}' --step "$FINISH") +DELEGATE_AGENT=$(stub_agent qa-f08-delegate \ + --step 'tool:task:{"subagent_type":"qa-f08-delegate","prompt":"Run the command.","description":"qa delegate"}' \ + --step 'tool:terminal:{"command":"printf %s \"$QA_F08_DELEGATED\" > qa-f08-delegated.txt; echo \"delegated=$QA_F08_DELEGATED\""}' \ + --step 'tool:finish:{"message":"QA_F08_DELEGATE_DONE"}' --step "$FINISH" | jq -c '.tools = [{"name": "task_tool_set"}]') +MESSAGE_AGENT=$(stub_agent qa-f08-message --step 'reply:QA_F08_SAYS qa-f08-message-value END') ROTATE_STEP='tool:terminal:{"command":"printf %s \"$QA_F08_ROT\" > qa-f08-rot.txt"}' ROTATE_AGENT=$(stub_agent qa-f08-rotate --step "$ROTATE_STEP" --step "$FINISH") LOOKUP_STEP='tool:terminal:{"command":"printf %s \"$QA_F08_LOOKUP\" > qa-f08-lookup.txt; echo \"lookup=$QA_F08_LOOKUP\""}' @@ -501,6 +520,78 @@ KID=$(control-agent-server api POST /api/conversations \ recorded model requests carry the masked text and never the value, and no server file holds the plaintext although the name is stored (the positive control for the glob). The run saved the state, so GET lists the name. +- **Secrets in a delegated task (`F08.secrets-task-delegate`).** The + conversation's agent has only the task tool; its create request defines + the sub-agent `qa-f08-delegate` with the terminal. The stub delegates, + and the delegate's command writes `$QA_F08_DELEGATED` to a file and echoes + it. + ```sh + DID=$(control-agent-server api POST /api/conversations \ + --json "{\"workspace\": {\"working_dir\": \"$W/delegate\"}, \"agent\": $DELEGATE_AGENT, \"autotitle\": false, \"agent_definitions\": [{\"name\": \"qa-f08-delegate\", \"description\": \"QA delegate.\", \"tools\": [\"terminal\"], \"system_prompt\": \"You are the QA delegate.\"}]}" \ + --expect 201 --field id) + control-agent-server api POST "/api/conversations/$DID/secrets" --json '{"secrets": {"QA_F08_DELEGATED": "qa-f08-delegated-value"}}' \ + --expect 200 --save F08.secrets-task-delegate/post + control-agent-server conversation send "$DID" --text 'Run the command.' --wait --until finished --timeout 120 + control-agent-server conversation events "$DID" --kinds ObservationEvent --contains '"subagent": "qa-f08-delegate"' \ + --expect-kind ObservationEvent --save F08.secrets-task-delegate/task + test "$(cat "$W/delegate/qa-f08-delegated.txt")" = qa-f08-delegated-value + control-agent-server state grep 'delegated=' --glob "server/workspace/conversations/${DID//-/}/subagents/*/events/*" + control-agent-server sink read --name qa-f08-delegate --contains 'delegated=' --expect-min 1 \ + --save F08.secrets-task-delegate/model-request + control-agent-server sink read --name qa-f08-delegate --contains qa-f08-delegated-value --expect-max 0 + control-agent-server state grep qa-f08-delegated-value --glob 'server/**/*' --expect-none + ``` + The task observation names `qa-f08-delegate`, and the file the delegate's + command wrote holds the real value: the secret posted to the conversation + was an environment variable in the sub-agent's terminal. The delegate's + own event log (under the conversation's `subagents/` folder) and the + model request that followed its command read `delegated=`; + no model request and no server file holds the plaintext. +- **Secret in the agent's reply (`F08.secrets-agent-message`).** The stub + answers with a text that contains the value of a secret posted to the + conversation. + ```sh + MID=$(control-agent-server api POST /api/conversations \ + --json "{\"workspace\": {\"working_dir\": \"$W/message\"}, \"agent\": $MESSAGE_AGENT, \"autotitle\": false}" --expect 201 --field id) + control-agent-server api POST "/api/conversations/$MID/secrets" --json '{"secrets": {"QA_F08_MSG": "qa-f08-message-value"}}' --expect 200 + control-agent-server conversation send "$MID" --text 'Say it.' --wait --until finished --timeout 120 + control-agent-server sink read --name qa-f08-message --expect-min 1 --expect-max 1 + control-agent-server api GET "/api/conversations/$MID/events/search" --query source=agent --query kind=MessageEvent \ + --check items len-eq 1 --check items.0.llm_message.content.0.text eq 'QA_F08_SAYS END' \ + --save F08.secrets-agent-message/message + control-agent-server api GET "/api/conversations/$MID/agent_final_response" --check response eq 'QA_F08_SAYS END' + control-agent-server state grep QA_F08_SAYS --glob "server/workspace/conversations/${MID//-/}/events/*" + control-agent-server state grep qa-f08-message-value --glob 'server/**/*' --expect-none + ``` + The model was called once and answered `QA_F08_SAYS qa-f08-message-value + END`; the agent's `MessageEvent` and the final response read `QA_F08_SAYS + END`, and the stored event carries the reply's marker (the + positive control for the search) while no server file holds the value. +- **Git uses a provider token secret (`F08.secrets-git-provider`), known bug #6814.** + A `GITHUB_TOKEN` runtime secret (a fake value), then a command that asks + git which credentials it would send to `https://github.com` + (`git credential fill`, no network) and never names the secret. The + stub's second command names it, as the positive control. + + Needs: `git` + ```sh + GID=$(control-agent-server api POST /api/conversations \ + --json "{\"workspace\": {\"working_dir\": \"$W/gitprov\"}, \"agent\": $GITPROV_AGENT, \"autotitle\": false}" --expect 201 --field id) + control-agent-server api POST "/api/conversations/$GID/secrets" --json '{"secrets": {"GITHUB_TOKEN": "qa-6814-token"}}' --expect 200 + control-agent-server conversation send "$GID" --text 'Run the command.' --wait --until finished --timeout 120 + control-agent-server conversation events "$GID" --kinds ObservationEvent --contains cred-done + test "$(cat "$W/gitprov/qa-f08-named.txt")" = qa-6814-token + control-agent-server state cat fixtures/qa-f08/gitprov/qa-f08-cred.txt --contains 'password=qa-6814-token' # bug: content contains 'could not read Username' + ``` + Expected: git's answer holds `password=qa-6814-token`, so an HTTPS + `git push`, `fetch` or `clone` against `github.com` authenticates with the + saved token although the command does not name it + ([#6814](https://github.com/OpenHands/software-agent-sdk/issues/6814)). + The file reads `fatal: could not read Username for + 'https://github.com': terminal prompts disabled`: the terminal exports a + secret only into a command whose text names it (the second command, which + does, wrote the token to `qa-f08-named.txt`), and nothing tells git to use + the token. - **Override a start-time secret (`F08.secrets-override`).** A conversation created with `QA_F08_ROT` = `qa-f08-rot-old`, then a runtime POST of a new value. @@ -731,7 +822,10 @@ KID=$(control-agent-server api POST /api/conversations \ without a key. - A command receives a secret only when its text mentions the secret's name (case-insensitive substring match); masking applies to every registered - secret's last resolved value, wherever it appears in terminal output. + secret's last resolved value, wherever it appears in terminal output or + in the text of the model's reply (`F08.secrets-agent-message`). A + sub-agent started by the task tool gets the conversation's secrets under + the same rules (`F08.secrets-task-delegate`). - Runtime secret changes made through `POST .../secrets` live only in `base_state.json` and are never written to `meta.json`, so a runtime secret survives a reload (restart, idle eviction, sandbox pause) only once a state diff --git a/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md b/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md index 4014dc4e62c..d040b8f8c7d 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md @@ -41,10 +41,11 @@ Routes: `POST /api/conversations/{conversation_id}/ask_agent`, - `F10.condense-status`: that refusal is a client error (4xx), not a 500. - `F10.condense-llm-failure`: when the condenser's model is unreachable, condense answers 500 with an `error_id`, leaves the `CondensationRequest` pending and records no `Condensation`. - `F10.condense-errors`: an unknown id is 404 `Conversation not found`, a malformed id 422, a missing key 401 (and records nothing). -- `F10.ask-agent`: `POST /ask_agent` answers a question about the conversation with DeepSeek (200 `{"response": ...}`), records no message and pushes no message on the socket, and leaves the status unchanged. +- `F10.ask-agent`: `POST /ask_agent` answers a question about the conversation with DeepSeek (200 `{"response": ...}`), records no message or action and pushes none on the socket, and leaves the status unchanged; the only event it adds is one `stats` `ConversationStateUpdateEvent` that reports the call's usage under `ask-agent-llm`, in the event log and on the socket. - `F10.ask-agent-errors`: a missing `question` is 422, a missing key 401, and an unreachable model 500 with an `error_id`. -- `F10.ask-agent-running`: while a run waits on its model, `ask_agent` answers at once (it does not wait for the step), records nothing and leaves the status `running`. +- `F10.ask-agent-running`: while a run waits on its model, `ask_agent` answers at once (it does not wait for the step), records nothing but its `stats` update (usage under `ask-agent-llm`) and leaves the status `running`. - `F10.ask-agent-unknown`: an unknown conversation id is 404, as the route documents. +- `F10.ask-agent-usage-saved`: the usage of an ask is part of the conversation's `stats` on `GET /api/conversations/{id}` right after the call (an `ask-agent-llm` entry, as in the `stats` event the ask recorded) and is still there after a restart. - `F10.restart-persist`: after a restart, forks keep their lineage, title, tags, HEAD and events, a fork id that names a persisted but not yet loaded conversation is still 409, and a navigated empty HEAD stays empty (the next message is still a new root). - `F10.restart-cold-calls`: after a restart, condense and ask_agent work on a conversation whose agent has not been initialized yet, and ask_agent still sees the history (the codename). @@ -110,6 +111,8 @@ NOPE=$(python3 -c 'import uuid; print(uuid.uuid4())') MSG=openhands.sdk.event.llm_convertible.message.MessageEvent CREQ=openhands.sdk.event.condenser.CondensationRequest COND=openhands.sdk.event.condenser.Condensation +ACT=openhands.sdk.event.llm_convertible.action.ActionEvent +STATE=openhands.sdk.event.conversation_state.ConversationStateUpdateEvent OFFLINE_LLM='{"model": "openai/qa-offline", "api_key": "qa-offline-key", "base_url": "http://127.0.0.1:9/v1", "num_retries": 0}' S=$(control-agent-server conversation start --placeholder-agent --no-autotitle --no-run --prompt QA_F10_M1 --print-id) control-agent-server api POST "/api/conversations/$S/events" --json '{"role": "user", "content": [{"type": "text", "text": "QA_F10_M2"}], "run": false}' --expect 200 --quiet @@ -450,6 +453,7 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key while its socket is open. ```sh N_MSG=$(control-agent-server api GET "/api/conversations/$A/events/count" --query kind=$MSG --field .) + N_ACT=$(control-agent-server api GET "/api/conversations/$A/events/count" --query kind=$ACT --field .) N_ALL=$(control-agent-server api GET "/api/conversations/$A/events/count" --field .) N_PELICAN=$(control-agent-server api GET "/api/conversations/$A/events/count" --query body=PELICAN --query source=agent --field .) control-agent-server ws start "/sockets/events/$A" --name qa-f10-ask --duration 120 @@ -458,18 +462,24 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key --expect 200 --check response matches '(?i)pelican' --timeout 120 --save F10.ask-agent/ask control-agent-server api GET "/api/conversations/$A/events/count" --query kind=$MSG --check . eq "$N_MSG" --save F10.ask-agent/messages control-agent-server api GET "/api/conversations/$A/events/count" --query body=PELICAN --query source=agent --check . eq "$N_PELICAN" - control-agent-server api GET "/api/conversations/$A/events/count" --check . eq "$N_ALL" + control-agent-server api GET "/api/conversations/$A/events/count" --query kind=$ACT --check . eq "$N_ACT" + control-agent-server api GET "/api/conversations/$A/events/count" --check . eq "$((N_ALL + 1))" + control-agent-server api GET "/api/conversations/$A/events/search" --query kind=$STATE --query sort_order=TIMESTAMP_DESC --query limit=1 --quiet \ + --check items.0.key eq stats --check items.0.value.usage_to_metrics.ask-agent-llm.accumulated_token_usage.completion_tokens ge 1 --save F10.ask-agent/stats-event control-agent-server api GET "/api/conversations/$A" --quiet --check execution_status eq finished + control-agent-server ws read qa-f10-ask --wait 15 --kinds ConversationStateUpdateEvent --contains '"ask-agent-llm"' --expect-min 1 --save F10.ask-agent/stats-frame control-agent-server ws stop qa-f10-ask --kinds MessageEvent,ActionEvent --expect-none --save F10.ask-agent/frames ``` The answer names the codename, which only the conversation's history holds - (the check is on that fact, not the wording). Nothing is recorded: the - message count, the count of agent events that mention the codename (the - source run's own reply or reasoning may already mention it, so the check - compares with the count before the call) and the total event count are - unchanged. Nothing is pushed: the socket was live before the call (its - `full_state` snapshot had arrived, and the same kind of socket delivered - `Condensation` frames in `F10.condense-llm`), and `A` is still `finished`. + (the check is on that fact, not the wording). Neither the question nor the + answer is recorded: the message count, the action count and the count of + agent events that mention the codename (the source run's own reply or + reasoning may already mention it, so the check compares with the count + before the call) are unchanged. The log grows by exactly one event, a + `stats` `ConversationStateUpdateEvent` whose `usage_to_metrics` has an + `ask-agent-llm` entry with the call's tokens; the socket, live before the + call (its `full_state` snapshot had arrived), delivers that frame and no + `MessageEvent` or `ActionEvent`, and `A` is still `finished`. - **Bad input, no key, unreachable model (`F10.ask-agent-errors`).** ```sh control-agent-server api POST "/api/conversations/$A/ask_agent" --json '{}' --expect 422 --check detail.0.loc.1 eq question --save F10.ask-agent-errors/missing-question @@ -491,7 +501,9 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key control-agent-server api POST "/api/conversations/$R/ask_agent" --json '{"question": "QA_F10_SIDE_Q"}' --expect 200 \ --check response eq QA_F10_SIDE --expect-max-ms 15000 --timeout 60 --save F10.ask-agent-running/ask control-agent-server api GET "/api/conversations/$R" --quiet --check execution_status eq running --save F10.ask-agent-running/still-running - control-agent-server api GET "/api/conversations/$R/events/count" --check . eq "$N_R" + control-agent-server api GET "/api/conversations/$R/events/count" --check . eq "$((N_R + 1))" + control-agent-server api GET "/api/conversations/$R/events/search" --query kind=$STATE --query sort_order=TIMESTAMP_DESC --query limit=1 --quiet \ + --check items.0.key eq stats --check items.0.value.usage_to_metrics.ask-agent-llm exists --save F10.ask-agent-running/stats-event control-agent-server sink read --name qa-f10-side --contains QA_F10_SIDE_Q --expect-min 1 --expect-max 1 control-agent-server api POST "/api/conversations/$R/interrupt" --expect 200 --quiet control-agent-server conversation wait "$R" --until paused --timeout 30 @@ -500,9 +512,9 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key The stub's first call (the run's step) is still hanging when the question arrives; the second call carries the question and answers `QA_F10_SIDE` within a fraction of a second, so `ask_agent` neither takes the state lock - the step holds nor waits for it. `R` is still `running`, its event count is - unchanged, and after the interrupt (which frees the run slot) no event - carries the side answer. + the step holds nor waits for it. `R` is still `running`, its log grew only + by the ask's `stats` update (an `ask-agent-llm` entry), and after the + interrupt (which frees the run slot) no event carries the side answer. - **Unknown conversation (`F10.ask-agent-unknown`).** The route documents 404, like condense and navigate. ```sh @@ -549,6 +561,40 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key A second `Condensation` is recorded and ask_agent still names the codename from the reloaded history (the condenser keeps the first two events, the system prompt and the user's message); `A` stays `finished`. +- **Ask usage on GET and after a restart (`F10.ask-agent-usage-saved`), known bug #6844.** + A conversation of its own on an `llm-stub` with fixed usage numbers: the + run's reply reports 11 prompt and 7 completion tokens, the ask's reply 5 + and 3. The bullet restarts the run, so it comes last. + ```sh + USAGE=$(control-agent-server fixture llm-stub --name qa-f10-usage --step 'reply:QA_F10_USAGE_RUN@usage=11,7' \ + --step 'reply:QA_F10_USAGE_ASK@usage=5,3' --print-path) + U=$(control-agent-server conversation start --no-autotitle --prompt QA_F10_USAGE --wait --until finished,error --timeout 60 --print-id \ + --body-json "{\"agent\": {\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key\", \"base_url\": \"$USAGE/v1\", \"num_retries\": 0}, \"tools\": []}}") + control-agent-server api GET "/api/conversations/$U" --quiet --check execution_status eq finished \ + --check stats.usage_to_metrics.default.accumulated_token_usage.completion_tokens eq 7 + control-agent-server api POST "/api/conversations/$U/ask_agent" --json '{"question": "QA_F10_USAGE_Q"}' --expect 200 \ + --check response eq QA_F10_USAGE_ASK --save F10.ask-agent-usage-saved/ask + control-agent-server api GET "/api/conversations/$U/events/search" --query kind=$STATE --query sort_order=TIMESTAMP_DESC --query limit=1 --quiet \ + --check items.0.key eq stats --check items.0.value.usage_to_metrics.ask-agent-llm.accumulated_token_usage.prompt_tokens eq 5 \ + --check items.0.value.usage_to_metrics.ask-agent-llm.accumulated_token_usage.completion_tokens eq 3 --save F10.ask-agent-usage-saved/stats-event + control-agent-server api GET "/api/conversations/$U" --max-chars 300 \ + --check stats.usage_to_metrics.default.accumulated_token_usage.completion_tokens eq 7 \ + --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.completion_tokens eq 3 \ + --save F10.ask-agent-usage-saved/get # bug: status eq 200 && checks.0.ok eq true && checks.1.actual eq null + control-agent-server restart + control-agent-server api GET "/api/conversations/$U" --max-chars 300 \ + --check stats.usage_to_metrics.default.accumulated_token_usage.completion_tokens eq 7 \ + --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.prompt_tokens eq 5 \ + --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.completion_tokens eq 3 --save F10.ask-agent-usage-saved/after-restart + ``` + The run's usage is on GET (7 completion tokens under `default`), the ask + answers, and the `stats` event the ask recorded reports 5 and 3 tokens + under `ask-agent-llm`. Expected: GET shows the same `ask-agent-llm` entry + right away and after a restart + ([#6844](https://github.com/OpenHands/software-agent-sdk/issues/6844), fix in [#6845](https://github.com/OpenHands/software-agent-sdk/pull/6845)). + Today GET still lists only `default`: the ask updates the live stats + without saving the conversation's state, GET reads the saved state, and a + restart right after the ask loses the entry. ## Gotchas @@ -584,12 +630,14 @@ SIDE_AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key it does not wait for the step (`F10.ask-agent-running`), unlike `condense`, `fork` and `navigate`, which take the state lock (from source; not driven here). - On a conversation that has not been reloaded it records no event at all - (`F10.ask-agent` checks the total count). On one reloaded after a restart, - each call records and pushes a `stats` `ConversationStateUpdateEvent` - whose `usage_to_metrics` includes an `ask-agent-llm` entry, while GET's - `stats` keeps only the stored `default` and `condenser` usage (seen on - DeepSeek; not asserted). It passes the agent's tools to + Each call records and pushes one `stats` `ConversationStateUpdateEvent` + whose `usage_to_metrics` includes an `ask-agent-llm` entry + (`F10.ask-agent` checks the total count), so a client that counts events + around an ask sees one more. GET's `stats` does not show the + `ask-agent-llm` entry until the conversation's state is next saved (a + message, a run), and a restart before that loses it + (`F10.ask-agent-usage-saved`, + [#6844](https://github.com/OpenHands/software-agent-sdk/issues/6844)). It passes the agent's tools to the model and returns the first text block, so on a conversation with no messages yet DeepSeek's answer came back as an empty `response` (200). - `fork` and `navigate` return `ConversationInfo` with diff --git a/.agents/skills/verify-agent-server/references/feature-map/F16-workspace-serving-and-vscode.md b/.agents/skills/verify-agent-server/references/feature-map/F16-workspace-serving-and-vscode.md index 19186f3b80f..008d10c0611 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F16-workspace-serving-and-vscode.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F16-workspace-serving-and-vscode.md @@ -9,12 +9,15 @@ resolves outside the workspace (encoded `..`, absolute paths, symlinks) is 400. These two routes sit in their own auth group: they accept the `X-Session-API-Key` header or the `oh_workspace_session_key` cookie, so Agent Canvas can embed agent-made HTML, images and PDFs in `