Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .agents/skills/verify-agent-server/references/daily.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,14 @@ nothing in `BASE..TARGET` touched is not driven: it waits for the weekly pass.
it is reproduced and gets its note, so bugs filed outside a pass reach the
map. Model-backed recipes run only inside the LLM budget; without a key
they are `blocked` with the missing prerequisite.
3. **Twins.** For every family step 2 drove, `control-agent-server map links
--file Fnn` lists its SDK twins (`Same promise:` links). When verify-sdk's
map is in the checkout, replay each twin in-process too (`sdk-map run
--file Snn --only Snn.x`) unless its family is replayed anyway. Put each
pair side by side: a pair whose results disagree is a finding, usually at
the seam between the SDK and the server (serialization, the async path,
persistence, restart), unless the difference is already a known bug on
one side. Drive each bullet once.

## Reading the result

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Routes: `GET /api/conversations/search`, `GET /api/conversations/count`,

## Sub-features

- `F04.create-settings`: a create from the saved settings (`agent_settings` plus `secrets_encrypted`, the Agent Canvas path) returns 201 `idle`, adds one to the count, and persists the agent only in `base_state.json`, never in `meta.json`.
- `F04.create-settings`: a create from the saved settings (`agent_settings` plus `secrets_encrypted`, the Agent Canvas path) returns 201 `idle`, adds one to the count, and persists the agent only in `base_state.json`, never in `meta.json`. Same promise: `S19.settings-create-agent`.
- `F04.create-inline-agent`: an inline `agent` without `kind` is accepted as `Agent`; the API key is never echoed back and the response trims inline skills.
- `F04.create-agent-profile`: `agent_profile_id` resolves a stored Agent Profile and its LLM profile and records `launched_agent_profile`.
- `F04.create-validation`: a missing agent source, two agent sources, a missing workspace, a bad tag key, `max_iterations` 0 and invalid `agent_settings` are 422; an unknown `agent_profile_id` is 404; a profile with a dangling LLM reference is 422 `unresolved_profile_references`; no key is 401; nothing is created.
Expand All @@ -33,14 +33,14 @@ Routes: `GET /api/conversations/search`, `GET /api/conversations/count`,
- `F04.create-initial-sender`: an `initial_message` with `sender_conversation_id` naming another conversation saves the first user message with `sender: "conversation:<sender id hex>"` and a first text part naming the sender (`[Sent by [<title>](conversation:<hex>), another conversation, not by the user. ...]`); an unknown sender or the new conversation's own id is 422 and creates nothing.
- `F04.create-parent`: `parent_conversation_id` links a child that the parent lists in `sub_conversation_ids`; an unknown parent, a different workspace and a self-parent are 422.
- `F04.create-worktree`: `worktree: true` inside a git repository moves the conversation into its own worktree on branch `openhands/<id>`.
- `F04.create-observability`: the `X-OpenHands-Observability-*` headers are validated (422 for non-object metadata) and stored in `meta.json`.
- `F04.create-observability-precedence`: body `observability_span_name`, `observability_tags` and `observability_parent_span_context` win over their headers, header metadata merges under the body's (body keys win), the parent-span header applies when the body has none, and an over-long or malformed span-name header, non-scalar header metadata and an empty body tag are 422.
- `F04.create-secrets`: start-time `StaticSecret` values appear masked in the create response and are encrypted at rest.
- `F04.create-observability`: the `X-OpenHands-Observability-*` headers are validated (422 for non-object metadata) and stored in `meta.json`. Same promise: `S20.conversation-args`, `S20.automation-headers`.
- `F04.create-observability-precedence`: body `observability_span_name`, `observability_tags` and `observability_parent_span_context` win over their headers, header metadata merges under the body's (body keys win), the parent-span header applies when the body has none, and an over-long or malformed span-name header, non-scalar header metadata and an empty body tag are 422. Same promise: `S20.conversation-args`.
- `F04.create-secrets`: start-time `StaticSecret` values appear masked in the create response and are encrypted at rest. Same promise: `S04.secrets-constructor`.
- `F04.create-secrets-get`: a start-time secret is listed by `GET /api/conversations/{id}` right after the create.
- `F04.create-secrets-plain`: a plain-string secret value in the create body is rejected with 422, never a 500.
- `F04.run-real`: an initial message runs on DeepSeek flash to `finished`; the events socket replays the run, the final response is non-empty and autotitle names the conversation.
- `F04.final-response`: `agent_final_response` is empty before the agent answers, carries the exact `finish` message afterwards, and is 404 for an unknown id.
- `F04.title-llm-profile`: with `title_llm_profile` naming a saved LLM profile, autotitle calls that profile and not the agent's LLM; an unknown profile name falls back to the agent's LLM and never fails the create.
- `F04.final-response`: `agent_final_response` is empty before the agent answers, carries the exact `finish` message afterwards, and is 404 for an unknown id. Same promise: `S04.final-response`.
- `F04.title-llm-profile`: with `title_llm_profile` naming a saved LLM profile, autotitle calls that profile and not the agent's LLM; an unknown profile name falls back to the agent's LLM and never fails the create. Same promise: `S04.title-and-tags`.
- `F04.get`: `GET /api/conversations/{id}` returns `runtime_info`, trims skills unless `include_skills=true`, accepts the 32-hex id form, and answers 404, 422 and 401.
- `F04.batch-get`: `GET /api/conversations?ids=..` returns a list aligned with the ids, `null` for unknown ones, and 422 without or with malformed ids.
- `F04.batch-get-limit`: 100 ids in one batch are refused with a client error, not a 500.
Expand All @@ -49,14 +49,14 @@ Routes: `GET /api/conversations/search`, `GET /api/conversations/count`,
- `F04.count`: `GET /api/conversations/count` equals the number of conversations search lists.
- `F04.search-at-scale`: with 2,000 conversations, search pages through every one exactly once in both orders, count is 2,000, the last page answers within 4× the first page's p95 (first page under 2 s), 50 listings grow the server's RSS by under 300 MB, and all 2,000 are listed again after a restart.
- `F04.patch-title`: PATCH `title` is stripped, stored in `meta.json`, bumps `updated_at` and moves the conversation to the top of `UPDATED_AT_DESC`.
- `F04.patch-tags`: PATCH `tags` replaces the whole map in `meta.json` and `base_state.json` and pushes a `tags` state update on the events socket.
- `F04.patch-tags`: PATCH `tags` replaces the whole map in `meta.json` and `base_state.json` and pushes a `tags` state update on the events socket. Same promise: `S04.title-and-tags`.
- `F04.patch-tags-while-running`: while a run holds the conversation's state for a tool step, a tags PATCH waits for it, but GET, the batch get and search already show the new tags (from `meta.json`) although `base_state.json` still has the old ones; once the step ends the PATCH answers 200 and both files agree.
- `F04.patch-errors`: an empty or 201-character title, a bad tag key and a malformed id are 422; no key is 401; the stored values are untouched.
- `F04.patch-whitespace-title`: a whitespace-only title is rejected instead of erasing the stored title.
- `F04.patch-unknown-id`: PATCH on an unknown id answers the 404 the OpenAPI document declares, not 200 `{"success": false}`.
- `F04.delete`: DELETE closes the conversation's open sockets with 4004, stops a running conversation, removes its state, keeps the workspace, removes a worktree conversation's worktree but keeps its branch, orphans children, and later reads are 404 (socket close 4004); a malformed id is 422 and no key is 401.
- `F04.delete-unknown-id`: DELETE on an unknown or already deleted id answers the 404 the OpenAPI document declares, not 400.
- `F04.restart-persist`: after a restart the catalog, statuses, titles, tags, final responses and event replay are unchanged, and deleted conversations stay deleted.
- `F04.restart-persist`: after a restart the catalog, statuses, titles, tags, final responses and event replay are unchanged, and deleted conversations stay deleted. Same promise: `S04.persist-events`, `S04.resume`.
- `F04.read-no-hydrate`: after a restart, GET, the batch get, search and count read a cold conversation without loading it (no `owner_lease.json`), while `agent_final_response` loads it and claims the lease.

- `F04.child-start`: `POST .../children` starts a real-model task with fresh history, inherited configuration and a parent link, using a shared directory, an isolated git worktree, or the shared fallback for a non-git workspace.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,33 +23,33 @@ Routes: `POST /api/conversations/{conversation_id}/run`,

## Sub-features

- `F05.run-queued`: a message appended with `run: false` leaves the conversation `idle`; `POST /run` returns 200 at once and the agent runs on DeepSeek flash through `running` to `finished`, both transitions pushed on the events socket.
- `F05.run-finished-noop`: `POST /run` on a `finished` conversation without new input returns 200 and the run ends at once: no `execution_status` change on the socket, still `finished`, no new events (no model call).
- `F05.run-new-input`: a new user message moves a `finished` conversation back to `idle`, and `POST /run` runs it to `finished` again.
- `F05.pause-idle`: pausing an idle conversation returns 200, reports `paused` on GET, the status search and count and the events socket, and persists one `PauseEvent`.
- `F05.pause-repeat`: pausing a `paused` conversation again returns 200 and adds no second `PauseEvent`.
- `F05.run-resume-paused`: `POST /run` resumes a `paused` conversation through `running` to `finished`.
- `F05.interrupt-idle`: interrupt with nothing running falls back to pause: `paused`, one `PauseEvent`, no `InterruptEvent`.
- `F05.run-error`: a run whose model provider is unreachable ends in `error` with a `ConversationErrorEvent` (also pushed on the events socket) and is listed by `status=error`.
- `F05.run-queued`: a message appended with `run: false` leaves the conversation `idle`; `POST /run` returns 200 at once and the agent runs on DeepSeek flash through `running` to `finished`, both transitions pushed on the events socket. Same promise: `S01.run-to-finish`.
- `F05.run-finished-noop`: `POST /run` on a `finished` conversation without new input returns 200 and the run ends at once: no `execution_status` change on the socket, still `finished`, no new events (no model call). Same promise: `S01.run-to-finish`.
- `F05.run-new-input`: a new user message moves a `finished` conversation back to `idle`, and `POST /run` runs it to `finished` again. Same promise: `S01.run-to-finish`.
- `F05.pause-idle`: pausing an idle conversation returns 200, reports `paused` on GET, the status search and count and the events socket, and persists one `PauseEvent`. Same promise: `S04.pause`.
- `F05.pause-repeat`: pausing a `paused` conversation again returns 200 and adds no second `PauseEvent`. Same promise: `S04.pause`.
- `F05.run-resume-paused`: `POST /run` resumes a `paused` conversation through `running` to `finished`. Same promise: `S04.pause`.
- `F05.interrupt-idle`: interrupt with nothing running falls back to pause: `paused`, one `PauseEvent`, no `InterruptEvent`. Same promise: `S04.arun-interrupt`.
- `F05.run-error`: a run whose model provider is unreachable ends in `error` with a `ConversationErrorEvent` (also pushed on the events socket) and is listed by `status=error`. Same promise: `S04.run-error`.
- `F05.pause-terminal`: pause and interrupt on a `finished` or `error` conversation return 200 and change neither the status nor the event log.
- `F05.run-error-responsive`: a run that fails does not stall the server: other requests keep answering while the error is handled.
- `F05.unknown-id-run`: `POST /run` on an unknown id returns 404.
- `F05.unknown-id-pause`: `POST /pause` and `POST /interrupt` on an unknown id return the documented 404.
- `F05.control-errors`: all three routes answer 401 without or with a wrong session key and 422 for a malformed id, and the idle conversation they targeted is untouched (still `idle`, no `PauseEvent`).
- `F05.run-conflict`: `POST /run` while a run is active returns 409 `Conversation already running...`, and the active run keeps going without a second model call.
- `F05.pause-running`: pause during a long model call returns 200 at once; the events socket, status search and event log say `paused`, and `/run` stays 409 while the step is still in flight.
- `F05.pause-running`: pause during a long model call returns 200 at once; the events socket, status search and event log say `paused`, and `/run` stays 409 while the step is still in flight. Same promise: `S04.pause`.
- `F05.pause-running-get`: after a pause during a long model call, `GET /api/conversations/{id}` and `base_state.json` report `paused` like every other view.
- `F05.pause-running-release`: interrupting a paused conversation whose step is still in flight cancels it (`InterruptEvent`), GET catches up to `paused`, and the run slot is free again (a create that needs the slot succeeds).
- `F05.interrupt-running`: interrupt cancels a hanging model call well within the server's 5 s wait: 200, `paused`, an `InterruptEvent` and no `PauseEvent`, pushed on the events socket.
- `F05.run-after-interrupt`: after an interrupt, `POST /run` is accepted at once and the resumed run reaches `finished`.
- `F05.pause-tool-step`: pause while a tool call runs lets that step finish (its observation is recorded), then stops the loop before the next model call: `paused`, one `PauseEvent`, one model request.
- `F05.interrupt-tool-call`: interrupt while a tool call runs returns at once: `paused`, an `InterruptEvent`, an `AgentErrorEvent` for the unfinished call (same `tool_call_id`) and no observation; the resumed run sends that error to the model as the tool result and finishes.
- `F05.interrupt-running`: interrupt cancels a hanging model call well within the server's 5 s wait: 200, `paused`, an `InterruptEvent` and no `PauseEvent`, pushed on the events socket. Same promise: `S04.arun-interrupt`.
- `F05.run-after-interrupt`: after an interrupt, `POST /run` is accepted at once and the resumed run reaches `finished`. Same promise: `S04.arun-interrupt`.
- `F05.pause-tool-step`: pause while a tool call runs lets that step finish (its observation is recorded), then stops the loop before the next model call: `paused`, one `PauseEvent`, one model request. Same promise: `S04.pause`.
- `F05.interrupt-tool-call`: interrupt while a tool call runs returns at once: `paused`, an `InterruptEvent`, an `AgentErrorEvent` for the unfinished call (same `tool_call_id`) and no observation; the resumed run sends that error to the model as the tool result and finishes. Same promise: `S04.cancel-token`.
- `F05.run-capacity`: with `max_concurrent_runs` 1, `POST /run` on a second conversation returns 429 while the first holds the slot, leaves it `idle`, and succeeds once the first is interrupted.
- `F05.stuck-detected`: with `stuck_detection` on (the default), a run whose model repeats the same tool call ends `stuck` after four identical action/observation pairs (four model calls), with no `ConversationErrorEvent`; GET, `status=stuck` search and count and the events socket say `stuck`.
- `F05.stuck-rerun`: `POST /run` on a `stuck` conversation with no new input returns 200 and resumes it with a fresh stuck-detection window: the model is called again, and a model that keeps looping ends `stuck` again after four more identical action/observation pairs (as the SDK does, `S01.stuck-recover`).
- `F05.stuck-new-input`: a new user message moves a `stuck` conversation to `idle`, and `POST /run` runs it again to `finished`.
- `F05.max-iterations-stop`: a conversation created with `max_iterations: 6` and `stuck_detection: false` whose model never finishes makes exactly six model calls (past the stuck threshold) and ends `error` with exactly one `ConversationErrorEvent` `code: MaxIterationsReached` naming the limit, on REST and the events socket, never `stuck`.
- `F05.max-iterations-resume`: `POST /run` after `MaxIterationsReached` is accepted and gets a fresh budget of six model calls (a second `MaxIterationsReached`), and once the model calls `finish` the next run ends `finished`.
- `F05.stuck-detected`: with `stuck_detection` on (the default), a run whose model repeats the same tool call ends `stuck` after four identical action/observation pairs (four model calls), with no `ConversationErrorEvent`; GET, `status=stuck` search and count and the events socket say `stuck`. Same promise: `S01.stuck-detect`.
- `F05.stuck-rerun`: `POST /run` on a `stuck` conversation with no new input returns 200 and resumes it with a fresh stuck-detection window: the model is called again, and a model that keeps looping ends `stuck` again after four more identical action/observation pairs (as the SDK does, `S01.stuck-recover`). Same promise: `S01.stuck-recover`.
- `F05.stuck-new-input`: a new user message moves a `stuck` conversation to `idle`, and `POST /run` runs it again to `finished`. Same promise: `S01.stuck-recover`.
- `F05.max-iterations-stop`: a conversation created with `max_iterations: 6` and `stuck_detection: false` whose model never finishes makes exactly six model calls (past the stuck threshold) and ends `error` with exactly one `ConversationErrorEvent` `code: MaxIterationsReached` naming the limit, on REST and the events socket, never `stuck`. Same promise: `S01.max-iterations`.
- `F05.max-iterations-resume`: `POST /run` after `MaxIterationsReached` is accepted and gets a fresh budget of six model calls (a second `MaxIterationsReached`), and once the model calls `finish` the next run ends `finished`. Same promise: `S01.max-iterations`.
- `F05.restart-running-graceful`: a conversation running at a graceful restart comes back `paused`, with a `PauseEvent` and an `InterruptEvent`.
- `F05.restart-keeps-status`: `finished`, `paused` and `error` statuses survive a restart unchanged.
- `F05.restart-hard-running`: a conversation running when the server is SIGKILLed comes back `error` on the next start, and `/run` resumes it.
Expand Down
Loading
Loading