diff --git a/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md b/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md index 32d53be0292..ca176d105ea 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md +++ b/.agents/skills/verify-agent-server/references/feature-map/F18-settings-and-secrets.md @@ -49,7 +49,7 @@ Routes: `GET /api/settings/agent-schema`, `GET /api/settings/conversation-schema - `F18.sdk-clone-repos`: with the server secret `GITHUB_TOKEN` stored, the SDK's `RemoteWorkspace.clone_repos()` reads it from `GET /api/settings/secrets/GITHUB_TOKEN` and clones with it (a local git remote reachable only through the token-bearing URL), a provider whose token secret is missing (`GITLAB_TOKEN`) clones anonymously, and `get_repos_context()` names both clones. - `F18.ts-client`: the TypeScript `SettingsClient` reads both schemas, reads settings redacted, `plaintext` and `encrypted`, applies a sparse `updateSettings` that a second read shows, upserts, lists (also with `agentProfileId`), reads as text and deletes a secret, and rejects an unknown secret (404), a bad name (422) and a wrong key (401) with an `HttpError`. - `F18.persist-restart`: settings (plaintext key, model, conversation settings) and secrets read back unchanged after a restart with the same cipher key. -- `F18.experimental-features`: `GET /api/settings/experimental` lists the experimental features the user can switch (none are registered yet, so `[]`) without creating `settings.json`, and `PATCH /api/settings/experimental/{key}` answers 404 for a key the server does not know and stores nothing. +- `F18.experimental-features`: `GET /api/settings/experimental` lists the experimental features the user can switch (`conversation_reactions`, off) without creating `settings.json`, and `PATCH /api/settings/experimental/{key}` answers 404 for a key the server does not know and stores nothing. - `F18.upgrade-from-release`: settings, an LLM key and a secret saved by the latest released server read back identical after the same state is restarted on this checkout (`restart --checkout`), the upgrade does not rewrite either file, and the experimental routes answer. - `F18.corrupted-files`: with an unparsable `settings.json`, GET silently returns defaults and PATCH is 409 without touching the file; with an unparsable `secrets.json` the list is empty, GET by name 404, PUT and DELETE 500; restoring the files restores the values. - `F18.no-cipher`: without any cipher key, `X-Expose-Secrets: encrypted` is 503, `plaintext` still works, both files store plaintext, and the secrets routes are open without a key. @@ -668,18 +668,18 @@ Preconditions: - **Experimental features (`F18.experimental-features`).** List them and try to switch one the server does not know. ```sh - control-agent-server api GET /api/settings/experimental --expect 200 --check . len-eq 0 --save F18.experimental-features/list + control-agent-server api GET /api/settings/experimental --expect 200 --check . len-eq 1 --check 0.key eq conversation_reactions \ + --check 0.enabled eq false --save F18.experimental-features/list control-agent-server api PATCH /api/settings/experimental/qa_f18_unknown --json '{"enabled": true}' --expect 404 \ --check detail eq 'Unknown experimental feature: qa_f18_unknown' --save F18.experimental-features/unknown control-agent-server api PATCH /api/settings/experimental/qa_f18_unknown --json '{}' --expect 422 control-agent-server api GET /api/settings/experimental --auth none --expect 401 if grep -qs experimental_features "$AGENT_SERVER_VERIFY_RUN/home/.openhands/settings.json"; then false; fi ``` - No feature is registered yet, so the list is `[]` and every key is unknown - (404). The refused switch leaves `settings.json` without an - `experimental_features` entry. A feature added to `EXPERIMENTAL_FEATURES` - appears in the list as `{key, label, description, enabled}` and its switch - is stored in `experimental_features` by key. + The list holds `conversation_reactions`, off (F37 switches it), as + `{key, label, description, enabled}`. A key the server does not know is a + 404, and the refused switch leaves `settings.json` without an + `experimental_features` entry. - **Upgrade from the latest release (`F18.upgrade-from-release`).** A second run starts on the newest release tag, saves settings as a user of that release would, and is restarted on this checkout over the same state. diff --git a/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md b/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md new file mode 100644 index 00000000000..fac242646fe --- /dev/null +++ b/.agents/skills/verify-agent-server/references/feature-map/F37-reactions.md @@ -0,0 +1,179 @@ +# Emoji reactions + +A reaction is a small emoji on a message: the agent's way to say "seen, working +on it" before it answers, and the user's way to acknowledge an answer without +writing back. Both are saved as a non-LLM `ReactionEvent` (`target_event_id`, +`emoji`, `removed`, `source`), so clients fold them into each message's current +set and the model never sees them. The agent reacts with the `react` and +`unreact` tools, whose executor only reads the log: the observation carries the +reaction and the agent records the `ReactionEvent` right after it, on its own +thread. The user reacts through `POST /api/conversations/{id}/reactions`, +which does not run the agent. Emoji are a fixed set of shortcodes: `eyes`, +`+1`, `white_check_mark`, `tada`, `thinking_face`, `heart`. + +Reactions sit behind the user-facing feature flag `conversation_reactions` +("Conversation reactions", off by default). While it is on, every new +OpenHands-agent conversation gets `react` and `unreact` without naming them, +and the REST route accepts reactions; while it is off, the route answers 409 +and new conversations do not get the tools. Flags are listed with +`GET /api/settings/experimental` and switched with +`PATCH /api/settings/experimental/{key}` `{"enabled": bool}`, stored in +`PersistedSettings.experimental_features`, and read when a conversation starts. + +Source: `openhands-sdk/openhands/sdk/event/reaction.py`, `openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py` (`react`), `openhands-sdk/openhands/sdk/agent/agent.py` (`follow_up_events`), `openhands-tools/openhands/tools/reaction/`, `openhands-agent-server/openhands/agent_server/event_router.py`, `openhands-agent-server/openhands/agent_server/experimental_features.py`, `openhands-agent-server/openhands/agent_server/experimental_features_router.py`, `openhands-sdk/openhands/sdk/launch/finalize.py` (`experimental_feature_tools`) + +Needs: `llm` + +Routes: `POST /api/conversations/{conversation_id}/reactions` + +## Sub-features + +- `F37.flag`: `GET /api/settings/experimental` lists `conversation_reactions` with `enabled: false` on a fresh server; `PATCH .../conversation_reactions` `{"enabled": true}` returns it on and the list agrees; an unknown key is 404. +- `F37.flag-gates`: with the flag off, `POST .../reactions` is 409 and a new conversation has no `react`/`unreact`; with it on, a new conversation started without naming them has both tools. +- `F37.rest`: `POST .../reactions` with `{emoji, target_event_id}` saves a user `ReactionEvent` on that message and returns it; `removed: true` saves the removal; the conversation stays `idle`. +- `F37.rest-validation`: an emoji outside the set is 422, an unknown target is 400, and leaving out the target with no agent message yet is 400; nothing is saved. +- `F37.tool`: an agent with `react` reacts to the user's latest message when asked to react first, and leaves the reaction on after it answers: the log has the tool's `ObservationEvent` followed by one agent `ReactionEvent` whose target is that message, and no removal. +- `F37.rest-while-confirming`: a reaction posted while a tool call waits for confirmation answers 200 but is held back: it joins the log right after the call's result once the call is accepted, never between the call and its result, and the run finishes. +- `F37.rest-default`: after the agent has answered, a reaction without `target_event_id` goes on its answer: the latest agent message or `finish`. + +## How to get to it (agent POV) + +- REST: `POST /api/conversations/{conversation_id}/reactions` with + `{"emoji", "target_event_id"?, "removed"?}` returns the saved + `ReactionEvent`. Without `target_event_id` it targets the agent's latest + message or `finish` action. It never runs the agent. +- Agent tools (registered by the agent-server, not part of the default + preset): `react` and `unreact` from `openhands.tools.reaction`, with + `{"emoji", "message_id"?}`; without `message_id` they target the user's + latest message. The `conversation_reactions` feature flag adds them to + every new OpenHands-agent conversation; ACP agents do not get them. +- Events: `ReactionEvent` in search, get-one and the socket like other events; + the latest event for a (message, emoji, source) is the current state. +- SDK: `LocalConversation.react(emoji, target_event_id=None, removed=False)`. + +## Driving it with control-agent-server + +Preconditions: + +- A baseline run is live and exported, and `doctor` is ok. The block below + activates the DeepSeek profile (`$DEEPSEEK_API_KEY`) for `F37.tool`. + + ```sh + control-agent-server llm preset deepseek + RC=$(control-agent-server conversation start --no-autotitle --tools none --print-id) + control-agent-server api POST "/api/conversations/$RC/events" \ + --json '{"role":"user","content":[{"type":"text","text":"QA_F37 hello"}],"run":false}' --expect 200 + RM=$(control-agent-server api GET "/api/conversations/$RC/events/search" --query body=QA_F37 --field items.0.id) + ``` + +- **The feature flag (`F37.flag`, `F37.flag-gates`).** + ```sh + control-agent-server api GET /api/settings/experimental \ + --check 0.key eq conversation_reactions --check 0.enabled eq false --save F37.flag/list + OFF=$(control-agent-server conversation start --no-autotitle --tools none --print-id) + control-agent-server api POST "/api/conversations/$OFF/reactions" --json '{"emoji": "eyes"}' --expect 409 + control-agent-server api PATCH /api/settings/experimental/conversation_reactions --json '{"enabled": true}' \ + --expect 200 --check enabled eq true + control-agent-server api PATCH /api/settings/experimental/time_travel --json '{"enabled": true}' --expect 404 + ON=$(control-agent-server conversation start --no-autotitle --tools none --print-id) + control-agent-server api GET "/api/conversations/$ON" --check agent.tools len-eq 2 --save F37.flag-gates/tools + control-agent-server api GET "/api/conversations/$OFF" --check agent.tools len-eq 0 + ``` + The flag stays on for the rest of this family. +- **React over REST (`F37.rest`).** + Requires: `F37.flag` + ```sh + control-agent-server api POST "/api/conversations/$RC/reactions" --json "{\"emoji\": \"heart\", \"target_event_id\": \"$RM\"}" \ + --expect 200 --check kind eq ReactionEvent --check source eq user --check emoji eq heart \ + --check target_event_id eq "$RM" --check removed eq false --save F37.rest/add + control-agent-server api POST "/api/conversations/$RC/reactions" --json "{\"emoji\": \"heart\", \"target_event_id\": \"$RM\", \"removed\": true}" \ + --expect 200 --check removed eq true + control-agent-server api GET "/api/conversations/$RC/events/search" --query kind=ReactionEvent --check items len-eq 2 + control-agent-server api GET "/api/conversations/$RC" --check execution_status eq idle + ``` + Both events are in the log and nothing ran. +- **Bad reactions (`F37.rest-validation`).** + Requires: `F37.rest` + ```sh + control-agent-server api POST "/api/conversations/$RC/reactions" --json "{\"emoji\": \"rocket\", \"target_event_id\": \"$RM\"}" --expect 422 + control-agent-server api POST "/api/conversations/$RC/reactions" --json '{"emoji": "eyes", "target_event_id": "missing"}' --expect 400 + control-agent-server api POST "/api/conversations/$RC/reactions" --json '{"emoji": "eyes"}' --expect 400 \ + --check detail eq 'There is no agent message to react to.' --save F37.rest-validation/no-agent-message + control-agent-server api GET "/api/conversations/$RC/events/search" --query kind=ReactionEvent --check items len-eq 2 + ``` + Nothing new is saved. +- **Model-backed: the agent reacts (`F37.tool`).** + Needs: `llm` + Requires: `F37.flag-gates` (it turns the reactions flag on; without it the + agent has no react tool) + ```sh + AC=$(control-agent-server conversation start --no-autotitle --tools none --print-id) + control-agent-server conversation send "$AC" --wait --timeout 300 --text 'QA_F37_TOOL Please react to this message with the eyes emoji first, then reply with the single word ok.' + AM=$(control-agent-server api GET "/api/conversations/$AC/events/search" --query body=QA_F37_TOOL --field items.0.id) + control-agent-server api GET "/api/conversations/$AC/events/search" --query kind=ReactionEvent \ + --check items len-eq 1 --check items.0.source eq agent --check items.0.emoji eq eyes \ + --check items.0.target_event_id eq "$AM" --check items.0.removed eq false --save F37.tool/reactions + control-agent-server conversation events "$AC" --kinds ObservationEvent --contains 'Reacted :eyes:' --expect-min 1 + ``` + The agent reacted to the user's message before answering and left the + reaction on: a reaction stays until it no longer fits, so an answer does not + take it off. +- **Default target is the agent's answer (`F37.rest-default`).** + Requires: `F37.tool` + ```sh + LAST_ACTION=$(control-agent-server api GET "/api/conversations/$AC/events/search" --query source=agent --query sort_order=TIMESTAMP_DESC \ + --query kind=ActionEvent --field items.0.id) + # An agent that answered with `finish` has no MessageEvent: --field then + # prints an error, so fall back to a value that matches nothing. + LAST_MESSAGE=$(control-agent-server api GET "/api/conversations/$AC/events/search" --query source=agent --query sort_order=TIMESTAMP_DESC \ + --query kind=MessageEvent --field items.0.id 2>/dev/null) || LAST_MESSAGE=none + control-agent-server api POST "/api/conversations/$AC/reactions" --json '{"emoji": "+1"}' --expect 200 \ + --check source eq user --check target_event_id matches "^($LAST_ACTION|$LAST_MESSAGE)\$" --save F37.rest-default/add + ``` + With no target the user's reaction goes on the agent's answer: its + `finish` action when it finished with the tool (then the latest agent + action), or its latest `MessageEvent` when it answered in plain text. +- **Reacting while a tool call waits (`F37.rest-while-confirming`).** A + conversation that confirms every action answers once, then stops at its + next tool call; the user reacts to the first answer while it waits. + Requires: `F37.flag-gates` + ```sh + WC=$(control-agent-server conversation start --confirmation-policy always --prompt 'Run: echo first' --wait --until waiting_for_confirmation --print-id) + control-agent-server api POST "/api/conversations/$WC/events/respond_to_confirmation" --json '{"accept": true}' --expect 200 + control-agent-server conversation wait "$WC" --until finished + control-agent-server conversation send "$WC" --text 'Now run: echo second' --wait --until waiting_for_confirmation + control-agent-server api POST "/api/conversations/$WC/reactions" --json '{"emoji": "heart"}' --expect 200 \ + --check source eq user --save F37.rest-while-confirming/add + control-agent-server api GET "/api/conversations/$WC/events/search" --query kind=ReactionEvent --check items len-eq 0 + control-agent-server api POST "/api/conversations/$WC/events/respond_to_confirmation" --json '{"accept": true}' --expect 200 + control-agent-server conversation wait "$WC" --until finished + control-agent-server api GET "/api/conversations/$WC/events/search" --query kind=ReactionEvent \ + --check items len-eq 1 --check items.0.emoji eq heart --save F37.rest-while-confirming/saved + control-agent-server api GET "/api/conversations/$WC/events/search" --query kind=Condensation --check items len-eq 0 + control-agent-server api GET "/api/conversations/$WC/events/search" --query kind=ObservationEvent --query limit=100 \ + --check items len-eq 2 --save F37.rest-while-confirming/observations + ``` + The reaction is not in the log while the call waits; it is saved right + after the call's result, and the run finishes with exactly the two commands + the user asked for and no condensation. A reaction between a call and + its result would make the model's API refuse the history. Held reactions + live in memory: a server restart while the call waits drops them. + +## Gotchas + +- The flag is read when a conversation starts: turning it on or off does not + change the tools of running conversations, only the REST route. Child + conversations reuse their parent's agent, and so its tools. + +- Reactions do not wake the agent. A user's `+1` on a question is not an + answer yet. +- The agent's `react` call is an ordinary tool call, so the `ActionEvent` and + its short `ObservationEvent` stay in the log next to the `ReactionEvent`; + clients that show reactions as chips can hide that pair. +- A user's reaction is saved under the conversation's state lock. The + server's run loop lets go of it during LLM calls, so during a run + `POST .../reactions` waits only while a tool or an event write holds it. +- Targets come from the active branch: after navigating back, messages on the + abandoned branch cannot be reacted to. +- A client older than this event kind cannot parse a conversation's events + once it holds a reaction, as with any new event kind. diff --git a/.agents/skills/verify-agent-server/references/feature-map/README.md b/.agents/skills/verify-agent-server/references/feature-map/README.md index 3a10afe2899..482b7cb7a78 100644 --- a/.agents/skills/verify-agent-server/references/feature-map/README.md +++ b/.agents/skills/verify-agent-server/references/feature-map/README.md @@ -10,7 +10,7 @@ recipe. Maintenance baseline: main@4db1ad742ad143ddc3411631f04efb70323f40f9 (2026-11-09). The first pass; the next maintenance pass starts from this commit, a pass proposes the next baseline in its PR, and merging that PR accepts it. -The map has **955 sub-features**. +The map has **962 sub-features**. ## Baseline preconditions @@ -99,6 +99,7 @@ them honest: [../maintenance.md](../maintenance.md). | F32 | [Event webhooks and product telemetry](F32-webhooks-and-telemetry.md) | event webhooks (event batches, conversation updates, headers, buffering, flush, limits, retries) and product telemetry (off by default, exporter lifecycle, sanitized payloads, consent) | behavior only: `webhooks` / `OH_WEBHOOKS_*`, `OH_TELEMETRY_*` | llm; launch `--webhook-sink` | 30 | | F33 | [Python SDK remote client (RemoteConversation, RemoteWorkspace)](F33-sdk-remote-client.md) | the Python SDK in this checkout against the server: RemoteWorkspace commands, files, git and settings readers; RemoteConversation create, run, callbacks, streaming, reattach, fork, confirmation, pause, interrupt, stop hooks, client tools, completion logs | behavior only: `Workspace(host=...)`, `RemoteWorkspace`, `RemoteConversation` | llm, tmux, git | 43 | | F34 | [TypeScript client (@openhands/typescript-client)](F34-typescript-client.md) | the TypeScript client against this checkout: server and version gate, workspace, bash, conversation, state, events and socket clients, hooks and MCP clients, endpoint audit and drift | behavior only: `@openhands/typescript-client` (`clients/typescript`) | llm, node, uvx, git, tmux, network | 32 | +| F37 | [Emoji reactions and feature flags](F37-reactions.md) | the `conversation_reactions` feature flag, the user's reactions over REST, validation, the agent's `react`/`unreact` tools, default targets | `GET /api/settings/experimental`, `PATCH /api/settings/experimental/{key}`, `POST /api/conversations/{id}/reactions` | llm | 7 | ## Not mapped diff --git a/clients/typescript/endpoint-audit.config.json b/clients/typescript/endpoint-audit.config.json index 36d15997d13..9424af46277 100644 --- a/clients/typescript/endpoint-audit.config.json +++ b/clients/typescript/endpoint-audit.config.json @@ -36,6 +36,12 @@ "owner": "OpenHands SDK maintainers", "tracking": "https://github.com/OpenHands/software-agent-sdk/issues/6758" }, + { + "endpoints": ["POST /api/conversations/{}/reactions"], + "reason": "Client-ahead API for conversation reactions, added with the Agent Server endpoint in the same PR.", + "owner": "OpenHands SDK maintainers", + "tracking": "https://github.com/OpenHands/software-agent-sdk/pull/6609" + }, { "endpoints": ["POST /api/acp/models"], "reason": "Client-ahead API stacked on the pending Agent Server ACP model discovery endpoint.", diff --git a/clients/typescript/src/__tests__/api-clients.test.ts b/clients/typescript/src/__tests__/api-clients.test.ts index 1a838d661a7..e6540a82003 100644 --- a/clients/typescript/src/__tests__/api-clients.test.ts +++ b/clients/typescript/src/__tests__/api-clients.test.ts @@ -5,6 +5,7 @@ import { ConversationManager, HttpClient, HttpError, + isReactionEvent, RemoteConversation, RemoteEventsList, RemoteWorkspace, @@ -2359,6 +2360,43 @@ describe('Auxiliary API clients', () => { ); }); + it('ConversationClient.react sends only the fields it is given', async () => { + const saved = { + id: 'r1', + kind: 'ReactionEvent', + source: 'user', + timestamp: '2026-11-07T00:00:00Z', + target_event_id: 'm1', + emoji: 'heart', + removed: false, + }; + global.fetch = vi.fn().mockImplementation(() => + Promise.resolve( + new Response(JSON.stringify(saved), { + status: 200, + headers: { 'content-type': 'application/json' }, + }) + ) + ) as typeof fetch; + + const client = new ConversationClient({ host: 'http://example.com' }); + const event = await client.react('conv-1', 'heart', { targetEventId: 'm1', removed: true }); + await client.react('conv-1', '+1'); + + expect(isReactionEvent(event)).toBe(true); + expect(isReactionEvent({ ...saved, kind: 'MessageEvent' })).toBe(false); + const fetchMock = global.fetch as Mock; + expect(fetchMock.mock.calls[0][0]).toBe( + 'http://example.com/api/conversations/conv-1/reactions' + ); + expect(JSON.parse(fetchMock.mock.calls[0][1].body)).toEqual({ + emoji: 'heart', + target_event_id: 'm1', + removed: true, + }); + expect(JSON.parse(fetchMock.mock.calls[1][1].body)).toEqual({ emoji: '+1' }); + }); + it('SettingsClient fetches and updates settings and secrets', async () => { const responses = [ { agent_settings: { llm_model: 'gpt-4o' }, conversation_settings: {} }, diff --git a/clients/typescript/src/client/conversation-client.ts b/clients/typescript/src/client/conversation-client.ts index 4ba759f8811..0ab4c684cf1 100644 --- a/clients/typescript/src/client/conversation-client.ts +++ b/clients/typescript/src/client/conversation-client.ts @@ -22,6 +22,7 @@ import type { UpdateSecretsRequest, } from '../models/conversation'; import type { ConfirmationPolicyBase } from '../types/base'; +import type { ReactionEmoji, ReactionEvent } from '../events/types'; export interface ConversationClientOptions { host: string; @@ -210,6 +211,28 @@ export class ConversationClient { }); } + /** + * Put the user's emoji reaction on a message, or take it off with + * `removed: true`. Without `targetEventId` it goes on the agent's latest + * message. Does not run the agent. The server answers 409 while the + * `conversation_reactions` feature is off. + */ + async react( + conversationId: string, + emoji: ReactionEmoji, + options: { targetEventId?: string; removed?: boolean } = {} + ): Promise { + const response = await this.client.post( + `/api/conversations/${conversationId}/reactions`, + { + emoji, + ...(options.targetEventId === undefined ? {} : { target_event_id: options.targetEventId }), + ...(options.removed === undefined ? {} : { removed: options.removed }), + } + ); + return response.data; + } + async pauseConversation(conversationId: string): Promise { const response = await this.client.post( `/api/conversations/${conversationId}/pause`, diff --git a/clients/typescript/src/events/types.ts b/clients/typescript/src/events/types.ts index 89e439174d2..8b34af7e7f4 100644 --- a/clients/typescript/src/events/types.ts +++ b/clients/typescript/src/events/types.ts @@ -165,11 +165,38 @@ export interface ThinkEvent extends BaseEvent { thought: string; } +/** Emoji shortcodes a reaction can use. */ +export type ReactionEmoji = 'eyes' | '+1' | 'white_check_mark' | 'tada' | 'thinking_face' | 'heart'; + +/** + * Reaction event - an emoji put on (or taken off) a message. + * + * The latest event for a (target, emoji, source) triple is the current state. + * The agent sees the user's reactions on its next turn, as one line with the + * end of the message (`target_text_ending`); its own stay out of what it sees. + * Needs the `conversation_reactions` feature. + */ +export interface ReactionEvent extends BaseEvent { + kind: 'ReactionEvent'; + /** The message or finish action the reaction is on */ + target_event_id: EventID; + emoji: ReactionEmoji; + /** True when this event takes the reaction off */ + removed: boolean; + /** Who reacted: the agent or the user */ + source: 'agent' | 'user'; + /** The end of the reacted message's text (at most 500 characters), on the user's reactions */ + target_text_ending?: string | null; + /** True when the message is longer than `target_text_ending` */ + target_text_clipped?: boolean; +} + /** * Union type of all conversation events */ export type ConversationEvent = | AgentServerEvent + | ReactionEvent | ConfirmationRequestEvent | ConfirmationResponseEvent | StuckDetectionEvent @@ -183,6 +210,13 @@ export function isMessageEvent(event: BaseEvent): event is MessageEvent { return event.kind === 'MessageEvent'; } +/** + * Type guard to check if an event is a ReactionEvent + */ +export function isReactionEvent(event: BaseEvent): event is ReactionEvent { + return event.kind === 'ReactionEvent'; +} + /** * Type guard to check if an event is an ActionEvent */ diff --git a/clients/typescript/src/index.ts b/clients/typescript/src/index.ts index 12abb61eb81..5cca2ac6914 100644 --- a/clients/typescript/src/index.ts +++ b/clients/typescript/src/index.ts @@ -69,6 +69,7 @@ export { isConversationErrorEvent, isCondensationEvent, isHookExecutionEvent, + isReactionEvent, } from './events/types'; export type { EventID, @@ -98,6 +99,8 @@ export type { StuckDetectionEvent, FinishEvent, ThinkEvent, + ReactionEmoji, + ReactionEvent, HookExecutionEvent, HookExecutionEventType, ConversationEvent, diff --git a/openhands-agent-server/openhands/agent_server/conversation_service.py b/openhands-agent-server/openhands/agent_server/conversation_service.py index 6dc0bc001e7..39003f51c78 100644 --- a/openhands-agent-server/openhands/agent_server/conversation_service.py +++ b/openhands-agent-server/openhands/agent_server/conversation_service.py @@ -25,6 +25,10 @@ EventService, RunSlot, ) +from openhands.agent_server.experimental_features import ( + experimental_feature_tools, + host_experimental_features, +) from openhands.agent_server.launch import ( is_codex_source, launch_source, @@ -1583,6 +1587,11 @@ async def _create_conversation( # ``ACPAgentSettings.agent_context`` is nullable, hence the guard. stored_context = settings.agent_settings.agent_context load_memory = bool(stored_context and stored_context.load_memory) + # In a Docker runtime the user's switches live on the host. + host_features = host_experimental_features() + experimental_features = ( + settings.experimental_features if host_features is None else host_features + ) source = await asyncio.to_thread( launch_source, @@ -1650,6 +1659,9 @@ async def _create_conversation( additions=request.agent_launch_additions, extra_suffixes=[worktree.guidance] if worktree else (), client_tools=client_tools, + experimental_feature_tools=experimental_feature_tools( + experimental_features + ), load_memory=load_memory, managed_secrets=( [CODEX_AUTH_SECRET_NAME] if managed_codex_credential else () diff --git a/openhands-agent-server/openhands/agent_server/docker_runtime/registry.py b/openhands-agent-server/openhands/agent_server/docker_runtime/registry.py index 912b62040d9..8ffbe276c84 100644 --- a/openhands-agent-server/openhands/agent_server/docker_runtime/registry.py +++ b/openhands-agent-server/openhands/agent_server/docker_runtime/registry.py @@ -4,6 +4,7 @@ import asyncio import hashlib +import json import os import subprocess import time @@ -20,12 +21,16 @@ from openhands.agent_server.conversation_registry import ConversationRegistry from openhands.agent_server.docker_runtime.provisioning import RuntimeProvisioningStore from openhands.agent_server.docker_runtime.storage import DockerRuntimeStorage +from openhands.agent_server.experimental_features import ( + RUNTIME_EXPERIMENTAL_FEATURES_ENV, +) from openhands.agent_server.launch import container_browser_enabled from openhands.agent_server.local_storage import remove_conversation_worktree from openhands.agent_server.models import ( ConversationRuntimeInfo, ConversationRuntimeStatus, ) +from openhands.agent_server.persistence import get_settings_store from openhands.agent_server.persistence.store import _get_persistence_dir from openhands.agent_server.storage import Reclaimer from openhands.sdk.conversation.request import ConversationContainerOptions @@ -614,6 +619,18 @@ async def _evict_idle_runtimes(self, ttl_seconds: float) -> None: ) await service.refresh_persisted_conversation(conversation_id) + def _experimental_features(self) -> dict[str, bool]: + """The user's experimental-feature switches, for a new container.""" + try: + settings = get_settings_store(self.config).load() + except (PermissionError, OSError): + logger.warning( + "Cannot read settings; the runtime starts with features off", + exc_info=True, + ) + return {} + return dict(settings.experimental_features) if settings else {} + def _build_container(self, conversation_id: UUID) -> ConversationContainer: identity = self.provisioning.load(conversation_id) options = identity.container or ConversationContainerOptions() @@ -653,6 +670,9 @@ def _build_container(self, conversation_id: UUID) -> ConversationContainer: if identity.launched_agent_profile else "" ), + RUNTIME_EXPERIMENTAL_FEATURES_ENV: json.dumps( + self._experimental_features() + ), } ) if "DEBUG" in os.environ: @@ -678,6 +698,7 @@ def _build_container(self, conversation_id: UUID) -> ConversationContainer: "OH_SECRET_KEY", V1_SESSION_API_KEY_ENV, "OH_RUNTIME_LAUNCHED_PROFILE", + RUNTIME_EXPERIMENTAL_FEATURES_ENV, "DEBUG", ] forwarded_env_names.extend( diff --git a/openhands-agent-server/openhands/agent_server/docker_runtime/routers.py b/openhands-agent-server/openhands/agent_server/docker_runtime/routers.py index 1bc30c29358..2fe72a865f1 100644 --- a/openhands-agent-server/openhands/agent_server/docker_runtime/routers.py +++ b/openhands-agent-server/openhands/agent_server/docker_runtime/routers.py @@ -26,6 +26,7 @@ DockerConversationRegistry, RuntimeArchivedError, ) +from openhands.agent_server.event_router import require_reactions_enabled from openhands.agent_server.launch import ( forward_to_runtime, launch_http_exception, @@ -428,6 +429,10 @@ async def proxy_conversation( }: raise HTTPException(501, "This operation is unavailable in Docker runtime mode") registry = get_registry(request) + if tail == "reactions" and request.method == "POST": + # The runtime's server has no settings of its own to check: the + # user's switch lives here and follows the switch at once. + await require_reactions_enabled(registry.config) container = await _container(registry, conversation_id) upstream_path = _upstream_path( request, diff --git a/openhands-agent-server/openhands/agent_server/event_router.py b/openhands-agent-server/openhands/agent_server/event_router.py index 6f1070fc03b..dab297962e7 100644 --- a/openhands-agent-server/openhands/agent_server/event_router.py +++ b/openhands-agent-server/openhands/agent_server/event_router.py @@ -19,22 +19,30 @@ from starlette.requests import Request from starlette.responses import JSONResponse +from openhands.agent_server._secrets_exposure import get_config +from openhands.agent_server.config import Config from openhands.agent_server.conversation_service import ConversationService from openhands.agent_server.dependencies import ( get_conversation_service, get_event_service, ) from openhands.agent_server.event_service import EventService +from openhands.agent_server.experimental_features import ( + host_experimental_features, + is_enabled, +) from openhands.agent_server.models import ( ConfirmationResponseRequest, EventSortOrder, + ReactionRequest, SendMessageRequest, Success, ) from openhands.agent_server.openai.usage import token_usage_delta +from openhands.agent_server.persistence import PersistedSettings, get_settings_store from openhands.sdk import Message from openhands.sdk.conversation.state import ConversationExecutionStatus -from openhands.sdk.event import Event +from openhands.sdk.event import Event, ReactionEvent event_read_router = APIRouter( @@ -351,6 +359,67 @@ async def respond_to_confirmation( return Success() +reaction_router = APIRouter( + prefix="/conversations/{conversation_id}/reactions", tags=["Events"] +) + + +async def require_reactions_enabled(config: Config) -> None: + """Refuse with 409 unless the user has conversation reactions on.""" + try: + settings = await asyncio.to_thread( + lambda: get_settings_store(config).load() or PersistedSettings() + ) + except (PermissionError, OSError): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail="Cannot read the settings file to check the feature flag", + ) + if not is_enabled(settings.experimental_features, "conversation_reactions"): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail="Conversation reactions are turned off", + ) + + +@reaction_router.post( + "", + responses={ + 400: {"description": "There is no such message to react to"}, + 404: {"description": "Item not found"}, + 409: {"description": "Conversation reactions are turned off"}, + }, +) +async def react_to_message( + request: ReactionRequest, + http_request: Request, + event_service: EventService = Depends(get_event_service), +) -> ReactionEvent: + """Add or remove the user's emoji reaction on a message. + + The reaction is saved as a `ReactionEvent` and streamed like other events; + it does not run the agent. While tool calls wait for their results (a + pending confirmation) it is held back and saved once they are answered. + Without `target_event_id` it goes on the agent's latest message. Needs + the `conversation_reactions` feature on; + unlike the agent's tools, which a conversation gets when it starts, this + check follows the switch at once. + """ + # A Docker runtime's server finds the user's switch on the host, which + # checks it before forwarding the request. + if host_experimental_features() is None: + await require_reactions_enabled(get_config(http_request)) + try: + return await event_service.react( + request.emoji, request.target_event_id, removed=request.removed + ) + except ValueError as exc: + if str(exc) == "inactive_service": + raise + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) + + event_router = APIRouter() event_router.include_router(event_read_router) event_router.include_router(event_write_router) +event_router.include_router(reaction_router) diff --git a/openhands-agent-server/openhands/agent_server/event_service.py b/openhands-agent-server/openhands/agent_server/event_service.py index dba0221adf0..39b5487e1bb 100644 --- a/openhands-agent-server/openhands/agent_server/event_service.py +++ b/openhands-agent-server/openhands/agent_server/event_service.py @@ -75,6 +75,7 @@ from openhands.sdk.event import ( AgentErrorEvent, ObservationBaseEvent, + ReactionEvent, StreamingDeltaEvent, ) from openhands.sdk.event.conversation_error import ConversationErrorEvent @@ -2134,6 +2135,27 @@ async def respond_to_confirmation(self, request: ConfirmationResponseRequest): else: await self.reject_pending_actions(request.reason) + async def react( + self, + emoji: str, + target_event_id: str | None = None, + *, + removed: bool = False, + ) -> ReactionEvent: + """Record the user's reaction on a message; it does not run the agent. + + Raises: + ValueError: The emoji is not allowed or there is no such message. + """ + if not self._conversation: + raise ValueError("inactive_service") + conversation = self._conversation + loop = asyncio.get_running_loop() + return await loop.run_in_executor( + None, + partial(conversation.react, emoji, target_event_id, removed=removed), + ) + async def reject_pending_actions(self, reason: str): """Reject all pending actions and publish updated state.""" if not self._conversation: diff --git a/openhands-agent-server/openhands/agent_server/experimental_features.py b/openhands-agent-server/openhands/agent_server/experimental_features.py index 6f5b574b106..9e77e6a966e 100644 --- a/openhands-agent-server/openhands/agent_server/experimental_features.py +++ b/openhands-agent-server/openhands/agent_server/experimental_features.py @@ -8,13 +8,26 @@ To add a feature, append an :class:`ExperimentalFeatureDefinition` to ``EXPERIMENTAL_FEATURES`` and read it where it takes effect with :func:`is_enabled`. + +The tools a feature gives the agent are chosen when a conversation starts +(:func:`experimental_feature_tools`), so they follow the switch for +conversations started after it; a route that checks the switch follows it at +once. Child conversations reuse their parent's agent and tools. + +A Docker runtime's server has its own, empty settings: the host passes it the +user's switches in ``OH_RUNTIME_EXPERIMENTAL_FEATURES`` and checks the live +ones itself before forwarding a request (see :func:`host_experimental_features`). """ +import json +import os from collections.abc import Mapping from dataclasses import dataclass from pydantic import BaseModel +from openhands.sdk.tool.spec import Tool + @dataclass(frozen=True) class ExperimentalFeatureDefinition: @@ -36,7 +49,17 @@ class ExperimentalFeatureUpdate(BaseModel): enabled: bool -EXPERIMENTAL_FEATURES: list[ExperimentalFeatureDefinition] = [] +CONVERSATION_REACTIONS = ExperimentalFeatureDefinition( + key="conversation_reactions", + label="Conversation reactions", + description=( + "The agent can put an emoji reaction on your messages, for example 👀 " + "when it starts on something before it answers, and you can react to " + "its messages. Applies to conversations started after you turn it on." + ), +) + +EXPERIMENTAL_FEATURES: list[ExperimentalFeatureDefinition] = [CONVERSATION_REACTIONS] def find_experimental_feature(key: str) -> ExperimentalFeatureDefinition | None: @@ -59,3 +82,37 @@ def experimental_feature_flags( ) for f in EXPERIMENTAL_FEATURES ] + + +def experimental_feature_tools(features: Mapping[str, bool]) -> list[Tool]: + """Tools to add to an OpenHands agent for the features that are on.""" + if is_enabled(features, CONVERSATION_REACTIONS.key): + # Tool names as registered by openhands.tools.reaction. + return [Tool(name="react"), Tool(name="unreact")] + return [] + + +RUNTIME_EXPERIMENTAL_FEATURES_ENV = "OH_RUNTIME_EXPERIMENTAL_FEATURES" + + +def host_experimental_features() -> dict[str, bool] | None: + """The host's switches, in a Docker runtime's server; None elsewhere. + + The host sets them when it starts the runtime's container, so they pick + the tools of the conversations started there. A route that follows a + switch at once is checked by the host before it forwards the request. + """ + raw = os.getenv(RUNTIME_EXPERIMENTAL_FEATURES_ENV) + if raw is None: + return None + try: + parsed = json.loads(raw) if raw else {} + except ValueError: + return {} + if not isinstance(parsed, dict): + return {} + return { + key: value + for key, value in parsed.items() + if find_experimental_feature(key) is not None and isinstance(value, bool) + } diff --git a/openhands-agent-server/openhands/agent_server/models.py b/openhands-agent-server/openhands/agent_server/models.py index 078bacc7a02..ae490ac2837 100644 --- a/openhands-agent-server/openhands/agent_server/models.py +++ b/openhands-agent-server/openhands/agent_server/models.py @@ -23,6 +23,8 @@ from openhands.sdk.conversation.state import ConversationExecutionStatus from openhands.sdk.conversation.types import ConversationTags from openhands.sdk.event.base import Event +from openhands.sdk.event.reaction import ReactionEmoji +from openhands.sdk.event.types import EventID from openhands.sdk.hooks import HookConfig from openhands.sdk.llm.message import ( # re-export ImageContent as ImageContent, @@ -460,6 +462,17 @@ def trim_conversation_response_skills(info: ConversationInfo) -> ConversationInf return info.model_copy(update={"agent": trimmed_agent}) +class ReactionRequest(BaseModel): + """Payload to add or remove the user's emoji reaction on a message.""" + + emoji: ReactionEmoji = Field(description="The emoji, as a shortcode.") + target_event_id: EventID | None = Field( + default=None, + description="The message to react to; the agent's latest when left out.", + ) + removed: bool = Field(default=False, description="True to remove the reaction.") + + class ConfirmationResponseRequest(BaseModel): """Payload to accept or reject a pending action.""" diff --git a/openhands-agent-server/openhands/agent_server/tool_router.py b/openhands-agent-server/openhands/agent_server/tool_router.py index e2df6e2f6c5..ee7c42d9346 100644 --- a/openhands-agent-server/openhands/agent_server/tool_router.py +++ b/openhands-agent-server/openhands/agent_server/tool_router.py @@ -8,6 +8,10 @@ # Registered only in the agent-server: the tool needs a server to create the # child on, so it is deliberately absent from get_default_tools(). import openhands.tools.child_conversation # noqa: E402, F401 + +# Reactions work in any conversation; registering them here makes them +# available to agents this server runs. +import openhands.tools.reaction # noqa: E402, F401 from openhands.agent_server.config import Config from openhands.agent_server.launch import ( can_probe_tools, diff --git a/openhands-sdk/openhands/sdk/agent/agent.py b/openhands-sdk/openhands/sdk/agent/agent.py index f10231f7e03..1adfebb75b7 100644 --- a/openhands-sdk/openhands/sdk/agent/agent.py +++ b/openhands-sdk/openhands/sdk/agent/agent.py @@ -1616,7 +1616,7 @@ def _execute_action_event( tool_name=tool.name, tool_call_id=action_event.tool_call.id, ) - return [obs_event] + return [obs_event, *observation.follow_up_events()] def _maybe_emit_vllm_tokens( self, llm_response: LLMResponse, on_event: ConversationCallbackType diff --git a/openhands-sdk/openhands/sdk/context/view/view.py b/openhands-sdk/openhands/sdk/context/view/view.py index ad9838ab6e4..fa66f841718 100644 --- a/openhands-sdk/openhands/sdk/context/view/view.py +++ b/openhands-sdk/openhands/sdk/context/view/view.py @@ -12,6 +12,7 @@ Condensation, CondensationRequest, LLMConvertibleEvent, + ReactionEvent, ) from openhands.sdk.event.base import Event @@ -125,6 +126,10 @@ def append_event(self, event: Event) -> None: case CondensationRequest(): self.unhandled_condensation_request = True + # The agent's own reactions: its react call is already in view. + case ReactionEvent(source="agent"): + pass + case LLMConvertibleEvent(): self.events.append(event) diff --git a/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py b/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py index 52b99e02385..ec577898016 100644 --- a/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py +++ b/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py @@ -6,6 +6,7 @@ import uuid from collections.abc import Mapping, Sequence from concurrent.futures import ThreadPoolExecutor +from datetime import datetime from pathlib import Path, PurePath from typing import Any, Final, TypeGuard, cast @@ -46,12 +47,19 @@ EventID, InterruptEvent, MessageEvent, + ObservationBaseEvent, ObservationEvent, PauseEvent, + ReactionEvent, UserRejectObservation, ) from openhands.sdk.event.conversation_error import ConversationErrorEvent from openhands.sdk.event.error_classification import AGENT_OUTCOME +from openhands.sdk.event.reaction import ( + check_reaction_emoji, + find_reaction_target, + reaction_target_text, +) from openhands.sdk.hooks import HookConfig, HookEventProcessor, create_hook_callback from openhands.sdk.io import FileStore, LocalFileStore from openhands.sdk.llm import LLM, Message, TextContent, content_to_str @@ -475,6 +483,25 @@ def _track_last_user_message(e): # Stop-hook feedback is emitted with source="environment". self._state.last_user_message_id = e.id + # The user's reactions that arrived while tool calls were waiting + # for their results. They join the log once every call is answered: + # nothing may sit between a tool call and its result. In memory only. + self._pending_reactions: list[ReactionEvent] = [] + + def _flush_pending_reactions(e): + if not self._pending_reactions or not isinstance(e, ObservationBaseEvent): + return + if ConversationState.get_unmatched_actions(self._state.active_branch()): + return + pending, self._pending_reactions = self._pending_reactions, [] + for reaction in pending: + # Restamped: the log's order is also its timestamp order. + self._on_event( + reaction.model_copy( + update={"timestamp": datetime.now().isoformat()} + ) + ) + callback_list = list(callbacks) if callbacks else [] # _default_callback (persist) runs before the caller-supplied callbacks # (e.g. a PubSub publish), so no subscriber is told about an event that @@ -483,7 +510,11 @@ def _track_last_user_message(e): # list never run. The visualizer is prepended below and so still renders # ahead of persist — that is local terminal output, not an announcement # a client can act on. - composed_list = [_default_callback] + callback_list + [_track_last_user_message] + composed_list = ( + [_default_callback] + + callback_list + + [_track_last_user_message, _flush_pending_reactions] + ) # Handle visualization configuration if isinstance(visualizer, ConversationVisualizerBase): # Use custom visualizer instance @@ -2102,6 +2133,45 @@ def send_message(self, message: str | Message, sender: str | None = None) -> Non ) self._on_event(user_msg_event) + def react( + self, + emoji: str, + target_event_id: EventID | None = None, + *, + removed: bool = False, + ) -> ReactionEvent: + """Record the user's emoji reaction on a message, or remove it. + + Without ``target_event_id`` it reacts to the agent's latest message. + The agent reacts through the ``react``/``unreact`` tools instead. + A reaction does not wake the agent; it sees it on its next turn, with + the end of the message's text. While tool calls wait for their results + (a pending confirmation, a pause) the reaction is held back, and joins + the events once every call is answered. + + Raises: + ValueError: The emoji is not allowed, or there is no such message. + """ + checked = check_reaction_emoji(emoji) + with self._state: + target = find_reaction_target( + self._state.active_branch(), target_event_id, source="user" + ) + ending, clipped = reaction_target_text(target) + event = ReactionEvent( + source="user", + target_event_id=target.id, + emoji=checked, + removed=removed, + target_text_ending=ending, + target_text_clipped=clipped, + ) + if ConversationState.get_unmatched_actions(self._state.active_branch()): + self._pending_reactions.append(event) + else: + self._on_event(event) + return event + def _on_event_with_state_lock(self, event: Event) -> None: """Emit an event while holding the conversation state lock.""" with self._state: @@ -3355,6 +3425,8 @@ def rerun_actions( action_id=event.id, ) rerun_log.append(obs_event) + for follow_up in observation.follow_up_events(): + rerun_log.append(follow_up) except Exception as e: logger.error( f"Action {action_count} ({tool_name}) failed during rerun: {e}" diff --git a/openhands-sdk/openhands/sdk/conversation/visualizer/default.py b/openhands-sdk/openhands/sdk/conversation/visualizer/default.py index a1317e72ac1..5b9f687cf09 100644 --- a/openhands-sdk/openhands/sdk/conversation/visualizer/default.py +++ b/openhands-sdk/openhands/sdk/conversation/visualizer/default.py @@ -21,6 +21,7 @@ MessageEvent, ObservationEvent, PauseEvent, + ReactionEvent, SystemPromptEvent, UserRejectObservation, ) @@ -242,6 +243,10 @@ def _get_message_color(event: Event) -> str: title="User Paused", color=_PAUSE_COLOR, ), + ReactionEvent: EventVisualizationConfig( + title="Reaction", + color=_SYSTEM_COLOR, + ), Condensation: EventVisualizationConfig( title="Condensation", color="white", diff --git a/openhands-sdk/openhands/sdk/event/__init__.py b/openhands-sdk/openhands/sdk/event/__init__.py index 4616dc7bc54..d2ce96abd02 100644 --- a/openhands-sdk/openhands/sdk/event/__init__.py +++ b/openhands-sdk/openhands/sdk/event/__init__.py @@ -18,6 +18,11 @@ SystemPromptEvent, UserRejectObservation, ) +from openhands.sdk.event.reaction import ( + REACTION_EMOJI, + ReactionEmoji, + ReactionEvent, +) from openhands.sdk.event.resume_transcript import ( RESUME_CONTEXT_MARKER, render_resume_transcript, @@ -44,6 +49,9 @@ "RejectionSource", "InterruptEvent", "PauseEvent", + "ReactionEvent", + "ReactionEmoji", + "REACTION_EMOJI", "StreamingDeltaEvent", "Condensation", "CondensationRequest", diff --git a/openhands-sdk/openhands/sdk/event/reaction.py b/openhands-sdk/openhands/sdk/event/reaction.py new file mode 100644 index 00000000000..7ae50fc9b2a --- /dev/null +++ b/openhands-sdk/openhands/sdk/event/reaction.py @@ -0,0 +1,196 @@ +"""Emoji reactions on messages.""" + +from collections.abc import Sequence +from typing import Final, Literal, get_args + +from pydantic import Field, field_validator +from rich.text import Text + +from openhands.sdk.event.base import Event, LLMConvertibleEvent +from openhands.sdk.event.llm_convertible import ActionEvent, MessageEvent +from openhands.sdk.event.types import EventID, SourceType +from openhands.sdk.llm import Message, TextContent + + +ReactionEmoji = Literal[ + "eyes", "+1", "white_check_mark", "tada", "thinking_face", "heart" +] +"""The emoji a reaction may use, as shortcodes.""" + +REACTION_EMOJI: Final[tuple[str, ...]] = get_args(ReactionEmoji) + +_GLYPHS: Final[dict[str, str]] = { + "eyes": "\N{EYES}", + "+1": "\N{THUMBS UP SIGN}", + "white_check_mark": "\N{WHITE HEAVY CHECK MARK}", + "tada": "\N{PARTY POPPER}", + "thinking_face": "\N{THINKING FACE}", + "heart": "\N{HEAVY BLACK HEART}\N{VARIATION SELECTOR-16}", +} + + +REACTION_TEXT_ENDING_MAX: Final[int] = 500 +"""How much of the end of a reacted message a user's reaction keeps.""" + +CLIPPED_MESSAGE_START: Final[str] = ( + "" +) + + +class ReactionEvent(LLMConvertibleEvent): + """An emoji reaction added to, or removed from, a message. + + The agent reacts with the ``react``/``unreact`` tools, and a user + through ``POST /api/conversations/{id}/reactions``. Clients fold these + events into each message's current set of reactions: the latest event for + a (message, emoji, source) wins. + + The agent sees the user's reactions on its next turn, as one line naming + the emoji and the end of the message (``target_text_ending``); a reaction + does not start a run. Its own reactions stay out of what it sees + (``View`` skips them): its ``react`` call is already there. A user's + reaction that arrives while tool calls wait for their results (for example + during a confirmation) is held back and becomes an event once those calls + are answered, so it never sits between a call and its result. + """ + + source: SourceType = "user" + target_event_id: EventID = Field( + description=( + "The message reacted to: a `MessageEvent`, or the agent's `finish` " + "`ActionEvent`." + ) + ) + emoji: ReactionEmoji = Field(description="The emoji, as a shortcode.") + removed: bool = Field( + default=False, description="True when this removes the reaction." + ) + target_text_ending: str | None = Field( + default=None, + description=( + "The end of the reacted message's text, at most " + f"{REACTION_TEXT_ENDING_MAX} characters, saved with the user's " + "reaction so the agent can tell which message it is about." + ), + ) + target_text_clipped: bool = Field( + default=False, + description="True when the message is longer than `target_text_ending`.", + ) + + @field_validator("source") + @classmethod + def _agent_or_user(cls, source: SourceType) -> SourceType: + if source not in ("agent", "user"): + raise ValueError("A reaction comes from the agent or the user.") + return source + + def to_llm_message(self) -> Message: + who = "The user" if self.source == "user" else "You" + which = "a message." if self.target_text_ending is None else "this message:" + if self.removed: + line = f"[{who} took back the {self.glyph} reaction on {which}]" + else: + line = f"[{who} reacted {self.glyph} to {which}]" + if self.target_text_ending is None: + text = line + elif self.target_text_clipped: + text = f"{line}\n{CLIPPED_MESSAGE_START}\n{self.target_text_ending}" + else: + text = f"{line}\n{self.target_text_ending}" + return Message(role="user", content=[TextContent(text=text)]) + + @property + def glyph(self) -> str: + return _GLYPHS[self.emoji] + + @property + def visualize(self) -> Text: + verb = "removed" if self.removed else "reacted" + return Text(f"{self.source} {verb} {self.glyph}", style="dim") + + def __str__(self) -> str: + verb = "removed" if self.removed else "reacted" + return ( + f"{self.__class__.__name__} ({self.source}): {verb} :{self.emoji}: " + f"on {self.target_event_id}" + ) + + +_FINISH_TOOL_NAME: Final[str] = "finish" + + +def is_reactable(event: Event) -> bool: + """A message, or the agent's ``finish`` (how an agent usually answers).""" + if isinstance(event, MessageEvent): + return True + return ( + isinstance(event, ActionEvent) + and event.source == "agent" + and event.tool_name == _FINISH_TOOL_NAME + ) + + +def reaction_target_text(event: Event) -> tuple[str, bool]: + """The end of a reactable message's text, and whether it was clipped.""" + from openhands.sdk.tool.builtins.finish import FinishAction + + if isinstance(event, MessageEvent): + text = "\n".join( + c.text for c in event.llm_message.content if isinstance(c, TextContent) + ) + elif isinstance(event, ActionEvent) and isinstance(event.action, FinishAction): + text = event.action.message + else: + text = "" + text = text.strip() + if len(text) <= REACTION_TEXT_ENDING_MAX: + return text, False + return text[-REACTION_TEXT_ENDING_MAX:], True + + +def find_reaction_target( + events: Sequence[Event], target_event_id: EventID | None, *, source: SourceType +) -> Event: + """The message a reaction from ``source`` is about. + + ``events`` is the conversation's active branch (``state.active_branch()``), + so a message on a branch the conversation navigated away from is never a + target. Without ``target_event_id``, the latest message from the other + side: the latest user message for the agent (the user's, or a + coordinator's), and for the user the agent's latest message or + ``finish``. + + Raises: + ValueError: There is no such message. + """ + if target_event_id is None: + wanted = "user" if source == "agent" else "agent" + for index in range(len(events) - 1, -1, -1): + event = events[index] + if is_reactable(event) and event.source == wanted: + return event + raise ValueError(f"There is no {wanted} message to react to.") + event = _find_event(events, target_event_id) + if event is None: + raise ValueError(f"No message with id {target_event_id}.") + if not is_reactable(event): + raise ValueError(f"Event {target_event_id} is not a message.") + return event + + +def _find_event(events: Sequence[Event], event_id: EventID) -> Event | None: + return next((e for e in reversed(events) if e.id == event_id), None) + + +def check_reaction_emoji(emoji: str) -> ReactionEmoji: + """Return ``emoji`` if reactions may use it. + + Raises: + ValueError: It is not one of :data:`REACTION_EMOJI`; the message lists + them. + """ + if emoji not in REACTION_EMOJI: + allowed = ", ".join(REACTION_EMOJI) + raise ValueError(f"Unknown emoji {emoji!r}; use one of: {allowed}.") + return emoji # type: ignore[return-value] diff --git a/openhands-sdk/openhands/sdk/launch/finalize.py b/openhands-sdk/openhands/sdk/launch/finalize.py index 1296ee2074e..e2a5087f14e 100644 --- a/openhands-sdk/openhands/sdk/launch/finalize.py +++ b/openhands-sdk/openhands/sdk/launch/finalize.py @@ -68,6 +68,7 @@ def finalize( additions: AgentLaunchAdditions | None = None, extra_suffixes: Sequence[str] = (), client_tools: Sequence[Tool] = (), + experimental_feature_tools: Sequence[Tool] = (), load_memory: bool = False, managed_secrets: Collection[str] = (), launched_at: datetime | None = None, @@ -78,6 +79,10 @@ def finalize( runtime can run it), ``current_datetime``, ``load_memory``, ACP skill sourcing, launch additions, client tools and credentials the runtime manages itself. + + ``experimental_feature_tools`` are tools the user turned on through a feature flag. + Only an OpenHands :class:`~openhands.sdk.agent.Agent` gets them: an ACP + agent runs its own tools, so it is left as it is. """ if isinstance(source, ResolvedLaunch): settings = _settings_for_runtime(source.settings, runtime) @@ -104,6 +109,8 @@ def finalize( agent = _without_context_secrets(agent, managed_secrets) if client_tools: agent = _with_client_tools(agent, client_tools) + if experimental_feature_tools and not isinstance(agent, ACPAgent): + agent = _with_client_tools(agent, experimental_feature_tools) return LaunchedAgent(agent=agent, profile=profile, _token=_FINALIZE_TOKEN) diff --git a/openhands-sdk/openhands/sdk/tool/schema.py b/openhands-sdk/openhands/sdk/tool/schema.py index 27841d725a7..deb8db4442e 100644 --- a/openhands-sdk/openhands/sdk/tool/schema.py +++ b/openhands-sdk/openhands/sdk/tool/schema.py @@ -18,6 +18,8 @@ if TYPE_CHECKING: from typing import Self + from openhands.sdk.event.base import Event + logger = get_logger(__name__) S = TypeVar("S", bound="Schema") @@ -399,6 +401,15 @@ class Observation(Schema, ABC): default=False, description="Whether the observation indicates an error" ) + def follow_up_events(self) -> "Sequence[Event]": + """Events to record right after this observation, in order. + + The agent emits them on its own thread after the tool returns, so a + tool can add a non-LLM event (such as a reaction) to the conversation + without touching the conversation from a worker thread. + """ + return () + @classmethod def from_text( cls, diff --git a/openhands-tools/openhands/tools/reaction/__init__.py b/openhands-tools/openhands/tools/reaction/__init__.py new file mode 100644 index 00000000000..91b5396bac1 --- /dev/null +++ b/openhands-tools/openhands/tools/reaction/__init__.py @@ -0,0 +1,33 @@ +"""Emoji reactions for agents. + +``react`` puts an emoji on a message (by default the user's latest one) and +``unreact`` takes it off. A reaction is a small visible sign that the agent saw +a message, for when the work comes before the answer. It is recorded as a +:class:`~openhands.sdk.event.ReactionEvent`, which clients show on the message +and the LLM never sees. + +Usage: + from openhands.tools.reaction import ReactTool, UnreactTool + + agent = Agent( + llm=llm, + tools=[Tool(name=ReactTool.name), Tool(name=UnreactTool.name)], + ) +""" + +from openhands.tools.reaction.definition import ( + ReactAction, + ReactionObservation, + ReactTool, + UnreactAction, + UnreactTool, +) + + +__all__ = [ + "ReactAction", + "ReactTool", + "ReactionObservation", + "UnreactAction", + "UnreactTool", +] diff --git a/openhands-tools/openhands/tools/reaction/definition.py b/openhands-tools/openhands/tools/reaction/definition.py new file mode 100644 index 00000000000..0248921ef8e --- /dev/null +++ b/openhands-tools/openhands/tools/reaction/definition.py @@ -0,0 +1,184 @@ +"""The ``react`` and ``unreact`` tools.""" + +from collections.abc import Sequence +from typing import TYPE_CHECKING, ClassVar, Self + +from pydantic import Field +from rich.text import Text + +from openhands.sdk.event import Event, EventID, ReactionEmoji, ReactionEvent +from openhands.sdk.tool.registry import register_tool +from openhands.sdk.tool.tool import ( + Action, + Observation, + ToolAnnotations, + ToolDefinition, +) + + +if TYPE_CHECKING: + from openhands.sdk.conversation.state import ConversationState + + +_MESSAGE_ID = Field( + default=None, + description=( + "Id of the message to react to. Leave it out to use the user's " + "latest message, which is almost always the one you mean." + ), +) + + +class ReactAction(Action): + """Put an emoji reaction on a message.""" + + emoji: ReactionEmoji = Field(description="The emoji, as a shortcode.") + message_id: EventID | None = _MESSAGE_ID + + @property + def visualize(self) -> Text: + return Text(f"React :{self.emoji}:", style="dim") + + +class UnreactAction(Action): + """Take an emoji reaction off a message.""" + + emoji: ReactionEmoji = Field(description="The emoji to remove, as a shortcode.") + message_id: EventID | None = _MESSAGE_ID + + @property + def visualize(self) -> Text: + return Text(f"Unreact :{self.emoji}:", style="dim") + + +class ReactionObservation(Observation): + """The result of ``react`` or ``unreact``. + + On success it carries the reaction, which the agent records as a + :class:`ReactionEvent` right after this observation. + """ + + target_event_id: EventID | None = None + emoji: ReactionEmoji | None = None + removed: bool = False + + def follow_up_events(self) -> Sequence[Event]: + if self.is_error or self.target_event_id is None or self.emoji is None: + return () + return ( + ReactionEvent( + source="agent", + target_event_id=self.target_event_id, + emoji=self.emoji, + removed=self.removed, + ), + ) + + @property + def visualize(self) -> Text: + if self.is_error: + return Text(self.text, style="red") + return Text() + + +_REACT_DESCRIPTION = """\ +Put an emoji reaction on a message; by default the user's latest one. +The user sees it on their message right away, while you keep working, so a +reaction is the smallest sign that you got a message and what you make of it. + +Use the reaction that fits. Examples: +- +1 when the user corrects you, changes the plan or adds to it while you + are working ("wait, actually...", "also...", "use X instead"): react +1 + first, then carry on with the change. No need to stop and reply. Also +1 a + note or go-ahead that needs no reply. +- eyes when a message needs work before you can answer it: react when you + start, and answer when you have the result. +- white_check_mark when you have done what a message asked and there is + nothing more to say about it. +- thinking_face when a message raises something you need to look into + before you can agree with it. +- tada when something the user cares about lands: a fix works, a release + is out, good news. +- heart when the user thanks you or shares something kind. + +These are only examples, you make your judgment when your user can benefit +from a more interactive and non-intrusive conversation as reactions make it. + +A reaction never replaces an answer: a question still gets a reply. React +when it tells the user something, not on every message. A reaction stays +on the message. Reacting does not end your turn.""" + +_UNREACT_DESCRIPTION = """\ +Take an emoji reaction you added off a message; by default the user's +latest one. Do this only when it no longer fits (for example eyes on work you +decided not to do) or when the user asks: a reaction is meant to stay.""" + + +def _annotations() -> ToolAnnotations: + return ToolAnnotations( + readOnlyHint=False, + destructiveHint=False, + # Each call records a new ReactionEvent. + idempotentHint=False, + openWorldHint=False, + ) + + +class ReactTool(ToolDefinition[ReactAction, ReactionObservation]): + """Tool that puts an emoji reaction on a message.""" + + # Given by the conversation_reactions feature, not picked in agent settings. + user_selectable: ClassVar[bool] = False + + @classmethod + def create( + cls, + conv_state: "ConversationState | None" = None, # noqa: ARG003 + **params, + ) -> Sequence[Self]: + if params: + raise ValueError(f"ReactTool does not accept parameters: {sorted(params)}") + from openhands.tools.reaction.impl import ReactionExecutor + + return [ + cls( + description=_REACT_DESCRIPTION, + action_type=ReactAction, + observation_type=ReactionObservation, + executor=ReactionExecutor(), + annotations=_annotations(), + ) + ] + + +class UnreactTool(ToolDefinition[UnreactAction, ReactionObservation]): + """Tool that takes an emoji reaction off a message.""" + + # Given by the conversation_reactions feature, not picked in agent settings. + user_selectable: ClassVar[bool] = False + + @classmethod + def create( + cls, + conv_state: "ConversationState | None" = None, # noqa: ARG003 + **params, + ) -> Sequence[Self]: + if params: + raise ValueError( + f"UnreactTool does not accept parameters: {sorted(params)}" + ) + from openhands.tools.reaction.impl import ReactionExecutor + + return [ + cls( + description=_UNREACT_DESCRIPTION, + action_type=UnreactAction, + observation_type=ReactionObservation, + executor=ReactionExecutor(), + annotations=_annotations(), + ) + ] + + +register_tool(ReactTool.name, ReactTool) +register_tool(UnreactTool.name, UnreactTool) diff --git a/openhands-tools/openhands/tools/reaction/impl.py b/openhands-tools/openhands/tools/reaction/impl.py new file mode 100644 index 00000000000..dea3a7c5341 --- /dev/null +++ b/openhands-tools/openhands/tools/reaction/impl.py @@ -0,0 +1,49 @@ +"""Executor for the ``react`` and ``unreact`` tools.""" + +from typing import TYPE_CHECKING + +from openhands.sdk.event.reaction import find_reaction_target +from openhands.sdk.tool.tool import ToolExecutor +from openhands.tools.reaction.definition import ( + ReactAction, + ReactionObservation, + UnreactAction, +) + + +if TYPE_CHECKING: + from openhands.sdk.conversation.impl.local_conversation import LocalConversation + + +class ReactionExecutor(ToolExecutor[ReactAction | UnreactAction, ReactionObservation]): + """Find the message and return the reaction. + + The executor only reads the conversation; the agent records the + :class:`ReactionEvent` from the observation on its own thread, so this is + safe when tools run in parallel. + """ + + def __call__( + self, + action: ReactAction | UnreactAction, + conversation: "LocalConversation | None" = None, + ) -> ReactionObservation: + if conversation is None: + return ReactionObservation.from_text( + "Reactions need a conversation.", is_error=True + ) + removed = isinstance(action, UnreactAction) + message_id = (action.message_id or "").strip() or None + try: + target = find_reaction_target( + conversation.state.active_branch(), message_id, source="agent" + ) + except ValueError as exc: + return ReactionObservation.from_text(str(exc), is_error=True) + verb = "Removed" if removed else "Reacted" + return ReactionObservation.from_text( + f"{verb} :{action.emoji}:.", + target_event_id=target.id, + emoji=action.emoji, + removed=removed, + ) diff --git a/tests/agent_server/docker_runtime/test_registry.py b/tests/agent_server/docker_runtime/test_registry.py index 0d095dbd860..f0892ad8cd9 100644 --- a/tests/agent_server/docker_runtime/test_registry.py +++ b/tests/agent_server/docker_runtime/test_registry.py @@ -20,6 +20,7 @@ DockerConversationRegistry, ) from openhands.agent_server.models import StartConversationRequest +from openhands.agent_server.persistence import PersistedSettings, get_settings_store from openhands.agent_server.storage import trash as trash_module from openhands.sdk import LLM, Agent, Message, TextContent from openhands.sdk.conversation.request import ConversationContainerOptions @@ -650,6 +651,23 @@ def test_container_command_is_hardened_and_mounts_only_its_state(tmp_path, monke assert env["OH_SECRET_KEY"] != "outer-key" +def test_container_gets_the_hosts_experimental_feature_switches(tmp_path, monkeypatch): + runtime = registry(tmp_path, monkeypatch) + + def turn_on(settings: PersistedSettings) -> PersistedSettings: + settings.experimental_features = {"conversation_reactions": True} + return settings + + get_settings_store(runtime.config).update(turn_on) + + _, command, env = _build_container(runtime, monkeypatch, uuid4()) + + assert json.loads(env["OH_RUNTIME_EXPERIMENTAL_FEATURES"]) == { + "conversation_reactions": True + } + assert "OH_RUNTIME_EXPERIMENTAL_FEATURES" in command + + def test_worktree_root_resolves_identically_on_host_and_in_container( tmp_path, monkeypatch ): diff --git a/tests/agent_server/docker_runtime/test_routes.py b/tests/agent_server/docker_runtime/test_routes.py index f207401cb2c..1dc89106f88 100644 --- a/tests/agent_server/docker_runtime/test_routes.py +++ b/tests/agent_server/docker_runtime/test_routes.py @@ -28,6 +28,7 @@ ) from openhands.agent_server.event_router import event_read_router from openhands.agent_server.models import UpdateSecretsRequest +from openhands.agent_server.persistence import PersistedSettings, get_settings_store from openhands.sdk.conversation.request import ConversationContainerOptions from openhands.sdk.git.utils import run_git_command from openhands.sdk.profiles.agent_profile import LaunchedAgentProfile @@ -849,3 +850,52 @@ def test_invalid_container_options_are_rejected(tmp_path, monkeypatch): assert response.status_code == 422 assert registry.provisioning.load_optional(conversation_id) is None + + +@pytest.mark.parametrize("enabled", [False, True]) +def test_reactions_follow_the_host_switch_before_reaching_the_runtime( + tmp_path, monkeypatch, enabled +): + monkeypatch.setenv("OH_PERSISTENCE_DIR", str(tmp_path / "persistence")) + config = Config( + conversations_path=tmp_path / "conversations", + secret_key=SecretStr("outer-key"), + ) + + def apply(settings: PersistedSettings) -> PersistedSettings: + settings.experimental_features = {"conversation_reactions": enabled} + return settings + + get_settings_store(config).update(apply) + app = FastAPI() + app.state.conversation_registry = DockerConversationRegistry(config) + app.state.conversation_service = AsyncMock() + app.include_router(docker_conversation_router, prefix="/api") + conversation_id = uuid4() + forwarded = [] + + async def container(*_args): + return SimpleNamespace(host="http://inner", api_key="inner-key") + + async def proxy(*_args, **kwargs): + forwarded.append(kwargs["upstream_path"]) + return Response() + + monkeypatch.setattr( + "openhands.agent_server.docker_runtime.routers._container", container + ) + monkeypatch.setattr( + "openhands.agent_server.docker_runtime.routers.proxy_http", proxy + ) + + with TestClient(app) as client: + response = client.post( + f"/api/conversations/{conversation_id}/reactions", json={"emoji": "+1"} + ) + + if enabled: + assert response.status_code == 200 + assert forwarded == [f"/api/conversations/{conversation_id}/reactions"] + else: + assert response.status_code == 409 + assert forwarded == [] diff --git a/tests/agent_server/test_experimental_features_router.py b/tests/agent_server/test_experimental_features_router.py index 950bc23020d..564c444dd32 100644 --- a/tests/agent_server/test_experimental_features_router.py +++ b/tests/agent_server/test_experimental_features_router.py @@ -9,13 +9,22 @@ from openhands.agent_server import experimental_features from openhands.agent_server.config import Config +from openhands.agent_server.conversation_service import ConversationService from openhands.agent_server.experimental_features import ( + RUNTIME_EXPERIMENTAL_FEATURES_ENV, ExperimentalFeatureDefinition, + experimental_feature_tools, + host_experimental_features, ) from openhands.agent_server.experimental_features_router import ( experimental_features_router, ) +from openhands.agent_server.models import StartConversationRequest from openhands.agent_server.persistence import PersistedSettings, get_settings_store +from openhands.sdk import LLM, Agent +from openhands.sdk.agent import ACPAgent +from openhands.sdk.launch.finalize import LaunchRuntime, finalize +from openhands.sdk.workspace import LocalWorkspace QA_FEATURE = ExperimentalFeatureDefinition( @@ -152,7 +161,109 @@ def test_settings_saved_before_experimental_features_load_with_none_on(): @pytest.mark.asyncio -async def test_no_features_are_registered_yet(client): +async def test_conversation_reactions_is_listed_and_off(client): response = await client.get("/api/settings/experimental") - assert response.json() == [] + assert [(f["key"], f["enabled"]) for f in response.json()] == [ + ("conversation_reactions", False) + ] + + +def test_feature_tools_follow_the_flags(): + assert experimental_feature_tools({}) == [] + assert [ + t.name for t in experimental_feature_tools({"conversation_reactions": True}) + ] == [ + "react", + "unreact", + ] + + +def _runtime() -> LaunchRuntime: + return LaunchRuntime(acp_skill_sourcing="native", browser_available=False) + + +def test_finalize_adds_feature_tools_to_openhands_agents_only(): + tools = experimental_feature_tools({"conversation_reactions": True}) + agent = Agent(llm=LLM(model="gpt-4o", usage_id="test-llm"), tools=[]) + acp = ACPAgent(acp_command=["echo"]) + + launched = finalize(agent, _runtime(), experimental_feature_tools=tools) + launched_acp = finalize(acp, _runtime(), experimental_feature_tools=tools) + + assert [t.name for t in launched.agent.tools] == ["react", "unreact"] + assert launched_acp.agent.tools == acp.tools + + +@pytest.mark.asyncio +async def test_new_conversations_get_reaction_tools_only_while_on(tmp_path): + workspace = tmp_path / "workspace" + workspace.mkdir() + + def request() -> StartConversationRequest: + return StartConversationRequest( + agent=Agent(llm=LLM(model="gpt-4o", usage_id="test-llm"), tools=[]), + workspace=LocalWorkspace(working_dir=str(workspace)), + ) + + async with ConversationService( + conversations_dir=tmp_path / "conversations" + ) as service: + off, _ = await service.start_conversation(request()) + + def turn_on(settings: PersistedSettings) -> PersistedSettings: + settings.experimental_features = {"conversation_reactions": True} + return settings + + get_settings_store().update(turn_on) + on, _ = await service.start_conversation(request()) + + tool_names = {} + for info in (off, on): + event_service = await service.get_event_service(info.id) + assert event_service is not None + agent = event_service.get_conversation().agent + tool_names[info.id] = [t.name for t in agent.tools] + + assert tool_names[off.id] == [] + assert tool_names[on.id] == ["react", "unreact"] + + +def test_host_switches_are_read_only_inside_a_runtime(monkeypatch): + monkeypatch.delenv(RUNTIME_EXPERIMENTAL_FEATURES_ENV, raising=False) + assert host_experimental_features() is None + + monkeypatch.setenv( + RUNTIME_EXPERIMENTAL_FEATURES_ENV, + json.dumps({"conversation_reactions": True, "unknown": True}), + ) + assert host_experimental_features() == {"conversation_reactions": True} + + for broken in ("", "not json", "[1]", '{"conversation_reactions": "yes"}'): + monkeypatch.setenv(RUNTIME_EXPERIMENTAL_FEATURES_ENV, broken) + assert host_experimental_features() == {} + + +@pytest.mark.asyncio +async def test_a_runtime_takes_its_tools_from_the_host_switches(tmp_path, monkeypatch): + # The runtime container's own settings are empty (flag off); the host + # passes the user's switches in the environment. + monkeypatch.setenv( + RUNTIME_EXPERIMENTAL_FEATURES_ENV, json.dumps({"conversation_reactions": True}) + ) + workspace = tmp_path / "workspace" + workspace.mkdir() + async with ConversationService( + conversations_dir=tmp_path / "conversations" + ) as service: + info, _ = await service.start_conversation( + StartConversationRequest( + agent=Agent(llm=LLM(model="gpt-4o", usage_id="test-llm"), tools=[]), + workspace=LocalWorkspace(working_dir=str(workspace)), + ) + ) + event_service = await service.get_event_service(info.id) + assert event_service is not None + agent = event_service.get_conversation().agent + + assert [t.name for t in agent.tools] == ["react", "unreact"] diff --git a/tests/agent_server/test_reactions_router.py b/tests/agent_server/test_reactions_router.py new file mode 100644 index 00000000000..ba51bdc6d7c --- /dev/null +++ b/tests/agent_server/test_reactions_router.py @@ -0,0 +1,153 @@ +"""Tests for POST /api/conversations/{id}/reactions.""" + +from collections.abc import AsyncIterator +from uuid import UUID + +import httpx +import pytest +from fastapi import FastAPI + +from openhands.agent_server.config import Config +from openhands.agent_server.conversation_service import ConversationService +from openhands.agent_server.dependencies import get_conversation_service +from openhands.agent_server.event_router import event_router +from openhands.agent_server.experimental_features import ( + RUNTIME_EXPERIMENTAL_FEATURES_ENV, +) +from openhands.agent_server.models import StartConversationRequest +from openhands.agent_server.persistence import PersistedSettings, get_settings_store +from openhands.sdk import LLM, Agent, Message, TextContent +from openhands.sdk.event import MessageEvent, ReactionEvent +from openhands.sdk.workspace import LocalWorkspace + + +@pytest.fixture +async def service(tmp_path) -> AsyncIterator[ConversationService]: + async with ConversationService( + conversations_dir=tmp_path / "conversations" + ) as service: + yield service + + +def _set_reactions(enabled: bool) -> None: + def apply(settings: PersistedSettings) -> PersistedSettings: + settings.experimental_features = {"conversation_reactions": enabled} + return settings + + get_settings_store().update(apply) + + +@pytest.fixture(autouse=True) +def reactions_on(): + _set_reactions(True) + + +@pytest.fixture +async def client(service) -> AsyncIterator[httpx.AsyncClient]: + app = FastAPI() + app.state.config = Config( + static_files_path=None, session_api_keys=[], secret_key=None + ) + app.include_router(event_router, prefix="/api") + app.dependency_overrides[get_conversation_service] = lambda: service + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://test") as c: + yield c + + +async def _conversation_with_answer(service, tmp_path) -> tuple[UUID, MessageEvent]: + workspace = tmp_path / "workspace" + workspace.mkdir(exist_ok=True) + info, _ = await service.start_conversation( + StartConversationRequest( + agent=Agent(llm=LLM(model="gpt-4o", usage_id="test-llm"), tools=[]), + workspace=LocalWorkspace(working_dir=str(workspace)), + ) + ) + event_service = await service.get_event_service(info.id) + assert event_service is not None + answer = MessageEvent( + source="agent", + llm_message=Message(role="assistant", content=[TextContent(text="Done.")]), + ) + conversation = event_service.get_conversation() + with conversation.state: + conversation._on_event(answer) + return info.id, answer + + +async def _reactions(service, cid: UUID) -> list[ReactionEvent]: + event_service = await service.get_event_service(cid) + assert event_service is not None + state = await event_service.get_state() + return [e for e in state.events if isinstance(e, ReactionEvent)] + + +@pytest.mark.asyncio +async def test_user_reacts_and_unreacts(client, service, tmp_path): + cid, answer = await _conversation_with_answer(service, tmp_path) + + added = await client.post( + f"/api/conversations/{cid}/reactions", json={"emoji": "+1"} + ) + removed = await client.post( + f"/api/conversations/{cid}/reactions", + json={"emoji": "+1", "target_event_id": answer.id, "removed": True}, + ) + + assert added.status_code == 200, added.text + assert added.json()["kind"] == "ReactionEvent" + assert added.json()["target_event_id"] == answer.id + assert added.json()["source"] == "user" + assert removed.json()["removed"] is True + saved = await _reactions(service, cid) + assert [(e.emoji, e.removed) for e in saved] == [("+1", False), ("+1", True)] + + +@pytest.mark.asyncio +async def test_bad_reactions_are_refused(client, service, tmp_path): + cid, _ = await _conversation_with_answer(service, tmp_path) + + unknown_emoji = await client.post( + f"/api/conversations/{cid}/reactions", json={"emoji": "rocket"} + ) + unknown_message = await client.post( + f"/api/conversations/{cid}/reactions", + json={"emoji": "eyes", "target_event_id": "missing"}, + ) + + assert unknown_emoji.status_code == 422 + assert unknown_message.status_code == 400 + assert await _reactions(service, cid) == [] + + +@pytest.mark.asyncio +async def test_reactions_are_refused_while_the_feature_is_off( + client, service, tmp_path +): + cid, _ = await _conversation_with_answer(service, tmp_path) + _set_reactions(False) + + response = await client.post( + f"/api/conversations/{cid}/reactions", json={"emoji": "+1"} + ) + + assert response.status_code == 409 + assert await _reactions(service, cid) == [] + + +@pytest.mark.asyncio +async def test_a_docker_runtime_leaves_the_switch_to_its_host( + client, service, tmp_path, monkeypatch +): + # In a runtime container the host checks the switch before forwarding; + # the container's own settings are empty. + monkeypatch.setenv(RUNTIME_EXPERIMENTAL_FEATURES_ENV, "{}") + cid, _ = await _conversation_with_answer(service, tmp_path) + _set_reactions(False) + + response = await client.post( + f"/api/conversations/{cid}/reactions", json={"emoji": "+1"} + ) + + assert response.status_code == 200, response.text diff --git a/tests/tools/reaction/test_reaction_tools.py b/tests/tools/reaction/test_reaction_tools.py new file mode 100644 index 00000000000..de40d499dfd --- /dev/null +++ b/tests/tools/reaction/test_reaction_tools.py @@ -0,0 +1,411 @@ +"""Tests for emoji reactions: the event, the tools and LocalConversation.react.""" + +import json +from unittest.mock import patch + +import pytest +from litellm import ChatCompletionMessageToolCall +from litellm.types.utils import ( + Choices, + Function, + Message as LiteLLMMessage, + ModelResponse, +) +from pydantic import SecretStr + +from openhands.sdk.agent import Agent +from openhands.sdk.conversation import Conversation +from openhands.sdk.event import ( + REACTION_EMOJI, + ActionEvent, + Event, + MessageEvent, + ObservationEvent, + ReactionEvent, +) +from openhands.sdk.event.reaction import ( + CLIPPED_MESSAGE_START, + REACTION_TEXT_ENDING_MAX, + check_reaction_emoji, + find_reaction_target, +) +from openhands.sdk.llm import LLM, Message, MessageToolCall, TextContent +from openhands.sdk.tool import Tool +from openhands.sdk.tool.builtins.finish import FinishAction +from openhands.tools.reaction import ( + ReactAction, + ReactionObservation, + ReactTool, + UnreactAction, + UnreactTool, +) +from openhands.tools.reaction.impl import ReactionExecutor + + +def _llm() -> LLM: + return LLM( + usage_id="test-llm", + model="test-model", + api_key=SecretStr("test-key"), + base_url="http://test", + ) + + +def _message(source: str, text: str) -> MessageEvent: + return MessageEvent( + source=source, # type: ignore[arg-type] + llm_message=Message( + role="user" if source == "user" else "assistant", + content=[TextContent(text=text)], + ), + ) + + +# The event and its helpers + + +def test_emoji_are_checked(): + assert check_reaction_emoji("eyes") == "eyes" + with pytest.raises(ValueError, match="use one of: eyes"): + check_reaction_emoji("rocket") + assert "+1" in REACTION_EMOJI + + +def test_default_target_is_the_latest_message_from_the_other_side(): + first, answer, second = ( + _message("user", "a"), + _message("agent", "b"), + _message("user", "c"), + ) + events = [first, answer, second] + + assert find_reaction_target(events, None, source="agent") is second + assert find_reaction_target(events, None, source="user") is answer + assert find_reaction_target(events, first.id, source="agent") is first + + +def test_the_agents_finish_counts_as_its_answer(): + question = _message("user", "fix it") + finish = ActionEvent( + source="agent", + thought=[TextContent(text="")], + action=FinishAction(message="Fixed."), + tool_name="finish", + tool_call_id="finish-1", + tool_call=MessageToolCall( + id="finish-1", + name="finish", + arguments='{"message": "Fixed."}', + origin="completion", + ), + llm_response_id="resp-1", + ) + + assert find_reaction_target([question, finish], None, source="user") is finish + assert find_reaction_target([question, finish], finish.id, source="user") is finish + + +def test_target_must_be_a_message(): + other = ReactionEvent(target_event_id="x", emoji="eyes") + events: list[Event] = [_message("agent", "hi"), other] + + with pytest.raises(ValueError, match="not a message"): + find_reaction_target(events, other.id, source="user") + with pytest.raises(ValueError, match="No message"): + find_reaction_target(events, "missing", source="user") + with pytest.raises(ValueError, match="no user message"): + find_reaction_target(events, None, source="agent") + + +def test_reaction_round_trips_as_an_event(): + event = ReactionEvent(source="agent", target_event_id="m1", emoji="+1") + loaded = Event.model_validate(json.loads(event.model_dump_json())) + assert isinstance(loaded, ReactionEvent) + assert (loaded.emoji, loaded.removed, loaded.source) == ("+1", False, "agent") + + +def test_only_the_agent_and_the_user_react(): + with pytest.raises(ValueError, match="agent or the user"): + ReactionEvent(source="environment", target_event_id="m1", emoji="eyes") + + +def test_observations_have_no_follow_up_events_by_default(): + assert ReactionObservation.from_text("x", is_error=True).follow_up_events() == () + + +# The tools + + +@pytest.fixture +def conversation(tmp_path): + agent = Agent(llm=_llm(), tools=[Tool(name="react"), Tool(name="unreact")]) + conv = Conversation(agent=agent, workspace=str(tmp_path), callbacks=[]) + conv.send_message("Please look into the flaky test") + return conv + + +def _user_message(conversation) -> MessageEvent: + return next( + e + for e in conversation.state.events + if isinstance(e, MessageEvent) and e.source == "user" + ) + + +def test_executor_targets_the_latest_user_message(conversation): + observation = ReactionExecutor()(ReactAction(emoji="eyes"), conversation) + + assert not observation.is_error + (event,) = observation.follow_up_events() + assert isinstance(event, ReactionEvent) + assert event.target_event_id == _user_message(conversation).id + assert (event.source, event.emoji, event.removed) == ("agent", "eyes", False) + + +def test_unreact_records_a_removal(conversation): + target = _user_message(conversation) + + observation = ReactionExecutor()( + UnreactAction(emoji="eyes", message_id=target.id), conversation + ) + + (event,) = observation.follow_up_events() + assert isinstance(event, ReactionEvent) + assert event.removed and event.target_event_id == target.id + + +def test_unknown_message_is_an_error_without_a_reaction(conversation): + observation = ReactionExecutor()( + ReactAction(emoji="eyes", message_id="nope"), conversation + ) + + assert observation.is_error + assert observation.follow_up_events() == () + + +def _react_call() -> ModelResponse: + return ModelResponse( + id="resp-1", + choices=[ + Choices( + index=0, + message=LiteLLMMessage( + role="assistant", + content="", + tool_calls=[ + ChatCompletionMessageToolCall( + id="call-1", + type="function", + function=Function( + name="react", arguments='{"emoji": "eyes"}' + ), + ) + ], + ), + finish_reason="tool_calls", + ) + ], + created=0, + model="test-model", + object="chat.completion", + ) + + +def test_agent_records_the_reaction_after_the_observation(conversation): + emitted: list[Event] = [] + + with patch( + "openhands.sdk.llm.llm.litellm_completion", + side_effect=lambda messages, **kw: _react_call(), + ): + conversation.agent.step(conversation, on_event=emitted.append) + + kinds = [type(e).__name__ for e in emitted] + assert kinds[-2:] == ["ObservationEvent", "ReactionEvent"] + observation, reaction = emitted[-2:] + assert isinstance(observation, ObservationEvent) + assert isinstance(reaction, ReactionEvent) + assert reaction.target_event_id == _user_message(conversation).id + # The LLM sees the short observation, never the reaction event. + assert "Reacted :eyes:" in observation.to_llm_message().content[0].text # type: ignore[union-attr] + + +# LocalConversation.react, the user's side + + +def test_user_reacts_to_the_agents_latest_message(conversation): + with pytest.raises(ValueError, match="no agent message"): + conversation.react("+1") + + conversation.state.append_event(_message("agent", "Fixed it.")) + event = conversation.react("+1") + removal = conversation.react("+1", event.target_event_id, removed=True) + + assert event.source == "user" and not event.removed + assert removal.removed and removal.target_event_id == event.target_event_id + saved = [e for e in conversation.state.events if isinstance(e, ReactionEvent)] + assert [e.id for e in saved] == [event.id, removal.id] + + +def test_user_cannot_use_an_unknown_emoji(conversation): + with pytest.raises(ValueError, match="Unknown emoji"): + conversation.react("rocket") + + +def test_messages_off_the_active_branch_are_not_targets(conversation): + question = _user_message(conversation) + answer = _message("agent", "Old answer.") + conversation.state.append_event(answer) + + conversation.navigate_to(question.id) + + with pytest.raises(ValueError, match="no agent message"): + conversation.react("+1") + with pytest.raises(ValueError, match="No message"): + conversation.react("+1", answer.id) + + +def test_execute_tool_returns_the_reaction_without_recording_it(conversation): + observation = conversation.execute_tool("react", ReactAction(emoji="tada")) + + assert not observation.is_error + (follow_up,) = observation.follow_up_events() + assert isinstance(follow_up, ReactionEvent) and follow_up.emoji == "tada" + assert not [e for e in conversation.state.events if isinstance(e, ReactionEvent)] + + +def test_reaction_tools_are_not_offered_in_agent_settings(): + assert ReactTool.user_selectable is False + assert UnreactTool.user_selectable is False + + +# What the agent sees of reactions + + +def _messages_on_next_step(conversation) -> list[dict]: + """The messages the agent sends the LLM on its next step.""" + seen: list[list[dict]] = [] + + def complete(messages, **kw): + seen.append(messages) + return _react_call() + + with patch("openhands.sdk.llm.llm.litellm_completion", side_effect=complete): + conversation.agent.step(conversation, on_event=lambda e: None) + return seen[0] + + +def _texts(messages: list[dict]) -> list[str]: + texts = [] + for message in messages: + content = message.get("content") + if isinstance(content, str): + texts.append(content) + elif isinstance(content, list): + texts.extend(c.get("text", "") for c in content if isinstance(c, dict)) + return texts + + +def test_the_agent_sees_the_users_reaction_on_its_next_turn(conversation): + question = "Shall I delete the generated fixtures?" + conversation.state.append_event(_message("agent", question)) + status = conversation.state.execution_status + + conversation.react("+1") + + # Reacting saves the event and starts nothing. + assert conversation.state.execution_status == status + texts = _texts(_messages_on_next_step(conversation)) + assert f"[The user reacted 👍 to this message:]\n{question}" in texts + + +def test_a_reaction_during_a_pending_tool_call_waits_for_its_result(conversation): + """LLM APIs reject anything between a tool call and its result.""" + conversation.state.append_event(_message("agent", "Shall I run it?")) + emitted: list[Event] = [] + with patch( + "openhands.sdk.llm.llm.litellm_completion", + side_effect=lambda messages, **kw: _react_call(), + ): + conversation.agent.step(conversation, on_event=emitted.append) + call = next(e for e in emitted if isinstance(e, ActionEvent)) + result = next(e for e in emitted if isinstance(e, ObservationEvent)) + + def kinds(events) -> list[str]: + return [type(e).__name__ for e in events] + + # The call waits (a confirmation, a pause); the user reacts meanwhile. + conversation._on_event(call) + held = conversation.react("heart") + conversation.react("heart", removed=True) + assert kinds(conversation.state.events)[-1] == "ActionEvent" + + # The result arrives: the held reactions follow it, in order. + conversation._on_event(result) + + assert kinds(conversation.state.events)[-4:] == [ + "ActionEvent", + "ObservationEvent", + "ReactionEvent", + "ReactionEvent", + ] + first, second = conversation.state.events[-2:] + assert (first.id, first.removed, second.removed) == (held.id, False, True) + assert first.timestamp >= conversation.state.events[-3].timestamp + assert ( + kinds(conversation.state.view.events)[-4:] + == kinds(conversation.state.events)[-4:] + ) + roles = [m["role"] for m in _messages_on_next_step(conversation)] + assert roles[-3:] == ["assistant", "tool", "user"] + + # Nothing is open any more: the next reaction is an event at once. + conversation.react("tada") + assert kinds(conversation.state.events)[-1] == "ReactionEvent" + + +def test_a_long_message_is_clipped_to_its_ending(conversation): + long_answer = "Start of a long plan. " + "step " * 200 + "Shall I go ahead?" + conversation.state.append_event(_message("agent", long_answer)) + + event = conversation.react("+1") + + assert event.target_text_clipped + assert event.target_text_ending == long_answer[-REACTION_TEXT_ENDING_MAX:] + (content,) = event.to_llm_message().content + assert isinstance(content, TextContent) + assert content.text == ( + "[The user reacted 👍 to this message:]\n" + f"{CLIPPED_MESSAGE_START}\n{long_answer[-REACTION_TEXT_ENDING_MAX:]}" + ) + + +def test_removal_and_older_reactions_read_as_one_line(): + removal = ReactionEvent( + target_event_id="m1", + emoji="heart", + removed=True, + target_text_ending="Done.", + ) + older = ReactionEvent(target_event_id="m1", emoji="tada") + + assert removal.to_llm_message().content[0].text == ( # type: ignore[union-attr] + "[The user took back the ❤️ reaction on this message:]\nDone." + ) + assert older.to_llm_message().content[0].text == ( # type: ignore[union-attr] + "[The user reacted 🎉 to a message.]" + ) + + +def test_the_agents_own_reaction_is_not_repeated_to_it(conversation): + with patch( + "openhands.sdk.llm.llm.litellm_completion", + side_effect=lambda messages, **kw: _react_call(), + ): + conversation.agent.step(conversation, on_event=conversation._on_event) + assert any(isinstance(e, ReactionEvent) for e in conversation.state.events) + + texts = _texts(_messages_on_next_step(conversation)) + + assert not any("You reacted" in text for text in texts) + assert any("Reacted :eyes:" in text for text in texts)