Add OpenAI-compatible computer use tool support

[maralbahari]

Problem statement / motivation

The Server-Side Tool Execution roadmap lists Computer use as an expected tool area, but agentic-api does not currently have first-class support for the OpenAI Responses API computer tool.

The Rust request model has no typed computer tool variant, and the execution path has no representation for computer_call, computer_call_output, screenshots, or computer actions. As a result, clients cannot use the current OpenAI-compatible computer-use loop through agentic-api, and deployments cannot register a gateway-executed browser or VM harness.

OpenAI's current computer-use flow is:

1. Send a Responses request with tools: [{"type": "computer"}].
2. Receive a computer_call containing an ordered actions[] batch.
3. Execute those actions in a browser or VM.
4. Capture the updated screen and submit it as computer_call_output.
5. Continue until the model no longer emits a computer_call.

This fits the roadmap's tool loop and ownership model. A configured computer harness can be a gateway-executed built-in tool; without a configured handler, the gateway must not execute UI actions by default.

Proposed solution

Add first-class, OpenAI-compatible support for the current computer built-in tool.

Wire types

• Accept and preserve the request declaration {"type": "computer"}.
• Add typed input/output items for:
  • computer_call
  • computer_call_output
  • computer_screenshot
• Represent the current action set:
  • click
  • double_click
  • scroll
  • type
  • wait
  • keypress
  • drag
  • move
  • screenshot
• Keep the action representation non-exhaustive so future action types can be preserved safely.
• Preserve call IDs, action ordering, statuses, screenshots, and image detail across storage and continuation.

Tool routing and execution

• Register computer in the request-scoped tool registry.
• If upstream inference requires function-shaped tools, normalize the declaration for inference and restore model calls to the public typed computer_call shape.
• Execute computer calls only when an explicit gateway computer handler is configured.
• Run every action in actions[] in order, capture a full screenshot after the batch, append computer_call_output, and continue the inference loop.
• When no gateway handler is configured, preserve/return the call for client execution or pass it through according to the existing ownership rules. Never control the gateway host UI by default.
• Support both blocking and streaming Responses flows, including continuation through previous_response_id and stored response state.

Safety and configuration

• Require an isolated browser, container, or VM for gateway execution.
• Do not inherit host environment variables or expose the host filesystem by default.
• Support URL/domain allowlists, timeouts, action/round limits, cancellation, and audit logging.
• Treat screenshots, pages, documents, emails, chats, and other third-party content as untrusted input.
• Provide a human-review/approval hook for high-impact actions and sensitive-data transmission.

Acceptance criteria

• {"type": "computer"} round-trips through request parsing, storage, rehydration, and serialization.
• All documented computer action variants deserialize and serialize without losing fields or order.
• computer_call and computer_call_output are supported in blocking and streaming Responses flows.
• A configured test harness can complete a multi-round action/screenshot loop and produce a final assistant response.
• previous_response_id continuation preserves the computer call and its output.
• A request without a configured gateway handler never executes UI actions implicitly.
• Timeouts, cancellation, malformed actions, handler failures, and maximum-loop exhaustion return typed, test-covered failures.
• Documentation explains handler registration, isolation, approvals, and the client-executed fallback.
• Compatibility tests cover the OpenAI request and output shapes below.

Alternatives considered

No response

Additional context

OpenAI API examples

The model name below follows the current OpenAI documentation; a vLLM deployment would use a model capable of emitting the compatible tool-call shape.

Start a computer-use request

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")

response = client.responses.create(
    model="gpt-5.6",
    tools=[{"type": "computer"}],
    input=(
        "Check whether the Filters panel is open. If it is not, click Show filters. "
        "Then type penguin in the search box. Use the computer tool for UI interaction."
    ),
)

The first output may request a screenshot:

{
  "type": "computer_call",
  "call_id": "call_001",
  "actions": [
    { "type": "screenshot" }
  ],
  "status": "completed"
}

Return the updated screenshot

response = client.responses.create(
    model="gpt-5.6",
    tools=[{"type": "computer"}],
    previous_response_id=response.id,
    input=[
        {
            "type": "computer_call_output",
            "call_id": "call_001",
            "output": {
                "type": "computer_screenshot",
                "image_url": f"data:image/png;base64,{screenshot_base64}",
                "detail": "original",
            },
        }
    ],
)

[Triumph1701]

The typed computer-call loop is a good fit for a gateway, but the ownership boundary should stay explicit: execute actions only when an isolated handler is configured, otherwise preserve the call for client-side execution. Carry call IDs, action order, screenshots, continuation state, and an audit record across retries so a resumed tool loop cannot be mistaken for a new user turn.