Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -1565,7 +1565,7 @@ Frame sources feed images to the agent, and a pluggable annotator turns each fra

- The built-in `screen` source is computer-use screenshot streaming, registered whenever screenshot streaming is on.
- **Directory sources** watch a folder and serve the newest image file by modification time - whatever your camera or capture process writes there.
- `GetLatestFrame` reads the current frame from a source; `ImageDecode` handles arbitrary images.
- `GetLatestFrame` reads the current frame from a source. For an arbitrary image file on disk, a vision model uses [`Read`](#read) and a text-only model uses `ImageDecode`.

The annotator produces a scene summary plus a numbered element list with bounding boxes: a vision model gets the image, a text-only model gets the text. Annotation is a side-call through the gateway, so any vision model the gateway serves works - including a fully local one through Ollama, which keeps annotation offline.

Expand All @@ -1587,7 +1587,7 @@ vision:
max_age: 24h
```

With `vision.annotator.enabled: true`, `GetLatestFrame` defaults to annotated text output and `ImageDecode` adds a text description to the image it returns - no per-model configuration. `ImageDecode` itself is always available; vision-capable models receive the image directly (see [Vision-capable models and ImageDecode](#vision-capable-models-and-imagedecode)). `retention` prunes the source directory after reads.
With `vision.annotator.enabled: true`, `GetLatestFrame` defaults to annotated text output and `ImageDecode` adds a text description to the image it returns - no per-model configuration. `ImageDecode` itself is always available - it is the path for a text-only model, and for a text description on top of an image a vision model can already see (see [Vision-capable models and ImageDecode](#vision-capable-models-and-imagedecode)). `retention` prunes the source directory after reads.

Not to be confused with the gateway-side `gateway.vision_enabled` flag, which is unrelated.

Expand Down Expand Up @@ -1618,11 +1618,23 @@ In [Plan Mode](#plan-mode), `Write`, `Edit`, `MultiEdit` and `Delete` execute on

#### Read

Read a file from the local filesystem with an optional line range. Handles text files and PDFs.
Read a file from the local filesystem with an optional line range. Handles text files, PDFs and images.

- **Parameters**: `file_path` (required, absolute or relative), `limit` (default 2000 lines), `offset` (default 1)
- **Approval**: not required (read-only)
- **Notes**: lines longer than 2000 characters are truncated; output is returned in `cat -n` format
- **Notes**: lines longer than 2000 characters are truncated - output is returned in `cat -n` format

An image file - `png`, `jpg`, `jpeg`, `gif` or `webp`, or a file without an extension whose bytes are an image - is **attached to the model's next message** when the gateway lists image input among the model's modalities. The tool result carries the path and the pixel size:

```text
Read image 1920x1080 screenshot.png
```

A text-only model, or a session with no image service, gets an error pointing at [ImageDecode](#vision-capable-models-and-imagedecode) instead, which describes the image in words through the [vision annotator](#frame-sources-and-vision-annotation):

```text
This is an image and the model cannot see images. Use ImageDecode on the path to get a description of it.
```

A missing file does not return a bare `NOT_FOUND`. The error appends up to five candidate paths so the next call can correct a typo or a wrong directory instead of retrying the same path: siblings of the nearest existing directory ranked by name similarity, plus - when no sibling carries the missing base name - files with that base name found under the [allowed sandbox directory](#file-sandbox) containing it.

Expand Down Expand Up @@ -1967,6 +1979,8 @@ Vision-capable models (those with the `vision` label in the [model picker](#mode

The tool **remains executable** if called - a model that invokes it from conversation history or right after a model switch gets a working tool, not an error. Text-only models keep the tool and the "use ImageDecode to inspect it" note unchanged.

The same split applies to an image file on disk: a vision model reads it with [`Read`](#read), which attaches the image to its next message, while a text-only model is sent to `ImageDecode` for a description in words. `ImageDecode` works for every model either way.

### Audio Tools

#### TextToSpeech Tool
Expand Down
2 changes: 1 addition & 1 deletion github-action.md
Original file line number Diff line number Diff line change
Expand Up @@ -492,7 +492,7 @@ Opt in by setting `vision-model` to any vision-capable model your gateway serves
deepseek-api-key: ${{ secrets.DEEPSEEK_API_KEY }}
```

This enables the CLI's `ImageDecode` tool and adds guidance to the system prompt: when the issue or PR text embeds an image (`![...](url)`), the agent downloads it with WebFetch and reads it with `ImageDecode` before starting work. The default `web-fetch-domains` already include the `githubusercontent.com` hosts that `github.com/user-attachments` links redirect to.
This enables the CLI's `ImageDecode` tool and adds guidance to the system prompt: when the issue or PR text embeds an image (`![...](url)`), the agent downloads it with WebFetch before starting work. A vision session model then reads the downloaded file with `Read`, which attaches the image to its next message. A text-only session model reads it with `ImageDecode`, which side-calls `vision-model` for a description in words. The default `web-fetch-domains` already include the `githubusercontent.com` hosts that `github.com/user-attachments` links redirect to.

**Limitation:** attachments on **private** repositories redirect to signed URLs that WebFetch requests unauthenticated, so private-repo screenshots may fail to download; the agent is instructed to say so and continue with the text. Public repositories work.

Expand Down