Skip to content

feat(screenshot): add an optional viewport capture with a debugger-backed fallback - #116

Open
DightEndless wants to merge 4 commits into
omdsh-dev:mainfrom
DightEndless:feat/screenshot-viewport
Open

DightEndless wants to merge 4 commits into
omdsh-dev:mainfrom
DightEndless:feat/screenshot-viewport

Conversation

@DightEndless

Copy link
Copy Markdown

feat(screenshot): add an optional viewport capture, with a debugger-backed fallback

Adds browser_screenshot for the visible viewport, plus the debugger plumbing it needs when the browser refuses the ordinary capture call.

Why

Issue #75 asks for hover and coordinate clicks, and for a way to reach targets the numbered inventory cannot express. A capture is the other half of that: it lets the model (and the person watching) see the page as pixels, and it is the only way to read something that is rendered but never reaches the text snapshot.

The page channel stays text-first. The pixels are written to a local file by the bridge and only the path is handed to the model — base64 never enters the conversation.

The permission model is deliberately narrow

  • debugger is added to optional_permissions only. It is never in permissions, and the extension does not ask for it at install time.
  • The request happens from a button in the existing Settings area ("Screenshot capture"), i.e. from a real user gesture, as Chrome requires. Opening Settings re-reads the current grant with chrome.permissions.contains.
  • If the grant is missing, the tool does not fail obscurely — it says what to turn on.
  • A test asserts the permission stays optional in both manifests, so a later edit cannot quietly make it mandatory.

How the capture works

  1. Try chrome.tabs.captureVisibleTab(windowId, {format, quality}) — no extra permission needed, and it photographs what is on screen.
  2. If that call is refused for any reason (on current Chrome it can fail with a permission error even when the extension holds host access), fall back to Page.captureScreenshot over the DevTools protocol with {format, quality?, fromSurface: true}.
  3. The answer states which route produced the image, so a reading of the result is never ambiguous.

Both routes are covered by tests, including the fallback's exact command payload and the detach that follows it.

Two judgement calls worth your review

  • Protected pages (chrome://, extension stores, …) are exempted from the gate that otherwise blocks page tools there. The reasoning: captureVisibleTab is taken by the browser, not by the page's own script, so it does not inject anything into a protected document. Say the word and I will drop the exemption.
  • A documentation contradiction is corrected in the same change. protocol.ts said the extension renders page state "text only (no screenshots)" and both READMEs claimed browser tools never capture screenshots. The capability negotiation is unchanged (textOnly: true remains; it still means page state is not fed to the model as inline images), but the comments and READMEs now describe the capture accurately.

Not verified in a live browser

The tests use a mocked chrome, and the bundles build. I have not driven this branch against a real browser: the 120 ms activation settle before photographing a background tab, the fallback against a live tab, and the actual "being controlled" banner are unverified here. If you want, I can run a live pass and attach evidence before this is merged.

debugger in manifest.firefox.json is also unverified against AMO review; if AMO objects, dropping it there degrades the tool to a clear "not granted" failure rather than breaking anything.

Testing

# extension
extensions/dsh-browser> node_modules/.bin/tsc -p tsconfig.json --noEmit   # exit 0
extensions/dsh-browser> node_modules/.bin/vitest run                     # 403 passed (51 files)

# bridge
packages/browser/bridge-browser> node_modules/.bin/tsc -b --pretty false # exit 0
packages/browser/bridge-browser> node_modules/.bin/vitest run            # 150 passed / 151

The one bridge failure is pre-existing and environmental: tests/token.spec.ts asserts a file mode of 0600, which Windows cannot express (it reports 0666). It fails identically on a clean main in this environment — verified by stashing the branch and running that spec alone.

Scope

Viewport capture only. Full-page capture, hover, coordinate clicks, drag, batching, upload and eval are deliberately not in this change; they are separate increments so this one stays reviewable.

dsh-browser-dev added 3 commits October 10, 2026 03:02
The tool captures the visible area of the controlled tab as jpeg (default) or png with an optional 1-100 quality, and returns the pixels as base64 beside its text answer so the bridge can save them.

chrome.tabs.captureVisibleTab comes first; because it needs <all_urls>/activeTab for the photographed tab and always shoots the active tab, a refused call falls back to Page.captureScreenshot over the DevTools protocol, which needs no activation. The answer names the route that produced the image.

The debugger is an optional permission, declared in optional_permissions for both builds and granted on demand from a new Settings section that calls chrome.permissions.request from a user gesture. A missing grant fails with the exact switch that enables it.

A screenshot is a page read: it joins the sharing-off privacy gate and the ask-mode read prompt. It is also the one tool the protected-page gate lets through, because the browser takes the picture, not the page.
… its path

The extension returns image bytes beside its text answer, so this is the only place that turns them into a file: browser_screenshot writes the capture under a dsh-browser-screenshots directory in the OS temp directory and the model receives that path for read_image instead of base64.

The tool needs no new wire vocabulary (tool.result.result was already unknown) and registers as browser_screenshot with format and quality, forwarded only when the model set them. The caps comment no longer claims page state can never include a capture.
Both root READMEs and the extension and bridge READMEs stated that browser tools never capture screenshots, which is no longer true. They now describe the one image route: the extension captures, the bridge saves the bytes to a local file, and the model receives a path.

The permission section documents debugger as an optional grant, the protected-page paragraph notes that a capture still works there, and the bilingual-pair records are re-hashed for all three README pairs.
The assertion pulled the saved image's path out of the tool result with
`/([A-Za-z]:\\[^\n]+\.png)/`, which matches a Windows path and nothing else. The
suite passed on the author's machine and failed the Linux runner with "expected
undefined to be defined", which is how this reached CI in the first place.

It now looks for the directory the implementation documents, so both platforms
find the same file, and the assertion no longer encodes one operating system's
path shape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant