Repository navigation
feat(screenshot): add an optional viewport capture with a debugger-backed fallback - #116
Open
DightEndless wants to merge 4 commits into
Open
DightEndless wants to merge 4 commits into
DightEndless wants to merge 4 commits into
Conversation
added 3 commits
October 10, 2026 03:02
The tool captures the visible area of the controlled tab as jpeg (default) or png with an optional 1-100 quality, and returns the pixels as base64 beside its text answer so the bridge can save them. chrome.tabs.captureVisibleTab comes first; because it needs <all_urls>/activeTab for the photographed tab and always shoots the active tab, a refused call falls back to Page.captureScreenshot over the DevTools protocol, which needs no activation. The answer names the route that produced the image. The debugger is an optional permission, declared in optional_permissions for both builds and granted on demand from a new Settings section that calls chrome.permissions.request from a user gesture. A missing grant fails with the exact switch that enables it. A screenshot is a page read: it joins the sharing-off privacy gate and the ask-mode read prompt. It is also the one tool the protected-page gate lets through, because the browser takes the picture, not the page.
… its path The extension returns image bytes beside its text answer, so this is the only place that turns them into a file: browser_screenshot writes the capture under a dsh-browser-screenshots directory in the OS temp directory and the model receives that path for read_image instead of base64. The tool needs no new wire vocabulary (tool.result.result was already unknown) and registers as browser_screenshot with format and quality, forwarded only when the model set them. The caps comment no longer claims page state can never include a capture.
Both root READMEs and the extension and bridge READMEs stated that browser tools never capture screenshots, which is no longer true. They now describe the one image route: the extension captures, the bridge saves the bytes to a local file, and the model receives a path. The permission section documents debugger as an optional grant, the protected-page paragraph notes that a capture still works there, and the bilingual-pair records are re-hashed for all three README pairs.
The assertion pulled the saved image's path out of the tool result with `/([A-Za-z]:\\[^\n]+\.png)/`, which matches a Windows path and nothing else. The suite passed on the author's machine and failed the Linux runner with "expected undefined to be defined", which is how this reached CI in the first place. It now looks for the directory the implementation documents, so both platforms find the same file, and the assertion no longer encodes one operating system's path shape.
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(screenshot): add an optional viewport capture, with a debugger-backed fallback
Adds
browser_screenshotfor the visible viewport, plus the debugger plumbing it needs when the browser refuses the ordinary capture call.Why
Issue #75 asks for hover and coordinate clicks, and for a way to reach targets the numbered inventory cannot express. A capture is the other half of that: it lets the model (and the person watching) see the page as pixels, and it is the only way to read something that is rendered but never reaches the text snapshot.
The page channel stays text-first. The pixels are written to a local file by the bridge and only the path is handed to the model — base64 never enters the conversation.
The permission model is deliberately narrow
debuggeris added tooptional_permissionsonly. It is never inpermissions, and the extension does not ask for it at install time.chrome.permissions.contains.How the capture works
chrome.tabs.captureVisibleTab(windowId, {format, quality})— no extra permission needed, and it photographs what is on screen.Page.captureScreenshotover the DevTools protocol with{format, quality?, fromSurface: true}.Both routes are covered by tests, including the fallback's exact command payload and the detach that follows it.
Two judgement calls worth your review
chrome://, extension stores, …) are exempted from the gate that otherwise blocks page tools there. The reasoning:captureVisibleTabis taken by the browser, not by the page's own script, so it does not inject anything into a protected document. Say the word and I will drop the exemption.protocol.tssaid the extension renders page state "text only (no screenshots)" and both READMEs claimed browser tools never capture screenshots. The capability negotiation is unchanged (textOnly: trueremains; it still means page state is not fed to the model as inline images), but the comments and READMEs now describe the capture accurately.Not verified in a live browser
The tests use a mocked
chrome, and the bundles build. I have not driven this branch against a real browser: the 120 ms activation settle before photographing a background tab, the fallback against a live tab, and the actual "being controlled" banner are unverified here. If you want, I can run a live pass and attach evidence before this is merged.debuggerinmanifest.firefox.jsonis also unverified against AMO review; if AMO objects, dropping it there degrades the tool to a clear "not granted" failure rather than breaking anything.Testing
The one bridge failure is pre-existing and environmental:
tests/token.spec.tsasserts a file mode of0600, which Windows cannot express (it reports0666). It fails identically on a cleanmainin this environment — verified by stashing the branch and running that spec alone.Scope
Viewport capture only. Full-page capture, hover, coordinate clicks, drag, batching, upload and eval are deliberately not in this change; they are separate increments so this one stays reviewable.