feat(snapshot): add configurable image capture limits - #43
Open
aikins01 wants to merge 1 commit into
Open
Conversation
Collaborator
|
hi, please resolve conflicts |
There was a problem hiding this comment.
Pull request overview
This PR expands the Open Computer Use runtimes to better fit host context budgets by introducing configurable macOS screenshot capture limits and get_app_state output compaction controls, while also updating the cross-platform tool surface to include select_text and improving app-resolution ranking.
Changes:
- Add macOS screenshot capture configuration via environment variables (timeout, long-edge cap, byte budget, and byte-budget retry floor scaling).
- Add
get_app_stateoutput controls (include_image,force_image,max_text_chars,only_changes) with per-app output caching to dedupe screenshots / cap text. - Add
select_texttool across macOS/Windows/Linux runtimes + fixture/smoke coverage, and refine app discovery resolution ranking.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/open-computer-use/SKILL.md | Updates skill docs to include select_text usage guidance. |
| skills/open-computer-use/references/usage.md | Documents select_text and get_app_state output controls + Windows opt-in for UIA selection. |
| README.zh-CN.md | Adds Chinese documentation for macOS screenshot sizing env vars. |
| README.md | Adds documentation for macOS screenshot sizing env vars. |
| plugins/open-computer-use/.codex-plugin/plugin.json | Updates plugin metadata to reflect 10-tool surface. |
| packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift | Adds unit tests for image config parsing, PNG bounding, output controls, selection match, and tool/schema counts. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift | Extends schemas with select_text and new get_app_state output-control fields. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift | Updates server instructions to include the select_text tool. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift | Extends fixture command/state types to support select_text and selected text reporting. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift | Parses new get_app_state output options and adds select_text dispatch. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift | Implements output caching/compaction for get_app_state and adds macOS select_text behavior. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift | Improves app resolution to prefer regular apps and stronger name matches. |
| packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift | Introduces ImageCaptureConfig + applies configurable capture limits and improved byte-budget scaling logic. |
| docs/releases/feature-release-notes.md | Adds July 2026 feature release notes entries covering new controls and tools. |
| docs/QUALITY_SCORE.md | Updates quality score narrative from 9 tools to 10 tools. |
| docs/histories/2026-07/20260702-1837-compact-app-state-output.md | Adds history entry for compact app-state output behavior. |
| docs/histories/2026-07/20260702-1634-app-resolution-ranking.md | Adds history entry for app resolution ranking fix. |
| docs/histories/2026-07/20260702-1202-select-text-codex-parity.md | Adds history entry for select_text parity work. |
| docs/histories/2026-07/20260702-0126-configurable-image-capture.md | Adds history entry for configurable macOS image capture behavior. |
| docs/exec-plans/completed/20260702-select-text-codex-parity.md | Adds completed execution plan for select_text parity. |
| docs/ARCHITECTURE.md | Updates architecture docs for 10-tool surface, output controls, and macOS capture config. |
| apps/OpenComputerUseWindows/runtime.ps1 | Adds select_text implementation via UIA TextPattern with explicit opt-in. |
| apps/OpenComputerUseWindows/main.go | Adds select_text, get_app_state output controls, and app-state output caching on Windows. |
| apps/OpenComputerUseWindows/main_test.go | Adds/updates tests for schema parity and output-control behavior on Windows. |
| apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift | Extends end-to-end smoke suite to include select_text and 10-tool expectation. |
| apps/OpenComputerUseLinux/runtime.py | Adds select_text implementation via AT-SPI text selection/caret APIs. |
| apps/OpenComputerUseLinux/main.go | Adds select_text, get_app_state output controls, and app-state output caching on Linux. |
| apps/OpenComputerUseLinux/main_test.go | Adds/updates tests for schema parity and output-control behavior on Linux. |
| apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift | Extends fixture to handle select_text and export selected text for smoke validation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
229
to
240
| private func optionalNonNegativeInt(_ key: String, in arguments: [String: Any]) throws -> Int? { | ||
| guard arguments.keys.contains(key) else { | ||
| return nil | ||
| } | ||
|
|
||
| guard let value = normalizedNonNegativeIntegerArgument(arguments[key]) else { | ||
| throw ComputerUseError.invalidArguments("\(key) must be a non-negative integer") | ||
| } | ||
|
|
||
| return value | ||
| } | ||
| } |
Comment on lines
+128
to
+132
| ToolDefinition( | ||
| name: "select_text", | ||
| description: "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it.", | ||
| annotations: defaultAnnotations(), | ||
| inputSchema: objectSchema( |
aikins01
force-pushed
the
feat/image-capture-env-controls
branch
from
July 19, 2026 00:03
bd623b9 to
3db2f22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
macOS screenshots can now be bounded per MCP host so
get_app_stateand action results fit tighter context budgets without changing the default behavior.OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSIONas a hard returned-PNG cap while allowing byte-budget retries to shrink further within the configured scale floor