Skip to content

Commit 50dd74c

Browse files
authored
v0.9.57: cua improvements, Monaco CRM integration, Hex and Stripe validation, keyless subjects held until trust policies apply, AWS Marketplace replica entitlement check, Slack channel search
2 parents 87fefdf + 377e81b commit 50dd74c

243 files changed

Lines changed: 49485 additions & 1139 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apps/desktop/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -245,8 +245,8 @@ Terminal control),
245245
which the helper refuses. With **Ask before using a new
246246
app** on, each app's first use is approved for the task or persistently, and persistent approvals are
247247
listed in Settings and can be revoked. Sign-out and deployment changes clear opt-in, the ask choice
248-
and approvals. The conversation shows the
249-
active app/action and a Stop button; **⌘⇧Esc** also stops active work when the shortcut is available.
248+
and approvals. Stopping the chat response, or **⌘⇧Esc** when the shortcut is available, stops active
249+
work; Settings shows the action in progress.
250250
Stop cancels queued actions, in-flight authorization, approval prompts and the native helper.
251251
Already completed input cannot be undone.
252252

‎apps/desktop/native/computer-use/ComputerUse.swift‎

Lines changed: 194 additions & 23 deletions
Large diffs are not rendered by default.

‎apps/desktop/native/computer-use/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,15 +14,15 @@ Coordinates in action inputs are window-local points; node frames are global scr
1414

1515
AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A hit on a window's own background counts as that window. Background tabs and minimized windows share bounds with the window in front, so they take no coordinate input. An action that opens UI (AXPress, AXPick, AXShowMenu) and times out while the app shows a sheet, modal or menu (`cannotComplete`) counts as dispatched; any other action that times out fails as uncertain. A menu-item press that leaves the item's own menu open with the app highlighting that item (Calendar on macOS 26) is completed with Return. Sheets, including Go to Folder and out-of-process open and save panels whose contents report no window, count as the window they are attached to. Menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
1616

17-
`type_text` and `input_sequence` target a text control (text fields and areas, terminals, combo boxes, date and time pickers, or anything with an editable text selection), or a container whose focused editor already holds keyboard focus, such as a web composer; keystrokes are sent only while focus is in a text input. `input_sequence` executes 1–32 key/text steps against one such element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the batch before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.
17+
`type_text` and `input_sequence` target a text control (text fields and areas, terminals, combo boxes, date and time pickers, or anything with an editable text selection), or a container whose focused editor already holds keyboard focus, such as a web composer; keystrokes are sent only while focus is in a text input. When an app ignores the Accessibility focus request (SwiftUI sheets such as Clock's city search), a foreground app's editor is clicked at its center to take focus after 250 ms without it, only if that point hit-tests to the editor (or a non-secure text control inside it) within its observed window; a click refused by the mouse guards sends nothing, the caret is moved to the end and read back so typing inserts there (an empty editor needs no move, and an input_sequence starting with Cmd+A skips the check because it replaces the value); if the caret cannot be confirmed at the end, or focus does not settle, nothing is typed and the call fails with `caret_unconfirmed`. The usual focus checks then apply. `input_sequence` executes 1–32 key/text steps against one such element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the sequence before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. `batch` runs 2–8 click, type_text, set_value, press_key, scroll or perform_action steps planned from one snapshot, each through its own action's checks; it stops at the first step that fails and after any step that changes the app's window structure (on-screen windows, which include sheets, menus and popovers, or the focused window's direct children), reports progress the same way, and a first step that sent nothing is refused with `dispatchState: "not_started"`. Every step is validated before the first one runs. The structure check is a heuristic: it misses modals inside web content and UI that appears after the step's render wait, and stops conservatively on tooltips and autocomplete popovers. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.
1818

1919
## Cancellation
2020

2121
SIGTERM/SIGINT sets a cancellation flag. Traversal, typing and drag loops check it; mouse up is sent through a defer before returning from a cancelled drag. Keyboard down/up are paired in one function. The owner should close stdin and send SIGTERM, allowing a grace period before SIGKILL. Foreground drags also stop if the active app or window geometry changes; this does not detect every physical user input within the same app. Forced termination cannot guarantee a final release event; input routines deliberately avoid long waits with held buttons. Cancellation cannot undo an action already dispatched.
2222

2323
## Tests and limitations
2424

25-
Compile pure native tests with `swiftc -parse-as-library -D COMPUTER_USE_TEST ComputerUse.swift tests/NativeTests.swift -o native-tests`. They cover multi-display negative origins, coordinate bounds/nonfinite input, shortcut modifiers, invalid key handling and cooperative cancellation. Real status and app discovery outputs were validated against the generated worker Zod schema (156 apps, including 147 installed-only apps).
25+
Compile pure native tests with `swiftc -parse-as-library -D COMPUTER_USE_TEST ComputerUse.swift tests/NativeTests.swift -o native-tests`. They cover multi-display negative origins, coordinate bounds/nonfinite input, shortcut modifiers, invalid key handling, input-sequence and batch validation, protected apps and cooperative cancellation. The live fixture also covers batches and click-to-focus for an editor that ignores Accessibility focus. Real status and app discovery outputs were validated against the generated worker Zod schema (156 apps, including 147 installed-only apps).
2626

2727
`python3 tests/protocol.py <compiled-helper>` checks persistent framing, bounded-request recovery, Unicode decoding, no-prompt status, invalid methods/permissions, permission gates and clean EOF. `tests/Fixture.swift` compiles into a disposable AppKit app with a text field, secure field, increment button/counter, slider, scroll area and a visual-only marker. `python3 tests/live.py <helper>` requires both grants and verifies actual outcomes. `--allow-missing-screen` explicitly reports screenshots blocked rather than passing them; `--existing-fixture` leaves an existing fixture running for coordinated model trials. Give the fixture bundle ID `com.mothership.computer-use-fixture` when packaging it for integration tests.
2828

‎apps/desktop/native/computer-use/tests/Fixture.swift‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,14 @@ final class MarkerView: NSView {
88
}
99
}
1010

11+
/// A field that ignores Accessibility focus requests but takes focus from a click, as SwiftUI sheets do.
12+
class ClickFocusField: NSTextField {
13+
override func setAccessibilityFocused(_ accessibilityFocused: Bool) {}
14+
}
15+
/// A click-only editor that also ignores Accessibility caret moves, so its caret cannot be confirmed.
16+
final class FixedCaretField: ClickFocusField {
17+
override func setAccessibilitySelectedTextRange(_ accessibilitySelectedTextRange: NSRange) {}
18+
}
1119
final class FixtureDelegate: NSObject, NSApplicationDelegate {
1220
var window: NSWindow!
1321
var count = 0
@@ -31,6 +39,9 @@ final class FixtureDelegate: NSObject, NSApplicationDelegate {
3139
let field = NSTextField(string: "Fixture text"); field.setAccessibilityIdentifier("fixture-text"); field.setAccessibilityLabel("Fixture editable text")
3240
let secure = NSSecureTextField(string: "fixture-only-secret"); secure.setAccessibilityIdentifier("fixture-password"); secure.setAccessibilityLabel("Fixture password")
3341
let button = NSButton(title: "Increment", target: self, action: #selector(increment)); button.setAccessibilityIdentifier("fixture-increment")
42+
let clickField = ClickFocusField(string: ""); clickField.setAccessibilityLabel("Fixture click-focus field")
43+
let caretField = FixedCaretField(string: String(repeating: "keep ", count: 40)); caretField.setAccessibilityLabel("Fixture fixed-caret field")
44+
let sheetButton = NSButton(title: "Show sheet", target: self, action: #selector(showSheet)); sheetButton.setAccessibilityIdentifier("fixture-show-sheet")
3445
let slider = NSSlider(value: 25, minValue: 0, maxValue: 100, target: nil, action: nil); slider.setAccessibilityLabel("Fixture slider")
3546
let time = NSDatePicker(); time.datePickerStyle = .textField; time.datePickerElements = .hourMinute; time.setAccessibilityLabel("Fixture time")
3647
time.dateValue = Calendar.current.date(bySettingHour: 10, minute: 15, second: 0, of: Date()) ?? Date(); time.target = self; time.action = #selector(timeChanged(_:))
@@ -43,15 +54,16 @@ final class FixtureDelegate: NSObject, NSApplicationDelegate {
4354
scroll.documentView = document
4455
scroll.contentView.postsBoundsChangedNotifications = true
4556
NotificationCenter.default.addObserver(self, selector: #selector(scrolled(_:)), name: NSView.boundsDidChangeNotification, object: scroll.contentView)
46-
for view in [field, secure, button, counter, slider, time, timeLabel, marker, scrollCounter, scroll] { stack.addArrangedSubview(view) }
57+
for view in [field, secure, button, counter, clickField, caretField, sheetButton, slider, time, timeLabel, marker, scrollCounter, scroll] { stack.addArrangedSubview(view) }
4758
window.contentView!.addSubview(stack)
48-
NSLayoutConstraint.activate([stack.leadingAnchor.constraint(equalTo: window.contentView!.leadingAnchor, constant: 30), stack.trailingAnchor.constraint(equalTo: window.contentView!.trailingAnchor, constant: -30), stack.topAnchor.constraint(equalTo: window.contentView!.topAnchor, constant: 30), field.widthAnchor.constraint(equalTo: stack.widthAnchor), secure.widthAnchor.constraint(equalTo: stack.widthAnchor), slider.widthAnchor.constraint(equalTo: stack.widthAnchor), scroll.widthAnchor.constraint(equalTo: stack.widthAnchor), scroll.heightAnchor.constraint(equalToConstant: 230), marker.widthAnchor.constraint(equalToConstant: 100), marker.heightAnchor.constraint(equalToConstant: 70)])
59+
NSLayoutConstraint.activate([stack.leadingAnchor.constraint(equalTo: window.contentView!.leadingAnchor, constant: 30), stack.trailingAnchor.constraint(equalTo: window.contentView!.trailingAnchor, constant: -30), stack.topAnchor.constraint(equalTo: window.contentView!.topAnchor, constant: 30), field.widthAnchor.constraint(equalTo: stack.widthAnchor), secure.widthAnchor.constraint(equalTo: stack.widthAnchor), clickField.widthAnchor.constraint(equalTo: stack.widthAnchor), caretField.widthAnchor.constraint(equalTo: stack.widthAnchor), slider.widthAnchor.constraint(equalTo: stack.widthAnchor), scroll.widthAnchor.constraint(equalTo: stack.widthAnchor), scroll.heightAnchor.constraint(equalToConstant: 230), marker.widthAnchor.constraint(equalToConstant: 100), marker.heightAnchor.constraint(equalToConstant: 70)])
4960
window.makeKeyAndOrderFront(nil)
5061
window.makeFirstResponder(field)
5162
NSApplication.shared.activate()
5263
print("fixture-ready"); fflush(stdout)
5364
}
5465
@objc func increment() { count += 1; counter.stringValue = "Count: \(count)" }
66+
@objc func showSheet() { let alert = NSAlert(); alert.messageText = "Fixture sheet"; alert.addButton(withTitle: "Dismiss"); alert.beginSheetModal(for: window) }
5567
@objc func timeChanged(_ picker: NSDatePicker) {
5668
let time = Calendar.current.dateComponents([.hour, .minute], from: picker.dateValue)
5769
timeLabel.stringValue = "Time: \(time.hour ?? 0):\(String(format: "%02d", time.minute ?? 0))"

‎apps/desktop/native/computer-use/tests/NativeTests.swift‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,13 +45,25 @@ import AppKit
4545
precondition(bounded.next()?.element == 1)
4646
precondition(bounded.next()?.element == 2)
4747
precondition(bounded.next() == nil)
48-
try validateInputSteps([InputStep(action: "press_key", text: nil, key: "Cmd+A"), InputStep(action: "type_text", text: "hello 🌍", key: nil)])
49-
try validateInputSteps((1...5).flatMap { [InputStep(action: "type_text", text: String($0), key: nil), InputStep(action: "press_key", text: nil, key: "Enter")] })
48+
try validateInputSteps([Parameters(action: "press_key", key: "Cmd+A"), Parameters(action: "type_text", text: "hello 🌍")])
49+
try validateInputSteps((1...5).flatMap { [Parameters(action: "type_text", text: String($0)), Parameters(action: "press_key", key: "Enter")] })
5050
expectError("invalid_arguments") { try validateInputSteps([]) }
51-
expectError("invalid_arguments") { try validateInputSteps(Array(repeating: InputStep(action: "type_text", text: "x", key: nil), count: 33)) }
52-
expectError("invalid_arguments") { try validateInputSteps([InputStep(action: "type_text", text: String(repeating: "🌍", count: 16001), key: nil)]) }
53-
expectError("invalid_arguments") { try validateInputSteps([InputStep(action: "click", text: nil, key: nil)]) }
54-
expectError("unsupported_key") { try validateInputSteps([InputStep(action: "press_key", text: nil, key: "F99")]) }
51+
expectError("invalid_arguments") { try validateInputSteps(Array(repeating: Parameters(action: "type_text", text: "x"), count: 33)) }
52+
expectError("invalid_arguments") { try validateInputSteps([Parameters(action: "type_text", text: String(repeating: "🌍", count: 16001))]) }
53+
expectError("invalid_arguments") { try validateInputSteps([Parameters(action: "click")]) }
54+
expectError("unsupported_key") { try validateInputSteps([Parameters(action: "press_key", key: "F99")]) }
55+
try validateBatchSteps([Parameters(action: "set_value", elementId: "e1", value: "Name"), Parameters(action: "click", elementId: "e2")])
56+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "click", elementId: "e1")]) }
57+
expectError("invalid_arguments") { try validateBatchSteps(Array(repeating: Parameters(action: "press_key", key: "Tab"), count: 9)) }
58+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "drag"), Parameters(action: "press_key", key: "Enter")]) }
59+
expectError("unsupported_key") { try validateBatchSteps([Parameters(action: "click", elementId: "e1"), Parameters(action: "press_key", key: "F99")]) }
60+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "click", elementId: "e1"), Parameters(action: "click")]) }
61+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "click", elementId: "e1"), Parameters(action: "scroll", elementId: "e2")]) }
62+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "click", elementId: "e1"), Parameters(action: "type_text", elementId: "e2")]) }
63+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "set_value", value: "x"), Parameters(action: "click", elementId: "e1")]) }
64+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "click", elementId: "e1", clickCount: 4), Parameters(action: "press_key", key: "Enter")]) }
65+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "click", bundleId: "com.example", elementId: "e1"), Parameters(action: "press_key", key: "Enter")]) }
66+
expectError("invalid_arguments") { try validateBatchSteps([Parameters(action: "set_value", elementId: "e1", value: String(repeating: "🌍", count: 8001)), Parameters(action: "type_text", elementId: "e2", text: String(repeating: "🌍", count: 8000))]) }
5567
let tabFocus = try keyMayChangeFocus("Tab"); precondition(tabFocus)
5668
let submitFocus = try keyMayChangeFocus("Enter"); precondition(!submitFocus)
5769
let searchFocus = try keyMayChangeFocus("Cmd+K"); precondition(searchFocus)

0 commit comments

Comments
 (0)