Skip to content

Add selected-text voice editing on macOS and iOS - #124

Draft
IchenDEV wants to merge 11 commits into
mainfrom
research/voicecursor-voice-editing-20261008
Draft

IchenDEV wants to merge 11 commits into
mainfrom
research/voicecursor-voice-editing-20261008

Conversation

@IchenDEV

@IchenDEV IchenDEV commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Change

Desktop selected-text voice edits now replace the frozen selection automatically after generation and capture drain, through the existing Session/target lease/output transaction. The user explicitly removed candidate preview and Apply. The original dark HUD shows only progress, cancellation and a confirmed one-use Restore. Original recording controls and ordinary dictation retain their behavior.

Esc cancels a shown selected session before delivery. Command-Option-Z restores its current confirmed original once; host Command-Z is untouched. Native registrations are scoped to the shown eligible session/receipt, fail closed on conflict, and retire stale/repeated events. Selection, document, focus, Unicode, clipboard ownership and uncertain-write guards remain. Remote-model disclosure is still required before sending selected text; selected content stays out of generic history/follow-up even when generation fails before delivery.

The combined iPhone/iPad keyboard import retains its existing portable recipe, typed output acknowledgement and proposal adapter. This desktop direct-edit change does not claim mobile T0, final iOS validation or a platform-wide automatic replacement contract.

Verification

Current pushed head: da43ec844cc024c93d067bf9ae54e202052ca8a1.

Exact local check Result
Complete real SwiftPM build-tests, basic/SDLC and diff checks Passed
Owner/security/Session cases 81 passed,0 failed,0 skipped
Native HUD, interaction and shortcut cases 9 passed,0 failed,0 skipped
Retained private-runtime policy 9 named canaries passed
Native visual review 42 raw PNGs inspected, including12 long remote-disclosure locale/width/appearance variants

Native tests exercise the real Session automatic output, generation-time cancellation, stale selection/focus, privacy retention, failure/uncertainty/clipboard and exact-original restoration. Actual production HUD captures contain no original/candidate/Apply. Native AX labels/actions and process-local pointer Restore pass. Keyboard boundary tests use an injected OS registrar plus real owned Carbon event decoding; they do not prove global registration or physical key dispatch on the locked desktop. Long disclosure scrolling uses public NSView.scroll and inspected end glyphs, not a physical wheel.

Independent source review identified hidden-HUD shortcuts and an early delivering phase during capture drain. Both were fixed and covered by native behavior checks; final source review found no remaining issue. The first build's test-helper import error and an outdated screenshot utility failure remain recorded, followed by successful exact-source build/visual checks.

Source manifest: b9b45674c3ff4e0e2afdc76b62fea190a02a3f3ada97849b91b94788be91f836. Complete frozen native product aggregate: a605be004e1dfa496e2a4b89182b4f8c214ec80c02fd99208552706fd5e67e86. All19 local pins, native SwiftPM sandbox, no dependency/model downloads and continuous10GiBDatafloor were retained. The complete design law was reread and all157headings audited. Raw commands, failures, hashes and images stay ignored; verification records their scope.

Fresh GitHub full-suite and Release-style checks belong to this head. Previous27ffc8c checks are all green but are historical. Full local swift test was not retried because the retained SwiftTesting-loader retry limit is2/2.

Open acceptance gates

  • Actual global shortcut registration/dispatch, unlocked external-host voice/focus/delivery/restore, real VoiceOver speech/cursor/scrolling, physical wheel and live desktop-backdrop translucency remain unverified. No permission, authentication or accessibility-setting changes were made.
  • Mobile T0 and final integrated iOS/device validation remain pending. Cold mobile activation still requires manual host return and does not meet T0. No device was queried, reserved, installed or launched.
  • Valid but misrecognized speech can still cause an unintended rewrite; this user-requested direct flow uses existing recognition/generation checks and guarded original restoration, not a proven semantic guarantee.
  • Complete private-broker coverage, human/protected merge approval and distribution signing/notarization remain open. CI packaging is not distribution acceptance. This PR stays Draft.

Rollback is a normal revert of the increment; no data migration or user-data reset is involved. Preserve ordinary dictation, remote consent and the shared runtime/output owners.

IchenDEV and others added 11 commits October 6, 2026 16:49
…ecovery

Product: StandbyService keeps the main app serving keyboard commands in the
background through a video-call Picture in Picture, reports displacement
honestly, and re-takes standby after one keyboard link visit (waits for the
content-source scene to become foreground-active before starting PiP, with
diagnostics on every failure path). Device flow on iPad mini (A17 Pro):
control without PiP suspends within seconds while PiP stays alive for the
full five-minute idle; keyboard dictation after idle and after being
displaced by Safari's video PiP both recognize the fixed speech and insert
once. UITests park the system video window away from the Settings sidebar
instead of relying on its default top-left position.
…cument reads

Keyboard UI (SDLC 2026-10-06-keyboard-ui-redesign, direction B): a 64 pt voice
key, a status capsule, a seven-bar recording activity indicator and a full
editing row on a dynamic surface. Long results and large Dynamic Type wrap and
scroll, and Reduce Motion renders the bars static. Two operation hints are
localized in en and zh-Hans.

Lifecycle fix (SDLC 2026-10-06-keyboard-document-lifecycle): UIKit's text
document proxy is not valid in viewWillAppear, which crashed the keyboard when
returning to the host. visible, refresh and the standby watcher now start in
viewDidAppear; the lease, document and single-insert checks are unchanged.

Device acceptance on iPad mini (A17 Pro): light and dark states, VoiceOver,
largest accessibility text, Reduce Motion, Full Access off, ten host
return/restart rounds, plain and five-minute standby dictation, and recovery
after Safari's video PiP displaced standby. The fixed speech is recognized and
inserted once. The Safari LAN fixture needs the user to accept its
insecure-connection prompt, so that flow is not unattended.

Co-authored-by: Cursor <cursoragent@cursor.com>
The stack parent moved presentation contracts, extended the capture
contract and relocated the app icon. Keep the iOS app and keyboard
building on top of it:

- Guard AppKit-only declarations in UtterPresentationContracts so that
  UtterModels, and therefore UtterMobile, compiles for iOS. macOS
  declarations are unchanged.
- Implement the new OwnedRecording revoke/stopCapture requirements in
  MobileRecording and revoke the recording when the mobile job is revoked,
  so no input-unavailable callback fires after revocation and close()
  reuses the same idempotent stop that releases the audio session.
- Point the iOS project at the app icon's new location.

Co-authored-by: Cursor <cursoragent@cursor.com>
…gement

Keyboard: one-tap dictation with streaming insertion and undo, translucent keycaps,
native alignment. Models: HIG-style list over the desktop catalog, non-destructive
selection, downloadable MLX polish models, engine routing for MLX speech models.

Co-authored-by: Cursor <cursoragent@cursor.com>
MLX aborts on the Simulator's GPU (null Metal architecture name), so MLX speech
and text models are device-only there and a leftover polish choice can no longer
crash a dictation. Adds a debug polish probe for real devices, a simulator setup
helper for Full Access, and the verification findings.

Co-authored-by: Cursor <cursoragent@cursor.com>
…inner display matches

The keyboard picked its tablet key grid from the device idiom, which a foldable iPhone keeps as
iPhone on its regular-by-regular inner display. Use the size classes instead; iPad and iPhone
behavior is unchanged. Record the phase-two work (reserved regions, Duo simulator) that needs the
Xcode 27.1 SDK.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant