Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Tauri regenerates these on every dev build; pin them to LF so a Windows
# checkout with core.autocrlf does not report them as modified.
src-tauri/permissions/autogenerated/*.toml text eol=lf
src-tauri/gen/** text eol=lf
10 changes: 5 additions & 5 deletions docs/point-scan.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,25 @@

Point scan ports the Android line-only and grid-then-line techniques to Switchify PC. The reference source is `switchifyapp/switchify-android` commit `856720d8747e2f3d1724a572bf754ffae05df299`, especially `PointScanLineManager`, `PointScanBlockManager`, and `ContinuousLineSpeedUtils`.

Open **Settings → Scanning**, choose automatic or manual scanning, and enable it. Local keyboard-emulating switch interfaces can use Space to select, Enter to step forward, Backspace to step backward, and F8 to pause or resume. The four keys can be changed. Escape always disables scanning and releases the reserved keys. Scanning is off at every application startup.
Open **Settings → Switches** to add named keyboard switches and assign normal and hold actions. Then enable scanning in **Settings → Scanning**. Fresh installs have no assignments; old point-scan keys migrate once. See [switch assignments](switches.md). Scanning stays off at startup.

Focus the intended application and press Select to start. Line mode chooses X, then Y, and clicks once. Grid mode chooses a row, then a cell, before the same line sequence. Selecting happens on switch release; holding Select freezes the position and repeat keydowns do not select again. The scan resets after a click and waits for the next Select. Movement wraps at the selected region's edges. Android's five speeds are 45, 75, 120, 180, and 270 logical units per second, with delayed ticks capped at 250 ms.
Focus the intended application and press Select to start. Line mode chooses X, then Y, and clicks once. Grid mode chooses a row, then a cell, before the same line sequence. Selecting happens on switch release; holding any switch freezes the position and repeat keydowns do not select again. The scan resets after a click and waits for the next Select. Movement wraps at the selected region's edges. Automatic movement stops after three full passes of the current phase without a selection; the scan resets and waits for the next Select. Manual steps never trigger this limit, and each Select starts a fresh count for the next phase. Android's five speeds are 45, 75, 120, 180, and 270 logical units per second, with delayed ticks capped at 250 ms.

The scan uses the monitor under the pointer when it starts. Windows uses native physical coordinates and display scaling; macOS uses Core Graphics display units and converts overlay rectangles to AppKit coordinates. A monitor geometry change cancels scanning. Native overlay strips are topmost, click-through, and nonactivating. The pointer moves only for the final click.

This first port uses switches attached to the computer. Android must be disconnected before enabling it, and an Android connection cancels local scanning. Existing authenticated Bluetooth commands and switch-forwarding profiles are unchanged. Input permission is still required on macOS. Point settings use a separate `point-scan.json` in the existing Tauri application configuration directory, so old settings and pairing schemas remain unchanged.

Changes save automatically, including when you leave Settings. Enabling waits for pending saves; a failed save keeps your edits and offers Retry save. Configuration is locked while scanning is enabled. Navigating away does not stop scanning.

The engine is independent of OS input. Activation uses the existing `InputInjector` adapter. Automated tests use a fake adapter and never move the real pointer. The Tauri global-shortcut plugin supplies global press/release events on both platforms; the existing dependencies did not include a switch-key listener. Only the main window can configure scanning through IPC.
The engine is independent of OS input. Activation uses the existing `InputInjector` adapter. Automated tests use a fake adapter and never move the real pointer. Switchify's local `switch_input` adapter supplies press/release events on both platforms. See `switches.md` for Windows capture limitations. Only the main window can configure scanning through IPC.

Manual validation should cover both modes, every speed, manual movement, holding and releasing switches, Escape, focus retention, display changes, mixed scaling, a Bluetooth connection during scanning, and application exit. Run macOS input checks through `npm run macos:run` to retain the stable Accessibility identity. No physical switch or real desktop input is exercised by automated tests.

## Reusing scanning in Switchify PC

`scanning.rs` is the pure shared core. `Session<T>` owns start, pause, completion, cancellation and bounded automatic timing. `SwitchInput` turns presses and releases into semantic actions, suppresses repeat events and rejects stale shortcut generations. `Cycle` and `Interval` provide traversal and timing. A technique implements `start`, `advance`, `handle`, `reset`, `frame` and `phase`, with its own typed selection result. The test-only item technique exercises this contract without pointer coordinates or desktop input.
`scanning.rs` is the pure shared core. `Session<T>` owns start, pause, completion, cancellation and bounded automatic timing. The shared gesture engine turns physical presses and releases into normal and hold actions; the controller rejects stale capture generations. `Cycle` and `Interval` provide traversal and timing. A technique implements `start`, `advance`, `handle`, `reset`, `frame` and `phase`, with its own typed selection result. The test-only item technique exercises this contract without pointer coordinates or desktop input.

`point_scan.rs` implements row/cell and X/Y selection. Its engine receives only point settings; the existing flat `Config` maps into shared switch settings and point settings. `scanning_runtime.rs` owns shortcut registration, session ticking, persistence, event publication and shutdown. Its `Adapter` supplies configuration, environment validation and selection activation. `scan_host.rs` renders shared frame strips through nonactivating Windows and macOS windows. `point_scan_runtime.rs` supplies the display and click adapters, and `point_scan_activation.rs` checks input state before clicking through `InputInjector`.
`point_scan.rs` implements row/cell and X/Y selection. Its engine receives only point settings; the existing flat `Config` maps into shared switch settings and point settings. `scanning_runtime.rs` owns embedded switch dispatch, session ticking, persistence, event publication and shutdown. Its `Adapter` supplies configuration, environment validation and selection activation. `scan_host.rs` renders shared frame strips through nonactivating Windows and macOS windows. `point_scan_runtime.rs` supplies the display and click adapters, and `point_scan_activation.rs` checks input state before clicking through `InputInjector`.

Only one local technique is installed at a time. A future technique can use the shared controller with its own adapter; adding a technique chooser is separate work. Transport cleanup calls the shared scan service, which invalidates callbacks immediately and releases native resources on the main thread. Queued cancellation cannot stop a newer registration.

Expand Down
25 changes: 25 additions & 0 deletions docs/switches.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Switch assignments

In Settings → Switches, each switch is a row showing its name, key and actions; Edit expands it. Add switch starts learning immediately: press and release the physical key, then enter a name and choose actions, then save. Learning can also be restarted from Change key. Assign Select, Next, Previous, Reverse direction, Stop scanning or Pause / resume. Each key belongs to one switch; several switches can run the same action. Existing assignments save automatically. Disable scanning before editing. While learning, a modal dialog holds keyboard focus and swallows key events so a switch press cannot scroll Settings or activate a button; Cancel capture is mouse-only and Escape cancels through the backend. Learning ends when Settings loses focus, the panel closes, or Escape is pressed.

The normal action runs on release. Hold actions are offered in their configured order. The interval defaults to 1 second and accepts 250–5000 milliseconds; five presets are shown and the rest sit behind an exact-interval select. The editor shows the time at which each hold action is offered. Each interval offers the next action in a nonactivating desktop prompt; the last remains selected. Release executes only the offered action. Movement freezes while any switch is held. The first pressed switch owns the gesture; overlapping presses do not activate additional actions.

Stop resets the scan but leaves switches enabled. Select starts again. Pause preserves position. Reverse changes direction without stepping. Automatic scanning requires Select; manual scanning also requires Next and Previous. Hold assignments count toward those requirements.

Escape disables capture immediately. The emergency hold timeout is at least 4 seconds and extends to (longest hold list + 2) × interval when hold actions exist. Settings shows the resulting duration. Heartbeat loss, capture failure, overflow, Android connection and shutdown cancel gestures without executing release actions.

## Storage and integration

The separate switch-settings.json uses schema version 1. Existing point-scan keys migrate once into four named assignments with empty hold lists. Fresh installs persist an empty list. Malformed or unsupported settings are reported without overwriting them. Saves use atomic replacement. Scanning stays off at startup.

Existing point-scan commands and event fields remain compatible for geometry and automatic mode. Legacy key fields remain readable; changing them is rejected with a direction to Settings → Switches. New settings and learning commands are restricted to the main window capability and check its label. On Windows the learning command checks that the main window is the foreground window rather than tao's focus flag, which turns false once keyboard focus moves into the WebView2 child.

Switchify owns local input through `switch_input`, a platform adapter that supplies generation-tagged press/release events. There is no USAHP runtime or dependency. Output still uses Switchify's existing input adapters. The capture state and macOS event tap were adapted from MIT-licensed code; attribution is retained beside the source.

Windows reserves unmodified assigned keys and Escape with `RegisterHotKey`, uses `MOD_NOREPEAT` for press deduplication, and receives releases through Raw Input on a dedicated message thread. Startup fails and rolls back all registrations if an assigned key cannot be reserved. F12 is unavailable because Windows reserves it for the debugger. During learning, only keys successfully reserved for that capture can be learned; a key reserved by another app will not be learned. Disable, emergency cancellation and heartbeat loss release unused reservations on the input thread. Keys already consumed stay reserved until release, so holding a cancelled switch cannot leak autorepeat. App exit removes all reservations.

Windows does not provide complete suppression through this adapter: key releases can reach other applications, and combinations with Shift, Ctrl, Alt or Windows are not reserved. Use plain switch keys. Raw Input alone does not start scan actions. macOS retains its event-tap capture and suppression behavior. Physical Windows activation, hold behavior, background operation and cleanup need manual verification on each supported setup.

The reusable gesture engine handles normal and ordered hold actions. The shared scan controller dispatches them to Session<T>. Native prompts are click-through and nonactivating.

Automated tests use fake events and never send desktop input. Manual checks should cover physical suppression and learning, focus retention, hold timing, emergency exits, capture loss, permission changes, display changes and shutdown. Use npm run macos:run for macOS permission testing with the established app identity. This version supports keyboard switches and scanning actions.
37 changes: 3 additions & 34 deletions src-tauri/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 4 additions & 2 deletions src-tauri/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ crate-type = ["staticlib", "cdylib", "rlib"]
tauri-build = { version = "2", features = [] }

[dependencies]
anyhow = "1"
base64 = "0.22"
directories = "6.0.0"
enigo = "=0.6.1"
Expand All @@ -26,7 +27,6 @@ serde_json = "1"
sha2 = "0.10"
tauri = { version = "2.11.5", features = ["tray-icon", "image-png", "macos-private-api"] }
tauri-plugin-autostart = "2.5.1"
tauri-plugin-global-shortcut = "2"
tauri-plugin-single-instance = "2.4.3"
tauri-plugin-updater = "2.10.1"
tiny-skia = "0.11.4"
Expand All @@ -36,14 +36,16 @@ uuid = { version = "1", features = ["v4", "serde"] }
[target.'cfg(target_os = "macos")'.dependencies]
block2 = "0.6.2"
corebluetooth-rs = "=0.3.6"
core-foundation = "0.10"
core-graphics = { version = "0.25", features = ["highsierra"] }
libc = "0.2"
objc2 = "0.6.4"
objc2-app-kit = { version = "0.3.2", features = ["NSBitmapImageRep", "NSColor", "NSControl", "NSEvent", "NSGraphics", "NSImage", "NSImageRep", "NSImageView", "NSPanel", "NSResponder", "NSScreen", "NSView", "NSWindow", "NSWorkspace", "objc2-core-foundation"] }
objc2-app-kit = { version = "0.3.2", features = ["NSBitmapImageRep", "NSColor", "NSControl", "NSEvent", "NSGraphics", "NSImage", "NSImageRep", "NSImageView", "NSPanel", "NSTextField", "NSFont", "NSResponder", "NSScreen", "NSView", "NSWindow", "NSWorkspace", "objc2-core-foundation"] }
objc2-core-graphics = { version = "0.3.2", default-features = false, features = ["CGEventSource", "CGEventTypes"] }
objc2-foundation = { version = "0.3.2", features = ["NSArray", "NSGeometry", "NSHost", "NSNotification", "NSObject", "NSOperation", "NSString", "NSThread", "NSURL", "block2"] }

[target.'cfg(target_os = "windows")'.dependencies]
windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Graphics_Gdi", "Win32_System_LibraryLoader", "Win32_System_Threading", "Win32_UI_WindowsAndMessaging", "Win32_UI_Input_KeyboardAndMouse"] }
windows = { version = "0.62.2", features = [
"Devices_Bluetooth",
"Devices_Bluetooth_GenericAttributeProfile",
Expand Down
4 changes: 4 additions & 0 deletions src-tauri/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ fn main() {
add_command_line_tools_swift_library_path();
let app_manifest = tauri_build::AppManifest::new().commands(&[
"get_app_state",
"get_switches",
"save_switches",
"begin_switch_capture",
"cancel_switch_capture",
"get_point_scan",
"configure_point_scan",
"check_accessibility",
Expand Down
4 changes: 4 additions & 0 deletions src-tauri/capabilities/main.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@
"permissions": [
"core:default",
"allow-get-app-state",
"allow-get-switches",
"allow-save-switches",
"allow-begin-switch-capture",
"allow-cancel-switch-capture",
"allow-get-point-scan",
"allow-configure-point-scan",
"allow-check-accessibility",
Expand Down
11 changes: 11 additions & 0 deletions src-tauri/permissions/autogenerated/begin_switch_capture.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Automatically generated - DO NOT EDIT!

[[permission]]
identifier = "allow-begin-switch-capture"
description = "Enables the begin_switch_capture command without any pre-configured scope."
commands.allow = ["begin_switch_capture"]

[[permission]]
identifier = "deny-begin-switch-capture"
description = "Denies the begin_switch_capture command without any pre-configured scope."
commands.deny = ["begin_switch_capture"]
11 changes: 11 additions & 0 deletions src-tauri/permissions/autogenerated/cancel_switch_capture.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Automatically generated - DO NOT EDIT!

[[permission]]
identifier = "allow-cancel-switch-capture"
description = "Enables the cancel_switch_capture command without any pre-configured scope."
commands.allow = ["cancel_switch_capture"]

[[permission]]
identifier = "deny-cancel-switch-capture"
description = "Denies the cancel_switch_capture command without any pre-configured scope."
commands.deny = ["cancel_switch_capture"]
11 changes: 11 additions & 0 deletions src-tauri/permissions/autogenerated/get_switches.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Automatically generated - DO NOT EDIT!

[[permission]]
identifier = "allow-get-switches"
description = "Enables the get_switches command without any pre-configured scope."
commands.allow = ["get_switches"]

[[permission]]
identifier = "deny-get-switches"
description = "Denies the get_switches command without any pre-configured scope."
commands.deny = ["get_switches"]
11 changes: 11 additions & 0 deletions src-tauri/permissions/autogenerated/save_switches.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Automatically generated - DO NOT EDIT!

[[permission]]
identifier = "allow-save-switches"
description = "Enables the save_switches command without any pre-configured scope."
commands.allow = ["save_switches"]

[[permission]]
identifier = "deny-save-switches"
description = "Denies the save_switches command without any pre-configured scope."
commands.deny = ["save_switches"]
Loading