Add native QWERTY switch-scanning keyboard

docs/qwerty-keyboard.md

@@ -0,0 +1,32 @@
+# QWERTY scanning keyboard
+
+Switchify has its own UK English keyboard on Windows and macOS. It uses the saved local switches or the existing Android remote scanning actions. It does not use the operating system's on-screen keyboard.
+
+Choose a point and select **Type here** to click that point once and open the keyboard. To keep the existing input focus without clicking, select **More → Keyboard**. The keyboard does not take focus or accept mouse clicks.
+
+Scan a row, select it, then scan and select a key. The escape slot returns to rows. After typing, scanning starts again at the first row. Existing automatic/manual movement, speed, pause, reverse and inactivity suspension apply. The keyboard stays open after Space, Backspace, Enter and Tab. **Close** returns to point scanning.
+
+Pages:
+
+- **Letters:** UK QWERTY and punctuation, with Backspace beside the top letter row.
+- **Functions:** Esc, F1–F12, arrows, Home, End, Page Up, Page Down and Delete. Windows also has Insert, Print Screen, Scroll Lock and Pause.
+- **Numbers:** digits, decimal point, arithmetic operators, UK number-row symbols (including £), Enter and Backspace. Digits are independent of hardware Num Lock.
+
+The bottom control row starts with Close keyboard, followed by Letters, Navigation, Numbers and docking. Space has a dedicated wide key. Editing and modifier keys are wider than ordinary characters; duplicate Shift and Caps controls are removed from the letters page. A header names the current page and scan target. The return-to-rows slot highlights only the header. Active modifiers and the current page have an indicator separate from the scan highlight. Modifiers are available on every page. Shift, Ctrl, Alt/Option and Windows/Command cycle through **off → once → locked → off**. The label shows once or locked. Caps is a keyboard-local uppercase setting; Shift reverses its letter case. Page and position controls do not consume a one-shot modifier.
+
+Modifier keys are pressed only around each emitted shortcut and immediately released. Selecting a locked modifier does not hold that operating-system key while the scanner runs. Ordinary characters use text injection; command combinations and navigation use native key events. Native shortcuts retain the operating system's layout semantics.
+
+The keyboard closes when the foreground target or display environment changes, or its scan session ends. Failed input clears keyboard modifiers and requires Select to resume. Stop, disconnect and application exit use the shared deterministic input cleanup. No typed text is logged, and no prediction data, context reader, protocol extension or persistent keyboard settings are added.
+
+## Manual validation
+
+Use synthetic text in Notepad and a browser on Windows, and TextEdit and a browser on macOS. Launch macOS through `npm run macos:run` to retain its Accessibility identity.
+
+1. Use Type here and verify exactly one click focuses the intended field. Use More → Keyboard and verify no click or focus change occurs.
+2. Type lowercase letters, uppercase letters with Shift/Caps, and UK punctuation including £, @ and double quotes. Test Space, Backspace, Enter and Tab.
+3. Test Ctrl+A/C/V on Windows and Command+A/C/V on macOS. Cycle once/locked/off, including Shift with another modifier, and confirm no modifier remains physically held between selections.
+4. Visit every page. Verify row/key highlighting, reverse movement, row escape, suspension/resume, returning to the first row after a key and Close.
+5. Move the keyboard between the top and bottom. Check a scaled display and a secondary display, including negative coordinates. Verify the keyboard fits the work area and never activates its native windows.
+6. Switch foreground apps, disconnect remote scanning, change display configuration and close Switchify. Verify overlays disappear and owned input is released.
+
+Automated tests use fake input adapters only. Native manual results must be recorded separately; compilation and unit tests do not establish live application compatibility.

src-tauri/src/input.rs

@@ -154,6 +154,14 @@ fn pointer_button(button: MouseButton) -> Button {
 #[cfg(target_os = "windows")]
 fn windows_alphanumeric_key(name: &str) -> Option<Key> {
     Some(match name {
+        #[cfg(target_os = "windows")]
+        "Insert" => Key::Insert,
+        #[cfg(target_os = "windows")]
+        "PrintScreen" => Key::PrintScr,
+        #[cfg(target_os = "windows")]
+        "ScrollLock" => Key::Scroll,
+        #[cfg(target_os = "windows")]
+        "Pause" => Key::Pause,
         "=" => Key::OEMPlus,
         "-" => Key::OEMMinus,
         "[" => Key::OEM4,

src-tauri/src/lib.rs

@@ -24,6 +24,7 @@ mod protocol;
 mod remote_scan;
 mod scan_executor;
 mod scan_host;
+mod scan_keyboard;
 mod scan_menu;
 mod scan_tile;
 mod scan_tree;

src-tauri/src/modifier_overlay_windows.rs

@@ -13,7 +13,7 @@ use windows::Win32::Graphics::Gdi::{
     CreateCompatibleDC, CreateDIBSection, CreateFontW, DeleteDC, DeleteObject, DrawTextW, GetDC,
     GetMonitorInfoW, MonitorFromPoint, ReleaseDC, SelectObject, SetBkMode, SetTextColor,
     AC_SRC_ALPHA, AC_SRC_OVER, BITMAPINFO, BITMAPINFOHEADER, BI_RGB, BLENDFUNCTION,
-    CLIP_DEFAULT_PRECIS, DEFAULT_CHARSET, DEFAULT_QUALITY, DIB_RGB_COLORS, DT_CENTER,
+    CLIP_DEFAULT_PRECIS, DEFAULT_CHARSET, DEFAULT_QUALITY, DIB_RGB_COLORS, DT_CALCRECT, DT_CENTER,
     DT_SINGLELINE, DT_VCENTER, DT_WORDBREAK, FF_DONTCARE, FW_BOLD, HDC, HGDIOBJ, MONITORINFO,
     MONITOR_DEFAULTTONEAREST, OUT_DEFAULT_PRECIS, TRANSPARENT,
 };
@@ -411,6 +411,7 @@ fn present(window: HWND, labels: &[String], layout: &Layout) -> Result<(), Strin
         layout,
         pixmap.data(),
         labels.iter().any(|label| label.contains('\n')),
+        false,
     )
 }
 
@@ -466,6 +467,7 @@ fn present_pixmap_with_text(
     layout: &Layout,
     rgba: &[u8],
     wrap: bool,
+    center_wrapped: bool,
 ) -> Result<(), String> {
     unsafe {
         let screen = ScreenDc(GetDC(None));
@@ -534,8 +536,24 @@ fn present_pixmap_with_text(
         SetBkMode(memory.0, TRANSPARENT);
         SetTextColor(memory.0, COLORREF(0x00ff_ffff));
         for (label, chip) in labels.iter().zip(&layout.chips) {
+            if label.is_empty() {
+                continue;
+            }
             let mut text = label.encode_utf16().collect::<Vec<_>>();
             let mut text_rect = *chip;
+            if center_wrapped {
+                let mut measured = text_rect;
+                let height = DrawTextW(
+                    memory.0,
+                    &mut text,
+                    &mut measured,
+                    DT_CENTER | DT_WORDBREAK | DT_CALCRECT,
+                );
+                let vertical_padding = text_rect.top.min((layout.height - height).max(0) / 2);
+                text_rect.top = vertical_padding;
+                text_rect.bottom = layout.height - vertical_padding;
+                text_rect.top += ((text_rect.bottom - text_rect.top - height) / 2).max(0);
+            }
             let _ = DrawTextW(
                 memory.0,
                 &mut text,
@@ -595,6 +613,35 @@ pub(crate) fn present_scan_tile(
     tile: &crate::scanning::FrameTile,
 ) -> Result<(), String> {
     let pixmap = crate::scan_tile::bitmap(tile)?;
+    if tile.icon == crate::scan_menu::Item::KeyboardKey {
+        let width = pixmap.width() as i32;
+        let height = pixmap.height() as i32;
+        let horizontal_padding = (8.0 * tile.scale).round().max(1.0) as i32;
+        let vertical_padding = (6.0 * tile.scale).round().max(1.0) as i32;
+        let horizontal_padding = horizontal_padding.min((width - 1).max(0) / 2);
+        let vertical_padding = vertical_padding.min((height - 1).max(0) / 2);
+        let layout = Layout {
+            x: tile.rect.x.round() as i32,
+            y: tile.rect.y.round() as i32,
+            width,
+            height,
+            scale: tile.scale * crate::scan_tile::keyboard_font_size(tile) / FONT_SIZE,
+            chips: vec![RECT {
+                left: horizontal_padding,
+                top: vertical_padding,
+                right: width - horizontal_padding,
+                bottom: height - vertical_padding,
+            }],
+        };
+        return present_pixmap_with_text(
+            window,
+            std::slice::from_ref(&tile.text),
+            &layout,
+            pixmap.data(),
+            true,
+            true,
+        );
+    }
     let size = pixmap.width() as i32;
     let layout = Layout {
         x: tile.rect.x.round() as i32,
@@ -615,6 +662,7 @@ pub(crate) fn present_scan_tile(
         &layout,
         pixmap.data(),
         true,
+        false,
     )
 }
 
@@ -646,6 +694,34 @@ pub(crate) fn present_scan_prompt(
 
 #[cfg(test)]
 mod tests {
+    #[test]
+    fn empty_keyboard_backdrop_label_never_reaches_native_text_drawing() {
+        let layout = super::Layout {
+            x: 0,
+            y: 0,
+            width: 100,
+            height: 50,
+            scale: 1.0,
+            chips: vec![windows::Win32::Foundation::RECT {
+                left: 8,
+                top: 6,
+                right: 92,
+                bottom: 44,
+            }],
+        };
+        let pixels = vec![255; 100 * 50 * 4];
+        for wrapped in [false, true] {
+            let result = super::present_pixmap_with_text(
+                windows::Win32::Foundation::HWND::default(),
+                &[String::new()],
+                &layout,
+                &pixels,
+                wrapped,
+                wrapped,
+            );
+            assert!(result.is_err());
+        }
+    }
     use super::*;
     use windows::Win32::Foundation::{LPARAM, WPARAM};
     use windows::Win32::UI::WindowsAndMessaging::{
@@ -800,6 +876,24 @@ mod tests {
     #[test]
     fn native_window_render_smoke_becomes_visible_and_hides() {
         let mut host = NativeHost::new().unwrap();
+        let keyboard = crate::scan_keyboard::Keyboard::new(false);
+        let frame = keyboard.frame(
+            crate::scanning::Rect {
+                x: 0.0,
+                y: 0.0,
+                width: 1280.0,
+                height: 720.0,
+            },
+            1.0,
+            crate::scanning::ScannerColor::default(),
+        );
+        for tile in &frame.tiles {
+            super::present_scan_tile(host.window, tile).unwrap();
+            unsafe {
+                assert!(IsWindowVisible(host.window).as_bool());
+            }
+        }
+        host.hide();
         host.render(&["Ctrl".into(), "Shift".into()]).unwrap();
         unsafe {
             assert!(IsWindowVisible(host.window).as_bool());

src-tauri/src/overlay_macos.rs

@@ -228,6 +228,20 @@ pub(crate) fn image_from_rgba(
     height: usize,
     logical_size: f64,
 ) -> Result<Retained<NSImage>, String> {
+    image_from_rgba_rect(rgba, width, height, logical_size, logical_size)
+}
+
+pub(crate) fn image_from_rgba_rect(
+    rgba: &[u8],
+    width: usize,
+    height: usize,
+    logical_width: f64,
+    logical_height: f64,
+) -> Result<Retained<NSImage>, String> {
+    if !logical_height.is_finite() || logical_height <= 0.0 {
+        return Err("Invalid overlay height.".into());
+    }
+    let logical_size = logical_width;
     let layout = MacBitmapLayout::new(rgba.len(), width, height, logical_size)?;
     let representation = unsafe {
         NSBitmapImageRep::initWithBitmapDataPlanes_pixelsWide_pixelsHigh_bitsPerSample_samplesPerPixel_hasAlpha_isPlanar_colorSpaceName_bitmapFormat_bytesPerRow_bitsPerPixel(
@@ -251,13 +265,13 @@ pub(crate) fn image_from_rgba(
     }
     representation.setSize(NSSize {
         width: layout.logical_size,
-        height: layout.logical_size,
+        height: logical_height,
     });
     let image = NSImage::initWithSize(
         NSImage::alloc(),
         NSSize {
             width: layout.logical_size,
-            height: layout.logical_size,
+            height: logical_height,
         },
     );
     image.addRepresentation(&representation as &NSImageRep);

src-tauri/src/point_scan_runtime.rs

@@ -11,6 +11,26 @@ use tauri::{AppHandle, Manager};
 pub struct Environment {
     display: Display,
     foreground: usize,
+    keyboard_area: Rect,
+    handoff: Option<FocusHandoff>,
+}
+#[derive(Clone)]
+struct FocusHandoff {
+    target: usize,
+    ready_at: std::time::Instant,
+}
+impl FocusHandoff {
+    fn verify(&self, now: std::time::Instant, foreground: usize) -> Result<bool, String> {
+        if now < self.ready_at {
+            return Ok(false);
+        }
+        if foreground != self.target {
+            return Err(
+                "The selected target did not receive focus. Please select it again.".into(),
+            );
+        }
+        Ok(true)
+    }
 }
 pub struct PointScan;
 pub type Controller = scanning_runtime::Controller<PointScan>;
@@ -59,13 +79,43 @@ impl Adapter for PointScan {
         crate::scan_executor::cleanup()?;
         crate::point_scan_prepare(app)
     }
-    fn activate(app: &AppHandle, request: Request) -> Result<(), String> {
+    fn activate(app: &AppHandle, request: Request) -> Result<Option<Environment>, String> {
         crate::point_scan_ready(app)?;
         match request {
+            Request::OpenKeyboard { point: Some(point) } => {
+                let target = crate::scan_host::target_at(point)?;
+                crate::scan_executor::activate(request)?;
+                let (_, mut environment) =
+                    new_engine(app, app.state::<Controller>().view().config)?;
+                // Allow the one requested click to establish focus before accepting
+                // any key. No typing or scan advancement occurs during this handoff.
+                environment.handoff = Some(FocusHandoff {
+                    target,
+                    ready_at: std::time::Instant::now() + std::time::Duration::from_millis(150),
+                });
+                return Ok(Some(environment));
+            }
             Request::Setting(setting) => scanning_runtime::update_point_setting(app, setting),
             Request::Display(next) => scanning_runtime::restart_point_on_display(app, next),
             request => crate::scan_executor::activate(request),
         }
+        .map(|()| None)
+    }
+    fn settle_environment(
+        app: &AppHandle,
+        environment: Option<&mut Environment>,
+    ) -> Result<bool, String> {
+        if let Some(environment) = environment {
+            if let Some(handoff) = &environment.handoff {
+                crate::point_scan_ready(app)?;
+                if !handoff.verify(std::time::Instant::now(), crate::scan_host::foreground()?)? {
+                    return Ok(false);
+                }
+                environment.foreground = handoff.target;
+                environment.handoff = None;
+            }
+        }
+        Ok(true)
     }
     fn cleanup(_app: &AppHandle) -> Result<(), String> {
         crate::scan_executor::cleanup()
@@ -81,7 +131,7 @@ fn new_engine(app: &AppHandle, config: Config) -> Result<(Workflow, Environment)
     } else {
         1.0
     };
-    let e = Workflow::new(
+    let mut e = Workflow::new(
         config.point(),
         Rect {
             x: display.x.into(),
@@ -91,11 +141,20 @@ fn new_engine(app: &AppHandle, config: Config) -> Result<(Workflow, Environment)
         },
         units,
     )?;
+    let keyboard_area = crate::scan_host::work_area(Rect {
+        x: display.x.into(),
+        y: display.y.into(),
+        width: display.width.into(),
+        height: display.height.into(),
+    })?;
+    e.set_keyboard_area(keyboard_area);
     Ok((
         e,
         Environment {
             display,
             foreground: crate::scan_host::foreground()?,
+            keyboard_area,
+            handoff: None,
         },
     ))
 }
@@ -109,6 +168,36 @@ fn validate_display(app: &AppHandle, display: Option<&Environment>) -> Result<()
         if !displays.contains(&expected.display) {
             return Err("Display geometry changed. Scanning restarts.".into());
         }
+        let d = &expected.display;
+        if crate::scan_host::work_area(Rect {
+            x: d.x.into(),
+            y: d.y.into(),
+            width: d.width.into(),
+            height: d.height.into(),
+        })? != expected.keyboard_area
+        {
+            return Err("Display work area changed. Scanning restarts.".into());
+        }
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn focus_handoff_waits_and_only_accepts_the_clicked_target() {
+        let start = std::time::Instant::now();
+        let handoff = FocusHandoff {
+            target: 42,
+            ready_at: start + std::time::Duration::from_millis(150),
+        };
+        assert_eq!(handoff.verify(start, 7), Ok(false));
+        assert_eq!(handoff.verify(start, 42), Ok(false));
+        assert_eq!(handoff.verify(handoff.ready_at, 42), Ok(true));
+        // A failed activation and an unrelated app stealing focus both fail closed.
+        assert!(handoff.verify(handoff.ready_at, 7).is_err());
+        assert!(handoff.verify(handoff.ready_at, 99).is_err());
+    }
+}