diff --git a/docs/word-prediction.md b/docs/word-prediction.md index 6f9d3549..8055199b 100644 --- a/docs/word-prediction.md +++ b/docs/word-prediction.md @@ -1,6 +1,6 @@ # Word prediction -Word prediction is enabled by default in Scanning settings. Its five-position row appears only on the Letters page. Empty positions are skipped. Select the row and then a word using the existing switches. Acceptance appends the missing suffix and a space without deleting text, selecting text or using the clipboard. Shift and Caps affect completion casing; Ctrl, Alt/Option and Windows/Command suppress suggestions. +Word prediction is enabled by default in Scanning settings. Its five-position row appears only on the Letters page. Empty positions are skipped. Select the row and then a word using the existing switches. Acceptance appends the missing suffix and a space without selecting text or using the clipboard. It deletes nothing, with one bounded exception: when the word has a capital where a lowercase letter was typed, as in “lon” for London or “i” for I’m, it backspaces the typed prefix and types the whole word. Shift and Caps affect completion casing; Ctrl, Alt/Option and Windows/Command suppress suggestions. Predictions use only a temporary buffer of successful Switchify keyboard input, starting with the first letter. Existing text, pasted text and hardware keyboard typing are never read into it. Switchify does not inspect fields, selections, passwords or caret positions. Suggestions may therefore appear anywhere the keyboard is open, including password fields or applications without a text field. The buffer records successful input injection; it cannot verify what an application actually accepted. @@ -14,7 +14,7 @@ One Word prediction setting controls an offline SmolLM2-135M int8 ONNX model. Th For a first typed prefix, a fixed local context keeps the model from favoring website names at the start of a document; it adds no user text. The model loads in the worker while keyboard input remains available. Suggestions are blank until loading finishes. A passive badge beside the scan prompt distinguishes loading, a ready keyboard awaiting typed context, no matching suggestions, available suggestions, paused activity tracking, and prediction failure. It never shows typed text and is not a scan target. If loading or inference fails, or a call exceeds 1.5 seconds, the keyboard continues accepting input and offers **Retry predictions** in its toolbar. Retry restarts only the prediction worker, clears its private text context, and leaves the keyboard open; type a new prefix afterward. A 400 ms search budget bounds candidate exploration. A clipped buffer with no complete earlier word yields no suggestions. -Candidates contain ASCII letters and apostrophes and must have sufficient model probability. There is no vocabulary filter: any word the model finds likely can be suggested, including swearing, because the person typing chose it. Single-letter candidates are limited to “a” and “I”. The pronoun I and its contractions, such as I’m and I’ll, are completed with a capital I; letters already typed stay as typed. Up to five suggestions are shown: the first, third and fifth are the most likely single words, and the second and fourth are the two most likely two-word phrases that begin with one of the top three words, ranked by the probability of the pair. When fewer phrases are found within an extra 200 ms, single words fill the remaining slots, and the other way round. Accepting a phrase inserts the rest of its first word, a space, the second word and a trailing space. Ordinary words use lowercase. Names keep the model's capital: mixed case anywhere, such as WhatsApp, or an initial capital mid-sentence that the model clearly prefers, such as London or Monday, which it does for proper nouns and occasionally a rare word. At a sentence start the model capitalises every word, so the lowercase form wins there and the sentence rule decides. Casing is applied when the keyboard displays a candidate, and the label shows exactly what will be inserted. A typed prefix of two or more letters, all capitals, is completed in capitals whatever the modifiers now say. Otherwise Caps, or Shift locked, uppercases the whole completion, and together they cancel as they do on typed letters. Before any letter of the word is typed, Shift once changes only the first letter, the way it would change the next typed letter, and a sentence start capitalises it without any modifier. The keyboard alone decides what a sentence start is: a full stop, exclamation or question mark it typed, not yet followed by a letter and not cancelled by pressing Shift. Punctuation the worker merely sees in its buffer, or the bare start of that buffer after a reset, does not capitalise anything, so a suggestion never disagrees with the keyboard's own automatic Shift. Shift once under Caps gives a lowercase first letter there, as it would on a typed letter. After typed letters, a pending Shift once is left for the next letter and does not touch the completion. +Candidates contain ASCII letters and apostrophes and must have sufficient model probability. There is no vocabulary filter: any word the model finds likely can be suggested, including swearing, because the person typing chose it. Single-letter candidates are limited to “a” and “I”. The pronoun I and its contractions, such as I’m and I’ll, are completed with a capital I. Accepting restores a capital the typed prefix lacks by retyping that prefix: the prefix is text Switchify typed itself in the current window with no outside activity since, it is never more than one word, and the executor refuses more than 32 deletions. A capital the person typed is kept, a word typed in capitals is never retyped, and when the case already matches nothing is deleted. The label always reads as the text will after acceptance. Any keyboard or mouse activity not made by Switchify between choosing the suggestion and typing it cancels the acceptance, so nothing else is deleted. Deleting and typing cannot be atomic: if typing fails after the deletion, the prefix is lost and prediction context resets. The deletion assumes one Backspace removes one typed character; an application that auto-pairs or autocorrects what was typed can break that, and Switchify cannot verify what an application accepted. Up to five suggestions are shown: the first, third and fifth are the most likely single words, and the second and fourth are the two most likely two-word phrases that begin with one of the top three words, ranked by the probability of the pair. When fewer phrases are found within an extra 200 ms, single words fill the remaining slots, and the other way round. Accepting a phrase inserts the rest of its first word, a space, the second word and a trailing space. Ordinary words use lowercase. Names keep the model's capital: mixed case anywhere, such as WhatsApp, or an initial capital mid-sentence that the model clearly prefers, such as London or Monday, which it does for proper nouns and occasionally a rare word. At a sentence start the model capitalises every word, so the lowercase form wins there and the sentence rule decides. Casing is applied when the keyboard displays a candidate, and the label shows exactly what will be inserted. A typed prefix of two or more letters, all capitals, is completed in capitals whatever the modifiers now say. Otherwise Caps, or Shift locked, uppercases the whole completion, and together they cancel as they do on typed letters. Before any letter of the word is typed, Shift once changes only the first letter, the way it would change the next typed letter, and a sentence start capitalises it without any modifier. The keyboard alone decides what a sentence start is: a full stop, exclamation or question mark it typed, not yet followed by a letter and not cancelled by pressing Shift. Punctuation the worker merely sees in its buffer, or the bare start of that buffer after a reset, does not capitalise anything, so a suggestion never disagrees with the keyboard's own automatic Shift. Shift once under Caps gives a lowercase first letter there, as it would on a typed letter. After typed letters, a pending Shift once is left for the next letter and does not touch the completion. The model files are too large to commit. `npm run prediction-model` downloads them from a pinned upstream revision and verifies their SHA-256. The Tauri dev and build commands and CI run it automatically. Run it once before `cargo test` or `cargo clippy` in a fresh checkout. diff --git a/src-tauri/src/prediction/database.rs b/src-tauri/src/prediction/database.rs index d3365b22..200ec376 100644 --- a/src-tauri/src/prediction/database.rs +++ b/src-tauri/src/prediction/database.rs @@ -106,7 +106,7 @@ impl Predict for FakeModel { ) -> Result { let words: Vec = ["water", "waffle", "walk", "WhatsApp"] .into_iter() - .filter(|w| w.starts_with(&prefix.to_lowercase())) + .filter(|w| w.to_lowercase().starts_with(&prefix.to_lowercase())) .map(str::to_owned) .collect(); let phrases = words.iter().take(1).map(|w| format!("{w} is")).collect(); diff --git a/src-tauri/src/prediction/mod.rs b/src-tauri/src/prediction/mod.rs index 42f1a02a..bd76088f 100644 --- a/src-tauri/src/prediction/mod.rs +++ b/src-tauri/src/prediction/mod.rs @@ -147,6 +147,9 @@ struct Service { acknowledged_revision: u64, reset: bool, accept: Option<(u64, usize)>, + /// The activity epoch when the suggestion was chosen. A deletion needs it + /// unchanged and the observer healthy at the moment of injection. + accept_activity: (u64, bool), accepting: bool, case: Option<(worker::Shift, bool, bool, bool)>, tracking: bool, @@ -368,9 +371,19 @@ pub fn select(token: u64, index: usize) -> Result<(), String> { return Err("Prediction is unavailable.".into()); } s.accept = Some((token, index)); + s.accept_activity = activity::snapshot(); Ok(()) }) } +/// Whether an accepted suggestion may delete typed characters. The worker +/// checked for outside input up to its reply; this covers the time since the +/// suggestion was chosen. Any keyboard or mouse activity not made by +/// Switchify, or an observer that cannot vouch for the interval, refuses the +/// deletion. Appending without deleting keeps its existing checks. +fn may_delete(backspaces: usize, chosen: (u64, bool), now: (u64, bool)) -> bool { + backspaces == 0 || (chosen.1 && now == chosen) +} + /// The model's config file; its directory is the model. A file rather than /// the directory, so a half-copied bundle is not mistaken for a model. fn resource(app: &AppHandle) -> Result { @@ -463,6 +476,7 @@ pub fn poll(app: &AppHandle, keyboard: Option<&mut Keyboard>, enabled: bool, ign } Response::Insert { generation, + backspaces, text, foreground, } => { @@ -474,12 +488,20 @@ pub fn poll(app: &AppHandle, keyboard: Option<&mut Keyboard>, enabled: bool, ign if Some(scope.foreground) != foreground || !scope.unchanged() { return Err(()); } - crate::scan_executor::prediction_text(&text).map_err(|_| ())?; + if !may_delete(backspaces, s.accept_activity, activity::snapshot()) + { + return Err(()); + } + crate::scan_executor::prediction_replace(backspaces, &text) + .map_err(|_| ())?; if !scope.unchanged() { return Err(()); } let trailing_space = text.ends_with(' '); let contains_letter = text.chars().any(char::is_alphabetic); + for _ in 0..backspaces { + s.queue_edit(Edit::Backspace, scope); + } s.queue_edit(Edit::Append(text), scope); Ok((trailing_space, contains_letter)) }) @@ -573,6 +595,14 @@ mod tests { }); } #[test] + fn deleting_needs_an_unbroken_watch_since_the_suggestion_was_chosen() { + assert!(may_delete(0, (3, false), (9, false))); + assert!(may_delete(2, (3, true), (3, true))); + assert!(!may_delete(2, (3, true), (4, true))); + assert!(!may_delete(2, (3, true), (3, false))); + assert!(!may_delete(2, (3, false), (3, false))); + } + #[test] fn activity_observer_retries_after_failure_or_lost_hook() { let state = AtomicU8::new(0); assert!(claim_keyboard_activity_start(&state, false)); diff --git a/src-tauri/src/prediction/worker.rs b/src-tauri/src/prediction/worker.rs index 6f56d742..b3bb6ec6 100644 --- a/src-tauri/src/prediction/worker.rs +++ b/src-tauri/src/prediction/worker.rs @@ -73,6 +73,9 @@ pub enum Response { }, Insert { generation: u64, + /// Characters to delete before typing `text`: the typed prefix, when + /// accepting restores a capital in it. Never more than the prefix. + backspaces: usize, text: Option, foreground: Option, }, @@ -113,7 +116,8 @@ pub struct Engine { revision: u64, snapshot: Option, batch: Option, - suffixes: Vec, + /// Per suggestion: how many typed characters to delete, then the text. + inserts: Vec<(usize, String)>, token: u64, case: (Shift, bool, bool), } @@ -133,7 +137,7 @@ impl Engine { revision: 0, snapshot: None, batch: None, - suffixes: Vec::new(), + inserts: Vec::new(), token: 0, case: (Shift::Off, false, false), } @@ -143,7 +147,7 @@ impl Engine { self.clipped = false; self.snapshot = None; self.batch = None; - self.suffixes.clear(); + self.inserts.clear(); } fn stable(&self, target: usize, epoch: u64) -> bool { self.tracked && (self.observe)() == (epoch, true) && (self.foreground)() == Ok(target) @@ -232,7 +236,7 @@ impl Engine { let words = interleave(prediction.words, prediction.phrases); self.status = status; self.token = self.token.wrapping_add(1); - self.suffixes.clear(); + self.inserts.clear(); let mut labels = Vec::new(); for word in words { let offset = word @@ -253,8 +257,25 @@ impl Engine { caps, ) }; - labels.push(ctx.prefix.clone() + &suffix); - self.suffixes.push(suffix + " "); + // A word written in capitals keeps its typed prefix as it is. + let head = if shouting(&ctx.prefix) { + None + } else { + restored(&ctx.prefix, &word[..offset]) + }; + match head { + Some(head) => { + labels.push(format!("{head}{suffix}")); + self.inserts.push(( + ctx.prefix.graphemes(true).count(), + format!("{head}{suffix} "), + )); + } + None => { + labels.push(ctx.prefix.clone() + &suffix); + self.inserts.push((0, suffix + " ")); + } + } } if !self.stable(target, epoch) { self.clear(); @@ -268,17 +289,17 @@ impl Engine { }); self.batch.clone() } - fn accept(&mut self, token: u64, index: usize) -> Option { - if self.batch.as_ref()?.token != token || index >= self.suffixes.len() { + fn accept(&mut self, token: u64, index: usize) -> Option<(usize, String)> { + if self.batch.as_ref()?.token != token || index >= self.inserts.len() { return None; } if !self.stable(self.target?, self.activity) { self.clear(); return None; } - let suffix = self.suffixes[index].clone(); + let insert = self.inserts[index].clone(); self.batch = None; - Some(suffix) + Some(insert) } pub fn respond(&mut self, request: Request) -> Response { match request { @@ -308,19 +329,44 @@ impl Engine { token, revision, index, - } => Response::Insert { - generation, - foreground: self.target, - text: if revision == self.revision { + } => { + let insert = if revision == self.revision { self.accept(token, index) } else { self.clear(); None - }, - }, + }; + let (backspaces, text) = insert.map_or((0, None), |(n, text)| (n, Some(text))); + Response::Insert { + generation, + foreground: self.target, + backspaces, + text, + } + } } } } +/// The typed prefix as it should read once the word is accepted. A capital +/// the word has where the person typed a lowercase letter is restored, as in +/// "lon" for London or "i" for I'm. A capital the person typed is kept. +/// `None` when nothing would change, which is the usual case and deletes +/// nothing. +fn restored(typed: &str, head: &str) -> Option { + let merged: String = typed + .chars() + .zip(head.chars()) + .map(|(t, w)| { + if t.is_lowercase() && w.is_uppercase() { + w + } else { + t + } + }) + .collect(); + (merged != typed).then_some(merged) +} + /// A typed prefix of two or more letters, all capitals, is a word being /// written in capitals: its completion continues that way whatever the /// modifiers now say, so the label shows exactly what will be inserted. @@ -462,7 +508,7 @@ mod tests { .query(vec![append("wa")], 1, Shift::Off, false, false) .unwrap(); assert_eq!(b.words, w(&["water", "water is", "waffle", "walk"])); - assert_eq!(e.accept(b.token, 1).unwrap(), "ter is "); + assert_eq!(e.accept(b.token, 1).unwrap(), (0, "ter is ".to_owned())); let upper = e.query(vec![], 1, Shift::Off, true, false).unwrap(); assert_eq!(upper.words[1], "waTER IS"); } @@ -477,7 +523,8 @@ mod tests { .query(vec![append("a")], 2, Shift::Off, false, false) .unwrap(); assert_eq!(b.words[0], "water"); - let suffix = e.accept(b.token, 0).unwrap(); + let (deleted, suffix) = e.accept(b.token, 0).unwrap(); + assert_eq!(deleted, 0); assert_eq!(suffix, "ter "); assert!(e.accept(b.token, 0).is_none()); e.query(vec![append(&suffix)], 3, Shift::Off, false, false); @@ -598,6 +645,73 @@ mod tests { assert_ne!(upper.token, b.token); } #[test] + fn accepting_restores_a_capital_the_typed_prefix_lacks_and_nothing_else() { + let w = |s: &[&str]| s.iter().map(|s| (*s).to_owned()).collect::>(); + assert_eq!(restored("lon", "Lon").as_deref(), Some("Lon")); + assert_eq!(restored("i’", "I’").as_deref(), Some("I’")); + assert_eq!(restored("whats", "Whats").as_deref(), Some("Whats")); + assert_eq!(restored("Wa", "wa"), None); + assert_eq!(restored("wa", "wa"), None); + assert_eq!(restored("", ""), None); + let mut e = engine(); + // The label reads as the text will, and only the prefix is deleted. + let b = e + .query(vec![append("wh")], 1, Shift::Off, false, false) + .unwrap(); + assert_eq!(b.words, w(&["WhatsApp", "WhatsApp is"])); + assert_eq!(e.accept(b.token, 0).unwrap(), (2, "WhatsApp ".to_owned())); + // The edits the service queues afterwards leave the buffer reading + // as the screen does. + e.query( + vec![ + edit(Edit::Backspace), + edit(Edit::Backspace), + append("WhatsApp "), + ], + 2, + Shift::Off, + false, + false, + ); + assert_eq!(e.buffer, "WhatsApp "); + // A capital the person typed is kept, and nothing is deleted. + let b = e + .query( + vec![edit(Edit::Reset), append("Wh")], + 3, + Shift::Off, + false, + false, + ) + .unwrap(); + assert_eq!(b.words, w(&["WhatsApp", "WhatsApp is"])); + assert_eq!(e.accept(b.token, 0).unwrap(), (0, "atsApp ".to_owned())); + // A word in capitals is never retyped. + let b = e + .query( + vec![edit(Edit::Reset), append("WH")], + 4, + Shift::Off, + false, + false, + ) + .unwrap(); + assert_eq!(e.accept(b.token, 0).unwrap(), (0, "ATSAPP ".to_owned())); + match e.respond(Request::Accept { + generation: 0, + token: 0, + revision: 99, + index: 0, + }) { + Response::Insert { + backspaces, text, .. + } => { + assert_eq!((backspaces, text), (0, None)); + } + _ => panic!("expected an insert response"), + } + } + #[test] fn an_all_caps_prefix_continues_in_capitals_whatever_the_modifiers() { let mut e = engine(); e.query(vec![append("WA")], 1, Shift::Off, false, false) diff --git a/src-tauri/src/scan_executor.rs b/src-tauri/src/scan_executor.rs index 5b318c0e..c4d0dba9 100644 --- a/src-tauri/src/scan_executor.rs +++ b/src-tauri/src/scan_executor.rs @@ -168,20 +168,45 @@ pub fn toggle_mouse_drag(point: (i32, i32)) -> Result<(), String> { result }) } -pub fn prediction_text(text: &str) -> Result<(), String> { - if text.chars().count() > 64 || !crate::scan_host::modifiers_released() { +/// The most typed characters an accepted suggestion may delete. A prefix is +/// part of one word, so anything longer is refused rather than trusted. +const PREDICTION_BACKSPACES: usize = 32; + +/// Types an accepted suggestion, first deleting `backspaces` characters when +/// the worker is restoring a capital in the typed prefix. The worker only +/// ever asks to delete the prefix, which Switchify typed itself. +pub fn prediction_replace(backspaces: usize, text: &str) -> Result<(), String> { + if !crate::scan_host::modifiers_released() { return Err("Prediction input is unavailable.".into()); } INPUT.with(|slot| { let mut slot = slot.borrow_mut(); let input = slot.as_mut().ok_or("Prediction input is unavailable.")?; - if input.has_active_drag() || input.has_active_switch_session() { + if input.has_active_switch_session() { return Err("Prediction input is unavailable.".into()); } - input.release_all()?; - input.type_text(text) + replace_text(input, backspaces, text) }) } + +fn replace_text( + input: &mut DesktopInput, + backspaces: usize, + text: &str, +) -> Result<(), String> { + if text.chars().count() > 64 || backspaces > PREDICTION_BACKSPACES || input.has_active_drag() { + return Err("Prediction input is unavailable.".into()); + } + input.release_all()?; + let result = (0..backspaces) + .try_for_each(|_| input.scan_chord(&["Backspace"], None)) + .and_then(|()| input.type_text(text)); + if result.is_err() { + // Nothing stays held after a failure part-way through. + let _ = input.release_all(); + } + result +} pub fn cleanup() -> Result<(), String> { INPUT.with(|slot| { slot.borrow_mut() @@ -416,6 +441,47 @@ mod tests { } } #[test] + fn accepted_predictions_delete_only_what_they_are_told_then_type() { + let mut input = DesktopInput::new(Fake::default()); + replace_text(&mut input, 0, "ter ").unwrap(); + assert_eq!(input.injector.events, ["text ter "]); + input.injector.events.clear(); + replace_text(&mut input, 2, "WhatsApp ").unwrap(); + assert_eq!( + input.injector.events, + [ + "key Backspace true", + "key Backspace false", + "key Backspace true", + "key Backspace false", + "text WhatsApp " + ] + ); + input.injector.events.clear(); + assert!(replace_text(&mut input, PREDICTION_BACKSPACES + 1, "x ").is_err()); + assert!(replace_text(&mut input, 0, &"x".repeat(65)).is_err()); + assert!(input.injector.events.is_empty()); + // Typing fails after the deletion: the error is reported and no key + // is left down. + input.injector.fail_text = true; + assert!(replace_text(&mut input, 1, "I'm ").is_err()); + assert_eq!( + input.injector.events, + ["key Backspace true", "key Backspace false", "text I'm "] + ); + let downs = |events: &[String], down: &str| { + events + .iter() + .filter(|e| e.starts_with("key ") && e.ends_with(down)) + .count() + }; + assert_eq!( + downs(&input.injector.events, "true"), + downs(&input.injector.events, "false") + ); + assert!(!input.has_active_drag()); + } + #[test] fn opening_keyboard_releases_a_drag_without_clicking_or_typing() { let mut input = DesktopInput::new(Fake::default()); execute(&mut input, Request::DragStart((10, 20)), true).unwrap();