Measure neural prediction optimizations and FUTO native decoder

.github/workflows/neural-experiment.yml

@@ -0,0 +1,55 @@
+name: Neural experiment
+on:
+  pull_request:
+  workflow_dispatch:
+permissions:
+  contents: read
+concurrency:
+  group: neural-experiment-${{ github.ref }}
+  cancel-in-progress: true
+jobs:
+  neural-checks:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+      - uses: dtolnay/rust-toolchain@4716b85f2fac3e324e64fa2810f6b5c3905760a5
+        with:
+          components: rustfmt, clippy
+      - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6
+        with:
+          workspaces: experiments/neural -> ../../target/neural
+      - run: cargo fmt --manifest-path experiments/neural/Cargo.toml -- --check
+      - run: cargo clippy --locked --manifest-path experiments/neural/Cargo.toml --target-dir target/neural --all-targets -- -D warnings
+      - run: cargo test --locked --manifest-path experiments/neural/Cargo.toml --target-dir target/neural
+  neural-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+      - uses: dtolnay/rust-toolchain@4716b85f2fac3e324e64fa2810f6b5c3905760a5
+      - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6
+      - run: cargo install cargo-audit --locked --version 0.22.0
+      # Candle transitively uses this unmaintained macro crate; see the protocol.
+      - run: cargo audit --file experiments/neural/Cargo.lock --deny warnings --ignore RUSTSEC-2024-0436
+  futo-windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+      - name: Prepare pinned native source
+        shell: pwsh
+        run: |
+          New-Item -ItemType Directory -Force artifacts | Out-Null
+          curl.exe -fsSL --retry 3 https://codeload.github.com/futo-org/android-keyboard/zip/70a5d390c505a6bbcc4e14966e5628e43ca3f1fc -o artifacts/futo-source.zip
+          if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+          python scripts/prepare_futo_probe.py --archive artifacts/futo-source.zip --output artifacts/futo-probe
+      - name: Build isolated native probe
+        shell: pwsh
+        run: |
+          $sourcePath = (Resolve-Path artifacts/futo-probe/source).Path.Replace('\', '/')
+          $preparedPath = (Resolve-Path artifacts/futo-probe).Path.Replace('\', '/')
+          cmake -S experiments/futo -B artifacts/futo-build "-DFUTO_SOURCE=$sourcePath" "-DFUTO_PREPARED=$preparedPath"
+          if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+          cmake --build artifacts/futo-build --config Release --target futo-probe --parallel 2

README.md

@@ -170,6 +170,15 @@ remain. English alone is supported.
 
 The [AAC experiment protocol](docs/aac-experiment.md) and earlier
 [conversational results](docs/quality-results.md) document model-selection history.
+Switchify is a general-purpose typing app. The separate
+[neural experiment](docs/general-neural-protocol.md) tests messages, email,
+documents and search-style text equally, without changing the production model.
+Its [results](docs/general-neural-results.md) show the measured accuracy gain
+and CPU latency cost on synthetic fixtures.
+The [speed follow-up](docs/neural-latency-results.md) measures caching, batching,
+Q8/SIMD builds and a separate FUTO native decoder. FUTO met the initial speed
+target with four threads but still needs completion-specific evaluation;
+none of these experiments changes the production predictor.
 The `prepare` and `evaluate` commands retain the earlier comparison workflows;
 `prepare` alone does not prepare the promoted production corpus. Use the commands
 above for production. A new model-selection exercise needs a new protocol; these

docs/futo-investigation.md

@@ -0,0 +1,39 @@
+# FUTO keyboard model investigation
+
+Inspected on 2026-10-03. FUTO is a relevant general-purpose keyboard reference.
+The subsequent [native speed probe](neural-latency-results.md) benchmarks it on
+Windows. It remains separate from production and has not been integrated.
+
+The official source at commit
+`70a5d390c505a6bbcc4e14966e5628e43ca3f1fc` selects the English `ml4_q6_k` model.
+The resource is available from the official `keyboard-large-resources` submodule
+at commit `d87d9dbdf3966bbe18413be375dab2f6c7bbdfdd`, path
+`raw/ml4_q6_k.gguf`. GitLab's file metadata reports 30,662,880 bytes and content
+SHA-256 `6545c1c9ef2d76e9bfb87ad4fcf2061889513af84fcf30d907412be7fcdedb7b`.
+The subsequent probe downloaded the model and verified this hash locally. Its name
+identifies the Q6_K model; no comparison with the F32 SmolLM2 timings is implied.
+
+The model is not an arbitrary chat GGUF. The Android loader requires KeyboardLM
+metadata and recognized feature flags. The native prediction path includes
+special word-boundary behavior, candidate search, capitalization filters and
+`<XBU>`, `<XBC>`, `<XEC>` / character tokens for supported correction modes.
+Loading weights with a generic GGUF runtime alone would not reproduce FUTO.
+
+A standalone Windows/macOS benchmark appears feasible by isolating the native
+prediction implementation and its matching runtime behind a Rust adapter.
+That is an engineering assessment from the source, not a tested build. A useful
+next step is to reproduce its next-word and partial-word outputs on fixed
+fixtures before comparing latency or accuracy. Its correction mode must not
+silently violate Switchify's exact-prefix completion contract.
+
+The repository identifies its code licence as FUTO Source First 1.1. Model terms
+also need checking before any redistribution; this investigation neither copies
+FUTO code into the MIT library nor distributes its model.
+
+## Sources
+
+- [Pinned model loader](https://github.com/futo-org/android-keyboard/blob/70a5d390c505a6bbcc4e14966e5628e43ca3f1fc/java/src/org/futo/inputmethod/latin/xlm/ModelPaths.kt)
+- [Pinned native prediction path](https://github.com/futo-org/android-keyboard/blob/70a5d390c505a6bbcc4e14966e5628e43ca3f1fc/native/jni/org_futo_inputmethod_latin_xlm_LanguageModel.cpp)
+- [Pinned resource submodule declarations](https://github.com/futo-org/android-keyboard/blob/70a5d390c505a6bbcc4e14966e5628e43ca3f1fc/.gitmodules)
+- [Official model resource](https://gitlab.futo.org/keyboard/keyboard-large-resources/-/blob/d87d9dbdf3966bbe18413be375dab2f6c7bbdfdd/raw/ml4_q6_k.gguf)
+- [Pinned code licence](https://github.com/futo-org/android-keyboard/blob/70a5d390c505a6bbcc4e14966e5628e43ca3f1fc/LICENSE.md)

docs/futo-probe-protocol.md

@@ -0,0 +1,63 @@
+# FUTO native probe
+
+This is a local research harness, not a production Rust adapter or an Android
+equivalence certification. Upstream code and weights stay in ignored artifacts.
+Do not distribute either through the MIT prediction library.
+
+Use the pinned source and model from `futo-investigation.md`. The preparation
+script verifies the source archive SHA-256 and extracts its native code locally.
+The model SHA-256 is checked before execution. Keep upstream licence notices
+with the downloaded source. Redistribution terms remain a separate review.
+
+The harness keeps upstream `LanguageModelState`, `PredictNextWord`,
+`PredictCorrection`, tokenization, logit transforms, sampling and three-result
+limit. Typed ASCII prefixes use upstream's no-coordinate fallback with one
+character weight set to one; no pointer coordinates are synthesized or injected.
+Non-ASCII/nonletter prefixes are reported as unsupported, not silently dropped.
+Corrections may violate the typed prefix. Report native top-one/top-three and
+exact-prefix-filtered top-one/top-three separately, without backfill. Do not call
+these five-slot results or silently compare them as equivalent to the reranker.
+
+Use the same 1,280 Rust-exported synthetic queries and one initial warm-up query.
+Keep upstream's single-thread setting. Record load time, internal query latency,
+model bytes, peak working set and failures. A 30-minute process timeout terminates
+a stuck native run. No keyboard/pointer input, learning or network inference.
+
+Mechanical local source changes remove JNI includes/entry points, replace one
+variable-length C++ array with `std::vector`, and guard empty/whitespace trimming
+and unsigned character classification.
+Two bundled Abseil include paths are adjusted to the repository's layout.
+The harness retains the JNI wrapper's exact-match preference before sorting.
+All four decode calls are checked, and upstream empty/default error returns
+raise a native error. Each completed query emits an explicit success marker;
+the runner rejects missing/error markers and non-finite timings. These checks
+change failure handling only, never successful rankings.
+No weights, sampling thresholds or ranking are tuned against the fixtures.
+The native state maintains its own context reuse across the fixed query order.
+These timings are separate from the explicit hit/miss Candle measurements.
+
+Also run a fixed four-thread comparison via `llama_set_n_threads`, matching the
+Candle thread budget. Use `--threads 4` with a new output directory. This changes
+execution parallelism only; compare ordered-output digests with the one-thread
+run. Neither threading choice changes the native three-result policy.
+
+Preparation and build:
+
+```sh
+curl -fL https://codeload.github.com/futo-org/android-keyboard/zip/70a5d390c505a6bbcc4e14966e5628e43ca3f1fc -o futo-source.zip
+python scripts/prepare_futo_probe.py --archive futo-source.zip --output artifacts/futo-probe
+cmake -S experiments/futo -B artifacts/futo-build -DFUTO_SOURCE=/absolute/path/artifacts/futo-probe/source -DFUTO_PREPARED=/absolute/path/artifacts/futo-probe
+cmake --build artifacts/futo-build --config Release --target futo-probe
+```
+
+Export queries using the neural executable's normal pinned baseline/model/
+manifest/fixtures/training arguments, plus `--mode baseline --export-queries
+--output artifacts/futo-queries.json`. Then run:
+
+```sh
+python scripts/futo_experiment.py --executable /path/to/futo-probe --model /path/to/ml4_q6_k.gguf --queries artifacts/futo-queries.json --output artifacts/futo-results
+```
+
+Use an empty output directory. Raw fixture outputs remain local; commit only
+aggregate measurements and provenance. A failed native build/run must be
+reported as such, with no invented accuracy or latency.

docs/general-neural-protocol.md

@@ -0,0 +1,93 @@
+# General-purpose neural experiment
+
+Frozen 2026-10-03 before scoring. Switchify is a general-purpose typing app.
+Messages, email, documents and search-style text receive equal weight. AAC is
+not a selection objective. The existing released model is a control, including
+its historical training mixture; this experiment does not endorse that mixture.
+
+## Fixed comparison
+
+Compare the released predictor with a SmolLM2-135M base-model reranker using
+Candle 0.11.0, CPU F32, four Rayon threads and no GPU, personal learning or
+network access during inference. This initial implementation is unquantized.
+Weights, tokenizer and configuration are pinned by SHA-256 and upstream commit.
+The experiment is a separate, unpublished Rust crate. Production dependencies,
+APIs and model formats stay unchanged.
+
+Both modes request the existing engine's top eight prefix-matching words. The
+control returns its first five. The neural mode scores each entire candidate's
+token sequence, followed by the probability of a word boundary, and returns
+the five highest scores. It uses BOS plus the last 63 tokens of preceding text
+within the current sentence. It does not score only the first subword, normalize
+by word length, mix frequency scales or learn from targets. Ties retain baseline
+order. Each query starts with an empty KV cache; the context cache is shared only
+between candidate branches within that query. Prefixes filter candidates before
+neural scoring. This is a bounded reranker, not unrestricted neural generation;
+report top-eight candidate coverage as its accuracy ceiling.
+
+Boundary probability sums the next-token probabilities for EOS and tokens whose
+decoded text starts with whitespace, a number, or one of `. ! ? ; : , ( ) [ ] { }
+" - /`. Apostrophe continuations remain inside a word. This is an explicit
+approximation, not an exact tokenizer-independent word probability.
+
+## Workload and decision
+
+The checked-in fixtures contain 24 newly authored synthetic examples in each
+of four domains. They are diagnostic fixtures, not a representative user corpus
+or independent evidence of real-world accuracy. Normalize with the existing
+Unicode/sentence tokenizer. Reject exact full-sentence overlap with the pinned
+production training file. Unknown overlap with neural pretraining remains.
+
+For each domain and each prefix length 0 through 4 Unicode graphemes, hash-sort
+eligible target positions and take 64 distinct queries, excluding words already
+complete at that prefix. The fixed workload has 1,280 queries per mode. Duplicate
+preceding-text/prefix/target triples within a cell are removed. All words before
+the target are supplied identically; each predictor uses its supported context.
+Lowercasing and removing punctuation limit conclusions about realistic writing.
+
+Report top-one, top-five and candidate coverage per domain/prefix, cold load,
+warm median, p95, maximum, failures, model file size and process peak memory.
+Prime once without timing. Abort on inference errors instead of silently falling
+back. A smaller `--per-cell` run is calibration only, not a result used to tune
+the model, shortlist or ranking rule.
+
+A promising result requires higher mean top-five accuracy over prefixes 0,1,2
+and all four domains, no domain/prefix loss above one percentage point there,
+and warm p95 below 20 ms. Do not tune after seeing fixture scores or promote a
+model automatically. Describe tradeoffs even if the criterion fails. Windows
+is the local benchmark platform; macOS build/tests do not establish macOS speed.
+
+Investigate FUTO's official model integration separately. Do not equate generic
+GGUF loading with implementing its custom keyboard prompts and tokenizer rules.
+No FUTO accuracy claim without running its actual prediction path.
+
+## Reproduction
+
+Use Rust 1.97.1 and Python 3. Prepare the unchanged production partitions with
+`python scripts/aac_experiment.py --prepare-only` if they are not cached. Extract
+`english.sqlite` from the model bundle on the
+[v0.1.0 release](https://github.com/switchifyapp/switchify-prediction/releases/tag/v0.1.0).
+Then run from the repository root:
+
+```sh
+cargo build --release --locked --manifest-path experiments/neural/Cargo.toml --target-dir target/neural
+python scripts/general_neural_experiment.py --baseline /path/to/english.sqlite --output artifacts/general-neural
+```
+
+The Python command downloads and verifies the pinned model before launching the
+offline Rust executables. Use a fresh output directory for each full run.
+It records aggregate results and Windows process peak working set. Other
+platforms report memory as unavailable, not zero. No captured user text is used.
+
+Run formatting, Clippy and tests with the separate crate's manifest in addition
+to the repository checks. Its lockfile contains Candle's transitive `paste`
+1.0.15 dependency, which has the unmaintained-package advisory RUSTSEC-2024-0436.
+The production dependency audit remains unchanged. For this unpublished
+experiment only, audit the separate lockfile with that single advisory exception:
+
+```sh
+cargo audit --file experiments/neural/Cargo.lock --deny warnings --ignore RUSTSEC-2024-0436
+```
+
+This exception does not assert that the dependency is maintained. Reassess the
+dependency before considering production integration.