Measure neural prediction optimizations and FUTO native decoder

[enaboapps]

Result

FUTO's native decoder reached 12.83 ms p95 and 16.77 ms maximum with four threads on Windows, with exactly the same ordered suggestions as its one-thread run. Its three-result correction behavior is not yet a suitable direct replacement: one-letter top-three accuracy was 12.50%, and 789 suggestions across the run needed exact-prefix filtering. There were 1,279 supported inferences, one explicitly unsupported prefix and no decode failures per run.

Host-optimized Q8 reduced SmolLM2 cache-miss p95 from F32's 226.72 ms to 101.78 ms, retaining 37.50% early top-five accuracy on these synthetic fixtures. Batching improved cache-hit latency but made Q8 context preparation slower. No neural variant met 20 ms. Do not promote any model from this experiment.

Change

Adds context reuse, candidate batching, locally converted Q8 weights, explicit host-native build comparisons and an isolated FUTO native probe. Every completed neural mode measured 1,280 misses and 1,280 cache hits, with ordered-result equality checked for each pair. Cache-hit probes are immediate repeats, not a realistic typing trace. A failed initial portable Q8 batch attempt and missing supplementary memory metadata are documented rather than hidden.

This branch starts from current main and includes the unmerged experiment from #14 as a prerequisite. Production dependencies, prediction APIs, model files and Switchify PC remain unchanged. No model weights or third-party decoder sources are redistributed. macOS runtime performance and Android output equivalence remain untested.

Validation

• Root formatting, Clippy with warnings denied and 22 Rust tests passed.
• Separate neural crate formatting, Clippy and six Rust tests passed; 14 Python tests passed.
• F32 conversion prefill/continuation parity passed before quantized scoring.
• F32 batching and host-native Q8 batching each preserved their sequential counterpart's ordered outputs.
• Final FUTO Windows build and both full 1/4-thread runs passed. All native decode calls fail closed; explicit success markers and parser regression tests address the independent review finding.
• Production dependency audit passed. The isolated Candle audit retains one documented unmaintained-package exception, RUSTSEC-2024-0436 for paste.
• Independent review of latest head 9fcd9eb found no outstanding actionable findings. All latest-head CI passed: production checks, database validation and packaged smoke tests, plus experimental checks, audit and FUTO Windows build.

See docs/neural-latency-results.md and docs/neural-latency-results.json for measurements and limitations. Reproduction commands are in docs/neural-latency-protocol.md and docs/futo-probe-protocol.md.

Closes #15.