From 11e5de1c7f758ec42543dae1fd02eb2f80f1c1a5 Mon Sep 17 00:00:00 2001 From: Owen McGirr Date: Sat, 3 Oct 2026 09:36:31 +0100 Subject: [PATCH 1/4] Add experimental legacy slot filling and frozen comparison --- README.md | 20 ++ docs/legacy-comparison-protocol.md | 35 +++ scripts/compare_legacy.py | 90 ++++++++ scripts/test_production.py | 2 +- src/experimental.rs | 345 +++++++++++++++++++++++++++++ src/lib.rs | 1 + src/main.rs | 82 +++++++ tests/experimental.rs | 143 ++++++++++++ 8 files changed, 717 insertions(+), 1 deletion(-) create mode 100644 docs/legacy-comparison-protocol.md create mode 100644 scripts/compare_legacy.py create mode 100644 src/experimental.rs create mode 100644 tests/experimental.rs diff --git a/README.md b/README.md index 9c0633f..4cc678c 100644 --- a/README.md +++ b/README.md @@ -180,3 +180,23 @@ New code is MIT licensed. Corpus licences are separate: retain distributions. [Production guidance](docs/production.md) covers the recorded licensing evidence, OANC's differing historical/current notices, private-data handling, backup, upgrades, rollback and releases. + +## Experimental legacy backfill + +The opt-in `experimental-predict` command returns ordered `{word, source}` records. +Use `--mode newer --baseline english.sqlite`, `--mode original --legacy old.sqlite`, +or `--mode combined --baseline english.sqlite --legacy old.sqlite`, with the usual +`--before`, `--prefix`, `--limit` and `--min-chars` arguments. Combined mode preserves +all newer suggestions and fills only unused slots. Scores are never mixed. No +personal database is opened. Existing APIs and the default model are unchanged. + +`experimental::LegacyPredictor` and `experimental::CombinedPredictor` provide the +same opt-in behavior in Rust. Load once on a worker thread and reuse; predictions +perform no file access. Missing/corrupt sources fail opening the requested mode. +The original source is not distributed by this repository. Normalization follows +the newer tokenizer; invalid single words and words longer than 48 characters +are skipped. This differs intentionally from the original lowercase-only filter. + +For the reproducible comparison, see [the frozen protocol](docs/legacy-comparison-protocol.md) +and [results](docs/legacy-comparison-results.md). No desktop integration or model +promotion is part of this experiment. diff --git a/docs/legacy-comparison-protocol.md b/docs/legacy-comparison-protocol.md new file mode 100644 index 0000000..236dbff --- /dev/null +++ b/docs/legacy-comparison-protocol.md @@ -0,0 +1,35 @@ +# Legacy slot-fill protocol v1 + +Frozen before scoring: preserve every newer result and append normalized unique legacy results only to empty slots. Never interpolate scores, learn evaluation text, or tune on these partitions. Compare original, newer and combined at prefixes 0 through 4 with five slots and minimum prefix zero. Give all modes identical preceding text, preserving the original three-word context. Use all six existing AAC/general/conversation development/test partitions and record their hashes. Report OOV subsets and legacy fill rates alongside accuracy and resource costs. Warm timing uses at least 1,000 queries per mode, with a 20 ms p95 target. Accuracy must not regress against newer; no automatic promotion. + +The legacy database is recovered locally from switchify-pc tag v1.0.0-rc.14, path src-tauri/resources/WordData2017051601.db, SHA-256 dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318. It is never committed, packaged or uploaded. Only BASE_FREQUENCY is read. Ranking follows rc.14 lookup/database_reference: longest context first, frequency descending then target ID ascending, then unigram backoff. Modern Unicode normalization and single-word validation are intentional differences from the historical adapter. Source training overlap is unknown, so results are regression comparisons, not unseen-data estimates. + +## Reproduce + +Use Rust 1.97.1, Python 3.9+, curl, and Git. On Windows clone with +`git -c core.autocrlf=false clone ...` so checksum-pinned files retain exact bytes. +Obtain the v0.1.0 model bundle and verify it using its `verify_bundle.py`. +Recover the legacy database from an existing Switchify PC checkout without +changing that checkout, using Python binary output: + +```python +import subprocess +from pathlib import Path +Path("old.sqlite").write_bytes(subprocess.check_output([ + "git", "-C", "../switchify-pc", "show", + "v1.0.0-rc.14:src-tauri/resources/WordData2017051601.db"])) +``` + +```sh +cargo build --release --locked +python scripts/fetch_corpus.py +python scripts/aac_experiment.py --prepare-only +python scripts/compare_legacy.py --exe target/release/switchify-prediction.exe --baseline /path/to/english.sqlite --legacy old.sqlite --output artifacts/legacy-comparison.json +``` + +Omit `.exe` on macOS/Linux. The comparison command verifies both database hashes +and all six partition hashes, runs each mode in a separate process, checks +position preservation and top-five non-regression, and rechecks database hashes +after completion. Reports contain aggregate counts, never captured typing or +personal databases. Peak process memory includes evaluation bookkeeping; payload +bytes exclude allocator overhead. Non-Windows memory polling is a lower bound. diff --git a/scripts/compare_legacy.py b/scripts/compare_legacy.py new file mode 100644 index 0000000..16a5be7 --- /dev/null +++ b/scripts/compare_legacy.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +"""Run frozen slot-fill comparison. Never distributes or trains on the legacy data.""" +import argparse +import hashlib +import json +from pathlib import Path +import platform +import subprocess +import time + +LEGACY_SHA = 'dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318' +BASELINE_SHA = '222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad' + +def sha(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + +def peak_rss(pid): + if platform.system() == 'Windows': + import ctypes + from ctypes import wintypes + class Counters(ctypes.Structure): + _fields_ = [('cb', wintypes.DWORD), ('faults', wintypes.DWORD)] + [(x, ctypes.c_size_t) for x in ['peak', 'working', 'a', 'b', 'c', 'd', 'e', 'f']] + kernel = ctypes.WinDLL('kernel32', use_last_error=True) + kernel.OpenProcess.restype = wintypes.HANDLE + kernel.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD] + kernel.CloseHandle.argtypes = [wintypes.HANDLE] + psapi = ctypes.WinDLL('psapi') + psapi.GetProcessMemoryInfo.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters), wintypes.DWORD] + handle = kernel.OpenProcess(0x410, False, pid) + if not handle: + return None + try: + data = Counters(); data.cb = ctypes.sizeof(data) + if psapi.GetProcessMemoryInfo(handle, ctypes.byref(data), data.cb): + return data.peak + finally: + kernel.CloseHandle(handle) + else: + try: + return int(subprocess.check_output(['ps', '-o', 'rss=', '-p', str(pid)]).strip()) * 1024 + except (ValueError, subprocess.CalledProcessError): + return None + return None + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--exe', type=Path, required=True) + parser.add_argument('--baseline', type=Path, required=True) + parser.add_argument('--legacy', type=Path, required=True) + parser.add_argument('--partitions', type=Path, default=Path('data/aac-oanc/prepared')) + parser.add_argument('--output', type=Path, required=True) + args = parser.parse_args() + if sha(args.legacy) != LEGACY_SHA or sha(args.baseline) != BASELINE_SHA: + raise ValueError('Comparison requires pinned legacy and released baseline databases') + expected = json.loads(Path('production-quality.json').read_text()) + reports = [] + for split in ['dev', 'test']: + for domain in ['aac', 'general', 'conversation']: + path = args.partitions / f'{domain}-{split}.txt' + if sha(path) != expected[split][domain]['candidate']['evaluation_sha256']: + raise ValueError(f'Partition mismatch: {path}') + batch = {} + for mode in ['original', 'newer', 'combined']: + command = [str(args.exe.resolve()), 'compare-legacy', '--mode', mode, '--baseline', str(args.baseline), '--legacy', str(args.legacy), '--input', str(path)] + # Output is small; communicate drains the pipe while polling memory. + import tempfile + with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err: + process = subprocess.Popen(command, stdout=out, stderr=err) + peak = None + while process.poll() is None: + sample = peak_rss(process.pid) + if sample is not None: peak = max(peak or 0, sample) + time.sleep(0.05) + out.seek(0); err.seek(0) + if process.returncode: raise RuntimeError(err.read().decode()) + report = json.loads(out.read()) + report.update(domain=domain, split=split, peak_process_rss_bytes=peak, memory_method='Windows process peak working set' if platform.system() == 'Windows' else 'sampled RSS; lower bound', hardware=platform.platform() + ' ' + platform.processor()) + batch[mode] = report; reports.append(report) + print(domain, split, mode, report['warm_p95_ms'], flush=True) + if batch['combined']['position_violations']: + raise ValueError('Modern positions changed') + for new, combined in zip(batch['newer']['accuracy'], batch['combined']['accuracy']): + if combined['top5'] < new['top5']: raise ValueError('Top-five accuracy regressed') + if sha(args.legacy) != LEGACY_SHA or sha(args.baseline) != BASELINE_SHA: + raise ValueError('Source database changed') + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps({'protocol': 'legacy-slot-fill-v1', 'reports': reports}, indent=2) + '\n') + +if __name__ == '__main__': + main() diff --git a/scripts/test_production.py b/scripts/test_production.py index 2837267..5539562 100644 --- a/scripts/test_production.py +++ b/scripts/test_production.py @@ -98,7 +98,7 @@ def test_generated_or_private_support_files_are_not_packaged(self): allowed = {'.cargo_vcs_info.json', 'Cargo.lock', 'Cargo.toml', 'Cargo.toml.orig', 'LICENSE', 'README.md', 'production-model.json', 'quality-policy.json', 'source-manifest.json', 'src/corpus.rs', 'src/evaluation.rs', 'src/lib.rs', - 'src/main.rs', 'src/production.rs'} + 'src/main.rs', 'src/production.rs', 'src/experimental.rs'} self.assertEqual({name.replace('\\', '/') for name in listed}, allowed) diff --git a/src/experimental.rs b/src/experimental.rs new file mode 100644 index 0000000..276566c --- /dev/null +++ b/src/experimental.rs @@ -0,0 +1,345 @@ +//! Opt-in conservative legacy backfill. No personal data or shared score scale. +use crate::{Error, Options, Predictor, Result, normalize, sentences}; +use rusqlite::{Connection, OpenFlags}; +use serde::Serialize; +use std::{ + collections::{BTreeMap, HashMap, HashSet}, + path::Path, +}; +use unicode_segmentation::UnicodeSegmentation; + +#[derive(Clone, Copy, Debug, Serialize, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +pub enum Source { + Newer, + Original, +} +#[derive(Debug, Serialize, PartialEq, Eq)] +pub struct RankedWord { + pub word: String, + pub source: Source, +} + +/// Frequency/ID-ranked lists reproduce rc.14 longest-context-first backoff. +/// Normalization and valid-single-word filtering deliberately follow the new API. +pub struct LegacyPredictor { + words: Vec, + ids: HashMap, + groups: HashMap, Vec>, + prefixes: BTreeMap, +} +impl LegacyPredictor { + pub fn open(path: &Path) -> Result { + let db = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_ONLY)?; + let integrity: String = db.query_row("PRAGMA integrity_check", [], |r| r.get(0))?; + if integrity != "ok" { + return Err(Error::Invalid("legacy integrity check failed".into())); + } + let mut words = Vec::new(); + let mut ids = HashMap::new(); + let mut ranks = HashMap::new(); + let mut all_ids = HashSet::new(); + let mut prefixes = BTreeMap::new(); + let mut stmt = db.prepare( + "SELECT ID,WORD,BASE_FREQUENCY FROM WORDS ORDER BY BASE_FREQUENCY DESC,ID ASC", + )?; + let rows = stmt.query_map([], |r| { + Ok(( + r.get::<_, i64>(0)?, + r.get::<_, String>(1)?, + r.get::<_, i64>(2)?, + )) + })?; + for row in rows { + let (id, text, frequency) = row?; + if id <= 0 || frequency < 0 || !all_ids.insert(id) { + return Err(Error::Invalid("invalid legacy word record".into())); + } + let word = normalize(&text); + if word.chars().count() > 48 || sentences(&word) != vec![vec![word.clone()]] { + continue; + } + let rank = words.len(); + ids.entry(word.clone()).or_insert(id); + prefixes.entry(word.clone()).or_insert(rank); + ranks.insert(id, rank); + words.push(word); + } + if words.is_empty() { + return Err(Error::Invalid("empty legacy vocabulary".into())); + } + let mut groups: HashMap, Vec> = HashMap::new(); + for (n, table) in [(2, "BIGRAMS"), (3, "TRIGRAMS"), (4, "QUADGRAMS")] { + let fields = (1..=n) + .map(|i| format!("ID{i}")) + .collect::>() + .join(","); + let mut stmt = db.prepare(&format!( + "SELECT {fields},BASE_FREQUENCY FROM {table} ORDER BY BASE_FREQUENCY DESC,ID{n} ASC" + ))?; + let mut rows = stmt.query([])?; + while let Some(row) = rows.next()? { + let mut key = Vec::new(); + for i in 0..n { + key.push(row.get::<_, i64>(i)?); + } + if row.get::<_, i64>(n)? < 0 || key.iter().any(|id| !all_ids.contains(id)) { + return Err(Error::Invalid("invalid legacy n-gram".into())); + } + let target = key.pop().unwrap(); + if let Some(&rank) = ranks.get(&target) { + groups.entry(key).or_default().push(rank); + } + } + } + Ok(Self { + words, + ids, + groups, + prefixes, + }) + } + pub fn vocabulary(&self) -> impl Iterator { + self.prefixes.keys().map(String::as_str) + } + /// Key/list payload only, not allocator overhead or process RSS. + pub fn model_payload_bytes(&self) -> usize { + self.words.iter().map(String::len).sum::() + + self + .groups + .iter() + .map(|(k, v)| k.len() * 8 + v.len() * std::mem::size_of::()) + .sum::() + + self.ids.keys().map(|s| s.len() + 8).sum::() + + self + .prefixes + .keys() + .map(|s| s.len() + std::mem::size_of::()) + .sum::() + } + pub fn predict(&self, before: &str, prefix: &str, options: Options) -> Vec { + let prefix = normalize(prefix); + if options.limit == 0 || prefix.graphemes(true).count() < options.min_chars { + return Vec::new(); + } + let tail = before + .rsplit(['.', '!', '?', '\n', '\r']) + .next() + .unwrap_or(""); + let context = sentences(tail).pop().unwrap_or_default(); + let mut result = Vec::new(); + let mut seen = HashSet::new(); + let mut push = |rank: usize| { + let word = &self.words[rank]; + if word.starts_with(&prefix) && seen.insert(word.clone()) { + result.push(RankedWord { + word: word.clone(), + source: Source::Original, + }); + } + result.len() >= options.limit + }; + if !options.unigram_only { + for n in (1..=context.len().min(3)).rev() { + let key: Option> = context[context.len() - n..] + .iter() + .map(|w| self.ids.get(w).copied()) + .collect(); + if let Some(list) = key.and_then(|k| self.groups.get(&k)) { + for &rank in list { + if push(rank) { + return result; + } + } + } + } + } + let mut ranks: Vec<_> = self + .prefixes + .range(prefix.clone()..) + .take_while(|(w, _)| w.starts_with(&prefix)) + .map(|(_, r)| *r) + .collect(); + ranks.sort_unstable(); + for rank in ranks { + if push(rank) { + break; + } + } + result + } +} + +pub struct CombinedPredictor { + newer: Predictor, + legacy: LegacyPredictor, +} +impl CombinedPredictor { + pub fn open(baseline: &Path, legacy: &Path) -> Result { + Ok(Self { + newer: Predictor::open(baseline, None)?, + legacy: LegacyPredictor::open(legacy)?, + }) + } + pub fn predict(&self, before: &str, prefix: &str, options: Options) -> Vec { + combine(&self.newer, &self.legacy, before, prefix, options) + } +} +pub fn combine( + newer: &Predictor, + legacy: &LegacyPredictor, + before: &str, + prefix: &str, + options: Options, +) -> Vec { + let mut result: Vec<_> = newer + .predict(before, prefix, options) + .into_iter() + .map(|s| RankedWord { + word: s.word, + source: Source::Newer, + }) + .collect(); + if result.len() < options.limit { + let mut seen: HashSet<_> = result.iter().map(|r| normalize(&r.word)).collect(); + // Request the full limit so overlap cannot hide eligible fallback words. + for item in legacy.predict(before, prefix, options) { + if seen.insert(normalize(&item.word)) { + result.push(item); + } + if result.len() == options.limit { + break; + } + } + } + result +} + +/// One process per mode makes peak process memory comparable. Baseline vocabulary +/// is read separately for OOV accounting, without loading an unused predictor. +pub fn compare( + mode: &str, + baseline: &Path, + legacy: &Path, + input: &Path, +) -> Result { + use std::{fs, time::Instant}; + if !matches!(mode, "original" | "newer" | "combined") { + return Err(Error::Invalid("unknown comparison mode".into())); + } + let text = fs::read_to_string(input)?; + let tests: std::collections::BTreeSet<_> = sentences(&text).into_iter().collect(); + if tests.is_empty() { + return Err(Error::Invalid("empty evaluation partition".into())); + } + let modern_vocabulary: HashSet = { + let db = Connection::open_with_flags(baseline, OpenFlags::SQLITE_OPEN_READ_ONLY)?; + db.prepare("SELECT word FROM vocabulary")? + .query_map([], |r| r.get(0))? + .collect::>()? + }; + let start = Instant::now(); + let newer = if mode != "original" { + Some(Predictor::open(baseline, None)?) + } else { + None + }; + let original = if mode != "newer" { + Some(LegacyPredictor::open(legacy)?) + } else { + None + }; + let cold = start.elapsed().as_secs_f64() * 1000.0; + let vocabulary: HashSet<_> = modern_vocabulary + .iter() + .filter(|_| mode != "original") + .cloned() + .chain( + original + .as_ref() + .into_iter() + .flat_map(|p| p.vocabulary().map(str::to_owned)), + ) + .collect(); + let opts = Options { + limit: 5, + min_chars: 0, + unigram_only: false, + }; + let predict = |before: &str, prefix: &str| -> Vec { + match (newer.as_ref(), original.as_ref()) { + (Some(n), Some(o)) => combine(n, o, before, prefix, opts), + (Some(n), None) => n + .predict(before, prefix, opts) + .into_iter() + .map(|s| RankedWord { + word: s.word, + source: Source::Newer, + }) + .collect(), + (None, Some(o)) => o.predict(before, prefix, opts), + _ => unreachable!(), + } + }; + predict("I need ", "he"); + let mut counts = [[0usize; 8]; 5]; // queries, top1, top5, covered, OOV queries/top1/top5, filled queries + let mut timings = Vec::new(); + let mut queries = Vec::new(); + let mut position_violations = 0usize; + for words in &tests { + for (i, target) in words.iter().enumerate() { + let before = words[..i].join(" "); + let graphemes: Vec<_> = target.graphemes(true).collect(); + for n in 0..=4.min(graphemes.len().saturating_sub(1)) { + let prefix = graphemes[..n].concat(); + let start = Instant::now(); + let result = predict(&before, &prefix); + timings.push(start.elapsed().as_secs_f64() * 1000.0); + let rank = result.iter().position(|r| r.word == *target); + let c = &mut counts[n]; + c[0] += 1; + c[1] += usize::from(rank == Some(0)); + c[2] += usize::from(rank.is_some()); + c[3] += usize::from(vocabulary.contains(target)); + if !modern_vocabulary.contains(target) { + c[4] += 1; + c[5] += usize::from(rank == Some(0)); + c[6] += usize::from(rank.is_some()); + } + c[7] += usize::from(result.iter().any(|r| r.source == Source::Original)); + if mode == "combined" { + let modern = newer.as_ref().unwrap().predict(&before, &prefix, opts); + if !modern + .iter() + .zip(&result) + .all(|(a, b)| a.word == b.word && b.source == Source::Newer) + || result.len() < modern.len() + { + position_violations += 1; + } + } + queries.push((before.clone(), prefix)); + } + } + } + while timings.len() < 1000 { + for (before, prefix) in &queries { + let start = Instant::now(); + predict(before, prefix); + timings.push(start.elapsed().as_secs_f64() * 1000.0); + if timings.len() >= 1000 { + break; + } + } + } + timings.sort_by(f64::total_cmp); + let percentile = |p: usize| timings[(timings.len() * p).div_ceil(100).saturating_sub(1)]; + let accuracy:Vec<_>=counts.iter().enumerate().map(|(n,c)| serde_json::json!({"prefix_chars":n,"queries":c[0],"top1":c[1] as f64/c[0].max(1) as f64,"top5":c[2] as f64/c[0].max(1) as f64,"vocabulary_coverage":c[3] as f64/c[0].max(1) as f64,"newer_oov_queries":c[4],"newer_oov_top1_hits":c[5],"newer_oov_top5_hits":c[6],"legacy_fill_queries":c[7]})).collect(); + let payload = newer.as_ref().map_or(0, Predictor::model_payload_bytes) + + original + .as_ref() + .map_or(0, LegacyPredictor::model_payload_bytes); + Ok( + serde_json::json!({"protocol":"legacy-slot-fill-v1","mode":mode,"input_sha256":crate::digest(text.as_bytes()),"baseline_sha256":crate::digest(&fs::read(baseline)?),"legacy_sha256":crate::digest(&fs::read(legacy)?),"sentences":tests.len(),"accuracy":accuracy,"position_violations":position_violations,"warm_query_count":timings.len(),"cold_load_ms":cold,"warm_median_ms":percentile(50),"warm_p95_ms":percentile(95),"warm_max_ms":timings.last(),"warm_p95_target_met":percentile(95)<20.0,"model_payload_bytes":payload,"database_bytes":if mode=="original" {fs::metadata(legacy)?.len()} else if mode=="newer" {fs::metadata(baseline)?.len()} else {fs::metadata(legacy)?.len()+fs::metadata(baseline)?.len()}}), + ) +} diff --git a/src/lib.rs b/src/lib.rs index d1ae9b3..9b6e747 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -12,6 +12,7 @@ use unicode_segmentation::UnicodeSegmentation; pub mod corpus; pub mod evaluation; +pub mod experimental; pub mod production; #[derive(Debug, thiserror::Error)] diff --git a/src/main.rs b/src/main.rs index c5fdad7..97cf186 100644 --- a/src/main.rs +++ b/src/main.rs @@ -14,6 +14,34 @@ struct Cli { } #[derive(Subcommand)] enum Command { + /// Experimental ordered predictions; scores from different sources are not mixed. + ExperimentalPredict { + #[arg(long, value_parser = ["original", "newer", "combined"])] + mode: String, + #[arg(long)] + baseline: Option, + #[arg(long)] + legacy: Option, + #[arg(long, default_value = "")] + before: String, + #[arg(long, default_value = "")] + prefix: String, + #[arg(long, default_value_t = 5)] + limit: usize, + #[arg(long, default_value_t = 2)] + min_chars: usize, + }, + /// Compare one experimental mode on an explicit frozen partition. + CompareLegacy { + #[arg(long, value_parser = ["original", "newer", "combined"])] + mode: String, + #[arg(long)] + baseline: PathBuf, + #[arg(long)] + legacy: PathBuf, + #[arg(long)] + input: PathBuf, + }, /// Normalize a UTF-8 corpus using exactly the predictor's sentence tokenizer. Normalize { #[arg(long)] @@ -117,6 +145,60 @@ fn print(value: impl serde::Serialize) -> Result<()> { } fn run() -> Result<()> { match Cli::parse().command { + Command::ExperimentalPredict { + mode, + baseline, + legacy, + before, + prefix, + limit, + min_chars, + } => { + use switchify_prediction::experimental::{ + CombinedPredictor, LegacyPredictor, RankedWord, Source, + }; + let required = |p: Option, name: &str| { + p.ok_or_else(|| { + switchify_prediction::Error::Invalid(format!("{name} path required")) + }) + }; + let options = Options { + limit, + min_chars, + unigram_only: false, + }; + match mode.as_str() { + "original" => print( + LegacyPredictor::open(&required(legacy, "legacy")?)? + .predict(&before, &prefix, options), + ), + "newer" => print( + Predictor::open(&required(baseline, "baseline")?, None)? + .predict(&before, &prefix, options) + .into_iter() + .map(|s| RankedWord { + word: s.word, + source: Source::Newer, + }) + .collect::>(), + ), + _ => print( + CombinedPredictor::open( + &required(baseline, "baseline")?, + &required(legacy, "legacy")?, + )? + .predict(&before, &prefix, options), + ), + } + } + Command::CompareLegacy { + mode, + baseline, + legacy, + input, + } => print(switchify_prediction::experimental::compare( + &mode, &baseline, &legacy, &input, + )?), Command::Normalize { input, output } => { let text = fs::read_to_string(input)?; let normalized = switchify_prediction::sentences(&text) diff --git a/tests/experimental.rs b/tests/experimental.rs new file mode 100644 index 0000000..9525086 --- /dev/null +++ b/tests/experimental.rs @@ -0,0 +1,143 @@ +use rusqlite::Connection; +use switchify_prediction::{ + Options, Predictor, build, + experimental::{CombinedPredictor, LegacyPredictor, Source, compare}, +}; +fn fixture(path: &std::path::Path) { + let c = Connection::open(path).unwrap(); + c.execute_batch("CREATE TABLE WORDS(ID INTEGER,WORD TEXT,BASE_FREQUENCY INTEGER,USER_FREQUENCY INTEGER); INSERT INTO WORDS VALUES(1,'I',9,0),(2,'would',8,0),(3,'like',7,0),(4,'water',1,999999),(5,'wine',20,0),(6,'walk',30,0),(7,'CAFÉ',4,0),(8,'café',2,0),(9,'can’t',3,0),(10,'two words',999,0),(11,'123',998,0); CREATE TABLE BIGRAMS(ID1 INTEGER,ID2 INTEGER,BASE_FREQUENCY INTEGER,USER_FREQUENCY INTEGER); INSERT INTO BIGRAMS VALUES(3,5,9,0); CREATE TABLE TRIGRAMS(ID1 INTEGER,ID2 INTEGER,ID3 INTEGER,BASE_FREQUENCY INTEGER); CREATE TABLE QUADGRAMS(ID1 INTEGER,ID2 INTEGER,ID3 INTEGER,ID4 INTEGER,BASE_FREQUENCY INTEGER); INSERT INTO QUADGRAMS VALUES(1,2,3,4,1);").unwrap(); +} +fn opts() -> Options { + Options { + limit: 5, + min_chars: 0, + unigram_only: false, + } +} +#[test] +fn historical_ranking_normalization_and_no_database_access_after_load() { + let d = tempfile::tempdir().unwrap(); + let p = d.path().join("old.db"); + fixture(&p); + let bytes = std::fs::read(&p).unwrap(); + let legacy = LegacyPredictor::open(&p).unwrap(); + assert_eq!( + legacy + .predict("I would like ", "w", opts()) + .iter() + .map(|s| s.word.as_str()) + .collect::>(), + ["water", "wine", "walk", "would"] + ); + assert_eq!(legacy.predict("I would like.", "w", opts())[0].word, "walk"); + assert_eq!( + legacy.predict("I would like\n", "w", opts())[0].word, + "walk" + ); + assert_eq!(legacy.predict("", "CAFE\u{301}", opts()).len(), 1); + assert_eq!(legacy.predict("", "can’", opts())[0].word, "can't"); + assert!(legacy.predict("", "12", opts()).is_empty()); + assert!(legacy.predict("", "two", opts()).is_empty()); + assert!( + legacy + .predict( + "", + "w", + Options { + min_chars: 2, + ..opts() + } + ) + .is_empty() + ); + assert!( + legacy + .predict("", "", Options { limit: 0, ..opts() }) + .is_empty() + ); + assert_eq!( + legacy.predict( + "I would like ", + "w", + Options { + unigram_only: true, + ..opts() + } + )[0] + .word, + "walk" + ); + assert_eq!(std::fs::read(&p).unwrap(), bytes); + std::fs::remove_file(&p).unwrap(); + assert!(!legacy.predict("", "", opts()).is_empty()); +} +#[test] +fn combined_keeps_positions_fills_slots_and_ignores_personal_counts() { + let d = tempfile::tempdir().unwrap(); + let old = d.path().join("old.db"); + fixture(&old); + let new = d.path().join("new.db"); + build(&new, "I want water. Café can’t wait.", "fixture").unwrap(); + let old_bytes = std::fs::read(&old).unwrap(); + let new_bytes = std::fs::read(&new).unwrap(); + let mixed = CombinedPredictor::open(&new, &old).unwrap(); + let modern = Predictor::open(&new, None).unwrap(); + for prefix in ["", "w", "wa", "caf", "can’", "zz"] { + let a = modern.predict("I would like ", prefix, opts()); + let b = mixed.predict("I would like ", prefix, opts()); + assert!(b.len() <= 5); + for (i, s) in a.iter().enumerate() { + assert_eq!(s.word, b[i].word); + assert_eq!(b[i].source, Source::Newer); + } + let unique: std::collections::HashSet<_> = b.iter().map(|r| &r.word).collect(); + assert_eq!(unique.len(), b.len()); + } + let words = mixed.predict("I would like ", "w", opts()); + assert!( + words + .iter() + .any(|r| r.word == "wine" && r.source == Source::Original) + ); + let full = mixed.predict("", "", Options { limit: 1, ..opts() }); + assert_eq!(full.len(), 1); + assert_eq!(full[0].source, Source::Newer); + assert!( + mixed + .predict( + "", + "w", + Options { + min_chars: 2, + ..opts() + } + ) + .is_empty() + ); + assert_eq!(std::fs::read(&old).unwrap(), old_bytes); + assert_eq!(std::fs::read(&new).unwrap(), new_bytes); + let text = d.path().join("test.txt"); + std::fs::write(&text, "I would like wine.").unwrap(); + let report = compare("combined", &new, &old, &text).unwrap(); + assert_eq!(report["position_violations"], 0); + assert!(report["warm_query_count"].as_u64().unwrap() >= 1000); +} +#[test] +fn corrupt_missing_and_dangling_data_fail_and_cli_requires_paths() { + let d = tempfile::tempdir().unwrap(); + let p = d.path().join("old.db"); + assert!(LegacyPredictor::open(&p).is_err()); + fixture(&p); + Connection::open(&p) + .unwrap() + .execute("INSERT INTO BIGRAMS VALUES(1,9999,1,0)", []) + .unwrap(); + assert!(LegacyPredictor::open(&p).is_err()); + std::fs::write(&p, "bad sqlite").unwrap(); + assert!(LegacyPredictor::open(&p).is_err()); + let output = std::process::Command::new(env!("CARGO_BIN_EXE_switchify-prediction")) + .args(["experimental-predict", "--mode", "combined"]) + .output() + .unwrap(); + assert!(!output.status.success()); +} From 0df0a77c4a45e865c1a8e953df013e3de9ad5a2f Mon Sep 17 00:00:00 2001 From: Owen McGirr Date: Sat, 3 Oct 2026 09:42:05 +0100 Subject: [PATCH 2/4] Stream comparison hashes and test acceptance gates --- docs/legacy-comparison-protocol.md | 70 +++---- scripts/compare_legacy.py | 191 ++++++++++--------- scripts/test_compare_legacy.py | 32 ++++ src/experimental.rs | 18 +- tests/experimental.rs | 286 ++++++++++++++--------------- 5 files changed, 328 insertions(+), 269 deletions(-) create mode 100644 scripts/test_compare_legacy.py diff --git a/docs/legacy-comparison-protocol.md b/docs/legacy-comparison-protocol.md index 236dbff..ec38afc 100644 --- a/docs/legacy-comparison-protocol.md +++ b/docs/legacy-comparison-protocol.md @@ -1,35 +1,35 @@ -# Legacy slot-fill protocol v1 - -Frozen before scoring: preserve every newer result and append normalized unique legacy results only to empty slots. Never interpolate scores, learn evaluation text, or tune on these partitions. Compare original, newer and combined at prefixes 0 through 4 with five slots and minimum prefix zero. Give all modes identical preceding text, preserving the original three-word context. Use all six existing AAC/general/conversation development/test partitions and record their hashes. Report OOV subsets and legacy fill rates alongside accuracy and resource costs. Warm timing uses at least 1,000 queries per mode, with a 20 ms p95 target. Accuracy must not regress against newer; no automatic promotion. - -The legacy database is recovered locally from switchify-pc tag v1.0.0-rc.14, path src-tauri/resources/WordData2017051601.db, SHA-256 dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318. It is never committed, packaged or uploaded. Only BASE_FREQUENCY is read. Ranking follows rc.14 lookup/database_reference: longest context first, frequency descending then target ID ascending, then unigram backoff. Modern Unicode normalization and single-word validation are intentional differences from the historical adapter. Source training overlap is unknown, so results are regression comparisons, not unseen-data estimates. - -## Reproduce - -Use Rust 1.97.1, Python 3.9+, curl, and Git. On Windows clone with -`git -c core.autocrlf=false clone ...` so checksum-pinned files retain exact bytes. -Obtain the v0.1.0 model bundle and verify it using its `verify_bundle.py`. -Recover the legacy database from an existing Switchify PC checkout without -changing that checkout, using Python binary output: - -```python -import subprocess -from pathlib import Path -Path("old.sqlite").write_bytes(subprocess.check_output([ - "git", "-C", "../switchify-pc", "show", - "v1.0.0-rc.14:src-tauri/resources/WordData2017051601.db"])) -``` - -```sh -cargo build --release --locked -python scripts/fetch_corpus.py -python scripts/aac_experiment.py --prepare-only -python scripts/compare_legacy.py --exe target/release/switchify-prediction.exe --baseline /path/to/english.sqlite --legacy old.sqlite --output artifacts/legacy-comparison.json -``` - -Omit `.exe` on macOS/Linux. The comparison command verifies both database hashes -and all six partition hashes, runs each mode in a separate process, checks -position preservation and top-five non-regression, and rechecks database hashes -after completion. Reports contain aggregate counts, never captured typing or -personal databases. Peak process memory includes evaluation bookkeeping; payload -bytes exclude allocator overhead. Non-Windows memory polling is a lower bound. +# Legacy slot-fill protocol v1 + +Frozen before scoring: preserve every newer result and append normalized unique legacy results only to empty slots. Never interpolate scores, learn evaluation text, or tune on these partitions. Compare original, newer and combined at prefixes 0 through 4 with five slots and minimum prefix zero. Give all modes identical preceding text, preserving the original three-word context. Use all six existing AAC/general/conversation development/test partitions and record their hashes. Report OOV subsets and legacy fill rates alongside accuracy and resource costs. Warm timing uses at least 1,000 queries per mode, with a 20 ms p95 target. Accuracy must not regress against newer; no automatic promotion. + +The legacy database is recovered locally from switchify-pc tag v1.0.0-rc.14, path src-tauri/resources/WordData2017051601.db, SHA-256 dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318. It is never committed, packaged or uploaded. Only BASE_FREQUENCY is read. Ranking follows rc.14 lookup/database_reference: longest context first, frequency descending then target ID ascending, then unigram backoff. Modern Unicode normalization and single-word validation are intentional differences from the historical adapter. Source training overlap is unknown, so results are regression comparisons, not unseen-data estimates. + +## Reproduce + +Use Rust 1.97.1, Python 3.9+, curl, and Git. On Windows clone with +`git -c core.autocrlf=false clone ...` so checksum-pinned files retain exact bytes. +Obtain the v0.1.0 model bundle and verify it using its `verify_bundle.py`. +Recover the legacy database from an existing Switchify PC checkout without +changing that checkout, using Python binary output: + +```python +import subprocess +from pathlib import Path +Path("old.sqlite").write_bytes(subprocess.check_output([ + "git", "-C", "../switchify-pc", "show", + "v1.0.0-rc.14:src-tauri/resources/WordData2017051601.db"])) +``` + +```sh +cargo build --release --locked +python scripts/fetch_corpus.py +python scripts/aac_experiment.py --prepare-only +python scripts/compare_legacy.py --exe target/release/switchify-prediction.exe --baseline /path/to/english.sqlite --legacy old.sqlite --output artifacts/legacy-comparison.json +``` + +Omit `.exe` on macOS/Linux. The comparison command verifies both database hashes +and all six partition hashes, runs each mode in a separate process, checks +position preservation and top-five non-regression, and rechecks database hashes +after completion. Reports contain aggregate counts, never captured typing or +personal databases. Peak process memory includes evaluation bookkeeping; payload +bytes exclude allocator overhead. Non-Windows memory polling is a lower bound. diff --git a/scripts/compare_legacy.py b/scripts/compare_legacy.py index 16a5be7..84eb8c1 100644 --- a/scripts/compare_legacy.py +++ b/scripts/compare_legacy.py @@ -1,90 +1,101 @@ -#!/usr/bin/env python3 -"""Run frozen slot-fill comparison. Never distributes or trains on the legacy data.""" -import argparse -import hashlib -import json -from pathlib import Path -import platform -import subprocess -import time - -LEGACY_SHA = 'dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318' -BASELINE_SHA = '222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad' - -def sha(path): - return hashlib.sha256(path.read_bytes()).hexdigest() - -def peak_rss(pid): - if platform.system() == 'Windows': - import ctypes - from ctypes import wintypes - class Counters(ctypes.Structure): - _fields_ = [('cb', wintypes.DWORD), ('faults', wintypes.DWORD)] + [(x, ctypes.c_size_t) for x in ['peak', 'working', 'a', 'b', 'c', 'd', 'e', 'f']] - kernel = ctypes.WinDLL('kernel32', use_last_error=True) - kernel.OpenProcess.restype = wintypes.HANDLE - kernel.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD] - kernel.CloseHandle.argtypes = [wintypes.HANDLE] - psapi = ctypes.WinDLL('psapi') - psapi.GetProcessMemoryInfo.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters), wintypes.DWORD] - handle = kernel.OpenProcess(0x410, False, pid) - if not handle: - return None - try: - data = Counters(); data.cb = ctypes.sizeof(data) - if psapi.GetProcessMemoryInfo(handle, ctypes.byref(data), data.cb): - return data.peak - finally: - kernel.CloseHandle(handle) - else: - try: - return int(subprocess.check_output(['ps', '-o', 'rss=', '-p', str(pid)]).strip()) * 1024 - except (ValueError, subprocess.CalledProcessError): - return None - return None - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--exe', type=Path, required=True) - parser.add_argument('--baseline', type=Path, required=True) - parser.add_argument('--legacy', type=Path, required=True) - parser.add_argument('--partitions', type=Path, default=Path('data/aac-oanc/prepared')) - parser.add_argument('--output', type=Path, required=True) - args = parser.parse_args() - if sha(args.legacy) != LEGACY_SHA or sha(args.baseline) != BASELINE_SHA: - raise ValueError('Comparison requires pinned legacy and released baseline databases') - expected = json.loads(Path('production-quality.json').read_text()) - reports = [] - for split in ['dev', 'test']: - for domain in ['aac', 'general', 'conversation']: - path = args.partitions / f'{domain}-{split}.txt' - if sha(path) != expected[split][domain]['candidate']['evaluation_sha256']: - raise ValueError(f'Partition mismatch: {path}') - batch = {} - for mode in ['original', 'newer', 'combined']: - command = [str(args.exe.resolve()), 'compare-legacy', '--mode', mode, '--baseline', str(args.baseline), '--legacy', str(args.legacy), '--input', str(path)] - # Output is small; communicate drains the pipe while polling memory. - import tempfile - with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err: - process = subprocess.Popen(command, stdout=out, stderr=err) - peak = None - while process.poll() is None: - sample = peak_rss(process.pid) - if sample is not None: peak = max(peak or 0, sample) - time.sleep(0.05) - out.seek(0); err.seek(0) - if process.returncode: raise RuntimeError(err.read().decode()) - report = json.loads(out.read()) - report.update(domain=domain, split=split, peak_process_rss_bytes=peak, memory_method='Windows process peak working set' if platform.system() == 'Windows' else 'sampled RSS; lower bound', hardware=platform.platform() + ' ' + platform.processor()) - batch[mode] = report; reports.append(report) - print(domain, split, mode, report['warm_p95_ms'], flush=True) - if batch['combined']['position_violations']: - raise ValueError('Modern positions changed') - for new, combined in zip(batch['newer']['accuracy'], batch['combined']['accuracy']): - if combined['top5'] < new['top5']: raise ValueError('Top-five accuracy regressed') - if sha(args.legacy) != LEGACY_SHA or sha(args.baseline) != BASELINE_SHA: - raise ValueError('Source database changed') - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps({'protocol': 'legacy-slot-fill-v1', 'reports': reports}, indent=2) + '\n') - -if __name__ == '__main__': - main() +#!/usr/bin/env python3 +"""Run frozen slot-fill comparison. Never distributes or trains on the legacy data.""" +import argparse +import hashlib +import json +from pathlib import Path +import platform +import subprocess +import time + +LEGACY_SHA = 'dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318' +BASELINE_SHA = '222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad' + +def sha(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + +def peak_rss(pid): + if platform.system() == 'Windows': + import ctypes + from ctypes import wintypes + class Counters(ctypes.Structure): + _fields_ = [('cb', wintypes.DWORD), ('faults', wintypes.DWORD)] + [(x, ctypes.c_size_t) for x in ['peak', 'working', 'a', 'b', 'c', 'd', 'e', 'f']] + kernel = ctypes.WinDLL('kernel32', use_last_error=True) + kernel.OpenProcess.restype = wintypes.HANDLE + kernel.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD] + kernel.CloseHandle.argtypes = [wintypes.HANDLE] + psapi = ctypes.WinDLL('psapi') + psapi.GetProcessMemoryInfo.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters), wintypes.DWORD] + handle = kernel.OpenProcess(0x410, False, pid) + if not handle: + return None + try: + data = Counters(); data.cb = ctypes.sizeof(data) + if psapi.GetProcessMemoryInfo(handle, ctypes.byref(data), data.cb): + return data.peak + finally: + kernel.CloseHandle(handle) + else: + try: + return int(subprocess.check_output(['ps', '-o', 'rss=', '-p', str(pid)]).strip()) * 1024 + except (ValueError, subprocess.CalledProcessError): + return None + return None + + +def validate_batch(batch): + if batch['combined']['position_violations']: + raise ValueError('Modern positions changed') + newer = batch['newer']['accuracy'] + combined = batch['combined']['accuracy'] + if len(newer) != 5 or len(combined) != 5: + raise ValueError('Missing prefix results') + for new, mixed in zip(newer, combined): + if (new['prefix_chars'], new['queries']) != (mixed['prefix_chars'], mixed['queries']): + raise ValueError('Mismatched evaluation queries') + if mixed['top5'] < new['top5'] or mixed['top1'] < new['top1']: + raise ValueError('Prediction accuracy regressed') + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--exe', type=Path, required=True) + parser.add_argument('--baseline', type=Path, required=True) + parser.add_argument('--legacy', type=Path, required=True) + parser.add_argument('--partitions', type=Path, default=Path('data/aac-oanc/prepared')) + parser.add_argument('--output', type=Path, required=True) + args = parser.parse_args() + if sha(args.legacy) != LEGACY_SHA or sha(args.baseline) != BASELINE_SHA: + raise ValueError('Comparison requires pinned legacy and released baseline databases') + expected = json.loads(Path('production-quality.json').read_text()) + reports = [] + for split in ['dev', 'test']: + for domain in ['aac', 'general', 'conversation']: + path = args.partitions / f'{domain}-{split}.txt' + if sha(path) != expected[split][domain]['candidate']['evaluation_sha256']: + raise ValueError(f'Partition mismatch: {path}') + batch = {} + for mode in ['original', 'newer', 'combined']: + command = [str(args.exe.resolve()), 'compare-legacy', '--mode', mode, '--baseline', str(args.baseline), '--legacy', str(args.legacy), '--input', str(path)] + # Output is small; communicate drains the pipe while polling memory. + import tempfile + with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err: + process = subprocess.Popen(command, stdout=out, stderr=err) + peak = None + while process.poll() is None: + sample = peak_rss(process.pid) + if sample is not None: peak = max(peak or 0, sample) + time.sleep(0.05) + out.seek(0); err.seek(0) + if process.returncode: raise RuntimeError(err.read().decode()) + report = json.loads(out.read()) + report.update(domain=domain, split=split, peak_process_rss_bytes=peak, memory_method='Windows process peak working set' if platform.system() == 'Windows' else 'sampled RSS; lower bound', hardware=platform.platform() + ' ' + platform.processor()) + batch[mode] = report; reports.append(report) + print(domain, split, mode, report['warm_p95_ms'], flush=True) + validate_batch(batch) + if sha(args.legacy) != LEGACY_SHA or sha(args.baseline) != BASELINE_SHA: + raise ValueError('Source database changed') + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps({'protocol': 'legacy-slot-fill-v1', 'reports': reports}, indent=2) + '\n') + +if __name__ == '__main__': + main() diff --git a/scripts/test_compare_legacy.py b/scripts/test_compare_legacy.py new file mode 100644 index 0000000..51d0ac9 --- /dev/null +++ b/scripts/test_compare_legacy.py @@ -0,0 +1,32 @@ +import copy +import unittest +from compare_legacy import validate_batch + +class ComparisonAcceptance(unittest.TestCase): + def fixture(self): + rows = [dict(prefix_chars=i, queries=10, top1=0.2, top5=0.6) for i in range(5)] + return dict(newer=dict(accuracy=rows), combined=dict(accuracy=copy.deepcopy(rows), position_violations=0)) + + def test_unchanged_and_improved_results_pass(self): + value = self.fixture() + validate_batch(value) + value['combined']['accuracy'][2]['top5'] = 0.7 + validate_batch(value) + + def test_position_changes_and_regressions_fail(self): + value = self.fixture() + value['combined']['position_violations'] = 1 + with self.assertRaises(ValueError): validate_batch(value) + for metric in ['top1', 'top5']: + value = self.fixture() + value['combined']['accuracy'][2][metric] = 0.1 + with self.assertRaises(ValueError): validate_batch(value) + + def test_incomparable_or_incomplete_reports_fail(self): + for key in ['queries', 'prefix_chars']: + value = self.fixture() + value['combined']['accuracy'][2][key] = 99 + with self.assertRaises(ValueError): validate_batch(value) + value = self.fixture() + value['combined']['accuracy'].pop() + with self.assertRaises(ValueError): validate_batch(value) diff --git a/src/experimental.rs b/src/experimental.rs index 276566c..6184d6d 100644 --- a/src/experimental.rs +++ b/src/experimental.rs @@ -340,6 +340,22 @@ pub fn compare( .as_ref() .map_or(0, LegacyPredictor::model_payload_bytes); Ok( - serde_json::json!({"protocol":"legacy-slot-fill-v1","mode":mode,"input_sha256":crate::digest(text.as_bytes()),"baseline_sha256":crate::digest(&fs::read(baseline)?),"legacy_sha256":crate::digest(&fs::read(legacy)?),"sentences":tests.len(),"accuracy":accuracy,"position_violations":position_violations,"warm_query_count":timings.len(),"cold_load_ms":cold,"warm_median_ms":percentile(50),"warm_p95_ms":percentile(95),"warm_max_ms":timings.last(),"warm_p95_target_met":percentile(95)<20.0,"model_payload_bytes":payload,"database_bytes":if mode=="original" {fs::metadata(legacy)?.len()} else if mode=="newer" {fs::metadata(baseline)?.len()} else {fs::metadata(legacy)?.len()+fs::metadata(baseline)?.len()}}), + serde_json::json!({"protocol":"legacy-slot-fill-v1","mode":mode,"input_sha256":crate::digest(text.as_bytes()),"baseline_sha256":file_digest(baseline)?,"legacy_sha256":file_digest(legacy)?,"sentences":tests.len(),"accuracy":accuracy,"position_violations":position_violations,"warm_query_count":timings.len(),"cold_load_ms":cold,"warm_median_ms":percentile(50),"warm_p95_ms":percentile(95),"warm_max_ms":timings.last(),"warm_p95_target_met":percentile(95)<20.0,"model_payload_bytes":payload,"database_bytes":if mode=="original" {fs::metadata(legacy)?.len()} else if mode=="newer" {fs::metadata(baseline)?.len()} else {fs::metadata(legacy)?.len()+fs::metadata(baseline)?.len()}}), ) } + +fn file_digest(path: &Path) -> Result { + use sha2::{Digest, Sha256}; + use std::io::Read; + let mut file = std::fs::File::open(path)?; + let mut hash = Sha256::new(); + let mut buffer = [0; 65536]; + loop { + let n = file.read(&mut buffer)?; + if n == 0 { + break; + } + hash.update(&buffer[..n]); + } + Ok(format!("{:x}", hash.finalize())) +} diff --git a/tests/experimental.rs b/tests/experimental.rs index 9525086..a7287a7 100644 --- a/tests/experimental.rs +++ b/tests/experimental.rs @@ -1,143 +1,143 @@ -use rusqlite::Connection; -use switchify_prediction::{ - Options, Predictor, build, - experimental::{CombinedPredictor, LegacyPredictor, Source, compare}, -}; -fn fixture(path: &std::path::Path) { - let c = Connection::open(path).unwrap(); - c.execute_batch("CREATE TABLE WORDS(ID INTEGER,WORD TEXT,BASE_FREQUENCY INTEGER,USER_FREQUENCY INTEGER); INSERT INTO WORDS VALUES(1,'I',9,0),(2,'would',8,0),(3,'like',7,0),(4,'water',1,999999),(5,'wine',20,0),(6,'walk',30,0),(7,'CAFÉ',4,0),(8,'café',2,0),(9,'can’t',3,0),(10,'two words',999,0),(11,'123',998,0); CREATE TABLE BIGRAMS(ID1 INTEGER,ID2 INTEGER,BASE_FREQUENCY INTEGER,USER_FREQUENCY INTEGER); INSERT INTO BIGRAMS VALUES(3,5,9,0); CREATE TABLE TRIGRAMS(ID1 INTEGER,ID2 INTEGER,ID3 INTEGER,BASE_FREQUENCY INTEGER); CREATE TABLE QUADGRAMS(ID1 INTEGER,ID2 INTEGER,ID3 INTEGER,ID4 INTEGER,BASE_FREQUENCY INTEGER); INSERT INTO QUADGRAMS VALUES(1,2,3,4,1);").unwrap(); -} -fn opts() -> Options { - Options { - limit: 5, - min_chars: 0, - unigram_only: false, - } -} -#[test] -fn historical_ranking_normalization_and_no_database_access_after_load() { - let d = tempfile::tempdir().unwrap(); - let p = d.path().join("old.db"); - fixture(&p); - let bytes = std::fs::read(&p).unwrap(); - let legacy = LegacyPredictor::open(&p).unwrap(); - assert_eq!( - legacy - .predict("I would like ", "w", opts()) - .iter() - .map(|s| s.word.as_str()) - .collect::>(), - ["water", "wine", "walk", "would"] - ); - assert_eq!(legacy.predict("I would like.", "w", opts())[0].word, "walk"); - assert_eq!( - legacy.predict("I would like\n", "w", opts())[0].word, - "walk" - ); - assert_eq!(legacy.predict("", "CAFE\u{301}", opts()).len(), 1); - assert_eq!(legacy.predict("", "can’", opts())[0].word, "can't"); - assert!(legacy.predict("", "12", opts()).is_empty()); - assert!(legacy.predict("", "two", opts()).is_empty()); - assert!( - legacy - .predict( - "", - "w", - Options { - min_chars: 2, - ..opts() - } - ) - .is_empty() - ); - assert!( - legacy - .predict("", "", Options { limit: 0, ..opts() }) - .is_empty() - ); - assert_eq!( - legacy.predict( - "I would like ", - "w", - Options { - unigram_only: true, - ..opts() - } - )[0] - .word, - "walk" - ); - assert_eq!(std::fs::read(&p).unwrap(), bytes); - std::fs::remove_file(&p).unwrap(); - assert!(!legacy.predict("", "", opts()).is_empty()); -} -#[test] -fn combined_keeps_positions_fills_slots_and_ignores_personal_counts() { - let d = tempfile::tempdir().unwrap(); - let old = d.path().join("old.db"); - fixture(&old); - let new = d.path().join("new.db"); - build(&new, "I want water. Café can’t wait.", "fixture").unwrap(); - let old_bytes = std::fs::read(&old).unwrap(); - let new_bytes = std::fs::read(&new).unwrap(); - let mixed = CombinedPredictor::open(&new, &old).unwrap(); - let modern = Predictor::open(&new, None).unwrap(); - for prefix in ["", "w", "wa", "caf", "can’", "zz"] { - let a = modern.predict("I would like ", prefix, opts()); - let b = mixed.predict("I would like ", prefix, opts()); - assert!(b.len() <= 5); - for (i, s) in a.iter().enumerate() { - assert_eq!(s.word, b[i].word); - assert_eq!(b[i].source, Source::Newer); - } - let unique: std::collections::HashSet<_> = b.iter().map(|r| &r.word).collect(); - assert_eq!(unique.len(), b.len()); - } - let words = mixed.predict("I would like ", "w", opts()); - assert!( - words - .iter() - .any(|r| r.word == "wine" && r.source == Source::Original) - ); - let full = mixed.predict("", "", Options { limit: 1, ..opts() }); - assert_eq!(full.len(), 1); - assert_eq!(full[0].source, Source::Newer); - assert!( - mixed - .predict( - "", - "w", - Options { - min_chars: 2, - ..opts() - } - ) - .is_empty() - ); - assert_eq!(std::fs::read(&old).unwrap(), old_bytes); - assert_eq!(std::fs::read(&new).unwrap(), new_bytes); - let text = d.path().join("test.txt"); - std::fs::write(&text, "I would like wine.").unwrap(); - let report = compare("combined", &new, &old, &text).unwrap(); - assert_eq!(report["position_violations"], 0); - assert!(report["warm_query_count"].as_u64().unwrap() >= 1000); -} -#[test] -fn corrupt_missing_and_dangling_data_fail_and_cli_requires_paths() { - let d = tempfile::tempdir().unwrap(); - let p = d.path().join("old.db"); - assert!(LegacyPredictor::open(&p).is_err()); - fixture(&p); - Connection::open(&p) - .unwrap() - .execute("INSERT INTO BIGRAMS VALUES(1,9999,1,0)", []) - .unwrap(); - assert!(LegacyPredictor::open(&p).is_err()); - std::fs::write(&p, "bad sqlite").unwrap(); - assert!(LegacyPredictor::open(&p).is_err()); - let output = std::process::Command::new(env!("CARGO_BIN_EXE_switchify-prediction")) - .args(["experimental-predict", "--mode", "combined"]) - .output() - .unwrap(); - assert!(!output.status.success()); -} +use rusqlite::Connection; +use switchify_prediction::{ + Options, Predictor, build, + experimental::{CombinedPredictor, LegacyPredictor, Source, compare}, +}; +fn fixture(path: &std::path::Path) { + let c = Connection::open(path).unwrap(); + c.execute_batch("CREATE TABLE WORDS(ID INTEGER,WORD TEXT,BASE_FREQUENCY INTEGER,USER_FREQUENCY INTEGER); INSERT INTO WORDS VALUES(1,'I',9,0),(2,'would',8,0),(3,'like',7,0),(4,'water',1,999999),(5,'wine',20,0),(6,'walk',30,0),(7,'CAFÉ',4,0),(8,'café',2,0),(9,'can’t',3,0),(10,'two words',999,0),(11,'123',998,0); CREATE TABLE BIGRAMS(ID1 INTEGER,ID2 INTEGER,BASE_FREQUENCY INTEGER,USER_FREQUENCY INTEGER); INSERT INTO BIGRAMS VALUES(3,5,9,0); CREATE TABLE TRIGRAMS(ID1 INTEGER,ID2 INTEGER,ID3 INTEGER,BASE_FREQUENCY INTEGER); CREATE TABLE QUADGRAMS(ID1 INTEGER,ID2 INTEGER,ID3 INTEGER,ID4 INTEGER,BASE_FREQUENCY INTEGER); INSERT INTO QUADGRAMS VALUES(1,2,3,4,1);").unwrap(); +} +fn opts() -> Options { + Options { + limit: 5, + min_chars: 0, + unigram_only: false, + } +} +#[test] +fn historical_ranking_normalization_and_no_database_access_after_load() { + let d = tempfile::tempdir().unwrap(); + let p = d.path().join("old.db"); + fixture(&p); + let bytes = std::fs::read(&p).unwrap(); + let legacy = LegacyPredictor::open(&p).unwrap(); + assert_eq!( + legacy + .predict("I would like ", "w", opts()) + .iter() + .map(|s| s.word.as_str()) + .collect::>(), + ["water", "wine", "walk", "would"] + ); + assert_eq!(legacy.predict("I would like.", "w", opts())[0].word, "walk"); + assert_eq!( + legacy.predict("I would like\n", "w", opts())[0].word, + "walk" + ); + assert_eq!(legacy.predict("", "CAFE\u{301}", opts()).len(), 1); + assert_eq!(legacy.predict("", "can’", opts())[0].word, "can't"); + assert!(legacy.predict("", "12", opts()).is_empty()); + assert!(legacy.predict("", "two", opts()).is_empty()); + assert!( + legacy + .predict( + "", + "w", + Options { + min_chars: 2, + ..opts() + } + ) + .is_empty() + ); + assert!( + legacy + .predict("", "", Options { limit: 0, ..opts() }) + .is_empty() + ); + assert_eq!( + legacy.predict( + "I would like ", + "w", + Options { + unigram_only: true, + ..opts() + } + )[0] + .word, + "walk" + ); + assert_eq!(std::fs::read(&p).unwrap(), bytes); + std::fs::remove_file(&p).unwrap(); + assert!(!legacy.predict("", "", opts()).is_empty()); +} +#[test] +fn combined_keeps_positions_fills_slots_and_ignores_personal_counts() { + let d = tempfile::tempdir().unwrap(); + let old = d.path().join("old.db"); + fixture(&old); + let new = d.path().join("new.db"); + build(&new, "I want water. Café can’t wait.", "fixture").unwrap(); + let old_bytes = std::fs::read(&old).unwrap(); + let new_bytes = std::fs::read(&new).unwrap(); + let mixed = CombinedPredictor::open(&new, &old).unwrap(); + let modern = Predictor::open(&new, None).unwrap(); + for prefix in ["", "w", "wa", "caf", "can’", "zz"] { + let a = modern.predict("I would like ", prefix, opts()); + let b = mixed.predict("I would like ", prefix, opts()); + assert!(b.len() <= 5); + for (i, s) in a.iter().enumerate() { + assert_eq!(s.word, b[i].word); + assert_eq!(b[i].source, Source::Newer); + } + let unique: std::collections::HashSet<_> = b.iter().map(|r| &r.word).collect(); + assert_eq!(unique.len(), b.len()); + } + let words = mixed.predict("I would like ", "w", opts()); + assert!( + words + .iter() + .any(|r| r.word == "wine" && r.source == Source::Original) + ); + let full = mixed.predict("", "", Options { limit: 1, ..opts() }); + assert_eq!(full.len(), 1); + assert_eq!(full[0].source, Source::Newer); + assert!( + mixed + .predict( + "", + "w", + Options { + min_chars: 2, + ..opts() + } + ) + .is_empty() + ); + assert_eq!(std::fs::read(&old).unwrap(), old_bytes); + assert_eq!(std::fs::read(&new).unwrap(), new_bytes); + let text = d.path().join("test.txt"); + std::fs::write(&text, "I would like wine.").unwrap(); + let report = compare("combined", &new, &old, &text).unwrap(); + assert_eq!(report["position_violations"], 0); + assert!(report["warm_query_count"].as_u64().unwrap() >= 1000); +} +#[test] +fn corrupt_missing_and_dangling_data_fail_and_cli_requires_paths() { + let d = tempfile::tempdir().unwrap(); + let p = d.path().join("old.db"); + assert!(LegacyPredictor::open(&p).is_err()); + fixture(&p); + Connection::open(&p) + .unwrap() + .execute("INSERT INTO BIGRAMS VALUES(1,9999,1,0)", []) + .unwrap(); + assert!(LegacyPredictor::open(&p).is_err()); + std::fs::write(&p, "bad sqlite").unwrap(); + assert!(LegacyPredictor::open(&p).is_err()); + let output = std::process::Command::new(env!("CARGO_BIN_EXE_switchify-prediction")) + .args(["experimental-predict", "--mode", "combined"]) + .output() + .unwrap(); + assert!(!output.status.success()); +} From ed45e8964596d88d3ec408043db26a36643f02f0 Mon Sep 17 00:00:00 2001 From: Owen McGirr Date: Sat, 3 Oct 2026 09:51:19 +0100 Subject: [PATCH 3/4] Record frozen three-way prediction comparison --- docs/legacy-comparison-results.json | 1427 +++++++++++++++++++++++++++ docs/legacy-comparison-results.md | 42 + 2 files changed, 1469 insertions(+) create mode 100644 docs/legacy-comparison-results.json create mode 100644 docs/legacy-comparison-results.md diff --git a/docs/legacy-comparison-results.json b/docs/legacy-comparison-results.json new file mode 100644 index 0000000..264a126 --- /dev/null +++ b/docs/legacy-comparison-results.json @@ -0,0 +1,1427 @@ +{ + "protocol": "legacy-slot-fill-v1", + "reports": [ + { + "accuracy": [ + { + "legacy_fill_queries": 2611, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 2611, + "top1": 0.14170815779394869, + "top5": 0.2903102259670624, + "vocabulary_coverage": 0.9950210647261586 + }, + { + "legacy_fill_queries": 2390, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 2390, + "top1": 0.40543933054393305, + "top5": 0.6493723849372385, + "vocabulary_coverage": 0.99581589958159 + }, + { + "legacy_fill_queries": 1888, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 1888, + "top1": 0.5254237288135594, + "top5": 0.7658898305084746, + "vocabulary_coverage": 0.9957627118644068 + }, + { + "legacy_fill_queries": 1295, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 2, + "newer_oov_top5_hits": 2, + "prefix_chars": 3, + "queries": 1295, + "top1": 0.6911196911196911, + "top5": 0.8926640926640926, + "vocabulary_coverage": 0.9961389961389961 + }, + { + "legacy_fill_queries": 695, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 3, + "newer_oov_top5_hits": 9, + "prefix_chars": 4, + "queries": 697, + "top1": 0.7360114777618364, + "top5": 0.9426111908177905, + "vocabulary_coverage": 0.9928263988522238 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 4079.1671, + "database_bytes": 102690816, + "input_sha256": "d85649bf5f81343b7cf32d5d930c39c482a378c64e711c7b3c1c46cbccbbe77d", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "original", + "model_payload_bytes": 44244214, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 451, + "warm_max_ms": 5.9923, + "warm_median_ms": 0.0059, + "warm_p95_ms": 2.2978, + "warm_p95_target_met": true, + "warm_query_count": 8881, + "domain": "aac", + "split": "dev", + "peak_process_rss_bytes": 196698112, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 2611, + "top1": 0.2386058981233244, + "top5": 0.42129452317119875, + "vocabulary_coverage": 0.991574109536576 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 2390, + "top1": 0.4803347280334728, + "top5": 0.7305439330543934, + "vocabulary_coverage": 0.9907949790794979 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 1888, + "top1": 0.6038135593220338, + "top5": 0.8416313559322034, + "vocabulary_coverage": 0.9883474576271186 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 1295, + "top1": 0.7351351351351352, + "top5": 0.9305019305019305, + "vocabulary_coverage": 0.9853281853281853 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 4, + "queries": 697, + "top1": 0.7833572453371592, + "top5": 0.9540889526542324, + "vocabulary_coverage": 0.9727403156384505 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 2588.1085000000003, + "database_bytes": 29802496, + "input_sha256": "d85649bf5f81343b7cf32d5d930c39c482a378c64e711c7b3c1c46cbccbbe77d", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "newer", + "model_payload_bytes": 16821218, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 451, + "warm_max_ms": 12.782, + "warm_median_ms": 0.2636, + "warm_p95_ms": 6.662000000000001, + "warm_p95_target_met": true, + "warm_query_count": 8881, + "domain": "aac", + "split": "dev", + "peak_process_rss_bytes": 382214144, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 2611, + "top1": 0.2386058981233244, + "top5": 0.42129452317119875, + "vocabulary_coverage": 0.9973190348525469 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 2390, + "top1": 0.4803347280334728, + "top5": 0.7305439330543934, + "vocabulary_coverage": 0.997071129707113 + }, + { + "legacy_fill_queries": 1, + "newer_oov_queries": 22, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 1888, + "top1": 0.6038135593220338, + "top5": 0.8416313559322034, + "vocabulary_coverage": 0.996292372881356 + }, + { + "legacy_fill_queries": 62, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 2, + "prefix_chars": 3, + "queries": 1295, + "top1": 0.7359073359073359, + "top5": 0.932046332046332, + "vocabulary_coverage": 0.9961389961389961 + }, + { + "legacy_fill_queries": 174, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 8, + "prefix_chars": 4, + "queries": 697, + "top1": 0.7847919655667145, + "top5": 0.9655667144906743, + "vocabulary_coverage": 0.9928263988522238 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 6703.5419, + "database_bytes": 132493312, + "input_sha256": "d85649bf5f81343b7cf32d5d930c39c482a378c64e711c7b3c1c46cbccbbe77d", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "combined", + "model_payload_bytes": 61065432, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 451, + "warm_max_ms": 10.1484, + "warm_median_ms": 0.2626, + "warm_p95_ms": 6.3606, + "warm_p95_target_met": true, + "warm_query_count": 8881, + "domain": "aac", + "split": "dev", + "peak_process_rss_bytes": 440262656, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 7164, + "newer_oov_queries": 336, + "newer_oov_top1_hits": 2, + "newer_oov_top5_hits": 6, + "prefix_chars": 0, + "queries": 7164, + "top1": 0.15591848129536573, + "top5": 0.29871580122836405, + "vocabulary_coverage": 0.9812953657174762 + }, + { + "legacy_fill_queries": 6786, + "newer_oov_queries": 336, + "newer_oov_top1_hits": 7, + "newer_oov_top5_hits": 12, + "prefix_chars": 1, + "queries": 6786, + "top1": 0.33760683760683763, + "top5": 0.5374300029472443, + "vocabulary_coverage": 0.9814323607427056 + }, + { + "legacy_fill_queries": 5607, + "newer_oov_queries": 334, + "newer_oov_top1_hits": 10, + "newer_oov_top5_hits": 22, + "prefix_chars": 2, + "queries": 5608, + "top1": 0.41957917261055633, + "top5": 0.6385520684736091, + "vocabulary_coverage": 0.9780670470756063 + }, + { + "legacy_fill_queries": 4092, + "newer_oov_queries": 329, + "newer_oov_top1_hits": 19, + "newer_oov_top5_hits": 74, + "prefix_chars": 3, + "queries": 4097, + "top1": 0.5264827922870393, + "top5": 0.7749572858188919, + "vocabulary_coverage": 0.9707102758115694 + }, + { + "legacy_fill_queries": 2722, + "newer_oov_queries": 318, + "newer_oov_top1_hits": 61, + "newer_oov_top5_hits": 157, + "prefix_chars": 4, + "queries": 2758, + "top1": 0.5888324873096447, + "top5": 0.8589557650471356, + "vocabulary_coverage": 0.9593908629441624 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 4177.4609, + "database_bytes": 102690816, + "input_sha256": "99cb104dbe893608313b300cca933600c97f2c1bedc9799c6e12e148dcd01f3e", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "original", + "model_payload_bytes": 44244214, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 804, + "warm_max_ms": 5.6778, + "warm_median_ms": 0.0063999999999999994, + "warm_p95_ms": 2.3247, + "warm_p95_target_met": true, + "warm_query_count": 26413, + "domain": "general", + "split": "dev", + "peak_process_rss_bytes": 198864896, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 336, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 7164, + "top1": 0.16150195421552205, + "top5": 0.29885538805136797, + "vocabulary_coverage": 0.9530988274706867 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 336, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 6786, + "top1": 0.32964927792514, + "top5": 0.5611553197760094, + "vocabulary_coverage": 0.9504862953138815 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 334, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 5608, + "top1": 0.43901569186875894, + "top5": 0.6768901569186876, + "vocabulary_coverage": 0.9404422253922967 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 329, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 4097, + "top1": 0.5430803026604832, + "top5": 0.7944837686111789, + "vocabulary_coverage": 0.9196973395167195 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 318, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 4, + "queries": 2758, + "top1": 0.5939086294416244, + "top5": 0.8368382886149384, + "vocabulary_coverage": 0.8846990572878898 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 2535.4922, + "database_bytes": 29802496, + "input_sha256": "99cb104dbe893608313b300cca933600c97f2c1bedc9799c6e12e148dcd01f3e", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "newer", + "model_payload_bytes": 16821218, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 804, + "warm_max_ms": 14.3581, + "warm_median_ms": 0.192, + "warm_p95_ms": 6.302099999999999, + "warm_p95_target_met": true, + "warm_query_count": 26413, + "domain": "general", + "split": "dev", + "peak_process_rss_bytes": 382488576, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 336, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 7164, + "top1": 0.16150195421552205, + "top5": 0.29885538805136797, + "vocabulary_coverage": 0.9932998324958124 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 336, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 6786, + "top1": 0.32964927792514, + "top5": 0.5611553197760094, + "vocabulary_coverage": 0.9929266136162688 + }, + { + "legacy_fill_queries": 9, + "newer_oov_queries": 334, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 5608, + "top1": 0.43901569186875894, + "top5": 0.6768901569186876, + "vocabulary_coverage": 0.9916191155492154 + }, + { + "legacy_fill_queries": 297, + "newer_oov_queries": 329, + "newer_oov_top1_hits": 2, + "newer_oov_top5_hits": 19, + "prefix_chars": 3, + "queries": 4097, + "top1": 0.5435684647302904, + "top5": 0.7991213082743471, + "vocabulary_coverage": 0.988772272394435 + }, + { + "legacy_fill_queries": 956, + "newer_oov_queries": 318, + "newer_oov_top1_hits": 21, + "newer_oov_top5_hits": 102, + "prefix_chars": 4, + "queries": 2758, + "top1": 0.6015228426395939, + "top5": 0.873821609862219, + "vocabulary_coverage": 0.9847715736040609 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 6796.8067, + "database_bytes": 132493312, + "input_sha256": "99cb104dbe893608313b300cca933600c97f2c1bedc9799c6e12e148dcd01f3e", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "combined", + "model_payload_bytes": 61065432, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 804, + "warm_max_ms": 14.335199999999999, + "warm_median_ms": 0.2418, + "warm_p95_ms": 6.3264000000000005, + "warm_p95_target_met": true, + "warm_query_count": 26413, + "domain": "general", + "split": "dev", + "peak_process_rss_bytes": 442404864, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 4991, + "newer_oov_queries": 57, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 4991, + "top1": 0.12422360248447205, + "top5": 0.2740933680625125, + "vocabulary_coverage": 0.9765578040472851 + }, + { + "legacy_fill_queries": 4665, + "newer_oov_queries": 57, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 4665, + "top1": 0.3333333333333333, + "top5": 0.5607717041800643, + "vocabulary_coverage": 0.9770632368703108 + }, + { + "legacy_fill_queries": 3690, + "newer_oov_queries": 56, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 3690, + "top1": 0.4514905149051491, + "top5": 0.683739837398374, + "vocabulary_coverage": 0.9766937669376694 + }, + { + "legacy_fill_queries": 2682, + "newer_oov_queries": 54, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 2, + "prefix_chars": 3, + "queries": 2685, + "top1": 0.5940409683426443, + "top5": 0.8126629422718809, + "vocabulary_coverage": 0.9705772811918063 + }, + { + "legacy_fill_queries": 1597, + "newer_oov_queries": 50, + "newer_oov_top1_hits": 5, + "newer_oov_top5_hits": 12, + "prefix_chars": 4, + "queries": 1608, + "top1": 0.6486318407960199, + "top5": 0.8911691542288557, + "vocabulary_coverage": 0.9651741293532339 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 4127.1048, + "database_bytes": 102690816, + "input_sha256": "8991ecbdeb263e6345b2e4e0b2301cf3287930ea4289cca9d649aad89144faac", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "original", + "model_payload_bytes": 44244214, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 599, + "warm_max_ms": 5.472099999999999, + "warm_median_ms": 0.0055, + "warm_p95_ms": 2.2393, + "warm_p95_target_met": true, + "warm_query_count": 17639, + "domain": "conversation", + "split": "dev", + "peak_process_rss_bytes": 197955584, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 57, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 4991, + "top1": 0.24924864756561813, + "top5": 0.4604287717892206, + "vocabulary_coverage": 0.9885794429973953 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 57, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 4665, + "top1": 0.49732047159699894, + "top5": 0.7541264737406217, + "vocabulary_coverage": 0.9877813504823151 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 56, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 3690, + "top1": 0.6585365853658537, + "top5": 0.8615176151761518, + "vocabulary_coverage": 0.9848238482384823 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 54, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 2685, + "top1": 0.7694599627560521, + "top5": 0.939292364990689, + "vocabulary_coverage": 0.9798882681564246 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 50, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 4, + "queries": 1608, + "top1": 0.8072139303482587, + "top5": 0.9533582089552238, + "vocabulary_coverage": 0.9689054726368159 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 2505.3296, + "database_bytes": 29802496, + "input_sha256": "8991ecbdeb263e6345b2e4e0b2301cf3287930ea4289cca9d649aad89144faac", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "newer", + "model_payload_bytes": 16821218, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 599, + "warm_max_ms": 9.1116, + "warm_median_ms": 0.2407, + "warm_p95_ms": 6.3534999999999995, + "warm_p95_target_met": true, + "warm_query_count": 17639, + "domain": "conversation", + "split": "dev", + "peak_process_rss_bytes": 382201856, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 57, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 4991, + "top1": 0.24924864756561813, + "top5": 0.4604287717892206, + "vocabulary_coverage": 0.9945902624724504 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 57, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 4665, + "top1": 0.49732047159699894, + "top5": 0.7541264737406217, + "vocabulary_coverage": 0.9942122186495177 + }, + { + "legacy_fill_queries": 6, + "newer_oov_queries": 56, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 3690, + "top1": 0.6585365853658537, + "top5": 0.8615176151761518, + "vocabulary_coverage": 0.9926829268292683 + }, + { + "legacy_fill_queries": 151, + "newer_oov_queries": 54, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 2685, + "top1": 0.7694599627560521, + "top5": 0.939292364990689, + "vocabulary_coverage": 0.9906890130353817 + }, + { + "legacy_fill_queries": 351, + "newer_oov_queries": 50, + "newer_oov_top1_hits": 3, + "newer_oov_top5_hits": 10, + "prefix_chars": 4, + "queries": 1608, + "top1": 0.8090796019900498, + "top5": 0.9595771144278606, + "vocabulary_coverage": 0.986318407960199 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 6702.6052, + "database_bytes": 132493312, + "input_sha256": "8991ecbdeb263e6345b2e4e0b2301cf3287930ea4289cca9d649aad89144faac", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "combined", + "model_payload_bytes": 61065432, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 599, + "warm_max_ms": 11.2401, + "warm_median_ms": 0.25789999999999996, + "warm_p95_ms": 6.4013, + "warm_p95_target_met": true, + "warm_query_count": 17639, + "domain": "conversation", + "split": "dev", + "peak_process_rss_bytes": 441249792, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 2239, + "newer_oov_queries": 20, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 2239, + "top1": 0.14381420276909335, + "top5": 0.2996873604287628, + "vocabulary_coverage": 0.9955337204108977 + }, + { + "legacy_fill_queries": 2074, + "newer_oov_queries": 20, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 2074, + "top1": 0.4021215043394407, + "top5": 0.6677917068466731, + "vocabulary_coverage": 0.9956605593056895 + }, + { + "legacy_fill_queries": 1672, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 2, + "prefix_chars": 2, + "queries": 1672, + "top1": 0.562200956937799, + "top5": 0.7924641148325359, + "vocabulary_coverage": 0.9952153110047847 + }, + { + "legacy_fill_queries": 1124, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 4, + "newer_oov_top5_hits": 7, + "prefix_chars": 3, + "queries": 1124, + "top1": 0.7357651245551602, + "top5": 0.9074733096085409, + "vocabulary_coverage": 0.9937722419928826 + }, + { + "legacy_fill_queries": 577, + "newer_oov_queries": 16, + "newer_oov_top1_hits": 6, + "newer_oov_top5_hits": 9, + "prefix_chars": 4, + "queries": 577, + "top1": 0.7885615251299827, + "top5": 0.9584055459272097, + "vocabulary_coverage": 0.9948006932409013 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 4114.9504, + "database_bytes": 102690816, + "input_sha256": "eda6e129d1f6e45e4e5ac27cf19d1a5ab18a4cd1dd3829e92db99cb9d6fafe11", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "original", + "model_payload_bytes": 44244214, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 424, + "warm_max_ms": 5.9312000000000005, + "warm_median_ms": 0.006, + "warm_p95_ms": 2.3965, + "warm_p95_target_met": true, + "warm_query_count": 7686, + "domain": "aac", + "split": "test", + "peak_process_rss_bytes": 196149248, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 20, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 2239, + "top1": 0.2219740955783832, + "top5": 0.41089772219740955, + "vocabulary_coverage": 0.9910674408217954 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 20, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 2074, + "top1": 0.4831243972999036, + "top5": 0.7343297974927676, + "vocabulary_coverage": 0.9903567984570878 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 1672, + "top1": 0.6339712918660287, + "top5": 0.8516746411483254, + "vocabulary_coverage": 0.9886363636363636 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 1124, + "top1": 0.7508896797153025, + "top5": 0.9279359430604982, + "vocabulary_coverage": 0.9830960854092526 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 16, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 4, + "queries": 577, + "top1": 0.7746967071057193, + "top5": 0.9428076256499134, + "vocabulary_coverage": 0.9722703639514731 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 2519.5838999999996, + "database_bytes": 29802496, + "input_sha256": "eda6e129d1f6e45e4e5ac27cf19d1a5ab18a4cd1dd3829e92db99cb9d6fafe11", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "newer", + "model_payload_bytes": 16821218, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 424, + "warm_max_ms": 8.0584, + "warm_median_ms": 0.2281, + "warm_p95_ms": 6.2276, + "warm_p95_target_met": true, + "warm_query_count": 7686, + "domain": "aac", + "split": "test", + "peak_process_rss_bytes": 382226432, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 20, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 2239, + "top1": 0.2219740955783832, + "top5": 0.41089772219740955, + "vocabulary_coverage": 0.9973202322465387 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 20, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 2074, + "top1": 0.4831243972999036, + "top5": 0.7343297974927676, + "vocabulary_coverage": 0.9971070395371263 + }, + { + "legacy_fill_queries": 1, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 1672, + "top1": 0.6339712918660287, + "top5": 0.8516746411483254, + "vocabulary_coverage": 0.9970095693779905 + }, + { + "legacy_fill_queries": 57, + "newer_oov_queries": 19, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 1124, + "top1": 0.7508896797153025, + "top5": 0.9279359430604982, + "vocabulary_coverage": 0.9955516014234875 + }, + { + "legacy_fill_queries": 158, + "newer_oov_queries": 16, + "newer_oov_top1_hits": 2, + "newer_oov_top5_hits": 8, + "prefix_chars": 4, + "queries": 577, + "top1": 0.7781629116117851, + "top5": 0.9566724436741768, + "vocabulary_coverage": 0.9965337954939342 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 6758.3623, + "database_bytes": 132493312, + "input_sha256": "eda6e129d1f6e45e4e5ac27cf19d1a5ab18a4cd1dd3829e92db99cb9d6fafe11", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "combined", + "model_payload_bytes": 61065432, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 424, + "warm_max_ms": 10.846, + "warm_median_ms": 0.2484, + "warm_p95_ms": 6.2661, + "warm_p95_target_met": true, + "warm_query_count": 7686, + "domain": "aac", + "split": "test", + "peak_process_rss_bytes": 439595008, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 6859, + "newer_oov_queries": 363, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 5, + "prefix_chars": 0, + "queries": 6859, + "top1": 0.1510424260096224, + "top5": 0.28969237498177575, + "vocabulary_coverage": 0.981192593672547 + }, + { + "legacy_fill_queries": 6542, + "newer_oov_queries": 363, + "newer_oov_top1_hits": 6, + "newer_oov_top5_hits": 11, + "prefix_chars": 1, + "queries": 6542, + "top1": 0.33720574747783555, + "top5": 0.5385203301742586, + "vocabulary_coverage": 0.9811984102720881 + }, + { + "legacy_fill_queries": 5411, + "newer_oov_queries": 361, + "newer_oov_top1_hits": 12, + "newer_oov_top5_hits": 33, + "prefix_chars": 2, + "queries": 5411, + "top1": 0.42080946220661614, + "top5": 0.6340787285159859, + "vocabulary_coverage": 0.97856218813528 + }, + { + "legacy_fill_queries": 3890, + "newer_oov_queries": 351, + "newer_oov_top1_hits": 40, + "newer_oov_top5_hits": 98, + "prefix_chars": 3, + "queries": 3898, + "top1": 0.5271934325295023, + "top5": 0.7739866598255516, + "vocabulary_coverage": 0.9715238583889174 + }, + { + "legacy_fill_queries": 2635, + "newer_oov_queries": 333, + "newer_oov_top1_hits": 78, + "newer_oov_top5_hits": 177, + "prefix_chars": 4, + "queries": 2679, + "top1": 0.6009705113848451, + "top5": 0.8678611422172452, + "vocabulary_coverage": 0.961926091825308 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 4013.9832, + "database_bytes": 102690816, + "input_sha256": "a5ade2d5cf0217a6aab881877e26ebe66837ff31f75ec622c0642984d9b58292", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "original", + "model_payload_bytes": 44244214, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 804, + "warm_max_ms": 5.6838, + "warm_median_ms": 0.0063999999999999994, + "warm_p95_ms": 2.3035, + "warm_p95_target_met": true, + "warm_query_count": 25389, + "domain": "general", + "split": "test", + "peak_process_rss_bytes": 198545408, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 363, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 6859, + "top1": 0.1494386936871264, + "top5": 0.2853185595567867, + "vocabulary_coverage": 0.9470768333576323 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 363, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 6542, + "top1": 0.32130846835830024, + "top5": 0.5498318557016203, + "vocabulary_coverage": 0.9445123815346989 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 361, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 5411, + "top1": 0.4304195158011458, + "top5": 0.6640177416374052, + "vocabulary_coverage": 0.9332840510072076 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 351, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 3898, + "top1": 0.5282195997947665, + "top5": 0.7914315033350436, + "vocabulary_coverage": 0.9099538224730631 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 333, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 4, + "queries": 2679, + "top1": 0.5841731989548339, + "top5": 0.8324001493094438, + "vocabulary_coverage": 0.8756998880179171 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 2464.7988, + "database_bytes": 29802496, + "input_sha256": "a5ade2d5cf0217a6aab881877e26ebe66837ff31f75ec622c0642984d9b58292", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "newer", + "model_payload_bytes": 16821218, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 804, + "warm_max_ms": 18.1844, + "warm_median_ms": 0.1973, + "warm_p95_ms": 6.2538, + "warm_p95_target_met": true, + "warm_query_count": 25389, + "domain": "general", + "split": "test", + "peak_process_rss_bytes": 382390272, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 363, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 6859, + "top1": 0.1494386936871264, + "top5": 0.2853185595567867, + "vocabulary_coverage": 0.9921271322350197 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 363, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 6542, + "top1": 0.32130846835830024, + "top5": 0.5498318557016203, + "vocabulary_coverage": 0.9917456435340875 + }, + { + "legacy_fill_queries": 10, + "newer_oov_queries": 361, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 2, + "prefix_chars": 2, + "queries": 5411, + "top1": 0.43060432452411757, + "top5": 0.6643873590833488, + "vocabulary_coverage": 0.9900203289595269 + }, + { + "legacy_fill_queries": 274, + "newer_oov_queries": 351, + "newer_oov_top1_hits": 3, + "newer_oov_top5_hits": 29, + "prefix_chars": 3, + "queries": 3898, + "top1": 0.5289892252437147, + "top5": 0.7988712160082093, + "vocabulary_coverage": 0.986659825551565 + }, + { + "legacy_fill_queries": 968, + "newer_oov_queries": 333, + "newer_oov_top1_hits": 31, + "newer_oov_top5_hits": 122, + "prefix_chars": 4, + "queries": 2679, + "top1": 0.5957446808510638, + "top5": 0.8779395296752519, + "vocabulary_coverage": 0.9824561403508771 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 6663.5256, + "database_bytes": 132493312, + "input_sha256": "a5ade2d5cf0217a6aab881877e26ebe66837ff31f75ec622c0642984d9b58292", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "combined", + "model_payload_bytes": 61065432, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 804, + "warm_max_ms": 9.665099999999999, + "warm_median_ms": 0.2241, + "warm_p95_ms": 6.2364, + "warm_p95_target_met": true, + "warm_query_count": 25389, + "domain": "general", + "split": "test", + "peak_process_rss_bytes": 442540032, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 4689, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 4689, + "top1": 0.12049477500533162, + "top5": 0.26636809554275964, + "vocabulary_coverage": 0.9707826828748134 + }, + { + "legacy_fill_queries": 4369, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 4369, + "top1": 0.32936598764019226, + "top5": 0.5628290226596475, + "vocabulary_coverage": 0.9725337605859464 + }, + { + "legacy_fill_queries": 3437, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 3, + "prefix_chars": 2, + "queries": 3437, + "top1": 0.44311899912714575, + "top5": 0.6828629618853651, + "vocabulary_coverage": 0.9717777131219086 + }, + { + "legacy_fill_queries": 2480, + "newer_oov_queries": 56, + "newer_oov_top1_hits": 2, + "newer_oov_top5_hits": 9, + "prefix_chars": 3, + "queries": 2486, + "top1": 0.5905068382944489, + "top5": 0.8153660498793243, + "vocabulary_coverage": 0.9629927594529365 + }, + { + "legacy_fill_queries": 1488, + "newer_oov_queries": 50, + "newer_oov_top1_hits": 8, + "newer_oov_top5_hits": 15, + "prefix_chars": 4, + "queries": 1492, + "top1": 0.6380697050938338, + "top5": 0.8840482573726541, + "vocabulary_coverage": 0.9597855227882037 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 4216.0855, + "database_bytes": 102690816, + "input_sha256": "17fb860b67c0a65da61af4c18011f4968d741a62bcf0fadd0ec67eb4275189a3", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "original", + "model_payload_bytes": 44244214, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 599, + "warm_max_ms": 5.858099999999999, + "warm_median_ms": 0.0059, + "warm_p95_ms": 2.347, + "warm_p95_target_met": true, + "warm_query_count": 16473, + "domain": "conversation", + "split": "test", + "peak_process_rss_bytes": 197783552, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 4689, + "top1": 0.25037321390488376, + "top5": 0.4523352527191299, + "vocabulary_coverage": 0.9872040946896993 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 4369, + "top1": 0.49576562142366676, + "top5": 0.7523460746166171, + "vocabulary_coverage": 0.9862668802929733 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 2, + "queries": 3437, + "top1": 0.6424207157404713, + "top5": 0.8623799825429154, + "vocabulary_coverage": 0.9825429153331393 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 56, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 3, + "queries": 2486, + "top1": 0.7642799678197908, + "top5": 0.9392598551890587, + "vocabulary_coverage": 0.9774738535800482 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 50, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 4, + "queries": 1492, + "top1": 0.8109919571045576, + "top5": 0.9537533512064343, + "vocabulary_coverage": 0.9664879356568364 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 2583.7326000000003, + "database_bytes": 29802496, + "input_sha256": "17fb860b67c0a65da61af4c18011f4968d741a62bcf0fadd0ec67eb4275189a3", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "newer", + "model_payload_bytes": 16821218, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 599, + "warm_max_ms": 8.352500000000001, + "warm_median_ms": 0.22139999999999999, + "warm_p95_ms": 6.3496999999999995, + "warm_p95_target_met": true, + "warm_query_count": 16473, + "domain": "conversation", + "split": "test", + "peak_process_rss_bytes": 382255104, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + }, + { + "accuracy": [ + { + "legacy_fill_queries": 0, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 0, + "queries": 4689, + "top1": 0.25037321390488376, + "top5": 0.4523352527191299, + "vocabulary_coverage": 0.9944551076988697 + }, + { + "legacy_fill_queries": 0, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 0, + "prefix_chars": 1, + "queries": 4369, + "top1": 0.49576562142366676, + "top5": 0.7523460746166171, + "vocabulary_coverage": 0.9940489814602884 + }, + { + "legacy_fill_queries": 11, + "newer_oov_queries": 60, + "newer_oov_top1_hits": 0, + "newer_oov_top5_hits": 1, + "prefix_chars": 2, + "queries": 3437, + "top1": 0.6424207157404713, + "top5": 0.8626709339540297, + "vocabulary_coverage": 0.992435263311027 + }, + { + "legacy_fill_queries": 141, + "newer_oov_queries": 56, + "newer_oov_top1_hits": 1, + "newer_oov_top5_hits": 4, + "prefix_chars": 3, + "queries": 2486, + "top1": 0.7646822204344328, + "top5": 0.9408688656476267, + "vocabulary_coverage": 0.9903459372485921 + }, + { + "legacy_fill_queries": 346, + "newer_oov_queries": 50, + "newer_oov_top1_hits": 3, + "newer_oov_top5_hits": 12, + "prefix_chars": 4, + "queries": 1492, + "top1": 0.8130026809651475, + "top5": 0.9617962466487936, + "vocabulary_coverage": 0.9865951742627346 + } + ], + "baseline_sha256": "222253417d0a7a705823ffb7e599a3bcf5d5d3daf4a9d76161ac6b3e555aeaad", + "cold_load_ms": 6559.9878, + "database_bytes": 132493312, + "input_sha256": "17fb860b67c0a65da61af4c18011f4968d741a62bcf0fadd0ec67eb4275189a3", + "legacy_sha256": "dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318", + "mode": "combined", + "model_payload_bytes": 61065432, + "position_violations": 0, + "protocol": "legacy-slot-fill-v1", + "sentences": 599, + "warm_max_ms": 9.543299999999999, + "warm_median_ms": 0.24850000000000003, + "warm_p95_ms": 6.3612, + "warm_p95_target_met": true, + "warm_query_count": 16473, + "domain": "conversation", + "split": "test", + "peak_process_rss_bytes": 441393152, + "memory_method": "Windows process peak working set", + "hardware": "Windows-11-10.0.26200-SP0 AMD64 Family 26 Model 36 Stepping 0, AuthenticAMD" + } + ] +} diff --git a/docs/legacy-comparison-results.md b/docs/legacy-comparison-results.md new file mode 100644 index 0000000..e213b91 --- /dev/null +++ b/docs/legacy-comparison-results.md @@ -0,0 +1,42 @@ +# Legacy slot-fill comparison results + +Measured on Windows x86-64 on 2026-10-03 using Rust 1.97.1, the released en-aac-oanc-v1 database and the recovered rc.14 legacy database. The mixing rule was frozen before scoring. Both source hashes remained unchanged. No personal data was loaded, desktop integration performed or model promoted. + +## Accuracy after two characters + +| Partition | Original top five | Newer top five | Combined top five | Gain over newer | +| --- | ---: | ---: | ---: | ---: | +| aac dev | 76.59% | 84.16% | 84.16% | +0.00 percentage points | +| general dev | 63.86% | 67.69% | 67.69% | +0.00 percentage points | +| conversation dev | 68.37% | 86.15% | 86.15% | +0.00 percentage points | +| aac test | 79.25% | 85.17% | 85.17% | +0.00 percentage points | +| general test | 63.41% | 66.40% | 66.44% | +0.04 percentage points | +| conversation test | 68.29% | 86.24% | 86.27% | +0.03 percentage points | + +All six partitions passed position-preservation and top-five non-regression checks at every prefix length from zero through four. This conservative policy cannot displace an existing newer suggestion. When all five slots are occupied, legacy data cannot help, even if it knows the intended word. + +## Words missing from the newer model + +At two typed characters, these are query occurrences, not unique words. The last column counts queries where combined mode appended at least one legacy suggestion, whether correct or not. + +| Test partition | Missing-word queries | Original hits | Combined hits | Queries receiving legacy suggestions | +| --- | ---: | ---: | ---: | ---: | +| aac | 19 | 2 | 0 | 1 | +| general | 361 | 33 | 2 | 10 | +| conversation | 60 | 3 | 1 | 11 | + +## Resource costs + +Each mode ran in a separate process on the same machine. Warm timings cover all scored queries after priming, with at least 1,000 per process. Values below show ranges across partitions for cold load and the worst partition for warm p95, maximum and peak working set. Memory includes evaluation bookkeeping and vocabulary sets, so it is whole-command memory, not pure resident-model size. Database hashing is streamed. + +| Mode | Cold load range | Worst p95 | Maximum query | Peak process working set | Database bytes | +| --- | ---: | ---: | ---: | ---: | ---: | +| original | 4014 to 4216 ms | 2.40 ms | 5.99 ms | 189.7 MiB | 102,690,816 | +| newer | 2465 to 2588 ms | 6.66 ms | 18.18 ms | 364.8 MiB | 29,802,496 | +| combined | 6560 to 6797 ms | 6.40 ms | 14.34 ms | 422.0 MiB | 132,493,312 | + +All modes met the initial 20 ms warm p95 target. The combined predictor adds legacy loading and storage costs while retaining the newer predictor's ranking. These measurements cover this Windows machine only, not macOS/Linux performance. + +The original training corpus is unknown and may overlap these sentences. These are regression comparisons, not evidence of unseen-data accuracy or measured AAC-user benefit. The legacy adapter deliberately applies modern normalization and single-word filtering, so its results are not a byte-for-byte reproduction of historical display text. + +The aggregate [JSON report](legacy-comparison-results.json) includes all prefix lengths, top-one/top-five accuracy, vocabulary coverage, missing-word hits, fill counts, timing distributions, payload sizes and source hashes. The [protocol](legacy-comparison-protocol.md) gives the reproduction command. Retain this as an experiment; no automatic promotion is recommended from these results alone. From 63d7867f994d38b04fdadffd0cf529ba7f133d71 Mon Sep 17 00:00:00 2001 From: Owen McGirr Date: Sat, 3 Oct 2026 09:52:36 +0100 Subject: [PATCH 4/4] Explain larger backfill gains at longer prefixes --- docs/legacy-comparison-results.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/legacy-comparison-results.md b/docs/legacy-comparison-results.md index e213b91..5f22cd1 100644 --- a/docs/legacy-comparison-results.md +++ b/docs/legacy-comparison-results.md @@ -15,6 +15,16 @@ Measured on Windows x86-64 on 2026-10-03 using Rust 1.97.1, the released en-aac- All six partitions passed position-preservation and top-five non-regression checks at every prefix length from zero through four. This conservative policy cannot displace an existing newer suggestion. When all five slots are occupied, legacy data cannot help, even if it knows the intended word. +## Longer prefixes + +The benefit is larger after four typed characters, when fewer newer words match and more slots become available. These are completion-only queries for words longer than four graphemes, so they should not be compared directly with the two-character query population. + +| Test partition | Newer top five at four characters | Combined top five at four characters | Gain | +| --- | ---: | ---: | ---: | +| aac | 94.28% | 95.67% | +1.39 percentage points | +| general | 83.24% | 87.79% | +4.55 percentage points | +| conversation | 95.38% | 96.18% | +0.80 percentage points | + ## Words missing from the newer model At two typed characters, these are query occurrences, not unique words. The last column counts queries where combined mode appended at least one legacy suggestion, whether correct or not.