Skip to content

Evaluate Taskmaster-2 data mixtures for word prediction - #12

Closed
enaboapps wants to merge 2 commits into
mainfrom
experiment/taskmaster2-data
Closed

enaboapps wants to merge 2 commits into
mainfrom
experiment/taskmaster2-data

Conversation

@enaboapps

@enaboapps enaboapps commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Result

Adding 21,000 Taskmaster-2 USER sentences at 1x and 3x did not pass the frozen development criteria. Mean early top-five accuracy changed by -0.128 and -0.751 percentage points respectively. The lighter candidate was evaluated diagnostically on AAC, general English, conversation and a filtered COMM2 set. At two graphemes, top-five accuracy fell on all four sets, although some top-one and longer-prefix results improved.

The 1x database grew by 13.9%; worst warm p95 was 6.885 ms against 6.425 ms for the baseline. Keep the shipping model unchanged.

Change

Adds a reproducible experiment command, pinned source checksums, a predeclared selection protocol, synthetic tests, aggregate JSON results and a report. COMM2 stays out of training; exact overlap checks preserve all existing development/test partitions. Prior legacy-backfill accuracy is included only for matching test hashes, with no cross-run timing comparison.

Production APIs, model files, packaging and Switchify PC are unchanged. No corpus text or experimental databases are committed or published.

Validation

  • Completed 17 local scoring runs, each with more than 1,000 queries at prefixes 0 through 4; no personal learning.
  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets -- -D warnings
  • cargo test --locked: 22 tests passed
  • Python unittest discovery: 15 tests passed
  • cargo audit --deny warnings
  • Verified unchanged production training and baseline hashes; prior newer-mode test accuracy exactly reproduced.
  • Independent read-only review of final head 5d595cf: no actionable findings. All required CI passed on this head: https://github.com/switchifyapp/switchify-prediction/actions/runs/37123756202 .

See docs/taskmaster2-results.md for gains, regressions, costs and limitations, and docs/taskmaster2-protocol.md for the reproduction command.

Closes #11.

@enaboapps enaboapps added this to the v0.2.0 milestone Oct 3, 2026
@enaboapps
enaboapps marked this pull request as ready for review October 3, 2026 12:52
@enaboapps enaboapps closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate Taskmaster-2 training mixtures for early word prediction

2 participants