Skip to content

Add optional local Select development scorer prototype - #34

Merged
snchimata merged 4 commits into
mainfrom
feat/select-v2-runtime-prototype-v1
Oct 16, 2026
Merged

snchimata merged 4 commits into
mainfrom
feat/select-v2-runtime-prototype-v1

Conversation

@snchimata

@snchimata snchimata commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Add the optional source-only packages/tokenfold-select development prototype using the already-shipped schema-1 Select approved-scorer file protocol. Scorer schemas, compression rendering, CLI/binding/npm source and production defaults are unchanged. Separate required CI repairs below include rustls/npm lock metadata and explicit observation-ledger lock release.
  • Package byte-exact canonical model/packer modules without research-checkout imports. Include bounded metadata/hash checks, explicit development opt-in, single-complete-window refusal, finite aligned score responses, isolated native bootstrap and bounded offline direct-child execution.
  • Add Windows/Linux Python 3.12 CPU-only installation/contract CI and an unreleased changelog entry. Existing publication workflows and released versions are unchanged.

Validation

  • Main-based wheel built and installed privately without ML dependencies; 8 stdlib contract tests passed from an unrelated directory in an isolated native interpreter.
  • Wheel inspection verified packaged Python bytes against source and absence of weights, datasets and developer-local paths.
  • cargo fmt --all --check, python eval/run_fidelity.py --gate --profile smoke-first-consumer and git diff --cached --check passed.
  • The same package Python sources previously passed one frozen synthetic local FP32 CUDA integration check in the development checkout: bit-identical canonical/installed scores and a positive native model receipt with exact allocation/text/budget parity. This is not representative quality/latency or a main-binary GPU qualification. CPU CI does not execute models.
  • After the required CI security finding, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace --locked, cargo audit, and cargo deny check advisories bans licenses sources all passed locally. Existing allowed duplicate-version warnings remain warnings; no advisory was ignored.
  • After the required npm lockfile finding, local npm ci and npm test passed (12 tests, Node 26.2.0/npm 11.13.0, using the freshly built local main-based CLI). Node 22/24 remote checks must pass on the updated head.
  • Local PyO3 extension-specific tests were not rerun because those sources are unchanged; existing full PR CI covers them and must be green before landing.

Explicit non-goals and remaining gates

This is not the Select v2 developer-preview release. No checkpoint, corpus, ML dependency, hosted inference, model download, package publication, release or tag is included. Redistribution/release/quality qualification remains false. Model artifacts and the separately provisioned supported GPU environment are caller-owned.

The scorer does not certify generic JSON grouping or a retention ratio. Controller integration, representative CUDA/lifecycle validation, clean-user GPU provisioning, rights/final notices and complete prospective matched-budget quality and full compression latency evidence remain open. Full v2 still requires untouched five-workload superiority against pinned Select v1 and healthy standalone Kompress v2, with the declared safety/support/raw-harm/statistical gates and separate cold/warm/contiguous-pipeline reporting.

The incomplete FP16 launch comparison has not been resumed or retried; its requested fresh-run approval remains pending. The large comparison branch and unrelated dirty work are not included in this PR.

Required security repair

The first required security job failed on main's pre-existing rustls 0.23.45 lock entry via ureq/tokenfold-proxy. A separate reviewable commit updates only rustls to the minimum patched 0.23.46 (version/checksum only; unchanged dependency list). This addresses RUSTSEC-2026-0346, 0347, 0348 and 0349. No security-gate suppression or bypass is added. Both new Windows/Linux package contract jobs passed on the first commit; all checks must pass on the updated head before merge.

Required npm clean-install repair

The rustls-updated head passed security and CodeQL, but Node 24 failed at npm ci because main's five existing platform entries were {optional:true} placeholders lacking version/integrity metadata; matrix fail-fast cancelled Node 22. npm's package-lock-only resolver restored complete metadata for the already-declared/published exact 0.5.1 packages. Root constraints and all dev-dependency lock entries are unchanged. No npm source, publication workflow, released version, optional-validation suppression or package publication is involved. Actual clean install and all 12 npm tests passed locally; all required remote checks still must pass on this new head.

Required ledger lock-release repair

With Node jobs repaired, coverage exposed an existing immediate-reopen failure in the persistent session ledger. The private lock file relied only on descriptor close. Linux locks persist through duplicated/fork-inherited open-file descriptions (flock semantics); a parallel child before exec can temporarily retain ownership. A bounded local WSL/Linux OS control demonstrated close-only refusal and explicit-unlock success while the child remained alive, and waited all child processes. This was an OS control, not a local Rust/Linux build.

A small Persistence::drop explicitly unlocks before the normal file close, binding ownership to the ledger lifetime. Live-owner refusal, corruption handling, schema, snapshots and fail-closed trust remain unchanged. A deterministic Unix regression retains a duplicated descriptor, verifies live-owner refusal, then immediate reopening after owner drop and continued exclusivity of the new owner. No retry loop, ignore or gate suppression is added. Windows formatting, clippy/all-targets, locked workspace tests (692 passed, one existing ignored) and deterministic fidelity smoke passed. The Unix regression and Linux coverage must pass in CI; local WSL lacks a Rust toolchain/compiler.

@snchimata
snchimata merged commit 1f242a2 into main Oct 16, 2026
24 checks passed
@snchimata
snchimata deleted the feat/select-v2-runtime-prototype-v1 branch October 16, 2026 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant