Repository navigation
explore(kev): qualify decision replacements against current runtime inputs and canonical Scout evidence #4535
Description
Goal
Establish a reproducible Kev-4B baseline that can decide which existing TraceDecay decisions are worth replacing. Reuse the current SystemOne client and evaluation tools. The result should support a small number of measured replacements and identify the machinery each replacement would remove.
Current evidence
Source inspected at 90909cd9d0f2ced7be95d924ce574380303cef80:
- The existing decision client already supports Noul, Choice, Score, typed failure outcomes, bounded concurrency and deadlines. The intent pass batches its questions in one request.
- Runtime reference questions now contain ten intents, including
build_diagnosticsandatomic_edit. The checked-in benchmark has nine categories, includingrefactor_edit, and a different state renderer. The historical scores/thresholds do not qualify today's exact requests. - Plan 41 promises a digest-checked canonical-session exporter to Kev JSONL. No corresponding implementation was found in the inspected scripts/evals/CLI paths.
- Delivery analytics describe delivered candidates. No-candidate exits do not retain the same prediction/window receipt. Delivered-hint data alone misses negative decisions and missed opportunities.
- run_http.py and score.py already provide remote evaluation and useful classification/calibration metrics. Extend them instead of creating another harness.
The official Kev-4B card describes a model that scores supplied options without generating text. Its v1.0 checkpoint is 139fdd94, with temperature 2.41 and 8,192 validated context tokens; the larger serving limit is not equivalent to validated quality. Qualification must pin the actual checkpoint and serving configuration used.
Bounded work
- Freeze the actual runtime question vocabulary, instructions, renderer/settings identity, input limits and thresholds in an evaluation manifest. Reuse the existing canonical settings digest and production rendering contract; avoid a third hand-maintained question list.
- Implement the missing bounded export from existing Scout provenance and outcomes. Rehydrate through the owning transcript API, reproduce the original window bounds and verify its digest. Report missing, changed, pruned or expired evidence explicitly; do not silently approximate an old request or create a parallel transcript store.
- Keep predictions, observed action facts, delivery decisions, explicit feedback, acted/ignored behavior and independent correctness labels separate. Only verified labels may enter training data. Include independently labelled negatives, near neighbors, ambiguity and misleading instructions inside tool output.
- Split related examples by session/project, freeze held-out cases before choosing thresholds, and preserve the historical benchmark as historical evidence. Extend existing calibration/error/coverage metrics and the existing task-effectiveness harness where a replacement claims user benefit.
- Record checkpoint/server identity, backend/dtype/temperature, question/renderer/settings digests, input size, truncation/refusal, measured latency and available usage. Reject
.partialresumes from a different experiment identity; unknown costs remain unknown. - Run inference on a separately provisioned compatible host through the existing typed HTTP contract, with supported authenticated access where required. No model download or local execution in this cloud workspace is required.
- For candidate selection, evaluate option-order sensitivity and an explicit
noneoutcome. Before a Choice result can select a live candidate, tighten the existing Choice decoder around the caller's supplied option set and required probability contract; the currently used intent path is Noul-only.
Acceptance
Produce a reproducible baseline/report and a keep, replace or defer recommendation for each evaluated application. Freeze application-specific error tolerances before reading held-out results. Report task success, unwanted suggestions or missed work, coverage/abstention, p50/p95 latency, cold/warm resource costs and contention with the existing intent pass.
A positive replacement decision must identify the existing code/tests/calls that disappear. If the existing deterministic behavior is as effective and cheaper, retain it. Fine-tuning is a later option justified by measured residual errors and independently labelled data.
Relationship to existing work
This completes the concrete qualification/export gap in Plan 41, whose Phase 1 was merged in #4121. Reuse the completed outcome work in #3654, #3655, #4216 and #4219; this issue does not reopen their broad task-effectiveness scope. Application-specific labels, decisions and deletion targets belong in the linked exploration issues.
This issue is based on source inspection and primary model documentation. No new Kev inference, training or performance measurements were performed while filing it.
Application explorations
- explore(scout): replace generative hint rewriting with bounded Kev candidate selection #4536 — Scout candidate choice/abstention and rewrite-path removal.
- explore(automation): use Kev to diagnose skill work and avoid unnecessary writer calls #4538 — Skill-work diagnosis and unnecessary writer-call avoidance.
- explore(memory): use Kev to triage curation pages before general-agent review #4539 — Memory-page work triage and unnecessary review-call avoidance.
- explore(intent): separate observed actions from keyword inference before expanding Kev routing #4540 — Semantic intent arbitration and legacy hint consolidation.
Suggested order: establish this baseline, then prioritize Scout selection and the existing intent disagreement; evaluate background authoring/curation gates with separate missed-work tolerances.
Activity
fix(scout): validate choice distributions and picked labels
fix(scout): reject explicitly malformed answer type tags
fix(evals): publish complete HTTP results without partial files
fix(evals): preserve foreign files during link probes
fix(evals): atomically publish scored summaries
fix(evals): validate typed HTTP decisions before recording results
fix(evals): constrain HTTP decisions to the selected endpoint
fix(evals): reject malformed scoring evidence
fix(evals): reject inconsistent calibrated score evidence
fix(evals): verify HTTP score experiment fingerprints
docs(evals): correct provenance note and test newline bytes
fix(evals): serialize concurrent HTTP result resumes
feat(evals): support secure bearer authentication for HTTP qualification
perf(evals): avoid quadratic AUC and leave-one-out rescans
fix(evals): hold writer lock through completed publication
111 remaining items
fix(evals): forbid client calibration in captured Scout parity
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4535 — published 79c3aae84f directly to existing DRAFT / NEVER MERGE #4797 (70 commits, targeting master). Single parent
c1adb1fac8, checked exact prior remote head and published withexpected_sha,force=false; re-read confirmed remote DRAFT head and open #4535. No duplicate PR or integration bypass.Production parity defect fixed: captured Scout qualification allowed evaluator-only Noul client-temperature calibration, although the Rust Scout decision client forwards raw Noul probabilities to
decide_intentswithout that transformation. Example: raw 0.55 is below a 0.60 threshold while client temperature 0.25 converts it to approximately 0.6905; the prior qualified report could falsely pass replacement recall. Captured arms now demand frozennoul_temperature: null(and no CLI calibration) through shared manifest validation both before HTTP and at independently scored publication; historical noncaptured evaluations remain unchanged. New Bazel-wired falsifiable test constructs a coherently rehashed captured result, proves raw positive recall 0 versus calibrated 1, checks scorer rejection and checks runner rejection with zero model calls and no output. Three scoped files, no test weakening.Verification boundary: source diff, commit parent, DRAFT state and single real arithmetic example checked; focused Bazel tests could not execute in this checkout (no Bazel / repository checkout). Exact-head hosted CI #42707454580 completed skipped (Release PR integrity only). No nonzero Bazel passing run, independently approved review or actual Kev inference claimed. Still blocked on owning-store canonical Scout transcript export/digest-checked rehydration, genuinely independently reviewed positive/negative labels, effective ten-question runtime settings/renderer parity, pinned live Kev inference, p50/p95 and cold/warm resource/contention, task effectiveness and application keep/replace/defer decisions. Issue stays OPEN under recurring ownership. #4797 and mixed #4776 stay DRAFT — NEVER MERGE/AUTO-MERGE, including integration routes. Release #4763 untouched; no Cargo, Kody/Copilot/external delegation, production, credentials, infrastructure, tags or releases.
fix(evals): verify raw Scout samples against scored evidence
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4535 — published commit
6ac8e822d1ccdirectly to existing DRAFT / NEVER MERGE #4797, targetingmaster(71 commits). Exact prior remote head79c3aae84f42was the only parent;expected_shaplusforce=falseupdated the ref. Remote re-read confirmed #4797 remains open/DRAFT and #4535 remains open; mixed #4776 remains DRAFT / NO MERGE.Concrete evidence-integrity gap closed: mutually rehashing a raw first-response sample's row digest and the result row's sample-response digest could previously pass independent qualification even when the original sample's Noul probabilities, reported model, input tokens or truncation counters contradicted the scored row. The sample verifier now optionally reconstructs these raw fields, including native usage flags and server timing, and compares them with the persisted first result row. The independent scorer enforces this; qualified resume enforces it before more model calls; final result publication rechecks it. Falsifiable, Bazel-wired regression exercises an accepted unmodified control and four coherently rehashed contradictory responses (probability/model/input tokens/drop count). Three evaluation/test files changed. The sample remains self-reported evidence, not a server attestation.
Verification boundary: exact published commit and one-parent fast-forward inspected; author-side PR review only. No Bazel checkout is available here, and exact-head CI #42719916100 only ran the skipped release-PR integrity workflow; no passing exact-head focused Bazel tests, native Rust or independent review is claimed.
Acceptance blockers remain: canonical owning-store Scout transcript export/digest-verified rehydration, genuinely reviewed positive/negative labels, effective ten-question production parity, pinned live Kev inference, p50/p95 cold/warm resource contention/task-effectiveness measurements and per-application keep/replace/defer evidence. Issue stays OPEN. All decision-model changes in #4797 and #4776 remain DRAFT — NEVER MERGE/AUTO-MERGE, including integration bypasses; release #4763 read-only. No Cargo, Kody/Copilot/external delegation, release/tag, credentials, production or infrastructure changes.
fix(scout): require typed Choice picks at the probability maximum
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4535 — published
0a7381c4c10cto the existing issue-specific DRAFT / NEVER MERGE #4797, targetingmaster. It is one commit ahead of exact prior head6ac8e822d1cc; the branch update usedexpected_shaandforce=false. Re-read confirms 72 commits and DRAFT status; mixed #4776 is still DRAFT/NO MERGE, and #4535 remains open.Concrete decoder defect fixed (issue bounded work #7): The production
DecisionEngineClientV1parsed a Choice's declaredchoiceas a string or converted JSON booleans/numbers into string labels. Thustruecould select the offered label"true", and1could select"1"even though neither was a typed string label. It also allowed a declared pick to be lower than another offered probability as long as the gap was <=0.001, conflating distribution-sum rounding tolerance with the selected-option argmax invariant. The decoder now requires a string label from the supplied option set and a probability that is an actual maximum; distribution-total rounding tolerance and omitted-choice behavior remain unchanged. Removed the now-unused scalar-to-label coercion helper.Added an actual Rust test
choice_pick_requires_string_label_and_maximum_probabilityin the same production module: three previously accepted invalid responses (booleantrue, number1, and an almost-tied but lower explicit pick) must beMalformed; a true string-label tie remains accepted. The new test is regression-backed and reachable via the agent-hosts Bazel unit target; no Bazel execution is claimed in this connector-only environment.Exact-head hosted CI #42723832240 completed skipped. No exact-head native Bazel pass, Clippy/rustfmt run, independent reviewer sign-off, real pinned Kev inference, canonical Scout transcript exporter, verified labels, ten-question runtime parity, or resource/contention evidence is claimed. These remain completion blockers, not grounds to weaken tests or merge. Refs #4535 only — do not merge/auto-merge, and do not route via #4776 or integration. No Cargo, release #4763, production, credentials or infrastructure changes.
fix(scout): reject duplicate keys in decision HTTP answers
style(scout): align strict decoder with Rust formatting
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published two exact-expected-head, non-force fast-forward commits to the existing issue-specific DRAFT / NEVER MERGE #4797, targeting
master: 642015a1 and 1a3058c5 (Rust formatting follow-up). Verified 74-commit head1a3058c561ae4d0e10dc2d3090f6f1bca5fa493f, DRAFT, #4535 still open, and mixed #4776 separately DRAFT/NO MERGE; no duplicate issue PR or integration bypass.Root cause / source repair: The qualification HTTP evaluator explicitly rejects ambiguous JSON object keys, but production
DecisionEngineClientV1::ask_blockingdecoded response bytes straight intoserde_json::Value, which silently retained the last repeated key. A response could therefore rewrite a reportedmodel, Noulnoulprobability, or Choice option probability via duplicate-key ambiguity and still become a successful Scout decision. Production now uses a recursive Serde visitor that rejects duplicate object keys at every nesting level before interpreting any answer; all otherwise valid typed responses retain the original parser. This closes a concrete production/qualification evidence admission mismatch.Falsifiable regression: New native loopback Rust test sends three otherwise valid duplicate-key responses (Noul probability, top-level model, nested Choice distribution), asserts each is
Malformedafter exactly one request, and proves a regular Noul response remainsAnswered. These negatives fail on the previousserde_json::Valuedecoder. No dependencies or protocol changes.Verification limits: No Bazel toolchain/checkout is provided in this execution environment; I did NOT invoke Cargo or claim any passing exact-head Bazel/native tests, Clippy, rustfmt, or independent reviewer approval. Latest head-specific CI #42735804663 only executed Release PR integrity and was skipped. Prior green CI is not evidence for this head. Required next evidence includes nonzero
bazel test --config=ci --test_output=errors //crates/tracedecay-agent-hosts:...(select the actual native target), repository Bazel rustfmt/Clippy and independent source review.Outstanding acceptance: canonical owning-transcript export/digest-verified window rehydration, independently verified actual labels, pinned Kev inference over authenticated compatible HTTP serving, current ten-question runtime parity, task outcomes, cold/warm resource and contention measurements, and quantified keep/replace/defer decisions. No releases/tags/credentials/production/infrastructure touched. DRAFT / NO MERGE or AUTO-MERGE for #4797 and mixed #4776 (including integration paths); issue stays OPEN.
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published
ab5acd645281on the existing issue-owned DRAFT #4797, basemaster, as a single-parent, exact-expected_sha,force=falsefast-forward from1a3058c561ae4d0e10dc2d3090f6f1bca5fa493f. Verified remote ref/PR headab5acd645281bb3d748f0009770e53e7f43135ab, 75 commits, open/DRAFT, and mixed #4776 separately open/DRAFT. NO MERGE / NO AUTO-MERGE, including integration paths. Issue remains open.Concrete production parity defect repaired: the Rust
parse_answersdecoder silently accepted surplus answer keys outside the caller's requested question set, while the existing qualified HTTP evaluator strictly requires exactly the requested keys. A successful Scout result could therefore represent a batch the qualification evaluator would reject. Production now refuses surplus response answers asMalformed. New native Rust regression makes two previously accepted extra-answer responses fail (unrequested Noul and Score); a valid exact two-question batch remains accepted with both predictions preserved. Source/test change is one file, with no weakened assertions or compatibility shims.Verification: commit and remote patch independently re-read. Latest exact-head hosted Actions #42739869913 completed with its only job, release-PR integrity, skipped. There is no Bazel executable/checkout in this execution environment; the new focused Rust test, rustfmt, Clippy, native host execution and an independent reviewer check are not claimed passing. No Cargo or external delegation was used. Canonical owning-transcript export/digest-checked rehydration, verified labels and actual frozen Kev inference, ten-question replay, task-effectiveness and cold/warm resource/contention measurements, and keep/replace/defer findings still block acceptance. No release #4763, production, secrets or infrastructure were changed.
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — new verification blocker at existing DRAFT #4797 head
ab5acd645281bb3d748f0009770e53e7f43135ab: the #4776/#4797 readiness worker reports that native/Bazel HTTP and qualification fixtures/expectations fail under the stricter admission contract. The worker has not yet published exact failing test names/output, so this is not a passing exact-head run, an independently approved repair, or enough evidence for safe edits. I inspected currentmasterruntime/policy/question definitions, PR source/tests, current root/nested instructions, related PRs, reviews and available Actions; exact-head public Actions #42739869913 is only a skipped release-integrity job. I acknowledged the worker's coordination request in this PR response and paused all #4797 ref/source writes pending their exact failure report and explicit handoff. No merge, auto-merge, Cargo, altered tests, branch rewrite, release, or production work. Keep #4535 OPEN and #4797 plus mixed #4776 DRAFT / NEVER MERGE including integration bypass. Existing issue-specific PR remains the only implementation lane; no duplicate PR.ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published
74bb6b8a5573to the existing DRAFT / NO MERGE PR #4797 after the readiness worker completed its native/Bazel diagnosis and explicitly returned writer control. Exact remote headab5acd645281advanced withexpected_sha,force=false; six-file patch verified and PR still DRAFT targeting master.Concrete repair: production request commitment was hashing compact JSON while the real HTTP request sent pretty JSON, so canonical Scout evidence incorrectly certified the wrong request bytes (observed 100 vs 143 bytes). One Rust serializer now supplies the exact UTF-8 bytes for both HTTP and digest; captured qualification and documentation now validate the same pretty wire bytes, including noncanonical refusal. Corrected the legacy 3-question successful test fixture (without weakening exact-batch rejection), fixed a sample's missing real usage telemetry, preserved adversarial raw-sample and counter tests at their real strict validation boundaries, and updated sample-publication refusal assertion to the secure no-clobber behavior.
Not yet verified on new head: the worker's prior 29/31 native, 65/66 HTTP, 73/77 qualification, 6/7 CLI, 26/26 scoring describe old head only. This execution environment has no Bazel/checkout; no new passing Bazel/native, rustfmt/Clippy, hosted CI or independent approval is claimed. The issue's actual canonical Scout export, verified real labels, 10-question source parity, authorized pinned Kev inference, effectiveness/resource/contention evidence and application recommendations remain absent. Keep #4535 OPEN and #4797 + mixed #4776 DRAFT / NEVER MERGE or auto-merge, including indirect integration. Release #4763 untouched; no Cargo/delegation/secrets/production/infrastructure.
fix(evals): reject in-flight substitution of raw qualification sample
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published
65d9828047e8to existing DRAFT #4797 with an exact-head nonforce fast-forward from74bb6b8a5573. Found a new resumable qualification evidence gap: replacing a first-response sample during inference with semantically identical, re-encoded JSON passed its prior parsed-response/row checks. The runner now binds the original raw sample byte SHA-256 across admission and final publication, and sample writing uses byte-exact UTF-8 across operating systems. Added a falsifiable mid-inference substitution regression with existing recovery control, plus updated direct sample digest expectations. Verified remote two-file diff, parent/head and draft states. No passing exact-head Bazel/native/CI or independent review is claimed; real Scout canonical export, independent labels, pinned Kev inference, current ten-question parity, and measurement/report acceptance remain unproved. Issue OPEN, #4797 and mixed #4776 DRAFT / NEVER MERGE or auto-merge (including indirect integration), release #4763 untouched; no Cargo/delegation/prod/credentials/infrastructure.