[ScriptedAlchemy]

Refs #4535 — partial evidence/qualification integrity only; NOT a Kev replacement or completion.

Root cause

The historical intent HTTP runner scores nine hand-maintained categories and permits case-level threshold exploration. Its old leave-one-out rates do not prove independent session/project generalization, and there was no fail-closed path binding separately reviewed labels, project-isolated holdout cases and precommitted error tolerances to the precise HTTP experiment. Existing scorer reports p90 rather than the p95 requested for qualification. A historical report must not silently become a current-runtime replacement recommendation.

Changes

• Add optional frozen qualification manifest to the existing HTTP runner. It binds the original dataset digest, exact question/instruction digest, independently declared annotation reviewer and review reference per case, complete case coverage, project/session-disjoint tune versus holdout groups, reviewed positives and negatives, fixed per-question thresholds and predeclared error tolerances.
• Reject missing/changed labels, self-review, drifted thresholds, overlapping groups and invalid manifests before a model call; send only frozen held-out cases. Bind raw manifest SHA-256 into resumable experiment identity.
• Extend existing scorer via score.py --qualified MANIFEST DATASET RESULT: refuse mismatched experiment, annotation and holdout order; report exclusively frozen-threshold false-hint/positive-case recall and measured p50/p95 HTTP latency with pass/fail against frozen tolerances; keep historical output untouched.
• Add //evals/intent_engine:qualification_test with falsifiable positive/negative, group leakage, threshold drift, tampering, exact-identity HTTP subset and no-network-refusal tests. Update usage/limitations documentation.

Additional provenance fix (2026-11-07, 09915be9)

• The scorer formerly trusted a result's declared dataset digest and self-consistent request hashes without reopening original frozen cases. An incorrect prompt or request fingerprint could be scored after rehashing the result metadata.
• run_http.py --dataset ORIGINAL.json now accepts a separately frozen original dataset. score.py --qualified MANIFEST ORIGINAL.json RESULT requires the same original bytes, checks every annotation against the complete file, and reconstructs each held-out request using the runner's own question/state renderer. Refuse dataset drift or a forged-but-rehashed request fingerprint before publishing a qualified report.
• Regression tests cover both refusals and an explicit non-default dataset file; historical scoring remains separate. This still does not establish ten-question production renderer parity, or prove the declared reviewers/remote server identity.

Additional qualification-integrity fix (2026-11-07, c7b6e0ee)

• An all-negative-only false-hint metric failed to count wrong extra suggestions on positive cases; a model could pass recall and negative-case false-hint thresholds while producing unwanted additional guidance. Freeze a separate max_unwanted_hint_rate, report all-case unwanted suggestions, missed positive cases, coverage and abstention.
• A server could omit state_tokens_dropped or report actual truncation while a frozen-threshold score passed. Now measured zero drops on every holdout case is required to report a passing qualification; unknown truncation is never treated as zero.
• Enforce session ID disjointness across tune/holdout even when a misleading project ID differs. Add targeted falsifiable regressions for all three failure modes. This remains historical evaluation only, not canonical runtime parity.

Additional group-provenance repair (2026-11-07, 711a18f8)

• The freeze manifest previously allowed its own project_id and session_id claims to define train/holdout isolation without checking the original dataset's source group identifiers. A rehashed manifest could therefore invent an independent group even when the underlying case belonged to another project/session.
• Qualification now requires every frozen source case to contain matching project/session identifiers, refuses substituted or missing identities, and retains the original dataset-byte digest. Regression tests cover missing group evidence and a forged, rehashed manifest/result pair.
• This establishes consistency of declared groups with frozen input, not independent verification of real Scout session provenance, production renderer parity or independent labeling.

Frozen client-calibration integrity (2026-11-08, 1259eef2)

• Qualification previously froze per-question thresholds but allowed an arbitrary --noul-temperature for client-side probability calibration. Re-running the holdout with different calibration could select an effective decision boundary post hoc despite an unchanged frozen threshold/holdout manifest.
• The frozen manifest now requires noul_temperature (null or a positive finite value); the runner rejects a mismatch before HTTP work, and qualification refuses a recomputed result/manifest pair with inconsistent calibration. Dedicated regression cases cover invalid/missing declarations, mismatched no-network refusal, a matching calibrated run and post-hoc rehashing. Still historical nine-question evaluation, not actual Kev/runtime parity.

Contiguous resume-order validation (2026-11-08, 1e27a2c6)

• Resumable HTTP evaluation previously accepted a syntactically valid .partial containing the second case before the first, or only the second. It then synthesized a final result in canonical order even though the actual inference/warm-up sequence was different.
• Require every resumed row to form a contiguous prefix of the frozen case order, before HTTP calls, appends or completed publication. Add a falsifiable runner test for skipped/reversed rows, unchanged partial bytes, zero network calls and a genuinely interrupted prefix that still resumes. Updated the HTTP runner contract documentation.
• This is replay/evidence correctness, not a runtime-parity experiment, authenticated source provenance or a real Kev measurement.

Frozen HTTP arm integrity (2026-11-08, f4494312)

• Found that a frozen held-out manifest bound labels, thresholds and calibration but not the HTTP renderer/config, requested model, remote endpoint or complete serving declaration. Multiple renderer/model/host arms could be tried on the same holdout, and only the best reported, despite internally consistent per-run experiment hashes.
• Manifest arm now precommits one renderer, nullable model selector, endpoint SHA-256 and complete sorted serving-config digest. The runner refuses mismatched or multiple arms before making HTTP requests; independently supplied results are checked against the same frozen arm at scoring. Rehashed endpoint/server changes cannot certify an alternative arm.
• Added falsifiable Bazel-wired tests for missing/malformed arms, changed renderer, extra renderer, model, URL, serving checkpoint, and forged-rehashed result metadata. Existing positive run fixtures now use the same manifest-frozen server declaration. This is not independent source authorization or remote checkpoint attestation, and does not prove production Scout parity.

Frozen HTTP request-size evidence (2026-11-08, 3f1b3322 + 990a88bc)

• The HTTP evaluator previously recorded a request digest but not the exact submitted JSON body size; qualification could not establish an input-byte ceiling or report comparable measured input sizes. The model's validated context-token capacity cannot be inferred from a JSON byte count.
• A frozen, strictly positive max_request_bytes now rejects oversized held-out requests before creating result files or sending model traffic. Each result records the actual serialized JSON byte count. Scoring reconstructs the exact request, rejects absent/forged counts and over-limit bodies even under self-consistently rehashed result/manifest metadata, and reports min/p50/max/total request bytes plus the frozen limit.
• Resumed qualified partials with a modified/missing count are rejected before HTTP append; the old partial remains unchanged. Falsifiable Bazel-wired regressions cover malformed limits, zero-network refusal, forged sizes, a post-hoc tighter limit, emitted sizes, and tampered partial denial with successful recovery. This is byte-size provenance, not verified input token count, current Scout runtime parity, or actual Kev model evidence.

Ambiguous JSON evidence refusal (2026-11-08, 23cd79ce)

• A last-key-wins JSON parse allowed an original source case, frozen qualification manifest, serving declaration, partial log, HTTP answer, or qualified result to contain duplicate keys even when independently reviewed tooling might interpret a different key. Python's default JSON parser also accepted nonfinite NaN/Infinity constants.
• Reject repeated object keys (including duplicates with identical values) and nonfinite constants at these evaluation boundaries. A malformed frozen input refuses before network/results creation; malformed partial rows preserve the existing bytes; qualified scoring refuses ambiguous result JSON before publishing a report. Preserve the historical scorer's existing output format.
• Add falsifiable regressions for manifest/dataset/server duplicates with consistent frozen hashes, nonfinite manifest extension, malformed real HTTP parsing, ambiguous partial plus genuine resume, and duplicate qualified result keys. Source/test change is limited to evaluation files; it does not prove independent reviewer authenticity, current ten-question Scout parity or Kev quality.

Numeric source truncation evidence (2026-11-08, a45f85a5)

• Found a qualification false positive: a result with per-case dropped: 0 but no retained numeric server drop/used-token counter passed the aggregate zero-truncation check. That normalized field alone can be forged and is not original response telemetry.
• Qualification now needs a direct server-reported dropped-token count or the native original/used state-token pair on every held-out row before untruncated_input_evidence can pass. Reports disclose source counter coverage without discarding the measured rows. This is still self-reported telemetry, not remote attestation.
• Added falsifiable regression for a forged zero-drop result with one source counter removed, a positive native-counter control, and a CLI failing-qualification case; updated test fixture counts consistently for measured dropped/unknown cases. Historical scorer and frozen tolerance semantics are unchanged.

Refuse missing model evidence before additional held-out inference (2026-11-08, 0b7ead4a)

• A qualified run with a null --send-model could append a response with no model name and continue querying the holdout. A matching .partial could also lose its model name and resume. Scoring would only reject the incomplete reported-model evidence after spending inference.
• Require a nonblank server-reported model in every qualified response, including default-selector runs, before append; check the same invariant on resumed rows before new calls. Historical unqualified behavior is untouched.
• Falsifiable Bazel-wired regressions cover missing model on first/second response, zero appended invalid rows, no completed result, missing/null/blank model in persisted partials, zero extra HTTP calls and byte-identical partials on refusal, plus successful recovery from a valid prefix. Existing fixture responses now report a model identity. Model names remain unverified server declarations, not checkpoint attestation.

Precommitted rendered request bodies (2026-11-08, d5527caf)

• A frozen arm previously pinned only a renderer name, questions, server and dataset. A changed implementation of the same renderer could produce different prompts under the same held-out manifest, and its after-the-fact request fingerprints would self-consistently describe the changed experiment.
• Require holdout_requests_sha256 predeclaring the ordered held-out [case_id, SHA-256(exact HTTP request bytes)] pairs. The runner refuses a changed body before inference and before outputs, and independent scoring checks the recorded experiment's request digest against the frozen manifest. Re-freeze manifests when renderer behavior legitimately changes.
• Added two falsifiable regressions: same-name renderer drift must make zero model calls and zero results directories; self-consistently rehashed scoring results still cannot contradict the frozen request digest. Updated existing calibrated/explicit-model/ten-question fixtures and documentation. Four files changed.
• This is a manifest commitment, not canonical Scout origin proof or independent attestation. No focused Bazel test pass, independent review, canonical Scout exporter, real Kev inference or runtime parity is claimed. Prior exact-head CI #42505651465 completed non-native gates successfully, with native tests skipped.

Owner-only Scout HTTP evidence files (2026-11-08, eb09b12d)

• Identified a source-data confidentiality gap: the eval runner used create modes 0666 for raw request/response samples, resumable files and published results. Under ordinary umask 022, that exposed sensitive transcript input and evaluation case information to other local users.
• All three new artifact types now use mode 0600 on POSIX; safely locked earlier permissive partials have group/other access revoked without rewriting their evidence. Existing no-clobber hard-link publication, exact resumability and header validation remain in place.
• Added nonzero, falsifiable Bazel-wired regression tests under umask 000 for sample/final permissions, interrupted partials, safe permission tightening before resumed network work, byte-identical evidence and successful resume. No real Scout export or inference was performed.

Frozen, byte-exact Scout HTTP capture replay (2026-11-08, 8af44e0f)

• The historical Python renderers cannot recreate the exact Rust Scout state/typed HTTP request. A new optional S_scout_captured arm accepts only the previously frozen, ordered original HTTP request JSON bytes, preserving field order, whitespace and escaping when sending to the model. This does not capture or independently attest the originating Scout request.
• The manifest binds both the original capture file SHA-256 and the ordered per-case wire request digests. Runner and scorer fail closed for edited/missing capture bytes, reordered heldout cases, mismatched Noul/model contract, extra request keys and request-byte overflow; the heldout scorer independently rechecks each transmitted wire fingerprint. Historical 9-question evaluations remain separate.
• Added Bazel-wired nonzero regression cases for positive byte-exact send and scoring, altered/forged capture refusal before any network or partial write, and scoring CLI denial without the original capture followed by successful frozen replay. Tests use synthetic requests, not Kev; no executed Bazel pass or production parity is claimed.
• Published six evaluation-only source/test/doc files with exact-head, nonforce fast-forward. This remains Refs explore(kev): qualify decision replacements against current runtime inputs and canonical Scout evidence #4535, partial work, DRAFT — NEVER MERGE. The canonical owning-transcript exporter and digest rehydration, independently verified real Scout capture/ten-question settings, real Kev model inference and cold/warm resource measurements, independent correctness labels, and application-specific replacement decisions are still missing. Hosted CI #42513291027 was queued on the exact head and native tests are not yet established.

Verification

The issue-specific branch targets master and remains DRAFT — NEVER MERGE; latest published tip 8af44e0f375dfeaa68328ac7242b1a4c40cf4a03 is 25 commits with 8 changed evaluation files, safely extended from a45f85a5 through an exact expected-head, non-force fast-forward. No Cargo was invoked. No independent review, actual Kev inference, canonical Scout source export, current ten-question parity, or replacement report is claimed.

Prior hosted CI #42480141209 completed with repository, dashboard and benchmark gates green but native jobs skipped. Exact-head CI #42505651465 completed successfully for non-native gates; native lanes were skipped for native=none. New exact-head CI #42508352324 is queued with native=none; its results must be checked independently. the draft workflow reports native=none and normally skips the focused Bazel tests. Green repository/frontend gates are not behavioral proof of the qualification tests. A complete local Bazel checkout is unavailable in this runtime. Required nonzero focused Bazel execution on the exact head, plus applicable native CI, remains:

bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test

Independent reviewer verification of labels and source provenance is outstanding.

Deliberate limitations — do not merge

This still uses the historical nine-question renderer. It does not implement canonical Scout transcript-window rehydration/digest verification, actual current ten-question runtime parity, negative/no-candidate Scout receipts, independently verified label truth, inference on a pinned Kev host, task-success trials, cold/warm resource contention, or keep/replace/defer decisions. Reviewer identity/reference fields are declarations, not cryptographic proof. This PR is a safety-critical but partial evaluation slice, must stay DRAFT, and must never be merged or auto-merged under current Kev/decision-model hold. Issue remains open.

Separate from mixed DRAFT #4776 (which also contains a #4535 partial HTTP transport fix and is explicitly nonmergeable), and release #4763 remains read-only. No release/tag/production/infrastructure/dependency changes.

Captured Scout endpoint parity (2026-11-09, 29d9a297)

• Root cause: a manifest could refreeze the endpoint digest to an arbitrary HTTP route while calling a byte-exact request a Scout replay. Actual DecisionEngineClientV1 appends /v1/systemone to its base URL, so a different path/query/fragment is not the same runtime contract.
• The qualified captured arm now refuses a noncanonical route, query or fragment before network traffic or result creation. Historical non-Scout evaluation endpoints are unaffected. Added Bazel-wired positive and falsifiable rehashed-negative regression cases. See 29d9a29.
• This is a transport-contract guard, not canonical transcript export, checkpoint attestation, independently verified labels or actual Kev inference. The scorer still has only the endpoint digest, not its original URL; independent source attestation remains unavailable.
• NO MERGE or auto-merge. Refs explore(kev): qualify decision replacements against current runtime inputs and canonical Scout evidence #4535; not Fixes explore(kev): qualify decision replacements against current runtime inputs and canonical Scout evidence #4535.

At f6cde284e6, all 156 native macOS Bazel evaluation-harness tests passed across the HTTP runner (65), qualification (59), qualification CLI (6), and scoring (26). The truncation regression now checks the actual HTTP timeout against the frozen manifest. These tests use fixtures. Independent holdout labels, canonical transcript provenance, pinned Kev inference, and the draft hold remain outstanding.

At c4f38a113b, the repository rustfmt target passed after formatting the new Scout receipt implementation. The native macOS agent-hosts suite passed 574 tests, including retention of the negative decision, raw model probability and measured latency without transcript text. Its one live-provider evaluation is opt-in. The four Bazel evaluation-harness targets also passed all 156 cases (65 HTTP, 59 qualification, 6 qualification CLI, 26 scoring). The independent qualification inputs and owner hold still apply.

At b0af164914 the complete native macOS agent-hosts suite passed 576 cases, including the exact 1 MiB decision response, one extra byte, and an open-ended over-limit stream. One opt-in live-provider test was ignored. The repository-wide Bazel formatter gate passed after correcting the formatting of the new loopback fixture. The previous 156 evaluation-harness cases and the labelled model qualification limits are documented above.

At eabbdc6e60, strict Clippy passed for the production agent-hosts library and its unit target after removing one redundant import. The native unit target also passed all 576 tests (one opt-in live-provider test ignored, identical binary retrieved from Bazel cache).

[changeset-bot]

⚠️ No Changeset found

Latest commit: 65d9828

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Published direct source/test fix d656712ccff897cbab79ace4a6e26c38a927f928 — Refs #4535. DRAFT / NO MERGE remains mandatory.

Root cause: score.py --qualified MANIFEST DATASET RESULT unconditionally exited 0 after printing the qualification report, including when frozen false-hint, recall, latency, or complete-input checks rejected the experiment. A shell/CI caller could therefore treat a rejected model as successful.

The command now publishes the full JSON measurements as before, names failed checks on stderr, and exits 1 when qualification fails. Passing qualification still exits 0. Also corrected the unqualified-mode refusal to include the required DATASET argument.

Added //evals/intent_engine:qualification_cli_test: real subprocess coverage of the scoring entry point, a passing control, six rejection variants covering all five checks, replacement of an earlier passing report with the current failing report, mismatched original dataset refusal, and the complete required-command diagnostic. These are evaluator fixtures, not Kev inference measurements. Existing tests/assertions and frozen tolerances are unchanged.

Concurrency: the first expected-head update did not land because the branch advanced to 711a18f8. I inspected that split-group provenance fix and published a child of it with expected head 711a18f8 and force=false. Its source/test/documentation changes are preserved. No duplicate PR or branch rewrite.

Verification limitation: original score.py bytes were verified against the Git blob before editing; changed Python files were statically parsed. No Bazel test execution or actual Kev inference is claimed. This runtime has no Bazel/full checkout and GitHub DNS fails from the shell. Current hosted CI is queued with native=none; it is not evidence of these tests passing. Independent review remains outstanding. The existing workflow supports a manual native dispatch, but the connected GitHub actions exposed here do not include workflow dispatch; no workflow or infrastructure was changed to bypass that.

Required nonzero Bazel execution on this exact head:

bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errors

For sweep/PR-maintenance owners: retain the same issue branch and this commit; keep #4797 and mixed #4776 DRAFT, do not merge/auto-merge or route the changes through another branch. #4535 remains open, and release #4763 was untouched.

[ScriptedAlchemy]

Refs #4535 — published direct issue-owned fix f4494312 to this existing DRAFT / NEVER MERGE PR. Exact parent 1e27a2c6, expected-head nonforce update, no branch rewrite/duplicate PR.

Found: frozen holdouts still allowed post-hoc cherry-picking across the historical HTTP runner's three request renderers, requested model selectors, endpoint URLs and serving declarations. Individually self-consistent hashes did not bind the arm selected before inspecting the held-out predictions.

Fixed: a required arm contract now freezes one renderer, nullable model selector, endpoint SHA-256 and full serving-config digest. HTTP preflight rejects multiple/changed arms before any inference or partial output; qualified scoring rechecks the experiment against the frozen arm. Regression cases cover missing/malformed arms, renderer/model/endpoint/server drift with zero network calls, and self-consistently rehashed results attempting to swap endpoint/server metadata. Tests are in the existing Bazel //evals/intent_engine:qualification_test target.

This prevents one class of evaluator-side holdout selection leakage, not independently verified labels, authenticated checkpoint identity, canonical Scout transcript-window rehydration, ten-question production renderer parity, actual Kev inference, task-success trials or resource contention measurement. No acceptance completion or Kev performance claim.

Verification pending: new hosted CI run #42455050174 was queued with native=none; draft native Bazel targets are normally skipped. The current runtime lacks a complete checkout/Bazel, so no successful Bazel test run is claimed. Independent review has not been submitted. Require nonzero execution of bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test on this exact head.

Existing sweep/PR maintenance: keep #4797 and mixed #4776 DRAFT and never merge/auto-merge or bypass via an integration branch. #4535 remains open, release #4763 remains read-only, no Cargo, credentials, infrastructure, release or production actions.

[ScriptedAlchemy]

Refs #4535 — published ed80f99c to the existing DRAFT #4797 using expected-head 990a88bc, force=false (14 commits; still targets master). DO NOT MERGE / AUTO-MERGE this PR or mixed DRAFT #4776, including through integration branches.

Root cause fixed: the frozen HTTP arm pinned the requested model and operator-supplied server configuration but never recorded/checked the model reported by each response. The service could silently switch its default between held-out requests, or contradict an explicit --send-model, while qualification still reported success. The evaluator now records observed_model in every new response row, refuses an explicitly contradictory model before appending evidence, and fails the qualified report on missing, malformed, inconsistent, or mismatched reported models. The report identifies the consistent self-reported model if present. Added falsifiable positive/negative and end-to-end HTTP runner regressions plus documentation.

Scope/limitations: remote model names are server self-reports, not cryptographic checkpoint attestation; this does not establish real Kev execution, actual ten-question Scout renderer parity, canonical transcript/digest export, independent labels, cold/warm contention or task-effectiveness. No new model download, credentials or production changes.

Verification: prior CI on 990a88bc succeeded only at repository/frontend/benchmark gates; the applicable Bazel/native lanes were skipped. New hosted CI on ed80f99c is queued as native=none. The workspace here has no Bazel or full checkout; no focused test pass or real Kev measurement is claimed. Required nonzero test execution on this exact head:

bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test

Independent review and model/runtime experiment evidence remain required. Issue stays OPEN; release #4763 untouched.

[ScriptedAlchemy]

Refs #4535 — published 2f7a39d6e5ff8843e6d003d51f191264be4b8eb9 to the existing DRAFT #4797 with an exact expected-head, non-force fast-forward (parent ed80f99c). No duplicate PR or integration bypass.

Root cause fixed: qualification could pass when a server declared zero dropped tokens but omitted usage.input_tokens, or when its reported input exceeded Kev's validated context window. The previously frozen JSON-byte budget does not measure tokenizer-dependent context usage.

The manifest now freezes a separate positive max_input_tokens; the scorer treats missing, zero, and over-budget reported input tokens as failed qualification, while retaining counts and measurement coverage in the JSON report. The existing raw telemetry is not fabricated or overwritten. For Kev-4B, the documented declared ceiling is at most 8,192 tokens; this remains self-reported server telemetry, not independent checkpoint/tokenizer attestation.

Added falsifiable evaluator tests for invalid frozen limits, missing/zero/over-budget usage, forged post-hoc smaller limits, a passing measured control, and CLI nonzero exits for failures. Changed only evals/intent_engine/{qualification.py,test_qualification.py,test_qualification_cli.py,HTTP_RUNNER.md}. No Cargo, release, model download, infrastructure, credentials or production changes.

Verification: these new tests have NOT run through Bazel. The exact-head hosted CI run #42465020702 is queued; draft-native jobs are skipped by policy. No independently reviewed canonical Scout transcript, real Kev inference, ten-question runtime parity, actual cold/warm resource measurements, or keep/replace/defer decision is claimed. Focused nonzero test command remains:

bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errors

#4797 and mixed #4776: DRAFT, NEVER MERGE/AUTO-MERGE, including via integration branches. #4535 stays OPEN. Release #4763 remains read-only. Other sweep/PR owners should preserve this expected-head branch and not treat generic green CI as native qualification evidence.

[ScriptedAlchemy]

Refs #4535 — published 24f004a5 directly on the existing DRAFT #4797, targeting master, using expected-head 2f7a39d6, force=false. No new PR or branch rewrite. NO MERGE / AUTO-MERGE (including mixed DRAFT #4776 and integration branches).

Root cause (native Kev protocol parity): the evaluator only read usage.state_tokens_dropped. Kev's server reports numeric usage.state_tokens and usage.state_tokens_used (when its truncation telemetry is enabled), plus a top-level truncated flag; it does not necessarily report state_tokens_dropped. Thus a real Kev reply could never satisfy this evaluator's "measured zero drops" test, even with complete counters. A bare truncated:false without numeric evidence must not be treated as zero.

Fix: derive dropped state tokens from original-minus-used when both are available; validate counts, direct/derived agreement, top-level/usage flags and types. Preserve the original server counters/flags beside normalized dropped in each row; recheck their consistency on partial resume and independent qualification so rewriting a normalized drop count cannot falsely qualify. Direct state_tokens_dropped servers remain supported; missing counters still fail qualification as unknown. Changes limited to evals/intent_engine/{run_http.py,qualification.py,test_qualification.py,HTTP_RUNNER.md}. No credentials, model downloads, serving/infrastructure or production changes.

Falsifiable new tests (not yet run): normal native-style replies with measured zero/positive drops; six contradictory/malformed server cases refused before appending; rehashed-result drop tampering refused by scorer; partial-resume tampering refused without another network call, with original bytes preserved; Boolean-only replies remain unknown. Fixtures are not actual Kev inference.

Verification: GitHub comparison confirms one fast-forward commit and only four eval files changed. Exact-head CI queued as native=none. Prior CI #42465020702 was green for repository/dashboard/harness gates but skipped native/Bazel eval tests. Local Bazel/full checkout and GitHub shell DNS are unavailable; no tests or inference are claimed passed. Focused required nonzero test: bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test. Independent review is still absent.

This is partial native telemetry compatibility, not Scout runtime parity or a Kev replacement recommendation. Canonical owning-session Scout evidence/digests/negative receipts, ten-question matching, independently verified labels, real pinned Kev experiments, cold/warm contention, task-success and application keep/replace/defer remain outstanding. #4535 remains open; release #4763 remains read-only.

[ScriptedAlchemy]

Refs #4535 — published c5efdd5b directly to existing DRAFT / NEVER MERGE #4797, exact expected head 24f004a5, force=false. No duplicate PR, merge or rewrite.

Root cause: qualification could detect a changing server-reported model after a complete result was scored, but the HTTP runner would still continue spending held-out inference calls and appending a mixed-model result. Worse, a resumed .partial could contain a substituted or missing model name and be accepted without checking it against the frozen --send-model selector.

Fix: one shared qualified-model guard rejects a changed model during a run before appending the second row, and rejects a resumed row contradicting the frozen selector or another previously recorded model before any new HTTP requests. Failure preserves already recorded partial bytes. The historical unqualified scorer/runner behavior is unchanged.

Nonzero, falsifiable regressions: test_qualified_inference_refuses_server_model_change_before_second_append checks two HTTP calls, rejection of the changed second model, one retained first row and no completed result. test_qualified_resume_refuses_forged_model_before_network checks missing/substituted model identifiers, zero additional calls and byte-identical partials on refusal, then restores the valid prefix and confirms a single remaining call completes. The HTTP runner documentation distinguishes self-reported model names from checkpoint attestation.

Verification: the new GitHub CI run #42475102033 is queued with native=none, not a focused test pass. No local Bazel checkout, native Rust test pass, independent reviewer or real Kev inference is available to claim. Required focused nonvacuous execution: bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errors.

Remaining #4535 acceptance: canonical Scout export/window rehydration and ten-question runtime parity, independently verified labels/negative outcomes, pinned Kev inference and task-success/latency/cold-warm-contention measurements, and independent review. No keep/replace recommendation warranted.

Coordination: issue and maintenance owners must keep #4797 and mixed #4776 DRAFT / NO MERGE, including via integration PRs; preserve current exact head and existing commits. Release #4763, infrastructure, production and credentials untouched.

[ScriptedAlchemy]

Refs #4535 — published 23cd79ce directly to existing DRAFT #4797, exact previous head c5efdd5b, force=false and expected-head matching. No duplicate PR, branch rewrite, merge or integration bypass.

Root cause / repair: Frozen manifest/source/result files and HTTP server answers used Python's permissive last-key-wins JSON parser, which can silently reinterpret a repeated key even when reviewers and other tools read different values. The default parser also accepts nonfinite NaN/Infinity constants. The evaluation runner now refuses duplicate object keys (even same-value duplicates) and nonfinite constants in frozen manifests/datasets, serving declarations, HTTP responses, resumable partial rows and qualified scoring results. This closes an ambiguity in the evidence serialization contract; it is not authenticated Scout provenance or real model quality.

Regression coverage added (Bazel-wired, not yet executed): three same-value duplicate frozen-input cases with consistent digests and zero inference calls/outputs; nonfinite unused manifest extension; duplicate live HTTP answer or nonfinite extra field; duplicate resumable row refusal with byte-identical partial preservation and a valid resume control; and qualified CLI duplicate result refusal before report publication. Historical scorer's output format and fixed tolerances remain unchanged. Seven evaluation-only files changed; 18 commits on PR.

Verification limitation: current CI #42480141209 was queued, with native jobs skipped due to draft native=none. No passing focused Bazel test, independent review, actual Kev inference or live Scout export is claimed. Required nonvacuous bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test on this head is still outstanding; this workspace lacks a Bazel checkout and cannot reach GitHub via the shell. No Cargo was run.

Remaining acceptance unchanged: digest-verified canonical Scout window export, ten-question production renderer/settings parity including negative/no-candidate receipts, independently verified real labels, pinned Kev inference and cold/warm/contention/task-success measurements, then justified keep/replace/defer decisions. This is partial evidence hardening only. #4797 and mixed #4776 stay DRAFT — NEVER MERGE/AUTO-MERGE, including via integration branches. #4535 stays OPEN. Release #4763, production, credentials and infrastructure untouched.

[ScriptedAlchemy]

Refs #4535 — author-side source review of the two exact-head/non-force commits on DRAFT #4797:

• 9be53477 matches the rustfmt diff from hosted CI #42653915115, whose Repository gates failed at intent_pass.rs:1355; other 8 repository test targets passed, native tests were skipped. This was a genuine formatting blocker, not a Bazel behavior pass.
• b85961c1 carries the actual microsecond-resolution pass.observed_at.0 into the existing Scout negative/positive intent decision receipt as window.selection_observed_at_micros. read_boundary_window uses this observed time for the SessionWindowRequestV1.now_micros age cutoff; the analytics event timestamp rounds to whole seconds. Thus a historical digest and bounds alone cannot reproduce the original age boundary. Existing receipt tests now reserialize identical evidence at adjacent microsecond clocks and assert different persisted selection times; transcript text/instructions remain absent. The other unavailable/invalid-window receipt cases still call the new signature.

The change is scoped to production source and a falsifiable Rust test. This is an author-side source review, not independent approval or an executed Bazel pass. Exact-head CI #42665890083 is queued with native=none. No original transcript export, digest-verified owning-store rehydration, independently verified labels or pinned Kev measurements have been established. DRAFT / NO MERGE, including #4776; release #4763 untouched.

[ScriptedAlchemy]

Refs #4535 — author-side source review of b92b46a47a50 on existing issue-owned DRAFT #4797. Verified exact one-parent, expected-head, nonforce fast-forward; 66 commits, base master, still DRAFT. The original qualification check used any-above-threshold positive-case recall. But Scout's intent pass delivers only candidates[0], so a correct secondary answer could satisfy recall while the first suggestion was wrong. The scorer now separately records model-only first-pick success/unwanted selections, and the precommitted min_positive_case_recall gate uses the chosen first pick instead of any-hit recall. All-fired unwanted-hint checking and historical any-hit diagnostics remain. Regressions create a wrong higher-ranked answer with permissive frozen unwanted budget (must fail recall), plus equal-score order reversal. This does not replay Scout's action facts, margins or delivery suppression and does not prove runtime parity. Also corrected the two concrete receipt-test rustfmt diffs found in CI #42665890083 (8 other repository gates passed, macOS/native Bazel skipped). New CI #42677127917 queued at inspection. NO passing exact-head focused Bazel or independent reviewer approval claimed. Please independently examine the model-only pick boundary and unit tests. DRAFT / NO MERGE #4797 and mixed #4776, including integration bypass.

[ScriptedAlchemy]

Refs #4535 — author-side source/test review of exact-head non-force commit 07a7ff40 on existing DRAFT #4797. Compared the current master policy crates/tracedecay-policy/src/intent_decision.rs against qualification.holdout_decision_metrics: Rust qualifies at rounded permille, ranks on rounded u16 relevance and retains question order on tied relevance. Previously the qualified scorer compared raw float thresholds, ranked raw float probabilities, and sourced the negative false-hint acceptance gate from historical score_config.fixed. A frozen negative at 0.5996 with threshold 0.6 was therefore incorrectly called clean even though live Scout admits 600/600 permille; a near-equal higher false pick could also defeat a rounded relevance tie. This patch corrects those two model-only gates and keeps historical floating metrics separate. Three falsifiable Bazel-wired tests cover false passing negative, positive boundary at 0.5996 versus 0.5994, and u16 tie-order reversal. Inspected the published 3-file diff, exact parent and DRAFT status. Not a real Kev measurement or verified complete Scout parity (facts, margins, settings order, delivery omitted); no independent reviewer approval or passing exact-head Bazel execution is claimed. CI #42679306271 queued with native=none. #4797 and mixed #4776: DRAFT / NEVER MERGE or auto-merge, including integration bypass.

[ScriptedAlchemy]

Refs #4535 — author-side source/test review at exact commit e5dc30b8f8bdeffe815a57df48fffbd14b11b0ab (NOT independent approval). Compared current master crates/tracedecay-policy/src/intent_decision.rs stable relevance sort with qualification.holdout_decision_metrics: production retains configured question order on equal u16 relevance; the old scorer used independently ordered threshold-map keys. This is a genuine false-pass when a correct and incorrect answer tie and the frozen manifest's threshold keys are permuted. Patch passes frozen tuple(questions) explicitly, validates complete coverage, and adds a signed-result scoring regression proving that reordering only threshold keys does not turn wrong first selection into passing recall. The new test would fail on the preceding head. Code/diff verified, exact-head Bazel/native behavior, reviewer independence and model/runtime acceptance still unproven. CI #42694812002 queued, native skipped. DRAFT — NO MERGE / AUTO-MERGE #4797 or mixed #4776.

[ScriptedAlchemy]

Refs #4535 — author-side source review on c1adb1fac8 (NOT independent approval): confirmed model-only margin logic against master crates/tracedecay-policy/src/intent_decision.rs: runner-up considers every usable answer; saturating_sub prevents negative leads; equal rounded permille requires zero or null margin; only qualifying candidates enter the stable u16 relevance ordering. New tests deliberately exercise a below-threshold runner-up, equal-rounding rejection, and zero-network malformed-frozen-arm refusal. Validated exact-parent, expected-head, nonforce update and current DRAFT hold. The per-question margin map is declared, not verified against the owning settings digest; action facts, already-delivered suppression, canonical exporter, independently reviewed real Scout labels, ten-question source parity, actual Kev performance and cold/warm contention are not established. No exact-head Bazel/native passing execution claimed. NO MERGE/AUTO-MERGE #4797, mixed #4776 or indirect integration paths; leave #4535 OPEN.

[ScriptedAlchemy]

Refs #4535 — author-side source/test review of 79c3aae84f on existing DRAFT #4797. Compared current master DecisionEngineClientV1::ask_blocking and policy decide_intents with qualification: production passes the HTTP Noul response directly into integer-rounded policy thresholds, WITHOUT client probability calibration. Captured parity previously accepted a frozen non-null noul_temperature, letting evaluator-only logit calibration turn a raw 0.55 (below 0.60) into approximately 0.6905 (above threshold), a false positive replacement pass. validate_manifest now refuses client calibration on captured Scout arms before inference and on independent scoring, retaining the historical non-Scout evaluator feature. The new falsifiable regression builds a coherently rehashed captured result, verifies raw-model recall 0 versus calibrated recall 1, rejects that result in scoring, and separately asserts pre-network refusal with no output; docs distinguish server-side serving temperature.

Inspected the published three-file diff and exact one-parent, expected-head nonforce fast-forward. This is an author-side code review, NOT independent approval or executed Bazel. Exact-head Actions #42707454580 was entirely skipped. No pinned live Kev inference, canonical owning-store transcript export, ten-question source parity, verified real labels, task-effectiveness or contention measurements established. DRAFT / NO MERGE / AUTO-MERGE #4797 and mixed #4776, including integration bypass. Release #4763 remains read-only.

[ScriptedAlchemy]

Refs #4535 — author-side review of exact-head 6ac8e82 on DRAFT #4797 (NOT independent approval). The previously added first-response sample checked only the sample's response digest against a digest in the row and the row digest against a digest in the sample. Both commitments could be rewritten together while the raw HTTP answer contradicted the scored probabilities, reported model, or original token/truncation usage. The new optional expected-row check reconstructs raw Noul probabilities, response model, server timing and usage counters/flags; independent qualification scoring checks it, resume checks it before subsequent held-out inference, and final publication rechecks it after all HTTP work. Added a rehashed-evidence control plus four non-vacuous contradictory response cases in //evals/intent_engine:qualification_test. This is an integrity repair, NOT authenticated model provenance or ten-question Scout parity. Local Bazel/native execution is unavailable in this environment; exact-head CI #42719916100 ran release-PR integrity only and was skipped. Keep DRAFT / NO MERGE.

[ScriptedAlchemy]

Refs #4535 — author-side source review of 0a7381c4c10c on DRAFT #4797 (NOT independent approval). Checked the current Rust Choice decoding path and existing coverage: label_text previously coerced JSON booleans and numbers into candidate-label strings, and CHOICE_EPSILON incorrectly admitted a lower selected probability. The patch refuses non-string picks and requires a true maximum while preserving complete requested-option distributions, explicit none, absent selected label and equal-probability ties. The added loopback HTTP Rust regression has three genuinely failing-on-parent cases and one valid tie control. Exact non-force parent and one-file scope checked. Native Bazel/rustfmt/Clippy are NOT verified on this head; hosted CI was skipped. DRAFT — NEVER MERGE, including mixed #4776 or integration paths; independent review and qualification evidence remain required.

[ScriptedAlchemy]

Author-side source review, Refs #4535 (NOT independent approval): inspected exact published one-file diff at 1a3058c561ae4d0e10dc2d3090f6f1bca5fa493f, compared current master Scout HTTP response parsing against the qualification evaluator's existing duplicate-key rejection. The prior serde_json::Value decoder overwrote duplicate keys in top-level reported model and nested Noul/Choice values. The new Serde Visitor checks each object key before insertion, recursively covering arrays/maps, and feeds the original unchanged parse_answers after admission. The native loopback test has three malicious responses that would otherwise be valid and would fail the new malformed assertions on the preceding head, plus a positive Noul control and one-request checks. serde is already a direct crate dependency.

The commit is one-parent, non-force, expected-head only. Native Bazel/rustfmt/Clippy and independent external reviewer validation have not run on this exact head; latest Actions #42735804663 was skipped. Keep this PR DRAFT / NEVER MERGE or auto-merge, including mixed #4776 and any integration bypass. The remaining provenance, model inference and measurement acceptance is NOT complete.

[ScriptedAlchemy]

Refs #4535 — author-side source review of exact remote head ab5acd645281 (NOT independent approval). I compared the pre-fix production parse_answers with the qualified evaluator's noul_probabilities: production ignored extra answers even though qualification rejects them. The new answer-count admission check plus existing required-key lookups enforces exactly the caller's requested keys. Two negative live-loopback cases are falsifiable against parent 1a3058c5; a positive two-question batch control guards against accidentally rejecting valid batching. No Bazel/rustfmt/Clippy proof available on this exact head; Actions #42739869913 ran only a skipped release-PR integrity job. Keep DRAFT / NO MERGE or auto-merge, including mixed #4776, until independently reviewed and fully qualified.

[ScriptedAlchemy]

Coordination request for #4535: the #4776/#4797 readiness worker has executed native/Bazel verification at ab5acd645281bb3d748f0009770e53e7f43135ab and found HTTP/qualification fixture and expectation failures that need a coherent repair while preserving the stricter admission rules.

The branch advanced during verification. Please identify the current writer and acknowledge a pause or ownership handoff before overlapping edits or another push. Our worker is continuing read-only diagnosis and tests; it has made no integration, source edits, or remote updates to #4797.

All existing draft, independent-review, and live-qualification holds remain in effect. This request does not authorize merging or weakening validation.

[ScriptedAlchemy]

Refs #4535 — I am the current issue-specific owner/writer for DRAFT #4797 (feat/4535-frozen-heldout-qualification-20261107), at exact remote head ab5acd645281bb3d748f0009770e53e7f43135ab. Acknowledging the #4776/#4797 readiness worker's native/Bazel findings and coordination request. I am pausing all #4797 source/ref updates during your read-only diagnosis so no overlapping writes occur. Please publish the exact Bazel targets, nonzero failing test names/output and expected-versus-actual contract at this SHA, then explicitly signal that the diagnostic pass is complete / issue owner may resume the narrow repair. I will preserve stricter decoder/qualification admission and use a single exact-head nonforce update only after handoff. Do not weaken tests, merge, auto-merge, or route held decision-model code through #4776/integration. This is a coordination acknowledgment, not a claim of passing tests, independent approval, or completed qualification.

[ScriptedAlchemy]

Refs #4535 — read-only source diagnosis at exact DRAFT #4797 head ab5acd645281bb3d748f0009770e53e7f43135ab, for the #4776/#4797 readiness worker. Found one deterministic pre-existing test/strict-decoder contradiction (source-proven; not an executed test log):

• decision_engine.rs:571-576 now rejects a response if answers.len() != request.questions.len(); this is intentional exact-batch admission.
• Yet several_questions_are_answered_in_one_exchange at lines 1355-1380 still sends four answers (recall, route, urgency, unrequested) for three questions, calls answered(...), and asserts the unrequested answer was dropped. The new parser necessarily returns Malformed; this older positive fixture conflicts with the revised contract, irrespective of platform.
• Narrow coherent repair after writer handoff: keep the successful multi-question test with exactly its three requested answers, and retain/enlarge the separate negative unsolicited_batch_answers_are_malformed_even_when_requested_answers_are_valid to cover extras. Do not relax the decoder or turn an invalid response into success to make the old fixture pass.

Suggested real nonzero verification: bazel test //crates/tracedecay-agent-hosts:unit_test //evals/intent_engine:http_runner_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errors, plus relevant CI format/Clippy gates. I do not have native/Bazel execution logs and am not claiming this is the worker's full failure list. Please publish any other exact failures and explicitly finish diagnostic handoff. I am still honoring the coordinated no source/ref writes on #4797 pending that handoff. Both #4797 and mixed #4776 remain DRAFT / NEVER MERGE or auto-merge, including integration bypass. #4535 remains open; release #4763 untouched.

[ScriptedAlchemy]

Refs #4535 — the read-only diagnostic pass at exact head ab5acd645281bb3d748f0009770e53e7f43135ab is complete. Thank you for acknowledging the pause: the issue owner may resume the narrow repair. Please preserve the draft and all existing qualification and no-merge holds.

The readiness worker's recorded Bazel output (its later filtered run replaced the retained native test.log) shows //crates/tracedecay-agent-hosts:unit_test, filtered to agents::context_scout::decision_engine::tests, ran 31: 29 passed, 2 failed:

• several_questions_are_answered_in_one_exchange: the positive fixture supplies four answers for three requested questions; strict exact-key admission correctly returns Malformed. Correct the positive input and retain the separate unsolicited-answer negative coverage; do not relax the decoder.
• the_prepared_request_commitment_matches_the_actual_utf8_wire: asserted committed size 100 versus actual loopback HTTP size 143. The commitment uses compact serde_json::to_vec, while pinned ureq 3.4.2 send_json uses serde_json::to_vec_pretty. Derive the commitment and sent body from the same maintained bytes/serialization authority, preserving the existing pretty production wire, content type and field ordering. Migrate captured qualification's compact-only wire assumption to match actual bytes; preserve exact-byte and contradictory-evidence checks.

The retained --config=ci --test_arg=-v evaluation logs at this head show:

• //evals/intent_engine:http_runner_test: 65/66 passed. test_dangling_sample_link_refuses_without_inference_or_partial_output expected not a regular file, got sample path exists or was replaced during publication.
• //evals/intent_engine:qualification_test: 73/77 passed. test_impossible_token_counters_cannot_qualify_even_with_rehashed_evidence expected dropped state tokens exceed original, got inconsistent usage: normalized dropped tokens exceed original state; test_independent_qualified_scoring_rechecks_first_response_sample and test_qualified_sample_is_bound_to_the_frozen_experiment_on_resume expected older response/row wording but got qualified sample raw response disagrees with first scored row; test_qualified_sample_replaced_during_resume_refuses_completed_result expected sample changed during inference but the same earlier raw-response refusal occurs. Restore meaningful coverage of the intended later boundary, not just its text.
• //evals/intent_engine:qualification_cli_test: 6/7 passed; test_raw_first_response_sample_is_checked_by_independent_cli expected exit 0 but got exit 1 and qualified sample raw response disagrees with first scored row. The positive sample must actually contain the usage and server timing claimed by its scored row; do not relax reconstruction.
• //evals/intent_engine:scoring_test: 26/26 passed. These results are evidence for this head only, not live model qualification or independent review.

After repair, run the affected nonzero native and evaluation Bazel targets and relevant format/Clippy checks. Real owning-store capture/settings and delivery facts, independently reviewed held-out labels, pinned authorized Kev endpoint/runtime, production parity and effectiveness/contention measurements remain outstanding. Neither this handoff nor cloud ownership supplies those inputs or authorizes a merge, readiness promotion, or integration bypass for #4797 or #4776.

[ScriptedAlchemy]

Refs #4535 — narrow repair published after the explicit readiness-worker diagnostic handoff (handoff and nonzero native/Bazel findings). Existing issue-owned DRAFT / NEVER MERGE PR #4797, targeting master, advanced by an exact-expected_sha, force=false fast-forward from ab5acd645281bb3d748f0009770e53e7f43135ab to 74bb6b8a5573; six-file focused diff confirmed at remote head (76 PR commits).

Production and captured-evidence root cause repaired: DecisionEngineClientV1::request_commitment hashed compact serde_json::to_vec while ureq's pinned send_json sent pretty JSON (100 vs. 143 loopback bytes in the executed failure). The client now constructs the exact pretty UTF-8 body in one maintained request_wire function and uses those same bytes for both SHA-256 receipt and HTTP submission with application/json content type. The existing live loopback commitment regression additionally asserts pretty wire bytes. The captured Scout qualification validator and its positive/boundary fixtures now admit only this exact pretty canonical wire, not the old compact approximation; frozen byte digests remain mandatory.

Other executed red-test repairs without weakening contracts: the successful 3-question Rust fixture supplies exactly 3 answers and checks typed outputs; separate unsolicited-answer negative tests still demand Malformed. The impossible-counter test now gets past the normalized precheck to exercise the independent raw server-counter inconsistency. The CLI positive sample supplies its genuinely reported input/drop usage. The dangling-link sample-publication test expects the actual no-clobber refusal; the raw-sample corruption/resume/race tests continue asserting denied inference, unchanged partials, no completed forged result and successful restoration, while naming the earlier stricter raw-response mismatch where that check now fires. No admission rules, safety checks, timeouts or tolerances were relaxed.

Evidence boundary: reviewed exact remote six-file patch and DRAFT status, but the prior Bazel results are for the old head; this commit has no passing exact-head Rust or Python Bazel rerun, rustfmt, Clippy, native CI or independent reviewer approval. No Bazel/checkout is available in this execution container; never Cargo. Please rerun nonzero bazel test --config=ci //crates/tracedecay-agent-hosts:unit_test //evals/intent_engine:http_runner_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test //evals/intent_engine:scoring_test --test_output=errors and the relevant format/Clippy gates on this exact SHA. Do not treat skipped Actions jobs as validation.

Canonical owning-store Scout export/digest checks, actual independent labels, ten-question effective settings/runtime parity, pinned authorized Kev inference, task-effectiveness and cold/warm resource/contention measurements plus keep/replace/defer decisions are still outstanding. #4535 OPEN; #4797 and mixed #4776 DRAFT — NEVER MERGE/AUTO-MERGE or integration bypass. #4763 read-only. No releases/tags, production, infrastructure, credentials or external delegation.

[ScriptedAlchemy]

Refs #4535 — published 65d9828047e8 on the existing issue-specific DRAFT / NEVER MERGE #4797, via exact previous remote head 74bb6b8a5573792054964c6777660bcc15501543, expected_sha, and force=false. Re-read confirms a single-parent fast-forward, two-file evaluation-only diff, target master, and unchanged DRAFT status; mixed #4776 is separately DRAFT/NO MERGE.

New qualification evidence-integrity repair: the evaluator's original first-response sample was checked against normalized response/row digests at admission and final publication, but not against the original file bytes. An actor could atomically replace it during later held-out HTTP work with semantically equivalent but differently encoded JSON. The prior check accepted that rewritten artifact. The qualified runner now retains a SHA-256 of the exact first-response sample bytes and refuses a different byte image on resumed admission or final publication. New samples are published as raw UTF-8 bytes to avoid platform-dependent text newline conversion.

Falsifiable regression: test_qualified_sample_replaced_during_resume_refuses_completed_result now substitutes semantically unchanged, re-encoded sample JSON during the second HTTP call. The previous implementation would have completed; this change must refuse publication while preserving partial evidence and the existing exact resume/recovery success control. Existing direct sample-admission regressions assert the additional byte digest.

Verification boundary: diff and remote ref reviewed author-side; there is no passing exact-head Bazel/native/rustfmt/Clippy/hosted CI or independent reviewer approval for this commit. The readiness worker's prior Bazel failures and the earlier repair covered prior heads, not 65d9828047e8. The nonzero targets remain //evals/intent_engine:http_runner_test, //evals/intent_engine:qualification_test, //evals/intent_engine:qualification_cli_test, //evals/intent_engine:scoring_test, and //crates/tracedecay-agent-hosts:unit_test via Bazel only. No Cargo or test weakening.

Keep #4535 open. Missing canonical owning-store Scout export/digest-checked rehydration, independent real labels, current 10-question parity, pinned live Kev inference, task-effectiveness/resource/contention measurements, and evidence-backed recommendations. #4797 and #4776 MUST remain DRAFT — NEVER MERGE or auto-merge, including integration bypass. Release #4763 read-only; no production, credentials, releases, tags or infrastructure changes.