[jatseng-ai]

Description

This adds an optional, fail-closed gate that runs Kernel Trust Harness (KTH) qualification on every Controller-integrated kernel patch before it can reach the benchmark or KEEP path.

It integrates the consolidated KTH mainline from AMD-AI/kernel-trust-harness#9, merge SHA 49154fb80508bb37d008f87f8bfae8759209742f. The KTH revision is not built in: operators pin it with HYPERLOOM_KTH_EXPECTED_SHA.

This PR replaces #1566 (the primary source) and #1550 (safeguards only). #1540 stays closed.

Responsibility split

Component	Owns
GEAK	Generates candidates.
Hyperloom	Builds the candidate envelope once the exact applied bytes exist. Invokes KTH before any benchmark or KEEP. Validates the attestation's binding to that candidate. Enforces the verdict and records diagnostics.
KTH	AutoSpec, specs, plans, detectors, policy, verdict and attestation.

Hyperloom has no kernel-specific, issue-specific, threshold or detector logic.

Where the gate sits

In integrate_controller_patches (src/hyperloom/orchestrator/kernel/controller_patch_integration.py), in this order:

1. apply_patch_resolving_conflicts(...) lands the candidate in the working tree.
2. KTH qualification runs on the bytes that actually landed: a git diff against HEAD built with a temporary index, not the published change.patch. Any result other than Eligible reverts the candidate with status reverted_kth_<status>. The performance validator is never called.
3. validate(publication) runs the existing performance validator (Eligible only).
4. Re-check before KEEP. If the applied diff digest no longer matches the qualified one, the candidate is reverted as reverted_kth_failed ("the candidate changed after it was qualified").
5. The KEEP commit and record_keep(...).

phases/kernel.py passes KthQualificationProvider.from_env(). When KTH is disabled this is None, and the loop takes exactly the path it took before.

Verdict and action

KTH outcome	Hyperloom status	Action
Eligible (exit 0) and every binding check passes	eligible	Goes to the performance validator; KEEP only if performance also passes.
Blocked (exit 2)	blocked	Revert. No benchmark, no KEEP. The KTH reason becomes repair feedback.
Inconclusive (exit 3)	inconclusive	Revert. No benchmark, no KEEP.
Exit 1, timeout, missing executable, malformed or non-object output, unknown exit code	failed	Revert. No benchmark, no KEEP.
Mismatch in request ID, subject digest, envelope digest, patch digest, base commit, plan or revision; unsupported schema; insufficient coverage or evidence; candidate-supplied control fields	failed	Revert. No benchmark, no KEEP.

GEAK observations (mean_case_speedup, best_wall_ms, iteration, correctness_passed) are never read as qualification input and cannot override a KTH verdict.

Security and trust

• Candidate identity. Each attempt gets a fresh UUID request_id. KTH's answer must bind back to that exact request, the applied patch bytes, the base commit, the plan (reviewed path) or the normalized envelope (adaptive path), and the pinned KTH revision. The digest framing (kth-subject-v1 / kth-subject-v2) is recomputed on the Hyperloom side and checked against golden values produced by real KTH.
• Stale answers. A replayed attestation from an earlier candidate fails the request, envelope or subject check.
• Plans are host-owned. Plans come from HYPERLOOM_KTH_PLANS (kernel path to KTH plan ID). A publication that declares its own plan, verdict, attestation or other control field is rejected. The changed paths must fall inside the plan's allowed_paths, and an empty change set is rejected.
• Revision pinning. An unknown revision or one that differs from HYPERLOOM_KTH_EXPECTED_SHA is rejected.
• Execution mode. execution_mode must be real or simulated and is recorded; KTH decides what it means for the verdict.
• Environment. The KTH subprocess runs with the scrubbed environment from env_safety, and the logged output is redacted. The request, attestation and result.json are written atomically under the session directory.

Linked issue(s)

None; there are no Jira keys or GitHub issues on #1566, #1550 or #1540. Supersedes #1566 and #1550.

Inclusion matrix

Retained from #1566

• HYPERLOOM_KTH_ENABLE, ..._QUALIFY_EXECUTABLE, ..._TIMEOUT_S and ..._EXPECTED_SHA.
• The kth-qualify --request --out subprocess contract and the exit codes 0/1/2/3.
• The adaptive envelope path (schema 2.0.0).
• The gate position: after apply, before the performance validator.
• The v1 reviewed-plan checks: digest, identity, plan and coverage.
• The adaptive checks: fully_verified, no uncertainty, nothing uncovered.
• The artifact files and the performance-admission marker (renamed performance_admitted).
• The repair-feedback reason, and kth in the integration result.

Uniquely retained from #1550 (strengthened where noted)

• Host-owned plan mapping. A plan declared by the publication is now rejected outright.
• Changed-path scope, now checked against the KTH plan's allowed_paths.
• Rejection of an empty scope.
• Post-apply identity. This is now the applied-diff digest, re-checked immediately before the KEEP commit.

Deliberately dropped

• kth_qualification.plan_id in the publication. A candidate must not choose its own plan.
• HYPERLOOM_KTH_ADAPTIVE. Adaptive is simply the path for any kernel without a host plan.
• Deterministic request IDs derived from the patch SHA. They allowed replays; IDs are now a fresh UUID per attempt.
• Qualifying change.patch. After a conflict-resolving merge it is not what landed.
• Accepting an unknown KTH revision.
• Deriving the revision from HYPERLOOM_KTH_ROOT.
• The OPERATION_PLANS name mapping.
• HYPERLOOM_KTH_ALLOWED_PATHS. The scope comes from the KTH plan.
• [HYPERLOOM] banner logging.
• Broad except Exception.
• The needs_review and kth_* statuses, replaced by reverted_kth_*.
• Optional KTH AutoSpec qualification KEEP gate #1566's runtime_observation artifact provenance. It claimed a runtime source that did not exist.

Demo material excluded

• examples/kth_qualification_gate/ (run_demo.py, README, expected_terminal.svg).
• Everything from Gate Controller patches with KTH qualification #1540.

Current APIs that replaced old assumptions

• apply_patch_resolving_conflicts merges. Qualification runs on the landed diff.
• Task-priority ordering, and the required record_keep/Coordinator writeback.
• _settle_apply_manifest.
• The strict env helpers.
• The env_safety scrub and redact functions.
• atomic_write.
• The publication's changed_files.

Tests

The new tests cover these 20 cases:

1. Disabled means unchanged.
2. Eligible benchmarks exactly once.
3. Blocked never benchmarks.
4. Inconclusive never benchmarks.
5. Provider failure never benchmarks.
6. Timeout.
7. Missing executable.
8. Malformed or non-object output.
9. Unknown exit code.
10. Request, subject, envelope, patch, base, plan and revision mismatches.
11. Unsupported schema.
12. Insufficient coverage or evidence.
13. Candidate-supplied control fields.
14. GEAK observations, including correctness_passed, cannot override Blocked.
15. Stale or replayed attestation.
16. A candidate changed after qualification.
17. A reconstructed merge qualifies the landed bytes.
18. A retry is re-qualified with a fresh request.
19. Secrets are not leaked to the child process or the logs.
20. Configuration validation and execution-mode handling.

There are also golden digests shared with KTH and opt-in live tests against an installed kth-qualify.

Commands run (all passing):

ruff format --check .           # 1761 files already formatted
ruff check .                    # All checks passed
mypy kth_contract.py kth_qualification.py controller_patch_integration.py   # no issues
pytest -n 8 test_kth_contract.py test_kth_qualification.py test_controller_patch_integration.py \
       test_controller_publication.py test_integrate_controller_preapplied_patch.py
                                # 160 passed, 6 skipped (the skips are the opt-in live tests);
                                # 161 with the added exit-code case
KTH_SHA=49154fb… HYPERLOOM_TEST_KTH_EXECUTABLE=…/kth-qualify pytest -k installed ...
                                # 6 passed against KTH 49154fb
pytest -n 16 --timeout=900 (full suite)
                                # 23968 passed, 60 skipped, 17 failed (see below)
scripts/check_broad_suppress.py, compileall, pylint --errors-only, bandit (changed modules): clean
test_packaging_lint.py; wheel build; scripts/check_wheel_contents.py: pass (both KTH modules ship)
gitleaks v8.28.0 over origin/main..HEAD: no leaks

None of the 17 full-suite failures are in KTH or Controller-integration code. They fall into two groups:

• 16 fail identically on a clean origin/main checkout on this host. They involve host-environment assumptions: missing-file, multi-node state ownership and git worktree discovery tests.
• 1 (test_grid_runner.py::...test_the_budget_is_re_checked_after_the_uncapped_server_restart) uses a 6-second wall-clock budget. It fails only under -n 16 load and passes alone on both this branch and main.

CI on this PR is the authoritative run.

CPU end-to-end run against KTH 49154fb

This used unmodified KTH, a host plan registered through a wrapper, and a real git repository:

Candidate	KTH verdict	Integration status	Performance-validator calls
Correct quantization patch (reviewed plan)	Eligible	kept	1
Numerically wrong patch (reviewed plan)	Blocked (NUMERICAL_POLICY)	reverted_kth_blocked	0
Kernel without a host plan (adaptive)	Inconclusive (no host-owned adapter)	reverted_kth_inconclusive	0

Size/complexity

• integrate_controller_patches: ruff C901 is 19 on main and 20 here. The gate adds one branch; the qualification and admission work lives in _admit_to_performance and _still_qualified.
• KthQualificationProvider._check_reviewed (13) and _check_adaptive (14) are flat sequences of one-field attestation comparisons, each with its own rejection reason. Splitting them would scatter the contract without removing any decision.
• Module sizes: kth_qualification.py 533 lines, controller_patch_integration.py 633 lines, kth_contract.py 264 lines. All are under the 800-line trigger. qualify is under 60 lines since the subprocess step was extracted into _run.

If this simplifies or refactors

This does not refactor existing behaviour; it is additive behind HYPERLOOM_KTH_ENABLE.

Observable effect

• Enabled: each Controller patch gets kth_qualification/<request_id>/ (request.json, attestation.json, result.json and the redacted KTH logs) under the session directory. The integration result carries a kth record. Non-eligible candidates show reverted_kth_blocked, reverted_kth_inconclusive or reverted_kth_failed, with a revert note and the KTH reason as repair feedback.
• Disabled (the default): nothing changes.

Documentation is in docs/reference/kth-qualification.md. It covers how to enable the gate, where qualification sits, what each verdict means, how to pin the revision, where artifacts land and the responsibility split, and links KTH's limitations and capability states.

Known limitations

• Controller path only. Only the Controller patch integration path is gated. The legacy agent-lane integrate, the pending drain and stack validation are not.
• Adaptive path. Kernels without a host plan in HYPERLOOM_KTH_PLANS go down the adaptive path. KTH currently returns Inconclusive there when no host-owned adapter exists, so those candidates are reverted (fail closed).
• KTH schema gap. KTH's v1 attestation JSON schema omits execution_binding, which the KTH CLI emits and Hyperloom validates. That is a KTH-side schema gap.
• Revision source. The KTH revision comes from a git checkout of KTH or KTH_SHA. An installation without either reports unknown, which Hyperloom rejects.

Breaking changes

No. The gate is off unless HYPERLOOM_KTH_ENABLE=1.

PR addresses single concern

Yes.

Root cause is upstream

Not applicable. The KTH schema gap above is a KTH issue, not a Hyperloom defect.

[chaojhou]

Requesting changes for one must-fix; I found no blockers. The gate itself is careful: every failure path I traced reverts before the benchmark, the validator measures the same landed bytes KTH qualified (the pre-applied snapshot is copied from the worktree), the digest is re-checked before the KEEP commit, and a malformed HYPERLOOM_KTH_* setting stops integration instead of silently disabling the gate.

Must-fix: kth-qualify runs on the coordinator event loop

integrate_controller_patches is a coroutine awaited directly by the kernel phase (phases/kernel.py:3811), but the gate calls _admit_to_performance synchronously:

# controller_patch_integration.py:469
qualified = _admit_to_performance(
    kth_provider,
    publication,
    ...
)

That reaches subprocess.run(..., timeout=self.timeout_s) (kth_qualification.py:256-261), so the event-loop thread is held for the whole qualification: up to HYPERLOOM_KTH_TIMEOUT_S (default 300 s, and operators can raise it) per candidate. A real-mode KTH run executes the kernel against its mandatory oracles, so this is not a short call.

The codebase already guards against this: test_setup_replay_runs_off_the_event_loop_thread (actions/executors/tests/test_integrate_patch_executor.py:1881) exists because a blocking subprocess.run on the loop thread freezes "concurrent in-flight LLM streams, the dispatcher's re-scan poll, and cancel grace". The rest of this flow follows that convention: the Controller runs under asyncio.to_thread (phases/kernel.py:3764), and the E2E validator is awaited.

The fix is one line:

qualified = await asyncio.to_thread(
    _admit_to_performance,
    kth_provider,
    publication,
    repo=repo,
    head=head_before,
    touched=touched,
    session_dir=Path(session_dir),
)

_still_qualified is a single git diff and can stay synchronous like the neighbouring Git calls. A thread-identity test in the style of the one above would keep this from regressing.

[chaojhou]

@jatseng-ai this PR conflicts with current main. Could you rebase it?

Files that conflict: src/hyperloom/orchestrator/kernel/controller_patch_integration.py, src/hyperloom/orchestrator/phases/kernel.py, src/hyperloom/orchestrator/kernel/tests/test_controller_patch_integration.py, docs/release-notes.md.

#1977, #1992, #2051 and #2090 reworked controller_patch_integration.py and phases/kernel.py after the branch point; the release-note entry also needs to move under ## Unreleased.

Please address the must-fix from the 2026-09-27 review in the same push.

Once it merges cleanly I'll review it.