Skip to content

feat(kernel): KTH shadow mode — record the gate's verdict, enforce nothing - #1685

Draft
jatseng-ai wants to merge 1 commit into
feature/kth-qualification-gate-v2from
feature/kth-autospec-v2-shadow
Draft

jatseng-ai wants to merge 1 commit into
feature/kth-qualification-gate-v2from
feature/kth-autospec-v2-shadow

Conversation

@jatseng-ai

Copy link
Copy Markdown
Collaborator

What this is

One flag, HYPERLOOM_KTH_SHADOW_ENABLE, that runs the KTH qualification for
each applied patch and enforces nothing. No patch is reverted, no benchmark
is skipped, no KEEP is affected, and every patch's status is exactly what it
would have been with the flag unset.

Based on feature/kth-qualification-gate-v2 (the head of #1672) so this diff
shows only the shadow work. Please do not merge, and please do not treat
this as part of #1672.

Why

#1672 adds a gate that can revert a patch. Before giving it that authority over
integration it is worth knowing what it would actually have done on the patches
this repository produces: how often it would have refused, and whether it would
have been right. Finding that out by turning the gate on is the wrong order of
operations.

The property that matters

The failure mode worth designing against is shadow mode quietly becoming
enforcement — through a later refactor, a copied line, or an operator setting
the wrong variable. Three things prevent it, by construction rather than by
convention:

  1. An observation carries nothing to enforce on. The enforcement path
    admits a patch by reading qualified.eligible. KthShadowObservation has no
    eligible attribute, no performance_admitted, and no other boolean
    summarising the outcome — recorded_only is the only bool on it, and it is
    always True. An observation cannot be substituted for a qualification
    without failing at the point it is written. A test asserts the absence.

  2. The observation never enters the integration result.
    _observe_in_shadow returns None, so there is no value for the loop to
    branch on, and the record is written to its own
    integration/results/NNNN.kth_shadow.json rather than into
    PatchIntegrationResult.kth. The two join on the index without either being
    able to contaminate the other.

  3. A recording instrument cannot fail the thing it records. Every failure —
    in the observer, in the provider, in reading the candidate, in writing the
    artifact — becomes a recorded observation with shadow_error set. The
    containment is at the call site as well as inside the observer, deliberately:
    the integration path must survive any observer, including one a later
    change makes less careful. A test injects an observer that raises and asserts
    both patches are still kept.

Enabling shadow mode alongside HYPERLOOM_KTH_ENABLE is a KthConfigurationError
rather than a silent precedence rule. An operator who asked to observe should
not end up enforcing, and one who asked to enforce should not quietly stop.

Shadow observations are never bound to a performance admission: qualify is
called, mark_performance_admitted is not.

What lands

File
kernel/kth_shadow.py New. KthShadowObservation, KthShadowObserver.
kernel/controller_patch_integration.py _observe_in_shadow, one call site, one new optional parameter.
phases/kernel.py kth_shadow_observer=KthShadowObserver.from_env().
docs/reference/environment-variables.md Two rows.
docs/reference/kth-qualification.md A "Shadow mode" section.

Seven new tests. src/hyperloom/orchestrator/kernel/tests/ is green at 387
passed / 6 skipped, and the phases suite too (424 / 6 together). ruff check is
clean on every file touched; the six pre-existing BLE001 findings in
conc_sweep.py and request_handlers.py are unchanged and unrelated.

Off by default. With both flags unset nothing in this diff executes.

Scope

  • No hackathon, demo or presentation material.
  • No change to the enforcement path's behaviour. The existing KTH tests pass
    unmodified.
  • The KTH side of this work is a separate research branch and is not proposed
    for merge either.

What this does not establish

Nothing has been observed yet. This is the instrument, not a result. The number
worth having — how often KTH would have refused a patch this repository kept,
and how often that refusal would have been correct — needs the flag turned on
for a real run, which has not happened.

Before giving the qualification gate authority over integration it is worth
knowing what it would have done. HYPERLOOM_KTH_SHADOW_ENABLE qualifies each
applied patch for the record and changes nothing: no patch is reverted, no
benchmark is skipped, no KEEP is affected, and every patch's status is what
it would have been with the flag unset.

The failure mode worth designing against is shadow mode quietly becoming
enforcement, so three properties are enforced by construction rather than by
convention. KthShadowObservation has no 'eligible' attribute and no other
boolean summarising the outcome, and the enforcement path admits a patch by
reading qualified.eligible, so an observation cannot be substituted for a
qualification. _observe_in_shadow returns None, so there is no value for the
integration loop to branch on and the observation never enters the same
object as the integration result. Every failure - in the observer, in the
provider, in writing the artifact - becomes a recorded observation, because
a recording instrument must not be able to fail the thing it records.

Enabling it alongside HYPERLOOM_KTH_ENABLE is a configuration error rather
than a silent precedence rule: an operator who asked to observe should not
end up enforcing, or the reverse.

Observations land in integration/results/NNNN.kth_shadow.json beside that
publication's NNNN.json, each carrying recorded_only and a label that says
it decided nothing. Off by default; behaviour is unchanged when unset.

Co-authored-by: Cursor <cursoragent@cursor.com>
artifacts_root=session_dir / "kth_shadow",
session_id=session_dir.name,
)
except BaseException as error: # noqa: BLE001
artifacts_root=artifacts_root,
session_id=session_id,
)
except BaseException as error: # noqa: BLE001

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants