Repository navigation
feat(kernel): KTH shadow mode — record the gate's verdict, enforce nothing - #1685
Draft
jatseng-ai wants to merge 1 commit into
Draft
jatseng-ai wants to merge 1 commit into
jatseng-ai wants to merge 1 commit into
Conversation
Before giving the qualification gate authority over integration it is worth knowing what it would have done. HYPERLOOM_KTH_SHADOW_ENABLE qualifies each applied patch for the record and changes nothing: no patch is reverted, no benchmark is skipped, no KEEP is affected, and every patch's status is what it would have been with the flag unset. The failure mode worth designing against is shadow mode quietly becoming enforcement, so three properties are enforced by construction rather than by convention. KthShadowObservation has no 'eligible' attribute and no other boolean summarising the outcome, and the enforcement path admits a patch by reading qualified.eligible, so an observation cannot be substituted for a qualification. _observe_in_shadow returns None, so there is no value for the integration loop to branch on and the observation never enters the same object as the integration result. Every failure - in the observer, in the provider, in writing the artifact - becomes a recorded observation, because a recording instrument must not be able to fail the thing it records. Enabling it alongside HYPERLOOM_KTH_ENABLE is a configuration error rather than a silent precedence rule: an operator who asked to observe should not end up enforcing, or the reverse. Observations land in integration/results/NNNN.kth_shadow.json beside that publication's NNNN.json, each carrying recorded_only and a label that says it decided nothing. Off by default; behaviour is unchanged when unset. Co-authored-by: Cursor <cursoragent@cursor.com>
| artifacts_root=session_dir / "kth_shadow", | ||
| session_id=session_dir.name, | ||
| ) | ||
| except BaseException as error: # noqa: BLE001 |
| artifacts_root=artifacts_root, | ||
| session_id=session_id, | ||
| ) | ||
| except BaseException as error: # noqa: BLE001 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
One flag,
HYPERLOOM_KTH_SHADOW_ENABLE, that runs the KTH qualification foreach applied patch and enforces nothing. No patch is reverted, no benchmark
is skipped, no KEEP is affected, and every patch's status is exactly what it
would have been with the flag unset.
Based on
feature/kth-qualification-gate-v2(the head of #1672) so this diffshows only the shadow work. Please do not merge, and please do not treat
this as part of #1672.
Why
#1672 adds a gate that can revert a patch. Before giving it that authority over
integration it is worth knowing what it would actually have done on the patches
this repository produces: how often it would have refused, and whether it would
have been right. Finding that out by turning the gate on is the wrong order of
operations.
The property that matters
The failure mode worth designing against is shadow mode quietly becoming
enforcement — through a later refactor, a copied line, or an operator setting
the wrong variable. Three things prevent it, by construction rather than by
convention:
An observation carries nothing to enforce on. The enforcement path
admits a patch by reading
qualified.eligible.KthShadowObservationhas noeligibleattribute, noperformance_admitted, and no other booleansummarising the outcome —
recorded_onlyis the only bool on it, and it isalways
True. An observation cannot be substituted for a qualificationwithout failing at the point it is written. A test asserts the absence.
The observation never enters the integration result.
_observe_in_shadowreturnsNone, so there is no value for the loop tobranch on, and the record is written to its own
integration/results/NNNN.kth_shadow.jsonrather than intoPatchIntegrationResult.kth. The two join on the index without either beingable to contaminate the other.
A recording instrument cannot fail the thing it records. Every failure —
in the observer, in the provider, in reading the candidate, in writing the
artifact — becomes a recorded observation with
shadow_errorset. Thecontainment is at the call site as well as inside the observer, deliberately:
the integration path must survive any observer, including one a later
change makes less careful. A test injects an observer that raises and asserts
both patches are still kept.
Enabling shadow mode alongside
HYPERLOOM_KTH_ENABLEis aKthConfigurationErrorrather than a silent precedence rule. An operator who asked to observe should
not end up enforcing, and one who asked to enforce should not quietly stop.
Shadow observations are never bound to a performance admission:
qualifyiscalled,
mark_performance_admittedis not.What lands
kernel/kth_shadow.pyKthShadowObservation,KthShadowObserver.kernel/controller_patch_integration.py_observe_in_shadow, one call site, one new optional parameter.phases/kernel.pykth_shadow_observer=KthShadowObserver.from_env().docs/reference/environment-variables.mddocs/reference/kth-qualification.mdSeven new tests.
src/hyperloom/orchestrator/kernel/tests/is green at 387passed / 6 skipped, and the phases suite too (424 / 6 together).
ruff checkisclean on every file touched; the six pre-existing
BLE001findings inconc_sweep.pyandrequest_handlers.pyare unchanged and unrelated.Off by default. With both flags unset nothing in this diff executes.
Scope
unmodified.
for merge either.
What this does not establish
Nothing has been observed yet. This is the instrument, not a result. The number
worth having — how often KTH would have refused a patch this repository kept,
and how often that refusal would have been correct — needs the flag turned on
for a real run, which has not happened.