feat(testing): an E2E scenario board, derived from the runs rather than written - #421
Merged
Merged
Conversation
…an written The Core's cross-surface explorer and the Tracker's RoboSoft robots each emit machine-readable evidence, into two places, and nobody reads them together. This renders both onto one page. It is NOT a second backlog, and the line is the point. `gap-tracking.md` stays the only place a defect's status lives; this page owns "which scenarios exist, which ran, what was observed" and CITES GT ids without ever owning one. Two pages claiming truth about one defect drift, and the drift is always found late — which is what that board spends its time catching. Derived rather than maintained, for the same reason the interface how-to is generated from live capture: a hand-written matrix is stale the first time a suite changes and nobody remembers the file. Deliberately NOT wired into the derived-artifact chain (guard 46). Both inputs are gitignored and exist only after a local run, so a fresh CI checkout has none — and a chain link nobody can satisfy is a gate that gets routed around. Instead each section states when it was measured and from which run, and a missing input renders as "NOT RUN", which is a different statement from "nothing failed". Two readings of the RoboSoft file were wrong before this one, and both are pinned in comments: `c.ok` is a field it does not emit (so `undefined !== false` counted every check as a pass), and a soft check that PASSED belongs under `passed` — `soft` is reserved for one that failed and was downgraded. The page now carries a reconciliation banner that fires when the per-robot sums disagree with the run's own totals; it is what caught the second reading. What it says today: Core 73 operations declared, 48 bound and executed across 66 invocations, 4 declared on all three surfaces and never reached, and 12 consistency observations (5 P1, 7 P2) — the surfaces disagree on success for the same operation. Tracker 229/0/1 across 10 robots against a live cluster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
📊 Bilingual Coverage ImpactPR Changes
Repository Coverage
✅ Good: All EN changes have ES counterparts. Generated by GitHub Actions |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Core's cross-surface explorer and the Tracker's RoboSoft robots each emit machine-readable evidence, into two places, and nobody reads them together. This renders both onto one page.
It is not a second backlog
That line is the point.
gap-tracking.mdstays the only place a defect's status lives. This page owns which scenarios exist, which ran, what was observed and citesGT-NNNwithout ever owning one. Two pages claiming truth about one defect drift, and the drift is always found late — which is what that board spends its time catching.Derived, not maintained
Same reasoning as the interface how-to, which is generated from live capture: a hand-written matrix is stale the first time a suite changes and nobody remembers the file.
Deliberately NOT wired into the derived-artifact chain (guard 46). Both inputs are gitignored and exist only after a local run, so a fresh CI checkout has none — and a chain link nobody can satisfy is a gate that gets routed around. Instead each section states when it was measured and from which run, and a missing input renders as
NOT RUN, which is a different statement from "nothing failed".Two wrong readings, both pinned
The RoboSoft evidence was misread twice before this landed:
c.ok— a field it does not emit.undefined !== falseis true, so every check counted as a pass.softcheck as its own bucket — but a soft check that passed belongs underpassed;softis reserved for one that failed and was downgraded to non-blocking. Measured: 226 pass+hard, 3 pass+soft, 1 fail+soft, 0 fail+hard → the runner's 229/0/1.The page now carries a reconciliation banner that fires when per-robot sums disagree with the run's own totals. It is what caught the second reading.
What it says today
The 12 observations are worth a look on their own: the surfaces disagree on
successfor the same operation (gate-evaluate,sdlc-status,validate-satellite,dora-metrics,agents-listat P1). They are not registered as gaps here — that is the board's job and a separate, human-reviewed step.ci-runner.mjs governance04-check-bilingual-parity--checkmode🤖 Generated with Claude Code