Skip to content

fix(testing): the E2E board published 12 divergences that do not exist #423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
beyondnetPeru merged 1 commit into from
Aug 4, 2026
Merged

fix(testing): the E2E board published 12 divergences that do not exist #423

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions .harness/scripts/generate-e2e-scenario-board.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,29 @@ function readJson(p) {
}
}

/**
* When the file was written, ISO, or null.
*
* The Core's `coverage.json` carries no timestamp of its own, and the first
* version of this page therefore rendered it with no indication of age. That is
* not cosmetic: on 2026-08-03 it published 12 "cross-surface divergences" from a
* capture made hours earlier by a run whose MCP OPA engine had crashed for want
* of `--experimental-vm-modules`, so every `tools/call` fail-closed with
* FORBIDDEN. Re-run correctly, the same suite reports ZERO. The page was
* faithful to its input and the input was junk, which is exactly the failure a
* derived artifact is supposed to make impossible.
*
* Age is now stated. A reader can see at a glance that the measurement predates
* the tree it claims to describe.
*/
function mtimeIso(p) {
try {
return fs.statSync(p).mtime.toISOString();
} catch {
return null;
}
}

function readJsonl(p) {
try {
return fs
Expand Down Expand Up @@ -141,6 +164,7 @@ const T = {
'These carry a binding on every surface and no invocation reached them. They are the honest edge of this run, listed rather than rounded away.',
measuredAt: 'Measured',
fromRun: 'from run',
howGen: 'Produced by',
},
es: {
title: 'Cuadro de escenarios E2E',
Expand Down Expand Up @@ -172,10 +196,11 @@ const T = {
'Tienen binding en cada superficie y ninguna invocación llegó a ellas. Son el borde honesto de esta corrida: se listan en vez de redondearse.',
measuredAt: 'Medido',
fromRun: 'de la corrida',
howGen: 'Producido por',
},
};

function render(lang, { coverage, findings, robo }) {
function render(lang, { coverage, findings, coverageAt, robo }) {
const t = T[lang];
const L = [];
const other = lang === 'en' ? './e2e-scenario-board.es.md' : './e2e-scenario-board.md';
Expand All @@ -200,6 +225,8 @@ function render(lang, { coverage, findings, robo }) {
L.push('');
L.push(`${t.howTo}: \`npm run test:exploration\``);
} else {
L.push(`${t.measuredAt} \`${coverageAt ?? '?'}\`. ${t.howGen}: \`npm run test:exploration\`.`);
L.push('');
L.push(`| ${t.metric} | ${t.value} |`);
L.push('|---|---|');
L.push(`| Operations declared | ${coverage.totalOperations} |`);
Expand Down Expand Up @@ -322,11 +349,12 @@ function main(argv = process.argv.slice(2)) {

const coverage = readJson(path.join(CORE_OUT, 'coverage.json'));
const findings = readJsonl(path.join(CORE_OUT, 'findings.jsonl'));
const coverageAt = mtimeIso(path.join(CORE_OUT, 'coverage.json'));
const robo = latestRoboSoftRun();

const outputs = [
[OUT_EN, render('en', { coverage, findings, robo })],
[OUT_ES, render('es', { coverage, findings, robo })],
[OUT_EN, render('en', { coverage, findings, coverageAt, robo })],
[OUT_ES, render('es', { coverage, findings, coverageAt, robo })],
];

let stale = false;
Expand Down
17 changes: 3 additions & 14 deletions reference/core/control-center/testing/e2e-scenario-board.es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ Regenerar con: `node .harness/scripts/generate-e2e-scenario-board.mjs`

## Core — exploración cross-superficie (CLI · MCP · REST)

Medido `2026-08-04T10:24:38.541Z`. Producido por: `npm run test:exploration`.

| Medida | Valor |
|---|---|
| Operations declared | 73 |
Expand All @@ -31,20 +33,7 @@ Tienen binding en cada superficie y ninguna invocación llegó a ellas. Son el b

### Observaciones

| Severidad | Tipo | Operación | Superficies | Observación |
|---|---|---|---|---|
| P1 | consistency | `gate-evaluate` | cli · mcp · rest | Cross-surface success divergence on gate-evaluate |
| P1 | consistency | `sdlc-status` | cli · mcp | Cross-surface success divergence on sdlc-status |
| P1 | consistency | `validate-satellite` | cli · mcp · rest | Cross-surface success divergence on validate-satellite |
| P1 | consistency | `dora-metrics` | cli · mcp | Cross-surface success divergence on dora-metrics |
| P1 | consistency | `agents-list` | cli · mcp | Cross-surface success divergence on agents-list |
| P2 | consistency | `phase-advance` | cli · mcp · rest | Cross-surface success divergence on phase-advance |
| P2 | consistency | `detect-drift` | cli · mcp · rest | Cross-surface success divergence on detect-drift |
| P2 | consistency | `evaluate` | cli · mcp · rest | Cross-surface success divergence on evaluate |
| P2 | consistency | `composable-validate` | cli · mcp · rest | Cross-surface success divergence on composable-validate |
| P2 | consistency | `recommend-topology` | cli · mcp · rest | Cross-surface success divergence on recommend-topology |
| P2 | consistency | `phase-artifacts-evaluate` | cli · mcp · rest | Cross-surface success divergence on phase-artifacts-evaluate |
| P2 | consistency | `topology-list` | mcp · rest | Cross-surface success divergence on topology-list |
Sin observaciones en la corrida registrada.

## Tracker — robots RoboSoft contra un clúster vivo

Expand Down
17 changes: 3 additions & 14 deletions reference/core/control-center/testing/e2e-scenario-board.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ Regenerate with: `node .harness/scripts/generate-e2e-scenario-board.mjs`

## Core — cross-surface exploration (CLI · MCP · REST)

Measured `2026-08-04T10:24:38.541Z`. Produced by: `npm run test:exploration`.

| Measure | Value |
|---|---|
| Operations declared | 73 |
Expand All @@ -31,20 +33,7 @@ These carry a binding on every surface and no invocation reached them. They are

### Observations

| Severity | Type | Operation | Surfaces | Observation |
|---|---|---|---|---|
| P1 | consistency | `gate-evaluate` | cli · mcp · rest | Cross-surface success divergence on gate-evaluate |
| P1 | consistency | `sdlc-status` | cli · mcp | Cross-surface success divergence on sdlc-status |
| P1 | consistency | `validate-satellite` | cli · mcp · rest | Cross-surface success divergence on validate-satellite |
| P1 | consistency | `dora-metrics` | cli · mcp | Cross-surface success divergence on dora-metrics |
| P1 | consistency | `agents-list` | cli · mcp | Cross-surface success divergence on agents-list |
| P2 | consistency | `phase-advance` | cli · mcp · rest | Cross-surface success divergence on phase-advance |
| P2 | consistency | `detect-drift` | cli · mcp · rest | Cross-surface success divergence on detect-drift |
| P2 | consistency | `evaluate` | cli · mcp · rest | Cross-surface success divergence on evaluate |
| P2 | consistency | `composable-validate` | cli · mcp · rest | Cross-surface success divergence on composable-validate |
| P2 | consistency | `recommend-topology` | cli · mcp · rest | Cross-surface success divergence on recommend-topology |
| P2 | consistency | `phase-artifacts-evaluate` | cli · mcp · rest | Cross-surface success divergence on phase-artifacts-evaluate |
| P2 | consistency | `topology-list` | mcp · rest | Cross-surface success divergence on topology-list |
No observations in the recorded run.

## Tracker — RoboSoft robots against a live cluster

Expand Down
Loading