Repository navigation
Open
Labels
blockedBlocked on an upstream gate or dependencyBlocked on an upstream gate or dependencybugSomething isn't workingSomething isn't working
Description
Problem
The CyberBattleSim baseline reproduction cannot produce a valid mediated attempt on dev (0272949), so it cannot produce the Revision 3 comparisons.
_execute_mediated_attempt(src/raes_adapters/cyberbattlesim/reproduction.pyL2352-2412) runspython -m raes_adapters.cli run --mode smokefor thecyberbattlesim-chainpack once per attempt.- The CLI admission gate rejects that command. A local run recorded in docs(cyberbattlesim): inventory evidence-capture requirements #100 printed
researcher.validation.evidence-unverifiable, exited3and wrote nothing. CI's clean-install probe expects the same rejection fromvalidate --mode smokefor this pack (noxfile.pyL667-702,_expect_evidence_rejection). - Every mediated attempt is therefore terminalized with
disposition="failed"andreproduction.mediated.execution-failed(L1530-1567). The lane still writes its collection withvalid_count0 and seals its inventory (L1657-1681). - Each comparison needs
_ATTEMPTS_PER_LANE(10, L96) attempts in both lanes and otherwise returns"result": "unavailable"(L1823-1824, L1853-1854). - Bypassing the admission gate would not restore the lane. After the episode files, the CLI calls
validate_experiment_run_against_task(cli.pyL1633) before it writesrun.json. Since fix: remove hardcoded capability and evidence gates #90 the run record's evidence artifact carries nosatisfies_refs, so the join rejectsattacker-action-logandavailability-seriesand the command exits6.
#30, which added this reproduction, is closed, so nothing tracks the regression. The evidence reconciliation that would let run --mode smoke pass both gates is #86, which is under the 2026-08-13 hold until OpenRAE/rae#1023 is settled.
Scope
- Restore valid mediated attempts in the baseline reproduction once the
cyberbattlesim-chainrun path passes the admission gate and the post-run join. - Add a check that fails when a mediated lane seals a collection with no valid attempt, so a regression like this one is reported rather than recorded as an empty lane.
Done when
- On
dev, the reproduction's mediated lane produces ten valid attempts with the packaged controls. - The Revision 3 comparisons report a pass, fail or weakened result instead of
unavailable. - A test fails when the mediated lane cannot produce a valid attempt.
Related
- docs(cyberbattlesim): inventory evidence-capture requirements #100 (records the failing mediated lane in the CyberBattleSim inventory)
- fix(cyberbattlesim): reconcile SDL evidence requirements, simulator capture, and backend manifest #86, test(cyberbattlesim): reproduce the selected public baseline through RAES #30, fix: remove hardcoded capability and evidence gates #90
Metadata
Metadata
Assignees
Labels
blockedBlocked on an upstream gate or dependencyBlocked on an upstream gate or dependencybugSomething isn't workingSomething isn't working
Activity