Run Level 2 validation probes

[burtenshaw]

This PR runs local Level 2 probes, grades startup/reward/observation/state, and records replayable evidence with explicit skips for unfinished checks. Builds on #1179; third of three stacked PRs tracked in #1177.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Schema worker truncates Unicode payloads
  • The grader now serializes worker payloads without ASCII escaping, with a regression test covering Unicode traces within the wire budget.

Or push these changes by commenting:

@cursor push c8f6b6ffb8

Preview (c8f6b6ffb8)

diff --git a/src/openenv/validation/graders/runtime/basic.py b/src/openenv/validation/graders/runtime/basic.py
--- a/src/openenv/validation/graders/runtime/basic.py
+++ b/src/openenv/validation/graders/runtime/basic.py
@@ -143,7 +143,10 @@
         try:
             checked = subprocess.run(
                 [sys.executable, "-I", str(worker)],
-                input=json.dumps({"schema": schema, "observations": observations}),
+                input=json.dumps(
+                    {"schema": schema, "observations": observations},
+                    ensure_ascii=False,
+                ),
                 capture_output=True,
                 text=True,
                 timeout=5,

diff --git a/tests/test_validation/test_runtime_grading.py b/tests/test_validation/test_runtime_grading.py
--- a/tests/test_validation/test_runtime_grading.py
+++ b/tests/test_validation/test_runtime_grading.py
@@ -219,6 +219,45 @@
     )
 
 
+def test_unicode_trace_within_wire_budget_reaches_schema_worker_intact(tmp_path):
+    text = "\U0001f600" * 150_000
+    rows = []
+    for index in range(6):
+        operation = "reset" if index == 0 else "step"
+        rows.append(
+            WireExchange(
+                operation,
+                json.dumps({"type": operation}),
+                json.dumps(
+                    {
+                        "type": "observation",
+                        "data": {
+                            "observation": {"counter": index, "text": text},
+                            "reward": None if index == 0 else 0,
+                            "done": False,
+                        },
+                    },
+                    ensure_ascii=False,
+                ),
+            )
+        )
+    assert (
+        sum(
+            len(row.request_json.encode()) + len(row.response_json.encode())
+            for row in rows
+        )
+        < 8 * 1024 * 1024
+    )
+    assert len(json.dumps([text] * len(rows))) > 10 * 1024 * 1024
+    schema = copy.deepcopy(OBSERVATION_SCHEMA)
+    schema["properties"]["text"] = {"type": "string"}
+    schema["required"].append("text")
+
+    result = ObservationSchemaGrader().run(subject_with(tmp_path, rows, schema=schema))
+
+    assert result.status is CheckStatus.PASS
+
+
 @pytest.mark.parametrize(
     "field,value", [("done", None), ("done", "false"), ("done", 0), ("observation", [])]
 )

_{You can send follow-ups to the cloud agent here.}

[cursor]

Alignment Review Report

Two-tier review of the RFC-008 Level 2 execution slice 3 (startup, reward, observation, state checks). This is a stacked PR (base ben/rfc008-l2-02-docker), so the repo's test/lint workflows don't gate it — the results below are from running the suites locally, plus the dedicated runtime-validation CI job.

Automated Checks

• Lint: PASS (PR files). ruff format --check, usort check, and ruff check are all clean on the 14 changed .py files. The full-tree lint.sh failure (57 ruff format + usort exit 2) is pre-existing envs/ + test_grid_world.py/test_julia_env.py drift, not introduced here.
• Debug code: CLEAN. No print/breakpoint/pdb/TODO/FIXME in the PR's src/ files.
• Tests (run locally): green. 97 new runtime unit tests pass; full tests/test_validation = 292 passed (incl. test_policy, test_schema_sync, test_runner); tests/test_cli/test_validate.py = 11 passed (no regression — contrast slice 1). The Docker integration test (test_runtime_cli.py, @pytest.mark.docker) isn't runnable in this env, but the dedicated "Runtime validation / Linux Docker" CI job passed.

Open RFCs Context

• RFC 008 — Environment Auto-Validation (Status: In Review, @zkwentz). This PR implements the "Level 2 execution amendment", slice 3, and is faithful to it: bounded public probe loader (65 536 B / depth 32 / 10 000 nodes / 1–100 actions, duplicate-key + non-finite rejection, no seed/episode_id override); a single-session collector recording raw uncoerced request/response strings; reset-reward-may-be-null / step-reward-must-be-a-finite-number-in-range-excluding-booleans; the state identity + step_count contract; policy v2 adding runtime.startup (v1 rejected for runtime); the FAIL/ERROR/SKIP taxonomy; unsupported network/GPU refused before build; an explicit SKIP inventory for not-yet-implemented Level-2 checks; and a redacted evidence bundle.
• No other RFC (001/002/003/004) is in conflict; RFC 004 rubrics remain leveraged-not-required.

Tier 1: Fixes Required

• None. No mechanical, type, import, or security defects found.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. The change actively reinforces several invariants — worth affirming:

• Dual API boundary / agent isolation: the collector drives the Gym-like reset/step/state loop as validation orchestration (agent_boundary: api), never as an agent tool — no agent-facing exposure of simulation control.
• Client–server separation: validation talks to the subject purely over the wire (httpx/websockets); it imports nothing from server/.
• Rewards in environment: graders only observe the environment's own reward against the declared range; no external reward computation.
• No credential exposure: strong throughout — httpx(trust_env=False), LaunchSpec inherits no host env, artifact redaction (secret keys + token regex), failure reasons carry only exception types, and tests assert secrets never reach reports/traces.
• Security depth (nice work): JSON-Schema evaluation is sandboxed in a disposable python -I subprocess with CPU/AS rlimits + a wall-clock deadline (ReDoS → finding, not hang); non-local $ref/$dynamicRef are rejected and the worker uses an empty referencing.Registry (no SSRF/file retrieval); source_digest opens files O_NOFOLLOW and rejects symlinks.

Three small, non-blocking notes are left inline (report coherence on mid-run source change; --policy exit-code semantics; HTTP /schema vs the WebSocket-only direction).

RFC Conflicts

ALIGNMENT FLAG: Implementation landing while RFC 008 is still In Review

• Principle/RFC at stake: RFC 008 (Status: In Review) — Level 2 execution amendment
• The concern: the standing sequencing flag for this stack — the slice freezes real behavior (manifest schema v2, severity policy v2, the validation.execution sidecar contract) against an amendment that is not yet Accepted. The implementation matches the current amendment text; the ask is to confirm those L2 contracts are settled before further slices stack on them. Non-blocking.
• Suggested reviewer: @zkwentz (RFC author); @Darktex (principles/invariants owner, git blame b1a92e1 on INVARIANTS.md)

Summary

• 0 mechanical issues to fix (Tier 1)
• 1 alignment point for human review (RFC-008 In-Review sequencing) + 3 minor non-blocking inline notes
• 0 hard RFC conflicts — the implementation is faithful to the RFC 008 Level 2 amendment

  

_{Sent by Cursor Automation: Pre-review}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Stale comment

Exact-head CI is red on the same inherited failure as #1178: test_write_report_validation_report_only. Fix lands in #1178; please rebase after that greens. Holding deeper Level-2 probe review until then.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale comment

REQUEST_CHANGES at c7e89896

Still blocked on inherited #1178 CI (test_write_report_validation_report_only). Separately, Bugbot’s High finding on this head is real:

collector treats exceptions from the final close send / WebSocket context exit as collection failures (failure_reason), and runtime graders then FAIL the episode even when reset/step/state already succeeded. Teardown noise must not fail a measured episode.

Also track (Medium):

1. Schema worker stdin.read(10MB) can truncate large observation batches under the 8MB trace cap
2. socket.send is not bound by episode_timeout_s (only pre-checked via remaining())

Fix close-failure handling here; rebase after #1178 greens. Not Thursday 0.6.0 cargo.

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Malformed envelopes crash runtime graders
  • Runtime graders now convert AttributeError from non-object envelope data into a FAIL finding while preserving the collector failure reason.

Or push these changes by commenting:

@cursor push ea46021702

Preview (ea46021702)

diff --git a/src/openenv/validation/graders/runtime/basic.py b/src/openenv/validation/graders/runtime/basic.py
--- a/src/openenv/validation/graders/runtime/basic.py
+++ b/src/openenv/validation/graders/runtime/basic.py
@@ -58,7 +58,14 @@
             problems.append(evidence.failure_reason)
         try:
             problems.extend(self.check(subject, evidence))
-        except (ValueError, TypeError, KeyError, RecursionError, OverflowError):
+        except (
+            ValueError,
+            TypeError,
+            KeyError,
+            AttributeError,
+            RecursionError,
+            OverflowError,
+        ):
             problems.append("malformed runtime evidence")
         return CheckResult(
             check_id=self.check_id,

_{You can send follow-ups to the cloud agent here.}

[cursor]

Stale comment

REQUEST_CHANGES at 2950de74

Inherited #1178 CI is green now. The High teardown issue remains on this head: close/WS-exit exceptions still set failure_reason and fail graders after a successful reset/step/state episode.

Also still open (Medium): schema-worker 10MB stdin truncate; socket.send not bound by episode_timeout_s.

Not Thursday 0.6.0 cargo.

  

_{Sent by Cursor Automation: Release}

[cursor]

Stale comment

REQUEST_CHANGES at d724b7e5 (High/Medium fixed; Low residual)

Fixed (prior High/Medium):

• Completed-episode teardown: complete + best-effort close/send no longer sets failure_reason / fails graders; regressions in test_runtime_transport.py cover close-send, close-failure, and genuine in-episode failure precedence.
• Transport sends are bounded via _bounded_call + raw-socket abort against episode_timeout_s.
• Schema worker input budget raised / wire JSON preserved (response_json / schema_json) so unicode and compact 1e9 episodes within wire limits no longer truncate.

Still open (Low): graders call data.get(...) after data = response["data"] without isinstance(data, dict). The collector can store a raw exchange then raise on a non-object data envelope; ObservationSchemaGrader / state graders then AttributeError outside the (ValueError, TypeError, …) catch → grader ERROR instead of contract FAIL. Guard with isinstance(data, dict) (or equivalent) before .get.

Holding approval for that guard (+ exact-head CI still in flight). Not Thursday 0.6.0 cargo.

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Episode timeout silently capped at 300s
  • Runtime collection now honors the manifest's validated episode timeout without imposing an unrelated 300-second cap.
• ✅ Fixed: Schema worker stdin encoding can fail
  • The schema worker subprocess now explicitly encodes text input as UTF-8 regardless of the parent locale.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Stale comment

REQUEST_CHANGES at d724b7e5 (Bugbot refresh)

Prior High/Medium (teardown / bounded send / schema wire budget) remain fixed.

New Medium: runner.py silently caps collection at min(..., 300.0) while policy bounds allow max_episode_timeout_s=3600. Legal longer episodes can false-FAIL. Use the policy-validated declared timeout (or align the cap with the bound).

Still open Lows:

1. Malformed non-object data → AttributeError in graders (ERROR vs FAIL).
2. Schema-worker stdin: parent text=True/ensure_ascii=False vs worker UTF-8 stdin.buffer (locale-dependent).

Bugbot autofix is active — will re-review on the next head. Not Thursday 0.6.0 cargo.

  

_{Sent by Cursor Automation: Release}

[cursor]

Stale comment

REQUEST_CHANGES at f405e4ec (autofix progress)

Fixed this head:

• Medium timeout cap: collection now uses manifest.resources.episode_timeout_s directly; regression asserts 600.0 is passed through.
• Low schema-worker encoding: parent subprocess.run(..., encoding="utf-8") aligns with UTF-8 stdin.buffer reads.

Still open (Low): graders still call data.get(...) without isinstance(data, dict) after loading a possibly non-object envelope the collector stored before rejecting. That AttributeError escapes the grader catch set → ERROR instead of contract FAIL.

Holding approval for that guard. Exact-head CI still running. Not Thursday 0.6.0 cargo.

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Startup failures hide provider diagnostics
  • Runtime startup outcomes now retain the sanitized exception details for runtime-plan, startup, and provider failures, with regression coverage for each path.

Or push these changes by commenting:

@cursor push 0577dc4585

Preview (0577dc4585)

diff --git a/src/openenv/validation/runner.py b/src/openenv/validation/runner.py
--- a/src/openenv/validation/runner.py
+++ b/src/openenv/validation/runner.py
@@ -185,27 +185,27 @@
             "runtime.startup", CheckStatus.SKIP, str(exc), started=started
         )
         checks = []
-    except RuntimePlanError:
+    except RuntimePlanError as exc:
         result = _outcome(
             "runtime.startup",
             CheckStatus.FAIL,
-            "runtime plan is missing, unsafe or invalid",
+            str(exc),
             started=started,
         )
         checks = []
-    except StartupError:
+    except StartupError as exc:
         result = _outcome(
             "runtime.startup",
             CheckStatus.FAIL,
-            "subject failed build or readiness; inspect the Docker fixture/build inputs",
+            str(exc),
             started=started,
         )
         checks = []
-    except ProviderError:
+    except ProviderError as exc:
         result = _outcome(
             "runtime.startup",
             CheckStatus.ERROR,
-            "provider could not complete a bounded operation",
+            str(exc),
             started=started,
         )
         checks = []

diff --git a/tests/test_validation/test_runtime_execution.py b/tests/test_validation/test_runtime_execution.py
--- a/tests/test_validation/test_runtime_execution.py
+++ b/tests/test_validation/test_runtime_execution.py
@@ -7,7 +7,7 @@
 
 import pytest
 from openenv.validation.policy import load_policy, PolicyError
-from openenv.validation.providers import StartupError
+from openenv.validation.providers import ProviderError, StartupError
 from openenv.validation.report import CheckResult
 from openenv.validation.runner import run_validation, source_digest
 from openenv.validation.runtime.artifacts import write_runtime_bundle
@@ -172,6 +172,8 @@
     provider = FakeRuntimeProvider()
     report = run_validation(package, max_level=Level.RUNTIME, provider=provider)
     assert report.verdict.value == "fail"
+    result = next(r for r in report.results if r.check_id == "runtime.startup")
+    assert "invalid runtime plan" in result.evidence[0]
     assert not provider.builds
 
 
@@ -185,9 +187,23 @@
     report = run_validation(package, max_level=Level.RUNTIME, provider=provider)
     result = next(r for r in report.results if r.check_id == "runtime.startup")
     assert result.status is CheckStatus.FAIL
+    assert result.evidence == ["subject build failed"]
     assert report.verdict.value == "fail"
 
 
+def test_provider_failure_diagnostics_are_visible(package):
+    provider = FakeRuntimeProvider()
+
+    def failed_build(*args):
+        raise ProviderError("provider deadline elapsed")
+
+    provider.build = failed_build
+    report = run_validation(package, max_level=Level.RUNTIME, provider=provider)
+    result = next(r for r in report.results if r.check_id == "runtime.startup")
+    assert result.status is CheckStatus.ERROR
+    assert result.evidence == ["provider deadline elapsed"]
+
+
 @pytest.mark.parametrize("failure", [RuntimeError, KeyboardInterrupt])
 def test_collector_crash_or_cancel_always_tears_down(package, monkeypatch, failure):
     provider = FakeRuntimeProvider()

_{You can send follow-ups to the cloud agent here.}

[cursor]

Stale comment

REQUEST_CHANGES at 16d68689 (CI green; Bugbot refresh)

Prior High/Medium (teardown, bounded send, schema wire budget, timeout cap, UTF-8 encoding) remain fixed. Exact-head CI is green.

New Medium: runtime.startup swallows sanitized StartupError / RuntimePlanError / ProviderError details into generic strings — authors lose actionable diagnostics. Surface a bounded sanitized reason.

Still open (Low): malformed non-object data → grader AttributeError (ERROR vs FAIL).

Bugbot autofix is active. Not Thursday 0.6.0 cargo.

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Teardown failure hides startup findings
  • Teardown failures now only mark cleanup incomplete, preserving the existing runtime startup and collection findings.

Or push these changes by commenting:

@cursor push 61b156dfda

Preview (61b156dfda)

diff --git a/src/openenv/validation/runner.py b/src/openenv/validation/runner.py
--- a/src/openenv/validation/runner.py
+++ b/src/openenv/validation/runner.py
@@ -234,12 +234,6 @@
                 cleanup["completed"] = True
             except Exception:
                 cleanup["completed"] = False
-                result = _outcome(
-                    "runtime.startup",
-                    CheckStatus.ERROR,
-                    "subject teardown failed",
-                    started=started,
-                )
     return [result, *checks], attempted, plan, evidence, inspection, cleanup

_{You can send follow-ups to the cloud agent here.}

[cursor]

Stale comment

REQUEST_CHANGES at eb7a40b1 (Bugbot Medium)

Prior APPROVE at this tip is withdrawn. Autofix is still in flight; do not merge until the teardown overwrite is fixed.

Tier 1

• src/openenv/validation/runner.py _runtime() finally: when running.stop() fails, the existing runtime.startup result is replaced with a generic teardown ERROR. That discards prior PASS/FAIL/ERROR evidence (provider reason, collection failure_reason, etc.) and contradicts the helper docstring (“retain failure evidence through teardown”).

Required: keep the original result (and its evidence); record cleanup failure via cleanup["completed"] = False and/or append a separate note without overwriting attribution. Add a regression that fails primary startup/collection then fails stop(), asserting the primary finding remains visible.

Not Thursday 0.6.0 cargo until merge.

  

_{Sent by Cursor Automation: Release}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Initial digest errors skip the report

Medium Severity

The initial source_digest call is not on the report-preserving path. A symlink or non-regular file now raises before any checks run, so authors get an internal-error exit instead of a report. The same ValueError and OSError cases after collection are caught and recorded.

Additional Locations (2)

• src/openenv/validation/runner.py#L52-L64
• src/openenv/validation/runner.py#L347-L350

 

^{Reviewed by Cursor Bugbot for commit bea5115. Configure here.}

[cursor]

Stale comment

REQUEST_CHANGES at exact tip bea5115d — Bugbot Medium (valid):

Initial source_digest skips the report path. digest_before = source_digest(target) (~L282) is uncaught. Symlink / non-regular-file ValueErrors (and OSError from os.open) abort with an internal error before any checks run. Post-collection source_digest already catches (ValueError, OSError) and records a reportable failure — initial digest should do the same (or otherwise emit a ValidationReport ERROR) for symmetry with the Autofix that landed in ccca191e.

Autofix is already kicked off — will re-review when the tip advances. Keep out of Thursday 0.6.0 cargo until green.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Failed digest still reads rejected source
  • Validation now reports the digest failure without parsing, re-digesting, or running the rejected package source.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Stale comment

REQUEST_CHANGES still holds at 65be5dd8 — prior Autofix fixed the uncaught initial digest, but Bugbot Medium is valid on the new shape:

Failed digest still reads rejected source. Catching source_digest and setting digest_before = "" then continues into parser.parse and _runtime before stamping static.manifest ERROR. Those paths can follow the same symlinks / non-regular files the digest rejected, so host content may enter the report or build context before the ERROR is recorded.

Fail closed: on initial digest (ValueError, OSError), emit a ValidationReport with a reportable ERROR (and empty/"" source_digest) and return without parse/runtime. Keep the regression that asserts no internal-error exit and no private path leakage.

Autofix already kicked off — will re-review when the tip advances. Keep out of Thursday 0.6.0.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Stale comment

@zkwentz — Ben wants your review before folding this into Thursday’s 0.6.0 cut. Tip refreshed onto current main (16240f19) so you’re not reviewing a stale branch; CI will re-run on this head.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Interrupt during close drops episode
  • WebSocket close teardown now swallows KeyboardInterrupt like other best-effort teardown failures, preserving completed or previously failed episode evidence.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Stale comment

REQUEST_CHANGES at exact tip 16240f19 — Bugbot Medium (valid) after merge-main re-scan:

Interrupt during post-episode websocket close drops a finished episode. _bounded_call re-raises KeyboardInterrupt; the teardown except Exception handlers miss it. A completed collection then surfaces as interrupt: startup → ERROR, graders never run, failure_phase still names the last successful op.

Post-episode close/teardown should treat KeyboardInterrupt as best-effort (same as other teardown failures) once complete is True, or convert to RuntimeCollectionInterrupted without discarding measured evidence. Autofix is already kicked off — will re-review on the new tip.

Hold fold into Thursday 0.6.0 until this is green + Zach’s review.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Re-APPROVE at dd59275f (Bugbot Autofix)

Confirmed the Medium was real: post-episode KeyboardInterrupt during _bounded_call close/teardown bypassed except Exception, so a finished collection was surfaced as RuntimeCollectionInterrupted and graders never ran.

Autofix is correct and minimal — teardown except (Exception, KeyboardInterrupt) on both the graceful close send and the socket close — with a focused regression in test_runtime_transport.py. Local transport suite: 10 passed.

Still waiting on @zkwentz for the human gate before folding into Thursday 0.6.0. Exact-head CI is re-running on this tip.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Re-APPROVE at d94f0218

Autofix force-pushed the same close-interrupt fix (collector teardown catches KeyboardInterrupt; transport suite 10/10). Prior APPROVE was on superseded tip dd59275f (+1 test-line churn only).

Still waiting on @zkwentz + exact-head CI before Thursday 0.6.0 fold.

View PR

  

_{Sent by Cursor Automation: Release}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Source digest uses Unix-only flags
  • Made O_NONBLOCK optional in source_digest and added a regression test for platforms where the flag is unavailable.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit f84f8a3. Configure here.}

[rycerzes]

Reviewed at 297b578b and ran the pinned lab on macOS arm64 (Python 3.12.13, uv 0.9.3), using Podman 5.8.4 behind the Docker CLI:

• fast: 384 passed, no skips; artifacts verify. protocol: pass.
• docker: once the Podman portability issues in the merged provider were patched locally (filing those separately against main), the Echo canary and all five grader fault cases pass. The graders' verdicts are correct.
• I also added validation: overlays to three real envs, each built from its own Dockerfile:
  • maze_env: surfaced a real bug. reset() ignores episode_id, and state_contract catches it.
  • finqa_env and qed_math_env (both MCP): gave the results behind the inline comments.

Comments 1, 4 and 5 still apply at the #1248 head, so fix them wherever suits the stack. I'm happy to take 1 and 4 as a follow-up if that helps.

[rycerzes]

1 — should fix. Every runtime grader appends evidence.failure_reason, so a single transport failure FAILs reward_well_formed, observation_schema and state_contract together with the same message. The hung-step fixture and qed_math (reset error) both produce three identical FAILs. Could the collection failure be reported once, on runtime.startup or a dedicated episode outcome, with the dependent checks SKIPping as "unmet dependency"?

[rycerzes]

2 — question. Reset observations are validated against /schema's single observation_cls. MCP envs reset with a base Observation but advertise CallToolObservation, which requires tool_name, so every MCP env fails here. Echo isn't special: finqa fails the same way, and non-MCP maze passes. Should reset be skipped or validated against base Observation until /schema includes a reset schema (#1215)?

[rycerzes]

3 — question. This follows the RFC amendment (no null or bool step rewards). But core's Observation.reward is bool | int | float | None and defaults to None, and envs return null on intermediate tool steps (finqa's get_descriptions, plus at least 7 others). Do we tighten the core contract, or allow null on non-terminal steps?

[rycerzes]

4 — diagnosability. finqa's actual error is 'tool_name' is a required property, but the report says schema mismatch at /. The keyword (error.validator) and the schema-side property names come from the schema, not from subject values, so they're safe to include. Relatedly, at collector.py:196 a typed {"type":"error","data":{"code":"FACTORY_ERROR"}} reply becomes reset failed (ValueError); surfacing data.code would help authors.

[rycerzes]

5 — should fix or document. request_timeout_s is fixed at 5 s, and runner.py:162 never overrides it, so it applies whatever episode_timeout_s says. A judged step or a slow reset will time out. I found this by reading the code; I couldn't exercise it (see 6).

[rycerzes]

6 — question, heads-up for later slices. qed_math can't reset: its rubric builds the OpenAI client at session creation and raises OpenAIError because no key is set. #1248 adds LaunchSpec.env_vars, but only for the validation token. Is a manifest-declared secret mechanism planned before the rubric and semantic checks, since LLM-judge envs can't reach a graded step without one?

[rycerzes]

7 — nit. Check IDs are hard-coded in _applicable, in the two dependency-reason sets (:356, :385), in the grader classes, and in the severity JSON. A small registry, or grader-declared metadata, would stop these drifting as more slices land.

[rycerzes]

Nits.

• --local does nothing except reject --url, but the docs present it as a mode.
• Teardown failure records only completed: false, with no reason.
• The redaction key regex catches prompt_token_ids and max_tokens.
• Each run leaves one untagged image behind.