[michaelgold]

Summary

• publish the complete 27-task, 270-round BlenderBench direct protocol and reference results
• add hash-bound generation, visual inspection, checkpoint attestation, independent CUDA audit, and standalone scoring
• publish reproducibility documentation, immutable result links, evaluator attribution, and portable CI

Verification

• 76 focused direct/publication tests passed
• 41 MCP CLI/tool-listing tests passed
• static validation: 22 Python files, 6 JSON files, 27 tasks, 108 assets, 11 implementation hashes
• CPython 3.13 x86-64 hash-lock dry-run resolved all 41 packages
• broad suite: 435 tests; 8 known environment-only errors from missing Blender/Pillow; 23 skips; no assertion failures
• independent final review: APPROVED

No benchmark generation or goal/solution code was executed during release verification.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings remain in scoring, provenance, checkpoint validation, orchestration, CI coverage, and runtime handling.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Publishes a reproducible, hash-bound 27-task, 270-round BlenderBench direct suite with MCP/Codex execution, independent auditing, scoring, reference artifacts, and portable CI.

Changes:

• Adds benchmark orchestration, datasets, manifests, scoring, recovery, and video tooling.
• Adds checkpoint attestation, runtime auditing, Codex controls, and focused tests.
• Publishes reproducibility documentation, reference results, provenance metadata, and CI.

File summaries

File	Changes and review notes
tools/direct_suite.py	Compatibility runner. Moderate (2 votes): direct execution may fail to import the repository package unless the repository root is added to sys.path.
tools/DIRECT_SUITE.md	Direct-suite usage documentation.
tools/direct_runtime.py	Independent Blender audit and rendering runtime. Moderate (1 vote): missing reviewed markers should fail closed.
tools/direct_codex.py	Codex launcher and evidence validation. Critical (1 vote): preflight trusts model-controlled values instead of binding to MCP arguments and server-side evidence. Critical (1 vote): reject symlink or non-regular rounds.jsonl ledgers.
tools/direct_codex.md	Launcher documentation. Nit (3 votes): documents workspace-write although the launcher uses --sandbox read-only.
tools/direct_codex.example.toml	Example MCP configuration.
tests/test_tool_listing.py	Tool-listing tests.
tests/test_direct_unrestricted.py	Unrestricted execution tests.
tests/test_direct_suite.py	Direct-suite orchestration tests.
tests/test_direct_saved_cleanup.py	Saved-scene cleanup tests.
tests/test_direct_recovery.py	Recovery tests.
tests/test_direct_codex.py	Codex and checkpoint validation tests.
tests/test_blender_cli.py	Blender CLI behavior tests.
readme.md	Published benchmark result and provenance links.
mcp/blmcp/tools/execute_blender_code.py	Checkpoint execution and rendering. Critical (1 vote): rehash the source after arbitrary code runs and reject changes before creating checkpoint attestations.
benchmarks/token_efficiency/creative_smoke.py	Creative token-efficiency smoke benchmark.
benchmarks/token_efficiency/codex_runner.py	Codex launch specification.
benchmarks/blenderbench_direct/VIGA-LICENSE.txt	VIGA license text.
benchmarks/blenderbench_direct/video.py	Compilation video generation. Moderate (2 votes): clamp positive-duration frame counts to at least one. Moderate (1 vote): explicitly reject negative elapsed time.
benchmarks/blenderbench_direct/TDD_EVIDENCE.md	Verification evidence record.
benchmarks/blenderbench_direct/suite.py	Generation, recovery, and scoring orchestration. Moderate (2 votes): persist cancellation for the current and remaining tasks before stopping. Critical (1 vote): recheck source identity after generation and before auditing or scoring.
benchmarks/blenderbench_direct/scoring.py	Standalone scoring and validation. Critical (1 vote): require the exact unique round identifier set, not merely the count of successful rows.
benchmarks/blenderbench_direct/score_pair.py	Pinned GPU scorer.
benchmarks/blenderbench_direct/runner.py	Benchmark CLI commands.
benchmarks/blenderbench_direct/requirements-scoring.in	Scoring dependency inputs.
benchmarks/blenderbench_direct/requirements-fixture.lock	Fixture dependency lockfile.
benchmarks/blenderbench_direct/requirements-fixture.in	Fixture dependency inputs.
benchmarks/blenderbench_direct/ref_based_eval.py	Vendored reference evaluator.
benchmarks/blenderbench_direct/README.md	Reproduction documentation.
benchmarks/blenderbench_direct/publication/results-summary.json	Published aggregate results.
benchmarks/blenderbench_direct/publication/reference-results.json	Published reference results. Moderate (3 votes): replace mutable resolve/main links with the immutable dataset revision. The related artifact manifest has the same Moderate (3 votes) issue.
benchmarks/blenderbench_direct/protocol.py	Protocol and source-identity manifests. Moderate (1 vote): reject dirty checkouts during recovery or scoring as well as generation.
benchmarks/blenderbench_direct/dataset.py	Dataset staging and validation.
benchmarks/blenderbench_direct/clip-snapshot-manifest.json	CLIP snapshot hashes.
benchmarks/blenderbench_direct/__init__.py	Package initialization.
.gitignore	Local benchmark output exclusions.
.github/workflows/blenderbench-portable.yml	Portable fixture CI. Moderate (2 votes): include mcp/blmcp/** in both duplicated path filters.
.gitattributes	Evaluator hash protections.

Review details

Suppressed comments (4)

benchmarks/blenderbench_direct/protocol.py:235

• The manifest check rejects a run recorded from a dirty checkout, but it does not reject a dirty checkout used for recovery or scoring: current = collect_source_identity() includes git_dirty, yet this condition only inspects source. That allows the claimed clean provenance to be consumed from an unclean tree, contrary to the documented invariant; require current['git_dirty'] to be false as well.

    if source["git_dirty"] is not False:
        raise ValueError("publication run manifest requires a clean Git source identity")

benchmarks/blenderbench_direct/video.py:15

• The validation error promises a nonnegative elapsed time, but this condition never rejects negative values. A caller can receive timeline state with a negative elapsed_seconds and a misleading zero-token/round-0 state; explicitly check elapsed < 0.

    if not math.isfinite(elapsed) or not math.isfinite(duration) or duration <= 0:
        raise ValueError("finite nonnegative elapsed and positive duration required")

tools/direct_codex.py:105

• execute_blender_code_for_cli executes arbitrary Python via exec, but this gate only regex-scans the serialized arguments. Dynamically constructed paths or strings (for example, concatenating goal and .py, decoding bytes, or reading a path from Blender data) bypass the regex while still allowing evaluator access, so the run can claim no feedback when it did receive it. Enforce the input-only filesystem boundary at the execution/OS boundary or record an execution-level access audit instead of relying on lexical matching.
  tools/direct_runtime.py:377
• This check accepts a missing reviewed field because only an explicit False is rejected. Since initialize() is the trust boundary for parent-reviewed literal operations, an omitted marker should fail closed just like an unreviewed policy.

• Files reviewed: 39/43 changed files
• Comments generated: 12
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.