[tardigrde]

Summary

Comprehensive update folding together the run-all multi-suite layer, static validation, expanded reporting, and side-effect contract enforcement.

Changes

Run-all / multi-suite

• Concurrent eval-suite execution with paired skill↔suite discovery
• Recursive eval discovery, changed-only mode, plugin eval layout
• Dry-run planning without agent calls, planned-case preview, plan totals
• Cost/duration totals, pass-rate gating, JUnit reports
• Failed-case rerun commands with context preservation
• Artifact archive + cross-run compare

Validation & assertions

• Strict eval-suite linting and structured-field validation
• Negative artifact assertions, command-assertion false-positive fix
• Run-all assertion-failure reporting

Grading

• Constrained OpenWebUI grader backend (localhost default)

Side effects & reporting

• Enforced side-effect contracts, GitLab cleanup manifests
• Static HTML reports, skill portfolio report
• Skill-hash + run-trajectory recording
• doctor command, JSON discovery output

New modules: fixtures.py, validation.py

Generics

Corporate-specific fixtures removed — tool is fully generic and usable by everyone. Default LLM grader backend is OpenRouter (public). Pre-existing lint nits folded in.

Verification

• pytest: 386 passed, 7 deselected (live-only)
• ruff check + ruff format --check: clean

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces significant enhancements to the agent-skill-eval harness, including support for structured deterministic assertions, side-effect level classification, and GitLab MR integration. It also adds new CLI commands such as run-all and doctor, alongside improved validation for eval suites and documentation. The review identified several critical bugs: a logic error in test failure detection that could cause false negatives, a potential IndexError crash in the harness when parsing prompts, incorrect git status parsing for renamed files, and improper error handling for git commands that could lead to false-positive test results. All identified issues include actionable code suggestions to improve robustness.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The failure marker " failed" (with a leading space) will match successful test summary outputs like "5 passed, 0 failed", causing a false-negative test failure. Consider refining the failure markers or using a regex to ensure we don't match "0 failed".

Suggested change

	failure_markers = (" failed", "failure", "traceback", "panic:", "error:")
	pass_markers = (
	" passed",
	"all tests passed",
	"test result: ok",
	"build succeeded",
	"tests passed",
	"ok ",
	)
	if any(marker in logs_lower for marker in failure_markers):
	return self._structured_result(assertion, False, "Failure marker found in logs")
	failure_markers = ("failure", "traceback", "panic:", "error:")
	if any(marker in logs_lower for marker in failure_markers) or re.search(r"\b[1-9]\d*\s+failed", logs_lower):
	return self._structured_result(assertion, False, "Failure marker found in logs")
	pass_markers = (
	" passed",
	"all tests passed",
	"test result: ok",
	"build succeeded",
	"tests passed",
	"ok ",
	)

[gemini-code-assist]

If the prompt ends with the marker (e.g., "fake_create_file:") without any trailing filename, prompt.split(marker, 1)[1].split() will return an empty list, causing an IndexError: list index out of range and crashing the harness. Consider checking if the split list is non-empty before accessing the first element.

Suggested change

	rel = prompt.split(marker, 1)[1].split()[0].strip().strip("`'\"")
	target = self.workspace / rel
	target.parent.mkdir(parents=True, exist_ok=True)
	target.write_text("created by fake harness\n", encoding="utf-8")
	parts = prompt.split(marker, 1)[1].split()
	if parts:
	rel = parts[0].strip().strip("\"'\\\"")
	target = self.workspace / rel
	target.parent.mkdir(parents=True, exist_ok=True)
	target.write_text("created by fake harness\\n", encoding="utf-8")

[gemini-code-assist]

git status --porcelain formats renamed files as R orig_path -> new_path. Simply slicing line[3:] will result in "orig_path -> new_path" being treated as a single filename, which will cause downstream file existence or contract checks to fail. Consider splitting the path by -> when a rename is detected.

    changed_files_set = set()
    for line in status_lines:
        if len(line) >= 4:
            path = line[3:].strip()
            if " -> " in path:
                path = path.split(" -> ", 1)[1]
            if path:
                changed_files_set.add(path)
    changed_files = sorted(changed_files_set)

[gemini-code-assist]

In _check_structured_file_unchanged, if git status or git diff fails (returns a non-zero exit code), the errors are silently ignored, and the method incorrectly falls through to returning a successful AssertionResult ("No change detected"). If the git commands fail, the assertion should fail or report the error as evidence.

        status = subprocess.run(
            ["git", "status", "--porcelain", "--", assertion.path],
            cwd=ws,
            capture_output=True,
            text=True,
            check=False,
        )
        if status.returncode != 0:
            return self._structured_result(assertion, False, f"git status failed: {status.stderr.strip()}")
        if status.stdout.strip():
            return self._structured_result(assertion, False, f"Uncommitted change detected: {status.stdout.strip()}")

        if self.pre_state and self.post_state and self.pre_state.head_sha and self.post_state.head_sha:
            diff = subprocess.run(
                ["git", "diff", "--name-only", self.pre_state.head_sha, self.post_state.head_sha, "--", assertion.path],
                cwd=ws,
                capture_output=True,
                text=True,
                check=False,
            )
            if diff.returncode != 0:
                return self._structured_result(assertion, False, f"git diff failed: {diff.stderr.strip()}")
            if diff.stdout.strip():
                return self._structured_result(assertion, False, f"Committed change detected: {diff.stdout.strip()}")