028: benchmark expansion — agent/skill/generate commands

[rotnov]

Summary

• AC-1: cortex benchmark agent <overlay-path> — runs promptfoo with kernel+overlay combined system prompt, prints 6-dimension table, exits 0
• AC-2: Agent scores stored under scores.json["agents"]["<name>"]["baseline"] + mutations array; runs.jsonl entries tagged type=agent
• AC-3: cortex benchmark skill <skill-path> — looks up benchmarks/skills/<name>.yaml, injects skill as system prompt, prints per-test pass/fail + overall, always exits 0
• AC-4: benchmarks/skills/ievo.yaml reference config created (2 test cases); skill scores stored under scores.json["skills"]
• AC-5: cortex benchmark generate <rule-text> [--dimension] [--append] — generates promptfoo test case via Anthropic API, stdout by default, --append does YAML round-trip into promptfooconfig.yaml
• AC-6: Reuses check_api_key() + .env loading; lazy import anthropic; anthropic added to dev deps
• AC-7: 197 tests total, 0 failures; existing 24 tests untouched; 173 new tests (unit + CLI + integration)

Architecture changes

• benchmark.py: new dataclasses (AgentScores, SkillBenchmarkEntry, SkillScores), extended ScoresFile (backward compatible — missing agents/skills keys default to {}), refactored run_promptfoo(config_path=None), new parse_skill_results(), infer_dimension(), generate_test_case(), append_skill_run_log()
• cli.py: 3 new benchmark_app commands (agent, skill, generate), import tempfile promoted to module level

Test plan

• uv run pytest tests/ -q → 197 passed, 0 failed
• uv run ruff check src/cortex/ tests/ → all checks passed
• Backward compat: existing scores.json (without agents/skills) loads without error
• Temp file cleanup: TemporaryDirectory context manager ensures cleanup even on failure
• Agent + skill scores coexist in scores.json without overwriting each other

🧬 Evolved with iEVO

[rotnov]

Direction check FAIL — the following acceptance criteria are not fully addressed in the PR description:

AC-1 — Error handling for missing files not confirmed. The AC requires:

• Missing dist/iEVO.md → prints "Error: dist/iEVO.md not found — run cortex compile first", exits 1
• Missing <overlay-path> → prints "Error: overlay file not found: ", exits 1
  The PR description confirms the happy path only.

AC-2 — runs.jsonl entry format incomplete. The AC requires both "type": "agent" AND "agent": "<name>" fields. The PR description mentions only type=agent.

AC-3 — Error handling not confirmed. The AC requires:

• Missing skill path → exits 1
• Missing benchmarks/skills/<name>.yaml → prints "No skill tests found: benchmarks/skills/.yaml — create test file first", exits 1
  The PR description confirms the happy path only.

AC-5 — Generated test case structure not confirmed. The AC requires the output to have:

• description containing the rule keyword and dimension name
• vars.prompt creating a scenario where the rule applies
• assert with at least one llm-rubric AND one icontains-any
• vars.dimension set to the detected/provided dimension name
  The PR description only says "generates promptfoo test case via Anthropic API".

AC-6 — Error messages and generation config not confirmed. The AC requires:

• API key missing → "Error: ANTHROPIC_API_KEY not set — add it to .env or export it", exits 1
• API call fails → prints error message, exits 1
• Temperature 0, model claude-haiku-4-5-20251001
  The PR description confirms the API setup reuse but not these specifics.

Please update the PR description to explicitly address each of these points.

[rotnov]

Architect Review — FAIL

Task: 028 | Branch: feat/028-benchmark-expansion

Plan verified

All 8 plan points verified against implementation:

• ScoresFile extended with agents: dict[str, AgentScores] and skills: dict[str, SkillScores] — AgentScores, SkillScores, SkillBenchmarkEntry dataclasses all present. from_dict() uses .get() with defaults (backward compat). (benchmark.py:130-214)
• run_promptfoo() refactored with optional config_path: Path | None = None — backward compatible, existing callers pass no argument. (benchmark.py:281)
• tempfile.TemporaryDirectory used for agent and skill temp files — both commands use context manager, cleanup is automatic. (cli.py:336, cli.py:458)
• Skill YAML (benchmarks/skills/ievo.yaml) has only tests key, provider injected at runtime. (cli.py:441-452)
• anthropic added to [project.optional-dependencies] dev. Lazy import inside generate_test_case(). (pyproject.toml:17, benchmark.py:560)
• Dimension inference via keyword map — _DIMENSION_KEYWORDS dict + infer_dimension(), falls back to structure_adherence. (benchmark.py:483-502)
• YAML round-trip with pyyaml for --append — yaml.safe_load() → append to tests list → yaml.dump(..., default_flow_style=False). (cli.py:540-549)

---

Deviation found

Q-001-arch — append_run_log() not extended for skill runs

Plan (Architecture Notes):

Skill runs pass run_type="skill" and extra={"skill": name} to the extended append_run_log().

Implementation:

• append_run_log() was extended for kernel and agent runs as planned.
• Skill runs use a separate append_skill_run_log(skill_name, scores, overall) function instead.

The deviation is intentional — skill runs lack the naked/kernel dual-score structure that append_run_log(naked, kernel, ...) requires. The coder added a dedicated function rather than contort skill data into the kernel-oriented signature.

Impact: runs.jsonl now has two entry formats (dual-score for kernel/agent, flat scores dict for skill). Any downstream consumer must handle both.

Full deviation report: .ievo/tasks/028/reports/Q-001-arch.md

---

Decision required: Accept the deviation (plan was wrong about skill data shape) or redesign append_run_log() to accept a flexible data dict for unified logging.