Repository navigation
026: cognitive benchmark for mutation validation - #11
Merged
Merged
Conversation
Subtask 01: DimensionScores, BenchmarkEntry, MutationEntry, ScoresFile with JSON serialization, baseline seeding, and regression detection. 15 tests. promptfoo added to dev deps. Co-Authored-By: iEVO <noreply@ievo.ai>
Subtask 02: structure_adherence, challenge_reflex, plan_first, decision_logging, ac_verification, evolution_awareness. Mix of deterministic (icontains) and llm-rubric assertions. `promptfoo validate` passes. Co-Authored-By: iEVO <noreply@ievo.ai>
Add benchmark sub-app to cortex CLI (cortex benchmark run, cortex benchmark compare). Fix unused imports in benchmark.py and cli.py caught by ruff. Co-Authored-By: iEVO architect <noreply@ievo.ai>
Co-Authored-By: iEVO architect <noreply@ievo.ai>
- Add `prompts: ["{{prompt}}"]` and `prompt` vars to each test case
(promptfoo needs explicit prompt template, not just description)
- Don't treat non-zero exit as fatal — promptfoo returns 100 when
assertions fail, which is expected behavior. Check output file instead.
- Seed baseline scores (overall: 0.50)
Co-Authored-By: iEVO architect <noreply@ievo.ai>
parse_results() now filters by provider label ("with-kernel") instead
of averaging both providers together. Re-seeded baseline: overall 0.67.
Co-Authored-By: iEVO architect <noreply@ievo.ai>
Co-Authored-By: iEVO architect <noreply@ievo.ai>
Co-Authored-By: iEVO architect <noreply@ievo.ai>
- Show Naked vs Kernel scores table in `cortex benchmark run/compare` - Add `naked` field to scores.json (LLM without kernel) - Append every run to benchmarks/runs.jsonl (gitignored, local log) - 147 tests pass Co-Authored-By: iEVO architect <noreply@ievo.ai>
- Replace Ollama provider with anthropic:messages:claude-haiku-4-5-20251001 - Add .env loading for ANTHROPIC_API_KEY - Replace check_ollama() with check_api_key() - temperature: 0 for deterministic results Co-Authored-By: iEVO architect <noreply@ievo.ai>
Rewrite all 6 test cases to target iEvo-specific behaviors: - structure_adherence: iEvo task spec format with YAML frontmatter - challenge_reflex: vague request that should trigger questions - plan_first: explicit "start coding now" to test planning reflex - decision_logging: pick a tool and document WHY - ac_verification: implement with AC-1..AC-4 checklist - evolution_awareness: discover problems during implementation Results: naked=0.17, kernel=0.33, delta=+0.17 Key win: evolution_awareness naked=0 kernel=1.0 Also: max_tokens 4096, broader icontains-any matches. Co-Authored-By: iEVO architect <noreply@ievo.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cortex benchmark runandcortex benchmark compareCLI commands for kernel mutation validationsrc/cortex/benchmark.py) with dataclasses for scores, baseline tracking, and comparison logicTest plan
uv run pytest tests/— 146 passeduv run ruff check src/cortex/— cleanollama serve+ollama pull qwen2.5:7b+cortex benchmark run(requires local Ollama)🧬 Evolved with iEVO