What
Add evalview validate [path] that checks test YAML/TOML files for schema errors, missing required fields, and invalid values — without calling any agent.
Why
Catching bad test files before running saves time and API costs. Useful in CI as a fast pre-check.
Scope
- New command:
evalview validate [path]
- Validates each file against the Pydantic
TestCase model
- Reports all errors with file name + line hint, not just the first one
- Exit code 0 = all valid, 1 = errors found
Getting Started
- TestCase model with validators:
evalview/core/types.py lines 159-319
- Field validators already exist: name validation (line 263), adapter whitelist (line 284), tag normalization (line 298)
- Reference for validation command:
evalview/commands/skill_cmd.py lines 42-96 (skill validate does something similar for skills)
- Loader:
evalview/core/loader.py
Acceptance Criteria
What
Add
evalview validate [path]that checks test YAML/TOML files for schema errors, missing required fields, and invalid values — without calling any agent.Why
Catching bad test files before running saves time and API costs. Useful in CI as a fast pre-check.
Scope
evalview validate [path]TestCasemodelGetting Started
evalview/core/types.pylines 159-319evalview/commands/skill_cmd.pylines 42-96 (skill validatedoes something similar for skills)evalview/core/loader.pyAcceptance Criteria
evalview validate tests/reports all errors across filesevalview validate tests/ --jsonoutputs machine-readable results