Skip to content

evalview validate — lint test files without running them #144

@hidai25

Description

@hidai25

What

Add evalview validate [path] that checks test YAML/TOML files for schema errors, missing required fields, and invalid values — without calling any agent.

Why

Catching bad test files before running saves time and API costs. Useful in CI as a fast pre-check.

Scope

  • New command: evalview validate [path]
  • Validates each file against the Pydantic TestCase model
  • Reports all errors with file name + line hint, not just the first one
  • Exit code 0 = all valid, 1 = errors found

Getting Started

  • TestCase model with validators: evalview/core/types.py lines 159-319
  • Field validators already exist: name validation (line 263), adapter whitelist (line 284), tag normalization (line 298)
  • Reference for validation command: evalview/commands/skill_cmd.py lines 42-96 (skill validate does something similar for skills)
  • Loader: evalview/core/loader.py

Acceptance Criteria

  • evalview validate tests/ reports all errors across files
  • Shows filename and specific field errors
  • evalview validate tests/ --json outputs machine-readable results
  • Exit code 1 on any validation error (CI-friendly)
  • Works without an API key or running agent

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions