Skip to content

usability: make CLI help defaults portable and sync init docs #173

Description

@changliu2

Reporter perspective

As a first-time user, I ran CLI help commands before running a demo. Two things were confusing: the docs disagree with the actual init --max-turns default, and results help prints an absolute path from the local checkout instead of a portable default.

Evidence

init --max-turns default drift

  • assert-eval init --help prints:
    --max-turns INTEGER  Maximum conversation turns.  [default: 30]
    
  • assert_eval/init/_command.py:73-78 sets default=30.
  • docs/cli/commands.md:37 says:
    - `--max-turns <int>` optional, default `20`

Absolute local paths in help output

  • assert-eval results status --help prints a default like:
    --results-dir PATH  Results root to inspect.  [default: C:\Users\changliu2\project\adaptive-eval-fleet-usability\artifacts\results]
    
  • The default comes from assert_eval/cli.py:30:
    DEFAULT_RESULTS_DIR = ROOT / "artifacts" / "results"
  • This absolute path is technically correct for the checkout, but it is noisy and looks machine-specific in copy-pasted help output. The same pattern shows up on other results commands and related artifact commands.

Recommended fix

  • Update docs/cli/commands.md:37 to default 30.
  • Make Click display a portable default such as ./artifacts/results or artifacts/results even if the internal default remains absolute.
  • Spot-check:
    • assert-eval results status --help
    • assert-eval results list --help
    • assert-eval results compare --help
    • assert-eval results compare-suites --help
    • assert-eval judge-traces --help

Slice rollup

Found by slice 2:

  • C:\Users\changliu2\.copilot\session-state\3714f9ab-3680-4990-a750-a80c932203f2\files\usability-slice-2-rollup.md

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    StalebugSomething isn't workingdocumentationImprovements or additions to documentationshould-fixConfusing or visibly rough but not launch-blocking

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions