Reporter perspective
As a first-time user, I ran CLI help commands before running a demo. Two things were confusing: the docs disagree with the actual init --max-turns default, and results help prints an absolute path from the local checkout instead of a portable default.
Evidence
init --max-turns default drift
assert-eval init --help prints:
--max-turns INTEGER Maximum conversation turns. [default: 30]
assert_eval/init/_command.py:73-78 sets default=30.
docs/cli/commands.md:37 says:
- `--max-turns <int>` optional, default `20`
Absolute local paths in help output
assert-eval results status --help prints a default like:
--results-dir PATH Results root to inspect. [default: C:\Users\changliu2\project\adaptive-eval-fleet-usability\artifacts\results]
- The default comes from
assert_eval/cli.py:30:
DEFAULT_RESULTS_DIR = ROOT / "artifacts" / "results"
- This absolute path is technically correct for the checkout, but it is noisy and looks machine-specific in copy-pasted help output. The same pattern shows up on other
results commands and related artifact commands.
Recommended fix
- Update
docs/cli/commands.md:37 to default 30.
- Make Click display a portable default such as
./artifacts/results or artifacts/results even if the internal default remains absolute.
- Spot-check:
assert-eval results status --help
assert-eval results list --help
assert-eval results compare --help
assert-eval results compare-suites --help
assert-eval judge-traces --help
Slice rollup
Found by slice 2:
C:\Users\changliu2\.copilot\session-state\3714f9ab-3680-4990-a750-a80c932203f2\files\usability-slice-2-rollup.md
Reporter perspective
As a first-time user, I ran CLI help commands before running a demo. Two things were confusing: the docs disagree with the actual
init --max-turnsdefault, andresultshelp prints an absolute path from the local checkout instead of a portable default.Evidence
init --max-turnsdefault driftassert-eval init --helpprints:assert_eval/init/_command.py:73-78setsdefault=30.docs/cli/commands.md:37says:Absolute local paths in help output
assert-eval results status --helpprints a default like:assert_eval/cli.py:30:resultscommands and related artifact commands.Recommended fix
docs/cli/commands.md:37todefault 30../artifacts/resultsorartifacts/resultseven if the internal default remains absolute.assert-eval results status --helpassert-eval results list --helpassert-eval results compare --helpassert-eval results compare-suites --helpassert-eval judge-traces --helpSlice rollup
Found by slice 2:
C:\Users\changliu2\.copilot\session-state\3714f9ab-3680-4990-a750-a80c932203f2\files\usability-slice-2-rollup.md