[napetrov]

Design note and its first implementation: docs/grading-tiers.md. Implemented: grading checks on atoms and probes with the roles veto, sufficient and evidence; the identifiers check; skill novelty ground (a veto from a tool's own --help); key verification (meta.key_status). Still proposed: the exec and source checks, derived answer slots, source drift.

Problem. Novelty probes are graded by one LLM judge, and the only check on that judge is inter-judge agreement. Two judges can agree and both be wrong.

Proposal. Grade each probe with the most objective grader available, as a ladder: (1) verifiable (exec, cli_help), (2) grounded (source snapshot from official docs, man pages, Sphinx inventories), (3) judge. The checks reuse the grading.checks schema from eval score reference. Each check has a role (veto / sufficient / evidence). The judge always runs in shadow, so on the subset a check decides, judge accuracy against ground truth comes out of the same run.

Prototype (no model calls, stored lab probe results, 7 public-tool skills):

• 26% of 1,371 public atoms have an identifier a source can check. Every hand-reviewed not_found came from a negative claim, a version-bound claim, the wrong tool or a router gap; none was a skill error.
• Treating slot presence as a positive grader agrees with the judges less than the judges agree with each other (75.8% / 80.0% vs 87.6%). In all 6 hand-checked disagreements, the judge was right.
• Veto works: 3 of 137 perf answers named flags that don't exist in perf 6.17 (perf report --stdin and others). Judge 1 graded all three correct. The sample is tiny.

Open questions are listed in the note. This round fixed the review on 8e7de9c (the veto's negation and scope, the bracketed help, key verification, the atomic write of ground, the sufficient-check tests) and corrected the doc claims the review named. The one question it raised (a screen reference) is fixed.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 95.74468% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.73%. Comparing base (37ba217) to head (1ea8665).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/novelty/grounding.py	93.84%	5 Missing and 3 partials ⚠️
dokimi/dokimi/eval/reference_checks.py	95.78%	2 Missing and 2 partials ⚠️
dokimi/dokimi/commands/novelty_ground.py	94.11%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #472    +/-   ##
========================================
  Coverage   95.73%   95.73%            
========================================
  Files         403      405     +2     
  Lines       54061    54375   +314     
  Branches     9308     9364    +56     
========================================
+ Hits        51755    52056   +301     
- Misses       1421     1430     +9     
- Partials      885      889     +4     

Files with missing lines	Coverage Δ
dokimi/dokimi/cli_routes.py	98.03% <ø> (ø)
dokimi/dokimi/commands/skill.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/judge2.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/measure.py	98.73% <100.00%> (+0.19%)	⬆️
dokimi/dokimi/novelty/probes.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/report.py	98.78% <ø> (ø)
dokimi/dokimi/commands/novelty_ground.py	94.11% <94.11%> (ø)
dokimi/dokimi/eval/reference_checks.py	95.62% <95.78%> (+0.28%)	⬆️
dokimi/dokimi/novelty/grounding.py	93.84% <93.84%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi skill novelty ground`
    + option `--atoms`
    + option `--help/-h`
    + option `--out`
    + option `--tool`
    + option `--tool-version`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[Copilot]

🟡 Changes recommended

Unresolved critical defects can reject correct answers, accept invented flags, crash grading, and stall measurement workers through unbounded regex scans.

9 open issues

• Apply regex timeouts to within and pattern scans · New
• Handle unmatched optional regex capture groups · New
• Require explicit support for negated options · New
• Distinguish rejected help output from a help snapshot · New
• Allowlist help spellings only when demonstrated · New
• Restrict vetoes to positively recommended invocations · New
• Validate parsed commands before spawning · New
• Select atoms by tool name, not answer option syntax · New
• Run checks after judge failures on successful answers · New

What changed in this PR

12 new findings: 9 must-fix and 3 lower-priority documentation nits.

This PR adds programmatic checks alongside dokimi’s novelty judge and documents the broader grading-ladder proposal. Unlike the description’s docs-only scope, it implements identifier vetoes, verdict resolution, and check reporting.

Changes:

• Add skill novelty ground to attach identifier checks derived from tool help.
• Carry grading checks through probes, resolve verdicts by check role, and record judge/check comparisons.
• Add regression tests, user documentation, a design note, and a changelog entry.

File	Description
dokimi/​tests/​test_novelty_checks.py	Tests grounding, check roles, verdicts, and serialization.
dokimi/​mkdocs.yml	Adds the grading-ladder page to navigation.
dokimi/​dokimi/​novelty/​report.py	Includes check summaries in probe records.
dokimi/​dokimi/​novelty/​probes.py	Validates and attaches atom grading to probes.
dokimi/​dokimi/​novelty/​measure.py	Runs checks, resolves verdicts, and summarizes comparisons.
dokimi/​dokimi/​novelty/​grounding.py	Builds and attaches help-derived identifier vetoes.
dokimi/​dokimi/​eval/​reference_checks.py	Adds identifier checks and role/tier validation.
dokimi/​dokimi/​commands/​skill.py	Registers the grounding command and arguments.
dokimi/​dokimi/​commands/​novelty.py	Implements grounding of saved atom records.
dokimi/​docs/​skill-novelty.md	Documents program checks in novelty evaluation.
dokimi/​docs/​README.md	Links the grading-ladder design note.
dokimi/​docs/​grading-tiers.md	Describes implementation, proposals, and prototype evidence.
dokimi/​changelog.d/​20261108_120000_napetrov_grading-tiers.md	Announces program checks and the grounding command.

_{Review effort: Balanced}

[exai-bot]

OpenCodeReview: Commit 1ea8665 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `1ea8665`: 13 still running, 4 passing, 1 skipped

Check	Status
Build docs (strict)	still running
CLI interface diff	still running
CodeQL (actions)	still running
CodeQL (python)	still running
External catalog smoke test (uxlfoundation/skills)	still running
Linux Python 3.11	still running
Linux Python 3.14	still running
Linux Wheels Python 3.11	still running
Linux Wheels Python 3.14	still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com...	still running

3 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (cdefa30): 5 medium</summary>

Open threads from the bot: 0

</details>

[napetrov]

Open risks at 4aef7d8, and where each one stands:

• Negation earlier in the clause (aJoyg, vlad's thread): asked on the review; no change until you choose.
• Contractions and curly apostrophes (YCl-): fixed in f801b50 and 4479e10.
• Bare -- with no comma (YCl4): known limit, documented in 4aef7d8.
• Two nits on da817b9 and 488d95a: already fixed, marker f07dbb2.

[vlad-perevezentsev]

Review at 7755042. Seven of my eight points from the last round are fixed and tested: the judge2 comparison, the sufficient refusal, the docs in answer-formats.md, --, regex validation, the write error and the full missing list. The negation point is still open, and I answered your (a)/(b) question in its thread. One new gap: a run kept at the spend-cap stop skips its checks (line comment).

What I ran, at 7755042:

• make ci-local --offline: 18,588 passed, 0 failed, 72 skipped; every row passes.
• Repros with mocked clients: the veto on the negation answers in the thread, and a judge reply refused as SpendCapReached on a vetoed answer. The proposed negation rule, patched in at test time: all 75 tests in test_novelty_checks.py pass.
• On Windows (Server 2025, Python 3.13): not run. The VM cannot make a venv from the lock (litellm==1.106.4 needs filelock<4, the lock pins 4.2.4); main has the same pins, so it is not from this PR.

Checked, no issues

• judge2: regrade samples and compares the first judge's verdict, and the new test fails on the old code (2 sampled, agreement 0).
• Denials after a flag: does not exist / is not an option / doesn't / curly apostrophes skip only the flag they follow, also past a closing backtick.
• Key verification: verify_keys takes missing from identifier_names, so every unknown name is kept, and it reads through the same scan as the veto.

[napetrov]

Re: review by vlad-perevezentsev

One new gap: a run kept at the spend-cap stop skips its checks
Fixed in 31113d6, the same change as the thread above; pinned by a new test.

[napetrov]

Re: review by antonwolfy

a negation word earlier in the clause skips the command after it
Declined for this PR: without X, use cmd stays a documented known false pass (8fdc1b4). The narrow rule for without, no and if you can't ... <command> forms is #669.

[vlad-perevezentsev]

LGTM

I checked f1d9202. 4e85279 fixes the capped run: the veto now grades a judge reply the spend cap refused, and a reply with no verdict still counts as ungraded, so the file stays partial. The negation point is a documented known limit, with #669 for the narrow rule. I agree with keeping it out of this PR, since a miss there leaves the judge's verdict standing.

Merge: safe now: it merges cleanly into current main, 10 commits behind, and its tests pass on the merge (test_novelty_checks.py, test_novelty_provenance.py, test_reference_checks.py, test_cli_surface.py: 247 passed). Once it is in, #489, #588, #627, #633 and #482 conflict with it (docs/README.md, mkdocs.yml, cli_flags_allowlist.txt, novelty/probes.py) and need a rebase.

What I ran, at f1d9202:

• make ci-local --offline: 19,011 passed, 0 failed, 72 skipped; every row passes.
• The capped-run cases: a refused reply that names a verdict and one that names none, on a vetoed answer.

Checked, no issues

• Capped runs: _capped_grade_run runs the checks either way. A vetoed answer is wrong_other, and ungraded still counts a reply with no verdict, so analyze and probe report the stop.
• Docs: the known-limit paragraph in grading-tiers.md names the :, No TUI needed: and without the TUI cases, as the code behaves.

[xaleryb]

• Ran skill novelty ground at 091e01c on atoms files that carry a malformed meta.grading.
• {"checks": null}, {"checks": ["x"]} or "x" as an atom's grading -> exit 5 with TypeError/AttributeError from add_checks, or from verify_keys when the atom does not name the tool. Expected: exit 2 with atom <id>: invalid grading: ..., which is what skill novelty probe gives for the same file (probes.atom_grading).
• Pushed 620dc9c (091e01c..620dc9c): every atom's grading is validated with validate_grading before add_checks and verify_keys run; tests cover both steps and the command, for a named and an unnamed tool. Full suite, ruff, mypy, bandit and docs check --strict pass with it.
• Static checks fails at check-added-large-files on docs/reference/cli-interface.json (504 KB here; main is at 501 KB against the 500 KB limit, fixed by fix(cli): the CLI snapshot writes one line per parameter #737). The snapshot may need regenerating after fix(cli): the CLI snapshot writes one line per parameter #737 lands.

[xaleryb]

Pushed 1ea8665 (620dc9c..1ea8665): merges main and regenerates docs/reference/cli-interface.json with make cli-snapshot on Python 3.13. The text merge applied cleanly but left the new skill novelty ground block in the old multi-line layout; the regenerated file is 294 KB and passes the large-file hook. On the merged tree the full suite (19830 passed), ruff, mypy, bandit, docs check --strict and the "CLI snapshot is current" step pass; cli_surface.py diff against main lists only the new skill novelty ground command.