[napetrov]

Make Dokimi’s capabilities easier to discover and understand: start with a concise README, explore a linked feature tree, and open individual feature cards for details.

What changes

• Add 48 illustrated feature cards across eight topic guides.
• Give each feature a description, implementation status, limitations, and links to detailed documentation.
• Add a feature tree connecting the overview to individual cards.
• Shorten the root README and update the package README, documentation home, and MkDocs navigation.
• Correct outdated descriptions of repository evaluation, task generation, coverage, and adherence.
• Extend documentation link checks to cover nested feature pages and the root README.

Feature availability

The guide distinguishes functionality available on main, partial implementations, open proposals, and research directions. Availability was reviewed against the code and open feature PRs on November 11, 2026.

“Available on main” does not imply inclusion in a published release. Illustrations explain behavior; they are not product screenshots or measured results.

Validation

• Strict MkDocs build passed.
• All 67 documentation checks passed.
• Verified 48 feature anchors, 48 image assets, and all nine new feature pages.
• Confirmed that a broken link in a nested feature page fails the link checks.

The full test run finished with 18,554 passed, 18 skipped, and 10 failed. All 10 failures also reproduce on the unchanged base in the same environment.

Full local CI remains non-green due to those failures and additional environment/tooling failures in package and smoke-test steps. Keeping this PR as a draft pending CI review.

No product implementation changes.

[exai-bot]

OpenCodeReview: Commit f9a7a17 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `f9a7a17`: 10 still running, 8 passing, 2 skipped

Check	Status
CodeQL (python)	still running
External catalog smoke test (uxlfoundation/skills)	still running
Linux Python 3.11	still running
Linux Python 3.14	still running
Linux Wheels Python 3.11	still running
Linux Wheels Python 3.14	still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com...	still running
benchmark	still running
mypy	still running
self / repo validate	still running

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (f0cd431): no findings</summary>

Open threads from the bot: 0

</details>

[Copilot]

🟡 Changes recommended

Documentation still mislabels knowledge retention as task value and conflates answer-grading red-team checks with adversarial controls for executable-task verifiers.

2 open issues

• Clarify which verifier red-team workflows are available · New
• Label the value axis as a retention proxy · New

What changed in this PR

2 new findings.

This PR adds a question-based capability map for Dokimi’s agent-context evaluation workflows, distinguishing implemented features from partial workflows and planned work.

Changes:

• Adds 48 feature cards across eight topics, with illustrations, availability boundaries and guide links.
• Updates README entry points, measurement guidance and repository-evaluation status.
• Adds feature navigation and expands link checking to nested documentation and both READMEs.

File	Description
README.md	Introduces question-based navigation and free starting checks.
dokimi/​tests/​test_docs_links.py	Checks nested documentation and both READMEs.
dokimi/​README.md	Links the capability map and updates implementation status.
dokimi/​mkdocs.yml	Adds feature pages to site navigation.
dokimi/​docs/​what-it-measures.md	Links the map and clarifies novelty interpretation.
dokimi/​docs/​user-journeys/​journeys.md	Updates task-generation, coverage and adherence boundaries.
dokimi/​docs/​repo-evaluation-contract.md	Identifies the implemented repository-evaluation core.
dokimi/​docs/​README.md	Streamlines the documentation home around capabilities and evidence.
dokimi/​docs/​features/​traces.md	Documents context inspection and execution diagnostics.
dokimi/​docs/​features/​skills.md	Covers skill analysis, shortening and validation.
dokimi/​docs/​features/​repositories.md	Covers repository guidance and evaluation workflows.
dokimi/​docs/​features/​README.md	Indexes all 48 cards and defines availability labels.
dokimi/​docs/​features/​judging.md	Explains grading methods and judge checks.
dokimi/​docs/​features/​experiments.md	Covers controlled comparisons, harnesses and cost.
dokimi/​docs/​features/​eval-design.md	Covers task creation, coverage and grader controls.
dokimi/​docs/​features/​discovery.md	Covers skill selection and instruction loading.
dokimi/​docs/​features/​continuous-evaluation.md	Covers reports, regression checks and development directions.

_{Review effort: Balanced}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (f12057b) to head (f9a7a17).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #627   +/-   ##
=======================================
  Coverage   95.71%   95.72%           
=======================================
  Files         415      415           
  Lines       55346    55362   +16     
  Branches     9514     9516    +2     
=======================================
+ Hits        52977    52993   +16     
  Misses       1466     1466           
  Partials      903      903           

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

Ran: the start guides followed on Linux with dokimi 0.2.0 from PyPI and with this branch; the cards checked against the code at main; mkdocs build --strict, the docs link tests and the docs check gate (passes, score 0.6256 before this commit, 0.6267 after).

Pushed f0cd431:

• docs/quickstart.md, docs/start/README.md: the seed example's --out-dir fails on 0.2.0 (unrecognized arguments), and 0.2.0 is what pip install dokimi installs. Both pages now give the 0.2.0 spelling, and development/interfaces.md gains a row for the merged output-flag rename.
• AGENTS.md, CONTRIBUTING.md, docs/features/README.md, docs/start/contributors.md: they told contributors to link a new page from docs/README.md. The index test now reads documentation-index.md, so a contributor who followed them turned test_the_index_links_every_doc_in_the_directory red.
• docs/features/traces.md card 37 had the candidates reversed: status_of makes a rule the sessions never brought up the candidate, and a rule they triggered or acted on in_use.
• Wrong behaviour statements: traces cards 35/36 (input is stream-json output, and without --outcomes a pass only means the harness exited cleanly); discovery card 14 (novelty routing scores only should-select queries, lexically without --model; negatives and real selection are catalog trigger-probe); skills cards 10 (the routing axis is always 1.0), 11 (--ignore takes finding codes only) and 46 (per-atom relevance for one skill); eval-design cards 26 (file/line F1, no grader) and 30 (eval check needs a repo manifest and refuses Harbor directories); experiments card 04 (tasks run takes one --model); continuous-evaluation card 40 (the reusable workflow runs validate, scan and eval check only); repositories card 24 input; architecture.md (the contract is no longer marked PROPOSED); start/experiments.md (arms run and tasks run have no cap); development/limitations.md (on main atoms and probe keep capped work).
• Commands that fail or mislead as written: quickstart step 6 (dashboard generate reads results/ by default and exits 2 after step 4; now --results-dir .); quickstart content setup (onetbb ships with the package and preflight from an empty directory fails only on the keys; --questions-root is named); start/README.md (linked guides still write python cli.py); start/repositories.md (the text output lists only fail claims; repo validate alone passes a manifest eval check refuses; --harness is named); start/skills.md (skill check for one skill); start/automation.md (the reusable repo-eval.yml workflow); features/README.md and docs/README.md (--version in a checkout prints the last release).
• Wrong links: dokimi/README.md, the PyPI readme, called docs/README.md the full explanation and the index; card links on 02, 07, 18, 21, 25, 26, 31, 32 and 33 pointed at pages or modules that do not cover the feature (methodology.md has nothing on headroom, the leak gate or judge validation); --plugins, --no-shorten and the dokimi-design-evals skill had no prose page left; documentation-index.md cited an unrelated PR (fix(judge): read a qualified or negated figure verdict #67); skill-novelty.md still said three free steps where the walkthrough says two.
• Statuses: discovery card 18 is Partial (catalog check already prints free merge/split hints); the flow map's J4 adherence edge now says what exists (declared process contracts).

Left for the author (images, not touched here):

• 30 shows a verifier challenge (redteam_print_pass.py) that the card says is not implemented.
• 31 shows a run_eval notebook that dokimi does not have.
• Crop artifacts: 25, 26, 35, 43 and 44 show the edge of a neighbouring frame or lose their own.
• 25 carries the GitHub mark and 27 an "Office" label; generic icons would avoid vendor marks.
• The alt texts repeat the heading and subtitle instead of describing the scene.

DCO fails: 49f1f17 ("docs: address PR #627 review threads and self-review findings") has no Signed-off-by line, so the PR cannot merge until that commit is signed off.

#472, #482, #489 and #588 add pages linked from docs/README.md; after this PR the index is docs/documentation-index.md, so they need an entry there.

[vchamarthi]

Review at f0cd431. 3 new findings: 1 must-fix, 2 nits. In the line comments. Reorganizes the docs into an illustrated feature tree of 39 cards and shortens the docs home. The docs link tests pass on the PR and it merges without conflict, but the eight topic guides under docs/features/ are missing from the documentation index that the docs say must link them, and the docs home drops sections while keeping their anchors. The PR text says 48 cards and 48 images; the tree holds 39 of each.

Verified

• Docs link tests: All 11 tests in dokimi/tests/test_docs_links.py pass on the PR head, including the relative-link and same-page anchor checks over the changed pages. The index test passes only because it covers top-level pages (see the first finding).
• Merge: The PR merges onto main with no conflict.
• Card count: The feature tree links 39 distinct cards (dokimi/docs/features/README.md), and dokimi/docs/assets/features/ holds 39 .webp images. dokimi/docs/README.md:32 and dokimi/docs/documentation-index.md:27 both say 39. The PR description's 48 cards and 48 images do not match the tree.

Not verified

• The availability labels on the 39 cards ("available on main", partial, proposed) were not traced to code feature by feature; only the commands and flags the cards cite were looked at: Cite the file or test that shows each "available on main" label on its card, or mark the label as the author's reading.

[napetrov]

@xaleryb two open questions before I change anything here:

1. Card count: the PR text says 48 cards and 48 images, but the tree links and ships 39 of each (vchamarthi's review). Should I (a) correct the description to 39, or (b) add the 9 missing cards and images?
2. Commit 49f1f17 has no Signed-off-by, so DCO blocks the PR. OK to re-sign it (history rewrite on this branch, lease push), or should the commit be squashed into a signed one?

[xaleryb]

@napetrov on your two questions:

1. (a): correct the description to 39. The tree, docs/README.md:32 and documentation-index.md:27 already say 39, and the 9 merged cards were the duplicate-image pairs, so adding cards back to reach 48 would only restore what the merge removed.
2. 49f1f17 is authored as Claude <noreply@anthropic.com>, and DCO checks the sign-off against the commit's author, so --signoff alone will not pass. Amend it with git commit --amend --reset-author -s (an interactive rebase that stops at that commit), replay the later commits and push with --force-with-lease. The repository merges by squash only, so the rewritten branch history does not reach main.

Re-checked f0cd431 merged with current main: the full suite, ruff, mypy, bandit and docs check --strict.

• Pushed 1c5c042 for vchamarthi's must-fix: documentation-index.md linked none of the eight docs/features/*.md guides that features/README.md:72 points to, and test_the_index_links_every_doc_in_the_directory checked only top-level pages, so it stayed green. The index now links the eight guides, the test walks every subfolder, and a new test checks that an unlinked page in a subfolder is reported.
• Left for the author: images 30 and 31, the crops on 25/26/35/43/44, the vendor marks on 25 and 27, and the alt texts, as listed earlier.

[xaleryb]

Pushed 2caa3ed (769b532..2caa3ed): merges main. Main had added one grading-tiers.md entry to the docs/README.md index (#472); since this PR moves the index to docs/documentation-index.md, that entry goes there, right after skill-novelty-evidence.md, and README.md keeps this PR's landing page. On the merged tree the full suite (19934 passed), ruff, mypy, bandit, docs check --strict, the CLI snapshot step and mkdocs build --strict pass. DCO is still red on 49f1f17, which only the author can re-sign.

[antonwolfy]

Several cards report as open or proposed work that is already merged, one of them names a command that no longer runs, and a .agent-eval task still points at the old docs index. Fix those before merge, together with the unsigned commit. The structure is sound. The feature tree, the topic guides and the nested link checks all work.

What I ran, on a Linux review host at 2caa3ed:

• make ci-local: 30 passed, 2 failed, 14 skipped. The two failures are the gitleaks download and the commit scan that needs it
• The new docs link tests, with 11 mutations (broken nested link, broken nested anchor, unindexed nested page, unlinked subfolder README, broken root README link): each one fails a test.
• mkdocs build --strict: passes, and every new page is in nav:.
• Every CLI command and flag the cards name, checked against --help.

Findings that cannot be anchored inline:

1. The docs-new-page task in .agent-eval still treats dokimi/docs/README.md as the docs index. .agent-eval/tasks/docs-new-page/check.py:26 reads docs/README.md for in_docs_index, and oracle.patch adds its line there. This PR turns that file into a start page and moves the index to documentation-index.md. git apply --check oracle.patch on the PR tree prints patch failed: dokimi/docs/README.md:249 and patch failed: dokimi/mkdocs.yml:113. An agent that follows the updated AGENTS.md and indexes its page in documentation-index.md is graded in_docs_index=False. Suggestion: read docs/documentation-index.md in check.py and regenerate the oracle patch.

2. The same stale-status problem runs through development/work-index.md and development/README.md. work-index.md:41,54,56,57,59 list #441, #590, #593, #594 and #596 as "Open PR", and development/README.md:44,66 call them open implementations. All five are merged into the base (b4d5471, d2b5a91, 60044a8, dcd4b18, f5cd927). Fix these together with the card text in the inline comments.

3. Commit 49f1f17 ("docs: address PR #627 review threads and self-review findings") has no Signed-off-by. Its author is Claude, so DCO fails and the merge waits for it. The other 17 non-merge commits are signed. @napetrov's question on the re-sign waits for the author.

4. The PR description says 48 cards and 48 images; the tree has 39 of each. git ls-files dokimi/docs/assets/features lists 39 images. The docs themselves say 39. Correct the description.

I agree with @vchamarthi's point on docs/README.md:46-49: the anchors #how-the-evaluation-flows and #why-this-matters still sit above a single paragraph with no section under them. It is a nit, not a blocker.

Checked, no issues

• Pins: the nested link checks. _unindexed handles #anchor suffixes, duplicates and ./ paths, and it does not exempt subfolder READMEs.
• Workflow: the new upload-artifact step uses the same pinned SHA as the other workflows and runs only on pull_request.
• Commands: the quickstart, the start guides and the card commands exist with the flags shown.
• Links: all 39 image references and every card anchor in the feature tree resolve.

[antonwolfy]

LGTM. Everything from my earlier review is fixed: the merged PRs are marked merged on the cards and development pages, the discovery card no longer names novelty routing, the docs-new-page check reads the new index, and every commit now carries a Signed-off-by. Four nits, two inline and two below.

What I ran, on a Linux review host at e993757:

• make ci-local: 33 passed, 0 failed, 14 skipped (the skips need Docker, network or another Python). Run tests and Coverage of changed lines pass; mkdocs build --strict passes.
• The docs link tests against 6 planted breakages (broken nested link, broken nested anchor, unindexed page in features/, unindexed page in start/, broken root README link, README.md missing from the skip set): each one fails a test. Replacing rglob with glob in SOURCES fails none (inline).
• Every command and flag the cards, the start guides and the quickstart name, against the CLI parser. Every PR link in features/ and development/ against the base's git log: each "merged" PR is there, each "open" one is not.
• oracle.patch of the docs-new-page task on the task's pinned base: applies, and the updated check.py grades it all true.

Nits not anchored inline:

1. Three images are orphaned. dokimi/docs/assets/readme/annotated-skill.png, shortening-strategies.png and three-questions.png were only referenced from the two READMEs this PR rewrites (about 527 KB, still copied into the site). git grep for the three names at the head finds nothing. Delete them or reuse them.
2. The PR description is stale. It says 48 cards and 48 images; the tree has 39 of each (git ls-files dokimi/docs/assets/features lists 39), as the docs themselves say.

Checked, no issues

• Pins: .github/workflows/docs.yml uses the same upload-artifact SHA as the other workflows, runs only on pull_request, and zizmor reports nothing.
• Index rule: AGENTS.md, CONTRIBUTING.md and start/contributors.md name documentation-index.md, which is what _unindexed checks; every new page is in nav:.
• Links: all 39 card images and anchors resolve.

[vchamarthi]

Pushed 1 commit to docs/feature-cards-and-map (3cf74da..f9a7a17).

• features/skills.md states novelty view colours by span and category only.
• features/skills.md states novelty view labels spans by structure only, without outcome or category colours.
  Run: the full suite with main merged in, 20491 passed.

The PR description needs these corrections:

• "Add 48 illustrated feature cards across eight topic guides." should read: Add 39 illustrated feature cards across eight topic guides.
• "Verified 48 feature anchors, 48 image assets, and all nine new feature pages." should read: Verified 39 feature anchors, 39 image assets, and all nine new feature pages.

[antonwolfy]

Pushed two commits for my nits: 3cf74da pins the widened SOURCES in tests/test_docs_links.py, and 1c62e3f rewords "Everything above" in how-evaluation-works.md. I resolved both threads. I also read f9a7a17 (the novelty view wording in features/skills.md and its test): it does what its message says and I found no problem in it. I am leaving this as a comment without an approval, since the other reviewers' review is still pending.