[antonwolfy]

Summary

Issue #679: the novelty docs, the --competitors help and the note written into every lexical routing result present "every dpnp-* skill scored 0.0 recall@1 against its siblings" as a measured finding. The queries behind it were drawn from the skills' section headings, and the 8 dpnp skills share one template ("Quick Start", "Implementation Guide", "Gotchas & Limitations"), so every skill's queries are identical across the family and every skill gets the same rank. The figure measures the template, not the descriptions.

On main, every lexical result carries "note": "A floor, and free. ... 0.0 for every dpnp-* sibling: descriptions discriminate a domain, not a sibling.", and catalog routing with a --queries-by-skill file that lists the same queries for each sibling runs without a word about it.

Closes #679

Changes

• Reworded the four sites, one sentence each, to keep merge conflicts small with the open PRs that edit those files: docs/skill-novelty.md ("Competitors are mandatory"), docs/skill-novelty-walkthrough.md, the --competitors help in commands/routing_parser.py, and the lexical note plus the module docstring in novelty/routing.py. They now say that siblings built from one template have identical heading queries, so heading-query recall at the family level measures the template, and that sibling confusion is still a failure mode to check with queries written independently of the skills. No new number is cited, and the advice to keep the competitors in the field stays.
• New novelty/query_overlap.py: shared_queries_warning counts a query as shared when the lexical scorer reads it as the same content words (tokenize(): case, spacing, punctuation and word order aside, so Quick Start, Quick Start. and start quick are one query while Straße and STRASSE are two) as a query of another skill, and returns a warning when at least SHARED_QUERY_WARN_SHARE = 0.5 of the given queries are shared. warn_shared_queries in commands/novelty_routing_report.py prints it on stderr, from drop_queries_left_out, before the --estimate-only return and before any model call. --queries-from-headings is refused under --all-vs-all and has no multi-skill mode, so the check applies to the --queries-by-skill file, whatever made it. main moved novelty routing to skill routing and catalog routing; the all-vs-all form is now catalog routing ROOT --queries-by-skill, and the --competitors help rewording lives in its new parser.
• Tests in a new file, tests/test_novelty_routing_shared_queries.py: both sides of the threshold (2 of 4 warns, 2 of 6 does not), case and spacing, one skill or no queries, the stderr line, and the note.
• The catalog routing result records shared_queries ({"shared": 9, "total": 9}) next to n_queries, counted by shared_query_counts, the function the warning uses (queries with no content word are in neither count). It is always recorded, below the threshold too. The artifact has no schema, so no version changes.
• Changelog fragments: Fixed (the command catalog routing as it is at this head) and Added (shared_queries).

Risk

The note text and two help and doc sentences change. catalog routing can print one more stderr line and its saved result gains the shared_queries key; the exit code is unchanged, and files saved earlier lack the key. A run whose skills legitimately share half or more of their queries now gets the warning.

Testing

On a Linux review host, at d206c17 (on main 1bb8978):

• make ci-local: 33 passed, 0 failed, 14 skipped; the full test run passed (20357 passed, 19 skipped). The skips are steps that need docker, a network, micromamba or semgrep, matrix legs fed by other jobs, and jobs that do not run on a PR.
• Mutants: the new tests fail when the threshold < becomes <=, when the case and spacing normalisation is dropped, and when the old note text is restored. test_all_vs_all_result_records_the_shared_query_counts and test_a_saved_all_vs_all_file_carries_the_shared_query_counts fail when the shared_queries entry is removed from the all_vs_all result; test_all_vs_all_result_records_the_shared_query_counts and test_queries_with_no_content_word_are_not_counted fail when queries with no content word are counted. The tests import query_overlap, which does not exist on main, so they cannot pass there. The fix for the shared-query key adds test_trailing_punctuation_does_not_hide_a_shared_query and test_queries_the_scorer_tokenizes_differently_are_not_shared; both fail when the key is the casefolded, space-collapsed text. test_word_order_does_not_hide_a_shared_query fails when the key keeps the token order, and test_a_repeated_word_keeps_a_query_apart_from_the_single_word fails when the key drops a repeated word (a set of tokens). With the warning call and the shared_queries entry removed, the six tests that run catalog routing or read the result fail.
• Not run: the other Python versions (3.11, 3.12, 3.14).

Revert

A plain revert restores the old wording and drops the warning and the shared_queries key. Result files already saved keep that key, which nothing reads.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

CI at d206c17: 25 passing, 3 skipped.

Review details

• Commit reviewed: d206c17 against merge base 2e0f942
• Coverage: 3 files in 1 groups
• Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
• Usage: 63k tokens (36k from cache), 18s
• Tool calls: 9 (7 file_read, 1 code_search, 1 local_check_results)
• small change set: 3 files

[vlad-perevezentsev]

Review at 0ce0e09. Removing the dpnp 0.0 claim from the docs, the help and the lexical note is right. The new warning needs work: its text overstates the problem, it comes too late for --model runs, and it is lost in the saved result. Details are in the line comments.

Agreed with @exai-bot: the "same rank" text in the warning and the changelog is wrong.

What I ran, at 0ce0e09:

• make ci-local --offline: 19,019 passed, 0 failed, 72 skipped. schema_compat fails on novelty.v1.json, but that comes from main (#633 widened it after this branch's base). This PR does not touch schemas; a rebase clears it.
• A repro of the shared-query warning against all_vs_all: shared queries give own_rate 1.0 / 0.0, not equal ranks; contentless queries are counted in the share.

Checked, no issues

• Docs and help: every place that quoted the dpnp 0.0 (skill-novelty.md, the walkthrough, --competitors help, the routing.py docstring and note) is changed. The 0.225 figure is kept, with a caveat that it measures the template for a family.
• New file and tests: the SPDX header is there, and the new tests fail on the base, since query_overlap does not exist there.

[vchamarthi]

Pushed 3 commits to docs/heading-query-artifact (0ce0e09..bd594a7).

• The warning no longer says every skill gets the same rank.
• Queries with no content word are not counted as shared or in the total.
• The shared-query warning is printed before the --estimate-only return.
• The --all-vs-all call site is pinned by tests.

Threads: 3 fixed, 0 answered as not a defect.
Run: the full suite with main merged in, 19150 passed.

Questions for @antonwolfy

1. Should the saved all-vs-all result record the shared-query counts, as result["shared_queries"] = {"shared": 9, "total": 9}? (dokimi/dokimi/commands/novelty_routing_report.py:658)
The warning goes to stderr only, so a JSON saved with --out from a fully shared run reads as a normal own_rate table. A new key in the result is an output-format choice.

• A: Add shared_queries to the result: the file carries the counts.
• B: Keep stderr only: no format change.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.71%. Comparing base (1bb8978) to head (d206c17).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #712    +/-   ##
========================================
  Coverage   95.71%   95.71%            
========================================
  Files         413      415     +2     
  Lines       55236    55339   +103     
  Branches     9501     9514    +13     
========================================
+ Hits        52867    52970   +103     
  Misses       1466     1466            
  Partials      903      903            

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/novelty_routing_report.py	98.18% <100.00%> (+0.02%)	⬆️
dokimi/dokimi/commands/routing_parser.py	100.00% <ø> (ø)
dokimi/dokimi/novelty/all_vs_all.py	99.32% <100.00%> (+<0.01%)	⬆️
dokimi/dokimi/novelty/query_overlap.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/routing.py	96.55% <ø> (ø)

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

(commented) Review at 0ce0e09. Removing the dpnp 0.0 claim from the docs, the help and the lexical note is

@vlad-perevezentsev (#712 (review)): The review's points are its threads 4565021052, 4565021613, 4563358011 and 4565022120, answered above. The schema_compat failure on novelty.v1.json came from main and this PR does not touch schemas; make ci-local at 793118c on main 8e48ed7 gives 32 passed, 0 failed, 13 skipped.

[antonwolfy]

Pushed 3 commits to docs/heading-query-artifact (0ce0e09..bd594a7).

@vchamarthi (#712 (comment)): Question 1 is answered with option A in 793118c: the saved all-vs-all result records shared_queries ({"shared": 9, "total": 9}). It needs no schema or version change, since the routing result has no schema, and files saved earlier lack the key. See thread 4563358011 for the reasons and the tests.

[napetrov]

Change class: not computed (no receipt profile for this repo, or receipt.py failed)

Verdict: comment. One question and one nit, no blocking. Head 793118c, merge base 8e48ed7, lane L2.
Ran: targeted probes on head with the shared-query counter and the CLI on fixtures (base vs head). The new tests pass on head per the pass (219 with the related routing tests). Full pytest was not run here: regex and litellm are missing in this sandbox.

CI: 0 failing Actions jobs on head. The merge base is behind base tip (a stale base, not a defect).

Raised before:

• @exai-bot T1 (query_overlap.py, the warning's rank claim): fixed on head (bd594a7 reworded it).
• @vlad-perevezentsev T2 (warning after the run): fixed on head; the warning now prints before the --estimate-only return.
• @vlad-perevezentsev T3 (saved JSON lacks the share): fixed on head; shared_queries is in the saved result.
• @vlad-perevezentsev T4 (contentless queries counted): fixed on head (bd594a7).
• @vlad-perevezentsev R2 (review at 0ce0e09): answered by the follow-up commits.

Coverage: checked the counter on head (exact-text key, casefold key, empty and contentless inputs), the warning and saved-key wiring, the estimate-only path. Not looked at: Python 3.11, 3.12, 3.14; make ci-local; the docs wording beyond the four sites.

Design: none. Follow-up issues: none.

[xaleryb]

• Ran: the full suite, ruff, mypy, bandit, the growth and comment-block checks, the docs benchmark gate and the CLI snapshot on main merged with 793118c. Also ran differential CLI runs against main on template and independent catalogs, and a 10-mutant pass on the new share counting.
• No must-fix or mid findings. All earlier points (warning wording, warning under --estimate-only, shared_queries in the saved result, contentless queries) re-run fixed at the head.
• Left for the author: napetrov's open question on punctuation variants (Quick Start vs Quick Start. count as different queries). This matches the documented 'case and spacing aside' rule. Recommendation: keep it and reply in the thread so it can be resolved.
• The branch now conflicts with main after #472 and #595 landed; a merge of main is needed before it can land.

[antonwolfy]

(commented) Change class: not computed (no receipt profile for this repo, or receipt.py failed)

@napetrov (#712 (review)): The review lists its own threads (T1 to T4, R2) as fixed and asks for no change; I re-ran the shared-query tests on the merged head. Its one question and one nit are the two threads above, both fixed in 2e0f942.

[antonwolfy]

(approved) - Ran: the full suite, ruff, mypy, bandit, the growth and comment-block checks, the docs benchmark gate and the CLI snapshot on main merged with 793118c. Also ran differential CLI runs agai

@xaleryb (#712 (review)): The approval lists its own checks and two open items, both handled in their own concerns. The punctuation question is settled in its thread by keying on the scorer's tokens (2e0f942), so Quick Start and Quick Start. are now one query instead of the kept behaviour the review suggested. The conflict with main is resolved in merge commit 1347abd: main moved novelty routing to skill routing and catalog routing, so the parser conflict took main's removal and the --competitors help rewording moved into commands/routing_parser.py.