Skip to content

fix(validate): stop reading capability words as description-no-when cues #743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xaleryb wants to merge 6 commits into
base: main
Choose a base branch
from
+93 −5
Open

fix(validate): stop reading capability words as description-no-when cues #743

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1ba06a4
fix(validate): count the ja いつ cue only before a use verb
xaleryb Nov 13, 2026
0b9fe76
fix(validate): do not read 耗时, 超时 or 延时 as a zh when cue
xaleryb Nov 13, 2026
cc8e3f7
fix(validate): count de beim only next to a use verb
xaleryb Nov 13, 2026
fee007b
docs(external): say which capability words are not when cues
xaleryb Nov 14, 2026
96705f7
Merge branch 'main' into fix/validate-when-cue-followups
xaleryb Nov 14, 2026
927c4ec
fix(validate): keep when cues for de beim, zh 超时 and ja いつ
vchamarthi Nov 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions dokimi/changelog.d/20261113_120000_xaleryb_when-cue-followups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
### Fixed

- **`description-no-when` no longer takes a capability word for a "when" cue** — a description with no
"when" clause lost the `warn` when it held Japanese `思いつく`/`追いつく`/`こいつ` (read as `いつ`), Chinese
`耗时调用`/`超时调用`/`延时执行`/`当事人`, or German `Hilft beim …`/`Unterstützt beim …`. Now `いつ` counts only
before a use verb in the same sentence, `时` after `耗`/`超`/`延` is no cue unless a comma or `请` and a use verb
follow, and `beim` counts only with `verwenden`, `nutzen`, `benutzen`, `einsetzen`, `aufrufen` (also `verwende`,
`nutze`, `benutze`, `ruf … auf`, `setz … ein`) in the same sentence, in either order. Run `catalog check` again to see the warnings that come back.
4 changes: 3 additions & 1 deletion dokimi/docs/external-content.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,9 @@ skill, since those commands then run with no prompt, and that pair fails `--stri
half and shares three words (`description-near-duplicate`). The `description-no-when` cue is read in English,
Spanish, French, German, Portuguese, Russian, Chinese, Japanese and Korean, and a description in a script none
of them covers (Arabic, Thai, Hebrew, …) is not checked: it gets a `note` (`description-when-not-checked`), so not checked does not read as a pass: its row says `note:` instead of `ok`, and the counts line `1 ok (1 not checked)`. A Latin-script language without a list of its own
(Polish, Dutch, Turkish, Vietnamese) is checked against these cues. Two skills in different paths with the same
(Polish, Dutch, Turkish, Vietnamese) is checked against these cues. A word that only states a capability is not a cue:
Japanese いつ counts only before a use verb, Chinese 耗时, 超时 and 延时 are not 时 + verb, and German `beim`
counts only next to a use verb (verwenden, nutzen, ...). Two skills in different paths with the same
directory name and description but different `SKILL.md` text are a `warn` on both
(`mirror-drift`), naming the other copies. Skills with
`disable-model-invocation: true` are exempt from the description checks.
Expand Down
19 changes: 15 additions & 4 deletions dokimi/dokimi/external/when_cues.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,17 @@
import re
from typing import Optional

# German `beim` is a cue only in a sentence with a use verb, in either order; a full stop that is
# followed by a capitalised word or an acronym ends the sentence, so `z. B.` does not.
_DE_S = r"(?:[^.;!?]|\.(?!\s*(?-i:[A-ZÄÖÜ][A-Za-zäöüß])))*?"

@exai-bot exai-bot Nov 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — German sentence-boundary detection produces incorrect validation verdicts. Trigger: Beim Deployen mit z. B. GitHub Actions verwende diesen Skill. is rejected, while Nutze das Tool. „Hilft beim Schreiben“ beschreibt seine Funktion. is accepted. Evidence: the negative lookahead blocks the period after B because GitHub starts with an uppercase letter followed by another letter, but permits the actual sentence-ending period before „Hilft because the quote bypasses that check. Neither example has another matching cue. Impact: catalog check emits description-no-when for the valid instruction and suppresses it for the capability-only description. Handle abbreviations and quoted sentence starts explicitly, and add both regression cases (including warning assertions). Workaround: use an explicit wenn clause for instructions. Severity: medium, because this is a concrete but bounded validation-verdict failure.

_DE_USE = r"\b(?:verwende[nst]?|nutze[nst]?|benutze[nst]?|einsetzen|aufrufen)\b"
_DE_BEIM = (
rf"{_DE_USE}{_DE_S}\bbeim\b|\bbeim\b{_DE_S}{_DE_USE}|"
rf"\bruf(?:e|en|t)?\b{_DE_S}\bbeim\b{_DE_S}\bauf\b|\bbeim\b{_DE_S}\bruf(?:e|en|t)?\b{_DE_S}\bauf\b|"
rf"\bsetz(?:e|en|t)?\b{_DE_S}\bbeim\b{_DE_S}\bein\b|\bbeim\b{_DE_S}\bsetz(?:e|en|t)?\b{_DE_S}\bein\b|"
rf"\bruf(?:e|en|t)?\b{_DE_S}\bauf\b{_DE_S}\bbeim\b|\bsetz(?:e|en|t)?\b{_DE_S}\bein\b{_DE_S}\bbeim\b"
)

# A "when" clause or use phrase per language: the model chooses a skill from its description alone.
# Latin cues take word boundaries, CJK cues are substrings; bare 使用 (use) or 用于 (for) are too broad.
WHEN_CUES = {
Expand All @@ -19,14 +30,14 @@
r"whenever\b)",
"es": r"(?<!vez en )\bcuando\b|\bcuándo\b|\bsiempre que\b|\bcada vez que\b|\ben caso de\b|\bal (?:usar|utilizar|invocar|ejecutar)\b",
"fr": r"\bquand\b(?!\s+même)|\blorsqu\w*|\bdès que\b|\bchaque fois que\b|\ben cas de\b|\blors (?:de|d')",
"de": r"\bwenn\b|\bbeim\b|\bsobald\b|\bimmer dann\b",
"de": r"\bwenn\b|" + _DE_BEIM + r"|\bsobald\b|\bimmer dann\b",
"pt": r"(?<!vez em )\bquando\b|\bsempre que\b|\bcada vez que\b|\bem caso de\b|\bao (?:usar|utilizar|invocar|executar)\b",
"ru": r"\bкогда\b(?![-‑])|\bкогато\b|\bколи\b|\bвсякий раз\b|\bкаждый раз\b|"
r"\bпри (?:работе|использовании|необходимости|запросе|наличии|вызове|запуске|получении)\b",
"zh": r"(?<!任)何时|(?<!任)何時|什么时候|什麼時候|的时候|的時候|(?<![相适適恰应應妥得充担擔])[当當](?![前然即作做成地天日晚年初今中])[^。；！？;!?]{1,80}?(?<![实實同隨随及定按即暂暫不准準])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)[时時](?![间間区區])|(?<![实實同隨随及定按即暂暫不准準])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)[时時][，,]?(?:请|請)?(?:使用|调用|調用|触发|觸發|启用|啟用|激活|运行|運行|执行|執行)|"
r"适用于[^。]{0,20}?(?:场景|场合|情况|(?<![实實同隨随及定按即暂暫不准準])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)时(?![间間段区區序代]))|適用於[^。]{0,20}?(?:場景|場合|情況|(?<![實実同隨随及定按即暫暂不準准])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)時(?![間间段區区序代]))|(?:适用|適用|使用)(?:场景|場景)[:：]|使用时机|使用時機|触发条件|觸發條件|用[户戶](?:[说說](?![明的])|提到(?!的)|要求(?!的)|[请請]求(?!的|数据|數據|时间|時間|参数|參數|内容|內容|日志|日誌|头|頭|量))",
"zh": r"(?<!任)何时|(?<!任)何時|什么时候|什麼時候|的时候|的時候|(?<![相适適恰应應妥得充担擔])[当當](?![前然即作做成地天日晚年初今中])(?!事人[^。；！？;!?时時]*[耗超延用][时時][^。；！？;!?时時]*(?:[。；！？;!?]|$))[^。；！？;!?]{1,80}?(?<![实實同隨随及定按即暂暫不准準])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)[时時](?![间間区區])|(?<![实實同隨随及定按即暂暫不准準耗超延])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)[时時][，,]?(?:请|請)?(?:使用|调用|調用|触发|觸發|启用|啟用|激活|运行|運行|执行|執行)|[耗超延][时時](?:[，,]\s*(?:请|請)?|[请請])(?:使用|调用|調用|触发|觸發|启用|啟用|激活|运行|運行|执行|執行)|"

@exai-bot exai-bot Nov 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — the new 当事人 exclusion stops working when a sentence contains two duration words. Trigger: 整理当事人信息，统计查询耗时和超时次数。 has no when clause. Evidence: both character classes in the negative lookahead exclude 时, so it cannot reach the sentence end past the second 时; the subsequent 当…时 alternative consequently accepts 当事人信息，统计查询耗时. Impact: when_clause_found returns True and _description_findings suppresses description-no-when, retaining the false pass this change promises to remove. Exclude capability-only closing times without requiring exactly one 时 in the sentence, and add this case to NO_WHEN while preserving 当事人签字时. Workaround: omit 当事人 or split the capability list into separate sentences. Medium severity: a bounded but incorrect catalog validation verdict.

r"适用于[^。]{0,20}?(?:场景|场合|情况|(?<![实實同隨随及定按即暂暫不准準耗超延])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)时(?![间間段区區序代]))|適用於[^。]{0,20}?(?:場景|場合|情況|(?<![實実同隨随及定按即暫暂不準准耗超延])(?<!(?<![水])平)(?<!(?<![没沒])有)(?<!(?<![合来來])[适適临臨])(?<!(?<![较較很过過太更最])小)時(?![間间段區区序代]))|(?:适用|適用|使用)(?:场景|場景)[:：]|使用时机|使用時機|触发条件|觸發條件|用[户戶](?:[说說](?![明的])|提到(?!的)|要求(?!的)|[请請]求(?!的|数据|數據|时间|時間|参数|參數|内容|內容|日志|日誌|头|頭|量))",
"ja": r"(?<!ひ)(?:とき(?!どき|おり|たま|には)|(?<=[^\s、。，,.!?はもがでにをとやか])ときには)|(?<![同当一平瞬即随適臨戦日])(?<!(?<![一-鿿])定)(?<!(?<![一-鿿])常)(?:時に(?!は)|(?<!ある)時は|(?<=[^\s、。，,.!?はもがでにをとやか])時には)|(?<!ほとんどの)(?<!多くの)(?<!大半の)場合(?!によ|分け|の数)|(?:場面|シーン)で(?:使う|起動|発動)|使(?:う|用)タイミング|トリガー[:：]|(?:など|」|』|）)で(?:起動|発動)|"
r"(?<![実国交])際(?:に|は)|いつ(?!まで|も|でも|か|の間)",
r"(?<![実国交])際(?:に|は)|(?<![こそあど思追食つ])いつ(?!まで|も|でも|か|の間)(?=[^。！？!?]{0,20}?(?:使|呼び出|起動|発動|実行|参照|用))",
"ko": r"(?:(?<=할 )|(?<=하는 )|(?<=한 )|(?<=될 )|(?<=되는 )|(?<=된 )|(?<=받을 )|(?<=받는 )|(?<=받은 )|(?<=인 )|(?<=일 )|(?<=있는 )|(?<=있을 )|(?<=없는 )|(?<=없을 ))경우"
r"|(?<![그한이])(?<!그 )(?<!이 )때(?![문로때])|언제(?!나)|상황에서|상황이면|트리거(?:\s*(?:키워드|조건|문구|단어))?\s*[:：]|트리거(?:가|는)?\s*(?:되면|될 때|되었을 때)|사용 시점|사용 시기|(?<=[가-힣])(?<!(?<![가-힣])매)(?<!현대)(?<!고전)(?<!서울)(?<!부산)(?<!대구)(?<!인천)(?<!광주)(?<!대전)(?<!울산)(?<!세종) 시(?=[\s에,.:]|$)",
}
Expand Down
67 changes: 67 additions & 0 deletions dokimi/tests/test_description_when_cues.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,47 @@
pytest.param("当仓库中没有时自动创建目录。", id="zh-dang-closing-meiyou"),
pytest.param("当新版本来临时自动升级。", id="zh-dang-closing-lailin"),
pytest.param("배포 스크립트입니다. 필요한 때 사용합니다.", id="ko-piryohan-ttae"),
pytest.param("いつ使うか：リリース前。", id="ja-itsu-tsukau"),
pytest.param("当请求超时，自动重试并记录日志。", id="zh-dang-chaoshi"),
pytest.param("当事件到达时，发送通知。", id="zh-dang-shijian"),
pytest.param("Beim Erstellen eines Release verwenden.", id="de-beim-verwenden"),
pytest.param("Verwenden Sie diesen Skill zur Prüfung und Aktualisierung der Dokumentation beim Erstellen eines Release.", id="de-beim-long-gap"),
pytest.param("Beim Erstellen eines Release verwende diesen Skill.", id="de-beim-first-verwende"),
pytest.param("Beim Erstellen eines Release: nutze diesen Skill.", id="de-beim-first-nutze"),
pytest.param("Beim Erstellen eines Release benutze diesen Skill.", id="de-beim-first-benutze"),
pytest.param("Verwende diesen Skill z. B. beim Erstellen eines Release.", id="de-beim-abbreviation"),
pytest.param("Nutze diesen Skill beim Deployen.", id="de-nutze-beim"),
pytest.param("Benutze diesen Skill beim Deployen.", id="de-benutze-beim"),
pytest.param("Sie können diesen Skill beim Deployen nutzen.", id="de-nutzen-beim"),
pytest.param("Sie können diesen Skill beim Deployen benutzen.", id="de-benutzen-beim"),
pytest.param("Diesen Skill beim Deployen einsetzen.", id="de-einsetzen-beim"),
pytest.param("Beim Deployen diesen Skill einsetzen.", id="de-beim-einsetzen"),
pytest.param("Diesen Skill beim Deployen aufrufen.", id="de-aufrufen-beim"),
pytest.param("Beim Deployen diesen Skill aufrufen.", id="de-beim-aufrufen"),
pytest.param("Beim Deployen nutzen Sie diesen Skill.", id="de-beim-nutzen"),
pytest.param("Beim Deployen benutzen Sie diesen Skill.", id="de-beim-benutzen"),
pytest.param("Beim Deployen verwenden Sie diesen Skill.", id="de-beim-verwenden-sie"),
pytest.param("Rufe diesen Skill beim Deployen auf.", id="de-ruf-beim-auf"),
pytest.param("Setze diesen Skill beim Deployen ein.", id="de-setz-beim-ein"),
pytest.param("Beim Deployen rufe diesen Skill auf.", id="de-beim-ruf-auf"),
pytest.param("Beim Deployen setze diesen Skill ein.", id="de-beim-setz-ein"),
pytest.param("Rufe diesen Skill auf, z. B. beim Deployen.", id="de-ruf-auf-beim"),
pytest.param("Setze diesen Skill ein, z. B. beim Deployen.", id="de-setz-ein-beim"),
pytest.param("请求超时，请使用本技能重试。", id="zh-chaoshi-comma-use"),
pytest.param("超时，请使用本技能重试。", id="zh-chaoshi-comma-use-bare"),
pytest.param("延时，请使用此技能。", id="zh-yanshi-comma-use"),
pytest.param("遇到超时请使用本技能。", id="zh-chaoshi-qing-use"),
pytest.param("遇到耗时,调用本技能。", id="zh-haoshi-comma-use"),
pytest.param("当事人签字时，生成归档报告。", id="zh-dangshiren-shi"),
pytest.param("いつこのスキルを使うか：リリース前。", id="ja-itsu-object-tsukau"),
pytest.param("いつ、このスキルを使うか。", id="ja-itsu-comma-tsukau"),
pytest.param("いつこのスキルを呼び出すか。", id="ja-itsu-yobidasu"),
pytest.param("いつこのスキルを起動する？", id="ja-itsu-kidou"),
pytest.param("いつこのスキルを発動する？", id="ja-itsu-hatsudou"),
pytest.param("いつこのスキルを実行する？", id="ja-itsu-jikkou"),
pytest.param("いつ参照すべきか", id="ja-itsu-sanshou"),
pytest.param("いつ適用するか", id="ja-itsu-tekiyou"),
pytest.param("いつこれを利用する？", id="ja-itsu-riyou"),
]

# No "when" clause in the language's own words; also the wording a bare cue would wrongly accept.
Expand Down Expand Up @@ -196,6 +237,32 @@
pytest.param("适用于定时任务的调度工具。", id="zh-shi-dingshi-not-when"),
pytest.param("适用于时序数据分析的工具。", id="zh-shi-shixu-not-when"),
pytest.param("适用于时代背景的分析工具。", id="zh-shi-daibei-not-when"),
pytest.param("アイデアを思いつくためのブレインストーミング支援ツール。", id="ja-omoitsuku-not-itsu"),
pytest.param("最新の仕様に追いつくためのドキュメント要約ツール。", id="ja-oitsuku-not-itsu"),
pytest.param("ユーザーが食いつくキャッチコピーを生成する。", id="ja-kuitsuku-not-itsu"),
pytest.param("こいつはPDFを結合するツール。", id="ja-koitsu-not-itsu"),
pytest.param("分析耗时调用链路并生成报告。", id="zh-haoshi-not-when"),
pytest.param("监控超时调用并发出告警。", id="zh-chaoshi-not-when"),
pytest.param("支持延时执行的任务队列管理工具。", id="zh-yanshi-not-when"),
pytest.param("整理当事人信息，统计查询耗时。", id="zh-dangshiren-not-when"),
pytest.param("Hilft beim Schreiben von Commit-Nachrichten.", id="de-hilft-beim-not-when"),
pytest.param("Unterstützt Sie beim Erstellen von Berichten.", id="de-unterstuetzt-beim-not-when"),
pytest.param("Nutze das Tool. Hilft beim Schreiben.", id="de-sentence-boundary"),
pytest.param("Nutze das Tool. KI hilft beim Schreiben.", id="de-sentence-boundary-acronym"),
pytest.param("整理当事人信息，统计查询用时。", id="zh-dangshiren-yongshi-not-when"),
pytest.param("そいつを使って生成したログを解析するツール。", id="ja-soitsu-not-itsu"),
pytest.param("あいつを使うツール。", id="ja-aitsu-not-itsu"),
pytest.param("どいつを使うか判定するツール。", id="ja-doitsu-not-itsu"),
pytest.param("適用於延時的工具。", id="zh-hant-shiyongyu-yanshi-not-when"),
pytest.param("适用于监控超时的工具。", id="zh-shiyongyu-chaoshi-not-when"),
pytest.param("适用于统计耗时的工具。", id="zh-shiyongyu-haoshi-not-when"),
pytest.param("適用於統計耗時的工具。", id="zh-hant-shiyongyu-haoshi-not-when"),
pytest.param("適用於監控超時的工具。", id="zh-hant-shiyongyu-chaoshi-not-when"),
pytest.param("适用于延时的工具。", id="zh-shiyongyu-yanshi-not-when"),
pytest.param("整理当事人信息，统计查询耗時。", id="zh-hant-dangshiren-haoshi-not-when"),
pytest.param("ついつい使いたくなるアイデアを思いつくツール。", id="ja-omoitsuku-tsukau-not-itsu"),
pytest.param("こいつを使うツール。", id="ja-koitsu-tsukau-not-itsu"),
pytest.param("いつも使えるPDF結合ツール。", id="ja-itsumo-tsukae-not-itsu"),
]


Expand Down
Loading