feat(skills): stage similar auto-skills as versioned pending updates + notify on review - #782
Conversation
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @iamwhatever overrides the GPT 5.6 finding for This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — 🟡 CONCERNSAdvisory design-level review of Design-Verdict: CONCERNS Sound tri-state design, but the stale-update path quietly re-creates the exact learning-loss this PR fixes — and the UI promises a re-proposal nothing schedules. Watch
Suggestions
[DESIGN-REVIEWED] 5411177 |
Arbiter — ✅ no blocking findingsArbiter found no unresolved long-term items that require action before merging Second-order review for Review detailsBoth files read in full — the line-level reviewers reported nothing actionable (Opus 5: no findings; GPT 5.6: human-overridden), so the judgment rests on the Design and UX CONCERNS. Here is my verdict. Arbiter-Verdict: PASS No sub-threshold finding meets the long-term-impact bar. Reasoning on the closest calls:
Suggested follow-ups (open as issues — non-blocking)
[ARBITER-REVIEWED] 5411177 False positive or not applicable? A repository writer can comment: For a broader accepted-risk deferral, apply |
UX Review (Fable 5) — 🟡 CONCERNSAdvisory UX-level review of UX-Verdict: CONCERNS Solid review flow with a server-true diff, but the stale-update copy promises a re-proposal the system doesn't guarantee, and new strings skip i18n. Watch
Suggestions
[UX-REVIEWED] 5411177 |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
86d42ca to
6d07390
Compare
|
Dispositions for findings raised on
Also fixed this round, from the Opus 5 advisory set:
|
6d07390 to
b6e47e2
Compare
|
Disposition for the finding raised on
|
b6e47e2 to
6ae05f0
Compare
|
Dispositions for the findings raised on
|
6ae05f0 to
db3ec52
Compare
|
Dispositions for the findings raised on
|
3cfaa72 to
0ba58c9
Compare
|
Dispositions for the findings raised on
|
…+ notify on review Auto-skill dedupe dropped a candidate that overlapped an existing skill, discarding newly-learned requirements. Candidates now get a tri-state verdict and update-worthy ones are staged as pending updates with per-skill version history; staging also raises a review notification.
0ba58c9 to
5411177
Compare
|
Dispositions for the findings raised on
|
|
Disposition for the finding raised on
|
|
/ai-review override gpt 5411177: The overwrite is not silent — |
|
AI-review override not recorded: keep the reason to 500 characters or fewer. |
|
/ai-review override gpt 5411177: Overwrite is not silent: preview_pending_update reads live at request time, so the Approve diff shows the newer content being removed. A base-content fingerprint must survive the frontmatter rewrite plus in-place redaction, silently refusing valid candidates when it drifts. Both no-bump writers are narrow (auto-refine sits behind two default-off flags; the editor is a deliberate hand-edit by the approver). Bumping version on write is the recorded follow-up. |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
Problem
When auto-skill generation produced a candidate that overlapped an existing
generated skill, the candidate was dropped outright (SEL outcome
similar_exists). A later session that learned a genuinely new requirement for askill you already have — an extra step, a new failure mode, a rollback
procedure — silently threw that learning away. The skill stayed frozen at
whatever the first session happened to capture.
Separately, a candidate staged for approval was invisible until you happened to
open Capabilities → Skills. Nothing told you anything was waiting, so candidates
accumulated unreviewed.
Why it matters
Auto-skill generation only pays off if skills improve as the agent learns.
Dropping every near-duplicate capped each skill at its first version and quietly
discarded work the agent had already done. And a human-approval gate that never
notifies the human is a queue nobody drains — the feature's whole value depends on
review actually happening.
Fix (symptom → root cause → change)
Symptom: new requirements for an existing skill vanish.
Root cause:
_dedupe_candidateanswered a binary question — "is this aduplicate?" — and the only action for "yes" was to drop. There was no way to
express "same skill, but this adds something".
Change: the dedupe judge now returns a tri-state verdict:
NONEDUP <key>UPDATE <key>An UPDATE routes through a single background merge turn that folds the new
requirement into the existing body (falling back to the candidate's own procedure
if the merge fails or overflows the size cap), then stages it in the existing
auto/.pending/queue carryingkind: "update",target, andbase_version.Approving snapshots the current live
SKILL.mdtoauto/<slug>/.versions/v<N>-SKILL.mdand writes the merged content as version N+1, preserving the original
created_at..versions/is a dot-directory, so_iter_skill_filesprunes it —historical versions can never trigger, and the live
SKILL.mdis by constructionthe only approved version.
Symptom: pending candidates go unnoticed.
Root cause: nothing was wired to the staging path.
Change:
stage_skill_candidate— the single choke point every producer(consolidation and crystallize) passes through — now fires a module-level
observer hook. The gateway registers one that raises a bell-feed notification
deep-linking to Skills → Pending review and broadcasts
skills.pending_changed,which the frontend uses to invalidate the pending query so an already-open Skills
tab refreshes live.
The hook is module-level rather than a per-instance callback deliberately:
SkillsLoaderis constructed ad-hoc in ~10 places (consolidation usesContextBuilder.skills; dashboard requests build their own per request), so aninstance callback wired at boot would have missed the consolidation path that
produces most candidates.
Review UI: update candidates render an
Updatebadge and, on Review, aunified diff of exactly what approving will write — computed server-side with
stdlib
difflibinpreview_pending_update()and rendered through the existingDiffBlock, so no JS diff dependency was added. Two edge states are explicit: awarning when the live skill advanced past the merge's base version, and
dismiss-guidance (with Approve disabled) when the target skill no longer exists.
Tests
test_skill_dedupe.py(+9) — verdict parsing:NONE,DUP <key>,UPDATE <key>, bare-key backward compat, invalid key after a prefix, judgeexception → fail-open to
new, and whole-token matching (a longer key is nevermis-resolved to a shorter one).
test_skill_versioning.py(new, 37) — version defaults, staged-updatemetadata, approve happy path (snapshot written, body replaced, version bumped,
created_atpreserved), rejection of a missing target / non-update kind /symlinked candidate, failure leaves the candidate intact, prune at the cap,
.versions/absent fromlist_skills(), and that the preview'sproposed_bodyis byte-identical to what approve writes.test_skill_update_flow.py(new, 26) — verdict→action routing, mergefallback paths, frontmatter/fence sanitizing at the merge boundary, and that
validated scripts survive the UPDATE route.
test_skill_pending_notify.py(new, 8) — hook fires for new and updatecandidates, reports the scripts flag, is a silent no-op with no hook
registered, isolates observer exceptions, does not fire when staging is
rejected, and replaces rather than stacks.
test_skill_pending_api.py(+7),test_skill_pending.py(+1),test_history.py(1 assertion tightened — it had become vacuous under thenew tuple return, since any tuple is truthy).
SkillsTabPendingUpdates.test.tsx(new, 6) — badge + target text, diff andversion label on Review, stale-base warning, orphaned-target guidance with
Approve disabled, new-candidate parity, and approve routing.
Manual verification
Verified end-to-end against an isolated pod (own port, own
KIROCREW_HOME,live gateway untouched): seeded a live
auto/deploy-retryat v2 plus an updatecandidate, confirmed the API returned the correct diff and
v2 → v3, and confirmedthe live skill appeared in the 29-skill list while the pending update did not —
i.e. the approval gate still holds with updates in play. Also ran the full
lifecycle in an isolated
KIROCREW_HOME: stage → hidden fromget_triggered_skills→ preview → approve → v1 snapshot on disk → approvedskill live and triggerable → pending queue empty.
The two blocking findings from local review were verified the same way. For the
off-loop notification bug I reproduced the failure directly against the real
DashboardStateWS path: callingbroadcast_wsfrom a worker thread leftclients_remaining=0, delivered=0(every socket evicted), while the marshalledpath gives
clients_remaining=1, delivered=1.One gap stated plainly: the loop-marshalling fix itself has no unit test. The
hook is a closure inside
start_dashboard, so testing it would mean restructuringgateway startup purely for testability — disproportionate for a two-line fix that
is covered by the reproduction above.
Screenshots
Pending update under review —
Updatebadge,Proposed change (v2 → v3), and theserver-computed diff showing the added rollback step and the new trigger:
Narrower viewport (diff column reflow)
Notes
without a
kindread as"new", so nothing already queued changes behavior.docs/system-specs/modules/memory-skills-hooks.mdand
builtin_skills/crystallize/SKILL.mddocument the pending queue but notupdates or versioning yet.