[AI-58] llm: add bitwarden-planning-tools plugin with consulting-adrs skill - #177
[AI-58] llm: add bitwarden-planning-tools plugin with consulting-adrs skill#177SaintPatrck wants to merge 4 commits into
Conversation
|
Claude Code is validating plugin components and security... If this comment does not update with results, check the Actions log. |
🤖 Bitwarden Claude Code ReviewOverall Assessment: APPROVE Reviewed the new Code Review Details
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Establish bitwarden-planning-tools as the pre-implementation planning home (counterpart to bitwarden-delivery-tools' post-implementation mechanics), and land consulting-adrs here as its first skill. consulting-adrs checks a design/change/plan/threat-model against Bitwarden's ADRs (or locates/summarizes the catalog) and returns structured conflict/gap/aligned findings with cited ADRs. Ships the full eval harness (trigger/structure/behavior) with baselines on claude-opus-4-8: triggering 7/8 should-trigger, 6/6 should-not; behavior with-skill 1.00 vs baseline 0.78 over 9 cases. WebFetch scoped to contributing.bitwarden.com. Proposed as the future home for tech breakdowns, initiative-funnel navigation, and architecting-solutions as planning tools consolidate here.
Treat fetched ADR pages as untrusted data, matching the sibling architecting-solutions skill's rule for the same domain. Drop the Cross-Plugin Integration table's consumer inventory rather than correct its premature claim — a skill has no need to track which plugins call it, and the table's only other row duplicated the Overview's own counterpart framing.
ce76862 to
8e05440
Compare
Claude Configuration Validation — PR #177Provisional: plugin-validator and skill-reviewer agents still running. This file will be replaced with the final report. Direct checks completed so far on
|
d4374e1 to
928b0d5
Compare
928b0d5 to
ab8303f
Compare
| "should_not_trigger_pass": "4/4", | ||
| "reliability": { | ||
| "all_runs_agree_rate": 0.9166666666666666, | ||
| "should_trigger_reliable": "7/8", | ||
| "should_not_trigger_reliable": "4/4" | ||
| }, |
There was a problem hiding this comment.
❓ QUESTION: The two negative cases that over-triggered were dropped and the reliability figures recomputed — scope decision or worth re-running?
Details
ab8303f removed "Help me write a new ADR documenting the token-rotation decision we just landed on." and "What's an ADR and when should our team bother writing one?" from trigger-eval.json, and adjusted this block to match: should_not_trigger_pass 6/6 → 4/4, all_runs_agree_rate 0.786 → 0.917, should_not_trigger_reliable 4/6 → 4/4. Those were the only two negative cases that failed (each fired 1/3), and the _notes line disclosing them was trimmed at the same time.
The remaining 12 per-query records are self-consistent with the new totals, so nothing here misreports. Two things worth confirming:
- If the removal was a scope call (authoring and conceptual asks are arguably ambiguous labels rather than skill defects), a one-line note in
evals/README.mdunder "Known issues / boundaries" would keep that reasoning with the artifact — the under-trigger case is documented there, this one now isn't. - If they are still in scope, ADR-authoring is the nearest over-trigger boundary for this skill and now has only partial coverage via
"Update ADR-0012's status to superseded...".
Either resolution is fine; the ask is that the recorded baseline carries the reasoning, since it's what the next refresh will be compared against.
There was a problem hiding this comment.
Scope decision. The dropped tests were flakey and not relevant until competing or complimentary skills are identified.
The catalog fetch pulls untrusted public HTML into whatever conversation asks for a validation, so the skill runs in its own context and returns only findings. Isolation means the subject arrives as an explicit input rather than ambient context, and a templated output keeps the roll-up terminal. A cited URL is confirmed against the catalog index, because a derived URL that does not resolve is worse than no URL.
1 participant
🎟️ Tracking
AI-58
📔 Objective
Adds
bitwarden-planning-tools, a new plugin housing pre-implementation planning and preparation skills.Its first skill is
consulting-adrs: it checks a design, change, plan, or threat model against Bitwarden's Architecture Decision Records, or locates and summarizes the catalog, returning structured findings (conflict, gap, stale-reference, aligned) with cited ADRs. A cited URL is confirmed against the catalog index, so a finding never carries a link that does not resolve.The skill runs as a forked subagent and blocks until it returns, so consulting the catalog never pulls untrusted public HTML into the calling conversation. The caller passes the subject in and gets findings back. Inside the fork,
WebFetchis scoped tocontributing.bitwarden.comalongsideReadandGrep, whileWrite,Edit,NotebookEdit, andAgentare removed, the last of those because a forked agent otherwise delegates its way around its own tool scope.Ships a full eval harness per the AI Review Guidelines (triggering, structure, behavior) with baselines on
claude-opus-4-8, graded blind byclaude-opus-5andclaude-sonnet-5, recorded inskills/consulting-adrs/evals/.#178 wires
bitwarden-security-engineer's ADR-alignment checks to this skill and is stacked on this PR, which must land first.