diff --git a/skills/plugin-audit/SKILL.md b/skills/plugin-audit/SKILL.md index 370c55a..8c9f5e2 100644 --- a/skills/plugin-audit/SKILL.md +++ b/skills/plugin-audit/SKILL.md @@ -8,7 +8,7 @@ Audit the target plugin and deliver a scored, evidence-backed health report. ## 1. Locate and inventory 1. Find the plugin root: the directory containing `.claude-plugin/plugin.json` **or** `.codex-plugin/plugin.json` (a dual-harness plugin has both). If the user gave no path, search the current repo, then the harness plugin dir (`~/.claude/plugins/` or `~/.codex/plugins/`); on Codex also map registered sources: `grep -A3 '\[marketplaces' ~/.codex/config.toml`. Never audit the read-only installed copies under `~/.claude/plugins/cache/` or `~/.codex/plugins/cache/`. -2. Determine which harnesses the plugin targets (manifests present, README claims). If it claims dual-harness support but ships only one manifest, flag the missing manifest as a parity finding — then audit it anyway. +2. Determine which harnesses the plugin targets. If it claims dual-harness support but ships one manifest, flag the missing manifest as a parity finding, then audit anyway. 3. Inventory every component: manifest fields (both manifests when present), `skills/*/SKILL.md`, `hooks/hooks.json` (or manifest `hooks` entries), `.mcp.json`, `.app.json`, `assets/`, README. ## 2. Validate mechanics @@ -17,7 +17,7 @@ Check each item and record pass/fail with file paths as evidence: - Each present `plugin.json` parses as JSON; `name` present, kebab-case; `version` is semver. - Parity: if both manifests exist, `name` is identical and `version` agrees (ignoring any Codex `+build` suffix) — drift is a finding. -- All manifest paths (`skills`, `hooks`, `mcpServers`, `apps`, asset paths) start with `./`, resolve relative to the plugin root, and stay inside it. +- All manifest paths (`skills`, `hooks`, `mcpServers`, `apps`, asset paths) start with `./` and stay inside the plugin root. - Each manifest dir holds only its own `plugin.json` (plus `marketplace.json`); `skills/`, `hooks/`, `assets/` are at the plugin root. - Every skill directory contains a `SKILL.md` with `name` and `description` frontmatter; `name` matches its directory. - `hooks.json` (if present) parses and uses known events: SessionStart, PreToolUse, PostToolUse, UserPromptSubmit, Stop. @@ -34,7 +34,7 @@ python3 scripts/tokens.py # token & budget report python3 scripts/errscan.py # runtime errors from session logs ``` -- `score.py` emits `{dimension:{auto,max,needs_judgment}}` — the machine FLOOR: take each `auto` as fixed, score only the `needs_judgment` items. Mapping in `references/deterministic-scoring.md`. +- `score.py` emits `{dimension:{auto,max,needs_judgment}}` — the machine FLOOR, now higher for carrying graduated `skill_quality` signal: take each `auto` as fixed, score only `needs_judgment`. Mapping in `references/deterministic-scoring.md`. - `tokens.py` gives the session-tax headline, budget headroom, and baseline delta → feeds Context economy. - `errscan.py` aggregates runtime errors per plugin/skill → feeds Hooks health (hooks that throw lose points even when `hooks.json` shape is valid). @@ -42,7 +42,7 @@ If a script is missing, inspect manually and note it. ## 4. Score -Read `references/scoring-rubric.md` and score all six dimensions (100 points total) on top of the `score.py` floor. It awards cross-harness parity credit inside Manifest integrity and Distribution readiness. For distribution readiness, also run the visual-presentation checks in `references/presentation.md` (interface metadata, icons, starter prompts, and the Codex-only per-skill openai.yaml). Every deduction needs a concrete evidence line (file, and line number where useful), citing script output where it drives one. +Read `references/scoring-rubric.md` and score each applicable dimension on the `score.py` floor, mapping the total to the frozen 5-band scale there. An N/A dimension (e.g. no hooks) is dropped, its weight redistributed — never a free max. For distribution readiness, run the visual-presentation checks in `references/presentation.md` (interface metadata, icons, starter prompts, Codex-only per-skill openai.yaml). Every deduction needs a concrete evidence line (file, line number where useful), citing script output where it drives one. ## 5. Report diff --git a/skills/plugin-audit/references/report-style.md b/skills/plugin-audit/references/report-style.md index c371d5e..014591e 100644 --- a/skills/plugin-audit/references/report-style.md +++ b/skills/plugin-audit/references/report-style.md @@ -4,10 +4,10 @@ Reports are read in chat. Optimize for scannability: the verdict in the first 3 ## Scorecard -Lead with grade + total, then a compact table with bar sparklines: +Lead with band + total, then a compact table with bar sparklines: ``` -## Plugin health: my-plugin — 82/100 (Good) +## Plugin health: my-plugin — 82/100 (Strong) | Dimension | Score | | |---|---|---| @@ -19,7 +19,9 @@ Lead with grade + total, then a compact table with bar sparklines: | Distribution | 8/10 | ████████░░ | ``` -Bars: 10 chars wide, `█` per 10%, `▌` for half, `░` fill. Always same width so columns align. +Bands (frozen — use these labels only, never legacy words like "Good"/"Fair"): **92–100 Exceptional · 82–91 Strong · 68–81 Solid · 50–67 Needs work · <50 Poor**. A mature-but-improvable plugin lands in Solid/Strong, not Exceptional. The example totals 82 → **Strong**. + +Bars: 10 chars wide, `█` per 10%, `▌` for half, `░` fill. Always same width so columns align. When N/A dimensions are dropped (see Diagnostics), the table shows only the scored dimensions — fewer than 6 rows is expected, and each row's `/max` reflects the redistributed weight. ## Findings @@ -38,13 +40,14 @@ Severity: 🔴 breaks behavior, 🟡 degrades quality, 🟢 cosmetic. Sort 🔴 A compact block placed right under the scorecard, showing the machine floor vs judgment split and the two runtime signals. Keep it to 3–4 lines: ``` -🔎 Floor 71/100 auto (score.py) · +9 judged = 80/100 · needs_judgment: skill-quality, trigger-negativescope +🔎 Floor 74/100 auto (score.py, incl. skill_quality signal) · +8 judged = 82/100 (Strong) · needs_judgment: skill-quality depth, trigger-negativescope 🪙 Session tax 4.1k tok · body headroom OK · descriptions +180 tok vs baseline (tokens.py) 🩺 Runtime: 2 hook errors, 1 skill error in 14 sessions — `hooks/lint.sh` exit 1 ×2 (errscan.py) ✅ Runtime clean — no hook/tool/skill errors in scanned sessions ``` -- Floor line: `auto` total from `score.py`, points you judged on top, final, and which dimensions still carried `needs_judgment` notes. +- Floor line: `auto` total from `score.py`, points you judged on top, final, and which dimensions still carried `needs_judgment` notes. The floor now includes graduated `skill_quality` machine signal, so it reaches higher than the old manifest-only floor — do not treat a high `auto` as saturation. +- N/A dimensions: when a dimension does not apply (e.g. the plugin ships no hooks), it is DROPPED and its weight redistributed across the applicable dimensions — it does NOT earn a free max. Name any dropped dimension here so the scorecard's `/max` values (and its shorter-than-6 row count) are traceable. - Token line: session-tax headline, budget headroom (or the skill that blew it), and baseline delta — the evidence behind any Context economy deduction. - Runtime line: error counts per plugin/skill and the top offending handler; when clean, collapse to the single ✅ line. This is the evidence behind any Hooks health deduction.