Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions skills/plugin-audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Audit the target plugin and deliver a scored, evidence-backed health report.
## 1. Locate and inventory

1. Find the plugin root: the directory containing `.claude-plugin/plugin.json` **or** `.codex-plugin/plugin.json` (a dual-harness plugin has both). If the user gave no path, search the current repo, then the harness plugin dir (`~/.claude/plugins/` or `~/.codex/plugins/`); on Codex also map registered sources: `grep -A3 '\[marketplaces' ~/.codex/config.toml`. Never audit the read-only installed copies under `~/.claude/plugins/cache/` or `~/.codex/plugins/cache/`.
2. Determine which harnesses the plugin targets (manifests present, README claims). If it claims dual-harness support but ships only one manifest, flag the missing manifest as a parity finding — then audit it anyway.
2. Determine which harnesses the plugin targets. If it claims dual-harness support but ships one manifest, flag the missing manifest as a parity finding, then audit anyway.
3. Inventory every component: manifest fields (both manifests when present), `skills/*/SKILL.md`, `hooks/hooks.json` (or manifest `hooks` entries), `.mcp.json`, `.app.json`, `assets/`, README.

## 2. Validate mechanics
Expand All @@ -17,7 +17,7 @@ Check each item and record pass/fail with file paths as evidence:

- Each present `plugin.json` parses as JSON; `name` present, kebab-case; `version` is semver.
- Parity: if both manifests exist, `name` is identical and `version` agrees (ignoring any Codex `+build` suffix) — drift is a finding.
- All manifest paths (`skills`, `hooks`, `mcpServers`, `apps`, asset paths) start with `./`, resolve relative to the plugin root, and stay inside it.
- All manifest paths (`skills`, `hooks`, `mcpServers`, `apps`, asset paths) start with `./` and stay inside the plugin root.
- Each manifest dir holds only its own `plugin.json` (plus `marketplace.json`); `skills/`, `hooks/`, `assets/` are at the plugin root.
- Every skill directory contains a `SKILL.md` with `name` and `description` frontmatter; `name` matches its directory.
- `hooks.json` (if present) parses and uses known events: SessionStart, PreToolUse, PostToolUse, UserPromptSubmit, Stop.
Expand All @@ -34,15 +34,15 @@ python3 scripts/tokens.py <plugin-root> # token & budget report
python3 scripts/errscan.py <plugin-root> # runtime errors from session logs
```

- `score.py` emits `{dimension:{auto,max,needs_judgment}}` — the machine FLOOR: take each `auto` as fixed, score only the `needs_judgment` items. Mapping in `references/deterministic-scoring.md`.
- `score.py` emits `{dimension:{auto,max,needs_judgment}}` — the machine FLOOR, now higher for carrying graduated `skill_quality` signal: take each `auto` as fixed, score only `needs_judgment`. Mapping in `references/deterministic-scoring.md`.
- `tokens.py` gives the session-tax headline, budget headroom, and baseline delta → feeds Context economy.
- `errscan.py` aggregates runtime errors per plugin/skill → feeds Hooks health (hooks that throw lose points even when `hooks.json` shape is valid).

If a script is missing, inspect manually and note it.

## 4. Score

Read `references/scoring-rubric.md` and score all six dimensions (100 points total) on top of the `score.py` floor. It awards cross-harness parity credit inside Manifest integrity and Distribution readiness. For distribution readiness, also run the visual-presentation checks in `references/presentation.md` (interface metadata, icons, starter prompts, and the Codex-only per-skill openai.yaml). Every deduction needs a concrete evidence line (file, and line number where useful), citing script output where it drives one.
Read `references/scoring-rubric.md` and score each applicable dimension on the `score.py` floor, mapping the total to the frozen 5-band scale there. An N/A dimension (e.g. no hooks) is dropped, its weight redistributed — never a free max. For distribution readiness, run the visual-presentation checks in `references/presentation.md` (interface metadata, icons, starter prompts, Codex-only per-skill openai.yaml). Every deduction needs a concrete evidence line (file, line number where useful), citing script output where it drives one.

## 5. Report

Expand Down
13 changes: 8 additions & 5 deletions skills/plugin-audit/references/report-style.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ Reports are read in chat. Optimize for scannability: the verdict in the first 3

## Scorecard

Lead with grade + total, then a compact table with bar sparklines:
Lead with band + total, then a compact table with bar sparklines:

```
## Plugin health: my-plugin — 82/100 (Good)
## Plugin health: my-plugin — 82/100 (Strong)

| Dimension | Score | |
|---|---|---|
Expand All @@ -19,7 +19,9 @@ Lead with grade + total, then a compact table with bar sparklines:
| Distribution | 8/10 | ████████░░ |
```

Bars: 10 chars wide, `█` per 10%, `▌` for half, `░` fill. Always same width so columns align.
Bands (frozen — use these labels only, never legacy words like "Good"/"Fair"): **92–100 Exceptional · 82–91 Strong · 68–81 Solid · 50–67 Needs work · <50 Poor**. A mature-but-improvable plugin lands in Solid/Strong, not Exceptional. The example totals 82 → **Strong**.

Bars: 10 chars wide, `█` per 10%, `▌` for half, `░` fill. Always same width so columns align. When N/A dimensions are dropped (see Diagnostics), the table shows only the scored dimensions — fewer than 6 rows is expected, and each row's `/max` reflects the redistributed weight.

## Findings

Expand All @@ -38,13 +40,14 @@ Severity: 🔴 breaks behavior, 🟡 degrades quality, 🟢 cosmetic. Sort 🔴
A compact block placed right under the scorecard, showing the machine floor vs judgment split and the two runtime signals. Keep it to 3–4 lines:

```
🔎 Floor 71/100 auto (score.py) · +9 judged = 80/100 · needs_judgment: skill-quality, trigger-negativescope
🔎 Floor 74/100 auto (score.py, incl. skill_quality signal) · +8 judged = 82/100 (Strong) · needs_judgment: skill-quality depth, trigger-negativescope
🪙 Session tax 4.1k tok · body headroom OK · descriptions +180 tok vs baseline (tokens.py)
🩺 Runtime: 2 hook errors, 1 skill error in 14 sessions — `hooks/lint.sh` exit 1 ×2 (errscan.py)
✅ Runtime clean — no hook/tool/skill errors in scanned sessions
```

- Floor line: `auto` total from `score.py`, points you judged on top, final, and which dimensions still carried `needs_judgment` notes.
- Floor line: `auto` total from `score.py`, points you judged on top, final, and which dimensions still carried `needs_judgment` notes. The floor now includes graduated `skill_quality` machine signal, so it reaches higher than the old manifest-only floor — do not treat a high `auto` as saturation.
- N/A dimensions: when a dimension does not apply (e.g. the plugin ships no hooks), it is DROPPED and its weight redistributed across the applicable dimensions — it does NOT earn a free max. Name any dropped dimension here so the scorecard's `/max` values (and its shorter-than-6 row count) are traceable.
- Token line: session-tax headline, budget headroom (or the skill that blew it), and baseline delta — the evidence behind any Context economy deduction.
- Runtime line: error counts per plugin/skill and the top offending handler; when clean, collapse to the single ✅ line. This is the evidence behind any Hooks health deduction.

Expand Down
Loading