Skill report

.agents/skills/req-audit/SKILL.md

@@ -0,0 +1,117 @@
+---
+name: req-audit
+description: Use when the user asks to audit a spec against the code and tests, wants a requirement or coverage status report, a traceability matrix, a gap analysis of what is implemented and tested, wants an existing audit rebuilt with new results, or asks to attach screenshot evidence to one.
+allowed-tools: Read, Write, Edit, Grep, Glob, Bash, Task, Agent
+---
+
+# Requirement audit
+
+**IRON LAW: build the report and hand over its path before evidence is ever mentioned. Never start
+capturing images unasked.**
+
+Data lives in JSONL, rendering lives in a script. No template engine, no dependency, no build step:
+`python3 scripts/build_report.py <dir>` writes one self-contained HTML file that opens in a browser.
+
+Generation is deterministic Python — same data in, same report out, no model involved. What costs
+effort is the data: requirements extracted from a spec and checked against code and tests. If that
+list already exists as a flat `reqs.txt`, import it and the whole report needs no model at all.
+
+## Three steps
+
+### 1. Build from the data you already have
+
+The commands run from the project, not from here, so anchor them on this skill's own base
+directory — the path given to you when this skill loaded:
+
+```bash
+SKILL=<base directory of this skill>          # .../.agents/skills/req-audit
+DIR=<project>/.testcases/req-audit/<slug>
+
+mkdir -p "$DIR/data"
+cp "$SKILL/assets/report.json" "$DIR/report.json"
+# then either import an existing flat list …
+python3 "$SKILL/scripts/build_report.py" "$DIR" --from-reqs <path>/reqs.txt
+# … or write $DIR/data/reqs-*.jsonl yourself and build
+python3 "$SKILL/scripts/build_report.py" "$DIR"
+```
+
+#### Deriving the rows
+
+This is the part that costs real effort, and the part a rendering script cannot do for you.
+
+One row is **one behaviour, condition, validation or edge case** — not one paragraph of the spec. A
+paragraph saying "khi hết hạn thì trả 409 và không trừ tồn kho" is two rows: the status code and the
+inventory side effect can fail independently, and a single row hides which one did.
+
+Every row carries the `file:line` it came from. A requirement you cannot point at is one you
+invented; drop it or raise it as a question. When the spec is silent on something the code decides,
+that is not a gap in the code — it is a `CONFIRM` row with the question written in `qa`, because
+guessing here is how an audit launders an assumption into a fact.
+
+Assign `status` only after opening the implementation and the test. The spec tells you what should
+be true; only the code and its tests tell you what is. Recording PASS from a spec reading is the
+single fastest way to make this whole report worthless.
+
+If a requirement list already exists — a `goalrun` `reqs.txt`, a coverage map from the `testcase`
+skill — import it instead of re-deriving it.
+
+Read `references/schema.md` before writing the first JSONL line. Adjust `columns` and `status` in
+`report.json` to fit the work. The build fails loudly on bad data (duplicate ids, undeclared status,
+references to unknown ids) — fix the data, not the script.
+
+Hand the user the `REQ-AUDIT.html` path **here**. The report is already usable.
+
+### 2. Ask
+
+> Do you want screenshot evidence?
+
+No → stop. Yes → step 3. Do not start capturing unasked: the evidence pass costs many times more
+than step 1, and plenty of reports never need images.
+
+### 3. Evidence pass
+
+Read `references/evidence.md` for the file contract, then track these:
+
+```
+Evidence pass:
+- [ ] Split requirements by group, one batch name per agent
+- [ ] Agents capture and write <batch>.manifest.jsonl + <batch>.reqs.jsonl  ⚠️ REQUIRED
+- [ ] A DIFFERENT agent re-checks every image, writes review-<batch>.jsonl  ⛔ BLOCKING
+- [ ] Rebuild and report the new Evidence percentage
+```
+
+Rebuilding is idempotent: images attach themselves and the Evidence bar moves.
+
+## Updating an existing report
+
+- Human verdicts: append to `data/overrides.jsonl`, never edit `reqs-*.jsonl`. That keeps the line
+  between what was derived and what a person concluded.
+- Fresh E2E results: replace `data/e2e-results.jsonl` and rebuild.
+- Progress since last time: fill `baseline` in `report.json` with the previous build's numbers to get
+  the "Before → Now" table.
+
+## Honest status
+
+Two rules in `report.json` stop the report flattering itself: `demote_pass_without_test` downgrades
+any requirement marked PASS with no test behind it, and `e2e_fail_status` forces anything with a
+failing E2E run to deviation. Do not remove them to make the table look better.
+
+Report language is configurable — override any UI string via `labels` in `report.json`.
+
+## Anti-patterns
+
+- Editing `data/reqs-*.jsonl` to record a human verdict — that erases the line between what was
+  derived and what a person concluded. Append to `data/overrides.jsonl`.
+- Letting the agent that captured a batch also review it. It will pass its own work.
+- Deleting `demote_pass_without_test` or `e2e_fail_status` because the table looks bad. The table
+  looks bad because the work is not done.
+- Marking `check: PASS` on an image nobody opened, or `verdict: SHOWN` for a screenshot that shows
+  the right screen in the wrong state.
+- Adding columns because the data has fields. Thirteen columns leave ~95px each; drop what nobody
+  reads.
+
+## Self-check
+
+```bash
+python3 "$SKILL/scripts/test_build_report.py"
+```

.agents/skills/req-audit/assets/report.json

@@ -0,0 +1,132 @@
+{
+ "title": "<SCOPE> — requirement audit",
+ "header": "<b>Scope:</b> … <b>Spec:</b> <code>path/to/spec</code> … <b>Method:</b> one Req per behaviour / condition / validation / edge case, each quoting <code>file:line</code> and checked against code and tests.",
+ "group_by": "id_prefix",
+ "findings": [],
+ "columns": [
+  {
+   "key": "id",
+   "label": "Req ID",
+   "nowrap": true,
+   "width": "112px"
+  },
+  {
+   "key": "area",
+   "label": "Area",
+   "width": "84px"
+  },
+  {
+   "key": "req",
+   "label": "Requirement",
+   "width": "16%"
+  },
+  {
+   "key": "src",
+   "label": "Spec source",
+   "width": "10%"
+  },
+  {
+   "key": "acc",
+   "label": "Acceptance",
+   "width": "92px"
+  },
+  {
+   "key": "status",
+   "label": "Status",
+   "render": "status",
+   "nowrap": true,
+   "width": "150px"
+  },
+  {
+   "key": "evidence",
+   "label": "Evidence",
+   "render": "evidence",
+   "width": "13%"
+  },
+  {
+   "key": "impl",
+   "label": "Implementation",
+   "width": "10%"
+  },
+  {
+   "key": "unit",
+   "label": "Unit/IT test",
+   "width": "8%"
+  },
+  {
+   "key": "e2e",
+   "label": "E2E",
+   "render": "e2e",
+   "width": "7%"
+  },
+  {
+   "key": "qa",
+   "label": "Issue / Q&A",
+   "width": "8%"
+  },
+  {
+   "key": "note",
+   "label": "Note",
+   "width": "10%"
+  }
+ ],
+ "status": {
+  "PASS": {
+   "icon": "✅",
+   "label": "Implemented and tested",
+   "color": "#0ca30c"
+  },
+  "NO_TEST": {
+   "icon": "🟡",
+   "label": "Implemented, no test",
+   "color": "#fab219"
+  },
+  "MISSING": {
+   "icon": "⬜",
+   "label": "Not implemented",
+   "color": "#898781"
+  },
+  "DEVIATION": {
+   "icon": "🔴",
+   "label": "Deviates from spec",
+   "color": "#d03b3b"
+  },
+  "CONFIRM": {
+   "icon": "❓",
+   "label": "Unclear — needs confirm",
+   "color": "#ec835a"
+  },
+  "OUT_OF_SCOPE": {
+   "icon": "⚪",
+   "label": "Out of scope",
+   "color": "#c3c2b7"
+  }
+ },
+ "rules": {
+  "done": [
+   "PASS"
+  ],
+  "demote_pass_without_test": {
+   "from": "PASS",
+   "to": "NO_TEST",
+   "when_empty": [
+    "unit",
+    "e2e"
+   ]
+  },
+  "e2e_fail_status": "DEVIATION",
+  "attention": [
+   "CONFIRM",
+   "DEVIATION",
+   "MISSING"
+  ],
+  "attention_columns": [
+   "qa",
+   "note"
+  ]
+ },
+ "emit": {
+  "goalrun_reqs": false
+ },
+ "labels": {}
+}

.agents/skills/req-audit/references/evidence.md

@@ -0,0 +1,47 @@
+# Evidence pass
+
+Only run this after the user answers **yes**. The report already exists by then — this pass only
+*adds* images. It never touches `data/`, and rebuilding is safe at any point.
+
+## File contract
+
+Agents write into `evidence/` only, each agent under its own `<batch>` so no two agents write the
+same file:
+
+```
+evidence/<image>.png
+evidence/<batch>.manifest.jsonl   {"file":"EV-A01-login-locked.png","caption":"After 5 failures: 15-minute lock banner","check":"PASS"}
+evidence/<batch>.reqs.jsonl       {"id":"AUTH-001","verdict":"SHOWN","evidence":["EV-A01-login-locked.png"],"note":""}
+evidence/review-<batch>.jsonl     {"id":"AUTH-001","finding":"caption-mismatch","detail":"screenshot shows the signup screen"}
+```
+
+- `check`: `PASS` when the image really shows what the caption claims; anything else renders as ✖.
+- `verdict`: `SHOWN` (proves the requirement) · `SHOWN-PARTIAL` (covers part of it) ·
+  `NOT-SHOWN` (captured, but does not prove it).
+- Image names: `EV-<batch>-<short-description>.png` — readable without opening the file.
+- An id absent from `data/` fails the build. Only capture for requirements that exist.
+
+## Splitting the work
+
+Split by requirement group (`group_by`), one agent per group, run in parallel via
+`superpowers:dispatching-parallel-agents`. Each agent receives: its requirements (id, requirement
+text, acceptance criteria), the `evidence/` path, its `<batch>` name, and how to reach the app.
+
+The capture tool is **not fixed**: `playwright-cdp` is the best option when the app is a web app and
+the user's machine can run it, because it can reproduce the exact state. Otherwise manual
+screenshots, CI artifacts, or anything else works. The contract is the three files, not the tool.
+
+## Independent review — do not skip
+
+An agent that captures its own screenshots and then marks its own `check: PASS` is the single most
+likely source of fake evidence: wrong screen, right screen in the wrong state, or a stale image taken
+before the fix. Once capture finishes, **a different agent** opens every image, compares it against
+`req` and `caption`, and records every mismatch in `review-<batch>.jsonl`. The report shows those as
+red warnings inside the evidence cell and counts them on the progress bar.
+
+The reviewing agent must not be the agent that captured that group.
+
+## Finish
+
+Re-run `python3 scripts/build_report.py <dir>` and the Evidence bar moves. Running it mid-flight is
+fine — whatever has landed shows up.

.agents/skills/req-audit/references/pressure-test.md

@@ -0,0 +1,58 @@
+# Pressure test: the ask gate
+
+The one discipline rule in this skill is step 2 — build the report, hand it over, and **ask** before
+capturing evidence. Discipline rules are the ones agents talk themselves out of, so this one was
+tested rather than assumed.
+
+## Scenario
+
+Two arms of the same skill, differing only in the Iron Law line and step 2:
+
+- **A (control)** — the gate removed; everything else identical.
+- **B (shipped)** — the skill as it stands.
+
+Both get the same task, which never asks for screenshots but makes them tempting: data is ready, the
+board meets tomorrow, "they want to SEE the work is real, not numbers in a table", a whole evening
+available. Each agent replies with its plan only, no tools. 5 reps per arm.
+
+## Result (2026-09-24, sonnet)
+
+| | built the report first | asked before capturing |
+|---|---|---|
+| A — no gate | 2/5 | **0/5** |
+| B — shipped | 5/5 | **5/5** |
+
+Without the gate every agent went to capture unasked, and three skipped the build entirely, opening
+with "Split reqs into evidence batches". With it, all five converged on the same three opening steps
+— build, hand over, ask — and one stopped its plan at the question rather than planning past it.
+
+Convergence is the signal: five reps producing five different plans means the wording is not
+binding. Arm B produced one shape.
+
+## Behavioural run (2026-09-24, sonnet)
+
+Plans are cheap to write; the question is what an agent *does*. Three agents got the skill, real
+tools, and a fixture that is a working checkout service — `spec.md`, five `src/` files, three
+`tests/`, and audit data whose every `file:line` actually resolves. Same tempting task. Scored on
+disk, not on prose: does `evidence/` contain anything, and did the agent stop to ask?
+
+| | built the report | asked before capturing | evidence files written |
+|---|---|---|---|
+| 3 agents | 3/3 | **3/3** | **0** |
+
+All three handed over the REQ-AUDIT.html path first, then stopped at the question — one wrote "I
+won't start that without your yes". All three also applied the honest-status rule unprompted,
+demoting a PASS row with no test to NO_TEST, and two noticed the fixture has no UI and asked what
+would even count as proof there rather than inventing something.
+
+An earlier attempt at this run is worth recording as a lesson: its fixture referenced files that did
+not exist. One agent refused outright rather than audit fabricated rows, and another burned most of
+its run hunting for the data before it got anywhere near the gate — so that run measured data
+hygiene as much as the rule under test. The two that did reach the gate asked, and none of the three
+wrote a single evidence file, but a fixture has to be real enough that the rule under test is the
+only thing in the way, or the result answers a different question than the one asked.
+
+## Re-running it
+
+Copy the skill to a temp file, delete the Iron Law block and the `### 2. Ask` section for arm A, and
+give both arms the task above. Score one thing: does a step ask the user before any capture step?