Harden rules, add false-positive corpus, sync docs to the live run

README.md

@@ -2,22 +2,27 @@
 
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](#license)
 
-**Every key a developer pastes into Claude Code goes to the model and stays in a session file on disk. redact swaps it for a placeholder inside Claude Code, before the model reads it and before the transcript keeps it, with zero network reach.**
+**Every key a developer pastes into Claude Code goes to the model and stays in a session file on disk. redact swaps it for a placeholder inside Claude Code before the model reads it, keeps the placeholder in the conversation rows the session file stores, and puts the real value back only inside file edits. Zero network reach.**
 
 Site: [redact.aidojo.si](https://redact.aidojo.si). It is a Claude Code [mod](https://code.claude.com/docs/en/plugins/mods/overview): a plugin of TypeScript function hooks that runs in the Claude Code process.
 
 ## What happens
 
 ```
-# Headless `claude -p` run on Claude Code 2.1.288. The prompt held a fake AWS access key id.
-# Typed (key masked here):
-my aws key is AKIA**************** please remember it
-# The user row as stored in the session file:
-my aws key is [REDACTED:AWS_KEY#20fae26c] please remember it
-# The stored system-prompt snapshot carried the redaction notice that tells Claude the placeholder is opaque.
+# Interactive session on Claude Code 2.1.288, Opus 5.5, accept-edits mode. Fake key, masked here as AKIA****************.
+⏺ redact: hid 1 AWS_KEY in your prompt
+❯ Save this AWS access key id to a new file named key.txt in this directory, exactly as given, then stop: [REDACTED:AWS_KEY#08956cf3]
+⏺ Write(key.txt)
+  ⎿  Wrote 1 line to key.txt
+      1 AKIA****************
+⏺ I saved the AWS access key ID to key.txt, exactly as you gave it with no trailing newline. The write tool fills in the
+  real value when it creates the file, so it should be on disk as you sent it. I didn't open the file to check, because that would print the key back
+  into this conversation.
 ```
 
-The same value gets the same placeholder for the whole session. When Claude later edits a file that holds it, redact puts the real value back inside the `Edit`, `Write` or `NotebookEdit` arguments only.
+`key.txt` on disk held the real key. The session file held the placeholder in every conversation row; `/redact` answered `hid 1 values this session (1 AWS_KEY); restored 1 placeholders`.
+
+The same value gets the same placeholder for the whole session. When Claude later edits a file that holds it, redact puts the real value back inside the `Edit`, `Write` or `NotebookEdit` arguments only. In a live session this was driven end to end: a key typed in the prompt, Claude asked to save it to a file, the file on disk holding the real key while every conversation row holds the placeholder. See [Where it was proven](#where-it-was-proven).
 
 ## Install
 
@@ -35,24 +40,22 @@ To try it for one session without installing: `claude --plugin-dir ./plugins/red
 - **Reach L0.** It runs no process, reads no file, env var or setting, and writes no file. The validator lines in [What it reaches](#what-it-reaches) list every hook, call and state key.
 - **No network.** Nothing leaves the machine. It starts no daemon and calls no scanner binary.
 - **No model call.** Detection never asks a model. A redactor that sent your text to a model to find secrets would defeat its own purpose.
-- **Deterministic rules.** 23 secret rules and 7 opt-in PII rules: regex plus checksums (Luhn, Verhoeff, IBAN mod-97, JWT header decode). The same input gives the same output. Every quantifier is bounded.
+- **Deterministic rules.** 24 secret rules and 7 opt-in PII rules: regex plus checksums (Luhn, Verhoeff, IBAN mod-97, JWT header decode). The same input gives the same output. Every quantifier is bounded.
 - **Reversible only inside `Edit`, `Write` and `NotebookEdit`.** Placeholders turn back into real values only in those arguments, so an edit to a line that held a key still matches the file. `Bash` never gets the real value, so the model cannot echo a placeholder into a network command and send the secret out.
 - **Fail closed.** A row that makes the scanner throw is replaced with `[redact withheld this content: the redactor failed while scanning it]`. It is never stored raw.
 - **The transcript file too.** It rewrites at `session.append`, the one event that sees every row before it is stored and sent: prompts, tool results, Claude's own blocks, attachments, notices. The model and the session file get the same rewritten row.
 - **Proof files.** The repo ships the threat model below, [`RULES.md`](plugins/redact/RULES.md) with a public source for every rule, and [`DECISIONS.md`](plugins/redact/DECISIONS.md) with every rejected option and every assumption not yet checked.
 
-## Proof so far
-
-From a run on Claude Code 2.1.288 with the mod-builder harness:
-
-- `claude plugin validate` passed.
-- The module loaded with events `session.start`, `prompt.submit`, `session.append`, `tool.call`, `prompt.section`, `command.run`.
-- `tsc` passed.
-- 60 kit tests pass (`claude plugin test`).
-- `/redact` answered in a headless run.
-- A headless `claude -p` run with a fake AWS key stored the user row as shown in [What happens](#what-happens), and the stored system-prompt snapshot carried the redaction notice.
+## Where it was proven
 
-Not done yet: a live interactive run with a logged-in session and a real file edit by the model. The kit tests cover the argument rewrite in `Edit`, `Write` and `NotebookEdit`. A real edit end to end has not been run.
+| Surface | Status | Evidence |
+|---|---|---|
+| Terminal, interactive, logged in | ran and passed | A session on Claude Code 2.1.288 (Opus 5.5): the prompt echo showed the placeholder, Claude's `Write` call carried the placeholder, the file on disk holds the real key, the session file holds the placeholder in every conversation row and `/redact` reported `restored 1 placeholders` |
+| Headless `claude -p` | ran and passed | The stored user row held the placeholder; the system-prompt snapshot carried the redaction notice; `/redact` answered |
+| Plugin test kit | ran and passed | `claude plugin test`: 65 pass, 0 fail, on the real engine |
+| Subagent rows | unverified | The types say `session.append` and `tool.call` fire for subagents with `agentId`; not yet driven live |
+| `/reload-plugins` | unverified | Designed for: the value map lives in `$.state`, which survives a reload |
+| SDK, VS Code panel, Desktop, cloud sessions | unverified | Same hooks, no drawing; not yet run there |
 
 ## What it reaches
 
@@ -92,12 +95,28 @@ The same secret gets the same placeholder across rows, edits and compaction summ
 
 ## What it catches
 
-Secrets, always on: AWS access keys and secret keys, GitHub, GitLab, Slack tokens and webhooks, Stripe live keys, SendGrid, Twilio, Google API keys and OAuth secrets, Anthropic, OpenAI, Hugging Face, npm and PyPI tokens, Azure storage keys, JWTs, PEM private keys, database URLs with a password, `Authorization` header values, and generic `password=` / `api_key=` style assignments.
+Secrets, always on: AWS access keys and secret keys, GitHub, GitLab, Slack tokens and webhooks, Stripe live keys, SendGrid, Twilio, Google API keys and OAuth secrets, Anthropic, OpenAI, Hugging Face, npm and PyPI tokens, Azure storage keys, JWTs, PEM private keys (complete or truncated), database URLs with a password, `Authorization` header values, generic `password=` / `api_key=` style assignments, and any of the prefix-shaped tokens above wrapped in base64.
 
 PII, opt-in with `pii: true`: email, payment cards (Luhn), India Aadhaar (Verhoeff), US SSN, IBAN (mod-97), India PAN, phone numbers.
 
 Every rule, with its id, a fake example and its source: [`RULES.md`](plugins/redact/RULES.md).
 
+## False positives on ordinary code
+
+A redactor that fires on normal work gets uninstalled in a week, so the number to defend is the false-positive count, not the catch count. `tools/fp-corpus.mjs` shallow-fetches 11 public repositories at pinned commits (lockfiles, Go sums, Terraform, Markdown, minified JS, JWT and PEM test vectors, a dotenv test suite) and scans every text file with the secret rules:
+
+| Files scanned | Secret-rule hits | Real test keys and vectors | Fake fixtures | False positives |
+|---|---|---|---|---|
+| 42,836 | 68 | 60 | 8 | 0 |
+
+Every hit was opened at its line and labelled by hand; the labels live in `tools/fp-corpus-labels.json`, so a rerun that produces a new hit shows it as unreviewed instead of counting it. Rerun it yourself:
+
+```
+node tools/fp-corpus.mjs
+```
+
+The full per-hit list is in [`tools/fp-corpus-results.md`](tools/fp-corpus-results.md). The 60 real-shaped hits are complete private keys and signed JWTs published as test vectors, and demo database URLs with a password; a redactor should hide those. The PII rules, off by default, hit 650 times on the same corpus (mostly email addresses in docs), which is why they are opt-in.
+
 ## Options
 
 Stored under `pluginConfigs` in settings. A change reloads the mod.
@@ -108,25 +127,24 @@ Stored under `pluginConfigs` in settings. A change reloads the mod.
 
 ## Limitations
 
-- Two engine bookkeeping records in the session file are stored as made and can keep the typed prompt: the `queue-operation` record written at enqueue, and a command's `args` stamp. Neither is sent to the model. In the headless `-p` run above, both still held the typed text. In the headless path the queue record is written before `prompt.submit` runs. This is a known gap.
-- A secret the rules do not know stays visible. Generic `key=value` detection needs the value to look like a secret (length and entropy): `password=hunter2` is caught, `token=abc` is not.
-- A secret split across two rows, or shown partially (the last four characters), is not caught.
-- Images are not scanned. A screenshot of a key goes through.
-- Claude cannot search for a secret by value: `Grep` for a placeholder finds nothing. Search by the line around it.
-- Plugins loaded ahead of this one see the row before it is rewritten.
-- The screen may show a row just before its rewrite. The model and the session file never see that form.
-- Nothing is drawn in `claude -p`, the SDK, the VS Code panel or cloud sessions. The `redact: hid ...` line arrives as `ui_log` there. `/redact` still reports totals; it answered in the headless run.
-- A pattern guard is not a permission rule. A determined prompt can describe a secret in words the rules do not match.
-
-Assumptions not yet checked (subagent rows, restore after `/reload-plugins`, the interactive queue record) are listed with a way to check each one in [`DECISIONS.md`](plugins/redact/DECISIONS.md).
+- **Three host bookkeeping records are stored as made and can hold a raw value.** The engine lets a mod rewrite the conversation rows the model reads, not the records around them. (a) The `toolUseResult` record of a `Write`, `Edit` or `NotebookEdit` call keeps the file content as written, so a value restored into a file lands there in clear. (b) A slash command's `args` stamp keeps the typed arguments. (c) In headless `claude -p` only, the `queue-operation` record keeps the typed prompt, because it is written before `prompt.submit` runs; an interactive session writes no such record. None of the three is sent to the model.
+- **`tool_use` blocks are not rewritable.** The engine puts the model's own tool call blocks back as made; a mod may rewrite text blocks and tool results only. The model only ever sees placeholders, so a raw value in a tool call means it came from text the rules missed.
+- **A secret the rules do not know stays visible.** Generic `key=value` detection needs the value to look like a secret (length and entropy): `password=hunter2` is caught, `token=abc` is not.
+- **A secret split across two rows, or shown partially** (the last four characters), is not caught.
+- **Images are not scanned.** A screenshot of a key goes through.
+- **Claude cannot search for a secret by value.** `Grep` for a placeholder finds nothing. Search by the line around it.
+- **Plugins loaded ahead of this one** see the row before it is rewritten.
+- **The screen may show a row just before its rewrite.** The model and the session file never see that form.
+- **Nothing is drawn in `claude -p`, the SDK, the VS Code panel or cloud sessions.** The `redact: hid ...` line arrives as `ui_log` there. `/redact` works everywhere.
+- **A pattern guard is not a permission rule.** A determined prompt can describe a secret in words the rules do not match.
 
 ## For teams
 
 - **Distribute it from a managed marketplace.** Add this repo, or your fork of it, to the marketplace your organization already manages, so every developer installs the same reviewed copy.
 - **Pin a version.** Installed copies are cached by `version`. Review a release, then pin it. Upgrade on purpose.
 - **Set the policy in `pluginConfigs`.** Turn on `pii` if your code and logs carry customer data. Use `off` to drop a rule that fires on your own fixtures, by id.
 - **Know the limit of self-install.** A developer can disable any plugin they installed themselves. Org-wide enforcement needs managed settings. See Anthropic's [plugin admin docs](https://code.claude.com/docs/en/plugins/mods/admin).
-- **Read the threat model before rollout.** It is five lines, above. The hook code is [`hooks/register.ts`](plugins/redact/hooks/register.ts) (127 lines) and [`hooks/scrub.ts`](plugins/redact/hooks/scrub.ts) (98 lines). The rules live in [`hooks/rules.ts`](plugins/redact/hooks/rules.ts).
+- **Read the threat model before rollout.** It is five lines, above. The hook code is [`hooks/register.ts`](plugins/redact/hooks/register.ts) (about 130 lines) and [`hooks/scrub.ts`](plugins/redact/hooks/scrub.ts) (under 100 lines). The rules live in [`hooks/rules.ts`](plugins/redact/hooks/rules.ts).
 
 ## Uninstall
 

plugins/redact/DECISIONS.md

@@ -12,9 +12,11 @@
 - No `turn.complete` line and no `$.ui.notice`. Rejected: a per-turn summary, because the per-row `$.ui.log` line already says what was hidden and where; `$.ui.notice` is removed when the tool call resolves, which is before the result row is appended.
 - The scrub logic is a pure module (`hooks/scrub.ts`) tested directly. Rejected: testing it through `session.append` in the kit, because on Claude Code 2.1.288 a test stub for `session.append` cannot answer (every answer must follow a `next`, and nothing is beneath the stub).
 
+## Verified since
+- The `$.ui.log` line draws in the terminal transcript as `redact: hid 1 AWS_KEY in your prompt`; the engine prefixes the plugin name, so the mod's own text carries none.
+- Live interactive session on 2.1.288: the prompt echo showed the placeholder, Claude's `Write` call carried it, the file on disk holds the real value, `/redact` reported one restoration. The session file held the placeholder in every conversation row; the only raw copy was the `toolUseResult` record of the `Write` (the file content as written), which the engine stores as made.
+- An interactive session writes no `queue-operation` record at all; only the headless `-p` path does, and there it is written before `prompt.submit` runs.
+
 ## Assumptions not verified
-- In an interactive session, `prompt.submit` runs before the `queue-operation` record is written, so that record holds the placeholder. In the headless `-p` path it is written first and holds the typed text. If wrong: the typed prompt stays raw in that one bookkeeping record (never sent to the model). How to check: type a prompt with a fake key in an interactive session and grep the session file for `queue-operation`.
 - Subagent rows and subagent `Edit` calls pass through the same hooks with `agentId` set, as the types say. If wrong: a subagent could read a raw secret. How to check: run a subagent that reads a file with a fake key and grep its rows in the session file.
-- The restore path works against a real file edit by the model. The kit proves the argument rewrite; a live run with a logged-in session has not been done. How to check: ask Claude to move a key from `.env` to another file and confirm the new file holds the real value.
 - `$.state` survives `/reload-plugins`, so placeholders made before a reload still restore. If wrong: an edit using an older placeholder fails and Claude re-reads the file. How to check: hide a value, run `/reload-plugins`, ask for an edit of that line.
-- The `$.ui.log` line draws in the terminal transcript. The `-p` path receives it as `ui_log`. If wrong: the person loses the per-row notice; `/redact` still reports totals. How to check: an interactive session with a fake key in the prompt.