fix(telemetry): one call, one key; one repository, one id

aidd_docs/product/usage-contract.md

@@ -56,15 +56,15 @@ The ledger stores `StoredUsage`: the record plus `repository_id`, resolved at in
 
 | Rule | Value |
 | --- | --- |
-| Key | `message.id:requestId`; without `requestId`, `message.id:sessionId:timestamp`; each advisor iteration adds `#advisor<n>` |
+| Key | `message.id:requestId`; without `requestId`, `message.id:sessionId` (a message id is unique per call; nothing bounds how far apart its copies are written, so an ingest storing such a call reads every day, and the ledger holds it once, except copies an older CLI stores after the migration, which only a report with no period folds); each advisor iteration adds `#advisor<n>` |
 | Fold | a call is written several times (streamed snapshots, resumed sessions, sub-agents); one record per `(tool, key)`: largest sum of the four counters (`null` as `0` for this comparison only) |
 | Tie | earliest `at`, then lowest `session_id`, then serialised record. No input order changes the winner |
 | Dropped | model `<synthetic>` |
 | `cache_write` | summed over `message` iterations (the top-level split reflects only the first) |
 | Advisor | each `advisor_message` iteration is a record, `agent: advisor`, own model |
 | Sub-agent | `isSidechain` or an `agentId` |
 
-**Repository.** The call's working directory resolved at ingest: its git root, named by the hash of the `origin` address (`host/owner/repo`), else the root commit. The address is never kept.
+**Repository.** The call's working directory resolved at ingest: its git root, named by the hash of the `origin` address (`host/owner/repo`), else the root commit. The address is never kept. Every form (https, `ssh://`, `git://`, scp-like) gives one address: scheme, credentials, port, `.git` and trailing `/` dropped, host case-folded, `ssh.github.com` read as `github.com`. Owner and repo are case-folded on `github.com` only, which documents them as not case sensitive; any other host keeps their case, so two spellings there may still give two ids.
 
 **Not stored.** Storing follows [Consent](#consent). A call read and left out counts in `run.not_stored` of the report that read it, under one reason, and nothing of it is kept, except an `undated` call of a clone that consents (see [Left out](#left-out)):
 
@@ -124,7 +124,7 @@ Row and `totals`: `records`, `input`, `output`, `cache_read`, `cache_write` (eac
 - `AIDD_TELEMETRY=0`: nothing read, object is `{"version": 2, "refused": "AIDD_TELEMETRY=0"}`.
 - The text report labels an `absent` person `anonymous`; the envelope keeps `kind: absent`, `key: null`.
 - A report ingests first; what that read found is its `run`, see [the envelope reference](../../plugins/aidd-telemetry/skills/01-usage/references/envelope.md#run). `coverage.records` describes the period; `coverage.left_out` every ingest so far.
-- A report with `--days`, `--from` or `--to` reads only the days of its period and the day on each side, plus any monthly file one of them falls in, with every day of its month; with none it reads the whole ledger. The copies of one call have been observed less than a day apart, and under that bound both print the same figures. Outside it, only a report with no period folds every copy of a call. `run.skipped_ledger_lines` counts the lines of the partitions read.
+- A report with `--days`, `--from` or `--to` reads only the days of its period and the day on each side, plus any monthly file one of them falls in, with every day of its month; with none it reads the whole ledger. The copies of a call with a `requestId` have been observed less than a day apart, and under that bound both print the same figures; a call with none is held once by the ingest that stores it, so no bound is needed. Outside these, only a report with no period folds every copy of a call: copies an older CLI stores after the migration are one such case. `run.skipped_ledger_lines` counts the lines of the partitions read.
 - No path, working directory or branch. `session` keys are full session ids, `repository` keys are hashes.
 
 ### Filters
@@ -161,15 +161,16 @@ Directory: `AIDD_TELEMETRY_DIR`, else `AIDD_USER_CONFIG_DIR/telemetry`, else `XD
 
 | Store | Writer | Readers | Holds |
 | --- | --- | --- | --- |
-| `ledger/<YYYY-MM-DD>.jsonl` | `ingest`, `report` | `report`, `forget` | `StoredUsage` by UTC day of `at`, in time then key order, one call per line. Ingest reads its calls' days and the day on each side, and rewrites only a changed day, whole, by rename; no new bytes, no write. `report` rewrites, folded, the partitions it read that share a call or hold a line that is not a record; a report with no period reads every partition, so after it each call is on disk once. A line that is not a record (a newer version's, a hand edit, a torn one) is never removed: a rewrite keeps it in its day, after the records, in text order and once, and counts it in `skipped_ledger_lines` on every read. A day holding only such lines stays. A monthly `<YYYY-MM>.jsonl` from an earlier version is still read, split into its days by the first ingest or report whose window touches it, or by a report with no period, then removed; its lines that are not records go to the first day of its month |
+| `ledger/<YYYY-MM-DD>.jsonl` | `ingest`, `report` | `report`, `forget` | `StoredUsage` by UTC day of `at`, in time then key order, one call per line. Ingest reads its calls' days and the day on each side (every day, and so every monthly file, when one has no `requestId`), and rewrites only a changed day, whole, by rename; no new bytes, no write. `report` rewrites, folded, the partitions it read that share a call or hold a line that is not a record; a report with no period reads every partition, so after it each call is on disk once. A line that is not a record (a newer version's, a hand edit, a torn one) is never removed: a rewrite keeps it in its day, after the records, in text order and once, and counts it in `skipped_ledger_lines` on every read. A day holding only such lines stays. A monthly `<YYYY-MM>.jsonl` from an earlier version is still read, split into its days by the first ingest or report whose window touches it, or by a report with no period, then removed; its lines that are not records go to the first day of its month |
 | `ledger/left-out.txt` | `ingest` | `report`, `forget` | what ingests measured and could not store, by hash, see [Left out](#left-out). cli-v5.5.0 never reads it |
+| `ledger/repository-ids.json` | `ingest`, `report` | every reader of the ledger, `roots.json` and `branches.jsonl` | `{version: 1, complete, aliases, pending}`: the id an earlier version gave a repository, to its id now, both hashes or root commits. Made from each root `roots.json` names, by its `origin` now. Every read re-keys a Claude Code call stored as `message.id:sessionId:timestamp` and re-ids by this table; the first ingest or report also rewrites `roots.json` and every ledger partition so, then marks it `complete`. A run stopped part way is finished by the next with the same table. A root that could not be read, or was not there, stays in `pending`: every later ingest looks at those roots alone and, when one adds an id, rewrites the stores again. Until then its stored calls keep the earlier id while its new calls get the id of now: two ids for one repository. An earlier version reading migrated data reads it as written; what it stores afterwards is re-keyed, and re-ided when its repository is in the table, at every read. A requestless call it stores then may sit in two days: a report with a period may count that copy, until a report with no period settles the ledger. `branches.jsonl` is never rewritten, so it relies on this table for good: a damaged table is made again from the clones still there, and a snapshot of a repository none of whose clones remains keeps its earlier id, so the calls of now no longer find its declaration. An `undated` hash in `left-out.txt` keeps its earlier key |
 | `ledger/offsets.json` | `ingest`, `report` | same | per transcript: bytes consumed, size, file identity. Shrunk or changed identity: read whole |
 | `ledger/roots.json` | `ingest`, `on` | `ingest`, `forget` | `{version: 2, directories}`: per directory and per clone seen alive there, `repository_id`, `root`, `clone`, `seen_at`, for every directory a read call or line named, in any clone. Other format holds nothing: a live directory is remembered again, a gone one counts `never-seen-alive`. `on` records its own root for `forget` |
 | `ledger/consents.jsonl` | `on`, `off`, `ingest`, hooks (close only) | same, plus `forget` | consent intervals, see [Consent](#consent). Outlives the clone; removed with the ledger. Hooks parse it strictly, pinned by the shared fixture |
 | `ledger/.lock` | CLI | CLI | `{pid, created_at}`; dead process or older than ten minutes: cleared |
 | `bindings/.lock` | CLI | CLI | same format. Declarations and snapshots run under it; `task` never takes the ledger lock, which an ingest holds while reading transcripts |
 | `bindings/sessions.jsonl` | `task` | hooks, `report`, `task` | `{session_id, task, ticket, none, declared_at, by}`, `by` = `command` \| `hook-intercept` |
-| `bindings/branches.jsonl` | `task`, `ingest`, `report` | `report`, `forget` | branch declaration at a moment, with when the branch took its name, from its reflog |
+| `bindings/branches.jsonl` | `task`, `ingest`, `report` | `report`, `forget` | branch declaration at a moment, with when the branch took its name, from its reflog. Read with `repository-ids.json` applied, never rewritten |
 | `bindings/carries.jsonl` | `SessionStart` hook | hooks, `report` | `{session_id, from, at}` |
 | `bindings/processes.jsonl` | `SessionStart` hook | `SessionStart` hook | `{pid, session_id, source, at}` |
 | `identity.json` | `identity`, `on` | `report`, `status`, `on` | `{"person_id": "<id>"}`, owner-only |

aidd_docs/tasks/2026_10/2026_10_17_one-call-one-key-one-repository-one-id/contract.md

@@ -0,0 +1,104 @@
+---
+status: done
+---
+
+# One call, one key; one repository, one id
+
+## Need
+
+Found by the R1 telemetry audit. Maintainer decision: migrate the stored data, the measurement is a beta.
+
+| # | Hole | Where |
+| --- | --- | --- |
+| M7 | One repository gets several ids. The owner and repository keep their case on every host, and an `ssh://` or `git://` host keeps its case too (WHATWG `URL` folds only special schemes): `git@github.com:Example/X.git`, `ssh://git@GitHub.com/example/x` and `https://github.com/example/x` give three ids. | `domain/repository-identity.ts` |
+| M10 | A call with no `requestId` is keyed `message.id:sessionId:timestamp`. Its copies carry different timestamps, so they are never folded and the call is counted once per copy. | `domain/formats/claude-transcript-usage.ts` |
+
+## Rules, before and after
+
+| Rule | Before (cli-v5.5.0 to `616dbbea`) | After |
+| --- | --- | --- |
+| Scheme | dropped | dropped |
+| Userinfo | dropped | dropped |
+| Port | dropped (a scp-like address has none) | dropped |
+| Host | case-folded for `http(s)`, `ws(s)`, `ftp` and scp-like; kept for `ssh://`, `git://`, `git+ssh://` | case-folded on every form ([RFC 3986 §3.2.2](https://www.rfc-editor.org/rfc/rfc3986#section-3.2.2): the host is case-insensitive) |
+| `ssh.github.com` | its own host | `github.com`: GitHub's SSH over port 443 answers there for the same repositories ([GitHub docs](https://docs.github.com/en/authentication/troubleshooting-ssh/using-ssh-over-the-https-port)) |
+| Owner and repository | case kept | `github.com`: case-folded (GitHub REST: owner and repo, "The name is not case sensitive", [docs](https://docs.github.com/en/rest/repos/repos#get-a-repository)). Any other host, GitHub Enterprise included: case kept, since no primary source says otherwise there; two spellings may then still give two ids, never one id for two repositories |
+| `.git` suffix, leading and trailing `/` | dropped | dropped, after case-folding where the host folds it |
+| No origin | root commit | root commit |
+| Key, with `requestId` | `message.id:requestId` | unchanged |
+| Key, without `requestId` | `message.id:sessionId:timestamp` | `message.id:sessionId` |
+| Advisor iteration | key + `#advisor<n>` | unchanged |
+
+**Why `message.id:sessionId` is one key per call.** The Messages API: `id` is a "Unique object identifier" of the message ([reference](https://platform.claude.com/docs/en/api/messages)), so no two calls share a `message.id`, and none share it with a session added. The session stays in the key: dropping it, or `requestId`, would re-key every stored call and every `undated` hash, which this fix does not take on (see Limits).
+
+## Store inventory
+
+Everything under the telemetry directory, and what this fix does to it.
+
+| Store | Holds an id or a key | Migration |
+| --- | --- | --- |
+| `ledger/<YYYY-MM-DD>.jsonl` | `repository_id`, `key` | every read re-keys and re-ids; the first run rewrites them all, folded |
+| `ledger/<YYYY-MM>.jsonl` (earlier layout) | `repository_id`, `key` | same; split into days by that rewrite, as any full settle does |
+| `ledger/left-out.txt` | `undated <hash>` = hash of tool and key | none: the hash cannot be undone (see Limits). `unrecognised` hashes a line's text: no id, no key |
+| `ledger/roots.json` | `repository_id` per directory and clone | every read re-ids; the first run rewrites it |
+| `ledger/repository-ids.json` | earlier id to id, both hashes | new: the migration's journal and alias table |
+| `ledger/offsets.json` | transcript path to position | none |
+| `ledger/consents.jsonl` | token, clone, time | none |
+| `ledger/.lock`, `bindings/.lock` | pid, time | none |
+| `bindings/branches.jsonl` | `repository_id` per snapshot | every read re-ids; never rewritten, it stays append-only |
+| `bindings/sessions.jsonl`, `carries.jsonl`, `processes.jsonl` | session ids | none |
+| `identity.json` | person id | none |
+| git config `aidd.telemetry`, `branch.<name>.aidd*` | token, task | none |
+| plugin hooks (`plugins/aidd-telemetry/hooks/lib/`) | compute no repository id and no key | none |
+
+## Migration
+
+```mermaid
+flowchart TD
+  S["ingest or report, under the ledger lock"] --> J{"repository-ids.json complete?"}
+  J -->|"yes, none pending"| I["ingest as usual"]
+  J -->|"yes, roots pending"| P["locate the pending roots alone"]
+  P -->|"no new id"| I
+  P -->|"a new id"| W1
+  J -->|absent| M["locate each root roots.json names: earlier id to id, from its origin now"]
+  M --> W1["write repository-ids.json, not complete"]
+  J -->|not complete| R
+  W1 --> R["rewrite roots.json and every ledger partition: re-keyed, re-ided, folded"]
+  R --> W2["write repository-ids.json, complete"]
+  W2 --> I
+```
+
+| Requirement | How |
+| --- | --- |
+| Automatic | first step of every ingest, so of every report and of the plugin's hook |
+| Deterministic | the alias table is a function of each root's `origin` now, written before anything else; a rerun reuses it |
+| Idempotent | a key re-keys at most once (the new shape never ends in its own `:at`); an alias never names an id that is itself aliased: an earlier address differs from today's only by an upper-case letter or `ssh.github.com`, which no address of today holds |
+| Crash-safe | under the ledger lock; every write is the existing write-then-rename; the table is written first and marked complete last, so a run stopped at any step leaves it incomplete and the next run redoes the whole rewrite. Exercised: a failure at every write (`replace`). Reasoned, not exercised: a stop during the removals that follow the writes, and a `kill -9` |
+| Read-time | every reader of the ledger, `roots.json` and `branches.jsonl` applies the key rule and the table, so a state the migration has not finished, or lines an older CLI writes afterwards, still count each call once under one id in a report with no period; a bounded report may count a requestless copy an older CLI stored after the migration (see Limits) |
+| No double count | copies of a call re-keyed alike fold, by the existing fold, to the copy one ingest would have kept |
+
+**An older CLI on migrated data** (cli-v5.5.0): the lines have the same shape, so it reads and reports them as they are, and ignores `repository-ids.json`. What it ingests afterwards it stores under its own rules; the newer CLI then re-keys it, and re-ids it when its repository is in the table, at every read. A requestless call it stores is not held once: a report with no period settles it.
+
+## Acceptance
+
+1. `normaliseRemote` folds every form in the table above to one address, and keeps owner and repo case for any host but `github.com`.
+2. A requestless call's copies at different timestamps are one record.
+3. For any pre-migration store (mixed-case ids, timestamped keys, copies across days, partitions and the earlier monthly layout), the migrated report and ledger bytes equal those of one fresh ingest under the new rules.
+4. Migrating twice equals once; a failure at every write, then a rerun, equals a clean run.
+5. The id and key in `roots.json` and the ledger are rewritten on disk by the first run; `branches.jsonl` is aliased at read.
+6. Gates: `cd cli && pnpm test`, `node --test 'scripts/__tests__/**/*.test.js'`, `pnpm test:mutation:telemetry` at 96 or more, `pnpm smoke`, the tokens end-to-end checks, bundle within 709 KB.
+
+## Limits
+
+| Limit | Effect |
+| --- | --- |
+| A root unreadable or absent at the migration | kept pending and looked at again by every ingest; until it can be read, its stored calls keep the earlier id and its new calls get the new one: two ids. A clone deleted for good stays pending, at the cost of one look per ingest |
+| `repository-ids.json` damaged | made again from the clones still there; `branches.jsonl`, never rewritten, then loses the alias of a repository with no clone left, and its declarations stop matching the calls of now |
+| Requestless copies more than a day apart | measured: no requestless line in 512,820 real usage lines (3,837 transcripts); copies of a message with a request id, 184,502 of them, at most 0.27 h apart. Nothing bounds the requestless ones, so storing one reads the whole ledger (measured by the checker at 250,000 records: 1,672 ms, against 13 ms with a request id) and the ledger holds it once |
+| Requestless lines an older CLI writes after the migration | keyed by their timestamp, stored in their own day: a report with a period may count such a copy as well as the call's other copy, until a report with no period settles the ledger. Needs a downgrade or a pinned older binary, and no requestless line was measured |
+| A root whose `origin` changed since its calls were stored | the table maps the address it has now, so those calls keep their id, as before |
+| `undated` hashes in `left-out.txt` from before (unreleased: cli-v5.5.0 has no such file) | cannot be re-keyed; copies of one undated requestless call kept apart stay counted apart, and one read again from the start of its transcript adds a hash beside the old one |
+| A requestless call copied into another session | two keys, as before; seen copies across sessions carry a `requestId`, keyed without the session |
+| One call seen with and without a `requestId` | two keys, as before |
+| A host alias from `~/.ssh/config` (`git@work-github:o/r`) | its own host: nothing on disk says what it stands for |
+| Percent-encoding in a URL path | kept as written; a scp-like address is never encoded |