perf(telemetry): the ledger is kept by day, so an ingest costs what it adds

aidd_docs/product/usage-contract.md

@@ -76,7 +76,7 @@ The ledger stores `StoredUsage`: the record plus `repository_id`, resolved at in
 | `consent-closed` | its live key names a closed interval; says "run `aidd telemetry on`", never says the clone has not opted in |
 | `unreadable-consent` | git config or `.git` unreadable, directory or git dir the file system will not let ingest look at, clone with no identity, damaged consent log |
 | `no-cwd` | no working directory |
-| `undated` | timestamp not a readable date |
+| `undated` | timestamp not a readable date, with no zone (`Z` or `±hh:mm`), or outside four-digit years |
 
 ## Report envelope
 
@@ -111,7 +111,7 @@ Directory: `AIDD_TELEMETRY_DIR`, else `AIDD_USER_CONFIG_DIR/telemetry`, else `XD
 
 | Store | Writer | Readers | Holds |
 | --- | --- | --- | --- |
-| `ledger/<YYYY-MM>.jsonl` | `ingest`, `report` | `report`, `forget` | `StoredUsage` by month of `at`. Only a changed month is rewritten, whole, by rename; no new bytes, no write |
+| `ledger/<YYYY-MM-DD>.jsonl` | `ingest`, `report` | `report`, `forget` | `StoredUsage` by UTC day of `at`, in time then key order, one call per line. Ingest reads its calls' days and the day on each side, and rewrites only a changed day, whole, by rename; no new bytes, no write. `report` rewrites, folded, the partitions that share a call or hold a line that is not a record. A monthly `<YYYY-MM>.jsonl` from an earlier version is still read, split into its days by the first ingest whose window touches it or by a report, then removed |
 | `ledger/offsets.json` | `ingest`, `report` | same | per transcript: bytes consumed, size, file identity. Shrunk or changed identity: read whole |
 | `ledger/roots.json` | `ingest`, `on` | `ingest`, `forget` | `{version: 2, directories}`: per directory and per clone seen alive there, `repository_id`, `root`, `clone`, `seen_at`. Other format holds nothing: a live directory is remembered again, a gone one counts `never-seen-alive`. `on` records its own root for `forget` |
 | `ledger/consents.jsonl` | `on`, `off`, `ingest`, hooks (close only) | same, plus `forget` | consent intervals, see [Consent](#consent). Outlives the clone; removed with the ledger. Hooks parse it strictly, pinned by the shared fixture |

aidd_docs/tasks/2026_10/2026_10_15_ledger-by-day/contract.md

@@ -0,0 +1,69 @@
+---
+status: in-progress
+issue: 1001
+---
+
+# Ledger by day: an ingest costs what it adds
+
+## Need
+
+One ingested call costs the size of the whole ledger (#1001). Measured on `next` (`0a103e67`), a synthetic month of 265,000 calls (118 MB), one new call:
+
+| Step | Time | Why |
+| --- | --- | --- |
+| `load()` | 2.8 s | reads, validates and folds every partition |
+| `upsertUsage` | 2.7 s | refolds held and incoming together |
+| `save(month)` | 6.0 s | stringifies 265k records, re-reads the file to compare, rewrites it |
+| total | ~11.5 s | |
+
+The round-1 review re-ran the same bench on `next` code, with the real `PrivateStorageAdapter`, and got 0.41 s, 0.37 s and 0.87 s, ~1.65 s in all. The first figure is kept as measured; the setup that gave it is not recorded. On the ledger by day (`9b0f11c5`) the same bench gives 58-64 ms per call, and 1.6-1.7 s for the one-time split of a monthly file. Criterion #1 holds against either "before".
+
+## Decision
+
+Partition the ledger by the UTC **day** of a record's `at` (`ledger/<YYYY-MM-DD>.jsonl`) instead of its month. Everything else stays: a changed partition is rewritten whole by rename, an unchanged one is not touched, records are in canonical order, one call is one line.
+
+An alternative, append-only partitions compacted by `report`, was weighed and set aside. It is faster by about 0.1 s, but it needs a compaction rule, torn-line handling and a fold in every reader, and it gives up one-call-one-line between reports.
+
+## Evidence for the ingest window
+
+Ingest reads only the days of its incoming records, plus the day before and after. Whether that window is enough was measured, read-only, on a real Claude Code history of 3,621 transcripts:
+
+| Measure | Value |
+| --- | --- |
+| usage lines | 493,006 |
+| distinct keys (`message.id:requestId`) | 262,631 |
+| keys written more than once | 177,235 |
+| largest spread between a key's first and last line | 0.3 h |
+| keys whose lines fall on two UTC days | 3 |
+| keys spread over one day or more | 0 |
+
+A key's snapshots are minutes apart at most, so the day before and after holds any earlier snapshot. The key and its `at` come from `domain/formats/claude-transcript-usage.ts:87-96`.
+
+## The case never observed
+
+A key whose snapshots were more than a day apart would be outside the window. Ingest would then write it a second time, in another day, and nothing reported would change: `load()` still folds the whole ledger, so each call counts once, by the unchanged fold rule. What would change is the bytes on disk, which would then depend on the order of ingests.
+
+**`report` therefore restores them.** It already loads every partition. When it finds one call in two partitions, it rewrites the partitions involved, folded. After any report, the ledger is byte-identical to the one a single ingest of the same calls would have written.
+
+## Contract
+
+- **Format.** `ledger/<YYYY-MM-DD>.jsonl`, one `StoredUsage` per line, by the UTC day of `at`, in time then key order. An `at` is usable only when it parses, carries a zone (`Z` or `±hh:mm`) and falls in a four-digit year, so its day never depends on the machine's time zone. An incoming call without one is counted `undated`; a stored line without one is dropped as damage.
+- **Ingest.** It loads the partitions of its records' days and the days on each side, folds only those with the incoming records, and rewrites only the days that changed. Its cost does not grow with the ledger's size.
+- **Report.** It reads every partition, monthly and daily, and folds them. When a call sits in more than one partition, a monthly partition remains, or a partition holds a line that is not a record, it rewrites those partitions in canonical daily form, under the ledger lock.
+- **Migration.** A monthly `<YYYY-MM>.jsonl` is still read. The first ingest whose window touches a month splits it into its days, then removes it.
+  - A crash between the two leaves each call in both files. Fold-at-read counts it once, and the next ingest or report finishes the split.
+- **Damage.** A line that is not a record is still skipped, counted, and dropped when its partition is rewritten, whichever format it sits in.
+- **`forget`** removes daily and monthly partitions alike. `offsets.json` is unchanged.
+- **Unchanged.** The fold rule, the `added` and `updated` counts within the window, every report axis and the report envelope.
+- **Out of scope.** Report cost: `load()` still reads the whole ledger, 2.8 s at this size. It is a separate issue.
+
+## Acceptance
+
+| # | Criterion | Proof |
+| --- | --- | --- |
+| 1 | One call into a 265k-call month ingests in under 1 s | the same bench, before and after |
+| 2 | One set of calls gives byte-identical partitions whatever the order and grouping of its ingests, starting from monthly files or not, once a report has run | fast-check property |
+| 3 | No report total changes across the migration or across a crash at any point of it | tests |
+| 4 | The fold rule is unchanged, and every axis adds up to `totals` | existing suite, plus `scripts/check-claude-tokens-end-to-end.cjs` in both modes |
+| 5 | Every reader of the partitions is updated, and the contract and code map say "day" | grep, plus a fresh checker |
+| 6 | Tests are written first and seen to fail; telemetry mutation stays at 96 or above | mutation run |

aidd_docs/tasks/2026_10/2026_10_15_ledger-by-day/phase-1.md

@@ -0,0 +1,85 @@
+---
+status: done
+---
+
+<!-- Fill or omit these sections; never add, rename, or reorder one. -->
+
+# Instruction: A record belongs to a day, an ingest to a window
+
+## Architecture projection
+
+> Tree of the final files. ✅ create · ✏️ modify · ❌ delete
+
+```txt
+cli/
+├── src/contexts/telemetry/domain/
+│   └── stored-usage.ts                      ✏️ add dayOf, partitionByDay, daysAround (monthOf, partitionByMonth stay until phase 2)
+└── tests/contexts/telemetry/domain/
+    └── stored-usage.unit.test.ts            ✏️ day, partition-by-day and window cases, written first
+```
+
+## User Journey
+
+```mermaid
+flowchart TD
+  A[an incoming call with its at] --> B[dayOf => YYYY-MM-DD in UTC]
+  B --> C[daysAround => the day, the day before, the day after]
+  C --> D[the window an ingest will read in phase 2]
+  B --> E[partitionByDay => canonical bytes of a day]
+```
+
+## Test Scope
+
+```mermaid
+---
+title: Test scope
+---
+journey
+  section Setup
+    build records with known at values across a month and a year boundary => records ready: 5: system
+  section Happy path
+    dayOf of an instant with an offset => the UTC day: 5: system
+    daysAround of calls on two days => each day plus its neighbours, once each: 5: system
+    partitionByDay of shuffled records => days in order, each in time then key order: 5: system
+  section Edge case - unparseable time
+    a record whose at does not parse => dayOf and daysAround and partitionByDay => no day, left out: 1: system
+  section Edge case - boundaries
+    a call at 2026-03-01T00:30Z => daysAround => includes 2026-02-28: 1: system
+    a call at 2026-12-31T23:59Z => daysAround => includes 2027-01-01: 1: system
+```
+
+## Tasks to do
+
+### `1)` Measure "before"
+
+> Record the baseline criterion 1 is judged against, on unchanged code.
+
+1. From the repository root, run the bench as it stands (command in `phase-5.md`, task 1) and keep its per-step timings (`load() total`, `upsert one`, `save(month)`) for the phase 5 comparison. The contract's table (~11.5 s) is the reference if the run differs by machine.
+
+### `2)` Tests first
+
+> Each new test fails because the function does not exist or returns the month.
+
+1. In `stored-usage.unit.test.ts`, a `describe("a record belongs to the day of its own time, in UTC")`: reads the day from the instant; uses UTC at the day boundary whatever the offset (`2026-10-07T23:30:00-02:00` => `2026-10-08`); pads month and day; `null` for a time that is not one. Must fail with "dayOf is not exported".
+2. A `describe("an ingest reads its calls' days and the day on each side")` for `daysAround(records)`: one call => three days; two calls on adjacent days => four days, no duplicate; month boundary (`2026-03-01` => `2026-02-28`) and year boundary (`2026-12-31` => `2027-01-01`); an unparseable `at` contributes nothing; empty input => empty set. Must fail with "daysAround is not exported".
+3. A `describe("a partition is the day's records in a fixed order")` mirroring the month block (lines ~120-176): groups by day and orders by time then key; lists days in order whatever the input order; two records of one instant ordered by key; a record with no usable time in no partition; the linear-copy case kept. Must fail with "partitionByDay is not exported".
+4. Run `pnpm vitest run tests/contexts/telemetry/domain/stored-usage.unit.test.ts` and see exactly those failures.
+
+### `3)` Implement
+
+> Pure transforms, beside the month ones they will replace.
+
+1. `dayOf(at)`: `YYYY-MM-DD` of `Date.parse(at)` in UTC, `null` when it does not parse. Doc comment says why UTC (partition independent of the machine's zone).
+2. `partitionByDay(records)`: same algorithm as `partitionByMonth` (parse once, sort by instant then `tool\0key` with `compareText`, push by key, keys sorted), keyed by `dayOf`.
+3. `daysAround(records)`: for each record with a day, add the day and its UTC neighbours (`Date.parse(day + "T00:00:00.000Z") ± 86_400_000`, sliced back to a day). Doc comment cites the contract's evidence: a call's snapshots are minutes apart, so a neighbour day holds any earlier one.
+4. Re-run the file: green. `pnpm typecheck` green (old functions still used, nothing removed yet).
+
+## Test acceptance criteria
+
+<!-- Each criterion is an observable behavior, not a command. -->
+
+| Task | Acceptance criteria              |
+| ---- | -------------------------------- |
+| 1    | A "before" timing for one call into a 265k-call month is recorded, for contract #1 |
+| 2    | Each new test was seen failing for the missing export, before any implementation (contract #6) |
+| 3    | A call's day is its UTC day; a window holds each call's day and both neighbours across month and year boundaries; a day's records come out in time then key order whatever the input order (contract Format) |