Repository navigation
test(framework): a diagnostic that proves the chain fires, and a period that has met a hundred sessions #694
Description
As someone who turned measurement on
I want to know it is actually recording, and to trust a figure over a real stretch of time
So that an installation that reports itself healthy while producing nothing is impossible rather than merely unlikely
Why now
Everything shipped is verified and none of it has met load. 2614 CLI tests, 177 hook tests and two live probes — against three sessions at most, and a task tree of a handful of files.
"It scales" is currently a hope.
And #617 owns the diagnostic. It is deliberately sequenced after coverage: written before #681 and #680, it would spend its time reporting the two failures already known.
What this holds
A diagnostic — #617's substance, restated against what now exists. Not "is the switch on" but the four questions in order, each with a distinguishable answer:
| Question | Broken looks like |
|---|---|
| Is the hook registered? | the switch is on, sessions run, and no run file appears |
| Is a session journalled? | a run file exists but carries only session_start |
| Can the tool's own files be read? | no session found for every tool while the journal names sessions |
| Do the two join? | records stored, every one of them unattributed |
A period that has met a hundred sessions. Build a sink holding a year of day files and a hundred journalled sessions, then measure and write down: the period read, the sweep, and the turn-end walk over a real task tree. Nothing here has been run against more than three.
A budget for the turn-end walk. The observed pass walks aidd_docs/tasks once per turn, capped at 2000 entries and never measured on a real repository. A cap nobody has timed is a guess with a number on it.
A live multi-step flow. One skill gives two rows. A real SDLC chain gives several, and that is where interval closing, reconciliation across five steps, and interleaved skills stop being unit tests and become observations. Costs one real session on a small task.
Acceptance
- Each of the four failures above is deliberately induced and named as itself, never answered with a zero.
- A period holding a hundred sessions answers, and how long it takes is written down rather than assumed acceptable.
- The turn-end walk is timed on a repository with a real task tree, and its cap is set from that number.
- A live multi-step flow reports each step separately, and the breakdown reconciles to the total.
- The diagnostic never reports healthy while producing nothing — the mode the epic exists to make impossible.
Out of scope
Fixing what the diagnostic finds. Naming a failure is this ticket; #681, #680 and #676 own the failures already known.
Relations
| Field | Value |
|---|---|
| parent | #631 |
| depends_on | #681, #680 |
| related | #617, #686 |
Activity
blafourcade commentedon Aug 21, 2026
All three, done and written down
aidd_docs/tasks/2026_08/2026_08_21_telemetry-v1-close/measurements.mdholds the numbers.A period that has met a hundred sessions
365 day files, 100 sessions, 25 tasks, built through
sink.js's ownappend()and the journal's own line builders — never hand-written JSON, because a hand-written fixture only proves the reader works on hand-written fixtures.assert.equal, no toleranceA one-off at 1000 sessions and 1825 day files: 120ms and 976ms. Under a millisecond a session, amortised.
A finding that came out of it: the sweep's real cost is per-tool subprocess spawns, not the sink. With
$PATHcarrying an installedopencodebinary the same 1000-session sweep is materially slower than with$PATHcleared.The turn-end walk, timed
The premise in this ticket does not hold for this repository. It uses 172 of 2000 entries, 8.6% — not the hundreds of task folders assumed. The cap stays at 2000, now for a measured reason: ~13.5ms p95 to exhaust it, against a turn-end handler that budgets 200ms p95 and already spends ~9ms elsewhere.
Two bugs in the walk, both the exact shape this ticket exists to remove:
if (++seen >= MAX) breakcounted the boundary entry as scanned without ever opening it.truncatedby "is anything still queued" readsfalsewhen the cap lands mid-listing of one wide directory: the queue empties the moment that directory is popped, before its listing is read. Reproduced on a flat 2300-file tree, red then green against the buggy version.And the deeper one: a wide enough directory makes the walk find zero files faster than the healthy case — indistinguishable from an empty period on a stopwatch.
taskFilesModifiedSincenow returns{found, truncated, scanned}and the run file carries ascan_truncatedline, so reaching the cap says so.A live multi-step flow
Three skills in sequence —
05-rule-generate,07-command-generate,01-commit— threestep_startlines, one commit, 18 requests, 265.6s, $2.10.Recomputed independently from
report --json, all five token fields, integer-exact:The fourth row is
unattributed— the planning turn before the first skill call. It stays its own row with its own totals, never folded into the nearest step. That is the promise kept literally rather than approximately.The diagnostic, run against the same session through a different code path, agrees exactly:
records join ok — 17 of 18, 1 unattributed.The first attempt produced nothing, and looked fine
Filed as #703. A project set up entirely through the CLI, every command reporting success, loaded none of its plugins under
claude -p:Skipping orphaned enabledPlugins entry: marketplace not registered, on stderr, behind a debug flag. All three skills answered "Unknown skill" and the session hand-rolled the procedures off disk, which reads as success from outside. $1.27 for nothing.The measurement layer was never at fault — it correctly reported nothing, because nothing happened. But it is the same failure this milestone exists to make impossible, one layer down.
One more thing measured
Codex sets
CODEX_THREAD_IDin the shell a skill spawns, matching the valuerecord.jsalready parses as itsvendor_id, and confirmed not inherited from the launching shell. The diagnostic's session anchor now covers two tools instead of one. Where both variables are present that is the nested case by construction, and Codex's wins by ordering alone — proven with a fixture where the inherited Claude Code id matches a healthy old run file and the diagnostic still correctly reports this session as having left none.Acceptance
Gate: plugin scripts 295 · CLI unit 1,893 · integration 567 · e2e 178 · biome 0 · tsc 0 · 0 broken links.
blafourcade commentedon Aug 22, 2026
Done and awaiting review in #706, which closes this on merge.
The work is on
claude/aidd-telemetry-layer-e403uf, eleven commits, targetingnext. Gate at the time of push: 365 plugin specs, 1,931 CLI unit, 577 integration, 178 e2e,tscand biome clean, no broken markdown links.Read the eleven commits rather than the pull request's file count — the branch carries its own copy of the CLI migration that
nexthas since received, so the diff counts it twice. The range and the real figures are in the first comment on #706: 142 files, +10,518 −418.Every claim about a tool in this work rests on a session that was actually run. The negative results are kept in full in
aidd_docs/tasks/2026_08/2026_08_22_telemetry-every-tool/measurements.md, including the two probes that disagreed about OpenCode and the measurement that reconciled them.blafourcade commentedon Sep 2, 2026
#703's third box moves here, because it is this ticket's first row seen from the other
end.
#703 asks that "a session that resolved no skill from an installed plugin is
distinguishable from one that needed none". Claude Code is
translationMode: "marketplace",so when a plugin fails to register the telemetry plugin fails with it:
journal.cjsneverruns and no run file is written at all — which is what #703 measured, "it produced zero run
files". With no run file there is no session record to distinguish from another, so the only
thing that can answer is the diagnostic this ticket owns:
One thing the merged branch adds to that row, and it is worth carrying: the cause is not
always "the hook was never declared". It can be "the host CLI never registered what the AIDD
CLI declared", which #703 now owns as a comparison of the two install surfaces — the
declaration in the project's settings against
~/.claude/plugins/known_marketplaces.jsonand
installed_plugins.json. That is the earlier and more specific answer, and it costs nosession. This ticket's row should name it rather than stop at "no run file appeared", so a
person is told which of the two failed.
Nothing else here changes. Also for the record, so it is not re-derived:
aidd telemetry checkcurrently reads the declaration only —telemetry-evidence-adapter.ts:71-83— andnever the registry.
test(framework): the read path, met at a hundred sessions (#752)
fix(telemetry): a backlog declaration the report can actually read (#780