Repository navigation
feat(framework): a skill that proves the telemetry pipeline fires #617
Description
Activity
Verified against the five vendors' official docs (2026-08-11)
One agent per tool, one page per claim. Full write-up with citations lives in the dashboard repo at
aidd_docs/memory/kpi/probes/03-hooks-et-identifiants-session.md; the hook-authoring reference is corrected under #618.The finding that changes this issue
None of the five tools documents that the session id seen by a hook is the one in its telemetry export. Not Claude Code, not Codex, not Cursor, not Copilot, and OpenCode has neither. The join this issue assumed is nowhere guaranteed.
One cross-surface join is guaranteed, by Claude Code only: its hook
prompt_id"matches theprompt.idattribute on OpenTelemetry events, so you can correlate hook output with telemetry for a single prompt" (requires 2.1.196+; 2.1.218 observed locally). It is per turn, not per session — which fits an interval model better than the session ever did.Consequence for the scope above: the marker must carry an AIDD-owned run id, with the vendor identifier stored beside it as a typed attribute (
session_idon Claude Code and Codex,conversation_idon Cursor,sessionIdon Copilot — four names). Task-to-execution attribution then owes nothing to a vendor. Cost joining stays borrowed and stays explicit, per tool.Capability per tool
Tool Session structure Cost join Note Claude Code native derived session.idin OTLP, but disableable viaOTEL_METRICS_INCLUDE_SESSION_IDCodex native unknown telemetry key never published; docs say "conversation id" in prose only Cursor native unknown Admin API conversationIdcarries tokens and cost, but is never tied to hooksCopilot native unknown OTLP gen_ai.conversation.idwith cost, off by default; Metrics API is day/user onlyOpenCode unknown unavailable plugins only, no session-end event, no session id in plugin context New acceptance criteria this requires
- A telemetry handler exits
0unconditionally, on every tool. On Copilot apreToolUsehandler that exits non-zero denies the tool call — an observer that throws stops the agent from committing. This is a safety criterion, not hygiene. - The marker records
run_id, the vendor identifier, and the identifier kind. -
statusdetects an installation that is present but inert: Codex hook trust review pending or[features] hooks = false, CopilotdisableAllHooks, Claude CodeOTEL_METRICS_INCLUDE_SESSION_ID=false. - Per-tool support is declared as
native/derived/unavailable/unknown, andunknownblocks that tool rather than shipping a partial claim. - For the pilot, correlate on
prompt_id/prompt.idrather than on session equality.
Smaller facts worth keeping
- Copilot reads the repository's
.claude/settings.jsonfor inline hooks: one Claude-format entry covers two tools. - Codex
SessionEndexists (the earlier claim that it does not was wrong) but its timeout is 1 s, 3 s max, and it does not fire for subagents. - Codex documents subagent propagation explicitly ("Subagent hooks use the parent session id"). Cursor does the opposite: subagents get their own id plus a separate
parent_conversation_id. Claude Code documents neither. - Copilot's built-in
general-purposeagent emits neithersubagentStartnorsubagentStop. - Cursor exposes
beforeShellExecutionwith the command at the top level, cleaner than filtering a generic tool event. - On OpenCode,
tool.execute.beforecan mutateoutput.args.command: the only one of the five where a trailer could be injected without a git hook.
Still unverified, and load-bearing
No vendor documents what happens to the session identifier across a resume, clear, compaction or fork. All five expose a
source/reasondiscriminant at those boundaries and none says whether the value is kept. Measure before relying on it.- A telemetry handler exits
Resserrage pour une v1 testable, 2026-08-14
Échéance du jalon dans une semaine. Cette issue porte deux moitiés indépendantes ; seule la seconde entre dans la v1.
L'hypothèse porteuse est levée
Le troisième critère d'acceptation — « the stamped id matches the
session.idseen in the provider's OpenTelemetry export » — est mesuré, sur quatre outils.Outil Champ côté hook Attribut côté export Même valeur Coût du test Claude Code session_idsession.idoui, deux sessions deux sessions réelles Codex CLI session_idconversation.idsurcodex.sse_eventoui zéro token GitHub Copilot sessionIdgen_ai.conversation.idsurinvoke_agentoui zéro crédit OpenCode — session.idsurai.streamTextexport prouvé une session Les identifiants sont fabriqués côté client, avant l'appel au modèle : un fournisseur pointé sur une adresse morte démarre quand même la session, fait tirer le hook et émet la télémétrie. Codex et Copilot se vérifient donc gratuitement, ce qui met cette vérification dans l'intégration continue plutôt que dans une case à cocher de fin de parcours.
Le risque a changé de place
Aucune sonde n'a fonctionné du premier coup, et jamais pour une raison différente : chaque outil verrouille ses hooks à sa façon (détail dans #618). Un hook installé sans lever son verrou est muet et ne lève aucune erreur.
C'est exactement ce que cette issue voulait attraper avec « an installed-but-inert state must read as failed ». La campagne montre que ce n'est pas un cas limite mais le cas nominal. Donc
statusne doit pas vérifier qu'un fichier existe, mais qu'un hook a tiré.Ce qui reste dans cette issue pour la v1
La commande d'état, et elle seule. Une ligne par affirmation vérifiable, jamais un verdict agrégé, avec la détection des états inertes par outil.
Ce qui en sort
Le trailer de commit — marqueur
PreToolUse,prepare-commit-msg, détection du gestionnaire de hooks. Raison : le journal des exécutions (#620) couvre déjà les sessions sans commit, qui sont les plus chères et les plus mal comptées. Le trailer ajoute de la précision par commit, pas la capacité à mesurer. Suivi dans une issue dédiée.Contradiction interne à corriger au passage
La section « Scope » fait écrire au marqueur « the
session_idfrom the event payload », alors que la décision du 2026-08-12 pose que le trailer porte lerun_idAIDD. Deux valeurs pour un même trailer.Sur #585
Déclaré bloquant, mais
.aidd/config.ymln'existe dans aucun code et la CLI n'a aucun analyseur YAML. La v1 lit.aidd/telemetry.json, format déjà manipulé par la CLI. Quand #585 arrive, c'est une ligne de lecture à déplacer.Plan :
aidd_docs/plans/2026_08_14-telemetry-v1/.- changed the title
[-]feat(cli): stamp commits with the AI session id, installed and verified by the CLI[/-][+]feat(cli): prove the telemetry pipeline actually fires, per tool[/+]on Aug 14, 2026 - added a commit that references this issue
on Aug 14, 2026 - added a parent issue
on Aug 14, 2026 - changed the title
[-]feat(cli): prove the telemetry pipeline actually fires, per tool[/-][+]feat(framework): a skill that proves the telemetry pipeline fires[/+]on Aug 14, 2026 The skill exists:
aidd-telemetry/skills/02-checkFive claims, each independently checkable, none a summary of the others. What it prints on a healthy install:
hook fired ok 1 run file(s), most recent session_start 2026-08-20T09:00:00Z session journalled ok 1 of 1 run file(s) carry more than session_start tool files readable ok claude: 1 of 1 session(s) read; codex: 0 of 1 session(s) read records join ok 1 of 1 record(s) joined a step, 0 unattributed not covered: cursor -- It writes no token count in any file it produces.With measurement off it stops before checking anything, so four failures caused by the switch are never reported as four failures.
Every failure is induced, and proved distinguishable
Not "a FAIL appears" — exactly one claim reads FAIL per induced fault, asserted by a helper that fails if a second one does. That is the difference between a diagnostic and a smoke alarm that goes off in every room.
hook fired FAIL no run file in aidd_docs/runs — the hook has never been observed firing hook fired FAIL this session left no run file — the newest one is from 2026-07-01T09:00:00Z session journalled FAIL 1 run file(s), all carrying only session_start — nothing closed the turn tool files readable FAIL no session found for any journalled session, across every covered tool records join FAIL 1 record(s) found, joined: 0 — every record unattributedDownstream claims read
--rather than FAIL when an earlier one already failed —records join -- no record read to join. A cascade of failures reads as one fault, not four.The acceptance criterion that nearly did not land
A hook whose last observed write is older than the current session's start reads FAIL.
The first pass set this aside as needing an invented staleness threshold. It does not — but nor is it a clock comparison, which is what I asked for and was wrong about. Any journal write necessarily precedes the moment a later process reads it, so "newest write versus now" is true on every healthy run.
It is an identity question. The diagnostic knows which session it is running in, so the check is whether that session left a run file.
CLAUDE_CODE_SESSION_IDsupplies the anchor. Without it a project whose hook broke a month ago readsokforever — the exact silent health this skill exists to remove — and the fixture proving it is one that reads completely healthy on all four other claims.Bound, stated in the code: only Claude Code exposes such a variable today. Under the others the claim reads
--with "no session anchor available", neverok. And the same variable was already measured to be inherited by a Codex process spawned inside a Claude Code session; this script has no payload to disambiguate with, so under genuine nesting the anchor can be a stale inherited one.Scope
The skill runs only its own script, never the CLI and never another skill's — the same rule the other two follow, with a test enforcing it across all three. Uncovered tools are named from the same
TOOLSdeclaration the readers use, so a tool gained or lost changes in one place. That declaration is now physically copied into this skill, for reasons and with a guard described in #702.Gate
node --test "scripts/__tests__/*.test.js"— 286 pass, 0 fail.check-markdown-links— 0 broken in 766 files.#694 holds the rest: a period that has met a hundred sessions, the turn-end walk timed on a real repository, and a live multi-step flow. Both are in progress on the same branch.
Done and awaiting review in #706, which closes this on merge.
The work is on
claude/aidd-telemetry-layer-e403uf, eleven commits, targetingnext. Gate at the time of push: 365 plugin specs, 1,931 CLI unit, 577 integration, 178 e2e,tscand biome clean, no broken markdown links.Read the eleven commits rather than the pull request's file count — the branch carries its own copy of the CLI migration that
nexthas since received, so the diff counts it twice. The range and the real figures are in the first comment on #706: 142 files, +10,518 −418.Every claim about a tool in this work rests on a session that was actually run. The negative results are kept in full in
aidd_docs/tasks/2026_08/2026_08_22_telemetry-every-tool/measurements.md, including the two probes that disagreed about OpenCode and the measurement that reconciled them.- added 4 commits that reference this issue
on Aug 22, 2026 This issue's own "Why a skill and not a CLI command" argued the opposite of what shipped:
"The CLI keeps one job only: turning the export on. Everything that reads belongs to the
plugin."aidd telemetry checkis now a CLI command, not a skill running its own script.The reversal is a direct product decision, recorded as this plan's own source
(aidd_docs/tasks/2026_08/2026_08_26_cli-owns-the-read/plan.md): move every part of the
plugin that reads — cost reporting and this diagnostic alike — behind one CLI
implementation, after measuring that the plugin's own second implementation of the report
was a maintenance liability (two places a figure could disagree) with no corresponding
cost, since only the write path (the hooks) is what fires per tool call and has to stay
dependency-free.02-check's own skill still owns when to run it and how to present
what it prints; the plugin ships no script of its own any more.The mechanism this issue asked for is unaffected by where it runs. It still answers from
inside a running session:resolveSessionAnchorreads whichever ofCODEX_THREAD_ID/
CLAUDE_CODE_SESSION_IDthe shell inherits, and the six-claim diagnostic below is
per-session, not a fleet-wide report.hook fired ok 1 run file(s), most recent session_start ... session journalled ok 1 of 1 run file(s) carry more than session_start tool files readable ok ... records join ok ... export configured ok ... identifier joinable ok ...Checked against every item in this issue's own acceptance list:
- One line per independently verifiable claim, no summary line.
- A hook whose configuration exists but never fired reads FAIL, not ok.
- A run file older than the current session's start reads FAIL, distinct from never
having fired. -
OTEL_METRICS_INCLUDE_SESSION_ID=falseon an otherwise complete export reads FAIL,
naming the exact setting. - The four uncovered tools are named explicitly, each with its own reason.
- Reports disabled until the project has opted in (
measurement is off — nothing to check until it is turned on). - No model quota consumed, no network call beyond the local sink.
Closing as done, under
aidd telemetry checkrather than a plugin-only skill — the design
question this issue raised is answered, just not the way it originally proposed.- added a commit that references this issue
on Aug 27, 2026
As a developer who has just installed AIDD telemetry
I want a skill that tells me, line by line, whether each link actually produces data
So that I never build a decision on figures an inert installation left incomplete
Acceptance
OTEL_METRICS_INCLUDE_SESSION_ID=falsecase.Expected output
Why a skill and not a CLI command
The diagnostic answers a question asked from inside a session, about the session that is running. A skill can read the run journal and see whether the current session has a file — which is precisely the "did the hook fire" check, and it is stronger from inside than from outside.
The CLI keeps one job only: turning the export on. Everything that reads belongs to the plugin.
Why this is the most important issue in the milestone
The measurement campaign (#632) found that no probe worked on the first attempt, and never for a different reason. Every tool gates its hooks differently. A hook installed without lifting the gate is silent and raises nothing.
That is not an edge case, it is the nominal case. Hence the central requirement: the skill does not check that a file exists, it checks that a hook fired.
Out of scope
Relations