diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md new file mode 100644 index 000000000..3f8c7464f --- /dev/null +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -0,0 +1,54 @@ +# Observe agent telemetry locally + +Run Claude Code, Codex and OpenCode v2 with OpenTelemetry (OTel), then inspect the local Gouvernail dashboard. For OpenCode, an opted-in AIDD plugin converts public session events to OTLP; the standard exporter does not supply the session-linked usage observed by this plugin. The framework's separate local usage ledger is not the source for this view. + +## Steps to observe agent telemetry locally + +### 1) โš™๏ธ Prepare the collector destination + +Start local Gouvernail and configure each tool's connector. A passkey-verified administrator generates tokens in Sources for an explicit team, member and installation UUID. Rotating a token replaces the previous token for that tool and organization. The member is the token's declared assignee, not proof of the person using it. Existing shared tokens retain their previous declarative attribution. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. + +| Tool | Connector token | +| --- | --- | +| Claude Code | Claude Code OTLP connector | +| Codex | Codex OTLP connector | +| OpenCode v2 | OpenCode OTLP connector for the selected team | + +`--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. The file remains in the system temporary directory and can be removed by system cleanup; copy it to a protected location if it must be retained. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. + +The new assigned tokens require `--include-installation-id`, which sends the existing AIDD installation UUID as an AIDD-added `aidd.installation.id` resource attribute for that execution. It is an installation identifier, not an authenticated person. This flag fails if no installation UUID exists. Captures remain provider data and do not contain this added attribute; the global `otel init` configuration does not add it. + +### 2) ๐Ÿงช Send two messages in each session + +Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each command. Choose models available to your accounts. These examples opt in to sending the installation UUID. + +```sh +# Claude connector token +aidd telemetry otel run --config claude --model haiku --tenant ai-driven-dev \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture --include-installation-id \ + --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." + +# Codex connector token +aidd telemetry otel run --config codex --model gpt-6-luna --tenant ai-driven-dev \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture --include-installation-id \ + --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." + +# OpenCode connector token +aidd telemetry otel run --config opencode --model claude-code/claude-haiku-4-5-20251001 \ + --tenant ai-driven-dev --endpoint http://127.0.0.1:55857 --yes --keep-capture --include-installation-id \ + --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." +``` + +Each command uses an isolated collector and resumes the first session for the follow-up. Claude and Codex persist these conversations in their usual local session history; OpenCode uses a temporary database removed after the run. This history is used only by the tools to resume; Gouvernail receives OTLP, not the tools' session files. Verify both responses when checking continuity: a shared session ID alone does not prove the model used the first message. + +The command waits for an OTLP usage observation after each execution. Claude identifies the matching prompt explicitly; OpenCode supplies a completed step and message ID. Codex turn spans observed with CLI 0.166.0 carry session, turn and usage together. Turn usage can cover multiple requests or models; zero cache and output values can be provider defaults. Do not add the same turn's span and response-event counters. The run sends logs before spans and excludes spans from a different session without inferring a parent relationship. An emitted observation does not guarantee that every requested field is available. `otel init` and the global development collector are a separate configuration path; generic `otel sync` does not yet forward Codex turn spans. + +### 3) โœ… Check receipt + +Open the local AI KPI page and check the session IDs and observed fields. Delivery does not prove that every event was emitted or received. For OpenCode, the plugin reports the selected model and positive normalized token counts in separate events; an ambiguous zero stays unavailable. Missing or uncertain values remain unavailable. See the [field inventory](../../plugins/aidd-telemetry/OBSERVABILITY.md). + +```sh +open 'http://localhost:55857/enterprise/ai-driven-dev/kpi/sessions' +``` + +Sources accessed 2026-11-06: [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage), [Codex telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted), [OpenCode v2 CLI](https://dev.opencode.ai/v2/docs/cli) and [OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts). diff --git a/cli/aidd_docs/memory/codebase-map.md b/cli/aidd_docs/memory/codebase-map.md index 752a48546..f03fa23a7 100644 --- a/cli/aidd_docs/memory/codebase-map.md +++ b/cli/aidd_docs/memory/codebase-map.md @@ -64,6 +64,7 @@ src/ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # a clone's identity, and when it consented โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ identity/ # the identifier a person chose to be named by โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ legacy/ # what the previous version wrote, by name and location, and its key and id rules +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ observations/ # native telemetry fields before display or aggregation โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ports/ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ bindings/ # the declarations kept beside the ledger, and the lock their writers share โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ forget/ @@ -72,6 +73,8 @@ src/ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ report/ # attribution of a call, the days of a report, the axes it splits by โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ switch/ # the previous version's config block, Claude's retention โ”‚ โ”‚ โ””โ”€โ”€ infrastructure/ +โ”‚ โ”‚ โ”œโ”€โ”€ otel/ # shared native OTLP settings; JSON templates in config/ +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ actions/ # initialize configuration, launch export, verify receipt โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # reading a clone's identity from the file system, and the consent history file โ”‚ โ”‚ โ”œโ”€โ”€ declaration/ # where a declaration is kept: git config and the telemetry dir โ”‚ โ”‚ โ”œโ”€โ”€ forget/ # erasing the telemetry dir, the previous version's files, git config keys diff --git a/cli/assets/configs/otel-opencode-plugin.js.txt b/cli/assets/configs/otel-opencode-plugin.js.txt new file mode 100644 index 000000000..648f3acb5 --- /dev/null +++ b/cli/assets/configs/otel-opencode-plugin.js.txt @@ -0,0 +1,125 @@ +// OpenCode server plugin v2.2.5+. The framework enables it only for an opted-in run. +// Public event stream: https://dev.opencode.ai/v2/docs/build/plugins/server (2026-11-06). +const types = ["session.created", "session.step.started", "session.step.ended", "session.step.failed"]; + +const text = (key, value) => ({ key, value: { stringValue: value } }); +const integer = (key, value) => ({ key, value: { intValue: String(value) } }); +const id = (value, prefix) => + typeof value === "string" && new RegExp(`^${prefix}_[A-Za-z0-9]{1,64}$`).test(value); +const label = (value) => + typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/@+#-]{0,199}$/.test(value); + +function destination() { + const endpoint = process.env.AIDD_OPENCODE_OTEL_LOGS_ENDPOINT; + const authorization = /^Authorization=Bearer ([A-Za-z0-9_-]+)$/.exec( + process.env.AIDD_OPENCODE_OTEL_HEADERS ?? "", + ); + if (!endpoint || !authorization) return; + try { + const url = new URL(endpoint); + const loopback = ["127.0.0.1", "localhost", "[::1]"].includes(url.hostname); + if ((url.protocol !== "https:" && !(url.protocol === "http:" && loopback)) || + url.username || url.password || url.search || url.hash || !url.pathname.endsWith("/v1/logs")) return; + return { url: url.href, authorization: `Bearer ${authorization[1]}` }; + } catch { + return; + } +} + +// Source: https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts +// Usage's absent-to-zero normalization: https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts (2026-11-06). +function record(event) { + const data = event?.data; + const durable = event?.durable; + if (!types.includes(event?.type) || !id(event.id, "evt") || !id(data?.sessionID, "ses") || + !Number.isSafeInteger(event.created) || event.created <= 0 || + !Number.isSafeInteger(durable?.seq) || durable.seq < 0 || + !Number.isSafeInteger(durable?.version) || durable.version < 1 || + durable.aggregateID !== data.sessionID) return; + if (event.type !== "session.created" && !id(data.assistantMessageID, "msg")) return; + const nanos = BigInt(event.created) * 1000000n; + if (nanos > 18446744073709551615n) return; + const attributes = [ + text("event.name", event.type), + text("event.id", event.id), + text("session.id", data.sessionID), + integer("opencode.event.sequence", durable.seq), + integer("opencode.event.version", durable.version), + ]; + if (event.type === "session.created") { + if (label(data.projectID)) attributes.push(text("opencode.project.id", data.projectID)); + if (id(data.parentID, "ses")) attributes.push(text("opencode.parent.session.id", data.parentID)); + } else { + attributes.push(text("assistant.message.id", data.assistantMessageID)); + if (event.type === "session.step.started") { + if (label(data.model?.providerID)) attributes.push(text("opencode.model.provider", data.model.providerID)); + if (label(data.model?.id)) attributes.push(text("opencode.model.name", data.model.id)); + if (label(data.model?.variant)) attributes.push(text("opencode.model.variant", data.model.variant)); + if (label(data.agent)) attributes.push(text("opencode.agent.name", data.agent)); + } else { + const counters = [ + ["opencode.usage.input", data.tokens?.input], + ["opencode.usage.output", data.tokens?.output], + ["opencode.usage.reasoning", data.tokens?.reasoning], + ["opencode.usage.cache_read", data.tokens?.cache?.read], + ["opencode.usage.cache_write", data.tokens?.cache?.write], + ]; + for (const [key, value] of counters) { + // A zero may mean the upstream provider omitted the value. + if (Number.isSafeInteger(value) && value > 0) attributes.push(integer(key, value)); + } + } + } + return { timeUnixNano: String(nanos), attributes }; +} + +export default { + id: "aidd.otel", + setup(ctx) { + const target = destination(); + if (!target) { + if (process.env.AIDD_OPENCODE_OTEL_LOGS_ENDPOINT || process.env.AIDD_OPENCODE_OTEL_HEADERS) + console.warn("AIDD OpenTelemetry: the OpenCode exporter configuration is invalid."); + return; + } + const controller = new AbortController(); + const pending = new Set(); + let warned = false; + const warn = () => { + if (!warned) console.warn("AIDD OpenTelemetry: an OpenCode observation could not be exported."); + warned = true; + }; + const version = label(ctx.app?.version) ? ctx.app.version : undefined; + const resource = [text("service.name", "opencode")]; + if (version) resource.push(text("service.version", version)); + void (async () => { + try { + for await (const event of ctx.event.subscribe({ types, signal: controller.signal })) { + if (controller.signal.aborted) break; + const log = record(event); + if (!log) continue; + if (pending.size >= 64) { warn(); continue; } + const body = JSON.stringify({ resourceLogs: [{ + resource: { attributes: resource }, + scopeLogs: [{ scope: { name: "aidd.opencode.otel", version: "1" }, logRecords: [log] }], + }] }); + const work = fetch(target.url, { + method: "POST", + headers: { "content-type": "application/json", authorization: target.authorization }, + body, + redirect: "error", + signal: AbortSignal.timeout(1500), + }).then((response) => { if (!response.ok) warn(); }).catch(warn); + pending.add(work); + void work.finally(() => pending.delete(work)); + } + } catch { + if (!controller.signal.aborted) warn(); + } + })(); + return async () => { + controller.abort(); + await Promise.allSettled([...pending]); + }; + }, +}; diff --git a/cli/package.json b/cli/package.json index ccf272438..f4c933dc0 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 747, + "bundleBudgetKB": 839, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index fd2d69e9e..0eddbb0ca 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -55,8 +55,29 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // (+5.2 KB over its 712.9 KB skills profile); 0.3% measured headroom. // 722 KB: 720.2 KB, Antigravity portable agents and direct-local installation // (+2.1 KB over its 718.1 KB hooks build); 0.3% measured headroom. +// 728 KB: 726.7 KB, explicit native OTLP configuration for three tools and the +// OpenCode 1 launcher (+15.9 KB over the 710.8 KB build above); 0.2% measured headroom. +// 746 KB: 744.9 KB, provider observation transformers and a persistent installation ID +// (+18.2 KB over the 726.7 KB build above); 0.1% measured headroom. +// 784 KB: 783.7 KB, local OTLP collector and consent-aware resumable provider sync, +// explicit metric sanitization, retry and durable feature invalidation (+38.8 KB over 744.9 KB); +// 0.1% measured headroom. +// 799 KB: 798.8 KB with opt-in Codex titles, one-pass title synchronization and +// telemetry integrity guards (+15.1 KB over 783.7 KB); less than 0.1% measured headroom. +// 804 KB: 803.3 KB with strict OTLP attribute and metric validation (+4.5 KB); +// 0.1% measured headroom. +// 807 KB: 806.6 KB with opt-in native Codex project metadata and shared metadata publishing; +// +3.3 KB, less than 0.1% measured headroom. +// 808 KB: 807.49 KB with bounded, event-scoped Codex reasoning-effort forwarding; +// +0.53 KB over the preceding 806.96 KB build, less than 0.1% measured headroom. +// 810 KB: 809.79 KB with separately opted-in, sanitized Codex repository metadata. +// 813 KB: 812.66 KB with consent and session-admission checks on every delivery attempt. +// 819 KB: 818.39 KB with per-identifier provenance and isolated pending session ranges. +// 829 KB: 828.93 KB with repository uncertainty and clone-proven resume after origin changes. +// 839 KB: 838.97 KB with Claude token-point effort and merged Antigravity support. // 747 KB: 732.4 KB, the machine-wide install of `setup --global` (+12.2 KB over the 720.2 KB // above); 2.0% measured headroom. +// Merged with next: 823.5 KB, within the existing 839 KB budget. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/scripts/smoke-tools.sh b/cli/scripts/smoke-tools.sh index fe1cb498c..c58c508ef 100755 --- a/cli/scripts/smoke-tools.sh +++ b/cli/scripts/smoke-tools.sh @@ -30,6 +30,7 @@ ALL_COMMANDS=( "auth login" "auth logout" "auth status" "telemetry on" "telemetry off" "telemetry forget" "telemetry ingest" "telemetry task" "telemetry status" "telemetry report" "telemetry identity" + "telemetry otel init" "telemetry otel verify" "telemetry otel run" "telemetry otel collect" ) PASS=0; FAIL=0; SKIP=0 @@ -66,7 +67,7 @@ project_tree_hash() { PARENTS=" plugin marketplace auth framework telemetry " # A parent one level deeper than PARENTS, so a covered key needs three words there, not two. -GRANDPARENTS=" " +GRANDPARENTS=" telemetry otel " derive_key() { local first="$1" second="${2:-}" third="${3:-}" if [[ "$GRANDPARENTS" == *" $first $second "* ]]; then @@ -165,6 +166,8 @@ export AIDD_TOKEN="$TOKEN" # HOME does not move Codex: it reads that variable and falls back to the real `~/.codex`. export HOME="$TMPROOT/home"; mkdir -p "$HOME" export CODEX_HOME="$TMPROOT/codex-home"; mkdir -p "$CODEX_HOME" +export XDG_CONFIG_HOME="$HOME/.config" +unset OPENCODE_CONFIG OPENCODE_CONFIG_DIR OPENCODE_CONFIG_CONTENT # โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• # OFFLINE / LOCAL โ€” runs without a token @@ -225,6 +228,69 @@ run "telemetry forget (preview)" 0 "Nothing was removed" "$P_TEL" -- telemetry f run "telemetry forget --yes" 0 "Removed" "$P_TEL" -- telemetry forget --yes run "telemetry off (after forget)" 0 "Measurement was not on" "$P_TEL" -- telemetry off +section "provider OTEL (sandboxed profile)" +OTEL_CAPTURE="$HOME/.local/share/aidd-otel-observation" +mkdir -p "$OTEL_CAPTURE" "$TMPROOT/otel-bin" +chmod 700 "$OTEL_CAPTURE" +printf '%s\n' 'test-token' > "$OTEL_CAPTURE/collector-token" +chmod 600 "$OTEL_CAPTURE/collector-token" +run "telemetry otel init (Claude)" 0 "Updated" "$P_TEL" -- \ + telemetry otel init --config=claude --endpoint=dev +printf '%s\n' '{"signal":"logs","payload":{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"attributes":[{"key":"event.name","value":{"stringValue":"user_prompt"}}]}]}]}]}}' > "$OTEL_CAPTURE/claude.jsonl" +run "telemetry otel verify (Claude)" 0 "1 normalized observations" "$P_TEL" -- \ + telemetry otel verify --config=claude --endpoint=dev +# Exercise the actual built collector on a free loopback port using only the test profile. +if node - "$CLI" "$OTEL_CAPTURE/claude.jsonl" >"$TMPROOT/collector-smoke.log" 2>&1 <<'NODE' +const { spawn } = require("node:child_process"); +const { readFile } = require("node:fs/promises"); +const [cli, capture] = process.argv.slice(2); +const child = spawn(process.execPath, [cli, "telemetry", "otel", "collect", "--port", "0"]); +let output = ""; +child.stderr.on("data", data => { output += data; }); +const port = new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error("collector did not start")), 10000); + child.once("exit", () => { clearTimeout(timer); reject(new Error(`collector exited: ${output}`)); }); + child.stdout.on("data", data => { + output += data; + const match = output.match(/listening on http:\/\/127\.0\.0\.1:(\d+)/); + if (match) { clearTimeout(timer); resolve(match[1]); } + }); +}); +(async () => { + try { + const url = `http://127.0.0.1:${await port}/claude/v1/logs`; + const before = await readFile(capture, "utf8"); + const denied = await fetch(url, { method: "POST", body: "{}" }); + if (denied.status !== 403 || await readFile(capture, "utf8") !== before) throw new Error("unauthorized capture was not rejected"); + const accepted = await fetch(url, { method: "POST", headers: { Authorization: "Bearer test-token" }, body: "{}" }); + const lines = (await readFile(capture, "utf8")).trim().split("\n"); + const recorded = JSON.parse(lines.at(-1)); + if (accepted.status !== 200 || lines.length !== 2 || recorded.signal !== "logs" || JSON.stringify(recorded.payload) !== "{}") throw new Error("authorized capture was not recorded"); + if (output.includes("test-token")) throw new Error("collector printed its token"); + } catch (error) { console.error(error); process.exitCode = 1; } + finally { child.kill("SIGTERM"); } +})(); +NODE +then mark_covered "telemetry otel collect"; ok "telemetry otel collect (authenticated local capture)" +else bad "telemetry otel collect (authenticated local capture)" "$(cat "$TMPROOT/collector-smoke.log")"; fi +# The fake executable validates the environment handed to OpenCode 2 without using a real +# provider, changing the user's configuration, or relying on an installed OpenCode version. +cat > "$TMPROOT/otel-bin/opencode" <<'SCRIPT' +#!/usr/bin/env bash +if [[ "${1:-}" == "--version" ]]; then echo '2.2.5'; exit 0; fi +exit 1 +SCRIPT +chmod 700 "$TMPROOT/otel-bin/opencode" +SMOKE_ORIGINAL_PATH="$PATH" +export PATH="$TMPROOT/otel-bin:$PATH" +unset OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS +run "telemetry otel init (OpenCode 2)" 0 "Restart the OpenCode background service" "$P_TEL" -- \ + telemetry otel init --config=opencode --endpoint=dev +unset AIDD_OTEL_TOKEN +run "telemetry otel run (no execution consent)" 1 "Pass --yes to consent" "$P_TEL" -- \ + telemetry otel run --config=codex --tenant=smoke --endpoint=http://127.0.0.1:4318 -- "reply OK" +export PATH="$SMOKE_ORIGINAL_PATH" + section "update --check" out=$(cd "$ROOT" && node "$CLI" update --check 2>&1); rc=$? if [[ "$rc" -eq 0 || "$rc" -eq 1 ]]; then mark_covered "update"; ok "update --check (exit $rc)"; else bad "update crashed (exit $rc)" "$out"; fi diff --git a/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts b/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts new file mode 100644 index 000000000..672df5070 --- /dev/null +++ b/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts @@ -0,0 +1,70 @@ +import type { AiToolId } from "../../../../kernel/tool.js"; + +export type JsonValue = + | null + | boolean + | number + | string + | readonly JsonValue[] + | { readonly [key: string]: JsonValue }; + +/** A source observation. Missing values stay missing; identical observations are not merged. */ +export interface TelemetryObservation { + readonly tool: AiToolId; + readonly signal: "log" | "span" | "span-event" | "metric"; + readonly eventType?: string; + readonly timestamp?: string; + readonly observedTimestamp?: string; + readonly startedAt?: string; + readonly endedAt?: string; + readonly toolVersion?: string; + readonly sessionId?: string; + readonly threadId?: string; + readonly turnId?: string; + readonly promptId?: string; + readonly clientRequestId?: string; + readonly messageId?: string; + readonly traceId?: string; + readonly spanId?: string; + readonly parentSpanId?: string; + readonly requestId?: string; + readonly provider?: string; + readonly model?: string; + readonly prompt?: string; + readonly response?: string; + readonly toolCallId?: string; + readonly runtimeToolCallId?: string; + readonly toolName?: string; + readonly toolInput?: JsonValue; + readonly toolOutput?: JsonValue; + readonly success?: boolean; + readonly durationMs?: number; + readonly cwd?: string; + readonly gitBranch?: string; + readonly tokens?: { + readonly input?: number; + readonly inputNonCached?: number; + readonly output?: number; + readonly cacheRead?: number; + readonly cacheWrite?: number; + readonly cacheWrite5m?: number; + readonly cacheWrite1h?: number; + readonly reasoning?: number; + readonly total?: number; + readonly tool?: number; + }; + /** Snapshots and aggregates must not be summed as model calls. */ + readonly usageScope?: "event" | "span-snapshot" | "metric-aggregate"; + readonly tokenInputIncludesCache?: boolean; + readonly tokenOutputIncludesReasoning?: boolean; + /** Codex may report zero for cache counts missing from provider usage. */ + readonly cacheZeroMayMeanMissing?: boolean; + /** As reported by the source, not a calculated or validated price. */ + readonly reportedCostUsd?: number; + readonly reportedCostUsdMicros?: number; + readonly metricName?: string; + readonly metricType?: string; + readonly metricValue?: number; + /** The original observation, retained for fields not yet selected for a display. */ + readonly source?: JsonValue; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts new file mode 100644 index 000000000..aa85590ee --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts @@ -0,0 +1,165 @@ +import { randomBytes, timingSafeEqual } from "node:crypto"; +import { constants } from "node:fs"; +import { lstat, mkdir, open, writeFile } from "node:fs/promises"; +import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; +import { join } from "node:path"; +import { gunzipSync } from "node:zlib"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import { OTEL_CONFIGS, readToken } from "../settings.js"; + +const MAX_BYTES = 32 * 1024 * 1024; +const MAX_CLIENTS = 16; +const PATH = /^\/([^/]+)\/v1\/(logs|metrics|traces)$/; +const CAPTURE_TOOLS = new Set(OTEL_CONFIGS); + +async function tokenFor(home: string, directory: string): Promise { + await mkdir(directory, { recursive: true, mode: 0o700 }); + const info = await lstat(directory); + if (!info.isDirectory() || (info.mode & 0o077) !== 0) { + throw new Error("OTLP capture directory must be private."); + } + const path = join(directory, "collector-token"); + try { + await writeFile(path, `${randomBytes(32).toString("base64url")}\n`, { + flag: "wx", + mode: 0o600, + }); + } catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "EEXIST")) throw error; + } + return readToken(home); +} + +function authorized(request: IncomingMessage, token: string): boolean { + const given = request.headers.authorization ?? ""; + const expected = `Bearer ${token}`; + if (!/^[\x20-\x7e]+$/.test(given)) return false; + const a = Buffer.from(given, "ascii"); + const b = Buffer.from(expected, "ascii"); + return a.length === b.length && timingSafeEqual(a, b); +} + +class RequestError extends Error { + constructor(readonly status: number) { + super("Invalid OTLP request."); + } +} + +async function readPayload(request: IncomingMessage): Promise> { + const length = Number(request.headers["content-length"] ?? 0); + if (!Number.isSafeInteger(length) || length < 0) throw new RequestError(400); + if (length > MAX_BYTES) throw new RequestError(413); + const chunks: Buffer[] = []; + let size = 0; + for await (const chunk of request) { + const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + size += buffer.length; + if (size > MAX_BYTES) throw new RequestError(413); + chunks.push(buffer); + } + if (size === 0) throw new RequestError(400); + let raw = Buffer.concat(chunks); + const encoding = (request.headers["content-encoding"] ?? "").toLowerCase(); + if (encoding && encoding !== "identity" && encoding !== "gzip") throw new RequestError(415); + if (encoding === "gzip") { + try { + raw = gunzipSync(raw, { maxOutputLength: MAX_BYTES }); + } catch { + throw new RequestError(400); + } + } + let parsed: unknown; + try { + parsed = JSON.parse(raw.toString("utf8")); + } catch { + throw new RequestError(400); + } + const payload = asPlainObject(parsed); + if (!payload) throw new RequestError(400); + return payload; +} + +async function appendCapture(path: string, line: string): Promise { + const file = await open( + path, + constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW, + 0o600 + ); + try { + if (!(await file.stat()).isFile()) throw new Error("Capture must be a regular file."); + await file.chmod(0o600); + await file.writeFile(line, "utf8"); + } finally { + await file.close(); + } +} + +/** A local authenticated OTLP/HTTP JSON collector. It never records request headers. */ +export async function startOtelCollector(options: { + readonly home: string; + readonly port: number; +}) { + if (process.platform === "win32") + throw new Error("OTLP collection requires private POSIX files."); + if (!Number.isInteger(options.port) || options.port < 0 || options.port > 65535) + throw new Error("Collector port must be between 0 and 65535."); + const directory = join(options.home, ".local/share/aidd-otel-observation"); + const token = await tokenFor(options.home, directory); + for (const tool of OTEL_CONFIGS) { + await appendCapture(join(directory, `${tool}.jsonl`), ""); + } + let clients = 0; + let queue = Promise.resolve(); + const append = (path: string, line: string) => { + const work = queue.then(() => appendCapture(path, line)); + queue = work.catch(() => {}); + return work; + }; + const server = createServer((request, response) => { + if (clients >= MAX_CLIENTS) { + response.writeHead(503).end(); + return; + } + clients++; + void handle(request, response) + .finally(() => { + clients--; + }) + .catch(() => {}); + }); + server.requestTimeout = 10_000; + server.headersTimeout = 10_000; + async function handle(request: IncomingMessage, response: ServerResponse): Promise { + try { + if (!authorized(request, token)) throw new RequestError(403); + const path = PATH.exec(new URL(request.url ?? "/", "http://127.0.0.1").pathname); + if (request.method !== "POST" || !path || !CAPTURE_TOOLS.has(path[1])) + throw new RequestError(404); + const payload = await readPayload(request); + const line = `${JSON.stringify({ received_at: new Date().toISOString(), signal: path[2], payload })}\n`; + if (Buffer.byteLength(line) > MAX_BYTES) throw new RequestError(413); + await append(join(directory, `${path[1]}.jsonl`), line); + response.writeHead(200, { "content-type": "application/json" }).end("{}"); + } catch (error) { + if (!response.headersSent && !response.destroyed) + response.writeHead(error instanceof RequestError ? error.status : 500).end(); + } + } + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(options.port, "127.0.0.1", () => { + server.off("error", reject); + resolve(); + }); + }); + const address = server.address(); + if (!address || typeof address === "string") + throw new Error("Collector did not open a TCP port."); + return { + port: address.port, + close: () => + new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())) + ), + }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts new file mode 100644 index 000000000..93c107f6d --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts @@ -0,0 +1,55 @@ +import { randomUUID } from "node:crypto"; +import { mkdir, open, rename, rm, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; +import { readSettings } from "../files.js"; +import { + type Config, + configurationEdit, + type Edit, + type Endpoint, + type Host, + OPENCODE_CONFIG_ID, +} from "../settings.js"; +import { installOpenCodePlugin } from "./opencode-plugin-file.js"; + +async function applyEdit(edit: Edit): Promise { + const before = await readSettings(edit.path, edit.initial); + const next = edit.update(before.content); + if (next === before.content && before.private) return false; + + await mkdir(dirname(edit.path), { recursive: true, mode: 0o700 }); + const lockPath = `${edit.path}.aidd-otel.lock`; + const lock = await open(lockPath, "wx", 0o600); + try { + const current = await readSettings(edit.path, edit.initial); + if ( + current.content !== before.content || + current.private !== before.private || + current.exists !== before.exists + ) { + throw new Error("OTLP settings changed during initialization. Retry with the updated file."); + } + const temporary = `${edit.path}.${randomUUID()}.tmp`; + try { + await writeFile(temporary, next, { flag: "wx", mode: 0o600 }); + await rename(temporary, edit.path); + } finally { + await rm(temporary, { force: true }); + } + } finally { + await lock.close(); + await rm(lockPath, { force: true }); + } + return true; +} + +export async function initializeOtel(config: Config, endpoint: Endpoint, host: Host) { + const edit = await configurationEdit(config, endpoint, host); + if (config === OPENCODE_CONFIG_ID) { + const before = await readSettings(edit.path, edit.initial); + edit.update(before.content); + } + const pluginChanged = config === OPENCODE_CONFIG_ID ? await installOpenCodePlugin(host) : false; + const settingsChanged = await applyEdit(edit); + return { path: edit.path, launch: edit.launch, changed: pluginChanged || settingsChanged }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts new file mode 100644 index 000000000..fbc32773c --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts @@ -0,0 +1,50 @@ +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; + +const KEY = "aidd.installation.id"; +const UUID_V4 = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +/** A provider must not override the installation declared by the AIDD runner. */ +export function checkInstallationClaim(value: unknown, id: string): void { + if (!UUID_V4.test(id)) throw new Error("Installation ID is invalid."); + if (Array.isArray(value)) { + for (const item of value) checkInstallationClaim(item, id); + return; + } + const object = asPlainObject(value); + if (!object) return; + if (object.key === KEY && asPlainObject(object.value)?.stringValue !== id) + throw new Error("Conflicting OTLP installation ID."); + for (const item of Object.values(object)) checkInstallationClaim(item, id); +} + +/** Label an already selected transfer; it is an installation claim, not a user identity. */ +export function attachInstallationId( + payload: unknown, + field: "resourceLogs" | "resourceMetrics" | "resourceSpans", + id: string +): Record { + checkInstallationClaim(payload, id); + const source = asPlainObject(payload); + const resources = source?.[field]; + if (!source || !Array.isArray(resources)) throw new Error("Invalid OTLP resource envelope."); + return { + ...source, + [field]: resources.map((value) => { + const entry = asPlainObject(value); + if (!entry) throw new Error("Invalid OTLP resource."); + const resource = asPlainObject(entry.resource) ?? {}; + const attrs = resource.attributes; + if (attrs !== undefined && !Array.isArray(attrs)) throw new Error("Invalid OTLP attributes."); + return { + ...entry, + resource: { + ...resource, + attributes: [ + ...(attrs ?? []).filter((item) => asPlainObject(item)?.key !== KEY), + { key: KEY, value: { stringValue: id } }, + ], + }, + }; + }), + }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts new file mode 100644 index 000000000..03e4e4263 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts @@ -0,0 +1,62 @@ +import { constants } from "node:fs"; +import { lstat, mkdir, open, readdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import openCodePlugin from "../../../../../../assets/configs/otel-opencode-plugin.js.txt"; +import { type Host, openCodeConfigDir } from "../settings.js"; + +export const openCodePluginDirectory = (host: Host) => + join(openCodeConfigDir(host), "plugins", "aidd-otel"); + +export async function hasOpenCodePlugin(host: Host): Promise { + const directory = openCodePluginDirectory(host); + try { + const info = await lstat(directory); + if (!info.isDirectory() || info.isSymbolicLink()) + throw new Error("Invalid OpenCode plugin directory."); + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return false; + throw error; + } + for (const name of await readdir(directory)) { + if ( + name === "package.json" || + name === "server" || + name.startsWith("server.") || + (name !== "index.js" && (name === "index" || name.startsWith("index."))) + ) + throw new Error("OpenCode telemetry plugin has a competing entry point."); + } + try { + const file = await open( + join(directory, "index.js"), + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); + try { + const info = await file.stat(); + if ( + !info.isFile() || + (info.mode & 0o022) !== 0 || + (await file.readFile("utf8")) !== openCodePlugin + ) + throw new Error("OpenCode telemetry plugin is unsafe or has different content."); + return true; + } finally { + await file.close(); + } + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return false; + throw error; + } +} + +export async function installOpenCodePlugin(host: Host): Promise { + if (await hasOpenCodePlugin(host)) return false; + const directory = openCodePluginDirectory(host); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const info = await lstat(directory); + if (!info.isDirectory() || info.isSymbolicLink()) + throw new Error("Invalid OpenCode plugin directory."); + if (await hasOpenCodePlugin(host)) return false; + await writeFile(join(directory, "index.js"), openCodePlugin, { flag: "wx", mode: 0o600 }); + return true; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts new file mode 100644 index 000000000..a126a250e --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts @@ -0,0 +1,162 @@ +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import { OPENCODE_CONFIG_ID } from "../settings.js"; + +type Fields = Map; +const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); +const object = (value: unknown) => asPlainObject(value) ?? {}; +const string = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const integer = (key: string, value: number) => ({ key, value: { intValue: String(value) } }); +const label = (value: unknown): value is string => + typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/@+#-]{0,199}$/u.test(value); +const id = (value: unknown, prefix: string): value is string => + typeof value === "string" && new RegExp(`^${prefix}_[A-Za-z0-9]{1,64}$`).test(value); + +function attributes(value: unknown): Fields { + const result: Fields = new Map(); + const seen = new Set(); + for (const item of list(value)) { + const entry = object(item); + const data = object(entry.value); + if (typeof entry.key !== "string" || seen.has(entry.key)) return new Map(); + seen.add(entry.key); + const raw = data.stringValue ?? data.intValue; + if (typeof raw === "string" || typeof raw === "number") result.set(entry.key, raw); + } + return result; +} + +function numeric(value: unknown, minimum: number): number | undefined { + if (typeof value !== "number" && (typeof value !== "string" || !/^\d+$/.test(value))) return; + const result = Number(value); + if (Number.isSafeInteger(result) && result >= minimum) return result; + return undefined; +} + +/** Select only the plugin's documented technical fields; never forward raw provider logs. */ +export type OpenCodePluginLog = { session: string; parent?: string; payload: unknown }; + +export function openCodePluginLogs(line: string): OpenCodePluginLog[] { + const envelope = object(JSON.parse(line)); + if (envelope.signal !== "logs") return []; + const selected: OpenCodePluginLog[] = []; + for (const rawResource of list(object(envelope.payload).resourceLogs)) { + const resource = object(rawResource); + const resourceAttributes = attributes(object(resource.resource).attributes); + if (resourceAttributes.get("service.name") !== OPENCODE_CONFIG_ID) continue; + const version = resourceAttributes.get("service.version"); + for (const rawScope of list(resource.scopeLogs)) { + const scope = object(rawScope); + if (object(scope.scope).name !== "aidd.opencode.otel" || object(scope.scope).version !== "1") + continue; + for (const rawRecord of list(scope.logRecords)) { + const record = object(rawRecord); + const values = attributes(record.attributes); + const name = values.get("event.name"); + const eventId = values.get("event.id"); + const session = values.get("session.id"); + const message = values.get("assistant.message.id"); + const seq = numeric(values.get("opencode.event.sequence"), 0); + const eventVersion = numeric(values.get("opencode.event.version"), 1); + const time = record.timeUnixNano; + if ( + typeof name !== "string" || + ![ + "session.created", + "session.step.started", + "session.step.ended", + "session.step.failed", + ].includes(name) || + !id(eventId, "evt") || + !id(session, "ses") || + (name !== "session.created" && !id(message, "msg")) || + seq === undefined || + eventVersion !== 1 || + typeof time !== "string" || + !/^[1-9]\d{0,19}$/.test(time) || + BigInt(time) > 18446744073709551615n + ) + continue; + const safe = [ + string("event.name", name), + string("event.id", eventId), + string("session.id", session), + integer("opencode.event.sequence", seq), + integer("opencode.event.version", eventVersion), + ]; + if (typeof message === "string" && name !== "session.created") + safe.push(string("assistant.message.id", message)); + const strings = + name === "session.created" + ? ["opencode.project.id", "opencode.parent.session.id"] + : name === "session.step.started" + ? [ + "opencode.model.provider", + "opencode.model.name", + "opencode.model.variant", + "opencode.agent.name", + ] + : []; + for (const key of strings) { + const value = values.get(key); + if (label(value) && (key !== "opencode.parent.session.id" || id(value, "ses"))) + safe.push(string(key, value)); + } + if (name === "session.step.ended" || name === "session.step.failed") { + for (const key of [ + "opencode.usage.input", + "opencode.usage.output", + "opencode.usage.reasoning", + "opencode.usage.cache_read", + "opencode.usage.cache_write", + ]) { + const value = numeric(values.get(key), 1); + if (value !== undefined) safe.push(integer(key, value)); + } + } + const resourceAttrs = [string("service.name", OPENCODE_CONFIG_ID)]; + if (label(version)) resourceAttrs.push(string("service.version", version)); + const parent = values.get("opencode.parent.session.id"); + selected.push({ + session, + ...(name === "session.created" && id(parent, "ses") ? { parent } : {}), + payload: { + resourceLogs: [ + { + resource: { attributes: resourceAttrs }, + scopeLogs: [ + { + scope: { name: "aidd.opencode.otel", version: "1" }, + logRecords: [{ timeUnixNano: time, attributes: safe }], + }, + ], + }, + ], + }, + }); + } + } + } + return selected; +} + +/** Keep only descendants whose parent is explicitly present in a session.created observation. */ +export function relatedOpenCodeLogs(root: string, logs: OpenCodePluginLog[]): OpenCodePluginLog[] { + const parents = new Map>(); + for (const log of logs) { + if (!log.parent) continue; + const candidates = parents.get(log.session) ?? new Set(); + candidates.add(log.parent); + parents.set(log.session, candidates); + } + const known = new Set([root]); + let changed = true; + while (changed) { + changed = false; + for (const [session, candidates] of parents) { + if (known.has(session) || candidates.size !== 1 || !known.has([...candidates][0])) continue; + known.add(session); + changed = true; + } + } + return logs.filter((log) => known.has(log.session)); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts new file mode 100644 index 000000000..909ae4e7f --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts @@ -0,0 +1,58 @@ +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; + +const invalid = () => new Error("Invalid OTLP attribute value."); +const specialDoubles = new Set(["NaN", "Infinity", "-Infinity"]); + +/** Keep AnyValue's oneof type until it has been validated; never coerce containers. */ +export function readAttributeValue(raw: unknown, scalarRequired: boolean): string | undefined { + if (!scalarRequired && (raw === undefined || raw === null)) return undefined; + const value = asPlainObject(raw); + if (!value) throw invalid(); + const fields = Object.entries(value); + if (fields.length === 0 && !scalarRequired) return undefined; + const first = fields.at(0); + if (fields.length !== 1 || !first) throw invalid(); + const [kind, item] = first; + if (kind === "stringValue") { + if (typeof item !== "string") throw invalid(); + return item; + } + if (kind === "boolValue") { + if (typeof item !== "boolean") throw invalid(); + return String(item); + } + if (kind === "intValue") return readInteger(item); + if (kind === "doubleValue") { + if (typeof item === "number" && Number.isFinite(item)) return String(item); + if (!scalarRequired && typeof item === "string" && specialDoubles.has(item)) return item; + throw invalid(); + } + if (scalarRequired) throw invalid(); + if (kind === "bytesValue") { + if (typeof item !== "string") throw invalid(); + return undefined; + } + if (kind === "arrayValue" || kind === "kvlistValue") { + const container = asPlainObject(item); + if (!container || (container.values !== undefined && !Array.isArray(container.values))) + throw invalid(); + for (const child of container.values ?? []) { + const nested = kind === "arrayValue" ? child : asPlainObject(child)?.value; + readAttributeValue(nested, false); + } + return undefined; + } + // Profiling dictionary references and unknown optional extensions are not log values. + return undefined; +} + +function readInteger(value: unknown): string { + if (typeof value === "number") { + if (!Number.isSafeInteger(value)) throw invalid(); + } else if (typeof value !== "string" || !/^-?\d{1,20}$/.test(value)) { + throw invalid(); + } + const integer = BigInt(value); + if (integer < -9223372036854775808n || integer > 9223372036854775807n) throw invalid(); + return String(value); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts new file mode 100644 index 000000000..601cb5c62 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -0,0 +1,671 @@ +import { spawn } from "node:child_process"; +import { randomBytes, randomUUID } from "node:crypto"; +import { createReadStream } from "node:fs"; +import { mkdir, mkdtemp, open, rename, rm, writeFile } from "node:fs/promises"; +import { homedir, tmpdir } from "node:os"; +import { join } from "node:path"; +import { createInterface } from "node:readline"; +import openCodePlugin from "../../../../../../assets/configs/otel-opencode-plugin.js.txt"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import claude from "../config/claude.json" with { type: "json" }; +import codex from "../config/codex.json" with { type: "json" }; +import { + CLAUDE_CONFIG_ID, + CODEX_CONFIG_ID, + type Config, + OPENCODE_CONFIG_ID, + readToken, +} from "../settings.js"; +import { startOtelCollector } from "./collect.js"; +import { attachInstallationId, checkInstallationClaim } from "./installation-attribute.js"; +import { hasOpenCodePlugin } from "./opencode-plugin-file.js"; +import { + type OpenCodePluginLog, + openCodePluginLogs, + relatedOpenCodeLogs, +} from "./opencode-plugin-logs.js"; +import { destination, sessionIdsInCapture, syncOtel } from "./sync.js"; + +type RunOptions = { + config: Config; + endpoint: string; + tenant: string; + token: string; + cwd: string; + prompt: string; + followUp?: string; + includeRepository?: boolean; + installationId?: string; + model?: string; + retainCapture?: (path: string) => void; + signal: AbortSignal; +}; + +async function provider(options: RunOptions, collector: string, token: string, directory: string) { + const env = Object.fromEntries( + Object.entries(process.env).filter( + ([key]) => !key.startsWith("OTEL_") && key !== "AIDD_OTEL_TOKEN" + ) + ); + env.AIDD_TELEMETRY = "0"; + const model = options.model ? ["--model", options.model] : []; + if (options.config === CLAUDE_CONFIG_ID) { + const settings: Record = {}; + for (const [key, value] of Object.entries(claude.env)) { + settings[key] = value.replace("{collector}", collector).replace("{token}", token); + } + for (const key of Object.keys(claude.defaults)) { + if (key.startsWith("OTEL_LOG_")) settings[key] = "0"; + } + settings.OTEL_TRACES_EXPORTER = "none"; + settings.OTEL_METRIC_EXPORT_INTERVAL = "1000"; + settings.OTEL_LOGS_EXPORT_INTERVAL = "1000"; + settings.OTEL_METRICS_INCLUDE_REPOSITORY = options.includeRepository ? "true" : "false"; + const file = join(directory, "claude-settings.json"); + await writeFile(file, JSON.stringify({ env: settings }), { mode: 0o600, flag: "wx" }); + return { + env: { ...env, ...settings }, + args: [ + "--print", + ...(options.followUp ? ["--session-id", randomUUID()] : ["--no-session-persistence"]), + "--settings", + file, + ...model, + "--", + options.prompt, + ], + }; + } + if (options.config === CODEX_CONFIG_ID) { + const overrides = Object.entries(codex.exporters).flatMap(([key, url]) => [ + "-c", + `otel.${key}={otlp-http={endpoint="${url.replace("{collector}", collector)}",protocol="json",headers={authorization="Bearer ${token}"}}}`, + ]); + return { + env, + args: [ + "exec", + ...(options.followUp ? ["--json"] : ["--ephemeral"]), + ...model, + ...overrides, + "-c", + "otel.log_user_prompt=false", + "-", + ], + input: options.prompt, + }; + } + if (options.followUp) { + const databaseDir = join(directory, "opencode-data"); + await mkdir(databaseDir, { mode: 0o700 }); + env.OPENCODE_DB = join(databaseDir, "session.db"); + } + const hasInstalledPlugin = await hasOpenCodePlugin({ home: homedir(), env: process.env }); + if (!hasInstalledPlugin) { + const pluginDir = join(directory, "opencode-plugin"); + await mkdir(pluginDir, { mode: 0o700 }); + await writeFile(join(pluginDir, "index.js"), openCodePlugin, { mode: 0o600, flag: "wx" }); + let inline: Record = {}; + if (env.OPENCODE_CONFIG_CONTENT) { + try { + const parsed = asPlainObject(JSON.parse(env.OPENCODE_CONFIG_CONTENT)); + if (!parsed) throw new Error(); + inline = parsed; + } catch { + throw new Error( + "OPENCODE_CONFIG_CONTENT must be a JSON object to enable OpenCode telemetry for this run." + ); + } + } + if (inline.plugins !== undefined && !Array.isArray(inline.plugins)) + throw new Error("OpenCode inline plugins must be an array."); + env.OPENCODE_CONFIG_CONTENT = JSON.stringify({ + ...inline, + plugins: [...(inline.plugins ?? []), pluginDir], + }); + } + return { + env: { + ...env, + OTEL_EXPORTER_OTLP_ENDPOINT: `${collector}/${OPENCODE_CONFIG_ID}`, + OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${token}`, + AIDD_OPENCODE_OTEL_LOGS_ENDPOINT: `${collector}/${OPENCODE_CONFIG_ID}/v1/logs`, + AIDD_OPENCODE_OTEL_HEADERS: `Authorization=Bearer ${token}`, + }, + args: [ + "run", + "--standalone", + ...(options.followUp ? ["--session", `ses_${randomBytes(16).toString("hex")}`] : []), + ...model, + "--", + options.prompt, + ], + }; +} + +function list(value: unknown): unknown[] { + return Array.isArray(value) ? value : []; +} + +function attributes(value: unknown) { + const values = new Map(); + for (const entry of list(value)) { + const row = asPlainObject(entry); + const content = asPlainObject(row?.value); + const item = content?.stringValue ?? content?.intValue; + if (typeof row?.key === "string" && (typeof item === "string" || typeof item === "number")) { + if (values.has(row.key)) return new Map(); + values.set(row.key, item); + } + } + return values; +} + +function validSpanTimes(start: unknown, end: unknown): boolean { + if ( + typeof start !== "string" || + typeof end !== "string" || + !/^[1-9]\d{0,19}$/.test(start) || + !/^[1-9]\d{0,19}$/.test(end) + ) + return false; + return BigInt(start) <= BigInt(end) && BigInt(end) <= 18446744073709551615n; +} + +export function openCodePromptSpans(line: string): unknown[] { + const envelope = asPlainObject(JSON.parse(line)); + if (envelope?.signal !== "traces") return []; + const payload = asPlainObject(envelope.payload); + const result = []; + for (const item of list(payload?.resourceSpans)) { + const resource = asPlainObject(item); + const a = attributes(asPlainObject(resource?.resource)?.attributes); + const version = a.get("service.version"); + if ( + a.get("service.name") !== OPENCODE_CONFIG_ID || + typeof version !== "string" || + !/^2\.\d+\.\d+$/.test(version) + ) + continue; + for (const scope of list(resource?.scopeSpans)) { + for (const raw of list(asPlainObject(scope)?.spans)) { + const span = asPlainObject(raw); + const b = attributes(span?.attributes); + const path = b.get("url.path"); + const route = "/api/session/:sessionID/prompt"; + const kind = span?.kind; + if ( + (kind !== 2 && kind !== "SPAN_KIND_SERVER") || + b.get("http.route") !== route || + b.get("http.request.method") !== "POST" || + ![200, "200"].includes(b.get("http.response.status_code") ?? "") || + typeof path !== "string" || + !/^\/api\/session\/ses_[A-Za-z0-9]{1,64}\/prompt$/.test(path) || + typeof span?.traceId !== "string" || + !/^[a-f0-9]{32}$/i.test(span.traceId) || + typeof span.spanId !== "string" || + !/^[a-f0-9]{16}$/i.test(span.spanId) || + span.name !== "POST" || + !validSpanTimes(span.startTimeUnixNano, span.endTimeUnixNano) + ) + continue; + const string = (key: string, value: string) => ({ key, value: { stringValue: value } }); + result.push({ + resourceSpans: [ + { + resource: { + attributes: [ + string("service.name", OPENCODE_CONFIG_ID), + string("service.version", version), + ], + }, + scopeSpans: [ + { + spans: [ + { + traceId: span.traceId, + spanId: span.spanId, + name: span.name, + kind, + startTimeUnixNano: span.startTimeUnixNano, + endTimeUnixNano: span.endTimeUnixNano, + attributes: [ + string("http.route", route), + string("http.request.method", "POST"), + string("url.path", path), + { key: "http.response.status_code", value: { intValue: "200" } }, + ], + }, + ], + }, + ], + }, + ], + }); + } + } + } + return result; +} + +async function retainCapture(home: string, capture: string, config: Config): Promise { + const retained = join(home, `${config}.jsonl`); + try { + await rename(capture, retained); + await rm(join(home, ".local"), { recursive: true }); + await rm(join(home, "opencode-plugin"), { recursive: true, force: true }); + await rm(join(home, "opencode-data"), { recursive: true, force: true }); + await rm(join(home, "claude-settings.json"), { force: true }); + return retained; + } catch (error) { + await rm(home, { recursive: true, force: true }); + throw error; + } +} + +function promptSession(payload: unknown): string | undefined { + const resource = asPlainObject(list(asPlainObject(payload)?.resourceSpans)[0]); + const scope = asPlainObject(list(resource?.scopeSpans)[0]); + const span = asPlainObject(list(scope?.spans)[0]); + const path = attributes(span?.attributes).get("url.path"); + return typeof path === "string" ? path.split("/")[3] : undefined; +} + +/** Find the last complete JSONL record without consuming an in-progress append. */ +export async function completeCaptureEnd(capture: string, start: number): Promise { + const file = await open(capture, "r"); + try { + let end = (await file.stat()).size; + const buffer = Buffer.alloc(4096); + while (end > start) { + const length = Math.min(buffer.length, end - start); + const offset = end - length; + const { bytesRead } = await file.read(buffer, 0, length, offset); + const newline = buffer.subarray(0, bytesRead).lastIndexOf(10); + if (newline >= 0) return offset + newline + 1; + end = offset; + } + return start; + } finally { + await file.close(); + } +} + +async function captureSessions(capture: string, config: Config, start: number) { + const end = await completeCaptureEnd(capture, start); + const sessions = new Set(); + const prompts = new Set(); + const usage = new Set(); + if (end <= start) return { end, sessions, usage }; + const lines = createInterface({ + input: createReadStream(capture, { start, end: end - 1 }), + crlfDelay: Infinity, + }); + for await (const line of lines) { + if (!line.trim()) continue; + if (config === OPENCODE_CONFIG_ID) { + for (const payload of openCodePromptSpans(line)) { + const session = promptSession(payload); + if (session) sessions.add(session); + } + for (const log of openCodePluginLogs(line)) { + const resource = asPlainObject(list(asPlainObject(log.payload)?.resourceLogs)[0]); + const scope = asPlainObject(list(resource?.scopeLogs)[0]); + const record = asPlainObject(list(scope?.logRecords)[0]); + const fields = attributes(record?.attributes); + const message = fields.get("assistant.message.id"); + if (fields.get("event.name") === "session.step.ended" && typeof message === "string") + usage.add(`${log.session}:${message}`); + } + } else { + for (const session of sessionIdsInCapture(line, config)) sessions.add(session); + const envelope = asPlainObject(JSON.parse(line)); + for (const resource of list(asPlainObject(envelope?.payload)?.resourceLogs)) { + for (const scope of list(asPlainObject(resource)?.scopeLogs)) { + for (const record of list(asPlainObject(scope)?.logRecords)) { + const fields = attributes(asPlainObject(record)?.attributes); + const name = fields.get("event.name"); + const prompt = fields.get("prompt.id"); + if (config === CLAUDE_CONFIG_ID && typeof prompt === "string") { + if (name === "user_prompt") prompts.add(prompt); + if (name === "api_request") usage.add(prompt); + } + if ( + config === CODEX_CONFIG_ID && + name === "codex.sse_event" && + fields.get("event.kind") === "response.completed" && + fields.has("input_token_count") && + fields.has("output_token_count") + ) + usage.add(String(fields.get("event.timestamp") ?? "completed")); + } + } + } + } + } + if (config === CLAUDE_CONFIG_ID) + for (const item of usage) if (!prompts.has(item)) usage.delete(item); + return { end, sessions, usage }; +} + +async function waitForUsage( + capture: string, + config: Config, + start: number, + expected: string, + signal: AbortSignal +) { + const deadline = Date.now() + 10_000; + while (true) { + const observation = await captureSessions(capture, config, start); + if (config === OPENCODE_CONFIG_ID) + for (const item of observation.usage) + if (!item.startsWith(`${expected}:`)) observation.usage.delete(item); + if ( + observation.sessions.size === 1 && + observation.sessions.has(expected) && + observation.usage.size > 0 + ) + return observation; + if (Date.now() >= deadline || signal.aborted) + throw new Error( + "The provider did not emit a completed usage observation for the expected session." + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + } +} + +function requireSession(sessions: Set, expected: string): void { + if (sessions.size !== 1 || !sessions.has(expected)) + throw new Error("The execution did not emit exactly the expected OTLP session."); +} + +async function executeProvider(input: { + config: Config; + args: string[]; + env: NodeJS.ProcessEnv; + cwd: string; + prompt?: string; + signal: AbortSignal; + codexJson: boolean; +}): Promise { + let thread: string | undefined; + let invalidThread = false; + const code = await new Promise((resolve, reject) => { + const child = spawn(input.config, input.args, { + cwd: input.cwd, + env: input.env, + signal: input.signal, + stdio: [input.prompt ? "pipe" : "ignore", input.codexJson ? "pipe" : "inherit", "inherit"], + }); + if (child.stdout) { + const lines = createInterface({ input: child.stdout, crlfDelay: Infinity }); + lines.on("line", (line) => { + let event: Record | null = null; + try { + event = asPlainObject(JSON.parse(line)); + } catch { + invalidThread = true; + return; + } + if (event?.type === "thread.started") { + const value = event.thread_id; + if ( + thread || + typeof value !== "string" || + !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value) + ) + invalidThread = true; + else thread = value; + } + const item = asPlainObject(event?.item); + if ( + event?.type === "item.completed" && + item?.type === "agent_message" && + typeof item.text === "string" + ) + process.stdout.write(`${item.text}\n`); + }); + } + child.on("error", reject); + child.on("close", resolve); + child.stdin?.on("error", reject); + child.stdin?.end(input.prompt); + }); + if (code !== 0) throw new Error(`${input.config} execution did not complete successfully.`); + if (input.codexJson && (!thread || invalidThread)) + throw new Error("Codex did not report exactly one valid session ID."); + return thread; +} + +function resumeArgs(config: Config, first: string[], session: string, prompt: string): string[] { + if (config === CODEX_CONFIG_ID) return ["exec", "resume", ...first.slice(1, -1), session, "-"]; + const args = [...first]; + if (config === CLAUDE_CONFIG_ID) args[args.indexOf("--session-id")] = "--resume"; + args[args.length - 1] = prompt; + return args; +} + +/** Consent covers this one invocation and its isolated collector only. */ +export async function runOtel(options: RunOptions): Promise { + if (process.env.AIDD_TELEMETRY === "0") throw new Error("AIDD telemetry is disabled."); + if (!options.prompt.trim()) throw new Error("Pass a nonempty prompt."); + if (options.followUp !== undefined && !options.followUp.trim()) + throw new Error("Pass a nonempty follow-up prompt."); + if (options.includeRepository && options.config !== CLAUDE_CONFIG_ID) + throw new Error("Repository metadata is only supported for Claude Code."); + if (options.model && !/^[A-Za-z0-9][A-Za-z0-9._:/#-]{0,127}$/.test(options.model)) { + throw new Error("Provider model ID is invalid."); + } + const config = options.config === OPENCODE_CONFIG_ID ? CODEX_CONFIG_ID : options.config; + const base = { ...options, config, session: "validation", home: "", follow: false }; + const origin = destination(base, "logs").origin; + if (options.signal.aborted) throw new Error("OTLP execution was cancelled."); + const home = await mkdtemp(join(tmpdir(), "aidd-otel-run-")); + const capture = join(home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); + let captureCreated = false; + try { + const collector = await startOtelCollector({ home, port: 0 }); + captureCreated = true; + try { + const launch = await provider( + options, + `http://127.0.0.1:${collector.port}`, + await readToken(home), + home + ); + const first = await executeProvider({ + config: options.config, + args: launch.args, + env: launch.env, + cwd: options.cwd, + prompt: launch.input, + signal: options.signal, + codexJson: !!options.followUp && options.config === CODEX_CONFIG_ID, + }); + if (options.followUp) { + const session = + options.config === CODEX_CONFIG_ID + ? first + : launch.args[ + launch.args.indexOf( + options.config === CLAUDE_CONFIG_ID ? "--session-id" : "--session" + ) + 1 + ]; + if (!session) throw new Error("The provider session is missing."); + const firstCapture = await waitForUsage( + capture, + options.config, + 0, + session, + options.signal + ); + requireSession(firstCapture.sessions, session); + const second = await executeProvider({ + config: options.config, + args: resumeArgs(options.config, launch.args, session, options.followUp), + env: launch.env, + cwd: options.cwd, + prompt: options.config === CODEX_CONFIG_ID ? options.followUp : undefined, + signal: options.signal, + codexJson: options.config === CODEX_CONFIG_ID, + }); + if (options.config === CODEX_CONFIG_ID && second !== session) + throw new Error("Codex resumed a different session."); + const secondCapture = await waitForUsage( + capture, + options.config, + firstCapture.end, + session, + options.signal + ); + requireSession(secondCapture.sessions, session); + if ( + options.config !== CODEX_CONFIG_ID && + [...secondCapture.usage].some((id) => firstCapture.usage.has(id)) + ) + throw new Error("The follow-up did not emit a distinct usage observation."); + } + } finally { + await collector.close(); + } + const lines = createInterface({ input: createReadStream(capture), crlfDelay: Infinity }); + let sent = 0; + if (options.config === OPENCODE_CONFIG_ID) { + const url = new URL( + `/api/kpi/otel/${OPENCODE_CONFIG_ID}/${options.tenant}/v1/traces`, + origin + ); + const selected: unknown[] = []; + const logs: OpenCodePluginLog[] = []; + let rootSession: string | undefined; + for await (const line of lines) { + if (!line.trim()) continue; + if (options.installationId) + checkInstallationClaim(JSON.parse(line), options.installationId); + for (const payload of openCodePromptSpans(line)) { + const session = promptSession(payload); + if (!session || (rootSession && rootSession !== session)) + throw new Error("OpenCode prompt spans do not identify the same session."); + rootSession = session; + selected.push(payload); + if (selected.length > (options.followUp ? 2 : 1)) + throw new Error("The isolated OpenCode run emitted more prompt spans than expected."); + } + for (const log of openCodePluginLogs(line)) { + logs.push(log); + } + } + if (!rootSession || selected.length !== (options.followUp ? 2 : 1)) + throw new Error( + "OpenCode prompt trace is missing; session attribution cannot be verified." + ); + const related = relatedOpenCodeLogs(rootSession, logs); + if (related.length !== logs.length) + console.warn( + "OpenCode observations from sessions without a proven parent were not forwarded." + ); + if (!related.some((log) => log.session === rootSession)) + throw new Error("OpenCode telemetry plugin did not emit any session observations."); + const logsPayload = { + resourceLogs: related.flatMap((log) => list(asPlainObject(log.payload)?.resourceLogs)), + }; + const logsBody = JSON.stringify( + options.installationId + ? attachInstallationId(logsPayload, "resourceLogs", options.installationId) + : logsPayload + ); + if (Buffer.byteLength(logsBody, "utf8") > 2 * 1024 * 1024) + throw new Error( + "OpenCode observations exceed the receiver's 2 MiB limit; nothing was sent." + ); + for (const payload of selected) { + const response = await fetch(url, { + method: "POST", + redirect: "manual", + signal: AbortSignal.any([options.signal, AbortSignal.timeout(15000)]), + headers: { + authorization: `Bearer ${options.token}`, + "content-type": "application/json", + }, + body: JSON.stringify( + options.installationId + ? attachInstallationId(payload, "resourceSpans", options.installationId) + : payload + ), + }); + if (!response.ok) + throw new Error(`Gouvernail rejected OpenCode traces: HTTP ${response.status}.`); + const receipt = asPlainObject(await response.json()); + if (receipt?.accepted !== 1 || Object.keys(receipt).length !== 1) + throw new Error("Gouvernail did not retain the OpenCode span."); + sent++; + } + const logsUrl = new URL( + `/api/kpi/otel/${OPENCODE_CONFIG_ID}/${options.tenant}/v1/logs`, + origin + ); + try { + const response = await fetch(logsUrl, { + method: "POST", + redirect: "manual", + signal: AbortSignal.any([options.signal, AbortSignal.timeout(15000)]), + headers: { authorization: `Bearer ${options.token}`, "content-type": "application/json" }, + body: logsBody, + }); + if (!response.ok) + throw new Error(`Gouvernail rejected OpenCode logs: HTTP ${response.status}.`); + const receipt = asPlainObject(await response.json()); + if (receipt?.accepted !== related.length || Object.keys(receipt).length !== 1) + throw new Error("Gouvernail did not retain all OpenCode observations."); + } catch (error) { + const message = error instanceof Error ? error.message : "OpenCode logs could not be sent."; + throw new Error(`${message} The prompt trace may already have been retained.`); + } + sent += related.length; + } else { + const sessions = new Set(); + for await (const line of lines) { + if (line.trim()) + for (const session of sessionIdsInCapture(line, config)) sessions.add(session); + } + if (sessions.size > 1) + throw new Error("The isolated run emitted more than one OTLP session."); + for (const session of sessions) { + const result = await syncOtel( + { ...base, session, home }, + async () => { + if (options.signal.aborted) throw new Error("OTLP execution was cancelled."); + }, + async () => !options.signal.aborted + ); + sent += result.sent; + if (options.config === CODEX_CONFIG_ID) { + if (result.sent === 0) + throw new Error("Codex session logs were not retained; turn traces were not sent."); + try { + const traces = await syncOtel( + { ...base, session, home, traceOnly: true }, + async () => { + if (options.signal.aborted) throw new Error("OTLP execution was cancelled."); + }, + async () => !options.signal.aborted + ); + sent += traces.sent; + } catch (error) { + const message = + error instanceof Error ? error.message : "Codex traces could not be sent."; + throw new Error(`${message} Logs may already have been retained.`); + } + } + } + } + return sent; + } finally { + if (captureCreated && options.retainCapture) { + options.retainCapture(await retainCapture(home, capture, options.config)); + } else { + await rm(home, { recursive: true, force: true }); + } + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts new file mode 100644 index 000000000..030eb3660 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts @@ -0,0 +1,54 @@ +import { createHash } from "node:crypto"; + +export function seriesFingerprint(resource: unknown, scope: unknown, point: unknown): string { + const resourceEnvelope = record(resource); + const resourceInfo = record(resourceEnvelope.resource); + const scopeEnvelope = record(scope); + const scopeInfo = record(scopeEnvelope.scope); + const value = { + resource: { + schemaUrl: resourceEnvelope.schemaUrl ?? null, + attributes: canonicalAttributes(resourceInfo.attributes), + droppedAttributesCount: resourceInfo.droppedAttributesCount ?? null, + }, + scope: { + schemaUrl: scopeEnvelope.schemaUrl ?? null, + name: scopeInfo.name ?? null, + version: scopeInfo.version ?? null, + attributes: canonicalAttributes(scopeInfo.attributes), + droppedAttributesCount: scopeInfo.droppedAttributesCount ?? null, + }, + metric: canonical(scopeEnvelope.metric), + point: canonicalAttributes(point), + }; + return createHash("sha256").update(JSON.stringify(value)).digest("hex"); +} + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function canonicalAttributes(value: unknown): string[] { + if (!Array.isArray(value)) return []; + return value.map((item) => JSON.stringify(canonical(item))).sort(); +} + +function canonical(value: unknown): unknown { + if (Array.isArray(value)) return value.map(canonical); + if (!value || typeof value !== "object") return value ?? null; + return Object.fromEntries( + Object.entries(value) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + .map(([key, item]) => [key, key === "intValue" ? canonicalInteger(item) : canonical(item)]) + ); +} + +function canonicalInteger(value: unknown): string { + if (typeof value === "number" && !Number.isSafeInteger(value)) + throw new Error("OTLP integer attribute is not exactly representable."); + const input = String(value); + if (!/^-?\d{1,20}$/.test(input)) throw new Error("Invalid OTLP integer attribute."); + return BigInt(input).toString(); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts new file mode 100644 index 000000000..55a763320 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -0,0 +1,835 @@ +import { constants } from "node:fs"; +import { lstat, open } from "node:fs/promises"; +import { join } from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { isDeepStrictEqual } from "node:util"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; +import { codexTurnTransfers } from "../turn-traces.js"; +import { attachInstallationId, checkInstallationClaim } from "./installation-attribute.js"; +import { readAttributeValue } from "./read-attribute-value.js"; +import { seriesFingerprint } from "./series-fingerprint.js"; + +export interface SyncOptions { + readonly config: SyncConfig; + readonly session: string; + readonly tenant: string; + readonly endpoint: string; + readonly token: string; + readonly home: string; + readonly follow: boolean; + readonly signal?: AbortSignal; + readonly intervalMs?: number; + readonly installationId?: string; + /** Used by isolated Codex runs after logs have been delivered. */ + readonly traceOnly?: boolean; +} + +type RecordObject = Record; +const object = (value: unknown): RecordObject => asPlainObject(value) ?? {}; +const array = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); +const MAX_PAYLOAD = 2 * 1024 * 1024; +const MAX_LINE = 32 * 1024 * 1024; + +export class OtelObservationTooLargeError extends Error { + constructor() { + super("An OTLP observation exceeds the receiver's 2 MiB limit."); + } +} +const INTERVAL_MS = 5000; +const CLAUDE_METRICS = new Set([ + "claude_code.active_time.total", + "claude_code.code_edit_tool.decision", + "claude_code.commit.count", + "claude_code.cost.usage", + "claude_code.lines_of_code.count", + "claude_code.pull_request.count", + "claude_code.session.count", + "claude_code.token.usage", +]); +const CODEX_EVENTS = new Set(["codex.conversation_starts", "codex.sse_event", "codex.tool_result"]); +const IDENTITY_ATTRIBUTES = new Set(["user.email", "user.account_id", "organization.id"]); +const NON_TEXT_ATTRIBUTES = new Set([ + "event.sequence", + "success", + "duration_ms", + "input_token_count", + "output_token_count", + "cached_token_count", + "cache_write_token_count", + "reasoning_token_count", +]); +const METRIC_DATA_FIELDS = [ + "sum", + "gauge", + "histogram", + "exponentialHistogram", + "exponential_histogram", + "summary", +]; +const METRIC_POINT_FIELDS = [ + "startTimeUnixNano", + "timeUnixNano", + "asInt", + "asDouble", + "count", + "sum", + "bucketCounts", + "explicitBounds", + "min", + "max", + "flags", +]; +const SAFE_ATTRIBUTES = new Set([ + "user.email", + "user.account_id", + "session.id", + "conversation.id", + "organization.id", + "vcs.repository.url.full", + "event.name", + "event.kind", + "event.sequence", + "event.timestamp", + "aidd.original_event_timestamp", + "tool_use_id", + "type", + "model", + "tool_name", + "tool", + "decision", + "query_source", + "aidd.series_fingerprint", + "app.version", + "originator", + "success", + "duration_ms", + "call_id", + "input_token_count", + "output_token_count", + "cached_token_count", + "cache_write_token_count", + "reasoning_token_count", +]); + +function attributes(value: unknown): Map { + if (value !== undefined && !Array.isArray(value)) throw new Error("Invalid OTLP attributes."); + const out = new Map(); + const seen = new Map(); + for (const item of array(value)) { + const entry = object(item); + if (typeof entry.key === "string") { + if (seen.has(entry.key) && !isDeepStrictEqual(seen.get(entry.key), entry.value)) + throw new Error("Conflicting duplicate OTLP attributes."); + seen.set(entry.key, entry.value); + } + if (typeof entry.key === "string") { + const critical = SAFE_ATTRIBUTES.has(entry.key); + if ( + critical && + !NON_TEXT_ATTRIBUTES.has(entry.key) && + typeof object(entry.value).stringValue !== "string" + ) + throw new Error("Invalid OTLP attribute value."); + const raw = readAttributeValue(entry.value, critical); + if ( + raw === "" && + ["event.name", "event.kind", "session.id", "conversation.id"].includes(entry.key) + ) + throw new Error("Invalid OTLP attribute value."); + if (raw !== undefined) out.set(entry.key, raw); + } + } + return out; +} + +function merge(...maps: Map[]): Map { + const out = new Map(); + for (const map of maps) { + for (const [key, value] of map) { + const previous = out.get(key); + if ( + (IDENTITY_ATTRIBUTES.has(key) || key === "vcs.repository.url.full") && + previous !== undefined && + previous !== value + ) + throw new Error("Conflicting OTLP identity attributes."); + out.set(key, value); + } + } + return out; +} + +function safeAttributes(values: Map) { + return [...values] + .filter( + ([key, value]) => + SAFE_ATTRIBUTES.has(key) && + (key !== "vcs.repository.url.full" || isSafeRepositoryUrl(value)) + ) + .map(([key, value]) => ({ + key, + value: { stringValue: value }, + })); +} + +// Request fields are allowed only on Claude API request events, never on other logs. +function claudeRequestAttributes(raw: unknown) { + const result: { key: string; value: { stringValue: string } }[] = []; + const entries = array(raw).map(object); + for (const key of ["request_id", "prompt.id", "effort", "model"]) { + const value = object(entries.find((entry) => entry.key === key)?.value).stringValue; + if (typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9_.:-]{0,199}$/.test(value)) + result.push({ key, value: { stringValue: value } }); + } + for (const key of [ + "input_tokens", + "output_tokens", + "cache_read_tokens", + "cache_creation_tokens", + ]) { + const rawValue = object(entries.find((entry) => entry.key === key)?.value); + const rawNumber = rawValue.intValue ?? rawValue.stringValue; + if (typeof rawNumber !== "number" && typeof rawNumber !== "string") continue; + const value = String(rawNumber); + if (/^(0|[1-9]\d*)$/.test(value) && Number.isSafeInteger(Number(value))) + result.push({ key, value: { stringValue: value } }); + } + return result; +} + +function isSafeRepositoryUrl(value: string): boolean { + return ( + /^https:\/\/(?:github\.com|gitlab\.com)\/[A-Za-z0-9_.-]+(?:\/[A-Za-z0-9_.-]+)+$/.test(value) && + !value.split("/").some((part) => part === "." || part === "..") + ); +} + +function eventNameFromBody(body: unknown): string { + if (body === undefined) return ""; + if (typeof object(body).stringValue !== "string") + throw new Error("Invalid OTLP attribute value."); + const value = readAttributeValue(body, true); + if (!value) throw new Error("Invalid OTLP attribute value."); + return value; +} + +function numericFields(source: RecordObject, keys: readonly string[]): RecordObject { + const result: RecordObject = {}; + for (const key of keys) { + const value = source[key]; + if (value === undefined) continue; + validateIntegerField(key, value); + if (["asDouble", "sum", "min", "max"].includes(key) && !isFiniteDouble(value)) + throw new Error("Invalid OTLP numeric field."); + if (key === "explicitBounds" && (!Array.isArray(value) || !value.every(isFiniteDouble))) + throw new Error("Invalid OTLP numeric field."); + if ( + (typeof value === "number" && Number.isFinite(value)) || + (typeof value === "string" && /^-?\d+(?:\.\d+)?$/.test(value)) || + (Array.isArray(value) && + value.every( + (item) => + (typeof item === "number" && Number.isFinite(item)) || + (typeof item === "string" && /^-?\d+(?:\.\d+)?$/.test(item)) + )) + ) + result[key] = value; + } + return result; +} + +function isFiniteDouble(value: unknown): boolean { + return ( + (typeof value === "number" && Number.isFinite(value)) || + (typeof value === "string" && /^-?\d+(?:\.\d+)?$/.test(value) && Number.isFinite(Number(value))) + ); +} + +function metricKind(metric: RecordObject): "sum" | "gauge" | "histogram" { + const fields = METRIC_DATA_FIELDS.filter((key) => metric[key] !== undefined); + const kind = fields.at(0); + if (fields.length !== 1 || (kind !== "sum" && kind !== "gauge" && kind !== "histogram")) + throw new Error("Invalid or unsupported OTLP metric data."); + const data = asPlainObject(metric[kind]); + if (!data || (data.dataPoints !== undefined && !Array.isArray(data.dataPoints))) + throw new Error("Invalid or unsupported OTLP metric data."); + return kind; +} + +function validateMetricPoint(point: RecordObject, kind: string): void { + const values = Number(point.asInt !== undefined) + Number(point.asDouble !== undefined); + if ((kind === "histogram" && values !== 0) || (kind !== "histogram" && values !== 1)) + throw new Error("Invalid OTLP metric point value."); + numericFields(point, METRIC_POINT_FIELDS); + if (point.flags !== undefined && (BigInt(String(point.flags)) & 1n) !== 0n) + throw new Error("OTLP metric point has no recorded value."); +} + +function isExactInteger(value: unknown, minimum: bigint, maximum: bigint): boolean { + if (typeof value === "number" && !Number.isSafeInteger(value)) return false; + if (typeof value !== "number" && (typeof value !== "string" || !/^-?\d{1,20}$/.test(value))) + return false; + const integer = BigInt(value); + return integer >= minimum && integer <= maximum; +} + +function validateIntegerField(key: string, value: unknown): void { + const unsigned64 = 18446744073709551615n; + let valid = true; + if (["startTimeUnixNano", "timeUnixNano", "observedTimeUnixNano", "count"].includes(key)) + valid = isExactInteger(value, 0n, unsigned64); + else if (key === "asInt") + valid = isExactInteger(value, -9223372036854775808n, 9223372036854775807n); + else if (key === "flags") valid = isExactInteger(value, 0n, 4294967295n); + else if (key === "bucketCounts") + valid = Array.isArray(value) && value.every((item) => isExactInteger(item, 0n, unsigned64)); + if (!valid) throw new Error(`Invalid OTLP integer field: ${key}.`); +} + +function isExactNanoseconds(value: unknown): value is string | number { + return isExactInteger(value, 0n, 18446744073709551615n); +} + +function timeOf(entry: RecordObject) { + for (const key of ["timeUnixNano", "observedTimeUnixNano"]) { + if (entry[key] !== undefined) validateIntegerField(key, entry[key]); + } + return { + ...(isExactNanoseconds(entry.timeUnixNano) ? { timeUnixNano: entry.timeUnixNano } : {}), + ...(isExactNanoseconds(entry.observedTimeUnixNano) + ? { observedTimeUnixNano: entry.observedTimeUnixNano } + : {}), + }; +} + +export interface Transfer { + readonly signal: "logs" | "metrics" | "traces"; + readonly payload: RecordObject; + readonly at?: string; + readonly start?: string; + /** When set, the same clone must also consent at this instant. Never sent. */ + readonly consentAt?: string; +} + +/** Each item was separately consent-checked. The receiver's size limit remains mandatory. */ +export function packTransfers( + transfers: readonly Transfer[], + groupByTimestamp = true +): Array<{ transfer: Transfer; count: number; items: Transfer[] }> { + const groups = new Map>(); + for (const item of transfers) { + const key = groupByTimestamp ? `${item.signal}\0${item.at ?? ""}` : item.signal; + const root = + item.signal === "logs" + ? "resourceLogs" + : item.signal === "metrics" + ? "resourceMetrics" + : "resourceSpans"; + const resources = array(item.payload[root]); + const batches = groups.get(key) ?? []; + const last = batches.at(-1); + const merged = last ? [...array(last.transfer.payload[root]), ...resources] : resources; + const body = { [root]: merged }; + if (Buffer.byteLength(JSON.stringify(body)) > MAX_PAYLOAD) { + if (Buffer.byteLength(JSON.stringify(item.payload)) > MAX_PAYLOAD) + throw new OtelObservationTooLargeError(); + batches.push({ transfer: item, count: 1, items: [item] }); + } else if (last) { + batches[batches.length - 1] = { + transfer: { ...last.transfer, payload: body }, + count: last.count + 1, + items: [...last.items, item], + }; + } else { + batches.push({ transfer: item, count: 1, items: [item] }); + } + groups.set(key, batches); + } + return [...groups.values()].flat(); +} + +/** Names only session IDs; data from an unknown session is never sent until the ledger admits it. */ +export function sessionIdsInCapture(line: string, config: SyncOptions["config"]): string[] { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + throw new Error("Capture contains invalid JSON; sync stopped."); + } + const envelope = object(parsed); + const payload = object(envelope.payload); + const sessionKey = config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id"; + const ids = new Set(); + const add = (resource: Map, raw: unknown) => { + const session = merge(resource, attributes(object(raw).attributes)).get(sessionKey); + if (session && /^[A-Za-z0-9_-]{1,128}$/.test(session)) ids.add(session); + }; + if (envelope.signal === "logs") { + for (const raw of array(payload.resourceLogs)) { + const resource = object(raw); + const attrs = attributes(object(resource.resource).attributes); + for (const scope of array(resource.scopeLogs)) { + attributes(object(object(scope).scope).attributes); + for (const record of array(object(scope).logRecords)) add(attrs, record); + } + } + } + if (config === CLAUDE_CONFIG_ID && envelope.signal === "metrics") { + for (const raw of array(payload.resourceMetrics)) { + const resource = object(raw); + const attrs = attributes(object(resource.resource).attributes); + for (const scope of array(resource.scopeMetrics)) { + attributes(object(object(scope).scope).attributes); + for (const rawMetric of array(object(scope).metrics)) { + const metric = object(rawMetric); + if (typeof metric.name !== "string" || !CLAUDE_METRICS.has(metric.name)) continue; + for (const kind of [metricKind(metric)]) + for (const point of array(object(metric[kind]).dataPoints)) { + validateMetricPoint(object(point), kind); + add(attrs, point); + } + } + } + } + } + return [...ids]; +} + +function timestamp(value: unknown): string | undefined { + if (value === undefined) return undefined; + if (!isExactNanoseconds(value)) throw new Error("Invalid OTLP integer field: timeUnixNano."); + try { + const nanos = BigInt(value); + return nanos > 0n ? new Date(Number(nanos / 1_000_000n)).toISOString() : undefined; + } catch { + return undefined; + } +} + +function eventTimestamp(value: string | undefined): string | undefined { + if (!value) return undefined; + const date = + /^(\d{4}-\d{2}-\d{2})T(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?(?:Z|[+-](?:[01]\d|2[0-3]):[0-5]\d)$/i.exec( + value + )?.[1]; + if (!date) return undefined; + const calendar = new Date(`${date}T00:00:00.000Z`); + if (Number.isNaN(calendar.getTime()) || !calendar.toISOString().startsWith(`${date}T`)) + return undefined; + const time = Date.parse(value); + return Number.isFinite(time) ? new Date(time).toISOString() : undefined; +} + +/** Keep only supported facts for the requested session. Original capture files are not changed. */ +export function selectTransfers(line: string, options: SyncOptions): Transfer[] { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + throw new Error("Capture contains invalid JSON; sync stopped."); + } + const envelope = object(parsed); + const payload = object(envelope.payload); + if (options.installationId) checkInstallationClaim(payload, options.installationId); + if (options.traceOnly) { + if (options.config !== CODEX_CONFIG_ID || envelope.signal !== "traces") return []; + const transfers = codexTurnTransfers(payload, options.session); + const installationId = options.installationId; + return installationId + ? transfers.map((transfer) => ({ + ...transfer, + payload: attachInstallationId(transfer.payload, "resourceSpans", installationId), + })) + : transfers; + } + const out: Transfer[] = []; + const sessionKey = options.config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id"; + if (envelope.signal === "logs") { + for (const rawResource of array(payload.resourceLogs)) { + const resource = object(rawResource); + const resourceAttrs = attributes(object(resource.resource).attributes); + for (const rawScope of array(resource.scopeLogs)) { + attributes(object(object(rawScope).scope).attributes); + for (const rawRecord of array(object(rawScope).logRecords)) { + const record = object(rawRecord); + const a = merge(resourceAttrs, attributes(record.attributes)); + if (a.get(sessionKey) !== options.session) continue; + const event = a.get("event.name") ?? eventNameFromBody(record.body); + const wanted = + options.config === CLAUDE_CONFIG_ID + ? [ + "tool_result", + "claude_code.tool_result", + "api_request", + "claude_code.api_request", + ].includes(event) + : CODEX_EVENTS.has(event) && + (event !== "codex.sse_event" || a.get("event.kind") === "response.completed"); + if (!wanted) continue; + a.set("event.name", event); + const at = + options.config === CODEX_CONFIG_ID + ? (eventTimestamp(a.get("event.timestamp")) ?? timestamp(record.timeUnixNano)) + : timestamp(record.timeUnixNano); + if (options.config === CODEX_CONFIG_ID) { + const originalTimestamp = a.get("event.timestamp"); + a.delete("aidd.original_event_timestamp"); + if (originalTimestamp && eventTimestamp(originalTimestamp)) + a.set("aidd.original_event_timestamp", originalTimestamp); + if (at) a.set("event.timestamp", at); + else a.delete("event.timestamp"); + } + const safe = safeAttributes(a); + if ( + options.config === CLAUDE_CONFIG_ID && + (event === "api_request" || event === "claude_code.api_request") + ) { + const model = safe.findIndex((attribute) => attribute.key === "model"); + if (model !== -1) safe.splice(model, 1); + safe.push(...claudeRequestAttributes(record.attributes)); + } + if (options.config === CODEX_CONFIG_ID && event !== "codex.tool_result") { + const key = + event === "codex.conversation_starts" ? "reasoning_effort" : "model_reasoning_effort"; + const entries = array(record.attributes).map(object); + const source = entries.find((entry) => entry.key === key); + const model = object(entries.find((entry) => entry.key === "model")?.value).stringValue; + const value = object(source?.value).stringValue; + if ( + typeof model === "string" && + model.trim() && + typeof value === "string" && + /^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$/.test(value) + ) + safe.push({ key, value: { stringValue: value } }); + } + out.push({ + signal: "logs", + at, + payload: { + resourceLogs: [ + { + scopeLogs: [{ logRecords: [{ ...timeOf(record), attributes: safe }] }], + }, + ], + }, + }); + } + } + } + } + if (options.config === CLAUDE_CONFIG_ID && envelope.signal === "metrics") { + for (const rawResource of array(payload.resourceMetrics)) { + const resource = object(rawResource); + const resourceAttrs = attributes(object(resource.resource).attributes); + for (const rawScope of array(resource.scopeMetrics)) { + attributes(object(object(rawScope).scope).attributes); + for (const rawMetric of array(object(rawScope).metrics)) { + const metric = object(rawMetric); + if (typeof metric.name !== "string" || !CLAUDE_METRICS.has(metric.name)) continue; + for (const kind of [metricKind(metric)]) { + const aggregation = object(metric[kind]); + for (const rawPoint of array(aggregation.dataPoints)) { + const point = object(rawPoint); + validateMetricPoint(point, kind); + const pointAttrs = attributes(point.attributes); + const a = merge(resourceAttrs, pointAttrs); + if (a.get(sessionKey) !== options.session) continue; + a.set( + "aidd.series_fingerprint", + seriesFingerprint( + resource, + { + scope: object(rawScope).scope, + schemaUrl: object(rawScope).schemaUrl, + metric: { + name: metric.name, + unit: metric.unit, + kind, + aggregationTemporality: aggregation.aggregationTemporality, + isMonotonic: aggregation.isMonotonic, + }, + }, + point.attributes + ) + ); + const safe = safeAttributes(a); + const effort = array(point.attributes) + .map(object) + .find((entry) => entry.key === "effort"); + const effortValue = object(effort?.value).stringValue; + if ( + metric.name === "claude_code.token.usage" && + a.get("model")?.trim() && + typeof effortValue === "string" && + /^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$/.test(effortValue) + ) + safe.push({ key: "effort", value: { stringValue: effortValue } }); + const sanitized = { + ...numericFields(point, METRIC_POINT_FIELDS), + attributes: safe, + }; + out.push({ + signal: "metrics", + at: timestamp(point.timeUnixNano), + start: timestamp(point.startTimeUnixNano), + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: metric.name, + [kind]: { + ...numericFields(aggregation, ["aggregationTemporality"]), + ...([ + "AGGREGATION_TEMPORALITY_CUMULATIVE", + "AGGREGATION_TEMPORALITY_DELTA", + ].includes(String(aggregation.aggregationTemporality)) + ? { aggregationTemporality: aggregation.aggregationTemporality } + : {}), + ...(typeof aggregation.isMonotonic === "boolean" + ? { isMonotonic: aggregation.isMonotonic } + : {}), + dataPoints: [sanitized], + }, + }, + ], + }, + ], + }, + ], + }, + }); + } + } + } + } + } + } + const installationId = options.installationId; + return installationId + ? out.map((transfer) => ({ + ...transfer, + payload: attachInstallationId( + transfer.payload, + transfer.signal === "logs" ? "resourceLogs" : "resourceMetrics", + installationId + ), + })) + : out; +} + +export function destination(options: SyncOptions, signal: "logs" | "metrics" | "traces"): URL { + let endpoint: URL; + try { + endpoint = new URL(options.endpoint); + } catch { + throw new Error("Endpoint must be a loopback HTTP origin."); + } + if ( + endpoint.protocol !== "http:" || + !["127.0.0.1", "[::1]"].includes(endpoint.hostname) || + endpoint.username || + endpoint.password || + endpoint.pathname !== "/" || + endpoint.search || + endpoint.hash + ) { + throw new Error("Endpoint must be a loopback HTTP origin."); + } + if (!/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(options.tenant) || options.tenant.length > 63) { + throw new Error("Tenant must be a valid slug."); + } + if (!/^[A-Za-z0-9_-]{1,128}$/.test(options.session)) throw new Error("Session ID is invalid."); + if (!options.token || /[\r\n]/.test(options.token)) + throw new Error("AIDD_OTEL_TOKEN is required."); + const provider = options.config === CODEX_CONFIG_ID ? `${CODEX_CONFIG_ID}/` : ""; + return new URL(`/api/kpi/otel/${provider}${options.tenant}/v1/${signal}`, endpoint); +} + +class RetryableDeliveryError extends Error {} + +export class ConsentNotConfirmedError extends Error { + constructor() { + super("Consent is no longer confirmed; sync stopped before sending."); + } +} + +async function assertFullDelivery(response: Response, signal: Transfer["signal"]): Promise { + let result: unknown; + try { + result = await response.json(); + } catch { + throw new Error("Gouvernail returned an invalid OTLP response."); + } + const envelope = asPlainObject(result); + if (!envelope) throw new Error("Gouvernail returned an invalid OTLP response."); + if (Object.keys(envelope).some((key) => key !== "partialSuccess" && key !== "partial_success")) + throw new Error("Gouvernail returned an invalid OTLP response."); + if ("partialSuccess" in envelope && "partial_success" in envelope) + throw new Error("Gouvernail returned an invalid OTLP response."); + const raw = envelope.partialSuccess ?? envelope.partial_success; + if (raw === undefined || raw === null) return; + const partial = asPlainObject(raw); + if (!partial) throw new Error("Gouvernail returned an invalid OTLP response."); + const camel = + signal === "logs" + ? "rejectedLogRecords" + : signal === "metrics" + ? "rejectedDataPoints" + : "rejectedSpans"; + const snake = + signal === "logs" + ? "rejected_log_records" + : signal === "metrics" + ? "rejected_data_points" + : "rejected_spans"; + if ( + Object.keys(partial).some( + (key) => ![camel, snake, "errorMessage", "error_message"].includes(key) + ) + ) + throw new Error("Gouvernail returned an invalid OTLP response."); + if (camel in partial && snake in partial) + throw new Error("Gouvernail returned an invalid OTLP response."); + const count = partial[camel] ?? partial[snake] ?? 0; + const number = + typeof count === "string" && /^\d+(?:[eE]\+?\d+)?$/.test(count) ? Number(count) : count; + if (typeof number !== "number" || !Number.isSafeInteger(number) || number < 0) + throw new Error("Gouvernail returned an invalid OTLP response."); + if (number > 0) + throw new Error( + `Gouvernail reported partial ${signal} delivery; resolve the rejection before retrying.` + ); +} + +export async function send( + options: SyncOptions, + transfer: Transfer, + consent: () => Promise +): Promise { + const body = JSON.stringify(transfer.payload); + if (Buffer.byteLength(body) > MAX_PAYLOAD) + throw new Error("An OTLP observation exceeds the receiver's 2 MiB limit."); + const url = destination(options, transfer.signal); + let backoff = options.intervalMs ?? 1000; + for (;;) { + if (!(await consent())) throw new ConsentNotConfirmedError(); + try { + let response: Response; + try { + response = await fetch(url, { + method: "POST", + redirect: "manual", + signal: AbortSignal.timeout(15000), + headers: { authorization: `Bearer ${options.token}`, "content-type": "application/json" }, + body, + }); + } catch { + throw new RetryableDeliveryError("Gouvernail could not be reached or timed out."); + } + if (!response.ok) { + const message = `Gouvernail rejected ${options.config} ${transfer.signal}: HTTP ${response.status}.`; + if (response.status >= 500) throw new RetryableDeliveryError(message); + throw new Error(message); + } + await assertFullDelivery(response, transfer.signal); + return; + } catch (error) { + if (!options.follow || !(error instanceof RetryableDeliveryError) || options.signal?.aborted) + throw error; + console.error(`${error.message} Retrying in ${backoff} ms.`); + await delay(backoff, undefined, { signal: options.signal }); + backoff = Math.min(backoff * 2, 30_000); + } + } +} + +/** Replays one selected session and optionally follows newly appended bytes. A failed delivery + * never advances the in-memory cursor. Restart safely replays into the idempotent receiver. */ +export async function syncOtel( + options: SyncOptions, + revalidate: () => Promise, + permits: (transfer: Transfer) => Promise +) { + destination(options, "logs"); + const path = join(options.home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); + const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + const stillPermitted = async (items: readonly Transfer[]) => { + await revalidate(); + const consentAt = new Date().toISOString(); + for (const item of items) if (!(await permits({ ...item, consentAt }))) return false; + return true; + }; + let sent = 0; + let skipped = 0; + let offset = 0; + let pending = Buffer.alloc(0); + try { + const info = await handle.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Capture must be a private regular file."); + const buffer = Buffer.alloc(65536); + for (;;) { + const { bytesRead } = await handle.read(buffer, 0, buffer.length, offset + pending.length); + if (bytesRead > 0) { + pending = Buffer.concat([pending, buffer.subarray(0, bytesRead)]); + if (pending.length > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); + for (let newline = pending.indexOf(10); newline >= 0; newline = pending.indexOf(10)) { + const line = pending.subarray(0, newline).toString("utf8"); + if (line.trim()) { + const admitted: Transfer[] = []; + const verdicts = new Map(); + for (const transfer of selectTransfers(line, options)) { + if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { + skipped++; + continue; + } + const stamp = `${transfer.at}\0${transfer.start ?? ""}`; + let allowed = verdicts.get(stamp); + if (allowed === undefined) { + allowed = await permits(transfer); + verdicts.set(stamp, allowed); + } + if (!allowed) { + skipped++; + continue; + } + admitted.push(transfer); + } + for (const batch of packTransfers(admitted)) { + await send(options, batch.transfer, () => stillPermitted(batch.items)); + sent += batch.count; + } + } + offset += newline + 1; + pending = pending.subarray(newline + 1); + } + continue; + } + const current = await lstat(path); + if ( + !current.isFile() || + current.ino !== info.ino || + current.dev !== info.dev || + current.size < offset + pending.length + ) { + throw new Error("Capture was replaced or truncated; restart sync to replay it safely."); + } + if (!options.follow || options.signal?.aborted) + return { sent, skipped, offset, partialLine: pending.length > 0 }; + try { + await delay(options.intervalMs ?? INTERVAL_MS, undefined, { signal: options.signal }); + } catch (error) { + if (options.signal?.aborted) + return { sent, skipped, offset, partialLine: pending.length > 0 }; + throw error; + } + } + } finally { + await handle.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts new file mode 100644 index 000000000..865a59530 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts @@ -0,0 +1,63 @@ +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; +import { join } from "node:path"; +import { readSettings } from "../files.js"; +import { transformCapturedLine } from "../observations.js"; +import { + type Config, + configurationEdit, + type Endpoint, + type Host, + OPENCODE_CONFIG_ID, +} from "../settings.js"; +import { hasOpenCodePlugin } from "./opencode-plugin-file.js"; + +export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) { + const edit = await configurationEdit(config, endpoint, host); + const current = await readSettings(edit.path, edit.initial); + if (!current.exists || edit.update(current.content) !== current.content) { + throw new Error("OTLP configuration is not initialized. Run aidd telemetry otel init first."); + } + if (!current.private) { + throw new Error("OTLP configuration must be private (mode 0600)."); + } + if (config === OPENCODE_CONFIG_ID && !(await hasOpenCodePlugin(host))) { + throw new Error( + "OpenCode telemetry plugin is not installed. Run aidd telemetry otel init first." + ); + } + const capture = join(host.home, ".local/share/aidd-otel-observation", `${config}.jsonl`); + const handle = await open( + capture, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error.code === "ENOENT") + throw new Error("No OTLP capture found. Run a tool session and retry."); + throw error; + }); + try { + const info = await handle.stat(); + if (!info.isFile()) throw new Error("OTLP capture must be a regular file."); + let lines = 0; + let observations = 0; + let unrecognisedLines = 0; + for await (const line of handle.readLines({ autoClose: false })) { + if (!line.trim()) continue; + lines++; + const count = transformCapturedLine(config, line).length; + observations += count; + if (count === 0) unrecognisedLines++; + } + if (lines === 0) throw new Error("OTLP capture is empty. Run a tool session and retry."); + return { + path: edit.path, + capture, + lines, + observations, + unrecognisedLines, + modifiedAt: info.mtime.toISOString(), + }; + } finally { + await handle.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts b/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts new file mode 100644 index 000000000..407d5c6a7 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts @@ -0,0 +1,185 @@ +import { isDeepStrictEqual } from "node:util"; +import { parse, stringify } from "smol-toml"; +import { asPlainObject } from "../../../../kernel/reading/plain-object.js"; + +type Settings = Record; + +function tomlTable(values: Settings = {}): Settings { + return Object.setPrototypeOf(values, null); +} + +function requireObject(value: unknown): Settings { + const object = asPlainObject(value); + if (!object) throw new Error("Expected a configuration object."); + return object; +} + +export function rejectConflict(name: string, actual: unknown, expected: string): void { + if (actual !== undefined && actual !== expected) { + throw new Error(`Conflicting OTLP setting: ${name}. Resolve it before initializing.`); + } +} + +function readJsonSettings(content: string): Settings { + try { + return requireObject(JSON.parse(content)); + } catch { + throw new Error("Invalid JSON settings."); + } +} + +/** Locate a top-level section and refuse duplicate properties where we are about to edit. */ +function sectionOffset(content: string, section: string): number { + let depth = 0; + let offset: number | undefined; + let inSection = false; + const keys = new Set(); + for (const match of content.matchAll(/"(?:[^"\\]|\\.)*"|[{}[\]]/gs)) { + const token = match[0]; + if (token === "{" || token === "[") { + depth++; + } else if (token === "}" || token === "]") { + depth--; + if (depth === 1) inSection = false; + } else { + const tail = /^\s*:\s*/.exec(content.slice(match.index + token.length)); + if (!tail) continue; + const key: string = JSON.parse(token); + if (depth === 1 && key === section) { + if (offset !== undefined) throw new Error(`Duplicate ${section} in JSON settings.`); + offset = match.index + token.length + tail[0].length; + inSection = true; + } else if (depth === 2 && inSection) { + if (keys.has(key)) throw new Error(`Duplicate entry in ${section} settings.`); + keys.add(key); + } + } + } + return offset ?? content.indexOf("{"); +} + +function mergeJsonSection( + content: string, + section: string, + entries: (existing: Settings) => Settings +): string { + const settings = readJsonSettings(content); + const original = settings[section] === undefined ? settings : requireObject(settings[section]); + const existing = settings[section] === undefined ? {} : original; + const offset = sectionOffset(content, section) + 1; + const missing = entries(existing); + if (Object.keys(missing).length === 0) return content; + const patch = settings[section] === undefined ? { [section]: missing } : missing; + const indent = settings[section] === undefined ? "" : " "; + const entry = JSON.stringify(patch, null, 2).slice(1, -2).replaceAll("\n", `\n${indent}`); + const separator = Object.keys(original).length === 0 ? "\n" : ","; + const next = content.slice(0, offset) + entry + separator + content.slice(offset); + readJsonSettings(next); + return next; +} + +export function mergeJsonEnvironment( + content: string, + additions: Readonly>, + verify: (env: Settings) => void, + defaults: Readonly> = {} +): string { + return mergeJsonSection(content, "env", (env) => { + verify(env); + for (const [key, value] of Object.entries(additions)) rejectConflict(key, env[key], value); + return Object.fromEntries( + Object.entries({ ...defaults, ...additions }).filter(([key]) => env[key] === undefined) + ); + }); +} + +function readToml(content: string): Settings { + try { + return parse(content); + } catch { + throw new Error("Invalid TOML settings."); + } +} + +function matchesToml(candidate: string, expected: Settings): boolean { + try { + return isDeepStrictEqual(parse(candidate), expected); + } catch { + // An insertion candidate can redefine a table. Try the next candidate. + return false; + } +} + +function addTomlTable(content: string, table: string, body: string, expected: Settings): string { + const header = `[${table}]`; + const candidates = [`${content ? `${content}\n` : ""}${header}\n${body}`]; + for (const match of content.matchAll(/^\s*\[([^\]\n]+)\][ \t]*(?:#[^\n]*)?(?:\r?\n|$)/gm)) { + const path = match[1].split(".").map((part) => part.trim().replace(/^["']|["']$/g, "")); + if (path.join(".") !== table) continue; + const offset = match.index + match[0].length; + const newline = match[0].endsWith("\n") ? "" : "\n"; + candidates.push(content.slice(0, offset) + newline + body + content.slice(offset)); + } + for (const candidate of candidates) { + if (matchesToml(candidate, expected)) return candidate; + } + throw new Error(`Cannot merge ${table} while preserving TOML settings. Use explicit tables.`); +} + +export function mergeCodexExporters( + content: string, + exporters: Readonly>, + token: string, + defaults: Readonly>, + toolResult: Readonly> +): string { + const settings = readToml(content); + const otel = settings.otel === undefined ? tomlTable() : requireObject(settings.otel); + settings.otel = otel; + const missing = Object.fromEntries( + Object.entries(defaults).filter(([key]) => otel[key] === undefined) + ); + Object.assign(otel, missing); + let next = Object.keys(missing).length + ? addTomlTable(content, "otel", stringify(missing), settings) + : content; + const result = otel.tool_result === undefined ? tomlTable() : requireObject(otel.tool_result); + const resultDefaults = Object.fromEntries( + Object.entries(toolResult).filter(([key]) => result[key] === undefined) + ); + if (Object.keys(resultDefaults).length) { + otel.tool_result = Object.assign(result, resultDefaults); + next = addTomlTable(next, "otel.tool_result", stringify(resultDefaults), settings); + } + for (const [signal, endpoint] of Object.entries(exporters)) { + const exporter = otel[signal] === undefined ? {} : requireObject(otel[signal]); + if (Object.keys(exporter).some((name) => name !== "otlp-http")) { + throw new Error(`Conflicting OTLP exporter: otel.${signal}. Resolve it before initializing.`); + } + const http = exporter["otlp-http"] === undefined ? {} : requireObject(exporter["otlp-http"]); + const fields: Record = {}; + for (const [key, value] of Object.entries({ endpoint, protocol: "json" })) { + rejectConflict(`otel.${signal}.${key}`, http[key], value); + if (http[key] === undefined) fields[key] = value; + } + let body = stringify(fields); + if (http.headers === undefined) { + fields.headers = Object.setPrototypeOf({ authorization: `Bearer ${token}` }, null); + body += `headers = { authorization = ${JSON.stringify(`Bearer ${token}`)} }\n`; + } else { + const headers = requireObject(http.headers); + const auth = Object.entries(headers).filter( + ([name]) => name.toLowerCase() === "authorization" + ); + if (auth.length !== 1 || auth[0][1] !== `Bearer ${token}`) { + throw new Error( + `Conflicting OTLP authorization: otel.${signal}. Resolve it before initializing.` + ); + } + } + if (Object.keys(fields).length === 0) continue; + otel[signal] = tomlTable({ "otlp-http": tomlTable({ ...http, ...fields }) }); + next = addTomlTable(next, `otel.${signal}.otlp-http`, body, settings); + } + return next; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config/claude.json b/cli/src/contexts/telemetry/infrastructure/otel/config/claude.json new file mode 100644 index 000000000..a8ed9afa0 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/config/claude.json @@ -0,0 +1,23 @@ +{ + "env": { + "CLAUDE_CODE_ENABLE_TELEMETRY": "1", + "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1", + "OTEL_METRICS_EXPORTER": "otlp", + "OTEL_LOGS_EXPORTER": "otlp", + "OTEL_TRACES_EXPORTER": "otlp", + "OTEL_EXPORTER_OTLP_PROTOCOL": "http/json", + "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT": "{collector}/claude/v1/metrics", + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "{collector}/claude/v1/logs", + "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "{collector}/claude/v1/traces", + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}" + }, + "defaults": { + "OTEL_LOG_USER_PROMPTS": "1", + "OTEL_LOG_ASSISTANT_RESPONSES": "1", + "OTEL_LOG_TOOL_DETAILS": "1", + "OTEL_LOG_TOOL_CONTENT": "1", + "OTEL_LOG_RAW_API_BODIES": "1", + "CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH": "262144", + "OTEL_METRICS_INCLUDE_REPOSITORY": "true" + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config/codex.json b/cli/src/contexts/telemetry/infrastructure/otel/config/codex.json new file mode 100644 index 000000000..b238aab01 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/config/codex.json @@ -0,0 +1,16 @@ +{ + "defaults": { + "environment": "local", + "log_user_prompt": true, + "log_agent_responses": true, + "log_guardian_assessments": true + }, + "toolResult": { + "max_bytes": 262144 + }, + "exporters": { + "exporter": "{collector}/codex/v1/logs", + "metrics_exporter": "{collector}/codex/v1/metrics", + "trace_exporter": "{collector}/codex/v1/traces" + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json new file mode 100644 index 000000000..3782e5cef --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json @@ -0,0 +1,8 @@ +{ + "env": { + "OTEL_EXPORTER_OTLP_ENDPOINT": "{collector}/opencode", + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}", + "AIDD_OPENCODE_OTEL_LOGS_ENDPOINT": "{collector}/opencode/v1/logs", + "AIDD_OPENCODE_OTEL_HEADERS": "Authorization=Bearer {token}" + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/files.ts b/cli/src/contexts/telemetry/infrastructure/otel/files.ts new file mode 100644 index 000000000..3b3a13c3f --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/files.ts @@ -0,0 +1,24 @@ +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; + +export async function readSettings(path: string, initial: string) { + const handle = await open( + path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!handle) return { content: initial, private: false, exists: false }; + try { + const info = await handle.stat(); + if (!info.isFile()) throw new Error("OTLP configuration must be a regular file."); + return { + content: await handle.readFile("utf8"), + private: (info.mode & 0o077) === 0, + exists: true, + }; + } finally { + await handle.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/observations.ts b/cli/src/contexts/telemetry/infrastructure/otel/observations.ts new file mode 100644 index 000000000..e72ddad6c --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/observations.ts @@ -0,0 +1,367 @@ +import { asPlainObject } from "../../../../kernel/reading/plain-object.js"; +import type { + JsonValue, + TelemetryObservation, +} from "../../domain/observations/telemetry-observation.js"; +import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type Config, OPENCODE_CONFIG_ID } from "./settings.js"; + +interface OtelCapture { + readonly signal: "logs" | "traces" | "metrics"; + readonly payload: unknown; + readonly received_at?: string; +} +type Obj = Record; +function json(value: unknown): JsonValue | undefined { + if (value === null || typeof value === "boolean" || typeof value === "string") return value; + if (typeof value === "number") return Number.isFinite(value) ? value : undefined; + if (Array.isArray(value)) return value.map((item) => json(item) ?? null); + const input = asPlainObject(value); + if (input === null) return undefined; + const result: Record = {}; + for (const [key, entry] of Object.entries(input)) { + const parsed = json(entry); + if (parsed !== undefined) result[key] = parsed; + } + return result; +} +const object = (value: unknown): Obj => asPlainObject(value) ?? {}; +const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); +const text = (value: unknown): string | undefined => + typeof value === "string" && value !== "" ? value : undefined; +function number(value: unknown): number | undefined { + if (typeof value !== "number" && (typeof value !== "string" || value.trim() === "")) + return undefined; + const n = Number(value); + return Number.isFinite(n) && n >= 0 ? n : undefined; +} +function tokenCount(value: unknown): number | undefined { + if (typeof value === "string" && !/^\d+$/.test(value)) return undefined; + const count = number(value); + return count !== undefined && Number.isSafeInteger(count) ? count : undefined; +} +function bool(value: unknown): boolean | undefined { + if (value === true || value === "true") return true; + if (value === false || value === "false") return false; + return undefined; +} +function nanos(value: unknown): string | undefined { + try { + if (typeof value !== "string" && typeof value !== "number") return undefined; + const n = BigInt(value); + if (n <= 0n) return undefined; + return new Date(Number(n / 1_000_000n)).toISOString(); + } catch { + return undefined; + } +} +function decodeValue(raw: unknown): unknown { + const item = object(raw); + for (const key of ["stringValue", "boolValue", "intValue", "doubleValue"]) { + if (key in item) return item[key]; + } + if (item.arrayValue) return list(object(item.arrayValue).values).map(decodeValue); + if (item.kvlistValue) return attributes(object(item.kvlistValue).values); + return undefined; +} +function attributes(raw: unknown): Obj { + const result: Obj = {}; + for (const entry of list(raw)) { + const item = object(entry); + const key = text(item.key); + if (key !== undefined) result[key] = decodeValue(item.value); + } + return result; +} +function first(...values: (T | undefined)[]): T | undefined { + return values.find((item) => item !== undefined); +} +function pickText(attributes: Obj, ...keys: string[]): string | undefined { + return first(...keys.map((key) => text(attributes[key]))); +} +function pickTokenCount(attributes: Obj, ...keys: string[]): number | undefined { + return first(...keys.map((key) => tokenCount(attributes[key]))); +} +function tokens(fields: TelemetryObservation["tokens"]): TelemetryObservation["tokens"] { + return fields && Object.values(fields).some((item) => item !== undefined) ? fields : undefined; +} +function cacheZeroMayMeanMissing( + tool: TelemetryObservation["tool"], + usage: TelemetryObservation["tokens"] +): true | undefined { + return tool === CODEX_CONFIG_ID && (usage?.cacheRead === 0 || usage?.cacheWrite === 0) + ? true + : undefined; +} +function base( + tool: TelemetryObservation["tool"], + signal: TelemetryObservation["signal"], + a: Obj, + source: unknown +): TelemetryObservation { + const usage = tokens({ + input: pickTokenCount( + a, + "input_tokens", + "input_token_count", + "codex.turn.token_usage.input_tokens", + "gen_ai.usage.input_tokens", + "ai.usage.inputTokens" + ), + inputNonCached: pickTokenCount( + a, + "codex.turn.token_usage.non_cached_input_tokens", + "ai.usage.inputTokenDetails.noCacheTokens" + ), + output: pickTokenCount( + a, + "output_tokens", + "output_token_count", + "codex.turn.token_usage.output_tokens", + "gen_ai.usage.output_tokens", + "ai.usage.outputTokens" + ), + cacheRead: pickTokenCount( + a, + "cache_read_tokens", + "cached_token_count", + "codex.turn.token_usage.cached_input_tokens", + "gen_ai.usage.cache_read.input_tokens", + "ai.usage.cachedInputTokens", + "ai.usage.inputTokenDetails.cacheReadTokens" + ), + cacheWrite: pickTokenCount( + a, + "cache_creation_tokens", + "cache_write_token_count", + "codex.turn.token_usage.cache_write_input_tokens", + "gen_ai.usage.cache_write.input_tokens" + ), + cacheWrite5m: tokenCount(a.cache_creation_5m_tokens), + cacheWrite1h: tokenCount(a.cache_creation_1h_tokens), + reasoning: pickTokenCount( + a, + "reasoning_token_count", + "codex.turn.token_usage.reasoning_output_tokens", + "codex.usage.reasoning_output_tokens", + "ai.usage.reasoningTokens", + "ai.usage.outputTokenDetails.reasoningTokens" + ), + total: pickTokenCount( + a, + "codex.turn.token_usage.total_tokens", + "codex.usage.total_tokens", + "ai.usage.totalTokens" + ), + tool: tokenCount(a.tool_token_count), + }); + return { + tool, + signal, + source: json(source), + eventType: pickText(a, "event.name"), + timestamp: pickText(a, "event.timestamp"), + toolVersion: pickText(a, "app.version", "service.version"), + sessionId: pickText(a, "session.id", "conversation.id"), + threadId: text(a["thread.id"]), + turnId: tool === CODEX_CONFIG_ID ? text(a["turn.id"]) : undefined, + promptId: tool === CLAUDE_CONFIG_ID ? text(a["prompt.id"]) : undefined, + clientRequestId: tool === CLAUDE_CONFIG_ID ? text(a.client_request_id) : undefined, + messageId: tool === CLAUDE_CONFIG_ID ? text(a["message.uuid"]) : undefined, + provider: pickText( + a, + "anthropic.provider", + "provider_name", + "provider", + "gen_ai.provider.name", + "gen_ai.system", + "ai.model.provider" + ), + model: pickText(a, "model", "gen_ai.request.model", "ai.model.id"), + requestId: pickText(a, "request_id", "gen_ai.response.id", "ai.response.id"), + toolCallId: pickText(a, "tool_use_id", "call_id", "gen_ai.tool.call.id", "tool.call_id"), + runtimeToolCallId: pickText(a, "runtime_tool_call_id"), + toolName: pickText(a, "tool_name", "gen_ai.tool.name", "tool.name"), + toolInput: json(first(a.tool_input, a.tool_parameters, a.arguments)), + success: bool(a.success), + durationMs: number(a.duration_ms), + cwd: pickText(a, "cwd", "directory"), + tokens: usage, + usageScope: usage ? "event" : undefined, + cacheZeroMayMeanMissing: cacheZeroMayMeanMissing(tool, usage), + tokenInputIncludesCache: + usage?.input !== undefined && tool !== OPENCODE_CONFIG_ID + ? tool === CODEX_CONFIG_ID + : undefined, + tokenOutputIncludesReasoning: + usage?.output !== undefined && tool === CODEX_CONFIG_ID ? true : undefined, + reportedCostUsd: number(a.cost_usd), + reportedCostUsdMicros: number(a.cost_usd_micros), + }; +} +function fromLog( + tool: TelemetryObservation["tool"], + resource: Obj, + raw: unknown +): TelemetryObservation { + const entry = object(raw); + const a = { ...resource, ...attributes(entry.attributes) }; + const row = base(tool, "log", a, raw); + const name = row.eventType; + return { + ...row, + timestamp: first(row.timestamp, nanos(entry.timeUnixNano)), + observedTimestamp: nanos(entry.observedTimeUnixNano), + traceId: text(entry.traceId), + spanId: text(entry.spanId), + prompt: + name === "user_prompt" || name === "codex.user_prompt" + ? pickText(a, "prompt_text", "prompt") + : undefined, + response: + name === "assistant_response" || name === "codex.agent_response" + ? text(a.response) + : undefined, + toolOutput: name === "tool_result" || name === "codex.tool_result" ? json(a.output) : undefined, + }; +} +function fromSpan( + tool: TelemetryObservation["tool"], + resource: Obj, + raw: unknown +): TelemetryObservation[] { + const span = object(raw); + const a = { ...resource, ...attributes(span.attributes) }; + const baseRow = base(tool, "span", a, raw); + const row: TelemetryObservation = { + ...baseRow, + eventType: text(span.name), + timestamp: nanos(span.startTimeUnixNano), + startedAt: nanos(span.startTimeUnixNano), + endedAt: nanos(span.endTimeUnixNano), + traceId: text(span.traceId), + spanId: text(span.spanId), + parentSpanId: text(span.parentSpanId), + prompt: text(a.user_prompt), + response: text(a["ai.response.text"]), + usageScope: baseRow.tokens ? "span-snapshot" : undefined, + tokenInputIncludesCache: + a["ai.usage.inputTokens"] !== undefined ? true : baseRow.tokenInputIncludesCache, + tokenOutputIncludesReasoning: + a["ai.usage.outputTokens"] !== undefined ? true : baseRow.tokenOutputIncludesReasoning, + }; + return [ + row, + ...list(span.events).map((rawEvent): TelemetryObservation => { + const event = object(rawEvent); + const ea = attributes(event.attributes); + return { + tool, + signal: "span-event", + source: json(rawEvent), + eventType: text(event.name), + timestamp: nanos(event.timeUnixNano), + toolVersion: row.toolVersion, + sessionId: row.sessionId, + threadId: row.threadId, + turnId: row.turnId, + traceId: row.traceId, + spanId: row.spanId, + parentSpanId: row.parentSpanId, + toolCallId: row.toolCallId, + toolName: row.toolName, + toolOutput: event.name === "tool.output" ? json(ea.output) : undefined, + }; + }), + ]; +} +function fromMetric( + tool: TelemetryObservation["tool"], + resource: Obj, + raw: unknown +): TelemetryObservation[] { + const metric = object(raw); + const name = text(metric.name); + return ["sum", "gauge", "histogram"].flatMap((kind) => + list(object(metric[kind]).dataPoints).map((rawPoint): TelemetryObservation => { + const point = object(rawPoint); + const a = { ...resource, ...attributes(point.attributes) }; + const metricType = pickText(a, "type", "token_type"); + const rawAmount = [point.asInt, point.asDouble, point.sum].find( + (value) => number(value) !== undefined + ); + const amount = number(rawAmount); + const tokenKinds: Record> = { + input: "input", + output: "output", + cacheRead: "cacheRead", + cacheCreation: "cacheWrite", + cached_input: "cacheRead", + cache_write_input: "cacheWrite", + reasoning_output: "reasoning", + }; + const token = metricType && tokenKinds[metricType]; + const count = tokenCount(rawAmount); + const usage = + (name === "claude_code.token.usage" || name === "codex.turn.token_usage") && + token && + count !== undefined + ? { [token]: count } + : undefined; + return { + ...base(tool, "metric", a, rawPoint), + eventType: name, + timestamp: nanos(point.timeUnixNano), + startedAt: nanos(point.startTimeUnixNano), + metricName: name, + metricType, + metricValue: amount, + tokens: usage, + usageScope: usage ? "metric-aggregate" : undefined, + cacheZeroMayMeanMissing: cacheZeroMayMeanMissing(tool, usage), + }; + }) + ); +} +function otel(tool: TelemetryObservation["tool"], capture: OtelCapture): TelemetryObservation[] { + const payload = object(capture.payload); + if (capture.signal === "logs") + return list(payload.resourceLogs).flatMap((r) => { + const resource = object(r); + const attrs = attributes(object(resource.resource).attributes); + return list(resource.scopeLogs).flatMap((s) => + list(object(s).logRecords).map((entry) => fromLog(tool, attrs, entry)) + ); + }); + if (capture.signal === "traces") + return list(payload.resourceSpans).flatMap((r) => { + const resource = object(r); + const attrs = attributes(object(resource.resource).attributes); + return list(resource.scopeSpans).flatMap((s) => + list(object(s).spans).flatMap((entry) => fromSpan(tool, attrs, entry)) + ); + }); + return list(payload.resourceMetrics).flatMap((r) => { + const resource = object(r); + const attrs = attributes(object(resource.resource).attributes); + return list(resource.scopeMetrics).flatMap((s) => + list(object(s).metrics).flatMap((entry) => fromMetric(tool, attrs, entry)) + ); + }); +} +export const transformClaude = (source: OtelCapture) => otel(CLAUDE_CONFIG_ID, source); +export const transformCodex = (source: OtelCapture) => otel(CODEX_CONFIG_ID, source); +export const transformOpenCode = (source: OtelCapture) => otel(OPENCODE_CONFIG_ID, source); + +export function transformCapturedLine(tool: Config, line: string): TelemetryObservation[] { + let input: Obj; + try { + input = object(JSON.parse(line)); + } catch { + return []; + } + if (input.signal === "logs" || input.signal === "traces" || input.signal === "metrics") { + const source: OtelCapture = { signal: input.signal, payload: input.payload }; + return otel(tool, source); + } + return []; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts new file mode 100644 index 000000000..a160bd81b --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts @@ -0,0 +1,176 @@ +import { execFile } from "node:child_process"; +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { claudeConfigDir } from "../../domain/claude-projects-root.js"; +import type { UsageTool } from "../../domain/usage-record.js"; +import claudeTemplate from "./config/claude.json" with { type: "json" }; +import codexTemplate from "./config/codex.json" with { type: "json" }; +import opencodeTemplate from "./config/opencode.json" with { type: "json" }; +import { mergeCodexExporters, mergeJsonEnvironment, rejectConflict } from "./config-edits.js"; + +export const OPENCODE_CONFIG_ID = "opencode"; +export const CLAUDE_CONFIG_ID = "claude"; +export const CODEX_CONFIG_ID = "codex"; +export const OTEL_CONFIGS = [CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, OPENCODE_CONFIG_ID] as const; +export const OTEL_SYNC_CONFIGS = [CLAUDE_CONFIG_ID, CODEX_CONFIG_ID] as const; +export type SyncConfig = (typeof OTEL_SYNC_CONFIGS)[number]; +const usageToolByOtelConfig: Record = { + [CLAUDE_CONFIG_ID]: "claude-code", + [CODEX_CONFIG_ID]: CODEX_CONFIG_ID, +}; +export const usageToolForOtel = (config: SyncConfig): UsageTool => usageToolByOtelConfig[config]; +export const OTEL_ENDPOINTS = ["dev", "prod"] as const; +export type Config = (typeof OTEL_CONFIGS)[number]; +export type Endpoint = (typeof OTEL_ENDPOINTS)[number]; + +export interface Host { + readonly home: string; + readonly env: NodeJS.ProcessEnv; + readonly opencodeVersion?: () => Promise; +} + +export interface Edit { + readonly path: string; + readonly initial: string; + readonly update: (content: string) => string; + readonly launch: string; +} + +const COLLECTOR = "http://127.0.0.1:14318"; +const TOKEN_PATH = ".local/share/aidd-otel-observation/collector-token"; +const SIGNALS = ["LOGS", "METRICS", "TRACES"] as const; + +export function interpolate(template: Readonly>, token: string) { + return Object.fromEntries( + Object.entries(template).map(([key, value]) => [ + key, + value.replace("{collector}", COLLECTOR).replace("{token}", token), + ]) + ); +} + +export async function readToken(home: string): Promise { + try { + const handle = await open(join(home, TOKEN_PATH), constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const info = await handle.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) throw new Error(); + const token = (await handle.readFile("utf8")).trim(); + if (!/^[a-zA-Z0-9_-]+$/.test(token)) throw new Error(); + return token; + } finally { + await handle.close(); + } + } catch { + throw new Error(`A valid private collector token is required at ~/${TOKEN_PATH} (mode 0600).`); + } +} + +function claudeEdit(host: Host, token: string): Edit { + const additions = interpolate(claudeTemplate.env, token); + return { + path: join(claudeConfigDir(host.env.CLAUDE_CONFIG_DIR, host.home), "settings.json"), + initial: "{}\n", + launch: "Restart Claude Code to apply the configuration.", + update: (content) => + mergeJsonEnvironment( + content, + additions, + (env) => { + rejectConflict( + "OTEL_EXPORTER_OTLP_ENDPOINT", + env.OTEL_EXPORTER_OTLP_ENDPOINT, + `${COLLECTOR}/claude` + ); + for (const signal of SIGNALS) { + for (const key of ["HEADERS", "PROTOCOL"]) { + const name = `OTEL_EXPORTER_OTLP_${signal}_${key}`; + rejectConflict(name, env[name], additions[`OTEL_EXPORTER_OTLP_${key}`]); + } + } + }, + claudeTemplate.defaults + ), + }; +} + +async function requireOpenCodeV2(host: Host): Promise { + let output: string; + try { + output = host.opencodeVersion + ? await host.opencodeVersion() + : ( + await promisify(execFile)(OPENCODE_CONFIG_ID, ["--version"], { + env: host.env, + timeout: 5000, + }) + ).stdout; + } catch { + throw new Error("Cannot determine OpenCode version. Make opencode available on PATH."); + } + const version = /^\s*(?:opencode\s+)?v?2\.(\d+)\.(\d+)\s*$/i.exec(output); + if (version) { + const minor = Number(version[1]); + if (minor > 2 || (minor === 2 && Number(version[2]) >= 5)) return; + } + throw new Error("Unsupported OpenCode version. Use OpenCode 2.2.5 or newer within 2.x."); +} + +function openCodeGlobalDir(host: Host): string { + return join(host.env.XDG_CONFIG_HOME || join(host.home, ".config"), OPENCODE_CONFIG_ID); +} + +export function openCodeConfigDir(host: Host): string { + return host.env.OPENCODE_CONFIG_DIR || openCodeGlobalDir(host); +} + +async function openCodeEdit(host: Host, token: string): Promise { + await requireOpenCodeV2(host); + const directory = openCodeConfigDir(host); + const additions = interpolate(opencodeTemplate.env, token); + return { + path: join(directory, "service.json"), + initial: "{}\n", + launch: "Restart the OpenCode background service with opencode service restart.", + update: (content) => mergeJsonEnvironment(content, additions, () => {}), + }; +} + +export async function configurationEdit(config: Config, endpoint: Endpoint, host: Host) { + if (endpoint === "prod") { + throw new Error( + "OTLP production configuration is unavailable until the Gouvernail endpoint is published." + ); + } + if (process.platform === "win32") { + throw new Error("OTLP initialization requires private POSIX configuration files."); + } + const token = await readToken(host.home); + let edit: Edit; + switch (config) { + case "claude": + edit = claudeEdit(host, token); + break; + case "codex": + edit = { + path: join(host.env.CODEX_HOME || join(host.home, ".codex"), "config.toml"), + initial: "", + launch: "Restart Codex to apply the configuration.", + update: (content) => + mergeCodexExporters( + content, + interpolate(codexTemplate.exporters, token), + token, + codexTemplate.defaults, + codexTemplate.toolResult + ), + }; + break; + case OPENCODE_CONFIG_ID: + edit = await openCodeEdit(host, token); + break; + } + return edit; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts b/cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts new file mode 100644 index 000000000..47fa40b17 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts @@ -0,0 +1,138 @@ +import { asPlainObject } from "../../../../kernel/reading/plain-object.js"; +import type { Transfer } from "./actions/sync.js"; + +const UUID = /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/i; +const TEXT_KEYS = new Set(["thread.id", "turn.id", "model", "codex.turn.reasoning_effort"]); +const COUNTER_KEYS = new Set([ + "codex.turn.token_usage.input_tokens", + "codex.turn.token_usage.cached_input_tokens", + "codex.turn.token_usage.cache_write_input_tokens", + "codex.turn.token_usage.non_cached_input_tokens", + "codex.turn.token_usage.output_tokens", + "codex.turn.token_usage.reasoning_output_tokens", + "codex.turn.token_usage.total_tokens", +]); +const obj = (value: unknown) => asPlainObject(value) ?? {}; +const list = (value: unknown) => (Array.isArray(value) ? value : []); + +function safeAttributes( + value: unknown +): Map { + const result = new Map(); + for (const raw of list(value)) { + const entry = obj(raw); + const key = entry.key; + if (typeof key !== "string" || (!TEXT_KEYS.has(key) && !COUNTER_KEYS.has(key))) continue; + const content = obj(entry.value); + // tracing adds a runtime integer thread.id alongside Codex's string UUID thread.id. + if (key === "thread.id" && content.intValue !== undefined && content.stringValue === undefined) + continue; + let safe: { stringValue: string } | { intValue: string | number }; + if (TEXT_KEYS.has(key)) { + if (typeof content.stringValue !== "string" || Object.keys(content).length !== 1) + throw new Error("Invalid Codex turn attribute."); + const text = content.stringValue; + if ( + key === "thread.id" || key === "turn.id" + ? !UUID.test(text) + : !/^[A-Za-z0-9][A-Za-z0-9._:/#-]{0,127}$/.test(text) + ) + throw new Error("Invalid Codex turn attribute."); + safe = { stringValue: text }; + } else { + const number = content.intValue; + if ( + Object.keys(content).length !== 1 || + !( + (typeof number === "string" && + /^\d+$/.test(number) && + Number.isSafeInteger(Number(number))) || + (typeof number === "number" && Number.isSafeInteger(number) && number >= 0) + ) + ) + throw new Error("Invalid Codex turn counter."); + safe = { intValue: number as string | number }; + } + const previous = result.get(key); + if (previous && JSON.stringify(previous) !== JSON.stringify(safe)) + throw new Error("Conflicting Codex turn attribute."); + result.set(key, safe); + } + return result; +} + +function time(value: unknown): string | undefined { + if (typeof value !== "string" || !/^\d{1,20}$/.test(value)) return undefined; + const nanos = BigInt(value); + if (nanos <= 0n || nanos > 18446744073709551615n) return undefined; + try { + return new Date(Number(nanos / 1_000_000n)).toISOString(); + } catch { + return undefined; + } +} + +/** Only provider turn spans are exported: no events, links, paths or prompt content. */ +export function codexTurnTransfers(payload: unknown, session: string): Transfer[] { + const transfers: Transfer[] = []; + for (const rawResource of list(obj(payload).resourceSpans)) { + for (const rawScope of list(obj(rawResource).scopeSpans)) { + for (const rawSpan of list(obj(rawScope).spans)) { + const span = obj(rawSpan); + if (span.name !== "session_task.turn") continue; + const identity = safeAttributes( + list(span.attributes).filter((item) => obj(item).key === "thread.id") + ); + const thread = obj(identity.get("thread.id")).stringValue; + if (typeof thread !== "string") throw new Error("Codex turn session ID is missing."); + if (thread.toLowerCase() !== session.toLowerCase()) continue; + const attrs = safeAttributes(span.attributes); + if (!attrs.has("turn.id")) throw new Error("Codex turn ID is missing."); + if ( + typeof span.traceId !== "string" || + !/^[a-f0-9]{32}$/i.test(span.traceId) || + /^0+$/.test(span.traceId) || + typeof span.spanId !== "string" || + !/^[a-f0-9]{16}$/i.test(span.spanId) || + /^0+$/.test(span.spanId) + ) + throw new Error("Invalid Codex trace identifier."); + const start = time(span.startTimeUnixNano); + const at = time(span.endTimeUnixNano); + if ( + !start || + !at || + BigInt(String(span.startTimeUnixNano)) > BigInt(String(span.endTimeUnixNano)) + ) + throw new Error("Invalid Codex turn interval."); + const safe = [...attrs].map(([key, value]) => ({ key, value })); + transfers.push({ + signal: "traces", + start, + at, + payload: { + resourceSpans: [ + { + scopeSpans: [ + { + spans: [ + { + name: "session_task.turn", + traceId: span.traceId, + spanId: span.spanId, + startTimeUnixNano: span.startTimeUnixNano, + endTimeUnixNano: span.endTimeUnixNano, + attributes: safe, + }, + ], + }, + ], + }, + ], + }, + }); + } + } + } + return transfers; +} diff --git a/cli/src/presentation/commands/setup.ts b/cli/src/presentation/commands/setup.ts index bcd93efc1..cfbe9cb55 100644 --- a/cli/src/presentation/commands/setup.ts +++ b/cli/src/presentation/commands/setup.ts @@ -13,6 +13,7 @@ import { } from "../../contexts/tools/domain/registry.js"; import type { ToolId } from "../../kernel/tool.js"; import { AI_TOOL_IDS, IDE_TOOL_IDS } from "../../kernel/tool.js"; +import { finishInstallationId } from "../../runtime/filesystem/installation-identity.js"; import { createDeps } from "../../runtime/wiring/framework.js"; import { printDetectedContext, @@ -187,6 +188,7 @@ export function registerSetupCommand(program: Command): void { if (interactive) printWelcomeBanner(output); const deps = await createDeps(projectRoot, { verbose }, output); + await deps.checkInstallationId(); const result = await new SetupUseCase( deps.fs, @@ -207,6 +209,7 @@ export function registerSetupCommand(program: Command): void { printSetupOutcome(output, result, verbose); if (interactive) printNextSteps(output, result.install.results.length > 0); reportSyncActivation(output, result.activation); + await finishInstallationId(deps.ensureInstallationId, "Setup completed"); } catch (error) { errorHandler.handle(error); } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index f89bff892..e18752c9f 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -25,8 +25,16 @@ import { } from "../../contexts/telemetry/domain/report/usage-report.js"; import { USAGE_AGENTS, USAGE_TOOLS } from "../../contexts/telemetry/domain/usage-record.js"; import type { Prompter } from "../../kernel/ports/prompter.js"; +import { finishInstallationId } from "../../runtime/filesystem/installation-identity.js"; import { createDeps } from "../../runtime/wiring/framework.js"; import type { TelemetryDeps } from "../../runtime/wiring/telemetry.js"; +import { + checkOtel, + collectOtel, + configureOtel, + otelOptions, + runOtelSession, +} from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, printIdentityRule, @@ -128,6 +136,142 @@ export function registerTelemetryCommand(program: Command): void { const telemetry = program .command("telemetry") .description("Measure what Claude Code and Codex sessions consume, locally"); + const otel = telemetry.command("otel").description("Configure native OpenTelemetry exporters"); + for (const action of ["init", "verify"] as const) { + otel + .command(action) + .description( + action === "init" + ? "Enable detailed local OTLP export in one tool's user configuration" + : "Check native OTLP configuration and existing local capture without changing files" + ) + .addOption( + new Option( + "--config ", + `User configuration to ${action === "init" ? "merge" : "verify"}` + ) + .choices(otelOptions.configs) + .makeOptionMandatory() + ) + .addOption( + new Option("--endpoint ", "Collector destination") + .choices(otelOptions.endpoints) + .makeOptionMandatory() + ) + .action( + async (options: { + config: (typeof otelOptions.configs)[number]; + endpoint: (typeof otelOptions.endpoints)[number]; + }) => { + const { output } = parseGlobalOptions(program); + try { + if (action === "init") { + const result = await configureOtel(options.config, options.endpoint); + output.info( + result.changed ? `Updated ${result.path}` : `Already configured in ${result.path}` + ); + output.info( + "Local OTLP capture may include prompts, responses, commands and tool output." + ); + output.info(result.launch); + } else { + const result = await checkOtel(options.config, options.endpoint); + output.info(`Configuration verified in ${result.path}`); + output.info( + `${result.lines} captured lines in ${result.capture}; last modified ${result.modifiedAt}` + ); + output.info( + `${result.observations} normalized observations; ${result.unrecognisedLines} unrecognised lines` + ); + output.info("Existing capture does not prove delivery from the current session."); + } + } catch (error) { + new ErrorHandler(output).handle(error); + } + } + ); + } + + otel + .command("collect") + .description("Capture authenticated OTLP on a local loopback port") + .option("--port ", "Loopback TCP port; 0 selects a free port", "14318") + .action(async (options: { port: string }) => { + const { output } = parseGlobalOptions(program); + try { + if (!/^\d+$/.test(options.port)) throw new Error("Collector port must be an integer."); + const collector = await collectOtel(Number(options.port)); + output.info(`OTLP collector listening on http://127.0.0.1:${collector.port}`); + } catch (error) { + new ErrorHandler(output).handle(error); + } + }); + + otel + .command("run ") + .description("Run one fresh provider session and send its allowed OTLP observations") + .addOption( + new Option("--config ", "Tool to run") + .choices(otelOptions.configs) + .makeOptionMandatory() + ) + .requiredOption("--tenant ", "Gouvernail tenant slug") + .requiredOption("--endpoint ", "Loopback HTTP origin") + .option("--yes", "Consent to capture and send telemetry from this execution") + .option("--model ", "Provider model to use") + .option( + "--follow-up ", + "Send a second prompt in the same session (Claude and Codex retain local history)" + ) + .option("--include-repository", "Include repository metadata for Claude Code") + .option("--include-installation-id", "Send the AIDD installation ID (not a user identity)") + .option("--keep-capture", "Retain the private local OTLP capture") + .action( + async ( + prompt: string, + options: { + config: (typeof otelOptions.configs)[number]; + tenant: string; + endpoint: string; + yes?: boolean; + model?: string; + followUp?: string; + includeRepository?: boolean; + includeInstallationId?: boolean; + keepCapture?: boolean; + } + ) => { + const { output, projectRoot } = parseGlobalOptions(program); + const abort = new AbortController(); + const cancel = () => abort.abort(); + process.once("SIGINT", cancel); + process.once("SIGTERM", cancel); + try { + if (!options.yes) + throw new Error( + "Pass --yes to consent to OTLP capture and delivery for this execution." + ); + const sent = await runOtelSession({ + ...options, + cwd: projectRoot, + prompt, + signal: abort.signal, + token: process.env.AIDD_OTEL_TOKEN ?? "", + retainCapture: options.keepCapture + ? (path) => output.info(`Private OTLP capture: ${path}`) + : undefined, + }); + if (!sent) + throw new Error("The execution produced no supported OTLP observations to send."); + output.info(`Sent ${sent} OTLP observations from this execution.`); + } catch (error) { + new ErrorHandler(output).handle(error); + } finally { + process.off("SIGINT", cancel); + process.off("SIGTERM", cancel); + } + } + ); telemetry .command("on") @@ -173,7 +317,11 @@ export function registerTelemetryCommand(program: Command): void { ); process.exit(1); } + await deps.checkInstallationId(); const result = await deps.telemetry.telemetryOnUseCase.execute(projectRoot, choice); + if (result.status === "on") { + await finishInstallationId(deps.ensureInstallationId, "Measurement is on"); + } printOnResult(output, result); if (result.status === "refused") process.exit(1); }); @@ -372,6 +520,8 @@ export function registerTelemetryCommand(program: Command): void { ? await use.show() : await use.set(id); printIdentityResult(output, result); + if (id === undefined && !cmdOptions.off) + output.info(`Installation ID: ${(await deps.checkInstallationId()) ?? "unavailable"}.`); if (result.status === "refused") process.exit(1); }); }); diff --git a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts index bbc35007b..f6e2d407b 100644 --- a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts +++ b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts @@ -112,6 +112,9 @@ export function printOffResult(output: CLIOutput, result: OffResult): void { ? "Measurement is off for this clone. What was measured stays until `aidd telemetry forget --yes`." : "Measurement was not on for this clone." ); + output.info( + "Provider OTLP configuration and collector are unchanged and may still collect data." + ); } const WHAT: Readonly> = { diff --git a/cli/src/runtime/filesystem/installation-identity.ts b/cli/src/runtime/filesystem/installation-identity.ts new file mode 100644 index 000000000..574f8f8d0 --- /dev/null +++ b/cli/src/runtime/filesystem/installation-identity.ts @@ -0,0 +1,87 @@ +import { randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import { link, open, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { isErrnoException, tryParseJson } from "../../kernel/reading/json-file.js"; +import { asPlainObject } from "../../kernel/reading/plain-object.js"; +import { PrivateStorageAdapter } from "./private-storage-adapter.js"; + +const UUID_V4 = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +async function readInstallationId(path: string): Promise { + const handle = await open( + path, + process.platform === "win32" + ? constants.O_RDONLY + : constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (isErrnoException(error) && error.code === "ENOENT") return null; + throw error; + }); + if (handle === null) return null; + try { + const info = await handle.stat(); + if (!info.isFile() || (process.platform !== "win32" && (info.mode & 0o077) !== 0)) { + throw new Error("Installation identity must be a private regular file."); + } + const parsed = tryParseJson(await handle.readFile("utf8")); + const object = parsed.ok ? asPlainObject(parsed.value) : null; + const id = object?.installation_id; + if (typeof id !== "string" || !UUID_V4.test(id)) { + throw new Error("Installation identity is invalid; the existing file was left unchanged."); + } + return id; + } finally { + await handle.close(); + } +} + +/** Validate a pre-existing ID without creating any files. */ +export function checkInstallationId(userConfigDir: string): Promise { + return readInstallationId(join(userConfigDir, "installation", "id.json")); +} + +/** The caller's operation has already taken effect when this is called. */ +export async function finishInstallationId( + ensure: () => Promise, + outcome: string +): Promise { + try { + await ensure(); + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + throw new Error(`${outcome}, but the installation ID could not be saved: ${reason}. Retry.`); + } +} + +/** A machine-local installation identifier, not a person, credential or authentication token. + * A completed temporary file is linked into place without replacing a concurrent initializer. */ +export async function ensureInstallationId(userConfigDir: string): Promise { + const directory = join(userConfigDir, "installation"); + const path = join(directory, "id.json"); + await new PrivateStorageAdapter().ensureDirectory(directory); + const existing = await readInstallationId(path); + if (existing !== null) return existing; + + const id = randomUUID(); + const temporary = join(directory, `${id}.tmp`); + let created = false; + try { + await writeFile(temporary, `${JSON.stringify({ installation_id: id })}\n`, { + flag: "wx", + mode: 0o600, + }); + created = true; + try { + await link(temporary, path); + return id; + } catch (error) { + if (!isErrnoException(error) || error.code !== "EEXIST") throw error; + const winner = await readInstallationId(path); + if (winner === null) throw new Error("Installation identity disappeared during setup."); + return winner; + } + } finally { + if (created) await rm(temporary, { force: true }); + } +} diff --git a/cli/src/runtime/wiring/framework.ts b/cli/src/runtime/wiring/framework.ts index 0ffe39907..8b800cc16 100644 --- a/cli/src/runtime/wiring/framework.ts +++ b/cli/src/runtime/wiring/framework.ts @@ -108,6 +108,7 @@ import type { CredentialStore } from "../auth/ports/credential-store.js"; import { atomicWriteFile } from "../filesystem/atomic-write.js"; import { FileAdapter } from "../filesystem/file-adapter.js"; import { HasherAdapter } from "../filesystem/hasher-adapter.js"; +import { checkInstallationId, ensureInstallationId } from "../filesystem/installation-identity.js"; import { HttpClient } from "../http/http-client.js"; import { PlatformAdapter } from "../platform/platform-adapter.js"; import { InquirerPrompterAdapter, SilentPrompterAdapter } from "../prompter/prompter-adapter.js"; @@ -129,6 +130,8 @@ interface GlobalOptions { } interface Deps { + checkInstallationId: () => Promise; + ensureInstallationId: () => Promise; fs: FileReader & FileWriter & FileMerger; manifestRepo: ManifestRepository; /** `--scope user`'s own manifest repository: `userConfigDir()/manifest.json`, never nested @@ -602,6 +605,8 @@ export async function createDeps( const listInstalledRulesUseCase = new ListInstalledRulesUseCase(fs); const checkUpdateUseCase = new CheckUpdateUseCase(cliUpdater, currentVersionProvider, logger, fs); const deps: Deps = { + checkInstallationId: () => checkInstallationId(userConfigDir()), + ensureInstallationId: () => ensureInstallationId(userConfigDir()), validateRecipesUseCase: new ValidateRecipesUseCase(new RecipeFilesAdapter()), fs, manifestRepo, diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts new file mode 100644 index 000000000..bdff8de8f --- /dev/null +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -0,0 +1,54 @@ +import { homedir } from "node:os"; +import { startOtelCollector } from "../../contexts/telemetry/infrastructure/otel/actions/collect.js"; +import { initializeOtel } from "../../contexts/telemetry/infrastructure/otel/actions/init.js"; +import { runOtel } from "../../contexts/telemetry/infrastructure/otel/actions/run.js"; +import { verifyOtel } from "../../contexts/telemetry/infrastructure/otel/actions/verify.js"; +import { + OTEL_CONFIGS, + OTEL_ENDPOINTS, +} from "../../contexts/telemetry/infrastructure/otel/settings.js"; +import { + checkInstallationId, + ensureInstallationId, + finishInstallationId, +} from "../filesystem/installation-identity.js"; +import { userConfigDirOf } from "../user-config-dir.js"; + +export async function runOtelSession( + options: Parameters[0] & { includeInstallationId?: boolean } +) { + const installationId = options.includeInstallationId + ? await checkInstallationId(userConfigDirOf(process.env, homedir())) + : undefined; + if (options.includeInstallationId && !installationId) + throw new Error("No AIDD installation ID exists. Run framework setup before including it."); + return runOtel({ ...options, installationId: installationId ?? undefined }); +} + +export function collectOtel(port: number) { + return startOtelCollector({ home: homedir(), port }); +} + +export const otelOptions = { + configs: OTEL_CONFIGS, + endpoints: OTEL_ENDPOINTS, +}; + +export async function configureOtel( + config: (typeof OTEL_CONFIGS)[number], + endpoint: (typeof OTEL_ENDPOINTS)[number] +) { + const home = homedir(); + const directory = userConfigDirOf(process.env, home); + if (endpoint !== "prod") await checkInstallationId(directory); + const result = await initializeOtel(config, endpoint, { home, env: process.env }); + await finishInstallationId(() => ensureInstallationId(directory), "OTLP configuration completed"); + return result; +} + +export function checkOtel( + config: (typeof OTEL_CONFIGS)[number], + endpoint: (typeof OTEL_ENDPOINTS)[number] +) { + return verifyOtel(config, endpoint, { home: homedir(), env: process.env }); +} diff --git a/cli/tests/architecture/tool-addition-cost.arch.test.ts b/cli/tests/architecture/tool-addition-cost.arch.test.ts index bb6160712..bbb57b51b 100644 --- a/cli/tests/architecture/tool-addition-cost.arch.test.ts +++ b/cli/tests/architecture/tool-addition-cost.arch.test.ts @@ -92,6 +92,8 @@ const BASELINE: readonly { readonly path: string; readonly named: number }[] = [ }, // Claude Code's own settings files, read to say how long it keeps a transcript. { path: "src/contexts/telemetry/infrastructure/switch/claude-settings-adapter.ts", named: 1 }, + // OTLP is an explicit, separate support surface: three observed native exporters. + { path: "src/contexts/telemetry/infrastructure/otel/settings.ts", named: 3 }, // Cursor's project hooks file, named after the tool whose file it is: the directory it // writes into is Cursor's own, not a list a sixth tool joins. { path: "src/contexts/tools/domain/formats/cursor-hooks-project-merge.ts", named: 1 }, diff --git a/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts new file mode 100644 index 000000000..b8092e585 --- /dev/null +++ b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts @@ -0,0 +1,564 @@ +import { describe, expect, it } from "vitest"; +import { + transformCapturedLine, + transformClaude, + transformCodex, + transformOpenCode, +} from "../../../../../src/contexts/telemetry/infrastructure/otel/observations.js"; + +const attr = (key: string, value: unknown) => ({ key, value }); +const string = (value: string) => ({ stringValue: value }); +const integer = (value: number) => ({ intValue: String(value) }); + +function log(attributes: unknown[]) { + return { + signal: "logs" as const, + payload: { + resourceLogs: [ + { + resource: { attributes: [attr("service.version", string("1.2.3"))] }, + scopeLogs: [{ logRecords: [{ timeUnixNano: "1700000000000000000", attributes }] }], + }, + ], + }, + }; +} + +describe("telemetry observations", () => { + it("maps Claude request fields without assigning missing fields zero", () => { + const result = transformClaude( + log([ + attr("event.name", string("api_request")), + attr("session.id", string("s1")), + attr("model", string("m1")), + attr("input_tokens", integer(17)), + attr("cache_read_tokens", integer(9)), + ]) + ); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ + tool: "claude", + signal: "log", + eventType: "api_request", + sessionId: "s1", + model: "m1", + toolVersion: "1.2.3", + timestamp: "2023-11-14T22:13:20.000Z", + tokens: { input: 17, cacheRead: 9 }, + usageScope: "event", + }); + expect(result[0]?.tokens?.output).toBeUndefined(); + }); + + it("maps Codex tool records and treats success as a boolean", () => { + const result = transformCodex( + log([ + attr("event.name", string("codex.tool_result")), + attr("conversation.id", string("s2")), + attr("call_id", string("call-1")), + attr("tool_name", string("read_file")), + attr("arguments", string('{"path":"x"}')), + attr("output", string("done")), + attr("success", string("true")), + ]) + ); + expect(result[0]).toMatchObject({ + tool: "codex", + sessionId: "s2", + toolCallId: "call-1", + toolName: "read_file", + toolInput: '{"path":"x"}', + toolOutput: "done", + success: true, + }); + }); +}); + +describe("source fidelity", () => { + it("keeps metric sums separate from per-request token counts", () => { + const capture = { + signal: "metrics" as const, + payload: { + resourceMetrics: [ + { + resource: { attributes: [] }, + scopeMetrics: [ + { + metrics: [ + { + name: "codex.turn.token_usage", + histogram: { + dataPoints: [ + { sum: 42, attributes: [attr("token_type", string("cached_input"))] }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }; + expect(transformCodex(capture)[0]).toMatchObject({ + signal: "metric", + metricName: "codex.turn.token_usage", + metricValue: 42, + usageScope: "metric-aggregate", + tokens: { cacheRead: 42 }, + }); + }); + + it("keeps span tool output linked to its call without copying its token counts", () => { + const capture = { + signal: "traces" as const, + payload: { + resourceSpans: [ + { + resource: { attributes: [] }, + scopeSpans: [ + { + spans: [ + { + name: "claude_code.tool", + traceId: "trace-1", + spanId: "span-1", + attributes: [ + attr("tool_use_id", string("call-1")), + attr("tool_name", string("read")), + attr("input_tokens", integer(12)), + ], + events: [{ name: "tool.output", attributes: [attr("output", string("done"))] }], + }, + ], + }, + ], + }, + ], + }, + }; + const rows = transformClaude(capture); + expect(rows).toHaveLength(2); + expect(rows[1]).toMatchObject({ + signal: "span-event", + traceId: "trace-1", + toolCallId: "call-1", + toolOutput: "done", + }); + expect(rows[1]?.tokens).toBeUndefined(); + }); +}); + +describe("Token semantics", () => { + it("keeps input token semantics explicit for Claude and Codex", () => { + const claude = transformClaude(log([attr("input_tokens", integer(17))])); + const codex = transformCodex(log([attr("input_token_count", integer(17))])); + expect(claude[0]?.tokenInputIncludesCache).toBe(false); + expect(codex[0]?.tokenInputIncludesCache).toBe(true); + }); +}); + +describe("source-specific fields", () => { + it("maps an OpenCode OTLP AI SDK trace without treating parent and child spans as calls", () => { + const capture = { + signal: "traces" as const, + payload: { + resourceSpans: [ + { + scopeSpans: [ + { + spans: [ + { + name: "ai.streamText.doStream", + attributes: [ + attr("session.id", string("s")), + attr("ai.model.id", string("m")), + attr("ai.model.provider", string("p")), + attr("ai.response.text", string("ok")), + attr("ai.usage.inputTokens", integer(5)), + attr("ai.usage.outputTokens", integer(2)), + attr("ai.usage.cachedInputTokens", integer(1)), + attr("ai.usage.inputTokenDetails.noCacheTokens", integer(4)), + attr("ai.usage.totalTokens", integer(7)), + ], + }, + ], + }, + ], + }, + ], + }, + }; + expect(transformOpenCode(capture)[0]).toMatchObject({ + sessionId: "s", + model: "m", + provider: "p", + response: "ok", + usageScope: "span-snapshot", + tokenInputIncludesCache: true, + tokenOutputIncludesReasoning: true, + tokens: { input: 5, inputNonCached: 4, output: 2, cacheRead: 1, total: 7 }, + }); + }); + it("does not turn a Codex thread id into a session id", () => { + const [threadOnly] = transformCodex(log([attr("thread.id", string("child"))])); + expect(threadOnly?.sessionId).toBeUndefined(); + expect(threadOnly?.threadId).toBe("child"); + const [both] = transformCodex( + log([attr("conversation.id", string("parent")), attr("thread.id", string("child"))]) + ); + expect(both).toMatchObject({ sessionId: "parent", threadId: "child" }); + }); + + it("preserves Codex non-cached and total counters and its tool aliases", () => { + const [row] = transformCodex( + log([ + attr("codex.turn.token_usage.non_cached_input_tokens", integer(9)), + attr("codex.turn.token_usage.total_tokens", integer(23)), + attr("tool_token_count", integer(4)), + attr("tool.name", string("read")), + attr("tool.call_id", string("call")), + attr("cwd", string("/project")), + ]) + ); + expect(row).toMatchObject({ + toolName: "read", + toolCallId: "call", + cwd: "/project", + tokens: { inputNonCached: 9, total: 23, tool: 4 }, + }); + }); +}); + +describe("OTLP field aliases", () => { + it.each([ + ["input_tokens", "input"], + ["input_token_count", "input"], + ["codex.turn.token_usage.input_tokens", "input"], + ["gen_ai.usage.input_tokens", "input"], + ["ai.usage.inputTokens", "input"], + ["codex.turn.token_usage.non_cached_input_tokens", "inputNonCached"], + ["ai.usage.inputTokenDetails.noCacheTokens", "inputNonCached"], + ["output_tokens", "output"], + ["output_token_count", "output"], + ["codex.turn.token_usage.output_tokens", "output"], + ["gen_ai.usage.output_tokens", "output"], + ["ai.usage.outputTokens", "output"], + ["cache_read_tokens", "cacheRead"], + ["cached_token_count", "cacheRead"], + ["codex.turn.token_usage.cached_input_tokens", "cacheRead"], + ["gen_ai.usage.cache_read.input_tokens", "cacheRead"], + ["ai.usage.cachedInputTokens", "cacheRead"], + ["ai.usage.inputTokenDetails.cacheReadTokens", "cacheRead"], + ["cache_creation_tokens", "cacheWrite"], + ["cache_write_token_count", "cacheWrite"], + ["codex.turn.token_usage.cache_write_input_tokens", "cacheWrite"], + ["gen_ai.usage.cache_write.input_tokens", "cacheWrite"], + ["cache_creation_5m_tokens", "cacheWrite5m"], + ["cache_creation_1h_tokens", "cacheWrite1h"], + ["reasoning_token_count", "reasoning"], + ["codex.turn.token_usage.reasoning_output_tokens", "reasoning"], + ["codex.usage.reasoning_output_tokens", "reasoning"], + ["ai.usage.reasoningTokens", "reasoning"], + ["ai.usage.outputTokenDetails.reasoningTokens", "reasoning"], + ["codex.turn.token_usage.total_tokens", "total"], + ["codex.usage.total_tokens", "total"], + ["ai.usage.totalTokens", "total"], + ["tool_token_count", "tool"], + ])("maps %s to %s", (key, normalized) => { + const [row] = transformCodex(log([attr(key, integer(7))])); + expect(row?.tokens).toMatchObject({ [normalized]: 7 }); + }); + + it.each([ + ["anthropic.provider", "provider"], + ["provider_name", "provider"], + ["provider", "provider"], + ["gen_ai.provider.name", "provider"], + ["gen_ai.system", "provider"], + ["ai.model.provider", "provider"], + ["model", "model"], + ["gen_ai.request.model", "model"], + ["ai.model.id", "model"], + ["request_id", "requestId"], + ["gen_ai.response.id", "requestId"], + ["ai.response.id", "requestId"], + ["tool_use_id", "toolCallId"], + ["call_id", "toolCallId"], + ["gen_ai.tool.call.id", "toolCallId"], + ["tool.call_id", "toolCallId"], + ["tool_name", "toolName"], + ["gen_ai.tool.name", "toolName"], + ["tool.name", "toolName"], + ["session.id", "sessionId"], + ["conversation.id", "sessionId"], + ])("maps %s to %s", (key, normalized) => { + const [row] = transformClaude(log([attr(key, string("example"))])); + expect(row).toMatchObject({ [normalized]: "example" }); + }); + + it("retains zero counters but leaves invalid numbers and absent usage unknown", () => { + const [zero] = transformClaude(log([attr("input_tokens", integer(0))])); + expect(zero?.tokens?.input).toBe(0); + for (const invalid of ["", "-1", "NaN", "Infinity"]) { + const [row] = transformClaude(log([attr("input_tokens", string(invalid))])); + expect(row?.tokens).toBeUndefined(); + } + }); + + it("decodes source attributes including booleans, arrays and nested maps", () => { + const [row] = transformCodex( + log([ + attr("event.name", string("codex.tool_result")), + attr("success", { boolValue: false }), + attr("arguments", { kvlistValue: { values: [attr("path", string("src/x"))] } }), + attr("output", { arrayValue: { values: [string("done"), integer(2)] } }), + ]) + ); + expect(row).toMatchObject({ + success: false, + toolInput: { path: "src/x" }, + toolOutput: ["done", "2"], + }); + }); +}); + +describe("OTLP event and metric semantics", () => { + it.each([ + ["user_prompt", "prompt", "prompt"], + ["user_prompt", "prompt_text", "prompt"], + ["codex.user_prompt", "prompt", "prompt"], + ["assistant_response", "response", "response"], + ["codex.agent_response", "response", "response"], + ["tool_result", "output", "toolOutput"], + ["codex.tool_result", "output", "toolOutput"], + ])("maps %s %s only to its matching field", (event, key, output) => { + const [row] = transformClaude( + log([attr("event.name", string(event)), attr(key, string("example"))]) + ); + expect(row).toMatchObject({ eventType: event, [output]: "example" }); + }); + + it("does not expose a prompt or response attached to an unrelated log event", () => { + const [row] = transformClaude( + log([ + attr("event.name", string("api_request")), + attr("prompt", string("secret")), + attr("response", string("response")), + attr("output", string("output")), + ]) + ); + expect(row?.prompt).toBeUndefined(); + expect(row?.response).toBeUndefined(); + expect(row?.toolOutput).toBeUndefined(); + }); + + it.each([ + ["claude_code.token.usage", "type", "input", "input", "sum"], + ["claude_code.token.usage", "type", "output", "output", "gauge"], + ["claude_code.token.usage", "type", "cacheRead", "cacheRead", "sum"], + ["claude_code.token.usage", "type", "cacheCreation", "cacheWrite", "sum"], + ["codex.turn.token_usage", "token_type", "cached_input", "cacheRead", "histogram"], + ["codex.turn.token_usage", "token_type", "cache_write_input", "cacheWrite", "histogram"], + ["codex.turn.token_usage", "token_type", "reasoning_output", "reasoning", "histogram"], + ])("maps aggregate %s %s", (name, attribute, type, normalized, kind) => { + const metric = { + name, + [kind]: { + dataPoints: [ + { + asInt: "12", + timeUnixNano: "1700000000000000000", + startTimeUnixNano: "1699999999000000000", + attributes: [attr(attribute, string(type))], + }, + ], + }, + }; + const [row] = transformCodex({ + signal: "metrics", + payload: { resourceMetrics: [{ scopeMetrics: [{ metrics: [metric] }] }] }, + }); + expect(row).toMatchObject({ + eventType: name, + metricName: name, + metricType: type, + metricValue: 12, + timestamp: "2023-11-14T22:13:20.000Z", + startedAt: "2023-11-14T22:13:19.000Z", + usageScope: "metric-aggregate", + tokens: { [normalized]: 12 }, + }); + }); + + it("does not mistake an unrelated metric for a token aggregate", () => { + const metric = { + name: "other.metric", + sum: { dataPoints: [{ asInt: "9", attributes: [attr("type", string("input"))] }] }, + }; + const [row] = transformClaude({ + signal: "metrics", + payload: { resourceMetrics: [{ scopeMetrics: [{ metrics: [metric] }] }] }, + }); + expect(row).toMatchObject({ metricName: "other.metric", metricType: "input", metricValue: 9 }); + expect(row?.tokens).toBeUndefined(); + expect(row?.usageScope).toBeUndefined(); + }); +}); + +it("normalizes only OTLP capture envelopes", () => { + for (const [tool, payload] of [ + ["opencode", { type: "session.step.ended", data: { sessionID: "session" } }], + ["opencode", { type: "message.updated", properties: {} }], + ["opencode", { info: {}, messages: [] }], + ["claude", { type: "result", session_id: "session" }], + ["codex", { type: "turn.completed", usage: { input_tokens: 1 } }], + ] as const) { + expect(transformCapturedLine(tool, JSON.stringify(payload))).toEqual([]); + } + expect(transformCapturedLine("opencode", "not json")).toEqual([]); + expect(transformCapturedLine("opencode", JSON.stringify(log([])))).toHaveLength(1); +}); + +describe("session and trace details", () => { + it.each([ + ["false", false], + ["true", true], + ["unknown", undefined], + ])("normalizes observed tool success %s", (value, expected) => { + const [row] = transformCodex( + log([attr("event.name", string("codex.tool_result")), attr("success", string(value))]) + ); + expect(row?.success).toBe(expected); + }); + + it("does not invent timestamps, token semantics or usage for incomplete events", () => { + const source = log([attr("event.name", string("user_prompt"))]); + source.payload.resourceLogs[0].scopeLogs[0].logRecords[0].timeUnixNano = "not-a-time"; + const [claude] = transformClaude(source); + expect(claude?.timestamp).toBeUndefined(); + expect(claude?.usageScope).toBeUndefined(); + expect(claude?.tokenInputIncludesCache).toBeUndefined(); + expect(claude?.tokenOutputIncludesReasoning).toBeUndefined(); + const [opencode] = transformOpenCode(source); + expect(opencode?.timestamp).toBeUndefined(); + expect(opencode?.usageScope).toBeUndefined(); + expect(opencode?.tokenInputIncludesCache).toBeUndefined(); + expect(opencode?.tokenOutputIncludesReasoning).toBeUndefined(); + }); + + it("keeps provider correlation IDs from OTLP without deriving new IDs", () => { + const claude = log([ + attr("session.id", string("session")), + attr("prompt.id", string("prompt")), + attr("client_request_id", string("request")), + attr("message.uuid", string("message")), + ]); + const record = claude.payload.resourceLogs[0].scopeLogs[0].logRecords[0]; + Object.assign(record, { traceId: "trace", spanId: "span" }); + expect(transformClaude(claude)[0]).toMatchObject({ + sessionId: "session", + promptId: "prompt", + clientRequestId: "request", + messageId: "message", + traceId: "trace", + spanId: "span", + }); + const [codex] = transformCodex(log([attr("turn.id", string("turn"))])); + expect(codex?.turnId).toBe("turn"); + expect(codex?.sessionId).toBeUndefined(); + }); + + it("keeps Codex session, thread, model, request and token categories on one log record", () => { + const [row] = transformCodex( + log([ + attr("event.name", string("codex.sse_event")), + attr("event.timestamp", string("2026-11-04T01:00:00Z")), + attr("conversation.id", string("session")), + attr("thread.id", string("thread")), + attr("provider_name", string("provider")), + attr("model", string("model")), + attr("request_id", string("request")), + attr("input_token_count", integer(10)), + attr("output_token_count", integer(4)), + attr("cached_token_count", integer(7)), + attr("cache_write_token_count", integer(0)), + attr("reasoning_token_count", integer(2)), + attr("duration_ms", integer(300)), + attr("success", { boolValue: true }), + ]) + ); + expect(row).toMatchObject({ + timestamp: "2026-11-04T01:00:00Z", + sessionId: "session", + threadId: "thread", + provider: "provider", + model: "model", + requestId: "request", + durationMs: 300, + success: true, + tokenInputIncludesCache: true, + tokenOutputIncludesReasoning: true, + tokens: { input: 10, output: 4, cacheRead: 7, cacheWrite: 0, reasoning: 2 }, + }); + }); + + it("retains span timestamps, correlation and source response as a snapshot", () => { + const capture = { + signal: "traces" as const, + payload: { + resourceSpans: [ + { + scopeSpans: [ + { + spans: [ + { + name: "ai.streamText.doStream", + traceId: "trace", + spanId: "span", + parentSpanId: "parent", + startTimeUnixNano: "1700000000000000000", + endTimeUnixNano: "1700000001000000000", + attributes: [ + attr("session.id", string("session")), + attr("ai.response.text", string("done")), + attr("ai.usage.inputTokens", integer(9)), + ], + events: [ + { + name: "tool.output", + timeUnixNano: "1700000000500000000", + attributes: [attr("output", string("result"))], + }, + ], + }, + ], + }, + ], + }, + ], + }, + }; + const [span, event] = transformOpenCode(capture); + expect(span).toMatchObject({ + eventType: "ai.streamText.doStream", + timestamp: "2023-11-14T22:13:20.000Z", + startedAt: "2023-11-14T22:13:20.000Z", + endedAt: "2023-11-14T22:13:21.000Z", + traceId: "trace", + spanId: "span", + parentSpanId: "parent", + sessionId: "session", + response: "done", + usageScope: "span-snapshot", + tokenInputIncludesCache: true, + }); + expect(event).toMatchObject({ + eventType: "tool.output", + timestamp: "2023-11-14T22:13:20.500Z", + traceId: "trace", + spanId: "span", + toolOutput: "result", + }); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts b/cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts new file mode 100644 index 000000000..bae9a4456 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts @@ -0,0 +1,87 @@ +import { describe, expect, it } from "vitest"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; +import { codexTurnTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/turn-traces.js"; + +const session = "00000000-0000-4000-8000-000000000001"; +const turn = "00000000-0000-4000-8000-000000000002"; +const attr = (key: string, value: Record) => ({ key, value }); +function span() { + return { + name: "session_task.turn", + traceId: "1234567890abcdef1234567890abcdef", + spanId: "1234567890abcdef", + startTimeUnixNano: "1794000000000000000", + endTimeUnixNano: "1794000001000000000", + events: [{ name: "private prompt" }], + attributes: [ + attr("thread.id", { intValue: "7" }), + attr("thread.id", { stringValue: session }), + attr("turn.id", { stringValue: turn }), + attr("model", { stringValue: "gpt-6-luna" }), + attr("codex.turn.token_usage.input_tokens", { intValue: "42" }), + attr("code.file.path", { stringValue: "/private/path" }), + ], + }; +} +function payload(spans: unknown[]) { + return { + resourceSpans: [ + { + resource: { attributes: [attr("secret", { stringValue: "private" })] }, + scopeSpans: [{ spans }], + }, + ], + }; +} + +describe("Codex turn trace filtering", () => { + it("keeps a session's typed turn data and omits other trace content", () => { + const out = codexTurnTransfers(payload([span()]), session); + expect(out).toHaveLength(1); + expect(JSON.stringify(out)).not.toMatch(/private|code.file.path|events/); + const selected = out[0].payload.resourceSpans as Array<{ + scopeSpans: Array<{ spans: Array<{ attributes: Array<{ key: string; value: unknown }> }> }>; + }>; + const attrs = selected[0].scopeSpans[0].spans[0].attributes; + expect(attrs.filter((item) => item.key === "thread.id")).toEqual([ + attr("thread.id", { stringValue: session }), + ]); + expect(attrs).toContainEqual(attr("codex.turn.token_usage.input_tokens", { intValue: "42" })); + expect(codexTurnTransfers(payload([span()]), turn)).toEqual([]); + }); + it("rejects a missing or invalid session ID", () => { + const missing = span(); + missing.attributes = missing.attributes.filter((item) => item.key !== "thread.id"); + expect(() => codexTurnTransfers(payload([missing]), session)).toThrow("missing"); + const invalid = span(); + invalid.attributes = invalid.attributes.filter((item) => item.key !== "thread.id"); + invalid.attributes.push(attr("thread.id", { stringValue: "invalid" })); + expect(() => codexTurnTransfers(payload([invalid]), session)).toThrow("Invalid"); + }); + it("rejects conflicting session IDs", () => { + const item = span(); + item.attributes.push(attr("thread.id", { stringValue: turn })); + expect(() => codexTurnTransfers(payload([item]), session)).toThrow("Conflicting"); + }); + it("only selects traces in the second isolated-run pass", () => { + const line = JSON.stringify({ signal: "traces", payload: payload([span()]) }); + const options = { + config: "codex" as const, + session, + tenant: "test", + endpoint: "http://127.0.0.1:1", + token: "token", + home: "/tmp", + follow: false, + }; + expect(selectTransfers(line, options)).toEqual([]); + expect(selectTransfers(line, { ...options, traceOnly: true })).toHaveLength(1); + expect( + selectTransfers(line, { + ...options, + traceOnly: true, + installationId: "00000000-0000-4000-8000-000000000003", + })[0].payload.resourceSpans + ).toBeDefined(); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/collector.integration.test.ts b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts new file mode 100644 index 000000000..253d4a70d --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts @@ -0,0 +1,151 @@ +import { mkdir, mkdtemp, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; +import { Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { gzipSync } from "node:zlib"; +import { expect, it as test, vi } from "vitest"; +import { startOtelCollector } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js"; + +const it = test.skipIf(process.platform === "win32"); +const TOKEN = "example-private-collector-token"; + +async function seedToken(home: string): Promise { + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + await writeFile(join(directory, "collector-token"), `${TOKEN}\n`, { mode: 0o600 }); +} + +it("captures authenticated OTLP locally, rejects unauthenticated requests, and keeps files private", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-")); + await seedToken(home); + const listen = vi.spyOn(Server.prototype, "listen"); + const collector = await startOtelCollector({ home, port: 0 }); + try { + expect(listen).toHaveBeenCalledOnce(); + expect(listen).toHaveBeenCalledWith(0, "127.0.0.1", expect.any(Function)); + for (const tool of ["claude", "codex", "opencode"]) { + const file = join(home, `.local/share/aidd-otel-observation/${tool}.jsonl`); + expect(await readFile(file, "utf8")).toBe(""); + expect((await stat(file)).mode & 0o777).toBe(0o600); + } + expect( + ( + await readFile(join(home, ".local/share/aidd-otel-observation/collector-token"), "utf8") + ).trim() + ).toBe(TOKEN); + const url = `http://127.0.0.1:${collector.port}/claude/v1/logs`; + const body = JSON.stringify({ resourceLogs: [{ scopeLogs: [] }] }); + expect((await fetch(url, { method: "POST", body })).status).toBe(403); + expect( + (await fetch(url, { method: "POST", headers: { authorization: "Bearer wrong" }, body })) + .status + ).toBe(403); + expect( + ( + await fetch(url, { + method: "POST", + headers: { authorization: `Bearer ${TOKEN}`, "content-encoding": "gzip" }, + body: gzipSync(body), + }) + ).status + ).toBe(200); + expect( + ( + await fetch(url, { + method: "POST", + headers: { authorization: `Bearer ${TOKEN}` }, + body: "not json", + }) + ).status + ).toBe(400); + expect( + ( + await fetch(`http://127.0.0.1:${collector.port}/not-a-tool/v1/logs`, { + method: "POST", + headers: { authorization: `Bearer ${TOKEN}` }, + body, + }) + ).status + ).toBe(404); + const file = join(home, ".local/share/aidd-otel-observation/claude.jsonl"); + const captured = JSON.parse((await readFile(file, "utf8")).trim()); + expect(captured).toMatchObject({ + signal: "logs", + payload: { resourceLogs: [{ scopeLogs: [] }] }, + }); + expect((await stat(file)).mode & 0o777).toBe(0o600); + expect( + (await stat(join(home, ".local/share/aidd-otel-observation/collector-token"))).mode & 0o777 + ).toBe(0o600); + } finally { + listen.mockRestore(); + await collector.close(); + await rm(home, { recursive: true, force: true }); + } +}); + +it("creates a private random token when none exists", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-token-")); + const collector = await startOtelCollector({ home, port: 0 }); + try { + const path = join(home, ".local/share/aidd-otel-observation/collector-token"); + expect((await readFile(path, "utf8")).trim()).toMatch(/^[A-Za-z0-9_-]{43}$/); + expect((await stat(path)).mode & 0o777).toBe(0o600); + } finally { + await collector.close(); + await rm(home, { recursive: true, force: true }); + } +}); + +it("does not follow an existing token symlink or replace its target", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-link-")); + const directory = join(home, ".local/share/aidd-otel-observation"); + const target = join(home, "other-file"); + try { + await mkdir(directory, { recursive: true, mode: 0o700 }); + await writeFile(target, "do not touch", { mode: 0o600 }); + await symlink(target, join(directory, "collector-token")); + await expect(startOtelCollector({ home, port: 0 })).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("do not touch"); + } finally { + await rm(home, { recursive: true, force: true }); + } +}); + +it("serializes concurrent appends into complete records", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-concurrent-")); + await seedToken(home); + const collector = await startOtelCollector({ home, port: 0 }); + try { + const directory = join(home, ".local/share/aidd-otel-observation"); + const url = `http://127.0.0.1:${collector.port}/codex/v1/logs`; + const statuses = await Promise.all( + Array.from({ length: 8 }, (_, number) => + fetch(url, { + method: "POST", + headers: { authorization: `Bearer ${TOKEN}` }, + body: JSON.stringify({ number }), + }).then((response) => response.status) + ) + ); + expect(statuses).toEqual(Array(8).fill(200)); + const lines = (await readFile(join(directory, "codex.jsonl"), "utf8")).trim().split("\n"); + expect(lines.map((line) => JSON.parse(line).payload.number).sort()).toEqual([ + 0, 1, 2, 3, 4, 5, 6, 7, + ]); + } finally { + await collector.close(); + await rm(home, { recursive: true, force: true }); + } +}); + +test.runIf(process.platform === "win32")("clearly refuses collection on Windows", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-windows-")); + try { + await expect(startOtelCollector({ home, port: 0 })).rejects.toThrow( + "OTLP collection requires private POSIX files" + ); + } finally { + await rm(home, { recursive: true, force: true }); + } +}); diff --git a/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts b/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts new file mode 100644 index 000000000..da86a1536 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts @@ -0,0 +1,116 @@ +import { parse } from "smol-toml"; +import { describe, expect, it } from "vitest"; +import { + mergeCodexExporters, + mergeJsonEnvironment, + rejectConflict, +} from "../../../../src/contexts/telemetry/infrastructure/otel/config-edits.js"; + +const endpoint = "http://127.0.0.1:14318/codex/v1/logs"; +const token = "test-token"; +const exporters = { exporter: endpoint }; +const defaults = { environment: "local", log_user_prompt: true }; +const toolResult = { max_bytes: 262144 }; +const mergeCodex = (content: string) => + mergeCodexExporters(content, exporters, token, defaults, toolResult); + +describe("merging Claude and OpenCode settings", () => { + it("adds the environment without losing unrelated settings or overriding the user's defaults", () => { + const original = '{"model":"m1","env":{"USER":"keep","DETAILS":"0"}}'; + const merged = mergeJsonEnvironment(original, { OTEL_ENDPOINT: "local" }, () => {}, { + DETAILS: "1", + }); + expect(JSON.parse(merged)).toEqual({ + model: "m1", + env: { USER: "keep", DETAILS: "0", OTEL_ENDPOINT: "local" }, + }); + expect( + mergeJsonEnvironment(merged, { OTEL_ENDPOINT: "local" }, () => {}, { DETAILS: "1" }) + ).toBe(merged); + }); + + it("creates the missing section and rejects conflicting or duplicate environment keys", () => { + expect( + JSON.parse(mergeJsonEnvironment('{"model":"m1"}', { OTEL_ENDPOINT: "local" }, () => {})) + ).toEqual({ model: "m1", env: { OTEL_ENDPOINT: "local" } }); + for (const invalid of [ + '{"env":{"OTEL_ENDPOINT":"remote"}}', + '{"env":{},"env":{}}', + '{"env":{"USER":"one","USER":"two"}}', + '{"env":[]}', + '{"env":', + ]) { + expect(() => mergeJsonEnvironment(invalid, { OTEL_ENDPOINT: "local" }, () => {})).toThrow(); + } + expect(() => rejectConflict("endpoint", "remote", "local")).toThrow(/endpoint/); + }); +}); + +describe("merging Codex TOML exporters", () => { + it("adds missing exporters and defaults without changing existing options or comments", () => { + const original = + '# Keep this comment\nmodel = "m1"\n[otel]\nlog_user_prompt = false\n[otel.tool_result]\nmax_bytes = 1234\n'; + const merged = mergeCodex(original); + const parsed = parse(merged); + expect(parsed).toMatchObject({ + model: "m1", + otel: { + environment: "local", + log_user_prompt: false, + tool_result: { max_bytes: 1234 }, + exporter: { + "otlp-http": { + endpoint, + protocol: "json", + headers: { authorization: "Bearer test-token" }, + }, + }, + }, + }); + expect(merged).toContain("# Keep this comment"); + expect(mergeCodex(merged)).toBe(merged); + }); + + it("preserves already configured exporter headers and adds only missing values", () => { + const original = `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nheaders = { Authorization = "Bearer test-token", other = "keep" }\n`; + const merged = mergeCodex(original); + expect(parse(merged)).toMatchObject({ + otel: { + exporter: { + "otlp-http": { + endpoint, + protocol: "json", + headers: { Authorization: "Bearer test-token", other: "keep" }, + }, + }, + }, + }); + expect(mergeCodex(merged)).toBe(merged); + }); + + it.each([ + '[ "otel" ] # keep\nvalue = true\n', + "['otel']\nvalue = true\n", + "[otel] # keep", + "[otel]\r\nvalue = true\r\n", + "[otel]\nvalue = true", + ])("preserves valid existing table syntax %j", (original) => { + const merged = mergeCodex(original); + const parsed = parse(merged); + expect(parsed.otel).toMatchObject({ environment: "local", log_user_prompt: true }); + if (original.includes("value")) expect(parsed.otel).toMatchObject({ value: true }); + if (original.includes("# keep")) expect(merged).toContain("# keep"); + expect(mergeCodex(merged)).toBe(merged); + }); + + it.each([ + '[otel.exporter.otlp-http]\nendpoint = "https://other.example"\n', + `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nprotocol = "grpc"\n`, + '[otel.exporter.otlp-grpc]\nendpoint = "https://other.example"\n', + `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nheaders = { authorization = "Bearer other" }\n`, + `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nheaders = { authorization = "Bearer test-token", Authorization = "Bearer test-token" }\n`, + "[otel\n", + ])("refuses an incompatible or invalid existing exporter", (original) => { + expect(() => mergeCodex(original)).toThrow(); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts new file mode 100644 index 000000000..0409125d6 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -0,0 +1,239 @@ +import { execFileSync } from "node:child_process"; +import { constants } from "node:fs"; +import { mkdir, mkdtemp, open, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { hasOpenCodePlugin } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.js"; +import { verifyOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js"; + +const TOKEN = "example-private-token"; +let home: string; + +async function seed(path: string, content: string): Promise { + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, content, { mode: 0o600 }); +} + +beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-")); +}); +afterEach(async () => { + await rm(home, { recursive: true, force: true }); +}); + +function seedToken() { + return seed(join(home, ".local/share/aidd-otel-observation/collector-token"), TOKEN); +} + +describe.skipIf(process.platform === "win32")("OTLP initialization", () => { + it.each([ + "package.json", + "server", + "server.ts", + "server.tsx", + "server.js", + "server.mjs", + "index", + "index.ts", + "index.tsx", + "index.mts", + ])("refuses competing %s without changing it", async (name) => { + await seedToken(); + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + const entry = join(home, ".config/opencode/plugins/aidd-otel", name); + await seed(entry, "user content"); + await expect(initializeOtel("opencode", "dev", host)).rejects.toThrow(/competing entry point/); + expect(await readFile(entry, "utf8")).toBe("user content"); + await expect(stat(join(dirname(entry), "index.js"))).rejects.toMatchObject({ + code: "ENOENT", + }); + }); + it("rejects a competing symlink and FIFO without reading either", async () => { + const directory = join(home, ".config/opencode/plugins/aidd-otel"); + await mkdir(directory, { recursive: true }); + await symlink(join(home, "missing"), join(directory, "package.json")); + await expect(hasOpenCodePlugin({ home, env: {} })).rejects.toThrow(/competing entry point/); + await rm(join(directory, "package.json")); + execFileSync("/usr/bin/mkfifo", [join(directory, "server.js")]); + await expect(hasOpenCodePlugin({ home, env: {} })).rejects.toThrow(/competing entry point/); + }); + it("refuses a competing entry added after installation", async () => { + await seedToken(); + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + await initializeOtel("opencode", "dev", host); + const notes = join(home, ".config/opencode/plugins/aidd-otel/notes.txt"); + await writeFile(notes, "unrelated\n"); + await expect(hasOpenCodePlugin(host)).resolves.toBe(true); + const entry = join(home, ".config/opencode/plugins/aidd-otel/server.js"); + await writeFile(entry, "user plugin\n"); + await expect(hasOpenCodePlugin(host)).rejects.toThrow(/competing entry point/); + await expect(verifyOtel("opencode", "dev", host)).rejects.toThrow(/competing entry point/); + expect(await readFile(entry, "utf8")).toBe("user plugin\n"); + }); + it("rejects a FIFO plugin without blocking", async () => { + const path = join(home, ".config/opencode/plugins/aidd-otel/index.js"); + await mkdir(dirname(path), { recursive: true }); + execFileSync("/usr/bin/mkfifo", [path]); + const check = hasOpenCodePlugin({ home, env: {} }).then( + () => "accepted", + (error: Error) => error.message + ); + let timer: ReturnType | undefined; + const result = await Promise.race([ + check, + new Promise((resolve) => { + timer = setTimeout(() => resolve("blocked"), 1000); + }), + ]); + if (timer) clearTimeout(timer); + if (result === "blocked") { + // Release a regressed blocking read so this behavioral check can fail cleanly. + const writer = await open(path, constants.O_WRONLY | constants.O_NONBLOCK); + await check; + await writer.close(); + } + expect(result).toBe("OpenCode telemetry plugin is unsafe or has different content."); + }); + it("refuses production before even needing the local token or executable", async () => { + await expect(initializeOtel("opencode", "prod", { home, env: {} })).rejects.toThrow( + /production.*Gouvernail/ + ); + await expect(stat(join(home, ".config"))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("requires the collector token and leaves the target untouched", async () => { + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow( + /collector-token/ + ); + await expect(stat(join(home, ".claude"))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("merges Claude settings, preserves content options and writes a private file once", async () => { + await seedToken(); + const path = join(home, ".claude/settings.json"); + await seed( + path, + '{"env":{"OTEL_LOG_USER_PROMPTS":"0","CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH":"16000"},"permissions":{"allow":[]}}' + ); + const result = await initializeOtel("claude", "dev", { home, env: {} }); + expect(result.changed).toBe(true); + const content = await readFile(path, "utf8"); + expect(JSON.parse(content)).toMatchObject({ + permissions: { allow: [] }, + env: { + OTEL_LOG_USER_PROMPTS: "0", + OTEL_LOG_RAW_API_BODIES: "1", + CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH: "16000", + OTEL_METRICS_INCLUDE_REPOSITORY: "true", + OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: "http://127.0.0.1:14318/claude/v1/logs", + OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${TOKEN}`, + }, + }); + expect(await initializeOtel("claude", "dev", { home, env: {} })).toMatchObject({ + changed: false, + }); + expect(await readFile(path, "utf8")).toBe(content); + expect((await stat(path)).mode & 0o777).toBe(0o600); + }); + + it("writes Claude settings privately on the first initialization", async () => { + await seedToken(); + await initializeOtel("claude", "dev", { home, env: {} }); + const handle = await open(join(home, ".claude/settings.json"), "r"); + try { + expect((await handle.stat()).mode & 0o777).toBe(0o600); + } finally { + await handle.close(); + } + }); + + it("refuses a symlink without reading or replacing its target", async () => { + await seedToken(); + const target = join(home, "original.json"); + await seed(target, "{}"); + await mkdir(join(home, ".claude")); + await symlink(target, join(home, ".claude/settings.json")); + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("{}"); + }); + + it("refuses invalid config or a conflicting exporter without altering the file", async () => { + await seedToken(); + const path = join(home, ".claude/settings.json"); + for (const content of [ + '{"env":', + '{"env":{"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT":"https://collector.example/logs"}}', + ]) { + await seed(path, content); + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow(); + expect(await readFile(path, "utf8")).toBe(content); + } + }); + + it("adds Codex exporters without changing TOML comments or content flags", async () => { + await seedToken(); + const dir = join(home, "codex-custom"); + const path = join(dir, "config.toml"); + const original = + "# User config\n[otel]\nlog_user_prompt = false # Keep this\n\n[otel.tool_result]\nmax_bytes = 1234\n"; + await seed(path, original); + const options = { home, env: { CODEX_HOME: dir } }; + expect(await initializeOtel("codex", "dev", options)).toMatchObject({ changed: true }); + const updated = await readFile(path, "utf8"); + expect(updated).toContain("# User config\n[otel]\n"); + expect(updated).toContain( + "log_user_prompt = false # Keep this\n\n[otel.tool_result]\nmax_bytes = 1234\n" + ); + expect(updated).toContain("log_agent_responses = true"); + expect(updated).toContain('endpoint = "http://127.0.0.1:14318/codex/v1/traces"'); + expect(await initializeOtel("codex", "dev", options)).toMatchObject({ changed: false }); + expect(await readFile(path, "utf8")).toBe(updated); + }); + + it("merges v2 daemon environment and refuses an unknown major", async () => { + await seedToken(); + const dir = join(home, "opencode-custom"); + const path = join(dir, "service.json"); + await seed(path, '{"port":1234,"env":{"EXISTING":"keep"}}'); + const options = { home, env: { OPENCODE_CONFIG_DIR: dir } }; + await expect( + initializeOtel("opencode", "dev", { ...options, opencodeVersion: async () => "3.0.0" }) + ).rejects.toThrow(/OpenCode version/); + expect(await readFile(path, "utf8")).toBe('{"port":1234,"env":{"EXISTING":"keep"}}'); + await expect( + initializeOtel("opencode", "dev", { ...options, opencodeVersion: async () => "2.0.0" }) + ).rejects.toThrow(/OpenCode version/); + expect(await readFile(path, "utf8")).toBe('{"port":1234,"env":{"EXISTING":"keep"}}'); + const v2 = { ...options, opencodeVersion: async () => "opencode v2.2.5" }; + expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: true }); + expect(JSON.parse(await readFile(path, "utf8"))).toMatchObject({ + port: 1234, + env: { + EXISTING: "keep", + OTEL_EXPORTER_OTLP_ENDPOINT: "http://127.0.0.1:14318/opencode", + AIDD_OPENCODE_OTEL_LOGS_ENDPOINT: "http://127.0.0.1:14318/opencode/v1/logs", + }, + }); + const plugin = join(dir, "plugins/aidd-otel/index.js"); + expect(await readFile(plugin, "utf8")).toContain('id: "aidd.otel"'); + expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: false }); + await writeFile(plugin, "user plugin\n"); + await expect(initializeOtel("opencode", "dev", v2)).rejects.toThrow(/plugin.*different/); + expect(await readFile(plugin, "utf8")).toBe("user plugin\n"); + }); + + it("rejects OpenCode 1 and versions before 2.2.5", async () => { + await seedToken(); + for (const version of ["1.18.35", "1.19.5", "2.2.4", "not-a-version"]) { + await expect( + initializeOtel("opencode", "dev", { + home, + env: {}, + opencodeVersion: async () => version, + }) + ).rejects.toThrow(/Unsupported OpenCode version/); + } + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts b/cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts new file mode 100644 index 000000000..6f5e525c7 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts @@ -0,0 +1,104 @@ +import { describe, expect, it } from "vitest"; +import { + attachInstallationId, + checkInstallationClaim, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.js"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const id = "6ba7b810-9dad-41d1-80b4-00c04fd430c8"; +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "codex" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:3000", + token: "test", + home: "/tmp", + follow: false, +}; +const codex = (extra: unknown[] = []) => + JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: [attr("conversation.id", "session"), ...extra] }, + scopeLogs: [ + { logRecords: [{ attributes: [attr("event.name", "codex.conversation_starts")] }] }, + ], + }, + ], + }, + }); + +describe("installation ID opt-in", () => { + it("adds the ID to selected Codex resources only when requested", () => { + expect(JSON.stringify(selectTransfers(codex(), options))).not.toContain("aidd.installation.id"); + const output = selectTransfers(codex(), { ...options, installationId: id }); + expect(output).toHaveLength(1); + expect(JSON.stringify(output)).toContain(JSON.stringify(attr("aidd.installation.id", id))); + }); + it("refuses a conflicting provider claim", () => { + expect(() => + selectTransfers(codex([attr("aidd.installation.id", "wrong")]), { + ...options, + installationId: id, + }) + ).toThrow("Conflicting OTLP installation ID"); + expect(() => + checkInstallationClaim( + { resourceLogs: [{ attributes: [attr("aidd.installation.id", "wrong")] }] }, + id + ) + ).toThrow(); + }); + it.each(["logs", "metrics"] as const)("adds the ID to Claude %s resources", (signal) => { + const resource = { attributes: [attr("session.id", "session")] }; + const payload = + signal === "logs" + ? { + resourceLogs: [ + { + resource, + scopeLogs: [{ logRecords: [{ attributes: [attr("event.name", "tool_result")] }] }], + }, + ], + } + : { + resourceMetrics: [ + { + resource, + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { dataPoints: [{ asInt: "7", attributes: [attr("type", "input")] }] }, + }, + ], + }, + ], + }, + ], + }; + const output = selectTransfers(JSON.stringify({ signal, payload }), { + ...options, + config: "claude", + installationId: id, + }); + expect(output).toHaveLength(1); + expect(JSON.stringify(output)).toContain(JSON.stringify(attr("aidd.installation.id", id))); + }); + it.each(["resourceLogs", "resourceMetrics", "resourceSpans"] as const)( + "labels %s without overwriting other attributes", + (field) => { + const result = attachInstallationId( + { [field]: [{ resource: { attributes: [attr("service.name", "opencode")] } }] }, + field, + id + ); + expect(JSON.stringify(result)).toContain(JSON.stringify(attr("service.name", "opencode"))); + expect(JSON.stringify(result)).toContain(JSON.stringify(attr("aidd.installation.id", id))); + } + ); +}); diff --git a/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts new file mode 100644 index 000000000..9bf2ea8fd --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts @@ -0,0 +1,161 @@ +import { appendFile, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it } from "vitest"; +import { + openCodePluginLogs, + relatedOpenCodeLogs, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.js"; +import { + completeCaptureEnd, + openCodePromptSpans, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; + +const text = (key: string, stringValue: string) => ({ key, value: { stringValue } }); + +function capture(overrides: Record = {}) { + return JSON.stringify({ + signal: "traces", + payload: { + resourceSpans: [ + { + resource: { + attributes: [ + text("service.name", "opencode"), + text("service.version", "2.2.5"), + text("host.name", "private-host"), + ], + }, + scopeSpans: [ + { + spans: [ + { + traceId: "00000000000000000000000000000001", + spanId: "0000000000000002", + name: "POST", + kind: 2, + startTimeUnixNano: "1700000000000000000", + endTimeUnixNano: "1700000000000000001", + attributes: [ + text("http.route", "/api/session/:sessionID/prompt"), + text("url.path", "/api/session/ses_probe/prompt"), + text("http.request.method", "POST"), + { key: "http.response.status_code", value: { intValue: "200" } }, + text("url.full", "http://localhost/private?secret=value"), + ], + ...overrides, + }, + ], + }, + ], + }, + ], + }, + }); +} + +it("forwards only allowlisted OpenCode prompt span data", () => { + const output = openCodePromptSpans(capture()); + expect(output).toHaveLength(1); + expect(JSON.stringify(output)).toContain('"url.path"'); + expect(JSON.stringify(output)).toContain('"spanId":"0000000000000002"'); + expect(JSON.stringify(output)).not.toMatch(/url.full|host.name|secret/); +}); + +it("only keeps child sessions linked by an unambiguous explicit parent", () => { + const logs = [ + { session: "ses_root", payload: "root" }, + { session: "ses_child", parent: "ses_root", payload: "child" }, + { session: "ses_grandchild", parent: "ses_child", payload: "grandchild" }, + { session: "ses_unknown", payload: "unrelated" }, + { session: "ses_conflict", parent: "ses_root", payload: "conflict one" }, + { session: "ses_conflict", parent: "ses_unknown", payload: "conflict two" }, + ]; + expect(relatedOpenCodeLogs("ses_root", logs).map((item) => item.payload)).toEqual([ + "root", + "child", + "grandchild", + ]); +}); + +it("rejects client spans and invalid timestamps", () => { + expect(openCodePromptSpans(capture({ kind: 3 }))).toEqual([]); + expect(openCodePromptSpans(capture({ startTimeUnixNano: "1700000000000000002" }))).toEqual([]); +}); + +it("forwards only allowlisted plugin observations and never turns an ambiguous zero into usage", () => { + const value = (key: string, intValue: string) => ({ key, value: { intValue } }); + const attributes = [ + text("event.name", "session.step.ended"), + text("event.id", "evt_probe"), + text("session.id", "ses_probe"), + text("assistant.message.id", "msg_probe"), + value("opencode.event.sequence", "12"), + value("opencode.event.version", "1"), + value("opencode.usage.input", "15"), + value("opencode.usage.cache_read", "0"), + text("prompt", "private-prompt"), + ]; + const scope = { + scope: { name: "aidd.opencode.otel", version: "1" }, + logRecords: [ + { + timeUnixNano: "1700000000000000000", + body: { stringValue: "private-response" }, + attributes, + }, + ], + }; + const capture = JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { + attributes: [text("service.name", "opencode"), text("host.name", "private-host")], + }, + scopeLogs: [scope], + }, + ], + }, + }); + const output = openCodePluginLogs(capture); + expect(output).toHaveLength(1); + expect(output[0].session).toBe("ses_probe"); + expect(JSON.stringify(output)).toContain('"intValue":"15"'); + expect(JSON.stringify(output)).not.toMatch(/private|prompt|cache_read/); + scope.scope.version = "2"; + expect( + openCodePluginLogs( + JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: [text("service.name", "opencode")] }, + scopeLogs: [scope], + }, + ], + }, + }) + ) + ).toEqual([]); +}); + +it("only advances the capture boundary after a complete JSONL record", async () => { + const directory = await mkdtemp(join(tmpdir(), "aidd-otel-boundary-")); + const file = join(directory, "capture.jsonl"); + try { + const first = '{"value":"รฉ"}\n'; + await writeFile(file, `${first}{"value":`); + const firstEnd = Buffer.byteLength(first); + expect(await completeCaptureEnd(file, 0)).toBe(firstEnd); + expect(await completeCaptureEnd(file, firstEnd)).toBe(firstEnd); + await appendFile(file, '"complete"}\n'); + expect(await completeCaptureEnd(file, firstEnd)).toBe( + firstEnd + Buffer.byteLength('{"value":"complete"}\n') + ); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); diff --git a/cli/tests/contexts/telemetry/otel/settings.integration.test.ts b/cli/tests/contexts/telemetry/otel/settings.integration.test.ts new file mode 100644 index 000000000..23339b119 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/settings.integration.test.ts @@ -0,0 +1,59 @@ +import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { readToken } from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; + +let home: string; +const tokenPath = () => join(home, ".local/share/aidd-otel-observation/collector-token"); + +beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-settings-")); +}); +afterEach(async () => { + await rm(home, { recursive: true, force: true }); +}); + +describe.skipIf(process.platform === "win32")("collector credential and exporter conflicts", () => { + it("reads a valid private token and rejects missing, malformed or public tokens", async () => { + await expect(readToken(home)).rejects.toThrow(/private collector token/); + await mkdir(dirname(tokenPath()), { recursive: true }); + await writeFile(tokenPath(), "safe-token_123\n", { mode: 0o600 }); + expect(await readToken(home)).toBe("safe-token_123"); + await writeFile(tokenPath(), "invalid header\n"); + await expect(readToken(home)).rejects.toThrow(/private collector token/); + await writeFile(tokenPath(), "safe-token_123\n"); + await chmod(tokenPath(), 0o644); + await expect(readToken(home)).rejects.toThrow(/private collector token/); + }); + + it("never follows a symlink to a collector token", async () => { + const target = join(home, "target"); + await writeFile(target, "safe-token", { mode: 0o600 }); + await mkdir(dirname(tokenPath()), { recursive: true }); + await symlink(target, tokenPath()); + await expect(readToken(home)).rejects.toThrow(/private collector token/); + }); + + it.each([ + ["OTEL_EXPORTER_OTLP_ENDPOINT", "https://different.example"], + ["OTEL_EXPORTER_OTLP_LOGS_HEADERS", "Authorization=Bearer other"], + ["OTEL_EXPORTER_OTLP_METRICS_HEADERS", "Authorization=Bearer other"], + ["OTEL_EXPORTER_OTLP_TRACES_HEADERS", "Authorization=Bearer other"], + ["OTEL_EXPORTER_OTLP_LOGS_PROTOCOL", "grpc"], + ["OTEL_EXPORTER_OTLP_METRICS_PROTOCOL", "grpc"], + ["OTEL_EXPORTER_OTLP_TRACES_PROTOCOL", "grpc"], + ])("refuses conflicting Claude setting %s without replacing the file", async (key, value) => { + await mkdir(dirname(tokenPath()), { recursive: true }); + await writeFile(tokenPath(), "safe-token", { mode: 0o600 }); + const settings = join(home, ".claude/settings.json"); + await mkdir(dirname(settings)); + const original = JSON.stringify({ env: { [key]: value }, untouched: true }); + await writeFile(settings, original, { mode: 0o600 }); + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow( + /Conflicting OTLP/ + ); + expect(await readFile(settings, "utf8")).toBe(original); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts b/cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts new file mode 100644 index 000000000..83827717f --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts @@ -0,0 +1,7 @@ +import { expect, it } from "vitest"; +import { usageToolForOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; + +it("maps each OTLP configuration to its ledger provider", () => { + expect(usageToolForOtel("claude")).toBe("claude-code"); + expect(usageToolForOtel("codex")).toBe("codex"); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts new file mode 100644 index 000000000..0bfeffdcb --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -0,0 +1,427 @@ +import { expect, test } from "vitest"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const attribute = (key: string, value: unknown) => ({ key, value }); +const text = (key: string, value: string) => attribute(key, { stringValue: value }); +const options = { + config: "codex" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:1", + token: "test", + home: "/tmp", + follow: false, +}; +type Attributes = ReturnType[]; + +function capture( + record: Attributes, + resource: Attributes = [], + scope: Attributes = [], + body?: unknown +): string { + return JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: resource }, + scopeLogs: [ + { + scope: { attributes: scope }, + logRecords: [ + { + ...(body === undefined ? {} : { body }), + attributes: [ + text("conversation.id", "session"), + ...(body === undefined ? [text("event.name", "codex.sse_event")] : []), + text("event.kind", "response.completed"), + text("event.timestamp", "2026-11-05T00:00:00Z"), + ...record, + ], + }, + ], + }, + ], + }, + ], + }, + }); +} + +test.each([ + { stringValue: ["codex.sse_event"], intValue: "9" }, + { stringValue: ["codex.sse_event"] }, + { stringValue: "" }, + { intValue: "9" }, + {}, + null, +])("rejects invalid fallback event bodies without coercion", (body) => { + expect(() => select(capture([], [], [], body))).toThrow("Invalid OTLP attribute value."); +}); + +test("keeps a valid string log body as the event-name fallback", () => { + expect(select(capture([], [], [], { stringValue: "codex.sse_event" }))).toHaveLength(1); + expect( + selectTransfers( + capture([text("session.id", "session")], [], [], { stringValue: "tool_result" }), + { ...options, config: "claude" } + ) + ).toHaveLength(1); +}); + +function select(line: string) { + return selectTransfers(line, options); +} + +test("forwards each Codex effort only on the event that reported it", () => { + const start = capture([ + text("model", "gpt-test"), + text("reasoning_effort", "high"), + text("model_reasoning_effort", "private-wrong-event"), + ]).replace("codex.sse_event", "codex.conversation_starts"); + const response = capture([ + text("model", "gpt-test"), + text("reasoning_effort", "private-wrong-event"), + text("model_reasoning_effort", "xhigh"), + ]); + const [first] = select(start); + const [second] = select(response); + const startPayload = JSON.stringify(first.payload); + const responsePayload = JSON.stringify(second.payload); + expect(startPayload).toContain('"key":"reasoning_effort","value":{"stringValue":"high"}'); + expect(responsePayload).toContain( + '"key":"model_reasoning_effort","value":{"stringValue":"xhigh"}' + ); + for (const payload of [startPayload, responsePayload]) { + expect(payload).toContain('"key":"model","value":{"stringValue":"gpt-test"}'); + expect(payload).toContain('"key":"event.timestamp"'); + expect(payload).not.toContain("private-wrong-event"); + } +}); + +test("leaves the original usage transfer unchanged apart from the observed effort", () => { + const common = [text("model", "gpt-test"), text("input_token_count", "42")]; + const [without] = select(capture(common)); + const [withEffort] = select(capture([...common, text("model_reasoning_effort", "high")])); + const expected = JSON.stringify(without.payload); + const actual = JSON.stringify(withEffort.payload).replace( + ',{"key":"model_reasoning_effort","value":{"stringValue":"high"}}', + "" + ); + expect(actual).toBe(expected); +}); + +test("does not infer effort from a resource, another event, or a missing value", () => { + const resource = [text("reasoning_effort", "high"), text("model_reasoning_effort", "high")]; + const [transfer] = select(capture([text("model", "gpt-test")], resource)); + expect(JSON.stringify(transfer.payload)).not.toContain("reasoning_effort"); + const [tool] = select( + capture([text("model_reasoning_effort", "high")]).replace( + "codex.sse_event", + "codex.tool_result" + ) + ); + expect(JSON.stringify(tool.payload)).not.toContain("reasoning_effort"); + for (const resourceModel of [[], [text("model", "gpt-test")]]) { + const [missing] = select(capture([text("model_reasoning_effort", "high")], resourceModel)); + expect(JSON.stringify(missing.payload)).not.toContain("model_reasoning_effort"); + } +}); + +test("preserves bounded custom effort identifiers and omits free text", () => { + const [custom] = select( + capture([text("model", "gpt-test"), text("model_reasoning_effort", "model_defined-2")]) + ); + expect(JSON.stringify(custom.payload)).toContain('"stringValue":"model_defined-2"'); + for (const invalid of ["", "high with spaces", "x".repeat(33), "high\nsecret"]) { + const [transfer] = select( + capture([text("model", "gpt-test"), text("model_reasoning_effort", invalid)]) + ); + expect(JSON.stringify(transfer.payload)).not.toContain("model_reasoning_effort"); + expect(JSON.stringify(transfer.payload)).not.toContain(invalid || '"stringValue":""'); + } +}); + +test("rejects contradictory effort values and omits non-text values", () => { + expect(() => + select(capture([text("model_reasoning_effort", "high"), text("model_reasoning_effort", "low")])) + ).toThrow("Conflicting duplicate OTLP attributes."); + const [nonText] = select( + capture([text("model", "gpt-test"), attribute("model_reasoning_effort", { intValue: "1" })]) + ); + expect(JSON.stringify(nonText.payload)).not.toContain("model_reasoning_effort"); +}); + +function metricCapture( + point: Attributes, + resource: Attributes = [], + scope: Attributes = [] +): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + resource: { attributes: resource }, + scopeMetrics: [ + { + scope: { attributes: scope }, + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: "1793836800000000000", + timeUnixNano: "1793836801000000000", + asInt: "5", + attributes: [text("session.id", "session"), ...point], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +test("does not forward Codex effort fields on Claude metrics", () => { + const [transfer] = selectTransfers( + metricCapture([text("type", "input"), text("reasoning_effort", "high")]), + { ...options, config: "claude" } + ); + expect(JSON.stringify(transfer.payload)).not.toContain("reasoning_effort"); +}); + +test("forwards Claude effort only when the token point explicitly reports it", () => { + const claude = { ...options, config: "claude" as const }; + const point = [text("type", "input"), text("model", "claude-test")]; + const [reported] = selectTransfers(metricCapture([...point, text("effort", "medium")]), claude); + expect(JSON.stringify(reported.payload)).toContain( + '"key":"effort","value":{"stringValue":"medium"}' + ); + for (const invalid of [ + metricCapture(point, [text("effort", "medium")]), + metricCapture([...point, text("effort", "private text")]), + metricCapture([...point, attribute("effort", { intValue: 3 })]), + metricCapture([text("type", "input"), text("effort", "medium")]), + metricCapture([...point, text("effort", "medium")]).replace( + "claude_code.token.usage", + "claude_code.cost.usage" + ), + ]) { + const [transfer] = selectTransfers(invalid, claude); + expect(JSON.stringify(transfer.payload)).not.toContain('"key":"effort"'); + } +}); + +test.each(["point", "resource", "scope"])("rejects contradictory Claude %s attributes", (level) => { + const conflict = [text("type", "input"), text("type", "output")]; + expect(() => + selectTransfers( + metricCapture( + level === "point" ? conflict : [], + level === "resource" ? conflict : [], + level === "scope" ? conflict : [] + ), + { ...options, config: "claude" } + ) + ).toThrow("Conflicting duplicate OTLP attributes."); +}); + +test("rejects contradictory Claude resource and point identities", () => { + expect(() => + selectTransfers( + metricCapture([text("user.email", "private-b")], [text("user.email", "private-a")]), + { ...options, config: "claude" } + ) + ).toThrow("Conflicting OTLP identity attributes."); +}); + +test.each(["input_token_count", "user.email", "private.unforwarded.attribute"])( + "rejects conflicting duplicate attributes without exposing their contents: %s", + (key) => { + expect(() => select(capture([text(key, "private-a"), text(key, "private-b")]))).toThrow( + "Conflicting duplicate OTLP attributes." + ); + } +); + +test("does not equate differently typed OTLP values that stringify identically", () => { + expect(() => + select( + capture([attribute("input_token_count", { intValue: 1 }), text("input_token_count", "1")]) + ) + ).toThrow("Conflicting duplicate OTLP attributes."); +}); + +test.each([ + { stringValue: ["5"], intValue: "900" }, + { stringValue: ["5"] }, + { stringValue: 5 }, + { intValue: Number.MAX_SAFE_INTEGER + 1 }, + { intValue: 1.5 }, + { intValue: "9223372036854775808" }, + { doubleValue: "5" }, + { boolValue: "true" }, + [], + null, + {}, + { arrayValue: { values: [{ stringValue: "5" }] } }, + { kvlistValue: { values: [] } }, + { bytesValue: "NQ==" }, +])("refuses malformed or non-scalar critical AnyValue without coercion", (value) => { + expect(() => select(capture([attribute("input_token_count", value)]))).toThrow( + "Invalid OTLP attribute value." + ); +}); + +test("does not coerce an identity from a malformed stringValue", () => { + expect(() => + select(capture([attribute("user.email", { stringValue: ["person@example.test"] })])) + ).toThrow("Invalid OTLP attribute value."); +}); + +test.each([ + "user.email", + "user.account_id", + "organization.id", + "model", + "event.name", + "event.kind", + "session.id", +])("refuses non-text values for textual telemetry fields: %s", (key) => { + expect(() => select(capture([attribute(key, { intValue: "5" })]))).toThrow(); +}); + +test.each(["event.name", "event.kind", "session.id", "conversation.id"])( + "refuses an explicitly empty mandatory context value: %s", + (key) => { + const original = key === "session.id" ? metricCapture([]) : capture([]); + const initial = { + "event.name": "codex.sse_event", + "event.kind": "response.completed", + "session.id": "session", + "conversation.id": "session", + }[key]; + const body = original.replace( + JSON.stringify(text(key, initial ?? "")), + JSON.stringify(text(key, "")) + ); + expect(() => + selectTransfers(body, { ...options, config: key === "session.id" ? "claude" : "codex" }) + ).toThrow(); + } +); + +test("keeps supported scalar AnyValue types exact", () => { + const [transfer] = select( + capture([ + attribute("input_token_count", { intValue: "9007199254740993" }), + attribute("success", { boolValue: false }), + attribute("duration_ms", { doubleValue: 1.5 }), + ]) + ); + const payload = JSON.stringify(transfer.payload); + for (const value of ["9007199254740993", "false", "1.5"]) + expect(payload).toContain(`"stringValue":"${value}"`); +}); + +test("ignores valid empty and structured noncritical values", () => { + expect( + select( + capture([ + attribute("unused.empty", {}), + attribute("unused.array", { arrayValue: { values: [{ stringValue: "private" }] } }), + attribute("unused.list", { + kvlistValue: { values: [{ key: "item", value: { boolValue: true } }] }, + }), + attribute("unused.bytes", { bytesValue: "NQ==" }), + ]) + ) + ).toHaveLength(1); +}); + +test("accepts identical duplicate attributes", () => { + const [transfer] = select( + capture([text("input_token_count", "5"), text("input_token_count", "5")]) + ); + expect(JSON.stringify(transfer.payload)).toContain( + '"key":"input_token_count","value":{"stringValue":"5"}' + ); +}); + +test.each(["resource", "scope"])("rejects conflicting duplicates in %s attributes", (level) => { + const conflict = [text("user.email", "private-a"), text("user.email", "private-b")]; + expect(() => + select(capture([], level === "resource" ? conflict : [], level === "scope" ? conflict : [])) + ).toThrow("Conflicting duplicate OTLP attributes."); +}); + +test("preserves non-identity attribute overrides between resource and record", () => { + const [transfer] = select( + capture([text("model", "record-model")], [text("model", "resource-model")]) + ); + expect(JSON.stringify(transfer.payload)).toContain("record-model"); + expect(JSON.stringify(transfer.payload)).not.toContain("resource-model"); +}); + +test.each(["user.email", "user.account_id", "organization.id"])( + "does not choose between contradictory resource and record identities: %s", + (key) => { + expect(() => select(capture([text(key, "private-b")], [text(key, "private-a")]))).toThrow( + "Conflicting OTLP identity attributes." + ); + } +); + +test("keeps identical identity values across resource and record", () => { + expect( + select(capture([text("user.email", "a@example.test")], [text("user.email", "a@example.test")])) + ).toHaveLength(1); +}); + +test.each(["user.email", "user.account_id", "organization.id"])( + "does not assume case-insensitive identity semantics: %s", + (key) => { + expect(() => select(capture([text(key, "ABC")], [text(key, "abc")]))).toThrow( + "Conflicting OTLP identity attributes." + ); + } +); + +test.each([ + { + values: [text("input_token_count", "100"), text("input_token_count", "5")], + error: "Conflicting duplicate OTLP attributes.", + body: undefined, + }, + { + values: [attribute("input_token_count", { stringValue: ["5"], intValue: "900" })], + error: "Invalid OTLP attribute value.", + body: undefined, + }, + { + values: [attribute("input_token_count", { intValue: Number.MAX_SAFE_INTEGER + 1 })], + error: "Invalid OTLP attribute value.", + body: undefined, + }, + { + values: [attribute("user.email", { stringValue: ["person@example.test"] })], + error: "Invalid OTLP attribute value.", + body: undefined, + }, + { + values: [], + error: "Invalid OTLP attribute value.", + body: { stringValue: ["codex.sse_event"], intValue: "9" }, + }, +])("rejects malformed attributes before transfer selection", ({ values, error, body }) => { + expect(() => selectTransfers(capture(values, [], [], body), options)).toThrow(error); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts new file mode 100644 index 000000000..b4b778f53 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts @@ -0,0 +1,93 @@ +import { expect, test } from "vitest"; +import { seriesFingerprint } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.js"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "claude" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:3011", + token: "test", + home: "/tmp", + follow: false, +}; + +function capture(dimension: string, reverse = false, unit = "token"): string { + const dimensions = [ + attr("session.id", "session"), + attr("type", "input"), + attr("agent.name", dimension), + ]; + if (reverse) dimensions.reverse(); + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + unit, + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: "1784505600000000000", + timeUnixNano: "1784548800000000000", + asInt: "5", + attributes: dimensions, + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +function fingerprint(source: string): string { + const transfers = selectTransfers(source, options); + const body = JSON.stringify(transfers[0]?.payload); + expect(body).not.toContain("private-agent"); + return ( + body.match(/"key":"aidd.series_fingerprint","value":\{"stringValue":"([0-9a-f]{64})"\}/)?.[1] ?? + "" + ); +} + +test("keeps distinct Claude metric dimensions distinguishable without exposing their values", () => { + const first = fingerprint(capture("private-agent-one")); + const second = fingerprint(capture("private-agent-two")); + expect(first).toHaveLength(64); + expect(second).toHaveLength(64); + expect(first).not.toBe(second); + expect(fingerprint(capture("private-agent-one", true))).toBe(first); + expect(fingerprint(capture("private-agent-one", false, "byte"))).not.toBe(first); +}); + +test("normalizes scope attribute order and distinguishes schemas", () => { + const resource = { resource: { attributes: [attr("service", "app")] }, schemaUrl: "schema-a" }; + const attributes = [attr("one", "1"), attr("two", "2")]; + const scope = { scope: { name: "scope", attributes }, schemaUrl: "scope-a" }; + const reordered = { ...scope, scope: { ...scope.scope, attributes: [...attributes].reverse() } }; + expect(seriesFingerprint(resource, scope, [])).toBe(seriesFingerprint(resource, reordered, [])); + expect(seriesFingerprint(resource, scope, [])).not.toBe( + seriesFingerprint({ ...resource, schemaUrl: "schema-b" }, scope, []) + ); + expect(seriesFingerprint(resource, { ...scope, metric: { unit: "token" } }, [])).not.toBe( + seriesFingerprint(resource, { ...scope, metric: { unit: "byte" } }, []) + ); +}); + +test("normalizes equivalent integer attribute encodings", () => { + const numeric = [{ key: "attempt", value: { intValue: 5 } }]; + const decimal = [{ key: "attempt", value: { intValue: "5" } }]; + expect(seriesFingerprint({}, {}, numeric)).toBe(seriesFingerprint({}, {}, decimal)); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts new file mode 100644 index 000000000..66877a0f5 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -0,0 +1,501 @@ +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { + selectTransfers, + send, + syncOtel, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const posixTest = test.skipIf(process.platform === "win32"); +const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "codex" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:1", + token: "test", + home: "/tmp", + follow: false, +}; +let server: Server | undefined; +const dirs: string[] = []; +afterEach(async () => { + if (server) await new Promise((resolve) => server?.close(() => resolve())); + server = undefined; + await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +function codexLog(time: Record, timestamp?: string): string { + return JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + ...time, + attributes: [ + attribute("conversation.id", "session"), + attribute("event.name", "codex.sse_event"), + attribute("event.kind", "response.completed"), + ...(timestamp ? [attribute("event.timestamp", timestamp)] : []), + ], + }, + ], + }, + ], + }, + ], + }, + }); +} + +function claudeMetric(startTime: string | number, endTime: string | number): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: startTime, + timeUnixNano: endTime, + asInt: "42", + attributes: [attribute("session.id", "session")], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +function claudePoint(extra: Record): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: "1785044858922210000", + timeUnixNano: "1785044858922210000", + asInt: "42", + ...extra, + attributes: [attribute("session.id", "session")], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +function metricData(data: Record): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { scopeMetrics: [{ metrics: [{ name: "claude_code.token.usage", ...data }] }] }, + ], + }, + }); +} + +test.each([ + { asDouble: 5 }, + { asInt: undefined, asDouble: [5] }, + { asInt: undefined, asDouble: {} }, + { asInt: undefined, asDouble: true }, + { asInt: undefined, asDouble: null }, + { asInt: undefined, asDouble: "NaN" }, + { asInt: undefined }, + { flags: 1 }, + { flags: "3" }, + { sum: [] }, + { explicitBounds: ["bad"] }, +])("rejects malformed or unrecorded NumberDataPoint values", (extra) => { + expect(() => selectTransfers(claudePoint(extra), { ...options, config: "claude" })).toThrow( + /OTLP/ + ); +}); + +test.each([ + {}, + { summary: {} }, + { exponentialHistogram: {} }, + { sum: {}, gauge: {} }, + { sum: {}, histogram: {} }, + { sum: {}, summary: {} }, + { sum: {}, exponentialHistogram: {} }, + { sum: {}, exponential_histogram: {} }, + { sum: { dataPoints: {} } }, +])("rejects missing, unsupported or competing data on a supported metric", (data) => { + expect(() => selectTransfers(metricData(data), { ...options, config: "claude" })).toThrow( + /OTLP metric data/ + ); +}); + +test("keeps valid histograms separate from NumberDataPoint's scalar oneof", () => { + const point = { + startTimeUnixNano: "1785044858922210000", + timeUnixNano: "1785044858922210000", + count: "2", + sum: 5.5, + explicitBounds: [1.5], + bucketCounts: ["1", "1"], + attributes: [attribute("session.id", "session")], + }; + const [transfer] = selectTransfers(metricData({ histogram: { dataPoints: [point] } }), { + ...options, + config: "claude", + }); + expect(JSON.stringify(transfer.payload)).toContain('"count":"2"'); + expect(JSON.stringify(transfer.payload)).toContain('"sum":5.5'); +}); + +test.each([ + { asInt: 1785044858922210000 }, + { asInt: 1.5 }, + { asInt: "1.5" }, + { asInt: "private-invalid-value" }, + { asInt: null }, + { asInt: "9223372036854775808" }, + { count: 1785044858922210000 }, + { count: -1 }, + { count: "1.5" }, + { count: "18446744073709551616" }, + { bucketCounts: [1, 1785044858922210000] }, + { bucketCounts: [1, "bad"] }, + { bucketCounts: 1 }, + { flags: 1.5 }, +])("rejects invalid or inexact integer metric fields before exporting", (extra) => { + expect(() => selectTransfers(claudePoint(extra), { ...options, config: "claude" })).toThrow( + /invalid OTLP integer/i + ); +}); + +test("preserves exact integer strings for signed and unsigned metric values", () => { + const extra = { + asInt: "9223372036854775807", + count: "18446744073709551615", + bucketCounts: ["9007199254740993", "0"], + }; + const [transfer] = selectTransfers(claudePoint(extra), { ...options, config: "claude" }); + const payload = JSON.stringify(transfer.payload); + for (const value of [extra.asInt, extra.count, ...extra.bucketCounts]) + expect(payload).toContain(`"${value}"`); +}); + +test.each([ + claudePoint({ asInt: 1785044858922210000 }), + claudePoint({ asDouble: 5 }), + claudePoint({ flags: 1 }), + metricData({ sum: {}, summary: {} }), + metricData({ summary: {} }), +])("rejects malformed metrics before transfer selection", (line) => { + expect(() => selectTransfers(line, { ...options, config: "claude" })).toThrow(/OTLP/); +}); + +function metricWithMetadata(unit: string, temporality: number, monotonic: boolean): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + unit, + sum: { + aggregationTemporality: temporality, + isMonotonic: monotonic, + dataPoints: [ + { + startTimeUnixNano: "1785044858922210000", + timeUnixNano: "1785044858922210000", + asInt: "42", + attributes: [attribute("session.id", "session")], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +test("distinguishes metric unit, aggregation and monotonicity when computing series identity", () => { + const fingerprint = (unit: string, temporality: number, monotonic: boolean) => { + const [transfer] = selectTransfers(metricWithMetadata(unit, temporality, monotonic), { + ...options, + config: "claude", + }); + return JSON.stringify(transfer.payload).match( + /"aidd.series_fingerprint","value":\{"stringValue":"([0-9a-f]+)"/ + )?.[1]; + }; + const base = fingerprint("token", 1, true); + expect(base).toHaveLength(64); + expect(fingerprint("byte", 1, true)).not.toBe(base); + expect(fingerprint("token", 2, true)).not.toBe(base); + expect(fingerprint("token", 1, false)).not.toBe(base); +}); + +test("does not convert an unsafe numeric OTLP timestamp into a fabricated Codex event time", () => { + const unsafe = 1785044858922210000; + const line = codexLog({ timeUnixNano: unsafe }); + expect(() => selectTransfers(line, options)).toThrow(/invalid OTLP integer/i); +}); + +test("preserves exact decimal-string nanoseconds, including for Claude metric intervals", () => { + const time = "1785044858922210000"; + const [log] = selectTransfers(codexLog({ timeUnixNano: time }), options); + expect(log.at).toBe("2026-07-26T05:47:38.922Z"); + expect(JSON.stringify(log.payload)).toContain(time); + const [metric] = selectTransfers(claudeMetric(time, time), { ...options, config: "claude" }); + expect(metric.start).toBe("2026-07-26T05:47:38.922Z"); + expect(metric.at).toBe("2026-07-26T05:47:38.922Z"); + expect(JSON.stringify(metric.payload)).toContain(time); +}); + +test.each([ + [1785044858922210000, "1785044858922210000"], + ["1785044858922210000", 1785044858922210000], +] as const)("rejects unsafe numeric Claude metric bounds", (start, end) => { + expect(() => selectTransfers(claudeMetric(start, end), { ...options, config: "claude" })).toThrow( + /invalid OTLP integer/i + ); +}); + +test("does not forward an unsafe timestamp beside a valid Codex event timestamp", () => { + const line = codexLog({ timeUnixNano: 1785044858922210000 }, "2026-07-26T05:47:38.922Z"); + expect(() => selectTransfers(line, options)).toThrow(/invalid OTLP integer/i); +}); + +test("uses a genuine OTLP event time when a Codex calendar date is invalid", () => { + const line = codexLog({ timeUnixNano: "1785044858922210000" }, "2026-02-30T12:00:00Z"); + const transfers = selectTransfers(line, options); + expect(transfers[0].at).toBe("2026-07-26T05:47:38.922Z"); + expect(JSON.stringify(transfers[0].payload)).not.toContain("aidd.original_event_timestamp"); + expect(JSON.stringify(transfers[0].payload)).not.toContain("2026-02-30"); +}); + +test.each([ + "2026-02-30T12:00:00Z", + "2026-01-01T25:00:00Z", + "2026-01-01T12:00:00+24:00", + "2026-01-01T12:00:00", +])("does not normalize an invalid provider event time: %s", (time) => { + const transfers = selectTransfers(codexLog({}, time), options); + expect(transfers[0].at).toBeUndefined(); + expect(JSON.stringify(transfers[0].payload)).not.toContain("aidd.original_event_timestamp"); +}); + +test("keeps a valid offset and leap date as the original event time", () => { + const original = "2028-02-29T12:30:00.123456+02:30"; + const transfers = selectTransfers(codexLog({}, original), options); + expect(transfers[0].at).toBe("2028-02-29T10:00:00.123Z"); + expect(JSON.stringify(transfers[0].payload)).toContain(original); +}); + +posixTest("does not export an observation-only timestamp as event activity", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-missing-event-time-")); + dirs.push(home); + const capture = join(home, ".local/share/aidd-otel-observation"); + await mkdir(capture, { recursive: true, mode: 0o700 }); + await writeFile( + join(capture, "codex.jsonl"), + `${codexLog({ observedTimeUnixNano: "1785044858922210000" })}\n`, + { mode: 0o600 } + ); + let calls = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const result = await syncOtel( + { ...options, home, endpoint: `http://127.0.0.1:${address.port}` }, + async () => {}, + async () => true + ); + expect(result.sent).toBe(0); + expect(result.skipped).toBe(1); + expect(calls).toBe(0); +}); + +test("does not use Claude observation time as event activity", () => { + const line = JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + observedTimeUnixNano: "1785044858922210000", + attributes: [ + attribute("session.id", "session"), + attribute("event.name", "tool_result"), + ], + }, + ], + }, + ], + }, + ], + }, + }); + const transfers = selectTransfers(line, { ...options, config: "claude" }); + expect(transfers).toHaveLength(1); + expect(transfers[0].at).toBeUndefined(); +}); + +test("preserves distinct submillisecond Codex event timestamps in the transmitted attributes", () => { + const timestamps = ["2026-11-04T01:00:00.123456100Z", "2026-11-04T01:00:00.123456200Z"]; + const line = JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: timestamps.map((time) => ({ + attributes: [ + attribute("conversation.id", "session"), + attribute("event.name", "codex.sse_event"), + attribute("event.kind", "response.completed"), + attribute("event.timestamp", time), + ], + })), + }, + ], + }, + ], + }, + }); + const transfers = selectTransfers(line, options); + expect(transfers).toHaveLength(2); + for (const [index, transfer] of transfers.entries()) { + expect(JSON.stringify(transfer.payload)).toContain(timestamps[index]); + expect(JSON.stringify(transfer.payload)).toContain( + '"event.timestamp","value":{"stringValue":"2026-11-04T01:00:00.123Z"}' + ); + expect(JSON.stringify(transfer.payload)).toContain('"aidd.original_event_timestamp"'); + expect(transfer.at).toBe("2026-11-04T01:00:00.123Z"); + } +}); + +test.each([ + ["logs" as const, { partialSuccess: { rejectedLogRecords: "1" } }], + ["metrics" as const, { partial_success: { rejected_data_points: 2 } }], +])("does not retry or report success for partial %s delivery", async (signal, body) => { + let calls = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200, { "content-type": "application/json" }).end(JSON.stringify(body)); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + await expect( + send( + { ...options, endpoint: `http://127.0.0.1:${address.port}`, follow: true }, + { signal, payload: {} }, + async () => true + ) + ).rejects.toThrow(/partial/i); + expect(calls).toBe(1); +}); + +test.each([ + "not-json", + '{"partialSuccess":{"rejectedLogRecords":"-1"}}', + '{"partialSuccess":{"rejectedLogRecords":""}}', + '{"partialSuccess":{"rejectedLogRecords":"0x0"}}', + '{"partialSuccess":{"rejectedDataPoints":"1"}}', + '{"partialSuccess":{"rejectedLogRecord":"1"}}', +])("refuses an invalid successful OTLP response: %s", async (body) => { + server = createServer((request, response) => { + request.resume(); + request.on("end", () => response.writeHead(200).end(body)); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + await expect( + send( + { ...options, endpoint: `http://127.0.0.1:${address.port}` }, + { signal: "logs", payload: {} }, + async () => true + ) + ).rejects.toThrow(/response/i); +}); + +test("accepts a warning with zero rejected records", async () => { + server = createServer((request, response) => { + request.resume(); + request.on("end", () => + response.writeHead(200).end( + JSON.stringify({ + partialSuccess: { rejectedLogRecords: "0", errorMessage: "warning" }, + }) + ) + ); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + await expect( + send( + { ...options, endpoint: `http://127.0.0.1:${address.port}` }, + { signal: "logs", payload: {} }, + async () => true + ) + ).resolves.toBeUndefined(); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts new file mode 100644 index 000000000..bbc84a150 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts @@ -0,0 +1,352 @@ +import { appendFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it as test } from "vitest"; +import { + selectTransfers, + syncOtel, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const it = test.skipIf(process.platform === "win32"); +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const record = (session: string, event: string) => ({ + timeUnixNano: "1793763900000000000", + attributes: [ + attr("conversation.id", session), + attr("user.email", "developer@example.test"), + attr("event.name", event), + attr("event.kind", "response.completed"), + attr("input_token_count", "42"), + attr("tool_use_id", "toolu_123"), + attr("output", "sensitive output"), + ], +}); + +describe("OTLP capture sync", () => { + let server: Server | undefined; + const dirs: string[] = []; + afterEach(async () => { + if (server) await new Promise((resolve) => server?.close(() => resolve())); + await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); + server = undefined; + }); + + test("retains numeric metric data but excludes exemplars and unknown properties", () => { + const source = JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: "AGGREGATION_TEMPORALITY_CUMULATIVE", + unknown: "SECRET_AGGREGATE", + dataPoints: [ + { + startTimeUnixNano: "1793763000000000000", + timeUnixNano: "1793763900000000000", + asInt: "42", + attributes: [attr("session.id", "wanted")], + exemplars: [{ filteredAttributes: [attr("private", "SECRET_EXEMPLAR")] }], + unknown: "SECRET_POINT", + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); + const transfers = selectTransfers(source, { + config: "claude", + session: "wanted", + tenant: "tenant", + endpoint: "http://127.0.0.1:3011", + token: "test", + home: "/tmp", + follow: false, + }); + expect(transfers).toHaveLength(1); + const sent = JSON.stringify(transfers[0]); + expect(sent).toContain('"asInt":"42"'); + expect(sent).toContain("AGGREGATION_TEMPORALITY_CUMULATIVE"); + expect(sent).not.toContain("SECRET_"); + }); + + it("sends only the selected session's useful events to loopback, without tool output", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const wanted = record("wanted", "codex.sse_event"); + wanted.attributes.push(attr("model", "gpt-test"), attr("model_reasoning_effort", "high")); + const line = { + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + wanted, + record("other", "codex.sse_event"), + record("wanted", "codex.user_prompt"), + ], + }, + ], + }, + ], + }, + }; + const file = join(directory, "codex.jsonl"); + await writeFile(file, `${JSON.stringify(line)}\n`, { mode: 0o600 }); + const received: Array<{ url: string; auth: string; body: string }> = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push({ url: request.url ?? "", auth: request.headers.authorization ?? "", body }); + response.writeHead(200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const before = await readFile(file, "utf8"); + const syncOptions = { + config: "codex" as const, + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + }; + const result = await syncOtel( + syncOptions, + async () => {}, + async () => true + ); + expect(result.sent).toBe(1); + expect(received).toHaveLength(1); + expect(received[0].url).toBe("/api/kpi/otel/codex/tenant/v1/logs"); + expect(received[0].auth).toBe("Bearer test-token"); + expect(received[0].body).not.toContain("sensitive output"); + expect(received[0].body).toContain("toolu_123"); + expect(received[0].body).toContain('"key":"model_reasoning_effort"'); + expect(received[0].body).toContain('"stringValue":"high"'); + expect(received[0].body).not.toContain('"other"'); + expect(await readFile(file, "utf8")).toBe(before); + const denied = await syncOtel( + syncOptions, + async () => {}, + async () => false + ); + expect(denied.skipped).toBe(1); + expect(received).toHaveLength(1); + }); + + test("refuses a non-loopback destination before reading capture or sending credentials", async () => { + await expect( + syncOtel( + { + config: "claude", + session: "wanted", + tenant: "tenant", + endpoint: "http://example.com:3011", + token: "test-token", + home: "/missing", + follow: false, + }, + async () => {}, + async () => true + ) + ).rejects.toThrow(/loopback/); + }); + + it("replays provider events after a rejected delivery", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-retry-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const line = { + signal: "logs", + payload: { + resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }], + }, + }; + await writeFile(join(directory, "codex.jsonl"), `${JSON.stringify(line)}\n`, { mode: 0o600 }); + const received: string[] = []; + let reject = true; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(reject ? 503 : 200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + }; + await expect( + syncOtel( + options, + async () => {}, + async () => true + ) + ).rejects.toThrow("HTTP 503"); + reject = false; + await syncOtel( + options, + async () => {}, + async () => true + ); + expect(received.filter((body) => body.includes("codex.sse_event"))).toHaveLength(2); + expect(received).toHaveLength(2); + }); + + it("follows only newly appended complete lines for the chosen session", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-follow-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const line = `${JSON.stringify({ signal: "logs", payload: { resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }] } })}\n`; + const file = join(directory, "codex.jsonl"); + await writeFile(file, line, { mode: 0o600 }); + const abort = new AbortController(); + let received = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + received++; + response.writeHead(200).end("{}"); + if (received === 1) void appendFile(file, line); + if (received === 2) abort.abort(); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const result = await syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: true, + signal: abort.signal, + intervalMs: 10, + }, + async () => {}, + async () => true + ); + expect(received).toBe(2); + expect(result.sent).toBe(2); + }); + + it("never sends an observation outside consent", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-unconsented-identity-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const source = { + signal: "logs", + payload: { + resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }], + }, + }; + await writeFile(join(directory, "codex.jsonl"), `${JSON.stringify(source)}\n`, { mode: 0o600 }); + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + response.writeHead(200).end("{}"); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const result = await syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + }, + async () => {}, + async () => false + ); + expect(result).toMatchObject({ sent: 0, skipped: 1 }); + expect(calls).toBe(0); + }); + + it("stops before retrying a failed post after consent is withdrawn", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-withdrawn-retry-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const source = { + signal: "logs", + payload: { + resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }], + }, + }; + await writeFile(join(directory, "codex.jsonl"), `${JSON.stringify(source)}\n`, { mode: 0o600 }); + let consenting = true; + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + request.on("end", () => { + consenting = false; + response.writeHead(503).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + await expect( + syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: true, + intervalMs: 10, + }, + async () => {}, + async (transfer) => !transfer.consentAt || consenting + ) + ).rejects.toThrow(/Consent is no longer confirmed/); + expect(calls).toBe(1); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts new file mode 100644 index 000000000..1ba95a2e5 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts @@ -0,0 +1,107 @@ +import { describe, expect, it } from "vitest"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "claude" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:3000", + token: "test", + home: "/tmp", + follow: false, +}; +const payload = (signal: "logs" | "metrics", url: string, conflictingUrl?: string) => + JSON.stringify({ + signal, + payload: + signal === "logs" + ? { + resourceLogs: [ + { + resource: { + attributes: [attr("session.id", "session"), attr("vcs.repository.url.full", url)], + }, + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: "1793763900000000000", + attributes: [ + attr("event.name", "tool_result"), + ...(conflictingUrl + ? [attr("vcs.repository.url.full", conflictingUrl)] + : []), + ], + }, + ], + }, + ], + }, + ], + } + : { + resourceMetrics: [ + { + resource: { + attributes: [attr("session.id", "session"), attr("vcs.repository.url.full", url)], + }, + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + dataPoints: [ + { + timeUnixNano: "1793763900000000000", + asInt: "42", + attributes: [ + attr("type", "input"), + ...(conflictingUrl + ? [attr("vcs.repository.url.full", conflictingUrl)] + : []), + ], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); + +describe("optional OTLP repository", () => { + it.each(["logs", "metrics"] as const)("forwards a safe repository in %s", (signal) => { + const transfers = selectTransfers(payload(signal, "https://github.com/org/repo"), options); + expect(transfers).toHaveLength(1); + expect(JSON.stringify(transfers)).toContain("https://github.com/org/repo"); + }); + it.each([ + "https://user:secret@github.com/org/repo", + "https://github.com/org/repo?token=secret", + "https://github.com/org/repo#secret", + "https://evil.example/org/repo", + ])("does not forward unsafe URL %s", (url) => { + for (const signal of ["logs", "metrics"] as const) { + const transfers = selectTransfers(payload(signal, url), options); + expect(transfers).toHaveLength(1); + expect(JSON.stringify(transfers)).not.toContain("vcs.repository.url.full"); + expect(JSON.stringify(transfers)).not.toContain("secret"); + } + }); + it.each(["logs", "metrics"] as const)( + "rejects conflicting repository attributes in %s", + (signal) => { + expect(() => + selectTransfers( + payload(signal, "https://github.com/org/one", "https://github.com/org/two"), + options + ) + ).toThrow(); + } + ); +}); diff --git a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts new file mode 100644 index 000000000..3cbd5ee40 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts @@ -0,0 +1,129 @@ +import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { verifyOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js"; + +let home: string; +let capture: string; +beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-verify-")); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true }); + await writeFile(join(directory, "collector-token"), "test-token", { mode: 0o600 }); + capture = join(directory, "claude.jsonl"); +}); +afterEach(async () => { + await rm(home, { recursive: true, force: true }); +}); + +describe.skipIf(process.platform === "win32")("OTLP verification", () => { + it("requires the installed OpenCode plugin before reporting a capture", async () => { + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + await initializeOtel("opencode", "dev", host); + const plugin = join(home, ".config/opencode/plugins/aidd-otel/index.js"); + await writeFile( + join(home, ".local/share/aidd-otel-observation/opencode.jsonl"), + '{"signal":"logs"}\n' + ); + expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ lines: 1 }); + await rm(plugin); + await expect(verifyOtel("opencode", "dev", host)).rejects.toThrow(/plugin is not installed/); + }); + it("requires initialization without creating settings", async () => { + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/not initialized/); + await expect(readFile(join(home, ".claude/settings.json"))).rejects.toMatchObject({ + code: "ENOENT", + }); + }); + it("requires captured receipt and leaves configuration unchanged", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const path = join(home, ".claude/settings.json"); + const original = await readFile(path, "utf8"); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/capture/); + await writeFile(capture, "\n", { mode: 0o600 }); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/capture/); + await writeFile(capture, '{"signal":"logs"}\n{"signal":"metrics"}\n'); + expect(await verifyOtel("claude", "dev", { home, env: {} })).toMatchObject({ + capture, + lines: 2, + }); + expect(await readFile(path, "utf8")).toBe(original); + await chmod(path, 0o644); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/private/); + }); + + it("counts normalized observations without returning raw content", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const record = { + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: [] }, + scopeLogs: [ + { + logRecords: [ + { + attributes: [ + { key: "event.name", value: { stringValue: "user_prompt" } }, + { key: "prompt", value: { stringValue: "private-example" } }, + ], + }, + ], + }, + ], + }, + ], + }, + }; + await writeFile(capture, `${JSON.stringify(record)}\ninvalid\n`, { mode: 0o600 }); + const result = await verifyOtel("claude", "dev", { home, env: {} }); + expect(result).toMatchObject({ lines: 2, observations: 1, unrecognisedLines: 1 }); + expect(JSON.stringify(result)).not.toContain("private-example"); + }); + + it("refuses a capture symlink and a directory without reading either target", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const target = join(home, "outside.jsonl"); + await writeFile(target, "private\n", { mode: 0o600 }); + await symlink(target, capture); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("private\n"); + await rm(capture); + await mkdir(capture); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/regular file/); + }); + + it("reports counts for valid and malformed lines while ignoring blank lines", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const record = { + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { attributes: [{ key: "event.name", value: { stringValue: "user_prompt" } }] }, + { attributes: [{ key: "event.name", value: { stringValue: "tool_result" } }] }, + ], + }, + ], + }, + ], + }, + }; + await writeFile(capture, `\n${JSON.stringify(record)}\nnot-json\n\n`, { mode: 0o600 }); + expect(await verifyOtel("claude", "dev", { home, env: {} })).toMatchObject({ + lines: 2, + observations: 2, + unrecognisedLines: 1, + }); + }); + + it("refuses production", async () => { + await expect(verifyOtel("claude", "prod", { home, env: {} })).rejects.toThrow(/production/); + }); +}); diff --git a/cli/tests/e2e/clean-scope-user.e2e.test.ts b/cli/tests/e2e/clean-scope-user.e2e.test.ts index fd4119d91..d3f79ffda 100644 --- a/cli/tests/e2e/clean-scope-user.e2e.test.ts +++ b/cli/tests/e2e/clean-scope-user.e2e.test.ts @@ -89,6 +89,8 @@ describe("E2E: clean --scope user purges the shared source", () => { expect(setupResult.exitCode).toBe(0); const userConfigDir = join(fakeHome, ".config", "aidd"); + const installationIdPath = join(userConfigDir, "installation", "id.json"); + const installationIdBefore = await readFile(installationIdPath, "utf-8"); const marketplacesPath = join(userConfigDir, "marketplaces.json"); const beforeMarketplaces = await readJson(marketplacesPath); const beforeNames = (beforeMarketplaces.marketplaces as Array<{ name: string }>).map( @@ -150,8 +152,14 @@ describe("E2E: clean --scope user purges the shared source", () => { // whitelist has no business under fails here rather than going unnoticed. expect(before).not.toEqual(after); expect(after).toEqual( - ["auth.json", "marketplaces.json", join("telemetry", "2026-01-01.jsonl")].sort() + [ + "auth.json", + "marketplaces.json", + join("installation", "id.json"), + join("telemetry", "2026-01-01.jsonl"), + ].sort() ); + expect(await readFile(installationIdPath, "utf-8")).toBe(installationIdBefore); const afterMarketplaces = await readJson(marketplacesPath); const afterNames = (afterMarketplaces.marketplaces as Array<{ name: string }>).map( (m) => m.name diff --git a/cli/tests/e2e/installation-identity.e2e.test.ts b/cli/tests/e2e/installation-identity.e2e.test.ts new file mode 100644 index 000000000..5bd7b88e6 --- /dev/null +++ b/cli/tests/e2e/installation-identity.e2e.test.ts @@ -0,0 +1,136 @@ +import { mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createTestEnv, FRAMEWORK_PATH, gitInit, runCli } from "./helpers.js"; + +describe.skipIf(process.platform === "win32")("installation identity across commands", () => { + it("reuses the setup ID for telemetry and keeps a manually chosen person ID", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-id"); + const userConfig = join(fakeHome, ".config", "aidd"); + const path = join(userConfig, "installation", "id.json"); + const person = join(userConfig, "telemetry", "identity.json"); + const collector = join(fakeHome, ".local", "share", "aidd-otel-observation"); + try { + await mkdir(join(userConfig, "telemetry"), { recursive: true }); + await writeFile(person, '{"person_id":"existing-person"}\n', { mode: 0o600 }); + const setup = [ + "setup", + "--source", + "local", + "--path", + FRAMEWORK_PATH, + "--ai", + "claude", + "--plugins", + "none", + "--yes", + ]; + expect((await runCli(setup, projectDir, fakeHome)).exitCode).toBe(0); + const first = await readFile(path, "utf8"); + expect((await runCli(setup, projectDir, fakeHome)).exitCode).toBe(0); + expect(await readFile(path, "utf8")).toBe(first); + await gitInit(projectDir); + const turnOn = await runCli(["telemetry", "on", "--yes"], projectDir, fakeHome); + expect(turnOn.exitCode, turnOn.stderr).toBe(0); + expect(await readFile(path, "utf8")).toBe(first); + await mkdir(collector, { recursive: true }); + await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); + expect( + ( + await runCli( + ["telemetry", "otel", "init", "--config=codex", "--endpoint=dev"], + projectDir, + fakeHome + ) + ).exitCode + ).toBe(0); + expect(await readFile(path, "utf8")).toBe(first); + expect(await readFile(person, "utf8")).toBe('{"person_id":"existing-person"}\n'); + } finally { + await cleanup(); + } + }); + + it("does not create an installation ID when OTLP production initialization is refused", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-refused"); + try { + const result = await runCli( + ["telemetry", "otel", "init", "--config=codex", "--endpoint=prod"], + projectDir, + fakeHome + ); + expect(result.exitCode).not.toBe(0); + await expect( + stat(join(fakeHome, ".config", "aidd", "installation", "id.json")) + ).rejects.toThrow(); + } finally { + await cleanup(); + } + }); + + it("creates and reuses the ID through OTLP initialization for an existing installation", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-otel"); + const path = join(fakeHome, ".config", "aidd", "installation", "id.json"); + const collector = join(fakeHome, ".local", "share", "aidd-otel-observation"); + try { + await mkdir(collector, { recursive: true }); + await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); + const args = ["telemetry", "otel", "init", "--config=codex", "--endpoint=dev"]; + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + const initial = await readFile(path, "utf8"); + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + expect(await readFile(path, "utf8")).toBe(initial); + } finally { + await cleanup(); + } + }); + + it("creates the ID when measurement is enabled and not when it is refused", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-on"); + const path = join(fakeHome, ".config", "aidd", "installation", "id.json"); + const args = ["telemetry", "on", "--yes", "--anonymous"]; + try { + expect((await runCli(args, projectDir, fakeHome)).exitCode).not.toBe(0); + await expect(readFile(path, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); + await gitInit(projectDir); + const turnOn = await runCli(args, projectDir, fakeHome); + expect(turnOn.exitCode, turnOn.stderr).toBe(0); + const initial = await readFile(path, "utf8"); + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + expect(await readFile(path, "utf8")).toBe(initial); + } finally { + await cleanup(); + } + }); + + it("does not initialize a project over an invalid existing installation ID", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-invalid"); + const directory = join(fakeHome, ".config", "aidd", "installation"); + const path = join(directory, "id.json"); + try { + await mkdir(directory, { recursive: true }); + await writeFile(path, '{"installation_id":"invalid"}', { mode: 0o600 }); + const result = await runCli( + [ + "setup", + "--source", + "local", + "--path", + FRAMEWORK_PATH, + "--ai", + "claude", + "--plugins", + "none", + "--yes", + ], + projectDir, + fakeHome + ); + expect(result.exitCode).not.toBe(0); + await expect(stat(join(projectDir, ".aidd", "manifest.json"))).rejects.toThrow(); + expect(await readFile(path, "utf8")).toBe('{"installation_id":"invalid"}'); + } finally { + await cleanup(); + } + }); +}); diff --git a/cli/tests/e2e/telemetry-identity.e2e.test.ts b/cli/tests/e2e/telemetry-identity.e2e.test.ts index 5571d2d28..2293cd8f1 100644 --- a/cli/tests/e2e/telemetry-identity.e2e.test.ts +++ b/cli/tests/e2e/telemetry-identity.e2e.test.ts @@ -1,4 +1,4 @@ -import { mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { mkdir, readFile, rm, stat, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { @@ -13,6 +13,20 @@ import { useReportSandbox(); describe("aidd telemetry identity", () => { + it("shows the saved installation identifier separately from the measurement identity", async () => { + await rm(join(sandbox.userConfig, "installation", "id.json"), { force: true }); + const before = await aidd(["identity"]); + expect(before.stdout).toContain("Installation ID: unavailable"); + await aidd(["on", "--yes", "--as", "person-a"]); + const installationPath = join(sandbox.userConfig, "installation", "id.json"); + const installation = JSON.parse(await readFile(installationPath, "utf8")) as { + installation_id: string; + }; + const shown = await aidd(["identity"]); + expect(shown.exitCode).toBe(0); + expect(shown.stdout).toContain('"person-a"'); + expect(shown.stdout).toContain(`Installation ID: ${installation.installation_id}`); + }); it("shows the person axis as anonymous until an identity is chosen", async () => { const before = rowsOf(await json("person")); expect(Object.keys(before)).toEqual(["absent"]); diff --git a/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts b/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts new file mode 100644 index 000000000..9a79bbfe9 --- /dev/null +++ b/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts @@ -0,0 +1,133 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; +import { createInterface } from "node:readline"; +import { describe, expect, it } from "vitest"; +import { transformCapturedLine } from "../../src/contexts/telemetry/infrastructure/otel/observations.js"; +import { cliPath, createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); + +// These are synthetic OTLP records, not a claim that every provider exports every field. +describe.skipIf(process.platform === "win32")("OTLP collector on the built CLI", () => { + it("captures isolated authenticated provider records and verifies their normalized fields", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("otel-collector"); + const directory = join(fakeHome, ".local/share/aidd-otel-observation"); + const token = "test-only-otel-collector-token"; + await mkdir(directory, { recursive: true, mode: 0o700 }); + await writeFile(join(directory, "collector-token"), `${token}\n`, { mode: 0o600 }); + const bin = join(fakeHome, "bin"); + await mkdir(bin); + await writeFile(join(bin, "opencode"), `#!${process.execPath}\nconsole.log('2.2.5');\n`, { + mode: 0o700, + }); + const env = sandboxedEnv(fakeHome); + const path = [bin, env.PATH].join(delimiter); + const child = spawn(process.execPath, [cliPath(), "telemetry", "otel", "collect", "--port=0"], { + cwd: projectDir, + env: sandboxedEnv(fakeHome, { PATH: path }), + stdio: ["ignore", "pipe", "pipe"], + }); + const exited = once(child, "exit"); + const outputLines = createInterface({ input: child.stdout }); + try { + const output = await Promise.race([ + once(outputLines, "line", { signal: AbortSignal.timeout(5000) }).then(([line]) => + String(line) + ), + exited.then(() => { + throw new Error("Collector exited before listening."); + }), + ]); + const port = Number(/127\.0\.0\.1:(\d+)/.exec(output)?.[1]); + if (!Number.isInteger(port) || port < 1 || port > 65535) + throw new Error("Collector reported an invalid loopback port."); + const configEnv = { + PATH: path, + OPENCODE_CONFIG_DIR: "", + OPENCODE_CONFIG: "", + OPENCODE_CONFIG_CONTENT: "", + }; + const tools = [ + { tool: "claude" as const, sessionKey: "session.id", inputKey: "input_tokens" }, + { tool: "codex" as const, sessionKey: "conversation.id", inputKey: "input_token_count" }, + { + tool: "opencode" as const, + sessionKey: "session.id", + inputKey: "gen_ai.usage.input_tokens", + }, + ]; + for (const { tool, sessionKey, inputKey } of tools) { + const payload = { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: "1700000000000000000", + attributes: [ + attr("event.name", "synthetic"), + attr(sessionKey, `${tool}-session`), + attr(inputKey, "17"), + ], + }, + ], + }, + ], + }, + ], + }; + const url = `http://127.0.0.1:${port}/${tool}/v1/logs`; + const body = JSON.stringify(payload); + expect((await fetch(url, { method: "POST", body, redirect: "manual" })).status).toBe(403); + expect( + ( + await fetch(url, { + method: "POST", + body, + redirect: "manual", + headers: { authorization: `Bearer ${token}` }, + }) + ).status + ).toBe(200); + const capture = (await readFile(join(directory, `${tool}.jsonl`), "utf8")).trim(); + const observations = transformCapturedLine(tool, capture); + expect(observations).toHaveLength(1); + expect(observations[0]).toMatchObject({ + tool, + sessionId: `${tool}-session`, + tokens: { input: 17 }, + }); + expect(observations[0]?.tokens?.output).toBeUndefined(); + const init = await runCli( + ["telemetry", "otel", "init", `--config=${tool}`, "--endpoint=dev"], + projectDir, + fakeHome, + { env: configEnv } + ); + expect(init.exitCode, init.stderr + init.stdout).toBe(0); + const verified = await runCli( + ["telemetry", "otel", "verify", `--config=${tool}`, "--endpoint=dev"], + projectDir, + fakeHome, + { env: configEnv } + ); + expect(verified.exitCode, verified.stderr + verified.stdout).toBe(0); + expect(verified.stdout).toContain("1 captured lines"); + expect(verified.stdout).toContain("1 normalized observations"); + } + for (const { tool } of tools) { + const lines = (await readFile(join(directory, `${tool}.jsonl`), "utf8")).trim().split("\n"); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain(`${tool}-session`); + } + } finally { + outputLines.close(); + child.kill("SIGTERM"); + await exited; + await cleanup(); + } + }, 20_000); +}); diff --git a/cli/tests/e2e/telemetry-otel.e2e.test.ts b/cli/tests/e2e/telemetry-otel.e2e.test.ts new file mode 100644 index 000000000..51cd7c358 --- /dev/null +++ b/cli/tests/e2e/telemetry-otel.e2e.test.ts @@ -0,0 +1,53 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createTestEnv, runCli } from "./helpers.js"; + +describe.skipIf(process.platform === "win32")("native OTLP verification", () => { + it("verifies captured receipt through the CLI and fails when capture is absent", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("telemetry-verify"); + const directory = join(fakeHome, ".local/share/aidd-otel-observation"); + try { + await mkdir(directory, { recursive: true }); + await writeFile(join(directory, "collector-token"), "test-token", { mode: 0o600 }); + const args = ["--config=claude", "--endpoint=dev"]; + const init = await runCli(["telemetry", "otel", "init", ...args], projectDir, fakeHome); + expect(init.exitCode).toBe(0); + const missing = await runCli(["telemetry", "otel", "verify", ...args], projectDir, fakeHome); + expect(missing.exitCode).not.toBe(0); + await writeFile( + join(directory, "claude.jsonl"), + `${JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + attributes: [ + { key: "event.name", value: { stringValue: "user_prompt" } }, + { key: "prompt", value: { stringValue: "private-example" } }, + ], + }, + ], + }, + ], + }, + ], + }, + })}\n`, + { mode: 0o600 } + ); + const verified = await runCli(["telemetry", "otel", "verify", ...args], projectDir, fakeHome); + expect(verified.exitCode).toBe(0); + expect(verified.stdout).toContain("1 captured lines"); + expect(verified.stdout).toContain("1 normalized observations"); + expect(verified.stdout).not.toContain("private-example"); + expect(verified.stdout).not.toContain("test-token"); + } finally { + await cleanup(); + } + }); +}); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 953e3cf99..8a537b895 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -137,7 +137,7 @@ { "invocation": "aidd telemetry", "exitCode": 0, - "help": "Usage: aidd telemetry [options] [command]\n\nMeasure what Claude Code and Codex sessions consume, locally\n\nOptions:\n -h, --help display help for command\n\nCommands:\n on [options] Measure this clone, and remove what the previous\n version left in it\n off Stop measuring this clone; what was measured stays\n forget [options] Show everything measurement keeps on this machine, or\n remove it with --yes\n ingest [options] Read new transcripts into the local ledger, for\n clones that opted in\n task [options] [name] Declare the task the current work belongs to\n status Show if this clone is measured, since when, and its\n task\n report [options] Show what the work consumed, split along one axis,\n after reading new transcripts\n identity [options] [id] Choose to be named on your own measurement, or show\n or remove that choice\n help [command] display help for command" + "help": "Usage: aidd telemetry [options] [command]\n\nMeasure what Claude Code and Codex sessions consume, locally\n\nOptions:\n -h, --help display help for command\n\nCommands:\n otel Configure native OpenTelemetry exporters\n on [options] Measure this clone, and remove what the previous\n version left in it\n off Stop measuring this clone; what was measured stays\n forget [options] Show everything measurement keeps on this machine, or\n remove it with --yes\n ingest [options] Read new transcripts into the local ledger, for\n clones that opted in\n task [options] [name] Declare the task the current work belongs to\n status Show if this clone is measured, since when, and its\n task\n report [options] Show what the work consumed, split along one axis,\n after reading new transcripts\n identity [options] [id] Choose to be named on your own measurement, or show\n or remove that choice\n help [command] display help for command" }, { "invocation": "aidd telemetry forget", @@ -164,6 +164,31 @@ "exitCode": 0, "help": "Usage: aidd telemetry on [options]\n\nMeasure this clone, and remove what the previous version left in it\n\nOptions:\n --yes Do not ask first (default: false)\n --as Name you by this identifier on your own measurement\n --anonymous Name nobody on your own measurement (default: false)\n -h, --help display help for command" }, + { + "invocation": "aidd telemetry otel", + "exitCode": 0, + "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local\n capture without changing files\n collect [options] Capture authenticated OTLP on a local loopback port\n run [options] Run one fresh provider session and send its allowed\n OTLP observations\n help [command] display help for command" + }, + { + "invocation": "aidd telemetry otel collect", + "exitCode": 0, + "help": "Usage: aidd telemetry otel collect [options]\n\nCapture authenticated OTLP on a local loopback port\n\nOptions:\n --port Loopback TCP port; 0 selects a free port (default: \"14318\")\n -h, --help display help for command" + }, + { + "invocation": "aidd telemetry otel init", + "exitCode": 0, + "help": "Usage: aidd telemetry otel init [options]\n\nEnable detailed local OTLP export in one tool's user configuration\n\nOptions:\n --config User configuration to merge (choices: \"claude\",\n \"codex\", \"opencode\")\n --endpoint Collector destination (choices: \"dev\", \"prod\")\n -h, --help display help for command" + }, + { + "invocation": "aidd telemetry otel run", + "exitCode": 0, + "help": "Usage: aidd telemetry otel run [options] \n\nRun one fresh provider session and send its allowed OTLP observations\n\nOptions:\n --config Tool to run (choices: \"claude\", \"codex\",\n \"opencode\")\n --tenant Gouvernail tenant slug\n --endpoint Loopback HTTP origin\n --yes Consent to capture and send telemetry from this\n execution\n --model Provider model to use\n --follow-up Send a second prompt in the same session (Claude\n and Codex retain local history)\n --include-repository Include repository metadata for Claude Code\n --include-installation-id Send the AIDD installation ID (not a user identity)\n --keep-capture Retain the private local OTLP capture\n -h, --help display help for command" + }, + { + "invocation": "aidd telemetry otel verify", + "exitCode": 0, + "help": "Usage: aidd telemetry otel verify [options]\n\nCheck native OTLP configuration and existing local capture without changing\nfiles\n\nOptions:\n --config User configuration to verify (choices: \"claude\",\n \"codex\", \"opencode\")\n --endpoint Collector destination (choices: \"dev\", \"prod\")\n -h, --help display help for command" + }, { "invocation": "aidd telemetry report", "exitCode": 0, diff --git a/cli/tests/golden/snapshots/phase0/snapshot.json b/cli/tests/golden/snapshots/phase0/snapshot.json index 829b9d71e..33eaa8ed6 100644 --- a/cli/tests/golden/snapshots/phase0/snapshot.json +++ b/cli/tests/golden/snapshots/phase0/snapshot.json @@ -55,6 +55,7 @@ "cache/built//aidd-framework/claude/plugins/aidd-test/hooks/session-start.js", "cache/built//aidd-framework/claude/plugins/aidd-test/skills/commit/SKILL.md", "cache/built//aidd-framework/claude/plugins/aidd-test/skills/hello.md", + "installation/id.json", "manifest.json", "marketplaces.json", "references.json" @@ -704,6 +705,7 @@ "cache/built//aidd-framework/cursor/plugins/aidd-test/hooks/session-start.js", "cache/built//aidd-framework/cursor/plugins/aidd-test/skills/commit/SKILL.md", "cache/built//aidd-framework/cursor/plugins/aidd-test/skills/hello.md", + "installation/id.json", "manifest.json", "marketplaces.json", "references.json" @@ -827,7 +829,7 @@ "stderr": "Warning: claude CLI not found on PATH โ€” skipping native plugin activation.\n", "filesWritten": [], "manifest": null, - "userConfigFiles": ["manifest.json", "marketplaces.json"], + "userConfigFiles": ["installation/id.json", "manifest.json", "marketplaces.json"], "userManifest": { "version": 8, "tools": { @@ -847,7 +849,7 @@ "stderr": "", "filesWritten": [], "manifest": null, - "userConfigFiles": ["marketplaces.json"], + "userConfigFiles": ["installation/id.json", "marketplaces.json"], "userManifest": null }, { diff --git a/cli/tests/presentation/commands/setup-wiring.integration.test.ts b/cli/tests/presentation/commands/setup-wiring.integration.test.ts index 84d29583b..2f6d23e06 100644 --- a/cli/tests/presentation/commands/setup-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/setup-wiring.integration.test.ts @@ -20,9 +20,13 @@ const setupToolsPrompt = vi.fn(); const detectContext = vi.fn(); const setupMachineScope = vi.fn(); const loadManifest = vi.fn(); +const ensureInstallationId = vi.fn(); +const checkInstallationId = vi.fn(); vi.mock("../../../src/runtime/wiring/framework.js", () => ({ createDeps: vi.fn(async () => ({ + ensureInstallationId, + checkInstallationId, fs: {}, manifestRepo: { load: loadManifest }, setupMarketplaceRegistration: { resolveSourceIfNeeded, registerIfPresent }, @@ -100,6 +104,7 @@ async function run(...args: string[]): Promise { describe("aidd setup โ€” the flow a scripted run builds", () => { it("names the tools it was given, installs no plugin, and registers the default source", async () => { expect(await run("--ai", "claude")).toEqual(["Project is up to date."]); + expect(ensureInstallationId).toHaveBeenCalledOnce(); expect({ ...builtFlow() }).toEqual({ projectRoot: PROJECT_ROOT, source: undefined, @@ -330,6 +335,7 @@ describe("aidd setup โ€” what it refuses and what it reports", () => { "Warning: [claude] claude CLI refused\nError: Sync failed for: claude. See the warnings above.\n" ); expect(exit).toHaveBeenCalledWith(1); + expect(ensureInstallationId).not.toHaveBeenCalled(); }); it("names a failed install on stderr and fails the process", async () => { @@ -342,6 +348,30 @@ describe("aidd setup โ€” what it refuses and what it reports", () => { expect(errors.join("")).toBe("Error: assets are missing\n"); expect(exit).toHaveBeenCalledWith(1); + expect(ensureInstallationId).not.toHaveBeenCalled(); + }); + + it("refuses an invalid existing installation ID before setup changes anything", async () => { + checkInstallationId.mockRejectedValueOnce(new Error("invalid installation ID")); + vi.spyOn(process, "exit").mockImplementation(() => { + throw new Error("exited"); + }); + await expect(run("--ai", "claude")).rejects.toThrow("exited"); + expect(setupTools).not.toHaveBeenCalled(); + expect(ensureInstallationId).not.toHaveBeenCalled(); + expect(errors.join("")).toContain("invalid installation ID"); + }); + + it("reports that setup completed if saving the installation ID fails afterward", async () => { + ensureInstallationId.mockRejectedValueOnce(new Error("disk full")); + vi.spyOn(process, "exit").mockImplementation(() => { + throw new Error("exited"); + }); + await expect(run("--ai", "claude")).rejects.toThrow("exited"); + expect(setupTools).toHaveBeenCalled(); + expect(errors.join("")).toContain( + "Setup completed, but the installation ID could not be saved: disk full" + ); }); it("builds the graph for this project at this run's verbosity", async () => { diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index cef7703d9..7daae1c0a 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -6,6 +6,8 @@ import { conflictsOf, type ReportFilters, } from "../../../src/contexts/telemetry/domain/report/report-filter.js"; +import { createDeps } from "../../../src/runtime/wiring/framework.js"; +import * as otel from "../../../src/runtime/wiring/telemetry-otel.js"; const ingest = vi.fn(); const declare = vi.fn(); @@ -19,9 +21,13 @@ const confirm = vi.fn(); const select = vi.fn(); const input = vi.fn(); const propose = vi.fn(); +const ensureInstallationId = vi.fn(); +const checkInstallationId = vi.fn(); vi.mock("../../../src/runtime/wiring/framework.js", () => ({ createDeps: vi.fn(async () => ({ + ensureInstallationId, + checkInstallationId, prompter: { confirm, select, input }, telemetry: { telemetryOnUseCase: { execute: turnOn }, @@ -69,6 +75,7 @@ beforeEach(() => { }); afterEach(() => { vi.restoreAllMocks(); + vi.unstubAllEnvs(); }); async function run(...args: string[]): Promise { @@ -297,9 +304,12 @@ describe("aidd telemetry report, narrowed", () => { describe("aidd telemetry identity", () => { it("shows who is named when given nothing", async () => { identity.show.mockResolvedValue({ status: "unset" }); + checkInstallationId.mockResolvedValueOnce("installation-id"); expect(await run("identity")).toBeNull(); expect(identity.show).toHaveBeenCalled(); + expect(checkInstallationId).toHaveBeenCalledOnce(); expect(stdout()).toContain("No identity is set"); + expect(stdout()).toContain("Installation ID: installation-id"); }); it("names the person it is given", async () => { @@ -363,6 +373,12 @@ describe("aidd telemetry status", () => { expect(await run("status")).toBeNull(); expect(show).toHaveBeenCalledWith(process.cwd()); }); + + it("passes the global verbose setting to the telemetry dependencies", async () => { + show.mockResolvedValue({ status: "refused", reason: "no-consent" }); + expect(await run("status", "--verbose")).toBeNull(); + expect(createDeps).toHaveBeenCalledWith(process.cwd(), { verbose: true }, expect.anything()); + }); }); describe("aidd telemetry task", () => { @@ -375,7 +391,8 @@ describe("aidd telemetry task", () => { it("declares the task it is given, with its ticket", async () => { declare.mockResolvedValue(declared); - await run("task", "checkout", "--ticket", "P-1"); + expect(await run("task", "checkout", "--ticket", "P-1")).toBeNull(); + expect(stdout()).toContain('Declared task "checkout"'); expect(declare).toHaveBeenCalledWith({ cwd: process.cwd(), request: { kind: "task", task: "checkout", ticket: "P-1" }, @@ -410,6 +427,7 @@ describe("aidd telemetry task", () => { it("exits 1 when the declaration is refused", async () => { declare.mockResolvedValue({ status: "refused", reason: "no-consent" }); expect(await run("task", "x")).toBe(1); + expect(stderr()).toContain("no task was declared"); }); }); @@ -434,6 +452,7 @@ describe("aidd telemetry on", () => { it("turns measurement on in the directory it was run from, without asking, with --yes and a choice", async () => { expect(await run("on", "--yes", "--anonymous")).toBeNull(); expect(turnOn).toHaveBeenCalledWith(process.cwd(), "none"); + expect(ensureInstallationId).toHaveBeenCalledOnce(); expect(confirm).not.toHaveBeenCalled(); expect(stdout()).toContain("Measurement is on"); }); @@ -455,6 +474,7 @@ describe("aidd telemetry on", () => { expect(await run("on", "--yes")).toBe(1); expect(turnOn).not.toHaveBeenCalled(); expect(propose).not.toHaveBeenCalled(); + expect(ensureInstallationId).not.toHaveBeenCalled(); expect(stderr()).toContain("--as "); expect(stderr()).toContain("--anonymous"); }); @@ -466,7 +486,7 @@ describe("aidd telemetry on", () => { }); it("refuses an identifier the identity rules refuse, before asking anything", async () => { - expect(await run("on", "--as", "a".repeat(129))).toBe(1); + expect(await run("on", "--yes", "--as", "a".repeat(129))).toBe(1); expect(confirm).not.toHaveBeenCalled(); expect(turnOn).not.toHaveBeenCalled(); expect(stderr()).toContain( @@ -584,6 +604,16 @@ describe("aidd telemetry on", () => { turnOn.mockResolvedValue({ status: "refused", reason: "identity-set" }); expect(await run("on", "--yes", "--anonymous")).toBe(1); expect(stderr()).toContain("aidd telemetry identity --off"); + expect(ensureInstallationId).not.toHaveBeenCalled(); + }); + + it("reports that measurement is on if saving the installation ID fails afterward", async () => { + ensureInstallationId.mockRejectedValueOnce(new Error("disk full")); + expect(await run("on", "--yes", "--anonymous")).toBe(1); + expect(turnOn).toHaveBeenCalled(); + expect(stderr()).toContain( + "Measurement is on, but the installation ID could not be saved: disk full" + ); }); }); @@ -647,3 +677,129 @@ describe("the telemetry command surface", () => { expect(stderr()).toContain("disk full"); }); }); + +describe("telemetry otel init", () => { + it("refuses the production endpoint through the command without touching ledger wiring", async () => { + expect(await run("otel", "init", "--config=codex", "--endpoint=prod")).toBe(1); + expect(stderr()).toContain("Gouvernail"); + expect(turnOn).not.toHaveBeenCalled(); + }); +}); + +describe("native telemetry commands", () => { + let previousExitCode: typeof process.exitCode; + + beforeEach(() => { + previousExitCode = process.exitCode; + }); + + afterEach(() => { + process.exitCode = previousExitCode; + }); + + it("starts the local collector on its default or requested port", async () => { + const collect = vi.spyOn(otel, "collectOtel").mockResolvedValue({ + port: 54321, + close: async () => {}, + }); + expect(await run("otel", "collect")).toBeNull(); + expect(collect).toHaveBeenCalledWith(14318); + expect(stdout()).toContain("OTLP collector listening on http://127.0.0.1:54321"); + expect(await run("otel", "collect", "--port=0")).toBeNull(); + expect(collect).toHaveBeenLastCalledWith(0); + }); + + it("rejects malformed collector ports and reports binding failures", async () => { + const collect = vi.spyOn(otel, "collectOtel").mockRejectedValue(new Error("port in use")); + for (const value of ["abc", "prefix42", "42suffix"]) { + expect(await run("otel", "collect", `--port=${value}`)).toBe(1); + expect(stderr()).toContain("Collector port must be an integer"); + } + expect(collect).not.toHaveBeenCalled(); + expect(await run("otel", "collect", "--port=42")).toBe(1); + expect(stderr()).toContain("port in use"); + }); + + it("reports a new exporter, then reports the same configuration unchanged", async () => { + const configure = vi + .spyOn(otel, "configureOtel") + .mockResolvedValueOnce({ + path: "/config/claude", + launch: "Restart Claude Code", + changed: true, + }) + .mockResolvedValueOnce({ + path: "/config/claude", + launch: "Restart Claude Code", + changed: false, + }); + const args = ["otel", "init", "--config=claude", "--endpoint=dev"]; + expect(await run(...args), stderr()).toBeNull(); + expect(stdout()).toContain("Updated "); + expect(stdout()).toContain("Restart Claude Code"); + expect(stdout()).toContain( + "Local OTLP capture may include prompts, responses, commands and tool output." + ); + out = []; + expect(await run(...args), stderr()).toBeNull(); + expect(stdout()).toContain("Already configured in "); + expect(configure).toHaveBeenCalledTimes(2); + expect(configure).toHaveBeenCalledWith("claude", "dev"); + expect(turnOn).not.toHaveBeenCalled(); + }); + + it("reports capture counts without exposing their content, and fails when no capture exists", async () => { + const check = vi + .spyOn(otel, "checkOtel") + .mockRejectedValueOnce(new Error("No OTLP capture found")) + .mockResolvedValueOnce({ + path: "/config/claude", + capture: "/capture/claude.jsonl", + lines: 1, + modifiedAt: "2026-11-04T00:00:00.000Z", + observations: 1, + unrecognisedLines: 0, + }); + expect(await run("otel", "verify", "--config=claude", "--endpoint=dev")).toBe(1); + expect(stderr()).toContain("No OTLP capture found"); + out = []; + expect(await run("otel", "verify", "--config=claude", "--endpoint=dev")).toBeNull(); + expect(stdout()).toContain("Configuration verified in /config/claude"); + expect(stdout()).toContain("1 captured lines"); + expect(stdout()).toContain("1 normalized observations; 0 unrecognised lines"); + expect(stdout()).toContain( + "Existing capture does not prove delivery from the current session." + ); + expect(check).toHaveBeenCalledWith("claude", "dev"); + }); + + it("requires explicit execution consent before launching a provider", async () => { + const runOtel = vi.spyOn(otel, "runOtelSession"); + const args = [ + "otel", + "run", + "--config=claude", + "--tenant=acme", + "--endpoint=http://127.0.0.1:3000", + ]; + expect(await run(...args, "--", "reply OK")).toBe(1); + expect(stderr()).toContain("Pass --yes to consent"); + expect(runOtel).not.toHaveBeenCalled(); + runOtel.mockImplementation(async ({ retainCapture }) => { + retainCapture?.("/private/claude.jsonl"); + throw new Error("Provider exited early."); + }); + expect(await run(...args, "--yes", "--keep-capture", "--model=haiku", "--", "reply OK")).toBe( + 1 + ); + expect(runOtel).toHaveBeenCalledWith( + expect.objectContaining({ + config: "claude", + model: "haiku", + prompt: "reply OK", + retainCapture: expect.any(Function), + }) + ); + expect(stdout()).toContain("Private OTLP capture: /private/claude.jsonl"); + }); +}); diff --git a/cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts b/cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts new file mode 100644 index 000000000..f2c0e9021 --- /dev/null +++ b/cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts @@ -0,0 +1,52 @@ +import { link, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; +import { ensureInstallationId } from "../../../src/runtime/filesystem/installation-identity.js"; + +vi.mock("node:fs/promises", async (importOriginal) => { + const original = await importOriginal(); + return { ...original, link: vi.fn(original.link) }; +}); + +const roots: string[] = []; +afterEach(async () => { + vi.mocked(link).mockReset(); + await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); +}); + +async function makeRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "aidd-installation-race-")); + roots.push(root); + return root; +} + +it("keeps the identity installed by a concurrent initializer", async () => { + const root = await makeRoot(); + const winner = "a720868e-d80a-4c71-96aa-2ce56936b35d"; + vi.mocked(link).mockImplementationOnce(async (_temporary, path) => { + await writeFile(path, JSON.stringify({ installation_id: winner }), { mode: 0o600 }); + throw Object.assign(new Error("already exists"), { code: "EEXIST" }); + }); + + expect(await ensureInstallationId(root)).toBe(winner); + expect(await readdir(join(root, "installation"))).toEqual(["id.json"]); +}); + +it("cleans up the temporary identity if the filesystem rejects installation", async () => { + const root = await makeRoot(); + vi.mocked(link).mockRejectedValueOnce(Object.assign(new Error("disk error"), { code: "EIO" })); + + await expect(ensureInstallationId(root)).rejects.toMatchObject({ code: "EIO" }); + expect(await readdir(join(root, "installation"))).toEqual([]); +}); + +it("reports when a concurrently installed identity disappears", async () => { + const root = await makeRoot(); + vi.mocked(link).mockRejectedValueOnce( + Object.assign(new Error("already exists"), { code: "EEXIST" }) + ); + + await expect(ensureInstallationId(root)).rejects.toThrow("disappeared during setup"); + expect(await readdir(join(root, "installation"))).toEqual([]); +}); diff --git a/cli/tests/runtime/filesystem/installation-identity.integration.test.ts b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts new file mode 100644 index 000000000..a87291ec3 --- /dev/null +++ b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts @@ -0,0 +1,151 @@ +import { + chmod, + mkdir, + mkdtemp, + readdir, + readFile, + rm, + stat, + symlink, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + checkInstallationId, + ensureInstallationId, + finishInstallationId, +} from "../../../src/runtime/filesystem/installation-identity.js"; + +const roots: string[] = []; +async function makeRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "aidd-installation-id-")); + roots.push(root); + return root; +} +afterEach(async () => { + await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); +}); + +describe("installation identity", () => { + it("creates one private UUID across repeated and concurrent initializations", async () => { + const root = await makeRoot(); + const ids = await Promise.all(Array.from({ length: 12 }, () => ensureInstallationId(root))); + expect(new Set(ids).size).toBe(1); + expect(ids[0]).toMatch( + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i + ); + expect(await ensureInstallationId(root)).toBe(ids[0]); + const path = join(root, "installation", "id.json"); + expect(JSON.parse(await readFile(path, "utf8"))).toEqual({ installation_id: ids[0] }); + expect(await readdir(join(root, "installation"))).toEqual(["id.json"]); + if (process.platform !== "win32") expect((await stat(path)).mode & 0o777).toBe(0o600); + }); + + it("preserves a previously stored UUID and any additional fields", async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + const existing = '{"installation_id":"a720868e-d80a-4c71-96aa-2ce56936b35d","future":"keep"}\n'; + await mkdir(directory); + await writeFile(join(directory, "id.json"), existing, { mode: 0o600 }); + expect(await ensureInstallationId(root)).toBe("a720868e-d80a-4c71-96aa-2ce56936b35d"); + expect(await readFile(join(directory, "id.json"), "utf8")).toBe(existing); + }); + + it("refuses an invalid existing identity without replacing it", async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + await writeFile(join(directory, "id.json"), '{"installation_id":"invalid"}', { mode: 0o600 }); + await expect(checkInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + await expect(ensureInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + expect(await readFile(join(directory, "id.json"), "utf8")).toBe( + '{"installation_id":"invalid"}' + ); + }); + + it("checks an absent installation without creating one", async () => { + const root = await makeRoot(); + expect(await checkInstallationId(root)).toBeNull(); + expect(await readdir(root)).toEqual([]); + }); + + it.each([ + '{"installation_id":"prefix-a720868e-d80a-4c71-96aa-2ce56936b35d"}', + '{"installation_id":"a720868e-d80a-4c71-96aa-2ce56936b35d-suffix"}', + '{"installation_id":"a720868e-d80a-1c71-96aa-2ce56936b35d"}', + '{"installation_id":null}', + "{}", + "[]", + "{invalid", + ])("refuses malformed stored identity %s", async (stored) => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + await writeFile(join(directory, "id.json"), stored, { mode: 0o600 }); + await expect(checkInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + await expect(ensureInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + expect(await readFile(join(directory, "id.json"), "utf8")).toBe(stored); + }); + + it("refuses to initialize when the parent is not a directory", async () => { + const root = await makeRoot(); + await writeFile(join(root, "installation"), "not a directory"); + if (process.platform === "win32") await expect(checkInstallationId(root)).resolves.toBeNull(); + else await expect(checkInstallationId(root)).rejects.toMatchObject({ code: "ENOTDIR" }); + await expect(ensureInstallationId(root)).rejects.toThrow(); + expect(await readFile(join(root, "installation"), "utf8")).toBe("not a directory"); + }); + + it.skipIf(process.platform === "win32")("rejects a directory as the identity file", async () => { + const root = await makeRoot(); + await mkdir(join(root, "installation", "id.json"), { recursive: true }); + await expect(checkInstallationId(root)).rejects.toThrow("private regular file"); + }); + + it.skipIf(process.platform === "win32")("refuses a symlink without following it", async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + const target = join(root, "target.json"); + await writeFile(target, "untouched", { mode: 0o600 }); + await symlink(target, join(directory, "id.json")); + await expect(ensureInstallationId(root)).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("untouched"); + }); + + it.skipIf(process.platform === "win32")( + "refuses an existing file with broad permissions", + async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + await writeFile( + join(directory, "id.json"), + '{"installation_id":"a720868e-d80a-4c71-96aa-2ce56936b35d"}', + { mode: 0o644 } + ); + await chmod(join(directory, "id.json"), 0o644); + await expect(ensureInstallationId(root)).rejects.toThrow("private regular file"); + } + ); + + it("reports an already applied OTLP configuration if identity creation then fails", async () => { + await expect( + finishInstallationId(async () => { + throw new Error("disk full"); + }, "OTLP configuration completed") + ).rejects.toThrow( + "OTLP configuration completed, but the installation ID could not be saved: disk full" + ); + }); + + it("reports non-Error failures after the earlier action already succeeded", async () => { + await expect( + finishInstallationId(async () => { + throw "disk full"; + }, "Measurement is on") + ).rejects.toThrow("Measurement is on, but the installation ID could not be saved: disk full"); + }); +}); diff --git a/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts b/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts index da0c88c36..580b6fd07 100644 --- a/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts +++ b/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts @@ -141,6 +141,14 @@ describe("replacing a file", () => { await expect(storage.replace(path, "x")).rejects.toThrow(); expect(await readdir(dir)).toEqual([]); }); + + it("removes the temporary file when the completed write cannot replace a directory", async () => { + const path = join(dir, "target"); + await mkdir(path); + await expect(storage.replace(path, "x")).rejects.toThrow(); + expect(await readdir(dir)).toEqual(["target"]); + expect((await stat(path)).isDirectory()).toBe(true); + }); }); describe("appending to a file", () => { diff --git a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts new file mode 100644 index 000000000..60cc3483f --- /dev/null +++ b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts @@ -0,0 +1,98 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ + initialize: vi.fn(), + verify: vi.fn(), + checkId: vi.fn(), + ensureId: vi.fn(), + finishId: vi.fn(), +})); + +vi.mock("node:os", async (load) => ({ + ...(await load()), + homedir: () => "/profile", +})); +vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/init.js", () => ({ + initializeOtel: mocks.initialize, +})); +vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js", () => ({ + verifyOtel: mocks.verify, +})); +vi.mock("../../../src/runtime/filesystem/installation-identity.js", () => ({ + checkInstallationId: mocks.checkId, + ensureInstallationId: mocks.ensureId, + finishInstallationId: mocks.finishId, +})); +vi.mock("../../../src/runtime/user-config-dir.js", () => ({ + userConfigDirOf: () => "/profile/aidd", +})); + +const { collectOtel, configureOtel, checkOtel } = await import( + "../../../src/runtime/wiring/telemetry-otel.js" +); +const collector = await import( + "../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js" +); +beforeEach(() => { + vi.clearAllMocks(); + mocks.checkId.mockReset().mockResolvedValue(null); + mocks.initialize + .mockReset() + .mockResolvedValue({ path: "/config", changed: true, launch: "restart" }); + mocks.ensureId.mockReset().mockResolvedValue("installation-id"); + mocks.finishId.mockReset().mockImplementation(async (ensure: () => Promise) => { + await ensure(); + }); +}); + +describe("OTLP runtime wiring", () => { + it("places the collector in the user's profile and uses the requested port", async () => { + const start = vi.spyOn(collector, "startOtelCollector").mockResolvedValue({ + port: 4318, + close: async () => {}, + }); + await collectOtel(4318); + expect(start).toHaveBeenCalledWith({ home: "/profile", port: 4318 }); + start.mockRestore(); + }); + + it("validates identity before initialization and saves it only after a successful change", async () => { + const result = await configureOtel("claude", "dev"); + expect(result).toMatchObject({ path: "/config", changed: true }); + expect(mocks.checkId).toHaveBeenCalledWith("/profile/aidd"); + expect(mocks.initialize).toHaveBeenCalledWith("claude", "dev", { + home: "/profile", + env: process.env, + }); + expect(mocks.checkId.mock.invocationCallOrder[0]).toBeLessThan( + mocks.initialize.mock.invocationCallOrder[0] + ); + expect(mocks.finishId.mock.invocationCallOrder[0]).toBeGreaterThan( + mocks.initialize.mock.invocationCallOrder[0] + ); + expect(mocks.ensureId).toHaveBeenCalledWith("/profile/aidd"); + }); + + it("does not change exporter settings when an existing installation ID is invalid", async () => { + mocks.checkId.mockRejectedValue(new Error("invalid identity")); + await expect(configureOtel("codex", "dev")).rejects.toThrow("invalid identity"); + expect(mocks.initialize).not.toHaveBeenCalled(); + expect(mocks.ensureId).not.toHaveBeenCalled(); + }); + + it("does not create an ID when a production destination is refused", async () => { + mocks.initialize.mockRejectedValue(new Error("production unavailable")); + await expect(configureOtel("codex", "prod")).rejects.toThrow("production unavailable"); + expect(mocks.checkId).not.toHaveBeenCalled(); + expect(mocks.ensureId).not.toHaveBeenCalled(); + }); + + it("passes its runtime home and environment to verification", async () => { + mocks.verify.mockResolvedValue({ lines: 1 }); + await expect(checkOtel("codex", "dev")).resolves.toEqual({ lines: 1 }); + expect(mocks.verify).toHaveBeenCalledWith("codex", "dev", { + home: "/profile", + env: process.env, + }); + }); +}); diff --git a/plugins/aidd-telemetry/CATALOG.md b/plugins/aidd-telemetry/CATALOG.md index bbc7098ed..1dbcb6873 100644 --- a/plugins/aidd-telemetry/CATALOG.md +++ b/plugins/aidd-telemetry/CATALOG.md @@ -32,11 +32,14 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| -| `actions` | [forget.md](skills/00-init/actions/forget.md) | - | -| `actions` | [identity.md](skills/00-init/actions/identity.md) | - | -| `actions` | [start.md](skills/00-init/actions/start.md) | - | -| `actions` | [stop.md](skills/00-init/actions/stop.md) | - | -| `-` | [SKILL.md](skills/00-init/SKILL.md) | `Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Not for answering what a period consumed.` | +| `actions` | [01-probe.md](skills/00-init/actions/01-probe.md) | - | +| `actions` | [02-start.md](skills/00-init/actions/02-start.md) | - | +| `actions` | [03-identity.md](skills/00-init/actions/03-identity.md) | - | +| `actions` | [04-stop.md](skills/00-init/actions/04-stop.md) | - | +| `actions` | [05-forget.md](skills/00-init/actions/05-forget.md) | - | +| `actions` | [06-otel-init.md](skills/00-init/actions/06-otel-init.md) | - | +| `actions` | [07-otel-verify.md](skills/00-init/actions/07-otel-verify.md) | - | +| `-` | [SKILL.md](skills/00-init/SKILL.md) | `Configure and verify provider OpenTelemetry or manage local AIDD measurement. Use when the user wants to enable export, check receipt, start or stop usage measurement, change their identity, or delete measured data. Not for usage reports.` | #### `skills/01-usage` diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md new file mode 100644 index 000000000..d140b402d --- /dev/null +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -0,0 +1,56 @@ +# Provider OpenTelemetry + +Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2, an opted-in AIDD plugin converts public OpenCode server events into OTLP; these are not emitted by OpenCode's standard OTLP exporter. The framework's existing local usage ledger is separate and is not sent as OTel. An observation is not proof that every event was delivered; missing or ambiguous data remains unavailable. + +| Field | Claude Code OTel | Codex OTel | OpenCode v2 OTel | +| --- | --- | --- | --- | +| Session | `session.id`, when present | `conversation.id` in logs; `thread.id` in observed turn spans | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | +| Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | +| Usage | Forwarded `claude_code.token.usage` metrics and technical fields from `api_request` events | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | +| Identity | Provider attributes when emitted, or an explicit connector-token assignment | Provider attributes when emitted, or an explicit connector-token assignment | An explicit connector-token assignment | +| Repository and branch | Repository URL when explicitly enabled; no branch attribute in the local two-turn capture | No repository or branch attribute observed in the local two-turn capture | No repository or branch attribute observed in the local two-turn capture | + +Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. + +In local Gouvernail, an assigned connector token binds a tool to a declared member, team and installation UUID. This is not proof of the person holding the token. Older shared tokens retain declarative attribution. + +`otel run --yes` captures one provider execution in isolation and forwards supported observations and allowlisted attributes to loopback Gouvernail. It neither reads the local usage ledger nor asserts a clone or feature association. With `--keep-capture`, raw private files can contain sensitive content. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). + +In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. Five other HTTP spans per execution contained that session ID but were not forwarded. The prompt spans had no trace-linked descendants in these captures; the model and usage spans belonged to a separate server trace without a session ID. A startup log included the requested model, but no session ID or proof of the model actually used. Those runs forwarded only the validated prompt span. These observations do not prove what other versions or configurations emit. + +The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.started`, `session.step.ended` and `session.step.failed`, and exports each event separately as OTLP logs. It includes the original event and session IDs, message ID when present, selected model and variant at step start, and positive OpenCode-normalized counters at step end. A creation event can also carry an OpenCode project ID and parent session ID; neither is a repository name. It does not export prompts, responses, tool contents or cost. The plugin stream can miss events; event IDs allow deduplication of observations received, not proof of complete delivery. Do not match retries solely by assistant message ID. Production export remains disabled. + +One local setup uses `@openchamber/opencode-claude` v1.7.0 behind an OpenAI-compatible provider named `claude-code`. Its per-turn usage conversion maps a missing cache-write count and an explicit zero to the same output, before OpenCode receives it. An OpenCode HTTP hook cannot distinguish those cases in that conversion path. For a turn with multiple model calls, the proxy can report the last call's prompt context rather than the sum of their inputs. This is specific to that provider and version; it does not establish the behavior of every OpenCode provider. [Provider usage conversion](https://github.com/openchamber/opencode-claude/blob/v1.7.0/src/usage.ts) (read 2026-11-07). + +## Additional coverage investigated + +| Provider | Available or observed in OTLP | Current limit and next step | +| --- | --- | --- | +| Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort and separate token and cache counters. These fields appeared in both local captures and are forwarded when valid. | The receiver compares request and metric counts; it does not add the two signals. Missing or conflicting observations remain unavailable. Repository attributes require a separate opt-in. | +| Codex | In local two-turn captures with CLI 0.166.0 and 0.167.0, a `session_task.turn` span contained `thread.id`, `turn.id`, selected model, effort and turn usage together. | The span counts the session usage delta during the turn and may cover several requests or models. The model is selected, not confirmed for every request. Cache and output zeros can be normalized defaults. A turn ID can recur during recovery; retain trace and span IDs. `otel run` sends allowlisted turn spans after session logs; generic `otel sync` does not yet forward these spans. Never add those counters to response-event counters. | +| OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | The opted-in plugin sends session-linked OpenCode events as OTLP. Missing counters are normalized to zero upstream, so zeros cannot establish actual usage. A model variant is not necessarily reasoning effort. | + +An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. + +In one two-prompt Claude capture, the second request's input and cache-read metric series started one millisecond before the cache-creation series, although `api_request` reported all three counts together. The dashboard does not join metric windows by temporal proximity. It can display the observed input including cache when the deduplicated request counts agree with each corresponding metric count. Event delivery can still be incomplete. [Claude event correlation](https://code.claude.com/docs/en/monitoring-usage#event-correlation-attributes) (read 2026-11-07). + +## Sources + +Read 2026-11-06: + +- [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) +- [Claude Code API request event](https://code.claude.com/docs/en/monitoring-usage#api-request-event) +- [Codex telemetry configuration](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted) +- [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts) +- [OpenCode v2.2.5 usage projection](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/projector.ts#L81-L110) +- [OpenCode v2.2.5 session events](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts#L333-L389) and [usage normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts#L8-L19) +- [OpenCode v2 server plugins](https://dev.opencode.ai/v2/docs/build/plugins/server) and [retry handling](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/runner/llm.ts#L229-L334) +- [OpenCode v2 HTTP API audit](https://github.com/anomalyco/opencode/blob/dev/V2_HTTP_API_AUDIT.md) (current development branch; the v2.2.5 path was observed locally) +- [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) +- [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) + +Read 2026-11-07: [Codex 0.166.0 turn span and counters](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L369-L382), [turn counter capture](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L760-L810) and [cache defaulting](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/codex-api/src/sse/responses.rs#L131-L153). + +In the 0.167.0 capture, two turn spans and four response events report zero cache writes, but they share [the same upstream defaulting](https://github.com/openai/codex/blob/rust-v0.167.0/codex-rs/codex-api/src/sse/responses.rs#L124-L145), so they do not independently establish a zero. Codex's [`non_cached_input_tokens`](https://github.com/openai/codex/blob/rust-v0.167.0/codex-rs/protocol/src/protocol.rs#L2487-L2492) subtracts cache reads but can still include cache writes; it is not the dashboard's cache-excluding input. Sources read 2026-11-07. + +Read 2026-11-07: [Claude Code repository attributes](https://code.claude.com/docs/en/monitoring-usage#repository-attributes). diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 1d873fa43..79742e476 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -2,7 +2,7 @@ # aidd-telemetry -Counts the tokens your Claude Code and Codex sessions consume. +Measures Claude Code and Codex usage locally, and can configure provider OpenTelemetry export. | | | | --- | --- | @@ -24,6 +24,34 @@ flowchart LR Tasks -->|"attribute"| Report ``` +The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The framework's local ledger powers `aidd telemetry report` and remains separate. Do not add their totals: both sources can cover the same calls. See the [field inventory](OBSERVABILITY.md). + +## Choose a mode + +| Mode | Opt-in and destination | Identity | What is kept or sent | +| --- | --- | --- | --- | +| Local usage | `aidd telemetry on` explicitly enables measurement for the current clone from that time. The ledger and report stay on this machine. | An optional name labels the local report; anonymous use is supported. | Per-call usage read from Claude Code and Codex session files: session, time, model, agent, token counts and local attribution context. The ledger does not store prompts or responses. | +| OTLP development | `otel run --yes` explicitly opts in one isolated execution and sends supported observations to loopback Gouvernail. | A token can be explicitly assigned to a team, member and installation for each tool in local Gouvernail. It does not prove who holds it. Older shared tokens retain their declarative attribution. | With `--keep-capture`, the private capture can contain prompts, responses, commands and tool output. Forwarding filters fields but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | +| OTLP production | **Not available:** `--endpoint=prod` is refused. `https://gouvernail.eu` is an intended future destination, not an active exporter. | **Required before production is enabled:** an authenticated user identity. This is a product requirement, not yet implemented or enforced. | No production payload is sent by this feature today. Consent, authentication, retention and deletion behavior must be implemented before enabling it. | + +### Consent and scope + +- `on` opts in the separate local usage ledger for one clone. `otel run --yes` opts in the single provider execution independently; it does not attribute observations to a clone or feature. +- `otel init` separately configures a user-level exporter. Its global collector can capture other projects and does not check clone consent. It is not required for `otel run`. +- `aidd telemetry off` and `forget` affect local usage measurement; they do not disable provider exporters or delete retained OTLP captures or data already received by Gouvernail. +- Retained captures are private local files in the system temporary directory. The CLI does not delete them, but system cleanup can. They can contain sensitive content. Claude metric forwarding also sends a deterministic series fingerprint, which can expose equality or allow guesses of low-entropy values. +- Delivery after a timeout cannot guarantee exactly-once behavior without receiver-side idempotence. + +### Three different identifiers + +| Identifier | Purpose and storage | Sent to Gouvernail? | +| --- | --- | --- | +| Local report name | Optional label in `~/.config/aidd/telemetry/identity.json`; the current label is applied when reporting, not recorded as proof per call. | No. | +| Installation UUID | Stable ID in `~/.config/aidd/installation/id.json`; not a login or credential. | Only with `otel run --include-installation-id`, as `aidd.installation.id`. | +| Provider identity | Email, account or organization ID when emitted. Gouvernail must match an approved active member to attribute a session. | Allowlisted identifiers may be sent by `otel run`. | + +The default paths can change with the configured user directory. A connector bearer token authorizes the destination connection. In local Gouvernail, an assigned token also identifies its declared member, team and installation; it does not prove who holds the token. + ## Start 1. `npm install -g @ai-driven-dev/cli` @@ -43,12 +71,22 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. - Only clones that ran `on` are measured, from then on. - A task and a ticket come only from what you declare. - Each model call counts once, whatever the axis. +- If an origin change is observed in the same clone, usage before its detection keeps its tokens but loses repository and task attribution. Changes made and reversed between observations, or changes overwritten before this protection existed, cannot be detected retrospectively. +- Repository attribution is unavailable for usage older than the first recorded observation of its clone. If the local resolution history is lost, attribution remains unavailable for earlier usage. +- A directory that is no longer available cannot establish its current origin, so its usage keeps its tokens without repository or task attribution. - Where you are asked for a task, and the limits: see the usage contract. -## Privacy +## Local usage measurement privacy -- Nothing leaves your machine. +- This local usage ledger and its reports stay on your machine. Provider OTLP export is a separate opt-in described above. - Consent is per clone and never committed. - Data lives in `~/.config/aidd/telemetry` and the clone's git config. - You are named only if you choose to be. -- `AIDD_TELEMETRY=0` refuses measurement everywhere. +- `AIDD_TELEMETRY=0` refuses AIDD usage measurement everywhere. +- Framework setup creates a stable installation UUID at `~/.config/aidd/installation/id.json` (or under the configured user directory), independently of the optional person label. It is not a credential or a login; telemetry setup creates it for older installations. It is retained when telemetry is forgotten. It is not linked to an account or attached to local usage records. The optional `otel run --include-installation-id` adds it to that execution's outgoing OTLP resources; it does not authenticate a person. + +## Provider OpenTelemetry + +Configure an exporter with `aidd telemetry otel init --config= --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. + +OpenCode initialization installs an opt-in server plugin. It sends session and event IDs, selected model and variant, optional OpenCode project and parent IDs, and positive OpenCode-normalized token counts to the configured local collector. It does not send prompts, responses, tool contents or cost. A missing or zero counter remains unavailable; delivery is not guaranteed. Restart the OpenCode v2 service after initialization. The one-execution `otel run --yes` enables this plugin for that run; it does not require global initialization. diff --git a/plugins/aidd-telemetry/skills/00-init/SKILL.md b/plugins/aidd-telemetry/skills/00-init/SKILL.md index ba79d2197..397026006 100644 --- a/plugins/aidd-telemetry/skills/00-init/SKILL.md +++ b/plugins/aidd-telemetry/skills/00-init/SKILL.md @@ -1,40 +1,36 @@ --- name: 00-init -description: Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Not for answering what a period consumed. -argument-hint: opt in | my name | opt out | delete +description: Configure and verify provider OpenTelemetry or manage local AIDD measurement. Use when the user wants to enable export, check receipt, start or stop usage measurement, change their identity, or delete measured data. Not for usage reports. +argument-hint: provider export | verify receipt | local measurement | my name | stop | delete --- # Init ```mermaid flowchart TD - request([request]) --> probe{"aidd answers telemetry?"} - probe -->|"no"| absent([say it is missing, change nothing]) - probe -->|"yes"| route{which case} - route -->|"start measuring"| start --> done([reported as it ran]) - route -->|"name me, or stop naming me"| identity --> done - route -->|"stop measuring"| stop --> done - route -->|"delete what was measured"| forget - forget -->|"person confirms the preview"| done - forget -->|"person declines"| kept([nothing removed]) + export(["provider export"]) --> init["actions/06-otel-init.md"] + receipt(["verify receipt"]) --> verify["actions/07-otel-verify.md"] + init -->|"configured"| verify + init -->|"declined or unavailable"| unchanged(["nothing changed"]) + verify --> result(["reported CLI result"]) + local(["local usage measurement"]) --> probe["actions/01-probe.md"] + probe -->|"CLI unavailable"| unchanged + probe -->|"start"| start["actions/02-start.md"] + probe -->|"identity"| identity["actions/03-identity.md"] + probe -->|"stop"| stop["actions/04-stop.md"] + probe -->|"delete"| forget["actions/05-forget.md"] + start -->|"consent given"| result + start -->|"declined"| unchanged + identity --> result + stop -->|"confirmed"| result + stop -->|"declined"| unchanged + forget -->|"confirmed"| result + forget -->|"declined or empty"| unchanged ``` -## Actions - -Run the flow above. Read only the action file the case needs. - -| Action | Does | -| --- | --- | -| start | ask for consent, then turn measurement on for this project | -| identity | set, show or remove the name on the person's own measurement | -| stop | turn measurement off for this project, keeping what was measured | -| forget | preview, then on confirmation delete everything measured on this machine | - ## Transversal rules -- Probe first. Run `aidd telemetry on --help` before any action. When `aidd` is not found, the command exits non-zero, or its help does not list `--anonymous`, stop and say plainly that `aidd` is missing or too old to measure, name `@ai-driven-dev/cli` as what to install or update, and state that nothing was changed. Never go on, and never answer as if it had worked. -- Every change goes through `aidd telemetry`. Never edit git config, `.aidd/config.json` or any file by hand. -- Never run a command that changes anything before the person has said yes to what it does. A silence, a previous yes to another case, or a standing instruction is not a yes. -- Report what each command printed, including every warning, and never claim a result the output did not show. -- Never choose, derive or guess a name for the person: not from git, from the machine, nor from the session. Only the identifier the person types is used. `aidd telemetry on` run in a terminal offers the git name itself. -- When a command exits non-zero, show its message and stop. Do not retry with other flags. +- Keep provider export and local usage measurement separate; run only the requested path. +- Apply changes through `aidd telemetry`, with the person's explicit authorization for that operation. +- Report command output and warnings with credentials redacted; a non-zero exit ends the action without retrying with other flags. +- Use only identifiers supplied by the person, rather than deriving one from git, the machine or the session. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/01-probe.md b/plugins/aidd-telemetry/skills/00-init/actions/01-probe.md new file mode 100644 index 000000000..9460e7876 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/01-probe.md @@ -0,0 +1,17 @@ +# 01 - Probe local measurement + +Check that the installed CLI supports local measurement before choosing its action. + +## Input + +The requested local operation. + +## Process + +1. Run `aidd telemetry on --help` and check that it lists `--anonymous`. + - If unavailable, report that `@ai-driven-dev/cli` needs installing or updating and that nothing changed. +2. Continue to the requested action, [start](02-start.md), [identity](03-identity.md), [stop](04-stop.md), or [forget](05-forget.md). + +## Output + +A supported CLI and the next local action, or the availability failure. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/start.md b/plugins/aidd-telemetry/skills/00-init/actions/02-start.md similarity index 58% rename from plugins/aidd-telemetry/skills/00-init/actions/start.md rename to plugins/aidd-telemetry/skills/00-init/actions/02-start.md index ab6c09130..fe6f62d1b 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/start.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/02-start.md @@ -1,4 +1,4 @@ -# Start +# 02 - Start Turn measurement on for the current project, once the person has agreed to it. @@ -6,41 +6,30 @@ Turn measurement on for the current project, once the person has agreed to it. The project directory, which must be inside a git repository. -## Output - -Measurement is on for the project, or nothing changed and the reason is stated. The person knows what is measured, what they will be asked, where it stays, and under which name, if any, they are measured. - ## Process -1. **Explain.** Tell the person, before asking anything: +1. Tell the person, before asking anything: - what is measured: the token counts of Claude Code and Codex sessions run in this project, read from each tool's own session files (Codex's under `CODEX_HOME`, else `~/.codex`), split into input, output, cache read and cache write; - what is asked: one question per working branch, a task name and optionally a ticket, only when a person is at the keyboard, in Claude Code or the Codex TUI. Codex asks only once the person has reviewed and trusted this plugin's checks in its TUI, which lists them for review, and never in `codex exec`, its IDE extension, its desktop app or full-access mode; - when it starts: now. Measurement starts at the moment it is turned on, and what this project's sessions did before is not counted, whatever Claude Code still holds of it; - where it stays: on this machine, and nothing is sent anywhere; - whose consent it is: this clone's alone, kept in its git config and never committed, so a teammate is not measured until they run it in theirs. -2. **Ask.** Ask whether to turn measurement on for this project. Wait for an explicit yes. +2. Ask whether to turn measurement on for this project. Wait for an explicit yes. - A no, or no answer, ends the action with nothing changed. -3. **Choose the name.** Run `aidd telemetry identity` to read whether an identity is already set. +3. Run `aidd telemetry identity` to read whether an identity is already set. - Set: say the person is measured under it, and that the `identity` case changes or removes it. Nothing to ask. - Not set: explain that a name only labels the person axis of their own report and stays on this machine, then ask the person to type the identifier they want, or to stay anonymous. Wait for the answer; never suggest or pick one for them. Say that `aidd telemetry on`, run in their own terminal, offers their git name instead. -4. **Turn on.** The yes was given in step 2, so the command's own confirmation is skipped: +4. Run the matching command with the consent already given: - identity already set: `aidd telemetry on --yes`; - a name typed: `aidd telemetry on --yes --as ''`, with exactly the text typed, single-quoted for the shell, each `'` it holds written as `'\''` (`O'Brien` becomes `'O'\''Brien'`); - anonymous: `aidd telemetry on --yes --anonymous`. Staying anonymous is not stored: the next `on`, in another clone, asks again. - Outside a git repository, with a git config git cannot read, or with an identifier that is not one non-empty line of at most 128 characters, the command refuses: show its message and stop. -5. **Report.** Relay everything the command printed, the removals and the warnings included, in the person's words rather than pasted. +5. Relay everything the command printed, the removals and the warnings included, in the person's words rather than pasted. - When it warned that Claude Code keeps session files only briefly, say that a call is measured only if it is read before the files are removed, and give the setting it named. -6. **Prove.** Run `aidd telemetry report --days 1`, and state what it shows. +6. Run `aidd telemetry report --days 1`, and state what it shows. - No calls yet is a normal answer for a project that has just opted in: say so, and that the next Claude Code or Codex session in this project will be counted. -7. **Tell the rest.** Say that the plugin is what asks about the task and that it needs `node`, and that a task can always be declared by hand with `aidd telemetry task [--ticket ]` or `aidd telemetry task --none`, and that `aidd telemetry status` shows whether a clone is measured and its task. +7. Say that the plugin is what asks about the task and that it needs `node`, and that a task can always be declared by hand with `aidd telemetry task [--ticket ]` or `aidd telemetry task --none`, and that `aidd telemetry status` shows whether a clone is measured and its task. -## Test +## Output -| Case | Pass | -| --- | --- | -| The person says no | No `aidd telemetry on` was run and the project is unchanged | -| The person says yes, with no identity set | They were asked to type an identifier or stay anonymous, no name was suggested and git was not read, exactly one of `aidd telemetry on --yes --as ''`, with the text they typed, or `aidd telemetry on --yes --anonymous` ran, and the person was told that measurement starts now, that earlier history is not counted, and, if the command removed or deleted a block in `.aidd/config.json`, that the file may need committing | -| The person says yes, with an identity set | They were asked nothing about a name, were told the one they are measured under, `aidd telemetry on --yes` ran once, and the person was told the same as above about when measurement starts and `.aidd/config.json` | -| The CLI's `on` has no `--anonymous` | The skill said to update `@ai-driven-dev/cli` and ran nothing else | -| `aidd` has no `telemetry` | The skill said so and ran nothing else | -| Run outside a git repository | The refusal was shown and nothing was written | +Measurement is on for the project, or nothing changed and the reason is stated. The person knows what is measured, what they will be asked, where it stays, and under which name, if any, they are measured. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md b/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md new file mode 100644 index 000000000..c4a1aa5c1 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md @@ -0,0 +1,26 @@ +# 03 - Identity + +Let the person choose to be named on their own measurement, or stop being named. + +## Input + +What the person wants: to be named as an identifier they give, to see the current choice, or to be named no more. + +## Process + +1. Say that the local report name is optional, labels the person axis of their own report, and is kept on this machine. It is not the provider identity, the installation UUID, or a Gouvernail login. +2. Run `aidd telemetry identity` to read the current choice, and tell the person what it holds. +3. To set one, ask the person for the identifier they want and wait for it. Run `aidd telemetry identity ` with exactly that text. + - The command refuses an identifier that is empty, over 128 characters, or spans lines: show its message and stop. +4. To remove it, confirm that the person wants to be named no more, then run `aidd telemetry identity --off`. +5. Run `aidd telemetry identity` again after a change and check that it reports the requested identifier or its removal. Present the result in the table below; fill provider, endpoint and dates only when a relevant CLI command actually reports them. Do not infer an identity or expose credentials. + +## Output + +| Item | Tools | Endpoint | Created | Stored | +| --- | --- | --- | --- | --- | +| Local report name | Claude Code and Codex local report; not tool-specific | None | Not recorded by this command | `~/.config/aidd/telemetry/identity.json` by default | +| Installation UUID | Installation, not a person | Not currently sent | Not reported by this command | `~/.config/aidd/installation/id.json` by default | +| Provider identity | Provider and values only if verified | Dev destination only if verified; prod unavailable | Not reported by this command | Local OTLP capture if emitted; provider storage unknown | + +Show the local name or `anonymous` as reported. Paths may differ with the configured user directory. A provider's native identity can be forwarded to local Gouvernail even when the local report is anonymous. Production user identity is a future requirement, not implemented today. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md new file mode 100644 index 000000000..13e12141d --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md @@ -0,0 +1,23 @@ +# 04 - Stop + +Turn measurement off for the current project and keep what was already measured. + +## Input + +The project directory. + +## Process + +1. Say what stopping does: new Claude Code or Codex work in this clone is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. It does not disable provider OTLP exporters or the raw collector. Wait for a yes. +2. Run `aidd telemetry off` and relay its message. +3. Run `aidd telemetry status` and verify that measurement is off. Report the result by scope in the table below. The CLI does not report the exact stop timestamp; do not invent one. An independently consented `otel run` is unaffected. +4. Say that deleting what was measured is a separate step, and that it needs its own confirmation. If the person wants raw OTLP to stop too, explain that `off` does not do that: the exporter and collector must be stopped separately, and there is no automatic restore command in this feature. + +## Output + +| Scope | Result after `off` | Since | Existing data | +| --- | --- | --- | --- | +| Local usage for this clone | Stopped if the command and status confirm it | Exact time not reported by the CLI | Retained | +| OTLP export and raw local collection | Not stopped by `off`; current process state is unknown unless separately checked | Unknown | Captures retained | +| OTLP execution to local Gouvernail | Independent; `otel run --yes` consents to each execution | Not controlled by `off` | Already received data retained | +| Gouvernail production | Not available in this feature | Not applicable | Not managed by this command | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md new file mode 100644 index 000000000..f65b0ea43 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md @@ -0,0 +1,18 @@ +# 05 - Forget + +Delete local usage measurement after a preview and explicit confirmation. This command does not delete provider OTLP captures, the installation UUID or data already received by Gouvernail. + +## Process + +1. Run `aidd telemetry forget` with no flag. It removes nothing and lists what it would. + - When it says nothing measured is kept, relay that and stop. +2. Give the person the list as the command printed it, with the warnings about repositories it could not reach. +3. Ask whether to remove all of it. Wait for an explicit yes naming the removal. + - A no, or no answer, ends the action with nothing removed. +4. Run `aidd telemetry forget --yes` only after that yes, and relay what it printed. + - Say that it also ends local usage measurement until the person runs `aidd telemetry on`; provider exporters and raw OTLP collection may still run. +5. Run `aidd telemetry forget` again and check that its preview lists nothing remaining. + +## Output + +Either the list of what was removed, or the list of what would be, with nothing removed. Separately name OTLP captures and Gouvernail data as not removed. Gouvernail production is not implemented here; the requested future policy disallows production deletion and is not enforced by this local command. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md new file mode 100644 index 000000000..f4627e4cc --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md @@ -0,0 +1,28 @@ +# 06 - Initialize provider export + +Merge native OpenTelemetry settings into the selected provider's local configuration. + +## Input + +The provider (`claude`, `codex` or `opencode`) and destination (`dev` or `prod`) chosen by the person. + +## Process + +1. Run `aidd telemetry otel init --help` and report an unavailable command as requiring an update of `@ai-driven-dev/cli`. +2. Obtain both choices and explain the separate opt-in: provider telemetry goes to the selected collector and can include prompts, responses, commands and tool output. The provider's user-level export can cover other projects; the raw collector does not check clone consent. + - The CLI's development endpoint is `http://127.0.0.1:14318`. Start the collector with `aidd telemetry otel collect`; it creates the private token. Initialization does not start it. + - Production is refused. `https://gouvernail.eu` is an intended future destination, not a configured endpoint; requiring an authenticated person there is a future requirement, not enforced today. +3. Run `aidd telemetry otel init --config= --endpoint=` after an explicit request enabling that export. + - A request naming both choices is sufficient authorization. + - Let the CLI validate configuration and credentials without requesting a token in chat. +4. Report whether configuration changed and relay the CLI's restart or launch instructions. OpenCode v2 needs its service restarted. +5. Continue to [verify receipt](07-otel-verify.md) after the person runs a short session with the selected provider. + +## Output + +The configured provider and destination, or the CLI's refusal. + +## Rules + +- Keep the merged configuration and credentials out of the report. +- Configuration success alone establishes no receipt of telemetry. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md b/plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md new file mode 100644 index 000000000..f7a929489 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md @@ -0,0 +1,24 @@ +# 07 - Verify provider receipt + +Read the CLI's checks of provider configuration and captured telemetry. + +## Input + +The selected provider and destination, carried from initialization or supplied with a verification request. + +## Process + +1. Run `aidd telemetry otel verify --help` and report an unavailable command as requiring an update of `@ai-driven-dev/cli`. +2. Obtain any missing provider or destination choice. +3. Run `aidd telemetry otel verify --config= --endpoint=`. +4. Report the captured nonempty line count and modification time, or the CLI's failure. + - Historical records establish previous receipt, not delivery from the current session. + - If a fresh session has not been run, state that current-session delivery remains unchecked. + +## Output + +The CLI's configuration and receipt result, with its timing limitation. + +## Rules + +- Verification is read-only; configuration changes require a separate initialization request. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/forget.md b/plugins/aidd-telemetry/skills/00-init/actions/forget.md deleted file mode 100644 index edb9697b3..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/forget.md +++ /dev/null @@ -1,29 +0,0 @@ -# Forget - -Delete everything measurement keeps on this machine, after showing it and getting a confirmation. - -## Input - -Nothing. The command covers the whole machine, not one project. - -## Output - -Either the list of what was removed, or the list of what would be, with nothing removed. - -## Process - -1. **Preview.** Run `aidd telemetry forget` with no flag. It removes nothing and lists what it would. - - When it says nothing measured is kept, relay that and stop. -2. **Show.** Give the person the list as the command printed it, with the warnings about repositories it could not reach. -3. **Confirm.** Ask whether to remove all of it. Wait for an explicit yes naming the removal. - - A no, or no answer, ends the action with nothing removed. -4. **Remove.** Run `aidd telemetry forget --yes` only after that yes, and relay what it printed. - - Say that it also ends measurement: nothing is measured again until the person runs `aidd telemetry on`. - -## Test - -| Case | Pass | -| --- | --- | -| Preview | `aidd telemetry forget` ran without `--yes` and the data is still there afterwards | -| The person declines | `aidd telemetry forget --yes` was never run | -| The person confirms | `aidd telemetry forget --yes` ran once, after the preview, and a second preview lists nothing | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/identity.md b/plugins/aidd-telemetry/skills/00-init/actions/identity.md deleted file mode 100644 index 6c8f23bd2..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/identity.md +++ /dev/null @@ -1,27 +0,0 @@ -# Identity - -Let the person choose to be named on their own measurement, or stop being named. - -## Input - -What the person wants: to be named as an identifier they give, to see the current choice, or to be named no more. - -## Output - -The identity set, shown or removed, as the command reported it. - -## Process - -1. **Explain.** Say that an identity is optional, that it only labels the person axis of their own report, and that it is kept in one file on this machine. -2. **Show.** Run `aidd telemetry identity` to read the current choice, and tell the person what it holds. -3. **Name.** To set one, ask the person for the identifier they want and wait for it. Run `aidd telemetry identity ` with exactly that text. - - The command refuses an identifier that is empty, over 128 characters, or spans lines: show its message and ask again. -4. **Unname.** To remove it, confirm that the person wants to be named no more, then run `aidd telemetry identity --off`. - -## Test - -| Case | Pass | -| --- | --- | -| The person gives no identifier | Nothing was set and none was invented | -| An identifier is given | `aidd telemetry identity` afterwards prints that identifier | -| Removal is confirmed | `aidd telemetry identity` afterwards says no identity is set | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/stop.md b/plugins/aidd-telemetry/skills/00-init/actions/stop.md deleted file mode 100644 index af46127ab..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/stop.md +++ /dev/null @@ -1,24 +0,0 @@ -# Stop - -Turn measurement off for the current project and keep what was already measured. - -## Input - -The project directory. - -## Output - -Measurement is off for the project, and the person knows the measured data is still there. - -## Process - -1. **Confirm.** Say what stopping does: new Claude Code or Codex work in this project is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. Wait for a yes. -2. **Turn off.** Run `aidd telemetry off` and relay its message. -3. **Offer the rest.** Say that deleting what was measured is a separate step, and that it needs its own confirmation. - -## Test - -| Case | Pass | -| --- | --- | -| The person does not confirm | No `aidd telemetry off` was run | -| Confirmed | The command's message was relayed and the data was not deleted | diff --git a/scripts/__tests__/cli-opens-no-server.test.js b/scripts/__tests__/cli-opens-no-server.test.js index 7fed0bb24..ffa686cc2 100644 --- a/scripts/__tests__/cli-opens-no-server.test.js +++ b/scripts/__tests__/cli-opens-no-server.test.js @@ -6,12 +6,12 @@ const { describe, it } = require("node:test"); const ROOT = path.resolve(__dirname, "../.."); const CLI_SRC_PREFIX = "cli/src/"; +const COLLECTOR = "cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts"; /** - * Nothing this system runs opens a network listener, and the constraint has to stay true of - * the code rather than of a document. This walks every tracked source file under `cli/src` - * and greps for the literal patterns a listener is built from, so a future listener has to be - * a deliberate decision โ€” this assertion updated, with a reason โ€” never an accident. + * The local OTLP collector is the sole listener in the CLI. This walks every tracked source + * file under `cli/src` so another listener needs an explicit decision, never an accident. + * The collector's integration tests verify its loopback bind and required authentication. * * Deliberately narrow to server construction, not merely importing `node:http` or `node:net`: * this codebase makes plenty of outbound calls, and a client is not what this forbids. @@ -41,26 +41,53 @@ function trackedCliSourceFiles() { .filter((rel) => rel.endsWith(".ts") && fs.existsSync(path.join(ROOT, rel))); } -describe("nothing in cli/src opens a network listener", () => { - it("no source file constructs or binds a server", () => { +function listenersIn(files) { + const offenders = []; + let collectorConstructors = 0; + let collectorBinds = 0; + for (const [rel, text] of files) { + for (const { pattern, label } of FORBIDDEN_PATTERNS) { + const count = [...text.matchAll(new RegExp(pattern.source, "gu"))].length; + if (rel === COLLECTOR && label === "createServer(") collectorConstructors = count; + else if (rel === COLLECTOR && label === ".listen(") collectorBinds = count; + else if (count > 0) offenders.push(`${rel}: ${label}`); + } + } + return { offenders, collectorConstructors, collectorBinds }; +} + +describe("the OTLP collector is the only listener in cli/src", () => { + it("allows one collector server and forbids all other listeners", () => { const files = trackedCliSourceFiles(); // A walk that found nothing would pass this test while checking nothing at all. assert.ok(files.length > 300, `expected the CLI's source tree, found ${files.length} files`); + const { offenders, collectorConstructors, collectorBinds } = listenersIn( + files.map((rel) => [rel, fs.readFileSync(path.join(ROOT, rel), "utf8")]) + ); - const offenders = []; - for (const rel of files) { - const text = fs.readFileSync(path.join(ROOT, rel), "utf8"); - for (const { pattern, label } of FORBIDDEN_PATTERNS) { - if (pattern.test(text)) offenders.push(`${rel}: ${label}`); - } - } - + assert.equal(collectorConstructors, 1, "the collector must construct exactly one server"); + assert.equal(collectorBinds, 1, "the collector must bind exactly one listener"); assert.deepEqual( offenders, [], - `a source file opens or names a network listener - this codebase's one hard ` + - `constraint is that nothing it runs does that:\n ${offenders.join("\n ")}` + `only the local OTLP collector may open a listener:\n ${offenders.join("\n ")}` ); }); + + it("detects a second collector listener or a listener in another source file", () => { + const collector = "createServer(handler); server.listen(1);"; + const duplicate = listenersIn([[COLLECTOR, `${collector} server.listen(2);`]]); + assert.equal(duplicate.collectorConstructors, 1); + assert.equal(duplicate.collectorBinds, 2); + assert.equal(listenersIn([[COLLECTOR, `${collector} createServer(handler);`]]).collectorConstructors, 2); + const other = listenersIn([ + [COLLECTOR, collector], + ["cli/src/cli.ts", "createServer(handler); server.listen(3);"], + ]); + assert.deepEqual(other.offenders, ["cli/src/cli.ts: createServer(", "cli/src/cli.ts: .listen("]); + assert.deepEqual(listenersIn([[COLLECTOR, 'import net from "node:net";']]).offenders, [ + `${COLLECTOR}: import from "node:net"`, + ]); + }); }); diff --git a/scripts/__tests__/telemetry-skills-name-real-commands.test.js b/scripts/__tests__/telemetry-skills-name-real-commands.test.js index b1ae34000..fc23e9cd2 100644 --- a/scripts/__tests__/telemetry-skills-name-real-commands.test.js +++ b/scripts/__tests__/telemetry-skills-name-real-commands.test.js @@ -28,8 +28,8 @@ function telemetryHelp() { assert.ok(group, "the help golden has no `aidd telemetry` entry"); const commands = new Map(); for (const entry of entries) { - const name = entry.invocation.trim().split(/\s+/u)[2]; - if (name === undefined) continue; + const name = entry.invocation.trim().split(/\s+/u).slice(2).join(" "); + if (name === "") continue; const help = entry.help.replace(/\s+/gu, " "); const choices = new Map(); for (const match of help.matchAll(/(--[a-z-]+)(?:(?!--)[^(])*\(choices: ([^)]*)\)/gu)) { @@ -64,12 +64,27 @@ function codeRegions(markdown) { /** Every `aidd telemetry ...` a skill's code names, as `{ file, text, sub, rest }`. */ function mentions() { const found = []; + const commands = telemetryHelp(); for (const file of markdownUnder(SKILLS)) { const relative = path.relative(ROOT, file).split(path.sep).join("/"); for (const region of codeRegions(fs.readFileSync(file, "utf8"))) { for (const match of region.matchAll(COMMAND)) { const words = match[1].trim().split(/\s+/u).filter(Boolean); - found.push({ file: relative, text: `aidd telemetry${match[1]}`.trim(), sub: words[0], rest: words.slice(1) }); + let count = 0; + while (count < words.length && commands.has(words.slice(0, count + 1).join(" "))) { + count++; + } + // A command group cannot accept an unknown child as an argument. + const prefix = `${words.slice(0, count).join(" ")} `; + const hasChildren = [...commands.keys()].some((name) => name.startsWith(prefix)); + if (count > 0 && hasChildren && /^[a-z][a-z-]*$/u.test(words[count] ?? "")) count++; + if (count === 0 && words.length > 0) count = 1; + found.push({ + file: relative, + text: `aidd telemetry${match[1]}`.trim(), + sub: words.slice(0, count).join(" ") || undefined, + rest: words.slice(count), + }); } } } @@ -104,13 +119,23 @@ describe("the telemetry skills name only commands the CLI has", () => { const command = help.get(mention.sub); if (command === undefined) continue; mention.rest.forEach((word, at) => { - if (word.startsWith("--") && !command.flags.has(word)) { - wrong.push(`${mention.file}: \`${mention.text}\` has no ${word}`); + const [flag, inlineValue] = word.split("=", 2); + if (word.startsWith("--") && !command.flags.has(flag)) { + wrong.push(`${mention.file}: \`${mention.text}\` has no ${flag}`); } const allowed = command.choices.get(mention.rest[at - 1]); if (allowed !== undefined && /^[a-z]+$/u.test(word) && !allowed.has(word)) { wrong.push(`${mention.file}: \`${mention.text}\` gives ${mention.rest[at - 1]} the value ${word}`); } + const inlineChoices = command.choices.get(flag); + if ( + inlineChoices !== undefined && + inlineValue !== undefined && + /^[a-z]+$/u.test(inlineValue) && + !inlineChoices.has(inlineValue) + ) { + wrong.push(`${mention.file}: \`${mention.text}\` gives ${flag} the value ${inlineValue}`); + } }); } assert.deepEqual(wrong, []);