From c80215f534cc2dce4028407c6f29bc5682134990 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 09:56:49 +0800 Subject: [PATCH 01/56] feat(telemetry): configure native OTLP exports --- aidd_docs/recipes/observe-agent-telemetry.md | 59 ++++ cli/aidd_docs/memory/codebase-map.md | 1 + cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 2 + .../infrastructure/otel/claude-dev.json | 23 ++ .../infrastructure/otel/codex-dev.json | 16 + .../infrastructure/otel/config-edits.ts | 218 +++++++++++++ .../infrastructure/otel/init-otel.ts | 294 ++++++++++++++++++ .../infrastructure/otel/opencode-dev.json | 7 + cli/src/presentation/commands/telemetry.ts | 52 ++++ cli/src/runtime/wiring/telemetry-otel.ts | 25 ++ .../tool-addition-cost.arch.test.ts | 2 + .../otel/init-otel.integration.test.ts | 158 ++++++++++ cli/tests/e2e/telemetry-otel.e2e.test.ts | 80 +++++ cli/tests/golden/snapshots/help/surface.json | 17 +- .../telemetry-wiring.integration.test.ts | 8 + plugins/aidd-telemetry/CATALOG.md | 3 +- plugins/aidd-telemetry/README.md | 8 +- .../aidd-telemetry/skills/00-init/SKILL.md | 23 +- .../skills/00-init/actions/otel.md | 26 ++ ...elemetry-skills-name-real-commands.test.js | 35 ++- 21 files changed, 1039 insertions(+), 20 deletions(-) create mode 100644 aidd_docs/recipes/observe-agent-telemetry.md create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/claude-dev.json create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/codex-dev.json create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/opencode-dev.json create mode 100644 cli/src/runtime/wiring/telemetry-otel.ts create mode 100644 cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts create mode 100644 cli/tests/e2e/telemetry-otel.e2e.test.ts create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/otel.md diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md new file mode 100644 index 000000000..f50c79229 --- /dev/null +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -0,0 +1,59 @@ +# Observe agent telemetry locally + +Configure native telemetry for Claude Code, Codex and OpenCode against the private development collector, then compare observed fields. + +## Steps to observe agent telemetry locally + +### 1) ๐Ÿ”’ Check the destination + +Telemetry may contain prompts, replies, commands, tool results and secrets. Use it only with the consent of those being observed, protect access to the logs and decide when to delete them. The CLI changes the selected tool's user configuration; this is separate from the per-clone `aidd telemetry on` usage ledger. + +| Destination | Availability | Behavior | +| --- | --- | --- | +| `dev` | Private test machine with the existing loopback collector | Configure the selected tool to export locally | +| `prod` | Reserved for a future Gouvernail collector | Refuse without changing files until its destination is defined | + +The development collector and its private token must already exist. Initialization does not start a collector or set up a production endpoint. + +### 2) โš™๏ธ Initialize each tool + +Run the command for each tool you intend to observe. Existing settings must be preserved; repeat initialization without adding duplicate entries. Follow the CLI's output for version-specific limitations and restarts. + +```sh +aidd telemetry otel init --config=claude --endpoint=dev +aidd telemetry otel init --config=codex --endpoint=dev +aidd telemetry otel init --config=opencode --endpoint=dev +``` + +OpenCode v1 reads the OTLP destination and headers from the process environment. After initialization, launch it through `aidd telemetry otel run --config=opencode` and pass any OpenCode arguments after `--`. OpenCode v2 supports user service environment configuration and requires a service restart. The v1 `experimental.openTelemetry` option enables AI SDK instrumentation; v2.2.5 rejects that option. Native OTLP logs and traces are distinct from the v2 session SSE stream. See the [v1 instrumentation](https://github.com/anomalyco/opencode/blob/v1.19.5/packages/opencode/src/session/llm.ts#L343-L355), [v2 configuration](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/config/normalize.ts#L46) and [v2 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts), accessed 2026-11-04. + +Claude and Codex also support native OTLP logs, metrics and traces; Claude tracing is beta. See [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage) and [Codex observability](https://learn.chatgpt.com/docs/config-file/config-advanced#observability-and-telemetry), accessed 2026-11-04. + +### 3) ๐Ÿงช Verify receipt + +Run a short test session and a non-destructive tool, then check that the relevant local file grows. OpenCode session SSE and native OTLP are separate captures; the SSE capture depends on a running subscriber. + +```sh +for file in "$HOME/.local/share/aidd-otel-observation/claude.jsonl" \ + "$HOME/.local/share/aidd-otel-observation/codex.jsonl" \ + "$HOME/.local/share/aidd-otel-observation/opencode.jsonl" \ + "$HOME/.local/share/aidd-otel-observation/opencode/events.jsonl"; do + if [ -f "$file" ]; then wc -l "$file"; fi +done +``` + +A configured exporter does not prove delivery. Existing sessions may need restarting. OpenCode SSE misses events during disconnection and does not cover another server or `--standalone` session. + +### 4) ๐Ÿ“Š Compare observed fields + +| Field | Claude Code | Codex | OpenCode v2 SSE | +| --- | --- | --- | --- | +| Session | `session.id` | `conversation.id` | `data.sessionID` | +| Model | `model` | `model` | `data.model.id` | +| Input / output tokens | `input_tokens` / `output_tokens` | `input_token_count` / `output_token_count` | `data.tokens.input` / `data.tokens.output` | +| Cache reads | `cache_read_tokens` | `cached_token_count` | `data.tokens.cache.read` | +| Tool activity | `tool_result` | `codex.tool_result` | `session.tool.input.started`, `session.tool.success` | + +These fields were observed in local samples, not guaranteed in every session. Align token definitions before comparing them: for example, Claude input excludes cache reads and writes. OpenCode v2 can turn unavailable token values into zero, so zero is not proof of no usage. See [Claude token counters](https://code.claude.com/docs/en/monitoring-usage) and [OpenCode usage normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts), accessed 2026-11-04. + +Files grow without automatic rotation. Configuration and receipt do not establish a Gouvernail integration; forwarding, normalization and retention remain future work. diff --git a/cli/aidd_docs/memory/codebase-map.md b/cli/aidd_docs/memory/codebase-map.md index 86da7d334..edf71de9c 100644 --- a/cli/aidd_docs/memory/codebase-map.md +++ b/cli/aidd_docs/memory/codebase-map.md @@ -71,6 +71,7 @@ src/ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ report/ # attribution of a call, the days of a report, the axes it splits by โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ switch/ # the previous version's config block, Claude's retention โ”‚ โ”‚ โ””โ”€โ”€ infrastructure/ +โ”‚ โ”‚ โ”œโ”€โ”€ otel/ # explicit user configuration of native OTLP exporters โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # reading a clone's identity from the file system, and the consent history file โ”‚ โ”‚ โ”œโ”€โ”€ declaration/ # where a declaration is kept: git config and the telemetry dir โ”‚ โ”‚ โ”œโ”€โ”€ forget/ # erasing the telemetry dir, the previous version's files, git config keys diff --git a/cli/package.json b/cli/package.json index 50bb7d104..bf99cbd76 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 712, + "bundleBudgetKB": 728, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 13322a0d0..ce560d9af 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -49,6 +49,8 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 712 KB: 710.8 KB, a Stop block continuing the turn on OpenCode V1 and Kilo, and V2's // tool answers reaching the model through its tool hook (+1.8 KB over the 709 KB ceiling); // 0.2% measured headroom. +// 728 KB: 726.7 KB, explicit native OTLP configuration for three tools and the +// OpenCode 1 launcher (+15.9 KB over the 710.8 KB build above); 0.2% measured headroom. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/claude-dev.json b/cli/src/contexts/telemetry/infrastructure/otel/claude-dev.json new file mode 100644 index 000000000..a8ed9afa0 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/claude-dev.json @@ -0,0 +1,23 @@ +{ + "env": { + "CLAUDE_CODE_ENABLE_TELEMETRY": "1", + "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1", + "OTEL_METRICS_EXPORTER": "otlp", + "OTEL_LOGS_EXPORTER": "otlp", + "OTEL_TRACES_EXPORTER": "otlp", + "OTEL_EXPORTER_OTLP_PROTOCOL": "http/json", + "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT": "{collector}/claude/v1/metrics", + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "{collector}/claude/v1/logs", + "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "{collector}/claude/v1/traces", + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}" + }, + "defaults": { + "OTEL_LOG_USER_PROMPTS": "1", + "OTEL_LOG_ASSISTANT_RESPONSES": "1", + "OTEL_LOG_TOOL_DETAILS": "1", + "OTEL_LOG_TOOL_CONTENT": "1", + "OTEL_LOG_RAW_API_BODIES": "1", + "CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH": "262144", + "OTEL_METRICS_INCLUDE_REPOSITORY": "true" + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/codex-dev.json b/cli/src/contexts/telemetry/infrastructure/otel/codex-dev.json new file mode 100644 index 000000000..b238aab01 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/codex-dev.json @@ -0,0 +1,16 @@ +{ + "defaults": { + "environment": "local", + "log_user_prompt": true, + "log_agent_responses": true, + "log_guardian_assessments": true + }, + "toolResult": { + "max_bytes": 262144 + }, + "exporters": { + "exporter": "{collector}/codex/v1/logs", + "metrics_exporter": "{collector}/codex/v1/metrics", + "trace_exporter": "{collector}/codex/v1/traces" + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts b/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts new file mode 100644 index 000000000..eabd31d9a --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts @@ -0,0 +1,218 @@ +import { isDeepStrictEqual } from "node:util"; +import { parse, stringify } from "smol-toml"; +import { asPlainObject } from "../../../../kernel/reading/plain-object.js"; + +type Settings = Record; + +function tomlTable(values: Settings = {}): Settings { + return Object.setPrototypeOf(values, null); +} + +function requireObject(value: unknown): Settings { + const object = asPlainObject(value); + if (!object) throw new Error("Expected a configuration object."); + return object; +} + +export function rejectConflict(name: string, actual: unknown, expected: string): void { + if (actual !== undefined && actual !== expected) { + throw new Error(`Conflicting OTLP setting: ${name}. Resolve it before initializing.`); + } +} + +/** Keep offsets stable while hiding JSONC comments and trailing commas from JSON.parse. */ +function normalizeJsonc(content: string): string { + return content + .replace(/"(?:[^"\\]|\\.)*"|\/\/[^\r\n]*|\/\*[\s\S]*?\*\//g, (token) => + token.startsWith("/") ? token.replace(/[^\r\n]/g, " ") : token + ) + .replace(/"(?:[^"\\]|\\.)*"|,(?=\s*[}\]])/g, (token) => (token === "," ? " " : token)); +} + +function readJsonSettings(content: string): Settings { + try { + return requireObject(JSON.parse(content)); + } catch { + throw new Error("Invalid JSON settings."); + } +} + +/** Locate a top-level section and refuse duplicate properties where we are about to edit. */ +function sectionOffset(content: string, section: string): number { + let depth = 0; + let offset: number | undefined; + let inSection = false; + const keys = new Set(); + for (const match of content.matchAll(/"(?:[^"\\]|\\.)*"|[{}[\]]/gs)) { + const token = match[0]; + if (token === "{" || token === "[") { + depth++; + } else if (token === "}" || token === "]") { + depth--; + if (depth === 1) inSection = false; + } else { + const tail = /^\s*:\s*/.exec(content.slice(match.index + token.length)); + if (!tail) continue; + const key: string = JSON.parse(token); + if (depth === 1 && key === section) { + if (offset !== undefined) throw new Error(`Duplicate ${section} in JSON settings.`); + offset = match.index + token.length + tail[0].length; + inSection = true; + } else if (depth === 2 && inSection) { + if (keys.has(key)) throw new Error(`Duplicate entry in ${section} settings.`); + keys.add(key); + } + } + } + return offset ?? content.indexOf("{"); +} + +function mergeJsonSection( + content: string, + section: string, + entries: (existing: Settings) => Settings, + normalize: (source: string) => string = (source) => source +): string { + const normalized = normalize(content); + const settings = readJsonSettings(normalized); + const original = settings[section] === undefined ? settings : requireObject(settings[section]); + const existing = settings[section] === undefined ? {} : original; + const offset = sectionOffset(normalized, section) + 1; + const missing = entries(existing); + if (Object.keys(missing).length === 0) return content; + const patch = settings[section] === undefined ? { [section]: missing } : missing; + const indent = settings[section] === undefined ? "" : " "; + const entry = JSON.stringify(patch, null, 2).slice(1, -2).replaceAll("\n", `\n${indent}`); + const separator = Object.keys(original).length === 0 ? "\n" : ","; + const next = content.slice(0, offset) + entry + separator + content.slice(offset); + readJsonSettings(normalize(next)); + return next; +} + +export function mergeJsonEnvironment( + content: string, + additions: Readonly>, + verify: (env: Settings) => void, + defaults: Readonly> = {} +): string { + return mergeJsonSection(content, "env", (env) => { + verify(env); + for (const [key, value] of Object.entries(additions)) rejectConflict(key, env[key], value); + return Object.fromEntries( + Object.entries({ ...defaults, ...additions }).filter(([key]) => env[key] === undefined) + ); + }); +} + +export function openCodeInstrumentation(content: string): unknown { + const settings = readJsonSettings(normalizeJsonc(content)); + return settings.experimental === undefined + ? undefined + : requireObject(settings.experimental).openTelemetry; +} + +export function mergeOpenCodeInstrumentation(content: string, enabled: boolean): string { + return mergeJsonSection( + content, + "experimental", + (experimental) => { + if (experimental.openTelemetry === enabled) return {}; + if (experimental.openTelemetry !== undefined) { + throw new Error("OpenCode experimental.openTelemetry is disabled or invalid."); + } + return { openTelemetry: enabled }; + }, + normalizeJsonc + ); +} + +function readToml(content: string): Settings { + try { + return parse(content); + } catch { + throw new Error("Invalid TOML settings."); + } +} + +function matchesToml(candidate: string, expected: Settings): boolean { + try { + return isDeepStrictEqual(parse(candidate), expected); + } catch { + // An insertion candidate can redefine a table. Try the next candidate. + return false; + } +} + +function addTomlTable(content: string, table: string, body: string, expected: Settings): string { + const header = `[${table}]`; + const candidates = [`${content ? `${content}\n` : ""}${header}\n${body}`]; + for (const match of content.matchAll(/^\s*\[([^\]\n]+)\][ \t]*(?:#[^\n]*)?(?:\r?\n|$)/gm)) { + const path = match[1].split(".").map((part) => part.trim().replace(/^["']|["']$/g, "")); + if (path.join(".") !== table) continue; + const offset = match.index + match[0].length; + const newline = match[0].endsWith("\n") ? "" : "\n"; + candidates.push(content.slice(0, offset) + newline + body + content.slice(offset)); + } + for (const candidate of candidates) { + if (matchesToml(candidate, expected)) return candidate; + } + throw new Error(`Cannot merge ${table} while preserving TOML settings. Use explicit tables.`); +} + +export function mergeCodexExporters( + content: string, + exporters: Readonly>, + token: string, + defaults: Readonly>, + toolResult: Readonly> +): string { + const settings = readToml(content); + const otel = settings.otel === undefined ? tomlTable() : requireObject(settings.otel); + settings.otel = otel; + const missing = Object.fromEntries( + Object.entries(defaults).filter(([key]) => otel[key] === undefined) + ); + Object.assign(otel, missing); + let next = Object.keys(missing).length + ? addTomlTable(content, "otel", stringify(missing), settings) + : content; + const result = otel.tool_result === undefined ? tomlTable() : requireObject(otel.tool_result); + const resultDefaults = Object.fromEntries( + Object.entries(toolResult).filter(([key]) => result[key] === undefined) + ); + if (Object.keys(resultDefaults).length) { + otel.tool_result = Object.assign(result, resultDefaults); + next = addTomlTable(next, "otel.tool_result", stringify(resultDefaults), settings); + } + for (const [signal, endpoint] of Object.entries(exporters)) { + const exporter = otel[signal] === undefined ? {} : requireObject(otel[signal]); + if (Object.keys(exporter).some((name) => name !== "otlp-http")) { + throw new Error(`Conflicting OTLP exporter: otel.${signal}. Resolve it before initializing.`); + } + const http = exporter["otlp-http"] === undefined ? {} : requireObject(exporter["otlp-http"]); + const fields: Record = {}; + for (const [key, value] of Object.entries({ endpoint, protocol: "json" })) { + rejectConflict(`otel.${signal}.${key}`, http[key], value); + if (http[key] === undefined) fields[key] = value; + } + let body = stringify(fields); + if (http.headers === undefined) { + fields.headers = Object.setPrototypeOf({ authorization: `Bearer ${token}` }, null); + body += `headers = { authorization = ${JSON.stringify(`Bearer ${token}`)} }\n`; + } else { + const headers = requireObject(http.headers); + const auth = Object.entries(headers).filter( + ([name]) => name.toLowerCase() === "authorization" + ); + if (auth.length !== 1 || auth[0][1] !== `Bearer ${token}`) { + throw new Error( + `Conflicting OTLP authorization: otel.${signal}. Resolve it before initializing.` + ); + } + } + if (Object.keys(fields).length === 0) continue; + otel[signal] = tomlTable({ "otlp-http": tomlTable({ ...http, ...fields }) }); + next = addTomlTable(next, `otel.${signal}.otlp-http`, body, settings); + } + return next; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts b/cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts new file mode 100644 index 000000000..522f4de88 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts @@ -0,0 +1,294 @@ +import { execFile, spawn } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import { lstat, mkdir, open, readFile, rename, rm, writeFile } from "node:fs/promises"; +import { constants as osConstants } from "node:os"; +import { dirname, join } from "node:path"; +import { promisify } from "node:util"; +import { claudeConfigDir } from "../../domain/claude-projects-root.js"; +import claudeTemplate from "./claude-dev.json" with { type: "json" }; +import codexTemplate from "./codex-dev.json" with { type: "json" }; +import { + mergeCodexExporters, + mergeJsonEnvironment, + mergeOpenCodeInstrumentation, + openCodeInstrumentation, + rejectConflict, +} from "./config-edits.js"; +import opencodeTemplate from "./opencode-dev.json" with { type: "json" }; + +export const OPENCODE_CONFIG_ID = "opencode"; +export const OTEL_CONFIGS = ["claude", "codex", OPENCODE_CONFIG_ID] as const; +export const OTEL_ENDPOINTS = ["dev", "prod"] as const; +type Config = (typeof OTEL_CONFIGS)[number]; +type Endpoint = (typeof OTEL_ENDPOINTS)[number]; + +interface Host { + readonly home: string; + readonly env: NodeJS.ProcessEnv; + readonly opencodeVersion?: () => Promise; +} + +interface Edit { + readonly path: string; + readonly initial: string; + readonly update: (content: string) => string; + readonly launch: string; + readonly containsToken?: false; +} + +const COLLECTOR = "http://127.0.0.1:14318"; +const TOKEN_PATH = ".local/share/aidd-otel-observation/collector-token"; +const SIGNALS = ["LOGS", "METRICS", "TRACES"] as const; + +function interpolate(template: Readonly>, token: string) { + return Object.fromEntries( + Object.entries(template).map(([key, value]) => [ + key, + value.replace("{collector}", COLLECTOR).replace("{token}", token), + ]) + ); +} + +async function readToken(home: string): Promise { + try { + const handle = await open(join(home, TOKEN_PATH), constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const info = await handle.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) throw new Error(); + const token = (await handle.readFile("utf8")).trim(); + if (!/^[a-zA-Z0-9_-]+$/.test(token)) throw new Error(); + return token; + } finally { + await handle.close(); + } + } catch { + throw new Error(`A valid private collector token is required at ~/${TOKEN_PATH} (mode 0600).`); + } +} + +function claudeEdit(host: Host, token: string): Edit { + const additions = interpolate(claudeTemplate.env, token); + return { + path: join(claudeConfigDir(host.env.CLAUDE_CONFIG_DIR, host.home), "settings.json"), + initial: "{}\n", + launch: "Restart Claude Code to apply the configuration.", + update: (content) => + mergeJsonEnvironment( + content, + additions, + (env) => { + rejectConflict( + "OTEL_EXPORTER_OTLP_ENDPOINT", + env.OTEL_EXPORTER_OTLP_ENDPOINT, + `${COLLECTOR}/claude` + ); + for (const signal of SIGNALS) { + for (const key of ["HEADERS", "PROTOCOL"]) { + const name = `OTEL_EXPORTER_OTLP_${signal}_${key}`; + rejectConflict(name, env[name], additions[`OTEL_EXPORTER_OTLP_${key}`]); + } + } + }, + claudeTemplate.defaults + ), + }; +} + +async function openCodeVersion(host: Host): Promise<"1" | "2"> { + let output: string; + try { + output = host.opencodeVersion + ? await host.opencodeVersion() + : ( + await promisify(execFile)(OPENCODE_CONFIG_ID, ["--version"], { + env: host.env, + timeout: 5000, + }) + ).stdout; + } catch { + throw new Error("Cannot determine OpenCode version. Make opencode available on PATH."); + } + const version = /^\s*(?:opencode\s+)?v?([12])\.(\d+)\.(\d+)\s*$/i.exec(output); + if (version && (version[1] === "1" || version[1] === "2")) { + const minimumMinor = version[1] === "1" ? 19 : 2; + const minor = Number(version[2]); + if (minor > minimumMinor || (minor === minimumMinor && Number(version[3]) >= 5)) { + return version[1]; + } + } + throw new Error( + "Unsupported OpenCode version. Use an OpenCode 1 release from 1.19.5 or 2 from 2.2.5." + ); +} + +function openCodeGlobalDir(host: Host): string { + return join(host.env.XDG_CONFIG_HOME || join(host.home, ".config"), OPENCODE_CONFIG_ID); +} + +async function openCodeV1Edit(host: Host): Promise { + const global = openCodeGlobalDir(host); + const custom = host.env.OPENCODE_CONFIG_DIR; + const initial = "{}\n"; + const nativeFiles = ["opencode.json", "opencode.jsonc"]; + const files = ["config.json", ...nativeFiles].map((name) => join(global, name)); + if (host.env.OPENCODE_CONFIG) files.push(host.env.OPENCODE_CONFIG); + if (custom && custom !== global) files.push(...nativeFiles.map((name) => join(custom, name))); + let path: string | undefined; + let flag: unknown; + for (const file of new Set(files)) { + const existing = await readSettings(file, initial); + if (!existing.exists) continue; + path = file; + const value = openCodeInstrumentation(existing.content); + if (value !== undefined) flag = value; + } + if (host.env.OPENCODE_CONFIG_CONTENT) { + const inline = openCodeInstrumentation(host.env.OPENCODE_CONFIG_CONTENT); + if (inline !== undefined) flag = inline; + } + if (flag !== undefined && flag !== true) { + throw new Error("OpenCode experimental.openTelemetry is disabled or invalid."); + } + return { + path: path ?? join(custom || global, nativeFiles[0]), + initial, + launch: + "Run OpenCode 1 with aidd telemetry otel run --config=opencode -- [OpenCode arguments].", + containsToken: false, + update: (content) => + flag === true && openCodeInstrumentation(content) === undefined + ? content + : mergeOpenCodeInstrumentation(content, opencodeTemplate.experimental.openTelemetry), + }; +} + +async function openCodeEdit(host: Host, token: string): Promise { + if ((await openCodeVersion(host)) === "1") return openCodeV1Edit(host); + const directory = host.env.OPENCODE_CONFIG_DIR || openCodeGlobalDir(host); + const additions = interpolate(opencodeTemplate.env, token); + return { + path: join(directory, "service.json"), + initial: "{}\n", + launch: "Restart the OpenCode background service with opencode service restart.", + update: (content) => mergeJsonEnvironment(content, additions, () => {}), + }; +} + +export async function runOtelOpenCode(args: readonly string[], host: Host): Promise { + if ((await openCodeVersion(host)) !== "1") { + throw new Error("OTLP launch is for OpenCode 1. Use the background service with OpenCode 2."); + } + const edit = await openCodeV1Edit(host); + const current = await readSettings(edit.path, edit.initial); + if (edit.update(current.content) !== current.content) { + throw new Error( + "Initialize OpenCode 1 OTLP with aidd telemetry otel init --config=opencode --endpoint=dev." + ); + } + const token = await readToken(host.home); + const additions = interpolate(opencodeTemplate.env, token); + for (const [key, value] of Object.entries(additions)) + rejectConflict(key, host.env[key] || undefined, value); + const child = spawn(OPENCODE_CONFIG_ID, args, { + env: { ...host.env, ...additions }, + stdio: "inherit", + }); + const interrupt = () => child.kill("SIGINT"); + const terminate = () => child.kill("SIGTERM"); + process.on("SIGINT", interrupt); + process.on("SIGTERM", terminate); + try { + return await new Promise((resolve, reject) => { + child.once("error", () => reject(new Error("Could not launch OpenCode."))); + child.once("exit", (code, signal) => + resolve(code ?? (signal ? 128 + osConstants.signals[signal] : 1)) + ); + }); + } finally { + process.off("SIGINT", interrupt); + process.off("SIGTERM", terminate); + } +} + +async function readSettings(path: string, initial: string) { + const info = await lstat(path).catch((error) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!info) return { content: initial, private: false, exists: false }; + if (!info.isFile()) throw new Error("OTLP configuration must be a regular file."); + return { + content: await readFile(path, "utf8"), + private: (info.mode & 0o077) === 0, + exists: true, + }; +} + +async function applyEdit(edit: Edit): Promise { + const before = await readSettings(edit.path, edit.initial); + const next = edit.update(before.content); + if (next === before.content && (before.private || edit.containsToken === false)) return false; + + await mkdir(dirname(edit.path), { recursive: true, mode: 0o700 }); + const lockPath = `${edit.path}.aidd-otel.lock`; + const lock = await open(lockPath, "wx", 0o600); + try { + const current = await readSettings(edit.path, edit.initial); + if ( + current.content !== before.content || + current.private !== before.private || + current.exists !== before.exists + ) { + throw new Error("OTLP settings changed during initialization. Retry with the updated file."); + } + const temporary = `${edit.path}.${randomUUID()}.tmp`; + try { + await writeFile(temporary, next, { flag: "wx", mode: 0o600 }); + await rename(temporary, edit.path); + } finally { + await rm(temporary, { force: true }); + } + } finally { + await lock.close(); + await rm(lockPath, { force: true }); + } + return true; +} + +export async function initializeOtel(config: Config, endpoint: Endpoint, host: Host) { + if (endpoint === "prod") { + throw new Error( + "OTLP production configuration is unavailable until the Gouvernail endpoint is published." + ); + } + if (process.platform === "win32") { + throw new Error("OTLP initialization requires private POSIX configuration files."); + } + const token = await readToken(host.home); + let edit: Edit; + switch (config) { + case "claude": + edit = claudeEdit(host, token); + break; + case "codex": + edit = { + path: join(host.env.CODEX_HOME || join(host.home, ".codex"), "config.toml"), + initial: "", + launch: "Restart Codex to apply the configuration.", + update: (content) => + mergeCodexExporters( + content, + interpolate(codexTemplate.exporters, token), + token, + codexTemplate.defaults, + codexTemplate.toolResult + ), + }; + break; + case OPENCODE_CONFIG_ID: + edit = await openCodeEdit(host, token); + break; + } + return { path: edit.path, launch: edit.launch, changed: await applyEdit(edit) }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/opencode-dev.json b/cli/src/contexts/telemetry/infrastructure/otel/opencode-dev.json new file mode 100644 index 000000000..1ab4fd21f --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/opencode-dev.json @@ -0,0 +1,7 @@ +{ + "env": { + "OTEL_EXPORTER_OTLP_ENDPOINT": "{collector}/opencode", + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}" + }, + "experimental": { "openTelemetry": true } +} diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index f89bff892..5d5af2b80 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -27,6 +27,7 @@ import { USAGE_AGENTS, USAGE_TOOLS } from "../../contexts/telemetry/domain/usage import type { Prompter } from "../../kernel/ports/prompter.js"; import { createDeps } from "../../runtime/wiring/framework.js"; import type { TelemetryDeps } from "../../runtime/wiring/telemetry.js"; +import { configureOtel, launchOtel, otelOptions } from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, printIdentityRule, @@ -128,6 +129,57 @@ export function registerTelemetryCommand(program: Command): void { const telemetry = program .command("telemetry") .description("Measure what Claude Code and Codex sessions consume, locally"); + const otel = telemetry.command("otel").description("Configure native OpenTelemetry exporters"); + otel + .command("init") + .description("Enable detailed local OTLP export in one tool's user configuration") + .addOption( + new Option("--config ", "User configuration to merge") + .choices(otelOptions.configs) + .makeOptionMandatory() + ) + .addOption( + new Option("--endpoint ", "Collector destination") + .choices(otelOptions.endpoints) + .makeOptionMandatory() + ) + .action( + async (options: { + config: (typeof otelOptions.configs)[number]; + endpoint: (typeof otelOptions.endpoints)[number]; + }) => { + const { output } = parseGlobalOptions(program); + try { + const result = await configureOtel(options.config, options.endpoint); + output.info( + result.changed ? `Updated ${result.path}` : `Already configured in ${result.path}` + ); + output.info( + "Local OTLP capture may include prompts, responses, commands and tool output." + ); + output.info(result.launch); + } catch (error) { + new ErrorHandler(output).handle(error); + } + } + ); + + otel + .command("run [args...]") + .description("Launch OpenCode 1 with local OTLP export. Pass tool arguments after --") + .addOption( + new Option("--config ", "Tool to launch") + .choices(otelOptions.launchConfigs) + .makeOptionMandatory() + ) + .action(async (args: string[]) => { + const { output } = parseGlobalOptions(program); + try { + process.exitCode = await launchOtel(args); + } catch (error) { + new ErrorHandler(output).handle(error); + } + }); telemetry .command("on") diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts new file mode 100644 index 000000000..d05ca37a1 --- /dev/null +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -0,0 +1,25 @@ +import { homedir } from "node:os"; +import { + initializeOtel, + OPENCODE_CONFIG_ID, + OTEL_CONFIGS, + OTEL_ENDPOINTS, + runOtelOpenCode, +} from "../../contexts/telemetry/infrastructure/otel/init-otel.js"; + +export const otelOptions = { + configs: OTEL_CONFIGS, + endpoints: OTEL_ENDPOINTS, + launchConfigs: [OPENCODE_CONFIG_ID], +}; + +export function configureOtel( + config: (typeof OTEL_CONFIGS)[number], + endpoint: (typeof OTEL_ENDPOINTS)[number] +) { + return initializeOtel(config, endpoint, { home: homedir(), env: process.env }); +} + +export function launchOtel(args: readonly string[]) { + return runOtelOpenCode(args, { home: homedir(), env: process.env }); +} diff --git a/cli/tests/architecture/tool-addition-cost.arch.test.ts b/cli/tests/architecture/tool-addition-cost.arch.test.ts index e14ca8aa1..85a04bf47 100644 --- a/cli/tests/architecture/tool-addition-cost.arch.test.ts +++ b/cli/tests/architecture/tool-addition-cost.arch.test.ts @@ -92,6 +92,8 @@ const BASELINE: readonly { readonly path: string; readonly named: number }[] = [ }, // Claude Code's own settings files, read to say how long it keeps a transcript. { path: "src/contexts/telemetry/infrastructure/switch/claude-settings-adapter.ts", named: 1 }, + // OTLP is an explicit, separate support surface: three observed native exporters. + { path: "src/contexts/telemetry/infrastructure/otel/init-otel.ts", named: 3 }, // Cursor's project hooks file, named after the tool whose file it is: the directory it // writes into is Cursor's own, not a list a sixth tool joins. { path: "src/contexts/tools/domain/formats/cursor-hooks-project-merge.ts", named: 1 }, diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts new file mode 100644 index 000000000..e2ea1b4c7 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -0,0 +1,158 @@ +import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/init-otel.js"; + +const TOKEN = "example-private-token"; +let home: string; + +async function seed(path: string, content: string): Promise { + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, content, { mode: 0o600 }); +} + +beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-")); +}); +afterEach(async () => { + await rm(home, { recursive: true, force: true }); +}); + +function seedToken() { + return seed(join(home, ".local/share/aidd-otel-observation/collector-token"), TOKEN); +} + +describe.skipIf(process.platform === "win32")("OTLP initialization", () => { + it("refuses production before even needing the local token or executable", async () => { + await expect(initializeOtel("opencode", "prod", { home, env: {} })).rejects.toThrow( + /production.*Gouvernail/ + ); + await expect(stat(join(home, ".config"))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("requires the collector token and leaves the target untouched", async () => { + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow( + /collector-token/ + ); + await expect(stat(join(home, ".claude"))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("merges Claude settings, preserves content options and writes a private file once", async () => { + await seedToken(); + const path = join(home, ".claude/settings.json"); + await seed( + path, + '{"env":{"OTEL_LOG_USER_PROMPTS":"0","CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH":"16000"},"permissions":{"allow":[]}}' + ); + const result = await initializeOtel("claude", "dev", { home, env: {} }); + expect(result.changed).toBe(true); + expect((await stat(path)).mode & 0o777).toBe(0o600); + const content = await readFile(path, "utf8"); + expect(JSON.parse(content)).toMatchObject({ + permissions: { allow: [] }, + env: { + OTEL_LOG_USER_PROMPTS: "0", + OTEL_LOG_RAW_API_BODIES: "1", + CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH: "16000", + OTEL_METRICS_INCLUDE_REPOSITORY: "true", + OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: "http://127.0.0.1:14318/claude/v1/logs", + OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${TOKEN}`, + }, + }); + expect(await initializeOtel("claude", "dev", { home, env: {} })).toMatchObject({ + changed: false, + }); + expect(await readFile(path, "utf8")).toBe(content); + }); + + it("refuses invalid config or a conflicting exporter without altering the file", async () => { + await seedToken(); + const path = join(home, ".claude/settings.json"); + for (const content of [ + '{"env":', + '{"env":{"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT":"https://collector.example/logs"}}', + ]) { + await seed(path, content); + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow(); + expect(await readFile(path, "utf8")).toBe(content); + } + }); + + it("adds Codex exporters without changing TOML comments or content flags", async () => { + await seedToken(); + const dir = join(home, "codex-custom"); + const path = join(dir, "config.toml"); + const original = + "# User config\n[otel]\nlog_user_prompt = false # Keep this\n\n[otel.tool_result]\nmax_bytes = 1234\n"; + await seed(path, original); + const options = { home, env: { CODEX_HOME: dir } }; + expect(await initializeOtel("codex", "dev", options)).toMatchObject({ changed: true }); + const updated = await readFile(path, "utf8"); + expect(updated).toContain("# User config\n[otel]\n"); + expect(updated).toContain( + "log_user_prompt = false # Keep this\n\n[otel.tool_result]\nmax_bytes = 1234\n" + ); + expect(updated).toContain("log_agent_responses = true"); + expect(updated).toContain('endpoint = "http://127.0.0.1:14318/codex/v1/traces"'); + expect(await initializeOtel("codex", "dev", options)).toMatchObject({ changed: false }); + expect(await readFile(path, "utf8")).toBe(updated); + }); + + it("merges v2 daemon environment and refuses an unknown major", async () => { + await seedToken(); + const dir = join(home, "opencode-custom"); + const path = join(dir, "service.json"); + await seed(path, '{"port":1234,"env":{"EXISTING":"keep"}}'); + const options = { home, env: { OPENCODE_CONFIG_DIR: dir } }; + await expect( + initializeOtel("opencode", "dev", { ...options, opencodeVersion: async () => "3.0.0" }) + ).rejects.toThrow(/OpenCode version/); + expect(await readFile(path, "utf8")).toBe('{"port":1234,"env":{"EXISTING":"keep"}}'); + await expect( + initializeOtel("opencode", "dev", { ...options, opencodeVersion: async () => "2.0.0" }) + ).rejects.toThrow(/OpenCode version/); + expect(await readFile(path, "utf8")).toBe('{"port":1234,"env":{"EXISTING":"keep"}}'); + const v2 = { ...options, opencodeVersion: async () => "opencode v2.2.5" }; + expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: true }); + expect(JSON.parse(await readFile(path, "utf8"))).toMatchObject({ + port: 1234, + env: { EXISTING: "keep", OTEL_EXPORTER_OTLP_ENDPOINT: "http://127.0.0.1:14318/opencode" }, + }); + expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: false }); + }); + + it("merges the v1 instrumentation flag into the native JSONC config without touching shell profiles", async () => { + await seedToken(); + const profile = join(home, ".zshrc"); + await seed(profile, "# keep me"); + const path = join(home, ".config/opencode/opencode.jsonc"); + const original = + '{\n // keep this comment\n "experimental": { "todo": true, /* keep this too */ },\n}\n'; + await seed(path, original); + const options = { home, env: {}, opencodeVersion: async () => "1.19.5" }; + const result = await initializeOtel("opencode", "dev", options); + expect(result).toMatchObject({ path, changed: true }); + expect(result.launch).toContain("aidd telemetry otel run --config=opencode"); + const next = await readFile(path, "utf8"); + expect(next).toContain('"openTelemetry": true'); + expect(next).toContain("// keep this comment"); + expect(next).toContain('"todo": true, /* keep this too */'); + expect(next).not.toContain(TOKEN); + expect(await readFile(profile, "utf8")).toBe("# keep me"); + expect(await initializeOtel("opencode", "dev", options)).toMatchObject({ changed: false }); + expect(await readFile(path, "utf8")).toBe(next); + }); + + it("preserves a disabled v1 instrumentation flag and refuses to initialize", async () => { + await seedToken(); + const path = join(home, ".config/opencode/opencode.json"); + const original = '{"experimental":{"openTelemetry":false}}'; + await seed(path, original); + const options = { home, env: {}, opencodeVersion: async () => "1.19.5" }; + await expect(initializeOtel("opencode", "dev", options)).rejects.toThrow( + /openTelemetry.*disabled/ + ); + expect(await readFile(path, "utf8")).toBe(original); + }); +}); diff --git a/cli/tests/e2e/telemetry-otel.e2e.test.ts b/cli/tests/e2e/telemetry-otel.e2e.test.ts new file mode 100644 index 000000000..2374dfe47 --- /dev/null +++ b/cli/tests/e2e/telemetry-otel.e2e.test.ts @@ -0,0 +1,80 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { mkdir, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { cliPath, createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; + +describe.skipIf(process.platform === "win32")("native OTLP launch", () => { + it.each([ + ["SIGINT", 42], + ["SIGTERM", 43], + ] as const)( + "forwards %s sent to AIDD and preserves OpenCode's exit code", + async (signal, code) => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("telemetry-otel"); + const tools = join(fakeHome, "bin"); + const collector = join(fakeHome, ".local/share/aidd-otel-observation"); + const opts = sandboxedEnv(fakeHome); + const env = { + PATH: [tools, opts.PATH].join(delimiter), + OPENCODE_CONFIG: "", + OPENCODE_CONFIG_DIR: "", + OPENCODE_CONFIG_CONTENT: "", + OTEL_EXPORTER_OTLP_ENDPOINT: "", + OTEL_EXPORTER_OTLP_HEADERS: "", + }; + try { + await mkdir(tools); + await mkdir(collector, { recursive: true }); + await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); + await writeFile( + join(tools, "opencode"), + `#!${process.execPath} +if (process.argv.includes("--version")) { + console.log("1.19.5"); + process.exit(0); +} +if (JSON.stringify(process.argv.slice(2)) !== '["stats","--days","1"]') process.exit(20); +if (process.env.OTEL_EXPORTER_OTLP_ENDPOINT !== "http://127.0.0.1:14318/opencode") process.exit(21); +if (process.env.OTEL_EXPORTER_OTLP_HEADERS !== "Authorization=Bearer test-token") process.exit(22); +process.on("SIGINT", () => process.exit(42)); +process.on("SIGTERM", () => process.exit(43)); +console.log(process.pid); +setTimeout(() => process.exit(24), 2000); +`, + { mode: 0o700 } + ); + const init = await runCli( + ["telemetry", "otel", "init", "--config=opencode", "--endpoint=dev"], + projectDir, + fakeHome, + { env } + ); + expect(init.exitCode).toBe(0); + const wrapper = spawn( + process.execPath, + [ + cliPath(), + "telemetry", + "otel", + "run", + "--config=opencode", + "--", + "stats", + "--days", + "1", + ], + { cwd: projectDir, env: sandboxedEnv(fakeHome, env), stdio: ["ignore", "pipe", "pipe"] } + ); + const finished = once(wrapper, "exit"); + const ready = await Promise.race([once(wrapper.stdout, "data"), finished]); + expect(Number(String(ready[0]))).toBeGreaterThan(0); + expect(wrapper.kill(signal)).toBe(true); + expect(await finished).toEqual([code, null]); + } finally { + await cleanup(); + } + } + ); +}); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 449726e32..4ee59eaf9 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -137,7 +137,7 @@ { "invocation": "aidd telemetry", "exitCode": 0, - "help": "Usage: aidd telemetry [options] [command]\n\nMeasure what Claude Code and Codex sessions consume, locally\n\nOptions:\n -h, --help display help for command\n\nCommands:\n on [options] Measure this clone, and remove what the previous\n version left in it\n off Stop measuring this clone; what was measured stays\n forget [options] Show everything measurement keeps on this machine, or\n remove it with --yes\n ingest [options] Read new transcripts into the local ledger, for\n clones that opted in\n task [options] [name] Declare the task the current work belongs to\n status Show if this clone is measured, since when, and its\n task\n report [options] Show what the work consumed, split along one axis,\n after reading new transcripts\n identity [options] [id] Choose to be named on your own measurement, or show\n or remove that choice\n help [command] display help for command" + "help": "Usage: aidd telemetry [options] [command]\n\nMeasure what Claude Code and Codex sessions consume, locally\n\nOptions:\n -h, --help display help for command\n\nCommands:\n otel Configure native OpenTelemetry exporters\n on [options] Measure this clone, and remove what the previous\n version left in it\n off Stop measuring this clone; what was measured stays\n forget [options] Show everything measurement keeps on this machine, or\n remove it with --yes\n ingest [options] Read new transcripts into the local ledger, for\n clones that opted in\n task [options] [name] Declare the task the current work belongs to\n status Show if this clone is measured, since when, and its\n task\n report [options] Show what the work consumed, split along one axis,\n after reading new transcripts\n identity [options] [id] Choose to be named on your own measurement, or show\n or remove that choice\n help [command] display help for command" }, { "invocation": "aidd telemetry forget", @@ -164,6 +164,21 @@ "exitCode": 0, "help": "Usage: aidd telemetry on [options]\n\nMeasure this clone, and remove what the previous version left in it\n\nOptions:\n --yes Do not ask first (default: false)\n --as Name you by this identifier on your own measurement\n --anonymous Name nobody on your own measurement (default: false)\n -h, --help display help for command" }, + { + "invocation": "aidd telemetry otel", + "exitCode": 0, + "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n run [options] [args...] Launch OpenCode 1 with local OTLP export. Pass tool\n arguments after --\n help [command] display help for command" + }, + { + "invocation": "aidd telemetry otel init", + "exitCode": 0, + "help": "Usage: aidd telemetry otel init [options]\n\nEnable detailed local OTLP export in one tool's user configuration\n\nOptions:\n --config User configuration to merge (choices: \"claude\",\n \"codex\", \"opencode\")\n --endpoint Collector destination (choices: \"dev\", \"prod\")\n -h, --help display help for command" + }, + { + "invocation": "aidd telemetry otel run", + "exitCode": 0, + "help": "Usage: aidd telemetry otel run [options] [args...]\n\nLaunch OpenCode 1 with local OTLP export. Pass tool arguments after --\n\nOptions:\n --config Tool to launch (choices: \"opencode\")\n -h, --help display help for command" + }, { "invocation": "aidd telemetry report", "exitCode": 0, diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index cef7703d9..ac64b9652 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -647,3 +647,11 @@ describe("the telemetry command surface", () => { expect(stderr()).toContain("disk full"); }); }); + +describe("telemetry otel init", () => { + it("refuses the production endpoint through the command without touching ledger wiring", async () => { + expect(await run("otel", "init", "--config=codex", "--endpoint=prod")).toBe(1); + expect(stderr()).toContain("Gouvernail"); + expect(turnOn).not.toHaveBeenCalled(); + }); +}); diff --git a/plugins/aidd-telemetry/CATALOG.md b/plugins/aidd-telemetry/CATALOG.md index bbc7098ed..733196e81 100644 --- a/plugins/aidd-telemetry/CATALOG.md +++ b/plugins/aidd-telemetry/CATALOG.md @@ -34,9 +34,10 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---| | `actions` | [forget.md](skills/00-init/actions/forget.md) | - | | `actions` | [identity.md](skills/00-init/actions/identity.md) | - | +| `actions` | [otel.md](skills/00-init/actions/otel.md) | - | | `actions` | [start.md](skills/00-init/actions/start.md) | - | | `actions` | [stop.md](skills/00-init/actions/stop.md) | - | -| `-` | [SKILL.md](skills/00-init/SKILL.md) | `Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Not for answering what a period consumed.` | +| `-` | [SKILL.md](skills/00-init/SKILL.md) | `Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Also use for explicit requests to configure provider-native OpenTelemetry export. Not for answering what a period consumed.` | #### `skills/01-usage` diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 1d873fa43..581f2759c 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -45,10 +45,14 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. - Each model call counts once, whatever the axis. - Where you are asked for a task, and the limits: see the usage contract. -## Privacy +## Local usage measurement privacy - Nothing leaves your machine. - Consent is per clone and never committed. - Data lives in `~/.config/aidd/telemetry` and the clone's git config. - You are named only if you choose to be. -- `AIDD_TELEMETRY=0` refuses measurement everywhere. +- `AIDD_TELEMETRY=0` refuses AIDD usage measurement everywhere. + +## Provider OpenTelemetry + +`aidd telemetry otel init --config= --endpoint=` configures the selected provider to send its native OpenTelemetry to the selected collector. This is a separate opt-in from local usage measurement. Raw provider telemetry can contain sensitive data, including secrets. The private development collector listens at `http://127.0.0.1:14318` and requires a bearer token in `~/.local/share/aidd-otel-observation/collector-token`. Start the collector separately before sending telemetry. Production is refused until its destination is configured. diff --git a/plugins/aidd-telemetry/skills/00-init/SKILL.md b/plugins/aidd-telemetry/skills/00-init/SKILL.md index ba79d2197..2c5431755 100644 --- a/plugins/aidd-telemetry/skills/00-init/SKILL.md +++ b/plugins/aidd-telemetry/skills/00-init/SKILL.md @@ -1,20 +1,22 @@ --- name: 00-init -description: Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Not for answering what a period consumed. -argument-hint: opt in | my name | opt out | delete +description: Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Also use for explicit requests to configure provider-native OpenTelemetry export. Not for answering what a period consumed. +argument-hint: opt in | my name | opt out | delete | otel --config=tool --endpoint=environment --- # Init ```mermaid flowchart TD - request([request]) --> probe{"aidd answers telemetry?"} + request([request]) --> route{which case} + route -->|"configure provider OpenTelemetry"| otel --> done([reported as it ran]) + route -->|"local usage measurement"| probe{"aidd answers telemetry on?"} probe -->|"no"| absent([say it is missing, change nothing]) - probe -->|"yes"| route{which case} - route -->|"start measuring"| start --> done([reported as it ran]) - route -->|"name me, or stop naming me"| identity --> done - route -->|"stop measuring"| stop --> done - route -->|"delete what was measured"| forget + probe -->|"yes"| local{which local case} + local -->|"start measuring"| start --> done + local -->|"name me, or stop naming me"| identity --> done + local -->|"stop measuring"| stop --> done + local -->|"delete what was measured"| forget forget -->|"person confirms the preview"| done forget -->|"person declines"| kept([nothing removed]) ``` @@ -25,6 +27,7 @@ Run the flow above. Read only the action file the case needs. | Action | Does | | --- | --- | +| otel | configure the chosen provider to export native OpenTelemetry to the chosen collector | | start | ask for consent, then turn measurement on for this project | | identity | set, show or remove the name on the person's own measurement | | stop | turn measurement off for this project, keeping what was measured | @@ -32,9 +35,9 @@ Run the flow above. Read only the action file the case needs. ## Transversal rules -- Probe first. Run `aidd telemetry on --help` before any action. When `aidd` is not found, the command exits non-zero, or its help does not list `--anonymous`, stop and say plainly that `aidd` is missing or too old to measure, name `@ai-driven-dev/cli` as what to install or update, and state that nothing was changed. Never go on, and never answer as if it had worked. +- Probe first. For provider OpenTelemetry, use the probe in the `otel` action. For local usage measurement, run `aidd telemetry on --help` before any action. When `aidd` is not found, the command exits non-zero, or its help does not list `--anonymous`, stop and say plainly that `aidd` is missing or too old to measure, name `@ai-driven-dev/cli` as what to install or update, and state that nothing was changed. Never go on, and never answer as if it had worked. - Every change goes through `aidd telemetry`. Never edit git config, `.aidd/config.json` or any file by hand. - Never run a command that changes anything before the person has said yes to what it does. A silence, a previous yes to another case, or a standing instruction is not a yes. -- Report what each command printed, including every warning, and never claim a result the output did not show. +- Report what each command printed, including every warning, with credentials redacted, and never claim a result the output did not show. - Never choose, derive or guess a name for the person: not from git, from the machine, nor from the session. Only the identifier the person types is used. `aidd telemetry on` run in a terminal offers the git name itself. - When a command exits non-zero, show its message and stop. Do not retry with other flags. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/otel.md b/plugins/aidd-telemetry/skills/00-init/actions/otel.md new file mode 100644 index 000000000..3d4dd07b2 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/otel.md @@ -0,0 +1,26 @@ +# Configure provider OpenTelemetry + +## Input + +The provider (`claude`, `codex` or `opencode`) and destination (`dev` or `prod`) chosen by the person. + +## Output + +The selected provider and destination, any required restart or version limitation, and whether configuration changed, or the CLI's refusal. + +## Process + +1. Run `aidd telemetry otel init --help`. If unavailable, say that `@ai-driven-dev/cli` needs installing or updating and that nothing changed. +2. Obtain both choices. Explain that the selected provider sends native OpenTelemetry to the chosen collector and that raw telemetry can contain sensitive data, including secrets. The private development collector listens at `http://127.0.0.1:14318` and requires a bearer token in `~/.local/share/aidd-otel-observation/collector-token`. Production is refused until its destination is configured. This opt-in is separate from the local AIDD usage ledger. +3. Run `aidd telemetry otel init --config= --endpoint=` only after the person explicitly requests enabling that export. A request naming the provider and destination is sufficient; do not ask for it again. Let the CLI validate configuration and credentials. Never ask the person to paste a token into chat. +4. Report the configured destination and any warnings or the refusal. Keep credentials and the full merged configuration out of the report. A successful configuration does not establish that the collector received telemetry. + +## Test + +| Case | Pass | +| --- | --- | +| Missing choice or consent | No init command ran | +| Approved provider and destination | One init command ran with both choices, and its result was reported | +| CLI refused the setup | The refusal was reported without retrying or editing files | +| Production destination unavailable | The CLI's refusal is reported and no configuration is changed | +| Existing local AIDD measurement | No measurement, identity or deletion command runs | diff --git a/scripts/__tests__/telemetry-skills-name-real-commands.test.js b/scripts/__tests__/telemetry-skills-name-real-commands.test.js index b1ae34000..fc23e9cd2 100644 --- a/scripts/__tests__/telemetry-skills-name-real-commands.test.js +++ b/scripts/__tests__/telemetry-skills-name-real-commands.test.js @@ -28,8 +28,8 @@ function telemetryHelp() { assert.ok(group, "the help golden has no `aidd telemetry` entry"); const commands = new Map(); for (const entry of entries) { - const name = entry.invocation.trim().split(/\s+/u)[2]; - if (name === undefined) continue; + const name = entry.invocation.trim().split(/\s+/u).slice(2).join(" "); + if (name === "") continue; const help = entry.help.replace(/\s+/gu, " "); const choices = new Map(); for (const match of help.matchAll(/(--[a-z-]+)(?:(?!--)[^(])*\(choices: ([^)]*)\)/gu)) { @@ -64,12 +64,27 @@ function codeRegions(markdown) { /** Every `aidd telemetry ...` a skill's code names, as `{ file, text, sub, rest }`. */ function mentions() { const found = []; + const commands = telemetryHelp(); for (const file of markdownUnder(SKILLS)) { const relative = path.relative(ROOT, file).split(path.sep).join("/"); for (const region of codeRegions(fs.readFileSync(file, "utf8"))) { for (const match of region.matchAll(COMMAND)) { const words = match[1].trim().split(/\s+/u).filter(Boolean); - found.push({ file: relative, text: `aidd telemetry${match[1]}`.trim(), sub: words[0], rest: words.slice(1) }); + let count = 0; + while (count < words.length && commands.has(words.slice(0, count + 1).join(" "))) { + count++; + } + // A command group cannot accept an unknown child as an argument. + const prefix = `${words.slice(0, count).join(" ")} `; + const hasChildren = [...commands.keys()].some((name) => name.startsWith(prefix)); + if (count > 0 && hasChildren && /^[a-z][a-z-]*$/u.test(words[count] ?? "")) count++; + if (count === 0 && words.length > 0) count = 1; + found.push({ + file: relative, + text: `aidd telemetry${match[1]}`.trim(), + sub: words.slice(0, count).join(" ") || undefined, + rest: words.slice(count), + }); } } } @@ -104,13 +119,23 @@ describe("the telemetry skills name only commands the CLI has", () => { const command = help.get(mention.sub); if (command === undefined) continue; mention.rest.forEach((word, at) => { - if (word.startsWith("--") && !command.flags.has(word)) { - wrong.push(`${mention.file}: \`${mention.text}\` has no ${word}`); + const [flag, inlineValue] = word.split("=", 2); + if (word.startsWith("--") && !command.flags.has(flag)) { + wrong.push(`${mention.file}: \`${mention.text}\` has no ${flag}`); } const allowed = command.choices.get(mention.rest[at - 1]); if (allowed !== undefined && /^[a-z]+$/u.test(word) && !allowed.has(word)) { wrong.push(`${mention.file}: \`${mention.text}\` gives ${mention.rest[at - 1]} the value ${word}`); } + const inlineChoices = command.choices.get(flag); + if ( + inlineChoices !== undefined && + inlineValue !== undefined && + /^[a-z]+$/u.test(inlineValue) && + !inlineChoices.has(inlineValue) + ) { + wrong.push(`${mention.file}: \`${mention.text}\` gives ${flag} the value ${inlineValue}`); + } }); } assert.deepEqual(wrong, []); From 3be60d3e9a0ffb038067635497e101b2763b8af8 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 10:20:39 +0800 Subject: [PATCH 02/56] feat(telemetry): verify native export capture and split setup actions --- aidd_docs/recipes/observe-agent-telemetry.md | 29 ++--- cli/aidd_docs/memory/codebase-map.md | 3 +- cli/package.json | 2 +- .../infrastructure/otel/actions/init.ts | 47 +++++++ .../infrastructure/otel/actions/run.ts | 49 ++++++++ .../infrastructure/otel/actions/verify.ts | 40 ++++++ .../{claude-dev.json => config/claude.json} | 0 .../{codex-dev.json => config/codex.json} | 0 .../opencode.json} | 0 .../telemetry/infrastructure/otel/files.ts | 24 ++++ .../otel/{init-otel.ts => settings.ts} | 116 +++--------------- cli/src/presentation/commands/telemetry.ts | 89 +++++++++----- cli/src/runtime/wiring/telemetry-otel.ts | 14 ++- .../tool-addition-cost.arch.test.ts | 2 +- .../otel/init-otel.integration.test.ts | 20 ++- .../otel/verify-otel.integration.test.ts | 60 +++++++++ cli/tests/e2e/telemetry-otel.e2e.test.ts | 21 ++++ cli/tests/golden/snapshots/help/surface.json | 7 +- plugins/aidd-telemetry/CATALOG.md | 14 ++- plugins/aidd-telemetry/README.md | 2 +- .../aidd-telemetry/skills/00-init/SKILL.md | 55 ++++----- .../skills/00-init/actions/01-probe.md | 17 +++ .../00-init/actions/{start.md => 02-start.md} | 31 ++--- .../skills/00-init/actions/03-identity.md | 20 +++ .../skills/00-init/actions/04-stop.md | 18 +++ .../skills/00-init/actions/05-forget.md | 18 +++ .../skills/00-init/actions/06-otel-init.md | 28 +++++ .../skills/00-init/actions/07-otel-verify.md | 24 ++++ .../skills/00-init/actions/forget.md | 29 ----- .../skills/00-init/actions/identity.md | 27 ---- .../skills/00-init/actions/otel.md | 26 ---- .../skills/00-init/actions/stop.md | 24 ---- scripts/__tests__/architecture-rules.test.js | 66 ++++++++++ scripts/lib/architecture-rules.js | 36 +++++- 34 files changed, 623 insertions(+), 335 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts rename cli/src/contexts/telemetry/infrastructure/otel/{claude-dev.json => config/claude.json} (100%) rename cli/src/contexts/telemetry/infrastructure/otel/{codex-dev.json => config/codex.json} (100%) rename cli/src/contexts/telemetry/infrastructure/otel/{opencode-dev.json => config/opencode.json} (100%) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/files.ts rename cli/src/contexts/telemetry/infrastructure/otel/{init-otel.ts => settings.ts} (61%) create mode 100644 cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/01-probe.md rename plugins/aidd-telemetry/skills/00-init/actions/{start.md => 02-start.md} (58%) create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/03-identity.md create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/04-stop.md create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/05-forget.md create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md create mode 100644 plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md delete mode 100644 plugins/aidd-telemetry/skills/00-init/actions/forget.md delete mode 100644 plugins/aidd-telemetry/skills/00-init/actions/identity.md delete mode 100644 plugins/aidd-telemetry/skills/00-init/actions/otel.md delete mode 100644 plugins/aidd-telemetry/skills/00-init/actions/stop.md diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index f50c79229..a189b17fb 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -1,6 +1,6 @@ # Observe agent telemetry locally -Configure native telemetry for Claude Code, Codex and OpenCode against the private development collector, then compare observed fields. +Configure native telemetry for Claude Code, Codex and OpenCode against the private development collector, then verify receipt. ## Steps to observe agent telemetry locally @@ -31,29 +31,14 @@ Claude and Codex also support native OTLP logs, metrics and traces; Claude traci ### 3) ๐Ÿงช Verify receipt -Run a short test session and a non-destructive tool, then check that the relevant local file grows. OpenCode session SSE and native OTLP are separate captures; the SSE capture depends on a running subscriber. +Run a short session with the selected tool, then ask the CLI to check its configuration and captured telemetry: ```sh -for file in "$HOME/.local/share/aidd-otel-observation/claude.jsonl" \ - "$HOME/.local/share/aidd-otel-observation/codex.jsonl" \ - "$HOME/.local/share/aidd-otel-observation/opencode.jsonl" \ - "$HOME/.local/share/aidd-otel-observation/opencode/events.jsonl"; do - if [ -f "$file" ]; then wc -l "$file"; fi -done +aidd telemetry otel verify --config=claude --endpoint=dev +aidd telemetry otel verify --config=codex --endpoint=dev +aidd telemetry otel verify --config=opencode --endpoint=dev ``` -A configured exporter does not prove delivery. Existing sessions may need restarting. OpenCode SSE misses events during disconnection and does not cover another server or `--standalone` session. +Use only the command for the tool you configured. Verification reads the selected tool's local capture and reports its nonempty line count and modification time. Historical records prove previous receipt, not delivery from the current session; compare the reported time with your test session. Follow the CLI's failure message if configuration or captured lines are missing. -### 4) ๐Ÿ“Š Compare observed fields - -| Field | Claude Code | Codex | OpenCode v2 SSE | -| --- | --- | --- | --- | -| Session | `session.id` | `conversation.id` | `data.sessionID` | -| Model | `model` | `model` | `data.model.id` | -| Input / output tokens | `input_tokens` / `output_tokens` | `input_token_count` / `output_token_count` | `data.tokens.input` / `data.tokens.output` | -| Cache reads | `cache_read_tokens` | `cached_token_count` | `data.tokens.cache.read` | -| Tool activity | `tool_result` | `codex.tool_result` | `session.tool.input.started`, `session.tool.success` | - -These fields were observed in local samples, not guaranteed in every session. Align token definitions before comparing them: for example, Claude input excludes cache reads and writes. OpenCode v2 can turn unavailable token values into zero, so zero is not proof of no usage. See [Claude token counters](https://code.claude.com/docs/en/monitoring-usage) and [OpenCode usage normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts), accessed 2026-11-04. - -Files grow without automatic rotation. Configuration and receipt do not establish a Gouvernail integration; forwarding, normalization and retention remain future work. +Captured files need a retention policy; initialization and verification do not rotate or delete them. diff --git a/cli/aidd_docs/memory/codebase-map.md b/cli/aidd_docs/memory/codebase-map.md index edf71de9c..fdf195e31 100644 --- a/cli/aidd_docs/memory/codebase-map.md +++ b/cli/aidd_docs/memory/codebase-map.md @@ -71,7 +71,8 @@ src/ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ report/ # attribution of a call, the days of a report, the axes it splits by โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ switch/ # the previous version's config block, Claude's retention โ”‚ โ”‚ โ””โ”€โ”€ infrastructure/ -โ”‚ โ”‚ โ”œโ”€โ”€ otel/ # explicit user configuration of native OTLP exporters +โ”‚ โ”‚ โ”œโ”€โ”€ otel/ # shared native OTLP settings; JSON templates in config/ +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ actions/ # initialize configuration, launch export, verify receipt โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # reading a clone's identity from the file system, and the consent history file โ”‚ โ”‚ โ”œโ”€โ”€ declaration/ # where a declaration is kept: git config and the telemetry dir โ”‚ โ”‚ โ”œโ”€โ”€ forget/ # erasing the telemetry dir, the previous version's files, git config keys diff --git a/cli/package.json b/cli/package.json index bf99cbd76..d382d958f 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 728, + "bundleBudgetKB": 729, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts new file mode 100644 index 000000000..a473727f0 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts @@ -0,0 +1,47 @@ +import { randomUUID } from "node:crypto"; +import { mkdir, open, rename, rm, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; +import { readSettings } from "../files.js"; +import { + type Config, + configurationEdit, + type Edit, + type Endpoint, + type Host, +} from "../settings.js"; + +async function applyEdit(edit: Edit): Promise { + const before = await readSettings(edit.path, edit.initial); + const next = edit.update(before.content); + if (next === before.content && (before.private || edit.containsToken === false)) return false; + + await mkdir(dirname(edit.path), { recursive: true, mode: 0o700 }); + const lockPath = `${edit.path}.aidd-otel.lock`; + const lock = await open(lockPath, "wx", 0o600); + try { + const current = await readSettings(edit.path, edit.initial); + if ( + current.content !== before.content || + current.private !== before.private || + current.exists !== before.exists + ) { + throw new Error("OTLP settings changed during initialization. Retry with the updated file."); + } + const temporary = `${edit.path}.${randomUUID()}.tmp`; + try { + await writeFile(temporary, next, { flag: "wx", mode: 0o600 }); + await rename(temporary, edit.path); + } finally { + await rm(temporary, { force: true }); + } + } finally { + await lock.close(); + await rm(lockPath, { force: true }); + } + return true; +} + +export async function initializeOtel(config: Config, endpoint: Endpoint, host: Host) { + const edit = await configurationEdit(config, endpoint, host); + return { path: edit.path, launch: edit.launch, changed: await applyEdit(edit) }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts new file mode 100644 index 000000000..7398798fe --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -0,0 +1,49 @@ +import { spawn } from "node:child_process"; +import { constants as osConstants } from "node:os"; +import opencodeTemplate from "../config/opencode.json" with { type: "json" }; +import { rejectConflict } from "../config-edits.js"; +import { readSettings } from "../files.js"; +import { + type Host, + interpolate, + OPENCODE_CONFIG_ID, + openCodeV1Edit, + openCodeVersion, + readToken, +} from "../settings.js"; + +export async function runOtelOpenCode(args: readonly string[], host: Host): Promise { + if ((await openCodeVersion(host)) !== "1") { + throw new Error("OTLP launch is for OpenCode 1. Use the background service with OpenCode 2."); + } + const edit = await openCodeV1Edit(host); + const current = await readSettings(edit.path, edit.initial); + if (edit.update(current.content) !== current.content) { + throw new Error( + "Initialize OpenCode 1 OTLP with aidd telemetry otel init --config=opencode --endpoint=dev." + ); + } + const token = await readToken(host.home); + const additions = interpolate(opencodeTemplate.env, token); + for (const [key, value] of Object.entries(additions)) + rejectConflict(key, host.env[key] || undefined, value); + const child = spawn(OPENCODE_CONFIG_ID, args, { + env: { ...host.env, ...additions }, + stdio: "inherit", + }); + const interrupt = () => child.kill("SIGINT"); + const terminate = () => child.kill("SIGTERM"); + process.on("SIGINT", interrupt); + process.on("SIGTERM", terminate); + try { + return await new Promise((resolve, reject) => { + child.once("error", () => reject(new Error("Could not launch OpenCode."))); + child.once("exit", (code, signal) => + resolve(code ?? (signal ? 128 + osConstants.signals[signal] : 1)) + ); + }); + } finally { + process.off("SIGINT", interrupt); + process.off("SIGTERM", terminate); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts new file mode 100644 index 000000000..9ecd9bfce --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts @@ -0,0 +1,40 @@ +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; +import { join } from "node:path"; +import { readSettings } from "../files.js"; +import { type Config, configurationEdit, type Endpoint, type Host } from "../settings.js"; + +export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) { + const edit = await configurationEdit(config, endpoint, host); + const current = await readSettings(edit.path, edit.initial); + if ( + (!current.exists && edit.containsToken !== false) || + edit.update(current.content) !== current.content + ) { + throw new Error("OTLP configuration is not initialized. Run aidd telemetry otel init first."); + } + if (!current.private && edit.containsToken !== false) { + throw new Error("OTLP configuration must be private (mode 0600)."); + } + const capture = join(host.home, ".local/share/aidd-otel-observation", `${config}.jsonl`); + const handle = await open( + capture, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error.code === "ENOENT") + throw new Error("No OTLP capture found. Run a tool session and retry."); + throw error; + }); + try { + const info = await handle.stat(); + if (!info.isFile()) throw new Error("OTLP capture must be a regular file."); + let lines = 0; + for await (const line of handle.readLines({ autoClose: false })) { + if (line.trim()) lines++; + } + if (lines === 0) throw new Error("OTLP capture is empty. Run a tool session and retry."); + return { path: edit.path, capture, lines, modifiedAt: info.mtime.toISOString() }; + } finally { + await handle.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/claude-dev.json b/cli/src/contexts/telemetry/infrastructure/otel/config/claude.json similarity index 100% rename from cli/src/contexts/telemetry/infrastructure/otel/claude-dev.json rename to cli/src/contexts/telemetry/infrastructure/otel/config/claude.json diff --git a/cli/src/contexts/telemetry/infrastructure/otel/codex-dev.json b/cli/src/contexts/telemetry/infrastructure/otel/config/codex.json similarity index 100% rename from cli/src/contexts/telemetry/infrastructure/otel/codex-dev.json rename to cli/src/contexts/telemetry/infrastructure/otel/config/codex.json diff --git a/cli/src/contexts/telemetry/infrastructure/otel/opencode-dev.json b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json similarity index 100% rename from cli/src/contexts/telemetry/infrastructure/otel/opencode-dev.json rename to cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json diff --git a/cli/src/contexts/telemetry/infrastructure/otel/files.ts b/cli/src/contexts/telemetry/infrastructure/otel/files.ts new file mode 100644 index 000000000..3b3a13c3f --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/files.ts @@ -0,0 +1,24 @@ +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; + +export async function readSettings(path: string, initial: string) { + const handle = await open( + path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!handle) return { content: initial, private: false, exists: false }; + try { + const info = await handle.stat(); + if (!info.isFile()) throw new Error("OTLP configuration must be a regular file."); + return { + content: await handle.readFile("utf8"), + private: (info.mode & 0o077) === 0, + exists: true, + }; + } finally { + await handle.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts similarity index 61% rename from cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts rename to cli/src/contexts/telemetry/infrastructure/otel/settings.ts index 522f4de88..02dbd4832 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/init-otel.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts @@ -1,13 +1,12 @@ -import { execFile, spawn } from "node:child_process"; -import { randomUUID } from "node:crypto"; +import { execFile } from "node:child_process"; import { constants } from "node:fs"; -import { lstat, mkdir, open, readFile, rename, rm, writeFile } from "node:fs/promises"; -import { constants as osConstants } from "node:os"; -import { dirname, join } from "node:path"; +import { open } from "node:fs/promises"; +import { join } from "node:path"; import { promisify } from "node:util"; import { claudeConfigDir } from "../../domain/claude-projects-root.js"; -import claudeTemplate from "./claude-dev.json" with { type: "json" }; -import codexTemplate from "./codex-dev.json" with { type: "json" }; +import claudeTemplate from "./config/claude.json" with { type: "json" }; +import codexTemplate from "./config/codex.json" with { type: "json" }; +import opencodeTemplate from "./config/opencode.json" with { type: "json" }; import { mergeCodexExporters, mergeJsonEnvironment, @@ -15,21 +14,21 @@ import { openCodeInstrumentation, rejectConflict, } from "./config-edits.js"; -import opencodeTemplate from "./opencode-dev.json" with { type: "json" }; +import { readSettings } from "./files.js"; export const OPENCODE_CONFIG_ID = "opencode"; export const OTEL_CONFIGS = ["claude", "codex", OPENCODE_CONFIG_ID] as const; export const OTEL_ENDPOINTS = ["dev", "prod"] as const; -type Config = (typeof OTEL_CONFIGS)[number]; -type Endpoint = (typeof OTEL_ENDPOINTS)[number]; +export type Config = (typeof OTEL_CONFIGS)[number]; +export type Endpoint = (typeof OTEL_ENDPOINTS)[number]; -interface Host { +export interface Host { readonly home: string; readonly env: NodeJS.ProcessEnv; readonly opencodeVersion?: () => Promise; } -interface Edit { +export interface Edit { readonly path: string; readonly initial: string; readonly update: (content: string) => string; @@ -41,7 +40,7 @@ const COLLECTOR = "http://127.0.0.1:14318"; const TOKEN_PATH = ".local/share/aidd-otel-observation/collector-token"; const SIGNALS = ["LOGS", "METRICS", "TRACES"] as const; -function interpolate(template: Readonly>, token: string) { +export function interpolate(template: Readonly>, token: string) { return Object.fromEntries( Object.entries(template).map(([key, value]) => [ key, @@ -50,7 +49,7 @@ function interpolate(template: Readonly>, token: string) ); } -async function readToken(home: string): Promise { +export async function readToken(home: string): Promise { try { const handle = await open(join(home, TOKEN_PATH), constants.O_RDONLY | constants.O_NOFOLLOW); try { @@ -95,7 +94,7 @@ function claudeEdit(host: Host, token: string): Edit { }; } -async function openCodeVersion(host: Host): Promise<"1" | "2"> { +export async function openCodeVersion(host: Host): Promise<"1" | "2"> { let output: string; try { output = host.opencodeVersion @@ -126,7 +125,7 @@ function openCodeGlobalDir(host: Host): string { return join(host.env.XDG_CONFIG_HOME || join(host.home, ".config"), OPENCODE_CONFIG_ID); } -async function openCodeV1Edit(host: Host): Promise { +export async function openCodeV1Edit(host: Host): Promise { const global = openCodeGlobalDir(host); const custom = host.env.OPENCODE_CONFIG_DIR; const initial = "{}\n"; @@ -175,88 +174,7 @@ async function openCodeEdit(host: Host, token: string): Promise { }; } -export async function runOtelOpenCode(args: readonly string[], host: Host): Promise { - if ((await openCodeVersion(host)) !== "1") { - throw new Error("OTLP launch is for OpenCode 1. Use the background service with OpenCode 2."); - } - const edit = await openCodeV1Edit(host); - const current = await readSettings(edit.path, edit.initial); - if (edit.update(current.content) !== current.content) { - throw new Error( - "Initialize OpenCode 1 OTLP with aidd telemetry otel init --config=opencode --endpoint=dev." - ); - } - const token = await readToken(host.home); - const additions = interpolate(opencodeTemplate.env, token); - for (const [key, value] of Object.entries(additions)) - rejectConflict(key, host.env[key] || undefined, value); - const child = spawn(OPENCODE_CONFIG_ID, args, { - env: { ...host.env, ...additions }, - stdio: "inherit", - }); - const interrupt = () => child.kill("SIGINT"); - const terminate = () => child.kill("SIGTERM"); - process.on("SIGINT", interrupt); - process.on("SIGTERM", terminate); - try { - return await new Promise((resolve, reject) => { - child.once("error", () => reject(new Error("Could not launch OpenCode."))); - child.once("exit", (code, signal) => - resolve(code ?? (signal ? 128 + osConstants.signals[signal] : 1)) - ); - }); - } finally { - process.off("SIGINT", interrupt); - process.off("SIGTERM", terminate); - } -} - -async function readSettings(path: string, initial: string) { - const info = await lstat(path).catch((error) => { - if (error.code === "ENOENT") return null; - throw error; - }); - if (!info) return { content: initial, private: false, exists: false }; - if (!info.isFile()) throw new Error("OTLP configuration must be a regular file."); - return { - content: await readFile(path, "utf8"), - private: (info.mode & 0o077) === 0, - exists: true, - }; -} - -async function applyEdit(edit: Edit): Promise { - const before = await readSettings(edit.path, edit.initial); - const next = edit.update(before.content); - if (next === before.content && (before.private || edit.containsToken === false)) return false; - - await mkdir(dirname(edit.path), { recursive: true, mode: 0o700 }); - const lockPath = `${edit.path}.aidd-otel.lock`; - const lock = await open(lockPath, "wx", 0o600); - try { - const current = await readSettings(edit.path, edit.initial); - if ( - current.content !== before.content || - current.private !== before.private || - current.exists !== before.exists - ) { - throw new Error("OTLP settings changed during initialization. Retry with the updated file."); - } - const temporary = `${edit.path}.${randomUUID()}.tmp`; - try { - await writeFile(temporary, next, { flag: "wx", mode: 0o600 }); - await rename(temporary, edit.path); - } finally { - await rm(temporary, { force: true }); - } - } finally { - await lock.close(); - await rm(lockPath, { force: true }); - } - return true; -} - -export async function initializeOtel(config: Config, endpoint: Endpoint, host: Host) { +export async function configurationEdit(config: Config, endpoint: Endpoint, host: Host) { if (endpoint === "prod") { throw new Error( "OTLP production configuration is unavailable until the Gouvernail endpoint is published." @@ -290,5 +208,5 @@ export async function initializeOtel(config: Config, endpoint: Endpoint, host: H edit = await openCodeEdit(host, token); break; } - return { path: edit.path, launch: edit.launch, changed: await applyEdit(edit) }; + return edit; } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 5d5af2b80..1015977e1 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -27,7 +27,12 @@ import { USAGE_AGENTS, USAGE_TOOLS } from "../../contexts/telemetry/domain/usage import type { Prompter } from "../../kernel/ports/prompter.js"; import { createDeps } from "../../runtime/wiring/framework.js"; import type { TelemetryDeps } from "../../runtime/wiring/telemetry.js"; -import { configureOtel, launchOtel, otelOptions } from "../../runtime/wiring/telemetry-otel.js"; +import { + checkOtel, + configureOtel, + launchOtel, + otelOptions, +} from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, printIdentityRule, @@ -130,39 +135,57 @@ export function registerTelemetryCommand(program: Command): void { .command("telemetry") .description("Measure what Claude Code and Codex sessions consume, locally"); const otel = telemetry.command("otel").description("Configure native OpenTelemetry exporters"); - otel - .command("init") - .description("Enable detailed local OTLP export in one tool's user configuration") - .addOption( - new Option("--config ", "User configuration to merge") - .choices(otelOptions.configs) - .makeOptionMandatory() - ) - .addOption( - new Option("--endpoint ", "Collector destination") - .choices(otelOptions.endpoints) - .makeOptionMandatory() - ) - .action( - async (options: { - config: (typeof otelOptions.configs)[number]; - endpoint: (typeof otelOptions.endpoints)[number]; - }) => { - const { output } = parseGlobalOptions(program); - try { - const result = await configureOtel(options.config, options.endpoint); - output.info( - result.changed ? `Updated ${result.path}` : `Already configured in ${result.path}` - ); - output.info( - "Local OTLP capture may include prompts, responses, commands and tool output." - ); - output.info(result.launch); - } catch (error) { - new ErrorHandler(output).handle(error); + for (const action of ["init", "verify"] as const) { + otel + .command(action) + .description( + action === "init" + ? "Enable detailed local OTLP export in one tool's user configuration" + : "Check native OTLP configuration and existing local capture without changing files" + ) + .addOption( + new Option( + "--config ", + `User configuration to ${action === "init" ? "merge" : "verify"}` + ) + .choices(otelOptions.configs) + .makeOptionMandatory() + ) + .addOption( + new Option("--endpoint ", "Collector destination") + .choices(otelOptions.endpoints) + .makeOptionMandatory() + ) + .action( + async (options: { + config: (typeof otelOptions.configs)[number]; + endpoint: (typeof otelOptions.endpoints)[number]; + }) => { + const { output } = parseGlobalOptions(program); + try { + if (action === "init") { + const result = await configureOtel(options.config, options.endpoint); + output.info( + result.changed ? `Updated ${result.path}` : `Already configured in ${result.path}` + ); + output.info( + "Local OTLP capture may include prompts, responses, commands and tool output." + ); + output.info(result.launch); + } else { + const result = await checkOtel(options.config, options.endpoint); + output.info(`Configuration verified in ${result.path}`); + output.info( + `${result.lines} captured lines in ${result.capture}; last modified ${result.modifiedAt}` + ); + output.info("Existing capture does not prove delivery from the current session."); + } + } catch (error) { + new ErrorHandler(output).handle(error); + } } - } - ); + ); + } otel .command("run [args...]") diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index d05ca37a1..1f07491e4 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -1,11 +1,12 @@ import { homedir } from "node:os"; +import { initializeOtel } from "../../contexts/telemetry/infrastructure/otel/actions/init.js"; +import { runOtelOpenCode } from "../../contexts/telemetry/infrastructure/otel/actions/run.js"; +import { verifyOtel } from "../../contexts/telemetry/infrastructure/otel/actions/verify.js"; import { - initializeOtel, OPENCODE_CONFIG_ID, OTEL_CONFIGS, OTEL_ENDPOINTS, - runOtelOpenCode, -} from "../../contexts/telemetry/infrastructure/otel/init-otel.js"; +} from "../../contexts/telemetry/infrastructure/otel/settings.js"; export const otelOptions = { configs: OTEL_CONFIGS, @@ -23,3 +24,10 @@ export function configureOtel( export function launchOtel(args: readonly string[]) { return runOtelOpenCode(args, { home: homedir(), env: process.env }); } + +export function checkOtel( + config: (typeof OTEL_CONFIGS)[number], + endpoint: (typeof OTEL_ENDPOINTS)[number] +) { + return verifyOtel(config, endpoint, { home: homedir(), env: process.env }); +} diff --git a/cli/tests/architecture/tool-addition-cost.arch.test.ts b/cli/tests/architecture/tool-addition-cost.arch.test.ts index 85a04bf47..24c079130 100644 --- a/cli/tests/architecture/tool-addition-cost.arch.test.ts +++ b/cli/tests/architecture/tool-addition-cost.arch.test.ts @@ -93,7 +93,7 @@ const BASELINE: readonly { readonly path: string; readonly named: number }[] = [ // Claude Code's own settings files, read to say how long it keeps a transcript. { path: "src/contexts/telemetry/infrastructure/switch/claude-settings-adapter.ts", named: 1 }, // OTLP is an explicit, separate support surface: three observed native exporters. - { path: "src/contexts/telemetry/infrastructure/otel/init-otel.ts", named: 3 }, + { path: "src/contexts/telemetry/infrastructure/otel/settings.ts", named: 3 }, // Cursor's project hooks file, named after the tool whose file it is: the directory it // writes into is Cursor's own, not a list a sixth tool joins. { path: "src/contexts/tools/domain/formats/cursor-hooks-project-merge.ts", named: 1 }, diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index e2ea1b4c7..1c605b3bd 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -1,8 +1,8 @@ -import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { mkdir, mkdtemp, open, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/init-otel.js"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; const TOKEN = "example-private-token"; let home: string; @@ -47,8 +47,10 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { ); const result = await initializeOtel("claude", "dev", { home, env: {} }); expect(result.changed).toBe(true); - expect((await stat(path)).mode & 0o777).toBe(0o600); - const content = await readFile(path, "utf8"); + const handle = await open(path, "r"); + const content = await handle.readFile("utf8"); + expect((await handle.stat()).mode & 0o777).toBe(0o600); + await handle.close(); expect(JSON.parse(content)).toMatchObject({ permissions: { allow: [] }, env: { @@ -66,6 +68,16 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { expect(await readFile(path, "utf8")).toBe(content); }); + it("refuses a symlink without reading or replacing its target", async () => { + await seedToken(); + const target = join(home, "original.json"); + await seed(target, "{}"); + await mkdir(join(home, ".claude")); + await symlink(target, join(home, ".claude/settings.json")); + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("{}"); + }); + it("refuses invalid config or a conflicting exporter without altering the file", async () => { await seedToken(); const path = join(home, ".claude/settings.json"); diff --git a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts new file mode 100644 index 000000000..193dacaba --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts @@ -0,0 +1,60 @@ +import { chmod, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { verifyOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js"; + +let home: string; +let capture: string; +beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-verify-")); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true }); + await writeFile(join(directory, "collector-token"), "test-token", { mode: 0o600 }); + capture = join(directory, "claude.jsonl"); +}); +afterEach(async () => { + await rm(home, { recursive: true, force: true }); +}); + +describe.skipIf(process.platform === "win32")("OTLP verification", () => { + it("requires initialization without creating settings", async () => { + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/not initialized/); + await expect(readFile(join(home, ".claude/settings.json"))).rejects.toMatchObject({ + code: "ENOENT", + }); + }); + it("requires captured receipt and leaves configuration unchanged", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const path = join(home, ".claude/settings.json"); + const original = await readFile(path, "utf8"); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/capture/); + await writeFile(capture, "\n", { mode: 0o600 }); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/capture/); + await writeFile(capture, '{"signal":"logs"}\n{"signal":"metrics"}\n'); + expect(await verifyOtel("claude", "dev", { home, env: {} })).toMatchObject({ + capture, + lines: 2, + }); + expect(await readFile(path, "utf8")).toBe(original); + await chmod(path, 0o644); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/private/); + }); + it("accepts OpenCode 1 instrumentation supplied only through its inline environment", async () => { + const host = { + home, + env: { OPENCODE_CONFIG_CONTENT: '{"experimental":{"openTelemetry":true}}' }, + opencodeVersion: async () => "1.19.5", + }; + await initializeOtel("opencode", "dev", host); + await writeFile(join(home, ".local/share/aidd-otel-observation/opencode.jsonl"), "{}\n", { + mode: 0o600, + }); + expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ lines: 1 }); + }); + + it("refuses production", async () => { + await expect(verifyOtel("claude", "prod", { home, env: {} })).rejects.toThrow(/production/); + }); +}); diff --git a/cli/tests/e2e/telemetry-otel.e2e.test.ts b/cli/tests/e2e/telemetry-otel.e2e.test.ts index 2374dfe47..11d1aa836 100644 --- a/cli/tests/e2e/telemetry-otel.e2e.test.ts +++ b/cli/tests/e2e/telemetry-otel.e2e.test.ts @@ -6,6 +6,27 @@ import { describe, expect, it } from "vitest"; import { cliPath, createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; describe.skipIf(process.platform === "win32")("native OTLP launch", () => { + it("verifies captured receipt through the CLI and fails when capture is absent", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("telemetry-verify"); + const directory = join(fakeHome, ".local/share/aidd-otel-observation"); + try { + await mkdir(directory, { recursive: true }); + await writeFile(join(directory, "collector-token"), "test-token", { mode: 0o600 }); + const args = ["--config=claude", "--endpoint=dev"]; + const init = await runCli(["telemetry", "otel", "init", ...args], projectDir, fakeHome); + expect(init.exitCode).toBe(0); + const missing = await runCli(["telemetry", "otel", "verify", ...args], projectDir, fakeHome); + expect(missing.exitCode).not.toBe(0); + await writeFile(join(directory, "claude.jsonl"), '{"signal":"logs"}\n', { mode: 0o600 }); + const verified = await runCli(["telemetry", "otel", "verify", ...args], projectDir, fakeHome); + expect(verified.exitCode).toBe(0); + expect(verified.stdout).toContain("1 captured lines"); + expect(verified.stdout).not.toContain("test-token"); + } finally { + await cleanup(); + } + }); + it.each([ ["SIGINT", 42], ["SIGTERM", 43], diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 4ee59eaf9..17f229628 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -167,7 +167,7 @@ { "invocation": "aidd telemetry otel", "exitCode": 0, - "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n run [options] [args...] Launch OpenCode 1 with local OTLP export. Pass tool\n arguments after --\n help [command] display help for command" + "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local\n capture without changing files\n run [options] [args...] Launch OpenCode 1 with local OTLP export. Pass tool\n arguments after --\n help [command] display help for command" }, { "invocation": "aidd telemetry otel init", @@ -179,6 +179,11 @@ "exitCode": 0, "help": "Usage: aidd telemetry otel run [options] [args...]\n\nLaunch OpenCode 1 with local OTLP export. Pass tool arguments after --\n\nOptions:\n --config Tool to launch (choices: \"opencode\")\n -h, --help display help for command" }, + { + "invocation": "aidd telemetry otel verify", + "exitCode": 0, + "help": "Usage: aidd telemetry otel verify [options]\n\nCheck native OTLP configuration and existing local capture without changing\nfiles\n\nOptions:\n --config User configuration to verify (choices: \"claude\",\n \"codex\", \"opencode\")\n --endpoint Collector destination (choices: \"dev\", \"prod\")\n -h, --help display help for command" + }, { "invocation": "aidd telemetry report", "exitCode": 0, diff --git a/plugins/aidd-telemetry/CATALOG.md b/plugins/aidd-telemetry/CATALOG.md index 733196e81..1dbcb6873 100644 --- a/plugins/aidd-telemetry/CATALOG.md +++ b/plugins/aidd-telemetry/CATALOG.md @@ -32,12 +32,14 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| -| `actions` | [forget.md](skills/00-init/actions/forget.md) | - | -| `actions` | [identity.md](skills/00-init/actions/identity.md) | - | -| `actions` | [otel.md](skills/00-init/actions/otel.md) | - | -| `actions` | [start.md](skills/00-init/actions/start.md) | - | -| `actions` | [stop.md](skills/00-init/actions/stop.md) | - | -| `-` | [SKILL.md](skills/00-init/SKILL.md) | `Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Also use for explicit requests to configure provider-native OpenTelemetry export. Not for answering what a period consumed.` | +| `actions` | [01-probe.md](skills/00-init/actions/01-probe.md) | - | +| `actions` | [02-start.md](skills/00-init/actions/02-start.md) | - | +| `actions` | [03-identity.md](skills/00-init/actions/03-identity.md) | - | +| `actions` | [04-stop.md](skills/00-init/actions/04-stop.md) | - | +| `actions` | [05-forget.md](skills/00-init/actions/05-forget.md) | - | +| `actions` | [06-otel-init.md](skills/00-init/actions/06-otel-init.md) | - | +| `actions` | [07-otel-verify.md](skills/00-init/actions/07-otel-verify.md) | - | +| `-` | [SKILL.md](skills/00-init/SKILL.md) | `Configure and verify provider OpenTelemetry or manage local AIDD measurement. Use when the user wants to enable export, check receipt, start or stop usage measurement, change their identity, or delete measured data. Not for usage reports.` | #### `skills/01-usage` diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 581f2759c..c3f3187a4 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -55,4 +55,4 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. ## Provider OpenTelemetry -`aidd telemetry otel init --config= --endpoint=` configures the selected provider to send its native OpenTelemetry to the selected collector. This is a separate opt-in from local usage measurement. Raw provider telemetry can contain sensitive data, including secrets. The private development collector listens at `http://127.0.0.1:14318` and requires a bearer token in `~/.local/share/aidd-otel-observation/collector-token`. Start the collector separately before sending telemetry. Production is refused until its destination is configured. +Configure an exporter with `aidd telemetry otel init --config= --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits. diff --git a/plugins/aidd-telemetry/skills/00-init/SKILL.md b/plugins/aidd-telemetry/skills/00-init/SKILL.md index 2c5431755..397026006 100644 --- a/plugins/aidd-telemetry/skills/00-init/SKILL.md +++ b/plugins/aidd-telemetry/skills/00-init/SKILL.md @@ -1,43 +1,36 @@ --- name: 00-init -description: Turn AIDD measurement on for a project after explicit consent, stop it, choose to be named on your own measurement, or delete what was measured. Use when the user wants to start or stop measuring what their work consumes, add or remove their name, or forget measured data. Also use for explicit requests to configure provider-native OpenTelemetry export. Not for answering what a period consumed. -argument-hint: opt in | my name | opt out | delete | otel --config=tool --endpoint=environment +description: Configure and verify provider OpenTelemetry or manage local AIDD measurement. Use when the user wants to enable export, check receipt, start or stop usage measurement, change their identity, or delete measured data. Not for usage reports. +argument-hint: provider export | verify receipt | local measurement | my name | stop | delete --- # Init ```mermaid flowchart TD - request([request]) --> route{which case} - route -->|"configure provider OpenTelemetry"| otel --> done([reported as it ran]) - route -->|"local usage measurement"| probe{"aidd answers telemetry on?"} - probe -->|"no"| absent([say it is missing, change nothing]) - probe -->|"yes"| local{which local case} - local -->|"start measuring"| start --> done - local -->|"name me, or stop naming me"| identity --> done - local -->|"stop measuring"| stop --> done - local -->|"delete what was measured"| forget - forget -->|"person confirms the preview"| done - forget -->|"person declines"| kept([nothing removed]) + export(["provider export"]) --> init["actions/06-otel-init.md"] + receipt(["verify receipt"]) --> verify["actions/07-otel-verify.md"] + init -->|"configured"| verify + init -->|"declined or unavailable"| unchanged(["nothing changed"]) + verify --> result(["reported CLI result"]) + local(["local usage measurement"]) --> probe["actions/01-probe.md"] + probe -->|"CLI unavailable"| unchanged + probe -->|"start"| start["actions/02-start.md"] + probe -->|"identity"| identity["actions/03-identity.md"] + probe -->|"stop"| stop["actions/04-stop.md"] + probe -->|"delete"| forget["actions/05-forget.md"] + start -->|"consent given"| result + start -->|"declined"| unchanged + identity --> result + stop -->|"confirmed"| result + stop -->|"declined"| unchanged + forget -->|"confirmed"| result + forget -->|"declined or empty"| unchanged ``` -## Actions - -Run the flow above. Read only the action file the case needs. - -| Action | Does | -| --- | --- | -| otel | configure the chosen provider to export native OpenTelemetry to the chosen collector | -| start | ask for consent, then turn measurement on for this project | -| identity | set, show or remove the name on the person's own measurement | -| stop | turn measurement off for this project, keeping what was measured | -| forget | preview, then on confirmation delete everything measured on this machine | - ## Transversal rules -- Probe first. For provider OpenTelemetry, use the probe in the `otel` action. For local usage measurement, run `aidd telemetry on --help` before any action. When `aidd` is not found, the command exits non-zero, or its help does not list `--anonymous`, stop and say plainly that `aidd` is missing or too old to measure, name `@ai-driven-dev/cli` as what to install or update, and state that nothing was changed. Never go on, and never answer as if it had worked. -- Every change goes through `aidd telemetry`. Never edit git config, `.aidd/config.json` or any file by hand. -- Never run a command that changes anything before the person has said yes to what it does. A silence, a previous yes to another case, or a standing instruction is not a yes. -- Report what each command printed, including every warning, with credentials redacted, and never claim a result the output did not show. -- Never choose, derive or guess a name for the person: not from git, from the machine, nor from the session. Only the identifier the person types is used. `aidd telemetry on` run in a terminal offers the git name itself. -- When a command exits non-zero, show its message and stop. Do not retry with other flags. +- Keep provider export and local usage measurement separate; run only the requested path. +- Apply changes through `aidd telemetry`, with the person's explicit authorization for that operation. +- Report command output and warnings with credentials redacted; a non-zero exit ends the action without retrying with other flags. +- Use only identifiers supplied by the person, rather than deriving one from git, the machine or the session. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/01-probe.md b/plugins/aidd-telemetry/skills/00-init/actions/01-probe.md new file mode 100644 index 000000000..9460e7876 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/01-probe.md @@ -0,0 +1,17 @@ +# 01 - Probe local measurement + +Check that the installed CLI supports local measurement before choosing its action. + +## Input + +The requested local operation. + +## Process + +1. Run `aidd telemetry on --help` and check that it lists `--anonymous`. + - If unavailable, report that `@ai-driven-dev/cli` needs installing or updating and that nothing changed. +2. Continue to the requested action, [start](02-start.md), [identity](03-identity.md), [stop](04-stop.md), or [forget](05-forget.md). + +## Output + +A supported CLI and the next local action, or the availability failure. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/start.md b/plugins/aidd-telemetry/skills/00-init/actions/02-start.md similarity index 58% rename from plugins/aidd-telemetry/skills/00-init/actions/start.md rename to plugins/aidd-telemetry/skills/00-init/actions/02-start.md index ab6c09130..fe6f62d1b 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/start.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/02-start.md @@ -1,4 +1,4 @@ -# Start +# 02 - Start Turn measurement on for the current project, once the person has agreed to it. @@ -6,41 +6,30 @@ Turn measurement on for the current project, once the person has agreed to it. The project directory, which must be inside a git repository. -## Output - -Measurement is on for the project, or nothing changed and the reason is stated. The person knows what is measured, what they will be asked, where it stays, and under which name, if any, they are measured. - ## Process -1. **Explain.** Tell the person, before asking anything: +1. Tell the person, before asking anything: - what is measured: the token counts of Claude Code and Codex sessions run in this project, read from each tool's own session files (Codex's under `CODEX_HOME`, else `~/.codex`), split into input, output, cache read and cache write; - what is asked: one question per working branch, a task name and optionally a ticket, only when a person is at the keyboard, in Claude Code or the Codex TUI. Codex asks only once the person has reviewed and trusted this plugin's checks in its TUI, which lists them for review, and never in `codex exec`, its IDE extension, its desktop app or full-access mode; - when it starts: now. Measurement starts at the moment it is turned on, and what this project's sessions did before is not counted, whatever Claude Code still holds of it; - where it stays: on this machine, and nothing is sent anywhere; - whose consent it is: this clone's alone, kept in its git config and never committed, so a teammate is not measured until they run it in theirs. -2. **Ask.** Ask whether to turn measurement on for this project. Wait for an explicit yes. +2. Ask whether to turn measurement on for this project. Wait for an explicit yes. - A no, or no answer, ends the action with nothing changed. -3. **Choose the name.** Run `aidd telemetry identity` to read whether an identity is already set. +3. Run `aidd telemetry identity` to read whether an identity is already set. - Set: say the person is measured under it, and that the `identity` case changes or removes it. Nothing to ask. - Not set: explain that a name only labels the person axis of their own report and stays on this machine, then ask the person to type the identifier they want, or to stay anonymous. Wait for the answer; never suggest or pick one for them. Say that `aidd telemetry on`, run in their own terminal, offers their git name instead. -4. **Turn on.** The yes was given in step 2, so the command's own confirmation is skipped: +4. Run the matching command with the consent already given: - identity already set: `aidd telemetry on --yes`; - a name typed: `aidd telemetry on --yes --as ''`, with exactly the text typed, single-quoted for the shell, each `'` it holds written as `'\''` (`O'Brien` becomes `'O'\''Brien'`); - anonymous: `aidd telemetry on --yes --anonymous`. Staying anonymous is not stored: the next `on`, in another clone, asks again. - Outside a git repository, with a git config git cannot read, or with an identifier that is not one non-empty line of at most 128 characters, the command refuses: show its message and stop. -5. **Report.** Relay everything the command printed, the removals and the warnings included, in the person's words rather than pasted. +5. Relay everything the command printed, the removals and the warnings included, in the person's words rather than pasted. - When it warned that Claude Code keeps session files only briefly, say that a call is measured only if it is read before the files are removed, and give the setting it named. -6. **Prove.** Run `aidd telemetry report --days 1`, and state what it shows. +6. Run `aidd telemetry report --days 1`, and state what it shows. - No calls yet is a normal answer for a project that has just opted in: say so, and that the next Claude Code or Codex session in this project will be counted. -7. **Tell the rest.** Say that the plugin is what asks about the task and that it needs `node`, and that a task can always be declared by hand with `aidd telemetry task [--ticket ]` or `aidd telemetry task --none`, and that `aidd telemetry status` shows whether a clone is measured and its task. +7. Say that the plugin is what asks about the task and that it needs `node`, and that a task can always be declared by hand with `aidd telemetry task [--ticket ]` or `aidd telemetry task --none`, and that `aidd telemetry status` shows whether a clone is measured and its task. -## Test +## Output -| Case | Pass | -| --- | --- | -| The person says no | No `aidd telemetry on` was run and the project is unchanged | -| The person says yes, with no identity set | They were asked to type an identifier or stay anonymous, no name was suggested and git was not read, exactly one of `aidd telemetry on --yes --as ''`, with the text they typed, or `aidd telemetry on --yes --anonymous` ran, and the person was told that measurement starts now, that earlier history is not counted, and, if the command removed or deleted a block in `.aidd/config.json`, that the file may need committing | -| The person says yes, with an identity set | They were asked nothing about a name, were told the one they are measured under, `aidd telemetry on --yes` ran once, and the person was told the same as above about when measurement starts and `.aidd/config.json` | -| The CLI's `on` has no `--anonymous` | The skill said to update `@ai-driven-dev/cli` and ran nothing else | -| `aidd` has no `telemetry` | The skill said so and ran nothing else | -| Run outside a git repository | The refusal was shown and nothing was written | +Measurement is on for the project, or nothing changed and the reason is stated. The person knows what is measured, what they will be asked, where it stays, and under which name, if any, they are measured. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md b/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md new file mode 100644 index 000000000..be7257788 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md @@ -0,0 +1,20 @@ +# 03 - Identity + +Let the person choose to be named on their own measurement, or stop being named. + +## Input + +What the person wants: to be named as an identifier they give, to see the current choice, or to be named no more. + +## Process + +1. Say that an identity is optional, that it only labels the person axis of their own report, and that it is kept in one file on this machine. +2. Run `aidd telemetry identity` to read the current choice, and tell the person what it holds. +3. To set one, ask the person for the identifier they want and wait for it. Run `aidd telemetry identity ` with exactly that text. + - The command refuses an identifier that is empty, over 128 characters, or spans lines: show its message and stop. +4. To remove it, confirm that the person wants to be named no more, then run `aidd telemetry identity --off`. +5. Run `aidd telemetry identity` again after a change and check that it reports the requested identifier or its removal. + +## Output + +The identity set, shown or removed, as the command reported it. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md new file mode 100644 index 000000000..82c21a2d0 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md @@ -0,0 +1,18 @@ +# 04 - Stop + +Turn measurement off for the current project and keep what was already measured. + +## Input + +The project directory. + +## Process + +1. Say what stopping does: new Claude Code or Codex work in this project is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. Wait for a yes. +2. Run `aidd telemetry off` and relay its message. +3. Say that deleting what was measured is a separate step, and that it needs its own confirmation. +4. Run `aidd telemetry status` and verify that measurement is off. + +## Output + +Measurement is off for the project, and the person knows the measured data is still there. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md new file mode 100644 index 000000000..89f700562 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md @@ -0,0 +1,18 @@ +# 05 - Forget + +Delete all locally measured data after a preview and explicit confirmation. + +## Process + +1. Run `aidd telemetry forget` with no flag. It removes nothing and lists what it would. + - When it says nothing measured is kept, relay that and stop. +2. Give the person the list as the command printed it, with the warnings about repositories it could not reach. +3. Ask whether to remove all of it. Wait for an explicit yes naming the removal. + - A no, or no answer, ends the action with nothing removed. +4. Run `aidd telemetry forget --yes` only after that yes, and relay what it printed. + - Say that it also ends measurement: nothing is measured again until the person runs `aidd telemetry on`. +5. Run `aidd telemetry forget` again and check that its preview lists nothing remaining. + +## Output + +Either the list of what was removed, or the list of what would be, with nothing removed. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md new file mode 100644 index 000000000..28493156e --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md @@ -0,0 +1,28 @@ +# 06 - Initialize provider export + +Merge native OpenTelemetry settings into the selected provider's local configuration. + +## Input + +The provider (`claude`, `codex` or `opencode`) and destination (`dev` or `prod`) chosen by the person. + +## Process + +1. Run `aidd telemetry otel init --help` and report an unavailable command as requiring an update of `@ai-driven-dev/cli`. +2. Obtain both choices and explain that native telemetry goes to the selected collector and can include sensitive data. + - The development collector and its private token must already exist; initialization does not start it. + - Production is refused until its destination is configured. +3. Run `aidd telemetry otel init --config= --endpoint=` after an explicit request enabling that export. + - A request naming both choices is sufficient authorization. + - Let the CLI validate configuration and credentials without requesting a token in chat. +4. Report whether configuration changed and relay the CLI's restart or launch instructions. +5. Continue to [verify receipt](07-otel-verify.md) after the person runs a short session with the selected provider. + +## Output + +The configured provider and destination, or the CLI's refusal. + +## Rules + +- Keep the merged configuration and credentials out of the report. +- Configuration success alone establishes no receipt of telemetry. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md b/plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md new file mode 100644 index 000000000..f7a929489 --- /dev/null +++ b/plugins/aidd-telemetry/skills/00-init/actions/07-otel-verify.md @@ -0,0 +1,24 @@ +# 07 - Verify provider receipt + +Read the CLI's checks of provider configuration and captured telemetry. + +## Input + +The selected provider and destination, carried from initialization or supplied with a verification request. + +## Process + +1. Run `aidd telemetry otel verify --help` and report an unavailable command as requiring an update of `@ai-driven-dev/cli`. +2. Obtain any missing provider or destination choice. +3. Run `aidd telemetry otel verify --config= --endpoint=`. +4. Report the captured nonempty line count and modification time, or the CLI's failure. + - Historical records establish previous receipt, not delivery from the current session. + - If a fresh session has not been run, state that current-session delivery remains unchecked. + +## Output + +The CLI's configuration and receipt result, with its timing limitation. + +## Rules + +- Verification is read-only; configuration changes require a separate initialization request. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/forget.md b/plugins/aidd-telemetry/skills/00-init/actions/forget.md deleted file mode 100644 index edb9697b3..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/forget.md +++ /dev/null @@ -1,29 +0,0 @@ -# Forget - -Delete everything measurement keeps on this machine, after showing it and getting a confirmation. - -## Input - -Nothing. The command covers the whole machine, not one project. - -## Output - -Either the list of what was removed, or the list of what would be, with nothing removed. - -## Process - -1. **Preview.** Run `aidd telemetry forget` with no flag. It removes nothing and lists what it would. - - When it says nothing measured is kept, relay that and stop. -2. **Show.** Give the person the list as the command printed it, with the warnings about repositories it could not reach. -3. **Confirm.** Ask whether to remove all of it. Wait for an explicit yes naming the removal. - - A no, or no answer, ends the action with nothing removed. -4. **Remove.** Run `aidd telemetry forget --yes` only after that yes, and relay what it printed. - - Say that it also ends measurement: nothing is measured again until the person runs `aidd telemetry on`. - -## Test - -| Case | Pass | -| --- | --- | -| Preview | `aidd telemetry forget` ran without `--yes` and the data is still there afterwards | -| The person declines | `aidd telemetry forget --yes` was never run | -| The person confirms | `aidd telemetry forget --yes` ran once, after the preview, and a second preview lists nothing | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/identity.md b/plugins/aidd-telemetry/skills/00-init/actions/identity.md deleted file mode 100644 index 6c8f23bd2..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/identity.md +++ /dev/null @@ -1,27 +0,0 @@ -# Identity - -Let the person choose to be named on their own measurement, or stop being named. - -## Input - -What the person wants: to be named as an identifier they give, to see the current choice, or to be named no more. - -## Output - -The identity set, shown or removed, as the command reported it. - -## Process - -1. **Explain.** Say that an identity is optional, that it only labels the person axis of their own report, and that it is kept in one file on this machine. -2. **Show.** Run `aidd telemetry identity` to read the current choice, and tell the person what it holds. -3. **Name.** To set one, ask the person for the identifier they want and wait for it. Run `aidd telemetry identity ` with exactly that text. - - The command refuses an identifier that is empty, over 128 characters, or spans lines: show its message and ask again. -4. **Unname.** To remove it, confirm that the person wants to be named no more, then run `aidd telemetry identity --off`. - -## Test - -| Case | Pass | -| --- | --- | -| The person gives no identifier | Nothing was set and none was invented | -| An identifier is given | `aidd telemetry identity` afterwards prints that identifier | -| Removal is confirmed | `aidd telemetry identity` afterwards says no identity is set | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/otel.md b/plugins/aidd-telemetry/skills/00-init/actions/otel.md deleted file mode 100644 index 3d4dd07b2..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/otel.md +++ /dev/null @@ -1,26 +0,0 @@ -# Configure provider OpenTelemetry - -## Input - -The provider (`claude`, `codex` or `opencode`) and destination (`dev` or `prod`) chosen by the person. - -## Output - -The selected provider and destination, any required restart or version limitation, and whether configuration changed, or the CLI's refusal. - -## Process - -1. Run `aidd telemetry otel init --help`. If unavailable, say that `@ai-driven-dev/cli` needs installing or updating and that nothing changed. -2. Obtain both choices. Explain that the selected provider sends native OpenTelemetry to the chosen collector and that raw telemetry can contain sensitive data, including secrets. The private development collector listens at `http://127.0.0.1:14318` and requires a bearer token in `~/.local/share/aidd-otel-observation/collector-token`. Production is refused until its destination is configured. This opt-in is separate from the local AIDD usage ledger. -3. Run `aidd telemetry otel init --config= --endpoint=` only after the person explicitly requests enabling that export. A request naming the provider and destination is sufficient; do not ask for it again. Let the CLI validate configuration and credentials. Never ask the person to paste a token into chat. -4. Report the configured destination and any warnings or the refusal. Keep credentials and the full merged configuration out of the report. A successful configuration does not establish that the collector received telemetry. - -## Test - -| Case | Pass | -| --- | --- | -| Missing choice or consent | No init command ran | -| Approved provider and destination | One init command ran with both choices, and its result was reported | -| CLI refused the setup | The refusal was reported without retrying or editing files | -| Production destination unavailable | The CLI's refusal is reported and no configuration is changed | -| Existing local AIDD measurement | No measurement, identity or deletion command runs | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/stop.md b/plugins/aidd-telemetry/skills/00-init/actions/stop.md deleted file mode 100644 index af46127ab..000000000 --- a/plugins/aidd-telemetry/skills/00-init/actions/stop.md +++ /dev/null @@ -1,24 +0,0 @@ -# Stop - -Turn measurement off for the current project and keep what was already measured. - -## Input - -The project directory. - -## Output - -Measurement is off for the project, and the person knows the measured data is still there. - -## Process - -1. **Confirm.** Say what stopping does: new Claude Code or Codex work in this project is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. Wait for a yes. -2. **Turn off.** Run `aidd telemetry off` and relay its message. -3. **Offer the rest.** Say that deleting what was measured is a separate step, and that it needs its own confirmation. - -## Test - -| Case | Pass | -| --- | --- | -| The person does not confirm | No `aidd telemetry off` was run | -| Confirmed | The command's message was relayed and the data was not deleted | diff --git a/scripts/__tests__/architecture-rules.test.js b/scripts/__tests__/architecture-rules.test.js index 35d0a1ad9..a9e1f3ae8 100644 --- a/scripts/__tests__/architecture-rules.test.js +++ b/scripts/__tests__/architecture-rules.test.js @@ -277,6 +277,72 @@ test("a skill holding action files with no Actions section at all is refused", ( assert.match(violations[0].message, /no "## Actions" section/); }); +const leanRouterPath = "plugins/aidd-fixture-a/skills/01-lean/SKILL.md"; +const leanRouter = [ + "# Lean skill", + "", + "```mermaid", + "flowchart LR", + ' first["first
actions/01-first.md"] --> second["second
actions/02-second.md"]', + "```", + "", + "## Transversal rules", + "", + "- Keep scope.", +].join("\n"); + +test("a preamble Mermaid flowchart names its actions without a heading or table", () => { + assert.deepEqual(checkArchitecture(leanRouterPath, leanRouter, ["01-first.md", "02-second.md"]), []); +}); + +test("removing an action path from a headerless flow still names the missing action", () => { + const content = leanRouter.replace("
actions/02-second.md", ""); + const violations = checkArchitecture(leanRouterPath, content, ["01-first.md", "02-second.md"]); + assert.equal(violations.length, 1); + assert.match(violations[0].message, /never names action file "02-second\.md"/); + assert.equal(violations[0].line, 3); +}); + +test("a missing headerless action cannot be supplied by prose, an example or another flow", () => { + const incomplete = leanRouter.replace("
actions/02-second.md", ""); + for (const extra of [ + "actions/02-second.md", + "```md\nactions/02-second.md\n```", + '```mermaid\nflowchart LR\n second["actions/02-second.md"]\n```', + ]) { + for (const content of [ + incomplete.replace("## Transversal rules", `${extra}\n\n## Transversal rules`), + `${incomplete}\n\n${extra}`, + ]) { + const violations = checkArchitecture(leanRouterPath, content, ["01-first.md", "02-second.md"]); + assert.equal(violations.length, 1); + assert.match(violations[0].message, /never names action file "02-second\.md"/); + } + } +}); + +test("only a closed top-level preamble flowchart can replace the Actions section", () => { + for (const content of [ + leanRouter.replace("```\n\n## Transversal rules", "\n\n## Transversal rules"), + leanRouter.replace("```mermaid", "```md"), + leanRouter.replace("flowchart LR", "classDiagram"), + `## Examples\n\n${leanRouter}`, + `\`\`\`\`md\n${leanRouter}\n\`\`\`\``, + ]) { + const violations = checkArchitecture(leanRouterPath, content, ["01-first.md", "02-second.md"]); + assert.equal(violations.length, 1); + assert.match(violations[0].message, /no "## Actions" section/); + } +}); + +test("numbered paths in a headerless flow distinguish action files sharing a stem", () => { + const content = leanRouter.replaceAll("01-first.md", "01-plan.md").replaceAll("02-second.md", "04-plan.md"); + assert.deepEqual(checkArchitecture(leanRouterPath, content, ["01-plan.md", "04-plan.md"]), []); + const violations = checkArchitecture(leanRouterPath, content.replace("04-plan.md", "plan.md"), ["01-plan.md", "04-plan.md"]); + assert.equal(violations.length, 1); + assert.match(violations[0].message, /never names action file "04-plan\.md"/); +}); + test("a glossary table before the router does not blind the router's own column", () => { const content = [ "# Two table skill", diff --git a/scripts/lib/architecture-rules.js b/scripts/lib/architecture-rules.js index ea1183d00..24e379a1c 100644 --- a/scripts/lib/architecture-rules.js +++ b/scripts/lib/architecture-rules.js @@ -116,7 +116,7 @@ function checkOrthogonality(filePath, content) { ); } -// --- Reading a `## Actions` section -------------------------------------------------------- +// --- Reading a router --------------------------------------------------------------------- function fenceCharacter(line) { return line.match(FENCE_MARKER)?.[1][0] ?? null; @@ -157,12 +157,38 @@ function actionsSection(lines) { const nextHeading = after.findIndex((line) => SECOND_LEVEL_HEADING.test(line)); return { + label: "## Actions", headingLine: headingIndex + 1, startIndex: headingIndex + 1, endIndex: nextHeading === -1 ? lines.length : headingIndex + 1 + nextHeading, }; } +/** The first closed Mermaid flowchart before any H2 is a router; later examples cannot add citations. */ +function preambleFlowSection(lines) { + let fence = null; + for (let index = 0; index < lines.length; index += 1) { + const line = lines[index]; + if (fence === null) { + if (SECOND_LEVEL_HEADING.test(line)) return null; + const opening = line.match(/^ {0,3}(`{3,}|~{3,})(.*)$/); + if (opening) fence = { marker: opening[1], language: opening[2].trim(), start: index }; + continue; + } + + const closing = line.match(/^ {0,3}(`{3,}|~{3,})\s*$/); + if (!closing || closing[1][0] !== fence.marker[0] || closing[1].length < fence.marker.length) continue; + + const body = lines.slice(fence.start + 1, index); + const firstContent = body.find((value) => value.trim() && !/^\s*%%/.test(value)); + if (fence.language === "mermaid" && /^\s*(?:flowchart|graph)(?:\s+(?:TB|TD|BT|RL|LR))?\s*(?:;|$)/.test(firstContent ?? "")) { + return { label: "Mermaid router", headingLine: fence.start + 1, startIndex: fence.start + 1, endIndex: index }; + } + fence = null; + } + return null; +} + function tableCells(line) { return line.trim().replace(/^\|/, "").replace(/\|$/, "").split("|").map((cell) => cell.trim()); } @@ -280,14 +306,14 @@ function checkRouterCoherence(filePath, content, actionFileNames) { const rawLines = toLines(content); const lines = withoutFences(rawLines); - const section = actionsSection(lines); + const section = actionsSection(lines) ?? preambleFlowSection(rawLines); if (!section) { - const message = `${filePath} has action files but no "## Actions" section`; + const message = `${filePath} has action files but no "## Actions" section or preamble Mermaid flowchart`; return [violation("router-coherence", filePath, 1, surface.owner, message)]; } - const { startIndex, endIndex, headingLine } = section; + const { startIndex, endIndex, headingLine, label } = section; const sectionLines = lines.slice(startIndex, endIndex); const { cited, cutOff } = citationsFromActionColumns(sectionLines); @@ -309,7 +335,7 @@ function checkRouterCoherence(filePath, content, actionFileNames) { filePath, headingLine, surface.owner, - `${filePath}:${headingLine} "## Actions" never names action file "${name}"` + `${filePath}:${headingLine} "${label}" never names action file "${name}"` ) ); } From 5d87bcce9db02b8924138b3095529d59c8c001db Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 11:27:24 +0800 Subject: [PATCH 03/56] feat(cli): create a stable installation identifier --- cli/src/presentation/commands/setup.ts | 3 + cli/src/presentation/commands/telemetry.ts | 5 + .../filesystem/installation-identity.ts | 87 +++++++++++ cli/src/runtime/wiring/framework.ts | 5 + cli/src/runtime/wiring/telemetry-otel.ts | 15 +- cli/tests/e2e/clean-scope-user.e2e.test.ts | 10 +- .../e2e/installation-identity.e2e.test.ts | 136 ++++++++++++++++++ .../golden/snapshots/phase0/snapshot.json | 6 +- .../commands/setup-wiring.integration.test.ts | 30 ++++ .../telemetry-wiring.integration.test.ts | 16 +++ .../installation-identity.integration.test.ts | 93 ++++++++++++ 11 files changed, 401 insertions(+), 5 deletions(-) create mode 100644 cli/src/runtime/filesystem/installation-identity.ts create mode 100644 cli/tests/e2e/installation-identity.e2e.test.ts create mode 100644 cli/tests/runtime/filesystem/installation-identity.integration.test.ts diff --git a/cli/src/presentation/commands/setup.ts b/cli/src/presentation/commands/setup.ts index b789ffe52..78b6668fe 100644 --- a/cli/src/presentation/commands/setup.ts +++ b/cli/src/presentation/commands/setup.ts @@ -10,6 +10,7 @@ import { } from "../../contexts/tools/domain/registry.js"; import type { ToolId } from "../../kernel/tool.js"; import { AI_TOOL_IDS, IDE_TOOL_IDS } from "../../kernel/tool.js"; +import { finishInstallationId } from "../../runtime/filesystem/installation-identity.js"; import { createDeps } from "../../runtime/wiring/framework.js"; import { printDetectedContext, @@ -170,6 +171,7 @@ export function registerSetupCommand(program: Command): void { if (interactive) printWelcomeBanner(output); const deps = await createDeps(projectRoot, { verbose }, output); + await deps.checkInstallationId(); const result = await new SetupUseCase( deps.fs, @@ -190,6 +192,7 @@ export function registerSetupCommand(program: Command): void { printSetupOutcome(output, result, verbose); if (interactive) printNextSteps(output, result.install.results.length > 0); reportSyncActivation(output, result.activation); + await finishInstallationId(deps.ensureInstallationId, "Setup completed"); } catch (error) { errorHandler.handle(error); } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 1015977e1..6b8175c63 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -25,6 +25,7 @@ import { } from "../../contexts/telemetry/domain/report/usage-report.js"; import { USAGE_AGENTS, USAGE_TOOLS } from "../../contexts/telemetry/domain/usage-record.js"; import type { Prompter } from "../../kernel/ports/prompter.js"; +import { finishInstallationId } from "../../runtime/filesystem/installation-identity.js"; import { createDeps } from "../../runtime/wiring/framework.js"; import type { TelemetryDeps } from "../../runtime/wiring/telemetry.js"; import { @@ -248,7 +249,11 @@ export function registerTelemetryCommand(program: Command): void { ); process.exit(1); } + await deps.checkInstallationId(); const result = await deps.telemetry.telemetryOnUseCase.execute(projectRoot, choice); + if (result.status === "on") { + await finishInstallationId(deps.ensureInstallationId, "Measurement is on"); + } printOnResult(output, result); if (result.status === "refused") process.exit(1); }); diff --git a/cli/src/runtime/filesystem/installation-identity.ts b/cli/src/runtime/filesystem/installation-identity.ts new file mode 100644 index 000000000..574f8f8d0 --- /dev/null +++ b/cli/src/runtime/filesystem/installation-identity.ts @@ -0,0 +1,87 @@ +import { randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import { link, open, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { isErrnoException, tryParseJson } from "../../kernel/reading/json-file.js"; +import { asPlainObject } from "../../kernel/reading/plain-object.js"; +import { PrivateStorageAdapter } from "./private-storage-adapter.js"; + +const UUID_V4 = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +async function readInstallationId(path: string): Promise { + const handle = await open( + path, + process.platform === "win32" + ? constants.O_RDONLY + : constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (isErrnoException(error) && error.code === "ENOENT") return null; + throw error; + }); + if (handle === null) return null; + try { + const info = await handle.stat(); + if (!info.isFile() || (process.platform !== "win32" && (info.mode & 0o077) !== 0)) { + throw new Error("Installation identity must be a private regular file."); + } + const parsed = tryParseJson(await handle.readFile("utf8")); + const object = parsed.ok ? asPlainObject(parsed.value) : null; + const id = object?.installation_id; + if (typeof id !== "string" || !UUID_V4.test(id)) { + throw new Error("Installation identity is invalid; the existing file was left unchanged."); + } + return id; + } finally { + await handle.close(); + } +} + +/** Validate a pre-existing ID without creating any files. */ +export function checkInstallationId(userConfigDir: string): Promise { + return readInstallationId(join(userConfigDir, "installation", "id.json")); +} + +/** The caller's operation has already taken effect when this is called. */ +export async function finishInstallationId( + ensure: () => Promise, + outcome: string +): Promise { + try { + await ensure(); + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + throw new Error(`${outcome}, but the installation ID could not be saved: ${reason}. Retry.`); + } +} + +/** A machine-local installation identifier, not a person, credential or authentication token. + * A completed temporary file is linked into place without replacing a concurrent initializer. */ +export async function ensureInstallationId(userConfigDir: string): Promise { + const directory = join(userConfigDir, "installation"); + const path = join(directory, "id.json"); + await new PrivateStorageAdapter().ensureDirectory(directory); + const existing = await readInstallationId(path); + if (existing !== null) return existing; + + const id = randomUUID(); + const temporary = join(directory, `${id}.tmp`); + let created = false; + try { + await writeFile(temporary, `${JSON.stringify({ installation_id: id })}\n`, { + flag: "wx", + mode: 0o600, + }); + created = true; + try { + await link(temporary, path); + return id; + } catch (error) { + if (!isErrnoException(error) || error.code !== "EEXIST") throw error; + const winner = await readInstallationId(path); + if (winner === null) throw new Error("Installation identity disappeared during setup."); + return winner; + } + } finally { + if (created) await rm(temporary, { force: true }); + } +} diff --git a/cli/src/runtime/wiring/framework.ts b/cli/src/runtime/wiring/framework.ts index 59ce0367e..a48bee75a 100644 --- a/cli/src/runtime/wiring/framework.ts +++ b/cli/src/runtime/wiring/framework.ts @@ -105,6 +105,7 @@ import type { CredentialStore } from "../auth/ports/credential-store.js"; import { atomicWriteFile } from "../filesystem/atomic-write.js"; import { FileAdapter } from "../filesystem/file-adapter.js"; import { HasherAdapter } from "../filesystem/hasher-adapter.js"; +import { checkInstallationId, ensureInstallationId } from "../filesystem/installation-identity.js"; import { HttpClient } from "../http/http-client.js"; import { PlatformAdapter } from "../platform/platform-adapter.js"; import { InquirerPrompterAdapter, SilentPrompterAdapter } from "../prompter/prompter-adapter.js"; @@ -126,6 +127,8 @@ interface GlobalOptions { } interface Deps { + checkInstallationId: () => Promise; + ensureInstallationId: () => Promise; fs: FileReader & FileWriter & FileMerger; manifestRepo: ManifestRepository; /** `--scope user`'s own manifest repository: `userConfigDir()/manifest.json`, never nested @@ -589,6 +592,8 @@ export async function createDeps( const listInstalledRulesUseCase = new ListInstalledRulesUseCase(fs); const checkUpdateUseCase = new CheckUpdateUseCase(cliUpdater, currentVersionProvider, logger, fs); const deps: Deps = { + checkInstallationId: () => checkInstallationId(userConfigDir()), + ensureInstallationId: () => ensureInstallationId(userConfigDir()), validateRecipesUseCase: new ValidateRecipesUseCase(new RecipeFilesAdapter()), fs, manifestRepo, diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index 1f07491e4..2e9b06cc6 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -7,6 +7,12 @@ import { OTEL_CONFIGS, OTEL_ENDPOINTS, } from "../../contexts/telemetry/infrastructure/otel/settings.js"; +import { + checkInstallationId, + ensureInstallationId, + finishInstallationId, +} from "../filesystem/installation-identity.js"; +import { userConfigDirOf } from "../user-config-dir.js"; export const otelOptions = { configs: OTEL_CONFIGS, @@ -14,11 +20,16 @@ export const otelOptions = { launchConfigs: [OPENCODE_CONFIG_ID], }; -export function configureOtel( +export async function configureOtel( config: (typeof OTEL_CONFIGS)[number], endpoint: (typeof OTEL_ENDPOINTS)[number] ) { - return initializeOtel(config, endpoint, { home: homedir(), env: process.env }); + const home = homedir(); + const directory = userConfigDirOf(process.env, home); + if (endpoint !== "prod") await checkInstallationId(directory); + const result = await initializeOtel(config, endpoint, { home, env: process.env }); + await finishInstallationId(() => ensureInstallationId(directory), "OTLP configuration completed"); + return result; } export function launchOtel(args: readonly string[]) { diff --git a/cli/tests/e2e/clean-scope-user.e2e.test.ts b/cli/tests/e2e/clean-scope-user.e2e.test.ts index fd4119d91..d3f79ffda 100644 --- a/cli/tests/e2e/clean-scope-user.e2e.test.ts +++ b/cli/tests/e2e/clean-scope-user.e2e.test.ts @@ -89,6 +89,8 @@ describe("E2E: clean --scope user purges the shared source", () => { expect(setupResult.exitCode).toBe(0); const userConfigDir = join(fakeHome, ".config", "aidd"); + const installationIdPath = join(userConfigDir, "installation", "id.json"); + const installationIdBefore = await readFile(installationIdPath, "utf-8"); const marketplacesPath = join(userConfigDir, "marketplaces.json"); const beforeMarketplaces = await readJson(marketplacesPath); const beforeNames = (beforeMarketplaces.marketplaces as Array<{ name: string }>).map( @@ -150,8 +152,14 @@ describe("E2E: clean --scope user purges the shared source", () => { // whitelist has no business under fails here rather than going unnoticed. expect(before).not.toEqual(after); expect(after).toEqual( - ["auth.json", "marketplaces.json", join("telemetry", "2026-01-01.jsonl")].sort() + [ + "auth.json", + "marketplaces.json", + join("installation", "id.json"), + join("telemetry", "2026-01-01.jsonl"), + ].sort() ); + expect(await readFile(installationIdPath, "utf-8")).toBe(installationIdBefore); const afterMarketplaces = await readJson(marketplacesPath); const afterNames = (afterMarketplaces.marketplaces as Array<{ name: string }>).map( (m) => m.name diff --git a/cli/tests/e2e/installation-identity.e2e.test.ts b/cli/tests/e2e/installation-identity.e2e.test.ts new file mode 100644 index 000000000..8474bd8cc --- /dev/null +++ b/cli/tests/e2e/installation-identity.e2e.test.ts @@ -0,0 +1,136 @@ +import { execFileSync } from "node:child_process"; +import { mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createTestEnv, FRAMEWORK_PATH, runCli } from "./helpers.js"; + +describe.skipIf(process.platform === "win32")("installation identity across commands", () => { + it("reuses the setup ID for telemetry and keeps a manually chosen person ID", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-id"); + const userConfig = join(fakeHome, ".config", "aidd"); + const path = join(userConfig, "installation", "id.json"); + const person = join(userConfig, "telemetry", "identity.json"); + const collector = join(fakeHome, ".local", "share", "aidd-otel-observation"); + try { + await mkdir(join(userConfig, "telemetry"), { recursive: true }); + await writeFile(person, '{"person_id":"existing-person"}\n', { mode: 0o600 }); + const setup = [ + "setup", + "--source", + "local", + "--path", + FRAMEWORK_PATH, + "--ai", + "claude", + "--plugins", + "none", + "--yes", + ]; + expect((await runCli(setup, projectDir, fakeHome)).exitCode).toBe(0); + const first = await readFile(path, "utf8"); + expect((await runCli(setup, projectDir, fakeHome)).exitCode).toBe(0); + expect(await readFile(path, "utf8")).toBe(first); + execFileSync("git", ["init", "--quiet"], { cwd: projectDir }); + const turnOn = await runCli(["telemetry", "on", "--yes"], projectDir, fakeHome); + expect(turnOn.exitCode, turnOn.stderr).toBe(0); + expect(await readFile(path, "utf8")).toBe(first); + await mkdir(collector, { recursive: true }); + await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); + expect( + ( + await runCli( + ["telemetry", "otel", "init", "--config=codex", "--endpoint=dev"], + projectDir, + fakeHome + ) + ).exitCode + ).toBe(0); + expect(await readFile(path, "utf8")).toBe(first); + expect(await readFile(person, "utf8")).toBe('{"person_id":"existing-person"}\n'); + } finally { + await cleanup(); + } + }); + + it("does not create an installation ID when OTLP production initialization is refused", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-refused"); + try { + const result = await runCli( + ["telemetry", "otel", "init", "--config=codex", "--endpoint=prod"], + projectDir, + fakeHome + ); + expect(result.exitCode).not.toBe(0); + await expect( + stat(join(fakeHome, ".config", "aidd", "installation", "id.json")) + ).rejects.toThrow(); + } finally { + await cleanup(); + } + }); + + it("creates and reuses the ID through OTLP initialization for an existing installation", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-otel"); + const path = join(fakeHome, ".config", "aidd", "installation", "id.json"); + const collector = join(fakeHome, ".local", "share", "aidd-otel-observation"); + try { + await mkdir(collector, { recursive: true }); + await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); + const args = ["telemetry", "otel", "init", "--config=codex", "--endpoint=dev"]; + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + const initial = await readFile(path, "utf8"); + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + expect(await readFile(path, "utf8")).toBe(initial); + } finally { + await cleanup(); + } + }); + + it("creates the ID when measurement is enabled and not when it is refused", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-on"); + const path = join(fakeHome, ".config", "aidd", "installation", "id.json"); + const args = ["telemetry", "on", "--yes", "--anonymous"]; + try { + expect((await runCli(args, projectDir, fakeHome)).exitCode).not.toBe(0); + await expect(stat(path)).rejects.toThrow(); + execFileSync("git", ["init", "--quiet"], { cwd: projectDir }); + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + const initial = await readFile(path, "utf8"); + expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + expect(await readFile(path, "utf8")).toBe(initial); + } finally { + await cleanup(); + } + }); + + it("does not initialize a project over an invalid existing installation ID", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("installation-invalid"); + const directory = join(fakeHome, ".config", "aidd", "installation"); + const path = join(directory, "id.json"); + try { + await mkdir(directory, { recursive: true }); + await writeFile(path, '{"installation_id":"invalid"}', { mode: 0o600 }); + const result = await runCli( + [ + "setup", + "--source", + "local", + "--path", + FRAMEWORK_PATH, + "--ai", + "claude", + "--plugins", + "none", + "--yes", + ], + projectDir, + fakeHome + ); + expect(result.exitCode).not.toBe(0); + await expect(stat(join(projectDir, ".aidd", "manifest.json"))).rejects.toThrow(); + expect(await readFile(path, "utf8")).toBe('{"installation_id":"invalid"}'); + } finally { + await cleanup(); + } + }); +}); diff --git a/cli/tests/golden/snapshots/phase0/snapshot.json b/cli/tests/golden/snapshots/phase0/snapshot.json index 325cc6a4e..b79e6ebec 100644 --- a/cli/tests/golden/snapshots/phase0/snapshot.json +++ b/cli/tests/golden/snapshots/phase0/snapshot.json @@ -55,6 +55,7 @@ "cache/built//aidd-framework/claude/plugins/aidd-test/hooks/session-start.js", "cache/built//aidd-framework/claude/plugins/aidd-test/skills/commit/SKILL.md", "cache/built//aidd-framework/claude/plugins/aidd-test/skills/hello.md", + "installation/id.json", "manifest.json", "marketplaces.json", "references.json" @@ -704,6 +705,7 @@ "cache/built//aidd-framework/cursor/plugins/aidd-test/hooks/session-start.js", "cache/built//aidd-framework/cursor/plugins/aidd-test/skills/commit/SKILL.md", "cache/built//aidd-framework/cursor/plugins/aidd-test/skills/hello.md", + "installation/id.json", "manifest.json", "marketplaces.json", "references.json" @@ -827,7 +829,7 @@ "stderr": "Warning: claude CLI not found on PATH โ€” skipping native plugin activation.\n", "filesWritten": [], "manifest": null, - "userConfigFiles": ["manifest.json", "marketplaces.json"], + "userConfigFiles": ["installation/id.json", "manifest.json", "marketplaces.json"], "userManifest": { "version": 8, "tools": { @@ -847,7 +849,7 @@ "stderr": "", "filesWritten": [], "manifest": null, - "userConfigFiles": ["marketplaces.json"], + "userConfigFiles": ["installation/id.json", "marketplaces.json"], "userManifest": null }, { diff --git a/cli/tests/presentation/commands/setup-wiring.integration.test.ts b/cli/tests/presentation/commands/setup-wiring.integration.test.ts index ea31fe91e..9e3ed8567 100644 --- a/cli/tests/presentation/commands/setup-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/setup-wiring.integration.test.ts @@ -20,9 +20,13 @@ const setupToolsPrompt = vi.fn(); const detectContext = vi.fn(); const setupMachineScope = vi.fn(); const loadManifest = vi.fn(); +const ensureInstallationId = vi.fn(); +const checkInstallationId = vi.fn(); vi.mock("../../../src/runtime/wiring/framework.js", () => ({ createDeps: vi.fn(async () => ({ + ensureInstallationId, + checkInstallationId, fs: {}, manifestRepo: { load: loadManifest }, setupMarketplaceRegistration: { resolveSourceIfNeeded, registerIfPresent }, @@ -98,6 +102,7 @@ async function run(...args: string[]): Promise { describe("aidd setup โ€” the flow a scripted run builds", () => { it("names the tools it was given, installs no plugin, and registers the default source", async () => { expect(await run("--ai", "claude")).toEqual(["Project is up to date."]); + expect(ensureInstallationId).toHaveBeenCalledOnce(); expect({ ...builtFlow() }).toEqual({ projectRoot: PROJECT_ROOT, source: undefined, @@ -355,6 +360,7 @@ describe("aidd setup โ€” what it refuses and what it reports", () => { "Warning: [claude] claude CLI refused\nError: Sync failed for: claude. See the warnings above.\n" ); expect(exit).toHaveBeenCalledWith(1); + expect(ensureInstallationId).not.toHaveBeenCalled(); }); it("names a failed install on stderr and fails the process", async () => { @@ -367,6 +373,30 @@ describe("aidd setup โ€” what it refuses and what it reports", () => { expect(errors.join("")).toBe("Error: assets are missing\n"); expect(exit).toHaveBeenCalledWith(1); + expect(ensureInstallationId).not.toHaveBeenCalled(); + }); + + it("refuses an invalid existing installation ID before setup changes anything", async () => { + checkInstallationId.mockRejectedValueOnce(new Error("invalid installation ID")); + vi.spyOn(process, "exit").mockImplementation(() => { + throw new Error("exited"); + }); + await expect(run("--ai", "claude")).rejects.toThrow("exited"); + expect(setupTools).not.toHaveBeenCalled(); + expect(ensureInstallationId).not.toHaveBeenCalled(); + expect(errors.join("")).toContain("invalid installation ID"); + }); + + it("reports that setup completed if saving the installation ID fails afterward", async () => { + ensureInstallationId.mockRejectedValueOnce(new Error("disk full")); + vi.spyOn(process, "exit").mockImplementation(() => { + throw new Error("exited"); + }); + await expect(run("--ai", "claude")).rejects.toThrow("exited"); + expect(setupTools).toHaveBeenCalled(); + expect(errors.join("")).toContain( + "Setup completed, but the installation ID could not be saved: disk full" + ); }); it("builds the graph for this project at this run's verbosity", async () => { diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index ac64b9652..0381421fd 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -19,9 +19,13 @@ const confirm = vi.fn(); const select = vi.fn(); const input = vi.fn(); const propose = vi.fn(); +const ensureInstallationId = vi.fn(); +const checkInstallationId = vi.fn(); vi.mock("../../../src/runtime/wiring/framework.js", () => ({ createDeps: vi.fn(async () => ({ + ensureInstallationId, + checkInstallationId, prompter: { confirm, select, input }, telemetry: { telemetryOnUseCase: { execute: turnOn }, @@ -434,6 +438,7 @@ describe("aidd telemetry on", () => { it("turns measurement on in the directory it was run from, without asking, with --yes and a choice", async () => { expect(await run("on", "--yes", "--anonymous")).toBeNull(); expect(turnOn).toHaveBeenCalledWith(process.cwd(), "none"); + expect(ensureInstallationId).toHaveBeenCalledOnce(); expect(confirm).not.toHaveBeenCalled(); expect(stdout()).toContain("Measurement is on"); }); @@ -455,6 +460,7 @@ describe("aidd telemetry on", () => { expect(await run("on", "--yes")).toBe(1); expect(turnOn).not.toHaveBeenCalled(); expect(propose).not.toHaveBeenCalled(); + expect(ensureInstallationId).not.toHaveBeenCalled(); expect(stderr()).toContain("--as "); expect(stderr()).toContain("--anonymous"); }); @@ -584,6 +590,16 @@ describe("aidd telemetry on", () => { turnOn.mockResolvedValue({ status: "refused", reason: "identity-set" }); expect(await run("on", "--yes", "--anonymous")).toBe(1); expect(stderr()).toContain("aidd telemetry identity --off"); + expect(ensureInstallationId).not.toHaveBeenCalled(); + }); + + it("reports that measurement is on if saving the installation ID fails afterward", async () => { + ensureInstallationId.mockRejectedValueOnce(new Error("disk full")); + expect(await run("on", "--yes", "--anonymous")).toBe(1); + expect(turnOn).toHaveBeenCalled(); + expect(stderr()).toContain( + "Measurement is on, but the installation ID could not be saved: disk full" + ); }); }); diff --git a/cli/tests/runtime/filesystem/installation-identity.integration.test.ts b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts new file mode 100644 index 000000000..b9ca02b29 --- /dev/null +++ b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts @@ -0,0 +1,93 @@ +import { chmod, mkdir, mkdtemp, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + checkInstallationId, + ensureInstallationId, + finishInstallationId, +} from "../../../src/runtime/filesystem/installation-identity.js"; + +const roots: string[] = []; +async function makeRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "aidd-installation-id-")); + roots.push(root); + return root; +} +afterEach(async () => { + await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); +}); + +describe("installation identity", () => { + it("creates one private UUID across repeated and concurrent initializations", async () => { + const root = await makeRoot(); + const ids = await Promise.all(Array.from({ length: 12 }, () => ensureInstallationId(root))); + expect(new Set(ids).size).toBe(1); + expect(ids[0]).toMatch( + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i + ); + expect(await ensureInstallationId(root)).toBe(ids[0]); + const path = join(root, "installation", "id.json"); + expect(JSON.parse(await readFile(path, "utf8"))).toEqual({ installation_id: ids[0] }); + if (process.platform !== "win32") expect((await stat(path)).mode & 0o777).toBe(0o600); + }); + + it("preserves a previously stored UUID and any additional fields", async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + const existing = '{"installation_id":"a720868e-d80a-4c71-96aa-2ce56936b35d","future":"keep"}\n'; + await mkdir(directory); + await writeFile(join(directory, "id.json"), existing, { mode: 0o600 }); + expect(await ensureInstallationId(root)).toBe("a720868e-d80a-4c71-96aa-2ce56936b35d"); + expect(await readFile(join(directory, "id.json"), "utf8")).toBe(existing); + }); + + it("refuses an invalid existing identity without replacing it", async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + await writeFile(join(directory, "id.json"), '{"installation_id":"invalid"}', { mode: 0o600 }); + await expect(checkInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + await expect(ensureInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + expect(await readFile(join(directory, "id.json"), "utf8")).toBe( + '{"installation_id":"invalid"}' + ); + }); + + it.skipIf(process.platform === "win32")("refuses a symlink without following it", async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + const target = join(root, "target.json"); + await writeFile(target, "untouched", { mode: 0o600 }); + await symlink(target, join(directory, "id.json")); + await expect(ensureInstallationId(root)).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("untouched"); + }); + + it.skipIf(process.platform === "win32")( + "refuses an existing file with broad permissions", + async () => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + await writeFile( + join(directory, "id.json"), + '{"installation_id":"a720868e-d80a-4c71-96aa-2ce56936b35d"}', + { mode: 0o644 } + ); + await chmod(join(directory, "id.json"), 0o644); + await expect(ensureInstallationId(root)).rejects.toThrow("private regular file"); + } + ); + + it("reports an already applied OTLP configuration if identity creation then fails", async () => { + await expect( + finishInstallationId(async () => { + throw new Error("disk full"); + }, "OTLP configuration completed") + ).rejects.toThrow( + "OTLP configuration completed, but the installation ID could not be saved: disk full" + ); + }); +}); From 315c93f069d3001d02d1d9d842df044bbe42cda1 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 11:28:30 +0800 Subject: [PATCH 04/56] feat(cli): normalize native telemetry observations --- cli/aidd_docs/memory/codebase-map.md | 1 + cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 2 + .../observations/telemetry-observation.ts | 79 +++ .../infrastructure/otel/actions/verify.ts | 58 +- .../infrastructure/otel/observations.ts | 504 ++++++++++++++++++ .../telemetry/infrastructure/otel/settings.ts | 8 +- cli/src/presentation/commands/telemetry.ts | 3 + .../observations/transformers.unit.test.ts | 360 +++++++++++++ .../otel/init-otel.integration.test.ts | 12 + .../otel/verify-otel.integration.test.ts | 54 ++ cli/tests/e2e/telemetry-otel.e2e.test.ts | 28 +- plugins/aidd-telemetry/OBSERVABILITY.md | 46 ++ plugins/aidd-telemetry/README.md | 3 +- 14 files changed, 1152 insertions(+), 8 deletions(-) create mode 100644 cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/observations.ts create mode 100644 cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts create mode 100644 plugins/aidd-telemetry/OBSERVABILITY.md diff --git a/cli/aidd_docs/memory/codebase-map.md b/cli/aidd_docs/memory/codebase-map.md index fdf195e31..afa687e78 100644 --- a/cli/aidd_docs/memory/codebase-map.md +++ b/cli/aidd_docs/memory/codebase-map.md @@ -63,6 +63,7 @@ src/ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # a clone's identity, and when it consented โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ identity/ # the identifier a person chose to be named by โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ legacy/ # what the previous version wrote, by name and location, and its key and id rules +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ observations/ # native telemetry fields before display or aggregation โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ports/ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ bindings/ # the declarations kept beside the ledger, and the lock their writers share โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ forget/ diff --git a/cli/package.json b/cli/package.json index d382d958f..56e38fbdb 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 729, + "bundleBudgetKB": 746, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index ce560d9af..0ed90cbe8 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -51,6 +51,8 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 0.2% measured headroom. // 728 KB: 726.7 KB, explicit native OTLP configuration for three tools and the // OpenCode 1 launcher (+15.9 KB over the 710.8 KB build above); 0.2% measured headroom. +// 746 KB: 744.9 KB, provider observation transformers and a persistent installation ID +// (+18.2 KB over the 726.7 KB build above); 0.1% measured headroom. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts b/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts new file mode 100644 index 000000000..3caabf2a1 --- /dev/null +++ b/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts @@ -0,0 +1,79 @@ +import type { AiToolId } from "../../../../kernel/tool.js"; + +export type JsonValue = + | null + | boolean + | number + | string + | readonly JsonValue[] + | { readonly [key: string]: JsonValue }; + +/** A source observation. Missing values stay missing; identical observations are not merged. */ +export interface TelemetryObservation { + readonly tool: AiToolId; + readonly signal: "log" | "span" | "span-event" | "metric" | "event"; + readonly eventType?: string; + readonly timestamp?: string; + readonly observedTimestamp?: string; + readonly startedAt?: string; + readonly endedAt?: string; + readonly toolVersion?: string; + readonly openCodeVersion?: 1 | 2; + readonly sessionId?: string; + readonly threadId?: string; + readonly messageId?: string; + readonly partId?: string; + readonly traceId?: string; + readonly spanId?: string; + readonly parentSpanId?: string; + readonly requestId?: string; + readonly provider?: string; + readonly model?: string; + readonly role?: string; + readonly prompt?: string; + readonly response?: string; + /** Text whose user/assistant role cannot be established from this event alone. */ + readonly text?: string; + readonly toolCallId?: string; + readonly runtimeToolCallId?: string; + readonly toolName?: string; + readonly toolInput?: JsonValue; + readonly toolOutput?: JsonValue; + readonly success?: boolean; + readonly error?: JsonValue; + readonly durationMs?: number; + readonly cwd?: string; + readonly gitBranch?: string; + readonly tokens?: { + readonly input?: number; + readonly inputNonCached?: number; + readonly output?: number; + readonly cacheRead?: number; + readonly cacheWrite?: number; + readonly cacheWrite5m?: number; + readonly cacheWrite1h?: number; + readonly reasoning?: number; + readonly total?: number; + readonly tool?: number; + }; + /** Snapshots and aggregates must not be summed as model calls. */ + readonly usageScope?: + | "event" + | "span-snapshot" + | "step-snapshot" + | "message-snapshot" + | "session-cumulative" + | "metric-aggregate"; + readonly tokenInputIncludesCache?: boolean; + readonly tokenOutputIncludesReasoning?: boolean; + /** OpenCode substitutes zero when an upstream token count was not supplied. */ + readonly zeroMayMeanMissing?: boolean; + /** As reported by the source, not a calculated or validated price. */ + readonly reportedCostUsd?: number; + readonly reportedCostUsdMicros?: number; + readonly metricName?: string; + readonly metricType?: string; + readonly metricValue?: number; + /** The original observation, retained for fields not yet selected for a display. */ + readonly source?: JsonValue; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts index 9ecd9bfce..1e3e9348c 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts @@ -2,7 +2,14 @@ import { constants } from "node:fs"; import { open } from "node:fs/promises"; import { join } from "node:path"; import { readSettings } from "../files.js"; -import { type Config, configurationEdit, type Endpoint, type Host } from "../settings.js"; +import { transformCapturedLine } from "../observations.js"; +import { + type Config, + configurationEdit, + type Endpoint, + type Host, + OPENCODE_CONFIG_ID, +} from "../settings.js"; export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) { const edit = await configurationEdit(config, endpoint, host); @@ -29,11 +36,56 @@ export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) const info = await handle.stat(); if (!info.isFile()) throw new Error("OTLP capture must be a regular file."); let lines = 0; + let observations = 0; + let unrecognisedLines = 0; for await (const line of handle.readLines({ autoClose: false })) { - if (line.trim()) lines++; + if (!line.trim()) continue; + lines++; + const count = transformCapturedLine(config, line).length; + observations += count; + if (count === 0) unrecognisedLines++; } if (lines === 0) throw new Error("OTLP capture is empty. Run a tool session and retry."); - return { path: edit.path, capture, lines, modifiedAt: info.mtime.toISOString() }; + let eventLines = 0; + if (config === OPENCODE_CONFIG_ID) { + const eventsPath = join( + host.home, + ".local/share/aidd-otel-observation", + OPENCODE_CONFIG_ID, + "events.jsonl" + ); + const events = await open( + eventsPath, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (events) { + try { + if (!(await events.stat()).isFile()) + throw new Error("OpenCode event capture must be a regular file."); + for await (const line of events.readLines({ autoClose: false })) { + if (!line.trim()) continue; + eventLines++; + const count = transformCapturedLine(config, line).length; + observations += count; + if (count === 0) unrecognisedLines++; + } + } finally { + await events.close(); + } + } + } + return { + path: edit.path, + capture, + lines, + eventLines, + observations, + unrecognisedLines, + modifiedAt: info.mtime.toISOString(), + }; } finally { await handle.close(); } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/observations.ts b/cli/src/contexts/telemetry/infrastructure/otel/observations.ts new file mode 100644 index 000000000..060bf00a9 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/observations.ts @@ -0,0 +1,504 @@ +import { asPlainObject } from "../../../../kernel/reading/plain-object.js"; +import type { + JsonValue, + TelemetryObservation, +} from "../../domain/observations/telemetry-observation.js"; +import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type Config, OPENCODE_CONFIG_ID } from "./settings.js"; + +interface OtelCapture { + readonly signal: "logs" | "traces" | "metrics"; + readonly payload: unknown; + readonly received_at?: string; +} +export interface ClaudeTelemetrySource extends OtelCapture {} +export interface CodexTelemetrySource extends OtelCapture {} +export interface OpenCodeV1Source { + readonly type: string; + readonly properties: unknown; +} +export interface OpenCodeV2Source { + readonly type: string; + readonly data: unknown; + readonly created?: number; + readonly location?: unknown; +} +export type OpenCodeTelemetrySource = OpenCodeV1Source | OpenCodeV2Source | OtelCapture; + +type Obj = Record; +function json(value: unknown): JsonValue | undefined { + if (value === null || typeof value === "boolean" || typeof value === "string") return value; + if (typeof value === "number") return Number.isFinite(value) ? value : undefined; + if (Array.isArray(value)) return value.map((item) => json(item) ?? null); + const input = asPlainObject(value); + if (input === null) return undefined; + const result: Record = {}; + for (const [key, entry] of Object.entries(input)) { + const parsed = json(entry); + if (parsed !== undefined) result[key] = parsed; + } + return result; +} +const object = (value: unknown): Obj => asPlainObject(value) ?? {}; +const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); +const text = (value: unknown): string | undefined => + typeof value === "string" && value !== "" ? value : undefined; +function number(value: unknown): number | undefined { + if (typeof value !== "number" && (typeof value !== "string" || value.trim() === "")) + return undefined; + const n = Number(value); + return Number.isFinite(n) && n >= 0 ? n : undefined; +} +function bool(value: unknown): boolean | undefined { + if (value === true || value === "true") return true; + if (value === false || value === "false") return false; + return undefined; +} +function nanos(value: unknown): string | undefined { + try { + if (typeof value !== "string" && typeof value !== "number") return undefined; + const n = BigInt(value); + if (n <= 0n) return undefined; + return new Date(Number(n / 1_000_000n)).toISOString(); + } catch { + return undefined; + } +} +function millis(value: unknown): string | undefined { + const n = number(value); + if (n === undefined) return undefined; + try { + return new Date(n).toISOString(); + } catch { + return undefined; + } +} +function decodeValue(raw: unknown): unknown { + const item = object(raw); + for (const key of ["stringValue", "boolValue", "intValue", "doubleValue"]) { + if (key in item) return item[key]; + } + if (item.arrayValue) return list(object(item.arrayValue).values).map(decodeValue); + if (item.kvlistValue) return attributes(object(item.kvlistValue).values); + return undefined; +} +function attributes(raw: unknown): Obj { + const result: Obj = {}; + for (const entry of list(raw)) { + const item = object(entry); + const key = text(item.key); + if (key !== undefined) result[key] = decodeValue(item.value); + } + return result; +} +function first(...values: (T | undefined)[]): T | undefined { + return values.find((item) => item !== undefined); +} +function pickText(attributes: Obj, ...keys: string[]): string | undefined { + return first(...keys.map((key) => text(attributes[key]))); +} +function pickNumber(attributes: Obj, ...keys: string[]): number | undefined { + return first(...keys.map((key) => number(attributes[key]))); +} +function tokens(fields: TelemetryObservation["tokens"]): TelemetryObservation["tokens"] { + return fields && Object.values(fields).some((item) => item !== undefined) ? fields : undefined; +} +function base( + tool: TelemetryObservation["tool"], + signal: TelemetryObservation["signal"], + a: Obj, + source: unknown +): TelemetryObservation { + const usage = tokens({ + input: pickNumber( + a, + "input_tokens", + "input_token_count", + "codex.turn.token_usage.input_tokens", + "gen_ai.usage.input_tokens", + "ai.usage.inputTokens" + ), + inputNonCached: pickNumber( + a, + "codex.turn.token_usage.non_cached_input_tokens", + "ai.usage.inputTokenDetails.noCacheTokens" + ), + output: pickNumber( + a, + "output_tokens", + "output_token_count", + "codex.turn.token_usage.output_tokens", + "gen_ai.usage.output_tokens", + "ai.usage.outputTokens" + ), + cacheRead: pickNumber( + a, + "cache_read_tokens", + "cached_token_count", + "codex.turn.token_usage.cached_input_tokens", + "gen_ai.usage.cache_read.input_tokens", + "ai.usage.cachedInputTokens", + "ai.usage.inputTokenDetails.cacheReadTokens" + ), + cacheWrite: pickNumber( + a, + "cache_creation_tokens", + "cache_write_token_count", + "codex.turn.token_usage.cache_write_input_tokens", + "gen_ai.usage.cache_write.input_tokens" + ), + cacheWrite5m: number(a.cache_creation_5m_tokens), + cacheWrite1h: number(a.cache_creation_1h_tokens), + reasoning: pickNumber( + a, + "reasoning_token_count", + "codex.turn.token_usage.reasoning_output_tokens", + "codex.usage.reasoning_output_tokens", + "ai.usage.reasoningTokens", + "ai.usage.outputTokenDetails.reasoningTokens" + ), + total: pickNumber( + a, + "codex.turn.token_usage.total_tokens", + "codex.usage.total_tokens", + "ai.usage.totalTokens" + ), + tool: number(a.tool_token_count), + }); + return { + tool, + signal, + source: json(source), + eventType: pickText(a, "event.name"), + timestamp: pickText(a, "event.timestamp"), + toolVersion: pickText(a, "app.version", "service.version"), + sessionId: pickText(a, "session.id", "conversation.id"), + threadId: text(a["thread.id"]), + provider: pickText( + a, + "anthropic.provider", + "provider_name", + "provider", + "gen_ai.provider.name", + "gen_ai.system", + "ai.model.provider" + ), + model: pickText(a, "model", "gen_ai.request.model", "ai.model.id"), + requestId: pickText(a, "request_id", "gen_ai.response.id", "ai.response.id"), + toolCallId: pickText(a, "tool_use_id", "call_id", "gen_ai.tool.call.id", "tool.call_id"), + runtimeToolCallId: pickText(a, "runtime_tool_call_id"), + toolName: pickText(a, "tool_name", "gen_ai.tool.name", "tool.name"), + toolInput: json(first(a.tool_input, a.tool_parameters, a.arguments)), + success: bool(a.success), + durationMs: number(a.duration_ms), + cwd: pickText(a, "cwd", "directory"), + tokens: usage, + usageScope: usage ? "event" : undefined, + tokenInputIncludesCache: + usage?.input !== undefined && tool !== OPENCODE_CONFIG_ID + ? tool === CODEX_CONFIG_ID + : undefined, + tokenOutputIncludesReasoning: + usage?.output !== undefined && tool === CODEX_CONFIG_ID ? true : undefined, + reportedCostUsd: number(a.cost_usd), + reportedCostUsdMicros: number(a.cost_usd_micros), + }; +} +function fromLog( + tool: TelemetryObservation["tool"], + resource: Obj, + raw: unknown +): TelemetryObservation { + const entry = object(raw); + const a = { ...resource, ...attributes(entry.attributes) }; + const row = base(tool, "log", a, raw); + const name = row.eventType; + return { + ...row, + timestamp: first(row.timestamp, nanos(entry.timeUnixNano)), + observedTimestamp: nanos(entry.observedTimeUnixNano), + prompt: + name === "user_prompt" || name === "codex.user_prompt" + ? pickText(a, "prompt_text", "prompt") + : undefined, + response: + name === "assistant_response" || name === "codex.agent_response" + ? text(a.response) + : undefined, + toolOutput: name === "tool_result" || name === "codex.tool_result" ? json(a.output) : undefined, + }; +} +function fromSpan( + tool: TelemetryObservation["tool"], + resource: Obj, + raw: unknown +): TelemetryObservation[] { + const span = object(raw); + const a = { ...resource, ...attributes(span.attributes) }; + const baseRow = base(tool, "span", a, raw); + const row: TelemetryObservation = { + ...baseRow, + eventType: text(span.name), + timestamp: nanos(span.startTimeUnixNano), + startedAt: nanos(span.startTimeUnixNano), + endedAt: nanos(span.endTimeUnixNano), + traceId: text(span.traceId), + spanId: text(span.spanId), + parentSpanId: text(span.parentSpanId), + prompt: text(a.user_prompt), + response: text(a["ai.response.text"]), + usageScope: baseRow.tokens ? "span-snapshot" : undefined, + tokenInputIncludesCache: + a["ai.usage.inputTokens"] !== undefined ? true : baseRow.tokenInputIncludesCache, + tokenOutputIncludesReasoning: + a["ai.usage.outputTokens"] !== undefined ? true : baseRow.tokenOutputIncludesReasoning, + }; + return [ + row, + ...list(span.events).map((rawEvent): TelemetryObservation => { + const event = object(rawEvent); + const ea = attributes(event.attributes); + return { + tool, + signal: "span-event", + source: json(rawEvent), + eventType: text(event.name), + timestamp: nanos(event.timeUnixNano), + toolVersion: row.toolVersion, + sessionId: row.sessionId, + traceId: row.traceId, + spanId: row.spanId, + toolCallId: row.toolCallId, + toolName: row.toolName, + toolOutput: event.name === "tool.output" ? json(ea.output) : undefined, + }; + }), + ]; +} +function fromMetric( + tool: TelemetryObservation["tool"], + resource: Obj, + raw: unknown +): TelemetryObservation[] { + const metric = object(raw); + const name = text(metric.name); + return ["sum", "gauge", "histogram"].flatMap((kind) => + list(object(metric[kind]).dataPoints).map((rawPoint): TelemetryObservation => { + const point = object(rawPoint); + const a = { ...resource, ...attributes(point.attributes) }; + const metricType = pickText(a, "type", "token_type"); + const amount = first(number(point.asInt), number(point.asDouble), number(point.sum)); + const tokenKinds: Record> = { + input: "input", + output: "output", + cacheRead: "cacheRead", + cacheCreation: "cacheWrite", + cached_input: "cacheRead", + cache_write_input: "cacheWrite", + reasoning_output: "reasoning", + }; + const token = metricType && tokenKinds[metricType]; + const usage = + (name === "claude_code.token.usage" || name === "codex.turn.token_usage") && + token && + amount !== undefined + ? { [token]: amount } + : undefined; + return { + ...base(tool, "metric", a, rawPoint), + eventType: name, + timestamp: nanos(point.timeUnixNano), + startedAt: nanos(point.startTimeUnixNano), + metricName: name, + metricType, + metricValue: amount, + tokens: usage, + usageScope: usage ? "metric-aggregate" : undefined, + }; + }) + ); +} +function otel(tool: TelemetryObservation["tool"], capture: OtelCapture): TelemetryObservation[] { + const payload = object(capture.payload); + if (capture.signal === "logs") + return list(payload.resourceLogs).flatMap((r) => { + const resource = object(r); + const attrs = attributes(object(resource.resource).attributes); + return list(resource.scopeLogs).flatMap((s) => + list(object(s).logRecords).map((entry) => fromLog(tool, attrs, entry)) + ); + }); + if (capture.signal === "traces") + return list(payload.resourceSpans).flatMap((r) => { + const resource = object(r); + const attrs = attributes(object(resource.resource).attributes); + return list(resource.scopeSpans).flatMap((s) => + list(object(s).spans).flatMap((entry) => fromSpan(tool, attrs, entry)) + ); + }); + return list(payload.resourceMetrics).flatMap((r) => { + const resource = object(r); + const attrs = attributes(object(resource.resource).attributes); + return list(resource.scopeMetrics).flatMap((s) => + list(object(s).metrics).flatMap((entry) => fromMetric(tool, attrs, entry)) + ); + }); +} +export function transformClaude(source: ClaudeTelemetrySource): TelemetryObservation[] { + return otel(CLAUDE_CONFIG_ID, source); +} +export function transformCodex(source: CodexTelemetrySource): TelemetryObservation[] { + return otel(CODEX_CONFIG_ID, source); +} +function openCodeTokens(raw: unknown): TelemetryObservation["tokens"] { + const t = object(raw); + const cache = object(t.cache); + return tokens({ + input: number(t.input), + output: number(t.output), + cacheRead: number(cache.read), + cacheWrite: number(cache.write), + reasoning: number(t.reasoning), + total: number(t.total), + }); +} +function openCodeV1(source: OpenCodeV1Source): TelemetryObservation[] { + const p = object(source.properties); + const info = object(p.info); + const part = object(p.part); + const state = object(part.state); + const model = object(info.model); + const step = part.type === "step-finish"; + const usage = openCodeTokens(step ? part.tokens : info.tokens); + const status = text(state.status); + return [ + { + tool: OPENCODE_CONFIG_ID, + signal: "event", + source: json(source), + openCodeVersion: 1, + eventType: source.type, + timestamp: first(millis(p.time), millis(object(info.time).created)), + sessionId: first( + text(p.sessionID), + text(info.sessionID), + text(part.sessionID), + source.type === "session.created" ? text(info.id) : undefined + ), + messageId: first( + source.type === "message.updated" ? text(info.id) : undefined, + text(part.messageID) + ), + toolVersion: text(info.version), + partId: text(part.id), + provider: first(text(info.providerID), text(model.providerID)), + model: first(text(info.modelID), text(model.modelID)), + role: text(info.role), + text: part.type === "text" ? text(part.text) : undefined, + toolCallId: part.type === "tool" ? text(part.callID) : undefined, + toolName: part.type === "tool" ? text(part.tool) : undefined, + toolInput: part.type === "tool" ? json(state.input) : undefined, + toolOutput: part.type === "tool" ? json(state.output) : undefined, + success: status === "completed" ? true : status === "error" ? false : undefined, + error: status === "error" ? json(state.error) : undefined, + tokens: usage, + usageScope: usage + ? step + ? "step-snapshot" + : source.type === "session.created" || source.type === "session.updated" + ? "session-cumulative" + : "message-snapshot" + : undefined, + tokenInputIncludesCache: usage?.input !== undefined ? false : undefined, + tokenOutputIncludesReasoning: usage?.output !== undefined ? false : undefined, + zeroMayMeanMissing: usage ? true : undefined, + reportedCostUsd: number(step ? part.cost : info.cost), + cwd: text(object(info.path).cwd), + }, + ]; +} +function openCodeV2(source: OpenCodeV2Source): TelemetryObservation[] { + const d = object(source.data); + const model = object(d.model); + const item = object(d.item); + const itemPayload = object(item.payload); + const inboxModel = object(object(itemPayload.metadata).model); + const usage = openCodeTokens(d.tokens); + const kind = source.type; + return [ + { + tool: OPENCODE_CONFIG_ID, + signal: "event", + source: json(source), + openCodeVersion: 2, + eventType: kind, + timestamp: millis(source.created), + toolVersion: kind === "session.created" ? text(d.version) : undefined, + sessionId: text(d.sessionID), + messageId: text(d.assistantMessageID), + provider: first(text(model.providerID), text(inboxModel.providerID)), + model: first(text(model.id), text(inboxModel.modelID)), + cwd: text(object(source.location).directory), + role: kind === "session.inbox.enqueued" ? text(item.type) : undefined, + prompt: + kind === "session.inbox.enqueued" && item.type === "user" + ? text(itemPayload.text) + : undefined, + response: kind === "session.text.ended" ? text(d.text) : undefined, + text: kind === "session.text.delta" ? text(d.delta) : undefined, + toolCallId: kind.startsWith("session.tool.") ? text(d.id) : undefined, + toolName: kind === "session.tool.input.started" ? text(d.name) : undefined, + toolInput: + kind === "session.tool.called" + ? json(d.input) + : kind === "session.tool.input.ended" + ? json(d.text) + : undefined, + toolOutput: kind === "session.tool.success" ? json(d.content) : undefined, + success: + kind === "session.tool.success" ? true : kind === "session.tool.failed" ? false : undefined, + error: kind === "session.tool.failed" ? json(d.error) : undefined, + tokens: usage, + usageScope: usage + ? kind === "session.usage.updated" + ? "session-cumulative" + : "event" + : undefined, + tokenInputIncludesCache: usage?.input !== undefined ? false : undefined, + tokenOutputIncludesReasoning: usage?.output !== undefined ? false : undefined, + zeroMayMeanMissing: usage ? true : undefined, + reportedCostUsd: number(d.cost), + }, + ]; +} +export function transformOpenCode(source: OpenCodeTelemetrySource): TelemetryObservation[] { + if ("signal" in source) return otel(OPENCODE_CONFIG_ID, source); + if ("properties" in source) return openCodeV1(source); + return openCodeV2(source); +} + +/** Decode one captured JSON line. Unknown or malformed records remain visibly unrecognised. */ +export function transformCapturedLine(tool: Config, line: string): TelemetryObservation[] { + let input: Obj; + try { + input = object(JSON.parse(line)); + } catch { + return []; + } + if (input.signal === "logs" || input.signal === "traces" || input.signal === "metrics") { + const source: OtelCapture = { signal: input.signal, payload: input.payload }; + if (tool === CLAUDE_CONFIG_ID) return transformClaude(source); + if (tool === CODEX_CONFIG_ID) return transformCodex(source); + return transformOpenCode(source); + } + if (tool !== OPENCODE_CONFIG_ID || typeof input.type !== "string") return []; + if ("properties" in input) + return transformOpenCode({ type: input.type, properties: input.properties }); + if ("data" in input) + return transformOpenCode({ + type: input.type, + data: input.data, + created: number(input.created), + location: input.location, + }); + return []; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts index 02dbd4832..f0e54ff3e 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts @@ -17,7 +17,9 @@ import { import { readSettings } from "./files.js"; export const OPENCODE_CONFIG_ID = "opencode"; -export const OTEL_CONFIGS = ["claude", "codex", OPENCODE_CONFIG_ID] as const; +export const CLAUDE_CONFIG_ID = "claude"; +export const CODEX_CONFIG_ID = "codex"; +export const OTEL_CONFIGS = [CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, OPENCODE_CONFIG_ID] as const; export const OTEL_ENDPOINTS = ["dev", "prod"] as const; export type Config = (typeof OTEL_CONFIGS)[number]; export type Endpoint = (typeof OTEL_ENDPOINTS)[number]; @@ -110,6 +112,8 @@ export async function openCodeVersion(host: Host): Promise<"1" | "2"> { } const version = /^\s*(?:opencode\s+)?v?([12])\.(\d+)\.(\d+)\s*$/i.exec(output); if (version && (version[1] === "1" || version[1] === "2")) { + // Native export from 1.18.35 was verified locally; other older releases remain unverified. + if (version[1] === "1" && Number(version[2]) === 18 && Number(version[3]) === 35) return "1"; const minimumMinor = version[1] === "1" ? 19 : 2; const minor = Number(version[2]); if (minor > minimumMinor || (minor === minimumMinor && Number(version[3]) >= 5)) { @@ -117,7 +121,7 @@ export async function openCodeVersion(host: Host): Promise<"1" | "2"> { } } throw new Error( - "Unsupported OpenCode version. Use an OpenCode 1 release from 1.19.5 or 2 from 2.2.5." + "Unsupported OpenCode version. Use OpenCode 1.18.35, a 1.x release from 1.19.5, or 2.x from 2.2.5." ); } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 6b8175c63..c4d128a37 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -179,6 +179,9 @@ export function registerTelemetryCommand(program: Command): void { output.info( `${result.lines} captured lines in ${result.capture}; last modified ${result.modifiedAt}` ); + output.info( + `${result.observations} normalized observations; ${result.eventLines} event lines; ${result.unrecognisedLines} unrecognised lines` + ); output.info("Existing capture does not prove delivery from the current session."); } } catch (error) { diff --git a/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts new file mode 100644 index 000000000..8fd2766a5 --- /dev/null +++ b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts @@ -0,0 +1,360 @@ +import { describe, expect, it } from "vitest"; +import { + transformClaude, + transformCodex, + transformOpenCode, +} from "../../../../../src/contexts/telemetry/infrastructure/otel/observations.js"; + +const attr = (key: string, value: unknown) => ({ key, value }); +const string = (value: string) => ({ stringValue: value }); +const integer = (value: number) => ({ intValue: String(value) }); + +function log(attributes: unknown[]) { + return { + signal: "logs" as const, + payload: { + resourceLogs: [ + { + resource: { attributes: [attr("service.version", string("1.2.3"))] }, + scopeLogs: [{ logRecords: [{ timeUnixNano: "1700000000000000000", attributes }] }], + }, + ], + }, + }; +} + +describe("telemetry observations", () => { + it("maps Claude request fields without assigning missing fields zero", () => { + const result = transformClaude( + log([ + attr("event.name", string("api_request")), + attr("session.id", string("s1")), + attr("model", string("m1")), + attr("input_tokens", integer(17)), + attr("cache_read_tokens", integer(9)), + ]) + ); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ + tool: "claude", + signal: "log", + eventType: "api_request", + sessionId: "s1", + model: "m1", + toolVersion: "1.2.3", + timestamp: "2023-11-14T22:13:20.000Z", + tokens: { input: 17, cacheRead: 9 }, + usageScope: "event", + }); + expect(result[0]?.tokens?.output).toBeUndefined(); + }); + + it("maps Codex tool records and treats success as a boolean", () => { + const result = transformCodex( + log([ + attr("event.name", string("codex.tool_result")), + attr("conversation.id", string("s2")), + attr("call_id", string("call-1")), + attr("tool_name", string("read_file")), + attr("arguments", string('{"path":"x"}')), + attr("output", string("done")), + attr("success", string("true")), + ]) + ); + expect(result[0]).toMatchObject({ + tool: "codex", + sessionId: "s2", + toolCallId: "call-1", + toolName: "read_file", + toolInput: '{"path":"x"}', + toolOutput: "done", + success: true, + }); + }); + + it("maps OpenCode v2 step and cumulative updates without treating them as separate calls", () => { + const data = { + sessionID: "s3", + tokens: { input: 10, output: 4, cache: { read: 5, write: 0 } }, + }; + const step = transformOpenCode({ type: "session.step.ended", created: 1700000000000, data }); + const cumulative = transformOpenCode({ + type: "session.usage.updated", + created: 1700000000000, + data, + }); + expect(step[0]).toMatchObject({ + tool: "opencode", + openCodeVersion: 2, + sessionId: "s3", + usageScope: "event", + tokens: { input: 10, output: 4, cacheRead: 5, cacheWrite: 0 }, + }); + expect(cumulative[0]?.usageScope).toBe("session-cumulative"); + }); + + it("maps OpenCode v1 message and tool events from the installed CLI schema", () => { + const message = transformOpenCode({ + type: "message.updated", + properties: { + sessionID: "s4", + info: { + id: "m4", + role: "assistant", + modelID: "model", + providerID: "provider", + tokens: { input: 2, output: 3, cache: { read: 1, write: 0 } }, + }, + }, + }); + expect(message[0]).toMatchObject({ + openCodeVersion: 1, + sessionId: "s4", + messageId: "m4", + model: "model", + provider: "provider", + usageScope: "message-snapshot", + tokens: { input: 2, output: 3, cacheRead: 1, cacheWrite: 0 }, + }); + const tool = transformOpenCode({ + type: "message.part.updated", + properties: { + sessionID: "s4", + part: { + type: "tool", + callID: "c4", + tool: "read", + state: { status: "completed", input: { path: "x" }, output: "done" }, + }, + }, + }); + expect(tool[0]).toMatchObject({ + toolCallId: "c4", + toolName: "read", + toolInput: { path: "x" }, + toolOutput: "done", + success: true, + }); + }); +}); + +describe("source fidelity", () => { + it("keeps metric sums separate from per-request token counts", () => { + const capture = { + signal: "metrics" as const, + payload: { + resourceMetrics: [ + { + resource: { attributes: [] }, + scopeMetrics: [ + { + metrics: [ + { + name: "codex.turn.token_usage", + histogram: { + dataPoints: [ + { sum: 42, attributes: [attr("token_type", string("cached_input"))] }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }; + expect(transformCodex(capture)[0]).toMatchObject({ + signal: "metric", + metricName: "codex.turn.token_usage", + metricValue: 42, + usageScope: "metric-aggregate", + tokens: { cacheRead: 42 }, + }); + }); + + it("keeps span tool output linked to its call without copying its token counts", () => { + const capture = { + signal: "traces" as const, + payload: { + resourceSpans: [ + { + resource: { attributes: [] }, + scopeSpans: [ + { + spans: [ + { + name: "claude_code.tool", + traceId: "trace-1", + spanId: "span-1", + attributes: [ + attr("tool_use_id", string("call-1")), + attr("tool_name", string("read")), + attr("input_tokens", integer(12)), + ], + events: [{ name: "tool.output", attributes: [attr("output", string("done"))] }], + }, + ], + }, + ], + }, + ], + }, + }; + const rows = transformClaude(capture); + expect(rows).toHaveLength(2); + expect(rows[1]).toMatchObject({ + signal: "span-event", + traceId: "trace-1", + toolCallId: "call-1", + toolOutput: "done", + }); + expect(rows[1]?.tokens).toBeUndefined(); + }); + + it("does not label a v1 text part as an assistant response without its role", () => { + const result = transformOpenCode({ + type: "message.part.updated", + properties: { sessionID: "s", part: { type: "text", text: "text" } }, + }); + expect(result[0]).toMatchObject({ text: "text" }); + expect(result[0]?.response).toBeUndefined(); + }); + + it("keeps v2 synthetic inbox messages out of the user prompt", () => { + const result = transformOpenCode({ + type: "session.inbox.enqueued", + data: { item: { type: "synthetic", payload: { text: "text" } } }, + }); + expect(result[0]?.prompt).toBeUndefined(); + expect(result[0]?.role).toBe("synthetic"); + }); +}); + +describe("OpenCode v1 step and user records", () => { + it("maps user-selected model without inventing message content", () => { + const [row] = transformOpenCode({ + type: "message.updated", + properties: { info: { role: "user", model: { modelID: "m1", providerID: "p1" } } }, + }); + expect(row).toMatchObject({ role: "user", model: "m1", provider: "p1" }); + expect(row?.prompt).toBeUndefined(); + }); + it("records step-finish tokens as a step snapshot, not a message total", () => { + const [row] = transformOpenCode({ + type: "message.part.updated", + properties: { + part: { + type: "step-finish", + id: "part", + sessionID: "session", + messageID: "message", + cost: 0.25, + tokens: { input: 3, output: 4, reasoning: 2, cache: { read: 1, write: 0 } }, + }, + }, + }); + expect(row).toMatchObject({ + sessionId: "session", + messageId: "message", + partId: "part", + usageScope: "step-snapshot", + reportedCostUsd: 0.25, + tokenInputIncludesCache: false, + tokenOutputIncludesReasoning: false, + zeroMayMeanMissing: true, + tokens: { input: 3, output: 4, reasoning: 2, cacheRead: 1, cacheWrite: 0 }, + }); + }); + it("keeps input token semantics explicit for Claude and Codex", () => { + const claude = transformClaude(log([attr("input_tokens", integer(17))])); + const codex = transformCodex(log([attr("input_token_count", integer(17))])); + expect(claude[0]?.tokenInputIncludesCache).toBe(false); + expect(codex[0]?.tokenInputIncludesCache).toBe(true); + }); +}); + +describe("source-specific fields", () => { + it("maps OpenCode v1 native AI SDK trace without treating parent and child spans as calls", () => { + const capture = { + signal: "traces" as const, + payload: { + resourceSpans: [ + { + scopeSpans: [ + { + spans: [ + { + name: "ai.streamText.doStream", + attributes: [ + attr("session.id", string("s")), + attr("ai.model.id", string("m")), + attr("ai.model.provider", string("p")), + attr("ai.response.text", string("ok")), + attr("ai.usage.inputTokens", integer(5)), + attr("ai.usage.outputTokens", integer(2)), + attr("ai.usage.cachedInputTokens", integer(1)), + attr("ai.usage.inputTokenDetails.noCacheTokens", integer(4)), + attr("ai.usage.totalTokens", integer(7)), + ], + }, + ], + }, + ], + }, + ], + }, + }; + expect(transformOpenCode(capture)[0]).toMatchObject({ + sessionId: "s", + model: "m", + provider: "p", + response: "ok", + usageScope: "span-snapshot", + tokenInputIncludesCache: true, + tokenOutputIncludesReasoning: true, + tokens: { input: 5, inputNonCached: 4, output: 2, cacheRead: 1, total: 7 }, + }); + }); + it("does not turn a Codex thread id into a session id", () => { + const [threadOnly] = transformCodex(log([attr("thread.id", string("child"))])); + expect(threadOnly?.sessionId).toBeUndefined(); + expect(threadOnly?.threadId).toBe("child"); + const [both] = transformCodex( + log([attr("conversation.id", string("parent")), attr("thread.id", string("child"))]) + ); + expect(both).toMatchObject({ sessionId: "parent", threadId: "child" }); + }); + + it("preserves Codex non-cached and total counters and its tool aliases", () => { + const [row] = transformCodex( + log([ + attr("codex.turn.token_usage.non_cached_input_tokens", integer(9)), + attr("codex.turn.token_usage.total_tokens", integer(23)), + attr("tool_token_count", integer(4)), + attr("tool.name", string("read")), + attr("tool.call_id", string("call")), + attr("cwd", string("/project")), + ]) + ); + expect(row).toMatchObject({ + toolName: "read", + toolCallId: "call", + cwd: "/project", + tokens: { inputNonCached: 9, total: 23, tool: 4 }, + }); + }); + + it("keeps OpenCode inbox model when no step has been emitted yet", () => { + const [row] = transformOpenCode({ + type: "session.inbox.enqueued", + data: { + item: { + type: "user", + payload: { metadata: { model: { modelID: "model", providerID: "provider" } } }, + }, + }, + }); + expect(row).toMatchObject({ model: "model", provider: "provider" }); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 1c605b3bd..3c1b34a3f 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -167,4 +167,16 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { ); expect(await readFile(path, "utf8")).toBe(original); }); + + it("initializes the locally verified v1.18.35 without admitting unverified earlier versions", async () => { + await seedToken(); + const options = { home, env: {}, opencodeVersion: async () => "1.18.35" }; + expect(await initializeOtel("opencode", "dev", options)).toMatchObject({ changed: true }); + expect( + JSON.parse(await readFile(join(home, ".config/opencode/opencode.json"), "utf8")) + ).toMatchObject({ experimental: { openTelemetry: true } }); + await expect( + initializeOtel("opencode", "dev", { ...options, opencodeVersion: async () => "1.18.34" }) + ).rejects.toThrow(/OpenCode version/); + }); }); diff --git a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts index 193dacaba..adb22ca00 100644 --- a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts @@ -54,6 +54,60 @@ describe.skipIf(process.platform === "win32")("OTLP verification", () => { expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ lines: 1 }); }); + it("counts normalized observations without returning raw content", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const record = { + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: [] }, + scopeLogs: [ + { + logRecords: [ + { + attributes: [ + { key: "event.name", value: { stringValue: "user_prompt" } }, + { key: "prompt", value: { stringValue: "private-example" } }, + ], + }, + ], + }, + ], + }, + ], + }, + }; + await writeFile(capture, `${JSON.stringify(record)}\ninvalid\n`, { mode: 0o600 }); + const result = await verifyOtel("claude", "dev", { home, env: {} }); + expect(result).toMatchObject({ lines: 2, observations: 1, unrecognisedLines: 1 }); + expect(JSON.stringify(result)).not.toContain("private-example"); + }); + + it("also counts a local OpenCode SSE capture when present", async () => { + const host = { + home, + env: { OPENCODE_CONFIG_CONTENT: '{"experimental":{"openTelemetry":true}}' }, + opencodeVersion: async () => "1.19.5", + }; + await initializeOtel("opencode", "dev", host); + const directory = join(home, ".local/share/aidd-otel-observation"); + await writeFile(join(directory, "opencode.jsonl"), "{}\n", { mode: 0o600 }); + await mkdir(join(directory, "opencode")); + await writeFile( + join(directory, "opencode/events.jsonl"), + `${JSON.stringify({ + type: "session.step.ended", + data: { tokens: { input: 4 } }, + })}\n`, + { mode: 0o600 } + ); + expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ + observations: 1, + eventLines: 1, + }); + }); + it("refuses production", async () => { await expect(verifyOtel("claude", "prod", { home, env: {} })).rejects.toThrow(/production/); }); diff --git a/cli/tests/e2e/telemetry-otel.e2e.test.ts b/cli/tests/e2e/telemetry-otel.e2e.test.ts index 11d1aa836..302d5d6e1 100644 --- a/cli/tests/e2e/telemetry-otel.e2e.test.ts +++ b/cli/tests/e2e/telemetry-otel.e2e.test.ts @@ -17,10 +17,36 @@ describe.skipIf(process.platform === "win32")("native OTLP launch", () => { expect(init.exitCode).toBe(0); const missing = await runCli(["telemetry", "otel", "verify", ...args], projectDir, fakeHome); expect(missing.exitCode).not.toBe(0); - await writeFile(join(directory, "claude.jsonl"), '{"signal":"logs"}\n', { mode: 0o600 }); + await writeFile( + join(directory, "claude.jsonl"), + `${JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + attributes: [ + { key: "event.name", value: { stringValue: "user_prompt" } }, + { key: "prompt", value: { stringValue: "private-example" } }, + ], + }, + ], + }, + ], + }, + ], + }, + })}\n`, + { mode: 0o600 } + ); const verified = await runCli(["telemetry", "otel", "verify", ...args], projectDir, fakeHome); expect(verified.exitCode).toBe(0); expect(verified.stdout).toContain("1 captured lines"); + expect(verified.stdout).toContain("1 normalized observations"); + expect(verified.stdout).not.toContain("private-example"); expect(verified.stdout).not.toContain("test-token"); } finally { await cleanup(); diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md new file mode 100644 index 000000000..3a341df27 --- /dev/null +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -0,0 +1,46 @@ +# Provider telemetry field mapping + +This is an inventory of source fields, not a promise that the AIDD report ingests them. Claude Code and Codex fields below were observed in local OTLP captures; OpenCode 2 fields were observed in its session event stream. OpenCode 1 fields are from its published session schema, not a verified OTLP capture. Presence depends on version, configuration, event and provider. A missing value is unknown, not zero. + +The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 and `O2` is OpenCode 2. OTLP attribute names below are keys in the record, span or resource's `attributes` array, not necessarily JSON object properties. For O1, `info` denotes the message in `message.updated`, `part` the part in `message.part.updated`, and session information comes from `session.created` or `session.updated`; these are session-event schemas, not OTLP attributes. + +| Common element | Data | Example | Source properties | +| --- | --- | --- | --- | +| Time | Event or operation time | `2026-11-04T10:20:00+08:00` | **C/X:** `event.timestamp`, `timeUnixNano`, `observedTimeUnixNano`; spans have `startTimeUnixNano`, `endTimeUnixNano`. **O1:** `info.time.created`, `part.time.start` / `end` when provided. **O2:** `created`, `data.started` on step start. | +| Event type | Kind of event | `tool_result` | **C:** `event.name`, log `body.stringValue`, span or metric name. **X:** `event.name`; `eventName` may identify a source location instead. **O1/O2:** event `type`. | +| Application version | Client version | `2.2.5` | **C:** resource `service.version`. **X:** `app.version`, resource `service.version`. **O1:** session `info.version`. **O2:** `data.version` on `session.created`; OTLP resource `service.version`. | +| Session | Tool-specific session ID | `session-abc` | **C:** `session.id`. **X:** `conversation.id`; trace `thread.id`. **O1:** `info.sessionID`, `part.sessionID`. **O2:** `data.sessionID`; session-event `durable.aggregateID`. | +| Provider | Provider identifier | `openai` | **C:** `anthropic.provider`, `gen_ai.provider.name`, `gen_ai.system`. **X:** `provider_name`, trace `provider`. **O1:** assistant `info.providerID`, user `info.model.providerID`. **O2:** `data.model.providerID` on step start. Provider labels are not necessarily equivalent across tools. | +| Model | Model identifier | `model-xyz` | **C:** `model`, `gen_ai.request.model`. **X:** `model`. **O1:** assistant `info.modelID`, user `info.model.modelID`. **O2:** `data.model.id` on step start. | +| User input | User-role text when captured | `Fix the test` | **C:** `prompt`, `prompt_text`, span `user_prompt`. **X:** `prompt`. **O1:** text part `part.text`, joined to a user-role message via `messageID`. **O2:** `data.item.payload.text` in `session.inbox.enqueued` with `data.item.type=user`. A `user` role alone does not prove a human authored it. | +| Assistant response | Response text | `The test is fixed.` | **C/X:** `response`. **O1:** text part `part.text`, joined to an assistant-role message. **O2:** `data.text` on `session.text.ended`; `data.delta` on `session.text.delta`. | +| Input tokens | Input count | `1,200` | **C:** `input_tokens`; metric `claude_code.token.usage` with `type=input`. **X:** `input_token_count`, `codex.turn.token_usage.input_tokens`, `gen_ai.usage.input_tokens`; metric `codex.turn.token_usage` with `token_type=input`. Trace `codex.turn.token_usage.non_cached_input_tokens` is a separate count. **O1:** assistant `info.tokens.input` or step-finish `part.tokens.input`. **O2:** `data.tokens.input`. | +| Output tokens | Output count | `350` | **C:** `output_tokens`; metric `type=output`. **X:** `output_token_count`, `codex.turn.token_usage.output_tokens`, `gen_ai.usage.output_tokens`; metric `token_type=output`. **O1:** `info.tokens.output` or `part.tokens.output`. **O2:** `data.tokens.output`. | +| Cache read | Tokens read from cache | `800` | **C:** `cache_read_tokens`; metric `type=cacheRead`. **X:** `cached_token_count`, `codex.turn.token_usage.cached_input_tokens`, `gen_ai.usage.cache_read.input_tokens`; metric `token_type=cached_input`. **O1:** `info.tokens.cache.read` or `part.tokens.cache.read`. **O2:** `data.tokens.cache.read`. | +| Cache write | Tokens written to cache | `200` | **C:** `cache_creation_tokens`, `cache_creation_5m_tokens`, `cache_creation_1h_tokens`; metric `type=cacheCreation`. **X:** `cache_write_token_count`, `codex.turn.token_usage.cache_write_input_tokens`, `gen_ai.usage.cache_write.input_tokens`; metric `token_type=cache_write_input`. **O1:** `info.tokens.cache.write` or `part.tokens.cache.write`. **O2:** `data.tokens.cache.write`. | +| Tool call ID | Tool-specific call ID | `call-abc` | **C:** `tool_use_id`, `gen_ai.tool.call.id`. **X:** `call_id`; trace `runtime_tool_call_id` is a separate identifier. **O1:** tool `part.callID`. **O2:** `data.id` on tool events. | +| Tool name | Invoked tool | `read_file` | **C:** `tool_name`, `gen_ai.tool.name`. **X:** `tool_name`; metric label `tool`. **O1:** tool `part.tool`. **O2:** `data.name` on `session.tool.input.started`. | +| Tool input | Tool arguments | `{"path":"src/app.ts"}` | **C:** `tool_parameters`, `tool_input`. **X:** `arguments`. **O1:** tool `part.state.input`. **O2:** `data.input` on `session.tool.called`; serialized `data.text` on `session.tool.input.ended`. | +| Tool output | Returned content | `File contentsโ€ฆ` | **C:** `output` on span event `tool.output`. **X:** `output` on `codex.tool_result`. **O1:** completed tool `part.state.output`. **O2:** `data.content[].text` on `session.tool.success`. | +| Tool outcome | Success or failure | `success` | **C/X:** `success` on tool result. **O1:** `part.state.status` (`pending`, `running`, `completed`, `error`); error `part.state.error`. **O2:** event `session.tool.success` or `session.tool.failed`; failure `data.error.type`, `data.error.message`. | + +## Interpretation and coverage + +- The Claude and Codex metric `type` / `token_type` labels above belong to `claude_code.token.usage` and `codex.turn.token_usage`, respectively. OTLP logs, traces and metrics may describe the same operation; do not add them together. +- OpenCode 2 emits `data.tokens.*` both on `session.step.ended` and `session.usage.updated`. The latter is cumulative; summing every update overcounts. Its `input` is non-cached input and `output` is visible output; reasoning is separate. The implementation maps missing provider usage to zero, so a reported zero does not prove zero usage. +- Claude's `input_tokens` excludes cache reads and writes. Normalize token definitions before comparison; do not assume a similarly named field counts the same thing in all tools. +- In the examined capture, Codex cache-write fields were present but all zero. A nonzero write was not verified. Not every event includes every field; OpenCode 2 text and model data may need correlation across events. Outputs can be truncated. +- Claude can emit `cost_usd`; OpenCode 1 has assistant and step-finish `cost`, and OpenCode 2 exposes step and cumulative cost. A trustworthy equivalent was not established for Codex in the observed captures. These values do not by themselves establish an actual invoice: pricing basis and completeness must be known, and cumulative and per-step figures must not be combined as independent costs. +- A future feature may display an **estimated token cost**, not an invoice. It would require the provider and model, price source and effective date, currency, and rates for input, output, cache reads and cache writes. Subscription, included quota, negotiated and API pricing can differ. If usage, a compatible rate or a billing basis is unknown, the estimate is unknown rather than zero. Cost calculation and pricing are outside this feature. +- A tool session ID is not automatically a feature ID. Feature attribution requires the framework's session/branch declaration and must handle missing or ambiguous bindings. The existing local usage ledger and provider OTLP export are distinct pipelines. +- OpenCode 2's rich session events are SSE, not the observed OTLP logs. The O1 schema above does not prove its fields are emitted over OTLP. The schema event's `durable.version` is not an application major version. +- Prompts, responses, tool arguments and results can contain secrets. Capture and retain them only with the appropriate consent and local access controls. + +## Sources + +Read on 2026-11-04: + +- [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) +- [Codex observability and telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#observability-and-telemetry) +- [OpenCode 1.18.35 session schema](https://github.com/anomalyco/opencode/blob/v1.18.35/packages/schema/src/v1/session.ts) and [LLM instrumentation](https://github.com/anomalyco/opencode/blob/v1.18.35/packages/opencode/src/session/llm.ts) +- [OpenCode 2.2.5 session events](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts) and [token normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts) diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index c3f3187a4..05dca3769 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -52,7 +52,8 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. - Data lives in `~/.config/aidd/telemetry` and the clone's git config. - You are named only if you choose to be. - `AIDD_TELEMETRY=0` refuses AIDD usage measurement everywhere. +- Framework setup creates a stable installation UUID at `~/.config/aidd/installation/id.json` (or under the configured user directory), independently of the optional person label. It is not a credential or a login; telemetry setup creates it for older installations. It is retained when telemetry is forgotten. It is not yet attached to usage records or linked to an account. ## Provider OpenTelemetry -Configure an exporter with `aidd telemetry otel init --config= --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits. +Configure an exporter with `aidd telemetry otel init --config= --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. From 7f51b7e27bb3d6f9106673b355dffc03fa5b7fcc Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 11:32:32 +0800 Subject: [PATCH 05/56] test(cli): isolate installation identity fixtures --- cli/tests/e2e/installation-identity.e2e.test.ts | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cli/tests/e2e/installation-identity.e2e.test.ts b/cli/tests/e2e/installation-identity.e2e.test.ts index 8474bd8cc..d2961e355 100644 --- a/cli/tests/e2e/installation-identity.e2e.test.ts +++ b/cli/tests/e2e/installation-identity.e2e.test.ts @@ -1,8 +1,7 @@ -import { execFileSync } from "node:child_process"; import { mkdir, readFile, stat, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { createTestEnv, FRAMEWORK_PATH, runCli } from "./helpers.js"; +import { createTestEnv, FRAMEWORK_PATH, gitInit, runCli } from "./helpers.js"; describe.skipIf(process.platform === "win32")("installation identity across commands", () => { it("reuses the setup ID for telemetry and keeps a manually chosen person ID", async () => { @@ -30,7 +29,7 @@ describe.skipIf(process.platform === "win32")("installation identity across comm const first = await readFile(path, "utf8"); expect((await runCli(setup, projectDir, fakeHome)).exitCode).toBe(0); expect(await readFile(path, "utf8")).toBe(first); - execFileSync("git", ["init", "--quiet"], { cwd: projectDir }); + await gitInit(projectDir); const turnOn = await runCli(["telemetry", "on", "--yes"], projectDir, fakeHome); expect(turnOn.exitCode, turnOn.stderr).toBe(0); expect(await readFile(path, "utf8")).toBe(first); @@ -93,8 +92,9 @@ describe.skipIf(process.platform === "win32")("installation identity across comm try { expect((await runCli(args, projectDir, fakeHome)).exitCode).not.toBe(0); await expect(stat(path)).rejects.toThrow(); - execFileSync("git", ["init", "--quiet"], { cwd: projectDir }); - expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); + await gitInit(projectDir); + const turnOn = await runCli(args, projectDir, fakeHome); + expect(turnOn.exitCode, turnOn.stderr).toBe(0); const initial = await readFile(path, "utf8"); expect((await runCli(args, projectDir, fakeHome)).exitCode).toBe(0); expect(await readFile(path, "utf8")).toBe(initial); From 526e30d71e1b9fa6fe29b61dfc3e2b50e47da363 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 17:12:55 +0800 Subject: [PATCH 06/56] feat(telemetry): sync local sessions to Gouvernail --- cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 3 + cli/scripts/smoke-tools.sh | 72 +- .../application/report/session-consent.ts | 68 ++ .../report/session-feature-use-case.ts | 59 ++ .../domain/report/session-feature.ts | 65 ++ .../infrastructure/otel/actions/collect.ts | 165 ++++ .../infrastructure/otel/actions/sync-all.ts | 505 +++++++++++++ .../infrastructure/otel/actions/sync.ts | 703 ++++++++++++++++++ .../telemetry/infrastructure/otel/settings.ts | 8 + cli/src/presentation/commands/telemetry.ts | 97 +++ cli/src/runtime/wiring/telemetry-otel.ts | 43 ++ cli/src/runtime/wiring/telemetry.ts | 18 +- .../session-feature-use-case.unit.test.ts | 49 ++ .../observations/transformers.unit.test.ts | 433 +++++++++++ .../domain/session-feature.unit.test.ts | 71 ++ .../otel/collector.integration.test.ts | 115 +++ .../telemetry/otel/config-edits.unit.test.ts | 156 ++++ .../otel/init-otel.integration.test.ts | 7 +- .../otel/run-otel.integration.test.ts | 74 ++ .../otel/settings.integration.test.ts | 97 +++ .../otel/sync-all-otel.integration.test.ts | 568 ++++++++++++++ .../telemetry/otel/sync-config.unit.test.ts | 7 + .../otel/sync-otel.integration.test.ts | 263 +++++++ .../otel/verify-otel.integration.test.ts | 57 +- .../e2e/installation-identity.e2e.test.ts | 2 +- cli/tests/golden/snapshots/help/surface.json | 12 +- .../telemetry-wiring.integration.test.ts | 238 +++++- ...allation-identity-race.integration.test.ts | 52 ++ .../installation-identity.integration.test.ts | 57 +- ...rivate-storage-adapter.integration.test.ts | 8 + .../runtime/wiring/otel-wiring.unit.test.ts | 155 ++++ plugins/aidd-telemetry/OBSERVABILITY.md | 17 + scripts/__tests__/cli-opens-no-server.test.js | 59 +- 34 files changed, 4280 insertions(+), 25 deletions(-) create mode 100644 cli/src/contexts/telemetry/application/report/session-consent.ts create mode 100644 cli/src/contexts/telemetry/application/report/session-feature-use-case.ts create mode 100644 cli/src/contexts/telemetry/domain/report/session-feature.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts create mode 100644 cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts create mode 100644 cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/collector.integration.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/settings.integration.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts create mode 100644 cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts create mode 100644 cli/tests/runtime/wiring/otel-wiring.unit.test.ts diff --git a/cli/package.json b/cli/package.json index 56e38fbdb..20f63bb68 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 746, + "bundleBudgetKB": 784, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 0ed90cbe8..85ef92ade 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -53,6 +53,9 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // OpenCode 1 launcher (+15.9 KB over the 710.8 KB build above); 0.2% measured headroom. // 746 KB: 744.9 KB, provider observation transformers and a persistent installation ID // (+18.2 KB over the 726.7 KB build above); 0.1% measured headroom. +// 784 KB: 783.7 KB, local OTLP collector and consent-aware resumable provider sync, +// explicit metric sanitization, retry and durable feature invalidation (+38.8 KB over 744.9 KB); +// 0.1% measured headroom. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/scripts/smoke-tools.sh b/cli/scripts/smoke-tools.sh index 63afd9111..6f53b2130 100755 --- a/cli/scripts/smoke-tools.sh +++ b/cli/scripts/smoke-tools.sh @@ -30,6 +30,7 @@ ALL_COMMANDS=( "auth login" "auth logout" "auth status" "telemetry on" "telemetry off" "telemetry forget" "telemetry ingest" "telemetry task" "telemetry status" "telemetry report" "telemetry identity" + "telemetry otel init" "telemetry otel verify" "telemetry otel run" "telemetry otel sync" "telemetry otel collect" ) PASS=0; FAIL=0; SKIP=0 @@ -66,7 +67,7 @@ project_tree_hash() { PARENTS=" plugin marketplace auth framework telemetry " # A parent one level deeper than PARENTS, so a covered key needs three words there, not two. -GRANDPARENTS=" " +GRANDPARENTS=" telemetry otel " derive_key() { local first="$1" second="${2:-}" third="${3:-}" if [[ "$GRANDPARENTS" == *" $first $second "* ]]; then @@ -165,6 +166,8 @@ export AIDD_TOKEN="$TOKEN" # HOME does not move Codex: it reads that variable and falls back to the real `~/.codex`. export HOME="$TMPROOT/home"; mkdir -p "$HOME" export CODEX_HOME="$TMPROOT/codex-home"; mkdir -p "$CODEX_HOME" +export XDG_CONFIG_HOME="$HOME/.config" +unset OPENCODE_CONFIG OPENCODE_CONFIG_DIR OPENCODE_CONFIG_CONTENT # โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ• # OFFLINE / LOCAL โ€” runs without a token @@ -225,6 +228,73 @@ run "telemetry forget (preview)" 0 "Nothing was removed" "$P_TEL" -- telemetry f run "telemetry forget --yes" 0 "Removed" "$P_TEL" -- telemetry forget --yes run "telemetry off (after forget)" 0 "Measurement was not on" "$P_TEL" -- telemetry off +section "native telemetry (sandboxed profile)" +OTEL_CAPTURE="$HOME/.local/share/aidd-otel-observation" +mkdir -p "$OTEL_CAPTURE" "$TMPROOT/otel-bin" +chmod 700 "$OTEL_CAPTURE" +printf '%s\n' 'test-token' > "$OTEL_CAPTURE/collector-token" +chmod 600 "$OTEL_CAPTURE/collector-token" +run "telemetry otel init (Claude)" 0 "Updated" "$P_TEL" -- \ + telemetry otel init --config=claude --endpoint=dev +printf '%s\n' '{"signal":"logs","payload":{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"attributes":[{"key":"event.name","value":{"stringValue":"user_prompt"}}]}]}]}]}}' > "$OTEL_CAPTURE/claude.jsonl" +run "telemetry otel verify (Claude)" 0 "1 normalized observations" "$P_TEL" -- \ + telemetry otel verify --config=claude --endpoint=dev +# Exercise the actual built collector on a free loopback port using only the test profile. +if node - "$CLI" "$OTEL_CAPTURE/claude.jsonl" >"$TMPROOT/collector-smoke.log" 2>&1 <<'NODE' +const { spawn } = require("node:child_process"); +const { readFile } = require("node:fs/promises"); +const [cli, capture] = process.argv.slice(2); +const child = spawn(process.execPath, [cli, "telemetry", "otel", "collect", "--port", "0"]); +let output = ""; +child.stderr.on("data", data => { output += data; }); +const port = new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error("collector did not start")), 10000); + child.once("exit", () => { clearTimeout(timer); reject(new Error(`collector exited: ${output}`)); }); + child.stdout.on("data", data => { + output += data; + const match = output.match(/listening on http:\/\/127\.0\.0\.1:(\d+)/); + if (match) { clearTimeout(timer); resolve(match[1]); } + }); +}); +(async () => { + try { + const url = `http://127.0.0.1:${await port}/claude/v1/logs`; + const before = await readFile(capture, "utf8"); + const denied = await fetch(url, { method: "POST", body: "{}" }); + if (denied.status !== 403 || await readFile(capture, "utf8") !== before) throw new Error("unauthorized capture was not rejected"); + const accepted = await fetch(url, { method: "POST", headers: { Authorization: "Bearer test-token" }, body: "{}" }); + const lines = (await readFile(capture, "utf8")).trim().split("\n"); + const recorded = JSON.parse(lines.at(-1)); + if (accepted.status !== 200 || lines.length !== 2 || recorded.signal !== "logs" || JSON.stringify(recorded.payload) !== "{}") throw new Error("authorized capture was not recorded"); + if (output.includes("test-token")) throw new Error("collector printed its token"); + } catch (error) { console.error(error); process.exitCode = 1; } + finally { child.kill("SIGTERM"); } +})(); +NODE +then mark_covered "telemetry otel collect"; ok "telemetry otel collect (authenticated local capture)" +else bad "telemetry otel collect (authenticated local capture)" "$(cat "$TMPROOT/collector-smoke.log")"; fi +# The fake executable validates the environment handed to OpenCode 1 without using a real +# provider, changing the user's configuration, or relying on an installed OpenCode version. +cat > "$TMPROOT/otel-bin/opencode" <<'SCRIPT' +#!/usr/bin/env bash +if [[ "${1:-}" == "--version" ]]; then echo '1.18.35'; exit 0; fi +[[ "${OTEL_EXPORTER_OTLP_ENDPOINT:-}" == 'http://127.0.0.1:14318/opencode' ]] || exit 1 +[[ "${OTEL_EXPORTER_OTLP_HEADERS:-}" == 'Authorization=Bearer test-token' ]] || exit 1 +echo 'OpenCode received local OTLP settings' +SCRIPT +chmod 700 "$TMPROOT/otel-bin/opencode" +SMOKE_ORIGINAL_PATH="$PATH" +export PATH="$TMPROOT/otel-bin:$PATH" +unset OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS +run "telemetry otel init (OpenCode 1)" 0 "OpenCode 1" "$P_TEL" -- \ + telemetry otel init --config=opencode --endpoint=dev +run "telemetry otel run (OpenCode 1)" 0 "OpenCode received local OTLP settings" "$P_TEL" -- \ + telemetry otel run --config=opencode -- --version-check +unset AIDD_OTEL_TOKEN +run "telemetry otel sync (no consented session)" 1 "No consented usage found" "$P_TEL" -- \ + telemetry otel sync --config=codex --session=missing --tenant=smoke --endpoint=http://127.0.0.1:4318 +export PATH="$SMOKE_ORIGINAL_PATH" + section "update --check" out=$(cd "$ROOT" && node "$CLI" update --check 2>&1); rc=$? if [[ "$rc" -eq 0 || "$rc" -eq 1 ]]; then mark_covered "update"; ok "update --check (exit $rc)"; else bad "update crashed (exit $rc)" "$out"; fi diff --git a/cli/src/contexts/telemetry/application/report/session-consent.ts b/cli/src/contexts/telemetry/application/report/session-consent.ts new file mode 100644 index 000000000..964d72a09 --- /dev/null +++ b/cli/src/contexts/telemetry/application/report/session-consent.ts @@ -0,0 +1,68 @@ +import { sameClone } from "../../domain/consent/clone-identity.js"; +import { foldConsent } from "../../domain/consent/consent-history.js"; +import type { ConsentHistory } from "../../domain/ports/consent-history.js"; +import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; +import { coversConsentWindow, type SessionAdmission } from "../../domain/report/session-feature.js"; +import type { DirectoryResolver } from "../directory-resolver.js"; + +export class SessionConsent { + constructor( + private readonly ledger: UsageLedger, + private readonly directories: DirectoryResolver, + private readonly consents: ConsentHistory + ) {} + + /** Cumulative metrics are permitted only when their complete interval belongs to the same + * clone and one uninterrupted consent interval. */ + async permits( + admission: SessionAdmission, + at: string, + start?: string + ): Promise<"allow" | "deny" | "unknown"> { + const endTime = Date.parse(at); + const startTime = start === undefined ? endTime : Date.parse(start); + if ( + !admission.cwd || + !admission.repositoryId || + !Number.isFinite(endTime) || + !Number.isFinite(startTime) || + startTime > endTime + ) + return "unknown"; + return this.ledger.exclusively(async () => { + const run = await this.directories.open(); + try { + const first = await run.resolve(admission.cwd ?? "", new Date(startTime).toISOString()); + const last = await run.resolve(admission.cwd ?? "", new Date(endTime).toISOString()); + if ("skipped" in first || "skipped" in last) { + const reasons = [first, last].flatMap((value) => + "skipped" in value ? [value.skipped] : [] + ); + return reasons.some( + (reason) => reason === "unreadable-consent" || reason === "never-seen-alive" + ) + ? "unknown" + : "deny"; + } + if ( + !sameClone(first.stored.clone, last.stored.clone) || + first.stored.repository_id !== admission.repositoryId || + last.stored.repository_id !== admission.repositoryId + ) + return "deny"; + const history = await this.consents.read(); + if (history.damaged) return "unknown"; + return coversConsentWindow( + foldConsent(history.events), + first.stored.clone, + startTime, + endTime + ) + ? "allow" + : "deny"; + } finally { + await run.close(); + } + }); + } +} diff --git a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts new file mode 100644 index 000000000..13173c40d --- /dev/null +++ b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts @@ -0,0 +1,59 @@ +import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; +import { + type SessionAdmission, + type SessionFeature, + sessionFeatureOf, +} from "../../domain/report/session-feature.js"; +import type { StoredUsage } from "../../domain/stored-usage.js"; +import type { UsageTool } from "../../domain/usage-record.js"; +import type { IngestUsageUseCase } from "../ingest-usage-use-case.js"; +import { settleLedger } from "../ledger-upkeep.js"; +import type { DeclaredBindings } from "./declared-bindings.js"; +import type { SessionConsent } from "./session-consent.js"; + +/** Only usage already admitted by the clone's consent can be forwarded. */ +export class SessionFeatureUseCase { + constructor( + private readonly ingest: Pick, + private readonly ledger: UsageLedger, + private readonly declared: Pick, + private readonly consent: Pick + ) {} + + async snapshot(tool: UsageTool): Promise> { + if ((await this.ingest.execute()).refused) + throw new Error("AIDD telemetry is disabled; no capture was sent."); + const held = await this.ledger.exclusively(() => settleLedger(this.ledger)); + const facts = await this.declared.read(); + const sessions = new Map(); + for (const record of held.records.filter((item) => item.tool === tool)) { + const group = sessions.get(record.session_id) ?? []; + group.push(record); + sessions.set(record.session_id, group); + } + return new Map( + [...sessions].map(([id, records]) => { + const directories = new Set(records.map((record) => record.cwd)); + const repositories = new Set(records.map((record) => record.repository_id)); + return [ + id, + { + feature: sessionFeatureOf(records, facts), + cwd: directories.size === 1 ? records[0].cwd : null, + repositoryId: repositories.size === 1 ? records[0].repository_id : null, + }, + ]; + }) + ); + } + + async execute(session: string, tool: UsageTool): Promise { + const found = (await this.snapshot(tool)).get(session); + if (!found) throw new Error("No consented usage found for this session; no capture was sent."); + return found.feature; + } + + async permits(admission: SessionAdmission, at: string, start?: string) { + return this.consent.permits(admission, at, start); + } +} diff --git a/cli/src/contexts/telemetry/domain/report/session-feature.ts b/cli/src/contexts/telemetry/domain/report/session-feature.ts new file mode 100644 index 000000000..bdff8bc61 --- /dev/null +++ b/cli/src/contexts/telemetry/domain/report/session-feature.ts @@ -0,0 +1,65 @@ +import { type CloneIdentity, sameClone } from "../consent/clone-identity.js"; +import type { ConsentInterval } from "../consent/consent-history.js"; +import type { StoredUsage } from "../stored-usage.js"; +import { type AttributionFacts, attribute } from "./attribution.js"; + +export type SessionFeature = + | { readonly state: "unbound" | "ambiguous" } + | { readonly state: "confirmed"; readonly feature: string; readonly branch: string }; + +export interface SessionAdmission { + readonly feature: SessionFeature; + readonly cwd: string | null; + readonly repositoryId: string | null; +} + +// biome-ignore lint/suspicious/noControlCharactersInRegex: control characters cannot name a branch +const CONTROL = /[\u0000-\u001f\u007f]/u; + +/** A cumulative point may only cross time covered by one uninterrupted consent interval. */ +export function coversConsentWindow( + intervals: readonly ConsentInterval[], + clone: CloneIdentity, + start: number, + end: number +): boolean { + return ( + start <= end && + intervals.some( + (interval) => + sameClone(interval.clone, clone) && + interval.from <= start && + (interval.to === null || end < interval.to) + ) + ); +} + +/** A whole session is linked only when every recorded call agrees on an explicitly declared + * canonical ticket and an observed branch. An unknown call is never assigned by proximity. */ +export function sessionFeatureOf( + records: readonly StoredUsage[], + facts: AttributionFacts +): SessionFeature { + let candidate: { feature: string; branch: string } | undefined; + let missing = false; + for (const record of records) { + const resolution = attribute(record, facts); + const feature = resolution.state === "attributed" ? resolution.ticket : null; + const branch = record.git_branch; + if ( + !feature || + !/^[^\s/#]+\/[^\s/#]+#[1-9][0-9]*$/u.test(feature) || + !branch || + branch.length > 255 || + CONTROL.test(branch) + ) { + missing = true; + continue; + } + if (candidate && (candidate.feature !== feature || candidate.branch !== branch)) + return { state: "ambiguous" }; + candidate = { feature, branch }; + } + if (!candidate) return { state: "unbound" }; + return missing ? { state: "ambiguous" } : { state: "confirmed", ...candidate }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts new file mode 100644 index 000000000..aa85590ee --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts @@ -0,0 +1,165 @@ +import { randomBytes, timingSafeEqual } from "node:crypto"; +import { constants } from "node:fs"; +import { lstat, mkdir, open, writeFile } from "node:fs/promises"; +import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; +import { join } from "node:path"; +import { gunzipSync } from "node:zlib"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import { OTEL_CONFIGS, readToken } from "../settings.js"; + +const MAX_BYTES = 32 * 1024 * 1024; +const MAX_CLIENTS = 16; +const PATH = /^\/([^/]+)\/v1\/(logs|metrics|traces)$/; +const CAPTURE_TOOLS = new Set(OTEL_CONFIGS); + +async function tokenFor(home: string, directory: string): Promise { + await mkdir(directory, { recursive: true, mode: 0o700 }); + const info = await lstat(directory); + if (!info.isDirectory() || (info.mode & 0o077) !== 0) { + throw new Error("OTLP capture directory must be private."); + } + const path = join(directory, "collector-token"); + try { + await writeFile(path, `${randomBytes(32).toString("base64url")}\n`, { + flag: "wx", + mode: 0o600, + }); + } catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "EEXIST")) throw error; + } + return readToken(home); +} + +function authorized(request: IncomingMessage, token: string): boolean { + const given = request.headers.authorization ?? ""; + const expected = `Bearer ${token}`; + if (!/^[\x20-\x7e]+$/.test(given)) return false; + const a = Buffer.from(given, "ascii"); + const b = Buffer.from(expected, "ascii"); + return a.length === b.length && timingSafeEqual(a, b); +} + +class RequestError extends Error { + constructor(readonly status: number) { + super("Invalid OTLP request."); + } +} + +async function readPayload(request: IncomingMessage): Promise> { + const length = Number(request.headers["content-length"] ?? 0); + if (!Number.isSafeInteger(length) || length < 0) throw new RequestError(400); + if (length > MAX_BYTES) throw new RequestError(413); + const chunks: Buffer[] = []; + let size = 0; + for await (const chunk of request) { + const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + size += buffer.length; + if (size > MAX_BYTES) throw new RequestError(413); + chunks.push(buffer); + } + if (size === 0) throw new RequestError(400); + let raw = Buffer.concat(chunks); + const encoding = (request.headers["content-encoding"] ?? "").toLowerCase(); + if (encoding && encoding !== "identity" && encoding !== "gzip") throw new RequestError(415); + if (encoding === "gzip") { + try { + raw = gunzipSync(raw, { maxOutputLength: MAX_BYTES }); + } catch { + throw new RequestError(400); + } + } + let parsed: unknown; + try { + parsed = JSON.parse(raw.toString("utf8")); + } catch { + throw new RequestError(400); + } + const payload = asPlainObject(parsed); + if (!payload) throw new RequestError(400); + return payload; +} + +async function appendCapture(path: string, line: string): Promise { + const file = await open( + path, + constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW, + 0o600 + ); + try { + if (!(await file.stat()).isFile()) throw new Error("Capture must be a regular file."); + await file.chmod(0o600); + await file.writeFile(line, "utf8"); + } finally { + await file.close(); + } +} + +/** A local authenticated OTLP/HTTP JSON collector. It never records request headers. */ +export async function startOtelCollector(options: { + readonly home: string; + readonly port: number; +}) { + if (process.platform === "win32") + throw new Error("OTLP collection requires private POSIX files."); + if (!Number.isInteger(options.port) || options.port < 0 || options.port > 65535) + throw new Error("Collector port must be between 0 and 65535."); + const directory = join(options.home, ".local/share/aidd-otel-observation"); + const token = await tokenFor(options.home, directory); + for (const tool of OTEL_CONFIGS) { + await appendCapture(join(directory, `${tool}.jsonl`), ""); + } + let clients = 0; + let queue = Promise.resolve(); + const append = (path: string, line: string) => { + const work = queue.then(() => appendCapture(path, line)); + queue = work.catch(() => {}); + return work; + }; + const server = createServer((request, response) => { + if (clients >= MAX_CLIENTS) { + response.writeHead(503).end(); + return; + } + clients++; + void handle(request, response) + .finally(() => { + clients--; + }) + .catch(() => {}); + }); + server.requestTimeout = 10_000; + server.headersTimeout = 10_000; + async function handle(request: IncomingMessage, response: ServerResponse): Promise { + try { + if (!authorized(request, token)) throw new RequestError(403); + const path = PATH.exec(new URL(request.url ?? "/", "http://127.0.0.1").pathname); + if (request.method !== "POST" || !path || !CAPTURE_TOOLS.has(path[1])) + throw new RequestError(404); + const payload = await readPayload(request); + const line = `${JSON.stringify({ received_at: new Date().toISOString(), signal: path[2], payload })}\n`; + if (Buffer.byteLength(line) > MAX_BYTES) throw new RequestError(413); + await append(join(directory, `${path[1]}.jsonl`), line); + response.writeHead(200, { "content-type": "application/json" }).end("{}"); + } catch (error) { + if (!response.headersSent && !response.destroyed) + response.writeHead(error instanceof RequestError ? error.status : 500).end(); + } + } + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(options.port, "127.0.0.1", () => { + server.off("error", reject); + resolve(); + }); + }); + const address = server.address(); + if (!address || typeof address === "string") + throw new Error("Collector did not open a TCP port."); + return { + port: address.port, + close: () => + new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())) + ), + }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts new file mode 100644 index 000000000..7aaecb0ed --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -0,0 +1,505 @@ +import { createHash, randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import { lstat, mkdir, open, readdir, rename, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import type { SessionAdmission, SessionFeature } from "../../../domain/report/session-feature.js"; +import { LedgerLock } from "../../ledger-lock.js"; +import { + bindingTransfer, + destination, + type Identity, + packTransfers, + readBindingDetails, + type SyncOptions, + saveBindingState, + selectTransfers, + send, + sessionIdsInCapture, + type Transfer, +} from "./sync.js"; + +export type AllOptions = Omit & { + readonly stateDir: string; + readonly refreshMs?: number; +}; +type Consent = "allow" | "deny" | "unknown"; +type Snapshot = ReadonlyMap; +interface Pending { + readonly offset: number; + readonly length: number; +} +interface SessionState { + readonly identity: Identity; + mayHaveLinked: boolean; + binding: string; + needsReplay: boolean; +} +interface Checkpoint { + readonly device: number; + readonly inode: number; + readonly offset: number; +} +const MAX_LINE = 32 * 1024 * 1024; +const INTERVAL_MS = 5000; +const RETRY_MS = 30_000; +const BINDING_REFRESH_MS = 60_000; + +function key(options: AllOptions, session: string) { + return createHash("sha256") + .update(`${options.endpoint}\0${options.tenant}\0${options.config}\0${session}`) + .digest("hex"); +} +function asOptions(options: AllOptions, session: string): SyncOptions { + return { ...options, session, follow: true }; +} +async function privateDirectory(path: string) { + await mkdir(path, { recursive: true, mode: 0o700 }); + const info = await lstat(path); + if (!info.isDirectory() || (info.mode & 0o077) !== 0) + throw new Error("Sync state directory must be private."); +} +async function readCheckpoint(path: string): Promise { + const file = await open( + path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return null; + throw error; + }); + if (!file) return null; + try { + const info = await file.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Sync checkpoint must be a private regular file."); + let parsed: unknown; + try { + parsed = JSON.parse(await file.readFile("utf8")); + } catch { + throw new Error("Sync checkpoint is invalid."); + } + const record = asPlainObject(parsed); + if ( + !record || + !Number.isSafeInteger(record.device) || + !Number.isSafeInteger(record.inode) || + !Number.isSafeInteger(record.offset) || + Number(record.offset) < 0 + ) { + throw new Error("Sync checkpoint is invalid."); + } + return { + device: Number(record.device), + inode: Number(record.inode), + offset: Number(record.offset), + }; + } finally { + await file.close(); + } +} +async function saveCheckpoint(path: string, value: Checkpoint) { + const temporary = `${path}.${randomUUID()}.tmp`; + try { + await writeFile(temporary, `${JSON.stringify(value)}\n`, { mode: 0o600, flag: "wx" }); + await rename(temporary, path); + } finally { + await rm(temporary, { force: true }); + } +} + +/** A pending journal contains only byte ranges, not source text or provider identities. */ +async function appendPending(path: string, item: Pending) { + const file = await open( + path, + constants.O_RDWR | constants.O_CREAT | constants.O_APPEND | constants.O_NOFOLLOW, + 0o600 + ); + try { + const info = await file.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Sync pending journal must be a private regular file."); + if (info.size > 0) { + const length = Math.min(info.size, 64); + const tail = Buffer.alloc(length); + const { bytesRead } = await file.read(tail, 0, length, info.size - length); + if (bytesRead !== length) throw new Error("Sync pending journal changed unexpectedly."); + if (tail.at(-1) !== 10) { + const last = tail.lastIndexOf(10); + if (last < 0 && info.size > length) + throw new Error("Sync pending journal has an invalid entry."); + await file.truncate(last < 0 ? 0 : info.size - length + last + 1); + } + } + await file.writeFile(`${item.offset} ${item.length}\n`); + await file.sync(); + } finally { + await file.close(); + } +} + +async function pendingRanges(path: string): Promise { + const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + try { + const info = await file.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Sync pending journal must be a private regular file."); + const ranges = new Map(); + const contents = await file.readFile("utf8"); + const lines = contents.split("\n"); + lines.pop(); // A crash can leave an incomplete last append; its source line is replayed. + for (const line of lines) { + const parts = /^(\d+) (\d+)$/.exec(line); + if (!parts) throw new Error("Sync pending journal is invalid."); + const offset = Number(parts[1]); + const length = Number(parts[2]); + if ( + !Number.isSafeInteger(offset) || + !Number.isSafeInteger(length) || + length < 1 || + length > MAX_LINE + ) + throw new Error("Sync pending journal is invalid."); + ranges.set(line, { offset, length }); + } + return [...ranges.values()]; + } finally { + await file.close(); + } +} + +async function replacePending(path: string, ranges: readonly Pending[]) { + if (ranges.length === 0) { + await rm(path, { force: true }); + return; + } + const temporary = `${path}.${randomUUID()}.tmp`; + try { + const file = await open(temporary, "wx", 0o600); + try { + await file.writeFile(ranges.map((item) => `${item.offset} ${item.length}\n`).join("")); + await file.sync(); + } finally { + await file.close(); + } + await rename(temporary, path); + } finally { + await rm(temporary, { force: true }); + } +} + +/** On first launch start at the last complete line. A line being appended is retained. */ +async function initialOffset( + file: Awaited>, + size: number +): Promise { + if (size === 0) return 0; + const limit = Math.min(size, MAX_LINE); + const tail = Buffer.alloc(limit); + const { bytesRead } = await file.read(tail, 0, limit, size - limit); + if (bytesRead !== limit) throw new Error("Capture changed while sync was starting."); + const last = tail.lastIndexOf(10); + if (last < 0 && size > limit) throw new Error("Capture line exceeds 32 MiB."); + return last < 0 ? 0 : size - limit + last + 1; +} + +/** Follow all sessions admitted by the ledger. Unknown sessions leave private byte-range + * journals before the checkpoint advances; no raw prompt or tool output is copied. */ +export async function syncAllOtel( + options: AllOptions, + snapshot: () => Promise, + permits: (admission: SessionAdmission, at: string, start?: string) => Promise +) { + const base = asOptions(options, "auto"); + destination(base, "logs"); + const capture = join( + options.home, + ".local/share/aidd-otel-observation", + `${options.config}.jsonl` + ); + const directory = join(options.stateDir, "otel-sync"); + await privateDirectory(directory); + const checkpointPath = join(directory, `${key(options, "auto")}.checkpoint.json`); + const lock = new LedgerLock(`${checkpointPath}.lock`, { + name: "OTLP sync", + waitMs: 0, + staleAfterMs: Number.MAX_SAFE_INTEGER, + }); + const release = await lock.acquire(); + let file: Awaited> | undefined; + try { + file = await open(capture, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + const info = await file.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Capture must be a private regular file."); + const previous = await readCheckpoint(checkpointPath); + if ( + previous && + (previous.device !== info.dev || previous.inode !== info.ino || previous.offset > info.size) + ) { + throw new Error( + "Capture changed since the last sync; recover the previous capture before restarting." + ); + } + let offset = previous?.offset ?? (await initialOffset(file, info.size)); + const save = async (position: number) => + saveCheckpoint(checkpointPath, { device: info.dev, inode: info.ino, offset: position }); + await save(offset); + let held = await snapshot(); + let lastSnapshotAt = Date.now(); + let dirty = false; + let pendingBytes = Buffer.alloc(0); + const sessions = new Map(); + let sent = 0; + let denied = 0; + const pendingDirectory = join(directory, `${key(options, "auto")}.pending`); + await privateDirectory(pendingDirectory); + const pendingPath = (id: string) => join(pendingDirectory, `${key(options, id)}.jsonl`); + const sessionState = async (id: string): Promise => { + const existing = sessions.get(id); + if (existing) return existing; + const saved = await readBindingDetails(join(directory, `${key(options, id)}.json`)); + if (saved?.details && saved.details.session !== id) + throw new Error("Sync binding state does not match its session."); + const state = { + identity: { + values: new Map(saved?.details?.identity ?? []), + conflict: saved?.details?.conflict === true, + }, + mayHaveLinked: saved?.mayHaveLinked === true, + binding: saved?.details?.binding ?? "", + needsReplay: Boolean( + saved?.mayHaveLinked && saved.details && saved.details.binding !== "invalidated" + ), + }; + sessions.set(id, state); + return state; + }; + const loadBindings = async () => { + const files = new Set(await readdir(directory)); + for (const [id] of held) { + if (sessions.has(id)) continue; + const name = `${key(options, id)}.json`; + if (!files.has(name)) continue; + const saved = await readBindingDetails(join(directory, name)); + if (!saved?.details) continue; // Older states need a bounded session replay to recover identity. + if (saved.details.session !== id) + throw new Error("Sync binding state does not match its session."); + sessions.set(id, { + identity: { + values: new Map(saved.details.identity), + conflict: saved.details.conflict === true, + }, + mayHaveLinked: saved.mayHaveLinked, + binding: saved.details.binding, + needsReplay: saved.mayHaveLinked && saved.details.binding !== "invalidated", + }); + } + }; + const process = async ( + id: string, + items: readonly Transfer[], + admission: SessionAdmission + ): Promise => { + let wait = false; + const allowed: Transfer[] = []; + const verdicts = new Map(); + for (const transfer of items) { + if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { + denied++; + continue; + } + const stamp = `${transfer.at}\0${transfer.start ?? ""}`; + let verdict = verdicts.get(stamp); + if (verdict === undefined) { + verdict = await permits(admission, transfer.at, transfer.start); + verdicts.set(stamp, verdict); + } + if (verdict === "unknown") wait = true; + else if (verdict === "deny") denied++; + else allowed.push(transfer); + } + for (const batch of packTransfers(allowed)) { + await send(asOptions(options, id), batch.transfer); + sent += batch.count; + } + return wait; + }; + const bindings = async () => { + for (const [id, state] of sessions) { + const admission = held.get(id); + if (!admission) continue; + const feature: SessionFeature = admission.feature; + const next = + feature.state === "confirmed" ? `${feature.branch}\0${feature.feature}` : feature.state; + const invalid = + state.mayHaveLinked && + (state.identity.conflict || + feature.state !== "confirmed" || + (state.binding !== "" && state.binding !== next)); + if (invalid && state.binding === "invalidated") continue; + if ( + !invalid && + (feature.state !== "confirmed" || (state.binding === next && !state.needsReplay)) + ) + continue; + if ((await permits(admission, new Date().toISOString())) !== "allow") continue; + if (invalid && state.binding !== "invalidated") { + const event = bindingTransfer(asOptions(options, id), state.identity, { + state: "ambiguous", + }); + if (event) { + await send(asOptions(options, id), event); + sent++; + state.binding = "invalidated"; + await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { + session: id, + identity: [...state.identity.values], + binding: state.binding, + }); + } + } else if ( + !invalid && + feature.state === "confirmed" && + (state.binding !== next || state.needsReplay) + ) { + const event = bindingTransfer(asOptions(options, id), state.identity, feature); + if (event) { + await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { + session: id, + identity: [...state.identity.values], + binding: next, + }); + state.mayHaveLinked = true; + await send(asOptions(options, id), event); + sent++; + state.binding = next; + state.needsReplay = false; + } + } + } + }; + const flushPending = async () => { + const files = new Set(await readdir(pendingDirectory)); + for (const [id, admission] of held) { + const path = pendingPath(id); + if (!files.has(`${key(options, id)}.jsonl`)) continue; + const state = await sessionState(id); + const waiting: Pending[] = []; + for (const item of await pendingRanges(path)) { + const raw = Buffer.alloc(item.length); + const read = await file?.read(raw, 0, item.length, item.offset); + if (!read || read.bytesRead !== item.length || raw.at(-1) !== 10) + throw new Error("Pending capture range is unavailable; recover the capture."); + const transfers = selectTransfers( + raw.toString("utf8"), + asOptions(options, id), + state.identity + ); + if (state.identity.conflict) { + await saveBindingState( + join(directory, `${key(options, id)}.json`), + directory, + { + session: id, + identity: [...state.identity.values], + binding: state.binding, + conflict: true, + }, + state.mayHaveLinked + ); + } + if (await process(id, transfers, admission)) waiting.push(item); + } + await replacePending(path, waiting); + } + }; + await loadBindings(); + await flushPending(); + await bindings(); + const buffer = Buffer.alloc(65536); + for (;;) { + const { bytesRead } = await file.read(buffer, 0, buffer.length, offset + pendingBytes.length); + if (bytesRead > 0) { + pendingBytes = Buffer.concat([pendingBytes, buffer.subarray(0, bytesRead)]); + if (pendingBytes.length > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); + for ( + let newline = pendingBytes.indexOf(10); + newline >= 0; + newline = pendingBytes.indexOf(10) + ) { + const raw = pendingBytes.subarray(0, newline).toString("utf8"); + const position = offset; + if (raw.trim()) + for (const id of sessionIdsInCapture(raw, options.config)) { + const state = await sessionState(id); + const items = selectTransfers(raw, asOptions(options, id), state.identity); + if (state.identity.conflict) { + await saveBindingState( + join(directory, `${key(options, id)}.json`), + directory, + { + session: id, + identity: [...state.identity.values], + binding: state.binding, + conflict: true, + }, + state.mayHaveLinked + ); + } + const admission = held.get(id); + const waiting = + items.length > 0 && (!admission || (await process(id, items, admission))); + if (waiting) + await appendPending(pendingPath(id), { offset: position, length: newline + 1 }); + } + offset += newline + 1; + pendingBytes = pendingBytes.subarray(newline + 1); + dirty = true; + } + if (dirty) await save(offset); + continue; + } + const current = await lstat(capture); + if ( + !current.isFile() || + current.ino !== info.ino || + current.dev !== info.dev || + current.size < offset + pendingBytes.length + ) { + throw new Error( + "Capture was replaced or truncated; recover the previous capture before restarting." + ); + } + const elapsed = Date.now() - lastSnapshotAt; + const retryAfter = options.refreshMs ?? RETRY_MS; + const bindingAfter = options.refreshMs ?? BINDING_REFRESH_MS; + const pendingFiles = (await readdir(pendingDirectory)).length; + if ( + dirty || + (pendingFiles > 0 && elapsed >= retryAfter) || + (sessions.size > 0 && elapsed >= bindingAfter) + ) { + held = await snapshot(); + await loadBindings(); + lastSnapshotAt = Date.now(); + dirty = false; + await flushPending(); + await bindings(); + await save(offset); + } + if (options.signal?.aborted) + return { sent, denied, pending: (await readdir(pendingDirectory)).length }; + try { + await delay(options.intervalMs ?? INTERVAL_MS, undefined, { signal: options.signal }); + } catch (error) { + if (options.signal?.aborted) + return { sent, denied, pending: (await readdir(pendingDirectory)).length }; + throw error; + } + } + } finally { + await file?.close(); + await release(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts new file mode 100644 index 000000000..08df06d03 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -0,0 +1,703 @@ +import { createHash, randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import { lstat, mkdir, open, rename, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import type { SessionFeature } from "../../../domain/report/session-feature.js"; +import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; + +type FeatureState = SessionFeature; + +export interface SyncOptions { + readonly config: SyncConfig; + readonly session: string; + readonly tenant: string; + readonly endpoint: string; + readonly token: string; + readonly home: string; + readonly follow: boolean; + readonly stateDir?: string; + readonly signal?: AbortSignal; + readonly intervalMs?: number; +} + +type RecordObject = Record; +const object = (value: unknown): RecordObject => asPlainObject(value) ?? {}; +const array = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); +const MAX_PAYLOAD = 2 * 1024 * 1024; +const MAX_LINE = 32 * 1024 * 1024; +const INTERVAL_MS = 5000; +const CLAUDE_METRICS = new Set([ + "claude_code.active_time.total", + "claude_code.code_edit_tool.decision", + "claude_code.commit.count", + "claude_code.cost.usage", + "claude_code.lines_of_code.count", + "claude_code.pull_request.count", + "claude_code.session.count", + "claude_code.token.usage", +]); +const CODEX_EVENTS = new Set(["codex.conversation_starts", "codex.sse_event", "codex.tool_result"]); +const SAFE_ATTRIBUTES = new Set([ + "user.email", + "user.account_id", + "session.id", + "conversation.id", + "organization.id", + "event.name", + "event.kind", + "event.sequence", + "event.timestamp", + "tool_use_id", + "type", + "model", + "tool_name", + "tool", + "decision", + "query_source", + "app.version", + "originator", + "success", + "duration_ms", + "call_id", + "input_token_count", + "output_token_count", + "cached_token_count", + "cache_write_token_count", + "reasoning_token_count", +]); + +export interface Identity { + readonly values: Map; + conflict: boolean; +} + +function attributes(value: unknown): Map { + const out = new Map(); + for (const item of array(value)) { + const entry = object(item); + const v = object(entry.value); + const raw = v.stringValue ?? v.intValue ?? v.doubleValue ?? v.boolValue; + if (typeof entry.key === "string" && raw !== undefined && raw !== null) { + out.set(entry.key, String(raw)); + } + } + return out; +} + +function merge(...maps: Map[]): Map { + return new Map(maps.flatMap((map) => [...map])); +} + +function safeAttributes(values: Map) { + return [...values] + .filter(([key]) => SAFE_ATTRIBUTES.has(key)) + .map(([key, value]) => ({ + key, + value: { stringValue: value }, + })); +} + +function numericFields(source: RecordObject, keys: readonly string[]): RecordObject { + const result: RecordObject = {}; + for (const key of keys) { + const value = source[key]; + if ( + (typeof value === "number" && Number.isFinite(value)) || + (typeof value === "string" && /^-?\d+(?:\.\d+)?$/.test(value)) || + (Array.isArray(value) && + value.every( + (item) => + (typeof item === "number" && Number.isFinite(item)) || + (typeof item === "string" && /^-?\d+(?:\.\d+)?$/.test(item)) + )) + ) + result[key] = value; + } + return result; +} + +function captureIdentity(target: Identity, values: Map) { + for (const key of ["user.email", "user.account_id", "organization.id"]) { + const value = values.get(key); + if (!value) continue; + const existing = target.values.get(key); + if (existing !== undefined && existing.toLowerCase() !== value.toLowerCase()) + target.conflict = true; + else target.values.set(key, value); + } +} + +function timeOf(entry: RecordObject) { + return { + ...(entry.timeUnixNano === undefined ? {} : { timeUnixNano: entry.timeUnixNano }), + ...(entry.observedTimeUnixNano === undefined + ? {} + : { observedTimeUnixNano: entry.observedTimeUnixNano }), + }; +} + +export interface Transfer { + readonly signal: "logs" | "metrics"; + readonly payload: RecordObject; + readonly at?: string; + readonly start?: string; +} + +/** Each item was separately consent-checked. Only points at the same instant and from the + * same session are combined, with a hard upper bound matching the receiver. */ +export function packTransfers( + transfers: readonly Transfer[] +): Array<{ transfer: Transfer; count: number }> { + const groups = new Map>(); + for (const item of transfers) { + const key = `${item.signal}\0${item.at ?? ""}`; + const root = item.signal === "logs" ? "resourceLogs" : "resourceMetrics"; + const resources = array(item.payload[root]); + const batches = groups.get(key) ?? []; + const last = batches.at(-1); + const merged = last ? [...array(last.transfer.payload[root]), ...resources] : resources; + const body = { [root]: merged }; + if (Buffer.byteLength(JSON.stringify(body)) > MAX_PAYLOAD) { + if (Buffer.byteLength(JSON.stringify(item.payload)) > MAX_PAYLOAD) + throw new Error("An OTLP observation exceeds the receiver's 2 MiB limit."); + batches.push({ transfer: item, count: 1 }); + } else if (last) { + batches[batches.length - 1] = { + transfer: { ...last.transfer, payload: body }, + count: last.count + 1, + }; + } else { + batches.push({ transfer: item, count: 1 }); + } + groups.set(key, batches); + } + return [...groups.values()].flat(); +} + +/** Names only session IDs; data from an unknown session is never sent until the ledger admits it. */ +export function sessionIdsInCapture(line: string, config: SyncOptions["config"]): string[] { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + throw new Error("Capture contains invalid JSON; sync stopped."); + } + const envelope = object(parsed); + const payload = object(envelope.payload); + const sessionKey = config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id"; + const ids = new Set(); + const add = (resource: Map, raw: unknown) => { + const session = merge(resource, attributes(object(raw).attributes)).get(sessionKey); + if (session && /^[A-Za-z0-9_-]{1,128}$/.test(session)) ids.add(session); + }; + if (envelope.signal === "logs") { + for (const raw of array(payload.resourceLogs)) { + const resource = object(raw); + const attrs = attributes(object(resource.resource).attributes); + for (const scope of array(resource.scopeLogs)) + for (const record of array(object(scope).logRecords)) add(attrs, record); + } + } + if (config === CLAUDE_CONFIG_ID && envelope.signal === "metrics") { + for (const raw of array(payload.resourceMetrics)) { + const resource = object(raw); + const attrs = attributes(object(resource.resource).attributes); + for (const scope of array(resource.scopeMetrics)) + for (const rawMetric of array(object(scope).metrics)) { + const metric = object(rawMetric); + for (const kind of ["sum", "gauge", "histogram"]) + for (const point of array(object(metric[kind]).dataPoints)) add(attrs, point); + } + } + } + return [...ids]; +} + +function timestamp(value: unknown): string | undefined { + try { + if (typeof value !== "string" && typeof value !== "number") return undefined; + const nanos = BigInt(value); + return nanos > 0n ? new Date(Number(nanos / 1_000_000n)).toISOString() : undefined; + } catch { + return undefined; + } +} + +function eventTimestamp(value: string | undefined): string | undefined { + if (!value || !/(Z|[+-]\d\d:\d\d)$/i.test(value)) return undefined; + const time = Date.parse(value); + return Number.isFinite(time) ? new Date(time).toISOString() : undefined; +} + +/** Keep only supported facts for the requested session. Original capture files are not changed. */ +export function selectTransfers( + line: string, + options: SyncOptions, + identity: Identity +): Transfer[] { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + throw new Error("Capture contains invalid JSON; sync stopped."); + } + const envelope = object(parsed); + const payload = object(envelope.payload); + const out: Transfer[] = []; + const sessionKey = options.config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id"; + if (envelope.signal === "logs") { + for (const rawResource of array(payload.resourceLogs)) { + const resource = object(rawResource); + const resourceAttrs = attributes(object(resource.resource).attributes); + for (const rawScope of array(resource.scopeLogs)) { + for (const rawRecord of array(object(rawScope).logRecords)) { + const record = object(rawRecord); + const a = merge(resourceAttrs, attributes(record.attributes)); + if (a.get(sessionKey) !== options.session) continue; + captureIdentity(identity, a); + const event = a.get("event.name") ?? String(object(record.body).stringValue ?? ""); + const wanted = + options.config === CLAUDE_CONFIG_ID + ? event === "tool_result" || event === "claude_code.tool_result" + : CODEX_EVENTS.has(event) && + (event !== "codex.sse_event" || a.get("event.kind") === "response.completed"); + if (!wanted) continue; + a.set("event.name", event); + const at = + options.config === CODEX_CONFIG_ID + ? (eventTimestamp(a.get("event.timestamp")) ?? + timestamp(record.timeUnixNano ?? record.observedTimeUnixNano)) + : timestamp(record.timeUnixNano ?? record.observedTimeUnixNano); + if (options.config === CODEX_CONFIG_ID) { + if (at) a.set("event.timestamp", at); + else a.delete("event.timestamp"); + } + out.push({ + signal: "logs", + at, + payload: { + resourceLogs: [ + { + scopeLogs: [ + { logRecords: [{ ...timeOf(record), attributes: safeAttributes(a) }] }, + ], + }, + ], + }, + }); + } + } + } + } + if (options.config === CLAUDE_CONFIG_ID && envelope.signal === "metrics") { + for (const rawResource of array(payload.resourceMetrics)) { + const resource = object(rawResource); + const resourceAttrs = attributes(object(resource.resource).attributes); + for (const rawScope of array(resource.scopeMetrics)) { + for (const rawMetric of array(object(rawScope).metrics)) { + const metric = object(rawMetric); + if (typeof metric.name !== "string" || !CLAUDE_METRICS.has(metric.name)) continue; + for (const kind of ["sum", "gauge", "histogram"]) { + const aggregation = object(metric[kind]); + for (const rawPoint of array(aggregation.dataPoints)) { + const point = object(rawPoint); + const a = merge(resourceAttrs, attributes(point.attributes)); + if (a.get(sessionKey) !== options.session) continue; + captureIdentity(identity, a); + const sanitized = { + ...numericFields(point, [ + "startTimeUnixNano", + "timeUnixNano", + "asInt", + "asDouble", + "count", + "sum", + "bucketCounts", + "explicitBounds", + "min", + "max", + "flags", + ]), + attributes: safeAttributes(a), + }; + out.push({ + signal: "metrics", + at: timestamp(point.timeUnixNano), + start: timestamp(point.startTimeUnixNano), + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: metric.name, + [kind]: { + ...numericFields(aggregation, ["aggregationTemporality"]), + ...([ + "AGGREGATION_TEMPORALITY_CUMULATIVE", + "AGGREGATION_TEMPORALITY_DELTA", + ].includes(String(aggregation.aggregationTemporality)) + ? { aggregationTemporality: aggregation.aggregationTemporality } + : {}), + ...(typeof aggregation.isMonotonic === "boolean" + ? { isMonotonic: aggregation.isMonotonic } + : {}), + dataPoints: [sanitized], + }, + }, + ], + }, + ], + }, + ], + }, + }); + } + } + } + } + } + } + return out; +} + +export function destination(options: SyncOptions, signal: "logs" | "metrics"): URL { + let endpoint: URL; + try { + endpoint = new URL(options.endpoint); + } catch { + throw new Error("Endpoint must be a loopback HTTP origin."); + } + if ( + endpoint.protocol !== "http:" || + !["127.0.0.1", "[::1]"].includes(endpoint.hostname) || + endpoint.username || + endpoint.password || + endpoint.pathname !== "/" || + endpoint.search || + endpoint.hash + ) { + throw new Error("Endpoint must be a loopback HTTP origin."); + } + if (!/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(options.tenant) || options.tenant.length > 63) { + throw new Error("Tenant must be a valid slug."); + } + if (!/^[A-Za-z0-9_-]{1,128}$/.test(options.session)) throw new Error("Session ID is invalid."); + if (!options.token || /[\r\n]/.test(options.token)) + throw new Error("AIDD_OTEL_TOKEN is required."); + const provider = options.config === CODEX_CONFIG_ID ? `${CODEX_CONFIG_ID}/` : ""; + return new URL(`/api/kpi/otel/${provider}${options.tenant}/v1/${signal}`, endpoint); +} + +class RetryableDeliveryError extends Error {} + +export async function send(options: SyncOptions, transfer: Transfer): Promise { + const body = JSON.stringify(transfer.payload); + if (Buffer.byteLength(body) > MAX_PAYLOAD) + throw new Error("An OTLP observation exceeds the receiver's 2 MiB limit."); + const url = destination(options, transfer.signal); + let backoff = options.intervalMs ?? 1000; + for (;;) { + try { + let response: Response; + try { + response = await fetch(url, { + method: "POST", + redirect: "manual", + signal: AbortSignal.timeout(15000), + headers: { authorization: `Bearer ${options.token}`, "content-type": "application/json" }, + body, + }); + } catch { + throw new RetryableDeliveryError("Gouvernail could not be reached or timed out."); + } + if (!response.ok) { + const message = `Gouvernail rejected ${options.config} ${transfer.signal}: HTTP ${response.status}.`; + if (response.status >= 500) throw new RetryableDeliveryError(message); + throw new Error(message); + } + return; + } catch (error) { + if (!options.follow || !(error instanceof RetryableDeliveryError) || options.signal?.aborted) + throw error; + console.error(`${error.message} Retrying in ${backoff} ms.`); + await delay(backoff, undefined, { signal: options.signal }); + backoff = Math.min(backoff * 2, 30_000); + } + } +} + +export interface BindingDetails { + readonly session: string; + readonly identity: Array<[string, string]>; + readonly binding: string; + readonly conflict?: boolean; +} + +export async function readBindingDetails( + path: string +): Promise<{ mayHaveLinked: boolean; details: BindingDetails | null } | null> { + const file = await open( + path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ).catch((error) => { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return null; + throw error; + }); + if (!file) return null; + try { + const info = await file.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Sync state must be a private regular file."); + const parsed: unknown = JSON.parse(await file.readFile("utf8")); + const data = object(parsed); + if (typeof data.mayHaveLinked !== "boolean") throw new Error("Sync binding state is invalid."); + if (data.session === undefined && data.identity === undefined && data.binding === undefined) + return { mayHaveLinked: data.mayHaveLinked, details: null }; + if ( + typeof data.session !== "string" || + typeof data.binding !== "string" || + !Array.isArray(data.identity) || + !data.identity.every( + (item: unknown) => + Array.isArray(item) && + item.length === 2 && + ["user.email", "user.account_id", "organization.id"].includes(item[0]) && + typeof item[1] === "string" + ) + ) + throw new Error("Sync binding state is invalid."); + if (data.conflict !== undefined && typeof data.conflict !== "boolean") + throw new Error("Sync binding state is invalid."); + return { + mayHaveLinked: data.mayHaveLinked, + details: { + session: data.session, + identity: data.identity as Array<[string, string]>, + binding: data.binding, + conflict: data.conflict === true, + }, + }; + } finally { + await file.close(); + } +} + +export async function saveBindingState( + path: string, + directory: string, + details?: BindingDetails, + mayHaveLinked = true +): Promise { + await mkdir(directory, { recursive: true, mode: 0o700 }); + const info = await lstat(directory); + if (!info.isDirectory() || (info.mode & 0o077) !== 0) + throw new Error("Sync state directory must be private."); + const temporary = `${path}.${randomUUID()}.tmp`; + try { + await writeFile(temporary, `${JSON.stringify({ mayHaveLinked, ...details })}\n`, { + flag: "wx", + mode: 0o600, + }); + await rename(temporary, path); + } finally { + await rm(temporary, { force: true }); + } +} + +export function bindingTransfer( + options: SyncOptions, + identity: Identity, + state: FeatureState +): Transfer | null { + if ( + (identity.conflict && state.state === "confirmed") || + (!identity.values.get("user.email") && !identity.values.get("user.account_id")) + ) + return null; + if (options.config === CLAUDE_CONFIG_ID && !identity.values.get("user.email")) return null; + const a = new Map(identity.values); + a.set(options.config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id", options.session); + a.set( + "event.name", + state.state === "confirmed" ? "aidd.session_feature" : "aidd.session_feature_invalidated" + ); + a.set("event.timestamp", new Date().toISOString()); + if (state.state === "confirmed") { + a.set("aidd.feature", state.feature); + a.set("aidd.branch", state.branch); + } + const bindingAttrs = [...a].map(([key, value]) => ({ key, value: { stringValue: value } })); + return { + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: (BigInt(Date.now()) * 1_000_000n).toString(), + attributes: bindingAttrs, + }, + ], + }, + ], + }, + ], + }, + }; +} + +/** Replays one selected session and optionally follows newly appended bytes. A failed delivery + * never advances the in-memory cursor. Restart safely replays into the idempotent receiver. */ +export async function syncOtel( + options: SyncOptions, + resolve: () => Promise, + permits: (transfer: Transfer) => Promise +) { + destination(options, "logs"); + const path = join(options.home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); + const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + const identity: Identity = { values: new Map(), conflict: false }; + let sent = 0; + let skipped = 0; + let offset = 0; + let pending = Buffer.alloc(0); + let binding = ""; + let needsReplay = false; + const stateName = createHash("sha256") + .update(`${options.endpoint}\0${options.tenant}\0${options.config}\0${options.session}`) + .digest("hex"); + const statePath = options.stateDir + ? join(options.stateDir, "otel-sync", `${stateName}.json`) + : null; + let mayHaveLinked = false; + try { + const info = await handle.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Capture must be a private regular file."); + if (statePath !== null) { + const saved = await readBindingDetails(statePath); + mayHaveLinked = saved?.mayHaveLinked === true; + if (saved?.details) { + if (saved.details.session !== options.session) + throw new Error("Sync binding state does not match its session."); + for (const [key, value] of saved.details.identity) identity.values.set(key, value); + identity.conflict = saved.details.conflict === true; + binding = saved.details.binding; + needsReplay = mayHaveLinked && binding !== "invalidated"; + } + } + const buffer = Buffer.alloc(65536); + for (;;) { + const { bytesRead } = await handle.read(buffer, 0, buffer.length, offset + pending.length); + if (bytesRead > 0) { + pending = Buffer.concat([pending, buffer.subarray(0, bytesRead)]); + if (pending.length > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); + for (let newline = pending.indexOf(10); newline >= 0; newline = pending.indexOf(10)) { + const line = pending.subarray(0, newline).toString("utf8"); + if (line.trim()) { + const admitted: Transfer[] = []; + const verdicts = new Map(); + for (const transfer of selectTransfers(line, options, identity)) { + if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { + skipped++; + continue; + } + const stamp = `${transfer.at}\0${transfer.start ?? ""}`; + let allowed = verdicts.get(stamp); + if (allowed === undefined) { + allowed = await permits(transfer); + verdicts.set(stamp, allowed); + } + if (!allowed) { + skipped++; + continue; + } + admitted.push(transfer); + } + for (const batch of packTransfers(admitted)) { + await send(options, batch.transfer); + sent += batch.count; + } + } + offset += newline + 1; + pending = pending.subarray(newline + 1); + } + continue; + } + const current = await lstat(path); + if ( + !current.isFile() || + current.ino !== info.ino || + current.dev !== info.dev || + current.size < offset + pending.length + ) { + throw new Error("Capture was replaced or truncated; restart sync to replay it safely."); + } + const feature = await resolve(); + const nextBinding = + feature.state === "confirmed" ? `${feature.branch}\0${feature.feature}` : feature.state; + const shouldInvalidate = + mayHaveLinked && + (identity.conflict || + feature.state !== "confirmed" || + (binding !== "" && binding !== nextBinding)); + const bindingAllowed = await permits({ + signal: "logs", + payload: {}, + at: new Date().toISOString(), + }); + if (bindingAllowed && shouldInvalidate && binding !== "invalidated") { + const event = bindingTransfer(options, identity, { state: "ambiguous" }); + if (event) { + await send(options, event); + sent++; + binding = "invalidated"; + if (statePath !== null) + await saveBindingState(statePath, join(options.stateDir ?? "", "otel-sync"), { + session: options.session, + identity: [...identity.values], + binding, + }); + } + } else if ( + bindingAllowed && + !shouldInvalidate && + feature.state === "confirmed" && + (nextBinding !== binding || needsReplay) + ) { + const event = bindingTransfer(options, identity, feature); + if (event) { + if (statePath !== null) { + await saveBindingState(statePath, join(options.stateDir ?? "", "otel-sync"), { + session: options.session, + identity: [...identity.values], + binding: nextBinding, + }); + } + mayHaveLinked = true; + await send(options, event); + sent++; + binding = nextBinding; + needsReplay = false; + } + } + if (!options.follow || options.signal?.aborted) + return { sent, skipped, offset, partialLine: pending.length > 0 }; + try { + await delay(options.intervalMs ?? INTERVAL_MS, undefined, { signal: options.signal }); + } catch (error) { + if (options.signal?.aborted) + return { sent, skipped, offset, partialLine: pending.length > 0 }; + throw error; + } + } + } finally { + await handle.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts index f0e54ff3e..d744bd598 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts @@ -4,6 +4,7 @@ import { open } from "node:fs/promises"; import { join } from "node:path"; import { promisify } from "node:util"; import { claudeConfigDir } from "../../domain/claude-projects-root.js"; +import type { UsageTool } from "../../domain/usage-record.js"; import claudeTemplate from "./config/claude.json" with { type: "json" }; import codexTemplate from "./config/codex.json" with { type: "json" }; import opencodeTemplate from "./config/opencode.json" with { type: "json" }; @@ -20,6 +21,13 @@ export const OPENCODE_CONFIG_ID = "opencode"; export const CLAUDE_CONFIG_ID = "claude"; export const CODEX_CONFIG_ID = "codex"; export const OTEL_CONFIGS = [CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, OPENCODE_CONFIG_ID] as const; +export const OTEL_SYNC_CONFIGS = [CLAUDE_CONFIG_ID, CODEX_CONFIG_ID] as const; +export type SyncConfig = (typeof OTEL_SYNC_CONFIGS)[number]; +const usageToolByOtelConfig: Record = { + [CLAUDE_CONFIG_ID]: "claude-code", + [CODEX_CONFIG_ID]: CODEX_CONFIG_ID, +}; +export const usageToolForOtel = (config: SyncConfig): UsageTool => usageToolByOtelConfig[config]; export const OTEL_ENDPOINTS = ["dev", "prod"] as const; export type Config = (typeof OTEL_CONFIGS)[number]; export type Endpoint = (typeof OTEL_ENDPOINTS)[number]; diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index c4d128a37..2355f2b27 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -30,9 +30,12 @@ import { createDeps } from "../../runtime/wiring/framework.js"; import type { TelemetryDeps } from "../../runtime/wiring/telemetry.js"; import { checkOtel, + collectOtel, configureOtel, + followCapturedSessions, launchOtel, otelOptions, + syncCapturedSession, } from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, @@ -191,6 +194,100 @@ export function registerTelemetryCommand(program: Command): void { ); } + otel + .command("collect") + .description("Capture authenticated OTLP on a local loopback port") + .option("--port ", "Loopback TCP port; 0 selects a free port", "14318") + .action(async (options: { port: string }) => { + const { output } = parseGlobalOptions(program); + try { + if (!/^\d+$/.test(options.port)) throw new Error("Collector port must be an integer."); + const collector = await collectOtel(Number(options.port)); + output.info(`OTLP collector listening on http://127.0.0.1:${collector.port}`); + } catch (error) { + new ErrorHandler(output).handle(error); + } + }); + + otel + .command("sync") + .description("Send one consented session's local telemetry to a loopback Gouvernail") + .addOption( + new Option("--config ", "Tool to sync") + .choices(otelOptions.syncConfigs) + .makeOptionMandatory() + ) + .option("--session ", "Exact provider session ID; omit with --follow to discover sessions") + .requiredOption("--tenant ", "Gouvernail tenant slug") + .requiredOption("--endpoint ", "Loopback HTTP origin") + .option("--follow", "Follow new events for this session", false) + .action( + async (options: { + config: (typeof otelOptions.syncConfigs)[number]; + session?: string; + tenant: string; + endpoint: string; + follow: boolean; + }) => { + const { output, run } = commandOf(program); + if (!options.session && !options.follow) { + output.error("Pass --session for a bounded sync, or --follow to discover new sessions."); + process.exitCode = 1; + return; + } + await run(async (deps) => { + const tool = otelOptions.usageTool(options.config); + if (!options.session) { + output.info("Following new consented sessions; press Ctrl-C to stop."); + const result = await followCapturedSessions( + { + config: options.config, + tenant: options.tenant, + endpoint: options.endpoint, + token: process.env.AIDD_OTEL_TOKEN ?? "", + }, + () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), + (admission, at, start) => + deps.telemetry.sessionFeatureUseCase.permits(admission, at, start) + ); + output.info( + `Sent ${result.sent} observations; ${result.denied} were outside consent; ${result.pending} await confirmation.` + ); + return; + } + const session = options.session; + let admission = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get(session); + const resolve = async () => { + admission = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get(session); + if (!admission) + throw new Error("No consented usage found for this session; no capture was sent."); + return admission.feature; + }; + await resolve(); + if (options.follow) output.info("Following the selected session; press Ctrl-C to stop."); + const result = await syncCapturedSession( + { ...options, session, token: process.env.AIDD_OTEL_TOKEN ?? "" }, + resolve, + async (transfer) => + Boolean( + admission && + transfer.at && + (await deps.telemetry.sessionFeatureUseCase.permits( + admission, + transfer.at, + transfer.start + )) === "allow" + ) + ); + output.info( + `Sent ${result.sent} observations; ${result.skipped} were outside a known consent interval. Receiver acceptance does not prove member attribution.` + ); + if (result.partialLine) + output.info("The last incomplete capture line was left for the next sync."); + }); + } + ); + otel .command("run [args...]") .description("Launch OpenCode 1 with local OTLP export. Pass tool arguments after --") diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index 2e9b06cc6..520d8d2d7 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -1,11 +1,27 @@ import { homedir } from "node:os"; +import type { + SessionAdmission, + SessionFeature, +} from "../../contexts/telemetry/domain/report/session-feature.js"; +import { startOtelCollector } from "../../contexts/telemetry/infrastructure/otel/actions/collect.js"; import { initializeOtel } from "../../contexts/telemetry/infrastructure/otel/actions/init.js"; import { runOtelOpenCode } from "../../contexts/telemetry/infrastructure/otel/actions/run.js"; +import { + type SyncOptions, + syncOtel, + type Transfer, +} from "../../contexts/telemetry/infrastructure/otel/actions/sync.js"; +import { + type AllOptions, + syncAllOtel, +} from "../../contexts/telemetry/infrastructure/otel/actions/sync-all.js"; import { verifyOtel } from "../../contexts/telemetry/infrastructure/otel/actions/verify.js"; import { OPENCODE_CONFIG_ID, OTEL_CONFIGS, OTEL_ENDPOINTS, + OTEL_SYNC_CONFIGS, + usageToolForOtel, } from "../../contexts/telemetry/infrastructure/otel/settings.js"; import { checkInstallationId, @@ -13,9 +29,36 @@ import { finishInstallationId, } from "../filesystem/installation-identity.js"; import { userConfigDirOf } from "../user-config-dir.js"; +import { telemetryDir } from "./telemetry.js"; + +export function collectOtel(port: number) { + return startOtelCollector({ home: homedir(), port }); +} + +export function syncCapturedSession( + options: Omit, + resolve: () => Promise, + permits: (transfer: Transfer) => Promise +) { + return syncOtel({ ...options, home: homedir(), stateDir: telemetryDir() }, resolve, permits); +} + +export function followCapturedSessions( + options: Omit, + snapshot: () => Promise>, + permits: ( + admission: SessionAdmission, + at: string, + start?: string + ) => Promise<"allow" | "deny" | "unknown"> +) { + return syncAllOtel({ ...options, home: homedir(), stateDir: telemetryDir() }, snapshot, permits); +} export const otelOptions = { configs: OTEL_CONFIGS, + syncConfigs: OTEL_SYNC_CONFIGS, + usageTool: usageToolForOtel, endpoints: OTEL_ENDPOINTS, launchConfigs: [OPENCODE_CONFIG_ID], }; diff --git a/cli/src/runtime/wiring/telemetry.ts b/cli/src/runtime/wiring/telemetry.ts index ec01e63ea..3910a9a01 100644 --- a/cli/src/runtime/wiring/telemetry.ts +++ b/cli/src/runtime/wiring/telemetry.ts @@ -13,6 +13,8 @@ import { StoredUsageMigration } from "../../contexts/telemetry/application/legac import { ReadUsageUseCase } from "../../contexts/telemetry/application/read-usage-use-case.js"; import { DeclaredBindings } from "../../contexts/telemetry/application/report/declared-bindings.js"; import { ReportUsageUseCase } from "../../contexts/telemetry/application/report/report-usage-use-case.js"; +import { SessionConsent } from "../../contexts/telemetry/application/report/session-consent.js"; +import { SessionFeatureUseCase } from "../../contexts/telemetry/application/report/session-feature-use-case.js"; import { SnapshotBindingsUseCase } from "../../contexts/telemetry/application/snapshot-bindings-use-case.js"; import { TelemetryOffUseCase } from "../../contexts/telemetry/application/switch/telemetry-off-use-case.js"; import { TelemetryOnUseCase } from "../../contexts/telemetry/application/switch/telemetry-on-use-case.js"; @@ -59,6 +61,7 @@ export interface TelemetryDeps { ingestUsageUseCase: IngestUsageUseCase; snapshotBindingsUseCase: SnapshotBindingsUseCase; reportUsageUseCase: ReportUsageUseCase; + sessionFeatureUseCase: SessionFeatureUseCase; manageIdentityUseCase: ManageIdentityUseCase; proposeIdentityUseCase: ProposeIdentityUseCase; telemetryOnUseCase: TelemetryOnUseCase; @@ -144,6 +147,13 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { declaration, identity ); + const directoryResolver = new DirectoryResolver( + locator, + consents, + resolutionStore, + consentHistory, + environment + ); const ingestUsageUseCase = new IngestUsageUseCase( [ new ReadUsageUseCase( @@ -158,7 +168,7 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { ), ], ledger, - new DirectoryResolver(locator, consents, resolutionStore, consentHistory, environment), + directoryResolver, snapshotBindingsUseCase, { refusedByEnvironment }, new StoredUsageMigration(locator, resolutionStore, aliases, ledger) @@ -177,6 +187,12 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { locator, { refusedByEnvironment } ), + sessionFeatureUseCase: new SessionFeatureUseCase( + ingestUsageUseCase, + ledger, + new DeclaredBindings(sessions, snapshotStore), + new SessionConsent(ledger, directoryResolver, consentHistory) + ), manageIdentityUseCase: new ManageIdentityUseCase(identity), proposeIdentityUseCase: new ProposeIdentityUseCase( new GitUserSourceAdapter(gitEnv), diff --git a/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts new file mode 100644 index 000000000..3e5bfa5d3 --- /dev/null +++ b/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts @@ -0,0 +1,49 @@ +import { expect, it, vi } from "vitest"; +import type { IngestResult } from "../../../../../src/contexts/telemetry/application/ingest-usage-use-case.js"; +import { SessionFeatureUseCase } from "../../../../../src/contexts/telemetry/application/report/session-feature-use-case.js"; +import type { UsageLedger } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; + +it("refuses synchronization before reading the ledger when telemetry is disabled", async () => { + const unexpected = (): never => { + throw new Error("Unexpected read"); + }; + const exclusively = vi.fn(unexpected); + const ledger: UsageLedger = { + exclusively, + leftOut: { add: unexpected, counts: unexpected }, + load: unexpected, + save: unexpected, + leaveOutRequestless: unexpected, + positions: unexpected, + savePositions: unexpected, + }; + const refused: IngestResult = { + refused: true, + filesRead: 0, + added: 0, + updated: 0, + unrecognised: 0, + skippedLedgerLines: 0, + snapshots: 0, + notStored: { + "outside-repo": 0, + "never-seen-alive": 0, + "no-consent": 0, + "consent-closed": 0, + "unreadable-consent": 0, + "no-cwd": 0, + undated: 0, + }, + consentLogDamaged: false, + oldestTranscriptAt: null, + }; + const useCase = new SessionFeatureUseCase( + { execute: async () => refused }, + ledger, + { read: unexpected }, + { permits: unexpected } + ); + + await expect(useCase.snapshot("codex")).rejects.toThrow("telemetry is disabled"); + expect(exclusively).not.toHaveBeenCalled(); +}); diff --git a/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts index 8fd2766a5..8be088f06 100644 --- a/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts +++ b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; import { + transformCapturedLine, transformClaude, transformCodex, transformOpenCode, @@ -358,3 +359,435 @@ describe("source-specific fields", () => { expect(row).toMatchObject({ model: "model", provider: "provider" }); }); }); + +describe("OTLP field aliases", () => { + it.each([ + ["input_tokens", "input"], + ["input_token_count", "input"], + ["codex.turn.token_usage.input_tokens", "input"], + ["gen_ai.usage.input_tokens", "input"], + ["ai.usage.inputTokens", "input"], + ["codex.turn.token_usage.non_cached_input_tokens", "inputNonCached"], + ["ai.usage.inputTokenDetails.noCacheTokens", "inputNonCached"], + ["output_tokens", "output"], + ["output_token_count", "output"], + ["codex.turn.token_usage.output_tokens", "output"], + ["gen_ai.usage.output_tokens", "output"], + ["ai.usage.outputTokens", "output"], + ["cache_read_tokens", "cacheRead"], + ["cached_token_count", "cacheRead"], + ["codex.turn.token_usage.cached_input_tokens", "cacheRead"], + ["gen_ai.usage.cache_read.input_tokens", "cacheRead"], + ["ai.usage.cachedInputTokens", "cacheRead"], + ["ai.usage.inputTokenDetails.cacheReadTokens", "cacheRead"], + ["cache_creation_tokens", "cacheWrite"], + ["cache_write_token_count", "cacheWrite"], + ["codex.turn.token_usage.cache_write_input_tokens", "cacheWrite"], + ["gen_ai.usage.cache_write.input_tokens", "cacheWrite"], + ["cache_creation_5m_tokens", "cacheWrite5m"], + ["cache_creation_1h_tokens", "cacheWrite1h"], + ["reasoning_token_count", "reasoning"], + ["codex.turn.token_usage.reasoning_output_tokens", "reasoning"], + ["codex.usage.reasoning_output_tokens", "reasoning"], + ["ai.usage.reasoningTokens", "reasoning"], + ["ai.usage.outputTokenDetails.reasoningTokens", "reasoning"], + ["codex.turn.token_usage.total_tokens", "total"], + ["codex.usage.total_tokens", "total"], + ["ai.usage.totalTokens", "total"], + ["tool_token_count", "tool"], + ])("maps %s to %s", (key, normalized) => { + const [row] = transformCodex(log([attr(key, integer(7))])); + expect(row?.tokens).toMatchObject({ [normalized]: 7 }); + }); + + it.each([ + ["anthropic.provider", "provider"], + ["provider_name", "provider"], + ["provider", "provider"], + ["gen_ai.provider.name", "provider"], + ["gen_ai.system", "provider"], + ["ai.model.provider", "provider"], + ["model", "model"], + ["gen_ai.request.model", "model"], + ["ai.model.id", "model"], + ["request_id", "requestId"], + ["gen_ai.response.id", "requestId"], + ["ai.response.id", "requestId"], + ["tool_use_id", "toolCallId"], + ["call_id", "toolCallId"], + ["gen_ai.tool.call.id", "toolCallId"], + ["tool.call_id", "toolCallId"], + ["tool_name", "toolName"], + ["gen_ai.tool.name", "toolName"], + ["tool.name", "toolName"], + ["session.id", "sessionId"], + ["conversation.id", "sessionId"], + ])("maps %s to %s", (key, normalized) => { + const [row] = transformClaude(log([attr(key, string("example"))])); + expect(row).toMatchObject({ [normalized]: "example" }); + }); + + it("retains zero counters but leaves invalid numbers and absent usage unknown", () => { + const [zero] = transformClaude(log([attr("input_tokens", integer(0))])); + expect(zero?.tokens?.input).toBe(0); + for (const invalid of ["", "-1", "NaN", "Infinity"]) { + const [row] = transformClaude(log([attr("input_tokens", string(invalid))])); + expect(row?.tokens).toBeUndefined(); + } + }); + + it("decodes source attributes including booleans, arrays and nested maps", () => { + const [row] = transformCodex( + log([ + attr("event.name", string("codex.tool_result")), + attr("success", { boolValue: false }), + attr("arguments", { kvlistValue: { values: [attr("path", string("src/x"))] } }), + attr("output", { arrayValue: { values: [string("done"), integer(2)] } }), + ]) + ); + expect(row).toMatchObject({ + success: false, + toolInput: { path: "src/x" }, + toolOutput: ["done", "2"], + }); + }); +}); + +describe("OTLP event and metric semantics", () => { + it.each([ + ["user_prompt", "prompt", "prompt"], + ["user_prompt", "prompt_text", "prompt"], + ["codex.user_prompt", "prompt", "prompt"], + ["assistant_response", "response", "response"], + ["codex.agent_response", "response", "response"], + ["tool_result", "output", "toolOutput"], + ["codex.tool_result", "output", "toolOutput"], + ])("maps %s %s only to its matching field", (event, key, output) => { + const [row] = transformClaude( + log([attr("event.name", string(event)), attr(key, string("example"))]) + ); + expect(row).toMatchObject({ eventType: event, [output]: "example" }); + }); + + it("does not expose a prompt or response attached to an unrelated log event", () => { + const [row] = transformClaude( + log([ + attr("event.name", string("api_request")), + attr("prompt", string("secret")), + attr("response", string("response")), + attr("output", string("output")), + ]) + ); + expect(row?.prompt).toBeUndefined(); + expect(row?.response).toBeUndefined(); + expect(row?.toolOutput).toBeUndefined(); + }); + + it.each([ + ["claude_code.token.usage", "type", "input", "input", "sum"], + ["claude_code.token.usage", "type", "output", "output", "gauge"], + ["claude_code.token.usage", "type", "cacheRead", "cacheRead", "sum"], + ["claude_code.token.usage", "type", "cacheCreation", "cacheWrite", "sum"], + ["codex.turn.token_usage", "token_type", "cached_input", "cacheRead", "histogram"], + ["codex.turn.token_usage", "token_type", "cache_write_input", "cacheWrite", "histogram"], + ["codex.turn.token_usage", "token_type", "reasoning_output", "reasoning", "histogram"], + ])("maps aggregate %s %s", (name, attribute, type, normalized, kind) => { + const metric = { + name, + [kind]: { + dataPoints: [ + { + asInt: "12", + timeUnixNano: "1700000000000000000", + startTimeUnixNano: "1699999999000000000", + attributes: [attr(attribute, string(type))], + }, + ], + }, + }; + const [row] = transformCodex({ + signal: "metrics", + payload: { resourceMetrics: [{ scopeMetrics: [{ metrics: [metric] }] }] }, + }); + expect(row).toMatchObject({ + eventType: name, + metricName: name, + metricType: type, + metricValue: 12, + timestamp: "2023-11-14T22:13:20.000Z", + startedAt: "2023-11-14T22:13:19.000Z", + usageScope: "metric-aggregate", + tokens: { [normalized]: 12 }, + }); + }); + + it("does not mistake an unrelated metric for a token aggregate", () => { + const metric = { + name: "other.metric", + sum: { dataPoints: [{ asInt: "9", attributes: [attr("type", string("input"))] }] }, + }; + const [row] = transformClaude({ + signal: "metrics", + payload: { resourceMetrics: [{ scopeMetrics: [{ metrics: [metric] }] }] }, + }); + expect(row).toMatchObject({ metricName: "other.metric", metricType: "input", metricValue: 9 }); + expect(row?.tokens).toBeUndefined(); + expect(row?.usageScope).toBeUndefined(); + }); +}); + +describe("OpenCode v2 event fields", () => { + it("maps step start, assistant text and user input without mixing roles", () => { + const [start] = transformOpenCode({ + type: "session.step.started", + created: 1700000000000, + location: { directory: "/project" }, + data: { + sessionID: "s", + assistantMessageID: "m", + model: { id: "model", providerID: "provider" }, + }, + }); + expect(start).toMatchObject({ + timestamp: "2023-11-14T22:13:20.000Z", + sessionId: "s", + messageId: "m", + cwd: "/project", + model: "model", + provider: "provider", + }); + const [prompt] = transformOpenCode({ + type: "session.inbox.enqueued", + data: { item: { type: "user", payload: { text: "fix this" } } }, + }); + expect(prompt).toMatchObject({ role: "user", prompt: "fix this" }); + const [reply] = transformOpenCode({ type: "session.text.ended", data: { text: "done" } }); + expect(reply?.response).toBe("done"); + const [delta] = transformOpenCode({ type: "session.text.delta", data: { delta: "part" } }); + expect(delta?.text).toBe("part"); + expect(delta?.response).toBeUndefined(); + }); + + it("maps tool lifecycle, results and errors to the same call id", () => { + const [start] = transformOpenCode({ + type: "session.tool.input.started", + data: { id: "call", name: "read" }, + }); + const [called] = transformOpenCode({ + type: "session.tool.called", + data: { id: "call", input: { path: "x" } }, + }); + const [finished] = transformOpenCode({ + type: "session.tool.success", + data: { id: "call", content: [{ type: "text", text: "done" }] }, + }); + const [failed] = transformOpenCode({ + type: "session.tool.failed", + data: { id: "call", error: { type: "tool_error", message: "failed" } }, + }); + expect(start).toMatchObject({ toolCallId: "call", toolName: "read" }); + expect(called).toMatchObject({ toolCallId: "call", toolInput: { path: "x" } }); + expect(finished).toMatchObject({ + toolCallId: "call", + success: true, + toolOutput: [{ type: "text", text: "done" }], + }); + expect(failed).toMatchObject({ + toolCallId: "call", + success: false, + error: { type: "tool_error", message: "failed" }, + }); + }); + + it("rejects malformed or unrelated capture lines rather than inventing observations", () => { + expect(transformCapturedLine("opencode", "not json")).toEqual([]); + expect(transformCapturedLine("claude", '{"type":"session.step.ended","data":{}}')).toEqual([]); + expect(transformCapturedLine("opencode", '{"type":"session.step.ended"}')).toEqual([]); + expect( + transformCapturedLine("opencode", '{"type":"session.step.ended","data":{"sessionID":"s"}}')[0] + ?.sessionId + ).toBe("s"); + }); +}); + +describe("session and trace details", () => { + it("retains the original event including nested values and explicit nulls", () => { + const source = { + type: "session.tool.called", + data: { + id: "call", + input: { + path: "x", + enabled: true, + count: 0, + missing: null, + parts: ["a", 1, false, null, { nested: "value" }], + }, + }, + }; + const [row] = transformOpenCode(source); + expect(row?.source).toEqual(source); + expect(row?.toolInput).toEqual(source.data.input); + }); + + it.each([ + ["false", false], + ["true", true], + ["unknown", undefined], + ])("normalizes observed tool success %s", (value, expected) => { + const [row] = transformCodex( + log([attr("event.name", string("codex.tool_result")), attr("success", string(value))]) + ); + expect(row?.success).toBe(expected); + }); + + it("does not invent timestamps, token semantics or usage for incomplete events", () => { + const source = log([attr("event.name", string("user_prompt"))]); + source.payload.resourceLogs[0].scopeLogs[0].logRecords[0].timeUnixNano = "not-a-time"; + const [claude] = transformClaude(source); + expect(claude?.timestamp).toBeUndefined(); + expect(claude?.usageScope).toBeUndefined(); + expect(claude?.tokenInputIncludesCache).toBeUndefined(); + expect(claude?.tokenOutputIncludesReasoning).toBeUndefined(); + const [opencode] = transformOpenCode({ type: "session.step.started", data: {} }); + expect(opencode?.timestamp).toBeUndefined(); + expect(opencode?.usageScope).toBeUndefined(); + expect(opencode?.zeroMayMeanMissing).toBeUndefined(); + expect(opencode?.tokenInputIncludesCache).toBeUndefined(); + expect(opencode?.tokenOutputIncludesReasoning).toBeUndefined(); + }); + + it("keeps Codex session, thread, model, request and token categories on one log record", () => { + const [row] = transformCodex( + log([ + attr("event.name", string("codex.sse_event")), + attr("event.timestamp", string("2026-11-04T01:00:00Z")), + attr("conversation.id", string("session")), + attr("thread.id", string("thread")), + attr("provider_name", string("provider")), + attr("model", string("model")), + attr("request_id", string("request")), + attr("input_token_count", integer(10)), + attr("output_token_count", integer(4)), + attr("cached_token_count", integer(7)), + attr("cache_write_token_count", integer(0)), + attr("reasoning_token_count", integer(2)), + attr("duration_ms", integer(300)), + attr("success", { boolValue: true }), + ]) + ); + expect(row).toMatchObject({ + timestamp: "2026-11-04T01:00:00Z", + sessionId: "session", + threadId: "thread", + provider: "provider", + model: "model", + requestId: "request", + durationMs: 300, + success: true, + tokenInputIncludesCache: true, + tokenOutputIncludesReasoning: true, + tokens: { input: 10, output: 4, cacheRead: 7, cacheWrite: 0, reasoning: 2 }, + }); + }); + + it("retains span timestamps, correlation and source response as a snapshot", () => { + const capture = { + signal: "traces" as const, + payload: { + resourceSpans: [ + { + scopeSpans: [ + { + spans: [ + { + name: "ai.streamText.doStream", + traceId: "trace", + spanId: "span", + parentSpanId: "parent", + startTimeUnixNano: "1700000000000000000", + endTimeUnixNano: "1700000001000000000", + attributes: [ + attr("session.id", string("session")), + attr("ai.response.text", string("done")), + attr("ai.usage.inputTokens", integer(9)), + ], + events: [ + { + name: "tool.output", + timeUnixNano: "1700000000500000000", + attributes: [attr("output", string("result"))], + }, + ], + }, + ], + }, + ], + }, + ], + }, + }; + const [span, event] = transformOpenCode(capture); + expect(span).toMatchObject({ + eventType: "ai.streamText.doStream", + timestamp: "2023-11-14T22:13:20.000Z", + startedAt: "2023-11-14T22:13:20.000Z", + endedAt: "2023-11-14T22:13:21.000Z", + traceId: "trace", + spanId: "span", + parentSpanId: "parent", + sessionId: "session", + response: "done", + usageScope: "span-snapshot", + tokenInputIncludesCache: true, + }); + expect(event).toMatchObject({ + eventType: "tool.output", + timestamp: "2023-11-14T22:13:20.500Z", + traceId: "trace", + spanId: "span", + toolOutput: "result", + }); + }); + + it("keeps OpenCode v1 session creation, tool errors and v2 input completion distinct", () => { + const [created] = transformOpenCode({ + type: "session.created", + properties: { info: { id: "session", version: "1.18.35", time: { created: 1700000000000 } } }, + }); + expect(created).toMatchObject({ + sessionId: "session", + toolVersion: "1.18.35", + timestamp: "2023-11-14T22:13:20.000Z", + }); + const [failed] = transformOpenCode({ + type: "message.part.updated", + properties: { + part: { + id: "part", + messageID: "message", + sessionID: "session", + type: "tool", + tool: "read", + callID: "call", + state: { status: "error", error: "failed", input: { path: "x" } }, + }, + }, + }); + expect(failed).toMatchObject({ + partId: "part", + messageId: "message", + sessionId: "session", + toolName: "read", + toolCallId: "call", + success: false, + error: "failed", + toolInput: { path: "x" }, + }); + const [input] = transformOpenCode({ + type: "session.tool.input.ended", + data: { id: "call", text: '{"path":"x"}' }, + }); + expect(input).toMatchObject({ toolCallId: "call", toolInput: '{"path":"x"}' }); + }); +}); diff --git a/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts b/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts new file mode 100644 index 000000000..7af55405b --- /dev/null +++ b/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from "vitest"; +import type { AttributionFacts } from "../../../../src/contexts/telemetry/domain/report/attribution.js"; +import { + coversConsentWindow, + sessionFeatureOf, +} from "../../../../src/contexts/telemetry/domain/report/session-feature.js"; +import type { StoredUsage } from "../../../../src/contexts/telemetry/domain/stored-usage.js"; + +const facts: AttributionFacts = { declarations: [], carries: [], branches: new Map() }; +const call: StoredUsage = { + tool: "codex", + tool_version: null, + key: "request", + session_id: "session", + agent: "main", + at: "2026-11-04T00:00:00Z", + model: null, + input: 1, + output: 1, + cache_read: null, + cache_write: null, + cache_write_1h: null, + reasoning: null, + cwd: null, + git_branch: "feature/work", + repository_id: "repo", +}; + +describe("session feature binding", () => { + it("leaves initially unbound sessions unbound", () => { + expect(sessionFeatureOf([call], facts)).toEqual({ state: "unbound" }); + }); + it("confirms only the same declared ticket and branch for every call", () => { + const declared: AttributionFacts = { + ...facts, + declarations: [ + { + session_id: "session", + task: "work", + ticket: "owner/repo#7", + none: false, + declared_at: "2026-11-03T00:00:00Z", + by: "command", + }, + ], + }; + expect(sessionFeatureOf([call], declared)).toEqual({ + state: "confirmed", + feature: "owner/repo#7", + branch: "feature/work", + }); + expect(sessionFeatureOf([call, { ...call, git_branch: "feature/other" }], declared)).toEqual({ + state: "ambiguous", + }); + expect(sessionFeatureOf([call, { ...call, git_branch: null }], declared)).toEqual({ + state: "ambiguous", + }); + }); +}); + +it("never exports a cumulative metric across an off/on consent gap", () => { + const clone = { path: "/repo/.git", dev: "1", ino: "2", birthtimeMs: 0 }; + const intervals = [ + { token: "one", clone, from: 100, to: 200 }, + { token: "two", clone, from: 300, to: null }, + ]; + expect(coversConsentWindow(intervals, clone, 120, 190)).toBe(true); + expect(coversConsentWindow(intervals, clone, 120, 350)).toBe(false); + expect(coversConsentWindow(intervals, clone, 300, 350)).toBe(true); + expect(coversConsentWindow(intervals, clone, 190, 200)).toBe(false); +}); diff --git a/cli/tests/contexts/telemetry/otel/collector.integration.test.ts b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts new file mode 100644 index 000000000..d514095cb --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts @@ -0,0 +1,115 @@ +import { mkdir, mkdtemp, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; +import { Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { gzipSync } from "node:zlib"; +import { expect, it, vi } from "vitest"; +import { startOtelCollector } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js"; + +it("captures authenticated OTLP locally, rejects unauthenticated requests, and keeps files private", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-")); + const listen = vi.spyOn(Server.prototype, "listen"); + const collector = await startOtelCollector({ home, port: 0 }); + try { + expect(listen).toHaveBeenCalledOnce(); + expect(listen).toHaveBeenCalledWith(0, "127.0.0.1", expect.any(Function)); + for (const tool of ["claude", "codex", "opencode"]) { + const file = join(home, `.local/share/aidd-otel-observation/${tool}.jsonl`); + expect(await readFile(file, "utf8")).toBe(""); + expect((await stat(file)).mode & 0o777).toBe(0o600); + } + const token = ( + await readFile(join(home, ".local/share/aidd-otel-observation/collector-token"), "utf8") + ).trim(); + const url = `http://127.0.0.1:${collector.port}/claude/v1/logs`; + const body = JSON.stringify({ resourceLogs: [{ scopeLogs: [] }] }); + expect((await fetch(url, { method: "POST", body })).status).toBe(403); + expect( + (await fetch(url, { method: "POST", headers: { authorization: "Bearer wrong" }, body })) + .status + ).toBe(403); + expect( + ( + await fetch(url, { + method: "POST", + headers: { authorization: `Bearer ${token}`, "content-encoding": "gzip" }, + body: gzipSync(body), + }) + ).status + ).toBe(200); + expect( + ( + await fetch(url, { + method: "POST", + headers: { authorization: `Bearer ${token}` }, + body: "not json", + }) + ).status + ).toBe(400); + expect( + ( + await fetch(`http://127.0.0.1:${collector.port}/not-a-tool/v1/logs`, { + method: "POST", + headers: { authorization: `Bearer ${token}` }, + body, + }) + ).status + ).toBe(404); + const file = join(home, ".local/share/aidd-otel-observation/claude.jsonl"); + const captured = JSON.parse((await readFile(file, "utf8")).trim()); + expect(captured).toMatchObject({ + signal: "logs", + payload: { resourceLogs: [{ scopeLogs: [] }] }, + }); + expect((await stat(file)).mode & 0o777).toBe(0o600); + expect( + (await stat(join(home, ".local/share/aidd-otel-observation/collector-token"))).mode & 0o777 + ).toBe(0o600); + } finally { + listen.mockRestore(); + await collector.close(); + await rm(home, { recursive: true, force: true }); + } +}); + +it("does not follow an existing token symlink or replace its target", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-link-")); + const directory = join(home, ".local/share/aidd-otel-observation"); + const target = join(home, "other-file"); + try { + await mkdir(directory, { recursive: true, mode: 0o700 }); + await writeFile(target, "do not touch", { mode: 0o600 }); + await symlink(target, join(directory, "collector-token")); + await expect(startOtelCollector({ home, port: 0 })).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("do not touch"); + } finally { + await rm(home, { recursive: true, force: true }); + } +}); + +it("serializes concurrent appends into complete records", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-concurrent-")); + const collector = await startOtelCollector({ home, port: 0 }); + try { + const directory = join(home, ".local/share/aidd-otel-observation"); + const token = (await readFile(join(directory, "collector-token"), "utf8")).trim(); + const url = `http://127.0.0.1:${collector.port}/codex/v1/logs`; + const statuses = await Promise.all( + Array.from({ length: 8 }, (_, number) => + fetch(url, { + method: "POST", + headers: { authorization: `Bearer ${token}` }, + body: JSON.stringify({ number }), + }).then((response) => response.status) + ) + ); + expect(statuses).toEqual(Array(8).fill(200)); + const lines = (await readFile(join(directory, "codex.jsonl"), "utf8")).trim().split("\n"); + expect(lines.map((line) => JSON.parse(line).payload.number).sort()).toEqual([ + 0, 1, 2, 3, 4, 5, 6, 7, + ]); + } finally { + await collector.close(); + await rm(home, { recursive: true, force: true }); + } +}); diff --git a/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts b/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts new file mode 100644 index 000000000..f04defda7 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts @@ -0,0 +1,156 @@ +import { parse } from "smol-toml"; +import { describe, expect, it } from "vitest"; +import { + mergeCodexExporters, + mergeJsonEnvironment, + mergeOpenCodeInstrumentation, + openCodeInstrumentation, + rejectConflict, +} from "../../../../src/contexts/telemetry/infrastructure/otel/config-edits.js"; + +const endpoint = "http://127.0.0.1:14318/codex/v1/logs"; +const token = "test-token"; +const exporters = { exporter: endpoint }; +const defaults = { environment: "local", log_user_prompt: true }; +const toolResult = { max_bytes: 262144 }; +const mergeCodex = (content: string) => + mergeCodexExporters(content, exporters, token, defaults, toolResult); + +describe("merging Claude and OpenCode settings", () => { + it("adds the environment without losing unrelated settings or overriding the user's defaults", () => { + const original = '{"model":"m1","env":{"USER":"keep","DETAILS":"0"}}'; + const merged = mergeJsonEnvironment(original, { OTEL_ENDPOINT: "local" }, () => {}, { + DETAILS: "1", + }); + expect(JSON.parse(merged)).toEqual({ + model: "m1", + env: { USER: "keep", DETAILS: "0", OTEL_ENDPOINT: "local" }, + }); + expect( + mergeJsonEnvironment(merged, { OTEL_ENDPOINT: "local" }, () => {}, { DETAILS: "1" }) + ).toBe(merged); + }); + + it("creates the missing section and rejects conflicting or duplicate environment keys", () => { + expect( + JSON.parse(mergeJsonEnvironment('{"model":"m1"}', { OTEL_ENDPOINT: "local" }, () => {})) + ).toEqual({ model: "m1", env: { OTEL_ENDPOINT: "local" } }); + for (const invalid of [ + '{"env":{"OTEL_ENDPOINT":"remote"}}', + '{"env":{},"env":{}}', + '{"env":{"USER":"one","USER":"two"}}', + '{"env":[]}', + '{"env":', + ]) { + expect(() => mergeJsonEnvironment(invalid, { OTEL_ENDPOINT: "local" }, () => {})).toThrow(); + } + expect(() => rejectConflict("endpoint", "remote", "local")).toThrow(/endpoint/); + }); + + it("preserves JSONC comments, quoted punctuation and trailing commas in OpenCode settings", () => { + const original = `{ + // keep punctuation in values + "model": "provider/model//name", + "experimental": {"note": "literal /* text */ and comma,}", /* keep */}, + }`; + const merged = mergeOpenCodeInstrumentation(original, true); + expect(merged).toContain("// keep punctuation in values"); + expect(merged).toContain('"provider/model//name"'); + expect(merged).toContain('"literal /* text */ and comma,}"'); + expect(merged).toContain("/* keep */"); + expect(openCodeInstrumentation(merged)).toBe(true); + expect(mergeOpenCodeInstrumentation(merged, true)).toBe(merged); + expect(openCodeInstrumentation('{"other":true}')).toBeUndefined(); + }); + + it("keeps escaped quotes and comment-like strings when inserting into JSONC", () => { + const original = + '{\n "description": "line // not a comment",\n "experimental": {\n "labels": ["comma,}", "quoted \\"value\\"",], // real comment\n },\n}'; + const merged = mergeOpenCodeInstrumentation(original, true); + expect(merged).toContain('"line // not a comment"'); + expect(merged).toContain('"quoted \\"value\\""'); + expect(merged).toContain("// real comment"); + expect(openCodeInstrumentation(merged)).toBe(true); + }); + + it("refuses disabled, duplicate, malformed or non-object OpenCode settings", () => { + for (const invalid of [ + '{"experimental":{"openTelemetry":false}}', + '{"experimental":{},"experimental":{}}', + '{"experimental":{"x":1,"x":2}}', + '{"experimental":[]}', + '{"experimental":', + ]) { + expect(() => mergeOpenCodeInstrumentation(invalid, true)).toThrow(); + } + }); +}); + +describe("merging Codex TOML exporters", () => { + it("adds missing exporters and defaults without changing existing options or comments", () => { + const original = + '# Keep this comment\nmodel = "m1"\n[otel]\nlog_user_prompt = false\n[otel.tool_result]\nmax_bytes = 1234\n'; + const merged = mergeCodex(original); + const parsed = parse(merged); + expect(parsed).toMatchObject({ + model: "m1", + otel: { + environment: "local", + log_user_prompt: false, + tool_result: { max_bytes: 1234 }, + exporter: { + "otlp-http": { + endpoint, + protocol: "json", + headers: { authorization: "Bearer test-token" }, + }, + }, + }, + }); + expect(merged).toContain("# Keep this comment"); + expect(mergeCodex(merged)).toBe(merged); + }); + + it("preserves already configured exporter headers and adds only missing values", () => { + const original = `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nheaders = { Authorization = "Bearer test-token", other = "keep" }\n`; + const merged = mergeCodex(original); + expect(parse(merged)).toMatchObject({ + otel: { + exporter: { + "otlp-http": { + endpoint, + protocol: "json", + headers: { Authorization: "Bearer test-token", other: "keep" }, + }, + }, + }, + }); + expect(mergeCodex(merged)).toBe(merged); + }); + + it.each([ + '[ "otel" ] # keep\nvalue = true\n', + "['otel']\nvalue = true\n", + "[otel] # keep", + "[otel]\r\nvalue = true\r\n", + "[otel]\nvalue = true", + ])("preserves valid existing table syntax %j", (original) => { + const merged = mergeCodex(original); + const parsed = parse(merged); + expect(parsed.otel).toMatchObject({ environment: "local", log_user_prompt: true }); + if (original.includes("value")) expect(parsed.otel).toMatchObject({ value: true }); + if (original.includes("# keep")) expect(merged).toContain("# keep"); + expect(mergeCodex(merged)).toBe(merged); + }); + + it.each([ + '[otel.exporter.otlp-http]\nendpoint = "https://other.example"\n', + `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nprotocol = "grpc"\n`, + '[otel.exporter.otlp-grpc]\nendpoint = "https://other.example"\n', + `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nheaders = { authorization = "Bearer other" }\n`, + `[otel.exporter.otlp-http]\nendpoint = "${endpoint}"\nheaders = { authorization = "Bearer test-token", Authorization = "Bearer test-token" }\n`, + "[otel\n", + ])("refuses an incompatible or invalid existing exporter", (original) => { + expect(() => mergeCodex(original)).toThrow(); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 3c1b34a3f..53c19f4cd 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -65,7 +65,12 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { expect(await initializeOtel("claude", "dev", { home, env: {} })).toMatchObject({ changed: false, }); - expect(await readFile(path, "utf8")).toBe(content); + const after = await open(path, "r"); + try { + expect(await after.readFile("utf8")).toBe(content); + } finally { + await after.close(); + } }); it("refuses a symlink without reading or replacing its target", async () => { diff --git a/cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts new file mode 100644 index 000000000..c3722d867 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts @@ -0,0 +1,74 @@ +import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { runOtelOpenCode } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; +import type { Host } from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; + +describe.skipIf(process.platform === "win32")("OpenCode OTLP launcher", () => { + let home: string; + let bin: string; + let host: Host; + + beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-launch-")); + bin = join(home, "bin"); + const collector = join(home, ".local/share/aidd-otel-observation"); + await mkdir(collector, { recursive: true }); + await mkdir(bin); + await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); + host = { + home, + env: { PATH: bin, OPENCODE_CONFIG_DIR: join(home, "config") }, + opencodeVersion: async () => "1.18.35", + }; + }); + + afterEach(async () => { + await rm(home, { recursive: true, force: true }); + }); + + it("requires an initialized v1 configuration and rejects v2", async () => { + await expect(runOtelOpenCode([], host)).rejects.toThrow(/Initialize OpenCode 1 OTLP/); + await expect( + runOtelOpenCode([], { ...host, opencodeVersion: async () => "2.2.5" }) + ).rejects.toThrow(/OpenCode 1/); + }); + + it("passes local OTLP credentials and command arguments to v1", async () => { + await initializeOtel("opencode", "dev", host); + const executable = join(bin, "opencode"); + await writeFile( + executable, + `#!/bin/sh +[ "$1" = "check" ] || exit 31 +[ "$OTEL_EXPORTER_OTLP_ENDPOINT" = "http://127.0.0.1:14318/opencode" ] || exit 32 +[ "$OTEL_EXPORTER_OTLP_HEADERS" = "Authorization=Bearer test-token" ] || exit 33 +exit 0 +` + ); + await chmod(executable, 0o700); + expect(await runOtelOpenCode(["check"], host)).toBe(0); + expect(await runOtelOpenCode(["other"], host)).toBe(31); + }); + + it("refuses conflicting inherited endpoints and a missing executable", async () => { + await initializeOtel("opencode", "dev", host); + await expect( + runOtelOpenCode([], { + ...host, + env: { ...host.env, OTEL_EXPORTER_OTLP_ENDPOINT: "https://different.example" }, + }) + ).rejects.toThrow(/Conflicting OTLP setting/); + await expect(runOtelOpenCode([], host)).rejects.toThrow(/Could not launch OpenCode/); + }); + + it("propagates a child terminated by a signal", async () => { + await initializeOtel("opencode", "dev", host); + const executable = join(bin, "opencode"); + await writeFile(executable, "#!/bin/sh\nkill -TERM $$\n"); + await chmod(executable, 0o700); + expect(await runOtelOpenCode([], host)).toBe(143); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/settings.integration.test.ts b/cli/tests/contexts/telemetry/otel/settings.integration.test.ts new file mode 100644 index 000000000..cbcddb5f4 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/settings.integration.test.ts @@ -0,0 +1,97 @@ +import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { + openCodeVersion, + readToken, +} from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; + +let home: string; +const tokenPath = () => join(home, ".local/share/aidd-otel-observation/collector-token"); + +beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), "aidd-otel-settings-")); +}); +afterEach(async () => { + await rm(home, { recursive: true, force: true }); +}); + +describe("OpenCode version detection", () => { + it.each([ + ["1.18.35", "1"], + ["1.19.5", "1"], + ["1.20.0", "1"], + ["opencode v2.2.5", "2"], + ["2.3.0", "2"], + ])("accepts verified release %s", async (version, expected) => { + expect(await openCodeVersion({ home, env: {}, opencodeVersion: async () => version })).toBe( + expected + ); + }); + + it.each(["1.18.34", "1.19.4", "2.2.4", "2.1.99", "3.0.0", "not-a-version", "1.18.35 extra"])( + "rejects unsupported release %s", + async (version) => { + await expect( + openCodeVersion({ home, env: {}, opencodeVersion: async () => version }) + ).rejects.toThrow(/Unsupported OpenCode version/); + } + ); + + it("explains when the executable cannot be queried", async () => { + await expect( + openCodeVersion({ + home, + env: {}, + opencodeVersion: async () => { + throw new Error("missing"); + }, + }) + ).rejects.toThrow(/Cannot determine OpenCode version/); + }); +}); + +describe.skipIf(process.platform === "win32")("collector credential and exporter conflicts", () => { + it("reads a valid private token and rejects missing, malformed or public tokens", async () => { + await expect(readToken(home)).rejects.toThrow(/private collector token/); + await mkdir(dirname(tokenPath()), { recursive: true }); + await writeFile(tokenPath(), "safe-token_123\n", { mode: 0o600 }); + expect(await readToken(home)).toBe("safe-token_123"); + await writeFile(tokenPath(), "invalid header\n"); + await expect(readToken(home)).rejects.toThrow(/private collector token/); + await writeFile(tokenPath(), "safe-token_123\n"); + await chmod(tokenPath(), 0o644); + await expect(readToken(home)).rejects.toThrow(/private collector token/); + }); + + it("never follows a symlink to a collector token", async () => { + const target = join(home, "target"); + await writeFile(target, "safe-token", { mode: 0o600 }); + await mkdir(dirname(tokenPath()), { recursive: true }); + await symlink(target, tokenPath()); + await expect(readToken(home)).rejects.toThrow(/private collector token/); + }); + + it.each([ + ["OTEL_EXPORTER_OTLP_ENDPOINT", "https://different.example"], + ["OTEL_EXPORTER_OTLP_LOGS_HEADERS", "Authorization=Bearer other"], + ["OTEL_EXPORTER_OTLP_METRICS_HEADERS", "Authorization=Bearer other"], + ["OTEL_EXPORTER_OTLP_TRACES_HEADERS", "Authorization=Bearer other"], + ["OTEL_EXPORTER_OTLP_LOGS_PROTOCOL", "grpc"], + ["OTEL_EXPORTER_OTLP_METRICS_PROTOCOL", "grpc"], + ["OTEL_EXPORTER_OTLP_TRACES_PROTOCOL", "grpc"], + ])("refuses conflicting Claude setting %s without replacing the file", async (key, value) => { + await mkdir(dirname(tokenPath()), { recursive: true }); + await writeFile(tokenPath(), "safe-token", { mode: 0o600 }); + const settings = join(home, ".claude/settings.json"); + await mkdir(dirname(settings)); + const original = JSON.stringify({ env: { [key]: value }, untouched: true }); + await writeFile(settings, original, { mode: 0o600 }); + await expect(initializeOtel("claude", "dev", { home, env: {} })).rejects.toThrow( + /Conflicting OTLP/ + ); + expect(await readFile(settings, "utf8")).toBe(original); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts new file mode 100644 index 000000000..10008368c --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts @@ -0,0 +1,568 @@ +import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, it } from "vitest"; +import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; + +const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); +function line(session: string) { + return `${JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: "1793763900000000000", + attributes: [ + attribute("conversation.id", session), + attribute("user.email", "developer@example.test"), + attribute("event.name", "codex.sse_event"), + attribute("event.kind", "response.completed"), + attribute("input_token_count", "42"), + ], + }, + ], + }, + ], + }, + ], + }, + })}\n`; +} +let server: Server | undefined; +const dirs: string[] = []; +afterEach(async () => { + if (server) await new Promise((resolve) => server?.close(() => resolve())); + await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); + server = undefined; +}); + +it("discovers a newly consented session without transmitting a different session", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + let snapshotCount = 0; + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => { + snapshotCount++; + return snapshotCount > 1 ? new Map([["allowed", admission]]) : new Map(); + }, + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed") + line("not-consented")); + const result = await running; + expect(result.sent).toBe(1); + expect(received).toHaveLength(1); + expect(received[0]).toContain("allowed"); + expect(received[0]).not.toContain("not-consented"); +}); + +it("replays a failed post from its last durable checkpoint", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-retry-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let fail = true; + let calls = 0; + const abort = new AbortController(); + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(fail ? 403 : 200).end("{}"); + if (!fail) abort.abort(); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const snapshot = async () => new Map([["allowed", admission]]); + const first = syncAllOtel(options, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await expect(first).rejects.toThrow("HTTP 403"); + fail = false; + const result = await syncAllOtel( + { ...options, signal: abort.signal }, + snapshot, + async () => "allow" + ); + expect(result.sent).toBe(1); + expect(calls).toBe(2); +}); + +it("retries a transient HTTP failure without losing its checkpoint", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-retry-live-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const abort = new AbortController(); + let calls = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(calls === 1 ? 503 : 200).end("{}"); + if (calls > 1) abort.abort(); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["allowed", admission]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + const result = await running; + expect(calls).toBe(2); + expect(result.sent).toBe(1); +}); + +it("does not send an event outside the clone's consent interval", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-consent-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + response.writeHead(200).end("{}"); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const abort = new AbortController(); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["allowed", admission]]), + async () => "deny" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 100)); + abort.abort(); + const result = await running; + expect(result.denied).toBe(1); + expect(calls).toBe(0); +}); + +it("recovers an interrupted pending append without losing either session on the same line", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-pending-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const firstAbort = new AbortController(); + const first = syncAllOtel( + { ...options, signal: firstAbort.signal }, + async () => new Map([["allowed", admission]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await expect( + syncAllOtel( + options, + async () => new Map(), + async () => "allow" + ) + ).rejects.toThrow(/lock/i); + const combined = JSON.parse(line("allowed")); + combined.payload.resourceLogs[0].scopeLogs[0].logRecords.push( + JSON.parse(line("waiting")).payload.resourceLogs[0].scopeLogs[0].logRecords[0] + ); + await appendFile(capture, `${JSON.stringify(combined)}\n`); + await new Promise((resolve) => setTimeout(resolve, 100)); + firstAbort.abort(); + await first; + expect(received).toHaveLength(1); + expect(received[0]).toContain("allowed"); + const stateFiles = await readdir(join(home, "state", "otel-sync")); + const pendingDirectory = join( + home, + "state", + "otel-sync", + stateFiles.find((name) => name.endsWith(".pending")) ?? "missing" + ); + const pending = join(pendingDirectory, (await readdir(pendingDirectory))[0]); + await appendFile(pending, "123"); // interrupted append before its checkpoint + await appendFile(capture, line("waiting")); + const secondAbort = new AbortController(); + const second = syncAllOtel( + { ...options, signal: secondAbort.signal }, + async () => new Map(), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + secondAbort.abort(); + await second; + expect((await readFile(pending, "utf8")).trim().split("\n")).toHaveLength(2); + const thirdAbort = new AbortController(); + const third = syncAllOtel( + { ...options, signal: thirdAbort.signal }, + async () => new Map([["waiting", admission]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + thirdAbort.abort(); + await third; + expect(received).toHaveLength(3); + expect( + received.slice(1).every((body) => body.includes("waiting") && !body.includes('"allowed"')) + ).toBe(true); +}); + +it("invalidates a persisted feature after restart when the session becomes ambiguous", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-restart-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, + cwd: "/project", + repositoryId: "repo", + }; + const firstAbort = new AbortController(); + const first = syncAllOtel( + { ...options, signal: firstAbort.signal }, + async () => new Map([["allowed", admission]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 120)); + firstAbort.abort(); + await first; + expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); + const secondAbort = new AbortController(); + const second = syncAllOtel( + { ...options, signal: secondAbort.signal }, + async () => new Map([["allowed", { ...admission, feature: { state: "ambiguous" as const } }]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 120)); + secondAbort.abort(); + await second; + expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(true); +}); + +it("retries a feature link rejected after its intent was persisted", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-retry-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let reject = true; + let links = 0; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + const isLink = body.includes('aidd.session_feature"'); + if (isLink) links++; + response.writeHead(isLink && reject ? 403 : 200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, + cwd: "/project", + repositoryId: "repo", + }; + const snapshot = async () => new Map([["allowed", admission]]); + const first = syncAllOtel(options, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await expect(first).rejects.toThrow("HTTP 403"); + reject = false; + const abort = new AbortController(); + const second = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 120)); + abort.abort(); + await second; + expect(links).toBe(2); +}); + +it("retains an identity conflict across a failed invalidation and restart", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-conflict-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let rejectInvalidation = true; + let invalidations = 0; + let linked = false; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + const invalidation = body.includes("aidd.session_feature_invalidated"); + if (invalidation) invalidations++; + if (body.includes('aidd.session_feature"')) linked = true; + response.writeHead(invalidation && rejectInvalidation ? 403 : 200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, + cwd: "/project", + repositoryId: "repo", + }; + const snapshot = async () => new Map([["allowed", admission]]); + const first = syncAllOtel(options, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + for (let i = 0; i < 100 && !linked; i++) await new Promise((resolve) => setTimeout(resolve, 20)); + expect(linked).toBe(true); + await appendFile( + capture, + line("allowed").replace("developer@example.test", "other@example.test") + ); + await expect(first).rejects.toThrow("HTTP 403"); + rejectInvalidation = false; + const abort = new AbortController(); + const second = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 120)); + abort.abort(); + await second; + expect(invalidations).toBe(2); +}); + +it("does not link or invalidate when identity conflicts before the first link, even after restart", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-first-conflict-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, + cwd: "/project", + repositoryId: "repo", + }; + const snapshot = async () => new Map([["allowed", admission]]); + const firstAbort = new AbortController(); + const first = syncAllOtel( + { ...options, signal: firstAbort.signal }, + snapshot, + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + const combined = JSON.parse(line("allowed")); + combined.payload.resourceLogs[0].scopeLogs[0].logRecords.push( + JSON.parse(line("allowed").replace("developer@example.test", "other@example.test")).payload + .resourceLogs[0].scopeLogs[0].logRecords[0] + ); + await appendFile(capture, `${JSON.stringify(combined)}\n`); + await new Promise((resolve) => setTimeout(resolve, 120)); + firstAbort.abort(); + await first; + const secondAbort = new AbortController(); + const second = syncAllOtel( + { ...options, signal: secondAbort.signal }, + snapshot, + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 120)); + secondAbort.abort(); + await second; + expect(received.length).toBeGreaterThan(0); + expect(received.some((body) => body.includes("aidd.session_feature"))).toBe(false); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts b/cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts new file mode 100644 index 000000000..83827717f --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-config.unit.test.ts @@ -0,0 +1,7 @@ +import { expect, it } from "vitest"; +import { usageToolForOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; + +it("maps each OTLP configuration to its ledger provider", () => { + expect(usageToolForOtel("claude")).toBe("claude-code"); + expect(usageToolForOtel("codex")).toBe("codex"); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts new file mode 100644 index 000000000..7a9d419a6 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts @@ -0,0 +1,263 @@ +import { appendFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + selectTransfers, + syncOtel, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const record = (session: string, event: string) => ({ + timeUnixNano: "1793763900000000000", + attributes: [ + attr("conversation.id", session), + attr("user.email", "developer@example.test"), + attr("event.name", event), + attr("event.kind", "response.completed"), + attr("input_token_count", "42"), + attr("tool_use_id", "toolu_123"), + attr("output", "sensitive output"), + ], +}); + +describe("OTLP capture sync", () => { + let server: Server | undefined; + const dirs: string[] = []; + afterEach(async () => { + if (server) await new Promise((resolve) => server?.close(() => resolve())); + await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); + server = undefined; + }); + + it("retains numeric metric data but excludes exemplars and unknown properties", () => { + const source = JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: "AGGREGATION_TEMPORALITY_CUMULATIVE", + unknown: "SECRET_AGGREGATE", + dataPoints: [ + { + startTimeUnixNano: "1793763000000000000", + timeUnixNano: "1793763900000000000", + asInt: "42", + attributes: [attr("session.id", "wanted")], + exemplars: [{ filteredAttributes: [attr("private", "SECRET_EXEMPLAR")] }], + unknown: "SECRET_POINT", + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); + const transfers = selectTransfers( + source, + { + config: "claude", + session: "wanted", + tenant: "tenant", + endpoint: "http://127.0.0.1:3011", + token: "test", + home: "/tmp", + follow: false, + }, + { values: new Map(), conflict: false } + ); + expect(transfers).toHaveLength(1); + const sent = JSON.stringify(transfers[0]); + expect(sent).toContain('"asInt":"42"'); + expect(sent).toContain("AGGREGATION_TEMPORALITY_CUMULATIVE"); + expect(sent).not.toContain("SECRET_"); + }); + + it("sends only the selected session's useful events to loopback, without tool output", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const line = { + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + record("wanted", "codex.sse_event"), + record("other", "codex.sse_event"), + record("wanted", "codex.user_prompt"), + ], + }, + ], + }, + ], + }, + }; + const file = join(directory, "codex.jsonl"); + await writeFile(file, `${JSON.stringify(line)}\n`, { mode: 0o600 }); + const received: Array<{ url: string; auth: string; body: string }> = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push({ url: request.url ?? "", auth: request.headers.authorization ?? "", body }); + response.writeHead(200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const before = await readFile(file, "utf8"); + const result = await syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + }, + async () => ({ state: "unbound" }), + async () => true + ); + expect(result.sent).toBe(1); + expect(received).toHaveLength(1); + expect(received[0].url).toBe("/api/kpi/otel/codex/tenant/v1/logs"); + expect(received[0].auth).toBe("Bearer test-token"); + expect(received[0].body).not.toContain("sensitive output"); + expect(received[0].body).toContain("toolu_123"); + expect(received[0].body).not.toContain('"other"'); + expect(await readFile(file, "utf8")).toBe(before); + }); + + it("refuses a non-loopback destination before reading capture or sending credentials", async () => { + await expect( + syncOtel( + { + config: "claude", + session: "wanted", + tenant: "tenant", + endpoint: "http://example.com:3011", + token: "test-token", + home: "/missing", + follow: false, + }, + async () => ({ state: "unbound" }), + async () => true + ) + ).rejects.toThrow(/loopback/); + }); + + it("replays after a rejected delivery and invalidates a formerly confirmed session", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-retry-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const line = { + signal: "logs", + payload: { + resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }], + }, + }; + await writeFile(join(directory, "codex.jsonl"), `${JSON.stringify(line)}\n`, { mode: 0o600 }); + const received: string[] = []; + let reject = true; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(reject ? 503 : 200).end("{}"); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + stateDir: join(home, "state"), + }; + const confirmed = async () => ({ + state: "confirmed" as const, + feature: "owner/repo#7", + branch: "feature/work", + }); + await expect(syncOtel(options, confirmed, async () => true)).rejects.toThrow("HTTP 503"); + reject = false; + await syncOtel(options, confirmed, async () => true); + expect(received.filter((body) => body.includes("codex.sse_event"))).toHaveLength(2); + expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); + await syncOtel( + options, + async () => ({ state: "unbound" }), + async () => true + ); + expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(true); + }); + + it("follows only newly appended complete lines for the chosen session", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-follow-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const line = `${JSON.stringify({ signal: "logs", payload: { resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }] } })}\n`; + const file = join(directory, "codex.jsonl"); + await writeFile(file, line, { mode: 0o600 }); + const abort = new AbortController(); + let received = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + received++; + response.writeHead(200).end("{}"); + if (received === 1) void appendFile(file, line); + if (received === 2) abort.abort(); + }); + }); + await new Promise((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const result = await syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: true, + signal: abort.signal, + intervalMs: 10, + }, + async () => ({ state: "unbound" }), + async () => true + ); + expect(received).toBe(2); + expect(result.sent).toBe(2); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts index adb22ca00..a711a470c 100644 --- a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts @@ -1,4 +1,4 @@ -import { chmod, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; @@ -84,6 +84,46 @@ describe.skipIf(process.platform === "win32")("OTLP verification", () => { expect(JSON.stringify(result)).not.toContain("private-example"); }); + it("refuses a capture symlink and a directory without reading either target", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const target = join(home, "outside.jsonl"); + await writeFile(target, "private\n", { mode: 0o600 }); + await symlink(target, capture); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(); + expect(await readFile(target, "utf8")).toBe("private\n"); + await rm(capture); + await mkdir(capture); + await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/regular file/); + }); + + it("reports counts for valid and malformed lines while ignoring blank lines", async () => { + await initializeOtel("claude", "dev", { home, env: {} }); + const record = { + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { attributes: [{ key: "event.name", value: { stringValue: "user_prompt" } }] }, + { attributes: [{ key: "event.name", value: { stringValue: "tool_result" } }] }, + ], + }, + ], + }, + ], + }, + }; + await writeFile(capture, `\n${JSON.stringify(record)}\nnot-json\n\n`, { mode: 0o600 }); + expect(await verifyOtel("claude", "dev", { home, env: {} })).toMatchObject({ + lines: 2, + observations: 2, + unrecognisedLines: 1, + eventLines: 0, + }); + }); + it("also counts a local OpenCode SSE capture when present", async () => { const host = { home, @@ -108,6 +148,21 @@ describe.skipIf(process.platform === "win32")("OTLP verification", () => { }); }); + it("refuses an OpenCode event capture that is not a regular file", async () => { + const host = { + home, + env: { OPENCODE_CONFIG_CONTENT: '{"experimental":{"openTelemetry":true}}' }, + opencodeVersion: async () => "1.19.5", + }; + await initializeOtel("opencode", "dev", host); + const directory = join(home, ".local/share/aidd-otel-observation"); + await writeFile(join(directory, "opencode.jsonl"), "{}\n", { mode: 0o600 }); + await mkdir(join(directory, "opencode", "events.jsonl"), { recursive: true }); + await expect(verifyOtel("opencode", "dev", host)).rejects.toThrow( + /event capture must be a regular file/ + ); + }); + it("refuses production", async () => { await expect(verifyOtel("claude", "prod", { home, env: {} })).rejects.toThrow(/production/); }); diff --git a/cli/tests/e2e/installation-identity.e2e.test.ts b/cli/tests/e2e/installation-identity.e2e.test.ts index d2961e355..5bd7b88e6 100644 --- a/cli/tests/e2e/installation-identity.e2e.test.ts +++ b/cli/tests/e2e/installation-identity.e2e.test.ts @@ -91,7 +91,7 @@ describe.skipIf(process.platform === "win32")("installation identity across comm const args = ["telemetry", "on", "--yes", "--anonymous"]; try { expect((await runCli(args, projectDir, fakeHome)).exitCode).not.toBe(0); - await expect(stat(path)).rejects.toThrow(); + await expect(readFile(path, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); await gitInit(projectDir); const turnOn = await runCli(args, projectDir, fakeHome); expect(turnOn.exitCode, turnOn.stderr).toBe(0); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 17f229628..4cf7e601f 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -167,7 +167,12 @@ { "invocation": "aidd telemetry otel", "exitCode": 0, - "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local\n capture without changing files\n run [options] [args...] Launch OpenCode 1 with local OTLP export. Pass tool\n arguments after --\n help [command] display help for command" + "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local\n capture without changing files\n collect [options] Capture authenticated OTLP on a local loopback port\n sync [options] Send one consented session's local telemetry to a\n loopback Gouvernail\n run [options] [args...] Launch OpenCode 1 with local OTLP export. Pass tool\n arguments after --\n help [command] display help for command" + }, + { + "invocation": "aidd telemetry otel collect", + "exitCode": 0, + "help": "Usage: aidd telemetry otel collect [options]\n\nCapture authenticated OTLP on a local loopback port\n\nOptions:\n --port Loopback TCP port; 0 selects a free port (default: \"14318\")\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel init", @@ -179,6 +184,11 @@ "exitCode": 0, "help": "Usage: aidd telemetry otel run [options] [args...]\n\nLaunch OpenCode 1 with local OTLP export. Pass tool arguments after --\n\nOptions:\n --config Tool to launch (choices: \"opencode\")\n -h, --help display help for command" }, + { + "invocation": "aidd telemetry otel sync", + "exitCode": 0, + "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config Tool to sync (choices: \"claude\", \"codex\")\n --session Exact provider session ID; omit with --follow to discover\n sessions\n --tenant Gouvernail tenant slug\n --endpoint Loopback HTTP origin\n --follow Follow new events for this session (default: false)\n -h, --help display help for command" + }, { "invocation": "aidd telemetry otel verify", "exitCode": 0, diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index 0381421fd..624cbccc1 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -6,6 +6,7 @@ import { conflictsOf, type ReportFilters, } from "../../../src/contexts/telemetry/domain/report/report-filter.js"; +import * as otel from "../../../src/runtime/wiring/telemetry-otel.js"; const ingest = vi.fn(); const declare = vi.fn(); @@ -21,6 +22,8 @@ const input = vi.fn(); const propose = vi.fn(); const ensureInstallationId = vi.fn(); const checkInstallationId = vi.fn(); +const sessionSnapshot = vi.fn(); +const sessionPermits = vi.fn(); vi.mock("../../../src/runtime/wiring/framework.js", () => ({ createDeps: vi.fn(async () => ({ @@ -37,6 +40,7 @@ vi.mock("../../../src/runtime/wiring/framework.js", () => ({ reportUsageUseCase: { execute: report }, manageIdentityUseCase: identity, proposeIdentityUseCase: { execute: propose }, + sessionFeatureUseCase: { snapshot: sessionSnapshot, permits: sessionPermits }, }, })), createMenuDeps: vi.fn(), @@ -73,6 +77,7 @@ beforeEach(() => { }); afterEach(() => { vi.restoreAllMocks(); + vi.unstubAllEnvs(); }); async function run(...args: string[]): Promise { @@ -379,7 +384,8 @@ describe("aidd telemetry task", () => { it("declares the task it is given, with its ticket", async () => { declare.mockResolvedValue(declared); - await run("task", "checkout", "--ticket", "P-1"); + expect(await run("task", "checkout", "--ticket", "P-1")).toBeNull(); + expect(stdout()).toContain('Declared task "checkout"'); expect(declare).toHaveBeenCalledWith({ cwd: process.cwd(), request: { kind: "task", task: "checkout", ticket: "P-1" }, @@ -414,6 +420,7 @@ describe("aidd telemetry task", () => { it("exits 1 when the declaration is refused", async () => { declare.mockResolvedValue({ status: "refused", reason: "no-consent" }); expect(await run("task", "x")).toBe(1); + expect(stderr()).toContain("no task was declared"); }); }); @@ -671,3 +678,232 @@ describe("telemetry otel init", () => { expect(turnOn).not.toHaveBeenCalled(); }); }); + +describe("native telemetry commands", () => { + let previousExitCode: typeof process.exitCode; + + beforeEach(() => { + previousExitCode = process.exitCode; + }); + + afterEach(() => { + process.exitCode = previousExitCode; + }); + + it("starts the local collector on its default or requested port", async () => { + const collect = vi.spyOn(otel, "collectOtel").mockResolvedValue({ + port: 54321, + close: async () => {}, + }); + expect(await run("otel", "collect")).toBeNull(); + expect(collect).toHaveBeenCalledWith(14318); + expect(stdout()).toContain("OTLP collector listening on http://127.0.0.1:54321"); + expect(await run("otel", "collect", "--port=0")).toBeNull(); + expect(collect).toHaveBeenLastCalledWith(0); + }); + + it("rejects malformed collector ports and reports binding failures", async () => { + const collect = vi.spyOn(otel, "collectOtel").mockRejectedValue(new Error("port in use")); + for (const value of ["abc", "prefix42", "42suffix"]) { + expect(await run("otel", "collect", `--port=${value}`)).toBe(1); + expect(stderr()).toContain("Collector port must be an integer"); + } + expect(collect).not.toHaveBeenCalled(); + expect(await run("otel", "collect", "--port=42")).toBe(1); + expect(stderr()).toContain("port in use"); + }); + + it("reports a new exporter, then reports the same configuration unchanged", async () => { + const configure = vi + .spyOn(otel, "configureOtel") + .mockResolvedValueOnce({ + path: "/config/claude", + launch: "Restart Claude Code", + changed: true, + }) + .mockResolvedValueOnce({ + path: "/config/claude", + launch: "Restart Claude Code", + changed: false, + }); + const args = ["otel", "init", "--config=claude", "--endpoint=dev"]; + expect(await run(...args), stderr()).toBeNull(); + expect(stdout()).toContain("Updated "); + expect(stdout()).toContain("Restart Claude Code"); + expect(stdout()).toContain( + "Local OTLP capture may include prompts, responses, commands and tool output." + ); + out = []; + expect(await run(...args), stderr()).toBeNull(); + expect(stdout()).toContain("Already configured in "); + expect(configure).toHaveBeenCalledTimes(2); + expect(configure).toHaveBeenCalledWith("claude", "dev"); + expect(turnOn).not.toHaveBeenCalled(); + }); + + it("reports capture counts without exposing their content, and fails when no capture exists", async () => { + const check = vi + .spyOn(otel, "checkOtel") + .mockRejectedValueOnce(new Error("No OTLP capture found")) + .mockResolvedValueOnce({ + path: "/config/claude", + capture: "/capture/claude.jsonl", + lines: 1, + modifiedAt: "2026-11-04T00:00:00.000Z", + observations: 1, + eventLines: 0, + unrecognisedLines: 0, + }); + expect(await run("otel", "verify", "--config=claude", "--endpoint=dev")).toBe(1); + expect(stderr()).toContain("No OTLP capture found"); + out = []; + expect(await run("otel", "verify", "--config=claude", "--endpoint=dev")).toBeNull(); + expect(stdout()).toContain("Configuration verified in /config/claude"); + expect(stdout()).toContain("1 captured lines"); + expect(stdout()).toContain("1 normalized observations; 0 event lines; 0 unrecognised lines"); + expect(stdout()).toContain( + "Existing capture does not prove delivery from the current session." + ); + expect(check).toHaveBeenCalledWith("claude", "dev"); + }); + + it("forwards the launched OpenCode exit code", async () => { + const launch = vi + .spyOn(otel, "launchOtel") + .mockResolvedValueOnce(7) + .mockRejectedValueOnce(new Error("Cannot launch OpenCode")); + expect(await run("otel", "run", "--config=opencode", "--", "check")).toBeNull(); + expect(launch).toHaveBeenCalledWith(["check"]); + expect(process.exitCode).toBe(7); + expect(await run("otel", "run", "--config=opencode", "--", "check")).toBe(1); + expect(stderr()).toContain("Cannot launch OpenCode"); + }); + + it("forwards one consented session without printing its token", async () => { + vi.stubEnv("AIDD_OTEL_TOKEN", "private-test-token"); + const admission = { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }; + sessionSnapshot.mockResolvedValue(new Map([["s1", admission]])); + sessionPermits.mockResolvedValueOnce("allow").mockResolvedValueOnce("deny"); + const sync = vi.spyOn(otel, "syncCapturedSession").mockResolvedValue({ + sent: 2, + skipped: 0, + offset: 12, + partialLine: true, + }); + expect( + await run( + "otel", + "sync", + "--config=claude", + "--session=s1", + "--tenant=acme", + "--endpoint=http://127.0.0.1:3000" + ) + ).toBeNull(); + expect(sessionSnapshot).toHaveBeenCalledWith("claude-code"); + const call = sync.mock.calls[0]; + if (!call) throw new Error("the session was not passed to the exporter"); + expect(call[0]).toMatchObject({ + config: "claude", + session: "s1", + tenant: "acme", + endpoint: "http://127.0.0.1:3000", + token: "private-test-token", + follow: false, + }); + expect(await call[1]()).toEqual({ state: "unbound" }); + const transfer = { signal: "logs" as const, payload: {}, at: "2026-11-04T00:00:00Z" }; + expect(await call[2](transfer)).toBe(true); + expect(await call[2](transfer)).toBe(false); + expect(await call[2]({ signal: "logs", payload: {} })).toBe(false); + expect(stdout()).toContain("Sent 2 observations"); + expect(stdout()).toContain("last incomplete capture line"); + expect(stdout()).not.toContain("private-test-token"); + }); + + it("never sends a session that lacks consent", async () => { + sessionSnapshot.mockResolvedValue(new Map()); + const sync = vi.spyOn(otel, "syncCapturedSession"); + expect( + await run( + "otel", + "sync", + "--config=codex", + "--session=s2", + "--tenant=acme", + "--endpoint=http://127.0.0.1:3000" + ) + ).toBe(1); + expect(sessionSnapshot).toHaveBeenCalledWith("codex"); + expect(stderr()).toContain("No consented usage found"); + expect(sync).not.toHaveBeenCalled(); + }); + + it("stops admitting events when a followed session loses consent", async () => { + vi.stubEnv("AIDD_OTEL_TOKEN", ""); + const admission = { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }; + sessionSnapshot + .mockResolvedValueOnce(new Map([["s1", admission]])) + .mockResolvedValueOnce(new Map([["s1", admission]])) + .mockResolvedValueOnce(new Map()); + sessionPermits.mockResolvedValue("allow"); + const sync = vi.spyOn(otel, "syncCapturedSession").mockResolvedValue({ + sent: 1, + skipped: 0, + offset: 1, + partialLine: false, + }); + expect( + await run( + "otel", + "sync", + "--config=codex", + "--session=s1", + "--tenant=acme", + "--endpoint=http://127.0.0.1:3000", + "--follow" + ) + ).toBeNull(); + expect(stdout()).toContain("Following the selected session"); + expect(stdout()).not.toContain("last incomplete capture line"); + const call = sync.mock.calls[0]; + if (!call) throw new Error("the session was not passed to the exporter"); + expect(call[0]).toMatchObject({ follow: true, token: "" }); + await expect(call[1]()).rejects.toThrow("No consented usage found"); + expect(await call[2]({ signal: "logs", payload: {}, at: "2026-11-04T00:00:00Z" })).toBe(false); + }); + + it("requires a session for one-shot sync and discovers sessions only with follow", async () => { + vi.stubEnv("AIDD_OTEL_TOKEN", "private-test-token"); + const follow = vi + .spyOn(otel, "followCapturedSessions") + .mockResolvedValue({ sent: 3, denied: 1, pending: 2 }); + const args = [ + "otel", + "sync", + "--config=codex", + "--tenant=acme", + "--endpoint=http://127.0.0.1:3000", + ]; + expect(await run(...args)).toBeNull(); + expect(process.exitCode).toBe(1); + expect(stderr()).toContain("Pass --session"); + expect(follow).not.toHaveBeenCalled(); + process.exitCode = previousExitCode; + sessionSnapshot.mockResolvedValue(new Map()); + sessionPermits.mockResolvedValue("unknown"); + expect(await run(...args, "--follow")).toBeNull(); + const call = follow.mock.calls[0]; + if (!call) throw new Error("the session follower was not started"); + expect(call[0]).toMatchObject({ config: "codex", tenant: "acme", token: "private-test-token" }); + expect(await call[1]()).toEqual(new Map()); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + expect(await call[2](admission, "2026-11-04T00:00:00Z")).toBe("unknown"); + expect(stdout()).toContain("Sent 3 observations; 1 were outside consent; 2 await confirmation"); + expect(stdout()).not.toContain("private-test-token"); + }); +}); diff --git a/cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts b/cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts new file mode 100644 index 000000000..f2c0e9021 --- /dev/null +++ b/cli/tests/runtime/filesystem/installation-identity-race.integration.test.ts @@ -0,0 +1,52 @@ +import { link, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; +import { ensureInstallationId } from "../../../src/runtime/filesystem/installation-identity.js"; + +vi.mock("node:fs/promises", async (importOriginal) => { + const original = await importOriginal(); + return { ...original, link: vi.fn(original.link) }; +}); + +const roots: string[] = []; +afterEach(async () => { + vi.mocked(link).mockReset(); + await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); +}); + +async function makeRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "aidd-installation-race-")); + roots.push(root); + return root; +} + +it("keeps the identity installed by a concurrent initializer", async () => { + const root = await makeRoot(); + const winner = "a720868e-d80a-4c71-96aa-2ce56936b35d"; + vi.mocked(link).mockImplementationOnce(async (_temporary, path) => { + await writeFile(path, JSON.stringify({ installation_id: winner }), { mode: 0o600 }); + throw Object.assign(new Error("already exists"), { code: "EEXIST" }); + }); + + expect(await ensureInstallationId(root)).toBe(winner); + expect(await readdir(join(root, "installation"))).toEqual(["id.json"]); +}); + +it("cleans up the temporary identity if the filesystem rejects installation", async () => { + const root = await makeRoot(); + vi.mocked(link).mockRejectedValueOnce(Object.assign(new Error("disk error"), { code: "EIO" })); + + await expect(ensureInstallationId(root)).rejects.toMatchObject({ code: "EIO" }); + expect(await readdir(join(root, "installation"))).toEqual([]); +}); + +it("reports when a concurrently installed identity disappears", async () => { + const root = await makeRoot(); + vi.mocked(link).mockRejectedValueOnce( + Object.assign(new Error("already exists"), { code: "EEXIST" }) + ); + + await expect(ensureInstallationId(root)).rejects.toThrow("disappeared during setup"); + expect(await readdir(join(root, "installation"))).toEqual([]); +}); diff --git a/cli/tests/runtime/filesystem/installation-identity.integration.test.ts b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts index b9ca02b29..dcd923b52 100644 --- a/cli/tests/runtime/filesystem/installation-identity.integration.test.ts +++ b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts @@ -1,4 +1,14 @@ -import { chmod, mkdir, mkdtemp, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; +import { + chmod, + mkdir, + mkdtemp, + readdir, + readFile, + rm, + stat, + symlink, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, describe, expect, it } from "vitest"; @@ -29,6 +39,7 @@ describe("installation identity", () => { expect(await ensureInstallationId(root)).toBe(ids[0]); const path = join(root, "installation", "id.json"); expect(JSON.parse(await readFile(path, "utf8"))).toEqual({ installation_id: ids[0] }); + expect(await readdir(join(root, "installation"))).toEqual(["id.json"]); if (process.platform !== "win32") expect((await stat(path)).mode & 0o777).toBe(0o600); }); @@ -54,6 +65,42 @@ describe("installation identity", () => { ); }); + it("checks an absent installation without creating one", async () => { + const root = await makeRoot(); + expect(await checkInstallationId(root)).toBeNull(); + expect(await readdir(root)).toEqual([]); + }); + + it.each([ + '{"installation_id":"prefix-a720868e-d80a-4c71-96aa-2ce56936b35d"}', + '{"installation_id":"a720868e-d80a-4c71-96aa-2ce56936b35d-suffix"}', + '{"installation_id":"a720868e-d80a-1c71-96aa-2ce56936b35d"}', + '{"installation_id":null}', + "{}", + "[]", + "{invalid", + ])("refuses malformed stored identity %s", async (stored) => { + const root = await makeRoot(); + const directory = join(root, "installation"); + await mkdir(directory); + await writeFile(join(directory, "id.json"), stored, { mode: 0o600 }); + await expect(checkInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + await expect(ensureInstallationId(root)).rejects.toThrow("existing file was left unchanged"); + expect(await readFile(join(directory, "id.json"), "utf8")).toBe(stored); + }); + + it("does not confuse a non-directory parent with a missing identity", async () => { + const root = await makeRoot(); + await writeFile(join(root, "installation"), "not a directory"); + await expect(checkInstallationId(root)).rejects.toMatchObject({ code: "ENOTDIR" }); + }); + + it.skipIf(process.platform === "win32")("rejects a directory as the identity file", async () => { + const root = await makeRoot(); + await mkdir(join(root, "installation", "id.json"), { recursive: true }); + await expect(checkInstallationId(root)).rejects.toThrow("private regular file"); + }); + it.skipIf(process.platform === "win32")("refuses a symlink without following it", async () => { const root = await makeRoot(); const directory = join(root, "installation"); @@ -90,4 +137,12 @@ describe("installation identity", () => { "OTLP configuration completed, but the installation ID could not be saved: disk full" ); }); + + it("reports non-Error failures after the earlier action already succeeded", async () => { + await expect( + finishInstallationId(async () => { + throw "disk full"; + }, "Measurement is on") + ).rejects.toThrow("Measurement is on, but the installation ID could not be saved: disk full"); + }); }); diff --git a/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts b/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts index da0c88c36..580b6fd07 100644 --- a/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts +++ b/cli/tests/runtime/filesystem/private-storage-adapter.integration.test.ts @@ -141,6 +141,14 @@ describe("replacing a file", () => { await expect(storage.replace(path, "x")).rejects.toThrow(); expect(await readdir(dir)).toEqual([]); }); + + it("removes the temporary file when the completed write cannot replace a directory", async () => { + const path = join(dir, "target"); + await mkdir(path); + await expect(storage.replace(path, "x")).rejects.toThrow(); + expect(await readdir(dir)).toEqual(["target"]); + expect((await stat(path)).isDirectory()).toBe(true); + }); }); describe("appending to a file", () => { diff --git a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts new file mode 100644 index 000000000..c08c4df01 --- /dev/null +++ b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts @@ -0,0 +1,155 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ + initialize: vi.fn(), + verify: vi.fn(), + launch: vi.fn(), + checkId: vi.fn(), + ensureId: vi.fn(), + finishId: vi.fn(), +})); + +vi.mock("node:os", async (load) => ({ + ...(await load()), + homedir: () => "/profile", +})); +vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/init.js", () => ({ + initializeOtel: mocks.initialize, +})); +vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js", () => ({ + verifyOtel: mocks.verify, +})); +vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/run.js", () => ({ + runOtelOpenCode: mocks.launch, +})); +vi.mock("../../../src/runtime/filesystem/installation-identity.js", () => ({ + checkInstallationId: mocks.checkId, + ensureInstallationId: mocks.ensureId, + finishInstallationId: mocks.finishId, +})); +vi.mock("../../../src/runtime/user-config-dir.js", () => ({ + userConfigDirOf: () => "/profile/aidd", +})); +vi.mock("../../../src/runtime/wiring/telemetry.js", () => ({ + telemetryDir: () => "/profile/telemetry", +})); + +const { + collectOtel, + configureOtel, + checkOtel, + launchOtel, + syncCapturedSession, + followCapturedSessions, +} = await import("../../../src/runtime/wiring/telemetry-otel.js"); +const collector = await import( + "../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js" +); +const sync = await import("../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"); +const follow = await import( + "../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js" +); + +beforeEach(() => { + vi.clearAllMocks(); + mocks.checkId.mockReset().mockResolvedValue(null); + mocks.initialize + .mockReset() + .mockResolvedValue({ path: "/config", changed: true, launch: "restart" }); + mocks.ensureId.mockReset().mockResolvedValue("installation-id"); + mocks.finishId.mockReset().mockImplementation(async (ensure: () => Promise) => { + await ensure(); + }); +}); + +describe("OTLP runtime wiring", () => { + it("places the collector in the user's profile and uses the requested port", async () => { + const start = vi.spyOn(collector, "startOtelCollector").mockResolvedValue({ + port: 4318, + close: async () => {}, + }); + await collectOtel(4318); + expect(start).toHaveBeenCalledWith({ home: "/profile", port: 4318 }); + start.mockRestore(); + }); + + it("validates identity before initialization and saves it only after a successful change", async () => { + const result = await configureOtel("claude", "dev"); + expect(result).toMatchObject({ path: "/config", changed: true }); + expect(mocks.checkId).toHaveBeenCalledWith("/profile/aidd"); + expect(mocks.initialize).toHaveBeenCalledWith("claude", "dev", { + home: "/profile", + env: process.env, + }); + expect(mocks.checkId.mock.invocationCallOrder[0]).toBeLessThan( + mocks.initialize.mock.invocationCallOrder[0] + ); + expect(mocks.finishId.mock.invocationCallOrder[0]).toBeGreaterThan( + mocks.initialize.mock.invocationCallOrder[0] + ); + expect(mocks.ensureId).toHaveBeenCalledWith("/profile/aidd"); + }); + + it("does not change exporter settings when an existing installation ID is invalid", async () => { + mocks.checkId.mockRejectedValue(new Error("invalid identity")); + await expect(configureOtel("codex", "dev")).rejects.toThrow("invalid identity"); + expect(mocks.initialize).not.toHaveBeenCalled(); + expect(mocks.ensureId).not.toHaveBeenCalled(); + }); + + it("does not create an ID when a production destination is refused", async () => { + mocks.initialize.mockRejectedValue(new Error("production unavailable")); + await expect(configureOtel("codex", "prod")).rejects.toThrow("production unavailable"); + expect(mocks.checkId).not.toHaveBeenCalled(); + expect(mocks.ensureId).not.toHaveBeenCalled(); + }); + + it("passes its runtime home and environment to verification and launch", async () => { + mocks.verify.mockResolvedValue({ lines: 1 }); + mocks.launch.mockResolvedValue(7); + await expect(checkOtel("codex", "dev")).resolves.toEqual({ lines: 1 }); + await expect(launchOtel(["check"])).resolves.toBe(7); + expect(mocks.verify).toHaveBeenCalledWith("codex", "dev", { + home: "/profile", + env: process.env, + }); + expect(mocks.launch).toHaveBeenCalledWith(["check"], { home: "/profile", env: process.env }); + }); + + it("uses the private runtime profile for one session and automatic following", async () => { + const bounded = vi + .spyOn(sync, "syncOtel") + .mockResolvedValue({ sent: 1, skipped: 0, offset: 5, partialLine: false }); + const automatic = vi + .spyOn(follow, "syncAllOtel") + .mockResolvedValue({ sent: 2, denied: 0, pending: 0 }); + const options = { + config: "codex" as const, + tenant: "acme", + endpoint: "http://127.0.0.1:3000", + token: "test-token", + }; + const resolve = async () => ({ state: "unbound" as const }); + const allow = async () => true; + const snapshot = async () => new Map(); + const permits = async () => "allow" as const; + await syncCapturedSession({ ...options, session: "s1", follow: false }, resolve, allow); + await followCapturedSessions(options, snapshot, permits); + expect(bounded).toHaveBeenCalledWith( + { + ...options, + session: "s1", + follow: false, + home: "/profile", + stateDir: "/profile/telemetry", + }, + resolve, + allow + ); + expect(automatic).toHaveBeenCalledWith( + { ...options, home: "/profile", stateDir: "/profile/telemetry" }, + snapshot, + permits + ); + }); +}); diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 3a341df27..972610671 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -36,6 +36,23 @@ The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 - OpenCode 2's rich session events are SSE, not the observed OTLP logs. The O1 schema above does not prove its fields are emitted over OTLP. The schema event's `durable.version` is not an application major version. - Prompts, responses, tool arguments and results can contain secrets. Capture and retain them only with the appropriate consent and local access controls. +## Local Gouvernail sync + +With the local OTLP capture already running, start one process per configured provider: + +```sh +# Set the relevant connector token in the environment without putting it in command arguments. +export AIDD_OTEL_TOKEN='' +aidd telemetry otel sync --config=claude --tenant=acme --endpoint=http://127.0.0.1:3011 --follow +# In another shell, use the Codex connector token and --config=codex. +``` + +The origin must be loopback HTTP; no token or prompt content is printed. Only supported observations from sessions admitted by the local ledger and covered by an uninterrupted consent interval are sent. A Claude cumulative metric whose whole interval is not covered is excluded. A successful HTTP response does not prove that Gouvernail matched a member: check its session view. + +An automatic follower starts with new capture events and keeps a private checkpoint. Network errors and HTTP 5xx responses are reported and retried with bounded backoff; permanent HTTP failures stop the follower. Restarting resumes from the checkpoint, and a failed post is replayed, so the receiver must remain idempotent. Unknown sessions wait for their transcript to be admitted using private byte-range indexes, without holding up other sessions. These indexes can grow until admission and depend on retaining the original capture; replaced or truncated captures require recovery rather than silently skipping data. Running a follower for each provider is an explicit operation; it does not start automatically after a machine restart. Linked session identifiers and the minimum provider identity are stored privately to detect later ambiguity, including after restart. Older sync state without identity must be refreshed with a session replay. + +To replay one consented session already in the capture, pass `--session=` without `--follow`. Add `--follow` to follow only that session. A feature link is sent only when every recorded call agrees on a single observed branch and a declared canonical ticket such as `owner/repo#123`; otherwise it remains unlinked. If a linked session later becomes ambiguous, an irreversible invalidation keeps it unlinked even if events arrive out of order. + ## Sources Read on 2026-11-04: diff --git a/scripts/__tests__/cli-opens-no-server.test.js b/scripts/__tests__/cli-opens-no-server.test.js index ad8bf99f8..85b6e4f31 100644 --- a/scripts/__tests__/cli-opens-no-server.test.js +++ b/scripts/__tests__/cli-opens-no-server.test.js @@ -6,12 +6,12 @@ const { describe, it } = require("node:test"); const ROOT = path.resolve(__dirname, "../.."); const CLI_SRC_PREFIX = "cli/src/"; +const COLLECTOR = "cli/src/contexts/telemetry/infrastructure/otel/actions/collect.ts"; /** - * Nothing this system runs opens a network listener, and the constraint has to stay true of - * the code rather than of a document. This walks every tracked source file under `cli/src` - * and greps for the literal patterns a listener is built from, so a future listener has to be - * a deliberate decision โ€” this assertion updated, with a reason โ€” never an accident. + * The local OTLP collector is the sole listener in the CLI. This walks every tracked source + * file under `cli/src` so another listener needs an explicit decision, never an accident. + * The collector's integration tests verify its loopback bind and required authentication. * * Deliberately narrow to server construction, not merely importing `node:http` or `node:net`: * this codebase makes plenty of outbound calls, and a client is not what this forbids. @@ -41,26 +41,53 @@ function trackedCliSourceFiles() { .filter((rel) => rel.endsWith(".ts")); } -describe("nothing in cli/src opens a network listener", () => { - it("no source file constructs or binds a server", () => { +function listenersIn(files) { + const offenders = []; + let collectorConstructors = 0; + let collectorBinds = 0; + for (const [rel, text] of files) { + for (const { pattern, label } of FORBIDDEN_PATTERNS) { + const count = [...text.matchAll(new RegExp(pattern.source, "gu"))].length; + if (rel === COLLECTOR && label === "createServer(") collectorConstructors = count; + else if (rel === COLLECTOR && label === ".listen(") collectorBinds = count; + else if (count > 0) offenders.push(`${rel}: ${label}`); + } + } + return { offenders, collectorConstructors, collectorBinds }; +} + +describe("the OTLP collector is the only listener in cli/src", () => { + it("allows one collector server and forbids all other listeners", () => { const files = trackedCliSourceFiles(); // A walk that found nothing would pass this test while checking nothing at all. assert.ok(files.length > 300, `expected the CLI's source tree, found ${files.length} files`); + const { offenders, collectorConstructors, collectorBinds } = listenersIn( + files.map((rel) => [rel, fs.readFileSync(path.join(ROOT, rel), "utf8")]) + ); - const offenders = []; - for (const rel of files) { - const text = fs.readFileSync(path.join(ROOT, rel), "utf8"); - for (const { pattern, label } of FORBIDDEN_PATTERNS) { - if (pattern.test(text)) offenders.push(`${rel}: ${label}`); - } - } - + assert.equal(collectorConstructors, 1, "the collector must construct exactly one server"); + assert.equal(collectorBinds, 1, "the collector must bind exactly one listener"); assert.deepEqual( offenders, [], - `a source file opens or names a network listener - this codebase's one hard ` + - `constraint is that nothing it runs does that:\n ${offenders.join("\n ")}` + `only the local OTLP collector may open a listener:\n ${offenders.join("\n ")}` ); }); + + it("detects a second collector listener or a listener in another source file", () => { + const collector = "createServer(handler); server.listen(1);"; + const duplicate = listenersIn([[COLLECTOR, `${collector} server.listen(2);`]]); + assert.equal(duplicate.collectorConstructors, 1); + assert.equal(duplicate.collectorBinds, 2); + assert.equal(listenersIn([[COLLECTOR, `${collector} createServer(handler);`]]).collectorConstructors, 2); + const other = listenersIn([ + [COLLECTOR, collector], + ["cli/src/cli.ts", "createServer(handler); server.listen(3);"], + ]); + assert.deepEqual(other.offenders, ["cli/src/cli.ts: createServer(", "cli/src/cli.ts: .listen("]); + assert.deepEqual(listenersIn([[COLLECTOR, 'import net from "node:net";']]).offenders, [ + `${COLLECTOR}: import from "node:net"`, + ]); + }); }); From 82eae337e020cb86473ce23cd4590704c056a76a Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 17:19:48 +0800 Subject: [PATCH 07/56] test(telemetry): isolate collector fixtures --- .../otel/collector.integration.test.ts | 40 +++++++++++++++---- .../otel/init-otel.integration.test.ts | 13 +++++- 2 files changed, 44 insertions(+), 9 deletions(-) diff --git a/cli/tests/contexts/telemetry/otel/collector.integration.test.ts b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts index d514095cb..3e3886b3d 100644 --- a/cli/tests/contexts/telemetry/otel/collector.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts @@ -6,8 +6,17 @@ import { gzipSync } from "node:zlib"; import { expect, it, vi } from "vitest"; import { startOtelCollector } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js"; +const TOKEN = "example-private-collector-token"; + +async function seedToken(home: string): Promise { + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + await writeFile(join(directory, "collector-token"), `${TOKEN}\n`, { mode: 0o600 }); +} + it("captures authenticated OTLP locally, rejects unauthenticated requests, and keeps files private", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-collector-")); + await seedToken(home); const listen = vi.spyOn(Server.prototype, "listen"); const collector = await startOtelCollector({ home, port: 0 }); try { @@ -18,9 +27,11 @@ it("captures authenticated OTLP locally, rejects unauthenticated requests, and k expect(await readFile(file, "utf8")).toBe(""); expect((await stat(file)).mode & 0o777).toBe(0o600); } - const token = ( - await readFile(join(home, ".local/share/aidd-otel-observation/collector-token"), "utf8") - ).trim(); + expect( + ( + await readFile(join(home, ".local/share/aidd-otel-observation/collector-token"), "utf8") + ).trim() + ).toBe(TOKEN); const url = `http://127.0.0.1:${collector.port}/claude/v1/logs`; const body = JSON.stringify({ resourceLogs: [{ scopeLogs: [] }] }); expect((await fetch(url, { method: "POST", body })).status).toBe(403); @@ -32,7 +43,7 @@ it("captures authenticated OTLP locally, rejects unauthenticated requests, and k ( await fetch(url, { method: "POST", - headers: { authorization: `Bearer ${token}`, "content-encoding": "gzip" }, + headers: { authorization: `Bearer ${TOKEN}`, "content-encoding": "gzip" }, body: gzipSync(body), }) ).status @@ -41,7 +52,7 @@ it("captures authenticated OTLP locally, rejects unauthenticated requests, and k ( await fetch(url, { method: "POST", - headers: { authorization: `Bearer ${token}` }, + headers: { authorization: `Bearer ${TOKEN}` }, body: "not json", }) ).status @@ -50,7 +61,7 @@ it("captures authenticated OTLP locally, rejects unauthenticated requests, and k ( await fetch(`http://127.0.0.1:${collector.port}/not-a-tool/v1/logs`, { method: "POST", - headers: { authorization: `Bearer ${token}` }, + headers: { authorization: `Bearer ${TOKEN}` }, body, }) ).status @@ -72,6 +83,19 @@ it("captures authenticated OTLP locally, rejects unauthenticated requests, and k } }); +it("creates a private random token when none exists", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-token-")); + const collector = await startOtelCollector({ home, port: 0 }); + try { + const path = join(home, ".local/share/aidd-otel-observation/collector-token"); + expect((await readFile(path, "utf8")).trim()).toMatch(/^[A-Za-z0-9_-]{43}$/); + expect((await stat(path)).mode & 0o777).toBe(0o600); + } finally { + await collector.close(); + await rm(home, { recursive: true, force: true }); + } +}); + it("does not follow an existing token symlink or replace its target", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-collector-link-")); const directory = join(home, ".local/share/aidd-otel-observation"); @@ -89,16 +113,16 @@ it("does not follow an existing token symlink or replace its target", async () = it("serializes concurrent appends into complete records", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-collector-concurrent-")); + await seedToken(home); const collector = await startOtelCollector({ home, port: 0 }); try { const directory = join(home, ".local/share/aidd-otel-observation"); - const token = (await readFile(join(directory, "collector-token"), "utf8")).trim(); const url = `http://127.0.0.1:${collector.port}/codex/v1/logs`; const statuses = await Promise.all( Array.from({ length: 8 }, (_, number) => fetch(url, { method: "POST", - headers: { authorization: `Bearer ${token}` }, + headers: { authorization: `Bearer ${TOKEN}` }, body: JSON.stringify({ number }), }).then((response) => response.status) ) diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 53c19f4cd..6ee7dffe0 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -49,7 +49,6 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { expect(result.changed).toBe(true); const handle = await open(path, "r"); const content = await handle.readFile("utf8"); - expect((await handle.stat()).mode & 0o777).toBe(0o600); await handle.close(); expect(JSON.parse(content)).toMatchObject({ permissions: { allow: [] }, @@ -68,11 +67,23 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { const after = await open(path, "r"); try { expect(await after.readFile("utf8")).toBe(content); + expect((await after.stat()).mode & 0o777).toBe(0o600); } finally { await after.close(); } }); + it("writes Claude settings privately on the first initialization", async () => { + await seedToken(); + await initializeOtel("claude", "dev", { home, env: {} }); + const handle = await open(join(home, ".claude/settings.json"), "r"); + try { + expect((await handle.stat()).mode & 0o777).toBe(0o600); + } finally { + await handle.close(); + } + }); + it("refuses a symlink without reading or replacing its target", async () => { await seedToken(); const target = join(home, "original.json"); From b8b7e50ca2892d3edb2700b36b9542c1d1564372 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 17:26:01 +0800 Subject: [PATCH 08/56] test(telemetry): compare initialization contents directly --- .../telemetry/otel/init-otel.integration.test.ts | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 6ee7dffe0..d15f7a8f5 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -47,9 +47,7 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { ); const result = await initializeOtel("claude", "dev", { home, env: {} }); expect(result.changed).toBe(true); - const handle = await open(path, "r"); - const content = await handle.readFile("utf8"); - await handle.close(); + const content = await readFile(path, "utf8"); expect(JSON.parse(content)).toMatchObject({ permissions: { allow: [] }, env: { @@ -64,13 +62,8 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { expect(await initializeOtel("claude", "dev", { home, env: {} })).toMatchObject({ changed: false, }); - const after = await open(path, "r"); - try { - expect(await after.readFile("utf8")).toBe(content); - expect((await after.stat()).mode & 0o777).toBe(0o600); - } finally { - await after.close(); - } + expect(await readFile(path, "utf8")).toBe(content); + expect((await stat(path)).mode & 0o777).toBe(0o600); }); it("writes Claude settings privately on the first initialization", async () => { From ed6f7e6d3b007cadc734e3bedd4f288b7209e256 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 17:54:03 +0800 Subject: [PATCH 09/56] test(telemetry): cover platform and command behavior --- .../telemetry/otel/collector.integration.test.ts | 14 +++++++++++++- .../otel/sync-all-otel.integration.test.ts | 3 ++- .../telemetry/otel/sync-otel.integration.test.ts | 7 ++++--- .../commands/telemetry-wiring.integration.test.ts | 9 ++++++++- .../installation-identity.integration.test.ts | 7 +++++-- plugins/aidd-telemetry/OBSERVABILITY.md | 2 ++ 6 files changed, 34 insertions(+), 8 deletions(-) diff --git a/cli/tests/contexts/telemetry/otel/collector.integration.test.ts b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts index 3e3886b3d..253d4a70d 100644 --- a/cli/tests/contexts/telemetry/otel/collector.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/collector.integration.test.ts @@ -3,9 +3,10 @@ import { Server } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { gzipSync } from "node:zlib"; -import { expect, it, vi } from "vitest"; +import { expect, it as test, vi } from "vitest"; import { startOtelCollector } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js"; +const it = test.skipIf(process.platform === "win32"); const TOKEN = "example-private-collector-token"; async function seedToken(home: string): Promise { @@ -137,3 +138,14 @@ it("serializes concurrent appends into complete records", async () => { await rm(home, { recursive: true, force: true }); } }); + +test.runIf(process.platform === "win32")("clearly refuses collection on Windows", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-collector-windows-")); + try { + await expect(startOtelCollector({ home, port: 0 })).rejects.toThrow( + "OTLP collection requires private POSIX files" + ); + } finally { + await rm(home, { recursive: true, force: true }); + } +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts index 10008368c..656de1a88 100644 --- a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts @@ -2,9 +2,10 @@ import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "no import { createServer, type Server } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { afterEach, expect, it } from "vitest"; +import { afterEach, expect, it as test } from "vitest"; import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; +const it = test.skipIf(process.platform === "win32"); const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); function line(session: string) { return `${JSON.stringify({ diff --git a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts index 7a9d419a6..8d9d47275 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts @@ -2,12 +2,13 @@ import { appendFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/pro import { createServer, type Server } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { afterEach, describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it as test } from "vitest"; import { selectTransfers, syncOtel, } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; +const it = test.skipIf(process.platform === "win32"); const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); const record = (session: string, event: string) => ({ timeUnixNano: "1793763900000000000", @@ -31,7 +32,7 @@ describe("OTLP capture sync", () => { server = undefined; }); - it("retains numeric metric data but excludes exemplars and unknown properties", () => { + test("retains numeric metric data but excludes exemplars and unknown properties", () => { const source = JSON.stringify({ signal: "metrics", payload: { @@ -147,7 +148,7 @@ describe("OTLP capture sync", () => { expect(await readFile(file, "utf8")).toBe(before); }); - it("refuses a non-loopback destination before reading capture or sending credentials", async () => { + test("refuses a non-loopback destination before reading capture or sending credentials", async () => { await expect( syncOtel( { diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index 624cbccc1..e0e231d70 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -6,6 +6,7 @@ import { conflictsOf, type ReportFilters, } from "../../../src/contexts/telemetry/domain/report/report-filter.js"; +import { createDeps } from "../../../src/runtime/wiring/framework.js"; import * as otel from "../../../src/runtime/wiring/telemetry-otel.js"; const ingest = vi.fn(); @@ -372,6 +373,12 @@ describe("aidd telemetry status", () => { expect(await run("status")).toBeNull(); expect(show).toHaveBeenCalledWith(process.cwd()); }); + + it("passes the global verbose setting to the telemetry dependencies", async () => { + show.mockResolvedValue({ status: "refused", reason: "no-consent" }); + expect(await run("status", "--verbose")).toBeNull(); + expect(createDeps).toHaveBeenCalledWith(process.cwd(), { verbose: true }, expect.anything()); + }); }); describe("aidd telemetry task", () => { @@ -479,7 +486,7 @@ describe("aidd telemetry on", () => { }); it("refuses an identifier the identity rules refuse, before asking anything", async () => { - expect(await run("on", "--as", "a".repeat(129))).toBe(1); + expect(await run("on", "--yes", "--as", "a".repeat(129))).toBe(1); expect(confirm).not.toHaveBeenCalled(); expect(turnOn).not.toHaveBeenCalled(); expect(stderr()).toContain( diff --git a/cli/tests/runtime/filesystem/installation-identity.integration.test.ts b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts index dcd923b52..a87291ec3 100644 --- a/cli/tests/runtime/filesystem/installation-identity.integration.test.ts +++ b/cli/tests/runtime/filesystem/installation-identity.integration.test.ts @@ -89,10 +89,13 @@ describe("installation identity", () => { expect(await readFile(join(directory, "id.json"), "utf8")).toBe(stored); }); - it("does not confuse a non-directory parent with a missing identity", async () => { + it("refuses to initialize when the parent is not a directory", async () => { const root = await makeRoot(); await writeFile(join(root, "installation"), "not a directory"); - await expect(checkInstallationId(root)).rejects.toMatchObject({ code: "ENOTDIR" }); + if (process.platform === "win32") await expect(checkInstallationId(root)).resolves.toBeNull(); + else await expect(checkInstallationId(root)).rejects.toMatchObject({ code: "ENOTDIR" }); + await expect(ensureInstallationId(root)).rejects.toThrow(); + expect(await readFile(join(root, "installation"), "utf8")).toBe("not a directory"); }); it.skipIf(process.platform === "win32")("rejects a directory as the identity file", async () => { diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 972610671..5b4d07e2e 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -38,6 +38,8 @@ The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 ## Local Gouvernail sync +Local OTLP collection and sync require private POSIX files and are not supported on Windows. + With the local OTLP capture already running, start one process per configured provider: ```sh From ffbe27efb94365e5f7942efc97626bb99048ab75 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Wed, 4 Nov 2026 17:55:16 +0800 Subject: [PATCH 10/56] docs(telemetry): explain local collector startup --- plugins/aidd-telemetry/OBSERVABILITY.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 5b4d07e2e..b1d42abf9 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -40,7 +40,16 @@ The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 Local OTLP collection and sync require private POSIX files and are not supported on Windows. -With the local OTLP capture already running, start one process per configured provider: +Start the collector, then configure the tools in another shell: + +```sh +aidd telemetry otel collect +# In another shell: +aidd telemetry otel init --config=claude --endpoint=dev +aidd telemetry otel init --config=codex --endpoint=dev +``` + +Restart the tools to load their settings. With local telemetry measurement enabled, start one sync process per provider: ```sh # Set the relevant connector token in the environment without putting it in command arguments. From f49ba8e7fa4372b3f59304d34a07570de495ccf1 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Thu, 5 Nov 2026 07:51:21 +0800 Subject: [PATCH 11/56] docs(telemetry): explain consent, data and identity --- plugins/aidd-telemetry/OBSERVABILITY.md | 88 ++++++++++++++----- plugins/aidd-telemetry/README.md | 36 +++++++- .../skills/00-init/actions/03-identity.md | 12 ++- .../skills/00-init/actions/04-stop.md | 13 ++- .../skills/00-init/actions/05-forget.md | 6 +- .../skills/00-init/actions/06-otel-init.md | 8 +- 6 files changed, 126 insertions(+), 37 deletions(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index b1d42abf9..384497afb 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -1,28 +1,32 @@ # Provider telemetry field mapping +Use this page to see what each provider can emit, what the local collector keeps, and what the development sync actually sends. For the three modes, opt-in and identity rules, start with the [README](README.md#choose-a-mode). + This is an inventory of source fields, not a promise that the AIDD report ingests them. Claude Code and Codex fields below were observed in local OTLP captures; OpenCode 2 fields were observed in its session event stream. OpenCode 1 fields are from its published session schema, not a verified OTLP capture. Presence depends on version, configuration, event and provider. A missing value is unknown, not zero. The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 and `O2` is OpenCode 2. OTLP attribute names below are keys in the record, span or resource's `attributes` array, not necessarily JSON object properties. For O1, `info` denotes the message in `message.updated`, `part` the part in `message.part.updated`, and session information comes from `session.created` or `session.updated`; these are session-event schemas, not OTLP attributes. -| Common element | Data | Example | Source properties | -| --- | --- | --- | --- | -| Time | Event or operation time | `2026-11-04T10:20:00+08:00` | **C/X:** `event.timestamp`, `timeUnixNano`, `observedTimeUnixNano`; spans have `startTimeUnixNano`, `endTimeUnixNano`. **O1:** `info.time.created`, `part.time.start` / `end` when provided. **O2:** `created`, `data.started` on step start. | -| Event type | Kind of event | `tool_result` | **C:** `event.name`, log `body.stringValue`, span or metric name. **X:** `event.name`; `eventName` may identify a source location instead. **O1/O2:** event `type`. | -| Application version | Client version | `2.2.5` | **C:** resource `service.version`. **X:** `app.version`, resource `service.version`. **O1:** session `info.version`. **O2:** `data.version` on `session.created`; OTLP resource `service.version`. | -| Session | Tool-specific session ID | `session-abc` | **C:** `session.id`. **X:** `conversation.id`; trace `thread.id`. **O1:** `info.sessionID`, `part.sessionID`. **O2:** `data.sessionID`; session-event `durable.aggregateID`. | -| Provider | Provider identifier | `openai` | **C:** `anthropic.provider`, `gen_ai.provider.name`, `gen_ai.system`. **X:** `provider_name`, trace `provider`. **O1:** assistant `info.providerID`, user `info.model.providerID`. **O2:** `data.model.providerID` on step start. Provider labels are not necessarily equivalent across tools. | -| Model | Model identifier | `model-xyz` | **C:** `model`, `gen_ai.request.model`. **X:** `model`. **O1:** assistant `info.modelID`, user `info.model.modelID`. **O2:** `data.model.id` on step start. | -| User input | User-role text when captured | `Fix the test` | **C:** `prompt`, `prompt_text`, span `user_prompt`. **X:** `prompt`. **O1:** text part `part.text`, joined to a user-role message via `messageID`. **O2:** `data.item.payload.text` in `session.inbox.enqueued` with `data.item.type=user`. A `user` role alone does not prove a human authored it. | -| Assistant response | Response text | `The test is fixed.` | **C/X:** `response`. **O1:** text part `part.text`, joined to an assistant-role message. **O2:** `data.text` on `session.text.ended`; `data.delta` on `session.text.delta`. | -| Input tokens | Input count | `1,200` | **C:** `input_tokens`; metric `claude_code.token.usage` with `type=input`. **X:** `input_token_count`, `codex.turn.token_usage.input_tokens`, `gen_ai.usage.input_tokens`; metric `codex.turn.token_usage` with `token_type=input`. Trace `codex.turn.token_usage.non_cached_input_tokens` is a separate count. **O1:** assistant `info.tokens.input` or step-finish `part.tokens.input`. **O2:** `data.tokens.input`. | -| Output tokens | Output count | `350` | **C:** `output_tokens`; metric `type=output`. **X:** `output_token_count`, `codex.turn.token_usage.output_tokens`, `gen_ai.usage.output_tokens`; metric `token_type=output`. **O1:** `info.tokens.output` or `part.tokens.output`. **O2:** `data.tokens.output`. | -| Cache read | Tokens read from cache | `800` | **C:** `cache_read_tokens`; metric `type=cacheRead`. **X:** `cached_token_count`, `codex.turn.token_usage.cached_input_tokens`, `gen_ai.usage.cache_read.input_tokens`; metric `token_type=cached_input`. **O1:** `info.tokens.cache.read` or `part.tokens.cache.read`. **O2:** `data.tokens.cache.read`. | -| Cache write | Tokens written to cache | `200` | **C:** `cache_creation_tokens`, `cache_creation_5m_tokens`, `cache_creation_1h_tokens`; metric `type=cacheCreation`. **X:** `cache_write_token_count`, `codex.turn.token_usage.cache_write_input_tokens`, `gen_ai.usage.cache_write.input_tokens`; metric `token_type=cache_write_input`. **O1:** `info.tokens.cache.write` or `part.tokens.cache.write`. **O2:** `data.tokens.cache.write`. | -| Tool call ID | Tool-specific call ID | `call-abc` | **C:** `tool_use_id`, `gen_ai.tool.call.id`. **X:** `call_id`; trace `runtime_tool_call_id` is a separate identifier. **O1:** tool `part.callID`. **O2:** `data.id` on tool events. | -| Tool name | Invoked tool | `read_file` | **C:** `tool_name`, `gen_ai.tool.name`. **X:** `tool_name`; metric label `tool`. **O1:** tool `part.tool`. **O2:** `data.name` on `session.tool.input.started`. | -| Tool input | Tool arguments | `{"path":"src/app.ts"}` | **C:** `tool_parameters`, `tool_input`. **X:** `arguments`. **O1:** tool `part.state.input`. **O2:** `data.input` on `session.tool.called`; serialized `data.text` on `session.tool.input.ended`. | -| Tool output | Returned content | `File contentsโ€ฆ` | **C:** `output` on span event `tool.output`. **X:** `output` on `codex.tool_result`. **O1:** completed tool `part.state.output`. **O2:** `data.content[].text` on `session.tool.success`. | -| Tool outcome | Success or failure | `success` | **C/X:** `success` on tool result. **O1:** `part.state.status` (`pending`, `running`, `completed`, `error`); error `part.state.error`. **O2:** event `session.tool.success` or `session.tool.failed`; failure `data.error.type`, `data.error.message`. | +| Common element | Example | Claude Code | Codex | OpenCode (v1 / v2) | +| --- | --- | --- | --- | --- | +| Time | `2026-11-04T10:20:00+08:00` | `event.timestamp`, `timeUnixNano`, `observedTimeUnixNano`; span `startTimeUnixNano` / `endTimeUnixNano` | Same OTLP times | **v1:** `info.time.created`, `part.time.start` / `end`; **v2:** `created`, `data.started` on step start | +| Event type | `tool_result` | `event.name`, log body, span or metric name | `event.name`; `eventName` may instead identify a source location | **v1/v2:** event `type` | +| Application version | `2.2.5` | resource `service.version` | `app.version`, resource `service.version` | **v1:** session `info.version`; **v2:** `data.version` on `session.created` or resource `service.version` | +| Session | `session-abc` | `session.id` | `conversation.id`; trace `thread.id` | **v1:** `info.sessionID`, `part.sessionID`; **v2:** `data.sessionID`, event `durable.aggregateID` | +| Provider | `openai` | `anthropic.provider`, `gen_ai.provider.name`, `gen_ai.system` | `provider_name`, trace `provider` | **v1:** assistant `info.providerID`, user `info.model.providerID`; **v2:** `data.model.providerID` on step start | +| Model | `model-xyz` | `model`, `gen_ai.request.model` | `model` | **v1:** assistant `info.modelID`, user `info.model.modelID`; **v2:** `data.model.id` on step start | +| User input | `Fix the test` | `prompt`, `prompt_text`, span `user_prompt` | `prompt` | **v1:** `part.text` joined by `messageID` to a user message; **v2:** `data.item.payload.text` on `session.inbox.enqueued` with `data.item.type=user` | +| Assistant response | `The test is fixed.` | `response` | `response` | **v1:** `part.text` joined by `messageID` to an assistant message; **v2:** `data.text` on `session.text.ended`, `data.delta` on `session.text.delta` | +| Input tokens | `1,200` | `input_tokens`; `claude_code.token.usage` metric, `type=input` | `input_token_count`, `codex.turn.token_usage.input_tokens`, `gen_ai.usage.input_tokens`; metric `token_type=input` | **v1:** `info.tokens.input` or `part.tokens.input`; **v2:** `data.tokens.input` | +| Output tokens | `350` | `output_tokens`; metric `type=output` | `output_token_count`, `codex.turn.token_usage.output_tokens`, `gen_ai.usage.output_tokens`; metric `token_type=output` | **v1:** `info.tokens.output` or `part.tokens.output`; **v2:** `data.tokens.output` | +| Cache read | `800` | `cache_read_tokens`; metric `type=cacheRead` | `cached_token_count`, `codex.turn.token_usage.cached_input_tokens`, `gen_ai.usage.cache_read.input_tokens`; metric `token_type=cached_input` | **v1:** `info.tokens.cache.read` or `part.tokens.cache.read`; **v2:** `data.tokens.cache.read` | +| Cache write | `200` | `cache_creation_tokens`, `cache_creation_5m_tokens`, `cache_creation_1h_tokens`; metric `type=cacheCreation` | `cache_write_token_count`, `codex.turn.token_usage.cache_write_input_tokens`, `gen_ai.usage.cache_write.input_tokens`; metric `token_type=cache_write_input` | **v1:** `info.tokens.cache.write` or `part.tokens.cache.write`; **v2:** `data.tokens.cache.write` | +| Tool call ID | `call-abc` | `tool_use_id`, `gen_ai.tool.call.id` | `call_id`; trace `runtime_tool_call_id` is distinct | **v1:** `part.callID`; **v2:** `data.id` | +| Tool name | `read_file` | `tool_name`, `gen_ai.tool.name` | `tool_name`; metric `tool` | **v1:** `part.tool`; **v2:** `data.name` on `session.tool.input.started` | +| Tool input | `{"path":"src/app.ts"}` | `tool_parameters`, `tool_input` | `arguments` | **v1:** `part.state.input`; **v2:** `data.input` on `session.tool.called`, serialized `data.text` on `session.tool.input.ended` | +| Tool output | `File contentsโ€ฆ` | `output` on span event `tool.output` | `output` on `codex.tool_result` | **v1:** completed `part.state.output`; **v2:** `data.content[].text` on `session.tool.success` | +| Tool outcome | `success` | `success` on tool result | `success` on tool result | **v1:** `part.state.status` and `part.state.error`; **v2:** `session.tool.success` or `session.tool.failed`, with `data.error.type` / `data.error.message` | + +Provider labels are not necessarily equivalent across tools. A user role does not prove a human authored the input. Some v1 fields require correlating a part with its message; v2 text and model fields depend on the event type. Codex trace `codex.turn.token_usage.non_cached_input_tokens` is a separate count. ## Interpretation and coverage @@ -36,9 +40,53 @@ The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 - OpenCode 2's rich session events are SSE, not the observed OTLP logs. The O1 schema above does not prove its fields are emitted over OTLP. The schema event's `durable.version` is not an application major version. - Prompts, responses, tool arguments and results can contain secrets. Capture and retain them only with the appropriate consent and local access controls. +## What sync sends + +The collector keeps the provider's raw OTLP JSON locally. The separate dev sync selects only consented sessions and sends: + +- **Claude Code:** `tool_result` logs and selected metrics: token usage, active time, edit decisions, lines changed, sessions, commits, pull requests and the provider's `cost.usage` metric when present. The dashboard does not calculate a cost. +- **Codex:** `conversation_starts`, `tool_result` and `response.completed` events, including available token counts. +- **Both:** available event time, session, model, application version, provider email/account/organization IDs, tool name, call ID, outcome and duration. Confirmed framework feature bindings additionally carry the declared branch and ticket; an invalidation can be sent when a binding becomes ambiguous. + +The sync excludes fields for prompt and response text, tool arguments and output, raw API bodies, metric exemplars and traces. It filters attribute keys, not secrets or free text inside allowed values. OpenCode is not forwarded by this sync. These are forwarding rules, not guarantees about what the raw collector receives. + +### Example: a Claude metric sent locally + +The structure below was produced by `selectTransfers` from one real local Claude capture. The timestamps, metric value, session, organization, email, account, model and query source have been replaced with fictional values; attribute presence varies by event. It is the filtered request body, not the raw capture or its authorization header. + +```json +{ + "resourceMetrics": [{ + "scopeMetrics": [{ + "metrics": [{ + "name": "claude_code.token.usage", + "sum": { + "aggregationTemporality": 1, + "isMonotonic": true, + "dataPoints": [{ + "startTimeUnixNano": "1767225600000000000", + "timeUnixNano": "1767225660000000000", + "asDouble": 123, + "attributes": [ + { "key": "session.id", "value": { "stringValue": "session-example" } }, + { "key": "organization.id", "value": { "stringValue": "organization-example" } }, + { "key": "user.email", "value": { "stringValue": "person@example.test" } }, + { "key": "user.account_id", "value": { "stringValue": "account-example" } }, + { "key": "model", "value": { "stringValue": "model-example" } }, + { "key": "query_source", "value": { "stringValue": "example" } }, + { "key": "type", "value": { "stringValue": "input" } } + ] + }] + } + }] + }] + }] +} +``` + ## Local Gouvernail sync -Local OTLP collection and sync require private POSIX files and are not supported on Windows. +OTLP initialization, verification, local collection and sync require private POSIX files and are not supported on Windows. Start the collector, then configure the tools in another shell: diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 05dca3769..e4e2ec6ff 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -2,7 +2,7 @@ # aidd-telemetry -Counts the tokens your Claude Code and Codex sessions consume. +Measures Claude Code and Codex usage locally, and can configure provider OpenTelemetry export. | | | | --- | --- | @@ -24,6 +24,34 @@ flowchart LR Tasks -->|"attribute"| Report ``` +The local ledger and provider OTLP captures are separate sources. The OTLP sync reuses the ledger's session attribution, task declarations and consent history; it does not replace the local report. Do not add their totals together. + +## Choose a mode + +| Mode | Opt-in and destination | Identity | What is kept or sent | +| --- | --- | --- | --- | +| Local usage | `aidd telemetry on` explicitly enables measurement for the current clone from that time. The ledger and report stay on this machine. | An optional name labels the local report; anonymous use is supported. | Per-call usage read from Claude Code and Codex session files: session, time, model, agent, token counts and local attribution context. The ledger does not store prompts or responses. | +| OTLP development | Explicitly configure each provider with `otel init --endpoint=dev` and start the local collector. A separate `otel sync` sends supported data only to loopback Gouvernail for sessions covered by the clone's consent. | No local report name is required. Native provider identifiers may still be sent when present; attribution in Gouvernail needs a matched active member. The connector token is not a person identity. | The collector stores raw provider events, which may include prompts, responses, commands and tool output. The sync excludes fields for those contents and traces, but does not inspect allowed attribute values for secrets. See [exact fields](OBSERVABILITY.md#what-sync-sends). | +| OTLP production | **Not available:** `--endpoint=prod` is refused. `https://gouvernail.eu` is an intended future destination, not an active exporter. | **Required before production is enabled:** an authenticated user identity. This is a product requirement, not yet implemented or enforced. | No production payload is sent by this feature today. Consent, authentication, retention and deletion behavior must be implemented before enabling it. | + +### Consent and scope + +- `on` is an explicit opt-in for one clone, not a team-wide setting. The local ledger and OTLP sync use its consent intervals; an earlier consent does not cover later work after `off`. +- `otel init` is a separate, explicit action to configure a provider's user-level exporter. The CLI does not show a separate consent dialog or record separate OTLP consent; the init skill asks for authorization before running it. When that provider and the collector are running, raw OTLP may be captured from other projects too: **the collector does not check clone consent**. The sync checks consent before forwarding. +- `aidd telemetry off` stops local measurement for the current clone; it does **not** disable provider exporters or stop the OTLP collector. A process started with `AIDD_TELEMETRY=0` refuses local measurement and sync, not raw collection; changing an environment variable does not change an already running process. +- `aidd telemetry forget` previews or deletes local usage measurement. It does **not** delete OTLP capture files, sync state, or data already received by Gouvernail. Production deletion is not available; the requested production policy is to disallow it. +- Raw captures are private local files, but have no automatic retention limit. They can contain sensitive content. The local collector and sync processes do not restart automatically. + +### Three different identifiers + +| Identifier | Purpose and storage | Sent to Gouvernail? | +| --- | --- | --- | +| Local report name | Optional label in `~/.config/aidd/telemetry/identity.json`; the current label is applied when reporting, not recorded as proof per call. | No. | +| Installation UUID | Stable ID in `~/.config/aidd/installation/id.json`; not a login or credential. | Not currently. | +| Provider identity | Native email, account or organization ID when emitted. Gouvernail must match an approved active member to attribute a session. | The allowlisted identifiers may be sent by the dev sync. | + +The default paths can change with the configured user directory. A connector bearer token authorizes the destination connection; it does not identify the person. + ## Start 1. `npm install -g @ai-driven-dev/cli` @@ -47,7 +75,7 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. ## Local usage measurement privacy -- Nothing leaves your machine. +- This local usage ledger and its reports stay on your machine. Provider OTLP export is a separate opt-in described above. - Consent is per clone and never committed. - Data lives in `~/.config/aidd/telemetry` and the clone's git config. - You are named only if you choose to be. @@ -56,4 +84,6 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. ## Provider OpenTelemetry -Configure an exporter with `aidd telemetry otel init --config= --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. +Configure an exporter with `aidd telemetry otel init --config= --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. OpenCode v1/v2 capture support does not yet imply OpenCode-to-Gouvernail sync. + +Start OpenCode v1 with `aidd telemetry otel run --config=opencode` to supply its endpoint and token; pass OpenCode arguments after `--`. For OpenCode v2, restart its service after initialization. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md b/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md index be7257788..c4a1aa5c1 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/03-identity.md @@ -8,13 +8,19 @@ What the person wants: to be named as an identifier they give, to see the curren ## Process -1. Say that an identity is optional, that it only labels the person axis of their own report, and that it is kept in one file on this machine. +1. Say that the local report name is optional, labels the person axis of their own report, and is kept on this machine. It is not the provider identity, the installation UUID, or a Gouvernail login. 2. Run `aidd telemetry identity` to read the current choice, and tell the person what it holds. 3. To set one, ask the person for the identifier they want and wait for it. Run `aidd telemetry identity ` with exactly that text. - The command refuses an identifier that is empty, over 128 characters, or spans lines: show its message and stop. 4. To remove it, confirm that the person wants to be named no more, then run `aidd telemetry identity --off`. -5. Run `aidd telemetry identity` again after a change and check that it reports the requested identifier or its removal. +5. Run `aidd telemetry identity` again after a change and check that it reports the requested identifier or its removal. Present the result in the table below; fill provider, endpoint and dates only when a relevant CLI command actually reports them. Do not infer an identity or expose credentials. ## Output -The identity set, shown or removed, as the command reported it. +| Item | Tools | Endpoint | Created | Stored | +| --- | --- | --- | --- | --- | +| Local report name | Claude Code and Codex local report; not tool-specific | None | Not recorded by this command | `~/.config/aidd/telemetry/identity.json` by default | +| Installation UUID | Installation, not a person | Not currently sent | Not reported by this command | `~/.config/aidd/installation/id.json` by default | +| Provider identity | Provider and values only if verified | Dev destination only if verified; prod unavailable | Not reported by this command | Local OTLP capture if emitted; provider storage unknown | + +Show the local name or `anonymous` as reported. Paths may differ with the configured user directory. A provider's native identity can be forwarded to local Gouvernail even when the local report is anonymous. Production user identity is a future requirement, not implemented today. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md index 82c21a2d0..3a5a9220a 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md @@ -8,11 +8,16 @@ The project directory. ## Process -1. Say what stopping does: new Claude Code or Codex work in this project is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. Wait for a yes. +1. Say what stopping does: new Claude Code or Codex work in this clone is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. It does not disable provider OTLP exporters or the raw collector. Wait for a yes. 2. Run `aidd telemetry off` and relay its message. -3. Say that deleting what was measured is a separate step, and that it needs its own confirmation. -4. Run `aidd telemetry status` and verify that measurement is off. +3. Run `aidd telemetry status` and verify that measurement is off. Report the result by scope in the table below. The CLI does not report the exact stop timestamp; do not invent one. Previously consented observations can still be processed or sent after `off`. +4. Say that deleting what was measured is a separate step, and that it needs its own confirmation. If the person wants raw OTLP to stop too, explain that `off` does not do that: the exporter and collector must be stopped separately, and there is no automatic restore command in this feature. ## Output -Measurement is off for the project, and the person knows the measured data is still there. +| Scope | Result after `off` | Since | Existing data | +| --- | --- | --- | --- | +| Local usage for this clone | Stopped if the command and status confirm it | Exact time not reported by the CLI | Retained | +| OTLP export and raw local collection | Not stopped by `off`; current process state is unknown unless separately checked | Unknown | Captures retained | +| OTLP sync to local Gouvernail | New events outside consent are not eligible; previously consented events may still be sent | Consent interval closes on `off`; exact time not reported | Already received data retained | +| Gouvernail production | Not available in this feature | Not applicable | Not managed by this command | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md index 89f700562..7c8184d13 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md @@ -1,6 +1,6 @@ # 05 - Forget -Delete all locally measured data after a preview and explicit confirmation. +Delete local usage measurement after a preview and explicit confirmation. This command does not delete provider OTLP captures, OTLP sync state, installation UUID or data already received by Gouvernail. ## Process @@ -10,9 +10,9 @@ Delete all locally measured data after a preview and explicit confirmation. 3. Ask whether to remove all of it. Wait for an explicit yes naming the removal. - A no, or no answer, ends the action with nothing removed. 4. Run `aidd telemetry forget --yes` only after that yes, and relay what it printed. - - Say that it also ends measurement: nothing is measured again until the person runs `aidd telemetry on`. + - Say that it also ends local usage measurement until the person runs `aidd telemetry on`; provider exporters and raw OTLP collection may still run. 5. Run `aidd telemetry forget` again and check that its preview lists nothing remaining. ## Output -Either the list of what was removed, or the list of what would be, with nothing removed. +Either the list of what was removed, or the list of what would be, with nothing removed. Separately name OTLP captures and Gouvernail data as not removed. Gouvernail production is not implemented here; the requested future policy disallows production deletion and is not enforced by this local command. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md index 28493156e..7be16eb10 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md @@ -9,13 +9,13 @@ The provider (`claude`, `codex` or `opencode`) and destination (`dev` or `prod`) ## Process 1. Run `aidd telemetry otel init --help` and report an unavailable command as requiring an update of `@ai-driven-dev/cli`. -2. Obtain both choices and explain that native telemetry goes to the selected collector and can include sensitive data. - - The development collector and its private token must already exist; initialization does not start it. - - Production is refused until its destination is configured. +2. Obtain both choices and explain the separate opt-in: native telemetry goes to the selected collector and can include prompts, responses, commands and tool output. The provider's user-level export can cover other projects; the raw collector does not check clone consent. The separate sync checks consent before forwarding. + - The CLI's development endpoint is `http://127.0.0.1:14318`. Start the collector with `aidd telemetry otel collect`; it creates the private token. Initialization does not start it. + - Production is refused. `https://gouvernail.eu` is an intended future destination, not a configured endpoint; requiring an authenticated person there is a future requirement, not enforced today. 3. Run `aidd telemetry otel init --config= --endpoint=` after an explicit request enabling that export. - A request naming both choices is sufficient authorization. - Let the CLI validate configuration and credentials without requesting a token in chat. -4. Report whether configuration changed and relay the CLI's restart or launch instructions. +4. Report whether configuration changed and relay the CLI's restart or launch instructions. OpenCode major version 1 needs the launch command printed by the CLI to supply the endpoint and token; major version 2 needs its service restarted. 5. Continue to [verify receipt](07-otel-verify.md) after the person runs a short session with the selected provider. ## Output From c343f82219794f21385c17b72b1b59318940ade2 Mon Sep 17 00:00:00 2001 From: alexsoyes Date: Thu, 5 Nov 2026 10:21:42 +0800 Subject: [PATCH 12/56] fix(telemetry): protect sync accuracy and add session titles --- cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 2 + .../otel/actions/series-fingerprint.ts | 54 ++ .../otel/actions/session-title.ts | 211 +++++++ .../otel/actions/sync-all-titles.ts | 126 +++++ .../infrastructure/otel/actions/sync-all.ts | 27 +- .../infrastructure/otel/actions/sync.ts | 156 +++++- cli/src/presentation/commands/telemetry.ts | 62 ++- cli/src/runtime/wiring/telemetry-otel.ts | 18 + .../session-title-sync.integration.test.ts | 280 ++++++++++ .../telemetry/otel/session-title.unit.test.ts | 156 ++++++ .../sync-otel-dimensions.integration.test.ts | 93 ++++ .../sync-otel-integrity.integration.test.ts | 514 ++++++++++++++++++ cli/tests/golden/snapshots/help/surface.json | 2 +- plugins/aidd-telemetry/OBSERVABILITY.md | 3 + plugins/aidd-telemetry/README.md | 3 + 16 files changed, 1690 insertions(+), 19 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts create mode 100644 cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/session-title.unit.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts diff --git a/cli/package.json b/cli/package.json index 20f63bb68..c0757b9f8 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 784, + "bundleBudgetKB": 799, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 85ef92ade..4593c112a 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -56,6 +56,8 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 784 KB: 783.7 KB, local OTLP collector and consent-aware resumable provider sync, // explicit metric sanitization, retry and durable feature invalidation (+38.8 KB over 744.9 KB); // 0.1% measured headroom. +// 799 KB: 798.8 KB with opt-in Codex titles, one-pass title synchronization and +// telemetry integrity guards (+15.1 KB over 783.7 KB); less than 0.1% measured headroom. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts new file mode 100644 index 000000000..030eb3660 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.ts @@ -0,0 +1,54 @@ +import { createHash } from "node:crypto"; + +export function seriesFingerprint(resource: unknown, scope: unknown, point: unknown): string { + const resourceEnvelope = record(resource); + const resourceInfo = record(resourceEnvelope.resource); + const scopeEnvelope = record(scope); + const scopeInfo = record(scopeEnvelope.scope); + const value = { + resource: { + schemaUrl: resourceEnvelope.schemaUrl ?? null, + attributes: canonicalAttributes(resourceInfo.attributes), + droppedAttributesCount: resourceInfo.droppedAttributesCount ?? null, + }, + scope: { + schemaUrl: scopeEnvelope.schemaUrl ?? null, + name: scopeInfo.name ?? null, + version: scopeInfo.version ?? null, + attributes: canonicalAttributes(scopeInfo.attributes), + droppedAttributesCount: scopeInfo.droppedAttributesCount ?? null, + }, + metric: canonical(scopeEnvelope.metric), + point: canonicalAttributes(point), + }; + return createHash("sha256").update(JSON.stringify(value)).digest("hex"); +} + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function canonicalAttributes(value: unknown): string[] { + if (!Array.isArray(value)) return []; + return value.map((item) => JSON.stringify(canonical(item))).sort(); +} + +function canonical(value: unknown): unknown { + if (Array.isArray(value)) return value.map(canonical); + if (!value || typeof value !== "object") return value ?? null; + return Object.fromEntries( + Object.entries(value) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + .map(([key, item]) => [key, key === "intValue" ? canonicalInteger(item) : canonical(item)]) + ); +} + +function canonicalInteger(value: unknown): string { + if (typeof value === "number" && !Number.isSafeInteger(value)) + throw new Error("OTLP integer attribute is not exactly representable."); + const input = String(value); + if (!/^-?\d{1,20}$/.test(input)) throw new Error("Invalid OTLP integer attribute."); + return BigInt(input).toString(); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts new file mode 100644 index 000000000..bcbd03dc5 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts @@ -0,0 +1,211 @@ +import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process"; +import { createInterface } from "node:readline"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import { CODEX_CONFIG_ID } from "../settings.js"; +import type { Identity, Transfer } from "./sync.js"; + +type Request = (method: string, params: object) => Promise; +type Title = { available: true; title: string | null } | { available: false }; +const MAX_TITLE_LENGTH = 4096; +const REQUEST_TIMEOUT_MS = 5000; +const REFRESH_MS = 60_000; + +export async function readSessionTitle(request: Request, session: string): Promise { + let result: unknown; + try { + result = await request("thread/read", { threadId: session, includeTurns: false }); + } catch { + return { available: false }; + } + const thread = asPlainObject(asPlainObject(result)?.thread); + if (!thread || thread.id !== session || thread.sessionId !== session) return { available: false }; + if (thread.name == null || (typeof thread.name === "string" && !thread.name.trim())) + return { available: true, title: null }; + if (typeof thread.name !== "string" || thread.name.length > MAX_TITLE_LENGTH) + return { available: false }; + return { available: true, title: thread.name }; +} + +export function sessionTitleTransfer( + session: string, + identity: Identity, + title: string | null, + at: Date +): Transfer | null { + if ( + identity.conflict || + (!identity.values.get("user.email") && !identity.values.get("user.account_id")) + ) + return null; + const values = new Map(identity.values); + values.set("conversation.id", session); + values.set("event.name", "aidd.session_title"); + values.set("event.timestamp", at.toISOString()); + if (title !== null) values.set("aidd.session_title", title); + return { + signal: "logs", + at: at.toISOString(), + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: (BigInt(at.getTime()) * 1_000_000n).toString(), + attributes: [...values].map(([key, value]) => ({ + key, + value: { stringValue: value }, + })), + }, + ], + }, + ], + }, + ], + }, + }; +} + +class AppServer { + private child: ChildProcessWithoutNullStreams; + private nextId = 0; + private pending = new Map<number, { resolve: (result: unknown) => void; reject: () => void }>(); + private ready: Promise<unknown>; + private failed = false; + + constructor(home: string) { + this.child = spawn(CODEX_CONFIG_ID, ["app-server", "--listen", "stdio://"], { + env: { ...process.env, HOME: home }, + stdio: ["pipe", "pipe", "pipe"], + }); + this.child.stderr.resume(); + this.child.stdin.on("error", () => this.fail()); + createInterface({ input: this.child.stdout }).on("line", (line) => this.receive(line)); + this.child.on("error", () => this.fail()); + this.child.on("exit", () => this.fail()); + this.ready = this.request("initialize", { + clientInfo: { name: "aidd_telemetry", title: "AIDD telemetry", version: "1" }, + }).then(() => { + this.child.stdin.write(`${JSON.stringify({ method: "initialized", params: {} })}\n`); + }); + } + + private receive(line: string): void { + let message: Record<string, unknown> | null; + try { + message = asPlainObject(JSON.parse(line)); + } catch { + return; + } + if (!message || typeof message.id !== "number") return; + const pending = this.pending.get(message.id); + if (!pending) return; + this.pending.delete(message.id); + if (message.error) pending.reject(); + else pending.resolve(message.result); + } + + private fail(): void { + this.failed = true; + for (const pending of this.pending.values()) pending.reject(); + this.pending.clear(); + } + + private request(method: string, params: object): Promise<unknown> { + return new Promise((resolve, reject) => { + const id = ++this.nextId; + const timeout = setTimeout(() => { + this.pending.delete(id); + this.failed = true; + this.child.kill(); + reject(new Error("Codex app-server did not respond.")); + }, REQUEST_TIMEOUT_MS); + this.pending.set(id, { + resolve: (result) => { + clearTimeout(timeout); + resolve(result); + }, + reject: () => { + clearTimeout(timeout); + reject(new Error("Codex app-server is unavailable.")); + }, + }); + this.child.stdin.write(`${JSON.stringify({ id, method, params })}\n`); + }); + } + + async read(session: string): Promise<Title> { + try { + await this.ready; + return await readSessionTitle((method, params) => this.request(method, params), session); + } catch { + return { available: false }; + } + } + + close(): void { + this.fail(); + this.child.kill(); + } + + get unavailable(): boolean { + return this.failed; + } +} + +type TitleReader = { read(session: string): Promise<Title>; close(): void; unavailable?: boolean }; + +export function createSessionTitlePublisher( + home: string, + dependencies: { + now?: () => number; + createReader?: (home: string) => TitleReader; + onUnavailable?: () => void; + } = {} +) { + let server: TitleReader | undefined; + const checked = new Map<string, number>(); + const previous = new Map<string, string | null>(); + let warned = false; + let retryAfter = 0; + return { + async publish( + session: string, + identity: Identity, + send: (transfer: Transfer) => Promise<boolean> + ): Promise<boolean> { + if ( + identity.conflict || + (!identity.values.get("user.email") && !identity.values.get("user.account_id")) + ) + return false; + const now = (dependencies.now ?? Date.now)(); + const last = checked.get(session); + if ((last !== undefined && now - last < REFRESH_MS) || now < retryAfter) return false; + if (server?.unavailable) { + server.close(); + server = undefined; + } + server ??= (dependencies.createReader ?? ((path) => new AppServer(path)))(home); + const result = await server.read(session); + const observedAt = (dependencies.now ?? Date.now)(); + checked.set(session, observedAt); + if (!result.available) { + if (server.unavailable) retryAfter = observedAt + REFRESH_MS; + if (!warned) dependencies.onUnavailable?.(); + warned = true; + return false; + } + if (previous.has(session) && previous.get(session) === result.title) return false; + const transfer = sessionTitleTransfer(session, identity, result.title, new Date(observedAt)); + if (!transfer) return false; + if (!(await send(transfer))) return false; + previous.set(session, result.title); + return true; + }, + close(): void { + server?.close(); + }, + }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts new file mode 100644 index 000000000..b54753737 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts @@ -0,0 +1,126 @@ +import { constants } from "node:fs"; +import { lstat, open } from "node:fs/promises"; +import { join } from "node:path"; +import type { SessionAdmission } from "../../../domain/report/session-feature.js"; +import { CODEX_CONFIG_ID } from "../settings.js"; +import { createSessionTitlePublisher } from "./session-title.js"; +import { + destination, + type Identity, + type SyncOptions, + selectTransfers, + send, + sessionIdsInCapture, +} from "./sync.js"; + +type Consent = "allow" | "deny" | "unknown"; +export type TitleOptions = Omit<SyncOptions, "session" | "follow" | "titlesOnly">; +const MAX_LINE = 32 * 1024 * 1024; + +function alreadyKnown(identity: Identity | undefined, observed: Identity): boolean { + if (identity?.conflict) return true; + if (observed.conflict) return false; + return [...observed.values].every( + ([key, value]) => identity?.values.get(key)?.toLowerCase() === value.toLowerCase() + ); +} + +/** Read a fixed snapshot once; unlike follow, this never writes a capture checkpoint. */ +export async function syncAllSessionTitles( + options: TitleOptions, + snapshot: () => Promise<ReadonlyMap<string, SessionAdmission>>, + permits: (admission: SessionAdmission, at: string, start?: string) => Promise<Consent> +) { + if (options.config !== CODEX_CONFIG_ID || !options.includeSessionTitles) + throw new Error("Title-only sync requires Codex and explicit title opt-in."); + const base: SyncOptions = { ...options, session: "auto", follow: false }; + destination(base, "logs"); + const held = await snapshot(); + const path = join(options.home, ".local/share/aidd-otel-observation", "codex.jsonl"); + const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + const identities = new Map<string, Identity>(); + const titles = createSessionTitlePublisher(options.home, { + onUnavailable: options.onTitleUnavailable, + }); + try { + const info = await file.stat(); + if (!info.isFile() || (info.mode & 0o077) !== 0) + throw new Error("Capture must be a private regular file."); + const consume = async (line: string) => { + if (!line.trim()) return; + for (const session of sessionIdsInCapture(line, CODEX_CONFIG_ID)) { + const admission = held.get(session); + if (!admission) continue; + const sessionOptions: SyncOptions = { ...base, session }; + const scratch: Identity = { values: new Map(), conflict: false }; + const transfers = selectTransfers(line, sessionOptions, scratch); + for (const transfer of transfers) { + if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) continue; + const source = JSON.stringify({ signal: transfer.signal, payload: transfer.payload }); + const observed: Identity = { values: new Map(), conflict: false }; + selectTransfers(source, sessionOptions, observed); + const identity = identities.get(session); + if (alreadyKnown(identity, observed)) continue; + if ((await permits(admission, transfer.at, transfer.start)) !== "allow") continue; + const accepted = identity ?? { values: new Map(), conflict: false }; + selectTransfers(source, sessionOptions, accepted); + identities.set(session, accepted); + } + } + }; + let offset = 0; + let pending = Buffer.alloc(0); + const buffer = Buffer.alloc(65536); + while (offset < info.size) { + const length = Math.min(buffer.length, info.size - offset); + const { bytesRead } = await file.read(buffer, 0, length, offset); + if (!bytesRead) throw new Error("Capture changed while titles were being read."); + offset += bytesRead; + pending = Buffer.concat([pending, buffer.subarray(0, bytesRead)]); + for (let newline = pending.indexOf(10); newline >= 0; newline = pending.indexOf(10)) { + if (newline > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); + await consume(pending.subarray(0, newline).toString("utf8")); + pending = pending.subarray(newline + 1); + } + if (pending.length > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); + } + const current = await lstat(path); + if ( + !current.isFile() || + current.ino !== info.ino || + current.dev !== info.dev || + current.size < info.size + ) + throw new Error("Capture was replaced or truncated; restart title sync."); + let sent = 0; + let identified = 0; + let currentConsenting = 0; + for (const [session, identity] of identities) { + const admission = held.get(session); + if ( + !admission || + identity.conflict || + (!identity.values.get("user.email") && !identity.values.get("user.account_id")) + ) + continue; + identified++; + if ((await permits(admission, new Date().toISOString())) !== "allow") continue; + currentConsenting++; + if ( + await titles.publish(session, identity, async (event) => { + if (!event.at || (await permits(admission, event.at)) !== "allow") { + currentConsenting--; + return false; + } + await send({ ...base, session }, event); + return true; + }) + ) + sent++; + } + return { sent, identified, currentConsenting, partialLine: pending.length > 0 }; + } finally { + titles.close(); + await file.close(); + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts index 7aaecb0ed..34e820f19 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -6,6 +6,8 @@ import { setTimeout as delay } from "node:timers/promises"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import type { SessionAdmission, SessionFeature } from "../../../domain/report/session-feature.js"; import { LedgerLock } from "../../ledger-lock.js"; +import { CODEX_CONFIG_ID } from "../settings.js"; +import { createSessionTitlePublisher } from "./session-title.js"; import { bindingTransfer, destination, @@ -20,7 +22,7 @@ import { type Transfer, } from "./sync.js"; -export type AllOptions = Omit<SyncOptions, "session" | "follow" | "stateDir"> & { +export type AllOptions = Omit<SyncOptions, "session" | "follow" | "stateDir" | "titlesOnly"> & { readonly stateDir: string; readonly refreshMs?: number; }; @@ -226,6 +228,10 @@ export async function syncAllOtel( staleAfterMs: Number.MAX_SAFE_INTEGER, }); const release = await lock.acquire(); + const titles = + options.config === CODEX_CONFIG_ID && options.includeSessionTitles + ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) + : undefined; let file: Awaited<ReturnType<typeof open>> | undefined; try { file = await open(capture, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); @@ -379,6 +385,22 @@ export async function syncAllOtel( } } }; + const publishTitles = async () => { + if (!titles) return; + for (const [id, state] of sessions) { + const admission = held.get(id); + if (!admission || state.identity.conflict) continue; + if ((await permits(admission, new Date().toISOString())) !== "allow") continue; + if ( + await titles.publish(id, state.identity, async (event) => { + if (!event.at || (await permits(admission, event.at)) !== "allow") return false; + await send(asOptions(options, id), event); + return true; + }) + ) + sent++; + } + }; const flushPending = async () => { const files = new Set(await readdir(pendingDirectory)); for (const [id, admission] of held) { @@ -417,6 +439,7 @@ export async function syncAllOtel( await loadBindings(); await flushPending(); await bindings(); + await publishTitles(); const buffer = Buffer.alloc(65536); for (;;) { const { bytesRead } = await file.read(buffer, 0, buffer.length, offset + pendingBytes.length); @@ -486,6 +509,7 @@ export async function syncAllOtel( dirty = false; await flushPending(); await bindings(); + await publishTitles(); await save(offset); } if (options.signal?.aborted) @@ -499,6 +523,7 @@ export async function syncAllOtel( } } } finally { + titles?.close(); await file?.close(); await release(); } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index 08df06d03..4319c3032 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -6,6 +6,8 @@ import { setTimeout as delay } from "node:timers/promises"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import type { SessionFeature } from "../../../domain/report/session-feature.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; +import { seriesFingerprint } from "./series-fingerprint.js"; +import { createSessionTitlePublisher } from "./session-title.js"; type FeatureState = SessionFeature; @@ -20,6 +22,9 @@ export interface SyncOptions { readonly stateDir?: string; readonly signal?: AbortSignal; readonly intervalMs?: number; + readonly includeSessionTitles?: boolean; + readonly onTitleUnavailable?: () => void; + readonly titlesOnly?: boolean; } type RecordObject = Record<string, unknown>; @@ -49,6 +54,7 @@ const SAFE_ATTRIBUTES = new Set([ "event.kind", "event.sequence", "event.timestamp", + "aidd.original_event_timestamp", "tool_use_id", "type", "model", @@ -56,6 +62,7 @@ const SAFE_ATTRIBUTES = new Set([ "tool", "decision", "query_source", + "aidd.series_fingerprint", "app.version", "originator", "success", @@ -103,6 +110,8 @@ function numericFields(source: RecordObject, keys: readonly string[]): RecordObj const result: RecordObject = {}; for (const key of keys) { const value = source[key]; + if (value === undefined) continue; + validateIntegerField(key, value); if ( (typeof value === "number" && Number.isFinite(value)) || (typeof value === "string" && /^-?\d+(?:\.\d+)?$/.test(value)) || @@ -118,6 +127,31 @@ function numericFields(source: RecordObject, keys: readonly string[]): RecordObj return result; } +function isExactInteger(value: unknown, minimum: bigint, maximum: bigint): boolean { + if (typeof value === "number" && !Number.isSafeInteger(value)) return false; + if (typeof value !== "number" && (typeof value !== "string" || !/^-?\d{1,20}$/.test(value))) + return false; + const integer = BigInt(value); + return integer >= minimum && integer <= maximum; +} + +function validateIntegerField(key: string, value: unknown): void { + const unsigned64 = 18446744073709551615n; + let valid = true; + if (["startTimeUnixNano", "timeUnixNano", "observedTimeUnixNano", "count"].includes(key)) + valid = isExactInteger(value, 0n, unsigned64); + else if (key === "asInt") + valid = isExactInteger(value, -9223372036854775808n, 9223372036854775807n); + else if (key === "flags") valid = isExactInteger(value, 0n, 4294967295n); + else if (key === "bucketCounts") + valid = Array.isArray(value) && value.every((item) => isExactInteger(item, 0n, unsigned64)); + if (!valid) throw new Error(`Invalid OTLP integer field: ${key}.`); +} + +function isExactNanoseconds(value: unknown): value is string | number { + return isExactInteger(value, 0n, 18446744073709551615n); +} + function captureIdentity(target: Identity, values: Map<string, string>) { for (const key of ["user.email", "user.account_id", "organization.id"]) { const value = values.get(key); @@ -130,11 +164,14 @@ function captureIdentity(target: Identity, values: Map<string, string>) { } function timeOf(entry: RecordObject) { + for (const key of ["timeUnixNano", "observedTimeUnixNano"]) { + if (entry[key] !== undefined) validateIntegerField(key, entry[key]); + } return { - ...(entry.timeUnixNano === undefined ? {} : { timeUnixNano: entry.timeUnixNano }), - ...(entry.observedTimeUnixNano === undefined - ? {} - : { observedTimeUnixNano: entry.observedTimeUnixNano }), + ...(isExactNanoseconds(entry.timeUnixNano) ? { timeUnixNano: entry.timeUnixNano } : {}), + ...(isExactNanoseconds(entry.observedTimeUnixNano) + ? { observedTimeUnixNano: entry.observedTimeUnixNano } + : {}), }; } @@ -216,8 +253,9 @@ export function sessionIdsInCapture(line: string, config: SyncOptions["config"]) } function timestamp(value: unknown): string | undefined { + if (value === undefined) return undefined; + if (!isExactNanoseconds(value)) throw new Error("Invalid OTLP integer field: timeUnixNano."); try { - if (typeof value !== "string" && typeof value !== "number") return undefined; const nanos = BigInt(value); return nanos > 0n ? new Date(Number(nanos / 1_000_000n)).toISOString() : undefined; } catch { @@ -226,7 +264,15 @@ function timestamp(value: unknown): string | undefined { } function eventTimestamp(value: string | undefined): string | undefined { - if (!value || !/(Z|[+-]\d\d:\d\d)$/i.test(value)) return undefined; + if (!value) return undefined; + const date = + /^(\d{4}-\d{2}-\d{2})T(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?(?:Z|[+-](?:[01]\d|2[0-3]):[0-5]\d)$/i.exec( + value + )?.[1]; + if (!date) return undefined; + const calendar = new Date(`${date}T00:00:00.000Z`); + if (Number.isNaN(calendar.getTime()) || !calendar.toISOString().startsWith(`${date}T`)) + return undefined; const time = Date.parse(value); return Number.isFinite(time) ? new Date(time).toISOString() : undefined; } @@ -267,10 +313,13 @@ export function selectTransfers( a.set("event.name", event); const at = options.config === CODEX_CONFIG_ID - ? (eventTimestamp(a.get("event.timestamp")) ?? - timestamp(record.timeUnixNano ?? record.observedTimeUnixNano)) - : timestamp(record.timeUnixNano ?? record.observedTimeUnixNano); + ? (eventTimestamp(a.get("event.timestamp")) ?? timestamp(record.timeUnixNano)) + : timestamp(record.timeUnixNano); if (options.config === CODEX_CONFIG_ID) { + const originalTimestamp = a.get("event.timestamp"); + a.delete("aidd.original_event_timestamp"); + if (originalTimestamp && eventTimestamp(originalTimestamp)) + a.set("aidd.original_event_timestamp", originalTimestamp); if (at) a.set("event.timestamp", at); else a.delete("event.timestamp"); } @@ -306,6 +355,24 @@ export function selectTransfers( const a = merge(resourceAttrs, attributes(point.attributes)); if (a.get(sessionKey) !== options.session) continue; captureIdentity(identity, a); + a.set( + "aidd.series_fingerprint", + seriesFingerprint( + resource, + { + scope: object(rawScope).scope, + schemaUrl: object(rawScope).schemaUrl, + metric: { + name: metric.name, + unit: metric.unit, + kind, + aggregationTemporality: aggregation.aggregationTemporality, + isMonotonic: aggregation.isMonotonic, + }, + }, + point.attributes + ) + ); const sanitized = { ...numericFields(point, [ "startTimeUnixNano", @@ -394,6 +461,44 @@ export function destination(options: SyncOptions, signal: "logs" | "metrics"): U class RetryableDeliveryError extends Error {} +async function assertFullDelivery(response: Response, signal: Transfer["signal"]): Promise<void> { + let result: unknown; + try { + result = await response.json(); + } catch { + throw new Error("Gouvernail returned an invalid OTLP response."); + } + const envelope = asPlainObject(result); + if (!envelope) throw new Error("Gouvernail returned an invalid OTLP response."); + if (Object.keys(envelope).some((key) => key !== "partialSuccess" && key !== "partial_success")) + throw new Error("Gouvernail returned an invalid OTLP response."); + if ("partialSuccess" in envelope && "partial_success" in envelope) + throw new Error("Gouvernail returned an invalid OTLP response."); + const raw = envelope.partialSuccess ?? envelope.partial_success; + if (raw === undefined || raw === null) return; + const partial = asPlainObject(raw); + if (!partial) throw new Error("Gouvernail returned an invalid OTLP response."); + const camel = signal === "logs" ? "rejectedLogRecords" : "rejectedDataPoints"; + const snake = signal === "logs" ? "rejected_log_records" : "rejected_data_points"; + if ( + Object.keys(partial).some( + (key) => ![camel, snake, "errorMessage", "error_message"].includes(key) + ) + ) + throw new Error("Gouvernail returned an invalid OTLP response."); + if (camel in partial && snake in partial) + throw new Error("Gouvernail returned an invalid OTLP response."); + const count = partial[camel] ?? partial[snake] ?? 0; + const number = + typeof count === "string" && /^\d+(?:[eE]\+?\d+)?$/.test(count) ? Number(count) : count; + if (typeof number !== "number" || !Number.isSafeInteger(number) || number < 0) + throw new Error("Gouvernail returned an invalid OTLP response."); + if (number > 0) + throw new Error( + `Gouvernail reported partial ${signal} delivery; resolve the rejection before retrying.` + ); +} + export async function send(options: SyncOptions, transfer: Transfer): Promise<void> { const body = JSON.stringify(transfer.payload); if (Buffer.byteLength(body) > MAX_PAYLOAD) @@ -419,6 +524,7 @@ export async function send(options: SyncOptions, transfer: Transfer): Promise<vo if (response.status >= 500) throw new RetryableDeliveryError(message); throw new Error(message); } + await assertFullDelivery(response, transfer.signal); return; } catch (error) { if (!options.follow || !(error instanceof RetryableDeliveryError) || options.signal?.aborted) @@ -559,6 +665,11 @@ export async function syncOtel( resolve: () => Promise<FeatureState>, permits: (transfer: Transfer) => Promise<boolean> ) { + if ( + options.titlesOnly && + (options.config !== CODEX_CONFIG_ID || options.follow || !options.includeSessionTitles) + ) + throw new Error("Title-only sync requires one Codex session and title opt-in."); destination(options, "logs"); const path = join(options.home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); @@ -576,6 +687,10 @@ export async function syncOtel( ? join(options.stateDir, "otel-sync", `${stateName}.json`) : null; let mayHaveLinked = false; + const titles = + options.config === CODEX_CONFIG_ID && options.includeSessionTitles + ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) + : undefined; try { const info = await handle.stat(); if (!info.isFile() || (info.mode & 0o077) !== 0) @@ -620,10 +735,11 @@ export async function syncOtel( } admitted.push(transfer); } - for (const batch of packTransfers(admitted)) { - await send(options, batch.transfer); - sent += batch.count; - } + if (!options.titlesOnly) + for (const batch of packTransfers(admitted)) { + await send(options, batch.transfer); + sent += batch.count; + } } offset += newline + 1; pending = pending.subarray(newline + 1); @@ -652,7 +768,7 @@ export async function syncOtel( payload: {}, at: new Date().toISOString(), }); - if (bindingAllowed && shouldInvalidate && binding !== "invalidated") { + if (!options.titlesOnly && bindingAllowed && shouldInvalidate && binding !== "invalidated") { const event = bindingTransfer(options, identity, { state: "ambiguous" }); if (event) { await send(options, event); @@ -667,6 +783,7 @@ export async function syncOtel( } } else if ( bindingAllowed && + !options.titlesOnly && !shouldInvalidate && feature.state === "confirmed" && (nextBinding !== binding || needsReplay) @@ -687,6 +804,16 @@ export async function syncOtel( needsReplay = false; } } + if ( + bindingAllowed && + titles && + (await titles.publish(options.session, identity, async (event) => { + if (!(await permits(event))) return false; + await send(options, event); + return true; + })) + ) + sent++; if (!options.follow || options.signal?.aborted) return { sent, skipped, offset, partialLine: pending.length > 0 }; try { @@ -698,6 +825,7 @@ export async function syncOtel( } } } finally { + titles?.close(); await handle.close(); } } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 2355f2b27..47636d2ce 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -36,6 +36,7 @@ import { launchOtel, otelOptions, syncCapturedSession, + syncCapturedSessionTitles, } from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, @@ -221,6 +222,12 @@ export function registerTelemetryCommand(program: Command): void { .requiredOption("--tenant <slug>", "Gouvernail tenant slug") .requiredOption("--endpoint <origin>", "Loopback HTTP origin") .option("--follow", "Follow new events for this session", false) + .option( + "--include-session-titles", + "Send Codex chat titles, which may contain prompt text", + false + ) + .option("--titles-only", "Send only Codex titles for consented sessions", false) .action( async (options: { config: (typeof otelOptions.syncConfigs)[number]; @@ -228,15 +235,58 @@ export function registerTelemetryCommand(program: Command): void { tenant: string; endpoint: string; follow: boolean; + includeSessionTitles: boolean; + titlesOnly: boolean; }) => { const { output, run } = commandOf(program); - if (!options.session && !options.follow) { + if (!options.session && !options.follow && !options.titlesOnly) { output.error("Pass --session for a bounded sync, or --follow to discover new sessions."); process.exitCode = 1; return; } + if (options.titlesOnly && options.follow) { + output.error("Title-only sync does not support --follow."); + process.exitCode = 1; + return; + } + if ( + (options.includeSessionTitles || options.titlesOnly) && + options.config !== otelOptions.titleConfig + ) { + output.error("Session title sync is currently available only for Codex."); + process.exitCode = 1; + return; + } await run(async (deps) => { const tool = otelOptions.usageTool(options.config); + const onTitleUnavailable = () => + output.warn("A Codex session title could not be read; telemetry sync continues."); + if (options.titlesOnly && !options.session) { + const result = await syncCapturedSessionTitles( + { + config: options.config, + tenant: options.tenant, + endpoint: options.endpoint, + token: process.env.AIDD_OTEL_TOKEN ?? "", + includeSessionTitles: true, + onTitleUnavailable, + }, + () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), + (admission, at, start) => + deps.telemetry.sessionFeatureUseCase.permits(admission, at, start) + ); + output.info( + `Sent ${result.sent} session title updates; no metrics or feature bindings were sent.` + ); + if (!result.sent && !result.identified) + output.warn( + "No unambiguous provider identity was found in consented capture events." + ); + else if (!result.sent && !result.currentConsenting) + output.warn("No identified session has current clone consent."); + if (result.partialLine) output.info("The last incomplete capture line was not read."); + return; + } if (!options.session) { output.info("Following new consented sessions; press Ctrl-C to stop."); const result = await followCapturedSessions( @@ -245,6 +295,8 @@ export function registerTelemetryCommand(program: Command): void { tenant: options.tenant, endpoint: options.endpoint, token: process.env.AIDD_OTEL_TOKEN ?? "", + includeSessionTitles: options.includeSessionTitles, + onTitleUnavailable, }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), (admission, at, start) => @@ -266,7 +318,13 @@ export function registerTelemetryCommand(program: Command): void { await resolve(); if (options.follow) output.info("Following the selected session; press Ctrl-C to stop."); const result = await syncCapturedSession( - { ...options, session, token: process.env.AIDD_OTEL_TOKEN ?? "" }, + { + ...options, + includeSessionTitles: options.includeSessionTitles || options.titlesOnly, + session, + token: process.env.AIDD_OTEL_TOKEN ?? "", + onTitleUnavailable, + }, resolve, async (transfer) => Boolean( diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index 520d8d2d7..5a09d3d1b 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -15,8 +15,13 @@ import { type AllOptions, syncAllOtel, } from "../../contexts/telemetry/infrastructure/otel/actions/sync-all.js"; +import { + syncAllSessionTitles, + type TitleOptions, +} from "../../contexts/telemetry/infrastructure/otel/actions/sync-all-titles.js"; import { verifyOtel } from "../../contexts/telemetry/infrastructure/otel/actions/verify.js"; import { + CODEX_CONFIG_ID, OPENCODE_CONFIG_ID, OTEL_CONFIGS, OTEL_ENDPOINTS, @@ -55,9 +60,22 @@ export function followCapturedSessions( return syncAllOtel({ ...options, home: homedir(), stateDir: telemetryDir() }, snapshot, permits); } +export function syncCapturedSessionTitles( + options: Omit<TitleOptions, "home" | "stateDir">, + snapshot: () => Promise<ReadonlyMap<string, SessionAdmission>>, + permits: ( + admission: SessionAdmission, + at: string, + start?: string + ) => Promise<"allow" | "deny" | "unknown"> +) { + return syncAllSessionTitles({ ...options, home: homedir() }, snapshot, permits); +} + export const otelOptions = { configs: OTEL_CONFIGS, syncConfigs: OTEL_SYNC_CONFIGS, + titleConfig: CODEX_CONFIG_ID, usageTool: usageToolForOtel, endpoints: OTEL_ENDPOINTS, launchConfigs: [OPENCODE_CONFIG_ID], diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts new file mode 100644 index 000000000..18244a938 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts @@ -0,0 +1,280 @@ +import { access, appendFile, chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { createServer } from "node:http"; +import { tmpdir } from "node:os"; +import { delimiter, join } from "node:path"; +import { expect, it } from "vitest"; +import { syncOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; +import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; +import { syncAllSessionTitles } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); + +it.skipIf(process.platform === "win32")( + "sends titles only when opted in and the session is consented", + async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-title-sync-")); + const oldPath = process.env.PATH; + const oldMarker = process.env.AIDD_TEST_TITLE_MARKER; + const marker = join(home, "called"); + const received: string[] = []; + let onReceived = (_body: string) => {}; + const server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + onReceived(body); + response.writeHead(200).end("{}"); + }); + }); + try { + const bin = join(home, "bin"); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(bin); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const fake = join(bin, "codex"); + await writeFile( + fake, + `#!${process.execPath}\n` + + String.raw` +const fs = require('node:fs'); +const readline = require('node:readline'); +fs.writeFileSync(process.env.AIDD_TEST_TITLE_MARKER, 'called'); +readline.createInterface({ input: process.stdin }).on('line', (line) => { + const message = JSON.parse(line); + if (!message.id) return; + const result = message.method === 'initialize' ? {} : { + thread: { id: message.params.threadId, sessionId: message.params.threadId, + name: 'Synthetic title', turns: [] } + }; + process.stdout.write(JSON.stringify({ id: message.id, result }) + '\n'); +}); +` + ); + await chmod(fake, 0o700); + process.env.PATH = `${bin}${delimiter}${oldPath ?? ""}`; + process.env.AIDD_TEST_TITLE_MARKER = marker; + const capture = { + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: "1793763900000000000", + attributes: [ + attr("conversation.id", "session-1"), + attr("user.email", "person@example.test"), + attr("event.name", "codex.sse_event"), + attr("event.kind", "response.completed"), + attr("input_token_count", "3"), + attr("output_token_count", "1"), + ], + }, + ], + }, + ], + }, + ], + }, + }; + await writeFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`, { + mode: 0o600, + }); + await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + session: "session-1", + tenant: "test", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test", + home, + follow: false, + }; + const resolve = async () => ({ state: "unbound" as const }); + + await syncOtel(options, resolve, async () => true); + await expect(access(marker)).rejects.toThrow(); + expect(received).toHaveLength(1); + + received.length = 0; + await syncOtel( + { ...options, includeSessionTitles: true }, + resolve, + async (transfer) => Object.keys(transfer.payload).length > 0 + ); + await expect(access(marker)).rejects.toThrow(); + expect(received).toHaveLength(1); + + received.length = 0; + await syncOtel( + { ...options, includeSessionTitles: true, titlesOnly: true }, + resolve, + async () => false + ); + await expect(access(marker)).rejects.toThrow(); + expect(received).toHaveLength(0); + + received.length = 0; + await syncOtel({ ...options, includeSessionTitles: true }, resolve, async () => true); + await access(marker); + expect(received).toHaveLength(2); + expect(received[1]).toContain("aidd.session_title"); + expect(received[1]).toContain("Synthetic title"); + expect(received[1]).not.toContain("aidd.feature"); + + received.length = 0; + await syncOtel( + { ...options, includeSessionTitles: true, titlesOnly: true }, + async () => ({ state: "confirmed", feature: "owner/repo#42", branch: "feat/demo" }), + async () => true + ); + expect(received).toHaveLength(1); + expect(received[0]).toContain("Synthetic title"); + expect(received[0]).not.toContain("input_token_count"); + expect(received[0]).not.toContain("aidd.feature"); + + received.length = 0; + await syncOtel( + { ...options, includeSessionTitles: true, titlesOnly: true }, + resolve, + async (transfer) => !JSON.stringify(transfer.payload).includes("aidd.session_title") + ); + expect(received).toHaveLength(0); + + received.length = 0; + const captureFile = join(captureDir, "codex.jsonl"); + await appendFile(captureFile, `${JSON.stringify(capture)}\n`.repeat(10)); + await appendFile( + captureFile, + `${JSON.stringify(capture).replaceAll("session-1", "session-2")}\n` + ); + await appendFile( + captureFile, + `${JSON.stringify(capture).replaceAll("session-1", "session-4")}\n` + ); + const original = capture.payload.resourceLogs[0].scopeLogs[0].logRecords[0]; + const attrs = original.attributes.map((a) => + a.key === "conversation.id" ? attr(a.key, "session-3") : a + ); + const denied = { ...original, timeUnixNano: "1793763899000000000", attributes: attrs }; + const admitted = { ...original, attributes: attrs.filter((a) => a.key !== "user.email") }; + const incomplete = { + signal: "logs", + payload: { resourceLogs: [{ scopeLogs: [{ logRecords: [denied, admitted] }] }] }, + }; + await appendFile(captureFile, `${JSON.stringify(incomplete)}\n`); + const admissions = new Map([ + [ + "session-1", + { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "allow" }, + ], + [ + "session-2", + { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "deny" }, + ], + [ + "session-3", + { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "mixed" }, + ], + [ + "session-4", + { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "allow-2" }, + ], + ]); + const deniedTime = new Date(1_793_763_899_000).toISOString(); + let allowedChecks = 0; + const result = await syncAllSessionTitles( + { ...options, includeSessionTitles: true, stateDir: join(home, "state") }, + async () => admissions, + async (admission, at) => { + if (admission.repositoryId === "allow") allowedChecks++; + return admission.repositoryId === "deny" || + (admission.repositoryId === "mixed" && at === deniedTime) + ? "deny" + : "allow"; + } + ); + expect(result.sent).toBe(2); + expect(result.identified).toBe(2); + expect(result.currentConsenting).toBe(2); + expect(allowedChecks).toBe(3); + expect(received).toHaveLength(2); + expect(received.some((body) => body.includes("session-1"))).toBe(true); + expect(received.some((body) => body.includes("session-4"))).toBe(true); + expect(received.join()).not.toContain("session-2"); + expect(received.join()).not.toContain("session-3"); + expect(received.join()).not.toContain("input_token_count"); + expect(received.join()).not.toContain("aidd.feature"); + await expect(access(join(home, "state"))).rejects.toThrow(); + + received.length = 0; + let checks = 0; + const admittedSession = admissions.get("session-1"); + if (!admittedSession) throw new Error("Missing test admission."); + const withdrawn = await syncAllSessionTitles( + { ...options, includeSessionTitles: true }, + async () => new Map([["session-1", admittedSession]]), + async () => (++checks < 3 ? "allow" : "deny") + ); + expect(withdrawn.sent).toBe(0); + expect(withdrawn.currentConsenting).toBe(0); + expect(received).toHaveLength(0); + + const noSourceConsent = await syncAllSessionTitles( + { ...options, includeSessionTitles: true }, + async () => new Map([["session-1", admittedSession]]), + async () => "deny" + ); + expect(noSourceConsent).toMatchObject({ sent: 0, identified: 0, currentConsenting: 0 }); + + const captureTime = new Date(1_793_763_900_000).toISOString(); + const noCurrentConsent = await syncAllSessionTitles( + { ...options, includeSessionTitles: true }, + async () => new Map([["session-1", admittedSession]]), + async (_admission, at) => (at === captureTime ? "allow" : "deny") + ); + expect(noCurrentConsent).toMatchObject({ sent: 0, identified: 1, currentConsenting: 0 }); + + received.length = 0; + const abort = new AbortController(); + const timeout = setTimeout(() => abort.abort(), 3000); + onReceived = (body) => { + if (body.includes("aidd.session_title")) abort.abort(); + }; + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const running = syncAllOtel( + { + ...options, + stateDir: join(home, "state"), + includeSessionTitles: true, + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["session-1", admission]]), + async () => "allow" + ); + await new Promise((done) => setTimeout(done, 100)); + await appendFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`); + await running; + clearTimeout(timeout); + expect(received.some((body) => body.includes("Synthetic title"))).toBe(true); + } finally { + process.env.PATH = oldPath; + if (oldMarker === undefined) delete process.env.AIDD_TEST_TITLE_MARKER; + else process.env.AIDD_TEST_TITLE_MARKER = oldMarker; + await new Promise<void>((resolve) => server.close(() => resolve())); + await rm(home, { recursive: true, force: true }); + } + } +); diff --git a/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts new file mode 100644 index 000000000..e8266be29 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts @@ -0,0 +1,156 @@ +import { describe, expect, it } from "vitest"; +import { + createSessionTitlePublisher, + readSessionTitle, + sessionTitleTransfer, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/session-title.js"; + +describe("Codex session titles", () => { + it("dates a title when it is observed, not before a delayed read", async () => { + let now = 100_000; + const reader = { + read: async () => { + now = 103_000; + return { available: true as const, title: "New title" }; + }, + close: () => {}, + }; + const publisher = createSessionTitlePublisher("/unused", { + now: () => now, + createReader: () => reader, + }); + let time: string | undefined; + await publisher.publish( + "s1", + { values: new Map([["user.email", "person@example.test"]]), conflict: false }, + async (event) => { + time = event.at; + return true; + } + ); + expect(time).toBe(new Date(103_000).toISOString()); + publisher.close(); + }); + + it("backs off globally after an app-server failure instead of blocking every session", async () => { + let now = 100_000; + let starts = 0; + let reads = 0; + let warnings = 0; + const publisher = createSessionTitlePublisher("/unused", { + now: () => now, + createReader: () => { + starts++; + return { + unavailable: true, + read: async () => { + reads++; + now += 5_000; + return { available: false as const }; + }, + close: () => {}, + }; + }, + onUnavailable: () => { + warnings++; + }, + }); + const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; + const send = async () => true; + expect(await publisher.publish("s1", identity, send)).toBe(false); + expect(await publisher.publish("s2", identity, send)).toBe(false); + expect(await publisher.publish("s3", identity, send)).toBe(false); + expect({ starts, reads, warnings }).toEqual({ starts: 1, reads: 1, warnings: 1 }); + now += 60_000; + expect(await publisher.publish("s3", identity, send)).toBe(false); + expect({ starts, reads, warnings }).toEqual({ starts: 2, reads: 2, warnings: 1 }); + publisher.close(); + }); + + it("does not mark a title delivered when consent is withdrawn during the read", async () => { + let now = 100_000; + const publisher = createSessionTitlePublisher("/unused", { + now: () => now, + createReader: () => ({ + read: async () => ({ available: true, title: "Title" }), + close: () => {}, + }), + }); + const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; + expect(await publisher.publish("s1", identity, async () => false)).toBe(false); + now += 60_000; + expect(await publisher.publish("s1", identity, async () => true)).toBe(true); + publisher.close(); + }); + + it("reads metadata only and requires exact thread and session IDs", async () => { + const requests: Array<{ method: string; params: object }> = []; + const read = async (method: string, params: object) => { + requests.push({ method, params }); + return { thread: { id: "s1", sessionId: "s1", name: "A title", turns: [] } }; + }; + expect(await readSessionTitle(read, "s1")).toEqual({ available: true, title: "A title" }); + expect(requests).toEqual([ + { method: "thread/read", params: { threadId: "s1", includeTurns: false } }, + ]); + expect( + await readSessionTitle( + async () => ({ thread: { id: "s1", sessionId: "root", name: "x" } }), + "s1" + ) + ).toEqual({ available: false }); + }); + + it("sends an absent title explicitly without inventing a feature", () => { + const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; + const event = sessionTitleTransfer("s1", identity, null, new Date("2026-11-05T00:00:00Z")); + expect(event?.signal).toBe("logs"); + const attrs = event?.payload.resourceLogs; + expect(JSON.stringify(attrs)).toContain("aidd.session_title"); + expect(JSON.stringify(attrs)).not.toContain("aidd.feature"); + expect(sessionTitleTransfer("s1", { ...identity, conflict: true }, "x", new Date())).toBeNull(); + }); + + it("sends title changes and removal once, and keeps the last known value on read failure", async () => { + let now = 100_000; + let warnings = 0; + let current: { available: true; title: string | null } | { available: false } = { + available: true, + title: "First", + }; + const sent: string[] = []; + const reader = { read: async () => current, close: () => {} }; + const publisher = createSessionTitlePublisher("/unused", { + now: () => now, + createReader: () => reader, + onUnavailable: () => { + warnings++; + }, + }); + const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; + const send = async (event: { payload: object }) => { + sent.push(JSON.stringify(event.payload)); + return true; + }; + expect(await publisher.publish("s1", identity, send)).toBe(true); + expect(await publisher.publish("s1", identity, send)).toBe(false); + now += 60_000; + current = { available: false }; + expect(await publisher.publish("s1", identity, send)).toBe(false); + expect(warnings).toBe(1); + now += 60_000; + expect(await publisher.publish("s1", identity, send)).toBe(false); + expect(warnings).toBe(1); + now += 60_000; + current = { available: true, title: "Second" }; + expect(await publisher.publish("s1", identity, send)).toBe(true); + now += 60_000; + current = { available: true, title: null }; + expect(await publisher.publish("s1", identity, send)).toBe(true); + expect(sent).toHaveLength(3); + expect(sent[0]).toContain("First"); + expect(sent[1]).toContain("Second"); + expect(sent[2]).not.toContain('"key":"aidd.session_title"'); + publisher.close(); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts new file mode 100644 index 000000000..dc480221a --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts @@ -0,0 +1,93 @@ +import { expect, test } from "vitest"; +import { seriesFingerprint } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/series-fingerprint.js"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "claude" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:3011", + token: "test", + home: "/tmp", + follow: false, +}; + +function capture(dimension: string, reverse = false, unit = "token"): string { + const dimensions = [ + attr("session.id", "session"), + attr("type", "input"), + attr("agent.name", dimension), + ]; + if (reverse) dimensions.reverse(); + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + unit, + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: "1784505600000000000", + timeUnixNano: "1784548800000000000", + asInt: "5", + attributes: dimensions, + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +function fingerprint(source: string): string { + const transfers = selectTransfers(source, options, { values: new Map(), conflict: false }); + const body = JSON.stringify(transfers[0]?.payload); + expect(body).not.toContain("private-agent"); + return ( + body.match(/"key":"aidd.series_fingerprint","value":\{"stringValue":"([0-9a-f]{64})"\}/)?.[1] ?? + "" + ); +} + +test("keeps distinct Claude metric dimensions distinguishable without exposing their values", () => { + const first = fingerprint(capture("private-agent-one")); + const second = fingerprint(capture("private-agent-two")); + expect(first).toHaveLength(64); + expect(second).toHaveLength(64); + expect(first).not.toBe(second); + expect(fingerprint(capture("private-agent-one", true))).toBe(first); + expect(fingerprint(capture("private-agent-one", false, "byte"))).not.toBe(first); +}); + +test("normalizes scope attribute order and distinguishes schemas", () => { + const resource = { resource: { attributes: [attr("service", "app")] }, schemaUrl: "schema-a" }; + const attributes = [attr("one", "1"), attr("two", "2")]; + const scope = { scope: { name: "scope", attributes }, schemaUrl: "scope-a" }; + const reordered = { ...scope, scope: { ...scope.scope, attributes: [...attributes].reverse() } }; + expect(seriesFingerprint(resource, scope, [])).toBe(seriesFingerprint(resource, reordered, [])); + expect(seriesFingerprint(resource, scope, [])).not.toBe( + seriesFingerprint({ ...resource, schemaUrl: "schema-b" }, scope, []) + ); + expect(seriesFingerprint(resource, { ...scope, metric: { unit: "token" } }, [])).not.toBe( + seriesFingerprint(resource, { ...scope, metric: { unit: "byte" } }, []) + ); +}); + +test("normalizes equivalent integer attribute encodings", () => { + const numeric = [{ key: "attempt", value: { intValue: 5 } }]; + const decimal = [{ key: "attempt", value: { intValue: "5" } }]; + expect(seriesFingerprint({}, {}, numeric)).toBe(seriesFingerprint({}, {}, decimal)); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts new file mode 100644 index 000000000..f512bc225 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -0,0 +1,514 @@ +import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { + selectTransfers, + send, + syncOtel, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; +import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; + +const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "codex" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:1", + token: "test", + home: "/tmp", + follow: false, +}; +let server: Server | undefined; +const dirs: string[] = []; +afterEach(async () => { + if (server) await new Promise<void>((resolve) => server?.close(() => resolve())); + server = undefined; + await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +function codexLog(time: Record<string, string | number>, timestamp?: string): string { + return JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + ...time, + attributes: [ + attribute("conversation.id", "session"), + attribute("event.name", "codex.sse_event"), + attribute("event.kind", "response.completed"), + ...(timestamp ? [attribute("event.timestamp", timestamp)] : []), + ], + }, + ], + }, + ], + }, + ], + }, + }); +} + +function claudeMetric(startTime: string | number, endTime: string | number): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: startTime, + timeUnixNano: endTime, + asInt: "42", + attributes: [attribute("session.id", "session")], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +function claudePoint( + extra: Record<string, string | number | Array<string | number> | null | undefined> +): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: "1785044858922210000", + timeUnixNano: "1785044858922210000", + asInt: "42", + ...extra, + attributes: [attribute("session.id", "session")], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +test.each([ + { asInt: 1785044858922210000 }, + { asInt: 1.5 }, + { asInt: "1.5" }, + { asInt: "private-invalid-value" }, + { asInt: null }, + { asInt: "9223372036854775808" }, + { count: 1785044858922210000 }, + { count: -1 }, + { count: "1.5" }, + { count: "18446744073709551616" }, + { bucketCounts: [1, 1785044858922210000] }, + { bucketCounts: [1, "bad"] }, + { bucketCounts: 1 }, + { flags: 1.5 }, +])("rejects invalid or inexact integer metric fields before exporting", (extra) => { + expect(() => + selectTransfers( + claudePoint(extra), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toThrow(/invalid OTLP integer/i); +}); + +test("preserves exact integer strings for signed and unsigned metric values", () => { + const extra = { + asInt: "9223372036854775807", + count: "18446744073709551615", + bucketCounts: ["9007199254740993", "0"], + }; + const [transfer] = selectTransfers( + claudePoint(extra), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ); + const payload = JSON.stringify(transfer.payload); + for (const value of [extra.asInt, extra.count, ...extra.bucketCounts]) + expect(payload).toContain(`"${value}"`); +}); + +test("an inexact metric stops all-session sync before delivery or checkpoint advance", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-inexact-metric-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + const stateDir = join(home, "state"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const file = join(captureDir, "claude.jsonl"); + await writeFile(file, "", { mode: 0o600 }); + let calls = 0; + const abort = new AbortController(); + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const running = syncAllOtel( + { + config: "claude", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test", + home, + stateDir, + intervalMs: 10, + signal: abort.signal, + }, + async () => + new Map([ + ["session", { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }], + ]), + async () => "allow" + ); + const directory = join(stateDir, "otel-sync"); + let checkpoint: string | undefined; + for (let attempt = 0; attempt < 100 && !checkpoint; attempt++) { + const entries = await readdir(directory).catch(() => []); + checkpoint = entries.find((entry) => entry.endsWith(".checkpoint.json")); + if (!checkpoint) await new Promise((resolve) => setTimeout(resolve, 10)); + } + if (!checkpoint) throw new Error("No initial checkpoint"); + await appendFile(file, `${claudePoint({ asInt: 1785044858922210000 })}\n`); + await expect(running).rejects.toThrow(/invalid OTLP integer/i); + const saved = JSON.parse(await readFile(join(directory, checkpoint), "utf8")); + expect(saved.offset).toBe(0); + expect(calls).toBe(0); +}); + +function metricWithMetadata(unit: string, temporality: number, monotonic: boolean): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + unit, + sum: { + aggregationTemporality: temporality, + isMonotonic: monotonic, + dataPoints: [ + { + startTimeUnixNano: "1785044858922210000", + timeUnixNano: "1785044858922210000", + asInt: "42", + attributes: [attribute("session.id", "session")], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +test("distinguishes metric unit, aggregation and monotonicity when computing series identity", () => { + const fingerprint = (unit: string, temporality: number, monotonic: boolean) => { + const [transfer] = selectTransfers( + metricWithMetadata(unit, temporality, monotonic), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ); + return JSON.stringify(transfer.payload).match( + /"aidd.series_fingerprint","value":\{"stringValue":"([0-9a-f]+)"/ + )?.[1]; + }; + const base = fingerprint("token", 1, true); + expect(base).toHaveLength(64); + expect(fingerprint("byte", 1, true)).not.toBe(base); + expect(fingerprint("token", 2, true)).not.toBe(base); + expect(fingerprint("token", 1, false)).not.toBe(base); +}); + +test("does not convert an unsafe numeric OTLP timestamp into a fabricated Codex event time", () => { + const unsafe = 1785044858922210000; + const line = codexLog({ timeUnixNano: unsafe }); + expect(() => selectTransfers(line, options, { values: new Map(), conflict: false })).toThrow( + /invalid OTLP integer/i + ); +}); + +test("preserves exact decimal-string nanoseconds, including for Claude metric intervals", () => { + const time = "1785044858922210000"; + const [log] = selectTransfers(codexLog({ timeUnixNano: time }), options, { + values: new Map(), + conflict: false, + }); + expect(log.at).toBe("2026-07-26T05:47:38.922Z"); + expect(JSON.stringify(log.payload)).toContain(time); + const [metric] = selectTransfers( + claudeMetric(time, time), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ); + expect(metric.start).toBe("2026-07-26T05:47:38.922Z"); + expect(metric.at).toBe("2026-07-26T05:47:38.922Z"); + expect(JSON.stringify(metric.payload)).toContain(time); +}); + +test.each([ + [1785044858922210000, "1785044858922210000"], + ["1785044858922210000", 1785044858922210000], +] as const)("rejects unsafe numeric Claude metric bounds", (start, end) => { + expect(() => + selectTransfers( + claudeMetric(start, end), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toThrow(/invalid OTLP integer/i); +}); + +test("does not forward an unsafe timestamp beside a valid Codex event timestamp", () => { + const line = codexLog({ timeUnixNano: 1785044858922210000 }, "2026-07-26T05:47:38.922Z"); + expect(() => selectTransfers(line, options, { values: new Map(), conflict: false })).toThrow( + /invalid OTLP integer/i + ); +}); + +test("uses a genuine OTLP event time when a Codex calendar date is invalid", () => { + const line = codexLog({ timeUnixNano: "1785044858922210000" }, "2026-02-30T12:00:00Z"); + const transfers = selectTransfers(line, options, { values: new Map(), conflict: false }); + expect(transfers[0].at).toBe("2026-07-26T05:47:38.922Z"); + expect(JSON.stringify(transfers[0].payload)).not.toContain("aidd.original_event_timestamp"); + expect(JSON.stringify(transfers[0].payload)).not.toContain("2026-02-30"); +}); + +test.each([ + "2026-02-30T12:00:00Z", + "2026-01-01T25:00:00Z", + "2026-01-01T12:00:00+24:00", + "2026-01-01T12:00:00", +])("does not normalize an invalid provider event time: %s", (time) => { + const transfers = selectTransfers(codexLog({}, time), options, { + values: new Map(), + conflict: false, + }); + expect(transfers[0].at).toBeUndefined(); + expect(JSON.stringify(transfers[0].payload)).not.toContain("aidd.original_event_timestamp"); +}); + +test("keeps a valid offset and leap date as the original event time", () => { + const original = "2028-02-29T12:30:00.123456+02:30"; + const transfers = selectTransfers(codexLog({}, original), options, { + values: new Map(), + conflict: false, + }); + expect(transfers[0].at).toBe("2028-02-29T10:00:00.123Z"); + expect(JSON.stringify(transfers[0].payload)).toContain(original); +}); + +test("does not export an observation-only timestamp as event activity", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-missing-event-time-")); + dirs.push(home); + const capture = join(home, ".local/share/aidd-otel-observation"); + await mkdir(capture, { recursive: true, mode: 0o700 }); + await writeFile( + join(capture, "codex.jsonl"), + `${codexLog({ observedTimeUnixNano: "1785044858922210000" })}\n`, + { mode: 0o600 } + ); + let calls = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200).end("{}"); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const result = await syncOtel( + { ...options, home, endpoint: `http://127.0.0.1:${address.port}` }, + async () => ({ state: "unbound" }), + async () => true + ); + expect(result.sent).toBe(0); + expect(result.skipped).toBe(1); + expect(calls).toBe(0); +}); + +test("does not use Claude observation time as event activity", () => { + const line = JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + observedTimeUnixNano: "1785044858922210000", + attributes: [ + attribute("session.id", "session"), + attribute("event.name", "tool_result"), + ], + }, + ], + }, + ], + }, + ], + }, + }); + const transfers = selectTransfers( + line, + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ); + expect(transfers).toHaveLength(1); + expect(transfers[0].at).toBeUndefined(); +}); + +test("preserves distinct submillisecond Codex event timestamps in the transmitted attributes", () => { + const timestamps = ["2026-11-04T01:00:00.123456100Z", "2026-11-04T01:00:00.123456200Z"]; + const line = JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: timestamps.map((time) => ({ + attributes: [ + attribute("conversation.id", "session"), + attribute("event.name", "codex.sse_event"), + attribute("event.kind", "response.completed"), + attribute("event.timestamp", time), + ], + })), + }, + ], + }, + ], + }, + }); + const transfers = selectTransfers(line, options, { values: new Map(), conflict: false }); + expect(transfers).toHaveLength(2); + for (const [index, transfer] of transfers.entries()) { + expect(JSON.stringify(transfer.payload)).toContain(timestamps[index]); + expect(JSON.stringify(transfer.payload)).toContain( + '"event.timestamp","value":{"stringValue":"2026-11-04T01:00:00.123Z"}' + ); + expect(JSON.stringify(transfer.payload)).toContain('"aidd.original_event_timestamp"'); + expect(transfer.at).toBe("2026-11-04T01:00:00.123Z"); + } +}); + +test.each([ + ["logs" as const, { partialSuccess: { rejectedLogRecords: "1" } }], + ["metrics" as const, { partial_success: { rejected_data_points: 2 } }], +])("does not retry or report success for partial %s delivery", async (signal, body) => { + let calls = 0; + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200, { "content-type": "application/json" }).end(JSON.stringify(body)); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + await expect( + send( + { ...options, endpoint: `http://127.0.0.1:${address.port}`, follow: true }, + { signal, payload: {} } + ) + ).rejects.toThrow(/partial/i); + expect(calls).toBe(1); +}); + +test.each([ + "not-json", + '{"partialSuccess":{"rejectedLogRecords":"-1"}}', + '{"partialSuccess":{"rejectedLogRecords":""}}', + '{"partialSuccess":{"rejectedLogRecords":"0x0"}}', + '{"partialSuccess":{"rejectedDataPoints":"1"}}', + '{"partialSuccess":{"rejectedLogRecord":"1"}}', +])("refuses an invalid successful OTLP response: %s", async (body) => { + server = createServer((request, response) => { + request.resume(); + request.on("end", () => response.writeHead(200).end(body)); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + await expect( + send( + { ...options, endpoint: `http://127.0.0.1:${address.port}` }, + { signal: "logs", payload: {} } + ) + ).rejects.toThrow(/response/i); +}); + +test("accepts a warning with zero rejected records", async () => { + server = createServer((request, response) => { + request.resume(); + request.on("end", () => + response.writeHead(200).end( + JSON.stringify({ + partialSuccess: { rejectedLogRecords: "0", errorMessage: "warning" }, + }) + ) + ); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + await expect( + send( + { ...options, endpoint: `http://127.0.0.1:${address.port}` }, + { signal: "logs", payload: {} } + ) + ).resolves.toBeUndefined(); +}); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 4cf7e601f..ab09e3c38 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -187,7 +187,7 @@ { "invocation": "aidd telemetry otel sync", "exitCode": 0, - "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to discover\n sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default: false)\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to\n discover sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default: false)\n --include-session-titles Send Codex chat titles, which may contain prompt\n text (default: false)\n --titles-only Send only Codex titles for consented sessions\n (default: false)\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 384497afb..f9e0d9104 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -47,6 +47,8 @@ The collector keeps the provider's raw OTLP JSON locally. The separate dev sync - **Claude Code:** `tool_result` logs and selected metrics: token usage, active time, edit decisions, lines changed, sessions, commits, pull requests and the provider's `cost.usage` metric when present. The dashboard does not calculate a cost. - **Codex:** `conversation_starts`, `tool_result` and `response.completed` events, including available token counts. - **Both:** available event time, session, model, application version, provider email/account/organization IDs, tool name, call ID, outcome and duration. Confirmed framework feature bindings additionally carry the declared branch and ticket; an invalidation can be sent when a binding becomes ambiguous. +- **Codex titles:** only with `--include-session-titles`, the free-text chat title is sent as session metadata. It may contain part of a prompt. +- **Claude series:** a SHA-256 fingerprint of the source resource attributes, instrumentation scope and all point attributes helps detect different metric series that share a legacy key. The attributes used to compute it are not sent as additional plaintext, but the fingerprint can reveal equality or allow guesses about low-entropy values. Existing points lack it and cannot be certified retroactively. [OTLP metric data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) (read 2026-11-05). The sync excludes fields for prompt and response text, tool arguments and output, raw API bodies, metric exemplars and traces. It filters attribute keys, not secrets or free text inside allowed values. OpenCode is not forwarded by this sync. These are forwarding rules, not guarantees about what the raw collector receives. @@ -118,5 +120,6 @@ Read on 2026-11-04: - [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) - [Codex observability and telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#observability-and-telemetry) +- [Codex app-server thread metadata](https://learn.chatgpt.com/docs/app-server#read-a-stored-thread-without-resuming) and [OTLP delivery responses](https://opentelemetry.io/docs/specs/otlp/#partial-success-1) (read 2026-11-05) - [OpenCode 1.18.35 session schema](https://github.com/anomalyco/opencode/blob/v1.18.35/packages/schema/src/v1/session.ts) and [LLM instrumentation](https://github.com/anomalyco/opencode/blob/v1.18.35/packages/opencode/src/session/llm.ts) - [OpenCode 2.2.5 session events](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts) and [token normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts) diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index e4e2ec6ff..160523478 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -38,6 +38,9 @@ The local ledger and provider OTLP captures are separate sources. The OTLP sync - `on` is an explicit opt-in for one clone, not a team-wide setting. The local ledger and OTLP sync use its consent intervals; an earlier consent does not cover later work after `off`. - `otel init` is a separate, explicit action to configure a provider's user-level exporter. The CLI does not show a separate consent dialog or record separate OTLP consent; the init skill asks for authorization before running it. When that provider and the collector are running, raw OTLP may be captured from other projects too: **the collector does not check clone consent**. The sync checks consent before forwarding. +- Codex chat titles are excluded by default. `otel sync --include-session-titles` separately opts in to sending the current title of each eligible Codex session to the selected Gouvernail destination, even when the title predates the consent interval. Titles are free text and may contain part of a prompt. This option does not send chat messages or change feature attribution. +- To send only previously measured Codex session titles, use `otel sync --config=codex --titles-only` with the same tenant and endpoint options as sync. Add `--session=<id>` to restrict it to one session. This reads the local capture to verify identity and consent but sends no captured metrics or feature bindings and does not change the follow checkpoint. +- Claude metric sync also sends a deterministic fingerprint derived from the original series attributes; it can expose equality or allow guessing low-entropy values. [Details](OBSERVABILITY.md#what-sync-sends). - `aidd telemetry off` stops local measurement for the current clone; it does **not** disable provider exporters or stop the OTLP collector. A process started with `AIDD_TELEMETRY=0` refuses local measurement and sync, not raw collection; changing an environment variable does not change an already running process. - `aidd telemetry forget` previews or deletes local usage measurement. It does **not** delete OTLP capture files, sync state, or data already received by Gouvernail. Production deletion is not available; the requested production policy is to disallow it. - Raw captures are private local files, but have no automatic retention limit. They can contain sensitive content. The local collector and sync processes do not restart automatically. From bad1f0a890bca04e5f02b2d0fa57d19b85a05cce Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 11:02:58 +0800 Subject: [PATCH 13/56] fix(telemetry): reject ambiguous OTLP data --- cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 2 + .../otel/actions/read-attribute-value.ts | 58 +++ .../otel/actions/sync-all-titles.ts | 4 +- .../infrastructure/otel/actions/sync.ts | 153 +++++-- .../session-title-sync.integration.test.ts | 13 + .../sync-otel-attributes.integration.test.ts | 395 ++++++++++++++++++ .../sync-otel-integrity.integration.test.ts | 185 +++++--- plugins/aidd-telemetry/OBSERVABILITY.md | 2 + 9 files changed, 730 insertions(+), 84 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts create mode 100644 cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts diff --git a/cli/package.json b/cli/package.json index c0757b9f8..797cf1612 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 799, + "bundleBudgetKB": 804, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 4593c112a..e2cf971e5 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -58,6 +58,8 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 0.1% measured headroom. // 799 KB: 798.8 KB with opt-in Codex titles, one-pass title synchronization and // telemetry integrity guards (+15.1 KB over 783.7 KB); less than 0.1% measured headroom. +// 804 KB: 803.3 KB with strict OTLP attribute and metric validation (+4.5 KB); +// 0.1% measured headroom. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts new file mode 100644 index 000000000..909ae4e7f --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/read-attribute-value.ts @@ -0,0 +1,58 @@ +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; + +const invalid = () => new Error("Invalid OTLP attribute value."); +const specialDoubles = new Set(["NaN", "Infinity", "-Infinity"]); + +/** Keep AnyValue's oneof type until it has been validated; never coerce containers. */ +export function readAttributeValue(raw: unknown, scalarRequired: boolean): string | undefined { + if (!scalarRequired && (raw === undefined || raw === null)) return undefined; + const value = asPlainObject(raw); + if (!value) throw invalid(); + const fields = Object.entries(value); + if (fields.length === 0 && !scalarRequired) return undefined; + const first = fields.at(0); + if (fields.length !== 1 || !first) throw invalid(); + const [kind, item] = first; + if (kind === "stringValue") { + if (typeof item !== "string") throw invalid(); + return item; + } + if (kind === "boolValue") { + if (typeof item !== "boolean") throw invalid(); + return String(item); + } + if (kind === "intValue") return readInteger(item); + if (kind === "doubleValue") { + if (typeof item === "number" && Number.isFinite(item)) return String(item); + if (!scalarRequired && typeof item === "string" && specialDoubles.has(item)) return item; + throw invalid(); + } + if (scalarRequired) throw invalid(); + if (kind === "bytesValue") { + if (typeof item !== "string") throw invalid(); + return undefined; + } + if (kind === "arrayValue" || kind === "kvlistValue") { + const container = asPlainObject(item); + if (!container || (container.values !== undefined && !Array.isArray(container.values))) + throw invalid(); + for (const child of container.values ?? []) { + const nested = kind === "arrayValue" ? child : asPlainObject(child)?.value; + readAttributeValue(nested, false); + } + return undefined; + } + // Profiling dictionary references and unknown optional extensions are not log values. + return undefined; +} + +function readInteger(value: unknown): string { + if (typeof value === "number") { + if (!Number.isSafeInteger(value)) throw invalid(); + } else if (typeof value !== "string" || !/^-?\d{1,20}$/.test(value)) { + throw invalid(); + } + const integer = BigInt(value); + if (integer < -9223372036854775808n || integer > 9223372036854775807n) throw invalid(); + return String(value); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts index b54753737..d3c5dd269 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts @@ -20,9 +20,7 @@ const MAX_LINE = 32 * 1024 * 1024; function alreadyKnown(identity: Identity | undefined, observed: Identity): boolean { if (identity?.conflict) return true; if (observed.conflict) return false; - return [...observed.values].every( - ([key, value]) => identity?.values.get(key)?.toLowerCase() === value.toLowerCase() - ); + return [...observed.values].every(([key, value]) => identity?.values.get(key) === value); } /** Read a fixed snapshot once; unlike follow, this never writes a capture checkpoint. */ diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index 4319c3032..4eb8b215c 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -3,9 +3,11 @@ import { constants } from "node:fs"; import { lstat, mkdir, open, rename, rm, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { setTimeout as delay } from "node:timers/promises"; +import { isDeepStrictEqual } from "node:util"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import type { SessionFeature } from "../../../domain/report/session-feature.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; +import { readAttributeValue } from "./read-attribute-value.js"; import { seriesFingerprint } from "./series-fingerprint.js"; import { createSessionTitlePublisher } from "./session-title.js"; @@ -44,6 +46,38 @@ const CLAUDE_METRICS = new Set([ "claude_code.token.usage", ]); const CODEX_EVENTS = new Set(["codex.conversation_starts", "codex.sse_event", "codex.tool_result"]); +const IDENTITY_ATTRIBUTES = new Set(["user.email", "user.account_id", "organization.id"]); +const NON_TEXT_ATTRIBUTES = new Set([ + "event.sequence", + "success", + "duration_ms", + "input_token_count", + "output_token_count", + "cached_token_count", + "cache_write_token_count", + "reasoning_token_count", +]); +const METRIC_DATA_FIELDS = [ + "sum", + "gauge", + "histogram", + "exponentialHistogram", + "exponential_histogram", + "summary", +]; +const METRIC_POINT_FIELDS = [ + "startTimeUnixNano", + "timeUnixNano", + "asInt", + "asDouble", + "count", + "sum", + "bucketCounts", + "explicitBounds", + "min", + "max", + "flags", +]; const SAFE_ATTRIBUTES = new Set([ "user.email", "user.account_id", @@ -81,20 +115,47 @@ export interface Identity { } function attributes(value: unknown): Map<string, string> { + if (value !== undefined && !Array.isArray(value)) throw new Error("Invalid OTLP attributes."); const out = new Map<string, string>(); + const seen = new Map<string, unknown>(); for (const item of array(value)) { const entry = object(item); - const v = object(entry.value); - const raw = v.stringValue ?? v.intValue ?? v.doubleValue ?? v.boolValue; - if (typeof entry.key === "string" && raw !== undefined && raw !== null) { - out.set(entry.key, String(raw)); + if (typeof entry.key === "string") { + if (seen.has(entry.key) && !isDeepStrictEqual(seen.get(entry.key), entry.value)) + throw new Error("Conflicting duplicate OTLP attributes."); + seen.set(entry.key, entry.value); + } + if (typeof entry.key === "string") { + const critical = SAFE_ATTRIBUTES.has(entry.key); + if ( + critical && + !NON_TEXT_ATTRIBUTES.has(entry.key) && + typeof object(entry.value).stringValue !== "string" + ) + throw new Error("Invalid OTLP attribute value."); + const raw = readAttributeValue(entry.value, critical); + if ( + raw === "" && + ["event.name", "event.kind", "session.id", "conversation.id"].includes(entry.key) + ) + throw new Error("Invalid OTLP attribute value."); + if (raw !== undefined) out.set(entry.key, raw); } } return out; } function merge(...maps: Map<string, string>[]): Map<string, string> { - return new Map(maps.flatMap((map) => [...map])); + const out = new Map<string, string>(); + for (const map of maps) { + for (const [key, value] of map) { + const previous = out.get(key); + if (IDENTITY_ATTRIBUTES.has(key) && previous !== undefined && previous !== value) + throw new Error("Conflicting OTLP identity attributes."); + out.set(key, value); + } + } + return out; } function safeAttributes(values: Map<string, string>) { @@ -106,12 +167,25 @@ function safeAttributes(values: Map<string, string>) { })); } +function eventNameFromBody(body: unknown): string { + if (body === undefined) return ""; + if (typeof object(body).stringValue !== "string") + throw new Error("Invalid OTLP attribute value."); + const value = readAttributeValue(body, true); + if (!value) throw new Error("Invalid OTLP attribute value."); + return value; +} + function numericFields(source: RecordObject, keys: readonly string[]): RecordObject { const result: RecordObject = {}; for (const key of keys) { const value = source[key]; if (value === undefined) continue; validateIntegerField(key, value); + if (["asDouble", "sum", "min", "max"].includes(key) && !isFiniteDouble(value)) + throw new Error("Invalid OTLP numeric field."); + if (key === "explicitBounds" && (!Array.isArray(value) || !value.every(isFiniteDouble))) + throw new Error("Invalid OTLP numeric field."); if ( (typeof value === "number" && Number.isFinite(value)) || (typeof value === "string" && /^-?\d+(?:\.\d+)?$/.test(value)) || @@ -127,6 +201,33 @@ function numericFields(source: RecordObject, keys: readonly string[]): RecordObj return result; } +function isFiniteDouble(value: unknown): boolean { + return ( + (typeof value === "number" && Number.isFinite(value)) || + (typeof value === "string" && /^-?\d+(?:\.\d+)?$/.test(value) && Number.isFinite(Number(value))) + ); +} + +function metricKind(metric: RecordObject): "sum" | "gauge" | "histogram" { + const fields = METRIC_DATA_FIELDS.filter((key) => metric[key] !== undefined); + const kind = fields.at(0); + if (fields.length !== 1 || (kind !== "sum" && kind !== "gauge" && kind !== "histogram")) + throw new Error("Invalid or unsupported OTLP metric data."); + const data = asPlainObject(metric[kind]); + if (!data || (data.dataPoints !== undefined && !Array.isArray(data.dataPoints))) + throw new Error("Invalid or unsupported OTLP metric data."); + return kind; +} + +function validateMetricPoint(point: RecordObject, kind: string): void { + const values = Number(point.asInt !== undefined) + Number(point.asDouble !== undefined); + if ((kind === "histogram" && values !== 0) || (kind !== "histogram" && values !== 1)) + throw new Error("Invalid OTLP metric point value."); + numericFields(point, METRIC_POINT_FIELDS); + if (point.flags !== undefined && (BigInt(String(point.flags)) & 1n) !== 0n) + throw new Error("OTLP metric point has no recorded value."); +} + function isExactInteger(value: unknown, minimum: bigint, maximum: bigint): boolean { if (typeof value === "number" && !Number.isSafeInteger(value)) return false; if (typeof value !== "number" && (typeof value !== "string" || !/^-?\d{1,20}$/.test(value))) @@ -153,12 +254,11 @@ function isExactNanoseconds(value: unknown): value is string | number { } function captureIdentity(target: Identity, values: Map<string, string>) { - for (const key of ["user.email", "user.account_id", "organization.id"]) { + for (const key of IDENTITY_ATTRIBUTES) { const value = values.get(key); if (!value) continue; const existing = target.values.get(key); - if (existing !== undefined && existing.toLowerCase() !== value.toLowerCase()) - target.conflict = true; + if (existing !== undefined && existing !== value) target.conflict = true; else target.values.set(key, value); } } @@ -233,20 +333,28 @@ export function sessionIdsInCapture(line: string, config: SyncOptions["config"]) for (const raw of array(payload.resourceLogs)) { const resource = object(raw); const attrs = attributes(object(resource.resource).attributes); - for (const scope of array(resource.scopeLogs)) + for (const scope of array(resource.scopeLogs)) { + attributes(object(object(scope).scope).attributes); for (const record of array(object(scope).logRecords)) add(attrs, record); + } } } if (config === CLAUDE_CONFIG_ID && envelope.signal === "metrics") { for (const raw of array(payload.resourceMetrics)) { const resource = object(raw); const attrs = attributes(object(resource.resource).attributes); - for (const scope of array(resource.scopeMetrics)) + for (const scope of array(resource.scopeMetrics)) { + attributes(object(object(scope).scope).attributes); for (const rawMetric of array(object(scope).metrics)) { const metric = object(rawMetric); - for (const kind of ["sum", "gauge", "histogram"]) - for (const point of array(object(metric[kind]).dataPoints)) add(attrs, point); + if (typeof metric.name !== "string" || !CLAUDE_METRICS.has(metric.name)) continue; + for (const kind of [metricKind(metric)]) + for (const point of array(object(metric[kind]).dataPoints)) { + validateMetricPoint(object(point), kind); + add(attrs, point); + } } + } } } return [...ids]; @@ -298,12 +406,13 @@ export function selectTransfers( const resource = object(rawResource); const resourceAttrs = attributes(object(resource.resource).attributes); for (const rawScope of array(resource.scopeLogs)) { + attributes(object(object(rawScope).scope).attributes); for (const rawRecord of array(object(rawScope).logRecords)) { const record = object(rawRecord); const a = merge(resourceAttrs, attributes(record.attributes)); if (a.get(sessionKey) !== options.session) continue; captureIdentity(identity, a); - const event = a.get("event.name") ?? String(object(record.body).stringValue ?? ""); + const event = a.get("event.name") ?? eventNameFromBody(record.body); const wanted = options.config === CLAUDE_CONFIG_ID ? event === "tool_result" || event === "claude_code.tool_result" @@ -345,13 +454,15 @@ export function selectTransfers( const resource = object(rawResource); const resourceAttrs = attributes(object(resource.resource).attributes); for (const rawScope of array(resource.scopeMetrics)) { + attributes(object(object(rawScope).scope).attributes); for (const rawMetric of array(object(rawScope).metrics)) { const metric = object(rawMetric); if (typeof metric.name !== "string" || !CLAUDE_METRICS.has(metric.name)) continue; - for (const kind of ["sum", "gauge", "histogram"]) { + for (const kind of [metricKind(metric)]) { const aggregation = object(metric[kind]); for (const rawPoint of array(aggregation.dataPoints)) { const point = object(rawPoint); + validateMetricPoint(point, kind); const a = merge(resourceAttrs, attributes(point.attributes)); if (a.get(sessionKey) !== options.session) continue; captureIdentity(identity, a); @@ -374,19 +485,7 @@ export function selectTransfers( ) ); const sanitized = { - ...numericFields(point, [ - "startTimeUnixNano", - "timeUnixNano", - "asInt", - "asDouble", - "count", - "sum", - "bucketCounts", - "explicitBounds", - "min", - "max", - "flags", - ]), + ...numericFields(point, METRIC_POINT_FIELDS), attributes: safeAttributes(a), }; out.push({ diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts index 18244a938..416ad0315 100644 --- a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts @@ -242,6 +242,19 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { ); expect(noCurrentConsent).toMatchObject({ sent: 0, identified: 1, currentConsenting: 0 }); + received.length = 0; + await appendFile( + captureFile, + `${JSON.stringify(capture).replaceAll("person@example.test", "PERSON@EXAMPLE.TEST")}\n` + ); + const conflictingIdentity = await syncAllSessionTitles( + { ...options, includeSessionTitles: true }, + async () => new Map([["session-1", admittedSession]]), + async () => "allow" + ); + expect(conflictingIdentity).toMatchObject({ sent: 0, identified: 0 }); + expect(received).toHaveLength(0); + received.length = 0; const abort = new AbortController(); const timeout = setTimeout(() => abort.abort(), 3000); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts new file mode 100644 index 000000000..717ee0445 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -0,0 +1,395 @@ +import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; +import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; + +const attribute = (key: string, value: unknown) => ({ key, value }); +const text = (key: string, value: string) => attribute(key, { stringValue: value }); +const options = { + config: "codex" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:1", + token: "test", + home: "/tmp", + follow: false, +}; +type Attributes = ReturnType<typeof attribute>[]; + +function capture( + record: Attributes, + resource: Attributes = [], + scope: Attributes = [], + body?: unknown +): string { + return JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: resource }, + scopeLogs: [ + { + scope: { attributes: scope }, + logRecords: [ + { + ...(body === undefined ? {} : { body }), + attributes: [ + text("conversation.id", "session"), + ...(body === undefined ? [text("event.name", "codex.sse_event")] : []), + text("event.kind", "response.completed"), + text("event.timestamp", "2026-11-05T00:00:00Z"), + ...record, + ], + }, + ], + }, + ], + }, + ], + }, + }); +} + +test.each([ + { stringValue: ["codex.sse_event"], intValue: "9" }, + { stringValue: ["codex.sse_event"] }, + { stringValue: "" }, + { intValue: "9" }, + {}, + null, +])("rejects invalid fallback event bodies without coercion", (body) => { + expect(() => select(capture([], [], [], body))).toThrow("Invalid OTLP attribute value."); +}); + +test("keeps a valid string log body as the event-name fallback", () => { + expect(select(capture([], [], [], { stringValue: "codex.sse_event" }))).toHaveLength(1); + expect( + selectTransfers( + capture([text("session.id", "session")], [], [], { stringValue: "tool_result" }), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toHaveLength(1); +}); + +function select(line: string) { + return selectTransfers(line, options, { values: new Map(), conflict: false }); +} + +function metricCapture( + point: Attributes, + resource: Attributes = [], + scope: Attributes = [] +): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { + resource: { attributes: resource }, + scopeMetrics: [ + { + scope: { attributes: scope }, + metrics: [ + { + name: "claude_code.token.usage", + sum: { + aggregationTemporality: 1, + dataPoints: [ + { + startTimeUnixNano: "1793836800000000000", + timeUnixNano: "1793836801000000000", + asInt: "5", + attributes: [text("session.id", "session"), ...point], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); +} + +test.each(["point", "resource", "scope"])("rejects contradictory Claude %s attributes", (level) => { + const conflict = [text("type", "input"), text("type", "output")]; + expect(() => + selectTransfers( + metricCapture( + level === "point" ? conflict : [], + level === "resource" ? conflict : [], + level === "scope" ? conflict : [] + ), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toThrow("Conflicting duplicate OTLP attributes."); +}); + +test("rejects contradictory Claude resource and point identities", () => { + expect(() => + selectTransfers( + metricCapture([text("user.email", "private-b")], [text("user.email", "private-a")]), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toThrow("Conflicting OTLP identity attributes."); +}); + +test.each(["input_token_count", "user.email", "private.unforwarded.attribute"])( + "rejects conflicting duplicate attributes without exposing their contents: %s", + (key) => { + expect(() => select(capture([text(key, "private-a"), text(key, "private-b")]))).toThrow( + "Conflicting duplicate OTLP attributes." + ); + } +); + +test("does not equate differently typed OTLP values that stringify identically", () => { + expect(() => + select( + capture([attribute("input_token_count", { intValue: 1 }), text("input_token_count", "1")]) + ) + ).toThrow("Conflicting duplicate OTLP attributes."); +}); + +test.each([ + { stringValue: ["5"], intValue: "900" }, + { stringValue: ["5"] }, + { stringValue: 5 }, + { intValue: Number.MAX_SAFE_INTEGER + 1 }, + { intValue: 1.5 }, + { intValue: "9223372036854775808" }, + { doubleValue: "5" }, + { boolValue: "true" }, + [], + null, + {}, + { arrayValue: { values: [{ stringValue: "5" }] } }, + { kvlistValue: { values: [] } }, + { bytesValue: "NQ==" }, +])("refuses malformed or non-scalar critical AnyValue without coercion", (value) => { + expect(() => select(capture([attribute("input_token_count", value)]))).toThrow( + "Invalid OTLP attribute value." + ); +}); + +test("does not coerce an identity from a malformed stringValue", () => { + expect(() => + select(capture([attribute("user.email", { stringValue: ["person@example.test"] })])) + ).toThrow("Invalid OTLP attribute value."); +}); + +test.each([ + "user.email", + "user.account_id", + "organization.id", + "model", + "event.name", + "event.kind", + "session.id", +])("refuses non-text values for textual telemetry fields: %s", (key) => { + expect(() => select(capture([attribute(key, { intValue: "5" })]))).toThrow(); +}); + +test.each(["event.name", "event.kind", "session.id", "conversation.id"])( + "refuses an explicitly empty mandatory context value: %s", + (key) => { + const original = key === "session.id" ? metricCapture([]) : capture([]); + const initial = { + "event.name": "codex.sse_event", + "event.kind": "response.completed", + "session.id": "session", + "conversation.id": "session", + }[key]; + const body = original.replace( + JSON.stringify(text(key, initial ?? "")), + JSON.stringify(text(key, "")) + ); + expect(() => + selectTransfers( + body, + { ...options, config: key === "session.id" ? "claude" : "codex" }, + { values: new Map(), conflict: false } + ) + ).toThrow(); + } +); + +test("keeps supported scalar AnyValue types exact", () => { + const [transfer] = select( + capture([ + attribute("input_token_count", { intValue: "9007199254740993" }), + attribute("success", { boolValue: false }), + attribute("duration_ms", { doubleValue: 1.5 }), + ]) + ); + const payload = JSON.stringify(transfer.payload); + for (const value of ["9007199254740993", "false", "1.5"]) + expect(payload).toContain(`"stringValue":"${value}"`); +}); + +test("ignores valid empty and structured noncritical values", () => { + expect( + select( + capture([ + attribute("unused.empty", {}), + attribute("unused.array", { arrayValue: { values: [{ stringValue: "private" }] } }), + attribute("unused.list", { + kvlistValue: { values: [{ key: "item", value: { boolValue: true } }] }, + }), + attribute("unused.bytes", { bytesValue: "NQ==" }), + ]) + ) + ).toHaveLength(1); +}); + +test("accepts identical duplicate attributes", () => { + const [transfer] = select( + capture([text("input_token_count", "5"), text("input_token_count", "5")]) + ); + expect(JSON.stringify(transfer.payload)).toContain( + '"key":"input_token_count","value":{"stringValue":"5"}' + ); +}); + +test.each(["resource", "scope"])("rejects conflicting duplicates in %s attributes", (level) => { + const conflict = [text("user.email", "private-a"), text("user.email", "private-b")]; + expect(() => + select(capture([], level === "resource" ? conflict : [], level === "scope" ? conflict : [])) + ).toThrow("Conflicting duplicate OTLP attributes."); +}); + +test("preserves non-identity attribute overrides between resource and record", () => { + const [transfer] = select( + capture([text("model", "record-model")], [text("model", "resource-model")]) + ); + expect(JSON.stringify(transfer.payload)).toContain("record-model"); + expect(JSON.stringify(transfer.payload)).not.toContain("resource-model"); +}); + +test.each(["user.email", "user.account_id", "organization.id"])( + "does not choose between contradictory resource and record identities: %s", + (key) => { + expect(() => select(capture([text(key, "private-b")], [text(key, "private-a")]))).toThrow( + "Conflicting OTLP identity attributes." + ); + } +); + +test("keeps identical identity values across resource and record", () => { + expect( + select(capture([text("user.email", "a@example.test")], [text("user.email", "a@example.test")])) + ).toHaveLength(1); +}); + +test.each(["user.email", "user.account_id", "organization.id"])( + "does not assume case-insensitive identity semantics: %s", + (key) => { + expect(() => select(capture([text(key, "ABC")], [text(key, "abc")]))).toThrow( + "Conflicting OTLP identity attributes." + ); + } +); + +test("marks case-distinct session identities as conflicting across observations", () => { + const identity = { values: new Map([["user.account_id", "abc"]]), conflict: false }; + selectTransfers(capture([text("user.account_id", "ABC")]), options, identity); + expect(identity.conflict).toBe(true); + expect(identity.values.get("user.account_id")).toBe("abc"); +}); + +const dirs: string[] = []; +let server: Server | undefined; +afterEach(async () => { + if (server) await new Promise<void>((resolve) => server?.close(() => resolve())); + server = undefined; + await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +test.each([ + { + values: [text("input_token_count", "100"), text("input_token_count", "5")], + error: "Conflicting duplicate OTLP attributes.", + body: undefined, + }, + { + values: [attribute("input_token_count", { stringValue: ["5"], intValue: "900" })], + error: "Invalid OTLP attribute value.", + body: undefined, + }, + { + values: [attribute("input_token_count", { intValue: Number.MAX_SAFE_INTEGER + 1 })], + error: "Invalid OTLP attribute value.", + body: undefined, + }, + { + values: [attribute("user.email", { stringValue: ["person@example.test"] })], + error: "Invalid OTLP attribute value.", + body: undefined, + }, + { + values: [], + error: "Invalid OTLP attribute value.", + body: { stringValue: ["codex.sse_event"], intValue: "9" }, + }, +])( + "invalid attributes stop sync before sending or advancing its checkpoint", + async ({ values, error, body }) => { + const home = await mkdtemp(join(tmpdir(), "aidd-conflicting-attributes-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const file = join(captureDir, "codex.jsonl"); + await writeFile(file, "", { mode: 0o600 }); + let calls = 0; + const abort = new AbortController(); + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const stateDir = join(home, "state"); + const running = syncAllOtel( + { + ...options, + home, + stateDir, + endpoint: `http://127.0.0.1:${address.port}`, + intervalMs: 10, + signal: abort.signal, + }, + async () => + new Map([ + ["session", { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }], + ]), + async () => "allow" + ); + const directory = join(stateDir, "otel-sync"); + let checkpoint: string | undefined; + for (let attempt = 0; attempt < 100 && !checkpoint; attempt++) { + checkpoint = (await readdir(directory).catch(() => [])).find((entry) => + entry.endsWith(".checkpoint.json") + ); + if (!checkpoint) await new Promise((resolve) => setTimeout(resolve, 10)); + } + if (!checkpoint) throw new Error("No initial checkpoint"); + await appendFile(file, `${capture(values, [], [], body)}\n`); + await expect(running).rejects.toThrow(error); + expect(calls).toBe(0); + expect(JSON.parse(await readFile(join(directory, checkpoint), "utf8")).offset).toBe(0); + } +); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts index f512bc225..9c11ecacb 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -87,9 +87,7 @@ function claudeMetric(startTime: string | number, endTime: string | number): str }); } -function claudePoint( - extra: Record<string, string | number | Array<string | number> | null | undefined> -): string { +function claudePoint(extra: Record<string, unknown>): string { return JSON.stringify({ signal: "metrics", payload: { @@ -122,6 +120,78 @@ function claudePoint( }); } +function metricData(data: Record<string, unknown>): string { + return JSON.stringify({ + signal: "metrics", + payload: { + resourceMetrics: [ + { scopeMetrics: [{ metrics: [{ name: "claude_code.token.usage", ...data }] }] }, + ], + }, + }); +} + +test.each([ + { asDouble: 5 }, + { asInt: undefined, asDouble: [5] }, + { asInt: undefined, asDouble: {} }, + { asInt: undefined, asDouble: true }, + { asInt: undefined, asDouble: null }, + { asInt: undefined, asDouble: "NaN" }, + { asInt: undefined }, + { flags: 1 }, + { flags: "3" }, + { sum: [] }, + { explicitBounds: ["bad"] }, +])("rejects malformed or unrecorded NumberDataPoint values", (extra) => { + expect(() => + selectTransfers( + claudePoint(extra), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toThrow(/OTLP/); +}); + +test.each([ + {}, + { summary: {} }, + { exponentialHistogram: {} }, + { sum: {}, gauge: {} }, + { sum: {}, histogram: {} }, + { sum: {}, summary: {} }, + { sum: {}, exponentialHistogram: {} }, + { sum: {}, exponential_histogram: {} }, + { sum: { dataPoints: {} } }, +])("rejects missing, unsupported or competing data on a supported metric", (data) => { + expect(() => + selectTransfers( + metricData(data), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ) + ).toThrow(/OTLP metric data/); +}); + +test("keeps valid histograms separate from NumberDataPoint's scalar oneof", () => { + const point = { + startTimeUnixNano: "1785044858922210000", + timeUnixNano: "1785044858922210000", + count: "2", + sum: 5.5, + explicitBounds: [1.5], + bucketCounts: ["1", "1"], + attributes: [attribute("session.id", "session")], + }; + const [transfer] = selectTransfers( + metricData({ histogram: { dataPoints: [point] } }), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ); + expect(JSON.stringify(transfer.payload)).toContain('"count":"2"'); + expect(JSON.stringify(transfer.payload)).toContain('"sum":5.5'); +}); + test.each([ { asInt: 1785044858922210000 }, { asInt: 1.5 }, @@ -163,58 +233,67 @@ test("preserves exact integer strings for signed and unsigned metric values", () expect(payload).toContain(`"${value}"`); }); -test("an inexact metric stops all-session sync before delivery or checkpoint advance", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-inexact-metric-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - const stateDir = join(home, "state"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const file = join(captureDir, "claude.jsonl"); - await writeFile(file, "", { mode: 0o600 }); - let calls = 0; - const abort = new AbortController(); - server = createServer((request, response) => { - request.resume(); - request.on("end", () => { - calls++; - response.writeHead(200).end("{}"); - abort.abort(); +test.each([ + claudePoint({ asInt: 1785044858922210000 }), + claudePoint({ asDouble: 5 }), + claudePoint({ flags: 1 }), + metricData({ sum: {}, summary: {} }), + metricData({ summary: {} }), +])( + "an invalid metric stops all-session sync before delivery or checkpoint advance", + async (line) => { + const home = await mkdtemp(join(tmpdir(), "aidd-inexact-metric-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + const stateDir = join(home, "state"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const file = join(captureDir, "claude.jsonl"); + await writeFile(file, "", { mode: 0o600 }); + let calls = 0; + const abort = new AbortController(); + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + response.writeHead(200).end("{}"); + abort.abort(); + }); }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const running = syncAllOtel( - { - config: "claude", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test", - home, - stateDir, - intervalMs: 10, - signal: abort.signal, - }, - async () => - new Map([ - ["session", { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }], - ]), - async () => "allow" - ); - const directory = join(stateDir, "otel-sync"); - let checkpoint: string | undefined; - for (let attempt = 0; attempt < 100 && !checkpoint; attempt++) { - const entries = await readdir(directory).catch(() => []); - checkpoint = entries.find((entry) => entry.endsWith(".checkpoint.json")); - if (!checkpoint) await new Promise((resolve) => setTimeout(resolve, 10)); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const running = syncAllOtel( + { + config: "claude", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test", + home, + stateDir, + intervalMs: 10, + signal: abort.signal, + }, + async () => + new Map([ + ["session", { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }], + ]), + async () => "allow" + ); + const directory = join(stateDir, "otel-sync"); + let checkpoint: string | undefined; + for (let attempt = 0; attempt < 100 && !checkpoint; attempt++) { + const entries = await readdir(directory).catch(() => []); + checkpoint = entries.find((entry) => entry.endsWith(".checkpoint.json")); + if (!checkpoint) await new Promise((resolve) => setTimeout(resolve, 10)); + } + if (!checkpoint) throw new Error("No initial checkpoint"); + await appendFile(file, `${line}\n`); + await expect(running).rejects.toThrow(/OTLP/); + const saved = JSON.parse(await readFile(join(directory, checkpoint), "utf8")); + expect(saved.offset).toBe(0); + expect(calls).toBe(0); } - if (!checkpoint) throw new Error("No initial checkpoint"); - await appendFile(file, `${claudePoint({ asInt: 1785044858922210000 })}\n`); - await expect(running).rejects.toThrow(/invalid OTLP integer/i); - const saved = JSON.parse(await readFile(join(directory, checkpoint), "utf8")); - expect(saved.offset).toBe(0); - expect(calls).toBe(0); -}); +); function metricWithMetadata(unit: string, temporality: number, monotonic: boolean): string { return JSON.stringify({ diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index f9e0d9104..b398f5474 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -52,6 +52,8 @@ The collector keeps the provider's raw OTLP JSON locally. The separate dev sync The sync excludes fields for prompt and response text, tool arguments and output, raw API bodies, metric exemplars and traces. It filters attribute keys, not secrets or free text inside allowed values. OpenCode is not forwarded by this sync. These are forwarding rules, not guarantees about what the raw collector receives. +Codex can replace absent cache details with zeros before emitting OTLP. A zero cache count therefore does not prove that the provider reported zero; Gouvernail displays it, and any uncached-input total depending on it, as unavailable. Nonzero counts are still observations, not proof that every event was received. [Codex response parser](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/codex-api/src/sse/responses.rs), [OTLP emitter](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/otel/src/events/session_telemetry.rs) (read 2026-11-05). + ### Example: a Claude metric sent locally The structure below was produced by `selectTransfers` from one real local Claude capture. The timestamps, metric value, session, organization, email, account, model and query source have been replaced with fictional values; attribute presence varies by event. It is the filtered request body, not the raw capture or its authorization header. From 0a702ede272f9ed4b294c5b018018862761ccb16 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 11:40:01 +0800 Subject: [PATCH 14/56] feat(telemetry): sync verified native session projects --- cli/aidd_docs/memory/codebase-map.md | 1 + cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 2 + .../otel/actions/metadata/session-project.ts | 51 +++++++++ .../actions/{ => metadata}/session-title.ts | 107 +++++++++++++----- .../otel/actions/sync-all-titles.ts | 33 ++++-- .../infrastructure/otel/actions/sync-all.ts | 28 +++-- .../infrastructure/otel/actions/sync.ts | 32 ++++-- cli/src/presentation/commands/telemetry.ts | 27 ++++- .../otel/session-project.unit.test.ts | 81 +++++++++++++ .../session-title-sync.integration.test.ts | 56 ++++++++- .../telemetry/otel/session-title.unit.test.ts | 2 +- cli/tests/golden/snapshots/help/surface.json | 2 +- plugins/aidd-telemetry/OBSERVABILITY.md | 6 + plugins/aidd-telemetry/README.md | 2 + 15 files changed, 363 insertions(+), 69 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts rename cli/src/contexts/telemetry/infrastructure/otel/actions/{ => metadata}/session-title.ts (66%) create mode 100644 cli/tests/contexts/telemetry/otel/session-project.unit.test.ts diff --git a/cli/aidd_docs/memory/codebase-map.md b/cli/aidd_docs/memory/codebase-map.md index afa687e78..1fe6eb091 100644 --- a/cli/aidd_docs/memory/codebase-map.md +++ b/cli/aidd_docs/memory/codebase-map.md @@ -74,6 +74,7 @@ src/ โ”‚ โ”‚ โ””โ”€โ”€ infrastructure/ โ”‚ โ”‚ โ”œโ”€โ”€ otel/ # shared native OTLP settings; JSON templates in config/ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ actions/ # initialize configuration, launch export, verify receipt +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ metadata/ # consented native Codex session metadata โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # reading a clone's identity from the file system, and the consent history file โ”‚ โ”‚ โ”œโ”€โ”€ declaration/ # where a declaration is kept: git config and the telemetry dir โ”‚ โ”‚ โ”œโ”€โ”€ forget/ # erasing the telemetry dir, the previous version's files, git config keys diff --git a/cli/package.json b/cli/package.json index 797cf1612..fb2736573 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 804, + "bundleBudgetKB": 807, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index e2cf971e5..58c801476 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -60,6 +60,8 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // telemetry integrity guards (+15.1 KB over 783.7 KB); less than 0.1% measured headroom. // 804 KB: 803.3 KB with strict OTLP attribute and metric validation (+4.5 KB); // 0.1% measured headroom. +// 807 KB: 806.6 KB with opt-in native Codex project metadata and shared metadata publishing; +// +3.3 KB, less than 0.1% measured headroom. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts new file mode 100644 index 000000000..5d8adfee6 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts @@ -0,0 +1,51 @@ +import { asPlainObject } from "../../../../../../kernel/reading/plain-object.js"; +import { createSessionMetadataPublisher, type Metadata, type Request } from "./session-title.js"; + +function hasControlCharacter(text: string): boolean { + return [...text].some( + (character) => character.charCodeAt(0) < 32 || character.charCodeAt(0) === 127 + ); +} + +/** Native project membership only: roots and cwd are not evidence of membership. */ +export async function readSessionProject(request: Request, session: string): Promise<Metadata> { + try { + const readThread = async () => { + const result = await request("thread/read", { threadId: session, includeTurns: false }); + const thread = asPlainObject(asPlainObject(result)?.thread); + return thread?.id === session && thread.sessionId === session ? thread : null; + }; + const thread = await readThread(); + if (!thread) return { available: false }; + if (thread.projectId === null) return { available: true, values: {} }; + const id = thread.projectId; + if (typeof id !== "string" || !id.trim() || id.length > 1024 || hasControlCharacter(id)) + return { available: false }; + const result = await request("project/read", { projectId: id }); + const project = asPlainObject(asPlainObject(result)?.project); + if ( + project?.id !== id || + typeof project.name !== "string" || + !project.name.trim() || + project.name.length > 4096 || + hasControlCharacter(project.name) + ) + return { available: false }; + if ((await readThread())?.projectId !== id) return { available: false }; + return { + available: true, + values: { "aidd.project_id": id, "aidd.project_name": project.name }, + }; + } catch { + return { available: false }; + } +} + +export function createSessionProjectPublisher(home: string, onUnavailable?: () => void) { + return createSessionMetadataPublisher( + home, + "aidd.session_project", + readSessionProject, + onUnavailable + ); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts similarity index 66% rename from cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts rename to cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts index bcbd03dc5..274318a20 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/session-title.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts @@ -1,10 +1,10 @@ import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process"; import { createInterface } from "node:readline"; -import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; -import { CODEX_CONFIG_ID } from "../settings.js"; -import type { Identity, Transfer } from "./sync.js"; +import { asPlainObject } from "../../../../../../kernel/reading/plain-object.js"; +import { CODEX_CONFIG_ID } from "../../settings.js"; +import type { Identity, Transfer } from "../sync.js"; -type Request = (method: string, params: object) => Promise<unknown>; +export type Request = (method: string, params: object) => Promise<unknown>; type Title = { available: true; title: string | null } | { available: false }; const MAX_TITLE_LENGTH = 4096; const REQUEST_TIMEOUT_MS = 5000; @@ -42,6 +42,13 @@ export function sessionTitleTransfer( values.set("event.name", "aidd.session_title"); values.set("event.timestamp", at.toISOString()); if (title !== null) values.set("aidd.session_title", title); + return metadataTransfer(session, { ...identity, values }, at); +} + +function metadataTransfer(session: string, identity: Identity, at: Date): Transfer { + const values = new Map(identity.values); + values.set("conversation.id", session); + values.set("event.timestamp", at.toISOString()); return { signal: "logs", at: at.toISOString(), @@ -67,14 +74,18 @@ export function sessionTitleTransfer( }; } -class AppServer { +class AppServer<T> { private child: ChildProcessWithoutNullStreams; private nextId = 0; private pending = new Map<number, { resolve: (result: unknown) => void; reject: () => void }>(); private ready: Promise<unknown>; private failed = false; - constructor(home: string) { + constructor( + home: string, + private readValue: (request: Request, session: string) => Promise<T>, + experimental = false + ) { this.child = spawn(CODEX_CONFIG_ID, ["app-server", "--listen", "stdio://"], { env: { ...process.env, HOME: home }, stdio: ["pipe", "pipe", "pipe"], @@ -86,6 +97,7 @@ class AppServer { this.child.on("exit", () => this.fail()); this.ready = this.request("initialize", { clientInfo: { name: "aidd_telemetry", title: "AIDD telemetry", version: "1" }, + ...(experimental ? { capabilities: { experimentalApi: true } } : {}), }).then(() => { this.child.stdin.write(`${JSON.stringify({ method: "initialized", params: {} })}\n`); }); @@ -135,13 +147,9 @@ class AppServer { }); } - async read(session: string): Promise<Title> { - try { - await this.ready; - return await readSessionTitle((method, params) => this.request(method, params), session); - } catch { - return { available: false }; - } + async read(session: string): Promise<T> { + await this.ready; + return this.readValue((method, params) => this.request(method, params), session); } close(): void { @@ -154,19 +162,24 @@ class AppServer { } } -type TitleReader = { read(session: string): Promise<Title>; close(): void; unavailable?: boolean }; +type Reader<T> = { read(session: string): Promise<T>; close(): void; unavailable?: boolean }; -export function createSessionTitlePublisher( +export type Metadata = { available: true; values: Record<string, string> } | { available: false }; + +function createPublisher<T extends { available: boolean }>( home: string, + read: (request: Request, session: string) => Promise<T>, + transfer: (session: string, identity: Identity, result: T, at: Date) => Transfer | null, + experimental: boolean, dependencies: { now?: () => number; - createReader?: (home: string) => TitleReader; + createReader?: (home: string) => Reader<T>; onUnavailable?: () => void; } = {} ) { - let server: TitleReader | undefined; + let server: Reader<T> | undefined; const checked = new Map<string, number>(); - const previous = new Map<string, string | null>(); + const previous = new Map<string, string>(); let warned = false; let retryAfter = 0; return { @@ -187,21 +200,24 @@ export function createSessionTitlePublisher( server.close(); server = undefined; } - server ??= (dependencies.createReader ?? ((path) => new AppServer(path)))(home); - const result = await server.read(session); + server ??= (dependencies.createReader ?? ((path) => new AppServer(path, read, experimental)))( + home + ); + const result = await server.read(session).catch(() => null); const observedAt = (dependencies.now ?? Date.now)(); checked.set(session, observedAt); - if (!result.available) { + if (!result?.available) { if (server.unavailable) retryAfter = observedAt + REFRESH_MS; if (!warned) dependencies.onUnavailable?.(); warned = true; return false; } - if (previous.has(session) && previous.get(session) === result.title) return false; - const transfer = sessionTitleTransfer(session, identity, result.title, new Date(observedAt)); - if (!transfer) return false; - if (!(await send(transfer))) return false; - previous.set(session, result.title); + const key = JSON.stringify(result); + if (previous.get(session) === key) return false; + const event = transfer(session, identity, result, new Date(observedAt)); + if (!event) return false; + if (!(await send(event))) return false; + previous.set(session, key); return true; }, close(): void { @@ -209,3 +225,42 @@ export function createSessionTitlePublisher( }, }; } + +export function createSessionTitlePublisher( + home: string, + dependencies: { + now?: () => number; + createReader?: (home: string) => Reader<Title>; + onUnavailable?: () => void; + } = {} +) { + return createPublisher( + home, + readSessionTitle, + (session, identity, result, at) => + result.available ? sessionTitleTransfer(session, identity, result.title, at) : null, + false, + dependencies + ); +} + +export function createSessionMetadataPublisher( + home: string, + eventName: string, + read: (request: Request, session: string) => Promise<Metadata>, + onUnavailable?: () => void +) { + return createPublisher( + home, + read, + (session, identity, result, at) => { + if (!result.available) return null; + const values = new Map(identity.values); + values.set("event.name", eventName); + for (const [key, value] of Object.entries(result.values)) values.set(key, value); + return metadataTransfer(session, { ...identity, values }, at); + }, + true, + { onUnavailable } + ); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts index d3c5dd269..f81e3700a 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts @@ -3,7 +3,8 @@ import { lstat, open } from "node:fs/promises"; import { join } from "node:path"; import type { SessionAdmission } from "../../../domain/report/session-feature.js"; import { CODEX_CONFIG_ID } from "../settings.js"; -import { createSessionTitlePublisher } from "./session-title.js"; +import { createSessionProjectPublisher } from "./metadata/session-project.js"; +import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { destination, type Identity, @@ -37,6 +38,9 @@ export async function syncAllSessionTitles( const path = join(options.home, ".local/share/aidd-otel-observation", "codex.jsonl"); const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); const identities = new Map<string, Identity>(); + const projects = options.includeSessionProjects + ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) + : undefined; const titles = createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable, }); @@ -104,21 +108,26 @@ export async function syncAllSessionTitles( identified++; if ((await permits(admission, new Date().toISOString())) !== "allow") continue; currentConsenting++; - if ( - await titles.publish(session, identity, async (event) => { - if (!event.at || (await permits(admission, event.at)) !== "allow") { - currentConsenting--; - return false; - } - await send({ ...base, session }, event); - return true; - }) - ) - sent++; + let stillConsenting = true; + for (const publisher of [titles, projects]) + if ( + publisher && + (await publisher.publish(session, identity, async (event) => { + if (!event.at || (await permits(admission, event.at)) !== "allow") { + if (stillConsenting) currentConsenting--; + stillConsenting = false; + return false; + } + await send({ ...base, session }, event); + return true; + })) + ) + sent++; } return { sent, identified, currentConsenting, partialLine: pending.length > 0 }; } finally { titles.close(); + projects?.close(); await file.close(); } } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts index 34e820f19..a68a7bbb8 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -7,7 +7,8 @@ import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import type { SessionAdmission, SessionFeature } from "../../../domain/report/session-feature.js"; import { LedgerLock } from "../../ledger-lock.js"; import { CODEX_CONFIG_ID } from "../settings.js"; -import { createSessionTitlePublisher } from "./session-title.js"; +import { createSessionProjectPublisher } from "./metadata/session-project.js"; +import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { bindingTransfer, destination, @@ -228,6 +229,10 @@ export async function syncAllOtel( staleAfterMs: Number.MAX_SAFE_INTEGER, }); const release = await lock.acquire(); + const projects = + options.config === CODEX_CONFIG_ID && options.includeSessionProjects + ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) + : undefined; const titles = options.config === CODEX_CONFIG_ID && options.includeSessionTitles ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) @@ -386,19 +391,21 @@ export async function syncAllOtel( } }; const publishTitles = async () => { - if (!titles) return; + if (!titles && !projects) return; for (const [id, state] of sessions) { const admission = held.get(id); if (!admission || state.identity.conflict) continue; if ((await permits(admission, new Date().toISOString())) !== "allow") continue; - if ( - await titles.publish(id, state.identity, async (event) => { - if (!event.at || (await permits(admission, event.at)) !== "allow") return false; - await send(asOptions(options, id), event); - return true; - }) - ) - sent++; + for (const publisher of [titles, projects]) + if ( + publisher && + (await publisher.publish(id, state.identity, async (event) => { + if (!event.at || (await permits(admission, event.at)) !== "allow") return false; + await send(asOptions(options, id), event); + return true; + })) + ) + sent++; } }; const flushPending = async () => { @@ -524,6 +531,7 @@ export async function syncAllOtel( } } finally { titles?.close(); + projects?.close(); await file?.close(); await release(); } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index 4eb8b215c..a264ceda1 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -7,9 +7,10 @@ import { isDeepStrictEqual } from "node:util"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import type { SessionFeature } from "../../../domain/report/session-feature.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; +import { createSessionProjectPublisher } from "./metadata/session-project.js"; +import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { readAttributeValue } from "./read-attribute-value.js"; import { seriesFingerprint } from "./series-fingerprint.js"; -import { createSessionTitlePublisher } from "./session-title.js"; type FeatureState = SessionFeature; @@ -25,6 +26,8 @@ export interface SyncOptions { readonly signal?: AbortSignal; readonly intervalMs?: number; readonly includeSessionTitles?: boolean; + readonly includeSessionProjects?: boolean; + readonly onProjectUnavailable?: () => void; readonly onTitleUnavailable?: () => void; readonly titlesOnly?: boolean; } @@ -786,6 +789,10 @@ export async function syncOtel( ? join(options.stateDir, "otel-sync", `${stateName}.json`) : null; let mayHaveLinked = false; + const projects = + options.config === CODEX_CONFIG_ID && options.includeSessionProjects + ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) + : undefined; const titles = options.config === CODEX_CONFIG_ID && options.includeSessionTitles ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) @@ -903,16 +910,18 @@ export async function syncOtel( needsReplay = false; } } - if ( - bindingAllowed && - titles && - (await titles.publish(options.session, identity, async (event) => { - if (!(await permits(event))) return false; - await send(options, event); - return true; - })) - ) - sent++; + if (bindingAllowed) + for (const publisher of [titles, projects]) { + if ( + publisher && + (await publisher.publish(options.session, identity, async (event) => { + if (!(await permits(event))) return false; + await send(options, event); + return true; + })) + ) + sent++; + } if (!options.follow || options.signal?.aborted) return { sent, skipped, offset, partialLine: pending.length > 0 }; try { @@ -925,6 +934,7 @@ export async function syncOtel( } } finally { titles?.close(); + projects?.close(); await handle.close(); } } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 47636d2ce..53411d00e 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -227,7 +227,16 @@ export function registerTelemetryCommand(program: Command): void { "Send Codex chat titles, which may contain prompt text", false ) - .option("--titles-only", "Send only Codex titles for consented sessions", false) + .option( + "--include-session-projects", + "Send native Codex project IDs and names (may contain sensitive text)", + false + ) + .option( + "--titles-only", + "Send only Codex titles and opted-in projects for consented sessions", + false + ) .action( async (options: { config: (typeof otelOptions.syncConfigs)[number]; @@ -236,6 +245,7 @@ export function registerTelemetryCommand(program: Command): void { endpoint: string; follow: boolean; includeSessionTitles: boolean; + includeSessionProjects: boolean; titlesOnly: boolean; }) => { const { output, run } = commandOf(program); @@ -250,15 +260,19 @@ export function registerTelemetryCommand(program: Command): void { return; } if ( - (options.includeSessionTitles || options.titlesOnly) && + (options.includeSessionTitles || options.includeSessionProjects || options.titlesOnly) && options.config !== otelOptions.titleConfig ) { - output.error("Session title sync is currently available only for Codex."); + output.error("Session title and project sync is currently available only for Codex."); process.exitCode = 1; return; } await run(async (deps) => { const tool = otelOptions.usageTool(options.config); + const onProjectUnavailable = () => + output.warn( + "A native Codex session project could not be verified; telemetry sync continues." + ); const onTitleUnavailable = () => output.warn("A Codex session title could not be read; telemetry sync continues."); if (options.titlesOnly && !options.session) { @@ -269,6 +283,8 @@ export function registerTelemetryCommand(program: Command): void { endpoint: options.endpoint, token: process.env.AIDD_OTEL_TOKEN ?? "", includeSessionTitles: true, + includeSessionProjects: options.includeSessionProjects, + onProjectUnavailable, onTitleUnavailable, }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), @@ -276,7 +292,7 @@ export function registerTelemetryCommand(program: Command): void { deps.telemetry.sessionFeatureUseCase.permits(admission, at, start) ); output.info( - `Sent ${result.sent} session title updates; no metrics or feature bindings were sent.` + `Sent ${result.sent} session metadata updates; no metrics or feature bindings were sent.` ); if (!result.sent && !result.identified) output.warn( @@ -296,6 +312,8 @@ export function registerTelemetryCommand(program: Command): void { endpoint: options.endpoint, token: process.env.AIDD_OTEL_TOKEN ?? "", includeSessionTitles: options.includeSessionTitles, + includeSessionProjects: options.includeSessionProjects, + onProjectUnavailable, onTitleUnavailable, }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), @@ -321,6 +339,7 @@ export function registerTelemetryCommand(program: Command): void { { ...options, includeSessionTitles: options.includeSessionTitles || options.titlesOnly, + onProjectUnavailable, session, token: process.env.AIDD_OTEL_TOKEN ?? "", onTitleUnavailable, diff --git a/cli/tests/contexts/telemetry/otel/session-project.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-project.unit.test.ts new file mode 100644 index 000000000..bd8702158 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/session-project.unit.test.ts @@ -0,0 +1,81 @@ +import { describe, expect, it } from "vitest"; +import { readSessionProject } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.js"; + +describe("Codex native session project", () => { + it("uses the native association and rechecks it after reading the project", async () => { + const calls: Array<[string, object]> = []; + const result = await readSessionProject(async (method, params) => { + calls.push([method, params]); + return method === "thread/read" + ? { thread: { id: "s", sessionId: "s", projectId: "p" } } + : { project: { id: "p", name: "Native project" } }; + }, "s"); + expect(result).toEqual({ + available: true, + values: { "aidd.project_id": "p", "aidd.project_name": "Native project" }, + }); + expect(calls).toEqual([ + ["thread/read", { threadId: "s", includeTurns: false }], + ["project/read", { projectId: "p" }], + ["thread/read", { threadId: "s", includeTurns: false }], + ]); + }); + it("clears only an explicitly absent association", async () => { + expect( + await readSessionProject( + async () => ({ thread: { id: "s", sessionId: "s", projectId: null } }), + "s" + ) + ).toEqual({ available: true, values: {} }); + expect( + await readSessionProject( + async () => ({ thread: { id: "s", sessionId: "s", cwd: "/project" } }), + "s" + ) + ).toEqual({ available: false }); + }); + it.each([ + { id: "other", sessionId: "s", projectId: "p" }, + { id: "s", sessionId: "root", projectId: "p" }, + { id: "s", sessionId: "s", projectId: "" }, + { id: "s", sessionId: "s", projectId: "bad\nID" }, + { id: "s", sessionId: "s", projectId: "p".repeat(1025) }, + ])("rejects unverifiable thread identity or association %j", async (thread) => { + expect(await readSessionProject(async () => ({ thread }), "s")).toEqual({ available: false }); + }); + it.each([ + { id: "wrong", name: "Project" }, + { id: "p", name: " " }, + { id: "p", name: "bad\nname" }, + { id: "p" }, + ])("rejects a mismatched or unnamed project %j", async (project) => { + expect( + await readSessionProject( + async (method) => + method === "thread/read" + ? { thread: { id: "s", sessionId: "s", projectId: "p" } } + : { project }, + "s" + ) + ).toEqual({ available: false }); + }); + it("does not publish an association changed during the read", async () => { + let reads = 0; + expect( + await readSessionProject( + async (method) => + method === "thread/read" + ? { thread: { id: "s", sessionId: "s", projectId: ++reads === 1 ? "p" : "other" } } + : { project: { id: "p", name: "Project" } }, + "s" + ) + ).toEqual({ available: false }); + }); + it("keeps previous information on API failure", async () => { + expect( + await readSessionProject(async () => { + throw new Error("unsupported"); + }, "s") + ).toEqual({ available: false }); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts index 416ad0315..7a3884987 100644 --- a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts @@ -1,4 +1,13 @@ -import { access, appendFile, chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { + access, + appendFile, + chmod, + mkdir, + mkdtemp, + readFile, + rm, + writeFile, +} from "node:fs/promises"; import { createServer } from "node:http"; import { tmpdir } from "node:os"; import { delimiter, join } from "node:path"; @@ -45,9 +54,10 @@ fs.writeFileSync(process.env.AIDD_TEST_TITLE_MARKER, 'called'); readline.createInterface({ input: process.stdin }).on('line', (line) => { const message = JSON.parse(line); if (!message.id) return; - const result = message.method === 'initialize' ? {} : { + if (message.method === 'initialize') fs.writeFileSync(process.env.AIDD_TEST_TITLE_MARKER, JSON.stringify(message.params)); + const result = message.method === 'initialize' ? {} : message.method === 'project/read' ? { project: { id: 'native-project', name: 'Native project' } } : { thread: { id: message.params.threadId, sessionId: message.params.threadId, - name: 'Synthetic title', turns: [] } + name: 'Synthetic title', projectId: 'native-project', turns: [] } }; process.stdout.write(JSON.stringify({ id: message.id, result }) + '\n'); }); @@ -127,6 +137,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(received).toHaveLength(2); expect(received[1]).toContain("aidd.session_title"); expect(received[1]).toContain("Synthetic title"); + expect(await readFile(marker, "utf8")).not.toContain("experimentalApi"); expect(received[1]).not.toContain("aidd.feature"); received.length = 0; @@ -148,6 +159,24 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { ); expect(received).toHaveLength(0); + received.length = 0; + await syncOtel({ ...options, includeSessionProjects: true }, resolve, async () => true); + expect(received).toHaveLength(2); + expect(received[1]).toContain("aidd.session_project"); + expect(received[1]).toContain("aidd.project_id"); + expect(received[1]).toContain("Native project"); + expect(await readFile(marker, "utf8")).toContain('"experimentalApi":true'); + expect(received[1]).not.toContain("Synthetic title"); + expect(received[1]).not.toContain("input_token_count"); + received.length = 0; + await syncOtel( + { ...options, includeSessionProjects: true }, + resolve, + async (transfer) => !JSON.stringify(transfer.payload).includes("aidd.session_project") + ); + expect(received).toHaveLength(1); + expect(received.join()).not.toContain("aidd.project_name"); + received.length = 0; const captureFile = join(captureDir, "codex.jsonl"); await appendFile(captureFile, `${JSON.stringify(capture)}\n`.repeat(10)); @@ -214,6 +243,27 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(received.join()).not.toContain("aidd.feature"); await expect(access(join(home, "state"))).rejects.toThrow(); + received.length = 0; + const metadata = await syncAllSessionTitles( + { + ...options, + includeSessionTitles: true, + includeSessionProjects: true, + stateDir: join(home, "state"), + }, + async () => admissions, + async (admission, at) => + admission.repositoryId === "deny" || + (admission.repositoryId === "mixed" && at === deniedTime) + ? "deny" + : "allow" + ); + expect(metadata.sent).toBe(4); + expect(received.filter((body) => body.includes("aidd.session_project"))).toHaveLength(2); + expect(received.join()).not.toContain("input_token_count"); + expect(received.join()).not.toContain("aidd.feature"); + await expect(access(join(home, "state"))).rejects.toThrow(); + received.length = 0; let checks = 0; const admittedSession = admissions.get("session-1"); diff --git a/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts index e8266be29..04f52413a 100644 --- a/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts +++ b/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts @@ -3,7 +3,7 @@ import { createSessionTitlePublisher, readSessionTitle, sessionTitleTransfer, -} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/session-title.js"; +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.js"; describe("Codex session titles", () => { it("dates a title when it is observed, not before a delayed read", async () => { diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index ab09e3c38..e76184d7f 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -187,7 +187,7 @@ { "invocation": "aidd telemetry otel sync", "exitCode": 0, - "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to\n discover sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default: false)\n --include-session-titles Send Codex chat titles, which may contain prompt\n text (default: false)\n --titles-only Send only Codex titles for consented sessions\n (default: false)\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to\n discover sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default:\n false)\n --include-session-titles Send Codex chat titles, which may contain prompt\n text (default: false)\n --include-session-projects Send native Codex project IDs and names (may\n contain sensitive text) (default: false)\n --titles-only Send only Codex titles and opted-in projects for\n consented sessions (default: false)\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index b398f5474..89e2992d2 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -48,6 +48,12 @@ The collector keeps the provider's raw OTLP JSON locally. The separate dev sync - **Codex:** `conversation_starts`, `tool_result` and `response.completed` events, including available token counts. - **Both:** available event time, session, model, application version, provider email/account/organization IDs, tool name, call ID, outcome and duration. Confirmed framework feature bindings additionally carry the declared branch and ticket; an invalidation can be sent when a binding becomes ambiguous. - **Codex titles:** only with `--include-session-titles`, the free-text chat title is sent as session metadata. It may contain part of a prompt. +- **Codex native projects:** `--include-session-projects` separately opts into sending the native project ID and free-text name as `aidd.session_project`. Names can contain sensitive text. The association must come from `thread.projectId`, then an exact `project/read` result; a second thread read checks membership has not changed. No path, repository, title, parent-thread inheritance or prompt fallback is used. A confirmed `projectId: null` clears the association; missing fields, API failures and inconsistent IDs send nothing. Clone consent is checked again immediately before sending. + +Use `telemetry otel sync --config=codex --tenant=<slug> --endpoint=<loopback-origin> --titles-only --include-session-projects` for one metadata-only pass over consented captured sessions (both titles and projects are opted in). It sends no token metrics or feature bindings and writes no capture checkpoint. `--follow --include-session-projects` checks sessions loaded by the regular follower; it does not discover every historical chat. Native projects are separate from Git repositories. Claude Code has no universal native project field in this bridge; OpenCode project export is not integrated. + +Codex's project API is experimental. In the local 0.166.0 probe on 2026-11-05, `project/read` returned an exact ID and nonempty native name, but all ten sampled thread reads returned `projectId: null`. Desktop UI grouping does not establish app-server membership. Named project export is therefore schema-tested, not demonstrated end to end on those local sessions. [Official project protocol](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/app-server-protocol/src/protocol/v2/project.rs) (read 2026-11-05). + - **Claude series:** a SHA-256 fingerprint of the source resource attributes, instrumentation scope and all point attributes helps detect different metric series that share a legacy key. The attributes used to compute it are not sent as additional plaintext, but the fingerprint can reveal equality or allow guesses about low-entropy values. Existing points lack it and cannot be certified retroactively. [OTLP metric data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) (read 2026-11-05). The sync excludes fields for prompt and response text, tool arguments and output, raw API bodies, metric exemplars and traces. It filters attribute keys, not secrets or free text inside allowed values. OpenCode is not forwarded by this sync. These are forwarding rules, not guarantees about what the raw collector receives. diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 160523478..6beab39ac 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -90,3 +90,5 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. Configure an exporter with `aidd telemetry otel init --config=<claude|codex|opencode> --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. OpenCode v1/v2 capture support does not yet imply OpenCode-to-Gouvernail sync. Start OpenCode v1 with `aidd telemetry otel run --config=opencode` to supply its endpoint and token; pass OpenCode arguments after `--`. For OpenCode v2, restart its service after initialization. + +Native Codex project metadata requires the separate `--include-session-projects` opt-in. It transmits the verified native project ID and name, session/provider identity and observation time; names may contain sensitive text. No path-based project inference is made. See [the observation contract](OBSERVABILITY.md) for consent, absence handling and the metadata-only command. From f7b3e1020da8aa438e5403ce802e310d5a341951 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:32:04 +0800 Subject: [PATCH 15/56] test(telemetry): verify session metadata end to end --- cli/scripts/smoke-codex-project.mjs | 240 +++++++++++++++++ .../e2e/telemetry-otel-collector.e2e.test.ts | 129 +++++++++ .../e2e/telemetry-otel-metadata.e2e.test.ts | 254 ++++++++++++++++++ plugins/aidd-telemetry/OBSERVABILITY.md | 4 +- 4 files changed, 626 insertions(+), 1 deletion(-) create mode 100644 cli/scripts/smoke-codex-project.mjs create mode 100644 cli/tests/e2e/telemetry-otel-collector.e2e.test.ts create mode 100644 cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts diff --git a/cli/scripts/smoke-codex-project.mjs b/cli/scripts/smoke-codex-project.mjs new file mode 100644 index 000000000..af0bcca47 --- /dev/null +++ b/cli/scripts/smoke-codex-project.mjs @@ -0,0 +1,240 @@ +import assert from "node:assert/strict"; +import { spawn, spawnSync } from "node:child_process"; +import { once } from "node:events"; +import { access, mkdir, mkdtemp, realpath, rm } from "node:fs/promises"; +import { createServer } from "node:net"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { createInterface } from "node:readline"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { build } from "tsup"; + +// Opt-in native smoke: no provider credentials, prompts, turns, or network access. +// Unlike the deterministic mocked tests, this checks the installed Codex protocol. +const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const sandbox = "/usr/bin/sandbox-exec"; +const policy = "(version 1) (allow default) (deny network*)"; + +function executable(name) { + const result = spawnSync("which", [name], { encoding: "utf8" }); + if (result.status !== 0 || !result.stdout.trim()) throw new Error(`${name} is required`); + return realpath(result.stdout.trim().split("\n")[0]); +} + +async function run(command, args, env) { + const child = spawn(command, args, { env, stdio: ["ignore", "ignore", "pipe"] }); + let stderr = ""; + child.stderr.on("data", (data) => { + stderr += data; + }); + const [code] = await once(child, "exit"); + if (code !== 0) throw new Error(`${command} failed: ${stderr.trim()}`); +} + +async function verifyNetworkBlocked(env) { + const server = createServer((socket) => socket.end()); + server.listen(0, "127.0.0.1"); + await once(server, "listening"); + try { + const { port } = server.address(); + const probe = + `const s = require('node:net').connect(${port}, '127.0.0.1');\n` + + "s.on('connect', () => process.exit(2));\n" + + "s.on('error', e => process.exit(['EACCES', 'EPERM'].includes(e.code) ? 0 : 3));\n" + + "setTimeout(() => process.exit(4), 3000);"; + await run(sandbox, ["-p", policy, process.execPath, "-e", probe], env); + } finally { + await new Promise((done) => server.close(done)); + } +} + +function rpc(child) { + let nextId = 0; + let stderr = ""; + let failure; + const pending = new Map(); + child.stderr.on("data", (data) => { + stderr += data; + }); + const lines = createInterface({ input: child.stdout }); + lines.on("line", (line) => { + let response; + try { + response = JSON.parse(line); + } catch { + return; + } + const callback = pending.get(response.id); + if (!callback) return; + pending.delete(response.id); + if (response.error) callback.reject(new Error(response.error.message)); + else callback.resolve(response.result); + }); + const fail = (error) => { + failure ??= error; + for (const callback of pending.values()) callback.reject(failure); + pending.clear(); + }; + child.on("error", fail); + child.stdin.on("error", fail); + child.on("exit", () => fail(new Error(`Codex app-server exited: ${stderr.trim()}`))); + return { + request(method, params) { + if (failure) return Promise.reject(failure); + const id = ++nextId; + return new Promise((resolveRequest, reject) => { + const timer = setTimeout(() => { + pending.delete(id); + reject(new Error(`Codex request timed out: ${method}`)); + }, 10_000); + pending.set(id, { + resolve(value) { + clearTimeout(timer); + resolveRequest(value); + }, + reject(error) { + clearTimeout(timer); + reject(error); + }, + }); + child.stdin.write(`${JSON.stringify({ id, method, params })}\n`); + }); + }, + notify(method) { + child.stdin.write(`${JSON.stringify({ method, params: {} })}\n`); + }, + close() { + fail(new Error("Codex app-server closed")); + lines.close(); + }, + }; +} + +async function stop(child) { + if (!child?.pid || child.exitCode !== null || child.signalCode !== null) return; + const exited = once(child, "exit").then(() => true); + async function wait() { + let timer; + try { + return await Promise.race([ + exited, + new Promise((resolveWait) => { + timer = setTimeout(() => resolveWait(false), 2_000); + }), + ]); + } finally { + clearTimeout(timer); + } + } + child.kill(); + if (await wait()) return; + child.kill("SIGKILL"); + if (await wait()) return; + child.stdin.destroy(); + child.stdout.destroy(); + child.stderr.destroy(); + child.unref(); + throw new Error("Codex app-server did not exit after SIGKILL"); +} + +async function main() { + if (process.platform !== "darwin") { + throw new Error( + "This native smoke requires macOS sandbox-exec; no unsandboxed fallback is allowed" + ); + } + await access(sandbox); + const codex = await executable("codex"); + const temp = await mkdtemp(join(tmpdir(), "aidd-codex-native-")); + let child; + try { + const home = join(temp, "home"); + const work = join(temp, "work"); + const codexHome = join(home, ".codex"); + await Promise.all([mkdir(codexHome, { recursive: true }), mkdir(work)]); + const env = { + PATH: `${dirname(process.execPath)}:/usr/bin:/bin:/usr/sbin:/sbin`, + HOME: home, + CODEX_HOME: codexHome, + XDG_CONFIG_HOME: join(home, ".config"), + XDG_CACHE_HOME: join(home, ".cache"), + XDG_DATA_HOME: join(home, ".local/share"), + TMPDIR: temp, + }; + await verifyNetworkBlocked(env); + + // Bundle the production readers into the temporary directory; no test doubles. + const metadata = join( + packageRoot, + "src/contexts/telemetry/infrastructure/otel/actions/metadata" + ); + const outDir = join(temp, "readers"); + await build({ + entry: { + project: join(metadata, "session-project.ts"), + title: join(metadata, "session-title.ts"), + }, + outDir, + format: ["esm"], + outExtension: () => ({ js: ".mjs" }), + platform: "node", + target: "node22", + config: false, + clean: false, + dts: false, + silent: true, + splitting: false, + noExternal: [/.*/], + }); + const { readSessionProject } = await import(pathToFileURL(join(outDir, "project.mjs"))); + const { readSessionTitle } = await import(pathToFileURL(join(outDir, "title.mjs"))); + + child = spawn(sandbox, ["-p", policy, codex, "app-server", "--listen", "stdio://"], { + env, + stdio: ["pipe", "pipe", "pipe"], + }); + const server = rpc(child); + try { + await server.request("initialize", { + clientInfo: { name: "aidd_native_smoke", version: "1" }, + capabilities: { experimentalApi: true }, + }); + server.notify("initialized"); + const { project } = await server.request("project/create", { + name: "Synthetic project", + roots: [{ path: work }], + idempotencyKey: "aidd-native-smoke", + }); + const { thread } = await server.request("thread/start", { + projectId: project.id, + cwd: work, + model: "gpt-5", + approvalPolicy: "never", + }); + await server.request("thread/name/set", { threadId: thread.id, name: "Synthetic title" }); + const request = (method, params) => server.request(method, params); + assert.deepEqual(await readSessionTitle(request, thread.id), { + available: true, + title: "Synthetic title", + }); + assert.deepEqual(await readSessionProject(request, thread.id), { + available: true, + values: { "aidd.project_id": project.id, "aidd.project_name": "Synthetic project" }, + }); + console.log("PASS: native Codex session title and project readers (network blocked)"); + } finally { + server.close(); + } + } finally { + try { + await stop(child); + } finally { + await rm(temp, { recursive: true, force: true }); + } + } +} + +main().catch((error) => { + console.error(`FAIL: ${error.message}`); + process.exitCode = 1; +}); diff --git a/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts b/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts new file mode 100644 index 000000000..c1d306a66 --- /dev/null +++ b/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts @@ -0,0 +1,129 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; +import { createInterface } from "node:readline"; +import { describe, expect, it } from "vitest"; +import { transformCapturedLine } from "../../src/contexts/telemetry/infrastructure/otel/observations.js"; +import { cliPath, createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); + +// These are synthetic OTLP records, not a claim that every provider exports every field. +describe.skipIf(process.platform === "win32")("OTLP collector on the built CLI", () => { + it("captures isolated authenticated provider records and verifies their normalized fields", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("otel-collector"); + const bin = join(fakeHome, "bin"); + await mkdir(bin); + await writeFile(join(bin, "opencode"), `#!${process.execPath}\nconsole.log('2.2.5');\n`, { + mode: 0o700, + }); + const env = sandboxedEnv(fakeHome); + const path = [bin, env.PATH].join(delimiter); + const child = spawn(process.execPath, [cliPath(), "telemetry", "otel", "collect", "--port=0"], { + cwd: projectDir, + env: sandboxedEnv(fakeHome, { PATH: path }), + stdio: ["ignore", "pipe", "pipe"], + }); + const exited = once(child, "exit"); + const outputLines = createInterface({ input: child.stdout }); + try { + const output = await Promise.race([ + once(outputLines, "line", { signal: AbortSignal.timeout(5000) }).then(([line]) => + String(line) + ), + exited.then(() => { + throw new Error("Collector exited before listening."); + }), + ]); + const port = /127\.0\.0\.1:(\d+)/.exec(output)?.[1]; + expect(port).toBeDefined(); + const directory = join(fakeHome, ".local/share/aidd-otel-observation"); + const token = (await readFile(join(directory, "collector-token"), "utf8")).trim(); + const configEnv = { + PATH: path, + OPENCODE_CONFIG_DIR: "", + OPENCODE_CONFIG: "", + OPENCODE_CONFIG_CONTENT: "", + }; + const tools = [ + { tool: "claude" as const, sessionKey: "session.id", inputKey: "input_tokens" }, + { tool: "codex" as const, sessionKey: "conversation.id", inputKey: "input_token_count" }, + { + tool: "opencode" as const, + sessionKey: "session.id", + inputKey: "gen_ai.usage.input_tokens", + }, + ]; + for (const { tool, sessionKey, inputKey } of tools) { + const payload = { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: "1700000000000000000", + attributes: [ + attr("event.name", "synthetic"), + attr(sessionKey, `${tool}-session`), + attr(inputKey, "17"), + ], + }, + ], + }, + ], + }, + ], + }; + const url = `http://127.0.0.1:${port}/${tool}/v1/logs`; + const body = JSON.stringify(payload); + expect((await fetch(url, { method: "POST", body })).status).toBe(403); + expect( + ( + await fetch(url, { + method: "POST", + body, + headers: { authorization: `Bearer ${token}` }, + }) + ).status + ).toBe(200); + const capture = (await readFile(join(directory, `${tool}.jsonl`), "utf8")).trim(); + const observations = transformCapturedLine(tool, capture); + expect(observations).toHaveLength(1); + expect(observations[0]).toMatchObject({ + tool, + sessionId: `${tool}-session`, + tokens: { input: 17 }, + }); + expect(observations[0]?.tokens?.output).toBeUndefined(); + const init = await runCli( + ["telemetry", "otel", "init", `--config=${tool}`, "--endpoint=dev"], + projectDir, + fakeHome, + { env: configEnv } + ); + expect(init.exitCode, init.stderr + init.stdout).toBe(0); + const verified = await runCli( + ["telemetry", "otel", "verify", `--config=${tool}`, "--endpoint=dev"], + projectDir, + fakeHome, + { env: configEnv } + ); + expect(verified.exitCode, verified.stderr + verified.stdout).toBe(0); + expect(verified.stdout).toContain("1 captured lines"); + expect(verified.stdout).toContain("1 normalized observations"); + } + for (const { tool } of tools) { + const lines = (await readFile(join(directory, `${tool}.jsonl`), "utf8")).trim().split("\n"); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain(`${tool}-session`); + } + } finally { + outputLines.close(); + child.kill("SIGTERM"); + await exited; + await cleanup(); + } + }, 20_000); +}); diff --git a/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts b/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts new file mode 100644 index 000000000..4254ebb04 --- /dev/null +++ b/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts @@ -0,0 +1,254 @@ +import { appendFile, mkdir, rm, writeFile } from "node:fs/promises"; +import { createServer } from "node:http"; +import { delimiter, join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { initRepository, sandboxGitEnv } from "../helpers/git-sandbox.js"; +import { createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; + +interface LogEnvelope { + resourceLogs: { + scopeLogs: { + logRecords: { attributes: { key: string; value: { stringValue: string } }[] }[]; + }[]; + }[]; +} + +function logAttributes(body: string): Record<string, string> { + const payload = JSON.parse(body) as LogEnvelope; + return Object.fromEntries( + payload.resourceLogs.flatMap((resource) => + resource.scopeLogs.flatMap((scope) => + scope.logRecords.flatMap((record) => + record.attributes.map(({ key, value }) => [key, value.stringValue]) + ) + ) + ) + ); +} + +/** The AIDD binary and consent ledger are real. The Codex protocol and receiver are controlled + * stand-ins: this proves the integration contract, not what an installed Codex emits. */ +describe.skipIf(process.platform === "win32")("Codex metadata through the CLI", () => { + it("sends consented session metadata and rejects an inconsistent native project", async () => { + const { projectDir, fakeHome, cleanup } = await createTestEnv("otel-metadata"); + const telemetry = join(fakeHome, "telemetry"); + const bin = join(fakeHome, "bin"); + const session = "00000000-0000-4000-8000-000000000123"; + const received: { url: string; auth: string | undefined; body: string }[] = []; + const server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push({ url: request.url ?? "", auth: request.headers.authorization, body }); + response.writeHead(200, { "content-type": "application/json" }).end("{}"); + }); + }); + try { + initRepository(projectDir, sandboxGitEnv(fakeHome), { + remote: "git@github.com:example/project.git", + }); + await mkdir(bin); + await writeFile( + join(bin, "codex"), + `#!${process.execPath} +const readline = require('node:readline'); +readline.createInterface({ input: process.stdin }).on('line', (line) => { + const message = JSON.parse(line); + if (!message.id) return; + if (!['initialize', 'project/read', 'thread/read'].includes(message.method)) { + process.stdout.write(JSON.stringify({ id: message.id, error: { code: -32601, message: 'Unknown method' } }) + '\\n'); + return; + } + const result = message.method === 'initialize' ? {} : message.method === 'project/read' + ? { project: { id: process.env.AIDD_TEST_BAD_PROJECT ? 'other-project' : 'project-1', name: 'Example project' } } + : { thread: { id: message.params.threadId, sessionId: message.params.threadId, name: 'Example session', projectId: 'project-1', turns: [] } }; + process.stdout.write(JSON.stringify({ id: message.id, result }) + '\\n'); +}); +`, + { mode: 0o700 } + ); + const env = { + AIDD_TELEMETRY_DIR: telemetry, + AIDD_TELEMETRY: "", + AIDD_OTEL_TOKEN: "test-token", + AIDD_TEST_BAD_PROJECT: "", + PATH: [bin, sandboxedEnv(fakeHome).PATH].join(delimiter), + }; + const cli = (args: string[], extra: Record<string, string> = {}) => + runCli(["telemetry", ...args], projectDir, fakeHome, { env: { ...env, ...extra } }); + await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Receiver did not start"); + const sync = (extra: Record<string, string> = {}, titlesOnly = true) => + cli( + [ + "otel", + "sync", + "--config=codex", + `--session=${session}`, + "--tenant=test", + `--endpoint=http://127.0.0.1:${address.port}`, + ...(titlesOnly ? ["--titles-only"] : ["--include-session-titles"]), + "--include-session-projects", + ], + extra + ); + const transcriptDir = join(fakeHome, ".codex/sessions"); + await mkdir(transcriptDir, { recursive: true }); + const captureDir = join(fakeHome, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); + const capture = (at: string, input: string) => ({ + signal: "logs", + payload: { + resourceLogs: [ + { + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: (BigInt(Date.parse(at)) * 1_000_000n).toString(), + attributes: [ + attr("conversation.id", session), + attr("user.email", "person@example.test"), + attr("event.name", "codex.sse_event"), + attr("event.kind", "response.completed"), + attr("model", "model-test"), + attr("input_token_count", input), + attr("output_token_count", "2"), + attr("cached_token_count", "4"), + attr("prompt", "private prompt"), + ], + }, + ], + }, + ], + }, + ], + }, + }); + const writeSession = async (at: string) => { + const transcript = [ + { + timestamp: at, + type: "session_meta", + payload: { id: session, cwd: projectDir, cli_version: "test", source: "cli" }, + }, + { + timestamp: at, + type: "turn_context", + payload: { cwd: projectDir, model: "model-test" }, + }, + { + timestamp: at, + type: "token_usage_record", + payload: { + response_id: "response-1", + session_id: session, + thread_id: session, + usage: { + input_tokens: 10, + cached_input_tokens: 4, + cache_write_input_tokens: 0, + output_tokens: 2, + reasoning_output_tokens: 1, + }, + }, + }, + ]; + await writeFile( + join(transcriptDir, `rollout-test-${session}.jsonl`), + `${transcript.map((line) => JSON.stringify(line)).join("\n")}\n` + ); + await writeFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture(at, "10"))}\n`, { + mode: 0o600, + }); + }; + + await writeSession(new Date().toISOString()); + const withoutConsent = await sync(); + expect(withoutConsent.exitCode).not.toBe(0); + expect(received).toHaveLength(0); + + await rm(join(transcriptDir, `rollout-test-${session}.jsonl`)); + const on = await cli(["on", "--yes", "--anonymous"]); + expect(on.exitCode, on.stderr).toBe(0); + await writeSession(new Date().toISOString()); + + const sent = await sync(); + expect(sent.exitCode, sent.stderr + sent.stdout).toBe(0); + expect(received).toHaveLength(2); + expect( + received.every( + (request) => + request.url === "/api/kpi/otel/codex/test/v1/logs" && + request.auth === "Bearer test-token" + ) + ).toBe(true); + const attributes = received.map(({ body }) => logAttributes(body)); + expect(attributes).toContainEqual( + expect.objectContaining({ + "event.name": "aidd.session_title", + "conversation.id": session, + "aidd.session_title": "Example session", + }) + ); + expect(attributes).toContainEqual( + expect.objectContaining({ + "event.name": "aidd.session_project", + "conversation.id": session, + "aidd.project_id": "project-1", + "aidd.project_name": "Example project", + }) + ); + expect(JSON.stringify(received)).not.toContain("private prompt"); + expect(JSON.stringify(received)).not.toContain("input_token_count"); + + received.length = 0; + const full = await sync({}, false); + expect(full.exitCode, full.stderr + full.stdout).toBe(0); + expect(received).toHaveLength(3); + expect(received.map(({ body }) => logAttributes(body))).toContainEqual( + expect.objectContaining({ + "event.name": "codex.sse_event", + "event.kind": "response.completed", + "conversation.id": session, + model: "model-test", + input_token_count: "10", + output_token_count: "2", + cached_token_count: "4", + }) + ); + expect(JSON.stringify(received)).not.toContain("private prompt"); + + received.length = 0; + const inconsistent = await sync({ AIDD_TEST_BAD_PROJECT: "1" }); + expect(inconsistent.exitCode, inconsistent.stderr).toBe(0); + expect(received).toHaveLength(1); + expect(received[0].body).toContain("aidd.session_title"); + expect(received[0].body).not.toContain("project-1"); + + const off = await cli(["off"]); + expect(off.exitCode, off.stderr).toBe(0); + received.length = 0; + const revoked = await sync(); + expect(revoked.exitCode, revoked.stderr).toBe(0); + expect(received).toHaveLength(0); + + // The historical event remains permitted; the event after opt-out is not sent. + await appendFile( + join(captureDir, "codex.jsonl"), + `${JSON.stringify(capture(new Date().toISOString(), "999"))}\n` + ); + const afterOff = await sync({}, false); + expect(afterOff.exitCode, afterOff.stderr + afterOff.stdout).toBe(0); + expect(received).toHaveLength(1); + expect(logAttributes(received[0].body).input_token_count).toBe("10"); + } finally { + await new Promise<void>((resolve) => server.close(() => resolve())); + await cleanup(); + } + }); +}); diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 89e2992d2..b936c5ea8 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -52,7 +52,9 @@ The collector keeps the provider's raw OTLP JSON locally. The separate dev sync Use `telemetry otel sync --config=codex --tenant=<slug> --endpoint=<loopback-origin> --titles-only --include-session-projects` for one metadata-only pass over consented captured sessions (both titles and projects are opted in). It sends no token metrics or feature bindings and writes no capture checkpoint. `--follow --include-session-projects` checks sessions loaded by the regular follower; it does not discover every historical chat. Native projects are separate from Git repositories. Claude Code has no universal native project field in this bridge; OpenCode project export is not integrated. -Codex's project API is experimental. In the local 0.166.0 probe on 2026-11-05, `project/read` returned an exact ID and nonempty native name, but all ten sampled thread reads returned `projectId: null`. Desktop UI grouping does not establish app-server membership. Named project export is therefore schema-tested, not demonstrated end to end on those local sessions. [Official project protocol](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/app-server-protocol/src/protocol/v2/project.rs) (read 2026-11-05). +Codex's project API is experimental. In the local 0.166.0 probe on 2026-11-05, `project/read` returned an exact ID and nonempty native name, but all ten sampled thread reads returned `projectId: null`. A separate Desktop tool listed five chats with explicit project IDs matching its project list; app-server returned `projectId: null` for those same five chats. An independent CLI's access to that Desktop source is not established. Named project export is tested against a simulated app-server, not demonstrated end to end on those local sessions. [Official project protocol](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/app-server-protocol/src/protocol/v2/project.rs) (read 2026-11-05). + +On macOS, `cd cli && node scripts/smoke-codex-project.mjs` checks the installed Codex app-server and the framework's metadata readers with an isolated profile and blocked network. It creates an explicitly linked test session; it does not verify existing Desktop groupings. - **Claude series:** a SHA-256 fingerprint of the source resource attributes, instrumentation scope and all point attributes helps detect different metric series that share a legacy key. The attributes used to compute it are not sent as additional plaintext, but the fingerprint can reveal equality or allow guesses about low-entropy values. Existing points lack it and cannot be certified retroactively. [OTLP metric data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) (read 2026-11-05). From 28b9b62323523db835258482ed2dff48e8aceca8 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:34:46 +0800 Subject: [PATCH 16/56] test(telemetry): register native Codex smoke test --- cli/package.json | 1 + 1 file changed, 1 insertion(+) diff --git a/cli/package.json b/cli/package.json index fb2736573..db9a34146 100644 --- a/cli/package.json +++ b/cli/package.json @@ -54,6 +54,7 @@ "smoke": "pnpm build && bash scripts/smoke-tools.sh", "smoke:full": "pnpm build && SMOKE_REMOTE=1 bash scripts/smoke-tools.sh", "smoke:real": "pnpm build && bash scripts/smoke-real.sh", + "smoke:codex-project": "node scripts/smoke-codex-project.mjs", "smoke:collision": "pnpm build && bash scripts/smoke-collision.sh", "typecheck": "tsc --noEmit", "lint": "biome check .", From 5bd795062ef3a382eb7fc3724decad88973a4f3b Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:36:15 +0800 Subject: [PATCH 17/56] test(telemetry): align sync coverage with platform support --- .../otel/session-title-sync.integration.test.ts | 13 ++++++------- .../otel/sync-otel-attributes.integration.test.ts | 2 +- .../otel/sync-otel-integrity.integration.test.ts | 5 +++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts index 7a3884987..396b1fa67 100644 --- a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts @@ -1,9 +1,9 @@ import { - access, appendFile, chmod, mkdir, mkdtemp, + readdir, readFile, rm, writeFile, @@ -110,7 +110,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { const resolve = async () => ({ state: "unbound" as const }); await syncOtel(options, resolve, async () => true); - await expect(access(marker)).rejects.toThrow(); + await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); expect(received).toHaveLength(1); received.length = 0; @@ -119,7 +119,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { resolve, async (transfer) => Object.keys(transfer.payload).length > 0 ); - await expect(access(marker)).rejects.toThrow(); + await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); expect(received).toHaveLength(1); received.length = 0; @@ -128,12 +128,11 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { resolve, async () => false ); - await expect(access(marker)).rejects.toThrow(); + await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); expect(received).toHaveLength(0); received.length = 0; await syncOtel({ ...options, includeSessionTitles: true }, resolve, async () => true); - await access(marker); expect(received).toHaveLength(2); expect(received[1]).toContain("aidd.session_title"); expect(received[1]).toContain("Synthetic title"); @@ -241,7 +240,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(received.join()).not.toContain("session-3"); expect(received.join()).not.toContain("input_token_count"); expect(received.join()).not.toContain("aidd.feature"); - await expect(access(join(home, "state"))).rejects.toThrow(); + await expect(readdir(join(home, "state"))).rejects.toMatchObject({ code: "ENOENT" }); received.length = 0; const metadata = await syncAllSessionTitles( @@ -262,7 +261,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(received.filter((body) => body.includes("aidd.session_project"))).toHaveLength(2); expect(received.join()).not.toContain("input_token_count"); expect(received.join()).not.toContain("aidd.feature"); - await expect(access(join(home, "state"))).rejects.toThrow(); + await expect(readdir(join(home, "state"))).rejects.toMatchObject({ code: "ENOENT" }); received.length = 0; let checks = 0; diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts index 717ee0445..359434a7c 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -314,7 +314,7 @@ afterEach(async () => { await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); }); -test.each([ +test.skipIf(process.platform === "win32").each([ { values: [text("input_token_count", "100"), text("input_token_count", "5")], error: "Conflicting duplicate OTLP attributes.", diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts index 9c11ecacb..05bb1978a 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -10,6 +10,7 @@ import { } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; +const posixTest = test.skipIf(process.platform === "win32"); const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); const options = { config: "codex" as const, @@ -233,7 +234,7 @@ test("preserves exact integer strings for signed and unsigned metric values", () expect(payload).toContain(`"${value}"`); }); -test.each([ +posixTest.each([ claudePoint({ asInt: 1785044858922210000 }), claudePoint({ asDouble: 5 }), claudePoint({ flags: 1 }), @@ -425,7 +426,7 @@ test("keeps a valid offset and leap date as the original event time", () => { expect(JSON.stringify(transfers[0].payload)).toContain(original); }); -test("does not export an observation-only timestamp as event activity", async () => { +posixTest("does not export an observation-only timestamp as event activity", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-missing-event-time-")); dirs.push(home); const capture = join(home, ".local/share/aidd-otel-observation"); From 70a4d2fadb3ae0826faa6b01708e33ab0cd7dfce Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:40:21 +0800 Subject: [PATCH 18/56] fix(ci): isolate mutation shard reports --- cli/scripts/mutation-shards.d.mts | 19 ++++---- cli/scripts/mutation-shards.mjs | 24 ++++++---- cli/scripts/run-mutation.mjs | 2 +- .../architecture/mutation-shards.arch.test.ts | 48 ++++++++++++++++++- 4 files changed, 74 insertions(+), 19 deletions(-) diff --git a/cli/scripts/mutation-shards.d.mts b/cli/scripts/mutation-shards.d.mts index f36d2e72b..27b1519a6 100644 --- a/cli/scripts/mutation-shards.d.mts +++ b/cli/scripts/mutation-shards.d.mts @@ -21,18 +21,19 @@ export function shardFiles( of: number, weightOf?: WeightOf ): string[]; -export function mergeReports(reports: readonly MutationReport[]): Required<MutationReport>; +interface MutationShard { + readonly manifest: { + readonly shard: number; + readonly of: number; + readonly files: readonly string[]; + }; + readonly report: MutationReport; +} +export function mergeReports(shards: readonly MutationShard[]): Required<MutationReport>; export function aggregateVerdict(input: { readonly mutate: Mutate; readonly files: readonly string[]; readonly of: number; - readonly shards: readonly { - readonly manifest: { - readonly shard: number; - readonly of: number; - readonly files: readonly string[]; - }; - readonly report: MutationReport; - }[]; + readonly shards: readonly MutationShard[]; readonly weightOf?: WeightOf; }): string | null; diff --git a/cli/scripts/mutation-shards.mjs b/cli/scripts/mutation-shards.mjs index 749208029..337258a57 100644 --- a/cli/scripts/mutation-shards.mjs +++ b/cli/scripts/mutation-shards.mjs @@ -72,9 +72,16 @@ export function shardFiles(mutate, files, shard, of, weightOf = linesOf) { return partition(mutate, files, of, weightOf)[shard - 1]; } -/** The shards' reports as one: each file is in exactly one shard, so nothing overlaps. */ -export function mergeReports(reports) { - return { files: Object.assign({}, ...reports.map((report) => report.files ?? {})) }; +/** Score only the files assigned to each shard, never stale entries from an incremental cache. */ +export function mergeReports(shards) { + const files = {}; + for (const shard of shards) { + for (const path of shard.manifest.files) { + const report = shard.report.files?.[path]; + if (report !== undefined) files[path] = report; + } + } + return { files }; } /** Why a sharded scope cannot be scored, or null. Every shard must be there, each must have @@ -82,8 +89,10 @@ export function mergeReports(reports) { * mutated: a missing or drifted shard would lift the score by leaving mutants out. */ export function aggregateVerdict({ mutate, files, of, shards, weightOf = linesOf }) { for (let shard = 1; shard <= of; shard += 1) { - const found = shards.find((candidate) => candidate.manifest.shard === shard); - if (found === undefined) return `shard ${shard} of ${of} is missing`; + const matches = shards.filter((candidate) => candidate.manifest.shard === shard); + if (matches.length === 0) return `shard ${shard} of ${of} is missing`; + if (matches.length > 1) return `shard ${shard} of ${of} was supplied more than once`; + const found = matches[0]; if (found.manifest.of !== of) return `shard ${shard} was cut in ${found.manifest.of}, not ${of}`; const expected = shardFiles(mutate, files, shard, of, weightOf); @@ -92,8 +101,7 @@ export function aggregateVerdict({ mutate, files, of, shards, weightOf = linesOf return `shard ${shard} of ${of} mutated other files than the partition gives it`; } } - const mutants = Object.values(mergeReports(shards.map((s) => s.report)).files).flatMap( - (file) => file.mutants - ).length; + if (shards.length !== of) return `unexpected shard outside 1 to ${of}`; + const mutants = Object.values(mergeReports(shards).files).flatMap((file) => file.mutants).length; return mutants === 0 ? "the shards hold no mutant" : null; } diff --git a/cli/scripts/run-mutation.mjs b/cli/scripts/run-mutation.mjs index ccb9496c9..25d04bb89 100644 --- a/cli/scripts/run-mutation.mjs +++ b/cli/scripts/run-mutation.mjs @@ -276,7 +276,7 @@ function mainAggregate(scope, scopes, from) { console.error(`${scope}: ${problem}`); process.exit(1); } - const merged = mergeReports(shards.map((shard) => shard.report)); + const merged = mergeReports(shards); const score = scoreOf(merged); console.log( `\nReport: ${scope} over ${shards.length} shards (score ${score.toFixed(1)}, floor ${declared.break})` diff --git a/cli/tests/architecture/mutation-shards.arch.test.ts b/cli/tests/architecture/mutation-shards.arch.test.ts index da803dc0f..cf66a7115 100644 --- a/cli/tests/architecture/mutation-shards.arch.test.ts +++ b/cli/tests/architecture/mutation-shards.arch.test.ts @@ -117,11 +117,57 @@ describe("scoring a sharded scope once, over every shard", () => { }; it("merges the shards into the report one run would have written", () => { - const merged = mergeReports([shard(1).report, shard(2).report]); + const merged = mergeReports([shard(1), shard(2)]); expect(Object.keys(merged.files).sort()).toEqual([...files].sort()); expect(scoreOf(merged)).toBe(50); }); + it("scores only each shard's own files even when another shard carries stale results", () => { + const first = shard(1); + const second = shard(2); + const otherFile = first.manifest.files[0]; + const stale = { + ...second, + report: { + files: { + ...second.report.files, + [otherFile]: { mutants: [mutant("Killed"), mutant("Killed")] }, + "src/other/stale.ts": { mutants: [mutant("Killed")] }, + }, + }, + }; + const merged = mergeReports([first, stale]); + expect(Object.keys(merged.files).sort()).toEqual([...files].sort()); + expect(merged.files[otherFile]?.mutants.map((item) => item.status)).toEqual([ + "Killed", + "Survived", + ]); + expect(scoreOf(merged)).toBe(50); + expect(mergeReports([stale, first])).toEqual(merged); + }); + + it("does not count stale files when determining whether shards hold mutants", () => { + const empty = [1, 2].map((i) => { + const current = shard(i); + const foreign = shard(i === 1 ? 2 : 1).manifest.files[0]; + return { ...current, report: report({ [foreign]: ["Killed"] }) }; + }); + expect(aggregateVerdict({ mutate, files, of, weightOf, shards: empty })).toMatch(/no mutant/); + }); + + it("fails when the same shard is supplied more than once", () => { + expect( + aggregateVerdict({ mutate, files, of, weightOf, shards: [shard(1), shard(2), shard(2)] }) + ).toMatch(/shard 2 of 2.*more than once/); + }); + + it("fails when an unexpected shard is supplied", () => { + const unexpected = { ...shard(1), manifest: { ...shard(1).manifest, shard: 3 } }; + expect( + aggregateVerdict({ mutate, files, of, weightOf, shards: [shard(1), shard(2), unexpected] }) + ).toMatch(/unexpected shard/); + }); + it("passes when every shard is there and matches the partition", () => { expect( aggregateVerdict({ mutate, files, of, weightOf, shards: [shard(1), shard(2)] }) From 0a15aeb40dfd03c743a5034be7e7bdbfe3353609 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:44:27 +0800 Subject: [PATCH 19/56] test(telemetry): verify session consent intervals --- .../report/session-consent.unit.test.ts | 168 ++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts diff --git a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts new file mode 100644 index 000000000..d51e865fe --- /dev/null +++ b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts @@ -0,0 +1,168 @@ +import { describe, expect, it } from "vitest"; +import { DirectoryResolver } from "../../../../../src/contexts/telemetry/application/directory-resolver.js"; +import { SessionConsent } from "../../../../../src/contexts/telemetry/application/report/session-consent.js"; +import { resolutionKey } from "../../../../../src/contexts/telemetry/domain/repository-resolution.js"; +import { consentValue } from "../../../../../src/contexts/telemetry/domain/telemetry-consent.js"; +import { + cloneOf, + FakeConsents, + FakeLocator, + grantedBy, + InMemoryConsentHistory, + InMemoryLedger, + InMemoryResolutions, +} from "../../../../helpers/ports/in-memory-telemetry.js"; + +const cwd = "/work/repo"; +const repo = "repository-one"; +const at = "2026-10-07T10:30:00.000Z"; +const start = "2026-10-07T10:00:00.000Z"; +const admission = { feature: { state: "unbound" as const }, cwd, repositoryId: repo }; + +function setup(history = new InMemoryConsentHistory()) { + const ledger = new InMemoryLedger(); + const locator = new FakeLocator(); + const source = new FakeConsents(); + const resolutions = new InMemoryResolutions(); + const clone = cloneOf(`${cwd}/.git`); + locator.directories.set(cwd, { + status: "repository", + root: cwd, + mainRoot: cwd, + clone, + remote: null, + rootCommit: repo, + }); + source.cloneSays(clone, grantedBy(clone)); + history.consented(clone, "2026-10-07T09:00:00.000Z"); + const directories = new DirectoryResolver(locator, source, resolutions, history, { + caseInsensitiveFileSystem: false, + now: () => new Date("2026-10-07T12:00:00.000Z"), + }); + return { + checker: new SessionConsent(ledger, directories, history), + ledger, + locator, + source, + resolutions, + history, + clone, + }; +} + +describe("consent for an observed session", () => { + it("allows an instant or whole interval only inside one clone's consent", async () => { + const s = setup(); + expect(await s.checker.permits(admission, at)).toBe("allow"); + expect(await s.checker.permits(admission, at, start)).toBe("allow"); + expect(s.ledger.events).toEqual(["lock", "unlock", "lock", "unlock"]); + }); + + it("rejects a cumulative interval crossing an opt-out even after opt-in resumes", async () => { + const s = setup(); + const first = s.history.written[0]; + if (first?.kind !== "open") throw new Error("Missing initial consent"); + s.history.written.push( + { kind: "close", token: first.token, at: "2026-10-07T10:15:00.000Z" }, + { kind: "open", token: "new-consent", clone: s.clone, at: "2026-10-07T10:20:00.000Z" } + ); + s.source.cloneSays(s.clone, consentValue("new-consent")); + expect(await s.checker.permits(admission, at, start)).toBe("deny"); + expect(await s.checker.permits(admission, at, "2026-10-07T10:20:00.000Z")).toBe("allow"); + expect(await s.checker.permits(admission, "2026-10-07T10:15:00.000Z", start)).toBe("deny"); + }); + + it("denies a window spanning two different clones at the same path", async () => { + const s = setup(); + const newClone = cloneOf(`${cwd}/.git`, { + ino: "8", + birthtimeMs: Date.parse("2026-10-07T10:15:00.000Z"), + }); + s.resolutions.resolutions.set(resolutionKey(cwd, s.clone), { + dir: cwd, + root: cwd, + repository_id: repo, + clone: s.clone, + seen_at: "2026-10-07T09:00:00.000Z", + }); + s.locator.directories.set(cwd, { + status: "repository", + root: cwd, + mainRoot: cwd, + clone: newClone, + remote: null, + rootCommit: repo, + }); + s.history.consented(newClone, "2026-10-07T10:15:00.000Z"); + s.source.cloneSays(newClone, grantedBy(newClone)); + expect(await s.checker.permits(admission, at, start)).toBe("deny"); + expect(await s.checker.permits(admission, at)).toBe("allow"); + }); + + it("denies a repository other than the session's admitted repository", async () => { + const s = setup(); + expect(await s.checker.permits({ ...admission, repositoryId: "other" }, at)).toBe("deny"); + }); + + it.each([ + [{ ...admission, cwd: null }, at, undefined], + [{ ...admission, cwd: "" }, at, undefined], + [{ ...admission, repositoryId: null }, at, undefined], + [admission, "invalid", undefined], + [admission, at, "invalid"], + [admission, start, at], + ])( + "leaves incomplete or inconsistent evidence unknown without reading the ledger", + async (a, end, from) => { + const s = setup(); + expect(await s.checker.permits(a, end, from)).toBe("unknown"); + expect(s.ledger.exclusiveRuns).toBe(0); + } + ); + + it.each([ + ["gone", "unknown"], + ["unreadable", "unknown"], + ["outside-repository", "deny"], + ] as const)("does not permit an unresolved %s directory", async (status, expected) => { + const s = setup(); + s.locator.directories.set(cwd, { status }); + expect(await s.checker.permits(admission, at)).toBe(expected); + }); + + it("denies a clone without consent, and treats damaged history as unknown", async () => { + const s = setup(); + s.history.written.length = 0; + expect(await s.checker.permits(admission, at)).toBe("deny"); + s.history.damaged = true; + expect(await s.checker.permits(admission, at)).toBe("unknown"); + }); + + it("does not authorize when the history becomes damaged after directory resolution", async () => { + class BecomesDamaged extends InMemoryConsentHistory { + calls = 0; + override async read() { + const result = await super.read(); + return { ...result, damaged: ++this.calls > 1 }; + } + } + const s = setup(new BecomesDamaged()); + expect(await s.checker.permits(admission, at)).toBe("unknown"); + expect(s.resolutions.saves).toBe(1); + expect(s.ledger.events).toEqual(["lock", "unlock"]); + }); + + it("closes directory resolution and releases the lock when the history read fails", async () => { + class FailsOnRecheck extends InMemoryConsentHistory { + calls = 0; + override async read() { + if (++this.calls > 1) throw new Error("history unavailable"); + return super.read(); + } + } + const s = setup(new FailsOnRecheck()); + await expect(s.checker.permits(admission, at)).rejects.toThrow("history unavailable"); + expect(s.resolutions.saves).toBe(1); + expect(s.ledger.events).toEqual(["lock", "unlock"]); + }); +}); From 1d962379c8bb49c348eae4af518487681eeef7c4 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:44:42 +0800 Subject: [PATCH 20/56] fix(telemetry): clarify identity and consent status --- .../application/switch/telemetry-off-use-case.ts | 12 ++++++++---- cli/src/presentation/commands/telemetry.ts | 2 ++ .../telemetry/telemetry-lifecycle-display.ts | 2 ++ .../telemetry-switch-use-cases.unit.test.ts | 2 +- cli/tests/e2e/telemetry-identity.e2e.test.ts | 16 +++++++++++++++- cli/tests/e2e/telemetry-lifecycle.e2e.test.ts | 11 +++++++++++ .../telemetry-wiring.integration.test.ts | 3 +++ 7 files changed, 42 insertions(+), 6 deletions(-) diff --git a/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts b/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts index 212e4b64d..3b4828914 100644 --- a/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts +++ b/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts @@ -13,7 +13,7 @@ export type OffResult = readonly status: "refused"; readonly reason: "outside-repository" | "unreadable-git-config" | "damaged-consent-log"; } - | { readonly status: "off"; readonly changed: boolean }; + | { readonly status: "off"; readonly changed: boolean; readonly closedAt?: string }; /** Stops measuring this clone: its interval ends now, and the calls it made while on stay * stored, whenever the next ingest runs. What was already measured stays until `forget`. */ @@ -33,7 +33,7 @@ export class TelemetryOffUseCase { const { located } = clone; const token = tokenOfKey(clone.value); const granted = token !== null; - let closed = false; + let closedAt: string | undefined; let damaged = false; // The lock first: an ingest running now must see the key and the interval change together. await this.ledger.exclusively(async () => { @@ -58,12 +58,16 @@ export class TelemetryOffUseCase { ); for (const interval of ours) { await log.close(interval.token, at); - closed = true; + closedAt = at.toISOString(); } } if (granted) await this.writer.set(located.root, CONSENT_WITHDRAWN); }); if (damaged) return { status: "refused", reason: "damaged-consent-log" }; - return { status: "off", changed: granted || closed }; + return { + status: "off", + changed: granted || closedAt !== undefined, + ...(closedAt ? { closedAt } : {}), + }; } } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 53411d00e..b551ddcde 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -629,6 +629,8 @@ export function registerTelemetryCommand(program: Command): void { ? await use.show() : await use.set(id); printIdentityResult(output, result); + if (id === undefined && !cmdOptions.off) + output.info(`Installation ID: ${(await deps.checkInstallationId()) ?? "unavailable"}.`); if (result.status === "refused") process.exit(1); }); }); diff --git a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts index bbc35007b..2ca52ef8a 100644 --- a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts +++ b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts @@ -112,6 +112,8 @@ export function printOffResult(output: CLIOutput, result: OffResult): void { ? "Measurement is off for this clone. What was measured stays until `aidd telemetry forget --yes`." : "Measurement was not on for this clone." ); + if (result.closedAt) output.info(`Consent ended at ${result.closedAt}.`); + output.info("Native OTLP collection is not stopped and may include other projects."); } const WHAT: Readonly<Record<ErasureKind, string>> = { diff --git a/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts b/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts index 59f57aeeb..48ca3cc7f 100644 --- a/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts @@ -549,7 +549,7 @@ describe("aidd telemetry off, kept as the end of an interval", () => { await s.on.execute("/w/repo", "keep"); s.consents.values.set("/w/repo", "off"); s.clock.now = new Date(T2); - expect(await s.off.execute("/w/repo")).toEqual({ status: "off", changed: true }); + expect(await s.off.execute("/w/repo")).toEqual({ status: "off", changed: true, closedAt: T2 }); expect(s.history.written.at(-1)).toEqual({ kind: "close", token: "tok-1", at: T2 }); }); diff --git a/cli/tests/e2e/telemetry-identity.e2e.test.ts b/cli/tests/e2e/telemetry-identity.e2e.test.ts index 5571d2d28..2293cd8f1 100644 --- a/cli/tests/e2e/telemetry-identity.e2e.test.ts +++ b/cli/tests/e2e/telemetry-identity.e2e.test.ts @@ -1,4 +1,4 @@ -import { mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { mkdir, readFile, rm, stat, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { @@ -13,6 +13,20 @@ import { useReportSandbox(); describe("aidd telemetry identity", () => { + it("shows the saved installation identifier separately from the measurement identity", async () => { + await rm(join(sandbox.userConfig, "installation", "id.json"), { force: true }); + const before = await aidd(["identity"]); + expect(before.stdout).toContain("Installation ID: unavailable"); + await aidd(["on", "--yes", "--as", "person-a"]); + const installationPath = join(sandbox.userConfig, "installation", "id.json"); + const installation = JSON.parse(await readFile(installationPath, "utf8")) as { + installation_id: string; + }; + const shown = await aidd(["identity"]); + expect(shown.exitCode).toBe(0); + expect(shown.stdout).toContain('"person-a"'); + expect(shown.stdout).toContain(`Installation ID: ${installation.installation_id}`); + }); it("shows the person axis as anonymous until an identity is chosen", async () => { const before = rowsOf(await json("person")); expect(Object.keys(before)).toEqual(["absent"]); diff --git a/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts b/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts index 350b99a19..3d6d1ad95 100644 --- a/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts +++ b/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts @@ -183,9 +183,20 @@ describe("aidd telemetry off", () => { const run = await cli(["off"]); expect(run.exitCode).toBe(0); expect(run.stdout).toContain("Measurement is off"); + const history = readFileSync(join(box.telemetry, "ledger", "consents.jsonl"), "utf8") + .trim() + .split("\n") + .map((line) => JSON.parse(line) as { close?: string }); + const close = history.find((event) => event.close !== undefined); + expect(close?.close).toBeDefined(); + expect(run.stdout).toContain(`Consent ended at ${close?.close}`); + expect(run.stdout).toContain("Native OTLP collection is not stopped"); expect(git(repo, box.gitEnv, "config", "--local", "--get", "aidd.telemetry").trim()).toBe( "off" ); + const repeated = await cli(["off"]); + expect(repeated.stdout).not.toContain("Consent ended at"); + expect(repeated.stdout).toContain("Native OTLP collection is not stopped"); }); }); diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index e0e231d70..f3563761d 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -307,9 +307,12 @@ describe("aidd telemetry report, narrowed", () => { describe("aidd telemetry identity", () => { it("shows who is named when given nothing", async () => { identity.show.mockResolvedValue({ status: "unset" }); + checkInstallationId.mockResolvedValueOnce("installation-id"); expect(await run("identity")).toBeNull(); expect(identity.show).toHaveBeenCalled(); + expect(checkInstallationId).toHaveBeenCalledOnce(); expect(stdout()).toContain("No identity is set"); + expect(stdout()).toContain("Installation ID: installation-id"); }); it("names the person it is given", async () => { From 7100d42be55ef605105fdb20aa425837304d6ab9 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:46:06 +0800 Subject: [PATCH 21/56] fix(telemetry): describe recorded consent closure precisely --- .../display/telemetry/telemetry-lifecycle-display.ts | 4 ++-- cli/tests/e2e/telemetry-lifecycle.e2e.test.ts | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts index 2ca52ef8a..53e94f704 100644 --- a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts +++ b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts @@ -112,8 +112,8 @@ export function printOffResult(output: CLIOutput, result: OffResult): void { ? "Measurement is off for this clone. What was measured stays until `aidd telemetry forget --yes`." : "Measurement was not on for this clone." ); - if (result.closedAt) output.info(`Consent ended at ${result.closedAt}.`); - output.info("Native OTLP collection is not stopped and may include other projects."); + if (result.closedAt) output.info(`Consent close recorded at ${result.closedAt}.`); + output.info("Native OTLP configuration and collector are unchanged and may still collect data."); } const WHAT: Readonly<Record<ErasureKind, string>> = { diff --git a/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts b/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts index 3d6d1ad95..0a92f0733 100644 --- a/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts +++ b/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts @@ -189,14 +189,14 @@ describe("aidd telemetry off", () => { .map((line) => JSON.parse(line) as { close?: string }); const close = history.find((event) => event.close !== undefined); expect(close?.close).toBeDefined(); - expect(run.stdout).toContain(`Consent ended at ${close?.close}`); - expect(run.stdout).toContain("Native OTLP collection is not stopped"); + expect(run.stdout).toContain(`Consent close recorded at ${close?.close}`); + expect(run.stdout).toContain("Native OTLP configuration and collector are unchanged"); expect(git(repo, box.gitEnv, "config", "--local", "--get", "aidd.telemetry").trim()).toBe( "off" ); const repeated = await cli(["off"]); - expect(repeated.stdout).not.toContain("Consent ended at"); - expect(repeated.stdout).toContain("Native OTLP collection is not stopped"); + expect(repeated.stdout).not.toContain("Consent close recorded at"); + expect(repeated.stdout).toContain("Native OTLP configuration and collector are unchanged"); }); }); From 50bc899ff3a72898c90a01830c581f341a6e4130 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:49:34 +0800 Subject: [PATCH 22/56] test(telemetry): verify recorded consent closure --- .../telemetry/telemetry-lifecycle.integration.test.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts b/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts index 46f4b4b8e..c70a7eaa8 100644 --- a/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts +++ b/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts @@ -257,7 +257,11 @@ describe("opting in to a repository the previous version measured", () => { it("turns off with the git config value off, and leaves .aidd/config.json alone", async () => { await configure('{"keep":1}'); await deps.telemetryOnUseCase.execute(repo, "keep"); - expect(await deps.telemetryOffUseCase.execute(repo)).toEqual({ status: "off", changed: true }); + expect(await deps.telemetryOffUseCase.execute(repo)).toMatchObject({ + status: "off", + changed: true, + closedAt: expect.any(String), + }); expect(consentOf()).toBe("off"); expect(read(join(repo, ".aidd", "config.json"))).toBe('{"keep":1}'); }); From 629163fa48d88ba6277b972f8b7158e0e2e573c8 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 12:52:56 +0800 Subject: [PATCH 23/56] test(telemetry): avoid maintenance during clone copy --- scripts/__tests__/aidd-telemetry-hooks.test.js | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/scripts/__tests__/aidd-telemetry-hooks.test.js b/scripts/__tests__/aidd-telemetry-hooks.test.js index f1e1c578b..57b5ca959 100644 --- a/scripts/__tests__/aidd-telemetry-hooks.test.js +++ b/scripts/__tests__/aidd-telemetry-hooks.test.js @@ -51,9 +51,10 @@ const GIT_DIR_ON_PATH = gitDirectory(); /** git as a test sets a repository up: never through a variable a commit hook exported. */ function git(cwd, ...args) { const env = Object.fromEntries(Object.entries(process.env).filter(([k]) => !k.startsWith("GIT_"))); + // A commit can detach maintenance and change object files while the copy tests read them. const run = spawnSync( "git", - ["-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", ...args], + ["-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", "-c", "maintenance.auto=false", ...args], { cwd, env, encoding: "utf8" } ); assert.equal(run.status, 0, `git ${args.join(" ")}: ${run.stderr}`); @@ -662,6 +663,13 @@ test("consent: a key with no interval open for the clone grants nothing, asks no } }); +test("sandbox: disables automatic Git maintenance before copying a repository", () => { + withBox({}, (box) => { + git(box.repo, "config", "--local", "maintenance.auto", "true"); + assert.equal(git(box.repo, "config", "--get", "maintenance.auto").trim(), "false"); + }); +}); + test("consent: a cp -R copy carries the key and not the clone, so it asks nothing and closes nothing", () => { withBox({}, (box) => { git(box.repo, "commit", "--allow-empty", "-q", "-m", "x"); From 7c61d8173d4dac95aa298d9a47b6a9094edec330 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 15:02:15 +0800 Subject: [PATCH 24/56] fix(telemetry): guard consent and enrich codex metadata --- cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 4 + .../application/report/session-consent.ts | 29 ++- .../report/session-feature-use-case.ts | 4 +- .../domain/report/session-feature.ts | 23 ++ .../actions/metadata/session-repository.ts | 49 ++++ .../otel/actions/sync-all-titles.ts | 29 ++- .../infrastructure/otel/actions/sync-all.ts | 120 ++++++---- .../infrastructure/otel/actions/sync.ts | 103 ++++++-- cli/src/presentation/commands/telemetry.ts | 59 ++++- cli/src/runtime/wiring/telemetry-otel.ts | 7 +- .../report/session-consent.unit.test.ts | 48 ++++ .../otel/session-repository.unit.test.ts | 76 ++++++ .../session-title-sync.integration.test.ts | 71 +++++- .../otel/sync-all-otel.integration.test.ts | 224 ++++++++++++++++++ .../sync-otel-attributes.integration.test.ts | 88 +++++++ .../sync-otel-integrity.integration.test.ts | 9 +- .../otel/sync-otel.integration.test.ts | 114 ++++++++- .../e2e/telemetry-otel-metadata.e2e.test.ts | 8 +- cli/tests/golden/snapshots/help/surface.json | 2 +- plugins/aidd-telemetry/OBSERVABILITY.md | 7 +- plugins/aidd-telemetry/README.md | 2 + 22 files changed, 959 insertions(+), 119 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts create mode 100644 cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts diff --git a/cli/package.json b/cli/package.json index db9a34146..73adbc8cd 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 807, + "bundleBudgetKB": 813, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 58c801476..a79740a9a 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -62,6 +62,10 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 0.1% measured headroom. // 807 KB: 806.6 KB with opt-in native Codex project metadata and shared metadata publishing; // +3.3 KB, less than 0.1% measured headroom. +// 808 KB: 807.49 KB with bounded, event-scoped Codex reasoning-effort forwarding; +// +0.53 KB over the preceding 806.96 KB build, less than 0.1% measured headroom. +// 810 KB: 809.79 KB with separately opted-in, sanitized Codex repository metadata. +// 813 KB: 812.66 KB with consent and session-admission checks on every delivery attempt. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/application/report/session-consent.ts b/cli/src/contexts/telemetry/application/report/session-consent.ts index 964d72a09..944d422c1 100644 --- a/cli/src/contexts/telemetry/application/report/session-consent.ts +++ b/cli/src/contexts/telemetry/application/report/session-consent.ts @@ -17,16 +17,20 @@ export class SessionConsent { async permits( admission: SessionAdmission, at: string, - start?: string + start?: string, + current?: string ): Promise<"allow" | "deny" | "unknown"> { const endTime = Date.parse(at); const startTime = start === undefined ? endTime : Date.parse(start); + const currentTime = current === undefined ? endTime : Date.parse(current); if ( !admission.cwd || !admission.repositoryId || !Number.isFinite(endTime) || !Number.isFinite(startTime) || - startTime > endTime + !Number.isFinite(currentTime) || + startTime > endTime || + currentTime < endTime ) return "unknown"; return this.ledger.exclusively(async () => { @@ -34,8 +38,12 @@ export class SessionConsent { try { const first = await run.resolve(admission.cwd ?? "", new Date(startTime).toISOString()); const last = await run.resolve(admission.cwd ?? "", new Date(endTime).toISOString()); - if ("skipped" in first || "skipped" in last) { - const reasons = [first, last].flatMap((value) => + const now = + current === undefined + ? last + : await run.resolve(admission.cwd ?? "", new Date(currentTime).toISOString()); + if ("skipped" in first || "skipped" in last || "skipped" in now) { + const reasons = [first, last, now].flatMap((value) => "skipped" in value ? [value.skipped] : [] ); return reasons.some( @@ -46,18 +54,17 @@ export class SessionConsent { } if ( !sameClone(first.stored.clone, last.stored.clone) || + !sameClone(first.stored.clone, now.stored.clone) || first.stored.repository_id !== admission.repositoryId || - last.stored.repository_id !== admission.repositoryId + last.stored.repository_id !== admission.repositoryId || + now.stored.repository_id !== admission.repositoryId ) return "deny"; const history = await this.consents.read(); if (history.damaged) return "unknown"; - return coversConsentWindow( - foldConsent(history.events), - first.stored.clone, - startTime, - endTime - ) + const intervals = foldConsent(history.events); + return coversConsentWindow(intervals, first.stored.clone, startTime, endTime) && + coversConsentWindow(intervals, first.stored.clone, currentTime, currentTime) ? "allow" : "deny"; } finally { diff --git a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts index 13173c40d..b48fb8c8f 100644 --- a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts +++ b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts @@ -53,7 +53,7 @@ export class SessionFeatureUseCase { return found.feature; } - async permits(admission: SessionAdmission, at: string, start?: string) { - return this.consent.permits(admission, at, start); + async permits(admission: SessionAdmission, at: string, start?: string, current?: string) { + return this.consent.permits(admission, at, start, current); } } diff --git a/cli/src/contexts/telemetry/domain/report/session-feature.ts b/cli/src/contexts/telemetry/domain/report/session-feature.ts index bdff8bc61..80a1cb1ed 100644 --- a/cli/src/contexts/telemetry/domain/report/session-feature.ts +++ b/cli/src/contexts/telemetry/domain/report/session-feature.ts @@ -13,6 +13,29 @@ export interface SessionAdmission { readonly repositoryId: string | null; } +/** A session cannot move to another clone or repository during a pending delivery. */ +export function sameSessionAdmission( + expected: SessionAdmission | undefined, + current: SessionAdmission | undefined +): boolean { + return Boolean( + expected?.cwd && + expected.repositoryId && + current?.cwd === expected.cwd && + current.repositoryId === expected.repositoryId + ); +} + +export function sameSessionFeature(left: SessionFeature, right: SessionFeature): boolean { + return ( + left.state === right.state && + (left.state !== "confirmed" || + (right.state === "confirmed" && + left.feature === right.feature && + left.branch === right.branch)) + ); +} + // biome-ignore lint/suspicious/noControlCharactersInRegex: control characters cannot name a branch const CONTROL = /[\u0000-\u001f\u007f]/u; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts new file mode 100644 index 000000000..4b95182f2 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts @@ -0,0 +1,49 @@ +import { asPlainObject } from "../../../../../../kernel/reading/plain-object.js"; +import { createSessionMetadataPublisher, type Metadata, type Request } from "./session-title.js"; + +/** Keep identities on supported hosts. Never forward raw Git credentials or local paths. */ +function publicRepository(origin: string): string | null { + if (origin.length > 2048) return null; + const match = + /^(?:https:\/\/(github\.com|gitlab\.com)\/|git@(github\.com|gitlab\.com):)([A-Za-z0-9_.-]+(?:\/[A-Za-z0-9_.-]+)+)$/.exec( + origin + ); + if (!match || match[0] !== origin) return null; + const host = match[1] ?? match[2]; + const path = match[3].endsWith(".git") ? match[3].slice(0, -4) : match[3]; + const parts = path.split("/"); + if ( + (host === "github.com" && parts.length !== 2) || + parts.some((part) => !part || part === "." || part === "..") + ) + return null; + return `https://${host}/${path}`; +} + +/** The Git origin in the native thread is separate from its Codex project. */ +export async function readSessionRepository(request: Request, session: string): Promise<Metadata> { + try { + const result = await request("thread/read", { threadId: session, includeTurns: false }); + const thread = asPlainObject(asPlainObject(result)?.thread); + if (!thread || thread.id !== session || thread.sessionId !== session) + return { available: false }; + if (thread.gitInfo === null) return { available: true, values: {} }; + const git = asPlainObject(thread.gitInfo); + if (!git) return { available: false }; + if (git.originUrl === null) return { available: true, values: {} }; + if (typeof git.originUrl !== "string") return { available: false }; + const url = publicRepository(git.originUrl); + return { available: true, values: url ? { "aidd.repository_url": url } : {} }; + } catch { + return { available: false }; + } +} + +export function createSessionRepositoryPublisher(home: string, onUnavailable?: () => void) { + return createSessionMetadataPublisher( + home, + "aidd.session_repository", + readSessionRepository, + onUnavailable + ); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts index f81e3700a..0da63f644 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts @@ -1,9 +1,13 @@ import { constants } from "node:fs"; import { lstat, open } from "node:fs/promises"; import { join } from "node:path"; -import type { SessionAdmission } from "../../../domain/report/session-feature.js"; +import { + type SessionAdmission, + sameSessionAdmission, +} from "../../../domain/report/session-feature.js"; import { CODEX_CONFIG_ID } from "../settings.js"; import { createSessionProjectPublisher } from "./metadata/session-project.js"; +import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { destination, @@ -28,7 +32,12 @@ function alreadyKnown(identity: Identity | undefined, observed: Identity): boole export async function syncAllSessionTitles( options: TitleOptions, snapshot: () => Promise<ReadonlyMap<string, SessionAdmission>>, - permits: (admission: SessionAdmission, at: string, start?: string) => Promise<Consent> + permits: ( + admission: SessionAdmission, + at: string, + start?: string, + current?: string + ) => Promise<Consent> ) { if (options.config !== CODEX_CONFIG_ID || !options.includeSessionTitles) throw new Error("Title-only sync requires Codex and explicit title opt-in."); @@ -41,6 +50,9 @@ export async function syncAllSessionTitles( const projects = options.includeSessionProjects ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) : undefined; + const repositories = options.includeSessionRepositories + ? createSessionRepositoryPublisher(options.home, options.onRepositoryUnavailable) + : undefined; const titles = createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable, }); @@ -109,7 +121,7 @@ export async function syncAllSessionTitles( if ((await permits(admission, new Date().toISOString())) !== "allow") continue; currentConsenting++; let stillConsenting = true; - for (const publisher of [titles, projects]) + for (const publisher of [titles, projects, repositories]) if ( publisher && (await publisher.publish(session, identity, async (event) => { @@ -118,7 +130,15 @@ export async function syncAllSessionTitles( stillConsenting = false; return false; } - await send({ ...base, session }, event); + await send({ ...base, session }, event, async () => { + const current = (await snapshot()).get(session); + return ( + !!current && + sameSessionAdmission(admission, current) && + !!event.at && + (await permits(current, event.at, undefined, new Date().toISOString())) === "allow" + ); + }); return true; })) ) @@ -128,6 +148,7 @@ export async function syncAllSessionTitles( } finally { titles.close(); projects?.close(); + repositories?.close(); await file.close(); } } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts index a68a7bbb8..2f70a816c 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -4,12 +4,19 @@ import { lstat, mkdir, open, readdir, rename, rm, writeFile } from "node:fs/prom import { join } from "node:path"; import { setTimeout as delay } from "node:timers/promises"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; -import type { SessionAdmission, SessionFeature } from "../../../domain/report/session-feature.js"; +import { + type SessionAdmission, + type SessionFeature, + sameSessionAdmission, + sameSessionFeature, +} from "../../../domain/report/session-feature.js"; import { LedgerLock } from "../../ledger-lock.js"; import { CODEX_CONFIG_ID } from "../settings.js"; import { createSessionProjectPublisher } from "./metadata/session-project.js"; +import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { + admitTransferIdentity, bindingTransfer, destination, type Identity, @@ -211,7 +218,12 @@ async function initialOffset( export async function syncAllOtel( options: AllOptions, snapshot: () => Promise<Snapshot>, - permits: (admission: SessionAdmission, at: string, start?: string) => Promise<Consent> + permits: ( + admission: SessionAdmission, + at: string, + start?: string, + current?: string + ) => Promise<Consent> ) { const base = asOptions(options, "auto"); destination(base, "logs"); @@ -233,6 +245,10 @@ export async function syncAllOtel( options.config === CODEX_CONFIG_ID && options.includeSessionProjects ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) : undefined; + const repositories = + options.config === CODEX_CONFIG_ID && options.includeSessionRepositories + ? createSessionRepositoryPublisher(options.home, options.onRepositoryUnavailable) + : undefined; const titles = options.config === CODEX_CONFIG_ID && options.includeSessionTitles ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) @@ -307,10 +323,26 @@ export async function syncAllOtel( }); } }; + const stillPermitted = async ( + id: string, + admission: SessionAdmission, + items: readonly Transfer[], + expectedFeature?: SessionFeature + ) => { + const current = (await snapshot()).get(id); + if (!sameSessionAdmission(admission, current) || !current) return false; + if (expectedFeature && !sameSessionFeature(expectedFeature, current.feature)) return false; + const consentAt = new Date().toISOString(); + for (const item of items) + if (!item.at || (await permits(current, item.at, item.start, consentAt)) !== "allow") + return false; + return true; + }; const process = async ( id: string, items: readonly Transfer[], - admission: SessionAdmission + admission: SessionAdmission, + state: SessionState ): Promise<boolean> => { let wait = false; const allowed: Transfer[] = []; @@ -328,10 +360,27 @@ export async function syncAllOtel( } if (verdict === "unknown") wait = true; else if (verdict === "deny") denied++; - else allowed.push(transfer); + else { + allowed.push(transfer); + admitTransferIdentity(transfer, asOptions(options, id), state.identity); + } } + if (state.identity.conflict) + await saveBindingState( + join(directory, `${key(options, id)}.json`), + directory, + { + session: id, + identity: [...state.identity.values], + binding: state.binding, + conflict: true, + }, + state.mayHaveLinked + ); for (const batch of packTransfers(allowed)) { - await send(asOptions(options, id), batch.transfer); + await send(asOptions(options, id), batch.transfer, () => + stillPermitted(id, admission, batch.items) + ); sent += batch.count; } return wait; @@ -360,7 +409,9 @@ export async function syncAllOtel( state: "ambiguous", }); if (event) { - await send(asOptions(options, id), event); + await send(asOptions(options, id), event, () => + stillPermitted(id, admission, [event], feature) + ); sent++; state.binding = "invalidated"; await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { @@ -382,7 +433,9 @@ export async function syncAllOtel( binding: next, }); state.mayHaveLinked = true; - await send(asOptions(options, id), event); + await send(asOptions(options, id), event, () => + stillPermitted(id, admission, [event], feature) + ); sent++; state.binding = next; state.needsReplay = false; @@ -391,17 +444,19 @@ export async function syncAllOtel( } }; const publishTitles = async () => { - if (!titles && !projects) return; + if (!titles && !projects && !repositories) return; for (const [id, state] of sessions) { const admission = held.get(id); if (!admission || state.identity.conflict) continue; if ((await permits(admission, new Date().toISOString())) !== "allow") continue; - for (const publisher of [titles, projects]) + for (const publisher of [titles, projects, repositories]) if ( publisher && (await publisher.publish(id, state.identity, async (event) => { if (!event.at || (await permits(admission, event.at)) !== "allow") return false; - await send(asOptions(options, id), event); + await send(asOptions(options, id), event, () => + stillPermitted(id, admission, [event]) + ); return true; })) ) @@ -420,25 +475,11 @@ export async function syncAllOtel( const read = await file?.read(raw, 0, item.length, item.offset); if (!read || read.bytesRead !== item.length || raw.at(-1) !== 10) throw new Error("Pending capture range is unavailable; recover the capture."); - const transfers = selectTransfers( - raw.toString("utf8"), - asOptions(options, id), - state.identity - ); - if (state.identity.conflict) { - await saveBindingState( - join(directory, `${key(options, id)}.json`), - directory, - { - session: id, - identity: [...state.identity.values], - binding: state.binding, - conflict: true, - }, - state.mayHaveLinked - ); - } - if (await process(id, transfers, admission)) waiting.push(item); + const transfers = selectTransfers(raw.toString("utf8"), asOptions(options, id), { + values: new Map(), + conflict: false, + }); + if (await process(id, transfers, admission, state)) waiting.push(item); } await replacePending(path, waiting); } @@ -463,23 +504,13 @@ export async function syncAllOtel( if (raw.trim()) for (const id of sessionIdsInCapture(raw, options.config)) { const state = await sessionState(id); - const items = selectTransfers(raw, asOptions(options, id), state.identity); - if (state.identity.conflict) { - await saveBindingState( - join(directory, `${key(options, id)}.json`), - directory, - { - session: id, - identity: [...state.identity.values], - binding: state.binding, - conflict: true, - }, - state.mayHaveLinked - ); - } + const items = selectTransfers(raw, asOptions(options, id), { + values: new Map(), + conflict: false, + }); const admission = held.get(id); const waiting = - items.length > 0 && (!admission || (await process(id, items, admission))); + items.length > 0 && (!admission || (await process(id, items, admission, state))); if (waiting) await appendPending(pendingPath(id), { offset: position, length: newline + 1 }); } @@ -532,6 +563,7 @@ export async function syncAllOtel( } finally { titles?.close(); projects?.close(); + repositories?.close(); await file?.close(); await release(); } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index a264ceda1..e3c439ec9 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -5,9 +5,10 @@ import { join } from "node:path"; import { setTimeout as delay } from "node:timers/promises"; import { isDeepStrictEqual } from "node:util"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; -import type { SessionFeature } from "../../../domain/report/session-feature.js"; +import { type SessionFeature, sameSessionFeature } from "../../../domain/report/session-feature.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; import { createSessionProjectPublisher } from "./metadata/session-project.js"; +import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { readAttributeValue } from "./read-attribute-value.js"; import { seriesFingerprint } from "./series-fingerprint.js"; @@ -27,7 +28,9 @@ export interface SyncOptions { readonly intervalMs?: number; readonly includeSessionTitles?: boolean; readonly includeSessionProjects?: boolean; + readonly includeSessionRepositories?: boolean; readonly onProjectUnavailable?: () => void; + readonly onRepositoryUnavailable?: () => void; readonly onTitleUnavailable?: () => void; readonly titlesOnly?: boolean; } @@ -283,14 +286,16 @@ export interface Transfer { readonly payload: RecordObject; readonly at?: string; readonly start?: string; + /** When set, the same clone must also consent at this instant. Never sent. */ + readonly consentAt?: string; } /** Each item was separately consent-checked. Only points at the same instant and from the * same session are combined, with a hard upper bound matching the receiver. */ export function packTransfers( transfers: readonly Transfer[] -): Array<{ transfer: Transfer; count: number }> { - const groups = new Map<string, Array<{ transfer: Transfer; count: number }>>(); +): Array<{ transfer: Transfer; count: number; items: Transfer[] }> { + const groups = new Map<string, Array<{ transfer: Transfer; count: number; items: Transfer[] }>>(); for (const item of transfers) { const key = `${item.signal}\0${item.at ?? ""}`; const root = item.signal === "logs" ? "resourceLogs" : "resourceMetrics"; @@ -302,14 +307,15 @@ export function packTransfers( if (Buffer.byteLength(JSON.stringify(body)) > MAX_PAYLOAD) { if (Buffer.byteLength(JSON.stringify(item.payload)) > MAX_PAYLOAD) throw new Error("An OTLP observation exceeds the receiver's 2 MiB limit."); - batches.push({ transfer: item, count: 1 }); + batches.push({ transfer: item, count: 1, items: [item] }); } else if (last) { batches[batches.length - 1] = { transfer: { ...last.transfer, payload: body }, count: last.count + 1, + items: [...last.items, item], }; } else { - batches.push({ transfer: item, count: 1 }); + batches.push({ transfer: item, count: 1, items: [item] }); } groups.set(key, batches); } @@ -435,15 +441,29 @@ export function selectTransfers( if (at) a.set("event.timestamp", at); else a.delete("event.timestamp"); } + const safe = safeAttributes(a); + if (options.config === CODEX_CONFIG_ID && event !== "codex.tool_result") { + const key = + event === "codex.conversation_starts" ? "reasoning_effort" : "model_reasoning_effort"; + const entries = array(record.attributes).map(object); + const source = entries.find((entry) => entry.key === key); + const model = object(entries.find((entry) => entry.key === "model")?.value).stringValue; + const value = object(source?.value).stringValue; + if ( + typeof model === "string" && + model.trim() && + typeof value === "string" && + /^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$/.test(value) + ) + safe.push({ key, value: { stringValue: value } }); + } out.push({ signal: "logs", at, payload: { resourceLogs: [ { - scopeLogs: [ - { logRecords: [{ ...timeOf(record), attributes: safeAttributes(a) }] }, - ], + scopeLogs: [{ logRecords: [{ ...timeOf(record), attributes: safe }] }], }, ], }, @@ -533,6 +553,19 @@ export function selectTransfers( return out; } +/** Only a consent-admitted transfer can contribute identity to later metadata. */ +export function admitTransferIdentity( + transfer: Transfer, + options: SyncOptions, + identity: Identity +): void { + selectTransfers( + JSON.stringify({ signal: transfer.signal, payload: transfer.payload }), + options, + identity + ); +} + export function destination(options: SyncOptions, signal: "logs" | "metrics"): URL { let endpoint: URL; try { @@ -601,13 +634,19 @@ async function assertFullDelivery(response: Response, signal: Transfer["signal"] ); } -export async function send(options: SyncOptions, transfer: Transfer): Promise<void> { +export async function send( + options: SyncOptions, + transfer: Transfer, + consent: () => Promise<boolean> +): Promise<void> { const body = JSON.stringify(transfer.payload); if (Buffer.byteLength(body) > MAX_PAYLOAD) throw new Error("An OTLP observation exceeds the receiver's 2 MiB limit."); const url = destination(options, transfer.signal); let backoff = options.intervalMs ?? 1000; for (;;) { + if (!(await consent())) + throw new Error("Consent is no longer confirmed; sync stopped before sending."); try { let response: Response; try { @@ -684,7 +723,9 @@ export async function readBindingDetails( mayHaveLinked: data.mayHaveLinked, details: { session: data.session, - identity: data.identity as Array<[string, string]>, + // Older versions collected identity before checking consent; never reuse it. + identity: + data.identityConsentChecked === true ? (data.identity as Array<[string, string]>) : [], binding: data.binding, conflict: data.conflict === true, }, @@ -706,10 +747,14 @@ export async function saveBindingState( throw new Error("Sync state directory must be private."); const temporary = `${path}.${randomUUID()}.tmp`; try { - await writeFile(temporary, `${JSON.stringify({ mayHaveLinked, ...details })}\n`, { - flag: "wx", - mode: 0o600, - }); + await writeFile( + temporary, + `${JSON.stringify({ mayHaveLinked, ...details, identityConsentChecked: true })}\n`, + { + flag: "wx", + mode: 0o600, + } + ); await rename(temporary, path); } finally { await rm(temporary, { force: true }); @@ -733,7 +778,8 @@ export function bindingTransfer( "event.name", state.state === "confirmed" ? "aidd.session_feature" : "aidd.session_feature_invalidated" ); - a.set("event.timestamp", new Date().toISOString()); + const at = new Date().toISOString(); + a.set("event.timestamp", at); if (state.state === "confirmed") { a.set("aidd.feature", state.feature); a.set("aidd.branch", state.branch); @@ -741,6 +787,7 @@ export function bindingTransfer( const bindingAttrs = [...a].map(([key, value]) => ({ key, value: { stringValue: value } })); return { signal: "logs", + at, payload: { resourceLogs: [ { @@ -776,6 +823,13 @@ export async function syncOtel( const path = join(options.home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); const identity: Identity = { values: new Map(), conflict: false }; + const stillPermitted = async (items: readonly Transfer[], expectedFeature?: SessionFeature) => { + const currentFeature = await resolve(); + if (expectedFeature && !sameSessionFeature(expectedFeature, currentFeature)) return false; + const consentAt = new Date().toISOString(); + for (const item of items) if (!(await permits({ ...item, consentAt }))) return false; + return true; + }; let sent = 0; let skipped = 0; let offset = 0; @@ -793,6 +847,10 @@ export async function syncOtel( options.config === CODEX_CONFIG_ID && options.includeSessionProjects ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) : undefined; + const repositories = + options.config === CODEX_CONFIG_ID && options.includeSessionRepositories + ? createSessionRepositoryPublisher(options.home, options.onRepositoryUnavailable) + : undefined; const titles = options.config === CODEX_CONFIG_ID && options.includeSessionTitles ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) @@ -824,7 +882,8 @@ export async function syncOtel( if (line.trim()) { const admitted: Transfer[] = []; const verdicts = new Map<string, boolean>(); - for (const transfer of selectTransfers(line, options, identity)) { + const scratch: Identity = { values: new Map(), conflict: false }; + for (const transfer of selectTransfers(line, options, scratch)) { if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { skipped++; continue; @@ -840,10 +899,11 @@ export async function syncOtel( continue; } admitted.push(transfer); + admitTransferIdentity(transfer, options, identity); } if (!options.titlesOnly) for (const batch of packTransfers(admitted)) { - await send(options, batch.transfer); + await send(options, batch.transfer, () => stillPermitted(batch.items)); sent += batch.count; } } @@ -877,7 +937,7 @@ export async function syncOtel( if (!options.titlesOnly && bindingAllowed && shouldInvalidate && binding !== "invalidated") { const event = bindingTransfer(options, identity, { state: "ambiguous" }); if (event) { - await send(options, event); + await send(options, event, () => stillPermitted([event], feature)); sent++; binding = "invalidated"; if (statePath !== null) @@ -904,19 +964,19 @@ export async function syncOtel( }); } mayHaveLinked = true; - await send(options, event); + await send(options, event, () => stillPermitted([event], feature)); sent++; binding = nextBinding; needsReplay = false; } } if (bindingAllowed) - for (const publisher of [titles, projects]) { + for (const publisher of [titles, projects, repositories]) { if ( publisher && (await publisher.publish(options.session, identity, async (event) => { if (!(await permits(event))) return false; - await send(options, event); + await send(options, event, () => stillPermitted([event])); return true; })) ) @@ -935,6 +995,7 @@ export async function syncOtel( } finally { titles?.close(); projects?.close(); + repositories?.close(); await handle.close(); } } diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index b551ddcde..f3691198d 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -232,9 +232,14 @@ export function registerTelemetryCommand(program: Command): void { "Send native Codex project IDs and names (may contain sensitive text)", false ) + .option( + "--include-session-repositories", + "Send verified Codex GitHub/GitLab repository URLs (may identify private work)", + false + ) .option( "--titles-only", - "Send only Codex titles and opted-in projects for consented sessions", + "Send only Codex titles and opted-in metadata for consented sessions", false ) .action( @@ -246,6 +251,7 @@ export function registerTelemetryCommand(program: Command): void { follow: boolean; includeSessionTitles: boolean; includeSessionProjects: boolean; + includeSessionRepositories: boolean; titlesOnly: boolean; }) => { const { output, run } = commandOf(program); @@ -260,10 +266,13 @@ export function registerTelemetryCommand(program: Command): void { return; } if ( - (options.includeSessionTitles || options.includeSessionProjects || options.titlesOnly) && + (options.includeSessionTitles || + options.includeSessionProjects || + options.includeSessionRepositories || + options.titlesOnly) && options.config !== otelOptions.titleConfig ) { - output.error("Session title and project sync is currently available only for Codex."); + output.error("Session metadata sync is currently available only for Codex."); process.exitCode = 1; return; } @@ -273,6 +282,10 @@ export function registerTelemetryCommand(program: Command): void { output.warn( "A native Codex session project could not be verified; telemetry sync continues." ); + const onRepositoryUnavailable = () => + output.warn( + "A native Codex Git repository could not be verified; telemetry sync continues." + ); const onTitleUnavailable = () => output.warn("A Codex session title could not be read; telemetry sync continues."); if (options.titlesOnly && !options.session) { @@ -284,12 +297,14 @@ export function registerTelemetryCommand(program: Command): void { token: process.env.AIDD_OTEL_TOKEN ?? "", includeSessionTitles: true, includeSessionProjects: options.includeSessionProjects, + includeSessionRepositories: options.includeSessionRepositories, onProjectUnavailable, + onRepositoryUnavailable, onTitleUnavailable, }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), - (admission, at, start) => - deps.telemetry.sessionFeatureUseCase.permits(admission, at, start) + (admission, at, start, current) => + deps.telemetry.sessionFeatureUseCase.permits(admission, at, start, current) ); output.info( `Sent ${result.sent} session metadata updates; no metrics or feature bindings were sent.` @@ -313,12 +328,14 @@ export function registerTelemetryCommand(program: Command): void { token: process.env.AIDD_OTEL_TOKEN ?? "", includeSessionTitles: options.includeSessionTitles, includeSessionProjects: options.includeSessionProjects, + includeSessionRepositories: options.includeSessionRepositories, onProjectUnavailable, + onRepositoryUnavailable, onTitleUnavailable, }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), - (admission, at, start) => - deps.telemetry.sessionFeatureUseCase.permits(admission, at, start) + (admission, at, start, current) => + deps.telemetry.sessionFeatureUseCase.permits(admission, at, start, current) ); output.info( `Sent ${result.sent} observations; ${result.denied} were outside consent; ${result.pending} await confirmation.` @@ -327,10 +344,30 @@ export function registerTelemetryCommand(program: Command): void { } const session = options.session; let admission = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get(session); + const initialAdmission = admission; + if (!initialAdmission) + throw new Error("No consented usage found for this session; no capture was sent."); const resolve = async () => { - admission = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get(session); - if (!admission) + const current = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get( + session + ); + if (!current) { + admission = undefined; throw new Error("No consented usage found for this session; no capture was sent."); + } + if ( + !initialAdmission.cwd || + !initialAdmission.repositoryId || + current.cwd !== initialAdmission.cwd || + current.repositoryId !== initialAdmission.repositoryId + ) { + admission = undefined; + throw new Error( + "Session admission changed or is unavailable; sync stopped before sending." + ); + } + admission = current; + if (!admission) throw new Error("Session admission is unavailable."); return admission.feature; }; await resolve(); @@ -340,6 +377,7 @@ export function registerTelemetryCommand(program: Command): void { ...options, includeSessionTitles: options.includeSessionTitles || options.titlesOnly, onProjectUnavailable, + onRepositoryUnavailable, session, token: process.env.AIDD_OTEL_TOKEN ?? "", onTitleUnavailable, @@ -352,7 +390,8 @@ export function registerTelemetryCommand(program: Command): void { (await deps.telemetry.sessionFeatureUseCase.permits( admission, transfer.at, - transfer.start + transfer.start, + transfer.consentAt )) === "allow" ) ); diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index 5a09d3d1b..47b8ac32c 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -3,6 +3,7 @@ import type { SessionAdmission, SessionFeature, } from "../../contexts/telemetry/domain/report/session-feature.js"; + import { startOtelCollector } from "../../contexts/telemetry/infrastructure/otel/actions/collect.js"; import { initializeOtel } from "../../contexts/telemetry/infrastructure/otel/actions/init.js"; import { runOtelOpenCode } from "../../contexts/telemetry/infrastructure/otel/actions/run.js"; @@ -54,7 +55,8 @@ export function followCapturedSessions( permits: ( admission: SessionAdmission, at: string, - start?: string + start?: string, + current?: string ) => Promise<"allow" | "deny" | "unknown"> ) { return syncAllOtel({ ...options, home: homedir(), stateDir: telemetryDir() }, snapshot, permits); @@ -66,7 +68,8 @@ export function syncCapturedSessionTitles( permits: ( admission: SessionAdmission, at: string, - start?: string + start?: string, + current?: string ) => Promise<"allow" | "deny" | "unknown"> ) { return syncAllSessionTitles({ ...options, home: homedir() }, snapshot, permits); diff --git a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts index d51e865fe..5849b8f85 100644 --- a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { DirectoryResolver } from "../../../../../src/contexts/telemetry/application/directory-resolver.js"; import { SessionConsent } from "../../../../../src/contexts/telemetry/application/report/session-consent.js"; +import { cloneKey } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { resolutionKey } from "../../../../../src/contexts/telemetry/domain/repository-resolution.js"; import { consentValue } from "../../../../../src/contexts/telemetry/domain/telemetry-consent.js"; import { @@ -68,6 +69,7 @@ describe("consent for an observed session", () => { ); s.source.cloneSays(s.clone, consentValue("new-consent")); expect(await s.checker.permits(admission, at, start)).toBe("deny"); + expect(await s.checker.permits(admission, start, undefined, at)).toBe("allow"); expect(await s.checker.permits(admission, at, "2026-10-07T10:20:00.000Z")).toBe("allow"); expect(await s.checker.permits(admission, "2026-10-07T10:15:00.000Z", start)).toBe("deny"); }); @@ -97,6 +99,7 @@ describe("consent for an observed session", () => { s.source.cloneSays(newClone, grantedBy(newClone)); expect(await s.checker.permits(admission, at, start)).toBe("deny"); expect(await s.checker.permits(admission, at)).toBe("allow"); + expect(await s.checker.permits(admission, start, undefined, at)).toBe("deny"); }); it("denies a repository other than the session's admitted repository", async () => { @@ -104,6 +107,51 @@ describe("consent for an observed session", () => { expect(await s.checker.permits({ ...admission, repositoryId: "other" }, at)).toBe("deny"); }); + it.each(["start", "end"] as const)( + "keeps a window unknown when its %s is unreadable and its other end is denied", + async (unreadable) => { + const s = setup(); + const newClone = cloneOf(`${cwd}/.git`, { + ino: "8", + birthtimeMs: Date.parse("2026-10-07T10:15:00.000Z"), + }); + s.resolutions.resolutions.set(resolutionKey(cwd, s.clone), { + dir: cwd, + root: cwd, + repository_id: repo, + clone: s.clone, + seen_at: "2026-10-07T09:00:00.000Z", + }); + s.locator.directories.set(cwd, { + status: "repository", + root: cwd, + mainRoot: cwd, + clone: newClone, + remote: null, + rootCommit: repo, + }); + const initial = s.history.written[0]; + if (initial?.kind !== "open") throw new Error("Missing initial consent"); + if (unreadable === "end") + s.history.written[0] = { ...initial, at: "2026-10-07T10:10:00.000Z" }; + s.history.consented( + newClone, + unreadable === "start" ? "2026-10-07T10:45:00.000Z" : "2026-10-07T10:20:00.000Z" + ); + s.source.cloneSays(newClone, grantedBy(newClone)); + const unavailableClone = unreadable === "start" ? s.clone : newClone; + s.source.unreadableClones.add(cloneKey(unavailableClone)); + expect(await s.checker.permits(admission, at, start)).toBe("unknown"); + expect(await s.checker.permits(admission, start)).toBe( + unreadable === "start" ? "unknown" : "deny" + ); + expect(await s.checker.permits(admission, at)).toBe( + unreadable === "end" ? "unknown" : "deny" + ); + expect(s.ledger.events).toEqual(["lock", "unlock", "lock", "unlock", "lock", "unlock"]); + } + ); + it.each([ [{ ...admission, cwd: null }, at, undefined], [{ ...admission, cwd: "" }, at, undefined], diff --git a/cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts new file mode 100644 index 000000000..06e6afa71 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from "vitest"; +import { readSessionRepository } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.js"; + +describe("Codex native session repository", () => { + it.each([ + ["git@github.com:owner/repository.git", "https://github.com/owner/repository"], + ["https://github.com/owner/repository", "https://github.com/owner/repository"], + ["git@gitlab.com:group/subgroup/repo.git", "https://gitlab.com/group/subgroup/repo"], + ["https://gitlab.com/group/repo.git", "https://gitlab.com/group/repo"], + ])("publishes only the canonical supported identity for %s", async (originUrl, expected) => { + const calls: Array<[string, object]> = []; + const result = await readSessionRepository(async (method, params) => { + calls.push([method, params]); + return { thread: { id: "s", sessionId: "s", gitInfo: { originUrl } } }; + }, "s"); + expect(result).toEqual({ available: true, values: { "aidd.repository_url": expected } }); + expect(calls).toEqual([["thread/read", { threadId: "s", includeTurns: false }]]); + }); + it.each([ + "https://user:secret@github.com/owner/repo.git", + "https://github.com/owner/repo?token=secret", + "https://github.com/owner/repo#fragment", + "https://github.com:443/owner/repo", + "https://example.com/owner/repo", + "git@example.com:owner/repo", + "ssh://git@github.com/owner/repo", + "file:///owner/repo", + "https://github.com/owner/repo%2fextra", + "https://github.com/owner/../repo", + "https://github.com/owner/.", + "https://github.com/owner/repo/extra", + "https://github.com/owner/.git", + "https://github.com/owner/repo\n", + ])( + "clears a previous link without exposing an unsupported or unsafe remote %j", + async (originUrl) => { + expect( + await readSessionRepository( + async () => ({ thread: { id: "s", sessionId: "s", gitInfo: { originUrl } } }), + "s" + ) + ).toEqual({ available: true, values: {} }); + } + ); + it("clears only an explicitly absent native Git origin", async () => { + for (const gitInfo of [null, { originUrl: null }]) + expect( + await readSessionRepository( + async () => ({ thread: { id: "s", sessionId: "s", gitInfo } }), + "s" + ) + ).toEqual({ available: true, values: {} }); + for (const gitInfo of [undefined, {}]) + expect( + await readSessionRepository( + async () => ({ thread: { id: "s", sessionId: "s", gitInfo } }), + "s" + ) + ).toEqual({ available: false }); + }); + it.each([ + { id: "other", sessionId: "s", gitInfo: { originUrl: "git@github.com:a/b.git" } }, + { id: "s", sessionId: "other", gitInfo: { originUrl: "git@github.com:a/b.git" } }, + ])("rejects an unrelated thread", async (thread) => { + expect(await readSessionRepository(async () => ({ thread }), "s")).toEqual({ + available: false, + }); + }); + it("retains existing metadata on API errors", async () => { + expect( + await readSessionRepository(async () => { + throw new Error("unavailable"); + }, "s") + ).toEqual({ available: false }); + }); +}); diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts index 396b1fa67..e62de083c 100644 --- a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts @@ -27,6 +27,7 @@ it.skipIf(process.platform === "win32")( const marker = join(home, "called"); const received: string[] = []; let onReceived = (_body: string) => {}; + let responseFor = (_body: string) => 200; const server = createServer((request, response) => { let body = ""; request.on("data", (chunk: Buffer) => { @@ -35,7 +36,7 @@ it.skipIf(process.platform === "win32")( request.on("end", () => { received.push(body); onReceived(body); - response.writeHead(200).end("{}"); + response.writeHead(responseFor(body)).end("{}"); }); }); try { @@ -57,7 +58,8 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { if (message.method === 'initialize') fs.writeFileSync(process.env.AIDD_TEST_TITLE_MARKER, JSON.stringify(message.params)); const result = message.method === 'initialize' ? {} : message.method === 'project/read' ? { project: { id: 'native-project', name: 'Native project' } } : { thread: { id: message.params.threadId, sessionId: message.params.threadId, - name: 'Synthetic title', projectId: 'native-project', turns: [] } + name: 'Synthetic title', projectId: 'native-project', + gitInfo: { originUrl: 'git@github.com:example/repository.git' }, turns: [] } }; process.stdout.write(JSON.stringify({ id: message.id, result }) + '\n'); }); @@ -176,6 +178,22 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(received).toHaveLength(1); expect(received.join()).not.toContain("aidd.project_name"); + received.length = 0; + await syncOtel({ ...options, includeSessionRepositories: true }, resolve, async () => true); + expect(received).toHaveLength(2); + expect(received[1]).toContain("aidd.session_repository"); + expect(received[1]).toContain("https://github.com/example/repository"); + expect(received[1]).not.toContain("git@github.com"); + expect(received[1]).not.toContain("Synthetic title"); + received.length = 0; + await syncOtel( + { ...options, includeSessionRepositories: true }, + resolve, + async (transfer) => !JSON.stringify(transfer.payload).includes("aidd.session_repository") + ); + expect(received).toHaveLength(1); + expect(received.join()).not.toContain("aidd.repository_url"); + received.length = 0; const captureFile = join(captureDir, "codex.jsonl"); await appendFile(captureFile, `${JSON.stringify(capture)}\n`.repeat(10)); @@ -232,7 +250,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(result.sent).toBe(2); expect(result.identified).toBe(2); expect(result.currentConsenting).toBe(2); - expect(allowedChecks).toBe(3); + expect(allowedChecks).toBe(4); expect(received).toHaveLength(2); expect(received.some((body) => body.includes("session-1"))).toBe(true); expect(received.some((body) => body.includes("session-4"))).toBe(true); @@ -248,6 +266,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { ...options, includeSessionTitles: true, includeSessionProjects: true, + includeSessionRepositories: true, stateDir: join(home, "state"), }, async () => admissions, @@ -257,8 +276,9 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { ? "deny" : "allow" ); - expect(metadata.sent).toBe(4); + expect(metadata.sent).toBe(6); expect(received.filter((body) => body.includes("aidd.session_project"))).toHaveLength(2); + expect(received.filter((body) => body.includes("aidd.session_repository"))).toHaveLength(2); expect(received.join()).not.toContain("input_token_count"); expect(received.join()).not.toContain("aidd.feature"); await expect(readdir(join(home, "state"))).rejects.toMatchObject({ code: "ENOENT" }); @@ -308,7 +328,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { const abort = new AbortController(); const timeout = setTimeout(() => abort.abort(), 3000); onReceived = (body) => { - if (body.includes("aidd.session_title")) abort.abort(); + if (body.includes("aidd.session_repository")) abort.abort(); }; const admission = { feature: { state: "unbound" as const }, @@ -320,6 +340,7 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { ...options, stateDir: join(home, "state"), includeSessionTitles: true, + includeSessionRepositories: true, signal: abort.signal, intervalMs: 10, }, @@ -331,6 +352,46 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { await running; clearTimeout(timeout); expect(received.some((body) => body.includes("Synthetic title"))).toBe(true); + expect(received.some((body) => body.includes("https://github.com/example/repository"))).toBe( + true + ); + + received.length = 0; + let consenting = true; + await writeFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`, { + mode: 0o600, + }); + onReceived = () => {}; + responseFor = (body) => { + if (!body.includes("aidd.session_title")) return 200; + consenting = false; + return 503; + }; + await expect( + syncOtel( + { ...options, follow: true, intervalMs: 10, includeSessionTitles: true }, + resolve, + async () => consenting + ) + ).rejects.toThrow(/Consent is no longer confirmed/); + expect(received.filter((body) => body.includes("aidd.session_title"))).toHaveLength(1); + + received.length = 0; + consenting = true; + const retryAll = syncAllOtel( + { + ...options, + stateDir: join(home, "withdrawn-state"), + includeSessionTitles: true, + intervalMs: 10, + }, + async () => new Map([["session-1", admission]]), + async () => (consenting ? "allow" : "deny") + ); + await new Promise((done) => setTimeout(done, 100)); + await appendFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`); + await expect(retryAll).rejects.toThrow(/Consent is no longer confirmed/); + expect(received.filter((body) => body.includes("aidd.session_title"))).toHaveLength(1); } finally { process.env.PATH = oldPath; if (oldMarker === undefined) delete process.env.AIDD_TEST_TITLE_MARKER; diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts index 656de1a88..810e6bd69 100644 --- a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts @@ -193,6 +193,230 @@ it("retries a transient HTTP failure without losing its checkpoint", async () => expect(result.sent).toBe(1); }); +it("retains an unsent capture across consent withdrawal and resumes only after consent returns", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-withdrawn-all-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let consenting = true; + let calls = 0; + const abort = new AbortController(); + server = createServer((request, response) => { + request.resume(); + request.on("end", () => { + calls++; + if (calls === 1) { + consenting = false; + response.writeHead(503).end("{}"); + } else { + response.writeHead(200).end("{}"); + abort.abort(); + } + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const snapshot = async () => new Map([["allowed", admission]]); + const historical = new Date(1_793_763_900_000).toISOString(); + const permits = async (_admission: unknown, at: string, _start?: string, current?: string) => + at === historical && (!current || consenting) ? ("allow" as const) : ("deny" as const); + const first = syncAllOtel(options, snapshot, permits); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await expect(first).rejects.toThrow(/Consent is no longer confirmed/); + expect(calls).toBe(1); + consenting = true; + const result = await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + expect(result.sent).toBe(1); + expect(calls).toBe(2); +}); + +it.each(["missing", "cwd", "repository", "unavailable"] as const)( + "does not retry if the session admission is %s after a failed post", + async (change) => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-admission-retry-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let changed = false; + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + request.on("end", () => { + changed = true; + response.writeHead(503).end("{}"); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }, + async () => { + if (changed && change === "unavailable") throw new Error("Admission unavailable"); + if (changed && change === "missing") return new Map(); + return new Map([ + [ + "allowed", + { + ...admission, + cwd: changed && change === "cwd" ? "/elsewhere" : admission.cwd, + repositoryId: changed && change === "repository" ? "other" : admission.repositoryId, + }, + ], + ]); + }, + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await expect(running).rejects.toThrow(/Consent is no longer confirmed|Admission unavailable/); + expect(calls).toBe(1); + } +); + +it("does not use an outside-consent identity for a later feature binding", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-unconsented-all-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + response.writeHead(200).end("{}"); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const abort = new AbortController(); + const admission = { + feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }, + cwd: "/project", + repositoryId: "repo", + }; + const historical = new Date(1_793_763_900_000).toISOString(); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + signal: abort.signal, + }, + async () => new Map([["allowed", admission]]), + async (_admission, at) => (at === historical ? "deny" : "allow") + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 100)); + abort.abort(); + const result = await running; + expect(result).toMatchObject({ sent: 0, denied: 1 }); + expect(calls).toBe(0); +}); + +it("does not reuse legacy checkpoint identity until a new admitted observation supplies it", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-legacy-identity-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + let abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + if (body.includes("aidd.session_feature")) abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }, + cwd: "/project", + repositoryId: "repo", + }; + const snapshot = async () => new Map([["allowed", admission]]); + const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await first; + expect(received.some((body) => body.includes("aidd.session_feature"))).toBe(true); + const directory = join(home, "state", "otel-sync"); + const stateFile = (await readdir(directory)).find( + (name) => name.endsWith(".json") && !name.includes("checkpoint") + ); + if (!stateFile) throw new Error("Missing binding state"); + const path = join(directory, stateFile); + const legacy = JSON.parse(await readFile(path, "utf8")); + delete legacy.identityConsentChecked; + await writeFile(path, JSON.stringify(legacy), { mode: 0o600 }); + received.length = 0; + abort = new AbortController(); + const restart = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(received).toHaveLength(0); + await appendFile(capture, line("allowed")); + await restart; + expect(received.filter((body) => body.includes("aidd.session_feature"))).toHaveLength(1); + expect(received.some((body) => body.includes("developer@example.test"))).toBe(true); +}); + it("does not send an event outside the clone's consent interval", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-consent-")); dirs.push(home); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts index 359434a7c..50eda0175 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -80,6 +80,85 @@ function select(line: string) { return selectTransfers(line, options, { values: new Map(), conflict: false }); } +test("forwards each Codex effort only on the event that reported it", () => { + const start = capture([ + text("model", "gpt-test"), + text("reasoning_effort", "high"), + text("model_reasoning_effort", "private-wrong-event"), + ]).replace("codex.sse_event", "codex.conversation_starts"); + const response = capture([ + text("model", "gpt-test"), + text("reasoning_effort", "private-wrong-event"), + text("model_reasoning_effort", "xhigh"), + ]); + const [first] = select(start); + const [second] = select(response); + const startPayload = JSON.stringify(first.payload); + const responsePayload = JSON.stringify(second.payload); + expect(startPayload).toContain('"key":"reasoning_effort","value":{"stringValue":"high"}'); + expect(responsePayload).toContain( + '"key":"model_reasoning_effort","value":{"stringValue":"xhigh"}' + ); + for (const payload of [startPayload, responsePayload]) { + expect(payload).toContain('"key":"model","value":{"stringValue":"gpt-test"}'); + expect(payload).toContain('"key":"event.timestamp"'); + expect(payload).not.toContain("private-wrong-event"); + } +}); + +test("leaves the original usage transfer unchanged apart from the observed effort", () => { + const common = [text("model", "gpt-test"), text("input_token_count", "42")]; + const [without] = select(capture(common)); + const [withEffort] = select(capture([...common, text("model_reasoning_effort", "high")])); + const expected = JSON.stringify(without.payload); + const actual = JSON.stringify(withEffort.payload).replace( + ',{"key":"model_reasoning_effort","value":{"stringValue":"high"}}', + "" + ); + expect(actual).toBe(expected); +}); + +test("does not infer effort from a resource, another event, or a missing value", () => { + const resource = [text("reasoning_effort", "high"), text("model_reasoning_effort", "high")]; + const [transfer] = select(capture([text("model", "gpt-test")], resource)); + expect(JSON.stringify(transfer.payload)).not.toContain("reasoning_effort"); + const [tool] = select( + capture([text("model_reasoning_effort", "high")]).replace( + "codex.sse_event", + "codex.tool_result" + ) + ); + expect(JSON.stringify(tool.payload)).not.toContain("reasoning_effort"); + for (const resourceModel of [[], [text("model", "gpt-test")]]) { + const [missing] = select(capture([text("model_reasoning_effort", "high")], resourceModel)); + expect(JSON.stringify(missing.payload)).not.toContain("model_reasoning_effort"); + } +}); + +test("preserves bounded custom effort identifiers and omits free text", () => { + const [custom] = select( + capture([text("model", "gpt-test"), text("model_reasoning_effort", "model_defined-2")]) + ); + expect(JSON.stringify(custom.payload)).toContain('"stringValue":"model_defined-2"'); + for (const invalid of ["", "high with spaces", "x".repeat(33), "high\nsecret"]) { + const [transfer] = select( + capture([text("model", "gpt-test"), text("model_reasoning_effort", invalid)]) + ); + expect(JSON.stringify(transfer.payload)).not.toContain("model_reasoning_effort"); + expect(JSON.stringify(transfer.payload)).not.toContain(invalid || '"stringValue":""'); + } +}); + +test("rejects contradictory effort values and omits non-text values", () => { + expect(() => + select(capture([text("model_reasoning_effort", "high"), text("model_reasoning_effort", "low")])) + ).toThrow("Conflicting duplicate OTLP attributes."); + const [nonText] = select( + capture([text("model", "gpt-test"), attribute("model_reasoning_effort", { intValue: "1" })]) + ); + expect(JSON.stringify(nonText.payload)).not.toContain("model_reasoning_effort"); +}); + function metricCapture( point: Attributes, resource: Attributes = [], @@ -118,6 +197,15 @@ function metricCapture( }); } +test("does not forward Codex effort fields on Claude metrics", () => { + const [transfer] = selectTransfers( + metricCapture([text("type", "input"), text("reasoning_effort", "high")]), + { ...options, config: "claude" }, + { values: new Map(), conflict: false } + ); + expect(JSON.stringify(transfer.payload)).not.toContain("reasoning_effort"); +}); + test.each(["point", "resource", "scope"])("rejects contradictory Claude %s attributes", (level) => { const conflict = [text("type", "input"), text("type", "output")]; expect(() => diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts index 05bb1978a..6d954d8dc 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -542,7 +542,8 @@ test.each([ await expect( send( { ...options, endpoint: `http://127.0.0.1:${address.port}`, follow: true }, - { signal, payload: {} } + { signal, payload: {} }, + async () => true ) ).rejects.toThrow(/partial/i); expect(calls).toBe(1); @@ -566,7 +567,8 @@ test.each([ await expect( send( { ...options, endpoint: `http://127.0.0.1:${address.port}` }, - { signal: "logs", payload: {} } + { signal: "logs", payload: {} }, + async () => true ) ).rejects.toThrow(/response/i); }); @@ -588,7 +590,8 @@ test("accepts a warning with zero rejected records", async () => { await expect( send( { ...options, endpoint: `http://127.0.0.1:${address.port}` }, - { signal: "logs", payload: {} } + { signal: "logs", payload: {} }, + async () => true ) ).resolves.toBeUndefined(); }); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts index 8d9d47275..8ff1f961a 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts @@ -90,6 +90,8 @@ describe("OTLP capture sync", () => { dirs.push(home); const directory = join(home, ".local/share/aidd-otel-observation"); await mkdir(directory, { recursive: true, mode: 0o700 }); + const wanted = record("wanted", "codex.sse_event"); + wanted.attributes.push(attr("model", "gpt-test"), attr("model_reasoning_effort", "high")); const line = { signal: "logs", payload: { @@ -98,7 +100,7 @@ describe("OTLP capture sync", () => { scopeLogs: [ { logRecords: [ - record("wanted", "codex.sse_event"), + wanted, record("other", "codex.sse_event"), record("wanted", "codex.user_prompt"), ], @@ -125,16 +127,17 @@ describe("OTLP capture sync", () => { const address = server.address(); if (address === null || typeof address === "string") throw new Error("No test port"); const before = await readFile(file, "utf8"); + const syncOptions = { + config: "codex" as const, + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + }; const result = await syncOtel( - { - config: "codex", - session: "wanted", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - follow: false, - }, + syncOptions, async () => ({ state: "unbound" }), async () => true ); @@ -144,8 +147,17 @@ describe("OTLP capture sync", () => { expect(received[0].auth).toBe("Bearer test-token"); expect(received[0].body).not.toContain("sensitive output"); expect(received[0].body).toContain("toolu_123"); + expect(received[0].body).toContain('"key":"model_reasoning_effort"'); + expect(received[0].body).toContain('"stringValue":"high"'); expect(received[0].body).not.toContain('"other"'); expect(await readFile(file, "utf8")).toBe(before); + const denied = await syncOtel( + syncOptions, + async () => ({ state: "unbound" }), + async () => false + ); + expect(denied.skipped).toBe(1); + expect(received).toHaveLength(1); }); test("refuses a non-loopback destination before reading capture or sending credentials", async () => { @@ -261,4 +273,86 @@ describe("OTLP capture sync", () => { expect(received).toBe(2); expect(result.sent).toBe(2); }); + + it("never uses an identity from an observation outside consent to publish a binding", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-unconsented-identity-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const source = { + signal: "logs", + payload: { + resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }], + }, + }; + await writeFile(join(directory, "codex.jsonl"), `${JSON.stringify(source)}\n`, { mode: 0o600 }); + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + response.writeHead(200).end("{}"); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const result = await syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: false, + }, + async () => ({ state: "confirmed", feature: "owner/repo#7", branch: "feature/work" }), + async (transfer) => Object.keys(transfer.payload).length === 0 + ); + expect(result).toMatchObject({ sent: 0, skipped: 1 }); + expect(calls).toBe(0); + }); + + it("stops before retrying a failed post after consent is withdrawn", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-withdrawn-retry-")); + dirs.push(home); + const directory = join(home, ".local/share/aidd-otel-observation"); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const source = { + signal: "logs", + payload: { + resourceLogs: [{ scopeLogs: [{ logRecords: [record("wanted", "codex.sse_event")] }] }], + }, + }; + await writeFile(join(directory, "codex.jsonl"), `${JSON.stringify(source)}\n`, { mode: 0o600 }); + let consenting = true; + let calls = 0; + server = createServer((request, response) => { + calls++; + request.resume(); + request.on("end", () => { + consenting = false; + response.writeHead(503).end("{}"); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + await expect( + syncOtel( + { + config: "codex", + session: "wanted", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + follow: true, + intervalMs: 10, + }, + async () => ({ state: "unbound" }), + async (transfer) => !transfer.consentAt || consenting + ) + ).rejects.toThrow(/Consent is no longer confirmed/); + expect(calls).toBe(1); + }); }); diff --git a/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts b/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts index 4254ebb04..032860d32 100644 --- a/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts +++ b/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts @@ -237,15 +237,15 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { expect(revoked.exitCode, revoked.stderr).toBe(0); expect(received).toHaveLength(0); - // The historical event remains permitted; the event after opt-out is not sent. + // No historical or new event is sent while current consent is off. await appendFile( join(captureDir, "codex.jsonl"), `${JSON.stringify(capture(new Date().toISOString(), "999"))}\n` ); const afterOff = await sync({}, false); - expect(afterOff.exitCode, afterOff.stderr + afterOff.stdout).toBe(0); - expect(received).toHaveLength(1); - expect(logAttributes(received[0].body).input_token_count).toBe("10"); + expect(afterOff.exitCode).not.toBe(0); + expect(afterOff.stderr + afterOff.stdout).toContain("Consent is no longer confirmed"); + expect(received).toHaveLength(0); } finally { await new Promise<void>((resolve) => server.close(() => resolve())); await cleanup(); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index e76184d7f..cf7185199 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -187,7 +187,7 @@ { "invocation": "aidd telemetry otel sync", "exitCode": 0, - "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to\n discover sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default:\n false)\n --include-session-titles Send Codex chat titles, which may contain prompt\n text (default: false)\n --include-session-projects Send native Codex project IDs and names (may\n contain sensitive text) (default: false)\n --titles-only Send only Codex titles and opted-in projects for\n consented sessions (default: false)\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow\n to discover sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default:\n false)\n --include-session-titles Send Codex chat titles, which may contain\n prompt text (default: false)\n --include-session-projects Send native Codex project IDs and names (may\n contain sensitive text) (default: false)\n --include-session-repositories Send verified Codex GitHub/GitLab repository\n URLs (may identify private work) (default:\n false)\n --titles-only Send only Codex titles and opted-in metadata\n for consented sessions (default: false)\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index b936c5ea8..5fabebf08 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -14,6 +14,7 @@ The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 | Session | `session-abc` | `session.id` | `conversation.id`; trace `thread.id` | **v1:** `info.sessionID`, `part.sessionID`; **v2:** `data.sessionID`, event `durable.aggregateID` | | Provider | `openai` | `anthropic.provider`, `gen_ai.provider.name`, `gen_ai.system` | `provider_name`, trace `provider` | **v1:** assistant `info.providerID`, user `info.model.providerID`; **v2:** `data.model.providerID` on step start | | Model | `model-xyz` | `model`, `gen_ai.request.model` | `model` | **v1:** assistant `info.modelID`, user `info.model.modelID`; **v2:** `data.model.id` on step start | +| Reasoning effort | `high` | Not verified | `reasoning_effort` on `codex.conversation_starts`; `model_reasoning_effort` on `response.completed` | Not verified | | User input | `Fix the test` | `prompt`, `prompt_text`, span `user_prompt` | `prompt` | **v1:** `part.text` joined by `messageID` to a user message; **v2:** `data.item.payload.text` on `session.inbox.enqueued` with `data.item.type=user` | | Assistant response | `The test is fixed.` | `response` | `response` | **v1:** `part.text` joined by `messageID` to an assistant message; **v2:** `data.text` on `session.text.ended`, `data.delta` on `session.text.delta` | | Input tokens | `1,200` | `input_tokens`; `claude_code.token.usage` metric, `type=input` | `input_token_count`, `codex.turn.token_usage.input_tokens`, `gen_ai.usage.input_tokens`; metric `token_type=input` | **v1:** `info.tokens.input` or `part.tokens.input`; **v2:** `data.tokens.input` | @@ -31,6 +32,7 @@ Provider labels are not necessarily equivalent across tools. A user role does no ## Interpretation and coverage - The Claude and Codex metric `type` / `token_type` labels above belong to `claude_code.token.usage` and `codex.turn.token_usage`, respectively. OTLP logs, traces and metrics may describe the same operation; do not add them together. +- Codex effort is an observation on one event, not a value inherited by other turns. The sync forwards only bounded ASCII identifiers when the same event also names its model; a missing or invalid value stays unknown. [Codex event source](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/otel/src/events/session_telemetry.rs) (read 2026-11-05). - OpenCode 2 emits `data.tokens.*` both on `session.step.ended` and `session.usage.updated`. The latter is cumulative; summing every update overcounts. Its `input` is non-cached input and `output` is visible output; reasoning is separate. The implementation maps missing provider usage to zero, so a reported zero does not prove zero usage. - Claude's `input_tokens` excludes cache reads and writes. Normalize token definitions before comparison; do not assume a similarly named field counts the same thing in all tools. - In the examined capture, Codex cache-write fields were present but all zero. A nonzero write was not verified. Not every event includes every field; OpenCode 2 text and model data may need correlation across events. Outputs can be truncated. @@ -49,8 +51,9 @@ The collector keeps the provider's raw OTLP JSON locally. The separate dev sync - **Both:** available event time, session, model, application version, provider email/account/organization IDs, tool name, call ID, outcome and duration. Confirmed framework feature bindings additionally carry the declared branch and ticket; an invalidation can be sent when a binding becomes ambiguous. - **Codex titles:** only with `--include-session-titles`, the free-text chat title is sent as session metadata. It may contain part of a prompt. - **Codex native projects:** `--include-session-projects` separately opts into sending the native project ID and free-text name as `aidd.session_project`. Names can contain sensitive text. The association must come from `thread.projectId`, then an exact `project/read` result; a second thread read checks membership has not changed. No path, repository, title, parent-thread inheritance or prompt fallback is used. A confirmed `projectId: null` clears the association; missing fields, API failures and inconsistent IDs send nothing. Clone consent is checked again immediately before sending. +- **Codex Git repositories:** `--include-session-repositories` separately opts into sending the repository identity from the exact native thread's `gitInfo.originUrl` as `aidd.session_repository` / `aidd.repository_url`. Only canonical HTTPS links for public GitHub and GitLab hosts are sent; repository names can disclose private work. Credentials, local paths, other hosts and unsafe URLs are never forwarded. A confirmed null or unsupported Git origin clears the association; missing fields and API failures send nothing. Consent is checked again immediately before sending. [Codex app-server thread/read](https://learn.chatgpt.com/docs/app-server#read-a-stored-thread-without-resuming) (read 2026-11-05). -Use `telemetry otel sync --config=codex --tenant=<slug> --endpoint=<loopback-origin> --titles-only --include-session-projects` for one metadata-only pass over consented captured sessions (both titles and projects are opted in). It sends no token metrics or feature bindings and writes no capture checkpoint. `--follow --include-session-projects` checks sessions loaded by the regular follower; it does not discover every historical chat. Native projects are separate from Git repositories. Claude Code has no universal native project field in this bridge; OpenCode project export is not integrated. +Use `telemetry otel sync --config=codex --tenant=<slug> --endpoint=<loopback-origin> --titles-only --include-session-projects --include-session-repositories` for one metadata-only pass over consented captured sessions (titles, projects and repositories are opted in). Omit either metadata option to exclude it. It sends no token metrics or feature bindings and writes no capture checkpoint. `--follow` checks sessions loaded by the regular follower; it does not discover every historical chat. Native projects are separate from Git repositories. Claude Code has no universal native project field in this bridge; OpenCode project export is not integrated. Codex's project API is experimental. In the local 0.166.0 probe on 2026-11-05, `project/read` returned an exact ID and nonempty native name, but all ten sampled thread reads returned `projectId: null`. A separate Desktop tool listed five chats with explicit project IDs matching its project list; app-server returned `projectId: null` for those same five chats. An independent CLI's access to that Desktop source is not established. Named project export is tested against a simulated app-server, not demonstrated end to end on those local sessions. [Official project protocol](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/app-server-protocol/src/protocol/v2/project.rs) (read 2026-11-05). @@ -122,6 +125,8 @@ The origin must be loopback HTTP; no token or prompt content is printed. Only su An automatic follower starts with new capture events and keeps a private checkpoint. Network errors and HTTP 5xx responses are reported and retried with bounded backoff; permanent HTTP failures stop the follower. Restarting resumes from the checkpoint, and a failed post is replayed, so the receiver must remain idempotent. Unknown sessions wait for their transcript to be admitted using private byte-range indexes, without holding up other sessions. These indexes can grow until admission and depend on retaining the original capture; replaced or truncated captures require recovery rather than silently skipping data. Running a follower for each provider is an explicit operation; it does not start automatically after a machine restart. Linked session identifiers and the minimum provider identity are stored privately to detect later ambiguity, including after restart. Older sync state without identity must be refreshed with a session replay. +Every delivery attempt, including a retry, rechecks session admission and the same clone's current consent and original consent interval. If consent or admission cannot be confirmed, sync stops before the next POST and leaves unsent capture data at its previous checkpoint. An explicit restart after renewed consent can replay observations whose original intervals remain authorized for that same clone. Session identity is collected only from admitted observations. Identity saved by older sync versions has no reliable consent provenance and is ignored until a new admitted observation or an explicit session replay supplies it; links and metadata may remain unavailable in the meantime. + To replay one consented session already in the capture, pass `--session=<id>` without `--follow`. Add `--follow` to follow only that session. A feature link is sent only when every recorded call agrees on a single observed branch and a declared canonical ticket such as `owner/repo#123`; otherwise it remains unlinked. If a linked session later becomes ambiguous, an irreversible invalidation keeps it unlinked even if events arrive out of order. ## Sources diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 6beab39ac..0a962eeb7 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -92,3 +92,5 @@ Configure an exporter with `aidd telemetry otel init --config=<claude|codex|open Start OpenCode v1 with `aidd telemetry otel run --config=opencode` to supply its endpoint and token; pass OpenCode arguments after `--`. For OpenCode v2, restart its service after initialization. Native Codex project metadata requires the separate `--include-session-projects` opt-in. It transmits the verified native project ID and name, session/provider identity and observation time; names may contain sensitive text. No path-based project inference is made. See [the observation contract](OBSERVABILITY.md) for consent, absence handling and the metadata-only command. + +Codex Git repository metadata requires its own `--include-session-repositories` opt-in. It sends a verified, sanitized GitHub or GitLab repository link, session/provider identity and observation time. Repository names can reveal private work. Local paths, credentials and unsupported hosts are not sent. Repositories and native Codex projects are distinct. From a82e5de06f66f3488434ee318d69a456ed982a4a Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 15:26:07 +0800 Subject: [PATCH 25/56] fix(telemetry): isolate revoked sessions and verify identity provenance --- cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 1 + .../infrastructure/otel/actions/sync-all.ts | 194 ++++++++++-- .../infrastructure/otel/actions/sync.ts | 168 +++++++++- cli/src/presentation/commands/telemetry.ts | 9 +- .../session-title-sync.integration.test.ts | 27 +- .../otel/sync-all-otel.integration.test.ts | 294 +++++++++++++++++- .../sync-otel-integrity.integration.test.ts | 28 ++ .../otel/sync-otel.integration.test.ts | 1 + .../e2e/telemetry-otel-collector.e2e.test.ts | 14 +- plugins/aidd-telemetry/OBSERVABILITY.md | 2 +- 11 files changed, 674 insertions(+), 66 deletions(-) diff --git a/cli/package.json b/cli/package.json index 73adbc8cd..f99446faa 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 813, + "bundleBudgetKB": 819, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index a79740a9a..5da5a1d7c 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -66,6 +66,7 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // +0.53 KB over the preceding 806.96 KB build, less than 0.1% measured headroom. // 810 KB: 809.79 KB with separately opted-in, sanitized Codex repository metadata. // 813 KB: 812.66 KB with consent and session-admission checks on every delivery attempt. +// 819 KB: 818.39 KB with per-identifier provenance and isolated pending session ranges. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts index 2f70a816c..62635372b 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -16,10 +16,15 @@ import { createSessionProjectPublisher } from "./metadata/session-project.js"; import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { - admitTransferIdentity, + admissionFingerprint, + type BindingDetails, bindingTransfer, + ConsentNotConfirmedError, destination, + evidenceTransfers, type Identity, + type IdentityEvidence, + observeIdentity, packTransfers, readBindingDetails, type SyncOptions, @@ -39,9 +44,16 @@ type Snapshot = ReadonlyMap<string, SessionAdmission>; interface Pending { readonly offset: number; readonly length: number; + /** This range was admitted but could not be delivered; preserve it even if later denied. */ + readonly hold?: boolean; + readonly admissionKey?: string; } interface SessionState { readonly identity: Identity; + readonly identityEvidence: IdentityEvidence[]; + identityAdmissionKey: string | null; + previousLinkUnverified: boolean; + savedIdentity: BindingDetails | null; mayHaveLinked: boolean; binding: string; needsReplay: boolean; @@ -64,6 +76,9 @@ function key(options: AllOptions, session: string) { function asOptions(options: AllOptions, session: string): SyncOptions { return { ...options, session, follow: true }; } +function admissionKey(admission: SessionAdmission): string { + return admissionFingerprint(admission); +} async function privateDirectory(path: string) { await mkdir(path, { recursive: true, mode: 0o700 }); const info = await lstat(path); @@ -118,6 +133,11 @@ async function saveCheckpoint(path: string, value: Checkpoint) { } } +function pendingLine(item: Pending): string { + if (item.hold && !item.admissionKey) throw new Error("Sync pending admission is unavailable."); + return `${item.offset} ${item.length}${item.hold ? ` hold ${item.admissionKey}` : ""}\n`; +} + /** A pending journal contains only byte ranges, not source text or provider identities. */ async function appendPending(path: string, item: Pending) { const file = await open( @@ -130,7 +150,7 @@ async function appendPending(path: string, item: Pending) { if (!info.isFile() || (info.mode & 0o077) !== 0) throw new Error("Sync pending journal must be a private regular file."); if (info.size > 0) { - const length = Math.min(info.size, 64); + const length = Math.min(info.size, 256); const tail = Buffer.alloc(length); const { bytesRead } = await file.read(tail, 0, length, info.size - length); if (bytesRead !== length) throw new Error("Sync pending journal changed unexpectedly."); @@ -141,7 +161,7 @@ async function appendPending(path: string, item: Pending) { await file.truncate(last < 0 ? 0 : info.size - length + last + 1); } } - await file.writeFile(`${item.offset} ${item.length}\n`); + await file.writeFile(pendingLine(item)); await file.sync(); } finally { await file.close(); @@ -159,7 +179,7 @@ async function pendingRanges(path: string): Promise<Pending[]> { const lines = contents.split("\n"); lines.pop(); // A crash can leave an incomplete last append; its source line is replayed. for (const line of lines) { - const parts = /^(\d+) (\d+)$/.exec(line); + const parts = /^(\d+) (\d+)(?: hold ([a-f0-9]{64}))?$/.exec(line); if (!parts) throw new Error("Sync pending journal is invalid."); const offset = Number(parts[1]); const length = Number(parts[2]); @@ -170,7 +190,16 @@ async function pendingRanges(path: string): Promise<Pending[]> { length > MAX_LINE ) throw new Error("Sync pending journal is invalid."); - ranges.set(line, { offset, length }); + const rangeKey = `${offset} ${length}`; + const previous = ranges.get(rangeKey); + if (previous?.admissionKey && parts[3] && previous.admissionKey !== parts[3]) + throw new Error("Sync pending journal has conflicting admissions."); + ranges.set(rangeKey, { + offset, + length, + hold: !!parts[3] || previous?.hold, + admissionKey: parts[3] ?? previous?.admissionKey, + }); } return [...ranges.values()]; } finally { @@ -187,7 +216,7 @@ async function replacePending(path: string, ranges: readonly Pending[]) { try { const file = await open(temporary, "wx", 0o600); try { - await file.writeFile(ranges.map((item) => `${item.offset} ${item.length}\n`).join("")); + await file.writeFile(ranges.map(pendingLine).join("")); await file.sync(); } finally { await file.close(); @@ -290,9 +319,13 @@ export async function syncAllOtel( throw new Error("Sync binding state does not match its session."); const state = { identity: { - values: new Map<string, string>(saved?.details?.identity ?? []), + values: new Map<string, string>(), conflict: saved?.details?.conflict === true, }, + identityEvidence: [] as IdentityEvidence[], + identityAdmissionKey: null, + previousLinkUnverified: saved?.mayHaveLinked === true, + savedIdentity: saved?.details ?? null, mayHaveLinked: saved?.mayHaveLinked === true, binding: saved?.details?.binding ?? "", needsReplay: Boolean( @@ -314,9 +347,13 @@ export async function syncAllOtel( throw new Error("Sync binding state does not match its session."); sessions.set(id, { identity: { - values: new Map(saved.details.identity), + values: new Map(), conflict: saved.details.conflict === true, }, + identityEvidence: [], + identityAdmissionKey: null, + previousLinkUnverified: saved.mayHaveLinked, + savedIdentity: saved.details, mayHaveLinked: saved.mayHaveLinked, binding: saved.details.binding, needsReplay: saved.mayHaveLinked && saved.details.binding !== "invalidated", @@ -338,15 +375,63 @@ export async function syncAllOtel( return false; return true; }; + const restoreIdentity = async ( + id: string, + admission: SessionAdmission, + state: SessionState + ) => { + const saved = state.savedIdentity; + const proof = saved?.identityProof; + if ( + !saved || + !proof || + proof.admissionKey !== admissionKey(admission) || + !(await stillPermitted(id, admission, evidenceTransfers(proof.sources))) + ) + return; + state.identity.values.clear(); + for (const [name, value] of saved.identity) state.identity.values.set(name, value); + state.identity.conflict ||= saved.conflict === true; + state.identityEvidence.splice(0, state.identityEvidence.length, ...proof.sources); + state.identityAdmissionKey = proof.admissionKey; + state.previousLinkUnverified = false; + state.savedIdentity = null; + }; + const stateProof = (state: SessionState): BindingDetails["identityProof"] => + state.identityAdmissionKey && state.identityEvidence.length + ? { version: 1, admissionKey: state.identityAdmissionKey, sources: state.identityEvidence } + : undefined; + const deliver = async (id: string, transfer: Transfer, guard: () => Promise<boolean>) => { + try { + await send(asOptions(options, id), transfer, guard); + return true; + } catch (error) { + if (error instanceof ConsentNotConfirmedError) return false; + throw error; + } + }; const process = async ( id: string, items: readonly Transfer[], admission: SessionAdmission, - state: SessionState - ): Promise<boolean> => { + state: SessionState, + hold = false + ): Promise<"resolved" | "unknown" | "hold"> => { let wait = false; const allowed: Transfer[] = []; const verdicts = new Map<string, Consent>(); + await restoreIdentity(id, admission, state); + const currentAdmissionKey = admissionKey(admission); + if ( + state.identityEvidence.length > 0 && + (state.identityAdmissionKey !== currentAdmissionKey || + !(await stillPermitted(id, admission, evidenceTransfers(state.identityEvidence)))) + ) { + if (state.mayHaveLinked) state.previousLinkUnverified = true; + state.identity.values.clear(); + state.identityEvidence.length = 0; + state.identityAdmissionKey = null; + } for (const transfer of items) { if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { denied++; @@ -359,13 +444,16 @@ export async function syncAllOtel( verdicts.set(stamp, verdict); } if (verdict === "unknown") wait = true; - else if (verdict === "deny") denied++; - else { + else if (verdict === "deny") { + if (hold) wait = true; + else denied++; + } else { allowed.push(transfer); - admitTransferIdentity(transfer, asOptions(options, id), state.identity); + observeIdentity(transfer, asOptions(options, id), state.identity, state.identityEvidence); + if (state.identityEvidence.length) state.identityAdmissionKey = currentAdmissionKey; } } - if (state.identity.conflict) + if (state.identity.conflict && !state.previousLinkUnverified) await saveBindingState( join(directory, `${key(options, id)}.json`), directory, @@ -374,21 +462,22 @@ export async function syncAllOtel( identity: [...state.identity.values], binding: state.binding, conflict: true, + identityProof: stateProof(state), }, state.mayHaveLinked ); for (const batch of packTransfers(allowed)) { - await send(asOptions(options, id), batch.transfer, () => - stillPermitted(id, admission, batch.items) - ); + if (!(await deliver(id, batch.transfer, () => stillPermitted(id, admission, batch.items)))) + return "hold"; sent += batch.count; } - return wait; + return wait ? (hold ? "hold" : "unknown") : "resolved"; }; const bindings = async () => { for (const [id, state] of sessions) { const admission = held.get(id); - if (!admission) continue; + if (admission) await restoreIdentity(id, admission, state); + if (!admission || state.previousLinkUnverified) continue; const feature: SessionFeature = admission.feature; const next = feature.state === "confirmed" ? `${feature.branch}\0${feature.feature}` : feature.state; @@ -409,15 +498,24 @@ export async function syncAllOtel( state: "ambiguous", }); if (event) { - await send(asOptions(options, id), event, () => - stillPermitted(id, admission, [event], feature) - ); + if ( + !(await deliver(id, event, () => + stillPermitted( + id, + admission, + [event, ...evidenceTransfers(state.identityEvidence)], + feature + ) + )) + ) + continue; sent++; state.binding = "invalidated"; await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { session: id, identity: [...state.identity.values], binding: state.binding, + identityProof: stateProof(state), }); } } else if ( @@ -431,11 +529,20 @@ export async function syncAllOtel( session: id, identity: [...state.identity.values], binding: next, + identityProof: stateProof(state), }); state.mayHaveLinked = true; - await send(asOptions(options, id), event, () => - stillPermitted(id, admission, [event], feature) - ); + if ( + !(await deliver(id, event, () => + stillPermitted( + id, + admission, + [event, ...evidenceTransfers(state.identityEvidence)], + feature + ) + )) + ) + continue; sent++; state.binding = next; state.needsReplay = false; @@ -447,6 +554,7 @@ export async function syncAllOtel( if (!titles && !projects && !repositories) return; for (const [id, state] of sessions) { const admission = held.get(id); + if (admission) await restoreIdentity(id, admission, state); if (!admission || state.identity.conflict) continue; if ((await permits(admission, new Date().toISOString())) !== "allow") continue; for (const publisher of [titles, projects, repositories]) @@ -454,10 +562,9 @@ export async function syncAllOtel( publisher && (await publisher.publish(id, state.identity, async (event) => { if (!event.at || (await permits(admission, event.at)) !== "allow") return false; - await send(asOptions(options, id), event, () => - stillPermitted(id, admission, [event]) + return deliver(id, event, () => + stillPermitted(id, admission, [event, ...evidenceTransfers(state.identityEvidence)]) ); - return true; })) ) sent++; @@ -471,6 +578,10 @@ export async function syncAllOtel( const state = await sessionState(id); const waiting: Pending[] = []; for (const item of await pendingRanges(path)) { + if (item.hold && item.admissionKey !== admissionKey(admission)) { + waiting.push(item); + continue; + } const raw = Buffer.alloc(item.length); const read = await file?.read(raw, 0, item.length, item.offset); if (!read || read.bytesRead !== item.length || raw.at(-1) !== 10) @@ -479,7 +590,14 @@ export async function syncAllOtel( values: new Map(), conflict: false, }); - if (await process(id, transfers, admission, state)) waiting.push(item); + const status = await process(id, transfers, admission, state, item.hold); + if (status !== "resolved") + waiting.push({ + ...item, + hold: status === "hold", + admissionKey: + status === "hold" ? (item.admissionKey ?? admissionKey(admission)) : undefined, + }); } await replacePending(path, waiting); } @@ -509,10 +627,20 @@ export async function syncAllOtel( conflict: false, }); const admission = held.get(id); - const waiting = - items.length > 0 && (!admission || (await process(id, items, admission, state))); - if (waiting) - await appendPending(pendingPath(id), { offset: position, length: newline + 1 }); + const status = + items.length === 0 + ? "resolved" + : admission + ? await process(id, items, admission, state) + : "unknown"; + if (status !== "resolved") + await appendPending(pendingPath(id), { + offset: position, + length: newline + 1, + hold: status === "hold", + admissionKey: + status === "hold" && admission ? admissionKey(admission) : undefined, + }); } offset += newline + 1; pendingBytes = pendingBytes.subarray(newline + 1); diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index e3c439ec9..3f377a8c5 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -33,6 +33,7 @@ export interface SyncOptions { readonly onRepositoryUnavailable?: () => void; readonly onTitleUnavailable?: () => void; readonly titlesOnly?: boolean; + readonly admission?: { readonly cwd: string; readonly repositoryId: string }; } type RecordObject = Record<string, unknown>; @@ -120,6 +121,30 @@ export interface Identity { conflict: boolean; } +export interface IdentityEvidence { + readonly keys: string[]; + readonly at: string; + readonly start?: string; +} + +export function admissionFingerprint(admission: { + cwd: string | null; + repositoryId: string | null; +}): string { + return createHash("sha256") + .update(JSON.stringify([admission.cwd, admission.repositoryId])) + .digest("hex"); +} + +export function evidenceTransfers(evidence: readonly IdentityEvidence[]): Transfer[] { + return evidence.map(({ at, start }) => ({ + signal: start ? "metrics" : "logs", + payload: {}, + at, + start, + })); +} + function attributes(value: unknown): Map<string, string> { if (value !== undefined && !Array.isArray(value)) throw new Error("Invalid OTLP attributes."); const out = new Map<string, string>(); @@ -566,6 +591,22 @@ export function admitTransferIdentity( ); } +export function observeIdentity( + transfer: Transfer, + options: SyncOptions, + identity: Identity, + evidence: IdentityEvidence[] +): void { + const previous = new Map(identity.values); + const conflicted = identity.conflict; + admitTransferIdentity(transfer, options, identity); + const keys = [...identity.values] + .filter(([key, value]) => previous.get(key) !== value) + .map(([key]) => key); + if ((keys.length || conflicted !== identity.conflict) && transfer.at) + evidence.push({ keys, at: transfer.at, ...(transfer.start ? { start: transfer.start } : {}) }); +} + export function destination(options: SyncOptions, signal: "logs" | "metrics"): URL { let endpoint: URL; try { @@ -596,6 +637,12 @@ export function destination(options: SyncOptions, signal: "logs" | "metrics"): U class RetryableDeliveryError extends Error {} +export class ConsentNotConfirmedError extends Error { + constructor() { + super("Consent is no longer confirmed; sync stopped before sending."); + } +} + async function assertFullDelivery(response: Response, signal: Transfer["signal"]): Promise<void> { let result: unknown; try { @@ -645,8 +692,7 @@ export async function send( const url = destination(options, transfer.signal); let backoff = options.intervalMs ?? 1000; for (;;) { - if (!(await consent())) - throw new Error("Consent is no longer confirmed; sync stopped before sending."); + if (!(await consent())) throw new ConsentNotConfirmedError(); try { let response: Response; try { @@ -682,6 +728,55 @@ export interface BindingDetails { readonly identity: Array<[string, string]>; readonly binding: string; readonly conflict?: boolean; + readonly identityProof?: { + readonly version: 1; + readonly admissionKey: string; + readonly sources: IdentityEvidence[]; + }; +} + +function readIdentityProof( + value: unknown, + identity: Array<[string, string]> +): BindingDetails["identityProof"] { + const proof = object(value); + if ( + proof.version !== 1 || + typeof proof.admissionKey !== "string" || + !/^[a-f0-9]{64}$/.test(proof.admissionKey) || + !Array.isArray(proof.sources) || + proof.sources.length === 0 || + proof.sources.length > 4 + ) + return undefined; + const keys = new Set(identity.map(([key]) => key)); + const seen = new Set<string>(); + const sources: IdentityEvidence[] = []; + for (const raw of proof.sources) { + const source = object(raw); + if ( + !Array.isArray(source.keys) || + new Set(source.keys).size !== source.keys.length || + !source.keys.every( + (key: unknown) => typeof key === "string" && keys.has(key) && !seen.has(key) + ) || + typeof source.at !== "string" || + !Number.isFinite(Date.parse(source.at)) || + (source.start !== undefined && + (typeof source.start !== "string" || + !Number.isFinite(Date.parse(source.start)) || + Date.parse(source.start) > Date.parse(source.at))) + ) + return undefined; + for (const key of source.keys) seen.add(key); + sources.push({ + keys: source.keys as string[], + at: source.at, + ...(typeof source.start === "string" ? { start: source.start } : {}), + }); + } + if (seen.size !== keys.size || keys.size === 0) return undefined; + return { version: 1, admissionKey: proof.admissionKey, sources }; } export async function readBindingDetails( @@ -717,15 +812,20 @@ export async function readBindingDetails( ) ) throw new Error("Sync binding state is invalid."); + if ( + new Set(data.identity.map((item: [string, string]) => item[0])).size !== data.identity.length + ) + throw new Error("Sync binding state is invalid."); if (data.conflict !== undefined && typeof data.conflict !== "boolean") throw new Error("Sync binding state is invalid."); + const identity = data.identity as Array<[string, string]>; + const identityProof = readIdentityProof(data.identityProof, identity); return { mayHaveLinked: data.mayHaveLinked, details: { session: data.session, - // Older versions collected identity before checking consent; never reuse it. - identity: - data.identityConsentChecked === true ? (data.identity as Array<[string, string]>) : [], + identity: identityProof ? identity : [], + identityProof, binding: data.binding, conflict: data.conflict === true, }, @@ -749,7 +849,7 @@ export async function saveBindingState( try { await writeFile( temporary, - `${JSON.stringify({ mayHaveLinked, ...details, identityConsentChecked: true })}\n`, + `${JSON.stringify({ mayHaveLinked, ...details, identity: details?.identityProof ? details.identity : [] })}\n`, { flag: "wx", mode: 0o600, @@ -823,6 +923,7 @@ export async function syncOtel( const path = join(options.home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); const identity: Identity = { values: new Map(), conflict: false }; + const identityEvidence: IdentityEvidence[] = []; const stillPermitted = async (items: readonly Transfer[], expectedFeature?: SessionFeature) => { const currentFeature = await resolve(); if (expectedFeature && !sameSessionFeature(expectedFeature, currentFeature)) return false; @@ -843,6 +944,7 @@ export async function syncOtel( ? join(options.stateDir, "otel-sync", `${stateName}.json`) : null; let mayHaveLinked = false; + let previousLinkUnverified = false; const projects = options.config === CODEX_CONFIG_ID && options.includeSessionProjects ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) @@ -862,10 +964,21 @@ export async function syncOtel( if (statePath !== null) { const saved = await readBindingDetails(statePath); mayHaveLinked = saved?.mayHaveLinked === true; + previousLinkUnverified = mayHaveLinked; if (saved?.details) { if (saved.details.session !== options.session) throw new Error("Sync binding state does not match its session."); - for (const [key, value] of saved.details.identity) identity.values.set(key, value); + const proof = saved.details.identityProof; + if ( + proof && + options.admission && + proof.admissionKey === admissionFingerprint(options.admission) && + (await stillPermitted(evidenceTransfers(proof.sources))) + ) { + for (const [key, value] of saved.details.identity) identity.values.set(key, value); + identityEvidence.push(...proof.sources); + previousLinkUnverified = false; + } identity.conflict = saved.details.conflict === true; binding = saved.details.binding; needsReplay = mayHaveLinked && binding !== "invalidated"; @@ -899,7 +1012,7 @@ export async function syncOtel( continue; } admitted.push(transfer); - admitTransferIdentity(transfer, options, identity); + observeIdentity(transfer, options, identity, identityEvidence); } if (!options.titlesOnly) for (const batch of packTransfers(admitted)) { @@ -934,10 +1047,18 @@ export async function syncOtel( payload: {}, at: new Date().toISOString(), }); - if (!options.titlesOnly && bindingAllowed && shouldInvalidate && binding !== "invalidated") { + if ( + !previousLinkUnverified && + !options.titlesOnly && + bindingAllowed && + shouldInvalidate && + binding !== "invalidated" + ) { const event = bindingTransfer(options, identity, { state: "ambiguous" }); if (event) { - await send(options, event, () => stillPermitted([event], feature)); + await send(options, event, () => + stillPermitted([event, ...evidenceTransfers(identityEvidence)], feature) + ); sent++; binding = "invalidated"; if (statePath !== null) @@ -945,10 +1066,20 @@ export async function syncOtel( session: options.session, identity: [...identity.values], binding, + ...(options.admission && identityEvidence.length + ? { + identityProof: { + version: 1, + admissionKey: admissionFingerprint(options.admission), + sources: identityEvidence, + }, + } + : {}), }); } } else if ( bindingAllowed && + !previousLinkUnverified && !options.titlesOnly && !shouldInvalidate && feature.state === "confirmed" && @@ -961,10 +1092,21 @@ export async function syncOtel( session: options.session, identity: [...identity.values], binding: nextBinding, + ...(options.admission && identityEvidence.length + ? { + identityProof: { + version: 1, + admissionKey: admissionFingerprint(options.admission), + sources: identityEvidence, + }, + } + : {}), }); } mayHaveLinked = true; - await send(options, event, () => stillPermitted([event], feature)); + await send(options, event, () => + stillPermitted([event, ...evidenceTransfers(identityEvidence)], feature) + ); sent++; binding = nextBinding; needsReplay = false; @@ -976,7 +1118,9 @@ export async function syncOtel( publisher && (await publisher.publish(options.session, identity, async (event) => { if (!(await permits(event))) return false; - await send(options, event, () => stillPermitted([event])); + await send(options, event, () => + stillPermitted([event, ...evidenceTransfers(identityEvidence)]) + ); return true; })) ) diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index f3691198d..40b5d01e4 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -367,7 +367,6 @@ export function registerTelemetryCommand(program: Command): void { ); } admission = current; - if (!admission) throw new Error("Session admission is unavailable."); return admission.feature; }; await resolve(); @@ -379,6 +378,14 @@ export function registerTelemetryCommand(program: Command): void { onProjectUnavailable, onRepositoryUnavailable, session, + ...(initialAdmission.cwd && initialAdmission.repositoryId + ? { + admission: { + cwd: initialAdmission.cwd, + repositoryId: initialAdmission.repositoryId, + }, + } + : {}), token: process.env.AIDD_OTEL_TOKEN ?? "", onTitleUnavailable, }, diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts index e62de083c..19c1b7385 100644 --- a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts @@ -17,6 +17,16 @@ import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/o import { syncAllSessionTitles } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.js"; const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +function logAttributes(body: string): ReturnType<typeof attr>[] { + const envelope = JSON.parse(body) as { + resourceLogs: Array<{ + scopeLogs: Array<{ logRecords: Array<{ attributes: ReturnType<typeof attr>[] }> }>; + }>; + }; + return envelope.resourceLogs.flatMap((resource) => + resource.scopeLogs.flatMap((scope) => scope.logRecords.flatMap((record) => record.attributes)) + ); +} it.skipIf(process.platform === "win32")( "sends titles only when opted in and the session is consented", @@ -182,7 +192,9 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { await syncOtel({ ...options, includeSessionRepositories: true }, resolve, async () => true); expect(received).toHaveLength(2); expect(received[1]).toContain("aidd.session_repository"); - expect(received[1]).toContain("https://github.com/example/repository"); + expect( + logAttributes(received[1]).filter((entry) => entry.key === "aidd.repository_url") + ).toEqual([attr("aidd.repository_url", "https://github.com/example/repository")]); expect(received[1]).not.toContain("git@github.com"); expect(received[1]).not.toContain("Synthetic title"); received.length = 0; @@ -352,9 +364,10 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { await running; clearTimeout(timeout); expect(received.some((body) => body.includes("Synthetic title"))).toBe(true); - expect(received.some((body) => body.includes("https://github.com/example/repository"))).toBe( - true - ); + const receivedAttributes = received.flatMap(logAttributes); + expect(receivedAttributes.filter((entry) => entry.key === "aidd.repository_url")).toEqual([ + attr("aidd.repository_url", "https://github.com/example/repository"), + ]); received.length = 0; let consenting = true; @@ -378,19 +391,23 @@ readline.createInterface({ input: process.stdin }).on('line', (line) => { received.length = 0; consenting = true; + const retryAbort = new AbortController(); const retryAll = syncAllOtel( { ...options, stateDir: join(home, "withdrawn-state"), includeSessionTitles: true, intervalMs: 10, + signal: retryAbort.signal, }, async () => new Map([["session-1", admission]]), async () => (consenting ? "allow" : "deny") ); await new Promise((done) => setTimeout(done, 100)); await appendFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`); - await expect(retryAll).rejects.toThrow(/Consent is no longer confirmed/); + await new Promise((done) => setTimeout(done, 100)); + retryAbort.abort(); + await retryAll; expect(received.filter((body) => body.includes("aidd.session_title"))).toHaveLength(1); } finally { process.env.PATH = oldPath; diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts index 810e6bd69..2d27b8e15 100644 --- a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts @@ -203,6 +203,7 @@ it("retains an unsent capture across consent withdrawal and resumes only after c let consenting = true; let calls = 0; const abort = new AbortController(); + const firstAbort = new AbortController(); server = createServer((request, response) => { request.resume(); request.on("end", () => { @@ -210,6 +211,7 @@ it("retains an unsent capture across consent withdrawal and resumes only after c if (calls === 1) { consenting = false; response.writeHead(503).end("{}"); + setTimeout(() => firstAbort.abort(), 100); } else { response.writeHead(200).end("{}"); abort.abort(); @@ -237,10 +239,10 @@ it("retains an unsent capture across consent withdrawal and resumes only after c const historical = new Date(1_793_763_900_000).toISOString(); const permits = async (_admission: unknown, at: string, _start?: string, current?: string) => at === historical && (!current || consenting) ? ("allow" as const) : ("deny" as const); - const first = syncAllOtel(options, snapshot, permits); + const first = syncAllOtel({ ...options, signal: firstAbort.signal }, snapshot, permits); await new Promise((resolve) => setTimeout(resolve, 80)); await appendFile(capture, line("allowed")); - await expect(first).rejects.toThrow(/Consent is no longer confirmed/); + expect(await first).toMatchObject({ sent: 0, pending: 1 }); expect(calls).toBe(1); consenting = true; const result = await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); @@ -248,6 +250,159 @@ it("retains an unsent capture across consent withdrawal and resumes only after c expect(calls).toBe(2); }); +it("isolates a revoked session across restarts while another session continues", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-two-sessions-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let consenting = true; + let historicalAllowed = true; + let firstFailure = true; + let abort = new AbortController(); + const accepted: string[] = []; + const attempts: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + const id = body.includes('"stringValue":"one"') ? "one" : "two"; + attempts.push(id); + if (id === "one" && firstFailure) { + firstFailure = false; + consenting = false; + response.writeHead(503).end("{}"); + } else { + accepted.push(id); + response.writeHead(200).end("{}"); + abort.abort(); + } + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + refreshMs: 20, + }; + const one = { feature: { state: "unbound" as const }, cwd: "/one", repositoryId: "one-repo" }; + const two = { feature: { state: "unbound" as const }, cwd: "/two", repositoryId: "two-repo" }; + const snapshot = async () => + new Map([ + ["one", one], + ["two", two], + ]); + const permits = async ( + admission: { repositoryId: string | null }, + _at: string, + _start?: string, + current?: string + ) => + admission.repositoryId === "one-repo" && (!historicalAllowed || (!!current && !consenting)) + ? ("deny" as const) + : ("allow" as const); + const initial = JSON.parse(line("one")); + initial.payload.resourceLogs[0].scopeLogs[0].logRecords.push( + JSON.parse(line("two")).payload.resourceLogs[0].scopeLogs[0].logRecords[0] + ); + const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, `${JSON.stringify(initial)}\n`); + expect(await first).toMatchObject({ sent: 1, pending: 1 }); + expect(attempts).toEqual(["one", "two"]); + expect(accepted).toEqual(["two"]); + + historicalAllowed = false; + abort = new AbortController(); + const second = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("two")); + expect(await second).toMatchObject({ sent: 1, pending: 1 }); + expect(attempts).toEqual(["one", "two", "two"]); + + historicalAllowed = true; + consenting = true; + abort = new AbortController(); + const third = await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + expect(third).toMatchObject({ sent: 1, pending: 0 }); + expect(accepted).toEqual(["two", "two", "one"]); +}); + +it("defers one revoked binding without blocking another and restores a proven identity after restart", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-two-bindings-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let consenting = true; + let failOne = true; + let abort = new AbortController(); + const bindings: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + if (!body.includes("aidd.session_feature")) return void response.writeHead(200).end("{}"); + const id = body.includes('"stringValue":"one"') ? "one" : "two"; + bindings.push(id); + if (id === "one" && failOne) { + failOne = false; + consenting = false; + response.writeHead(503).end("{}"); + } else { + response.writeHead(200).end("{}"); + abort.abort(); + } + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + refreshMs: 20, + }; + const feature = { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }; + const one = { feature, cwd: "/one", repositoryId: "one-repo" }; + const two = { feature, cwd: "/two", repositoryId: "two-repo" }; + const snapshot = async () => + new Map([ + ["one", one], + ["two", two], + ]); + const permits = async (admission: { repositoryId: string | null }) => + admission.repositoryId === "one-repo" && !consenting ? ("deny" as const) : ("allow" as const); + const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("one") + line("two")); + await first; + expect(bindings).toEqual(["one", "two"]); + consenting = true; + abort = new AbortController(); + setTimeout(() => abort.abort(), 150); + await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + expect(bindings).toEqual(["one", "two", "one", "two"]); +}); + it.each(["missing", "cwd", "repository", "unavailable"] as const)( "does not retry if the session admission is %s after a failed post", async (change) => { @@ -259,12 +414,14 @@ it.each(["missing", "cwd", "repository", "unavailable"] as const)( await writeFile(capture, "", { mode: 0o600 }); let changed = false; let calls = 0; + const abort = new AbortController(); server = createServer((request, response) => { calls++; request.resume(); request.on("end", () => { changed = true; response.writeHead(503).end("{}"); + setTimeout(() => abort.abort(), 100); }); }); await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); @@ -284,6 +441,7 @@ it.each(["missing", "cwd", "repository", "unavailable"] as const)( home, stateDir: join(home, "state"), intervalMs: 10, + signal: abort.signal, }, async () => { if (changed && change === "unavailable") throw new Error("Admission unavailable"); @@ -303,7 +461,8 @@ it.each(["missing", "cwd", "repository", "unavailable"] as const)( ); await new Promise((resolve) => setTimeout(resolve, 80)); await appendFile(capture, line("allowed")); - await expect(running).rejects.toThrow(/Consent is no longer confirmed|Admission unavailable/); + if (change === "unavailable") await expect(running).rejects.toThrow(/Admission unavailable/); + else expect(await running).toMatchObject({ sent: 0, pending: 1 }); expect(calls).toBe(1); } ); @@ -405,6 +564,7 @@ it("does not reuse legacy checkpoint identity until a new admitted observation s const path = join(directory, stateFile); const legacy = JSON.parse(await readFile(path, "utf8")); delete legacy.identityConsentChecked; + delete legacy.identityProof; await writeFile(path, JSON.stringify(legacy), { mode: 0o600 }); received.length = 0; abort = new AbortController(); @@ -412,11 +572,89 @@ it("does not reuse legacy checkpoint identity until a new admitted observation s await new Promise((resolve) => setTimeout(resolve, 100)); expect(received).toHaveLength(0); await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 100)); + abort.abort(); await restart; - expect(received.filter((body) => body.includes("aidd.session_feature"))).toHaveLength(1); + expect(received.filter((body) => body.includes("aidd.session_feature"))).toHaveLength(0); expect(received.some((body) => body.includes("developer@example.test"))).toBe(true); }); +it("does not reuse an identity when a clone at the same path and repository is replaced", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-replaced-clone-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + refreshMs: 20, + }; + let changed = false; + const oldAt = new Date(1_793_763_900_000).toISOString(); + const snapshot = async () => + new Map([ + [ + "allowed", + { + feature: changed + ? { state: "unbound" as const } + : { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }, + cwd: "/project", + repositoryId: "repo", + }, + ], + ]); + const permits = async (_admission: unknown, at: string, _start?: string, current?: string) => + changed && current && at === oldAt ? ("deny" as const) : ("allow" as const); + const abort = new AbortController(); + const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + for (let i = 0; i < 100 && !received.some((body) => body.includes('aidd.session_feature"')); i++) + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); + changed = true; + const replacement = JSON.parse( + line("allowed").replace("developer@example.test", "replacement@example.test") + ); + replacement.payload.resourceLogs[0].scopeLogs[0].logRecords[0].timeUnixNano = + "1793763901000000000"; + await appendFile(capture, `${JSON.stringify(replacement)}\n`); + await new Promise((resolve) => setTimeout(resolve, 150)); + abort.abort(); + await first; + expect(received.some((body) => body.includes("replacement@example.test"))).toBe(true); + expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(false); + const beforeRestart = received.length; + const restartAbort = new AbortController(); + const restart = syncAllOtel({ ...options, signal: restartAbort.signal }, snapshot, permits); + await new Promise((resolve) => setTimeout(resolve, 120)); + restartAbort.abort(); + await restart; + expect(received).toHaveLength(beforeRestart); +}); + it("does not send an event outside the clone's consent interval", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-consent-")); dirs.push(home); @@ -666,7 +904,7 @@ it("retries a feature link rejected after its intent was persisted", async () => expect(links).toBe(2); }); -it("retains an identity conflict across a failed invalidation and restart", async () => { +it("revalidates every identity source before resuming a conflicted binding", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-conflict-")); dirs.push(home); const captureDir = join(home, ".local/share/aidd-otel-observation"); @@ -708,7 +946,20 @@ it("retains an identity conflict across a failed invalidation and restart", asyn const snapshot = async () => new Map([["allowed", admission]]); const first = syncAllOtel(options, snapshot, async () => "allow"); await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); + const initial = JSON.parse(line("allowed")); + const original = initial.payload.resourceLogs[0].scopeLogs[0].logRecords[0]; + const second = structuredClone(original); + second.timeUnixNano = "1793763900001000000"; + second.attributes = second.attributes.filter( + (item: { key: string }) => item.key !== "user.email" + ); + second.attributes.push(attribute("user.account_id", "account-1")); + const third = structuredClone(original); + third.timeUnixNano = "1793763900002000000"; + third.attributes = third.attributes.filter((item: { key: string }) => item.key !== "user.email"); + third.attributes.push(attribute("organization.id", "org-1")); + initial.payload.resourceLogs[0].scopeLogs[0].logRecords.push(second, third); + await appendFile(capture, `${JSON.stringify(initial)}\n`); for (let i = 0; i < 100 && !linked; i++) await new Promise((resolve) => setTimeout(resolve, 20)); expect(linked).toBe(true); await appendFile( @@ -716,12 +967,39 @@ it("retains an identity conflict across a failed invalidation and restart", asyn line("allowed").replace("developer@example.test", "other@example.test") ); await expect(first).rejects.toThrow("HTTP 403"); + const directory = join(home, "state", "otel-sync"); + const stateName = (await readdir(directory)).find( + (name) => name.endsWith(".json") && !name.includes("checkpoint") + ); + if (!stateName) throw new Error("Missing binding state"); + const saved = JSON.parse(await readFile(join(directory, stateName), "utf8")); + expect(saved.identityProof.sources.map((source: { keys: string[] }) => source.keys)).toEqual([ + ["user.email"], + ["user.account_id"], + ["organization.id"], + [], + ]); rejectInvalidation = false; const abort = new AbortController(); - const second = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); + const accountAt = new Date(1_793_763_900_001).toISOString(); + const blocked = syncAllOtel( + { ...options, signal: abort.signal }, + snapshot, + async (_admission, at, _start, current) => (current && at === accountAt ? "deny" : "allow") + ); await new Promise((resolve) => setTimeout(resolve, 120)); abort.abort(); - await second; + await blocked; + expect(invalidations).toBe(1); + const retryAbort = new AbortController(); + const retry = syncAllOtel( + { ...options, signal: retryAbort.signal }, + snapshot, + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 120)); + retryAbort.abort(); + await retry; expect(invalidations).toBe(2); }); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts index 6d954d8dc..1d6d5fb0f 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, expect, test } from "vitest"; import { + readBindingDetails, selectTransfers, send, syncOtel, @@ -29,6 +30,33 @@ afterEach(async () => { await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); }); +posixTest( + "rejects duplicate identity keys in a persisted binding, even with a v1 proof", + async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-duplicate-identity-")); + dirs.push(home); + const state = join(home, "binding.json"); + const payload = { + mayHaveLinked: true, + session: "session", + binding: "branch\u0000owner/repo#1", + identity: [["user.email", "first@example.invalid"]], + identityProof: { + version: 1, + admissionKey: "a".repeat(64), + sources: [{ keys: ["user.email"], at: "2026-11-05T00:00:00.000Z" }], + }, + }; + await writeFile(state, JSON.stringify(payload), { mode: 0o600 }); + await expect(readBindingDetails(state)).resolves.toMatchObject({ + details: { identity: [["user.email", "first@example.invalid"]] }, + }); + payload.identity.push(["user.email", "second@example.invalid"]); + await writeFile(state, JSON.stringify(payload), { mode: 0o600 }); + await expect(readBindingDetails(state)).rejects.toThrow("Sync binding state is invalid"); + } +); + function codexLog(time: Record<string, string | number>, timestamp?: string): string { return JSON.stringify({ signal: "logs", diff --git a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts index 8ff1f961a..171410752 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts @@ -214,6 +214,7 @@ describe("OTLP capture sync", () => { home, follow: false, stateDir: join(home, "state"), + admission: { cwd: "/project", repositoryId: "repo" }, }; const confirmed = async () => ({ state: "confirmed" as const, diff --git a/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts b/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts index c1d306a66..9a79bbfe9 100644 --- a/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts +++ b/cli/tests/e2e/telemetry-otel-collector.e2e.test.ts @@ -13,6 +13,10 @@ const attr = (key: string, value: string) => ({ key, value: { stringValue: value describe.skipIf(process.platform === "win32")("OTLP collector on the built CLI", () => { it("captures isolated authenticated provider records and verifies their normalized fields", async () => { const { projectDir, fakeHome, cleanup } = await createTestEnv("otel-collector"); + const directory = join(fakeHome, ".local/share/aidd-otel-observation"); + const token = "test-only-otel-collector-token"; + await mkdir(directory, { recursive: true, mode: 0o700 }); + await writeFile(join(directory, "collector-token"), `${token}\n`, { mode: 0o600 }); const bin = join(fakeHome, "bin"); await mkdir(bin); await writeFile(join(bin, "opencode"), `#!${process.execPath}\nconsole.log('2.2.5');\n`, { @@ -36,10 +40,9 @@ describe.skipIf(process.platform === "win32")("OTLP collector on the built CLI", throw new Error("Collector exited before listening."); }), ]); - const port = /127\.0\.0\.1:(\d+)/.exec(output)?.[1]; - expect(port).toBeDefined(); - const directory = join(fakeHome, ".local/share/aidd-otel-observation"); - const token = (await readFile(join(directory, "collector-token"), "utf8")).trim(); + const port = Number(/127\.0\.0\.1:(\d+)/.exec(output)?.[1]); + if (!Number.isInteger(port) || port < 1 || port > 65535) + throw new Error("Collector reported an invalid loopback port."); const configEnv = { PATH: path, OPENCODE_CONFIG_DIR: "", @@ -78,12 +81,13 @@ describe.skipIf(process.platform === "win32")("OTLP collector on the built CLI", }; const url = `http://127.0.0.1:${port}/${tool}/v1/logs`; const body = JSON.stringify(payload); - expect((await fetch(url, { method: "POST", body })).status).toBe(403); + expect((await fetch(url, { method: "POST", body, redirect: "manual" })).status).toBe(403); expect( ( await fetch(url, { method: "POST", body, + redirect: "manual", headers: { authorization: `Bearer ${token}` }, }) ).status diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 5fabebf08..bb0556d28 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -125,7 +125,7 @@ The origin must be loopback HTTP; no token or prompt content is printed. Only su An automatic follower starts with new capture events and keeps a private checkpoint. Network errors and HTTP 5xx responses are reported and retried with bounded backoff; permanent HTTP failures stop the follower. Restarting resumes from the checkpoint, and a failed post is replayed, so the receiver must remain idempotent. Unknown sessions wait for their transcript to be admitted using private byte-range indexes, without holding up other sessions. These indexes can grow until admission and depend on retaining the original capture; replaced or truncated captures require recovery rather than silently skipping data. Running a follower for each provider is an explicit operation; it does not start automatically after a machine restart. Linked session identifiers and the minimum provider identity are stored privately to detect later ambiguity, including after restart. Older sync state without identity must be refreshed with a session replay. -Every delivery attempt, including a retry, rechecks session admission and the same clone's current consent and original consent interval. If consent or admission cannot be confirmed, sync stops before the next POST and leaves unsent capture data at its previous checkpoint. An explicit restart after renewed consent can replay observations whose original intervals remain authorized for that same clone. Session identity is collected only from admitted observations. Identity saved by older sync versions has no reliable consent provenance and is ignored until a new admitted observation or an explicit session replay supplies it; links and metadata may remain unavailable in the meantime. +Every delivery attempt, including a retry, rechecks session admission and the same clone's current consent and original consent interval. If consent or admission cannot be confirmed, a single-session sync stops before the next POST. The multi-session follower retains an already-admitted but unsent capture range in a private per-session journal and continues other sessions. The journal temporarily stores a SHA-256 hash of the admitted working directory and repository ID, not their plaintext; the hash may reveal equality or permit guesses and is not sent. The range and hash survive restarts, provided the original capture remains available. After renewed consent, the follower can replay it only for the same admission and clone while its original interval remains authorized; otherwise it stays pending. Other delivery errors and invalid state still stop the follower. Session identity is collected only from admitted observations. New binding state privately retains the minimum identity, admission hash and source times for each identifier, and restores them only after rechecking each source and the current consent for the same clone. Legacy bindings without that proof are not automatically replayed or invalidated: an old feature may remain visible but cannot be certified. This unresolved limitation blocks merging the feature. To replay one consented session already in the capture, pass `--session=<id>` without `--follow`. Add `--follow` to follow only that session. A feature link is sent only when every recorded call agrees on a single observed branch and a declared canonical ticket such as `owner/repo#123`; otherwise it remains unlinked. If a linked session later becomes ambiguous, an irreversible invalidation keeps it unlinked even if events arrive out of order. From d79afa72e10742519182d7cc8e1826c9e1ab7402 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 18:35:31 +0800 Subject: [PATCH 26/56] fix(telemetry): preserve uncertain session usage Avoid attributing historical usage to an unproven repository or clone while retaining observations for safe replay. --- cli/package.json | 2 +- cli/scripts/check-bundle-size.mjs | 1 + .../application/directory-resolver.ts | 32 +- .../application/ingest-usage-use-case.ts | 8 +- .../report/report-usage-use-case.ts | 20 +- .../report/repository-uncertainty.ts | 61 ++ .../application/report/session-consent.ts | 57 +- .../report/session-feature-use-case.ts | 22 +- .../application/switch/remember-own-root.ts | 25 +- .../telemetry/domain/report/attribution.ts | 2 + .../domain/report/session-feature.ts | 8 +- .../telemetry/domain/report/usage-report.ts | 2 +- .../telemetry/domain/repository-resolution.ts | 120 +++- .../contexts/telemetry/domain/stored-usage.ts | 3 + .../infrastructure/otel/actions/sync-all.ts | 362 +++++++++-- .../infrastructure/otel/actions/sync.ts | 38 +- cli/src/presentation/commands/telemetry.ts | 19 +- cli/src/runtime/wiring/telemetry-otel.ts | 15 +- cli/src/runtime/wiring/telemetry.ts | 5 +- .../ingest-usage-use-case.unit.test.ts | 38 +- ...age-migration.property.integration.test.ts | 18 +- .../left-out.property.integration.test.ts | 3 +- ...eport-figures.property.integration.test.ts | 3 +- ...eport-filters.property.integration.test.ts | 3 +- .../report/report-usage-use-case.unit.test.ts | 65 +- ...report-window.property.integration.test.ts | 9 +- .../report/session-consent.unit.test.ts | 59 +- .../session-feature-use-case.unit.test.ts | 97 ++- .../domain/repository-resolution.unit.test.ts | 23 +- .../domain/session-feature.unit.test.ts | 15 + .../otel/sync-all-otel.integration.test.ts | 577 ++++++++++++++++++ .../runtime/wiring/otel-wiring.unit.test.ts | 3 +- plugins/aidd-telemetry/README.md | 4 + 33 files changed, 1538 insertions(+), 181 deletions(-) create mode 100644 cli/src/contexts/telemetry/application/report/repository-uncertainty.ts diff --git a/cli/package.json b/cli/package.json index f99446faa..dc8d35ed3 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,7 +36,7 @@ "node": ">=22.12" }, "packageManager": "pnpm@12.16.1", - "bundleBudgetKB": 819, + "bundleBudgetKB": 829, "scripts": { "build": "tsup && node scripts/check-bundle-size.mjs", "build:check-size": "node scripts/check-bundle-size.mjs", diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 5da5a1d7c..f14867e3a 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -67,6 +67,7 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 810 KB: 809.79 KB with separately opted-in, sanitized Codex repository metadata. // 813 KB: 812.66 KB with consent and session-admission checks on every delivery attempt. // 819 KB: 818.39 KB with per-identifier provenance and isolated pending session ranges. +// 829 KB: 828.93 KB with repository uncertainty and clone-proven resume after origin changes. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/application/directory-resolver.ts b/cli/src/contexts/telemetry/application/directory-resolver.ts index 220a69918..ab7f83511 100644 --- a/cli/src/contexts/telemetry/application/directory-resolver.ts +++ b/cli/src/contexts/telemetry/application/directory-resolver.ts @@ -7,10 +7,10 @@ import { repositoryIdOf } from "../domain/repository-identity.js"; import { cwdKey, type NotStoredReason, - ownerAt, + ownersAt, type RepositoryResolution, + rememberResolution, resolutionKey, - sameResolution, } from "../domain/repository-resolution.js"; import { tokenOfKey } from "../domain/telemetry-consent.js"; import { ConsentLog } from "./consent-log.js"; @@ -140,7 +140,13 @@ export class ResolutionRun { const directory = await this.directoryOf(cwd); if ("skipped" in directory) return directory; const instant = Date.parse(at); - const owner = ownerAt(directory.owners, instant); + const candidates = ownersAt(directory.owners, instant); + // Equal birth times cannot establish which clone made the call. Consent alone + // cannot identify the owner, so leave it pending rather than guessing. + if (candidates.length !== 1) return { skipped: "unreadable-consent" }; + const selected = candidates[0] as RepositoryResolution; + // Another directory in this run may have revealed a changed origin for this clone. + const owner = this.remembered.get(resolutionKey(selected.dir, selected.clone)) ?? selected; if (this.unreadable.has(cloneKey(owner.clone))) return { skipped: "unreadable-consent" }; const live = directory.current !== null && sameClone(directory.current.clone, owner.clone); if (!this.consents.covers(owner.clone, instant)) { @@ -196,19 +202,13 @@ export class ResolutionRun { root: string, clone: CloneIdentity ): RepositoryResolution { - const key = resolutionKey(dir, clone); - const held = this.remembered.get(key); - const seen = { - dir, - repository_id: repositoryId, - root, - clone, - seen_at: held?.seen_at ?? this.environment.now().toISOString(), - }; - if (sameResolution(held, seen)) return seen; - this.remembered.set(key, seen); - this.changed = true; - return seen; + const result = rememberResolution( + this.remembered, + { dir, repository_id: repositoryId, root, clone }, + this.environment.now().toISOString() + ); + if (result.changed) this.changed = true; + return result.resolution; } private ownersOf(dir: string): RepositoryResolution[] { diff --git a/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts b/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts index 2cb1fef99..0d65c0710 100644 --- a/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts +++ b/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts @@ -1,6 +1,6 @@ import { isRequestless, requestlessCallOf } from "../domain/formats/claude-transcript-usage.js"; import type { LeftOutEntry, UsageLedger } from "../domain/ports/usage-ledger.js"; -import type { NotStoredReason } from "../domain/repository-resolution.js"; +import { type NotStoredReason, repositoryUncertain } from "../domain/repository-resolution.js"; import { dayOf, type StoredUsage } from "../domain/stored-usage.js"; import type { TranscriptPosition } from "../domain/transcript-position.js"; import type { UsageRecord } from "../domain/usage-record.js"; @@ -149,7 +149,11 @@ export class IngestUsageUseCase { leftOut.push({ reason: "requestless", of: requestlessCallOf(record) }); continue; } - toStore.push({ ...record, repository_id: outcome.stored.repository_id }); + toStore.push({ + ...record, + repository_id: outcome.stored.repository_id, + ...(repositoryUncertain(outcome.stored, record.at) ? { repository_uncertain: true } : {}), + }); if (outcome.live) live.set(outcome.stored.repository_id, outcome.stored.root); } diff --git a/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts b/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts index 7baf36e2e..c8b277a63 100644 --- a/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts +++ b/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts @@ -1,6 +1,6 @@ import type { PersonIdentityStore } from "../../domain/ports/identity/person-identity-store.js"; import type { RepositoryLocator } from "../../domain/ports/repository-locator.js"; -import type { LeftOutReason, UsageLedger } from "../../domain/ports/usage-ledger.js"; +import type { LeftOutReason } from "../../domain/ports/usage-ledger.js"; import { attribute } from "../../domain/report/attribution.js"; import { inPeriod, type Period, windowOf } from "../../domain/report/period.js"; import { @@ -20,8 +20,8 @@ import { import { repositoryIdOf } from "../../domain/repository-identity.js"; import type { StoredUsage } from "../../domain/stored-usage.js"; import type { IngestResult, IngestUsageUseCase } from "../ingest-usage-use-case.js"; -import { settleLedger } from "../ledger-upkeep.js"; import type { DeclaredBindings } from "./declared-bindings.js"; +import type { ReportLedger } from "./repository-uncertainty.js"; /** The filters as typed: a session or repository may be the start of an id. A repository is * either typed, or `--repository .`, the directory `here` whose repository is asked for; never @@ -105,7 +105,7 @@ export class ReportUsageUseCase { constructor( /** Only its run is used: whatever it found is the report's `run`, never a figure. */ private readonly ingest: Pick<IngestUsageUseCase, "execute">, - private readonly ledger: UsageLedger, + private readonly ledger: ReportLedger, private readonly declared: DeclaredBindings, private readonly identity: PersonIdentityStore, private readonly locator: RepositoryLocator, @@ -123,10 +123,8 @@ export class ReportUsageUseCase { if (located === null) return { status: "not-a-repository" }; const { filesRead, unrecognised, notStored, consentLogDamaged, oldestTranscriptAt } = await this.ingest.execute(); - const [held, leftOut] = await this.ledger.exclusively(async () => [ - await settleLedger(this.ledger, windowOf(request.period)), - await this.ledger.leftOut.counts(), - ]); + const held = await this.ledger.read(windowOf(request.period), undefined, true); + const leftOut = held.leftOut as Readonly<Record<LeftOutReason, number>>; const records = held.records.filter((record) => inPeriod(record.at, request.period)); const session = resolvedIn(records, "session", asked.session); if ("status" in session) return session; @@ -174,7 +172,13 @@ function resolvedIn( if (typed === null) return { id: null }; const found = resolveId( typed, - records.map((record) => (flag === "session" ? record.session_id : record.repository_id)) + records.flatMap((record) => + flag === "session" + ? [record.session_id] + : record.repository_uncertain + ? [] + : [record.repository_id] + ) ); if (found.kind === "ambiguous") return { status: "ambiguous", flag, typed, ids: found.ids }; return { id: found.kind === "found" ? found.id : typed }; diff --git a/cli/src/contexts/telemetry/application/report/repository-uncertainty.ts b/cli/src/contexts/telemetry/application/report/repository-uncertainty.ts new file mode 100644 index 000000000..67bb42b87 --- /dev/null +++ b/cli/src/contexts/telemetry/application/report/repository-uncertainty.ts @@ -0,0 +1,61 @@ +import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; +import { repositoryUncertain } from "../../domain/repository-resolution.js"; +import type { StoredUsage } from "../../domain/stored-usage.js"; +import type { UsageTool } from "../../domain/usage-record.js"; +import type { DirectoryResolver } from "../directory-resolver.js"; +import { settleLedger } from "../ledger-upkeep.js"; + +/** Settles and annotates a report's ledger under the same lock. */ +export class ReportLedger { + constructor( + private readonly ledger: UsageLedger, + private readonly directories?: Pick<DirectoryResolver, "open"> + ) {} + + async read(holds?: (day: string) => boolean, tool?: UsageTool, includeLeftOut = false) { + return this.ledger.exclusively(async () => { + const held = await settleLedger(this.ledger, holds); + const selected = tool ? held.records.filter((record) => record.tool === tool) : held.records; + const records = this.directories + ? await withRepositoryUncertainty(selected, this.directories) + : selected.map((record) => ({ ...record, repository_uncertain: true as const })); + const leftOut = includeLeftOut ? await this.ledger.leftOut.counts() : undefined; + return { records, skippedLines: held.skippedLines, leftOut }; + }); + } +} + +/** Annotates stored usage without changing token counts or rewriting the ledger. + * The caller holds the ledger lock while the resolver observes and saves changes. */ +export async function withRepositoryUncertainty( + records: readonly StoredUsage[], + directories: Pick<DirectoryResolver, "open"> +): Promise<readonly StoredUsage[]> { + const run = await directories.open(); + try { + // Observe every directory before attributing any record: another working tree may + // reveal a changed origin for a clone already visited in this same run. + const observed = new Set<string>(); + for (const record of records) { + if (record.cwd && !observed.has(record.cwd)) { + observed.add(record.cwd); + await run.resolve(record.cwd, record.at); + } + } + const annotated: StoredUsage[] = []; + for (const record of records) { + const outcome = record.cwd ? await run.resolve(record.cwd, record.at) : null; + const uncertain = + record.repository_uncertain === true || + !outcome || + "skipped" in outcome || + !outcome.live || + outcome.stored.repository_id !== record.repository_id || + repositoryUncertain(outcome.stored, record.at); + annotated.push(uncertain ? { ...record, repository_uncertain: true } : record); + } + return annotated; + } finally { + await run.close(); + } +} diff --git a/cli/src/contexts/telemetry/application/report/session-consent.ts b/cli/src/contexts/telemetry/application/report/session-consent.ts index 944d422c1..98692c5b1 100644 --- a/cli/src/contexts/telemetry/application/report/session-consent.ts +++ b/cli/src/contexts/telemetry/application/report/session-consent.ts @@ -1,4 +1,5 @@ -import { sameClone } from "../../domain/consent/clone-identity.js"; +import { createHash } from "node:crypto"; +import { cloneKey, sameClone } from "../../domain/consent/clone-identity.js"; import { foldConsent } from "../../domain/consent/consent-history.js"; import type { ConsentHistory } from "../../domain/ports/consent-history.js"; import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; @@ -20,19 +21,39 @@ export class SessionConsent { start?: string, current?: string ): Promise<"allow" | "deny" | "unknown"> { + return (await this.assess(admission, at, start, current)).verdict; + } + + /** A private fingerprint of the consented clone, only after the entire interval is verified. */ + async proof( + admission: SessionAdmission, + at: string, + start?: string, + current?: string + ): Promise<string | null> { + return (await this.assess(admission, at, start, current)).clone ?? null; + } + + private async assess( + admission: SessionAdmission, + at: string, + start?: string, + current?: string + ): Promise<{ verdict: "allow" | "deny" | "unknown"; clone?: string }> { const endTime = Date.parse(at); const startTime = start === undefined ? endTime : Date.parse(start); const currentTime = current === undefined ? endTime : Date.parse(current); if ( !admission.cwd || - !admission.repositoryId || + (!admission.repositoryId && + !(admission.repositoryId === null && admission.feature.state !== "confirmed")) || !Number.isFinite(endTime) || !Number.isFinite(startTime) || !Number.isFinite(currentTime) || startTime > endTime || currentTime < endTime ) - return "unknown"; + return { verdict: "unknown" }; return this.ledger.exclusively(async () => { const run = await this.directories.open(); try { @@ -46,27 +67,33 @@ export class SessionConsent { const reasons = [first, last, now].flatMap((value) => "skipped" in value ? [value.skipped] : [] ); - return reasons.some( - (reason) => reason === "unreadable-consent" || reason === "never-seen-alive" - ) - ? "unknown" - : "deny"; + return { + verdict: reasons.some( + (reason) => reason === "unreadable-consent" || reason === "never-seen-alive" + ) + ? "unknown" + : "deny", + }; } if ( !sameClone(first.stored.clone, last.stored.clone) || !sameClone(first.stored.clone, now.stored.clone) || - first.stored.repository_id !== admission.repositoryId || - last.stored.repository_id !== admission.repositoryId || - now.stored.repository_id !== admission.repositoryId + (admission.repositoryId !== null && + (first.stored.repository_id !== admission.repositoryId || + last.stored.repository_id !== admission.repositoryId || + now.stored.repository_id !== admission.repositoryId)) ) - return "deny"; + return { verdict: "deny" }; const history = await this.consents.read(); - if (history.damaged) return "unknown"; + if (history.damaged) return { verdict: "unknown" }; const intervals = foldConsent(history.events); return coversConsentWindow(intervals, first.stored.clone, startTime, endTime) && coversConsentWindow(intervals, first.stored.clone, currentTime, currentTime) - ? "allow" - : "deny"; + ? { + verdict: "allow", + clone: createHash("sha256").update(cloneKey(first.stored.clone)).digest("hex"), + } + : { verdict: "deny" }; } finally { await run.close(); } diff --git a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts index b48fb8c8f..c69f0f804 100644 --- a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts +++ b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts @@ -1,4 +1,3 @@ -import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; import { type SessionAdmission, type SessionFeature, @@ -7,26 +6,27 @@ import { import type { StoredUsage } from "../../domain/stored-usage.js"; import type { UsageTool } from "../../domain/usage-record.js"; import type { IngestUsageUseCase } from "../ingest-usage-use-case.js"; -import { settleLedger } from "../ledger-upkeep.js"; import type { DeclaredBindings } from "./declared-bindings.js"; +import type { ReportLedger } from "./repository-uncertainty.js"; import type { SessionConsent } from "./session-consent.js"; /** Only usage already admitted by the clone's consent can be forwarded. */ export class SessionFeatureUseCase { constructor( private readonly ingest: Pick<IngestUsageUseCase, "execute">, - private readonly ledger: UsageLedger, + private readonly ledger: ReportLedger, private readonly declared: Pick<DeclaredBindings, "read">, - private readonly consent: Pick<SessionConsent, "permits"> + private readonly consent: Pick<SessionConsent, "permits"> & + Partial<Pick<SessionConsent, "proof">> ) {} async snapshot(tool: UsageTool): Promise<Map<string, SessionAdmission>> { if ((await this.ingest.execute()).refused) throw new Error("AIDD telemetry is disabled; no capture was sent."); - const held = await this.ledger.exclusively(() => settleLedger(this.ledger)); + const { records } = await this.ledger.read(undefined, tool); const facts = await this.declared.read(); const sessions = new Map<string, StoredUsage[]>(); - for (const record of held.records.filter((item) => item.tool === tool)) { + for (const record of records) { const group = sessions.get(record.session_id) ?? []; group.push(record); sessions.set(record.session_id, group); @@ -35,12 +35,14 @@ export class SessionFeatureUseCase { [...sessions].map(([id, records]) => { const directories = new Set(records.map((record) => record.cwd)); const repositories = new Set(records.map((record) => record.repository_id)); + const uncertain = + repositories.size !== 1 || records.some((record) => record.repository_uncertain); return [ id, { - feature: sessionFeatureOf(records, facts), + feature: uncertain ? { state: "ambiguous" as const } : sessionFeatureOf(records, facts), cwd: directories.size === 1 ? records[0].cwd : null, - repositoryId: repositories.size === 1 ? records[0].repository_id : null, + repositoryId: uncertain ? null : records[0].repository_id, }, ]; }) @@ -56,4 +58,8 @@ export class SessionFeatureUseCase { async permits(admission: SessionAdmission, at: string, start?: string, current?: string) { return this.consent.permits(admission, at, start, current); } + + async cloneProof(admission: SessionAdmission, at: string, start?: string, current?: string) { + return this.consent.proof?.(admission, at, start, current) ?? null; + } } diff --git a/cli/src/contexts/telemetry/application/switch/remember-own-root.ts b/cli/src/contexts/telemetry/application/switch/remember-own-root.ts index cef802429..a517789af 100644 --- a/cli/src/contexts/telemetry/application/switch/remember-own-root.ts +++ b/cli/src/contexts/telemetry/application/switch/remember-own-root.ts @@ -1,12 +1,7 @@ import type { LocatedDirectory } from "../../domain/ports/repository-locator.js"; import type { ResolutionStore } from "../../domain/ports/resolution-store.js"; import { repositoryIdOf } from "../../domain/repository-identity.js"; -import { - cwdKey, - type RepositoryResolution, - resolutionKey, - sameResolution, -} from "../../domain/repository-resolution.js"; +import { cwdKey, rememberResolution } from "../../domain/repository-resolution.js"; type Located = Extract<LocatedDirectory, { status: "repository" }>; @@ -24,16 +19,10 @@ export async function rememberOwnRoot( if (id === null || located.clone === null) return; const held = await store.load(); const dir = cwdKey(located.root, caseInsensitive); - const key = resolutionKey(dir, located.clone); - const earlier = held.get(key); - const own: RepositoryResolution = { - dir, - repository_id: id, - root: located.root, - clone: located.clone, - seen_at: earlier?.seen_at ?? at.toISOString(), - }; - if (sameResolution(earlier, own)) return; - held.set(key, own); - await store.save(held); + const result = rememberResolution( + held, + { dir, repository_id: id, root: located.root, clone: located.clone }, + at.toISOString() + ); + if (result.changed) await store.save(held); } diff --git a/cli/src/contexts/telemetry/domain/report/attribution.ts b/cli/src/contexts/telemetry/domain/report/attribution.ts index d19a75f43..18cb8754c 100644 --- a/cli/src/contexts/telemetry/domain/report/attribution.ts +++ b/cli/src/contexts/telemetry/domain/report/attribution.ts @@ -26,6 +26,7 @@ export interface AttributableUsage { readonly at: string; readonly git_branch: string | null; readonly repository_id: string; + readonly repository_uncertain?: true; } const NO_BINDING: Attribution = { state: "unattributed", reason: "no-binding" }; @@ -153,6 +154,7 @@ function branchVerdict( * session's declaration, else the session's carry, else the branch's declaration, else no * task. Sub-agent and advisor calls name their parent's session, so they follow it. */ export function attribute(record: AttributableUsage, facts: AttributionFacts): Attribution { + if (record.repository_uncertain) return NO_BINDING; const at = Date.parse(record.at); return sessionVerdict(record, at, facts) ?? branchVerdict(record, at, facts) ?? NO_BINDING; } diff --git a/cli/src/contexts/telemetry/domain/report/session-feature.ts b/cli/src/contexts/telemetry/domain/report/session-feature.ts index 80a1cb1ed..4d7f0cea4 100644 --- a/cli/src/contexts/telemetry/domain/report/session-feature.ts +++ b/cli/src/contexts/telemetry/domain/report/session-feature.ts @@ -20,9 +20,12 @@ export function sameSessionAdmission( ): boolean { return Boolean( expected?.cwd && - expected.repositoryId && current?.cwd === expected.cwd && - current.repositoryId === expected.repositoryId + current.repositoryId === expected.repositoryId && + (expected.repositoryId || + (expected.repositoryId === null && + expected.feature.state !== "confirmed" && + current.feature.state !== "confirmed")) ); } @@ -66,6 +69,7 @@ export function sessionFeatureOf( let candidate: { feature: string; branch: string } | undefined; let missing = false; for (const record of records) { + if (record.repository_uncertain) return { state: "ambiguous" }; const resolution = attribute(record, facts); const feature = resolution.state === "attributed" ? resolution.ticket : null; const branch = record.git_branch; diff --git a/cli/src/contexts/telemetry/domain/report/usage-report.ts b/cli/src/contexts/telemetry/domain/report/usage-report.ts index d4335f418..7b5d1f5ec 100644 --- a/cli/src/contexts/telemetry/domain/report/usage-report.ts +++ b/cli/src/contexts/telemetry/domain/report/usage-report.ts @@ -116,7 +116,7 @@ export function axisValueOf( case "day": return value(new Date(record.at).toISOString().slice(0, 10)); case "repository": - return value(record.repository_id); + return value(record.repository_uncertain ? null : record.repository_id); case "task": return attribution.state === "attributed" ? value(attribution.task) diff --git a/cli/src/contexts/telemetry/domain/repository-resolution.ts b/cli/src/contexts/telemetry/domain/repository-resolution.ts index c7516a9fb..6fc62f15a 100644 --- a/cli/src/contexts/telemetry/domain/repository-resolution.ts +++ b/cli/src/contexts/telemetry/domain/repository-resolution.ts @@ -1,6 +1,11 @@ import { tryParseJson } from "../../../kernel/reading/json-file.js"; import { asPlainObject } from "../../../kernel/reading/plain-object.js"; -import { type CloneIdentity, cloneKey, parseCloneIdentity } from "./consent/clone-identity.js"; +import { + type CloneIdentity, + cloneKey, + parseCloneIdentity, + sameClone, +} from "./consent/clone-identity.js"; import { compareText } from "./text-order.js"; /** What a working directory was found to be, the first time it was seen alive in a clone. A @@ -16,6 +21,8 @@ export interface RepositoryResolution { readonly clone: CloneIdentity; /** When ingest first saw the directory in this clone. */ readonly seen_at: string; + /** Before a detected change, the origin of a call cannot be established. */ + readonly repository_uncertain_before?: string | null; } /** The key a directory is remembered under. File systems that ignore case (macOS, Windows) @@ -34,7 +41,73 @@ export function sameResolution( a: RepositoryResolution | undefined, b: RepositoryResolution ): boolean { - return a !== undefined && a.repository_id === b.repository_id && a.root === b.root; + return ( + a !== undefined && + a.repository_id === b.repository_id && + a.root === b.root && + a.repository_uncertain_before === b.repository_uncertain_before + ); +} + +/** Remember a detected repository change across all directories in the same clone. */ +export function rememberResolution( + resolutions: Map<string, RepositoryResolution>, + found: Omit<RepositoryResolution, "seen_at" | "repository_uncertain_before">, + at: string +): { readonly resolution: RepositoryResolution; readonly changed: boolean } { + const key = resolutionKey(found.dir, found.clone); + const previous = resolutions.get(key); + const related = [...resolutions.entries()].filter(([, value]) => + sameClone(value.clone, found.clone) + ); + const changedRepository = related.some( + ([, value]) => value.repository_id !== found.repository_id + ); + const boundaries = related.flatMap(([, value]) => + value.repository_uncertain_before ? [value.repository_uncertain_before] : [] + ); + if (changedRepository) boundaries.push(at); + const uncertainBefore = related.some(([, value]) => value.repository_uncertain_before === null) + ? null + : boundaries.sort((a, b) => Date.parse(a) - Date.parse(b)).at(-1); + let changed = false; + if (changedRepository) { + for (const [otherKey, value] of related) { + const replacement = { + ...value, + repository_id: found.repository_id, + repository_uncertain_before: uncertainBefore, + }; + if (!sameResolution(value, replacement)) { + resolutions.set(otherKey, replacement); + changed = true; + } + } + } + const resolution: RepositoryResolution = { + ...found, + // A root observed during opt-in establishes the first observation for its clone, + // including working directories first visited later. + seen_at: + previous?.seen_at ?? + related.map(([, value]) => value.seen_at).sort((a, b) => Date.parse(a) - Date.parse(b))[0] ?? + at, + ...(uncertainBefore !== undefined ? { repository_uncertain_before: uncertainBefore } : {}), + }; + if (!sameResolution(resolutions.get(key), resolution)) { + resolutions.set(key, resolution); + changed = true; + } + return { resolution, changed }; +} + +export function repositoryUncertain(resolution: RepositoryResolution, at: string): boolean { + return ( + Date.parse(at) < Date.parse(resolution.seen_at) || + resolution.repository_uncertain_before === null || + (resolution.repository_uncertain_before !== undefined && + Date.parse(at) <= Date.parse(resolution.repository_uncertain_before)) + ); } /** The instant from which a clone answered for a directory: its birth, else the time ingest @@ -45,22 +118,20 @@ function answeringFrom(resolution: RepositoryResolution): number { : Date.parse(resolution.seen_at); } -/** The clone that answers for a call made at `at` in a directory several clones have lived in: - * the latest to have started by then. A call older than all of them goes to the first, because +/** The clones that may answer for a call made at `at` in a directory several clones have lived in: + * those latest to have started by then. A call older than all of them goes to the earliest, because * a clone with no birth time is known only from the day ingest first saw a directory in it, and * a call made before that day in a directory of its own is still its call. That is no consent * on its own: the owner's interval must cover the call, and none opens before the clone's `on`. */ -export function ownerAt( +export function ownersAt( owners: readonly [RepositoryResolution, ...RepositoryResolution[]], at: number -): RepositoryResolution { - const [first, ...later] = [...owners].sort( - (a, b) => - answeringFrom(a) - answeringFrom(b) || compareText(resolutionKeyOf(a), resolutionKeyOf(b)) - ); - let owner = first as RepositoryResolution; - for (const candidate of later) if (answeringFrom(candidate) <= at) owner = candidate; - return owner; +): readonly RepositoryResolution[] { + const eligible = owners.filter((owner) => answeringFrom(owner) <= at); + const candidates = eligible.length ? eligible : owners; + const times = candidates.map(answeringFrom); + const selected = eligible.length ? Math.max(...times) : Math.min(...times); + return candidates.filter((owner) => answeringFrom(owner) === selected); } function resolutionKeyOf(resolution: RepositoryResolution): string { @@ -72,12 +143,31 @@ function parseResolution(value: unknown): RepositoryResolution | null { if (object === null) return null; const clone = parseCloneIdentity(object.clone); if (clone === null) return null; - const { dir, repository_id: repositoryId, root, seen_at: seenAt } = object; + const { + dir, + repository_id: repositoryId, + root, + seen_at: seenAt, + repository_uncertain_before: uncertainBefore, + } = object; if (typeof dir !== "string" || dir === "") return null; if (typeof repositoryId !== "string" || repositoryId === "") return null; if (typeof root !== "string") return null; if (typeof seenAt !== "string" || Number.isNaN(Date.parse(seenAt))) return null; - return { dir, repository_id: repositoryId, root, clone, seen_at: seenAt }; + const uncertainty = + uncertainBefore === undefined + ? undefined + : typeof uncertainBefore === "string" && !Number.isNaN(Date.parse(uncertainBefore)) + ? uncertainBefore + : null; + return { + dir, + repository_id: repositoryId, + root, + clone, + seen_at: seenAt, + ...(uncertainty !== undefined ? { repository_uncertain_before: uncertainty } : {}), + }; } /** The format of `roots.json`. A file of another format, among them every file written before diff --git a/cli/src/contexts/telemetry/domain/stored-usage.ts b/cli/src/contexts/telemetry/domain/stored-usage.ts index f7181e684..62d715bfe 100644 --- a/cli/src/contexts/telemetry/domain/stored-usage.ts +++ b/cli/src/contexts/telemetry/domain/stored-usage.ts @@ -10,6 +10,8 @@ import { USAGE_AGENTS, USAGE_TOOLS, type UsageRecord } from "./usage-record.js"; * be gone by the time anyone reports. */ export interface StoredUsage extends UsageRecord { readonly repository_id: string; + /** The ID is retained for clone consent, but must not be displayed or attributed. */ + readonly repository_uncertain?: true; } const COUNTERS = [ @@ -144,6 +146,7 @@ function isStoredUsage( typeof object.at === "string" && dayOf(object.at) !== null && isText(object.repository_id) && + (object.repository_uncertain === undefined || object.repository_uncertain === true) && isNullableText(object.tool_version) && isNullableText(object.model) && isNullableText(object.cwd) && diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts index 62635372b..f584e906e 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -24,6 +24,7 @@ import { evidenceTransfers, type Identity, type IdentityEvidence, + OtelObservationTooLargeError, observeIdentity, packTransfers, readBindingDetails, @@ -47,11 +48,15 @@ interface Pending { /** This range was admitted but could not be delivered; preserve it even if later denied. */ readonly hold?: boolean; readonly admissionKey?: string; + readonly cwdKey?: string; + readonly cloneKey?: string; } interface SessionState { readonly identity: Identity; readonly identityEvidence: IdentityEvidence[]; identityAdmissionKey: string | null; + identityAdmission: SessionAdmission | null; + identityCloneKey: string | null; previousLinkUnverified: boolean; savedIdentity: BindingDetails | null; mayHaveLinked: boolean; @@ -79,6 +84,9 @@ function asOptions(options: AllOptions, session: string): SyncOptions { function admissionKey(admission: SessionAdmission): string { return admissionFingerprint(admission); } +function cwdKey(cwd: string): string { + return createHash("sha256").update(cwd).digest("hex"); +} async function privateDirectory(path: string) { await mkdir(path, { recursive: true, mode: 0o700 }); const info = await lstat(path); @@ -135,7 +143,7 @@ async function saveCheckpoint(path: string, value: Checkpoint) { function pendingLine(item: Pending): string { if (item.hold && !item.admissionKey) throw new Error("Sync pending admission is unavailable."); - return `${item.offset} ${item.length}${item.hold ? ` hold ${item.admissionKey}` : ""}\n`; + return `${item.offset} ${item.length}${item.hold ? ` hold ${item.admissionKey}${item.cwdKey && item.cloneKey ? ` ${item.cwdKey} ${item.cloneKey}` : ""}` : ""}\n`; } /** A pending journal contains only byte ranges, not source text or provider identities. */ @@ -179,7 +187,8 @@ async function pendingRanges(path: string): Promise<Pending[]> { const lines = contents.split("\n"); lines.pop(); // A crash can leave an incomplete last append; its source line is replayed. for (const line of lines) { - const parts = /^(\d+) (\d+)(?: hold ([a-f0-9]{64}))?$/.exec(line); + const parts = + /^(\d+) (\d+)(?: hold ([a-f0-9]{64})(?: ([a-f0-9]{64}) ([a-f0-9]{64}))?)?$/.exec(line); if (!parts) throw new Error("Sync pending journal is invalid."); const offset = Number(parts[1]); const length = Number(parts[2]); @@ -194,11 +203,18 @@ async function pendingRanges(path: string): Promise<Pending[]> { const previous = ranges.get(rangeKey); if (previous?.admissionKey && parts[3] && previous.admissionKey !== parts[3]) throw new Error("Sync pending journal has conflicting admissions."); + if ( + (previous?.cwdKey && parts[4] && previous.cwdKey !== parts[4]) || + (previous?.cloneKey && parts[5] && previous.cloneKey !== parts[5]) + ) + throw new Error("Sync pending journal has conflicting clone proofs."); ranges.set(rangeKey, { offset, length, hold: !!parts[3] || previous?.hold, admissionKey: parts[3] ?? previous?.admissionKey, + cwdKey: parts[4] ?? previous?.cwdKey, + cloneKey: parts[5] ?? previous?.cloneKey, }); } return [...ranges.values()]; @@ -252,7 +268,13 @@ export async function syncAllOtel( at: string, start?: string, current?: string - ) => Promise<Consent> + ) => Promise<Consent>, + cloneProof?: ( + admission: SessionAdmission, + at: string, + start?: string, + current?: string + ) => Promise<string | null> ) { const base = asOptions(options, "auto"); destination(base, "logs"); @@ -308,6 +330,7 @@ export async function syncAllOtel( const sessions = new Map<string, SessionState>(); let sent = 0; let denied = 0; + const oversized = new Set<string>(); const pendingDirectory = join(directory, `${key(options, "auto")}.pending`); await privateDirectory(pendingDirectory); const pendingPath = (id: string) => join(pendingDirectory, `${key(options, id)}.jsonl`); @@ -324,6 +347,8 @@ export async function syncAllOtel( }, identityEvidence: [] as IdentityEvidence[], identityAdmissionKey: null, + identityAdmission: null, + identityCloneKey: null, previousLinkUnverified: saved?.mayHaveLinked === true, savedIdentity: saved?.details ?? null, mayHaveLinked: saved?.mayHaveLinked === true, @@ -352,6 +377,8 @@ export async function syncAllOtel( }, identityEvidence: [], identityAdmissionKey: null, + identityAdmission: null, + identityCloneKey: null, previousLinkUnverified: saved.mayHaveLinked, savedIdentity: saved.details, mayHaveLinked: saved.mayHaveLinked, @@ -382,6 +409,55 @@ export async function syncAllOtel( ) => { const saved = state.savedIdentity; const proof = saved?.identityProof; + if ( + saved && + proof?.version === 2 && + proof.admissionKey !== admissionKey(admission) && + admission.cwd && + admission.repositoryId === null && + admission.feature.state === "ambiguous" && + proof.cwdKey === cwdKey(admission.cwd) && + proof.cloneKey && + cloneProof && + state.mayHaveLinked && + saved.binding !== "invalidated" + ) { + const matches = async () => { + if ( + !(await stillPermitted( + id, + admission, + evidenceTransfers(proof.sources), + admission.feature + )) + ) + return false; + const now = new Date().toISOString(); + for (const source of proof.sources) + if ((await cloneProof(admission, source.at, source.start, now)) !== proof.cloneKey) + return false; + return true; + }; + if (!(await matches())) return; + const event = bindingTransfer( + asOptions(options, id), + { values: new Map(saved.identity), conflict: saved.conflict === true }, + { state: "ambiguous" } + ); + if (!event || !(await deliver(id, event, matches))) return; + sent++; + await saveBindingState( + join(directory, `${key(options, id)}.json`), + directory, + { session: id, identity: [], binding: "invalidated" }, + false + ); + state.binding = "invalidated"; + state.mayHaveLinked = false; + state.previousLinkUnverified = false; + state.savedIdentity = null; + return; + } if ( !saved || !proof || @@ -389,18 +465,56 @@ export async function syncAllOtel( !(await stillPermitted(id, admission, evidenceTransfers(proof.sources))) ) return; + if (proof.version === 2) { + if ( + !admission.cwd || + proof.cwdKey !== cwdKey(admission.cwd) || + !proof.cloneKey || + !cloneProof + ) + return; + const now = new Date().toISOString(); + for (const source of proof.sources) + if ((await cloneProof(admission, source.at, source.start, now)) !== proof.cloneKey) + return; + } state.identity.values.clear(); for (const [name, value] of saved.identity) state.identity.values.set(name, value); state.identity.conflict ||= saved.conflict === true; state.identityEvidence.splice(0, state.identityEvidence.length, ...proof.sources); state.identityAdmissionKey = proof.admissionKey; + state.identityAdmission = admission; + state.identityCloneKey = proof.version === 2 ? (proof.cloneKey ?? null) : null; state.previousLinkUnverified = false; state.savedIdentity = null; }; - const stateProof = (state: SessionState): BindingDetails["identityProof"] => - state.identityAdmissionKey && state.identityEvidence.length - ? { version: 1, admissionKey: state.identityAdmissionKey, sources: state.identityEvidence } - : undefined; + const stateProof = async (state: SessionState): Promise<BindingDetails["identityProof"]> => { + if (!state.identityAdmissionKey || !state.identityEvidence.length) return undefined; + const admission = state.identityAdmission; + if (cloneProof && admission?.cwd) { + let clone: string | null = null; + const now = new Date().toISOString(); + for (const source of state.identityEvidence) { + const found = await cloneProof(admission, source.at, source.start, now); + if (!found || (clone && found !== clone)) return undefined; + clone = found; + } + if (clone && state.identityCloneKey === clone) + return { + version: 2, + admissionKey: state.identityAdmissionKey, + cwdKey: cwdKey(admission.cwd), + cloneKey: clone, + sources: state.identityEvidence, + }; + return undefined; + } + return { + version: 1, + admissionKey: state.identityAdmissionKey, + sources: state.identityEvidence, + }; + }; const deliver = async (id: string, transfer: Transfer, guard: () => Promise<boolean>) => { try { await send(asOptions(options, id), transfer, guard); @@ -410,31 +524,107 @@ export async function syncAllOtel( throw error; } }; + const proofForTransfers = async ( + admission: SessionAdmission, + transfers: readonly Transfer[] + ) => { + if (!cloneProof || !admission.cwd || transfers.length === 0) return null; + const now = new Date().toISOString(); + let clone: string | null = null; + for (const transfer of transfers) { + if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) return null; + const found = await cloneProof(admission, transfer.at, transfer.start, now); + if (!found || (clone && found !== clone)) return null; + clone = found; + } + return clone; + }; + const reconcileIdentity = async ( + id: string, + admission: SessionAdmission, + state: SessionState + ) => { + if (!state.identityEvidence.length) return true; + const current = (await snapshot()).get(id); + if (!current || !sameSessionAdmission(admission, current)) return false; + if (state.identityAdmissionKey === admissionKey(admission)) { + const consentAt = new Date().toISOString(); + let allowed = true; + for (const item of evidenceTransfers(state.identityEvidence)) + if (!item.at || (await permits(current, item.at, item.start, consentAt)) !== "allow") { + allowed = false; + break; + } + if (allowed) return true; + } + + const previous = state.identityAdmission; + if ( + state.mayHaveLinked && + state.binding !== "invalidated" && + previous?.cwd && + previous.repositoryId && + state.identityAdmissionKey === admissionKey(previous) && + admission.cwd === previous.cwd && + admission.repositoryId === null && + admission.feature.state === "ambiguous" && + (!cloneProof || + (state.identityCloneKey && + (await proofForTransfers(admission, evidenceTransfers(state.identityEvidence))) === + state.identityCloneKey)) + ) { + const event = bindingTransfer(asOptions(options, id), state.identity, { + state: "ambiguous", + }); + if (event) { + const evidence = [event, ...evidenceTransfers(state.identityEvidence)]; + if ( + !(await deliver(id, event, () => + stillPermitted(id, admission, evidence, admission.feature) + )) + ) + return false; + sent++; + state.binding = "invalidated"; + await saveBindingState( + join(directory, `${key(options, id)}.json`), + directory, + { + session: id, + identity: [], + binding: state.binding, + }, + false + ); + state.mayHaveLinked = false; + } + } + if (state.mayHaveLinked) state.previousLinkUnverified = true; + state.identity.values.clear(); + state.identityEvidence.length = 0; + state.identityAdmissionKey = null; + state.identityAdmission = null; + state.identityCloneKey = null; + return true; + }; const process = async ( id: string, items: readonly Transfer[], admission: SessionAdmission, state: SessionState, - hold = false + hold = false, + expectedClone?: string, + captured?: { clone: string | null } ): Promise<"resolved" | "unknown" | "hold"> => { let wait = false; + let rejected = 0; const allowed: Transfer[] = []; const verdicts = new Map<string, Consent>(); await restoreIdentity(id, admission, state); - const currentAdmissionKey = admissionKey(admission); - if ( - state.identityEvidence.length > 0 && - (state.identityAdmissionKey !== currentAdmissionKey || - !(await stillPermitted(id, admission, evidenceTransfers(state.identityEvidence)))) - ) { - if (state.mayHaveLinked) state.previousLinkUnverified = true; - state.identity.values.clear(); - state.identityEvidence.length = 0; - state.identityAdmissionKey = null; - } + if (!(await reconcileIdentity(id, admission, state))) return "unknown"; for (const transfer of items) { if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { - denied++; + rejected++; continue; } const stamp = `${transfer.at}\0${transfer.start ?? ""}`; @@ -445,12 +635,49 @@ export async function syncAllOtel( } if (verdict === "unknown") wait = true; else if (verdict === "deny") { + // A repository can change between the snapshot and the consent check. A denial + // against that stale admission is not evidence that the call lacked consent. + if (!sameSessionAdmission(admission, (await snapshot()).get(id))) return "unknown"; if (hold) wait = true; - else denied++; + else rejected++; } else { allowed.push(transfer); - observeIdentity(transfer, asOptions(options, id), state.identity, state.identityEvidence); - if (state.identityEvidence.length) state.identityAdmissionKey = currentAdmissionKey; + } + } + // Retrying a partly sent source line would resend its earlier observations. + if (wait) return hold ? "hold" : "unknown"; + let batches: ReturnType<typeof packTransfers>; + try { + batches = packTransfers(allowed, false); + } catch (error) { + if (!(error instanceof OtelObservationTooLargeError)) throw error; + batches = []; + } + if (allowed.length > 0 && batches.length !== 1) { + if (!oversized.has(id)) { + console.error( + "A captured OTLP line exceeds the 2 MiB delivery limit; it remains pending." + ); + oversized.add(id); + } + return hold ? "hold" : "unknown"; + } + const clone = allowed.length ? await proofForTransfers(admission, allowed) : null; + if ( + allowed.length && + cloneProof && + (!clone || + (expectedClone && clone !== expectedClone) || + (state.identityCloneKey && state.identityCloneKey !== clone)) + ) + return hold ? "hold" : "unknown"; + if (captured) captured.clone = clone; + for (const transfer of allowed) { + observeIdentity(transfer, asOptions(options, id), state.identity, state.identityEvidence); + if (state.identityEvidence.length) { + state.identityAdmissionKey = admissionKey(admission); + state.identityAdmission = admission; + state.identityCloneKey ??= clone; } } if (state.identity.conflict && !state.previousLinkUnverified) @@ -462,21 +689,31 @@ export async function syncAllOtel( identity: [...state.identity.values], binding: state.binding, conflict: true, - identityProof: stateProof(state), + identityProof: await stateProof(state), }, state.mayHaveLinked ); - for (const batch of packTransfers(allowed)) { - if (!(await deliver(id, batch.transfer, () => stillPermitted(id, admission, batch.items)))) + for (const batch of batches) { + if ( + !(await deliver( + id, + batch.transfer, + async () => + (await stillPermitted(id, admission, batch.items)) && + (!clone || (await proofForTransfers(admission, batch.items)) === clone) + )) + ) return "hold"; sent += batch.count; } - return wait ? (hold ? "hold" : "unknown") : "resolved"; + denied += rejected; + return "resolved"; }; const bindings = async () => { for (const [id, state] of sessions) { const admission = held.get(id); if (admission) await restoreIdentity(id, admission, state); + if (admission && !(await reconcileIdentity(id, admission, state))) continue; if (!admission || state.previousLinkUnverified) continue; const feature: SessionFeature = admission.feature; const next = @@ -515,7 +752,7 @@ export async function syncAllOtel( session: id, identity: [...state.identity.values], binding: state.binding, - identityProof: stateProof(state), + identityProof: await stateProof(state), }); } } else if ( @@ -525,21 +762,31 @@ export async function syncAllOtel( ) { const event = bindingTransfer(asOptions(options, id), state.identity, feature); if (event) { + const proof = await stateProof(state); + if (cloneProof && proof?.version !== 2) continue; await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { session: id, identity: [...state.identity.values], binding: next, - identityProof: stateProof(state), + identityProof: proof, }); state.mayHaveLinked = true; if ( - !(await deliver(id, event, () => - stillPermitted( - id, - admission, - [event, ...evidenceTransfers(state.identityEvidence)], - feature - ) + !(await deliver( + id, + event, + async () => + (await stillPermitted( + id, + admission, + [event, ...evidenceTransfers(state.identityEvidence)], + feature + )) && + (!proof?.cloneKey || + (await proofForTransfers( + admission, + evidenceTransfers(state.identityEvidence) + )) === proof.cloneKey) )) ) continue; @@ -578,10 +825,6 @@ export async function syncAllOtel( const state = await sessionState(id); const waiting: Pending[] = []; for (const item of await pendingRanges(path)) { - if (item.hold && item.admissionKey !== admissionKey(admission)) { - waiting.push(item); - continue; - } const raw = Buffer.alloc(item.length); const read = await file?.read(raw, 0, item.length, item.offset); if (!read || read.bytesRead !== item.length || raw.at(-1) !== 10) @@ -590,13 +833,40 @@ export async function syncAllOtel( values: new Map(), conflict: false, }); - const status = await process(id, transfers, admission, state, item.hold); + const changedAdmission = item.hold && item.admissionKey !== admissionKey(admission); + if ( + changedAdmission && + (!item.cwdKey || + !item.cloneKey || + !admission.cwd || + admission.repositoryId !== null || + admission.feature.state !== "ambiguous" || + item.cwdKey !== cwdKey(admission.cwd)) + ) { + waiting.push(item); + continue; + } + const captured = { clone: null as string | null }; + const status = await process( + id, + transfers, + admission, + state, + item.hold, + item.cloneKey, + captured + ); if (status !== "resolved") waiting.push({ ...item, - hold: status === "hold", + hold: item.hold || status === "hold", admissionKey: - status === "hold" ? (item.admissionKey ?? admissionKey(admission)) : undefined, + item.hold || status === "hold" + ? (item.admissionKey ?? admissionKey(admission)) + : undefined, + ...(status === "hold" && admission.cwd && captured.clone && !item.hold + ? { cwdKey: cwdKey(admission.cwd), cloneKey: captured.clone } + : {}), }); } await replacePending(path, waiting); @@ -627,11 +897,12 @@ export async function syncAllOtel( conflict: false, }); const admission = held.get(id); + const captured = { clone: null as string | null }; const status = items.length === 0 ? "resolved" : admission - ? await process(id, items, admission, state) + ? await process(id, items, admission, state, false, undefined, captured) : "unknown"; if (status !== "resolved") await appendPending(pendingPath(id), { @@ -640,6 +911,9 @@ export async function syncAllOtel( hold: status === "hold", admissionKey: status === "hold" && admission ? admissionKey(admission) : undefined, + ...(status === "hold" && admission?.cwd && captured.clone + ? { cwdKey: cwdKey(admission.cwd), cloneKey: captured.clone } + : {}), }); } offset += newline + 1; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index 3f377a8c5..e0ace5b44 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -33,7 +33,7 @@ export interface SyncOptions { readonly onRepositoryUnavailable?: () => void; readonly onTitleUnavailable?: () => void; readonly titlesOnly?: boolean; - readonly admission?: { readonly cwd: string; readonly repositoryId: string }; + readonly admission?: { readonly cwd: string; readonly repositoryId: string | null }; } type RecordObject = Record<string, unknown>; @@ -41,6 +41,12 @@ const object = (value: unknown): RecordObject => asPlainObject(value) ?? {}; const array = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); const MAX_PAYLOAD = 2 * 1024 * 1024; const MAX_LINE = 32 * 1024 * 1024; + +export class OtelObservationTooLargeError extends Error { + constructor() { + super("An OTLP observation exceeds the receiver's 2 MiB limit."); + } +} const INTERVAL_MS = 5000; const CLAUDE_METRICS = new Set([ "claude_code.active_time.total", @@ -315,14 +321,14 @@ export interface Transfer { readonly consentAt?: string; } -/** Each item was separately consent-checked. Only points at the same instant and from the - * same session are combined, with a hard upper bound matching the receiver. */ +/** Each item was separately consent-checked. The receiver's size limit remains mandatory. */ export function packTransfers( - transfers: readonly Transfer[] + transfers: readonly Transfer[], + groupByTimestamp = true ): Array<{ transfer: Transfer; count: number; items: Transfer[] }> { const groups = new Map<string, Array<{ transfer: Transfer; count: number; items: Transfer[] }>>(); for (const item of transfers) { - const key = `${item.signal}\0${item.at ?? ""}`; + const key = groupByTimestamp ? `${item.signal}\0${item.at ?? ""}` : item.signal; const root = item.signal === "logs" ? "resourceLogs" : "resourceMetrics"; const resources = array(item.payload[root]); const batches = groups.get(key) ?? []; @@ -331,7 +337,7 @@ export function packTransfers( const body = { [root]: merged }; if (Buffer.byteLength(JSON.stringify(body)) > MAX_PAYLOAD) { if (Buffer.byteLength(JSON.stringify(item.payload)) > MAX_PAYLOAD) - throw new Error("An OTLP observation exceeds the receiver's 2 MiB limit."); + throw new OtelObservationTooLargeError(); batches.push({ transfer: item, count: 1, items: [item] }); } else if (last) { batches[batches.length - 1] = { @@ -729,8 +735,10 @@ export interface BindingDetails { readonly binding: string; readonly conflict?: boolean; readonly identityProof?: { - readonly version: 1; + readonly version: 1 | 2; readonly admissionKey: string; + readonly cwdKey?: string; + readonly cloneKey?: string; readonly sources: IdentityEvidence[]; }; } @@ -741,9 +749,14 @@ function readIdentityProof( ): BindingDetails["identityProof"] { const proof = object(value); if ( - proof.version !== 1 || + (proof.version !== 1 && proof.version !== 2) || typeof proof.admissionKey !== "string" || !/^[a-f0-9]{64}$/.test(proof.admissionKey) || + (proof.version === 2 && + (typeof proof.cwdKey !== "string" || + !/^[a-f0-9]{64}$/.test(proof.cwdKey) || + typeof proof.cloneKey !== "string" || + !/^[a-f0-9]{64}$/.test(proof.cloneKey))) || !Array.isArray(proof.sources) || proof.sources.length === 0 || proof.sources.length > 4 @@ -776,7 +789,14 @@ function readIdentityProof( }); } if (seen.size !== keys.size || keys.size === 0) return undefined; - return { version: 1, admissionKey: proof.admissionKey, sources }; + return { + version: proof.version as 1 | 2, + admissionKey: proof.admissionKey, + ...(proof.version === 2 + ? { cwdKey: proof.cwdKey as string, cloneKey: proof.cloneKey as string } + : {}), + sources, + }; } export async function readBindingDetails( diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 40b5d01e4..397a0be0c 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -335,7 +335,9 @@ export function registerTelemetryCommand(program: Command): void { }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), (admission, at, start, current) => - deps.telemetry.sessionFeatureUseCase.permits(admission, at, start, current) + deps.telemetry.sessionFeatureUseCase.permits(admission, at, start, current), + (admission, at, start, current) => + deps.telemetry.sessionFeatureUseCase.cloneProof(admission, at, start, current) ); output.info( `Sent ${result.sent} observations; ${result.denied} were outside consent; ${result.pending} await confirmation.` @@ -357,9 +359,15 @@ export function registerTelemetryCommand(program: Command): void { } if ( !initialAdmission.cwd || - !initialAdmission.repositoryId || + (!initialAdmission.repositoryId && + !( + initialAdmission.repositoryId === null && + initialAdmission.feature.state !== "confirmed" + )) || current.cwd !== initialAdmission.cwd || - current.repositoryId !== initialAdmission.repositoryId + (current.repositoryId !== initialAdmission.repositoryId && + !(current.repositoryId === null && current.feature.state === "ambiguous")) || + (current.repositoryId === null && current.feature.state === "confirmed") ) { admission = undefined; throw new Error( @@ -378,7 +386,10 @@ export function registerTelemetryCommand(program: Command): void { onProjectUnavailable, onRepositoryUnavailable, session, - ...(initialAdmission.cwd && initialAdmission.repositoryId + ...(initialAdmission.cwd && + (initialAdmission.repositoryId || + (initialAdmission.repositoryId === null && + initialAdmission.feature.state !== "confirmed")) ? { admission: { cwd: initialAdmission.cwd, diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index 47b8ac32c..69b8abdc3 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -57,9 +57,20 @@ export function followCapturedSessions( at: string, start?: string, current?: string - ) => Promise<"allow" | "deny" | "unknown"> + ) => Promise<"allow" | "deny" | "unknown">, + proof?: ( + admission: SessionAdmission, + at: string, + start?: string, + current?: string + ) => Promise<string | null> ) { - return syncAllOtel({ ...options, home: homedir(), stateDir: telemetryDir() }, snapshot, permits); + return syncAllOtel( + { ...options, home: homedir(), stateDir: telemetryDir() }, + snapshot, + permits, + proof + ); } export function syncCapturedSessionTitles( diff --git a/cli/src/runtime/wiring/telemetry.ts b/cli/src/runtime/wiring/telemetry.ts index 3910a9a01..e9a355683 100644 --- a/cli/src/runtime/wiring/telemetry.ts +++ b/cli/src/runtime/wiring/telemetry.ts @@ -13,6 +13,7 @@ import { StoredUsageMigration } from "../../contexts/telemetry/application/legac import { ReadUsageUseCase } from "../../contexts/telemetry/application/read-usage-use-case.js"; import { DeclaredBindings } from "../../contexts/telemetry/application/report/declared-bindings.js"; import { ReportUsageUseCase } from "../../contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../contexts/telemetry/application/report/repository-uncertainty.js"; import { SessionConsent } from "../../contexts/telemetry/application/report/session-consent.js"; import { SessionFeatureUseCase } from "../../contexts/telemetry/application/report/session-feature-use-case.js"; import { SnapshotBindingsUseCase } from "../../contexts/telemetry/application/snapshot-bindings-use-case.js"; @@ -181,7 +182,7 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { snapshotBindingsUseCase, reportUsageUseCase: new ReportUsageUseCase( ingestUsageUseCase, - ledger, + new ReportLedger(ledger, directoryResolver), new DeclaredBindings(sessions, snapshotStore), identity, locator, @@ -189,7 +190,7 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { ), sessionFeatureUseCase: new SessionFeatureUseCase( ingestUsageUseCase, - ledger, + new ReportLedger(ledger, directoryResolver), new DeclaredBindings(sessions, snapshotStore), new SessionConsent(ledger, directoryResolver, consentHistory) ), diff --git a/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts index b9938e1b9..5b756949b 100644 --- a/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts @@ -653,8 +653,7 @@ describe("where a call was made", () => { it("answers two spellings of a directory with one remembered resolution on a case-insensitive one", async () => { const s = setup({ caseInsensitive: true }); - s.locator.directories.set("/Work/A", repository("/Work/A")); - s.optIn("/Work/A/.git"); + s.locator.directories.set("/Work/A", repository("/Work/A", { clone: A })); s.transcripts.files.set("/t/1.jsonl", [line("A", 1, { cwd: "/Work/A" })]); await s.ingest.execute(); s.locator.directories.delete("/Work/A"); @@ -1049,6 +1048,41 @@ describe("a call is judged at its own time", () => { const EPOCH = "1970-01-01T00:00:00.000Z"; const B = cloneOf("/work/a/.git", { ino: "99", birthtimeMs: 5_000 }); + it("keeps tied macOS clone identities pending even when only one has consented", async () => { + const s = setup(); + const current = cloneOf(A.path, { dev: "0" }); + s.history.written.length = 0; + s.history.consented(A, "2026-10-01T00:00:00.000Z"); + s.history.written.push({ kind: "close", token: tokenOf(A), at: "2026-10-05T00:00:00.000Z" }); + s.history.consented(current, "2026-10-06T00:00:00.000Z"); + s.resolutions.resolutions.set(resolutionKey("/work/a", A), sighting("/work/a", A)); + s.locator.directories.set("/work/a", repository("/work/a", { clone: current })); + s.consents.clones.delete(cloneKey(A)); + s.consents.cloneSays(current, grantedBy(current)); + s.transcripts.files.set("/t/1.jsonl", [ + line("old", 1, {}, "2026-10-04T00:00:00.000Z"), + line("gap", 2, {}, "2026-10-05T00:00:00.000Z"), + line("new", 3, {}, "2026-10-07T00:00:00.000Z"), + ]); + const result = await s.ingest.execute(); + expect(s.ledger.records).toEqual([]); + expect(result.notStored["unreadable-consent"]).toBe(3); + }); + + it("does not choose between tied clones whose consent intervals overlap", async () => { + const s = setup(); + const current = cloneOf(A.path, { dev: "0" }); + s.history.consented(current); + s.resolutions.resolutions.set(resolutionKey("/work/a", A), sighting("/work/a", A)); + s.locator.directories.set("/work/a", repository("/work/a", { clone: current })); + s.consents.clones.delete(cloneKey(A)); + s.consents.cloneSays(current, grantedBy(current)); + s.transcripts.files.set("/t/1.jsonl", [line("ambiguous", 1)]); + const result = await s.ingest.execute(); + expect(s.ledger.records).toEqual([]); + expect(result.notStored["unreadable-consent"]).toBe(1); + }); + it("gives a clone two owners of one directory, each its own calls", async () => { const s = setup(); s.history.written.length = 0; diff --git a/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts b/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts index 4c8984c49..87f27080d 100644 --- a/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts @@ -10,6 +10,7 @@ import { ReadUsageUseCase } from "../../../../../src/contexts/telemetry/applicat import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; import { ReportUsageUseCase } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import type { CloneIdentity } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { readClaudeUsageLine } from "../../../../../src/contexts/telemetry/domain/formats/claude-transcript-usage.js"; @@ -241,13 +242,14 @@ function machine(name: string, storage: PrivateStorage = new PrivateStorageAdapt history.consented(clone); } const snapshots = new BindingSnapshotStoreAdapter(bindings, storage, aliases); + const directories = new DirectoryResolver(locator, consents, resolutions, history, { + caseInsensitiveFileSystem: false, + now: () => NOW, + }); const ingest = new IngestUsageUseCase( [new ReadUsageUseCase(transcripts, () => readClaudeUsageLine)], ledger, - new DirectoryResolver(locator, consents, resolutions, history, { - caseInsensitiveFileSystem: false, - now: () => NOW, - }), + directories, new SnapshotBindingsUseCase( new FakeBindings(), new InMemorySnapshots(), @@ -259,7 +261,7 @@ function machine(name: string, storage: PrivateStorage = new PrivateStorageAdapt ); const report = new ReportUsageUseCase( ingest, - ledger, + new ReportLedger(ledger, directories), new DeclaredBindings(new InMemorySessions(), snapshots), new InMemoryIdentity(), locator, @@ -287,7 +289,11 @@ async function bytes(dir: string): Promise<Record<string, string>> { for (const entry of await readdir(dir, { withFileTypes: true })) { if (!entry.isFile() || ["offsets.json", ".lock", "repository-ids.json"].includes(entry.name)) continue; - found[entry.name] = await readFile(join(dir, entry.name), "utf8"); + const text = await readFile(join(dir, entry.name), "utf8"); + // Earlier versions did not persist an uncertainty marker; reports recompute it. + found[entry.name] = /^\d{4}-\d{2}-\d{2}\.jsonl$/.test(entry.name) + ? text.replace(/,"repository_uncertain":true(?=\})/gu, "") + : text; } return Object.fromEntries(Object.entries(found).sort(([a], [b]) => (a < b ? -1 : 1))); } diff --git a/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts index f2baa7588..1abc2e821 100644 --- a/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts @@ -9,6 +9,7 @@ import { ReadUsageUseCase } from "../../../../../src/contexts/telemetry/applicat import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; import { ReportUsageUseCase } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import { cloneKey } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { readClaudeUsageLine } from "../../../../../src/contexts/telemetry/domain/formats/claude-transcript-usage.js"; @@ -206,7 +207,7 @@ function machine() { ); const report = new ReportUsageUseCase( ingest, - ledger, + new ReportLedger(ledger), new DeclaredBindings(new InMemorySessions(), new InMemorySnapshots()), new InMemoryIdentity(), locator, diff --git a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts index 4f18af287..2b808d6d7 100644 --- a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts @@ -15,6 +15,7 @@ import { type ReportRequest, ReportUsageUseCase, } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { declarationOf } from "../../../../../src/contexts/telemetry/domain/declaration/task-declaration.js"; import type { LeftOutEntry } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; import type { Period } from "../../../../../src/contexts/telemetry/domain/report/period.js"; @@ -274,7 +275,7 @@ async function report( return inZone(zone, async () => { const result = await new ReportUsageUseCase( ingest, - ledger, + new ReportLedger(ledger), new DeclaredBindings(sessions, new InMemorySnapshots()), identity, new FakeLocator(), diff --git a/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts index 9137092fb..23f82fd39 100644 --- a/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts @@ -13,6 +13,7 @@ import { type ReportRequest, ReportUsageUseCase, } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import { declarationOf } from "../../../../../src/contexts/telemetry/domain/declaration/task-declaration.js"; import type { Period } from "../../../../../src/contexts/telemetry/domain/report/period.js"; @@ -140,7 +141,7 @@ async function reporterOf({ calls, declarations, person }: Ledger) { { refusedByEnvironment: false }, ALREADY_MIGRATED ), - ledger, + new ReportLedger(ledger), new DeclaredBindings(sessions, snapshots), identity, new FakeLocator(), diff --git a/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts index e800a22ce..95a6d21be 100644 --- a/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts @@ -7,6 +7,7 @@ import { ReportUsageUseCase, type RequestedFilters, } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import { cloneKey } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { @@ -21,6 +22,7 @@ import { } from "../../../../../src/contexts/telemetry/domain/report/report-filter.js"; import type { ReportAxis } from "../../../../../src/contexts/telemetry/domain/report/usage-report.js"; import { repositoryIdOf } from "../../../../../src/contexts/telemetry/domain/repository-identity.js"; +import { rememberResolution } from "../../../../../src/contexts/telemetry/domain/repository-resolution.js"; import { ALREADY_MIGRATED, AUTUMN, @@ -74,13 +76,15 @@ function setup(options: { refused?: boolean } = {}) { const snapshotStore = new InMemorySnapshots(events); const sessions = new InMemorySessions(events); const identity = new InMemoryIdentity(); + const resolutions = new InMemoryResolutions(); + const directories = new DirectoryResolver(locator, consents, resolutions, history, { + caseInsensitiveFileSystem: false, + now: () => new Date("2026-10-09T00:00:00.000Z"), + }); const ingest = new IngestUsageUseCase( [new ReadUsageUseCase(transcripts, () => readClaudeUsageLine)], ledger, - new DirectoryResolver(locator, consents, new InMemoryResolutions(), history, { - caseInsensitiveFileSystem: false, - now: () => new Date("2026-10-09T00:00:00.000Z"), - }), + directories, new SnapshotBindingsUseCase( bindings, snapshotStore, @@ -98,17 +102,33 @@ function setup(options: { refused?: boolean } = {}) { remote: "https://github.com/acme/widgets.git", rootCommit: "c0ffee", }); + rememberResolution( + resolutions.resolutions, + { dir: "/work/a", root: "/work/a", clone: cloneOf("/work/a/.git"), repository_id: widgets() }, + "2026-01-01T00:00:00.000Z" + ); consents.cloneSays(cloneOf("/work/a/.git"), grantedBy(cloneOf("/work/a/.git"))); history.consented(cloneOf("/work/a/.git")); const report = new ReportUsageUseCase( ingest, - ledger, + new ReportLedger(ledger, directories), new DeclaredBindings(sessions, snapshotStore), identity, locator, { refusedByEnvironment: options.refused ?? false } ); - return { transcripts, ledger, bindings, sessions, identity, report, events, locator, consents }; + return { + transcripts, + ledger, + bindings, + sessions, + identity, + report, + events, + locator, + consents, + resolutions, + }; } async function rows(s: ReturnType<typeof setup>, axis: ReportAxis, period = ANY) { @@ -118,6 +138,39 @@ async function rows(s: ReturnType<typeof setup>, axis: ReportAxis, period = ANY) } describe("reporting usage", () => { + it("keeps tokens but stops attributing old usage after an origin change, even when it changes back", async () => { + const s = setup(); + s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); + expect(await rows(s, "repository")).toEqual([[{ kind: "value", value: widgets() }, 11]]); + const original = s.locator.directories.get("/work/a"); + if (original?.status !== "repository") throw new Error("missing repository"); + s.locator.directories.set("/work/a", { + ...original, + remote: "https://github.com/acme/other.git", + }); + expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); + expect(await rows(s, "task")).toEqual([[{ kind: "unattributed", reason: "no-binding" }, 11]]); + s.locator.directories.set("/work/a", original); + expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); + }); + + it("keeps tokens without repository attribution when resolution history was lost", async () => { + const s = setup(); + s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); + expect(await rows(s, "repository")).toEqual([[{ kind: "value", value: widgets() }, 11]]); + s.resolutions.resolutions.clear(); + expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); + }); + + it("keeps tokens without repository or task attribution when a directory is gone", async () => { + const s = setup(); + s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); + expect(await rows(s, "repository")).toEqual([[{ kind: "value", value: widgets() }, 11]]); + s.locator.directories.delete("/work/a"); + expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); + expect(await rows(s, "task")).toEqual([[{ kind: "unattributed", reason: "no-binding" }, 11]]); + }); + it("settles a call held in two partitions under the ledger lock", async () => { const s = setup(); s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); diff --git a/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts index 4b91d1039..107b6eede 100644 --- a/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts @@ -12,6 +12,7 @@ import { storeCalls } from "../../../../../src/contexts/telemetry/application/le import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; import { ReportUsageUseCase } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import type { UsageLedger } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; import { attribute } from "../../../../../src/contexts/telemetry/domain/report/attribution.js"; @@ -246,7 +247,7 @@ describe("a report on a bounded period", () => { await settled.leftOut.add(leftOut); const bounded = await new ReportUsageUseCase( ingestOf(settled), - settled, + new ReportLedger(settled), declared, identity, new FakeLocator(), @@ -256,7 +257,9 @@ describe("a report on a bounded period", () => { const fullLedger = await ledgerWith(files); await fullLedger.leftOut.add(leftOut); const full = await fullLedger.load(); - const kept = full.records.filter((record) => inPeriod(record.at, period)); + const kept = full.records + .filter((record) => inPeriod(record.at, period)) + .map((record) => ({ ...record, repository_uncertain: true as const })); // A session or repository typed by its start names a value among the period's calls. const resolved: { -readonly [K in keyof ReportFilters]: ReportFilters[K] } = { ...filters }; for (const flag of ["session", "repository"] as const) { @@ -264,7 +267,7 @@ describe("a report on a bounded period", () => { if (typed === null) continue; const found = resolveId( typed, - kept.map((record) => (flag === "session" ? record.session_id : record.repository_id)) + kept.flatMap((record) => (flag === "session" ? [record.session_id] : [])) ); if (found.kind === "ambiguous") { expect(bounded).toEqual({ status: "ambiguous", flag, typed, ids: found.ids }); diff --git a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts index 5849b8f85..55fe9a45b 100644 --- a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts @@ -52,6 +52,28 @@ function setup(history = new InMemoryConsentHistory()) { } describe("consent for an observed session", () => { + it("proves the consented clone and refuses a replacement at the same path", async () => { + const s = setup(); + const original = await s.checker.proof(admission, at, start, "2026-10-07T11:00:00.000Z"); + expect(original).toMatch(/^[a-f0-9]{64}$/); + expect(await s.checker.proof(admission, at, start, "2026-10-07T11:00:00.000Z")).toBe(original); + const replacement = cloneOf(`${cwd}/.git`, { + ino: "9", + birthtimeMs: Date.parse("2026-10-07T10:45:00.000Z"), + }); + s.locator.directories.set(cwd, { + status: "repository", + root: cwd, + mainRoot: cwd, + clone: replacement, + remote: null, + rootCommit: repo, + }); + s.history.consented(replacement, "2026-10-07T10:45:00.000Z"); + s.source.cloneSays(replacement, grantedBy(replacement)); + expect(await s.checker.proof(admission, at, start, "2026-10-07T11:00:00.000Z")).toBeNull(); + }); + it("allows an instant or whole interval only inside one clone's consent", async () => { const s = setup(); expect(await s.checker.permits(admission, at)).toBe("allow"); @@ -107,6 +129,42 @@ describe("consent for an observed session", () => { expect(await s.checker.permits({ ...admission, repositoryId: "other" }, at)).toBe("deny"); }); + it("allows unlinked token observations without a repository only for the same consented clone", async () => { + const s = setup(); + const uncertain = { ...admission, repositoryId: null }; + expect(await s.checker.permits(uncertain, at, start)).toBe("allow"); + expect( + await s.checker.permits( + { ...uncertain, feature: { state: "confirmed", feature: "owner/repo#1", branch: "work" } }, + at + ) + ).toBe("unknown"); + const replacement = cloneOf(`${cwd}/.git`, { + ino: "9", + birthtimeMs: Date.parse("2026-10-07T10:45:00.000Z"), + }); + s.resolutions.resolutions.set(resolutionKey(cwd, s.clone), { + dir: cwd, + root: cwd, + repository_id: repo, + clone: s.clone, + seen_at: "2026-10-07T09:00:00.000Z", + }); + s.locator.directories.set(cwd, { + status: "repository", + root: cwd, + mainRoot: cwd, + clone: replacement, + remote: null, + rootCommit: repo, + }); + s.history.consented(replacement, "2026-10-07T10:45:00.000Z"); + s.source.cloneSays(replacement, grantedBy(replacement)); + expect(await s.checker.permits(uncertain, at, undefined, "2026-10-07T11:00:00.000Z")).toBe( + "deny" + ); + }); + it.each(["start", "end"] as const)( "keeps a window unknown when its %s is unreadable and its other end is denied", async (unreadable) => { @@ -155,7 +213,6 @@ describe("consent for an observed session", () => { it.each([ [{ ...admission, cwd: null }, at, undefined], [{ ...admission, cwd: "" }, at, undefined], - [{ ...admission, repositoryId: null }, at, undefined], [admission, "invalid", undefined], [admission, at, "invalid"], [admission, start, at], diff --git a/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts index 3e5bfa5d3..2dbfe7546 100644 --- a/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts @@ -1,7 +1,19 @@ import { expect, it, vi } from "vitest"; +import { DirectoryResolver } from "../../../../../src/contexts/telemetry/application/directory-resolver.js"; import type { IngestResult } from "../../../../../src/contexts/telemetry/application/ingest-usage-use-case.js"; +import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; +import { SessionConsent } from "../../../../../src/contexts/telemetry/application/report/session-consent.js"; import { SessionFeatureUseCase } from "../../../../../src/contexts/telemetry/application/report/session-feature-use-case.js"; import type { UsageLedger } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; +import { + cloneOf, + FakeConsents, + FakeLocator, + grantedBy, + InMemoryConsentHistory, + InMemoryLedger, + InMemoryResolutions, +} from "../../../../helpers/ports/in-memory-telemetry.js"; it("refuses synchronization before reading the ledger when telemetry is disabled", async () => { const unexpected = (): never => { @@ -39,7 +51,7 @@ it("refuses synchronization before reading the ledger when telemetry is disabled }; const useCase = new SessionFeatureUseCase( { execute: async () => refused }, - ledger, + new ReportLedger(ledger), { read: unexpected }, { permits: unexpected } ); @@ -47,3 +59,86 @@ it("refuses synchronization before reading the ledger when telemetry is disabled await expect(useCase.snapshot("codex")).rejects.toThrow("telemetry is disabled"); expect(exclusively).not.toHaveBeenCalled(); }); + +it("withdraws a feature after an observed origin change without losing consented usage", async () => { + const cwd = "/work/repo"; + const clone = cloneOf(`${cwd}/.git`); + const locator = new FakeLocator(); + const source = new FakeConsents(); + const history = new InMemoryConsentHistory(); + const ledger = new InMemoryLedger(); + let now = "2026-10-07T09:30:00.000Z"; + const repository = { + status: "repository" as const, + root: cwd, + mainRoot: cwd, + clone, + remote: null, + rootCommit: "repo-a", + }; + locator.directories.set(cwd, repository); + source.cloneSays(clone, grantedBy(clone)); + history.consented(clone, "2026-10-07T09:00:00.000Z"); + const directories = new DirectoryResolver(locator, source, new InMemoryResolutions(), history, { + caseInsensitiveFileSystem: false, + now: () => new Date(now), + }); + const remembered = await directories.open(); + await remembered.resolve(cwd, now); + await remembered.close(); + now = "2026-10-07T11:00:00.000Z"; + ledger.records = [ + { + tool: "codex", + tool_version: null, + key: "request", + session_id: "session", + agent: "main", + at: "2026-10-07T10:00:00.000Z", + model: null, + input: 42, + output: 9, + cache_read: null, + cache_write: null, + cache_write_1h: null, + reasoning: null, + cwd, + git_branch: "work", + repository_id: "repo-a", + }, + ]; + const facts = { + declarations: [ + { + session_id: "session", + task: "work", + ticket: "owner/repo#7", + none: false, + declared_at: "2026-10-07T09:00:00.000Z", + by: "command" as const, + }, + ], + carries: [], + branches: new Map(), + }; + const useCase = new SessionFeatureUseCase( + { execute: async () => ({ refused: false }) as IngestResult }, + new ReportLedger(ledger, directories), + { read: async () => facts }, + new SessionConsent(ledger, directories, history) + ); + expect((await useCase.snapshot("codex")).get("session")).toMatchObject({ + repositoryId: "repo-a", + feature: { state: "confirmed", feature: "owner/repo#7" }, + }); + + now = "2026-10-07T12:00:00.000Z"; + locator.directories.set(cwd, { ...repository, rootCommit: "repo-b" }); + const admission = (await useCase.snapshot("codex")).get("session"); + expect(admission).toEqual({ cwd, repositoryId: null, feature: { state: "ambiguous" } }); + if (!admission) throw new Error("Missing session admission"); + expect(await useCase.permits(admission, "2026-10-07T10:00:00.000Z", undefined, now)).toBe( + "allow" + ); + expect(ledger.records[0]).toMatchObject({ input: 42, output: 9 }); +}); diff --git a/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts b/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts index 52edf4a8c..cb3ab83fe 100644 --- a/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts +++ b/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest"; import type { CloneIdentity } from "../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { cwdKey, - ownerAt, + ownersAt, parseResolutions, type RepositoryResolution, renderResolutions, @@ -119,18 +119,18 @@ describe("the clone that answers for a call", () => { const next = { ...resolution, clone: { ...CLONE, ino: "2", birthtimeMs: 5_000 } }; it("is the only clone there is, whenever the call was made", () => { - expect(ownerAt([old], 0)).toBe(old); - expect(ownerAt([old], 1e12)).toBe(old); + expect(ownersAt([old], 0)).toEqual([old]); + expect(ownersAt([old], 1e12)).toEqual([old]); }); it("is the latest clone born by then", () => { - expect(ownerAt([next, old], 4_999)).toBe(old); - expect(ownerAt([next, old], 5_000)).toBe(next); - expect(ownerAt([old, next], 9_000)).toBe(next); + expect(ownersAt([next, old], 4_999)).toEqual([old]); + expect(ownersAt([next, old], 5_000)).toEqual([next]); + expect(ownersAt([old, next], 9_000)).toEqual([next]); }); it("is the first when the call is older than every clone", () => { - expect(ownerAt([next, old], 10)).toBe(old); + expect(ownersAt([next, old], 10)).toEqual([old]); }); it("goes by the time ingest first saw the directory, where a clone has no birth time", () => { @@ -144,12 +144,13 @@ describe("the clone that answers for a call", () => { clone: { ...next.clone, birthtimeMs: 0 }, seen_at: "2026-10-05T00:00:00.000Z", }; - expect(ownerAt([late, early], Date.parse("2026-10-04T00:00:00.000Z"))).toBe(early); - expect(ownerAt([early, late], Date.parse("2026-10-05T00:00:00.000Z"))).toBe(late); + expect(ownersAt([late, early], Date.parse("2026-10-04T00:00:00.000Z"))).toEqual([early]); + expect(ownersAt([early, late], Date.parse("2026-10-05T00:00:00.000Z"))).toEqual([late]); }); - it("is the same whichever order the clones are given in", () => { + it("leaves tied birth times unresolved", () => { const tie = { ...old, clone: { ...old.clone, ino: "3" } }; - expect(ownerAt([old, tie], 1_000)).toBe(ownerAt([tie, old], 1_000)); + expect(ownersAt([old, tie], 1_000)).toEqual([old, tie]); + expect(ownersAt([tie, old], 1_000)).toEqual([tie, old]); }); }); diff --git a/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts b/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts index 7af55405b..b450503da 100644 --- a/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts +++ b/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest"; import type { AttributionFacts } from "../../../../src/contexts/telemetry/domain/report/attribution.js"; import { coversConsentWindow, + sameSessionAdmission, sessionFeatureOf, } from "../../../../src/contexts/telemetry/domain/report/session-feature.js"; import type { StoredUsage } from "../../../../src/contexts/telemetry/domain/stored-usage.js"; @@ -55,9 +56,23 @@ describe("session feature binding", () => { expect(sessionFeatureOf([call, { ...call, git_branch: null }], declared)).toEqual({ state: "ambiguous", }); + expect(sessionFeatureOf([call, { ...call, repository_uncertain: true }], declared)).toEqual({ + state: "ambiguous", + }); }); }); +it("keeps an uncertain repository unlinked while allowing the same session to continue", () => { + const uncertain = { feature: { state: "ambiguous" as const }, cwd: "/work", repositoryId: null }; + expect(sameSessionAdmission(uncertain, uncertain)).toBe(true); + expect(sameSessionAdmission(uncertain, { ...uncertain, cwd: "/elsewhere" })).toBe(false); + const confirmed = { + ...uncertain, + feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "work" }, + }; + expect(sameSessionAdmission(confirmed, confirmed)).toBe(false); +}); + it("never exports a cumulative metric across an off/on consent gap", () => { const clone = { path: "/repo/.git", dev: "1", ino: "2", birthtimeMs: 0 }; const intervals = [ diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts index 2d27b8e15..f065d04a9 100644 --- a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts @@ -97,6 +97,506 @@ it("discovers a newly consented session without transmitting a different session expect(received[0]).not.toContain("not-consented"); }); +it("forwards token observations for a consented session whose repository is uncertain", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-uncertain-repo-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "ambiguous" as const }, + cwd: "/project", + repositoryId: null, + }; + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["allowed", admission]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + const result = await running; + expect(result.sent).toBe(1); + expect(received[0]).toContain("input_token_count"); + expect(received[0]).not.toContain("aidd.session_feature"); +}); + +it("invalidates an in-memory binding after a repository change before forwarding new tokens", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-origin-change-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + let changed = false; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", async () => { + received.push(body); + if (body.includes("aidd.session_feature_invalidated")) { + await appendFile(capture, line("allowed")); + } else if (body.includes("aidd.session_feature")) { + changed = true; + await appendFile(capture, line("allowed")); + } + response.writeHead(200).end("{}"); + if ( + changed && + !body.includes("aidd.session_feature") && + received.some((item) => item.includes("aidd.session_feature_invalidated")) + ) + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const known = { + feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "work" }, + cwd: "/project", + repositoryId: "repo-a", + }; + const uncertain = { + feature: { state: "ambiguous" as const }, + cwd: "/project", + repositoryId: null, + }; + const timeout = setTimeout(() => abort.abort(), 3000); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + refreshMs: 20, + }, + async () => new Map([["allowed", changed ? uncertain : known]]), + async () => "allow" + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + const result = await running; + clearTimeout(timeout); + const invalidation = received.findIndex((body) => + body.includes("aidd.session_feature_invalidated") + ); + expect(invalidation).toBeGreaterThan(0); + expect(received.slice(invalidation + 1).some((body) => body.includes("input_token_count"))).toBe( + true + ); + expect(result.pending).toBe(0); +}); + +it("retries a denial caused by an admission changing during the consent check", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-stale-admission-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const known = { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "repo" }; + const uncertain = { + feature: { state: "ambiguous" as const }, + cwd: "/project", + repositoryId: null, + }; + let changed = false; + const timeout = setTimeout(() => abort.abort(), 3000); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + refreshMs: 20, + }, + async () => new Map([["allowed", changed ? uncertain : known]]), + async (admission) => { + if (admission.repositoryId !== null) { + changed = true; + return "deny"; + } + return "allow"; + } + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + const result = await running; + clearTimeout(timeout); + expect(result.denied).toBe(0); + expect(received.some((body) => body.includes("input_token_count"))).toBe(true); +}); + +it("forwards consented observations when the same capture line also contains denied or undated records", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-mixed-consent-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const validAt = new Date(1793763900000).toISOString(); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["allowed", admission]]), + async (_admission, at) => (at === validAt ? "allow" : "deny"), + async (_admission, at) => (at === validAt ? "a".repeat(64) : null) + ); + const payload = JSON.parse(line("allowed")); + const records = payload.payload.resourceLogs[0].scopeLogs[0].logRecords; + const valid = records[0]; + const denied = structuredClone(valid); + denied.timeUnixNano = "1793763800000000000"; + denied.attributes.find( + (entry: { key: string }) => entry.key === "input_token_count" + ).value.stringValue = "17"; + const undated = structuredClone(valid); + delete undated.timeUnixNano; + undated.attributes.find( + (entry: { key: string }) => entry.key === "input_token_count" + ).value.stringValue = "28"; + records.push(denied, undated); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, `${JSON.stringify(payload)}\n`); + const result = await running; + expect(result.denied).toBe(2); + expect(result.pending).toBe(0); + expect(received).toHaveLength(1); + expect(received[0]).toContain('"42"'); + expect(received[0]).not.toContain('"17"'); + expect(received[0]).not.toContain('"28"'); +}); + +it("waits for every consent verdict before sending a capture line once", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-unknown-batch-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const firstAt = new Date(1793763900000).toISOString(); + let ready = false; + const timeout = setTimeout(() => abort.abort(), 3000); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + refreshMs: 20, + }, + async () => new Map([["allowed", admission]]), + async (_admission, at) => (at === firstAt || ready ? "allow" : "unknown"), + async () => "a".repeat(64) + ); + const payload = JSON.parse(line("allowed")); + const records = payload.payload.resourceLogs[0].scopeLogs[0].logRecords; + const second = structuredClone(records[0]); + second.timeUnixNano = "1793763901000000000"; + records.push(second); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, `${JSON.stringify(payload)}\n`); + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(received).toHaveLength(0); + ready = true; + const result = await running; + clearTimeout(timeout); + expect(result.pending).toBe(0); + expect(received).toHaveLength(1); + expect(received[0]).toContain("1793763900000000000"); + expect(received[0]).toContain("1793763901000000000"); +}); + +it("keeps an oversized line pending without sending a partial batch", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-oversize-batch-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + let received = 0; + server = createServer((request, response) => { + request.resume(); + received++; + response.writeHead(200).end("{}"); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const abort = new AbortController(); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["allowed", admission]]), + async () => "allow", + async () => "a".repeat(64) + ); + const payload = JSON.parse(line("allowed")); + const records = payload.payload.resourceLogs[0].scopeLogs[0].logRecords; + records[0].attributes.push(attribute("model", "x".repeat(1_100_000))); + records.push(structuredClone(records[0])); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, `${JSON.stringify(payload)}\n`); + await new Promise((resolve) => setTimeout(resolve, 200)); + abort.abort(); + const result = await running; + expect(result.pending).toBe(1); + expect(received).toBe(0); +}); + +it("keeps one oversized observation pending and continues with the next line", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-single-oversize-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + const abort = new AbortController(); + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + abort.abort(); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const admission = { + feature: { state: "unbound" as const }, + cwd: "/project", + repositoryId: "repo", + }; + const timeout = setTimeout(() => abort.abort(), 3000); + const running = syncAllOtel( + { + config: "codex", + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + signal: abort.signal, + intervalMs: 10, + }, + async () => new Map([["allowed", admission]]), + async () => "allow", + async () => "a".repeat(64) + ); + const payload = JSON.parse(line("allowed")); + payload.payload.resourceLogs[0].scopeLogs[0].logRecords[0].attributes.push( + attribute("model", "x".repeat(2_100_000)) + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, `${JSON.stringify(payload)}\n${line("allowed")}`); + const result = await running; + clearTimeout(timeout); + expect(result.pending).toBe(1); + expect(received).toHaveLength(1); + expect(received[0]).toContain("input_token_count"); + expect(received[0]).not.toContain("xxxxxxxxxxxxxxxxxxxxxxxx"); +}); + +it("replays a held v2 range after restart only for the original consented clone", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-pending-v2-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + refreshMs: 20, + }; + const known = { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "repo" }; + const uncertain = { + feature: { state: "ambiguous" as const }, + cwd: "/project", + repositoryId: null, + }; + const firstAbort = new AbortController(); + const first = syncAllOtel( + { ...options, signal: firstAbort.signal }, + async () => new Map([["allowed", known]]), + async (_admission, _at, _start, current) => (current ? "unknown" : "allow"), + async () => "a".repeat(64) + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 100)); + firstAbort.abort(); + expect((await first).pending).toBe(1); + const secondAbort = new AbortController(); + const second = syncAllOtel( + { ...options, signal: secondAbort.signal }, + async () => new Map([["allowed", uncertain]]), + async () => "allow", + async () => "b".repeat(64) + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + secondAbort.abort(); + expect((await second).pending).toBe(1); + expect(received).toHaveLength(0); + const thirdAbort = new AbortController(); + const third = syncAllOtel( + { ...options, signal: thirdAbort.signal }, + async () => new Map([["allowed", uncertain]]), + async () => "allow", + async () => "a".repeat(64) + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + thirdAbort.abort(); + expect((await third).pending).toBe(0); + expect(received.some((body) => body.includes("input_token_count"))).toBe(true); +}); + it("replays a failed post from its last durable checkpoint", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-retry-")); dirs.push(home); @@ -794,6 +1294,83 @@ it("recovers an interrupted pending append without losing either session on the ).toBe(true); }); +it("invalidates a v2 binding after restart only for the original consented clone", async () => { + const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-v2-")); + dirs.push(home); + const captureDir = join(home, ".local/share/aidd-otel-observation"); + await mkdir(captureDir, { recursive: true, mode: 0o700 }); + const capture = join(captureDir, "codex.jsonl"); + await writeFile(capture, "", { mode: 0o600 }); + const received: string[] = []; + server = createServer((request, response) => { + let body = ""; + request.on("data", (chunk: Buffer) => { + body += chunk.toString(); + }); + request.on("end", () => { + received.push(body); + response.writeHead(200).end("{}"); + }); + }); + await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") throw new Error("No test port"); + const options = { + config: "codex" as const, + tenant: "tenant", + endpoint: `http://127.0.0.1:${address.port}`, + token: "test-token", + home, + stateDir: join(home, "state"), + intervalMs: 10, + }; + const admission = { + feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, + cwd: "/project", + repositoryId: "repo", + }; + const uncertain = { + feature: { state: "ambiguous" as const }, + cwd: "/project", + repositoryId: null, + }; + const firstAbort = new AbortController(); + const first = syncAllOtel( + { ...options, signal: firstAbort.signal }, + async () => new Map([["allowed", admission]]), + async () => "allow", + async () => "a".repeat(64) + ); + await new Promise((resolve) => setTimeout(resolve, 80)); + await appendFile(capture, line("allowed")); + await new Promise((resolve) => setTimeout(resolve, 120)); + firstAbort.abort(); + await first; + expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); + const secondAbort = new AbortController(); + const second = syncAllOtel( + { ...options, signal: secondAbort.signal }, + async () => new Map([["allowed", uncertain]]), + async () => "allow", + async () => "b".repeat(64) + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + secondAbort.abort(); + await second; + expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(false); + const thirdAbort = new AbortController(); + const third = syncAllOtel( + { ...options, signal: thirdAbort.signal }, + async () => new Map([["allowed", uncertain]]), + async () => "allow", + async () => "a".repeat(64) + ); + await new Promise((resolve) => setTimeout(resolve, 100)); + thirdAbort.abort(); + await third; + expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(true); +}); + it("invalidates a persisted feature after restart when the session becomes ambiguous", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-restart-")); dirs.push(home); diff --git a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts index c08c4df01..2f2e880dd 100644 --- a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts +++ b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts @@ -149,7 +149,8 @@ describe("OTLP runtime wiring", () => { expect(automatic).toHaveBeenCalledWith( { ...options, home: "/profile", stateDir: "/profile/telemetry" }, snapshot, - permits + permits, + undefined ); }); }); diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 0a962eeb7..f81bb9e80 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -44,6 +44,7 @@ The local ledger and provider OTLP captures are separate sources. The OTLP sync - `aidd telemetry off` stops local measurement for the current clone; it does **not** disable provider exporters or stop the OTLP collector. A process started with `AIDD_TELEMETRY=0` refuses local measurement and sync, not raw collection; changing an environment variable does not change an already running process. - `aidd telemetry forget` previews or deletes local usage measurement. It does **not** delete OTLP capture files, sync state, or data already received by Gouvernail. Production deletion is not available; the requested production policy is to disallow it. - Raw captures are private local files, but have no automatic retention limit. They can contain sensitive content. The local collector and sync processes do not restart automatically. +- Automatic sync sends each session's capture line as one request. A line over 2 MiB remains pending and is reported; later lines continue. A timeout after receiver acceptance cannot guarantee exactly-once delivery without receiver-side idempotence. ### Three different identifiers @@ -74,6 +75,9 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. - Only clones that ran `on` are measured, from then on. - A task and a ticket come only from what you declare. - Each model call counts once, whatever the axis. +- If an origin change is observed in the same clone, usage before its detection keeps its tokens but loses repository and task attribution. Changes made and reversed between observations, or changes overwritten before this protection existed, cannot be detected retrospectively. +- Repository attribution is unavailable for usage older than the first recorded observation of its clone. If the local resolution history is lost, attribution remains unavailable for earlier usage. +- A directory that is no longer available cannot establish its current origin, so its usage keeps its tokens without repository or task attribution. - Where you are asked for a task, and the limits: see the usage contract. ## Local usage measurement privacy From c1dcad9e85a884385ea13fb71b5e59c03a25eb8d Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 18:39:19 +0800 Subject: [PATCH 27/56] test(telemetry): align report observation timeline --- cli/tests/e2e/telemetry-report-sandbox.ts | 13 +++++++++++++ cli/tests/e2e/telemetry-report.e2e.test.ts | 16 ++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/cli/tests/e2e/telemetry-report-sandbox.ts b/cli/tests/e2e/telemetry-report-sandbox.ts index 772c3e519..9da1d5908 100644 --- a/cli/tests/e2e/telemetry-report-sandbox.ts +++ b/cli/tests/e2e/telemetry-report-sandbox.ts @@ -265,12 +265,25 @@ export function useReportSandbox(): void { // Opting in is `on`: a key set by hand opens no consent interval, and nothing is stored. expect((await aidd(["on", "--yes", "--anonymous"])).exitCode).toBe(0); await backdateConsent(telemetry); + // The synthetic calls also need a repository observation from before they ran. + await dateRootObservation("2026-09-01T00:00:00.000Z"); }); afterEach(async () => { await env.cleanup(); }); } +/** Simulate when the sandbox's clone was first observed, alongside its backdated consent. */ +export async function dateRootObservation(at: string): Promise<void> { + const path = join(telemetry, "ledger", "roots.json"); + const roots = JSON.parse(await readFile(path, "utf8")) as { + directories: Array<{ seen_at: string }>; + }; + if (roots.directories.length !== 1) throw new Error("Expected one remembered clone"); + roots.directories[0].seen_at = at; + await writeFile(path, `${JSON.stringify(roots, null, 2)}\n`); +} + /** Appends a call to the sandbox's main transcript, as Claude would write it later. */ export async function addCall(call: Call): Promise<void> { await appendFile(join(claude, "projects", "sandbox", "main.jsonl"), `${usageLine(call, repo)}\n`); diff --git a/cli/tests/e2e/telemetry-report.e2e.test.ts b/cli/tests/e2e/telemetry-report.e2e.test.ts index 60f03c3a9..dba34813b 100644 --- a/cli/tests/e2e/telemetry-report.e2e.test.ts +++ b/cli/tests/e2e/telemetry-report.e2e.test.ts @@ -5,6 +5,7 @@ import { CALLS, type Call, COUNTERS, + dateRootObservation, declareBranch, type Envelope, held, @@ -19,6 +20,21 @@ import { useReportSandbox(); describe("aidd telemetry report", () => { + it("keeps tokens without repository or task attribution before the first clone observation", async () => { + await dateRootObservation("2026-10-09T00:00:00.000Z"); + const repository = await json("repository"); + expect(repository.totals.records).toBe(CALLS.length); + expect(repository.rows).toHaveLength(1); + expect(repository.rows[0]).toMatchObject({ kind: "absent", records: CALLS.length }); + const tasks = await json("task"); + expect(tasks.rows).toHaveLength(1); + expect(tasks.rows[0]).toMatchObject({ + kind: "unattributed", + reason: "no-binding", + records: CALLS.length, + }); + }); + it("answers on every axis, and every axis adds up to the same totals", async () => { await declareBranch(); const known = (pick: (call: Call) => number | null) => From fd35066a303bcb9e3ced9d6175ac4d13fde00ef4 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Thu, 5 Nov 2026 22:25:36 +0800 Subject: [PATCH 28/56] fix(telemetry): forward observed Claude effort Keep the effort attached to its source point so it is not inferred for other observations. --- cli/scripts/check-bundle-size.mjs | 2 +- .../infrastructure/otel/actions/sync.ts | 17 +++++++++++-- .../sync-otel-attributes.integration.test.ts | 25 +++++++++++++++++++ plugins/aidd-telemetry/OBSERVABILITY.md | 5 ++-- 4 files changed, 44 insertions(+), 5 deletions(-) diff --git a/cli/scripts/check-bundle-size.mjs b/cli/scripts/check-bundle-size.mjs index 051e8e45a..ae07558de 100644 --- a/cli/scripts/check-bundle-size.mjs +++ b/cli/scripts/check-bundle-size.mjs @@ -74,7 +74,7 @@ const pkg = JSON.parse(readFileSync(resolve(root, "package.json"), "utf8")); // 813 KB: 812.66 KB with consent and session-admission checks on every delivery attempt. // 819 KB: 818.39 KB with per-identifier provenance and isolated pending session ranges. // 829 KB: 828.93 KB with repository uncertainty and clone-proven resume after origin changes. -// 839 KB: 838.61 KB after merging Antigravity support into the telemetry branch. +// 839 KB: 838.97 KB with Claude token-point effort and merged Antigravity support. const budgetKB = pkg.bundleBudgetKB ?? 500; const budgetBytes = budgetKB * 1024; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index e0ace5b44..23292026a 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -517,7 +517,8 @@ export function selectTransfers( for (const rawPoint of array(aggregation.dataPoints)) { const point = object(rawPoint); validateMetricPoint(point, kind); - const a = merge(resourceAttrs, attributes(point.attributes)); + const pointAttrs = attributes(point.attributes); + const a = merge(resourceAttrs, pointAttrs); if (a.get(sessionKey) !== options.session) continue; captureIdentity(identity, a); a.set( @@ -538,9 +539,21 @@ export function selectTransfers( point.attributes ) ); + const safe = safeAttributes(a); + const effort = array(point.attributes) + .map(object) + .find((entry) => entry.key === "effort"); + const effortValue = object(effort?.value).stringValue; + if ( + metric.name === "claude_code.token.usage" && + a.get("model")?.trim() && + typeof effortValue === "string" && + /^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$/.test(effortValue) + ) + safe.push({ key: "effort", value: { stringValue: effortValue } }); const sanitized = { ...numericFields(point, METRIC_POINT_FIELDS), - attributes: safeAttributes(a), + attributes: safe, }; out.push({ signal: "metrics", diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts index 50eda0175..ea9a23b38 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -206,6 +206,31 @@ test("does not forward Codex effort fields on Claude metrics", () => { expect(JSON.stringify(transfer.payload)).not.toContain("reasoning_effort"); }); +test("forwards Claude effort only when the token point explicitly reports it", () => { + const claude = { ...options, config: "claude" as const }; + const point = [text("type", "input"), text("model", "claude-test")]; + const [reported] = selectTransfers(metricCapture([...point, text("effort", "medium")]), claude, { + values: new Map(), + conflict: false, + }); + expect(JSON.stringify(reported.payload)).toContain( + '"key":"effort","value":{"stringValue":"medium"}' + ); + for (const invalid of [ + metricCapture(point, [text("effort", "medium")]), + metricCapture([...point, text("effort", "private text")]), + metricCapture([...point, attribute("effort", { intValue: 3 })]), + metricCapture([text("type", "input"), text("effort", "medium")]), + metricCapture([...point, text("effort", "medium")]).replace( + "claude_code.token.usage", + "claude_code.cost.usage" + ), + ]) { + const [transfer] = selectTransfers(invalid, claude, { values: new Map(), conflict: false }); + expect(JSON.stringify(transfer.payload)).not.toContain('"key":"effort"'); + } +}); + test.each(["point", "resource", "scope"])("rejects contradictory Claude %s attributes", (level) => { const conflict = [text("type", "input"), text("type", "output")]; expect(() => diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index bb0556d28..fb800c19e 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -14,7 +14,7 @@ The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 | Session | `session-abc` | `session.id` | `conversation.id`; trace `thread.id` | **v1:** `info.sessionID`, `part.sessionID`; **v2:** `data.sessionID`, event `durable.aggregateID` | | Provider | `openai` | `anthropic.provider`, `gen_ai.provider.name`, `gen_ai.system` | `provider_name`, trace `provider` | **v1:** assistant `info.providerID`, user `info.model.providerID`; **v2:** `data.model.providerID` on step start | | Model | `model-xyz` | `model`, `gen_ai.request.model` | `model` | **v1:** assistant `info.modelID`, user `info.model.modelID`; **v2:** `data.model.id` on step start | -| Reasoning effort | `high` | Not verified | `reasoning_effort` on `codex.conversation_starts`; `model_reasoning_effort` on `response.completed` | Not verified | +| Reasoning effort | `high` | `effort` on `claude_code.token.usage` points when supplied | `reasoning_effort` on `codex.conversation_starts`; `model_reasoning_effort` on `response.completed` | Not verified | | User input | `Fix the test` | `prompt`, `prompt_text`, span `user_prompt` | `prompt` | **v1:** `part.text` joined by `messageID` to a user message; **v2:** `data.item.payload.text` on `session.inbox.enqueued` with `data.item.type=user` | | Assistant response | `The test is fixed.` | `response` | `response` | **v1:** `part.text` joined by `messageID` to an assistant message; **v2:** `data.text` on `session.text.ended`, `data.delta` on `session.text.delta` | | Input tokens | `1,200` | `input_tokens`; `claude_code.token.usage` metric, `type=input` | `input_token_count`, `codex.turn.token_usage.input_tokens`, `gen_ai.usage.input_tokens`; metric `token_type=input` | **v1:** `info.tokens.input` or `part.tokens.input`; **v2:** `data.tokens.input` | @@ -32,10 +32,11 @@ Provider labels are not necessarily equivalent across tools. A user role does no ## Interpretation and coverage - The Claude and Codex metric `type` / `token_type` labels above belong to `claude_code.token.usage` and `codex.turn.token_usage`, respectively. OTLP logs, traces and metrics may describe the same operation; do not add them together. +- Claude effort is forwarded only when present on a token point with its model. It is an observation of that point, not a value inferred for other points. [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage#token-counter) (read 2026-11-05). - Codex effort is an observation on one event, not a value inherited by other turns. The sync forwards only bounded ASCII identifiers when the same event also names its model; a missing or invalid value stays unknown. [Codex event source](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/otel/src/events/session_telemetry.rs) (read 2026-11-05). - OpenCode 2 emits `data.tokens.*` both on `session.step.ended` and `session.usage.updated`. The latter is cumulative; summing every update overcounts. Its `input` is non-cached input and `output` is visible output; reasoning is separate. The implementation maps missing provider usage to zero, so a reported zero does not prove zero usage. - Claude's `input_tokens` excludes cache reads and writes. Normalize token definitions before comparison; do not assume a similarly named field counts the same thing in all tools. -- In the examined capture, Codex cache-write fields were present but all zero. A nonzero write was not verified. Not every event includes every field; OpenCode 2 text and model data may need correlation across events. Outputs can be truncated. +- In the examined capture, Codex cache-write fields were present but all zero. Codex can replace absent provider cache details with zeros, so a reported zero does not establish that the provider supplied a zero. [Codex source](https://github.com/openai/codex/blob/rust-v0.166.0-alpha.6/codex-rs/codex-api/src/sse/responses.rs#L115-L145) (read 2026-11-05). A nonzero write was not verified. Not every event includes every field; OpenCode 2 text and model data may need correlation across events. Outputs can be truncated. - Claude can emit `cost_usd`; OpenCode 1 has assistant and step-finish `cost`, and OpenCode 2 exposes step and cumulative cost. A trustworthy equivalent was not established for Codex in the observed captures. These values do not by themselves establish an actual invoice: pricing basis and completeness must be known, and cumulative and per-step figures must not be combined as independent costs. - A future feature may display an **estimated token cost**, not an invoice. It would require the provider and model, price source and effective date, currency, and rates for input, output, cache reads and cache writes. Subscription, included quota, negotiated and API pricing can differ. If usage, a compatible rate or a billing basis is unknown, the estimate is unknown rather than zero. Cost calculation and pricing are outside this feature. - A tool session ID is not automatically a feature ID. Feature attribution requires the framework's session/branch declaration and must handle missing or ambiguous bindings. The existing local usage ledger and provider OTLP export are distinct pipelines. From af42e1c6f4832dd779b81cb6e5fe329d94ed437e Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 09:58:58 +0800 Subject: [PATCH 29/56] refactor(telemetry): remove non-OTel session metadata --- aidd_docs/recipes/observe-agent-telemetry.md | 48 +- cli/aidd_docs/memory/codebase-map.md | 1 - cli/package.json | 1 - cli/scripts/smoke-codex-project.mjs | 240 ------- cli/scripts/smoke-tools.sh | 14 +- .../observations/telemetry-observation.ts | 23 +- .../infrastructure/otel/actions/init.ts | 2 +- .../otel/actions/metadata/session-project.ts | 51 -- .../actions/metadata/session-repository.ts | 49 -- .../otel/actions/metadata/session-title.ts | 266 -------- .../infrastructure/otel/actions/run.ts | 49 -- .../otel/actions/sync-all-titles.ts | 154 ----- .../infrastructure/otel/actions/sync-all.ts | 451 +------------ .../infrastructure/otel/actions/sync.ts | 451 +------------ .../infrastructure/otel/actions/verify.ts | 47 +- .../infrastructure/otel/config-edits.ts | 41 +- .../infrastructure/otel/config/opencode.json | 3 +- .../infrastructure/otel/observations.ts | 231 ++----- .../telemetry/infrastructure/otel/settings.ts | 68 +- cli/src/presentation/commands/telemetry.ts | 131 +--- cli/src/runtime/wiring/telemetry-otel.ts | 38 +- .../observations/transformers.unit.test.ts | 303 ++------- .../telemetry/otel/config-edits.unit.test.ts | 40 -- .../otel/init-otel.integration.test.ts | 53 +- .../otel/run-otel.integration.test.ts | 74 --- .../otel/session-project.unit.test.ts | 81 --- .../otel/session-repository.unit.test.ts | 76 --- .../session-title-sync.integration.test.ts | 420 ------------ .../telemetry/otel/session-title.unit.test.ts | 156 ----- .../otel/settings.integration.test.ts | 40 +- .../otel/sync-all-otel.integration.test.ts | 627 +----------------- .../sync-otel-attributes.integration.test.ts | 34 +- .../sync-otel-dimensions.integration.test.ts | 2 +- .../sync-otel-integrity.integration.test.ts | 133 +--- .../otel/sync-otel.integration.test.ts | 63 +- .../otel/verify-otel.integration.test.ts | 52 -- .../e2e/telemetry-otel-metadata.e2e.test.ts | 254 ------- cli/tests/e2e/telemetry-otel.e2e.test.ts | 80 +-- cli/tests/golden/snapshots/help/surface.json | 9 +- .../telemetry-wiring.integration.test.ts | 17 +- .../runtime/wiring/otel-wiring.unit.test.ts | 22 +- plugins/aidd-telemetry/OBSERVABILITY.md | 148 +---- plugins/aidd-telemetry/README.md | 16 +- 43 files changed, 286 insertions(+), 4773 deletions(-) delete mode 100644 cli/scripts/smoke-codex-project.mjs delete mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts delete mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts delete mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts delete mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts delete mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts delete mode 100644 cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts delete mode 100644 cli/tests/contexts/telemetry/otel/session-project.unit.test.ts delete mode 100644 cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts delete mode 100644 cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts delete mode 100644 cli/tests/contexts/telemetry/otel/session-title.unit.test.ts delete mode 100644 cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index a189b17fb..3ca4380e3 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -1,23 +1,14 @@ # Observe agent telemetry locally -Configure native telemetry for Claude Code, Codex and OpenCode against the private development collector, then verify receipt. +This procedure captures provider OpenTelemetry (OTel) for Claude Code, Codex and OpenCode v2. The framework's separate local usage ledger is not the source for the Gouvernail OTel view. ## Steps to observe agent telemetry locally -### 1) ๐Ÿ”’ Check the destination +### 1) โš™๏ธ Configure -Telemetry may contain prompts, replies, commands, tool results and secrets. Use it only with the consent of those being observed, protect access to the logs and decide when to delete them. The CLI changes the selected tool's user configuration; this is separate from the per-clone `aidd telemetry on` usage ledger. +Obtain consent from people whose activity will be observed. Raw local captures can contain prompts, responses, commands, tool output and secrets; restrict access and decide when to delete them. -| Destination | Availability | Behavior | -| --- | --- | --- | -| `dev` | Private test machine with the existing loopback collector | Configure the selected tool to export locally | -| `prod` | Reserved for a future Gouvernail collector | Refuse without changing files until its destination is defined | - -The development collector and its private token must already exist. Initialization does not start a collector or set up a production endpoint. - -### 2) โš™๏ธ Initialize each tool - -Run the command for each tool you intend to observe. Existing settings must be preserved; repeat initialization without adding duplicate entries. Follow the CLI's output for version-specific limitations and restarts. +Start the private development collector first, then configure each tool: ```sh aidd telemetry otel init --config=claude --endpoint=dev @@ -25,13 +16,32 @@ aidd telemetry otel init --config=codex --endpoint=dev aidd telemetry otel init --config=opencode --endpoint=dev ``` -OpenCode v1 reads the OTLP destination and headers from the process environment. After initialization, launch it through `aidd telemetry otel run --config=opencode` and pass any OpenCode arguments after `--`. OpenCode v2 supports user service environment configuration and requires a service restart. The v1 `experimental.openTelemetry` option enables AI SDK instrumentation; v2.2.5 rejects that option. Native OTLP logs and traces are distinct from the v2 session SSE stream. See the [v1 instrumentation](https://github.com/anomalyco/opencode/blob/v1.19.5/packages/opencode/src/session/llm.ts#L343-L355), [v2 configuration](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/config/normalize.ts#L46) and [v2 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts), accessed 2026-11-04. +`dev` requires the existing loopback collector and its private token. `prod` is reserved and refuses configuration until a destination is defined. Restart the OpenCode v2 service after initialization so its process receives the exporter settings. + +### 2) ๐Ÿงช Run two sessions per tool -Claude and Codex also support native OTLP logs, metrics and traces; Claude tracing is beta. See [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage) and [Codex observability](https://learn.chatgpt.com/docs/config-file/config-advanced#observability-and-telemetry), accessed 2026-11-04. +Run these from the opted-in project. The examples are the commands used in the local observation, with the isolated test settings omitted. They create separate sessions rather than two turns in the same session. + +```sh +for i in 1 2; do + claude --model haiku --tools "" --output-format json -p "Reply with only GOUVERNAIL-CLAUDE-PROBE-$i." +done + +for i in 1 2; do + codex exec --ephemeral --sandbox read-only --model gpt-6.1-terra \ + -c 'model_reasoning_effort="low"' --json "Reply with only GOUVERNAIL-CODEX-PROBE-$i." +done + +# Start the configured OpenCode v2 service separately with authentication enabled. +for i in 1 2; do + opencode run --server http://127.0.0.1:51394 --format json \ + --model openai/gpt-6-luna "Reply with only GOUVERNAIL-OPENCODE-PROBE-$i." +done +``` -### 3) ๐Ÿงช Verify receipt +The OpenCode example assumes the authenticated service is listening on port `51394` and the client has its credentials. Choose models available to the configured accounts. Claude and Codex emit token usage; in the observed OpenCode v2.2.5 capture, prompt spans contained session IDs but no model or token counts. See the [field inventory](../../plugins/aidd-telemetry/OBSERVABILITY.md). -Run a short session with the selected tool, then ask the CLI to check its configuration and captured telemetry: +### 3) โœ… Check receipt ```sh aidd telemetry otel verify --config=claude --endpoint=dev @@ -39,6 +49,6 @@ aidd telemetry otel verify --config=codex --endpoint=dev aidd telemetry otel verify --config=opencode --endpoint=dev ``` -Use only the command for the tool you configured. Verification reads the selected tool's local capture and reports its nonempty line count and modification time. Historical records prove previous receipt, not delivery from the current session; compare the reported time with your test session. Follow the CLI's failure message if configuration or captured lines are missing. +Verification reports capture line counts and modification times. Old records do not prove delivery from the current sessions. Collection and consent-aware forwarding to Gouvernail are separate; capture alone does not create dashboard rows. -Captured files need a retention policy; initialization and verification do not rotate or delete them. +Sources accessed 2026-11-06: [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage), [Codex telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted), [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts). diff --git a/cli/aidd_docs/memory/codebase-map.md b/cli/aidd_docs/memory/codebase-map.md index 30f38b435..f03fa23a7 100644 --- a/cli/aidd_docs/memory/codebase-map.md +++ b/cli/aidd_docs/memory/codebase-map.md @@ -75,7 +75,6 @@ src/ โ”‚ โ”‚ โ””โ”€โ”€ infrastructure/ โ”‚ โ”‚ โ”œโ”€โ”€ otel/ # shared native OTLP settings; JSON templates in config/ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ actions/ # initialize configuration, launch export, verify receipt -โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ metadata/ # consented native Codex session metadata โ”‚ โ”‚ โ”œโ”€โ”€ consent/ # reading a clone's identity from the file system, and the consent history file โ”‚ โ”‚ โ”œโ”€โ”€ declaration/ # where a declaration is kept: git config and the telemetry dir โ”‚ โ”‚ โ”œโ”€โ”€ forget/ # erasing the telemetry dir, the previous version's files, git config keys diff --git a/cli/package.json b/cli/package.json index 0354ff4fc..9f073e60c 100644 --- a/cli/package.json +++ b/cli/package.json @@ -54,7 +54,6 @@ "smoke": "pnpm build && bash scripts/smoke-tools.sh", "smoke:full": "pnpm build && SMOKE_REMOTE=1 bash scripts/smoke-tools.sh", "smoke:real": "pnpm build && bash scripts/smoke-real.sh", - "smoke:codex-project": "node scripts/smoke-codex-project.mjs", "smoke:collision": "pnpm build && bash scripts/smoke-collision.sh", "typecheck": "tsc --noEmit", "lint": "biome check .", diff --git a/cli/scripts/smoke-codex-project.mjs b/cli/scripts/smoke-codex-project.mjs deleted file mode 100644 index af0bcca47..000000000 --- a/cli/scripts/smoke-codex-project.mjs +++ /dev/null @@ -1,240 +0,0 @@ -import assert from "node:assert/strict"; -import { spawn, spawnSync } from "node:child_process"; -import { once } from "node:events"; -import { access, mkdir, mkdtemp, realpath, rm } from "node:fs/promises"; -import { createServer } from "node:net"; -import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; -import { createInterface } from "node:readline"; -import { fileURLToPath, pathToFileURL } from "node:url"; -import { build } from "tsup"; - -// Opt-in native smoke: no provider credentials, prompts, turns, or network access. -// Unlike the deterministic mocked tests, this checks the installed Codex protocol. -const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const sandbox = "/usr/bin/sandbox-exec"; -const policy = "(version 1) (allow default) (deny network*)"; - -function executable(name) { - const result = spawnSync("which", [name], { encoding: "utf8" }); - if (result.status !== 0 || !result.stdout.trim()) throw new Error(`${name} is required`); - return realpath(result.stdout.trim().split("\n")[0]); -} - -async function run(command, args, env) { - const child = spawn(command, args, { env, stdio: ["ignore", "ignore", "pipe"] }); - let stderr = ""; - child.stderr.on("data", (data) => { - stderr += data; - }); - const [code] = await once(child, "exit"); - if (code !== 0) throw new Error(`${command} failed: ${stderr.trim()}`); -} - -async function verifyNetworkBlocked(env) { - const server = createServer((socket) => socket.end()); - server.listen(0, "127.0.0.1"); - await once(server, "listening"); - try { - const { port } = server.address(); - const probe = - `const s = require('node:net').connect(${port}, '127.0.0.1');\n` + - "s.on('connect', () => process.exit(2));\n" + - "s.on('error', e => process.exit(['EACCES', 'EPERM'].includes(e.code) ? 0 : 3));\n" + - "setTimeout(() => process.exit(4), 3000);"; - await run(sandbox, ["-p", policy, process.execPath, "-e", probe], env); - } finally { - await new Promise((done) => server.close(done)); - } -} - -function rpc(child) { - let nextId = 0; - let stderr = ""; - let failure; - const pending = new Map(); - child.stderr.on("data", (data) => { - stderr += data; - }); - const lines = createInterface({ input: child.stdout }); - lines.on("line", (line) => { - let response; - try { - response = JSON.parse(line); - } catch { - return; - } - const callback = pending.get(response.id); - if (!callback) return; - pending.delete(response.id); - if (response.error) callback.reject(new Error(response.error.message)); - else callback.resolve(response.result); - }); - const fail = (error) => { - failure ??= error; - for (const callback of pending.values()) callback.reject(failure); - pending.clear(); - }; - child.on("error", fail); - child.stdin.on("error", fail); - child.on("exit", () => fail(new Error(`Codex app-server exited: ${stderr.trim()}`))); - return { - request(method, params) { - if (failure) return Promise.reject(failure); - const id = ++nextId; - return new Promise((resolveRequest, reject) => { - const timer = setTimeout(() => { - pending.delete(id); - reject(new Error(`Codex request timed out: ${method}`)); - }, 10_000); - pending.set(id, { - resolve(value) { - clearTimeout(timer); - resolveRequest(value); - }, - reject(error) { - clearTimeout(timer); - reject(error); - }, - }); - child.stdin.write(`${JSON.stringify({ id, method, params })}\n`); - }); - }, - notify(method) { - child.stdin.write(`${JSON.stringify({ method, params: {} })}\n`); - }, - close() { - fail(new Error("Codex app-server closed")); - lines.close(); - }, - }; -} - -async function stop(child) { - if (!child?.pid || child.exitCode !== null || child.signalCode !== null) return; - const exited = once(child, "exit").then(() => true); - async function wait() { - let timer; - try { - return await Promise.race([ - exited, - new Promise((resolveWait) => { - timer = setTimeout(() => resolveWait(false), 2_000); - }), - ]); - } finally { - clearTimeout(timer); - } - } - child.kill(); - if (await wait()) return; - child.kill("SIGKILL"); - if (await wait()) return; - child.stdin.destroy(); - child.stdout.destroy(); - child.stderr.destroy(); - child.unref(); - throw new Error("Codex app-server did not exit after SIGKILL"); -} - -async function main() { - if (process.platform !== "darwin") { - throw new Error( - "This native smoke requires macOS sandbox-exec; no unsandboxed fallback is allowed" - ); - } - await access(sandbox); - const codex = await executable("codex"); - const temp = await mkdtemp(join(tmpdir(), "aidd-codex-native-")); - let child; - try { - const home = join(temp, "home"); - const work = join(temp, "work"); - const codexHome = join(home, ".codex"); - await Promise.all([mkdir(codexHome, { recursive: true }), mkdir(work)]); - const env = { - PATH: `${dirname(process.execPath)}:/usr/bin:/bin:/usr/sbin:/sbin`, - HOME: home, - CODEX_HOME: codexHome, - XDG_CONFIG_HOME: join(home, ".config"), - XDG_CACHE_HOME: join(home, ".cache"), - XDG_DATA_HOME: join(home, ".local/share"), - TMPDIR: temp, - }; - await verifyNetworkBlocked(env); - - // Bundle the production readers into the temporary directory; no test doubles. - const metadata = join( - packageRoot, - "src/contexts/telemetry/infrastructure/otel/actions/metadata" - ); - const outDir = join(temp, "readers"); - await build({ - entry: { - project: join(metadata, "session-project.ts"), - title: join(metadata, "session-title.ts"), - }, - outDir, - format: ["esm"], - outExtension: () => ({ js: ".mjs" }), - platform: "node", - target: "node22", - config: false, - clean: false, - dts: false, - silent: true, - splitting: false, - noExternal: [/.*/], - }); - const { readSessionProject } = await import(pathToFileURL(join(outDir, "project.mjs"))); - const { readSessionTitle } = await import(pathToFileURL(join(outDir, "title.mjs"))); - - child = spawn(sandbox, ["-p", policy, codex, "app-server", "--listen", "stdio://"], { - env, - stdio: ["pipe", "pipe", "pipe"], - }); - const server = rpc(child); - try { - await server.request("initialize", { - clientInfo: { name: "aidd_native_smoke", version: "1" }, - capabilities: { experimentalApi: true }, - }); - server.notify("initialized"); - const { project } = await server.request("project/create", { - name: "Synthetic project", - roots: [{ path: work }], - idempotencyKey: "aidd-native-smoke", - }); - const { thread } = await server.request("thread/start", { - projectId: project.id, - cwd: work, - model: "gpt-5", - approvalPolicy: "never", - }); - await server.request("thread/name/set", { threadId: thread.id, name: "Synthetic title" }); - const request = (method, params) => server.request(method, params); - assert.deepEqual(await readSessionTitle(request, thread.id), { - available: true, - title: "Synthetic title", - }); - assert.deepEqual(await readSessionProject(request, thread.id), { - available: true, - values: { "aidd.project_id": project.id, "aidd.project_name": "Synthetic project" }, - }); - console.log("PASS: native Codex session title and project readers (network blocked)"); - } finally { - server.close(); - } - } finally { - try { - await stop(child); - } finally { - await rm(temp, { recursive: true, force: true }); - } - } -} - -main().catch((error) => { - console.error(`FAIL: ${error.message}`); - process.exitCode = 1; -}); diff --git a/cli/scripts/smoke-tools.sh b/cli/scripts/smoke-tools.sh index 5c05fb1a5..37d733215 100755 --- a/cli/scripts/smoke-tools.sh +++ b/cli/scripts/smoke-tools.sh @@ -30,7 +30,7 @@ ALL_COMMANDS=( "auth login" "auth logout" "auth status" "telemetry on" "telemetry off" "telemetry forget" "telemetry ingest" "telemetry task" "telemetry status" "telemetry report" "telemetry identity" - "telemetry otel init" "telemetry otel verify" "telemetry otel run" "telemetry otel sync" "telemetry otel collect" + "telemetry otel init" "telemetry otel verify" "telemetry otel sync" "telemetry otel collect" ) PASS=0; FAIL=0; SKIP=0 @@ -273,23 +273,19 @@ const port = new Promise((resolve, reject) => { NODE then mark_covered "telemetry otel collect"; ok "telemetry otel collect (authenticated local capture)" else bad "telemetry otel collect (authenticated local capture)" "$(cat "$TMPROOT/collector-smoke.log")"; fi -# The fake executable validates the environment handed to OpenCode 1 without using a real +# The fake executable validates the environment handed to OpenCode 2 without using a real # provider, changing the user's configuration, or relying on an installed OpenCode version. cat > "$TMPROOT/otel-bin/opencode" <<'SCRIPT' #!/usr/bin/env bash -if [[ "${1:-}" == "--version" ]]; then echo '1.18.35'; exit 0; fi -[[ "${OTEL_EXPORTER_OTLP_ENDPOINT:-}" == 'http://127.0.0.1:14318/opencode' ]] || exit 1 -[[ "${OTEL_EXPORTER_OTLP_HEADERS:-}" == 'Authorization=Bearer test-token' ]] || exit 1 -echo 'OpenCode received local OTLP settings' +if [[ "${1:-}" == "--version" ]]; then echo '2.2.5'; exit 0; fi +exit 1 SCRIPT chmod 700 "$TMPROOT/otel-bin/opencode" SMOKE_ORIGINAL_PATH="$PATH" export PATH="$TMPROOT/otel-bin:$PATH" unset OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS -run "telemetry otel init (OpenCode 1)" 0 "OpenCode 1" "$P_TEL" -- \ +run "telemetry otel init (OpenCode 2)" 0 "Restart the OpenCode background service" "$P_TEL" -- \ telemetry otel init --config=opencode --endpoint=dev -run "telemetry otel run (OpenCode 1)" 0 "OpenCode received local OTLP settings" "$P_TEL" -- \ - telemetry otel run --config=opencode -- --version-check unset AIDD_OTEL_TOKEN run "telemetry otel sync (no consented session)" 1 "No consented usage found" "$P_TEL" -- \ telemetry otel sync --config=codex --session=missing --tenant=smoke --endpoint=http://127.0.0.1:4318 diff --git a/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts b/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts index 3caabf2a1..672df5070 100644 --- a/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts +++ b/cli/src/contexts/telemetry/domain/observations/telemetry-observation.ts @@ -11,36 +11,33 @@ export type JsonValue = /** A source observation. Missing values stay missing; identical observations are not merged. */ export interface TelemetryObservation { readonly tool: AiToolId; - readonly signal: "log" | "span" | "span-event" | "metric" | "event"; + readonly signal: "log" | "span" | "span-event" | "metric"; readonly eventType?: string; readonly timestamp?: string; readonly observedTimestamp?: string; readonly startedAt?: string; readonly endedAt?: string; readonly toolVersion?: string; - readonly openCodeVersion?: 1 | 2; readonly sessionId?: string; readonly threadId?: string; + readonly turnId?: string; + readonly promptId?: string; + readonly clientRequestId?: string; readonly messageId?: string; - readonly partId?: string; readonly traceId?: string; readonly spanId?: string; readonly parentSpanId?: string; readonly requestId?: string; readonly provider?: string; readonly model?: string; - readonly role?: string; readonly prompt?: string; readonly response?: string; - /** Text whose user/assistant role cannot be established from this event alone. */ - readonly text?: string; readonly toolCallId?: string; readonly runtimeToolCallId?: string; readonly toolName?: string; readonly toolInput?: JsonValue; readonly toolOutput?: JsonValue; readonly success?: boolean; - readonly error?: JsonValue; readonly durationMs?: number; readonly cwd?: string; readonly gitBranch?: string; @@ -57,17 +54,11 @@ export interface TelemetryObservation { readonly tool?: number; }; /** Snapshots and aggregates must not be summed as model calls. */ - readonly usageScope?: - | "event" - | "span-snapshot" - | "step-snapshot" - | "message-snapshot" - | "session-cumulative" - | "metric-aggregate"; + readonly usageScope?: "event" | "span-snapshot" | "metric-aggregate"; readonly tokenInputIncludesCache?: boolean; readonly tokenOutputIncludesReasoning?: boolean; - /** OpenCode substitutes zero when an upstream token count was not supplied. */ - readonly zeroMayMeanMissing?: boolean; + /** Codex may report zero for cache counts missing from provider usage. */ + readonly cacheZeroMayMeanMissing?: boolean; /** As reported by the source, not a calculated or validated price. */ readonly reportedCostUsd?: number; readonly reportedCostUsdMicros?: number; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts index a473727f0..04d6818b1 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts @@ -13,7 +13,7 @@ import { async function applyEdit(edit: Edit): Promise<boolean> { const before = await readSettings(edit.path, edit.initial); const next = edit.update(before.content); - if (next === before.content && (before.private || edit.containsToken === false)) return false; + if (next === before.content && before.private) return false; await mkdir(dirname(edit.path), { recursive: true, mode: 0o700 }); const lockPath = `${edit.path}.aidd-otel.lock`; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts deleted file mode 100644 index 5d8adfee6..000000000 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.ts +++ /dev/null @@ -1,51 +0,0 @@ -import { asPlainObject } from "../../../../../../kernel/reading/plain-object.js"; -import { createSessionMetadataPublisher, type Metadata, type Request } from "./session-title.js"; - -function hasControlCharacter(text: string): boolean { - return [...text].some( - (character) => character.charCodeAt(0) < 32 || character.charCodeAt(0) === 127 - ); -} - -/** Native project membership only: roots and cwd are not evidence of membership. */ -export async function readSessionProject(request: Request, session: string): Promise<Metadata> { - try { - const readThread = async () => { - const result = await request("thread/read", { threadId: session, includeTurns: false }); - const thread = asPlainObject(asPlainObject(result)?.thread); - return thread?.id === session && thread.sessionId === session ? thread : null; - }; - const thread = await readThread(); - if (!thread) return { available: false }; - if (thread.projectId === null) return { available: true, values: {} }; - const id = thread.projectId; - if (typeof id !== "string" || !id.trim() || id.length > 1024 || hasControlCharacter(id)) - return { available: false }; - const result = await request("project/read", { projectId: id }); - const project = asPlainObject(asPlainObject(result)?.project); - if ( - project?.id !== id || - typeof project.name !== "string" || - !project.name.trim() || - project.name.length > 4096 || - hasControlCharacter(project.name) - ) - return { available: false }; - if ((await readThread())?.projectId !== id) return { available: false }; - return { - available: true, - values: { "aidd.project_id": id, "aidd.project_name": project.name }, - }; - } catch { - return { available: false }; - } -} - -export function createSessionProjectPublisher(home: string, onUnavailable?: () => void) { - return createSessionMetadataPublisher( - home, - "aidd.session_project", - readSessionProject, - onUnavailable - ); -} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts deleted file mode 100644 index 4b95182f2..000000000 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.ts +++ /dev/null @@ -1,49 +0,0 @@ -import { asPlainObject } from "../../../../../../kernel/reading/plain-object.js"; -import { createSessionMetadataPublisher, type Metadata, type Request } from "./session-title.js"; - -/** Keep identities on supported hosts. Never forward raw Git credentials or local paths. */ -function publicRepository(origin: string): string | null { - if (origin.length > 2048) return null; - const match = - /^(?:https:\/\/(github\.com|gitlab\.com)\/|git@(github\.com|gitlab\.com):)([A-Za-z0-9_.-]+(?:\/[A-Za-z0-9_.-]+)+)$/.exec( - origin - ); - if (!match || match[0] !== origin) return null; - const host = match[1] ?? match[2]; - const path = match[3].endsWith(".git") ? match[3].slice(0, -4) : match[3]; - const parts = path.split("/"); - if ( - (host === "github.com" && parts.length !== 2) || - parts.some((part) => !part || part === "." || part === "..") - ) - return null; - return `https://${host}/${path}`; -} - -/** The Git origin in the native thread is separate from its Codex project. */ -export async function readSessionRepository(request: Request, session: string): Promise<Metadata> { - try { - const result = await request("thread/read", { threadId: session, includeTurns: false }); - const thread = asPlainObject(asPlainObject(result)?.thread); - if (!thread || thread.id !== session || thread.sessionId !== session) - return { available: false }; - if (thread.gitInfo === null) return { available: true, values: {} }; - const git = asPlainObject(thread.gitInfo); - if (!git) return { available: false }; - if (git.originUrl === null) return { available: true, values: {} }; - if (typeof git.originUrl !== "string") return { available: false }; - const url = publicRepository(git.originUrl); - return { available: true, values: url ? { "aidd.repository_url": url } : {} }; - } catch { - return { available: false }; - } -} - -export function createSessionRepositoryPublisher(home: string, onUnavailable?: () => void) { - return createSessionMetadataPublisher( - home, - "aidd.session_repository", - readSessionRepository, - onUnavailable - ); -} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts deleted file mode 100644 index 274318a20..000000000 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.ts +++ /dev/null @@ -1,266 +0,0 @@ -import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process"; -import { createInterface } from "node:readline"; -import { asPlainObject } from "../../../../../../kernel/reading/plain-object.js"; -import { CODEX_CONFIG_ID } from "../../settings.js"; -import type { Identity, Transfer } from "../sync.js"; - -export type Request = (method: string, params: object) => Promise<unknown>; -type Title = { available: true; title: string | null } | { available: false }; -const MAX_TITLE_LENGTH = 4096; -const REQUEST_TIMEOUT_MS = 5000; -const REFRESH_MS = 60_000; - -export async function readSessionTitle(request: Request, session: string): Promise<Title> { - let result: unknown; - try { - result = await request("thread/read", { threadId: session, includeTurns: false }); - } catch { - return { available: false }; - } - const thread = asPlainObject(asPlainObject(result)?.thread); - if (!thread || thread.id !== session || thread.sessionId !== session) return { available: false }; - if (thread.name == null || (typeof thread.name === "string" && !thread.name.trim())) - return { available: true, title: null }; - if (typeof thread.name !== "string" || thread.name.length > MAX_TITLE_LENGTH) - return { available: false }; - return { available: true, title: thread.name }; -} - -export function sessionTitleTransfer( - session: string, - identity: Identity, - title: string | null, - at: Date -): Transfer | null { - if ( - identity.conflict || - (!identity.values.get("user.email") && !identity.values.get("user.account_id")) - ) - return null; - const values = new Map(identity.values); - values.set("conversation.id", session); - values.set("event.name", "aidd.session_title"); - values.set("event.timestamp", at.toISOString()); - if (title !== null) values.set("aidd.session_title", title); - return metadataTransfer(session, { ...identity, values }, at); -} - -function metadataTransfer(session: string, identity: Identity, at: Date): Transfer { - const values = new Map(identity.values); - values.set("conversation.id", session); - values.set("event.timestamp", at.toISOString()); - return { - signal: "logs", - at: at.toISOString(), - payload: { - resourceLogs: [ - { - scopeLogs: [ - { - logRecords: [ - { - timeUnixNano: (BigInt(at.getTime()) * 1_000_000n).toString(), - attributes: [...values].map(([key, value]) => ({ - key, - value: { stringValue: value }, - })), - }, - ], - }, - ], - }, - ], - }, - }; -} - -class AppServer<T> { - private child: ChildProcessWithoutNullStreams; - private nextId = 0; - private pending = new Map<number, { resolve: (result: unknown) => void; reject: () => void }>(); - private ready: Promise<unknown>; - private failed = false; - - constructor( - home: string, - private readValue: (request: Request, session: string) => Promise<T>, - experimental = false - ) { - this.child = spawn(CODEX_CONFIG_ID, ["app-server", "--listen", "stdio://"], { - env: { ...process.env, HOME: home }, - stdio: ["pipe", "pipe", "pipe"], - }); - this.child.stderr.resume(); - this.child.stdin.on("error", () => this.fail()); - createInterface({ input: this.child.stdout }).on("line", (line) => this.receive(line)); - this.child.on("error", () => this.fail()); - this.child.on("exit", () => this.fail()); - this.ready = this.request("initialize", { - clientInfo: { name: "aidd_telemetry", title: "AIDD telemetry", version: "1" }, - ...(experimental ? { capabilities: { experimentalApi: true } } : {}), - }).then(() => { - this.child.stdin.write(`${JSON.stringify({ method: "initialized", params: {} })}\n`); - }); - } - - private receive(line: string): void { - let message: Record<string, unknown> | null; - try { - message = asPlainObject(JSON.parse(line)); - } catch { - return; - } - if (!message || typeof message.id !== "number") return; - const pending = this.pending.get(message.id); - if (!pending) return; - this.pending.delete(message.id); - if (message.error) pending.reject(); - else pending.resolve(message.result); - } - - private fail(): void { - this.failed = true; - for (const pending of this.pending.values()) pending.reject(); - this.pending.clear(); - } - - private request(method: string, params: object): Promise<unknown> { - return new Promise((resolve, reject) => { - const id = ++this.nextId; - const timeout = setTimeout(() => { - this.pending.delete(id); - this.failed = true; - this.child.kill(); - reject(new Error("Codex app-server did not respond.")); - }, REQUEST_TIMEOUT_MS); - this.pending.set(id, { - resolve: (result) => { - clearTimeout(timeout); - resolve(result); - }, - reject: () => { - clearTimeout(timeout); - reject(new Error("Codex app-server is unavailable.")); - }, - }); - this.child.stdin.write(`${JSON.stringify({ id, method, params })}\n`); - }); - } - - async read(session: string): Promise<T> { - await this.ready; - return this.readValue((method, params) => this.request(method, params), session); - } - - close(): void { - this.fail(); - this.child.kill(); - } - - get unavailable(): boolean { - return this.failed; - } -} - -type Reader<T> = { read(session: string): Promise<T>; close(): void; unavailable?: boolean }; - -export type Metadata = { available: true; values: Record<string, string> } | { available: false }; - -function createPublisher<T extends { available: boolean }>( - home: string, - read: (request: Request, session: string) => Promise<T>, - transfer: (session: string, identity: Identity, result: T, at: Date) => Transfer | null, - experimental: boolean, - dependencies: { - now?: () => number; - createReader?: (home: string) => Reader<T>; - onUnavailable?: () => void; - } = {} -) { - let server: Reader<T> | undefined; - const checked = new Map<string, number>(); - const previous = new Map<string, string>(); - let warned = false; - let retryAfter = 0; - return { - async publish( - session: string, - identity: Identity, - send: (transfer: Transfer) => Promise<boolean> - ): Promise<boolean> { - if ( - identity.conflict || - (!identity.values.get("user.email") && !identity.values.get("user.account_id")) - ) - return false; - const now = (dependencies.now ?? Date.now)(); - const last = checked.get(session); - if ((last !== undefined && now - last < REFRESH_MS) || now < retryAfter) return false; - if (server?.unavailable) { - server.close(); - server = undefined; - } - server ??= (dependencies.createReader ?? ((path) => new AppServer(path, read, experimental)))( - home - ); - const result = await server.read(session).catch(() => null); - const observedAt = (dependencies.now ?? Date.now)(); - checked.set(session, observedAt); - if (!result?.available) { - if (server.unavailable) retryAfter = observedAt + REFRESH_MS; - if (!warned) dependencies.onUnavailable?.(); - warned = true; - return false; - } - const key = JSON.stringify(result); - if (previous.get(session) === key) return false; - const event = transfer(session, identity, result, new Date(observedAt)); - if (!event) return false; - if (!(await send(event))) return false; - previous.set(session, key); - return true; - }, - close(): void { - server?.close(); - }, - }; -} - -export function createSessionTitlePublisher( - home: string, - dependencies: { - now?: () => number; - createReader?: (home: string) => Reader<Title>; - onUnavailable?: () => void; - } = {} -) { - return createPublisher( - home, - readSessionTitle, - (session, identity, result, at) => - result.available ? sessionTitleTransfer(session, identity, result.title, at) : null, - false, - dependencies - ); -} - -export function createSessionMetadataPublisher( - home: string, - eventName: string, - read: (request: Request, session: string) => Promise<Metadata>, - onUnavailable?: () => void -) { - return createPublisher( - home, - read, - (session, identity, result, at) => { - if (!result.available) return null; - const values = new Map(identity.values); - values.set("event.name", eventName); - for (const [key, value] of Object.entries(result.values)) values.set(key, value); - return metadataTransfer(session, { ...identity, values }, at); - }, - true, - { onUnavailable } - ); -} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts deleted file mode 100644 index 7398798fe..000000000 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts +++ /dev/null @@ -1,49 +0,0 @@ -import { spawn } from "node:child_process"; -import { constants as osConstants } from "node:os"; -import opencodeTemplate from "../config/opencode.json" with { type: "json" }; -import { rejectConflict } from "../config-edits.js"; -import { readSettings } from "../files.js"; -import { - type Host, - interpolate, - OPENCODE_CONFIG_ID, - openCodeV1Edit, - openCodeVersion, - readToken, -} from "../settings.js"; - -export async function runOtelOpenCode(args: readonly string[], host: Host): Promise<number> { - if ((await openCodeVersion(host)) !== "1") { - throw new Error("OTLP launch is for OpenCode 1. Use the background service with OpenCode 2."); - } - const edit = await openCodeV1Edit(host); - const current = await readSettings(edit.path, edit.initial); - if (edit.update(current.content) !== current.content) { - throw new Error( - "Initialize OpenCode 1 OTLP with aidd telemetry otel init --config=opencode --endpoint=dev." - ); - } - const token = await readToken(host.home); - const additions = interpolate(opencodeTemplate.env, token); - for (const [key, value] of Object.entries(additions)) - rejectConflict(key, host.env[key] || undefined, value); - const child = spawn(OPENCODE_CONFIG_ID, args, { - env: { ...host.env, ...additions }, - stdio: "inherit", - }); - const interrupt = () => child.kill("SIGINT"); - const terminate = () => child.kill("SIGTERM"); - process.on("SIGINT", interrupt); - process.on("SIGTERM", terminate); - try { - return await new Promise((resolve, reject) => { - child.once("error", () => reject(new Error("Could not launch OpenCode."))); - child.once("exit", (code, signal) => - resolve(code ?? (signal ? 128 + osConstants.signals[signal] : 1)) - ); - }); - } finally { - process.off("SIGINT", interrupt); - process.off("SIGTERM", terminate); - } -} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts deleted file mode 100644 index 0da63f644..000000000 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.ts +++ /dev/null @@ -1,154 +0,0 @@ -import { constants } from "node:fs"; -import { lstat, open } from "node:fs/promises"; -import { join } from "node:path"; -import { - type SessionAdmission, - sameSessionAdmission, -} from "../../../domain/report/session-feature.js"; -import { CODEX_CONFIG_ID } from "../settings.js"; -import { createSessionProjectPublisher } from "./metadata/session-project.js"; -import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; -import { createSessionTitlePublisher } from "./metadata/session-title.js"; -import { - destination, - type Identity, - type SyncOptions, - selectTransfers, - send, - sessionIdsInCapture, -} from "./sync.js"; - -type Consent = "allow" | "deny" | "unknown"; -export type TitleOptions = Omit<SyncOptions, "session" | "follow" | "titlesOnly">; -const MAX_LINE = 32 * 1024 * 1024; - -function alreadyKnown(identity: Identity | undefined, observed: Identity): boolean { - if (identity?.conflict) return true; - if (observed.conflict) return false; - return [...observed.values].every(([key, value]) => identity?.values.get(key) === value); -} - -/** Read a fixed snapshot once; unlike follow, this never writes a capture checkpoint. */ -export async function syncAllSessionTitles( - options: TitleOptions, - snapshot: () => Promise<ReadonlyMap<string, SessionAdmission>>, - permits: ( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ) => Promise<Consent> -) { - if (options.config !== CODEX_CONFIG_ID || !options.includeSessionTitles) - throw new Error("Title-only sync requires Codex and explicit title opt-in."); - const base: SyncOptions = { ...options, session: "auto", follow: false }; - destination(base, "logs"); - const held = await snapshot(); - const path = join(options.home, ".local/share/aidd-otel-observation", "codex.jsonl"); - const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); - const identities = new Map<string, Identity>(); - const projects = options.includeSessionProjects - ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) - : undefined; - const repositories = options.includeSessionRepositories - ? createSessionRepositoryPublisher(options.home, options.onRepositoryUnavailable) - : undefined; - const titles = createSessionTitlePublisher(options.home, { - onUnavailable: options.onTitleUnavailable, - }); - try { - const info = await file.stat(); - if (!info.isFile() || (info.mode & 0o077) !== 0) - throw new Error("Capture must be a private regular file."); - const consume = async (line: string) => { - if (!line.trim()) return; - for (const session of sessionIdsInCapture(line, CODEX_CONFIG_ID)) { - const admission = held.get(session); - if (!admission) continue; - const sessionOptions: SyncOptions = { ...base, session }; - const scratch: Identity = { values: new Map(), conflict: false }; - const transfers = selectTransfers(line, sessionOptions, scratch); - for (const transfer of transfers) { - if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) continue; - const source = JSON.stringify({ signal: transfer.signal, payload: transfer.payload }); - const observed: Identity = { values: new Map(), conflict: false }; - selectTransfers(source, sessionOptions, observed); - const identity = identities.get(session); - if (alreadyKnown(identity, observed)) continue; - if ((await permits(admission, transfer.at, transfer.start)) !== "allow") continue; - const accepted = identity ?? { values: new Map(), conflict: false }; - selectTransfers(source, sessionOptions, accepted); - identities.set(session, accepted); - } - } - }; - let offset = 0; - let pending = Buffer.alloc(0); - const buffer = Buffer.alloc(65536); - while (offset < info.size) { - const length = Math.min(buffer.length, info.size - offset); - const { bytesRead } = await file.read(buffer, 0, length, offset); - if (!bytesRead) throw new Error("Capture changed while titles were being read."); - offset += bytesRead; - pending = Buffer.concat([pending, buffer.subarray(0, bytesRead)]); - for (let newline = pending.indexOf(10); newline >= 0; newline = pending.indexOf(10)) { - if (newline > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); - await consume(pending.subarray(0, newline).toString("utf8")); - pending = pending.subarray(newline + 1); - } - if (pending.length > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); - } - const current = await lstat(path); - if ( - !current.isFile() || - current.ino !== info.ino || - current.dev !== info.dev || - current.size < info.size - ) - throw new Error("Capture was replaced or truncated; restart title sync."); - let sent = 0; - let identified = 0; - let currentConsenting = 0; - for (const [session, identity] of identities) { - const admission = held.get(session); - if ( - !admission || - identity.conflict || - (!identity.values.get("user.email") && !identity.values.get("user.account_id")) - ) - continue; - identified++; - if ((await permits(admission, new Date().toISOString())) !== "allow") continue; - currentConsenting++; - let stillConsenting = true; - for (const publisher of [titles, projects, repositories]) - if ( - publisher && - (await publisher.publish(session, identity, async (event) => { - if (!event.at || (await permits(admission, event.at)) !== "allow") { - if (stillConsenting) currentConsenting--; - stillConsenting = false; - return false; - } - await send({ ...base, session }, event, async () => { - const current = (await snapshot()).get(session); - return ( - !!current && - sameSessionAdmission(admission, current) && - !!event.at && - (await permits(current, event.at, undefined, new Date().toISOString())) === "allow" - ); - }); - return true; - })) - ) - sent++; - } - return { sent, identified, currentConsenting, partialLine: pending.length > 0 }; - } finally { - titles.close(); - projects?.close(); - repositories?.close(); - await file.close(); - } -} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts index f584e906e..c36951659 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts @@ -6,37 +6,23 @@ import { setTimeout as delay } from "node:timers/promises"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import { type SessionAdmission, - type SessionFeature, sameSessionAdmission, - sameSessionFeature, } from "../../../domain/report/session-feature.js"; import { LedgerLock } from "../../ledger-lock.js"; -import { CODEX_CONFIG_ID } from "../settings.js"; -import { createSessionProjectPublisher } from "./metadata/session-project.js"; -import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; -import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { admissionFingerprint, - type BindingDetails, - bindingTransfer, ConsentNotConfirmedError, destination, - evidenceTransfers, - type Identity, - type IdentityEvidence, OtelObservationTooLargeError, - observeIdentity, packTransfers, - readBindingDetails, type SyncOptions, - saveBindingState, selectTransfers, send, sessionIdsInCapture, type Transfer, } from "./sync.js"; -export type AllOptions = Omit<SyncOptions, "session" | "follow" | "stateDir" | "titlesOnly"> & { +export type AllOptions = Omit<SyncOptions, "session" | "follow" | "stateDir"> & { readonly stateDir: string; readonly refreshMs?: number; }; @@ -51,18 +37,6 @@ interface Pending { readonly cwdKey?: string; readonly cloneKey?: string; } -interface SessionState { - readonly identity: Identity; - readonly identityEvidence: IdentityEvidence[]; - identityAdmissionKey: string | null; - identityAdmission: SessionAdmission | null; - identityCloneKey: string | null; - previousLinkUnverified: boolean; - savedIdentity: BindingDetails | null; - mayHaveLinked: boolean; - binding: string; - needsReplay: boolean; -} interface Checkpoint { readonly device: number; readonly inode: number; @@ -71,7 +45,6 @@ interface Checkpoint { const MAX_LINE = 32 * 1024 * 1024; const INTERVAL_MS = 5000; const RETRY_MS = 30_000; -const BINDING_REFRESH_MS = 60_000; function key(options: AllOptions, session: string) { return createHash("sha256") @@ -292,18 +265,6 @@ export async function syncAllOtel( staleAfterMs: Number.MAX_SAFE_INTEGER, }); const release = await lock.acquire(); - const projects = - options.config === CODEX_CONFIG_ID && options.includeSessionProjects - ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) - : undefined; - const repositories = - options.config === CODEX_CONFIG_ID && options.includeSessionRepositories - ? createSessionRepositoryPublisher(options.home, options.onRepositoryUnavailable) - : undefined; - const titles = - options.config === CODEX_CONFIG_ID && options.includeSessionTitles - ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) - : undefined; let file: Awaited<ReturnType<typeof open>> | undefined; try { file = await open(capture, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); @@ -327,194 +288,25 @@ export async function syncAllOtel( let lastSnapshotAt = Date.now(); let dirty = false; let pendingBytes = Buffer.alloc(0); - const sessions = new Map<string, SessionState>(); let sent = 0; let denied = 0; const oversized = new Set<string>(); const pendingDirectory = join(directory, `${key(options, "auto")}.pending`); await privateDirectory(pendingDirectory); const pendingPath = (id: string) => join(pendingDirectory, `${key(options, id)}.jsonl`); - const sessionState = async (id: string): Promise<SessionState> => { - const existing = sessions.get(id); - if (existing) return existing; - const saved = await readBindingDetails(join(directory, `${key(options, id)}.json`)); - if (saved?.details && saved.details.session !== id) - throw new Error("Sync binding state does not match its session."); - const state = { - identity: { - values: new Map<string, string>(), - conflict: saved?.details?.conflict === true, - }, - identityEvidence: [] as IdentityEvidence[], - identityAdmissionKey: null, - identityAdmission: null, - identityCloneKey: null, - previousLinkUnverified: saved?.mayHaveLinked === true, - savedIdentity: saved?.details ?? null, - mayHaveLinked: saved?.mayHaveLinked === true, - binding: saved?.details?.binding ?? "", - needsReplay: Boolean( - saved?.mayHaveLinked && saved.details && saved.details.binding !== "invalidated" - ), - }; - sessions.set(id, state); - return state; - }; - const loadBindings = async () => { - const files = new Set(await readdir(directory)); - for (const [id] of held) { - if (sessions.has(id)) continue; - const name = `${key(options, id)}.json`; - if (!files.has(name)) continue; - const saved = await readBindingDetails(join(directory, name)); - if (!saved?.details) continue; // Older states need a bounded session replay to recover identity. - if (saved.details.session !== id) - throw new Error("Sync binding state does not match its session."); - sessions.set(id, { - identity: { - values: new Map(), - conflict: saved.details.conflict === true, - }, - identityEvidence: [], - identityAdmissionKey: null, - identityAdmission: null, - identityCloneKey: null, - previousLinkUnverified: saved.mayHaveLinked, - savedIdentity: saved.details, - mayHaveLinked: saved.mayHaveLinked, - binding: saved.details.binding, - needsReplay: saved.mayHaveLinked && saved.details.binding !== "invalidated", - }); - } - }; const stillPermitted = async ( id: string, admission: SessionAdmission, - items: readonly Transfer[], - expectedFeature?: SessionFeature + items: readonly Transfer[] ) => { const current = (await snapshot()).get(id); if (!sameSessionAdmission(admission, current) || !current) return false; - if (expectedFeature && !sameSessionFeature(expectedFeature, current.feature)) return false; const consentAt = new Date().toISOString(); for (const item of items) if (!item.at || (await permits(current, item.at, item.start, consentAt)) !== "allow") return false; return true; }; - const restoreIdentity = async ( - id: string, - admission: SessionAdmission, - state: SessionState - ) => { - const saved = state.savedIdentity; - const proof = saved?.identityProof; - if ( - saved && - proof?.version === 2 && - proof.admissionKey !== admissionKey(admission) && - admission.cwd && - admission.repositoryId === null && - admission.feature.state === "ambiguous" && - proof.cwdKey === cwdKey(admission.cwd) && - proof.cloneKey && - cloneProof && - state.mayHaveLinked && - saved.binding !== "invalidated" - ) { - const matches = async () => { - if ( - !(await stillPermitted( - id, - admission, - evidenceTransfers(proof.sources), - admission.feature - )) - ) - return false; - const now = new Date().toISOString(); - for (const source of proof.sources) - if ((await cloneProof(admission, source.at, source.start, now)) !== proof.cloneKey) - return false; - return true; - }; - if (!(await matches())) return; - const event = bindingTransfer( - asOptions(options, id), - { values: new Map(saved.identity), conflict: saved.conflict === true }, - { state: "ambiguous" } - ); - if (!event || !(await deliver(id, event, matches))) return; - sent++; - await saveBindingState( - join(directory, `${key(options, id)}.json`), - directory, - { session: id, identity: [], binding: "invalidated" }, - false - ); - state.binding = "invalidated"; - state.mayHaveLinked = false; - state.previousLinkUnverified = false; - state.savedIdentity = null; - return; - } - if ( - !saved || - !proof || - proof.admissionKey !== admissionKey(admission) || - !(await stillPermitted(id, admission, evidenceTransfers(proof.sources))) - ) - return; - if (proof.version === 2) { - if ( - !admission.cwd || - proof.cwdKey !== cwdKey(admission.cwd) || - !proof.cloneKey || - !cloneProof - ) - return; - const now = new Date().toISOString(); - for (const source of proof.sources) - if ((await cloneProof(admission, source.at, source.start, now)) !== proof.cloneKey) - return; - } - state.identity.values.clear(); - for (const [name, value] of saved.identity) state.identity.values.set(name, value); - state.identity.conflict ||= saved.conflict === true; - state.identityEvidence.splice(0, state.identityEvidence.length, ...proof.sources); - state.identityAdmissionKey = proof.admissionKey; - state.identityAdmission = admission; - state.identityCloneKey = proof.version === 2 ? (proof.cloneKey ?? null) : null; - state.previousLinkUnverified = false; - state.savedIdentity = null; - }; - const stateProof = async (state: SessionState): Promise<BindingDetails["identityProof"]> => { - if (!state.identityAdmissionKey || !state.identityEvidence.length) return undefined; - const admission = state.identityAdmission; - if (cloneProof && admission?.cwd) { - let clone: string | null = null; - const now = new Date().toISOString(); - for (const source of state.identityEvidence) { - const found = await cloneProof(admission, source.at, source.start, now); - if (!found || (clone && found !== clone)) return undefined; - clone = found; - } - if (clone && state.identityCloneKey === clone) - return { - version: 2, - admissionKey: state.identityAdmissionKey, - cwdKey: cwdKey(admission.cwd), - cloneKey: clone, - sources: state.identityEvidence, - }; - return undefined; - } - return { - version: 1, - admissionKey: state.identityAdmissionKey, - sources: state.identityEvidence, - }; - }; const deliver = async (id: string, transfer: Transfer, guard: () => Promise<boolean>) => { try { await send(asOptions(options, id), transfer, guard); @@ -539,79 +331,10 @@ export async function syncAllOtel( } return clone; }; - const reconcileIdentity = async ( - id: string, - admission: SessionAdmission, - state: SessionState - ) => { - if (!state.identityEvidence.length) return true; - const current = (await snapshot()).get(id); - if (!current || !sameSessionAdmission(admission, current)) return false; - if (state.identityAdmissionKey === admissionKey(admission)) { - const consentAt = new Date().toISOString(); - let allowed = true; - for (const item of evidenceTransfers(state.identityEvidence)) - if (!item.at || (await permits(current, item.at, item.start, consentAt)) !== "allow") { - allowed = false; - break; - } - if (allowed) return true; - } - - const previous = state.identityAdmission; - if ( - state.mayHaveLinked && - state.binding !== "invalidated" && - previous?.cwd && - previous.repositoryId && - state.identityAdmissionKey === admissionKey(previous) && - admission.cwd === previous.cwd && - admission.repositoryId === null && - admission.feature.state === "ambiguous" && - (!cloneProof || - (state.identityCloneKey && - (await proofForTransfers(admission, evidenceTransfers(state.identityEvidence))) === - state.identityCloneKey)) - ) { - const event = bindingTransfer(asOptions(options, id), state.identity, { - state: "ambiguous", - }); - if (event) { - const evidence = [event, ...evidenceTransfers(state.identityEvidence)]; - if ( - !(await deliver(id, event, () => - stillPermitted(id, admission, evidence, admission.feature) - )) - ) - return false; - sent++; - state.binding = "invalidated"; - await saveBindingState( - join(directory, `${key(options, id)}.json`), - directory, - { - session: id, - identity: [], - binding: state.binding, - }, - false - ); - state.mayHaveLinked = false; - } - } - if (state.mayHaveLinked) state.previousLinkUnverified = true; - state.identity.values.clear(); - state.identityEvidence.length = 0; - state.identityAdmissionKey = null; - state.identityAdmission = null; - state.identityCloneKey = null; - return true; - }; const process = async ( id: string, items: readonly Transfer[], admission: SessionAdmission, - state: SessionState, hold = false, expectedClone?: string, captured?: { clone: string | null } @@ -620,8 +343,6 @@ export async function syncAllOtel( let rejected = 0; const allowed: Transfer[] = []; const verdicts = new Map<string, Consent>(); - await restoreIdentity(id, admission, state); - if (!(await reconcileIdentity(id, admission, state))) return "unknown"; for (const transfer of items) { if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { rejected++; @@ -663,36 +384,9 @@ export async function syncAllOtel( return hold ? "hold" : "unknown"; } const clone = allowed.length ? await proofForTransfers(admission, allowed) : null; - if ( - allowed.length && - cloneProof && - (!clone || - (expectedClone && clone !== expectedClone) || - (state.identityCloneKey && state.identityCloneKey !== clone)) - ) + if (allowed.length && cloneProof && (!clone || (expectedClone && clone !== expectedClone))) return hold ? "hold" : "unknown"; if (captured) captured.clone = clone; - for (const transfer of allowed) { - observeIdentity(transfer, asOptions(options, id), state.identity, state.identityEvidence); - if (state.identityEvidence.length) { - state.identityAdmissionKey = admissionKey(admission); - state.identityAdmission = admission; - state.identityCloneKey ??= clone; - } - } - if (state.identity.conflict && !state.previousLinkUnverified) - await saveBindingState( - join(directory, `${key(options, id)}.json`), - directory, - { - session: id, - identity: [...state.identity.values], - binding: state.binding, - conflict: true, - identityProof: await stateProof(state), - }, - state.mayHaveLinked - ); for (const batch of batches) { if ( !(await deliver( @@ -709,130 +403,18 @@ export async function syncAllOtel( denied += rejected; return "resolved"; }; - const bindings = async () => { - for (const [id, state] of sessions) { - const admission = held.get(id); - if (admission) await restoreIdentity(id, admission, state); - if (admission && !(await reconcileIdentity(id, admission, state))) continue; - if (!admission || state.previousLinkUnverified) continue; - const feature: SessionFeature = admission.feature; - const next = - feature.state === "confirmed" ? `${feature.branch}\0${feature.feature}` : feature.state; - const invalid = - state.mayHaveLinked && - (state.identity.conflict || - feature.state !== "confirmed" || - (state.binding !== "" && state.binding !== next)); - if (invalid && state.binding === "invalidated") continue; - if ( - !invalid && - (feature.state !== "confirmed" || (state.binding === next && !state.needsReplay)) - ) - continue; - if ((await permits(admission, new Date().toISOString())) !== "allow") continue; - if (invalid && state.binding !== "invalidated") { - const event = bindingTransfer(asOptions(options, id), state.identity, { - state: "ambiguous", - }); - if (event) { - if ( - !(await deliver(id, event, () => - stillPermitted( - id, - admission, - [event, ...evidenceTransfers(state.identityEvidence)], - feature - ) - )) - ) - continue; - sent++; - state.binding = "invalidated"; - await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { - session: id, - identity: [...state.identity.values], - binding: state.binding, - identityProof: await stateProof(state), - }); - } - } else if ( - !invalid && - feature.state === "confirmed" && - (state.binding !== next || state.needsReplay) - ) { - const event = bindingTransfer(asOptions(options, id), state.identity, feature); - if (event) { - const proof = await stateProof(state); - if (cloneProof && proof?.version !== 2) continue; - await saveBindingState(join(directory, `${key(options, id)}.json`), directory, { - session: id, - identity: [...state.identity.values], - binding: next, - identityProof: proof, - }); - state.mayHaveLinked = true; - if ( - !(await deliver( - id, - event, - async () => - (await stillPermitted( - id, - admission, - [event, ...evidenceTransfers(state.identityEvidence)], - feature - )) && - (!proof?.cloneKey || - (await proofForTransfers( - admission, - evidenceTransfers(state.identityEvidence) - )) === proof.cloneKey) - )) - ) - continue; - sent++; - state.binding = next; - state.needsReplay = false; - } - } - } - }; - const publishTitles = async () => { - if (!titles && !projects && !repositories) return; - for (const [id, state] of sessions) { - const admission = held.get(id); - if (admission) await restoreIdentity(id, admission, state); - if (!admission || state.identity.conflict) continue; - if ((await permits(admission, new Date().toISOString())) !== "allow") continue; - for (const publisher of [titles, projects, repositories]) - if ( - publisher && - (await publisher.publish(id, state.identity, async (event) => { - if (!event.at || (await permits(admission, event.at)) !== "allow") return false; - return deliver(id, event, () => - stillPermitted(id, admission, [event, ...evidenceTransfers(state.identityEvidence)]) - ); - })) - ) - sent++; - } - }; const flushPending = async () => { const files = new Set(await readdir(pendingDirectory)); for (const [id, admission] of held) { const path = pendingPath(id); if (!files.has(`${key(options, id)}.jsonl`)) continue; - const state = await sessionState(id); const waiting: Pending[] = []; for (const item of await pendingRanges(path)) { const raw = Buffer.alloc(item.length); const read = await file?.read(raw, 0, item.length, item.offset); if (!read || read.bytesRead !== item.length || raw.at(-1) !== 10) throw new Error("Pending capture range is unavailable; recover the capture."); - const transfers = selectTransfers(raw.toString("utf8"), asOptions(options, id), { - values: new Map(), - conflict: false, - }); + const transfers = selectTransfers(raw.toString("utf8"), asOptions(options, id)); const changedAdmission = item.hold && item.admissionKey !== admissionKey(admission); if ( changedAdmission && @@ -851,7 +433,6 @@ export async function syncAllOtel( id, transfers, admission, - state, item.hold, item.cloneKey, captured @@ -872,10 +453,7 @@ export async function syncAllOtel( await replacePending(path, waiting); } }; - await loadBindings(); await flushPending(); - await bindings(); - await publishTitles(); const buffer = Buffer.alloc(65536); for (;;) { const { bytesRead } = await file.read(buffer, 0, buffer.length, offset + pendingBytes.length); @@ -891,18 +469,14 @@ export async function syncAllOtel( const position = offset; if (raw.trim()) for (const id of sessionIdsInCapture(raw, options.config)) { - const state = await sessionState(id); - const items = selectTransfers(raw, asOptions(options, id), { - values: new Map(), - conflict: false, - }); + const items = selectTransfers(raw, asOptions(options, id)); const admission = held.get(id); const captured = { clone: null as string | null }; const status = items.length === 0 ? "resolved" : admission - ? await process(id, items, admission, state, false, undefined, captured) + ? await process(id, items, admission, false, undefined, captured) : "unknown"; if (status !== "resolved") await appendPending(pendingPath(id), { @@ -936,20 +510,12 @@ export async function syncAllOtel( } const elapsed = Date.now() - lastSnapshotAt; const retryAfter = options.refreshMs ?? RETRY_MS; - const bindingAfter = options.refreshMs ?? BINDING_REFRESH_MS; const pendingFiles = (await readdir(pendingDirectory)).length; - if ( - dirty || - (pendingFiles > 0 && elapsed >= retryAfter) || - (sessions.size > 0 && elapsed >= bindingAfter) - ) { + if (dirty || (pendingFiles > 0 && elapsed >= retryAfter)) { held = await snapshot(); - await loadBindings(); lastSnapshotAt = Date.now(); dirty = false; await flushPending(); - await bindings(); - await publishTitles(); await save(offset); } if (options.signal?.aborted) @@ -963,9 +529,6 @@ export async function syncAllOtel( } } } finally { - titles?.close(); - projects?.close(); - repositories?.close(); await file?.close(); await release(); } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index 23292026a..f6b34fe2f 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -1,20 +1,14 @@ -import { createHash, randomUUID } from "node:crypto"; +import { createHash } from "node:crypto"; import { constants } from "node:fs"; -import { lstat, mkdir, open, rename, rm, writeFile } from "node:fs/promises"; +import { lstat, open } from "node:fs/promises"; import { join } from "node:path"; import { setTimeout as delay } from "node:timers/promises"; import { isDeepStrictEqual } from "node:util"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; -import { type SessionFeature, sameSessionFeature } from "../../../domain/report/session-feature.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; -import { createSessionProjectPublisher } from "./metadata/session-project.js"; -import { createSessionRepositoryPublisher } from "./metadata/session-repository.js"; -import { createSessionTitlePublisher } from "./metadata/session-title.js"; import { readAttributeValue } from "./read-attribute-value.js"; import { seriesFingerprint } from "./series-fingerprint.js"; -type FeatureState = SessionFeature; - export interface SyncOptions { readonly config: SyncConfig; readonly session: string; @@ -23,17 +17,8 @@ export interface SyncOptions { readonly token: string; readonly home: string; readonly follow: boolean; - readonly stateDir?: string; readonly signal?: AbortSignal; readonly intervalMs?: number; - readonly includeSessionTitles?: boolean; - readonly includeSessionProjects?: boolean; - readonly includeSessionRepositories?: boolean; - readonly onProjectUnavailable?: () => void; - readonly onRepositoryUnavailable?: () => void; - readonly onTitleUnavailable?: () => void; - readonly titlesOnly?: boolean; - readonly admission?: { readonly cwd: string; readonly repositoryId: string | null }; } type RecordObject = Record<string, unknown>; @@ -122,17 +107,6 @@ const SAFE_ATTRIBUTES = new Set([ "reasoning_token_count", ]); -export interface Identity { - readonly values: Map<string, string>; - conflict: boolean; -} - -export interface IdentityEvidence { - readonly keys: string[]; - readonly at: string; - readonly start?: string; -} - export function admissionFingerprint(admission: { cwd: string | null; repositoryId: string | null; @@ -142,15 +116,6 @@ export function admissionFingerprint(admission: { .digest("hex"); } -export function evidenceTransfers(evidence: readonly IdentityEvidence[]): Transfer[] { - return evidence.map(({ at, start }) => ({ - signal: start ? "metrics" : "logs", - payload: {}, - at, - start, - })); -} - function attributes(value: unknown): Map<string, string> { if (value !== undefined && !Array.isArray(value)) throw new Error("Invalid OTLP attributes."); const out = new Map<string, string>(); @@ -290,16 +255,6 @@ function isExactNanoseconds(value: unknown): value is string | number { return isExactInteger(value, 0n, 18446744073709551615n); } -function captureIdentity(target: Identity, values: Map<string, string>) { - for (const key of IDENTITY_ATTRIBUTES) { - const value = values.get(key); - if (!value) continue; - const existing = target.values.get(key); - if (existing !== undefined && existing !== value) target.conflict = true; - else target.values.set(key, value); - } -} - function timeOf(entry: RecordObject) { for (const key of ["timeUnixNano", "observedTimeUnixNano"]) { if (entry[key] !== undefined) validateIntegerField(key, entry[key]); @@ -426,11 +381,7 @@ function eventTimestamp(value: string | undefined): string | undefined { } /** Keep only supported facts for the requested session. Original capture files are not changed. */ -export function selectTransfers( - line: string, - options: SyncOptions, - identity: Identity -): Transfer[] { +export function selectTransfers(line: string, options: SyncOptions): Transfer[] { let parsed: unknown; try { parsed = JSON.parse(line); @@ -451,7 +402,6 @@ export function selectTransfers( const record = object(rawRecord); const a = merge(resourceAttrs, attributes(record.attributes)); if (a.get(sessionKey) !== options.session) continue; - captureIdentity(identity, a); const event = a.get("event.name") ?? eventNameFromBody(record.body); const wanted = options.config === CLAUDE_CONFIG_ID @@ -520,7 +470,6 @@ export function selectTransfers( const pointAttrs = attributes(point.attributes); const a = merge(resourceAttrs, pointAttrs); if (a.get(sessionKey) !== options.session) continue; - captureIdentity(identity, a); a.set( "aidd.series_fingerprint", seriesFingerprint( @@ -597,35 +546,6 @@ export function selectTransfers( return out; } -/** Only a consent-admitted transfer can contribute identity to later metadata. */ -export function admitTransferIdentity( - transfer: Transfer, - options: SyncOptions, - identity: Identity -): void { - selectTransfers( - JSON.stringify({ signal: transfer.signal, payload: transfer.payload }), - options, - identity - ); -} - -export function observeIdentity( - transfer: Transfer, - options: SyncOptions, - identity: Identity, - evidence: IdentityEvidence[] -): void { - const previous = new Map(identity.values); - const conflicted = identity.conflict; - admitTransferIdentity(transfer, options, identity); - const keys = [...identity.values] - .filter(([key, value]) => previous.get(key) !== value) - .map(([key]) => key); - if ((keys.length || conflicted !== identity.conflict) && transfer.at) - evidence.push({ keys, at: transfer.at, ...(transfer.start ? { start: transfer.start } : {}) }); -} - export function destination(options: SyncOptions, signal: "logs" | "metrics"): URL { let endpoint: URL; try { @@ -742,224 +662,18 @@ export async function send( } } -export interface BindingDetails { - readonly session: string; - readonly identity: Array<[string, string]>; - readonly binding: string; - readonly conflict?: boolean; - readonly identityProof?: { - readonly version: 1 | 2; - readonly admissionKey: string; - readonly cwdKey?: string; - readonly cloneKey?: string; - readonly sources: IdentityEvidence[]; - }; -} - -function readIdentityProof( - value: unknown, - identity: Array<[string, string]> -): BindingDetails["identityProof"] { - const proof = object(value); - if ( - (proof.version !== 1 && proof.version !== 2) || - typeof proof.admissionKey !== "string" || - !/^[a-f0-9]{64}$/.test(proof.admissionKey) || - (proof.version === 2 && - (typeof proof.cwdKey !== "string" || - !/^[a-f0-9]{64}$/.test(proof.cwdKey) || - typeof proof.cloneKey !== "string" || - !/^[a-f0-9]{64}$/.test(proof.cloneKey))) || - !Array.isArray(proof.sources) || - proof.sources.length === 0 || - proof.sources.length > 4 - ) - return undefined; - const keys = new Set(identity.map(([key]) => key)); - const seen = new Set<string>(); - const sources: IdentityEvidence[] = []; - for (const raw of proof.sources) { - const source = object(raw); - if ( - !Array.isArray(source.keys) || - new Set(source.keys).size !== source.keys.length || - !source.keys.every( - (key: unknown) => typeof key === "string" && keys.has(key) && !seen.has(key) - ) || - typeof source.at !== "string" || - !Number.isFinite(Date.parse(source.at)) || - (source.start !== undefined && - (typeof source.start !== "string" || - !Number.isFinite(Date.parse(source.start)) || - Date.parse(source.start) > Date.parse(source.at))) - ) - return undefined; - for (const key of source.keys) seen.add(key); - sources.push({ - keys: source.keys as string[], - at: source.at, - ...(typeof source.start === "string" ? { start: source.start } : {}), - }); - } - if (seen.size !== keys.size || keys.size === 0) return undefined; - return { - version: proof.version as 1 | 2, - admissionKey: proof.admissionKey, - ...(proof.version === 2 - ? { cwdKey: proof.cwdKey as string, cloneKey: proof.cloneKey as string } - : {}), - sources, - }; -} - -export async function readBindingDetails( - path: string -): Promise<{ mayHaveLinked: boolean; details: BindingDetails | null } | null> { - const file = await open( - path, - constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK - ).catch((error) => { - if (error instanceof Error && "code" in error && error.code === "ENOENT") return null; - throw error; - }); - if (!file) return null; - try { - const info = await file.stat(); - if (!info.isFile() || (info.mode & 0o077) !== 0) - throw new Error("Sync state must be a private regular file."); - const parsed: unknown = JSON.parse(await file.readFile("utf8")); - const data = object(parsed); - if (typeof data.mayHaveLinked !== "boolean") throw new Error("Sync binding state is invalid."); - if (data.session === undefined && data.identity === undefined && data.binding === undefined) - return { mayHaveLinked: data.mayHaveLinked, details: null }; - if ( - typeof data.session !== "string" || - typeof data.binding !== "string" || - !Array.isArray(data.identity) || - !data.identity.every( - (item: unknown) => - Array.isArray(item) && - item.length === 2 && - ["user.email", "user.account_id", "organization.id"].includes(item[0]) && - typeof item[1] === "string" - ) - ) - throw new Error("Sync binding state is invalid."); - if ( - new Set(data.identity.map((item: [string, string]) => item[0])).size !== data.identity.length - ) - throw new Error("Sync binding state is invalid."); - if (data.conflict !== undefined && typeof data.conflict !== "boolean") - throw new Error("Sync binding state is invalid."); - const identity = data.identity as Array<[string, string]>; - const identityProof = readIdentityProof(data.identityProof, identity); - return { - mayHaveLinked: data.mayHaveLinked, - details: { - session: data.session, - identity: identityProof ? identity : [], - identityProof, - binding: data.binding, - conflict: data.conflict === true, - }, - }; - } finally { - await file.close(); - } -} - -export async function saveBindingState( - path: string, - directory: string, - details?: BindingDetails, - mayHaveLinked = true -): Promise<void> { - await mkdir(directory, { recursive: true, mode: 0o700 }); - const info = await lstat(directory); - if (!info.isDirectory() || (info.mode & 0o077) !== 0) - throw new Error("Sync state directory must be private."); - const temporary = `${path}.${randomUUID()}.tmp`; - try { - await writeFile( - temporary, - `${JSON.stringify({ mayHaveLinked, ...details, identity: details?.identityProof ? details.identity : [] })}\n`, - { - flag: "wx", - mode: 0o600, - } - ); - await rename(temporary, path); - } finally { - await rm(temporary, { force: true }); - } -} - -export function bindingTransfer( - options: SyncOptions, - identity: Identity, - state: FeatureState -): Transfer | null { - if ( - (identity.conflict && state.state === "confirmed") || - (!identity.values.get("user.email") && !identity.values.get("user.account_id")) - ) - return null; - if (options.config === CLAUDE_CONFIG_ID && !identity.values.get("user.email")) return null; - const a = new Map(identity.values); - a.set(options.config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id", options.session); - a.set( - "event.name", - state.state === "confirmed" ? "aidd.session_feature" : "aidd.session_feature_invalidated" - ); - const at = new Date().toISOString(); - a.set("event.timestamp", at); - if (state.state === "confirmed") { - a.set("aidd.feature", state.feature); - a.set("aidd.branch", state.branch); - } - const bindingAttrs = [...a].map(([key, value]) => ({ key, value: { stringValue: value } })); - return { - signal: "logs", - at, - payload: { - resourceLogs: [ - { - scopeLogs: [ - { - logRecords: [ - { - timeUnixNano: (BigInt(Date.now()) * 1_000_000n).toString(), - attributes: bindingAttrs, - }, - ], - }, - ], - }, - ], - }, - }; -} - /** Replays one selected session and optionally follows newly appended bytes. A failed delivery * never advances the in-memory cursor. Restart safely replays into the idempotent receiver. */ export async function syncOtel( options: SyncOptions, - resolve: () => Promise<FeatureState>, + revalidate: () => Promise<void>, permits: (transfer: Transfer) => Promise<boolean> ) { - if ( - options.titlesOnly && - (options.config !== CODEX_CONFIG_ID || options.follow || !options.includeSessionTitles) - ) - throw new Error("Title-only sync requires one Codex session and title opt-in."); destination(options, "logs"); const path = join(options.home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); - const identity: Identity = { values: new Map(), conflict: false }; - const identityEvidence: IdentityEvidence[] = []; - const stillPermitted = async (items: readonly Transfer[], expectedFeature?: SessionFeature) => { - const currentFeature = await resolve(); - if (expectedFeature && !sameSessionFeature(expectedFeature, currentFeature)) return false; + const stillPermitted = async (items: readonly Transfer[]) => { + await revalidate(); const consentAt = new Date().toISOString(); for (const item of items) if (!(await permits({ ...item, consentAt }))) return false; return true; @@ -968,55 +682,10 @@ export async function syncOtel( let skipped = 0; let offset = 0; let pending = Buffer.alloc(0); - let binding = ""; - let needsReplay = false; - const stateName = createHash("sha256") - .update(`${options.endpoint}\0${options.tenant}\0${options.config}\0${options.session}`) - .digest("hex"); - const statePath = options.stateDir - ? join(options.stateDir, "otel-sync", `${stateName}.json`) - : null; - let mayHaveLinked = false; - let previousLinkUnverified = false; - const projects = - options.config === CODEX_CONFIG_ID && options.includeSessionProjects - ? createSessionProjectPublisher(options.home, options.onProjectUnavailable) - : undefined; - const repositories = - options.config === CODEX_CONFIG_ID && options.includeSessionRepositories - ? createSessionRepositoryPublisher(options.home, options.onRepositoryUnavailable) - : undefined; - const titles = - options.config === CODEX_CONFIG_ID && options.includeSessionTitles - ? createSessionTitlePublisher(options.home, { onUnavailable: options.onTitleUnavailable }) - : undefined; try { const info = await handle.stat(); if (!info.isFile() || (info.mode & 0o077) !== 0) throw new Error("Capture must be a private regular file."); - if (statePath !== null) { - const saved = await readBindingDetails(statePath); - mayHaveLinked = saved?.mayHaveLinked === true; - previousLinkUnverified = mayHaveLinked; - if (saved?.details) { - if (saved.details.session !== options.session) - throw new Error("Sync binding state does not match its session."); - const proof = saved.details.identityProof; - if ( - proof && - options.admission && - proof.admissionKey === admissionFingerprint(options.admission) && - (await stillPermitted(evidenceTransfers(proof.sources))) - ) { - for (const [key, value] of saved.details.identity) identity.values.set(key, value); - identityEvidence.push(...proof.sources); - previousLinkUnverified = false; - } - identity.conflict = saved.details.conflict === true; - binding = saved.details.binding; - needsReplay = mayHaveLinked && binding !== "invalidated"; - } - } const buffer = Buffer.alloc(65536); for (;;) { const { bytesRead } = await handle.read(buffer, 0, buffer.length, offset + pending.length); @@ -1028,8 +697,7 @@ export async function syncOtel( if (line.trim()) { const admitted: Transfer[] = []; const verdicts = new Map<string, boolean>(); - const scratch: Identity = { values: new Map(), conflict: false }; - for (const transfer of selectTransfers(line, options, scratch)) { + for (const transfer of selectTransfers(line, options)) { if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { skipped++; continue; @@ -1045,13 +713,11 @@ export async function syncOtel( continue; } admitted.push(transfer); - observeIdentity(transfer, options, identity, identityEvidence); } - if (!options.titlesOnly) - for (const batch of packTransfers(admitted)) { - await send(options, batch.transfer, () => stillPermitted(batch.items)); - sent += batch.count; - } + for (const batch of packTransfers(admitted)) { + await send(options, batch.transfer, () => stillPermitted(batch.items)); + sent += batch.count; + } } offset += newline + 1; pending = pending.subarray(newline + 1); @@ -1067,98 +733,6 @@ export async function syncOtel( ) { throw new Error("Capture was replaced or truncated; restart sync to replay it safely."); } - const feature = await resolve(); - const nextBinding = - feature.state === "confirmed" ? `${feature.branch}\0${feature.feature}` : feature.state; - const shouldInvalidate = - mayHaveLinked && - (identity.conflict || - feature.state !== "confirmed" || - (binding !== "" && binding !== nextBinding)); - const bindingAllowed = await permits({ - signal: "logs", - payload: {}, - at: new Date().toISOString(), - }); - if ( - !previousLinkUnverified && - !options.titlesOnly && - bindingAllowed && - shouldInvalidate && - binding !== "invalidated" - ) { - const event = bindingTransfer(options, identity, { state: "ambiguous" }); - if (event) { - await send(options, event, () => - stillPermitted([event, ...evidenceTransfers(identityEvidence)], feature) - ); - sent++; - binding = "invalidated"; - if (statePath !== null) - await saveBindingState(statePath, join(options.stateDir ?? "", "otel-sync"), { - session: options.session, - identity: [...identity.values], - binding, - ...(options.admission && identityEvidence.length - ? { - identityProof: { - version: 1, - admissionKey: admissionFingerprint(options.admission), - sources: identityEvidence, - }, - } - : {}), - }); - } - } else if ( - bindingAllowed && - !previousLinkUnverified && - !options.titlesOnly && - !shouldInvalidate && - feature.state === "confirmed" && - (nextBinding !== binding || needsReplay) - ) { - const event = bindingTransfer(options, identity, feature); - if (event) { - if (statePath !== null) { - await saveBindingState(statePath, join(options.stateDir ?? "", "otel-sync"), { - session: options.session, - identity: [...identity.values], - binding: nextBinding, - ...(options.admission && identityEvidence.length - ? { - identityProof: { - version: 1, - admissionKey: admissionFingerprint(options.admission), - sources: identityEvidence, - }, - } - : {}), - }); - } - mayHaveLinked = true; - await send(options, event, () => - stillPermitted([event, ...evidenceTransfers(identityEvidence)], feature) - ); - sent++; - binding = nextBinding; - needsReplay = false; - } - } - if (bindingAllowed) - for (const publisher of [titles, projects, repositories]) { - if ( - publisher && - (await publisher.publish(options.session, identity, async (event) => { - if (!(await permits(event))) return false; - await send(options, event, () => - stillPermitted([event, ...evidenceTransfers(identityEvidence)]) - ); - return true; - })) - ) - sent++; - } if (!options.follow || options.signal?.aborted) return { sent, skipped, offset, partialLine: pending.length > 0 }; try { @@ -1170,9 +744,6 @@ export async function syncOtel( } } } finally { - titles?.close(); - projects?.close(); - repositories?.close(); await handle.close(); } } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts index 1e3e9348c..08d223d81 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts @@ -3,24 +3,15 @@ import { open } from "node:fs/promises"; import { join } from "node:path"; import { readSettings } from "../files.js"; import { transformCapturedLine } from "../observations.js"; -import { - type Config, - configurationEdit, - type Endpoint, - type Host, - OPENCODE_CONFIG_ID, -} from "../settings.js"; +import { type Config, configurationEdit, type Endpoint, type Host } from "../settings.js"; export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) { const edit = await configurationEdit(config, endpoint, host); const current = await readSettings(edit.path, edit.initial); - if ( - (!current.exists && edit.containsToken !== false) || - edit.update(current.content) !== current.content - ) { + if (!current.exists || edit.update(current.content) !== current.content) { throw new Error("OTLP configuration is not initialized. Run aidd telemetry otel init first."); } - if (!current.private && edit.containsToken !== false) { + if (!current.private) { throw new Error("OTLP configuration must be private (mode 0600)."); } const capture = join(host.home, ".local/share/aidd-otel-observation", `${config}.jsonl`); @@ -46,42 +37,10 @@ export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) if (count === 0) unrecognisedLines++; } if (lines === 0) throw new Error("OTLP capture is empty. Run a tool session and retry."); - let eventLines = 0; - if (config === OPENCODE_CONFIG_ID) { - const eventsPath = join( - host.home, - ".local/share/aidd-otel-observation", - OPENCODE_CONFIG_ID, - "events.jsonl" - ); - const events = await open( - eventsPath, - constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK - ).catch((error) => { - if (error.code === "ENOENT") return null; - throw error; - }); - if (events) { - try { - if (!(await events.stat()).isFile()) - throw new Error("OpenCode event capture must be a regular file."); - for await (const line of events.readLines({ autoClose: false })) { - if (!line.trim()) continue; - eventLines++; - const count = transformCapturedLine(config, line).length; - observations += count; - if (count === 0) unrecognisedLines++; - } - } finally { - await events.close(); - } - } - } return { path: edit.path, capture, lines, - eventLines, observations, unrecognisedLines, modifiedAt: info.mtime.toISOString(), diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts b/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts index eabd31d9a..407d5c6a7 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/config-edits.ts @@ -20,15 +20,6 @@ export function rejectConflict(name: string, actual: unknown, expected: string): } } -/** Keep offsets stable while hiding JSONC comments and trailing commas from JSON.parse. */ -function normalizeJsonc(content: string): string { - return content - .replace(/"(?:[^"\\]|\\.)*"|\/\/[^\r\n]*|\/\*[\s\S]*?\*\//g, (token) => - token.startsWith("/") ? token.replace(/[^\r\n]/g, " ") : token - ) - .replace(/"(?:[^"\\]|\\.)*"|,(?=\s*[}\]])/g, (token) => (token === "," ? " " : token)); -} - function readJsonSettings(content: string): Settings { try { return requireObject(JSON.parse(content)); @@ -70,14 +61,12 @@ function sectionOffset(content: string, section: string): number { function mergeJsonSection( content: string, section: string, - entries: (existing: Settings) => Settings, - normalize: (source: string) => string = (source) => source + entries: (existing: Settings) => Settings ): string { - const normalized = normalize(content); - const settings = readJsonSettings(normalized); + const settings = readJsonSettings(content); const original = settings[section] === undefined ? settings : requireObject(settings[section]); const existing = settings[section] === undefined ? {} : original; - const offset = sectionOffset(normalized, section) + 1; + const offset = sectionOffset(content, section) + 1; const missing = entries(existing); if (Object.keys(missing).length === 0) return content; const patch = settings[section] === undefined ? { [section]: missing } : missing; @@ -85,7 +74,7 @@ function mergeJsonSection( const entry = JSON.stringify(patch, null, 2).slice(1, -2).replaceAll("\n", `\n${indent}`); const separator = Object.keys(original).length === 0 ? "\n" : ","; const next = content.slice(0, offset) + entry + separator + content.slice(offset); - readJsonSettings(normalize(next)); + readJsonSettings(next); return next; } @@ -104,28 +93,6 @@ export function mergeJsonEnvironment( }); } -export function openCodeInstrumentation(content: string): unknown { - const settings = readJsonSettings(normalizeJsonc(content)); - return settings.experimental === undefined - ? undefined - : requireObject(settings.experimental).openTelemetry; -} - -export function mergeOpenCodeInstrumentation(content: string, enabled: boolean): string { - return mergeJsonSection( - content, - "experimental", - (experimental) => { - if (experimental.openTelemetry === enabled) return {}; - if (experimental.openTelemetry !== undefined) { - throw new Error("OpenCode experimental.openTelemetry is disabled or invalid."); - } - return { openTelemetry: enabled }; - }, - normalizeJsonc - ); -} - function readToml(content: string): Settings { try { return parse(content); diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json index 1ab4fd21f..30fafb505 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json +++ b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json @@ -2,6 +2,5 @@ "env": { "OTEL_EXPORTER_OTLP_ENDPOINT": "{collector}/opencode", "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}" - }, - "experimental": { "openTelemetry": true } + } } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/observations.ts b/cli/src/contexts/telemetry/infrastructure/otel/observations.ts index 060bf00a9..e72ddad6c 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/observations.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/observations.ts @@ -10,20 +10,6 @@ interface OtelCapture { readonly payload: unknown; readonly received_at?: string; } -export interface ClaudeTelemetrySource extends OtelCapture {} -export interface CodexTelemetrySource extends OtelCapture {} -export interface OpenCodeV1Source { - readonly type: string; - readonly properties: unknown; -} -export interface OpenCodeV2Source { - readonly type: string; - readonly data: unknown; - readonly created?: number; - readonly location?: unknown; -} -export type OpenCodeTelemetrySource = OpenCodeV1Source | OpenCodeV2Source | OtelCapture; - type Obj = Record<string, unknown>; function json(value: unknown): JsonValue | undefined { if (value === null || typeof value === "boolean" || typeof value === "string") return value; @@ -48,6 +34,11 @@ function number(value: unknown): number | undefined { const n = Number(value); return Number.isFinite(n) && n >= 0 ? n : undefined; } +function tokenCount(value: unknown): number | undefined { + if (typeof value === "string" && !/^\d+$/.test(value)) return undefined; + const count = number(value); + return count !== undefined && Number.isSafeInteger(count) ? count : undefined; +} function bool(value: unknown): boolean | undefined { if (value === true || value === "true") return true; if (value === false || value === "false") return false; @@ -63,15 +54,6 @@ function nanos(value: unknown): string | undefined { return undefined; } } -function millis(value: unknown): string | undefined { - const n = number(value); - if (n === undefined) return undefined; - try { - return new Date(n).toISOString(); - } catch { - return undefined; - } -} function decodeValue(raw: unknown): unknown { const item = object(raw); for (const key of ["stringValue", "boolValue", "intValue", "doubleValue"]) { @@ -96,12 +78,20 @@ function first<T>(...values: (T | undefined)[]): T | undefined { function pickText(attributes: Obj, ...keys: string[]): string | undefined { return first(...keys.map((key) => text(attributes[key]))); } -function pickNumber(attributes: Obj, ...keys: string[]): number | undefined { - return first(...keys.map((key) => number(attributes[key]))); +function pickTokenCount(attributes: Obj, ...keys: string[]): number | undefined { + return first(...keys.map((key) => tokenCount(attributes[key]))); } function tokens(fields: TelemetryObservation["tokens"]): TelemetryObservation["tokens"] { return fields && Object.values(fields).some((item) => item !== undefined) ? fields : undefined; } +function cacheZeroMayMeanMissing( + tool: TelemetryObservation["tool"], + usage: TelemetryObservation["tokens"] +): true | undefined { + return tool === CODEX_CONFIG_ID && (usage?.cacheRead === 0 || usage?.cacheWrite === 0) + ? true + : undefined; +} function base( tool: TelemetryObservation["tool"], signal: TelemetryObservation["signal"], @@ -109,7 +99,7 @@ function base( source: unknown ): TelemetryObservation { const usage = tokens({ - input: pickNumber( + input: pickTokenCount( a, "input_tokens", "input_token_count", @@ -117,12 +107,12 @@ function base( "gen_ai.usage.input_tokens", "ai.usage.inputTokens" ), - inputNonCached: pickNumber( + inputNonCached: pickTokenCount( a, "codex.turn.token_usage.non_cached_input_tokens", "ai.usage.inputTokenDetails.noCacheTokens" ), - output: pickNumber( + output: pickTokenCount( a, "output_tokens", "output_token_count", @@ -130,7 +120,7 @@ function base( "gen_ai.usage.output_tokens", "ai.usage.outputTokens" ), - cacheRead: pickNumber( + cacheRead: pickTokenCount( a, "cache_read_tokens", "cached_token_count", @@ -139,16 +129,16 @@ function base( "ai.usage.cachedInputTokens", "ai.usage.inputTokenDetails.cacheReadTokens" ), - cacheWrite: pickNumber( + cacheWrite: pickTokenCount( a, "cache_creation_tokens", "cache_write_token_count", "codex.turn.token_usage.cache_write_input_tokens", "gen_ai.usage.cache_write.input_tokens" ), - cacheWrite5m: number(a.cache_creation_5m_tokens), - cacheWrite1h: number(a.cache_creation_1h_tokens), - reasoning: pickNumber( + cacheWrite5m: tokenCount(a.cache_creation_5m_tokens), + cacheWrite1h: tokenCount(a.cache_creation_1h_tokens), + reasoning: pickTokenCount( a, "reasoning_token_count", "codex.turn.token_usage.reasoning_output_tokens", @@ -156,13 +146,13 @@ function base( "ai.usage.reasoningTokens", "ai.usage.outputTokenDetails.reasoningTokens" ), - total: pickNumber( + total: pickTokenCount( a, "codex.turn.token_usage.total_tokens", "codex.usage.total_tokens", "ai.usage.totalTokens" ), - tool: number(a.tool_token_count), + tool: tokenCount(a.tool_token_count), }); return { tool, @@ -173,6 +163,10 @@ function base( toolVersion: pickText(a, "app.version", "service.version"), sessionId: pickText(a, "session.id", "conversation.id"), threadId: text(a["thread.id"]), + turnId: tool === CODEX_CONFIG_ID ? text(a["turn.id"]) : undefined, + promptId: tool === CLAUDE_CONFIG_ID ? text(a["prompt.id"]) : undefined, + clientRequestId: tool === CLAUDE_CONFIG_ID ? text(a.client_request_id) : undefined, + messageId: tool === CLAUDE_CONFIG_ID ? text(a["message.uuid"]) : undefined, provider: pickText( a, "anthropic.provider", @@ -193,6 +187,7 @@ function base( cwd: pickText(a, "cwd", "directory"), tokens: usage, usageScope: usage ? "event" : undefined, + cacheZeroMayMeanMissing: cacheZeroMayMeanMissing(tool, usage), tokenInputIncludesCache: usage?.input !== undefined && tool !== OPENCODE_CONFIG_ID ? tool === CODEX_CONFIG_ID @@ -216,6 +211,8 @@ function fromLog( ...row, timestamp: first(row.timestamp, nanos(entry.timeUnixNano)), observedTimestamp: nanos(entry.observedTimeUnixNano), + traceId: text(entry.traceId), + spanId: text(entry.spanId), prompt: name === "user_prompt" || name === "codex.user_prompt" ? pickText(a, "prompt_text", "prompt") @@ -265,8 +262,11 @@ function fromSpan( timestamp: nanos(event.timeUnixNano), toolVersion: row.toolVersion, sessionId: row.sessionId, + threadId: row.threadId, + turnId: row.turnId, traceId: row.traceId, spanId: row.spanId, + parentSpanId: row.parentSpanId, toolCallId: row.toolCallId, toolName: row.toolName, toolOutput: event.name === "tool.output" ? json(ea.output) : undefined, @@ -286,7 +286,10 @@ function fromMetric( const point = object(rawPoint); const a = { ...resource, ...attributes(point.attributes) }; const metricType = pickText(a, "type", "token_type"); - const amount = first(number(point.asInt), number(point.asDouble), number(point.sum)); + const rawAmount = [point.asInt, point.asDouble, point.sum].find( + (value) => number(value) !== undefined + ); + const amount = number(rawAmount); const tokenKinds: Record<string, keyof NonNullable<TelemetryObservation["tokens"]>> = { input: "input", output: "output", @@ -297,11 +300,12 @@ function fromMetric( reasoning_output: "reasoning", }; const token = metricType && tokenKinds[metricType]; + const count = tokenCount(rawAmount); const usage = (name === "claude_code.token.usage" || name === "codex.turn.token_usage") && token && - amount !== undefined - ? { [token]: amount } + count !== undefined + ? { [token]: count } : undefined; return { ...base(tool, "metric", a, rawPoint), @@ -313,6 +317,7 @@ function fromMetric( metricValue: amount, tokens: usage, usageScope: usage ? "metric-aggregate" : undefined, + cacheZeroMayMeanMissing: cacheZeroMayMeanMissing(tool, usage), }; }) ); @@ -343,140 +348,10 @@ function otel(tool: TelemetryObservation["tool"], capture: OtelCapture): Telemet ); }); } -export function transformClaude(source: ClaudeTelemetrySource): TelemetryObservation[] { - return otel(CLAUDE_CONFIG_ID, source); -} -export function transformCodex(source: CodexTelemetrySource): TelemetryObservation[] { - return otel(CODEX_CONFIG_ID, source); -} -function openCodeTokens(raw: unknown): TelemetryObservation["tokens"] { - const t = object(raw); - const cache = object(t.cache); - return tokens({ - input: number(t.input), - output: number(t.output), - cacheRead: number(cache.read), - cacheWrite: number(cache.write), - reasoning: number(t.reasoning), - total: number(t.total), - }); -} -function openCodeV1(source: OpenCodeV1Source): TelemetryObservation[] { - const p = object(source.properties); - const info = object(p.info); - const part = object(p.part); - const state = object(part.state); - const model = object(info.model); - const step = part.type === "step-finish"; - const usage = openCodeTokens(step ? part.tokens : info.tokens); - const status = text(state.status); - return [ - { - tool: OPENCODE_CONFIG_ID, - signal: "event", - source: json(source), - openCodeVersion: 1, - eventType: source.type, - timestamp: first(millis(p.time), millis(object(info.time).created)), - sessionId: first( - text(p.sessionID), - text(info.sessionID), - text(part.sessionID), - source.type === "session.created" ? text(info.id) : undefined - ), - messageId: first( - source.type === "message.updated" ? text(info.id) : undefined, - text(part.messageID) - ), - toolVersion: text(info.version), - partId: text(part.id), - provider: first(text(info.providerID), text(model.providerID)), - model: first(text(info.modelID), text(model.modelID)), - role: text(info.role), - text: part.type === "text" ? text(part.text) : undefined, - toolCallId: part.type === "tool" ? text(part.callID) : undefined, - toolName: part.type === "tool" ? text(part.tool) : undefined, - toolInput: part.type === "tool" ? json(state.input) : undefined, - toolOutput: part.type === "tool" ? json(state.output) : undefined, - success: status === "completed" ? true : status === "error" ? false : undefined, - error: status === "error" ? json(state.error) : undefined, - tokens: usage, - usageScope: usage - ? step - ? "step-snapshot" - : source.type === "session.created" || source.type === "session.updated" - ? "session-cumulative" - : "message-snapshot" - : undefined, - tokenInputIncludesCache: usage?.input !== undefined ? false : undefined, - tokenOutputIncludesReasoning: usage?.output !== undefined ? false : undefined, - zeroMayMeanMissing: usage ? true : undefined, - reportedCostUsd: number(step ? part.cost : info.cost), - cwd: text(object(info.path).cwd), - }, - ]; -} -function openCodeV2(source: OpenCodeV2Source): TelemetryObservation[] { - const d = object(source.data); - const model = object(d.model); - const item = object(d.item); - const itemPayload = object(item.payload); - const inboxModel = object(object(itemPayload.metadata).model); - const usage = openCodeTokens(d.tokens); - const kind = source.type; - return [ - { - tool: OPENCODE_CONFIG_ID, - signal: "event", - source: json(source), - openCodeVersion: 2, - eventType: kind, - timestamp: millis(source.created), - toolVersion: kind === "session.created" ? text(d.version) : undefined, - sessionId: text(d.sessionID), - messageId: text(d.assistantMessageID), - provider: first(text(model.providerID), text(inboxModel.providerID)), - model: first(text(model.id), text(inboxModel.modelID)), - cwd: text(object(source.location).directory), - role: kind === "session.inbox.enqueued" ? text(item.type) : undefined, - prompt: - kind === "session.inbox.enqueued" && item.type === "user" - ? text(itemPayload.text) - : undefined, - response: kind === "session.text.ended" ? text(d.text) : undefined, - text: kind === "session.text.delta" ? text(d.delta) : undefined, - toolCallId: kind.startsWith("session.tool.") ? text(d.id) : undefined, - toolName: kind === "session.tool.input.started" ? text(d.name) : undefined, - toolInput: - kind === "session.tool.called" - ? json(d.input) - : kind === "session.tool.input.ended" - ? json(d.text) - : undefined, - toolOutput: kind === "session.tool.success" ? json(d.content) : undefined, - success: - kind === "session.tool.success" ? true : kind === "session.tool.failed" ? false : undefined, - error: kind === "session.tool.failed" ? json(d.error) : undefined, - tokens: usage, - usageScope: usage - ? kind === "session.usage.updated" - ? "session-cumulative" - : "event" - : undefined, - tokenInputIncludesCache: usage?.input !== undefined ? false : undefined, - tokenOutputIncludesReasoning: usage?.output !== undefined ? false : undefined, - zeroMayMeanMissing: usage ? true : undefined, - reportedCostUsd: number(d.cost), - }, - ]; -} -export function transformOpenCode(source: OpenCodeTelemetrySource): TelemetryObservation[] { - if ("signal" in source) return otel(OPENCODE_CONFIG_ID, source); - if ("properties" in source) return openCodeV1(source); - return openCodeV2(source); -} +export const transformClaude = (source: OtelCapture) => otel(CLAUDE_CONFIG_ID, source); +export const transformCodex = (source: OtelCapture) => otel(CODEX_CONFIG_ID, source); +export const transformOpenCode = (source: OtelCapture) => otel(OPENCODE_CONFIG_ID, source); -/** Decode one captured JSON line. Unknown or malformed records remain visibly unrecognised. */ export function transformCapturedLine(tool: Config, line: string): TelemetryObservation[] { let input: Obj; try { @@ -486,19 +361,7 @@ export function transformCapturedLine(tool: Config, line: string): TelemetryObse } if (input.signal === "logs" || input.signal === "traces" || input.signal === "metrics") { const source: OtelCapture = { signal: input.signal, payload: input.payload }; - if (tool === CLAUDE_CONFIG_ID) return transformClaude(source); - if (tool === CODEX_CONFIG_ID) return transformCodex(source); - return transformOpenCode(source); + return otel(tool, source); } - if (tool !== OPENCODE_CONFIG_ID || typeof input.type !== "string") return []; - if ("properties" in input) - return transformOpenCode({ type: input.type, properties: input.properties }); - if ("data" in input) - return transformOpenCode({ - type: input.type, - data: input.data, - created: number(input.created), - location: input.location, - }); return []; } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts index d744bd598..a1329eda5 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts @@ -8,14 +8,7 @@ import type { UsageTool } from "../../domain/usage-record.js"; import claudeTemplate from "./config/claude.json" with { type: "json" }; import codexTemplate from "./config/codex.json" with { type: "json" }; import opencodeTemplate from "./config/opencode.json" with { type: "json" }; -import { - mergeCodexExporters, - mergeJsonEnvironment, - mergeOpenCodeInstrumentation, - openCodeInstrumentation, - rejectConflict, -} from "./config-edits.js"; -import { readSettings } from "./files.js"; +import { mergeCodexExporters, mergeJsonEnvironment, rejectConflict } from "./config-edits.js"; export const OPENCODE_CONFIG_ID = "opencode"; export const CLAUDE_CONFIG_ID = "claude"; @@ -43,7 +36,6 @@ export interface Edit { readonly initial: string; readonly update: (content: string) => string; readonly launch: string; - readonly containsToken?: false; } const COLLECTOR = "http://127.0.0.1:14318"; @@ -104,7 +96,7 @@ function claudeEdit(host: Host, token: string): Edit { }; } -export async function openCodeVersion(host: Host): Promise<"1" | "2"> { +async function requireOpenCodeV2(host: Host): Promise<void> { let output: string; try { output = host.opencodeVersion @@ -118,64 +110,20 @@ export async function openCodeVersion(host: Host): Promise<"1" | "2"> { } catch { throw new Error("Cannot determine OpenCode version. Make opencode available on PATH."); } - const version = /^\s*(?:opencode\s+)?v?([12])\.(\d+)\.(\d+)\s*$/i.exec(output); - if (version && (version[1] === "1" || version[1] === "2")) { - // Native export from 1.18.35 was verified locally; other older releases remain unverified. - if (version[1] === "1" && Number(version[2]) === 18 && Number(version[3]) === 35) return "1"; - const minimumMinor = version[1] === "1" ? 19 : 2; - const minor = Number(version[2]); - if (minor > minimumMinor || (minor === minimumMinor && Number(version[3]) >= 5)) { - return version[1]; - } + const version = /^\s*(?:opencode\s+)?v?2\.(\d+)\.(\d+)\s*$/i.exec(output); + if (version) { + const minor = Number(version[1]); + if (minor > 2 || (minor === 2 && Number(version[2]) >= 5)) return; } - throw new Error( - "Unsupported OpenCode version. Use OpenCode 1.18.35, a 1.x release from 1.19.5, or 2.x from 2.2.5." - ); + throw new Error("Unsupported OpenCode version. Use OpenCode 2.2.5 or newer within 2.x."); } function openCodeGlobalDir(host: Host): string { return join(host.env.XDG_CONFIG_HOME || join(host.home, ".config"), OPENCODE_CONFIG_ID); } -export async function openCodeV1Edit(host: Host): Promise<Edit> { - const global = openCodeGlobalDir(host); - const custom = host.env.OPENCODE_CONFIG_DIR; - const initial = "{}\n"; - const nativeFiles = ["opencode.json", "opencode.jsonc"]; - const files = ["config.json", ...nativeFiles].map((name) => join(global, name)); - if (host.env.OPENCODE_CONFIG) files.push(host.env.OPENCODE_CONFIG); - if (custom && custom !== global) files.push(...nativeFiles.map((name) => join(custom, name))); - let path: string | undefined; - let flag: unknown; - for (const file of new Set(files)) { - const existing = await readSettings(file, initial); - if (!existing.exists) continue; - path = file; - const value = openCodeInstrumentation(existing.content); - if (value !== undefined) flag = value; - } - if (host.env.OPENCODE_CONFIG_CONTENT) { - const inline = openCodeInstrumentation(host.env.OPENCODE_CONFIG_CONTENT); - if (inline !== undefined) flag = inline; - } - if (flag !== undefined && flag !== true) { - throw new Error("OpenCode experimental.openTelemetry is disabled or invalid."); - } - return { - path: path ?? join(custom || global, nativeFiles[0]), - initial, - launch: - "Run OpenCode 1 with aidd telemetry otel run --config=opencode -- [OpenCode arguments].", - containsToken: false, - update: (content) => - flag === true && openCodeInstrumentation(content) === undefined - ? content - : mergeOpenCodeInstrumentation(content, opencodeTemplate.experimental.openTelemetry), - }; -} - async function openCodeEdit(host: Host, token: string): Promise<Edit> { - if ((await openCodeVersion(host)) === "1") return openCodeV1Edit(host); + await requireOpenCodeV2(host); const directory = host.env.OPENCODE_CONFIG_DIR || openCodeGlobalDir(host); const additions = interpolate(opencodeTemplate.env, token); return { diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 397a0be0c..35d5d9831 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -33,10 +33,8 @@ import { collectOtel, configureOtel, followCapturedSessions, - launchOtel, otelOptions, syncCapturedSession, - syncCapturedSessionTitles, } from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, @@ -184,7 +182,7 @@ export function registerTelemetryCommand(program: Command): void { `${result.lines} captured lines in ${result.capture}; last modified ${result.modifiedAt}` ); output.info( - `${result.observations} normalized observations; ${result.eventLines} event lines; ${result.unrecognisedLines} unrecognised lines` + `${result.observations} normalized observations; ${result.unrecognisedLines} unrecognised lines` ); output.info("Existing capture does not prove delivery from the current session."); } @@ -222,26 +220,6 @@ export function registerTelemetryCommand(program: Command): void { .requiredOption("--tenant <slug>", "Gouvernail tenant slug") .requiredOption("--endpoint <origin>", "Loopback HTTP origin") .option("--follow", "Follow new events for this session", false) - .option( - "--include-session-titles", - "Send Codex chat titles, which may contain prompt text", - false - ) - .option( - "--include-session-projects", - "Send native Codex project IDs and names (may contain sensitive text)", - false - ) - .option( - "--include-session-repositories", - "Send verified Codex GitHub/GitLab repository URLs (may identify private work)", - false - ) - .option( - "--titles-only", - "Send only Codex titles and opted-in metadata for consented sessions", - false - ) .action( async (options: { config: (typeof otelOptions.syncConfigs)[number]; @@ -249,75 +227,15 @@ export function registerTelemetryCommand(program: Command): void { tenant: string; endpoint: string; follow: boolean; - includeSessionTitles: boolean; - includeSessionProjects: boolean; - includeSessionRepositories: boolean; - titlesOnly: boolean; }) => { const { output, run } = commandOf(program); - if (!options.session && !options.follow && !options.titlesOnly) { + if (!options.session && !options.follow) { output.error("Pass --session for a bounded sync, or --follow to discover new sessions."); process.exitCode = 1; return; } - if (options.titlesOnly && options.follow) { - output.error("Title-only sync does not support --follow."); - process.exitCode = 1; - return; - } - if ( - (options.includeSessionTitles || - options.includeSessionProjects || - options.includeSessionRepositories || - options.titlesOnly) && - options.config !== otelOptions.titleConfig - ) { - output.error("Session metadata sync is currently available only for Codex."); - process.exitCode = 1; - return; - } await run(async (deps) => { const tool = otelOptions.usageTool(options.config); - const onProjectUnavailable = () => - output.warn( - "A native Codex session project could not be verified; telemetry sync continues." - ); - const onRepositoryUnavailable = () => - output.warn( - "A native Codex Git repository could not be verified; telemetry sync continues." - ); - const onTitleUnavailable = () => - output.warn("A Codex session title could not be read; telemetry sync continues."); - if (options.titlesOnly && !options.session) { - const result = await syncCapturedSessionTitles( - { - config: options.config, - tenant: options.tenant, - endpoint: options.endpoint, - token: process.env.AIDD_OTEL_TOKEN ?? "", - includeSessionTitles: true, - includeSessionProjects: options.includeSessionProjects, - includeSessionRepositories: options.includeSessionRepositories, - onProjectUnavailable, - onRepositoryUnavailable, - onTitleUnavailable, - }, - () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), - (admission, at, start, current) => - deps.telemetry.sessionFeatureUseCase.permits(admission, at, start, current) - ); - output.info( - `Sent ${result.sent} session metadata updates; no metrics or feature bindings were sent.` - ); - if (!result.sent && !result.identified) - output.warn( - "No unambiguous provider identity was found in consented capture events." - ); - else if (!result.sent && !result.currentConsenting) - output.warn("No identified session has current clone consent."); - if (result.partialLine) output.info("The last incomplete capture line was not read."); - return; - } if (!options.session) { output.info("Following new consented sessions; press Ctrl-C to stop."); const result = await followCapturedSessions( @@ -326,12 +244,6 @@ export function registerTelemetryCommand(program: Command): void { tenant: options.tenant, endpoint: options.endpoint, token: process.env.AIDD_OTEL_TOKEN ?? "", - includeSessionTitles: options.includeSessionTitles, - includeSessionProjects: options.includeSessionProjects, - includeSessionRepositories: options.includeSessionRepositories, - onProjectUnavailable, - onRepositoryUnavailable, - onTitleUnavailable, }, () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), (admission, at, start, current) => @@ -349,7 +261,7 @@ export function registerTelemetryCommand(program: Command): void { const initialAdmission = admission; if (!initialAdmission) throw new Error("No consented usage found for this session; no capture was sent."); - const resolve = async () => { + const revalidate = async () => { const current = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get( session ); @@ -375,32 +287,16 @@ export function registerTelemetryCommand(program: Command): void { ); } admission = current; - return admission.feature; }; - await resolve(); + await revalidate(); if (options.follow) output.info("Following the selected session; press Ctrl-C to stop."); const result = await syncCapturedSession( { ...options, - includeSessionTitles: options.includeSessionTitles || options.titlesOnly, - onProjectUnavailable, - onRepositoryUnavailable, session, - ...(initialAdmission.cwd && - (initialAdmission.repositoryId || - (initialAdmission.repositoryId === null && - initialAdmission.feature.state !== "confirmed")) - ? { - admission: { - cwd: initialAdmission.cwd, - repositoryId: initialAdmission.repositoryId, - }, - } - : {}), token: process.env.AIDD_OTEL_TOKEN ?? "", - onTitleUnavailable, }, - resolve, + revalidate, async (transfer) => Boolean( admission && @@ -422,23 +318,6 @@ export function registerTelemetryCommand(program: Command): void { } ); - otel - .command("run [args...]") - .description("Launch OpenCode 1 with local OTLP export. Pass tool arguments after --") - .addOption( - new Option("--config <tool>", "Tool to launch") - .choices(otelOptions.launchConfigs) - .makeOptionMandatory() - ) - .action(async (args: string[]) => { - const { output } = parseGlobalOptions(program); - try { - process.exitCode = await launchOtel(args); - } catch (error) { - new ErrorHandler(output).handle(error); - } - }); - telemetry .command("on") .description("Measure this clone, and remove what the previous version left in it") diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index 69b8abdc3..effa2c5e6 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -1,12 +1,7 @@ import { homedir } from "node:os"; -import type { - SessionAdmission, - SessionFeature, -} from "../../contexts/telemetry/domain/report/session-feature.js"; - +import type { SessionAdmission } from "../../contexts/telemetry/domain/report/session-feature.js"; import { startOtelCollector } from "../../contexts/telemetry/infrastructure/otel/actions/collect.js"; import { initializeOtel } from "../../contexts/telemetry/infrastructure/otel/actions/init.js"; -import { runOtelOpenCode } from "../../contexts/telemetry/infrastructure/otel/actions/run.js"; import { type SyncOptions, syncOtel, @@ -16,14 +11,8 @@ import { type AllOptions, syncAllOtel, } from "../../contexts/telemetry/infrastructure/otel/actions/sync-all.js"; -import { - syncAllSessionTitles, - type TitleOptions, -} from "../../contexts/telemetry/infrastructure/otel/actions/sync-all-titles.js"; import { verifyOtel } from "../../contexts/telemetry/infrastructure/otel/actions/verify.js"; import { - CODEX_CONFIG_ID, - OPENCODE_CONFIG_ID, OTEL_CONFIGS, OTEL_ENDPOINTS, OTEL_SYNC_CONFIGS, @@ -42,11 +31,11 @@ export function collectOtel(port: number) { } export function syncCapturedSession( - options: Omit<SyncOptions, "home" | "stateDir">, - resolve: () => Promise<SessionFeature>, + options: Omit<SyncOptions, "home">, + revalidate: () => Promise<void>, permits: (transfer: Transfer) => Promise<boolean> ) { - return syncOtel({ ...options, home: homedir(), stateDir: telemetryDir() }, resolve, permits); + return syncOtel({ ...options, home: homedir() }, revalidate, permits); } export function followCapturedSessions( @@ -73,26 +62,11 @@ export function followCapturedSessions( ); } -export function syncCapturedSessionTitles( - options: Omit<TitleOptions, "home" | "stateDir">, - snapshot: () => Promise<ReadonlyMap<string, SessionAdmission>>, - permits: ( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ) => Promise<"allow" | "deny" | "unknown"> -) { - return syncAllSessionTitles({ ...options, home: homedir() }, snapshot, permits); -} - export const otelOptions = { configs: OTEL_CONFIGS, syncConfigs: OTEL_SYNC_CONFIGS, - titleConfig: CODEX_CONFIG_ID, usageTool: usageToolForOtel, endpoints: OTEL_ENDPOINTS, - launchConfigs: [OPENCODE_CONFIG_ID], }; export async function configureOtel( @@ -107,10 +81,6 @@ export async function configureOtel( return result; } -export function launchOtel(args: readonly string[]) { - return runOtelOpenCode(args, { home: homedir(), env: process.env }); -} - export function checkOtel( config: (typeof OTEL_CONFIGS)[number], endpoint: (typeof OTEL_ENDPOINTS)[number] diff --git a/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts index 8be088f06..b8092e585 100644 --- a/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts +++ b/cli/tests/contexts/telemetry/domain/observations/transformers.unit.test.ts @@ -72,71 +72,6 @@ describe("telemetry observations", () => { success: true, }); }); - - it("maps OpenCode v2 step and cumulative updates without treating them as separate calls", () => { - const data = { - sessionID: "s3", - tokens: { input: 10, output: 4, cache: { read: 5, write: 0 } }, - }; - const step = transformOpenCode({ type: "session.step.ended", created: 1700000000000, data }); - const cumulative = transformOpenCode({ - type: "session.usage.updated", - created: 1700000000000, - data, - }); - expect(step[0]).toMatchObject({ - tool: "opencode", - openCodeVersion: 2, - sessionId: "s3", - usageScope: "event", - tokens: { input: 10, output: 4, cacheRead: 5, cacheWrite: 0 }, - }); - expect(cumulative[0]?.usageScope).toBe("session-cumulative"); - }); - - it("maps OpenCode v1 message and tool events from the installed CLI schema", () => { - const message = transformOpenCode({ - type: "message.updated", - properties: { - sessionID: "s4", - info: { - id: "m4", - role: "assistant", - modelID: "model", - providerID: "provider", - tokens: { input: 2, output: 3, cache: { read: 1, write: 0 } }, - }, - }, - }); - expect(message[0]).toMatchObject({ - openCodeVersion: 1, - sessionId: "s4", - messageId: "m4", - model: "model", - provider: "provider", - usageScope: "message-snapshot", - tokens: { input: 2, output: 3, cacheRead: 1, cacheWrite: 0 }, - }); - const tool = transformOpenCode({ - type: "message.part.updated", - properties: { - sessionID: "s4", - part: { - type: "tool", - callID: "c4", - tool: "read", - state: { status: "completed", input: { path: "x" }, output: "done" }, - }, - }, - }); - expect(tool[0]).toMatchObject({ - toolCallId: "c4", - toolName: "read", - toolInput: { path: "x" }, - toolOutput: "done", - success: true, - }); - }); }); describe("source fidelity", () => { @@ -212,61 +147,9 @@ describe("source fidelity", () => { }); expect(rows[1]?.tokens).toBeUndefined(); }); - - it("does not label a v1 text part as an assistant response without its role", () => { - const result = transformOpenCode({ - type: "message.part.updated", - properties: { sessionID: "s", part: { type: "text", text: "text" } }, - }); - expect(result[0]).toMatchObject({ text: "text" }); - expect(result[0]?.response).toBeUndefined(); - }); - - it("keeps v2 synthetic inbox messages out of the user prompt", () => { - const result = transformOpenCode({ - type: "session.inbox.enqueued", - data: { item: { type: "synthetic", payload: { text: "text" } } }, - }); - expect(result[0]?.prompt).toBeUndefined(); - expect(result[0]?.role).toBe("synthetic"); - }); }); -describe("OpenCode v1 step and user records", () => { - it("maps user-selected model without inventing message content", () => { - const [row] = transformOpenCode({ - type: "message.updated", - properties: { info: { role: "user", model: { modelID: "m1", providerID: "p1" } } }, - }); - expect(row).toMatchObject({ role: "user", model: "m1", provider: "p1" }); - expect(row?.prompt).toBeUndefined(); - }); - it("records step-finish tokens as a step snapshot, not a message total", () => { - const [row] = transformOpenCode({ - type: "message.part.updated", - properties: { - part: { - type: "step-finish", - id: "part", - sessionID: "session", - messageID: "message", - cost: 0.25, - tokens: { input: 3, output: 4, reasoning: 2, cache: { read: 1, write: 0 } }, - }, - }, - }); - expect(row).toMatchObject({ - sessionId: "session", - messageId: "message", - partId: "part", - usageScope: "step-snapshot", - reportedCostUsd: 0.25, - tokenInputIncludesCache: false, - tokenOutputIncludesReasoning: false, - zeroMayMeanMissing: true, - tokens: { input: 3, output: 4, reasoning: 2, cacheRead: 1, cacheWrite: 0 }, - }); - }); +describe("Token semantics", () => { it("keeps input token semantics explicit for Claude and Codex", () => { const claude = transformClaude(log([attr("input_tokens", integer(17))])); const codex = transformCodex(log([attr("input_token_count", integer(17))])); @@ -276,7 +159,7 @@ describe("OpenCode v1 step and user records", () => { }); describe("source-specific fields", () => { - it("maps OpenCode v1 native AI SDK trace without treating parent and child spans as calls", () => { + it("maps an OpenCode OTLP AI SDK trace without treating parent and child spans as calls", () => { const capture = { signal: "traces" as const, payload: { @@ -345,19 +228,6 @@ describe("source-specific fields", () => { tokens: { inputNonCached: 9, total: 23, tool: 4 }, }); }); - - it("keeps OpenCode inbox model when no step has been emitted yet", () => { - const [row] = transformOpenCode({ - type: "session.inbox.enqueued", - data: { - item: { - type: "user", - payload: { metadata: { model: { modelID: "model", providerID: "provider" } } }, - }, - }, - }); - expect(row).toMatchObject({ model: "model", provider: "provider" }); - }); }); describe("OTLP field aliases", () => { @@ -536,100 +406,21 @@ describe("OTLP event and metric semantics", () => { }); }); -describe("OpenCode v2 event fields", () => { - it("maps step start, assistant text and user input without mixing roles", () => { - const [start] = transformOpenCode({ - type: "session.step.started", - created: 1700000000000, - location: { directory: "/project" }, - data: { - sessionID: "s", - assistantMessageID: "m", - model: { id: "model", providerID: "provider" }, - }, - }); - expect(start).toMatchObject({ - timestamp: "2023-11-14T22:13:20.000Z", - sessionId: "s", - messageId: "m", - cwd: "/project", - model: "model", - provider: "provider", - }); - const [prompt] = transformOpenCode({ - type: "session.inbox.enqueued", - data: { item: { type: "user", payload: { text: "fix this" } } }, - }); - expect(prompt).toMatchObject({ role: "user", prompt: "fix this" }); - const [reply] = transformOpenCode({ type: "session.text.ended", data: { text: "done" } }); - expect(reply?.response).toBe("done"); - const [delta] = transformOpenCode({ type: "session.text.delta", data: { delta: "part" } }); - expect(delta?.text).toBe("part"); - expect(delta?.response).toBeUndefined(); - }); - - it("maps tool lifecycle, results and errors to the same call id", () => { - const [start] = transformOpenCode({ - type: "session.tool.input.started", - data: { id: "call", name: "read" }, - }); - const [called] = transformOpenCode({ - type: "session.tool.called", - data: { id: "call", input: { path: "x" } }, - }); - const [finished] = transformOpenCode({ - type: "session.tool.success", - data: { id: "call", content: [{ type: "text", text: "done" }] }, - }); - const [failed] = transformOpenCode({ - type: "session.tool.failed", - data: { id: "call", error: { type: "tool_error", message: "failed" } }, - }); - expect(start).toMatchObject({ toolCallId: "call", toolName: "read" }); - expect(called).toMatchObject({ toolCallId: "call", toolInput: { path: "x" } }); - expect(finished).toMatchObject({ - toolCallId: "call", - success: true, - toolOutput: [{ type: "text", text: "done" }], - }); - expect(failed).toMatchObject({ - toolCallId: "call", - success: false, - error: { type: "tool_error", message: "failed" }, - }); - }); - - it("rejects malformed or unrelated capture lines rather than inventing observations", () => { - expect(transformCapturedLine("opencode", "not json")).toEqual([]); - expect(transformCapturedLine("claude", '{"type":"session.step.ended","data":{}}')).toEqual([]); - expect(transformCapturedLine("opencode", '{"type":"session.step.ended"}')).toEqual([]); - expect( - transformCapturedLine("opencode", '{"type":"session.step.ended","data":{"sessionID":"s"}}')[0] - ?.sessionId - ).toBe("s"); - }); +it("normalizes only OTLP capture envelopes", () => { + for (const [tool, payload] of [ + ["opencode", { type: "session.step.ended", data: { sessionID: "session" } }], + ["opencode", { type: "message.updated", properties: {} }], + ["opencode", { info: {}, messages: [] }], + ["claude", { type: "result", session_id: "session" }], + ["codex", { type: "turn.completed", usage: { input_tokens: 1 } }], + ] as const) { + expect(transformCapturedLine(tool, JSON.stringify(payload))).toEqual([]); + } + expect(transformCapturedLine("opencode", "not json")).toEqual([]); + expect(transformCapturedLine("opencode", JSON.stringify(log([])))).toHaveLength(1); }); describe("session and trace details", () => { - it("retains the original event including nested values and explicit nulls", () => { - const source = { - type: "session.tool.called", - data: { - id: "call", - input: { - path: "x", - enabled: true, - count: 0, - missing: null, - parts: ["a", 1, false, null, { nested: "value" }], - }, - }, - }; - const [row] = transformOpenCode(source); - expect(row?.source).toEqual(source); - expect(row?.toolInput).toEqual(source.data.input); - }); - it.each([ ["false", false], ["true", true], @@ -649,14 +440,35 @@ describe("session and trace details", () => { expect(claude?.usageScope).toBeUndefined(); expect(claude?.tokenInputIncludesCache).toBeUndefined(); expect(claude?.tokenOutputIncludesReasoning).toBeUndefined(); - const [opencode] = transformOpenCode({ type: "session.step.started", data: {} }); + const [opencode] = transformOpenCode(source); expect(opencode?.timestamp).toBeUndefined(); expect(opencode?.usageScope).toBeUndefined(); - expect(opencode?.zeroMayMeanMissing).toBeUndefined(); expect(opencode?.tokenInputIncludesCache).toBeUndefined(); expect(opencode?.tokenOutputIncludesReasoning).toBeUndefined(); }); + it("keeps provider correlation IDs from OTLP without deriving new IDs", () => { + const claude = log([ + attr("session.id", string("session")), + attr("prompt.id", string("prompt")), + attr("client_request_id", string("request")), + attr("message.uuid", string("message")), + ]); + const record = claude.payload.resourceLogs[0].scopeLogs[0].logRecords[0]; + Object.assign(record, { traceId: "trace", spanId: "span" }); + expect(transformClaude(claude)[0]).toMatchObject({ + sessionId: "session", + promptId: "prompt", + clientRequestId: "request", + messageId: "message", + traceId: "trace", + spanId: "span", + }); + const [codex] = transformCodex(log([attr("turn.id", string("turn"))])); + expect(codex?.turnId).toBe("turn"); + expect(codex?.sessionId).toBeUndefined(); + }); + it("keeps Codex session, thread, model, request and token categories on one log record", () => { const [row] = transformCodex( log([ @@ -749,45 +561,4 @@ describe("session and trace details", () => { toolOutput: "result", }); }); - - it("keeps OpenCode v1 session creation, tool errors and v2 input completion distinct", () => { - const [created] = transformOpenCode({ - type: "session.created", - properties: { info: { id: "session", version: "1.18.35", time: { created: 1700000000000 } } }, - }); - expect(created).toMatchObject({ - sessionId: "session", - toolVersion: "1.18.35", - timestamp: "2023-11-14T22:13:20.000Z", - }); - const [failed] = transformOpenCode({ - type: "message.part.updated", - properties: { - part: { - id: "part", - messageID: "message", - sessionID: "session", - type: "tool", - tool: "read", - callID: "call", - state: { status: "error", error: "failed", input: { path: "x" } }, - }, - }, - }); - expect(failed).toMatchObject({ - partId: "part", - messageId: "message", - sessionId: "session", - toolName: "read", - toolCallId: "call", - success: false, - error: "failed", - toolInput: { path: "x" }, - }); - const [input] = transformOpenCode({ - type: "session.tool.input.ended", - data: { id: "call", text: '{"path":"x"}' }, - }); - expect(input).toMatchObject({ toolCallId: "call", toolInput: '{"path":"x"}' }); - }); }); diff --git a/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts b/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts index f04defda7..da86a1536 100644 --- a/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts +++ b/cli/tests/contexts/telemetry/otel/config-edits.unit.test.ts @@ -3,8 +3,6 @@ import { describe, expect, it } from "vitest"; import { mergeCodexExporters, mergeJsonEnvironment, - mergeOpenCodeInstrumentation, - openCodeInstrumentation, rejectConflict, } from "../../../../src/contexts/telemetry/infrastructure/otel/config-edits.js"; @@ -46,44 +44,6 @@ describe("merging Claude and OpenCode settings", () => { } expect(() => rejectConflict("endpoint", "remote", "local")).toThrow(/endpoint/); }); - - it("preserves JSONC comments, quoted punctuation and trailing commas in OpenCode settings", () => { - const original = `{ - // keep punctuation in values - "model": "provider/model//name", - "experimental": {"note": "literal /* text */ and comma,}", /* keep */}, - }`; - const merged = mergeOpenCodeInstrumentation(original, true); - expect(merged).toContain("// keep punctuation in values"); - expect(merged).toContain('"provider/model//name"'); - expect(merged).toContain('"literal /* text */ and comma,}"'); - expect(merged).toContain("/* keep */"); - expect(openCodeInstrumentation(merged)).toBe(true); - expect(mergeOpenCodeInstrumentation(merged, true)).toBe(merged); - expect(openCodeInstrumentation('{"other":true}')).toBeUndefined(); - }); - - it("keeps escaped quotes and comment-like strings when inserting into JSONC", () => { - const original = - '{\n "description": "line // not a comment",\n "experimental": {\n "labels": ["comma,}", "quoted \\"value\\"",], // real comment\n },\n}'; - const merged = mergeOpenCodeInstrumentation(original, true); - expect(merged).toContain('"line // not a comment"'); - expect(merged).toContain('"quoted \\"value\\""'); - expect(merged).toContain("// real comment"); - expect(openCodeInstrumentation(merged)).toBe(true); - }); - - it("refuses disabled, duplicate, malformed or non-object OpenCode settings", () => { - for (const invalid of [ - '{"experimental":{"openTelemetry":false}}', - '{"experimental":{},"experimental":{}}', - '{"experimental":{"x":1,"x":2}}', - '{"experimental":[]}', - '{"experimental":', - ]) { - expect(() => mergeOpenCodeInstrumentation(invalid, true)).toThrow(); - } - }); }); describe("merging Codex TOML exporters", () => { diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index d15f7a8f5..d3ed68d2b 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -143,49 +143,16 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: false }); }); - it("merges the v1 instrumentation flag into the native JSONC config without touching shell profiles", async () => { + it("rejects OpenCode 1 and versions before 2.2.5", async () => { await seedToken(); - const profile = join(home, ".zshrc"); - await seed(profile, "# keep me"); - const path = join(home, ".config/opencode/opencode.jsonc"); - const original = - '{\n // keep this comment\n "experimental": { "todo": true, /* keep this too */ },\n}\n'; - await seed(path, original); - const options = { home, env: {}, opencodeVersion: async () => "1.19.5" }; - const result = await initializeOtel("opencode", "dev", options); - expect(result).toMatchObject({ path, changed: true }); - expect(result.launch).toContain("aidd telemetry otel run --config=opencode"); - const next = await readFile(path, "utf8"); - expect(next).toContain('"openTelemetry": true'); - expect(next).toContain("// keep this comment"); - expect(next).toContain('"todo": true, /* keep this too */'); - expect(next).not.toContain(TOKEN); - expect(await readFile(profile, "utf8")).toBe("# keep me"); - expect(await initializeOtel("opencode", "dev", options)).toMatchObject({ changed: false }); - expect(await readFile(path, "utf8")).toBe(next); - }); - - it("preserves a disabled v1 instrumentation flag and refuses to initialize", async () => { - await seedToken(); - const path = join(home, ".config/opencode/opencode.json"); - const original = '{"experimental":{"openTelemetry":false}}'; - await seed(path, original); - const options = { home, env: {}, opencodeVersion: async () => "1.19.5" }; - await expect(initializeOtel("opencode", "dev", options)).rejects.toThrow( - /openTelemetry.*disabled/ - ); - expect(await readFile(path, "utf8")).toBe(original); - }); - - it("initializes the locally verified v1.18.35 without admitting unverified earlier versions", async () => { - await seedToken(); - const options = { home, env: {}, opencodeVersion: async () => "1.18.35" }; - expect(await initializeOtel("opencode", "dev", options)).toMatchObject({ changed: true }); - expect( - JSON.parse(await readFile(join(home, ".config/opencode/opencode.json"), "utf8")) - ).toMatchObject({ experimental: { openTelemetry: true } }); - await expect( - initializeOtel("opencode", "dev", { ...options, opencodeVersion: async () => "1.18.34" }) - ).rejects.toThrow(/OpenCode version/); + for (const version of ["1.18.35", "1.19.5", "2.2.4", "not-a-version"]) { + await expect( + initializeOtel("opencode", "dev", { + home, + env: {}, + opencodeVersion: async () => version, + }) + ).rejects.toThrow(/Unsupported OpenCode version/); + } }); }); diff --git a/cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts deleted file mode 100644 index c3722d867..000000000 --- a/cli/tests/contexts/telemetry/otel/run-otel.integration.test.ts +++ /dev/null @@ -1,74 +0,0 @@ -import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; -import { runOtelOpenCode } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; -import type { Host } from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; - -describe.skipIf(process.platform === "win32")("OpenCode OTLP launcher", () => { - let home: string; - let bin: string; - let host: Host; - - beforeEach(async () => { - home = await mkdtemp(join(tmpdir(), "aidd-otel-launch-")); - bin = join(home, "bin"); - const collector = join(home, ".local/share/aidd-otel-observation"); - await mkdir(collector, { recursive: true }); - await mkdir(bin); - await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); - host = { - home, - env: { PATH: bin, OPENCODE_CONFIG_DIR: join(home, "config") }, - opencodeVersion: async () => "1.18.35", - }; - }); - - afterEach(async () => { - await rm(home, { recursive: true, force: true }); - }); - - it("requires an initialized v1 configuration and rejects v2", async () => { - await expect(runOtelOpenCode([], host)).rejects.toThrow(/Initialize OpenCode 1 OTLP/); - await expect( - runOtelOpenCode([], { ...host, opencodeVersion: async () => "2.2.5" }) - ).rejects.toThrow(/OpenCode 1/); - }); - - it("passes local OTLP credentials and command arguments to v1", async () => { - await initializeOtel("opencode", "dev", host); - const executable = join(bin, "opencode"); - await writeFile( - executable, - `#!/bin/sh -[ "$1" = "check" ] || exit 31 -[ "$OTEL_EXPORTER_OTLP_ENDPOINT" = "http://127.0.0.1:14318/opencode" ] || exit 32 -[ "$OTEL_EXPORTER_OTLP_HEADERS" = "Authorization=Bearer test-token" ] || exit 33 -exit 0 -` - ); - await chmod(executable, 0o700); - expect(await runOtelOpenCode(["check"], host)).toBe(0); - expect(await runOtelOpenCode(["other"], host)).toBe(31); - }); - - it("refuses conflicting inherited endpoints and a missing executable", async () => { - await initializeOtel("opencode", "dev", host); - await expect( - runOtelOpenCode([], { - ...host, - env: { ...host.env, OTEL_EXPORTER_OTLP_ENDPOINT: "https://different.example" }, - }) - ).rejects.toThrow(/Conflicting OTLP setting/); - await expect(runOtelOpenCode([], host)).rejects.toThrow(/Could not launch OpenCode/); - }); - - it("propagates a child terminated by a signal", async () => { - await initializeOtel("opencode", "dev", host); - const executable = join(bin, "opencode"); - await writeFile(executable, "#!/bin/sh\nkill -TERM $$\n"); - await chmod(executable, 0o700); - expect(await runOtelOpenCode([], host)).toBe(143); - }); -}); diff --git a/cli/tests/contexts/telemetry/otel/session-project.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-project.unit.test.ts deleted file mode 100644 index bd8702158..000000000 --- a/cli/tests/contexts/telemetry/otel/session-project.unit.test.ts +++ /dev/null @@ -1,81 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { readSessionProject } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/metadata/session-project.js"; - -describe("Codex native session project", () => { - it("uses the native association and rechecks it after reading the project", async () => { - const calls: Array<[string, object]> = []; - const result = await readSessionProject(async (method, params) => { - calls.push([method, params]); - return method === "thread/read" - ? { thread: { id: "s", sessionId: "s", projectId: "p" } } - : { project: { id: "p", name: "Native project" } }; - }, "s"); - expect(result).toEqual({ - available: true, - values: { "aidd.project_id": "p", "aidd.project_name": "Native project" }, - }); - expect(calls).toEqual([ - ["thread/read", { threadId: "s", includeTurns: false }], - ["project/read", { projectId: "p" }], - ["thread/read", { threadId: "s", includeTurns: false }], - ]); - }); - it("clears only an explicitly absent association", async () => { - expect( - await readSessionProject( - async () => ({ thread: { id: "s", sessionId: "s", projectId: null } }), - "s" - ) - ).toEqual({ available: true, values: {} }); - expect( - await readSessionProject( - async () => ({ thread: { id: "s", sessionId: "s", cwd: "/project" } }), - "s" - ) - ).toEqual({ available: false }); - }); - it.each([ - { id: "other", sessionId: "s", projectId: "p" }, - { id: "s", sessionId: "root", projectId: "p" }, - { id: "s", sessionId: "s", projectId: "" }, - { id: "s", sessionId: "s", projectId: "bad\nID" }, - { id: "s", sessionId: "s", projectId: "p".repeat(1025) }, - ])("rejects unverifiable thread identity or association %j", async (thread) => { - expect(await readSessionProject(async () => ({ thread }), "s")).toEqual({ available: false }); - }); - it.each([ - { id: "wrong", name: "Project" }, - { id: "p", name: " " }, - { id: "p", name: "bad\nname" }, - { id: "p" }, - ])("rejects a mismatched or unnamed project %j", async (project) => { - expect( - await readSessionProject( - async (method) => - method === "thread/read" - ? { thread: { id: "s", sessionId: "s", projectId: "p" } } - : { project }, - "s" - ) - ).toEqual({ available: false }); - }); - it("does not publish an association changed during the read", async () => { - let reads = 0; - expect( - await readSessionProject( - async (method) => - method === "thread/read" - ? { thread: { id: "s", sessionId: "s", projectId: ++reads === 1 ? "p" : "other" } } - : { project: { id: "p", name: "Project" } }, - "s" - ) - ).toEqual({ available: false }); - }); - it("keeps previous information on API failure", async () => { - expect( - await readSessionProject(async () => { - throw new Error("unsupported"); - }, "s") - ).toEqual({ available: false }); - }); -}); diff --git a/cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts deleted file mode 100644 index 06e6afa71..000000000 --- a/cli/tests/contexts/telemetry/otel/session-repository.unit.test.ts +++ /dev/null @@ -1,76 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { readSessionRepository } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/metadata/session-repository.js"; - -describe("Codex native session repository", () => { - it.each([ - ["git@github.com:owner/repository.git", "https://github.com/owner/repository"], - ["https://github.com/owner/repository", "https://github.com/owner/repository"], - ["git@gitlab.com:group/subgroup/repo.git", "https://gitlab.com/group/subgroup/repo"], - ["https://gitlab.com/group/repo.git", "https://gitlab.com/group/repo"], - ])("publishes only the canonical supported identity for %s", async (originUrl, expected) => { - const calls: Array<[string, object]> = []; - const result = await readSessionRepository(async (method, params) => { - calls.push([method, params]); - return { thread: { id: "s", sessionId: "s", gitInfo: { originUrl } } }; - }, "s"); - expect(result).toEqual({ available: true, values: { "aidd.repository_url": expected } }); - expect(calls).toEqual([["thread/read", { threadId: "s", includeTurns: false }]]); - }); - it.each([ - "https://user:secret@github.com/owner/repo.git", - "https://github.com/owner/repo?token=secret", - "https://github.com/owner/repo#fragment", - "https://github.com:443/owner/repo", - "https://example.com/owner/repo", - "git@example.com:owner/repo", - "ssh://git@github.com/owner/repo", - "file:///owner/repo", - "https://github.com/owner/repo%2fextra", - "https://github.com/owner/../repo", - "https://github.com/owner/.", - "https://github.com/owner/repo/extra", - "https://github.com/owner/.git", - "https://github.com/owner/repo\n", - ])( - "clears a previous link without exposing an unsupported or unsafe remote %j", - async (originUrl) => { - expect( - await readSessionRepository( - async () => ({ thread: { id: "s", sessionId: "s", gitInfo: { originUrl } } }), - "s" - ) - ).toEqual({ available: true, values: {} }); - } - ); - it("clears only an explicitly absent native Git origin", async () => { - for (const gitInfo of [null, { originUrl: null }]) - expect( - await readSessionRepository( - async () => ({ thread: { id: "s", sessionId: "s", gitInfo } }), - "s" - ) - ).toEqual({ available: true, values: {} }); - for (const gitInfo of [undefined, {}]) - expect( - await readSessionRepository( - async () => ({ thread: { id: "s", sessionId: "s", gitInfo } }), - "s" - ) - ).toEqual({ available: false }); - }); - it.each([ - { id: "other", sessionId: "s", gitInfo: { originUrl: "git@github.com:a/b.git" } }, - { id: "s", sessionId: "other", gitInfo: { originUrl: "git@github.com:a/b.git" } }, - ])("rejects an unrelated thread", async (thread) => { - expect(await readSessionRepository(async () => ({ thread }), "s")).toEqual({ - available: false, - }); - }); - it("retains existing metadata on API errors", async () => { - expect( - await readSessionRepository(async () => { - throw new Error("unavailable"); - }, "s") - ).toEqual({ available: false }); - }); -}); diff --git a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts b/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts deleted file mode 100644 index 19c1b7385..000000000 --- a/cli/tests/contexts/telemetry/otel/session-title-sync.integration.test.ts +++ /dev/null @@ -1,420 +0,0 @@ -import { - appendFile, - chmod, - mkdir, - mkdtemp, - readdir, - readFile, - rm, - writeFile, -} from "node:fs/promises"; -import { createServer } from "node:http"; -import { tmpdir } from "node:os"; -import { delimiter, join } from "node:path"; -import { expect, it } from "vitest"; -import { syncOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; -import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; -import { syncAllSessionTitles } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all-titles.js"; - -const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); -function logAttributes(body: string): ReturnType<typeof attr>[] { - const envelope = JSON.parse(body) as { - resourceLogs: Array<{ - scopeLogs: Array<{ logRecords: Array<{ attributes: ReturnType<typeof attr>[] }> }>; - }>; - }; - return envelope.resourceLogs.flatMap((resource) => - resource.scopeLogs.flatMap((scope) => scope.logRecords.flatMap((record) => record.attributes)) - ); -} - -it.skipIf(process.platform === "win32")( - "sends titles only when opted in and the session is consented", - async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-title-sync-")); - const oldPath = process.env.PATH; - const oldMarker = process.env.AIDD_TEST_TITLE_MARKER; - const marker = join(home, "called"); - const received: string[] = []; - let onReceived = (_body: string) => {}; - let responseFor = (_body: string) => 200; - const server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - onReceived(body); - response.writeHead(responseFor(body)).end("{}"); - }); - }); - try { - const bin = join(home, "bin"); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(bin); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const fake = join(bin, "codex"); - await writeFile( - fake, - `#!${process.execPath}\n` + - String.raw` -const fs = require('node:fs'); -const readline = require('node:readline'); -fs.writeFileSync(process.env.AIDD_TEST_TITLE_MARKER, 'called'); -readline.createInterface({ input: process.stdin }).on('line', (line) => { - const message = JSON.parse(line); - if (!message.id) return; - if (message.method === 'initialize') fs.writeFileSync(process.env.AIDD_TEST_TITLE_MARKER, JSON.stringify(message.params)); - const result = message.method === 'initialize' ? {} : message.method === 'project/read' ? { project: { id: 'native-project', name: 'Native project' } } : { - thread: { id: message.params.threadId, sessionId: message.params.threadId, - name: 'Synthetic title', projectId: 'native-project', - gitInfo: { originUrl: 'git@github.com:example/repository.git' }, turns: [] } - }; - process.stdout.write(JSON.stringify({ id: message.id, result }) + '\n'); -}); -` - ); - await chmod(fake, 0o700); - process.env.PATH = `${bin}${delimiter}${oldPath ?? ""}`; - process.env.AIDD_TEST_TITLE_MARKER = marker; - const capture = { - signal: "logs", - payload: { - resourceLogs: [ - { - scopeLogs: [ - { - logRecords: [ - { - timeUnixNano: "1793763900000000000", - attributes: [ - attr("conversation.id", "session-1"), - attr("user.email", "person@example.test"), - attr("event.name", "codex.sse_event"), - attr("event.kind", "response.completed"), - attr("input_token_count", "3"), - attr("output_token_count", "1"), - ], - }, - ], - }, - ], - }, - ], - }, - }; - await writeFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`, { - mode: 0o600, - }); - await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - session: "session-1", - tenant: "test", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test", - home, - follow: false, - }; - const resolve = async () => ({ state: "unbound" as const }); - - await syncOtel(options, resolve, async () => true); - await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); - expect(received).toHaveLength(1); - - received.length = 0; - await syncOtel( - { ...options, includeSessionTitles: true }, - resolve, - async (transfer) => Object.keys(transfer.payload).length > 0 - ); - await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); - expect(received).toHaveLength(1); - - received.length = 0; - await syncOtel( - { ...options, includeSessionTitles: true, titlesOnly: true }, - resolve, - async () => false - ); - await expect(readFile(marker, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); - expect(received).toHaveLength(0); - - received.length = 0; - await syncOtel({ ...options, includeSessionTitles: true }, resolve, async () => true); - expect(received).toHaveLength(2); - expect(received[1]).toContain("aidd.session_title"); - expect(received[1]).toContain("Synthetic title"); - expect(await readFile(marker, "utf8")).not.toContain("experimentalApi"); - expect(received[1]).not.toContain("aidd.feature"); - - received.length = 0; - await syncOtel( - { ...options, includeSessionTitles: true, titlesOnly: true }, - async () => ({ state: "confirmed", feature: "owner/repo#42", branch: "feat/demo" }), - async () => true - ); - expect(received).toHaveLength(1); - expect(received[0]).toContain("Synthetic title"); - expect(received[0]).not.toContain("input_token_count"); - expect(received[0]).not.toContain("aidd.feature"); - - received.length = 0; - await syncOtel( - { ...options, includeSessionTitles: true, titlesOnly: true }, - resolve, - async (transfer) => !JSON.stringify(transfer.payload).includes("aidd.session_title") - ); - expect(received).toHaveLength(0); - - received.length = 0; - await syncOtel({ ...options, includeSessionProjects: true }, resolve, async () => true); - expect(received).toHaveLength(2); - expect(received[1]).toContain("aidd.session_project"); - expect(received[1]).toContain("aidd.project_id"); - expect(received[1]).toContain("Native project"); - expect(await readFile(marker, "utf8")).toContain('"experimentalApi":true'); - expect(received[1]).not.toContain("Synthetic title"); - expect(received[1]).not.toContain("input_token_count"); - received.length = 0; - await syncOtel( - { ...options, includeSessionProjects: true }, - resolve, - async (transfer) => !JSON.stringify(transfer.payload).includes("aidd.session_project") - ); - expect(received).toHaveLength(1); - expect(received.join()).not.toContain("aidd.project_name"); - - received.length = 0; - await syncOtel({ ...options, includeSessionRepositories: true }, resolve, async () => true); - expect(received).toHaveLength(2); - expect(received[1]).toContain("aidd.session_repository"); - expect( - logAttributes(received[1]).filter((entry) => entry.key === "aidd.repository_url") - ).toEqual([attr("aidd.repository_url", "https://github.com/example/repository")]); - expect(received[1]).not.toContain("git@github.com"); - expect(received[1]).not.toContain("Synthetic title"); - received.length = 0; - await syncOtel( - { ...options, includeSessionRepositories: true }, - resolve, - async (transfer) => !JSON.stringify(transfer.payload).includes("aidd.session_repository") - ); - expect(received).toHaveLength(1); - expect(received.join()).not.toContain("aidd.repository_url"); - - received.length = 0; - const captureFile = join(captureDir, "codex.jsonl"); - await appendFile(captureFile, `${JSON.stringify(capture)}\n`.repeat(10)); - await appendFile( - captureFile, - `${JSON.stringify(capture).replaceAll("session-1", "session-2")}\n` - ); - await appendFile( - captureFile, - `${JSON.stringify(capture).replaceAll("session-1", "session-4")}\n` - ); - const original = capture.payload.resourceLogs[0].scopeLogs[0].logRecords[0]; - const attrs = original.attributes.map((a) => - a.key === "conversation.id" ? attr(a.key, "session-3") : a - ); - const denied = { ...original, timeUnixNano: "1793763899000000000", attributes: attrs }; - const admitted = { ...original, attributes: attrs.filter((a) => a.key !== "user.email") }; - const incomplete = { - signal: "logs", - payload: { resourceLogs: [{ scopeLogs: [{ logRecords: [denied, admitted] }] }] }, - }; - await appendFile(captureFile, `${JSON.stringify(incomplete)}\n`); - const admissions = new Map([ - [ - "session-1", - { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "allow" }, - ], - [ - "session-2", - { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "deny" }, - ], - [ - "session-3", - { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "mixed" }, - ], - [ - "session-4", - { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "allow-2" }, - ], - ]); - const deniedTime = new Date(1_793_763_899_000).toISOString(); - let allowedChecks = 0; - const result = await syncAllSessionTitles( - { ...options, includeSessionTitles: true, stateDir: join(home, "state") }, - async () => admissions, - async (admission, at) => { - if (admission.repositoryId === "allow") allowedChecks++; - return admission.repositoryId === "deny" || - (admission.repositoryId === "mixed" && at === deniedTime) - ? "deny" - : "allow"; - } - ); - expect(result.sent).toBe(2); - expect(result.identified).toBe(2); - expect(result.currentConsenting).toBe(2); - expect(allowedChecks).toBe(4); - expect(received).toHaveLength(2); - expect(received.some((body) => body.includes("session-1"))).toBe(true); - expect(received.some((body) => body.includes("session-4"))).toBe(true); - expect(received.join()).not.toContain("session-2"); - expect(received.join()).not.toContain("session-3"); - expect(received.join()).not.toContain("input_token_count"); - expect(received.join()).not.toContain("aidd.feature"); - await expect(readdir(join(home, "state"))).rejects.toMatchObject({ code: "ENOENT" }); - - received.length = 0; - const metadata = await syncAllSessionTitles( - { - ...options, - includeSessionTitles: true, - includeSessionProjects: true, - includeSessionRepositories: true, - stateDir: join(home, "state"), - }, - async () => admissions, - async (admission, at) => - admission.repositoryId === "deny" || - (admission.repositoryId === "mixed" && at === deniedTime) - ? "deny" - : "allow" - ); - expect(metadata.sent).toBe(6); - expect(received.filter((body) => body.includes("aidd.session_project"))).toHaveLength(2); - expect(received.filter((body) => body.includes("aidd.session_repository"))).toHaveLength(2); - expect(received.join()).not.toContain("input_token_count"); - expect(received.join()).not.toContain("aidd.feature"); - await expect(readdir(join(home, "state"))).rejects.toMatchObject({ code: "ENOENT" }); - - received.length = 0; - let checks = 0; - const admittedSession = admissions.get("session-1"); - if (!admittedSession) throw new Error("Missing test admission."); - const withdrawn = await syncAllSessionTitles( - { ...options, includeSessionTitles: true }, - async () => new Map([["session-1", admittedSession]]), - async () => (++checks < 3 ? "allow" : "deny") - ); - expect(withdrawn.sent).toBe(0); - expect(withdrawn.currentConsenting).toBe(0); - expect(received).toHaveLength(0); - - const noSourceConsent = await syncAllSessionTitles( - { ...options, includeSessionTitles: true }, - async () => new Map([["session-1", admittedSession]]), - async () => "deny" - ); - expect(noSourceConsent).toMatchObject({ sent: 0, identified: 0, currentConsenting: 0 }); - - const captureTime = new Date(1_793_763_900_000).toISOString(); - const noCurrentConsent = await syncAllSessionTitles( - { ...options, includeSessionTitles: true }, - async () => new Map([["session-1", admittedSession]]), - async (_admission, at) => (at === captureTime ? "allow" : "deny") - ); - expect(noCurrentConsent).toMatchObject({ sent: 0, identified: 1, currentConsenting: 0 }); - - received.length = 0; - await appendFile( - captureFile, - `${JSON.stringify(capture).replaceAll("person@example.test", "PERSON@EXAMPLE.TEST")}\n` - ); - const conflictingIdentity = await syncAllSessionTitles( - { ...options, includeSessionTitles: true }, - async () => new Map([["session-1", admittedSession]]), - async () => "allow" - ); - expect(conflictingIdentity).toMatchObject({ sent: 0, identified: 0 }); - expect(received).toHaveLength(0); - - received.length = 0; - const abort = new AbortController(); - const timeout = setTimeout(() => abort.abort(), 3000); - onReceived = (body) => { - if (body.includes("aidd.session_repository")) abort.abort(); - }; - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const running = syncAllOtel( - { - ...options, - stateDir: join(home, "state"), - includeSessionTitles: true, - includeSessionRepositories: true, - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["session-1", admission]]), - async () => "allow" - ); - await new Promise((done) => setTimeout(done, 100)); - await appendFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`); - await running; - clearTimeout(timeout); - expect(received.some((body) => body.includes("Synthetic title"))).toBe(true); - const receivedAttributes = received.flatMap(logAttributes); - expect(receivedAttributes.filter((entry) => entry.key === "aidd.repository_url")).toEqual([ - attr("aidd.repository_url", "https://github.com/example/repository"), - ]); - - received.length = 0; - let consenting = true; - await writeFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`, { - mode: 0o600, - }); - onReceived = () => {}; - responseFor = (body) => { - if (!body.includes("aidd.session_title")) return 200; - consenting = false; - return 503; - }; - await expect( - syncOtel( - { ...options, follow: true, intervalMs: 10, includeSessionTitles: true }, - resolve, - async () => consenting - ) - ).rejects.toThrow(/Consent is no longer confirmed/); - expect(received.filter((body) => body.includes("aidd.session_title"))).toHaveLength(1); - - received.length = 0; - consenting = true; - const retryAbort = new AbortController(); - const retryAll = syncAllOtel( - { - ...options, - stateDir: join(home, "withdrawn-state"), - includeSessionTitles: true, - intervalMs: 10, - signal: retryAbort.signal, - }, - async () => new Map([["session-1", admission]]), - async () => (consenting ? "allow" : "deny") - ); - await new Promise((done) => setTimeout(done, 100)); - await appendFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture)}\n`); - await new Promise((done) => setTimeout(done, 100)); - retryAbort.abort(); - await retryAll; - expect(received.filter((body) => body.includes("aidd.session_title"))).toHaveLength(1); - } finally { - process.env.PATH = oldPath; - if (oldMarker === undefined) delete process.env.AIDD_TEST_TITLE_MARKER; - else process.env.AIDD_TEST_TITLE_MARKER = oldMarker; - await new Promise<void>((resolve) => server.close(() => resolve())); - await rm(home, { recursive: true, force: true }); - } - } -); diff --git a/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts b/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts deleted file mode 100644 index 04f52413a..000000000 --- a/cli/tests/contexts/telemetry/otel/session-title.unit.test.ts +++ /dev/null @@ -1,156 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { - createSessionTitlePublisher, - readSessionTitle, - sessionTitleTransfer, -} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/metadata/session-title.js"; - -describe("Codex session titles", () => { - it("dates a title when it is observed, not before a delayed read", async () => { - let now = 100_000; - const reader = { - read: async () => { - now = 103_000; - return { available: true as const, title: "New title" }; - }, - close: () => {}, - }; - const publisher = createSessionTitlePublisher("/unused", { - now: () => now, - createReader: () => reader, - }); - let time: string | undefined; - await publisher.publish( - "s1", - { values: new Map([["user.email", "person@example.test"]]), conflict: false }, - async (event) => { - time = event.at; - return true; - } - ); - expect(time).toBe(new Date(103_000).toISOString()); - publisher.close(); - }); - - it("backs off globally after an app-server failure instead of blocking every session", async () => { - let now = 100_000; - let starts = 0; - let reads = 0; - let warnings = 0; - const publisher = createSessionTitlePublisher("/unused", { - now: () => now, - createReader: () => { - starts++; - return { - unavailable: true, - read: async () => { - reads++; - now += 5_000; - return { available: false as const }; - }, - close: () => {}, - }; - }, - onUnavailable: () => { - warnings++; - }, - }); - const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; - const send = async () => true; - expect(await publisher.publish("s1", identity, send)).toBe(false); - expect(await publisher.publish("s2", identity, send)).toBe(false); - expect(await publisher.publish("s3", identity, send)).toBe(false); - expect({ starts, reads, warnings }).toEqual({ starts: 1, reads: 1, warnings: 1 }); - now += 60_000; - expect(await publisher.publish("s3", identity, send)).toBe(false); - expect({ starts, reads, warnings }).toEqual({ starts: 2, reads: 2, warnings: 1 }); - publisher.close(); - }); - - it("does not mark a title delivered when consent is withdrawn during the read", async () => { - let now = 100_000; - const publisher = createSessionTitlePublisher("/unused", { - now: () => now, - createReader: () => ({ - read: async () => ({ available: true, title: "Title" }), - close: () => {}, - }), - }); - const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; - expect(await publisher.publish("s1", identity, async () => false)).toBe(false); - now += 60_000; - expect(await publisher.publish("s1", identity, async () => true)).toBe(true); - publisher.close(); - }); - - it("reads metadata only and requires exact thread and session IDs", async () => { - const requests: Array<{ method: string; params: object }> = []; - const read = async (method: string, params: object) => { - requests.push({ method, params }); - return { thread: { id: "s1", sessionId: "s1", name: "A title", turns: [] } }; - }; - expect(await readSessionTitle(read, "s1")).toEqual({ available: true, title: "A title" }); - expect(requests).toEqual([ - { method: "thread/read", params: { threadId: "s1", includeTurns: false } }, - ]); - expect( - await readSessionTitle( - async () => ({ thread: { id: "s1", sessionId: "root", name: "x" } }), - "s1" - ) - ).toEqual({ available: false }); - }); - - it("sends an absent title explicitly without inventing a feature", () => { - const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; - const event = sessionTitleTransfer("s1", identity, null, new Date("2026-11-05T00:00:00Z")); - expect(event?.signal).toBe("logs"); - const attrs = event?.payload.resourceLogs; - expect(JSON.stringify(attrs)).toContain("aidd.session_title"); - expect(JSON.stringify(attrs)).not.toContain("aidd.feature"); - expect(sessionTitleTransfer("s1", { ...identity, conflict: true }, "x", new Date())).toBeNull(); - }); - - it("sends title changes and removal once, and keeps the last known value on read failure", async () => { - let now = 100_000; - let warnings = 0; - let current: { available: true; title: string | null } | { available: false } = { - available: true, - title: "First", - }; - const sent: string[] = []; - const reader = { read: async () => current, close: () => {} }; - const publisher = createSessionTitlePublisher("/unused", { - now: () => now, - createReader: () => reader, - onUnavailable: () => { - warnings++; - }, - }); - const identity = { values: new Map([["user.email", "person@example.test"]]), conflict: false }; - const send = async (event: { payload: object }) => { - sent.push(JSON.stringify(event.payload)); - return true; - }; - expect(await publisher.publish("s1", identity, send)).toBe(true); - expect(await publisher.publish("s1", identity, send)).toBe(false); - now += 60_000; - current = { available: false }; - expect(await publisher.publish("s1", identity, send)).toBe(false); - expect(warnings).toBe(1); - now += 60_000; - expect(await publisher.publish("s1", identity, send)).toBe(false); - expect(warnings).toBe(1); - now += 60_000; - current = { available: true, title: "Second" }; - expect(await publisher.publish("s1", identity, send)).toBe(true); - now += 60_000; - current = { available: true, title: null }; - expect(await publisher.publish("s1", identity, send)).toBe(true); - expect(sent).toHaveLength(3); - expect(sent[0]).toContain("First"); - expect(sent[1]).toContain("Second"); - expect(sent[2]).not.toContain('"key":"aidd.session_title"'); - publisher.close(); - }); -}); diff --git a/cli/tests/contexts/telemetry/otel/settings.integration.test.ts b/cli/tests/contexts/telemetry/otel/settings.integration.test.ts index cbcddb5f4..23339b119 100644 --- a/cli/tests/contexts/telemetry/otel/settings.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/settings.integration.test.ts @@ -3,10 +3,7 @@ import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; -import { - openCodeVersion, - readToken, -} from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; +import { readToken } from "../../../../src/contexts/telemetry/infrastructure/otel/settings.js"; let home: string; const tokenPath = () => join(home, ".local/share/aidd-otel-observation/collector-token"); @@ -18,41 +15,6 @@ afterEach(async () => { await rm(home, { recursive: true, force: true }); }); -describe("OpenCode version detection", () => { - it.each([ - ["1.18.35", "1"], - ["1.19.5", "1"], - ["1.20.0", "1"], - ["opencode v2.2.5", "2"], - ["2.3.0", "2"], - ])("accepts verified release %s", async (version, expected) => { - expect(await openCodeVersion({ home, env: {}, opencodeVersion: async () => version })).toBe( - expected - ); - }); - - it.each(["1.18.34", "1.19.4", "2.2.4", "2.1.99", "3.0.0", "not-a-version", "1.18.35 extra"])( - "rejects unsupported release %s", - async (version) => { - await expect( - openCodeVersion({ home, env: {}, opencodeVersion: async () => version }) - ).rejects.toThrow(/Unsupported OpenCode version/); - } - ); - - it("explains when the executable cannot be queried", async () => { - await expect( - openCodeVersion({ - home, - env: {}, - opencodeVersion: async () => { - throw new Error("missing"); - }, - }) - ).rejects.toThrow(/Cannot determine OpenCode version/); - }); -}); - describe.skipIf(process.platform === "win32")("collector credential and exporter conflicts", () => { it("reads a valid private token and rejects missing, malformed or public tokens", async () => { await expect(readToken(home)).rejects.toThrow(/private collector token/); diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts index f065d04a9..c06989243 100644 --- a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts @@ -147,81 +147,6 @@ it("forwards token observations for a consented session whose repository is unce expect(received[0]).not.toContain("aidd.session_feature"); }); -it("invalidates an in-memory binding after a repository change before forwarding new tokens", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-origin-change-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - let changed = false; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", async () => { - received.push(body); - if (body.includes("aidd.session_feature_invalidated")) { - await appendFile(capture, line("allowed")); - } else if (body.includes("aidd.session_feature")) { - changed = true; - await appendFile(capture, line("allowed")); - } - response.writeHead(200).end("{}"); - if ( - changed && - !body.includes("aidd.session_feature") && - received.some((item) => item.includes("aidd.session_feature_invalidated")) - ) - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const known = { - feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "work" }, - cwd: "/project", - repositoryId: "repo-a", - }; - const uncertain = { - feature: { state: "ambiguous" as const }, - cwd: "/project", - repositoryId: null, - }; - const timeout = setTimeout(() => abort.abort(), 3000); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - refreshMs: 20, - }, - async () => new Map([["allowed", changed ? uncertain : known]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - const result = await running; - clearTimeout(timeout); - const invalidation = received.findIndex((body) => - body.includes("aidd.session_feature_invalidated") - ); - expect(invalidation).toBeGreaterThan(0); - expect(received.slice(invalidation + 1).some((body) => body.includes("input_token_count"))).toBe( - true - ); - expect(result.pending).toBe(0); -}); - it("retries a denial caused by an admission changing during the consent check", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-stale-admission-")); dirs.push(home); @@ -838,71 +763,6 @@ it("isolates a revoked session across restarts while another session continues", expect(accepted).toEqual(["two", "two", "one"]); }); -it("defers one revoked binding without blocking another and restores a proven identity after restart", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-two-bindings-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let consenting = true; - let failOne = true; - let abort = new AbortController(); - const bindings: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - if (!body.includes("aidd.session_feature")) return void response.writeHead(200).end("{}"); - const id = body.includes('"stringValue":"one"') ? "one" : "two"; - bindings.push(id); - if (id === "one" && failOne) { - failOne = false; - consenting = false; - response.writeHead(503).end("{}"); - } else { - response.writeHead(200).end("{}"); - abort.abort(); - } - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - refreshMs: 20, - }; - const feature = { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }; - const one = { feature, cwd: "/one", repositoryId: "one-repo" }; - const two = { feature, cwd: "/two", repositoryId: "two-repo" }; - const snapshot = async () => - new Map([ - ["one", one], - ["two", two], - ]); - const permits = async (admission: { repositoryId: string | null }) => - admission.repositoryId === "one-repo" && !consenting ? ("deny" as const) : ("allow" as const); - const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("one") + line("two")); - await first; - expect(bindings).toEqual(["one", "two"]); - consenting = true; - abort = new AbortController(); - setTimeout(() => abort.abort(), 150); - await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - expect(bindings).toEqual(["one", "two", "one", "two"]); -}); - it.each(["missing", "cwd", "repository", "unavailable"] as const)( "does not retry if the session admission is %s after a failed post", async (change) => { @@ -967,194 +827,6 @@ it.each(["missing", "cwd", "repository", "unavailable"] as const)( } ); -it("does not use an outside-consent identity for a later feature binding", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-unconsented-all-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let calls = 0; - server = createServer((request, response) => { - calls++; - request.resume(); - response.writeHead(200).end("{}"); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const abort = new AbortController(); - const admission = { - feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }, - cwd: "/project", - repositoryId: "repo", - }; - const historical = new Date(1_793_763_900_000).toISOString(); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - signal: abort.signal, - }, - async () => new Map([["allowed", admission]]), - async (_admission, at) => (at === historical ? "deny" : "allow") - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 100)); - abort.abort(); - const result = await running; - expect(result).toMatchObject({ sent: 0, denied: 1 }); - expect(calls).toBe(0); -}); - -it("does not reuse legacy checkpoint identity until a new admitted observation supplies it", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-legacy-identity-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - let abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - if (body.includes("aidd.session_feature")) abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }, - cwd: "/project", - repositoryId: "repo", - }; - const snapshot = async () => new Map([["allowed", admission]]); - const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await first; - expect(received.some((body) => body.includes("aidd.session_feature"))).toBe(true); - const directory = join(home, "state", "otel-sync"); - const stateFile = (await readdir(directory)).find( - (name) => name.endsWith(".json") && !name.includes("checkpoint") - ); - if (!stateFile) throw new Error("Missing binding state"); - const path = join(directory, stateFile); - const legacy = JSON.parse(await readFile(path, "utf8")); - delete legacy.identityConsentChecked; - delete legacy.identityProof; - await writeFile(path, JSON.stringify(legacy), { mode: 0o600 }); - received.length = 0; - abort = new AbortController(); - const restart = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); - await new Promise((resolve) => setTimeout(resolve, 100)); - expect(received).toHaveLength(0); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 100)); - abort.abort(); - await restart; - expect(received.filter((body) => body.includes("aidd.session_feature"))).toHaveLength(0); - expect(received.some((body) => body.includes("developer@example.test"))).toBe(true); -}); - -it("does not reuse an identity when a clone at the same path and repository is replaced", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-replaced-clone-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - refreshMs: 20, - }; - let changed = false; - const oldAt = new Date(1_793_763_900_000).toISOString(); - const snapshot = async () => - new Map([ - [ - "allowed", - { - feature: changed - ? { state: "unbound" as const } - : { state: "confirmed" as const, feature: "owner/repo#7", branch: "feature/work" }, - cwd: "/project", - repositoryId: "repo", - }, - ], - ]); - const permits = async (_admission: unknown, at: string, _start?: string, current?: string) => - changed && current && at === oldAt ? ("deny" as const) : ("allow" as const); - const abort = new AbortController(); - const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - for (let i = 0; i < 100 && !received.some((body) => body.includes('aidd.session_feature"')); i++) - await new Promise((resolve) => setTimeout(resolve, 20)); - expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); - changed = true; - const replacement = JSON.parse( - line("allowed").replace("developer@example.test", "replacement@example.test") - ); - replacement.payload.resourceLogs[0].scopeLogs[0].logRecords[0].timeUnixNano = - "1793763901000000000"; - await appendFile(capture, `${JSON.stringify(replacement)}\n`); - await new Promise((resolve) => setTimeout(resolve, 150)); - abort.abort(); - await first; - expect(received.some((body) => body.includes("replacement@example.test"))).toBe(true); - expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(false); - const beforeRestart = received.length; - const restartAbort = new AbortController(); - const restart = syncAllOtel({ ...options, signal: restartAbort.signal }, snapshot, permits); - await new Promise((resolve) => setTimeout(resolve, 120)); - restartAbort.abort(); - await restart; - expect(received).toHaveLength(beforeRestart); -}); - it("does not send an event outside the clone's consent interval", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-consent-")); dirs.push(home); @@ -1294,293 +966,7 @@ it("recovers an interrupted pending append without losing either session on the ).toBe(true); }); -it("invalidates a v2 binding after restart only for the original consented clone", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-v2-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, - cwd: "/project", - repositoryId: "repo", - }; - const uncertain = { - feature: { state: "ambiguous" as const }, - cwd: "/project", - repositoryId: null, - }; - const firstAbort = new AbortController(); - const first = syncAllOtel( - { ...options, signal: firstAbort.signal }, - async () => new Map([["allowed", admission]]), - async () => "allow", - async () => "a".repeat(64) - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 120)); - firstAbort.abort(); - await first; - expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); - const secondAbort = new AbortController(); - const second = syncAllOtel( - { ...options, signal: secondAbort.signal }, - async () => new Map([["allowed", uncertain]]), - async () => "allow", - async () => "b".repeat(64) - ); - await new Promise((resolve) => setTimeout(resolve, 100)); - secondAbort.abort(); - await second; - expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(false); - const thirdAbort = new AbortController(); - const third = syncAllOtel( - { ...options, signal: thirdAbort.signal }, - async () => new Map([["allowed", uncertain]]), - async () => "allow", - async () => "a".repeat(64) - ); - await new Promise((resolve) => setTimeout(resolve, 100)); - thirdAbort.abort(); - await third; - expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(true); -}); - -it("invalidates a persisted feature after restart when the session becomes ambiguous", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-restart-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, - cwd: "/project", - repositoryId: "repo", - }; - const firstAbort = new AbortController(); - const first = syncAllOtel( - { ...options, signal: firstAbort.signal }, - async () => new Map([["allowed", admission]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 120)); - firstAbort.abort(); - await first; - expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); - const secondAbort = new AbortController(); - const second = syncAllOtel( - { ...options, signal: secondAbort.signal }, - async () => new Map([["allowed", { ...admission, feature: { state: "ambiguous" as const } }]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 120)); - secondAbort.abort(); - await second; - expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(true); -}); - -it("retries a feature link rejected after its intent was persisted", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-binding-retry-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let reject = true; - let links = 0; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - const isLink = body.includes('aidd.session_feature"'); - if (isLink) links++; - response.writeHead(isLink && reject ? 403 : 200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, - cwd: "/project", - repositoryId: "repo", - }; - const snapshot = async () => new Map([["allowed", admission]]); - const first = syncAllOtel(options, snapshot, async () => "allow"); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await expect(first).rejects.toThrow("HTTP 403"); - reject = false; - const abort = new AbortController(); - const second = syncAllOtel({ ...options, signal: abort.signal }, snapshot, async () => "allow"); - await new Promise((resolve) => setTimeout(resolve, 120)); - abort.abort(); - await second; - expect(links).toBe(2); -}); - -it("revalidates every identity source before resuming a conflicted binding", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-conflict-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let rejectInvalidation = true; - let invalidations = 0; - let linked = false; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - const invalidation = body.includes("aidd.session_feature_invalidated"); - if (invalidation) invalidations++; - if (body.includes('aidd.session_feature"')) linked = true; - response.writeHead(invalidation && rejectInvalidation ? 403 : 200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, - cwd: "/project", - repositoryId: "repo", - }; - const snapshot = async () => new Map([["allowed", admission]]); - const first = syncAllOtel(options, snapshot, async () => "allow"); - await new Promise((resolve) => setTimeout(resolve, 80)); - const initial = JSON.parse(line("allowed")); - const original = initial.payload.resourceLogs[0].scopeLogs[0].logRecords[0]; - const second = structuredClone(original); - second.timeUnixNano = "1793763900001000000"; - second.attributes = second.attributes.filter( - (item: { key: string }) => item.key !== "user.email" - ); - second.attributes.push(attribute("user.account_id", "account-1")); - const third = structuredClone(original); - third.timeUnixNano = "1793763900002000000"; - third.attributes = third.attributes.filter((item: { key: string }) => item.key !== "user.email"); - third.attributes.push(attribute("organization.id", "org-1")); - initial.payload.resourceLogs[0].scopeLogs[0].logRecords.push(second, third); - await appendFile(capture, `${JSON.stringify(initial)}\n`); - for (let i = 0; i < 100 && !linked; i++) await new Promise((resolve) => setTimeout(resolve, 20)); - expect(linked).toBe(true); - await appendFile( - capture, - line("allowed").replace("developer@example.test", "other@example.test") - ); - await expect(first).rejects.toThrow("HTTP 403"); - const directory = join(home, "state", "otel-sync"); - const stateName = (await readdir(directory)).find( - (name) => name.endsWith(".json") && !name.includes("checkpoint") - ); - if (!stateName) throw new Error("Missing binding state"); - const saved = JSON.parse(await readFile(join(directory, stateName), "utf8")); - expect(saved.identityProof.sources.map((source: { keys: string[] }) => source.keys)).toEqual([ - ["user.email"], - ["user.account_id"], - ["organization.id"], - [], - ]); - rejectInvalidation = false; - const abort = new AbortController(); - const accountAt = new Date(1_793_763_900_001).toISOString(); - const blocked = syncAllOtel( - { ...options, signal: abort.signal }, - snapshot, - async (_admission, at, _start, current) => (current && at === accountAt ? "deny" : "allow") - ); - await new Promise((resolve) => setTimeout(resolve, 120)); - abort.abort(); - await blocked; - expect(invalidations).toBe(1); - const retryAbort = new AbortController(); - const retry = syncAllOtel( - { ...options, signal: retryAbort.signal }, - snapshot, - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 120)); - retryAbort.abort(); - await retry; - expect(invalidations).toBe(2); -}); - -it("does not link or invalidate when identity conflicts before the first link, even after restart", async () => { +it("only sends provider events for a confirmed feature, including after restart", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-first-conflict-")); dirs.push(home); const captureDir = join(home, ".local/share/aidd-otel-observation"); @@ -1623,12 +1009,7 @@ it("does not link or invalidate when identity conflicts before the first link, e async () => "allow" ); await new Promise((resolve) => setTimeout(resolve, 80)); - const combined = JSON.parse(line("allowed")); - combined.payload.resourceLogs[0].scopeLogs[0].logRecords.push( - JSON.parse(line("allowed").replace("developer@example.test", "other@example.test")).payload - .resourceLogs[0].scopeLogs[0].logRecords[0] - ); - await appendFile(capture, `${JSON.stringify(combined)}\n`); + await appendFile(capture, line("allowed")); await new Promise((resolve) => setTimeout(resolve, 120)); firstAbort.abort(); await first; @@ -1643,6 +1024,6 @@ it("does not link or invalidate when identity conflicts before the first link, e await new Promise((resolve) => setTimeout(resolve, 120)); secondAbort.abort(); await second; - expect(received.length).toBeGreaterThan(0); - expect(received.some((body) => body.includes("aidd.session_feature"))).toBe(false); + expect(received).toHaveLength(2); + expect(received.every((body) => body.includes("codex.sse_event"))).toBe(true); }); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts index ea9a23b38..6a117301b 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -70,14 +70,13 @@ test("keeps a valid string log body as the event-name fallback", () => { expect( selectTransfers( capture([text("session.id", "session")], [], [], { stringValue: "tool_result" }), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } + { ...options, config: "claude" } ) ).toHaveLength(1); }); function select(line: string) { - return selectTransfers(line, options, { values: new Map(), conflict: false }); + return selectTransfers(line, options); } test("forwards each Codex effort only on the event that reported it", () => { @@ -200,8 +199,7 @@ function metricCapture( test("does not forward Codex effort fields on Claude metrics", () => { const [transfer] = selectTransfers( metricCapture([text("type", "input"), text("reasoning_effort", "high")]), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } + { ...options, config: "claude" } ); expect(JSON.stringify(transfer.payload)).not.toContain("reasoning_effort"); }); @@ -209,10 +207,7 @@ test("does not forward Codex effort fields on Claude metrics", () => { test("forwards Claude effort only when the token point explicitly reports it", () => { const claude = { ...options, config: "claude" as const }; const point = [text("type", "input"), text("model", "claude-test")]; - const [reported] = selectTransfers(metricCapture([...point, text("effort", "medium")]), claude, { - values: new Map(), - conflict: false, - }); + const [reported] = selectTransfers(metricCapture([...point, text("effort", "medium")]), claude); expect(JSON.stringify(reported.payload)).toContain( '"key":"effort","value":{"stringValue":"medium"}' ); @@ -226,7 +221,7 @@ test("forwards Claude effort only when the token point explicitly reports it", ( "claude_code.cost.usage" ), ]) { - const [transfer] = selectTransfers(invalid, claude, { values: new Map(), conflict: false }); + const [transfer] = selectTransfers(invalid, claude); expect(JSON.stringify(transfer.payload)).not.toContain('"key":"effort"'); } }); @@ -240,8 +235,7 @@ test.each(["point", "resource", "scope"])("rejects contradictory Claude %s attri level === "resource" ? conflict : [], level === "scope" ? conflict : [] ), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } + { ...options, config: "claude" } ) ).toThrow("Conflicting duplicate OTLP attributes."); }); @@ -250,8 +244,7 @@ test("rejects contradictory Claude resource and point identities", () => { expect(() => selectTransfers( metricCapture([text("user.email", "private-b")], [text("user.email", "private-a")]), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } + { ...options, config: "claude" } ) ).toThrow("Conflicting OTLP identity attributes."); }); @@ -327,11 +320,7 @@ test.each(["event.name", "event.kind", "session.id", "conversation.id"])( JSON.stringify(text(key, "")) ); expect(() => - selectTransfers( - body, - { ...options, config: key === "session.id" ? "claude" : "codex" }, - { values: new Map(), conflict: false } - ) + selectTransfers(body, { ...options, config: key === "session.id" ? "claude" : "codex" }) ).toThrow(); } ); @@ -412,13 +401,6 @@ test.each(["user.email", "user.account_id", "organization.id"])( } ); -test("marks case-distinct session identities as conflicting across observations", () => { - const identity = { values: new Map([["user.account_id", "abc"]]), conflict: false }; - selectTransfers(capture([text("user.account_id", "ABC")]), options, identity); - expect(identity.conflict).toBe(true); - expect(identity.values.get("user.account_id")).toBe("abc"); -}); - const dirs: string[] = []; let server: Server | undefined; afterEach(async () => { diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts index dc480221a..b4b778f53 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-dimensions.integration.test.ts @@ -53,7 +53,7 @@ function capture(dimension: string, reverse = false, unit = "token"): string { } function fingerprint(source: string): string { - const transfers = selectTransfers(source, options, { values: new Map(), conflict: false }); + const transfers = selectTransfers(source, options); const body = JSON.stringify(transfers[0]?.payload); expect(body).not.toContain("private-agent"); return ( diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts index 1d6d5fb0f..834972749 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -4,7 +4,6 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, expect, test } from "vitest"; import { - readBindingDetails, selectTransfers, send, syncOtel, @@ -30,33 +29,6 @@ afterEach(async () => { await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); }); -posixTest( - "rejects duplicate identity keys in a persisted binding, even with a v1 proof", - async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-duplicate-identity-")); - dirs.push(home); - const state = join(home, "binding.json"); - const payload = { - mayHaveLinked: true, - session: "session", - binding: "branch\u0000owner/repo#1", - identity: [["user.email", "first@example.invalid"]], - identityProof: { - version: 1, - admissionKey: "a".repeat(64), - sources: [{ keys: ["user.email"], at: "2026-11-05T00:00:00.000Z" }], - }, - }; - await writeFile(state, JSON.stringify(payload), { mode: 0o600 }); - await expect(readBindingDetails(state)).resolves.toMatchObject({ - details: { identity: [["user.email", "first@example.invalid"]] }, - }); - payload.identity.push(["user.email", "second@example.invalid"]); - await writeFile(state, JSON.stringify(payload), { mode: 0o600 }); - await expect(readBindingDetails(state)).rejects.toThrow("Sync binding state is invalid"); - } -); - function codexLog(time: Record<string, string | number>, timestamp?: string): string { return JSON.stringify({ signal: "logs", @@ -173,13 +145,9 @@ test.each([ { sum: [] }, { explicitBounds: ["bad"] }, ])("rejects malformed or unrecorded NumberDataPoint values", (extra) => { - expect(() => - selectTransfers( - claudePoint(extra), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ) - ).toThrow(/OTLP/); + expect(() => selectTransfers(claudePoint(extra), { ...options, config: "claude" })).toThrow( + /OTLP/ + ); }); test.each([ @@ -193,13 +161,9 @@ test.each([ { sum: {}, exponential_histogram: {} }, { sum: { dataPoints: {} } }, ])("rejects missing, unsupported or competing data on a supported metric", (data) => { - expect(() => - selectTransfers( - metricData(data), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ) - ).toThrow(/OTLP metric data/); + expect(() => selectTransfers(metricData(data), { ...options, config: "claude" })).toThrow( + /OTLP metric data/ + ); }); test("keeps valid histograms separate from NumberDataPoint's scalar oneof", () => { @@ -212,11 +176,10 @@ test("keeps valid histograms separate from NumberDataPoint's scalar oneof", () = bucketCounts: ["1", "1"], attributes: [attribute("session.id", "session")], }; - const [transfer] = selectTransfers( - metricData({ histogram: { dataPoints: [point] } }), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ); + const [transfer] = selectTransfers(metricData({ histogram: { dataPoints: [point] } }), { + ...options, + config: "claude", + }); expect(JSON.stringify(transfer.payload)).toContain('"count":"2"'); expect(JSON.stringify(transfer.payload)).toContain('"sum":5.5'); }); @@ -237,13 +200,9 @@ test.each([ { bucketCounts: 1 }, { flags: 1.5 }, ])("rejects invalid or inexact integer metric fields before exporting", (extra) => { - expect(() => - selectTransfers( - claudePoint(extra), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ) - ).toThrow(/invalid OTLP integer/i); + expect(() => selectTransfers(claudePoint(extra), { ...options, config: "claude" })).toThrow( + /invalid OTLP integer/i + ); }); test("preserves exact integer strings for signed and unsigned metric values", () => { @@ -252,11 +211,7 @@ test("preserves exact integer strings for signed and unsigned metric values", () count: "18446744073709551615", bucketCounts: ["9007199254740993", "0"], }; - const [transfer] = selectTransfers( - claudePoint(extra), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ); + const [transfer] = selectTransfers(claudePoint(extra), { ...options, config: "claude" }); const payload = JSON.stringify(transfer.payload); for (const value of [extra.asInt, extra.count, ...extra.bucketCounts]) expect(payload).toContain(`"${value}"`); @@ -360,11 +315,10 @@ function metricWithMetadata(unit: string, temporality: number, monotonic: boolea test("distinguishes metric unit, aggregation and monotonicity when computing series identity", () => { const fingerprint = (unit: string, temporality: number, monotonic: boolean) => { - const [transfer] = selectTransfers( - metricWithMetadata(unit, temporality, monotonic), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ); + const [transfer] = selectTransfers(metricWithMetadata(unit, temporality, monotonic), { + ...options, + config: "claude", + }); return JSON.stringify(transfer.payload).match( /"aidd.series_fingerprint","value":\{"stringValue":"([0-9a-f]+)"/ )?.[1]; @@ -379,24 +333,15 @@ test("distinguishes metric unit, aggregation and monotonicity when computing ser test("does not convert an unsafe numeric OTLP timestamp into a fabricated Codex event time", () => { const unsafe = 1785044858922210000; const line = codexLog({ timeUnixNano: unsafe }); - expect(() => selectTransfers(line, options, { values: new Map(), conflict: false })).toThrow( - /invalid OTLP integer/i - ); + expect(() => selectTransfers(line, options)).toThrow(/invalid OTLP integer/i); }); test("preserves exact decimal-string nanoseconds, including for Claude metric intervals", () => { const time = "1785044858922210000"; - const [log] = selectTransfers(codexLog({ timeUnixNano: time }), options, { - values: new Map(), - conflict: false, - }); + const [log] = selectTransfers(codexLog({ timeUnixNano: time }), options); expect(log.at).toBe("2026-07-26T05:47:38.922Z"); expect(JSON.stringify(log.payload)).toContain(time); - const [metric] = selectTransfers( - claudeMetric(time, time), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ); + const [metric] = selectTransfers(claudeMetric(time, time), { ...options, config: "claude" }); expect(metric.start).toBe("2026-07-26T05:47:38.922Z"); expect(metric.at).toBe("2026-07-26T05:47:38.922Z"); expect(JSON.stringify(metric.payload)).toContain(time); @@ -406,25 +351,19 @@ test.each([ [1785044858922210000, "1785044858922210000"], ["1785044858922210000", 1785044858922210000], ] as const)("rejects unsafe numeric Claude metric bounds", (start, end) => { - expect(() => - selectTransfers( - claudeMetric(start, end), - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ) - ).toThrow(/invalid OTLP integer/i); + expect(() => selectTransfers(claudeMetric(start, end), { ...options, config: "claude" })).toThrow( + /invalid OTLP integer/i + ); }); test("does not forward an unsafe timestamp beside a valid Codex event timestamp", () => { const line = codexLog({ timeUnixNano: 1785044858922210000 }, "2026-07-26T05:47:38.922Z"); - expect(() => selectTransfers(line, options, { values: new Map(), conflict: false })).toThrow( - /invalid OTLP integer/i - ); + expect(() => selectTransfers(line, options)).toThrow(/invalid OTLP integer/i); }); test("uses a genuine OTLP event time when a Codex calendar date is invalid", () => { const line = codexLog({ timeUnixNano: "1785044858922210000" }, "2026-02-30T12:00:00Z"); - const transfers = selectTransfers(line, options, { values: new Map(), conflict: false }); + const transfers = selectTransfers(line, options); expect(transfers[0].at).toBe("2026-07-26T05:47:38.922Z"); expect(JSON.stringify(transfers[0].payload)).not.toContain("aidd.original_event_timestamp"); expect(JSON.stringify(transfers[0].payload)).not.toContain("2026-02-30"); @@ -436,20 +375,14 @@ test.each([ "2026-01-01T12:00:00+24:00", "2026-01-01T12:00:00", ])("does not normalize an invalid provider event time: %s", (time) => { - const transfers = selectTransfers(codexLog({}, time), options, { - values: new Map(), - conflict: false, - }); + const transfers = selectTransfers(codexLog({}, time), options); expect(transfers[0].at).toBeUndefined(); expect(JSON.stringify(transfers[0].payload)).not.toContain("aidd.original_event_timestamp"); }); test("keeps a valid offset and leap date as the original event time", () => { const original = "2028-02-29T12:30:00.123456+02:30"; - const transfers = selectTransfers(codexLog({}, original), options, { - values: new Map(), - conflict: false, - }); + const transfers = selectTransfers(codexLog({}, original), options); expect(transfers[0].at).toBe("2028-02-29T10:00:00.123Z"); expect(JSON.stringify(transfers[0].payload)).toContain(original); }); @@ -477,7 +410,7 @@ posixTest("does not export an observation-only timestamp as event activity", asy if (address === null || typeof address === "string") throw new Error("No test port"); const result = await syncOtel( { ...options, home, endpoint: `http://127.0.0.1:${address.port}` }, - async () => ({ state: "unbound" }), + async () => {}, async () => true ); expect(result.sent).toBe(0); @@ -508,11 +441,7 @@ test("does not use Claude observation time as event activity", () => { ], }, }); - const transfers = selectTransfers( - line, - { ...options, config: "claude" }, - { values: new Map(), conflict: false } - ); + const transfers = selectTransfers(line, { ...options, config: "claude" }); expect(transfers).toHaveLength(1); expect(transfers[0].at).toBeUndefined(); }); @@ -540,7 +469,7 @@ test("preserves distinct submillisecond Codex event timestamps in the transmitte ], }, }); - const transfers = selectTransfers(line, options, { values: new Map(), conflict: false }); + const transfers = selectTransfers(line, options); expect(transfers).toHaveLength(2); for (const [index, transfer] of transfers.entries()) { expect(JSON.stringify(transfer.payload)).toContain(timestamps[index]); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts index 171410752..bbc84a150 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel.integration.test.ts @@ -65,19 +65,15 @@ describe("OTLP capture sync", () => { ], }, }); - const transfers = selectTransfers( - source, - { - config: "claude", - session: "wanted", - tenant: "tenant", - endpoint: "http://127.0.0.1:3011", - token: "test", - home: "/tmp", - follow: false, - }, - { values: new Map(), conflict: false } - ); + const transfers = selectTransfers(source, { + config: "claude", + session: "wanted", + tenant: "tenant", + endpoint: "http://127.0.0.1:3011", + token: "test", + home: "/tmp", + follow: false, + }); expect(transfers).toHaveLength(1); const sent = JSON.stringify(transfers[0]); expect(sent).toContain('"asInt":"42"'); @@ -138,7 +134,7 @@ describe("OTLP capture sync", () => { }; const result = await syncOtel( syncOptions, - async () => ({ state: "unbound" }), + async () => {}, async () => true ); expect(result.sent).toBe(1); @@ -153,7 +149,7 @@ describe("OTLP capture sync", () => { expect(await readFile(file, "utf8")).toBe(before); const denied = await syncOtel( syncOptions, - async () => ({ state: "unbound" }), + async () => {}, async () => false ); expect(denied.skipped).toBe(1); @@ -172,13 +168,13 @@ describe("OTLP capture sync", () => { home: "/missing", follow: false, }, - async () => ({ state: "unbound" }), + async () => {}, async () => true ) ).rejects.toThrow(/loopback/); }); - it("replays after a rejected delivery and invalidates a formerly confirmed session", async () => { + it("replays provider events after a rejected delivery", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-retry-")); dirs.push(home); const directory = join(home, ".local/share/aidd-otel-observation"); @@ -213,25 +209,22 @@ describe("OTLP capture sync", () => { token: "test-token", home, follow: false, - stateDir: join(home, "state"), - admission: { cwd: "/project", repositoryId: "repo" }, }; - const confirmed = async () => ({ - state: "confirmed" as const, - feature: "owner/repo#7", - branch: "feature/work", - }); - await expect(syncOtel(options, confirmed, async () => true)).rejects.toThrow("HTTP 503"); + await expect( + syncOtel( + options, + async () => {}, + async () => true + ) + ).rejects.toThrow("HTTP 503"); reject = false; - await syncOtel(options, confirmed, async () => true); - expect(received.filter((body) => body.includes("codex.sse_event"))).toHaveLength(2); - expect(received.some((body) => body.includes('aidd.session_feature"'))).toBe(true); await syncOtel( options, - async () => ({ state: "unbound" }), + async () => {}, async () => true ); - expect(received.some((body) => body.includes("aidd.session_feature_invalidated"))).toBe(true); + expect(received.filter((body) => body.includes("codex.sse_event"))).toHaveLength(2); + expect(received).toHaveLength(2); }); it("follows only newly appended complete lines for the chosen session", async () => { @@ -268,14 +261,14 @@ describe("OTLP capture sync", () => { signal: abort.signal, intervalMs: 10, }, - async () => ({ state: "unbound" }), + async () => {}, async () => true ); expect(received).toBe(2); expect(result.sent).toBe(2); }); - it("never uses an identity from an observation outside consent to publish a binding", async () => { + it("never sends an observation outside consent", async () => { const home = await mkdtemp(join(tmpdir(), "aidd-sync-unconsented-identity-")); dirs.push(home); const directory = join(home, ".local/share/aidd-otel-observation"); @@ -306,8 +299,8 @@ describe("OTLP capture sync", () => { home, follow: false, }, - async () => ({ state: "confirmed", feature: "owner/repo#7", branch: "feature/work" }), - async (transfer) => Object.keys(transfer.payload).length === 0 + async () => {}, + async () => false ); expect(result).toMatchObject({ sent: 0, skipped: 1 }); expect(calls).toBe(0); @@ -350,7 +343,7 @@ describe("OTLP capture sync", () => { follow: true, intervalMs: 10, }, - async () => ({ state: "unbound" }), + async () => {}, async (transfer) => !transfer.consentAt || consenting ) ).rejects.toThrow(/Consent is no longer confirmed/); diff --git a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts index a711a470c..585751b5a 100644 --- a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts @@ -41,18 +41,6 @@ describe.skipIf(process.platform === "win32")("OTLP verification", () => { await chmod(path, 0o644); await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/private/); }); - it("accepts OpenCode 1 instrumentation supplied only through its inline environment", async () => { - const host = { - home, - env: { OPENCODE_CONFIG_CONTENT: '{"experimental":{"openTelemetry":true}}' }, - opencodeVersion: async () => "1.19.5", - }; - await initializeOtel("opencode", "dev", host); - await writeFile(join(home, ".local/share/aidd-otel-observation/opencode.jsonl"), "{}\n", { - mode: 0o600, - }); - expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ lines: 1 }); - }); it("counts normalized observations without returning raw content", async () => { await initializeOtel("claude", "dev", { home, env: {} }); @@ -120,49 +108,9 @@ describe.skipIf(process.platform === "win32")("OTLP verification", () => { lines: 2, observations: 2, unrecognisedLines: 1, - eventLines: 0, - }); - }); - - it("also counts a local OpenCode SSE capture when present", async () => { - const host = { - home, - env: { OPENCODE_CONFIG_CONTENT: '{"experimental":{"openTelemetry":true}}' }, - opencodeVersion: async () => "1.19.5", - }; - await initializeOtel("opencode", "dev", host); - const directory = join(home, ".local/share/aidd-otel-observation"); - await writeFile(join(directory, "opencode.jsonl"), "{}\n", { mode: 0o600 }); - await mkdir(join(directory, "opencode")); - await writeFile( - join(directory, "opencode/events.jsonl"), - `${JSON.stringify({ - type: "session.step.ended", - data: { tokens: { input: 4 } }, - })}\n`, - { mode: 0o600 } - ); - expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ - observations: 1, - eventLines: 1, }); }); - it("refuses an OpenCode event capture that is not a regular file", async () => { - const host = { - home, - env: { OPENCODE_CONFIG_CONTENT: '{"experimental":{"openTelemetry":true}}' }, - opencodeVersion: async () => "1.19.5", - }; - await initializeOtel("opencode", "dev", host); - const directory = join(home, ".local/share/aidd-otel-observation"); - await writeFile(join(directory, "opencode.jsonl"), "{}\n", { mode: 0o600 }); - await mkdir(join(directory, "opencode", "events.jsonl"), { recursive: true }); - await expect(verifyOtel("opencode", "dev", host)).rejects.toThrow( - /event capture must be a regular file/ - ); - }); - it("refuses production", async () => { await expect(verifyOtel("claude", "prod", { home, env: {} })).rejects.toThrow(/production/); }); diff --git a/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts b/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts deleted file mode 100644 index 032860d32..000000000 --- a/cli/tests/e2e/telemetry-otel-metadata.e2e.test.ts +++ /dev/null @@ -1,254 +0,0 @@ -import { appendFile, mkdir, rm, writeFile } from "node:fs/promises"; -import { createServer } from "node:http"; -import { delimiter, join } from "node:path"; -import { describe, expect, it } from "vitest"; -import { initRepository, sandboxGitEnv } from "../helpers/git-sandbox.js"; -import { createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; - -interface LogEnvelope { - resourceLogs: { - scopeLogs: { - logRecords: { attributes: { key: string; value: { stringValue: string } }[] }[]; - }[]; - }[]; -} - -function logAttributes(body: string): Record<string, string> { - const payload = JSON.parse(body) as LogEnvelope; - return Object.fromEntries( - payload.resourceLogs.flatMap((resource) => - resource.scopeLogs.flatMap((scope) => - scope.logRecords.flatMap((record) => - record.attributes.map(({ key, value }) => [key, value.stringValue]) - ) - ) - ) - ); -} - -/** The AIDD binary and consent ledger are real. The Codex protocol and receiver are controlled - * stand-ins: this proves the integration contract, not what an installed Codex emits. */ -describe.skipIf(process.platform === "win32")("Codex metadata through the CLI", () => { - it("sends consented session metadata and rejects an inconsistent native project", async () => { - const { projectDir, fakeHome, cleanup } = await createTestEnv("otel-metadata"); - const telemetry = join(fakeHome, "telemetry"); - const bin = join(fakeHome, "bin"); - const session = "00000000-0000-4000-8000-000000000123"; - const received: { url: string; auth: string | undefined; body: string }[] = []; - const server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push({ url: request.url ?? "", auth: request.headers.authorization, body }); - response.writeHead(200, { "content-type": "application/json" }).end("{}"); - }); - }); - try { - initRepository(projectDir, sandboxGitEnv(fakeHome), { - remote: "git@github.com:example/project.git", - }); - await mkdir(bin); - await writeFile( - join(bin, "codex"), - `#!${process.execPath} -const readline = require('node:readline'); -readline.createInterface({ input: process.stdin }).on('line', (line) => { - const message = JSON.parse(line); - if (!message.id) return; - if (!['initialize', 'project/read', 'thread/read'].includes(message.method)) { - process.stdout.write(JSON.stringify({ id: message.id, error: { code: -32601, message: 'Unknown method' } }) + '\\n'); - return; - } - const result = message.method === 'initialize' ? {} : message.method === 'project/read' - ? { project: { id: process.env.AIDD_TEST_BAD_PROJECT ? 'other-project' : 'project-1', name: 'Example project' } } - : { thread: { id: message.params.threadId, sessionId: message.params.threadId, name: 'Example session', projectId: 'project-1', turns: [] } }; - process.stdout.write(JSON.stringify({ id: message.id, result }) + '\\n'); -}); -`, - { mode: 0o700 } - ); - const env = { - AIDD_TELEMETRY_DIR: telemetry, - AIDD_TELEMETRY: "", - AIDD_OTEL_TOKEN: "test-token", - AIDD_TEST_BAD_PROJECT: "", - PATH: [bin, sandboxedEnv(fakeHome).PATH].join(delimiter), - }; - const cli = (args: string[], extra: Record<string, string> = {}) => - runCli(["telemetry", ...args], projectDir, fakeHome, { env: { ...env, ...extra } }); - await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("Receiver did not start"); - const sync = (extra: Record<string, string> = {}, titlesOnly = true) => - cli( - [ - "otel", - "sync", - "--config=codex", - `--session=${session}`, - "--tenant=test", - `--endpoint=http://127.0.0.1:${address.port}`, - ...(titlesOnly ? ["--titles-only"] : ["--include-session-titles"]), - "--include-session-projects", - ], - extra - ); - const transcriptDir = join(fakeHome, ".codex/sessions"); - await mkdir(transcriptDir, { recursive: true }); - const captureDir = join(fakeHome, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); - const capture = (at: string, input: string) => ({ - signal: "logs", - payload: { - resourceLogs: [ - { - scopeLogs: [ - { - logRecords: [ - { - timeUnixNano: (BigInt(Date.parse(at)) * 1_000_000n).toString(), - attributes: [ - attr("conversation.id", session), - attr("user.email", "person@example.test"), - attr("event.name", "codex.sse_event"), - attr("event.kind", "response.completed"), - attr("model", "model-test"), - attr("input_token_count", input), - attr("output_token_count", "2"), - attr("cached_token_count", "4"), - attr("prompt", "private prompt"), - ], - }, - ], - }, - ], - }, - ], - }, - }); - const writeSession = async (at: string) => { - const transcript = [ - { - timestamp: at, - type: "session_meta", - payload: { id: session, cwd: projectDir, cli_version: "test", source: "cli" }, - }, - { - timestamp: at, - type: "turn_context", - payload: { cwd: projectDir, model: "model-test" }, - }, - { - timestamp: at, - type: "token_usage_record", - payload: { - response_id: "response-1", - session_id: session, - thread_id: session, - usage: { - input_tokens: 10, - cached_input_tokens: 4, - cache_write_input_tokens: 0, - output_tokens: 2, - reasoning_output_tokens: 1, - }, - }, - }, - ]; - await writeFile( - join(transcriptDir, `rollout-test-${session}.jsonl`), - `${transcript.map((line) => JSON.stringify(line)).join("\n")}\n` - ); - await writeFile(join(captureDir, "codex.jsonl"), `${JSON.stringify(capture(at, "10"))}\n`, { - mode: 0o600, - }); - }; - - await writeSession(new Date().toISOString()); - const withoutConsent = await sync(); - expect(withoutConsent.exitCode).not.toBe(0); - expect(received).toHaveLength(0); - - await rm(join(transcriptDir, `rollout-test-${session}.jsonl`)); - const on = await cli(["on", "--yes", "--anonymous"]); - expect(on.exitCode, on.stderr).toBe(0); - await writeSession(new Date().toISOString()); - - const sent = await sync(); - expect(sent.exitCode, sent.stderr + sent.stdout).toBe(0); - expect(received).toHaveLength(2); - expect( - received.every( - (request) => - request.url === "/api/kpi/otel/codex/test/v1/logs" && - request.auth === "Bearer test-token" - ) - ).toBe(true); - const attributes = received.map(({ body }) => logAttributes(body)); - expect(attributes).toContainEqual( - expect.objectContaining({ - "event.name": "aidd.session_title", - "conversation.id": session, - "aidd.session_title": "Example session", - }) - ); - expect(attributes).toContainEqual( - expect.objectContaining({ - "event.name": "aidd.session_project", - "conversation.id": session, - "aidd.project_id": "project-1", - "aidd.project_name": "Example project", - }) - ); - expect(JSON.stringify(received)).not.toContain("private prompt"); - expect(JSON.stringify(received)).not.toContain("input_token_count"); - - received.length = 0; - const full = await sync({}, false); - expect(full.exitCode, full.stderr + full.stdout).toBe(0); - expect(received).toHaveLength(3); - expect(received.map(({ body }) => logAttributes(body))).toContainEqual( - expect.objectContaining({ - "event.name": "codex.sse_event", - "event.kind": "response.completed", - "conversation.id": session, - model: "model-test", - input_token_count: "10", - output_token_count: "2", - cached_token_count: "4", - }) - ); - expect(JSON.stringify(received)).not.toContain("private prompt"); - - received.length = 0; - const inconsistent = await sync({ AIDD_TEST_BAD_PROJECT: "1" }); - expect(inconsistent.exitCode, inconsistent.stderr).toBe(0); - expect(received).toHaveLength(1); - expect(received[0].body).toContain("aidd.session_title"); - expect(received[0].body).not.toContain("project-1"); - - const off = await cli(["off"]); - expect(off.exitCode, off.stderr).toBe(0); - received.length = 0; - const revoked = await sync(); - expect(revoked.exitCode, revoked.stderr).toBe(0); - expect(received).toHaveLength(0); - - // No historical or new event is sent while current consent is off. - await appendFile( - join(captureDir, "codex.jsonl"), - `${JSON.stringify(capture(new Date().toISOString(), "999"))}\n` - ); - const afterOff = await sync({}, false); - expect(afterOff.exitCode).not.toBe(0); - expect(afterOff.stderr + afterOff.stdout).toContain("Consent is no longer confirmed"); - expect(received).toHaveLength(0); - } finally { - await new Promise<void>((resolve) => server.close(() => resolve())); - await cleanup(); - } - }); -}); diff --git a/cli/tests/e2e/telemetry-otel.e2e.test.ts b/cli/tests/e2e/telemetry-otel.e2e.test.ts index 302d5d6e1..51cd7c358 100644 --- a/cli/tests/e2e/telemetry-otel.e2e.test.ts +++ b/cli/tests/e2e/telemetry-otel.e2e.test.ts @@ -1,11 +1,9 @@ -import { spawn } from "node:child_process"; -import { once } from "node:events"; import { mkdir, writeFile } from "node:fs/promises"; -import { delimiter, join } from "node:path"; +import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { cliPath, createTestEnv, runCli, sandboxedEnv } from "./helpers.js"; +import { createTestEnv, runCli } from "./helpers.js"; -describe.skipIf(process.platform === "win32")("native OTLP launch", () => { +describe.skipIf(process.platform === "win32")("native OTLP verification", () => { it("verifies captured receipt through the CLI and fails when capture is absent", async () => { const { projectDir, fakeHome, cleanup } = await createTestEnv("telemetry-verify"); const directory = join(fakeHome, ".local/share/aidd-otel-observation"); @@ -52,76 +50,4 @@ describe.skipIf(process.platform === "win32")("native OTLP launch", () => { await cleanup(); } }); - - it.each([ - ["SIGINT", 42], - ["SIGTERM", 43], - ] as const)( - "forwards %s sent to AIDD and preserves OpenCode's exit code", - async (signal, code) => { - const { projectDir, fakeHome, cleanup } = await createTestEnv("telemetry-otel"); - const tools = join(fakeHome, "bin"); - const collector = join(fakeHome, ".local/share/aidd-otel-observation"); - const opts = sandboxedEnv(fakeHome); - const env = { - PATH: [tools, opts.PATH].join(delimiter), - OPENCODE_CONFIG: "", - OPENCODE_CONFIG_DIR: "", - OPENCODE_CONFIG_CONTENT: "", - OTEL_EXPORTER_OTLP_ENDPOINT: "", - OTEL_EXPORTER_OTLP_HEADERS: "", - }; - try { - await mkdir(tools); - await mkdir(collector, { recursive: true }); - await writeFile(join(collector, "collector-token"), "test-token", { mode: 0o600 }); - await writeFile( - join(tools, "opencode"), - `#!${process.execPath} -if (process.argv.includes("--version")) { - console.log("1.19.5"); - process.exit(0); -} -if (JSON.stringify(process.argv.slice(2)) !== '["stats","--days","1"]') process.exit(20); -if (process.env.OTEL_EXPORTER_OTLP_ENDPOINT !== "http://127.0.0.1:14318/opencode") process.exit(21); -if (process.env.OTEL_EXPORTER_OTLP_HEADERS !== "Authorization=Bearer test-token") process.exit(22); -process.on("SIGINT", () => process.exit(42)); -process.on("SIGTERM", () => process.exit(43)); -console.log(process.pid); -setTimeout(() => process.exit(24), 2000); -`, - { mode: 0o700 } - ); - const init = await runCli( - ["telemetry", "otel", "init", "--config=opencode", "--endpoint=dev"], - projectDir, - fakeHome, - { env } - ); - expect(init.exitCode).toBe(0); - const wrapper = spawn( - process.execPath, - [ - cliPath(), - "telemetry", - "otel", - "run", - "--config=opencode", - "--", - "stats", - "--days", - "1", - ], - { cwd: projectDir, env: sandboxedEnv(fakeHome, env), stdio: ["ignore", "pipe", "pipe"] } - ); - const finished = once(wrapper, "exit"); - const ready = await Promise.race([once(wrapper.stdout, "data"), finished]); - expect(Number(String(ready[0]))).toBeGreaterThan(0); - expect(wrapper.kill(signal)).toBe(true); - expect(await finished).toEqual([code, null]); - } finally { - await cleanup(); - } - } - ); }); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 3c93092a3..d6cd40d01 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -167,7 +167,7 @@ { "invocation": "aidd telemetry otel", "exitCode": 0, - "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local\n capture without changing files\n collect [options] Capture authenticated OTLP on a local loopback port\n sync [options] Send one consented session's local telemetry to a\n loopback Gouvernail\n run [options] [args...] Launch OpenCode 1 with local OTLP export. Pass tool\n arguments after --\n help [command] display help for command" + "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local capture\n without changing files\n collect [options] Capture authenticated OTLP on a local loopback port\n sync [options] Send one consented session's local telemetry to a loopback\n Gouvernail\n help [command] display help for command" }, { "invocation": "aidd telemetry otel collect", @@ -179,15 +179,10 @@ "exitCode": 0, "help": "Usage: aidd telemetry otel init [options]\n\nEnable detailed local OTLP export in one tool's user configuration\n\nOptions:\n --config <tool> User configuration to merge (choices: \"claude\",\n \"codex\", \"opencode\")\n --endpoint <environment> Collector destination (choices: \"dev\", \"prod\")\n -h, --help display help for command" }, - { - "invocation": "aidd telemetry otel run", - "exitCode": 0, - "help": "Usage: aidd telemetry otel run [options] [args...]\n\nLaunch OpenCode 1 with local OTLP export. Pass tool arguments after --\n\nOptions:\n --config <tool> Tool to launch (choices: \"opencode\")\n -h, --help display help for command" - }, { "invocation": "aidd telemetry otel sync", "exitCode": 0, - "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow\n to discover sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default:\n false)\n --include-session-titles Send Codex chat titles, which may contain\n prompt text (default: false)\n --include-session-projects Send native Codex project IDs and names (may\n contain sensitive text) (default: false)\n --include-session-repositories Send verified Codex GitHub/GitLab repository\n URLs (may identify private work) (default:\n false)\n --titles-only Send only Codex titles and opted-in metadata\n for consented sessions (default: false)\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to discover\n sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default: false)\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index f3563761d..90fb8d9ae 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -761,7 +761,6 @@ describe("native telemetry commands", () => { lines: 1, modifiedAt: "2026-11-04T00:00:00.000Z", observations: 1, - eventLines: 0, unrecognisedLines: 0, }); expect(await run("otel", "verify", "--config=claude", "--endpoint=dev")).toBe(1); @@ -770,25 +769,13 @@ describe("native telemetry commands", () => { expect(await run("otel", "verify", "--config=claude", "--endpoint=dev")).toBeNull(); expect(stdout()).toContain("Configuration verified in /config/claude"); expect(stdout()).toContain("1 captured lines"); - expect(stdout()).toContain("1 normalized observations; 0 event lines; 0 unrecognised lines"); + expect(stdout()).toContain("1 normalized observations; 0 unrecognised lines"); expect(stdout()).toContain( "Existing capture does not prove delivery from the current session." ); expect(check).toHaveBeenCalledWith("claude", "dev"); }); - it("forwards the launched OpenCode exit code", async () => { - const launch = vi - .spyOn(otel, "launchOtel") - .mockResolvedValueOnce(7) - .mockRejectedValueOnce(new Error("Cannot launch OpenCode")); - expect(await run("otel", "run", "--config=opencode", "--", "check")).toBeNull(); - expect(launch).toHaveBeenCalledWith(["check"]); - expect(process.exitCode).toBe(7); - expect(await run("otel", "run", "--config=opencode", "--", "check")).toBe(1); - expect(stderr()).toContain("Cannot launch OpenCode"); - }); - it("forwards one consented session without printing its token", async () => { vi.stubEnv("AIDD_OTEL_TOKEN", "private-test-token"); const admission = { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }; @@ -821,7 +808,7 @@ describe("native telemetry commands", () => { token: "private-test-token", follow: false, }); - expect(await call[1]()).toEqual({ state: "unbound" }); + await expect(call[1]()).resolves.toBeUndefined(); const transfer = { signal: "logs" as const, payload: {}, at: "2026-11-04T00:00:00Z" }; expect(await call[2](transfer)).toBe(true); expect(await call[2](transfer)).toBe(false); diff --git a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts index 2f2e880dd..57055ea98 100644 --- a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts +++ b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts @@ -3,7 +3,6 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; const mocks = vi.hoisted(() => ({ initialize: vi.fn(), verify: vi.fn(), - launch: vi.fn(), checkId: vi.fn(), ensureId: vi.fn(), finishId: vi.fn(), @@ -19,9 +18,6 @@ vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/init.js", ( vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js", () => ({ verifyOtel: mocks.verify, })); -vi.mock("../../../src/contexts/telemetry/infrastructure/otel/actions/run.js", () => ({ - runOtelOpenCode: mocks.launch, -})); vi.mock("../../../src/runtime/filesystem/installation-identity.js", () => ({ checkInstallationId: mocks.checkId, ensureInstallationId: mocks.ensureId, @@ -34,14 +30,8 @@ vi.mock("../../../src/runtime/wiring/telemetry.js", () => ({ telemetryDir: () => "/profile/telemetry", })); -const { - collectOtel, - configureOtel, - checkOtel, - launchOtel, - syncCapturedSession, - followCapturedSessions, -} = await import("../../../src/runtime/wiring/telemetry-otel.js"); +const { collectOtel, configureOtel, checkOtel, syncCapturedSession, followCapturedSessions } = + await import("../../../src/runtime/wiring/telemetry-otel.js"); const collector = await import( "../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js" ); @@ -104,16 +94,13 @@ describe("OTLP runtime wiring", () => { expect(mocks.ensureId).not.toHaveBeenCalled(); }); - it("passes its runtime home and environment to verification and launch", async () => { + it("passes its runtime home and environment to verification", async () => { mocks.verify.mockResolvedValue({ lines: 1 }); - mocks.launch.mockResolvedValue(7); await expect(checkOtel("codex", "dev")).resolves.toEqual({ lines: 1 }); - await expect(launchOtel(["check"])).resolves.toBe(7); expect(mocks.verify).toHaveBeenCalledWith("codex", "dev", { home: "/profile", env: process.env, }); - expect(mocks.launch).toHaveBeenCalledWith(["check"], { home: "/profile", env: process.env }); }); it("uses the private runtime profile for one session and automatic following", async () => { @@ -129,7 +116,7 @@ describe("OTLP runtime wiring", () => { endpoint: "http://127.0.0.1:3000", token: "test-token", }; - const resolve = async () => ({ state: "unbound" as const }); + const resolve = async () => {}; const allow = async () => true; const snapshot = async () => new Map(); const permits = async () => "allow" as const; @@ -141,7 +128,6 @@ describe("OTLP runtime wiring", () => { session: "s1", follow: false, home: "/profile", - stateDir: "/profile/telemetry", }, resolve, allow diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index fb800c19e..d1096c3bf 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -1,141 +1,29 @@ -# Provider telemetry field mapping +# Provider OpenTelemetry -Use this page to see what each provider can emit, what the local collector keeps, and what the development sync actually sends. For the three modes, opt-in and identity rules, start with the [README](README.md#choose-a-mode). +Gouvernail's AI KPI view uses provider OpenTelemetry (OTel) observations only. The framework's existing local usage ledger is separate and is not sent as OTel. A provider observation is not proof that every event was delivered; missing or ambiguous data remains unavailable. -This is an inventory of source fields, not a promise that the AIDD report ingests them. Claude Code and Codex fields below were observed in local OTLP captures; OpenCode 2 fields were observed in its session event stream. OpenCode 1 fields are from its published session schema, not a verified OTLP capture. Presence depends on version, configuration, event and provider. A missing value is unknown, not zero. +| Field | Claude Code OTel | Codex OTel | OpenCode v2 OTel | +| --- | --- | --- | --- | +| Session | `session.id`, when present | `conversation.id`, when present | In the `url.path` of observed `/api/session/{id}/prompt` HTTP spans | +| Model | `model`, when present | `model`, when present | Not yet established for all signals | +| Usage | `claude_code.token.usage` and `api_request` | `codex.sse_event` and turn traces | Not established in the observed OTLP capture | +| Identity | Provider attributes, when emitted | Provider attributes, when emitted | Not yet established | -The examples are fictional. `C` is Claude Code, `X` is Codex, `O1` is OpenCode 1 and `O2` is OpenCode 2. OTLP attribute names below are keys in the record, span or resource's `attributes` array, not necessarily JSON object properties. For O1, `info` denotes the message in `message.updated`, `part` the part in `message.part.updated`, and session information comes from `session.created` or `session.updated`; these are session-event schemas, not OTLP attributes. +Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. -| Common element | Example | Claude Code | Codex | OpenCode (v1 / v2) | -| --- | --- | --- | --- | --- | -| Time | `2026-11-04T10:20:00+08:00` | `event.timestamp`, `timeUnixNano`, `observedTimeUnixNano`; span `startTimeUnixNano` / `endTimeUnixNano` | Same OTLP times | **v1:** `info.time.created`, `part.time.start` / `end`; **v2:** `created`, `data.started` on step start | -| Event type | `tool_result` | `event.name`, log body, span or metric name | `event.name`; `eventName` may instead identify a source location | **v1/v2:** event `type` | -| Application version | `2.2.5` | resource `service.version` | `app.version`, resource `service.version` | **v1:** session `info.version`; **v2:** `data.version` on `session.created` or resource `service.version` | -| Session | `session-abc` | `session.id` | `conversation.id`; trace `thread.id` | **v1:** `info.sessionID`, `part.sessionID`; **v2:** `data.sessionID`, event `durable.aggregateID` | -| Provider | `openai` | `anthropic.provider`, `gen_ai.provider.name`, `gen_ai.system` | `provider_name`, trace `provider` | **v1:** assistant `info.providerID`, user `info.model.providerID`; **v2:** `data.model.providerID` on step start | -| Model | `model-xyz` | `model`, `gen_ai.request.model` | `model` | **v1:** assistant `info.modelID`, user `info.model.modelID`; **v2:** `data.model.id` on step start | -| Reasoning effort | `high` | `effort` on `claude_code.token.usage` points when supplied | `reasoning_effort` on `codex.conversation_starts`; `model_reasoning_effort` on `response.completed` | Not verified | -| User input | `Fix the test` | `prompt`, `prompt_text`, span `user_prompt` | `prompt` | **v1:** `part.text` joined by `messageID` to a user message; **v2:** `data.item.payload.text` on `session.inbox.enqueued` with `data.item.type=user` | -| Assistant response | `The test is fixed.` | `response` | `response` | **v1:** `part.text` joined by `messageID` to an assistant message; **v2:** `data.text` on `session.text.ended`, `data.delta` on `session.text.delta` | -| Input tokens | `1,200` | `input_tokens`; `claude_code.token.usage` metric, `type=input` | `input_token_count`, `codex.turn.token_usage.input_tokens`, `gen_ai.usage.input_tokens`; metric `token_type=input` | **v1:** `info.tokens.input` or `part.tokens.input`; **v2:** `data.tokens.input` | -| Output tokens | `350` | `output_tokens`; metric `type=output` | `output_token_count`, `codex.turn.token_usage.output_tokens`, `gen_ai.usage.output_tokens`; metric `token_type=output` | **v1:** `info.tokens.output` or `part.tokens.output`; **v2:** `data.tokens.output` | -| Cache read | `800` | `cache_read_tokens`; metric `type=cacheRead` | `cached_token_count`, `codex.turn.token_usage.cached_input_tokens`, `gen_ai.usage.cache_read.input_tokens`; metric `token_type=cached_input` | **v1:** `info.tokens.cache.read` or `part.tokens.cache.read`; **v2:** `data.tokens.cache.read` | -| Cache write | `200` | `cache_creation_tokens`, `cache_creation_5m_tokens`, `cache_creation_1h_tokens`; metric `type=cacheCreation` | `cache_write_token_count`, `codex.turn.token_usage.cache_write_input_tokens`, `gen_ai.usage.cache_write.input_tokens`; metric `token_type=cache_write_input` | **v1:** `info.tokens.cache.write` or `part.tokens.cache.write`; **v2:** `data.tokens.cache.write` | -| Tool call ID | `call-abc` | `tool_use_id`, `gen_ai.tool.call.id` | `call_id`; trace `runtime_tool_call_id` is distinct | **v1:** `part.callID`; **v2:** `data.id` | -| Tool name | `read_file` | `tool_name`, `gen_ai.tool.name` | `tool_name`; metric `tool` | **v1:** `part.tool`; **v2:** `data.name` on `session.tool.input.started` | -| Tool input | `{"path":"src/app.ts"}` | `tool_parameters`, `tool_input` | `arguments` | **v1:** `part.state.input`; **v2:** `data.input` on `session.tool.called`, serialized `data.text` on `session.tool.input.ended` | -| Tool output | `File contentsโ€ฆ` | `output` on span event `tool.output` | `output` on `codex.tool_result` | **v1:** completed `part.state.output`; **v2:** `data.content[].text` on `session.tool.success` | -| Tool outcome | `success` | `success` on tool result | `success` on tool result | **v1:** `part.state.status` and `part.state.error`; **v2:** `session.tool.success` or `session.tool.failed`, with `data.error.type` / `data.error.message` | +OTLP collection, verification and development forwarding are separate steps. The collector keeps provider OTLP locally; the consent-aware sync forwards only supported signals and allowlisted attributes to a loopback Gouvernail endpoint. The local collector does not check per-clone consent, so its private files may contain other sessions and sensitive data. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). -Provider labels are not necessarily equivalent across tools. A user role does not prove a human authored the input. Some v1 fields require correlating a part with its message; v2 text and model fields depend on the event type. Codex trace `codex.turn.token_usage.non_cached_input_tokens` is a separate count. +The current sync still reads the framework's local usage ledger to establish session-to-clone consent. It does not forward OpenCode traces. Consequently, it is not yet an OTel-only end-to-end pipeline for the three tools. -## Interpretation and coverage - -- The Claude and Codex metric `type` / `token_type` labels above belong to `claude_code.token.usage` and `codex.turn.token_usage`, respectively. OTLP logs, traces and metrics may describe the same operation; do not add them together. -- Claude effort is forwarded only when present on a token point with its model. It is an observation of that point, not a value inferred for other points. [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage#token-counter) (read 2026-11-05). -- Codex effort is an observation on one event, not a value inherited by other turns. The sync forwards only bounded ASCII identifiers when the same event also names its model; a missing or invalid value stays unknown. [Codex event source](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/otel/src/events/session_telemetry.rs) (read 2026-11-05). -- OpenCode 2 emits `data.tokens.*` both on `session.step.ended` and `session.usage.updated`. The latter is cumulative; summing every update overcounts. Its `input` is non-cached input and `output` is visible output; reasoning is separate. The implementation maps missing provider usage to zero, so a reported zero does not prove zero usage. -- Claude's `input_tokens` excludes cache reads and writes. Normalize token definitions before comparison; do not assume a similarly named field counts the same thing in all tools. -- In the examined capture, Codex cache-write fields were present but all zero. Codex can replace absent provider cache details with zeros, so a reported zero does not establish that the provider supplied a zero. [Codex source](https://github.com/openai/codex/blob/rust-v0.166.0-alpha.6/codex-rs/codex-api/src/sse/responses.rs#L115-L145) (read 2026-11-05). A nonzero write was not verified. Not every event includes every field; OpenCode 2 text and model data may need correlation across events. Outputs can be truncated. -- Claude can emit `cost_usd`; OpenCode 1 has assistant and step-finish `cost`, and OpenCode 2 exposes step and cumulative cost. A trustworthy equivalent was not established for Codex in the observed captures. These values do not by themselves establish an actual invoice: pricing basis and completeness must be known, and cumulative and per-step figures must not be combined as independent costs. -- A future feature may display an **estimated token cost**, not an invoice. It would require the provider and model, price source and effective date, currency, and rates for input, output, cache reads and cache writes. Subscription, included quota, negotiated and API pricing can differ. If usage, a compatible rate or a billing basis is unknown, the estimate is unknown rather than zero. Cost calculation and pricing are outside this feature. -- A tool session ID is not automatically a feature ID. Feature attribution requires the framework's session/branch declaration and must handle missing or ambiguous bindings. The existing local usage ledger and provider OTLP export are distinct pipelines. -- OpenCode 2's rich session events are SSE, not the observed OTLP logs. The O1 schema above does not prove its fields are emitted over OTLP. The schema event's `durable.version` is not an application major version. -- Prompts, responses, tool arguments and results can contain secrets. Capture and retain them only with the appropriate consent and local access controls. - -## What sync sends - -The collector keeps the provider's raw OTLP JSON locally. The separate dev sync selects only consented sessions and sends: - -- **Claude Code:** `tool_result` logs and selected metrics: token usage, active time, edit decisions, lines changed, sessions, commits, pull requests and the provider's `cost.usage` metric when present. The dashboard does not calculate a cost. -- **Codex:** `conversation_starts`, `tool_result` and `response.completed` events, including available token counts. -- **Both:** available event time, session, model, application version, provider email/account/organization IDs, tool name, call ID, outcome and duration. Confirmed framework feature bindings additionally carry the declared branch and ticket; an invalidation can be sent when a binding becomes ambiguous. -- **Codex titles:** only with `--include-session-titles`, the free-text chat title is sent as session metadata. It may contain part of a prompt. -- **Codex native projects:** `--include-session-projects` separately opts into sending the native project ID and free-text name as `aidd.session_project`. Names can contain sensitive text. The association must come from `thread.projectId`, then an exact `project/read` result; a second thread read checks membership has not changed. No path, repository, title, parent-thread inheritance or prompt fallback is used. A confirmed `projectId: null` clears the association; missing fields, API failures and inconsistent IDs send nothing. Clone consent is checked again immediately before sending. -- **Codex Git repositories:** `--include-session-repositories` separately opts into sending the repository identity from the exact native thread's `gitInfo.originUrl` as `aidd.session_repository` / `aidd.repository_url`. Only canonical HTTPS links for public GitHub and GitLab hosts are sent; repository names can disclose private work. Credentials, local paths, other hosts and unsafe URLs are never forwarded. A confirmed null or unsupported Git origin clears the association; missing fields and API failures send nothing. Consent is checked again immediately before sending. [Codex app-server thread/read](https://learn.chatgpt.com/docs/app-server#read-a-stored-thread-without-resuming) (read 2026-11-05). - -Use `telemetry otel sync --config=codex --tenant=<slug> --endpoint=<loopback-origin> --titles-only --include-session-projects --include-session-repositories` for one metadata-only pass over consented captured sessions (titles, projects and repositories are opted in). Omit either metadata option to exclude it. It sends no token metrics or feature bindings and writes no capture checkpoint. `--follow` checks sessions loaded by the regular follower; it does not discover every historical chat. Native projects are separate from Git repositories. Claude Code has no universal native project field in this bridge; OpenCode project export is not integrated. - -Codex's project API is experimental. In the local 0.166.0 probe on 2026-11-05, `project/read` returned an exact ID and nonempty native name, but all ten sampled thread reads returned `projectId: null`. A separate Desktop tool listed five chats with explicit project IDs matching its project list; app-server returned `projectId: null` for those same five chats. An independent CLI's access to that Desktop source is not established. Named project export is tested against a simulated app-server, not demonstrated end to end on those local sessions. [Official project protocol](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/app-server-protocol/src/protocol/v2/project.rs) (read 2026-11-05). - -On macOS, `cd cli && node scripts/smoke-codex-project.mjs` checks the installed Codex app-server and the framework's metadata readers with an isolated profile and blocked network. It creates an explicitly linked test session; it does not verify existing Desktop groupings. - -- **Claude series:** a SHA-256 fingerprint of the source resource attributes, instrumentation scope and all point attributes helps detect different metric series that share a legacy key. The attributes used to compute it are not sent as additional plaintext, but the fingerprint can reveal equality or allow guesses about low-entropy values. Existing points lack it and cannot be certified retroactively. [OTLP metric data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) (read 2026-11-05). - -The sync excludes fields for prompt and response text, tool arguments and output, raw API bodies, metric exemplars and traces. It filters attribute keys, not secrets or free text inside allowed values. OpenCode is not forwarded by this sync. These are forwarding rules, not guarantees about what the raw collector receives. - -Codex can replace absent cache details with zeros before emitting OTLP. A zero cache count therefore does not prove that the provider reported zero; Gouvernail displays it, and any uncached-input total depending on it, as unavailable. Nonzero counts are still observations, not proof that every event was received. [Codex response parser](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/codex-api/src/sse/responses.rs), [OTLP emitter](https://github.com/openai/codex/blob/c90354172c5c49535b122443e9a8ed52c2233b80/codex-rs/otel/src/events/session_telemetry.rs) (read 2026-11-05). - -### Example: a Claude metric sent locally - -The structure below was produced by `selectTransfers` from one real local Claude capture. The timestamps, metric value, session, organization, email, account, model and query source have been replaced with fictional values; attribute presence varies by event. It is the filtered request body, not the raw capture or its authorization header. - -```json -{ - "resourceMetrics": [{ - "scopeMetrics": [{ - "metrics": [{ - "name": "claude_code.token.usage", - "sum": { - "aggregationTemporality": 1, - "isMonotonic": true, - "dataPoints": [{ - "startTimeUnixNano": "1767225600000000000", - "timeUnixNano": "1767225660000000000", - "asDouble": 123, - "attributes": [ - { "key": "session.id", "value": { "stringValue": "session-example" } }, - { "key": "organization.id", "value": { "stringValue": "organization-example" } }, - { "key": "user.email", "value": { "stringValue": "person@example.test" } }, - { "key": "user.account_id", "value": { "stringValue": "account-example" } }, - { "key": "model", "value": { "stringValue": "model-example" } }, - { "key": "query_source", "value": { "stringValue": "example" } }, - { "key": "type", "value": { "stringValue": "input" } } - ] - }] - } - }] - }] - }] -} -``` - -## Local Gouvernail sync - -OTLP initialization, verification, local collection and sync require private POSIX files and are not supported on Windows. - -Start the collector, then configure the tools in another shell: - -```sh -aidd telemetry otel collect -# In another shell: -aidd telemetry otel init --config=claude --endpoint=dev -aidd telemetry otel init --config=codex --endpoint=dev -``` - -Restart the tools to load their settings. With local telemetry measurement enabled, start one sync process per provider: - -```sh -# Set the relevant connector token in the environment without putting it in command arguments. -export AIDD_OTEL_TOKEN='<connector token>' -aidd telemetry otel sync --config=claude --tenant=acme --endpoint=http://127.0.0.1:3011 --follow -# In another shell, use the Codex connector token and --config=codex. -``` - -The origin must be loopback HTTP; no token or prompt content is printed. Only supported observations from sessions admitted by the local ledger and covered by an uninterrupted consent interval are sent. A Claude cumulative metric whose whole interval is not covered is excluded. A successful HTTP response does not prove that Gouvernail matched a member: check its session view. - -An automatic follower starts with new capture events and keeps a private checkpoint. Network errors and HTTP 5xx responses are reported and retried with bounded backoff; permanent HTTP failures stop the follower. Restarting resumes from the checkpoint, and a failed post is replayed, so the receiver must remain idempotent. Unknown sessions wait for their transcript to be admitted using private byte-range indexes, without holding up other sessions. These indexes can grow until admission and depend on retaining the original capture; replaced or truncated captures require recovery rather than silently skipping data. Running a follower for each provider is an explicit operation; it does not start automatically after a machine restart. Linked session identifiers and the minimum provider identity are stored privately to detect later ambiguity, including after restart. Older sync state without identity must be refreshed with a session replay. - -Every delivery attempt, including a retry, rechecks session admission and the same clone's current consent and original consent interval. If consent or admission cannot be confirmed, a single-session sync stops before the next POST. The multi-session follower retains an already-admitted but unsent capture range in a private per-session journal and continues other sessions. The journal temporarily stores a SHA-256 hash of the admitted working directory and repository ID, not their plaintext; the hash may reveal equality or permit guesses and is not sent. The range and hash survive restarts, provided the original capture remains available. After renewed consent, the follower can replay it only for the same admission and clone while its original interval remains authorized; otherwise it stays pending. Other delivery errors and invalid state still stop the follower. Session identity is collected only from admitted observations. New binding state privately retains the minimum identity, admission hash and source times for each identifier, and restores them only after rechecking each source and the current consent for the same clone. Legacy bindings without that proof are not automatically replayed or invalidated: an old feature may remain visible but cannot be certified. This unresolved limitation blocks merging the feature. - -To replay one consented session already in the capture, pass `--session=<id>` without `--follow`. Add `--follow` to follow only that session. A feature link is sent only when every recorded call agrees on a single observed branch and a declared canonical ticket such as `owner/repo#123`; otherwise it remains unlinked. If a linked session later becomes ambiguous, an irreversible invalidation keeps it unlinked even if events arrive out of order. +In a local OpenCode v2.2.5 capture, two prompt spans included a session ID in their HTTP path; no model or token counts were observed. This observation does not establish that every OpenCode signal includes a session ID. OpenCode session-event feeds and session exports are not provider OTLP and are not used in this view. Production export remains disabled until authenticated user identity and the destination are implemented. ## Sources -Read on 2026-11-04: +Read 2026-11-06: - [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) -- [Codex observability and telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#observability-and-telemetry) -- [Codex app-server thread metadata](https://learn.chatgpt.com/docs/app-server#read-a-stored-thread-without-resuming) and [OTLP delivery responses](https://opentelemetry.io/docs/specs/otlp/#partial-success-1) (read 2026-11-05) -- [OpenCode 1.18.35 session schema](https://github.com/anomalyco/opencode/blob/v1.18.35/packages/schema/src/v1/session.ts) and [LLM instrumentation](https://github.com/anomalyco/opencode/blob/v1.18.35/packages/opencode/src/session/llm.ts) -- [OpenCode 2.2.5 session events](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts) and [token normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts) +- [Codex telemetry configuration](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted) +- [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts) +- [OpenCode v2 HTTP API audit](https://github.com/anomalyco/opencode/blob/dev/V2_HTTP_API_AUDIT.md) (current development branch; the v2.2.5 path was observed locally) +- [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) +- [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index f81bb9e80..2a3fdb545 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -24,23 +24,21 @@ flowchart LR Tasks -->|"attribute"| Report ``` -The local ledger and provider OTLP captures are separate sources. The OTLP sync reuses the ledger's session attribution, task declarations and consent history; it does not replace the local report. Do not add their totals together. +The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The framework's local ledger powers `aidd telemetry report` and remains separate. The development sync currently relies on the ledger to associate a session with a clone before checking consent. Do not add their totals: both sources can cover the same calls. See the [field inventory](OBSERVABILITY.md). ## Choose a mode | Mode | Opt-in and destination | Identity | What is kept or sent | | --- | --- | --- | --- | | Local usage | `aidd telemetry on` explicitly enables measurement for the current clone from that time. The ledger and report stay on this machine. | An optional name labels the local report; anonymous use is supported. | Per-call usage read from Claude Code and Codex session files: session, time, model, agent, token counts and local attribution context. The ledger does not store prompts or responses. | -| OTLP development | Explicitly configure each provider with `otel init --endpoint=dev` and start the local collector. A separate `otel sync` sends supported data only to loopback Gouvernail for sessions covered by the clone's consent. | No local report name is required. Native provider identifiers may still be sent when present; attribution in Gouvernail needs a matched active member. The connector token is not a person identity. | The collector stores raw provider events, which may include prompts, responses, commands and tool output. The sync excludes fields for those contents and traces, but does not inspect allowed attribute values for secrets. See [exact fields](OBSERVABILITY.md#what-sync-sends). | +| OTLP development | Explicitly configure each provider with `otel init --endpoint=dev` and start the local collector. A separate `otel sync` sends supported data only to loopback Gouvernail for sessions covered by the clone's consent. | No local report name is required. Provider identifiers may still be sent when present; attribution in Gouvernail needs a matched active member. The connector token is not a person identity. | The collector stores raw provider events, which may include prompts, responses, commands and tool output. The sync excludes fields for those contents and traces, but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | | OTLP production | **Not available:** `--endpoint=prod` is refused. `https://gouvernail.eu` is an intended future destination, not an active exporter. | **Required before production is enabled:** an authenticated user identity. This is a product requirement, not yet implemented or enforced. | No production payload is sent by this feature today. Consent, authentication, retention and deletion behavior must be implemented before enabling it. | ### Consent and scope - `on` is an explicit opt-in for one clone, not a team-wide setting. The local ledger and OTLP sync use its consent intervals; an earlier consent does not cover later work after `off`. - `otel init` is a separate, explicit action to configure a provider's user-level exporter. The CLI does not show a separate consent dialog or record separate OTLP consent; the init skill asks for authorization before running it. When that provider and the collector are running, raw OTLP may be captured from other projects too: **the collector does not check clone consent**. The sync checks consent before forwarding. -- Codex chat titles are excluded by default. `otel sync --include-session-titles` separately opts in to sending the current title of each eligible Codex session to the selected Gouvernail destination, even when the title predates the consent interval. Titles are free text and may contain part of a prompt. This option does not send chat messages or change feature attribution. -- To send only previously measured Codex session titles, use `otel sync --config=codex --titles-only` with the same tenant and endpoint options as sync. Add `--session=<id>` to restrict it to one session. This reads the local capture to verify identity and consent but sends no captured metrics or feature bindings and does not change the follow checkpoint. -- Claude metric sync also sends a deterministic fingerprint derived from the original series attributes; it can expose equality or allow guessing low-entropy values. [Details](OBSERVABILITY.md#what-sync-sends). +- Claude metric sync also sends a deterministic fingerprint derived from the original series attributes; it can expose equality or allow guessing low-entropy values. - `aidd telemetry off` stops local measurement for the current clone; it does **not** disable provider exporters or stop the OTLP collector. A process started with `AIDD_TELEMETRY=0` refuses local measurement and sync, not raw collection; changing an environment variable does not change an already running process. - `aidd telemetry forget` previews or deletes local usage measurement. It does **not** delete OTLP capture files, sync state, or data already received by Gouvernail. Production deletion is not available; the requested production policy is to disallow it. - Raw captures are private local files, but have no automatic retention limit. They can contain sensitive content. The local collector and sync processes do not restart automatically. @@ -91,10 +89,6 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. ## Provider OpenTelemetry -Configure an exporter with `aidd telemetry otel init --config=<claude|codex|opencode> --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. OpenCode v1/v2 capture support does not yet imply OpenCode-to-Gouvernail sync. +Configure an exporter with `aidd telemetry otel init --config=<claude|codex|opencode> --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. -Start OpenCode v1 with `aidd telemetry otel run --config=opencode` to supply its endpoint and token; pass OpenCode arguments after `--`. For OpenCode v2, restart its service after initialization. - -Native Codex project metadata requires the separate `--include-session-projects` opt-in. It transmits the verified native project ID and name, session/provider identity and observation time; names may contain sensitive text. No path-based project inference is made. See [the observation contract](OBSERVABILITY.md) for consent, absence handling and the metadata-only command. - -Codex Git repository metadata requires its own `--include-session-repositories` opt-in. It sends a verified, sanitized GitHub or GitLab repository link, session/provider identity and observation time. Repository names can reveal private work. Local paths, credentials and unsupported hosts are not sent. Repositories and native Codex projects are distinct. +Restart the OpenCode v2 service after initialization. Receipt of OTLP logs or traces does not by itself establish that session identifiers or token usage were exported. From c9866cd9f346c720d0c07bf7cdeb4f63423526ee Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 10:01:09 +0800 Subject: [PATCH 30/56] refactor(telemetry): remove unused session comparator --- .../telemetry/domain/report/session-feature.ts | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/cli/src/contexts/telemetry/domain/report/session-feature.ts b/cli/src/contexts/telemetry/domain/report/session-feature.ts index 4d7f0cea4..4cab286ec 100644 --- a/cli/src/contexts/telemetry/domain/report/session-feature.ts +++ b/cli/src/contexts/telemetry/domain/report/session-feature.ts @@ -29,16 +29,6 @@ export function sameSessionAdmission( ); } -export function sameSessionFeature(left: SessionFeature, right: SessionFeature): boolean { - return ( - left.state === right.state && - (left.state !== "confirmed" || - (right.state === "confirmed" && - left.feature === right.feature && - left.branch === right.branch)) - ); -} - // biome-ignore lint/suspicious/noControlCharactersInRegex: control characters cannot name a branch const CONTROL = /[\u0000-\u001f\u007f]/u; From 1517d40ca5d6e55599c989870c7d8a616a93e148 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 10:21:57 +0800 Subject: [PATCH 31/56] feat(telemetry): run isolated provider OTLP sessions --- aidd_docs/recipes/observe-agent-telemetry.md | 49 +- cli/scripts/mutation-shards.d.mts | 19 +- cli/scripts/mutation-shards.mjs | 24 +- cli/scripts/run-mutation.mjs | 2 +- cli/scripts/smoke-tools.sh | 8 +- .../application/directory-resolver.ts | 32 +- .../application/ingest-usage-use-case.ts | 8 +- .../report/report-usage-use-case.ts | 20 +- .../report/repository-uncertainty.ts | 61 - .../application/report/session-consent.ts | 102 -- .../report/session-feature-use-case.ts | 65 -- .../application/switch/remember-own-root.ts | 25 +- .../switch/telemetry-off-use-case.ts | 12 +- .../telemetry/domain/report/attribution.ts | 2 - .../domain/report/session-feature.ts | 82 -- .../telemetry/domain/report/usage-report.ts | 2 +- .../telemetry/domain/repository-resolution.ts | 120 +- .../contexts/telemetry/domain/stored-usage.ts | 3 - .../infrastructure/otel/actions/run.ts | 315 +++++ .../infrastructure/otel/actions/sync-all.ts | 535 --------- .../infrastructure/otel/actions/sync.ts | 10 - cli/src/presentation/commands/telemetry.ts | 146 +-- .../telemetry/telemetry-lifecycle-display.ts | 5 +- cli/src/runtime/wiring/telemetry-otel.ts | 52 +- cli/src/runtime/wiring/telemetry.ts | 21 +- .../architecture/mutation-shards.arch.test.ts | 48 +- .../ingest-usage-use-case.unit.test.ts | 38 +- ...age-migration.property.integration.test.ts | 18 +- .../left-out.property.integration.test.ts | 3 +- ...eport-figures.property.integration.test.ts | 3 +- ...eport-filters.property.integration.test.ts | 3 +- .../report/report-usage-use-case.unit.test.ts | 65 +- ...report-window.property.integration.test.ts | 9 +- .../report/session-consent.unit.test.ts | 273 ----- .../session-feature-use-case.unit.test.ts | 144 --- .../telemetry-switch-use-cases.unit.test.ts | 2 +- .../domain/repository-resolution.unit.test.ts | 23 +- .../domain/session-feature.unit.test.ts | 86 -- .../telemetry/otel/run-otel.unit.test.ts | 58 + .../otel/sync-all-otel.integration.test.ts | 1029 ----------------- .../sync-otel-attributes.integration.test.ts | 73 +- .../sync-otel-integrity.integration.test.ts | 63 +- .../telemetry-lifecycle.integration.test.ts | 6 +- cli/tests/e2e/telemetry-lifecycle.e2e.test.ts | 11 - cli/tests/e2e/telemetry-report-sandbox.ts | 13 - cli/tests/e2e/telemetry-report.e2e.test.ts | 16 - cli/tests/golden/snapshots/help/surface.json | 6 +- .../telemetry-wiring.integration.test.ts | 147 +-- .../runtime/wiring/otel-wiring.unit.test.ts | 50 +- plugins/aidd-telemetry/OBSERVABILITY.md | 4 +- plugins/aidd-telemetry/README.md | 18 +- .../skills/00-init/actions/04-stop.md | 4 +- .../skills/00-init/actions/05-forget.md | 2 +- .../skills/00-init/actions/06-otel-init.md | 4 +- .../__tests__/aidd-telemetry-hooks.test.js | 10 +- 55 files changed, 627 insertions(+), 3322 deletions(-) delete mode 100644 cli/src/contexts/telemetry/application/report/repository-uncertainty.ts delete mode 100644 cli/src/contexts/telemetry/application/report/session-consent.ts delete mode 100644 cli/src/contexts/telemetry/application/report/session-feature-use-case.ts delete mode 100644 cli/src/contexts/telemetry/domain/report/session-feature.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts delete mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts delete mode 100644 cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts delete mode 100644 cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts delete mode 100644 cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts create mode 100644 cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts delete mode 100644 cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index 3ca4380e3..eae21f6c2 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -1,54 +1,53 @@ # Observe agent telemetry locally -This procedure captures provider OpenTelemetry (OTel) for Claude Code, Codex and OpenCode v2. The framework's separate local usage ledger is not the source for the Gouvernail OTel view. +Run Claude Code, Codex and OpenCode v2 with provider OpenTelemetry (OTel), then inspect the local Gouvernail dashboard. The framework's separate local usage ledger is not the source for this view. ## Steps to observe agent telemetry locally -### 1) โš™๏ธ Configure +### 1) โš™๏ธ Prepare the collector destination -Obtain consent from people whose activity will be observed. Raw local captures can contain prompts, responses, commands, tool output and secrets; restrict access and decide when to delete them. +Start local Gouvernail and configure each tool's connector. OpenCode's connector is generated in the Sources page for an explicit team. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. -Start the private development collector first, then configure each tool: +| Tool | Connector token | +| --- | --- | +| Claude Code | Claude Code OTLP connector | +| Codex | Codex OTLP connector | +| OpenCode v2 | OpenCode OTLP connector for the selected team | -```sh -aidd telemetry otel init --config=claude --endpoint=dev -aidd telemetry otel init --config=codex --endpoint=dev -aidd telemetry otel init --config=opencode --endpoint=dev -``` - -`dev` requires the existing loopback collector and its private token. `prod` is reserved and refuses configuration until a destination is defined. Restart the OpenCode v2 service after initialization so its process receives the exporter settings. +`--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. ### 2) ๐Ÿงช Run two sessions per tool -Run these from the opted-in project. The examples are the commands used in the local observation, with the isolated test settings omitted. They create separate sessions rather than two turns in the same session. +Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each pair. Choose models available to your accounts. ```sh +# Claude connector token must be set before this pair. for i in 1 2; do - claude --model haiku --tools "" --output-format json -p "Reply with only GOUVERNAIL-CLAUDE-PROBE-$i." + aidd telemetry otel run --config claude --model haiku --tenant ai-driven-dev \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture -- "Reply with only CLAUDE-$i." done +# Codex connector token must be set before this pair. for i in 1 2; do - codex exec --ephemeral --sandbox read-only --model gpt-6.1-terra \ - -c 'model_reasoning_effort="low"' --json "Reply with only GOUVERNAIL-CODEX-PROBE-$i." + aidd telemetry otel run --config codex --model gpt-5.6-luna --tenant ai-driven-dev \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture -- "Reply with only CODEX-$i." done -# Start the configured OpenCode v2 service separately with authentication enabled. +# OpenCode connector token must be set before this pair. for i in 1 2; do - opencode run --server http://127.0.0.1:51394 --format json \ - --model openai/gpt-6-luna "Reply with only GOUVERNAIL-OPENCODE-PROBE-$i." + aidd telemetry otel run --config opencode --model claude-code/claude-haiku-4-5-20251001 \ + --tenant ai-driven-dev --endpoint http://127.0.0.1:55857 --yes --keep-capture -- "Reply with only OPENCODE-$i." done ``` -The OpenCode example assumes the authenticated service is listening on port `51394` and the client has its credentials. Choose models available to the configured accounts. Claude and Codex emit token usage; in the observed OpenCode v2.2.5 capture, prompt spans contained session IDs but no model or token counts. See the [field inventory](../../plugins/aidd-telemetry/OBSERVABILITY.md). +Each command launches a separate session with an isolated collector. It does not read the framework usage ledger or establish a project or feature association. `otel init` and the global development collector are a separate configuration path; they are not required for these commands. ### 3) โœ… Check receipt +Open the local AI KPI page and check the session IDs and observed fields. Delivery does not prove that every provider event was emitted or received. In the OpenCode v2.2.5 capture, HTTP prompt spans included session IDs but no model or token counts. Missing or uncertain values remain unavailable. See the [field inventory](../../plugins/aidd-telemetry/OBSERVABILITY.md). + ```sh -aidd telemetry otel verify --config=claude --endpoint=dev -aidd telemetry otel verify --config=codex --endpoint=dev -aidd telemetry otel verify --config=opencode --endpoint=dev +open 'http://localhost:55857/enterprise/ai-driven-dev/kpi/ai' ``` -Verification reports capture line counts and modification times. Old records do not prove delivery from the current sessions. Collection and consent-aware forwarding to Gouvernail are separate; capture alone does not create dashboard rows. - -Sources accessed 2026-11-06: [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage), [Codex telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted), [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts). +Sources accessed 2026-11-06: [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage), [Codex telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted), [OpenCode v2 CLI](https://dev.opencode.ai/v2/docs/cli) and [OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts). diff --git a/cli/scripts/mutation-shards.d.mts b/cli/scripts/mutation-shards.d.mts index 27b1519a6..f36d2e72b 100644 --- a/cli/scripts/mutation-shards.d.mts +++ b/cli/scripts/mutation-shards.d.mts @@ -21,19 +21,18 @@ export function shardFiles( of: number, weightOf?: WeightOf ): string[]; -interface MutationShard { - readonly manifest: { - readonly shard: number; - readonly of: number; - readonly files: readonly string[]; - }; - readonly report: MutationReport; -} -export function mergeReports(shards: readonly MutationShard[]): Required<MutationReport>; +export function mergeReports(reports: readonly MutationReport[]): Required<MutationReport>; export function aggregateVerdict(input: { readonly mutate: Mutate; readonly files: readonly string[]; readonly of: number; - readonly shards: readonly MutationShard[]; + readonly shards: readonly { + readonly manifest: { + readonly shard: number; + readonly of: number; + readonly files: readonly string[]; + }; + readonly report: MutationReport; + }[]; readonly weightOf?: WeightOf; }): string | null; diff --git a/cli/scripts/mutation-shards.mjs b/cli/scripts/mutation-shards.mjs index 337258a57..749208029 100644 --- a/cli/scripts/mutation-shards.mjs +++ b/cli/scripts/mutation-shards.mjs @@ -72,16 +72,9 @@ export function shardFiles(mutate, files, shard, of, weightOf = linesOf) { return partition(mutate, files, of, weightOf)[shard - 1]; } -/** Score only the files assigned to each shard, never stale entries from an incremental cache. */ -export function mergeReports(shards) { - const files = {}; - for (const shard of shards) { - for (const path of shard.manifest.files) { - const report = shard.report.files?.[path]; - if (report !== undefined) files[path] = report; - } - } - return { files }; +/** The shards' reports as one: each file is in exactly one shard, so nothing overlaps. */ +export function mergeReports(reports) { + return { files: Object.assign({}, ...reports.map((report) => report.files ?? {})) }; } /** Why a sharded scope cannot be scored, or null. Every shard must be there, each must have @@ -89,10 +82,8 @@ export function mergeReports(shards) { * mutated: a missing or drifted shard would lift the score by leaving mutants out. */ export function aggregateVerdict({ mutate, files, of, shards, weightOf = linesOf }) { for (let shard = 1; shard <= of; shard += 1) { - const matches = shards.filter((candidate) => candidate.manifest.shard === shard); - if (matches.length === 0) return `shard ${shard} of ${of} is missing`; - if (matches.length > 1) return `shard ${shard} of ${of} was supplied more than once`; - const found = matches[0]; + const found = shards.find((candidate) => candidate.manifest.shard === shard); + if (found === undefined) return `shard ${shard} of ${of} is missing`; if (found.manifest.of !== of) return `shard ${shard} was cut in ${found.manifest.of}, not ${of}`; const expected = shardFiles(mutate, files, shard, of, weightOf); @@ -101,7 +92,8 @@ export function aggregateVerdict({ mutate, files, of, shards, weightOf = linesOf return `shard ${shard} of ${of} mutated other files than the partition gives it`; } } - if (shards.length !== of) return `unexpected shard outside 1 to ${of}`; - const mutants = Object.values(mergeReports(shards).files).flatMap((file) => file.mutants).length; + const mutants = Object.values(mergeReports(shards.map((s) => s.report)).files).flatMap( + (file) => file.mutants + ).length; return mutants === 0 ? "the shards hold no mutant" : null; } diff --git a/cli/scripts/run-mutation.mjs b/cli/scripts/run-mutation.mjs index 25d04bb89..ccb9496c9 100644 --- a/cli/scripts/run-mutation.mjs +++ b/cli/scripts/run-mutation.mjs @@ -276,7 +276,7 @@ function mainAggregate(scope, scopes, from) { console.error(`${scope}: ${problem}`); process.exit(1); } - const merged = mergeReports(shards); + const merged = mergeReports(shards.map((shard) => shard.report)); const score = scoreOf(merged); console.log( `\nReport: ${scope} over ${shards.length} shards (score ${score.toFixed(1)}, floor ${declared.break})` diff --git a/cli/scripts/smoke-tools.sh b/cli/scripts/smoke-tools.sh index 37d733215..f3491dce4 100755 --- a/cli/scripts/smoke-tools.sh +++ b/cli/scripts/smoke-tools.sh @@ -30,7 +30,7 @@ ALL_COMMANDS=( "auth login" "auth logout" "auth status" "telemetry on" "telemetry off" "telemetry forget" "telemetry ingest" "telemetry task" "telemetry status" "telemetry report" "telemetry identity" - "telemetry otel init" "telemetry otel verify" "telemetry otel sync" "telemetry otel collect" + "telemetry otel init" "telemetry otel verify" "telemetry otel run" "telemetry otel collect" ) PASS=0; FAIL=0; SKIP=0 @@ -228,7 +228,7 @@ run "telemetry forget (preview)" 0 "Nothing was removed" "$P_TEL" -- telemetry f run "telemetry forget --yes" 0 "Removed" "$P_TEL" -- telemetry forget --yes run "telemetry off (after forget)" 0 "Measurement was not on" "$P_TEL" -- telemetry off -section "native telemetry (sandboxed profile)" +section "provider OTEL (sandboxed profile)" OTEL_CAPTURE="$HOME/.local/share/aidd-otel-observation" mkdir -p "$OTEL_CAPTURE" "$TMPROOT/otel-bin" chmod 700 "$OTEL_CAPTURE" @@ -287,8 +287,8 @@ unset OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS run "telemetry otel init (OpenCode 2)" 0 "Restart the OpenCode background service" "$P_TEL" -- \ telemetry otel init --config=opencode --endpoint=dev unset AIDD_OTEL_TOKEN -run "telemetry otel sync (no consented session)" 1 "No consented usage found" "$P_TEL" -- \ - telemetry otel sync --config=codex --session=missing --tenant=smoke --endpoint=http://127.0.0.1:4318 +run "telemetry otel run (no execution consent)" 1 "Pass --yes to consent" "$P_TEL" -- \ + telemetry otel run --config=codex --tenant=smoke --endpoint=http://127.0.0.1:4318 -- "reply OK" export PATH="$SMOKE_ORIGINAL_PATH" section "update --check" diff --git a/cli/src/contexts/telemetry/application/directory-resolver.ts b/cli/src/contexts/telemetry/application/directory-resolver.ts index ab7f83511..220a69918 100644 --- a/cli/src/contexts/telemetry/application/directory-resolver.ts +++ b/cli/src/contexts/telemetry/application/directory-resolver.ts @@ -7,10 +7,10 @@ import { repositoryIdOf } from "../domain/repository-identity.js"; import { cwdKey, type NotStoredReason, - ownersAt, + ownerAt, type RepositoryResolution, - rememberResolution, resolutionKey, + sameResolution, } from "../domain/repository-resolution.js"; import { tokenOfKey } from "../domain/telemetry-consent.js"; import { ConsentLog } from "./consent-log.js"; @@ -140,13 +140,7 @@ export class ResolutionRun { const directory = await this.directoryOf(cwd); if ("skipped" in directory) return directory; const instant = Date.parse(at); - const candidates = ownersAt(directory.owners, instant); - // Equal birth times cannot establish which clone made the call. Consent alone - // cannot identify the owner, so leave it pending rather than guessing. - if (candidates.length !== 1) return { skipped: "unreadable-consent" }; - const selected = candidates[0] as RepositoryResolution; - // Another directory in this run may have revealed a changed origin for this clone. - const owner = this.remembered.get(resolutionKey(selected.dir, selected.clone)) ?? selected; + const owner = ownerAt(directory.owners, instant); if (this.unreadable.has(cloneKey(owner.clone))) return { skipped: "unreadable-consent" }; const live = directory.current !== null && sameClone(directory.current.clone, owner.clone); if (!this.consents.covers(owner.clone, instant)) { @@ -202,13 +196,19 @@ export class ResolutionRun { root: string, clone: CloneIdentity ): RepositoryResolution { - const result = rememberResolution( - this.remembered, - { dir, repository_id: repositoryId, root, clone }, - this.environment.now().toISOString() - ); - if (result.changed) this.changed = true; - return result.resolution; + const key = resolutionKey(dir, clone); + const held = this.remembered.get(key); + const seen = { + dir, + repository_id: repositoryId, + root, + clone, + seen_at: held?.seen_at ?? this.environment.now().toISOString(), + }; + if (sameResolution(held, seen)) return seen; + this.remembered.set(key, seen); + this.changed = true; + return seen; } private ownersOf(dir: string): RepositoryResolution[] { diff --git a/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts b/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts index 0d65c0710..2cb1fef99 100644 --- a/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts +++ b/cli/src/contexts/telemetry/application/ingest-usage-use-case.ts @@ -1,6 +1,6 @@ import { isRequestless, requestlessCallOf } from "../domain/formats/claude-transcript-usage.js"; import type { LeftOutEntry, UsageLedger } from "../domain/ports/usage-ledger.js"; -import { type NotStoredReason, repositoryUncertain } from "../domain/repository-resolution.js"; +import type { NotStoredReason } from "../domain/repository-resolution.js"; import { dayOf, type StoredUsage } from "../domain/stored-usage.js"; import type { TranscriptPosition } from "../domain/transcript-position.js"; import type { UsageRecord } from "../domain/usage-record.js"; @@ -149,11 +149,7 @@ export class IngestUsageUseCase { leftOut.push({ reason: "requestless", of: requestlessCallOf(record) }); continue; } - toStore.push({ - ...record, - repository_id: outcome.stored.repository_id, - ...(repositoryUncertain(outcome.stored, record.at) ? { repository_uncertain: true } : {}), - }); + toStore.push({ ...record, repository_id: outcome.stored.repository_id }); if (outcome.live) live.set(outcome.stored.repository_id, outcome.stored.root); } diff --git a/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts b/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts index c8b277a63..7baf36e2e 100644 --- a/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts +++ b/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts @@ -1,6 +1,6 @@ import type { PersonIdentityStore } from "../../domain/ports/identity/person-identity-store.js"; import type { RepositoryLocator } from "../../domain/ports/repository-locator.js"; -import type { LeftOutReason } from "../../domain/ports/usage-ledger.js"; +import type { LeftOutReason, UsageLedger } from "../../domain/ports/usage-ledger.js"; import { attribute } from "../../domain/report/attribution.js"; import { inPeriod, type Period, windowOf } from "../../domain/report/period.js"; import { @@ -20,8 +20,8 @@ import { import { repositoryIdOf } from "../../domain/repository-identity.js"; import type { StoredUsage } from "../../domain/stored-usage.js"; import type { IngestResult, IngestUsageUseCase } from "../ingest-usage-use-case.js"; +import { settleLedger } from "../ledger-upkeep.js"; import type { DeclaredBindings } from "./declared-bindings.js"; -import type { ReportLedger } from "./repository-uncertainty.js"; /** The filters as typed: a session or repository may be the start of an id. A repository is * either typed, or `--repository .`, the directory `here` whose repository is asked for; never @@ -105,7 +105,7 @@ export class ReportUsageUseCase { constructor( /** Only its run is used: whatever it found is the report's `run`, never a figure. */ private readonly ingest: Pick<IngestUsageUseCase, "execute">, - private readonly ledger: ReportLedger, + private readonly ledger: UsageLedger, private readonly declared: DeclaredBindings, private readonly identity: PersonIdentityStore, private readonly locator: RepositoryLocator, @@ -123,8 +123,10 @@ export class ReportUsageUseCase { if (located === null) return { status: "not-a-repository" }; const { filesRead, unrecognised, notStored, consentLogDamaged, oldestTranscriptAt } = await this.ingest.execute(); - const held = await this.ledger.read(windowOf(request.period), undefined, true); - const leftOut = held.leftOut as Readonly<Record<LeftOutReason, number>>; + const [held, leftOut] = await this.ledger.exclusively(async () => [ + await settleLedger(this.ledger, windowOf(request.period)), + await this.ledger.leftOut.counts(), + ]); const records = held.records.filter((record) => inPeriod(record.at, request.period)); const session = resolvedIn(records, "session", asked.session); if ("status" in session) return session; @@ -172,13 +174,7 @@ function resolvedIn( if (typed === null) return { id: null }; const found = resolveId( typed, - records.flatMap((record) => - flag === "session" - ? [record.session_id] - : record.repository_uncertain - ? [] - : [record.repository_id] - ) + records.map((record) => (flag === "session" ? record.session_id : record.repository_id)) ); if (found.kind === "ambiguous") return { status: "ambiguous", flag, typed, ids: found.ids }; return { id: found.kind === "found" ? found.id : typed }; diff --git a/cli/src/contexts/telemetry/application/report/repository-uncertainty.ts b/cli/src/contexts/telemetry/application/report/repository-uncertainty.ts deleted file mode 100644 index 67bb42b87..000000000 --- a/cli/src/contexts/telemetry/application/report/repository-uncertainty.ts +++ /dev/null @@ -1,61 +0,0 @@ -import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; -import { repositoryUncertain } from "../../domain/repository-resolution.js"; -import type { StoredUsage } from "../../domain/stored-usage.js"; -import type { UsageTool } from "../../domain/usage-record.js"; -import type { DirectoryResolver } from "../directory-resolver.js"; -import { settleLedger } from "../ledger-upkeep.js"; - -/** Settles and annotates a report's ledger under the same lock. */ -export class ReportLedger { - constructor( - private readonly ledger: UsageLedger, - private readonly directories?: Pick<DirectoryResolver, "open"> - ) {} - - async read(holds?: (day: string) => boolean, tool?: UsageTool, includeLeftOut = false) { - return this.ledger.exclusively(async () => { - const held = await settleLedger(this.ledger, holds); - const selected = tool ? held.records.filter((record) => record.tool === tool) : held.records; - const records = this.directories - ? await withRepositoryUncertainty(selected, this.directories) - : selected.map((record) => ({ ...record, repository_uncertain: true as const })); - const leftOut = includeLeftOut ? await this.ledger.leftOut.counts() : undefined; - return { records, skippedLines: held.skippedLines, leftOut }; - }); - } -} - -/** Annotates stored usage without changing token counts or rewriting the ledger. - * The caller holds the ledger lock while the resolver observes and saves changes. */ -export async function withRepositoryUncertainty( - records: readonly StoredUsage[], - directories: Pick<DirectoryResolver, "open"> -): Promise<readonly StoredUsage[]> { - const run = await directories.open(); - try { - // Observe every directory before attributing any record: another working tree may - // reveal a changed origin for a clone already visited in this same run. - const observed = new Set<string>(); - for (const record of records) { - if (record.cwd && !observed.has(record.cwd)) { - observed.add(record.cwd); - await run.resolve(record.cwd, record.at); - } - } - const annotated: StoredUsage[] = []; - for (const record of records) { - const outcome = record.cwd ? await run.resolve(record.cwd, record.at) : null; - const uncertain = - record.repository_uncertain === true || - !outcome || - "skipped" in outcome || - !outcome.live || - outcome.stored.repository_id !== record.repository_id || - repositoryUncertain(outcome.stored, record.at); - annotated.push(uncertain ? { ...record, repository_uncertain: true } : record); - } - return annotated; - } finally { - await run.close(); - } -} diff --git a/cli/src/contexts/telemetry/application/report/session-consent.ts b/cli/src/contexts/telemetry/application/report/session-consent.ts deleted file mode 100644 index 98692c5b1..000000000 --- a/cli/src/contexts/telemetry/application/report/session-consent.ts +++ /dev/null @@ -1,102 +0,0 @@ -import { createHash } from "node:crypto"; -import { cloneKey, sameClone } from "../../domain/consent/clone-identity.js"; -import { foldConsent } from "../../domain/consent/consent-history.js"; -import type { ConsentHistory } from "../../domain/ports/consent-history.js"; -import type { UsageLedger } from "../../domain/ports/usage-ledger.js"; -import { coversConsentWindow, type SessionAdmission } from "../../domain/report/session-feature.js"; -import type { DirectoryResolver } from "../directory-resolver.js"; - -export class SessionConsent { - constructor( - private readonly ledger: UsageLedger, - private readonly directories: DirectoryResolver, - private readonly consents: ConsentHistory - ) {} - - /** Cumulative metrics are permitted only when their complete interval belongs to the same - * clone and one uninterrupted consent interval. */ - async permits( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ): Promise<"allow" | "deny" | "unknown"> { - return (await this.assess(admission, at, start, current)).verdict; - } - - /** A private fingerprint of the consented clone, only after the entire interval is verified. */ - async proof( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ): Promise<string | null> { - return (await this.assess(admission, at, start, current)).clone ?? null; - } - - private async assess( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ): Promise<{ verdict: "allow" | "deny" | "unknown"; clone?: string }> { - const endTime = Date.parse(at); - const startTime = start === undefined ? endTime : Date.parse(start); - const currentTime = current === undefined ? endTime : Date.parse(current); - if ( - !admission.cwd || - (!admission.repositoryId && - !(admission.repositoryId === null && admission.feature.state !== "confirmed")) || - !Number.isFinite(endTime) || - !Number.isFinite(startTime) || - !Number.isFinite(currentTime) || - startTime > endTime || - currentTime < endTime - ) - return { verdict: "unknown" }; - return this.ledger.exclusively(async () => { - const run = await this.directories.open(); - try { - const first = await run.resolve(admission.cwd ?? "", new Date(startTime).toISOString()); - const last = await run.resolve(admission.cwd ?? "", new Date(endTime).toISOString()); - const now = - current === undefined - ? last - : await run.resolve(admission.cwd ?? "", new Date(currentTime).toISOString()); - if ("skipped" in first || "skipped" in last || "skipped" in now) { - const reasons = [first, last, now].flatMap((value) => - "skipped" in value ? [value.skipped] : [] - ); - return { - verdict: reasons.some( - (reason) => reason === "unreadable-consent" || reason === "never-seen-alive" - ) - ? "unknown" - : "deny", - }; - } - if ( - !sameClone(first.stored.clone, last.stored.clone) || - !sameClone(first.stored.clone, now.stored.clone) || - (admission.repositoryId !== null && - (first.stored.repository_id !== admission.repositoryId || - last.stored.repository_id !== admission.repositoryId || - now.stored.repository_id !== admission.repositoryId)) - ) - return { verdict: "deny" }; - const history = await this.consents.read(); - if (history.damaged) return { verdict: "unknown" }; - const intervals = foldConsent(history.events); - return coversConsentWindow(intervals, first.stored.clone, startTime, endTime) && - coversConsentWindow(intervals, first.stored.clone, currentTime, currentTime) - ? { - verdict: "allow", - clone: createHash("sha256").update(cloneKey(first.stored.clone)).digest("hex"), - } - : { verdict: "deny" }; - } finally { - await run.close(); - } - }); - } -} diff --git a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts b/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts deleted file mode 100644 index c69f0f804..000000000 --- a/cli/src/contexts/telemetry/application/report/session-feature-use-case.ts +++ /dev/null @@ -1,65 +0,0 @@ -import { - type SessionAdmission, - type SessionFeature, - sessionFeatureOf, -} from "../../domain/report/session-feature.js"; -import type { StoredUsage } from "../../domain/stored-usage.js"; -import type { UsageTool } from "../../domain/usage-record.js"; -import type { IngestUsageUseCase } from "../ingest-usage-use-case.js"; -import type { DeclaredBindings } from "./declared-bindings.js"; -import type { ReportLedger } from "./repository-uncertainty.js"; -import type { SessionConsent } from "./session-consent.js"; - -/** Only usage already admitted by the clone's consent can be forwarded. */ -export class SessionFeatureUseCase { - constructor( - private readonly ingest: Pick<IngestUsageUseCase, "execute">, - private readonly ledger: ReportLedger, - private readonly declared: Pick<DeclaredBindings, "read">, - private readonly consent: Pick<SessionConsent, "permits"> & - Partial<Pick<SessionConsent, "proof">> - ) {} - - async snapshot(tool: UsageTool): Promise<Map<string, SessionAdmission>> { - if ((await this.ingest.execute()).refused) - throw new Error("AIDD telemetry is disabled; no capture was sent."); - const { records } = await this.ledger.read(undefined, tool); - const facts = await this.declared.read(); - const sessions = new Map<string, StoredUsage[]>(); - for (const record of records) { - const group = sessions.get(record.session_id) ?? []; - group.push(record); - sessions.set(record.session_id, group); - } - return new Map( - [...sessions].map(([id, records]) => { - const directories = new Set(records.map((record) => record.cwd)); - const repositories = new Set(records.map((record) => record.repository_id)); - const uncertain = - repositories.size !== 1 || records.some((record) => record.repository_uncertain); - return [ - id, - { - feature: uncertain ? { state: "ambiguous" as const } : sessionFeatureOf(records, facts), - cwd: directories.size === 1 ? records[0].cwd : null, - repositoryId: uncertain ? null : records[0].repository_id, - }, - ]; - }) - ); - } - - async execute(session: string, tool: UsageTool): Promise<SessionFeature> { - const found = (await this.snapshot(tool)).get(session); - if (!found) throw new Error("No consented usage found for this session; no capture was sent."); - return found.feature; - } - - async permits(admission: SessionAdmission, at: string, start?: string, current?: string) { - return this.consent.permits(admission, at, start, current); - } - - async cloneProof(admission: SessionAdmission, at: string, start?: string, current?: string) { - return this.consent.proof?.(admission, at, start, current) ?? null; - } -} diff --git a/cli/src/contexts/telemetry/application/switch/remember-own-root.ts b/cli/src/contexts/telemetry/application/switch/remember-own-root.ts index a517789af..cef802429 100644 --- a/cli/src/contexts/telemetry/application/switch/remember-own-root.ts +++ b/cli/src/contexts/telemetry/application/switch/remember-own-root.ts @@ -1,7 +1,12 @@ import type { LocatedDirectory } from "../../domain/ports/repository-locator.js"; import type { ResolutionStore } from "../../domain/ports/resolution-store.js"; import { repositoryIdOf } from "../../domain/repository-identity.js"; -import { cwdKey, rememberResolution } from "../../domain/repository-resolution.js"; +import { + cwdKey, + type RepositoryResolution, + resolutionKey, + sameResolution, +} from "../../domain/repository-resolution.js"; type Located = Extract<LocatedDirectory, { status: "repository" }>; @@ -19,10 +24,16 @@ export async function rememberOwnRoot( if (id === null || located.clone === null) return; const held = await store.load(); const dir = cwdKey(located.root, caseInsensitive); - const result = rememberResolution( - held, - { dir, repository_id: id, root: located.root, clone: located.clone }, - at.toISOString() - ); - if (result.changed) await store.save(held); + const key = resolutionKey(dir, located.clone); + const earlier = held.get(key); + const own: RepositoryResolution = { + dir, + repository_id: id, + root: located.root, + clone: located.clone, + seen_at: earlier?.seen_at ?? at.toISOString(), + }; + if (sameResolution(earlier, own)) return; + held.set(key, own); + await store.save(held); } diff --git a/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts b/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts index 3b4828914..212e4b64d 100644 --- a/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts +++ b/cli/src/contexts/telemetry/application/switch/telemetry-off-use-case.ts @@ -13,7 +13,7 @@ export type OffResult = readonly status: "refused"; readonly reason: "outside-repository" | "unreadable-git-config" | "damaged-consent-log"; } - | { readonly status: "off"; readonly changed: boolean; readonly closedAt?: string }; + | { readonly status: "off"; readonly changed: boolean }; /** Stops measuring this clone: its interval ends now, and the calls it made while on stay * stored, whenever the next ingest runs. What was already measured stays until `forget`. */ @@ -33,7 +33,7 @@ export class TelemetryOffUseCase { const { located } = clone; const token = tokenOfKey(clone.value); const granted = token !== null; - let closedAt: string | undefined; + let closed = false; let damaged = false; // The lock first: an ingest running now must see the key and the interval change together. await this.ledger.exclusively(async () => { @@ -58,16 +58,12 @@ export class TelemetryOffUseCase { ); for (const interval of ours) { await log.close(interval.token, at); - closedAt = at.toISOString(); + closed = true; } } if (granted) await this.writer.set(located.root, CONSENT_WITHDRAWN); }); if (damaged) return { status: "refused", reason: "damaged-consent-log" }; - return { - status: "off", - changed: granted || closedAt !== undefined, - ...(closedAt ? { closedAt } : {}), - }; + return { status: "off", changed: granted || closed }; } } diff --git a/cli/src/contexts/telemetry/domain/report/attribution.ts b/cli/src/contexts/telemetry/domain/report/attribution.ts index 18cb8754c..d19a75f43 100644 --- a/cli/src/contexts/telemetry/domain/report/attribution.ts +++ b/cli/src/contexts/telemetry/domain/report/attribution.ts @@ -26,7 +26,6 @@ export interface AttributableUsage { readonly at: string; readonly git_branch: string | null; readonly repository_id: string; - readonly repository_uncertain?: true; } const NO_BINDING: Attribution = { state: "unattributed", reason: "no-binding" }; @@ -154,7 +153,6 @@ function branchVerdict( * session's declaration, else the session's carry, else the branch's declaration, else no * task. Sub-agent and advisor calls name their parent's session, so they follow it. */ export function attribute(record: AttributableUsage, facts: AttributionFacts): Attribution { - if (record.repository_uncertain) return NO_BINDING; const at = Date.parse(record.at); return sessionVerdict(record, at, facts) ?? branchVerdict(record, at, facts) ?? NO_BINDING; } diff --git a/cli/src/contexts/telemetry/domain/report/session-feature.ts b/cli/src/contexts/telemetry/domain/report/session-feature.ts deleted file mode 100644 index 4cab286ec..000000000 --- a/cli/src/contexts/telemetry/domain/report/session-feature.ts +++ /dev/null @@ -1,82 +0,0 @@ -import { type CloneIdentity, sameClone } from "../consent/clone-identity.js"; -import type { ConsentInterval } from "../consent/consent-history.js"; -import type { StoredUsage } from "../stored-usage.js"; -import { type AttributionFacts, attribute } from "./attribution.js"; - -export type SessionFeature = - | { readonly state: "unbound" | "ambiguous" } - | { readonly state: "confirmed"; readonly feature: string; readonly branch: string }; - -export interface SessionAdmission { - readonly feature: SessionFeature; - readonly cwd: string | null; - readonly repositoryId: string | null; -} - -/** A session cannot move to another clone or repository during a pending delivery. */ -export function sameSessionAdmission( - expected: SessionAdmission | undefined, - current: SessionAdmission | undefined -): boolean { - return Boolean( - expected?.cwd && - current?.cwd === expected.cwd && - current.repositoryId === expected.repositoryId && - (expected.repositoryId || - (expected.repositoryId === null && - expected.feature.state !== "confirmed" && - current.feature.state !== "confirmed")) - ); -} - -// biome-ignore lint/suspicious/noControlCharactersInRegex: control characters cannot name a branch -const CONTROL = /[\u0000-\u001f\u007f]/u; - -/** A cumulative point may only cross time covered by one uninterrupted consent interval. */ -export function coversConsentWindow( - intervals: readonly ConsentInterval[], - clone: CloneIdentity, - start: number, - end: number -): boolean { - return ( - start <= end && - intervals.some( - (interval) => - sameClone(interval.clone, clone) && - interval.from <= start && - (interval.to === null || end < interval.to) - ) - ); -} - -/** A whole session is linked only when every recorded call agrees on an explicitly declared - * canonical ticket and an observed branch. An unknown call is never assigned by proximity. */ -export function sessionFeatureOf( - records: readonly StoredUsage[], - facts: AttributionFacts -): SessionFeature { - let candidate: { feature: string; branch: string } | undefined; - let missing = false; - for (const record of records) { - if (record.repository_uncertain) return { state: "ambiguous" }; - const resolution = attribute(record, facts); - const feature = resolution.state === "attributed" ? resolution.ticket : null; - const branch = record.git_branch; - if ( - !feature || - !/^[^\s/#]+\/[^\s/#]+#[1-9][0-9]*$/u.test(feature) || - !branch || - branch.length > 255 || - CONTROL.test(branch) - ) { - missing = true; - continue; - } - if (candidate && (candidate.feature !== feature || candidate.branch !== branch)) - return { state: "ambiguous" }; - candidate = { feature, branch }; - } - if (!candidate) return { state: "unbound" }; - return missing ? { state: "ambiguous" } : { state: "confirmed", ...candidate }; -} diff --git a/cli/src/contexts/telemetry/domain/report/usage-report.ts b/cli/src/contexts/telemetry/domain/report/usage-report.ts index 7b5d1f5ec..d4335f418 100644 --- a/cli/src/contexts/telemetry/domain/report/usage-report.ts +++ b/cli/src/contexts/telemetry/domain/report/usage-report.ts @@ -116,7 +116,7 @@ export function axisValueOf( case "day": return value(new Date(record.at).toISOString().slice(0, 10)); case "repository": - return value(record.repository_uncertain ? null : record.repository_id); + return value(record.repository_id); case "task": return attribution.state === "attributed" ? value(attribution.task) diff --git a/cli/src/contexts/telemetry/domain/repository-resolution.ts b/cli/src/contexts/telemetry/domain/repository-resolution.ts index 6fc62f15a..c7516a9fb 100644 --- a/cli/src/contexts/telemetry/domain/repository-resolution.ts +++ b/cli/src/contexts/telemetry/domain/repository-resolution.ts @@ -1,11 +1,6 @@ import { tryParseJson } from "../../../kernel/reading/json-file.js"; import { asPlainObject } from "../../../kernel/reading/plain-object.js"; -import { - type CloneIdentity, - cloneKey, - parseCloneIdentity, - sameClone, -} from "./consent/clone-identity.js"; +import { type CloneIdentity, cloneKey, parseCloneIdentity } from "./consent/clone-identity.js"; import { compareText } from "./text-order.js"; /** What a working directory was found to be, the first time it was seen alive in a clone. A @@ -21,8 +16,6 @@ export interface RepositoryResolution { readonly clone: CloneIdentity; /** When ingest first saw the directory in this clone. */ readonly seen_at: string; - /** Before a detected change, the origin of a call cannot be established. */ - readonly repository_uncertain_before?: string | null; } /** The key a directory is remembered under. File systems that ignore case (macOS, Windows) @@ -41,73 +34,7 @@ export function sameResolution( a: RepositoryResolution | undefined, b: RepositoryResolution ): boolean { - return ( - a !== undefined && - a.repository_id === b.repository_id && - a.root === b.root && - a.repository_uncertain_before === b.repository_uncertain_before - ); -} - -/** Remember a detected repository change across all directories in the same clone. */ -export function rememberResolution( - resolutions: Map<string, RepositoryResolution>, - found: Omit<RepositoryResolution, "seen_at" | "repository_uncertain_before">, - at: string -): { readonly resolution: RepositoryResolution; readonly changed: boolean } { - const key = resolutionKey(found.dir, found.clone); - const previous = resolutions.get(key); - const related = [...resolutions.entries()].filter(([, value]) => - sameClone(value.clone, found.clone) - ); - const changedRepository = related.some( - ([, value]) => value.repository_id !== found.repository_id - ); - const boundaries = related.flatMap(([, value]) => - value.repository_uncertain_before ? [value.repository_uncertain_before] : [] - ); - if (changedRepository) boundaries.push(at); - const uncertainBefore = related.some(([, value]) => value.repository_uncertain_before === null) - ? null - : boundaries.sort((a, b) => Date.parse(a) - Date.parse(b)).at(-1); - let changed = false; - if (changedRepository) { - for (const [otherKey, value] of related) { - const replacement = { - ...value, - repository_id: found.repository_id, - repository_uncertain_before: uncertainBefore, - }; - if (!sameResolution(value, replacement)) { - resolutions.set(otherKey, replacement); - changed = true; - } - } - } - const resolution: RepositoryResolution = { - ...found, - // A root observed during opt-in establishes the first observation for its clone, - // including working directories first visited later. - seen_at: - previous?.seen_at ?? - related.map(([, value]) => value.seen_at).sort((a, b) => Date.parse(a) - Date.parse(b))[0] ?? - at, - ...(uncertainBefore !== undefined ? { repository_uncertain_before: uncertainBefore } : {}), - }; - if (!sameResolution(resolutions.get(key), resolution)) { - resolutions.set(key, resolution); - changed = true; - } - return { resolution, changed }; -} - -export function repositoryUncertain(resolution: RepositoryResolution, at: string): boolean { - return ( - Date.parse(at) < Date.parse(resolution.seen_at) || - resolution.repository_uncertain_before === null || - (resolution.repository_uncertain_before !== undefined && - Date.parse(at) <= Date.parse(resolution.repository_uncertain_before)) - ); + return a !== undefined && a.repository_id === b.repository_id && a.root === b.root; } /** The instant from which a clone answered for a directory: its birth, else the time ingest @@ -118,20 +45,22 @@ function answeringFrom(resolution: RepositoryResolution): number { : Date.parse(resolution.seen_at); } -/** The clones that may answer for a call made at `at` in a directory several clones have lived in: - * those latest to have started by then. A call older than all of them goes to the earliest, because +/** The clone that answers for a call made at `at` in a directory several clones have lived in: + * the latest to have started by then. A call older than all of them goes to the first, because * a clone with no birth time is known only from the day ingest first saw a directory in it, and * a call made before that day in a directory of its own is still its call. That is no consent * on its own: the owner's interval must cover the call, and none opens before the clone's `on`. */ -export function ownersAt( +export function ownerAt( owners: readonly [RepositoryResolution, ...RepositoryResolution[]], at: number -): readonly RepositoryResolution[] { - const eligible = owners.filter((owner) => answeringFrom(owner) <= at); - const candidates = eligible.length ? eligible : owners; - const times = candidates.map(answeringFrom); - const selected = eligible.length ? Math.max(...times) : Math.min(...times); - return candidates.filter((owner) => answeringFrom(owner) === selected); +): RepositoryResolution { + const [first, ...later] = [...owners].sort( + (a, b) => + answeringFrom(a) - answeringFrom(b) || compareText(resolutionKeyOf(a), resolutionKeyOf(b)) + ); + let owner = first as RepositoryResolution; + for (const candidate of later) if (answeringFrom(candidate) <= at) owner = candidate; + return owner; } function resolutionKeyOf(resolution: RepositoryResolution): string { @@ -143,31 +72,12 @@ function parseResolution(value: unknown): RepositoryResolution | null { if (object === null) return null; const clone = parseCloneIdentity(object.clone); if (clone === null) return null; - const { - dir, - repository_id: repositoryId, - root, - seen_at: seenAt, - repository_uncertain_before: uncertainBefore, - } = object; + const { dir, repository_id: repositoryId, root, seen_at: seenAt } = object; if (typeof dir !== "string" || dir === "") return null; if (typeof repositoryId !== "string" || repositoryId === "") return null; if (typeof root !== "string") return null; if (typeof seenAt !== "string" || Number.isNaN(Date.parse(seenAt))) return null; - const uncertainty = - uncertainBefore === undefined - ? undefined - : typeof uncertainBefore === "string" && !Number.isNaN(Date.parse(uncertainBefore)) - ? uncertainBefore - : null; - return { - dir, - repository_id: repositoryId, - root, - clone, - seen_at: seenAt, - ...(uncertainty !== undefined ? { repository_uncertain_before: uncertainty } : {}), - }; + return { dir, repository_id: repositoryId, root, clone, seen_at: seenAt }; } /** The format of `roots.json`. A file of another format, among them every file written before diff --git a/cli/src/contexts/telemetry/domain/stored-usage.ts b/cli/src/contexts/telemetry/domain/stored-usage.ts index 62d715bfe..f7181e684 100644 --- a/cli/src/contexts/telemetry/domain/stored-usage.ts +++ b/cli/src/contexts/telemetry/domain/stored-usage.ts @@ -10,8 +10,6 @@ import { USAGE_AGENTS, USAGE_TOOLS, type UsageRecord } from "./usage-record.js"; * be gone by the time anyone reports. */ export interface StoredUsage extends UsageRecord { readonly repository_id: string; - /** The ID is retained for clone consent, but must not be displayed or attributed. */ - readonly repository_uncertain?: true; } const COUNTERS = [ @@ -146,7 +144,6 @@ function isStoredUsage( typeof object.at === "string" && dayOf(object.at) !== null && isText(object.repository_id) && - (object.repository_uncertain === undefined || object.repository_uncertain === true) && isNullableText(object.tool_version) && isNullableText(object.model) && isNullableText(object.cwd) && diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts new file mode 100644 index 000000000..8975d0515 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -0,0 +1,315 @@ +import { spawn } from "node:child_process"; +import { createReadStream } from "node:fs"; +import { mkdtemp, rename, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createInterface } from "node:readline"; +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import claude from "../config/claude.json" with { type: "json" }; +import codex from "../config/codex.json" with { type: "json" }; +import { + CLAUDE_CONFIG_ID, + CODEX_CONFIG_ID, + type Config, + OPENCODE_CONFIG_ID, + readToken, +} from "../settings.js"; +import { startOtelCollector } from "./collect.js"; +import { destination, sessionIdsInCapture, syncOtel } from "./sync.js"; + +type RunOptions = { + config: Config; + endpoint: string; + tenant: string; + token: string; + cwd: string; + prompt: string; + model?: string; + retainCapture?: (path: string) => void; + signal: AbortSignal; +}; + +async function provider(options: RunOptions, collector: string, token: string, directory: string) { + const env = Object.fromEntries( + Object.entries(process.env).filter( + ([key]) => !key.startsWith("OTEL_") && key !== "AIDD_OTEL_TOKEN" + ) + ); + env.AIDD_TELEMETRY = "0"; + const model = options.model ? ["--model", options.model] : []; + if (options.config === CLAUDE_CONFIG_ID) { + const settings: Record<string, string> = {}; + for (const [key, value] of Object.entries(claude.env)) { + settings[key] = value.replace("{collector}", collector).replace("{token}", token); + } + for (const key of Object.keys(claude.defaults)) { + if (key.startsWith("OTEL_LOG_")) settings[key] = "0"; + } + settings.OTEL_TRACES_EXPORTER = "none"; + settings.OTEL_METRIC_EXPORT_INTERVAL = "1000"; + settings.OTEL_LOGS_EXPORT_INTERVAL = "1000"; + settings.OTEL_METRICS_INCLUDE_REPOSITORY = "false"; + const file = join(directory, "claude-settings.json"); + await writeFile(file, JSON.stringify({ env: settings }), { mode: 0o600, flag: "wx" }); + return { + env: { ...env, ...settings }, + args: [ + "--print", + "--no-session-persistence", + "--settings", + file, + ...model, + "--", + options.prompt, + ], + }; + } + if (options.config === CODEX_CONFIG_ID) { + const overrides = Object.entries(codex.exporters).flatMap(([key, url]) => [ + "-c", + `otel.${key}={otlp-http={endpoint="${url.replace("{collector}", collector)}",protocol="json",headers={authorization="Bearer ${token}"}}}`, + ]); + return { + env, + args: [ + "exec", + "--ephemeral", + ...model, + ...overrides, + "-c", + "otel.log_user_prompt=false", + "-", + ], + input: options.prompt, + }; + } + return { + env: { + ...env, + OTEL_EXPORTER_OTLP_ENDPOINT: `${collector}/${OPENCODE_CONFIG_ID}`, + OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${token}`, + }, + args: ["run", "--standalone", ...model, "--", options.prompt], + }; +} + +function list(value: unknown): unknown[] { + return Array.isArray(value) ? value : []; +} + +function attributes(value: unknown) { + const values = new Map<string, string | number>(); + for (const entry of list(value)) { + const row = asPlainObject(entry); + const content = asPlainObject(row?.value); + const item = content?.stringValue ?? content?.intValue; + if (typeof row?.key === "string" && (typeof item === "string" || typeof item === "number")) { + if (values.has(row.key)) return new Map<string, string | number>(); + values.set(row.key, item); + } + } + return values; +} + +function validSpanTimes(start: unknown, end: unknown): boolean { + if ( + typeof start !== "string" || + typeof end !== "string" || + !/^[1-9]\d{0,19}$/.test(start) || + !/^[1-9]\d{0,19}$/.test(end) + ) + return false; + return BigInt(start) <= BigInt(end) && BigInt(end) <= 18446744073709551615n; +} + +export function openCodePromptSpans(line: string): unknown[] { + const envelope = asPlainObject(JSON.parse(line)); + if (envelope?.signal !== "traces") return []; + const payload = asPlainObject(envelope.payload); + const result = []; + for (const item of list(payload?.resourceSpans)) { + const resource = asPlainObject(item); + const a = attributes(asPlainObject(resource?.resource)?.attributes); + const version = a.get("service.version"); + if ( + a.get("service.name") !== OPENCODE_CONFIG_ID || + typeof version !== "string" || + !/^2\.\d+\.\d+$/.test(version) + ) + continue; + for (const scope of list(resource?.scopeSpans)) { + for (const raw of list(asPlainObject(scope)?.spans)) { + const span = asPlainObject(raw); + const b = attributes(span?.attributes); + const path = b.get("url.path"); + const route = "/api/session/:sessionID/prompt"; + const kind = span?.kind; + if ( + (kind !== 2 && kind !== "SPAN_KIND_SERVER") || + b.get("http.route") !== route || + b.get("http.request.method") !== "POST" || + ![200, "200"].includes(b.get("http.response.status_code") ?? "") || + typeof path !== "string" || + !/^\/api\/session\/ses_[A-Za-z0-9]{1,64}\/prompt$/.test(path) || + typeof span?.traceId !== "string" || + !/^[a-f0-9]{32}$/i.test(span.traceId) || + typeof span.spanId !== "string" || + !/^[a-f0-9]{16}$/i.test(span.spanId) || + span.name !== "POST" || + !validSpanTimes(span.startTimeUnixNano, span.endTimeUnixNano) + ) + continue; + const string = (key: string, value: string) => ({ key, value: { stringValue: value } }); + result.push({ + resourceSpans: [ + { + resource: { + attributes: [ + string("service.name", OPENCODE_CONFIG_ID), + string("service.version", version), + ], + }, + scopeSpans: [ + { + spans: [ + { + traceId: span.traceId, + spanId: span.spanId, + name: span.name, + kind, + startTimeUnixNano: span.startTimeUnixNano, + endTimeUnixNano: span.endTimeUnixNano, + attributes: [ + string("http.route", route), + string("http.request.method", "POST"), + string("url.path", path), + { key: "http.response.status_code", value: { intValue: "200" } }, + ], + }, + ], + }, + ], + }, + ], + }); + } + } + } + return result; +} + +async function retainCapture(home: string, capture: string, config: Config): Promise<string> { + const retained = join(home, `${config}.jsonl`); + try { + await rename(capture, retained); + await rm(join(home, ".local"), { recursive: true }); + await rm(join(home, "claude-settings.json"), { force: true }); + return retained; + } catch (error) { + await rm(home, { recursive: true, force: true }); + throw error; + } +} + +/** Consent covers this one invocation and its isolated collector only. */ +export async function runOtel(options: RunOptions): Promise<number> { + if (process.env.AIDD_TELEMETRY === "0") throw new Error("AIDD telemetry is disabled."); + if (!options.prompt.trim()) throw new Error("Pass a nonempty prompt."); + if (options.model && !/^[A-Za-z0-9][A-Za-z0-9._:/#-]{0,127}$/.test(options.model)) { + throw new Error("Provider model ID is invalid."); + } + const config = options.config === OPENCODE_CONFIG_ID ? CODEX_CONFIG_ID : options.config; + const base = { ...options, config, session: "validation", home: "", follow: false }; + const origin = destination(base, "logs").origin; + if (options.signal.aborted) throw new Error("OTLP execution was cancelled."); + const home = await mkdtemp(join(tmpdir(), "aidd-otel-run-")); + const capture = join(home, ".local/share/aidd-otel-observation", `${options.config}.jsonl`); + let captureCreated = false; + try { + const collector = await startOtelCollector({ home, port: 0 }); + captureCreated = true; + try { + const launch = await provider( + options, + `http://127.0.0.1:${collector.port}`, + await readToken(home), + home + ); + const code = await new Promise<number | null>((resolve, reject) => { + const child = spawn(options.config, launch.args, { + cwd: options.cwd, + env: launch.env, + signal: options.signal, + stdio: [launch.input ? "pipe" : "ignore", "inherit", "inherit"], + }); + child.on("error", reject); + child.on("close", resolve); + child.stdin?.on("error", reject); + child.stdin?.end(launch.input); + }); + if (code !== 0) throw new Error(`${options.config} execution did not complete successfully.`); + } finally { + await collector.close(); + } + const lines = createInterface({ input: createReadStream(capture), crlfDelay: Infinity }); + let sent = 0; + if (options.config === OPENCODE_CONFIG_ID) { + const url = new URL( + `/api/kpi/otel/${OPENCODE_CONFIG_ID}/${options.tenant}/v1/traces`, + origin + ); + let selected: unknown; + for await (const line of lines) { + if (!line.trim()) continue; + for (const payload of openCodePromptSpans(line)) { + if (selected) + throw new Error("The isolated OpenCode run emitted more than one prompt span."); + selected = payload; + } + } + if (selected) { + const response = await fetch(url, { + method: "POST", + redirect: "manual", + signal: AbortSignal.any([options.signal, AbortSignal.timeout(15000)]), + headers: { + authorization: `Bearer ${options.token}`, + "content-type": "application/json", + }, + body: JSON.stringify(selected), + }); + if (!response.ok) + throw new Error(`Gouvernail rejected OpenCode traces: HTTP ${response.status}.`); + const receipt = asPlainObject(await response.json()); + if (receipt?.accepted !== 1 || Object.keys(receipt).length !== 1) + throw new Error("Gouvernail did not retain the OpenCode span."); + sent++; + } + } else { + const sessions = new Set<string>(); + for await (const line of lines) { + if (line.trim()) + for (const session of sessionIdsInCapture(line, config)) sessions.add(session); + } + if (sessions.size > 1) + throw new Error("The isolated run emitted more than one OTLP session."); + for (const session of sessions) { + const result = await syncOtel( + { ...base, session, home }, + async () => { + if (options.signal.aborted) throw new Error("OTLP execution was cancelled."); + }, + async () => !options.signal.aborted + ); + sent += result.sent; + } + } + return sent; + } finally { + if (captureCreated && options.retainCapture) { + options.retainCapture(await retainCapture(home, capture, options.config)); + } else { + await rm(home, { recursive: true, force: true }); + } + } +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts deleted file mode 100644 index c36951659..000000000 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync-all.ts +++ /dev/null @@ -1,535 +0,0 @@ -import { createHash, randomUUID } from "node:crypto"; -import { constants } from "node:fs"; -import { lstat, mkdir, open, readdir, rename, rm, writeFile } from "node:fs/promises"; -import { join } from "node:path"; -import { setTimeout as delay } from "node:timers/promises"; -import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; -import { - type SessionAdmission, - sameSessionAdmission, -} from "../../../domain/report/session-feature.js"; -import { LedgerLock } from "../../ledger-lock.js"; -import { - admissionFingerprint, - ConsentNotConfirmedError, - destination, - OtelObservationTooLargeError, - packTransfers, - type SyncOptions, - selectTransfers, - send, - sessionIdsInCapture, - type Transfer, -} from "./sync.js"; - -export type AllOptions = Omit<SyncOptions, "session" | "follow" | "stateDir"> & { - readonly stateDir: string; - readonly refreshMs?: number; -}; -type Consent = "allow" | "deny" | "unknown"; -type Snapshot = ReadonlyMap<string, SessionAdmission>; -interface Pending { - readonly offset: number; - readonly length: number; - /** This range was admitted but could not be delivered; preserve it even if later denied. */ - readonly hold?: boolean; - readonly admissionKey?: string; - readonly cwdKey?: string; - readonly cloneKey?: string; -} -interface Checkpoint { - readonly device: number; - readonly inode: number; - readonly offset: number; -} -const MAX_LINE = 32 * 1024 * 1024; -const INTERVAL_MS = 5000; -const RETRY_MS = 30_000; - -function key(options: AllOptions, session: string) { - return createHash("sha256") - .update(`${options.endpoint}\0${options.tenant}\0${options.config}\0${session}`) - .digest("hex"); -} -function asOptions(options: AllOptions, session: string): SyncOptions { - return { ...options, session, follow: true }; -} -function admissionKey(admission: SessionAdmission): string { - return admissionFingerprint(admission); -} -function cwdKey(cwd: string): string { - return createHash("sha256").update(cwd).digest("hex"); -} -async function privateDirectory(path: string) { - await mkdir(path, { recursive: true, mode: 0o700 }); - const info = await lstat(path); - if (!info.isDirectory() || (info.mode & 0o077) !== 0) - throw new Error("Sync state directory must be private."); -} -async function readCheckpoint(path: string): Promise<Checkpoint | null> { - const file = await open( - path, - constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK - ).catch((error) => { - if (error instanceof Error && "code" in error && error.code === "ENOENT") return null; - throw error; - }); - if (!file) return null; - try { - const info = await file.stat(); - if (!info.isFile() || (info.mode & 0o077) !== 0) - throw new Error("Sync checkpoint must be a private regular file."); - let parsed: unknown; - try { - parsed = JSON.parse(await file.readFile("utf8")); - } catch { - throw new Error("Sync checkpoint is invalid."); - } - const record = asPlainObject(parsed); - if ( - !record || - !Number.isSafeInteger(record.device) || - !Number.isSafeInteger(record.inode) || - !Number.isSafeInteger(record.offset) || - Number(record.offset) < 0 - ) { - throw new Error("Sync checkpoint is invalid."); - } - return { - device: Number(record.device), - inode: Number(record.inode), - offset: Number(record.offset), - }; - } finally { - await file.close(); - } -} -async function saveCheckpoint(path: string, value: Checkpoint) { - const temporary = `${path}.${randomUUID()}.tmp`; - try { - await writeFile(temporary, `${JSON.stringify(value)}\n`, { mode: 0o600, flag: "wx" }); - await rename(temporary, path); - } finally { - await rm(temporary, { force: true }); - } -} - -function pendingLine(item: Pending): string { - if (item.hold && !item.admissionKey) throw new Error("Sync pending admission is unavailable."); - return `${item.offset} ${item.length}${item.hold ? ` hold ${item.admissionKey}${item.cwdKey && item.cloneKey ? ` ${item.cwdKey} ${item.cloneKey}` : ""}` : ""}\n`; -} - -/** A pending journal contains only byte ranges, not source text or provider identities. */ -async function appendPending(path: string, item: Pending) { - const file = await open( - path, - constants.O_RDWR | constants.O_CREAT | constants.O_APPEND | constants.O_NOFOLLOW, - 0o600 - ); - try { - const info = await file.stat(); - if (!info.isFile() || (info.mode & 0o077) !== 0) - throw new Error("Sync pending journal must be a private regular file."); - if (info.size > 0) { - const length = Math.min(info.size, 256); - const tail = Buffer.alloc(length); - const { bytesRead } = await file.read(tail, 0, length, info.size - length); - if (bytesRead !== length) throw new Error("Sync pending journal changed unexpectedly."); - if (tail.at(-1) !== 10) { - const last = tail.lastIndexOf(10); - if (last < 0 && info.size > length) - throw new Error("Sync pending journal has an invalid entry."); - await file.truncate(last < 0 ? 0 : info.size - length + last + 1); - } - } - await file.writeFile(pendingLine(item)); - await file.sync(); - } finally { - await file.close(); - } -} - -async function pendingRanges(path: string): Promise<Pending[]> { - const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); - try { - const info = await file.stat(); - if (!info.isFile() || (info.mode & 0o077) !== 0) - throw new Error("Sync pending journal must be a private regular file."); - const ranges = new Map<string, Pending>(); - const contents = await file.readFile("utf8"); - const lines = contents.split("\n"); - lines.pop(); // A crash can leave an incomplete last append; its source line is replayed. - for (const line of lines) { - const parts = - /^(\d+) (\d+)(?: hold ([a-f0-9]{64})(?: ([a-f0-9]{64}) ([a-f0-9]{64}))?)?$/.exec(line); - if (!parts) throw new Error("Sync pending journal is invalid."); - const offset = Number(parts[1]); - const length = Number(parts[2]); - if ( - !Number.isSafeInteger(offset) || - !Number.isSafeInteger(length) || - length < 1 || - length > MAX_LINE - ) - throw new Error("Sync pending journal is invalid."); - const rangeKey = `${offset} ${length}`; - const previous = ranges.get(rangeKey); - if (previous?.admissionKey && parts[3] && previous.admissionKey !== parts[3]) - throw new Error("Sync pending journal has conflicting admissions."); - if ( - (previous?.cwdKey && parts[4] && previous.cwdKey !== parts[4]) || - (previous?.cloneKey && parts[5] && previous.cloneKey !== parts[5]) - ) - throw new Error("Sync pending journal has conflicting clone proofs."); - ranges.set(rangeKey, { - offset, - length, - hold: !!parts[3] || previous?.hold, - admissionKey: parts[3] ?? previous?.admissionKey, - cwdKey: parts[4] ?? previous?.cwdKey, - cloneKey: parts[5] ?? previous?.cloneKey, - }); - } - return [...ranges.values()]; - } finally { - await file.close(); - } -} - -async function replacePending(path: string, ranges: readonly Pending[]) { - if (ranges.length === 0) { - await rm(path, { force: true }); - return; - } - const temporary = `${path}.${randomUUID()}.tmp`; - try { - const file = await open(temporary, "wx", 0o600); - try { - await file.writeFile(ranges.map(pendingLine).join("")); - await file.sync(); - } finally { - await file.close(); - } - await rename(temporary, path); - } finally { - await rm(temporary, { force: true }); - } -} - -/** On first launch start at the last complete line. A line being appended is retained. */ -async function initialOffset( - file: Awaited<ReturnType<typeof open>>, - size: number -): Promise<number> { - if (size === 0) return 0; - const limit = Math.min(size, MAX_LINE); - const tail = Buffer.alloc(limit); - const { bytesRead } = await file.read(tail, 0, limit, size - limit); - if (bytesRead !== limit) throw new Error("Capture changed while sync was starting."); - const last = tail.lastIndexOf(10); - if (last < 0 && size > limit) throw new Error("Capture line exceeds 32 MiB."); - return last < 0 ? 0 : size - limit + last + 1; -} - -/** Follow all sessions admitted by the ledger. Unknown sessions leave private byte-range - * journals before the checkpoint advances; no raw prompt or tool output is copied. */ -export async function syncAllOtel( - options: AllOptions, - snapshot: () => Promise<Snapshot>, - permits: ( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ) => Promise<Consent>, - cloneProof?: ( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ) => Promise<string | null> -) { - const base = asOptions(options, "auto"); - destination(base, "logs"); - const capture = join( - options.home, - ".local/share/aidd-otel-observation", - `${options.config}.jsonl` - ); - const directory = join(options.stateDir, "otel-sync"); - await privateDirectory(directory); - const checkpointPath = join(directory, `${key(options, "auto")}.checkpoint.json`); - const lock = new LedgerLock(`${checkpointPath}.lock`, { - name: "OTLP sync", - waitMs: 0, - staleAfterMs: Number.MAX_SAFE_INTEGER, - }); - const release = await lock.acquire(); - let file: Awaited<ReturnType<typeof open>> | undefined; - try { - file = await open(capture, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); - const info = await file.stat(); - if (!info.isFile() || (info.mode & 0o077) !== 0) - throw new Error("Capture must be a private regular file."); - const previous = await readCheckpoint(checkpointPath); - if ( - previous && - (previous.device !== info.dev || previous.inode !== info.ino || previous.offset > info.size) - ) { - throw new Error( - "Capture changed since the last sync; recover the previous capture before restarting." - ); - } - let offset = previous?.offset ?? (await initialOffset(file, info.size)); - const save = async (position: number) => - saveCheckpoint(checkpointPath, { device: info.dev, inode: info.ino, offset: position }); - await save(offset); - let held = await snapshot(); - let lastSnapshotAt = Date.now(); - let dirty = false; - let pendingBytes = Buffer.alloc(0); - let sent = 0; - let denied = 0; - const oversized = new Set<string>(); - const pendingDirectory = join(directory, `${key(options, "auto")}.pending`); - await privateDirectory(pendingDirectory); - const pendingPath = (id: string) => join(pendingDirectory, `${key(options, id)}.jsonl`); - const stillPermitted = async ( - id: string, - admission: SessionAdmission, - items: readonly Transfer[] - ) => { - const current = (await snapshot()).get(id); - if (!sameSessionAdmission(admission, current) || !current) return false; - const consentAt = new Date().toISOString(); - for (const item of items) - if (!item.at || (await permits(current, item.at, item.start, consentAt)) !== "allow") - return false; - return true; - }; - const deliver = async (id: string, transfer: Transfer, guard: () => Promise<boolean>) => { - try { - await send(asOptions(options, id), transfer, guard); - return true; - } catch (error) { - if (error instanceof ConsentNotConfirmedError) return false; - throw error; - } - }; - const proofForTransfers = async ( - admission: SessionAdmission, - transfers: readonly Transfer[] - ) => { - if (!cloneProof || !admission.cwd || transfers.length === 0) return null; - const now = new Date().toISOString(); - let clone: string | null = null; - for (const transfer of transfers) { - if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) return null; - const found = await cloneProof(admission, transfer.at, transfer.start, now); - if (!found || (clone && found !== clone)) return null; - clone = found; - } - return clone; - }; - const process = async ( - id: string, - items: readonly Transfer[], - admission: SessionAdmission, - hold = false, - expectedClone?: string, - captured?: { clone: string | null } - ): Promise<"resolved" | "unknown" | "hold"> => { - let wait = false; - let rejected = 0; - const allowed: Transfer[] = []; - const verdicts = new Map<string, Consent>(); - for (const transfer of items) { - if (!transfer.at || (transfer.signal === "metrics" && !transfer.start)) { - rejected++; - continue; - } - const stamp = `${transfer.at}\0${transfer.start ?? ""}`; - let verdict = verdicts.get(stamp); - if (verdict === undefined) { - verdict = await permits(admission, transfer.at, transfer.start); - verdicts.set(stamp, verdict); - } - if (verdict === "unknown") wait = true; - else if (verdict === "deny") { - // A repository can change between the snapshot and the consent check. A denial - // against that stale admission is not evidence that the call lacked consent. - if (!sameSessionAdmission(admission, (await snapshot()).get(id))) return "unknown"; - if (hold) wait = true; - else rejected++; - } else { - allowed.push(transfer); - } - } - // Retrying a partly sent source line would resend its earlier observations. - if (wait) return hold ? "hold" : "unknown"; - let batches: ReturnType<typeof packTransfers>; - try { - batches = packTransfers(allowed, false); - } catch (error) { - if (!(error instanceof OtelObservationTooLargeError)) throw error; - batches = []; - } - if (allowed.length > 0 && batches.length !== 1) { - if (!oversized.has(id)) { - console.error( - "A captured OTLP line exceeds the 2 MiB delivery limit; it remains pending." - ); - oversized.add(id); - } - return hold ? "hold" : "unknown"; - } - const clone = allowed.length ? await proofForTransfers(admission, allowed) : null; - if (allowed.length && cloneProof && (!clone || (expectedClone && clone !== expectedClone))) - return hold ? "hold" : "unknown"; - if (captured) captured.clone = clone; - for (const batch of batches) { - if ( - !(await deliver( - id, - batch.transfer, - async () => - (await stillPermitted(id, admission, batch.items)) && - (!clone || (await proofForTransfers(admission, batch.items)) === clone) - )) - ) - return "hold"; - sent += batch.count; - } - denied += rejected; - return "resolved"; - }; - const flushPending = async () => { - const files = new Set(await readdir(pendingDirectory)); - for (const [id, admission] of held) { - const path = pendingPath(id); - if (!files.has(`${key(options, id)}.jsonl`)) continue; - const waiting: Pending[] = []; - for (const item of await pendingRanges(path)) { - const raw = Buffer.alloc(item.length); - const read = await file?.read(raw, 0, item.length, item.offset); - if (!read || read.bytesRead !== item.length || raw.at(-1) !== 10) - throw new Error("Pending capture range is unavailable; recover the capture."); - const transfers = selectTransfers(raw.toString("utf8"), asOptions(options, id)); - const changedAdmission = item.hold && item.admissionKey !== admissionKey(admission); - if ( - changedAdmission && - (!item.cwdKey || - !item.cloneKey || - !admission.cwd || - admission.repositoryId !== null || - admission.feature.state !== "ambiguous" || - item.cwdKey !== cwdKey(admission.cwd)) - ) { - waiting.push(item); - continue; - } - const captured = { clone: null as string | null }; - const status = await process( - id, - transfers, - admission, - item.hold, - item.cloneKey, - captured - ); - if (status !== "resolved") - waiting.push({ - ...item, - hold: item.hold || status === "hold", - admissionKey: - item.hold || status === "hold" - ? (item.admissionKey ?? admissionKey(admission)) - : undefined, - ...(status === "hold" && admission.cwd && captured.clone && !item.hold - ? { cwdKey: cwdKey(admission.cwd), cloneKey: captured.clone } - : {}), - }); - } - await replacePending(path, waiting); - } - }; - await flushPending(); - const buffer = Buffer.alloc(65536); - for (;;) { - const { bytesRead } = await file.read(buffer, 0, buffer.length, offset + pendingBytes.length); - if (bytesRead > 0) { - pendingBytes = Buffer.concat([pendingBytes, buffer.subarray(0, bytesRead)]); - if (pendingBytes.length > MAX_LINE) throw new Error("Capture line exceeds 32 MiB."); - for ( - let newline = pendingBytes.indexOf(10); - newline >= 0; - newline = pendingBytes.indexOf(10) - ) { - const raw = pendingBytes.subarray(0, newline).toString("utf8"); - const position = offset; - if (raw.trim()) - for (const id of sessionIdsInCapture(raw, options.config)) { - const items = selectTransfers(raw, asOptions(options, id)); - const admission = held.get(id); - const captured = { clone: null as string | null }; - const status = - items.length === 0 - ? "resolved" - : admission - ? await process(id, items, admission, false, undefined, captured) - : "unknown"; - if (status !== "resolved") - await appendPending(pendingPath(id), { - offset: position, - length: newline + 1, - hold: status === "hold", - admissionKey: - status === "hold" && admission ? admissionKey(admission) : undefined, - ...(status === "hold" && admission?.cwd && captured.clone - ? { cwdKey: cwdKey(admission.cwd), cloneKey: captured.clone } - : {}), - }); - } - offset += newline + 1; - pendingBytes = pendingBytes.subarray(newline + 1); - dirty = true; - } - if (dirty) await save(offset); - continue; - } - const current = await lstat(capture); - if ( - !current.isFile() || - current.ino !== info.ino || - current.dev !== info.dev || - current.size < offset + pendingBytes.length - ) { - throw new Error( - "Capture was replaced or truncated; recover the previous capture before restarting." - ); - } - const elapsed = Date.now() - lastSnapshotAt; - const retryAfter = options.refreshMs ?? RETRY_MS; - const pendingFiles = (await readdir(pendingDirectory)).length; - if (dirty || (pendingFiles > 0 && elapsed >= retryAfter)) { - held = await snapshot(); - lastSnapshotAt = Date.now(); - dirty = false; - await flushPending(); - await save(offset); - } - if (options.signal?.aborted) - return { sent, denied, pending: (await readdir(pendingDirectory)).length }; - try { - await delay(options.intervalMs ?? INTERVAL_MS, undefined, { signal: options.signal }); - } catch (error) { - if (options.signal?.aborted) - return { sent, denied, pending: (await readdir(pendingDirectory)).length }; - throw error; - } - } - } finally { - await file?.close(); - await release(); - } -} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index f6b34fe2f..c5ca82f87 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -1,4 +1,3 @@ -import { createHash } from "node:crypto"; import { constants } from "node:fs"; import { lstat, open } from "node:fs/promises"; import { join } from "node:path"; @@ -107,15 +106,6 @@ const SAFE_ATTRIBUTES = new Set([ "reasoning_token_count", ]); -export function admissionFingerprint(admission: { - cwd: string | null; - repositoryId: string | null; -}): string { - return createHash("sha256") - .update(JSON.stringify([admission.cwd, admission.repositoryId])) - .digest("hex"); -} - function attributes(value: unknown): Map<string, string> { if (value !== undefined && !Array.isArray(value)) throw new Error("Invalid OTLP attributes."); const out = new Map<string, string>(); diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 35d5d9831..459e0e886 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -32,9 +32,8 @@ import { checkOtel, collectOtel, configureOtel, - followCapturedSessions, otelOptions, - syncCapturedSession, + runOtelSession, } from "../../runtime/wiring/telemetry-otel.js"; import { printForgetResult, @@ -209,112 +208,59 @@ export function registerTelemetryCommand(program: Command): void { }); otel - .command("sync") - .description("Send one consented session's local telemetry to a loopback Gouvernail") + .command("run <prompt>") + .description("Run one fresh provider session and send its allowed OTLP observations") .addOption( - new Option("--config <tool>", "Tool to sync") - .choices(otelOptions.syncConfigs) + new Option("--config <tool>", "Tool to run") + .choices(otelOptions.configs) .makeOptionMandatory() ) - .option("--session <id>", "Exact provider session ID; omit with --follow to discover sessions") .requiredOption("--tenant <slug>", "Gouvernail tenant slug") .requiredOption("--endpoint <origin>", "Loopback HTTP origin") - .option("--follow", "Follow new events for this session", false) + .option("--yes", "Consent to capture and send telemetry from this execution") + .option("--model <model>", "Provider model to use") + .option("--keep-capture", "Retain the private local OTLP capture") .action( - async (options: { - config: (typeof otelOptions.syncConfigs)[number]; - session?: string; - tenant: string; - endpoint: string; - follow: boolean; - }) => { - const { output, run } = commandOf(program); - if (!options.session && !options.follow) { - output.error("Pass --session for a bounded sync, or --follow to discover new sessions."); - process.exitCode = 1; - return; + async ( + prompt: string, + options: { + config: (typeof otelOptions.configs)[number]; + tenant: string; + endpoint: string; + yes?: boolean; + model?: string; + keepCapture?: boolean; } - await run(async (deps) => { - const tool = otelOptions.usageTool(options.config); - if (!options.session) { - output.info("Following new consented sessions; press Ctrl-C to stop."); - const result = await followCapturedSessions( - { - config: options.config, - tenant: options.tenant, - endpoint: options.endpoint, - token: process.env.AIDD_OTEL_TOKEN ?? "", - }, - () => deps.telemetry.sessionFeatureUseCase.snapshot(tool), - (admission, at, start, current) => - deps.telemetry.sessionFeatureUseCase.permits(admission, at, start, current), - (admission, at, start, current) => - deps.telemetry.sessionFeatureUseCase.cloneProof(admission, at, start, current) - ); - output.info( - `Sent ${result.sent} observations; ${result.denied} were outside consent; ${result.pending} await confirmation.` - ); - return; - } - const session = options.session; - let admission = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get(session); - const initialAdmission = admission; - if (!initialAdmission) - throw new Error("No consented usage found for this session; no capture was sent."); - const revalidate = async () => { - const current = (await deps.telemetry.sessionFeatureUseCase.snapshot(tool)).get( - session + ) => { + const { output, projectRoot } = parseGlobalOptions(program); + const abort = new AbortController(); + const cancel = () => abort.abort(); + process.once("SIGINT", cancel); + process.once("SIGTERM", cancel); + try { + if (!options.yes) + throw new Error( + "Pass --yes to consent to OTLP capture and delivery for this execution." ); - if (!current) { - admission = undefined; - throw new Error("No consented usage found for this session; no capture was sent."); - } - if ( - !initialAdmission.cwd || - (!initialAdmission.repositoryId && - !( - initialAdmission.repositoryId === null && - initialAdmission.feature.state !== "confirmed" - )) || - current.cwd !== initialAdmission.cwd || - (current.repositoryId !== initialAdmission.repositoryId && - !(current.repositoryId === null && current.feature.state === "ambiguous")) || - (current.repositoryId === null && current.feature.state === "confirmed") - ) { - admission = undefined; - throw new Error( - "Session admission changed or is unavailable; sync stopped before sending." - ); - } - admission = current; - }; - await revalidate(); - if (options.follow) output.info("Following the selected session; press Ctrl-C to stop."); - const result = await syncCapturedSession( - { - ...options, - session, - token: process.env.AIDD_OTEL_TOKEN ?? "", - }, - revalidate, - async (transfer) => - Boolean( - admission && - transfer.at && - (await deps.telemetry.sessionFeatureUseCase.permits( - admission, - transfer.at, - transfer.start, - transfer.consentAt - )) === "allow" - ) - ); - output.info( - `Sent ${result.sent} observations; ${result.skipped} were outside a known consent interval. Receiver acceptance does not prove member attribution.` - ); - if (result.partialLine) - output.info("The last incomplete capture line was left for the next sync."); - }); + const sent = await runOtelSession({ + ...options, + cwd: projectRoot, + prompt, + signal: abort.signal, + token: process.env.AIDD_OTEL_TOKEN ?? "", + retainCapture: options.keepCapture + ? (path) => output.info(`Private OTLP capture: ${path}`) + : undefined, + }); + if (!sent) + throw new Error("The execution produced no supported OTLP observations to send."); + output.info(`Sent ${sent} OTLP observations from this execution.`); + } catch (error) { + new ErrorHandler(output).handle(error); + } finally { + process.off("SIGINT", cancel); + process.off("SIGTERM", cancel); + } } ); diff --git a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts index 53e94f704..f6e2d407b 100644 --- a/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts +++ b/cli/src/presentation/display/telemetry/telemetry-lifecycle-display.ts @@ -112,8 +112,9 @@ export function printOffResult(output: CLIOutput, result: OffResult): void { ? "Measurement is off for this clone. What was measured stays until `aidd telemetry forget --yes`." : "Measurement was not on for this clone." ); - if (result.closedAt) output.info(`Consent close recorded at ${result.closedAt}.`); - output.info("Native OTLP configuration and collector are unchanged and may still collect data."); + output.info( + "Provider OTLP configuration and collector are unchanged and may still collect data." + ); } const WHAT: Readonly<Record<ErasureKind, string>> = { diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index effa2c5e6..b11e86838 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -1,22 +1,11 @@ import { homedir } from "node:os"; -import type { SessionAdmission } from "../../contexts/telemetry/domain/report/session-feature.js"; import { startOtelCollector } from "../../contexts/telemetry/infrastructure/otel/actions/collect.js"; import { initializeOtel } from "../../contexts/telemetry/infrastructure/otel/actions/init.js"; -import { - type SyncOptions, - syncOtel, - type Transfer, -} from "../../contexts/telemetry/infrastructure/otel/actions/sync.js"; -import { - type AllOptions, - syncAllOtel, -} from "../../contexts/telemetry/infrastructure/otel/actions/sync-all.js"; +import { runOtel } from "../../contexts/telemetry/infrastructure/otel/actions/run.js"; import { verifyOtel } from "../../contexts/telemetry/infrastructure/otel/actions/verify.js"; import { OTEL_CONFIGS, OTEL_ENDPOINTS, - OTEL_SYNC_CONFIGS, - usageToolForOtel, } from "../../contexts/telemetry/infrastructure/otel/settings.js"; import { checkInstallationId, @@ -24,48 +13,17 @@ import { finishInstallationId, } from "../filesystem/installation-identity.js"; import { userConfigDirOf } from "../user-config-dir.js"; -import { telemetryDir } from "./telemetry.js"; - -export function collectOtel(port: number) { - return startOtelCollector({ home: homedir(), port }); -} -export function syncCapturedSession( - options: Omit<SyncOptions, "home">, - revalidate: () => Promise<void>, - permits: (transfer: Transfer) => Promise<boolean> -) { - return syncOtel({ ...options, home: homedir() }, revalidate, permits); +export function runOtelSession(options: Parameters<typeof runOtel>[0]) { + return runOtel(options); } -export function followCapturedSessions( - options: Omit<AllOptions, "home" | "stateDir">, - snapshot: () => Promise<ReadonlyMap<string, SessionAdmission>>, - permits: ( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ) => Promise<"allow" | "deny" | "unknown">, - proof?: ( - admission: SessionAdmission, - at: string, - start?: string, - current?: string - ) => Promise<string | null> -) { - return syncAllOtel( - { ...options, home: homedir(), stateDir: telemetryDir() }, - snapshot, - permits, - proof - ); +export function collectOtel(port: number) { + return startOtelCollector({ home: homedir(), port }); } export const otelOptions = { configs: OTEL_CONFIGS, - syncConfigs: OTEL_SYNC_CONFIGS, - usageTool: usageToolForOtel, endpoints: OTEL_ENDPOINTS, }; diff --git a/cli/src/runtime/wiring/telemetry.ts b/cli/src/runtime/wiring/telemetry.ts index e9a355683..ec01e63ea 100644 --- a/cli/src/runtime/wiring/telemetry.ts +++ b/cli/src/runtime/wiring/telemetry.ts @@ -13,9 +13,6 @@ import { StoredUsageMigration } from "../../contexts/telemetry/application/legac import { ReadUsageUseCase } from "../../contexts/telemetry/application/read-usage-use-case.js"; import { DeclaredBindings } from "../../contexts/telemetry/application/report/declared-bindings.js"; import { ReportUsageUseCase } from "../../contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../contexts/telemetry/application/report/repository-uncertainty.js"; -import { SessionConsent } from "../../contexts/telemetry/application/report/session-consent.js"; -import { SessionFeatureUseCase } from "../../contexts/telemetry/application/report/session-feature-use-case.js"; import { SnapshotBindingsUseCase } from "../../contexts/telemetry/application/snapshot-bindings-use-case.js"; import { TelemetryOffUseCase } from "../../contexts/telemetry/application/switch/telemetry-off-use-case.js"; import { TelemetryOnUseCase } from "../../contexts/telemetry/application/switch/telemetry-on-use-case.js"; @@ -62,7 +59,6 @@ export interface TelemetryDeps { ingestUsageUseCase: IngestUsageUseCase; snapshotBindingsUseCase: SnapshotBindingsUseCase; reportUsageUseCase: ReportUsageUseCase; - sessionFeatureUseCase: SessionFeatureUseCase; manageIdentityUseCase: ManageIdentityUseCase; proposeIdentityUseCase: ProposeIdentityUseCase; telemetryOnUseCase: TelemetryOnUseCase; @@ -148,13 +144,6 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { declaration, identity ); - const directoryResolver = new DirectoryResolver( - locator, - consents, - resolutionStore, - consentHistory, - environment - ); const ingestUsageUseCase = new IngestUsageUseCase( [ new ReadUsageUseCase( @@ -169,7 +158,7 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { ), ], ledger, - directoryResolver, + new DirectoryResolver(locator, consents, resolutionStore, consentHistory, environment), snapshotBindingsUseCase, { refusedByEnvironment }, new StoredUsageMigration(locator, resolutionStore, aliases, ledger) @@ -182,18 +171,12 @@ export function wireTelemetry(homedir: () => string): TelemetryDeps { snapshotBindingsUseCase, reportUsageUseCase: new ReportUsageUseCase( ingestUsageUseCase, - new ReportLedger(ledger, directoryResolver), + ledger, new DeclaredBindings(sessions, snapshotStore), identity, locator, { refusedByEnvironment } ), - sessionFeatureUseCase: new SessionFeatureUseCase( - ingestUsageUseCase, - new ReportLedger(ledger, directoryResolver), - new DeclaredBindings(sessions, snapshotStore), - new SessionConsent(ledger, directoryResolver, consentHistory) - ), manageIdentityUseCase: new ManageIdentityUseCase(identity), proposeIdentityUseCase: new ProposeIdentityUseCase( new GitUserSourceAdapter(gitEnv), diff --git a/cli/tests/architecture/mutation-shards.arch.test.ts b/cli/tests/architecture/mutation-shards.arch.test.ts index cf66a7115..da803dc0f 100644 --- a/cli/tests/architecture/mutation-shards.arch.test.ts +++ b/cli/tests/architecture/mutation-shards.arch.test.ts @@ -117,57 +117,11 @@ describe("scoring a sharded scope once, over every shard", () => { }; it("merges the shards into the report one run would have written", () => { - const merged = mergeReports([shard(1), shard(2)]); + const merged = mergeReports([shard(1).report, shard(2).report]); expect(Object.keys(merged.files).sort()).toEqual([...files].sort()); expect(scoreOf(merged)).toBe(50); }); - it("scores only each shard's own files even when another shard carries stale results", () => { - const first = shard(1); - const second = shard(2); - const otherFile = first.manifest.files[0]; - const stale = { - ...second, - report: { - files: { - ...second.report.files, - [otherFile]: { mutants: [mutant("Killed"), mutant("Killed")] }, - "src/other/stale.ts": { mutants: [mutant("Killed")] }, - }, - }, - }; - const merged = mergeReports([first, stale]); - expect(Object.keys(merged.files).sort()).toEqual([...files].sort()); - expect(merged.files[otherFile]?.mutants.map((item) => item.status)).toEqual([ - "Killed", - "Survived", - ]); - expect(scoreOf(merged)).toBe(50); - expect(mergeReports([stale, first])).toEqual(merged); - }); - - it("does not count stale files when determining whether shards hold mutants", () => { - const empty = [1, 2].map((i) => { - const current = shard(i); - const foreign = shard(i === 1 ? 2 : 1).manifest.files[0]; - return { ...current, report: report({ [foreign]: ["Killed"] }) }; - }); - expect(aggregateVerdict({ mutate, files, of, weightOf, shards: empty })).toMatch(/no mutant/); - }); - - it("fails when the same shard is supplied more than once", () => { - expect( - aggregateVerdict({ mutate, files, of, weightOf, shards: [shard(1), shard(2), shard(2)] }) - ).toMatch(/shard 2 of 2.*more than once/); - }); - - it("fails when an unexpected shard is supplied", () => { - const unexpected = { ...shard(1), manifest: { ...shard(1).manifest, shard: 3 } }; - expect( - aggregateVerdict({ mutate, files, of, weightOf, shards: [shard(1), shard(2), unexpected] }) - ).toMatch(/unexpected shard/); - }); - it("passes when every shard is there and matches the partition", () => { expect( aggregateVerdict({ mutate, files, of, weightOf, shards: [shard(1), shard(2)] }) diff --git a/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts index 5b756949b..b9938e1b9 100644 --- a/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/ingest-usage-use-case.unit.test.ts @@ -653,7 +653,8 @@ describe("where a call was made", () => { it("answers two spellings of a directory with one remembered resolution on a case-insensitive one", async () => { const s = setup({ caseInsensitive: true }); - s.locator.directories.set("/Work/A", repository("/Work/A", { clone: A })); + s.locator.directories.set("/Work/A", repository("/Work/A")); + s.optIn("/Work/A/.git"); s.transcripts.files.set("/t/1.jsonl", [line("A", 1, { cwd: "/Work/A" })]); await s.ingest.execute(); s.locator.directories.delete("/Work/A"); @@ -1048,41 +1049,6 @@ describe("a call is judged at its own time", () => { const EPOCH = "1970-01-01T00:00:00.000Z"; const B = cloneOf("/work/a/.git", { ino: "99", birthtimeMs: 5_000 }); - it("keeps tied macOS clone identities pending even when only one has consented", async () => { - const s = setup(); - const current = cloneOf(A.path, { dev: "0" }); - s.history.written.length = 0; - s.history.consented(A, "2026-10-01T00:00:00.000Z"); - s.history.written.push({ kind: "close", token: tokenOf(A), at: "2026-10-05T00:00:00.000Z" }); - s.history.consented(current, "2026-10-06T00:00:00.000Z"); - s.resolutions.resolutions.set(resolutionKey("/work/a", A), sighting("/work/a", A)); - s.locator.directories.set("/work/a", repository("/work/a", { clone: current })); - s.consents.clones.delete(cloneKey(A)); - s.consents.cloneSays(current, grantedBy(current)); - s.transcripts.files.set("/t/1.jsonl", [ - line("old", 1, {}, "2026-10-04T00:00:00.000Z"), - line("gap", 2, {}, "2026-10-05T00:00:00.000Z"), - line("new", 3, {}, "2026-10-07T00:00:00.000Z"), - ]); - const result = await s.ingest.execute(); - expect(s.ledger.records).toEqual([]); - expect(result.notStored["unreadable-consent"]).toBe(3); - }); - - it("does not choose between tied clones whose consent intervals overlap", async () => { - const s = setup(); - const current = cloneOf(A.path, { dev: "0" }); - s.history.consented(current); - s.resolutions.resolutions.set(resolutionKey("/work/a", A), sighting("/work/a", A)); - s.locator.directories.set("/work/a", repository("/work/a", { clone: current })); - s.consents.clones.delete(cloneKey(A)); - s.consents.cloneSays(current, grantedBy(current)); - s.transcripts.files.set("/t/1.jsonl", [line("ambiguous", 1)]); - const result = await s.ingest.execute(); - expect(s.ledger.records).toEqual([]); - expect(result.notStored["unreadable-consent"]).toBe(1); - }); - it("gives a clone two owners of one directory, each its own calls", async () => { const s = setup(); s.history.written.length = 0; diff --git a/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts b/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts index 87f27080d..4c8984c49 100644 --- a/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/legacy/stored-usage-migration.property.integration.test.ts @@ -10,7 +10,6 @@ import { ReadUsageUseCase } from "../../../../../src/contexts/telemetry/applicat import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; import { ReportUsageUseCase } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import type { CloneIdentity } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { readClaudeUsageLine } from "../../../../../src/contexts/telemetry/domain/formats/claude-transcript-usage.js"; @@ -242,14 +241,13 @@ function machine(name: string, storage: PrivateStorage = new PrivateStorageAdapt history.consented(clone); } const snapshots = new BindingSnapshotStoreAdapter(bindings, storage, aliases); - const directories = new DirectoryResolver(locator, consents, resolutions, history, { - caseInsensitiveFileSystem: false, - now: () => NOW, - }); const ingest = new IngestUsageUseCase( [new ReadUsageUseCase(transcripts, () => readClaudeUsageLine)], ledger, - directories, + new DirectoryResolver(locator, consents, resolutions, history, { + caseInsensitiveFileSystem: false, + now: () => NOW, + }), new SnapshotBindingsUseCase( new FakeBindings(), new InMemorySnapshots(), @@ -261,7 +259,7 @@ function machine(name: string, storage: PrivateStorage = new PrivateStorageAdapt ); const report = new ReportUsageUseCase( ingest, - new ReportLedger(ledger, directories), + ledger, new DeclaredBindings(new InMemorySessions(), snapshots), new InMemoryIdentity(), locator, @@ -289,11 +287,7 @@ async function bytes(dir: string): Promise<Record<string, string>> { for (const entry of await readdir(dir, { withFileTypes: true })) { if (!entry.isFile() || ["offsets.json", ".lock", "repository-ids.json"].includes(entry.name)) continue; - const text = await readFile(join(dir, entry.name), "utf8"); - // Earlier versions did not persist an uncertainty marker; reports recompute it. - found[entry.name] = /^\d{4}-\d{2}-\d{2}\.jsonl$/.test(entry.name) - ? text.replace(/,"repository_uncertain":true(?=\})/gu, "") - : text; + found[entry.name] = await readFile(join(dir, entry.name), "utf8"); } return Object.fromEntries(Object.entries(found).sort(([a], [b]) => (a < b ? -1 : 1))); } diff --git a/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts index 1abc2e821..f2baa7588 100644 --- a/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/left-out.property.integration.test.ts @@ -9,7 +9,6 @@ import { ReadUsageUseCase } from "../../../../../src/contexts/telemetry/applicat import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; import { ReportUsageUseCase } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import { cloneKey } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { readClaudeUsageLine } from "../../../../../src/contexts/telemetry/domain/formats/claude-transcript-usage.js"; @@ -207,7 +206,7 @@ function machine() { ); const report = new ReportUsageUseCase( ingest, - new ReportLedger(ledger), + ledger, new DeclaredBindings(new InMemorySessions(), new InMemorySnapshots()), new InMemoryIdentity(), locator, diff --git a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts index 2b808d6d7..4f18af287 100644 --- a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts @@ -15,7 +15,6 @@ import { type ReportRequest, ReportUsageUseCase, } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { declarationOf } from "../../../../../src/contexts/telemetry/domain/declaration/task-declaration.js"; import type { LeftOutEntry } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; import type { Period } from "../../../../../src/contexts/telemetry/domain/report/period.js"; @@ -275,7 +274,7 @@ async function report( return inZone(zone, async () => { const result = await new ReportUsageUseCase( ingest, - new ReportLedger(ledger), + ledger, new DeclaredBindings(sessions, new InMemorySnapshots()), identity, new FakeLocator(), diff --git a/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts index 23f82fd39..9137092fb 100644 --- a/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-filters.property.integration.test.ts @@ -13,7 +13,6 @@ import { type ReportRequest, ReportUsageUseCase, } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import { declarationOf } from "../../../../../src/contexts/telemetry/domain/declaration/task-declaration.js"; import type { Period } from "../../../../../src/contexts/telemetry/domain/report/period.js"; @@ -141,7 +140,7 @@ async function reporterOf({ calls, declarations, person }: Ledger) { { refusedByEnvironment: false }, ALREADY_MIGRATED ), - new ReportLedger(ledger), + ledger, new DeclaredBindings(sessions, snapshots), identity, new FakeLocator(), diff --git a/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts index 95a6d21be..e800a22ce 100644 --- a/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts @@ -7,7 +7,6 @@ import { ReportUsageUseCase, type RequestedFilters, } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import { cloneKey } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { @@ -22,7 +21,6 @@ import { } from "../../../../../src/contexts/telemetry/domain/report/report-filter.js"; import type { ReportAxis } from "../../../../../src/contexts/telemetry/domain/report/usage-report.js"; import { repositoryIdOf } from "../../../../../src/contexts/telemetry/domain/repository-identity.js"; -import { rememberResolution } from "../../../../../src/contexts/telemetry/domain/repository-resolution.js"; import { ALREADY_MIGRATED, AUTUMN, @@ -76,15 +74,13 @@ function setup(options: { refused?: boolean } = {}) { const snapshotStore = new InMemorySnapshots(events); const sessions = new InMemorySessions(events); const identity = new InMemoryIdentity(); - const resolutions = new InMemoryResolutions(); - const directories = new DirectoryResolver(locator, consents, resolutions, history, { - caseInsensitiveFileSystem: false, - now: () => new Date("2026-10-09T00:00:00.000Z"), - }); const ingest = new IngestUsageUseCase( [new ReadUsageUseCase(transcripts, () => readClaudeUsageLine)], ledger, - directories, + new DirectoryResolver(locator, consents, new InMemoryResolutions(), history, { + caseInsensitiveFileSystem: false, + now: () => new Date("2026-10-09T00:00:00.000Z"), + }), new SnapshotBindingsUseCase( bindings, snapshotStore, @@ -102,33 +98,17 @@ function setup(options: { refused?: boolean } = {}) { remote: "https://github.com/acme/widgets.git", rootCommit: "c0ffee", }); - rememberResolution( - resolutions.resolutions, - { dir: "/work/a", root: "/work/a", clone: cloneOf("/work/a/.git"), repository_id: widgets() }, - "2026-01-01T00:00:00.000Z" - ); consents.cloneSays(cloneOf("/work/a/.git"), grantedBy(cloneOf("/work/a/.git"))); history.consented(cloneOf("/work/a/.git")); const report = new ReportUsageUseCase( ingest, - new ReportLedger(ledger, directories), + ledger, new DeclaredBindings(sessions, snapshotStore), identity, locator, { refusedByEnvironment: options.refused ?? false } ); - return { - transcripts, - ledger, - bindings, - sessions, - identity, - report, - events, - locator, - consents, - resolutions, - }; + return { transcripts, ledger, bindings, sessions, identity, report, events, locator, consents }; } async function rows(s: ReturnType<typeof setup>, axis: ReportAxis, period = ANY) { @@ -138,39 +118,6 @@ async function rows(s: ReturnType<typeof setup>, axis: ReportAxis, period = ANY) } describe("reporting usage", () => { - it("keeps tokens but stops attributing old usage after an origin change, even when it changes back", async () => { - const s = setup(); - s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); - expect(await rows(s, "repository")).toEqual([[{ kind: "value", value: widgets() }, 11]]); - const original = s.locator.directories.get("/work/a"); - if (original?.status !== "repository") throw new Error("missing repository"); - s.locator.directories.set("/work/a", { - ...original, - remote: "https://github.com/acme/other.git", - }); - expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); - expect(await rows(s, "task")).toEqual([[{ kind: "unattributed", reason: "no-binding" }, 11]]); - s.locator.directories.set("/work/a", original); - expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); - }); - - it("keeps tokens without repository attribution when resolution history was lost", async () => { - const s = setup(); - s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); - expect(await rows(s, "repository")).toEqual([[{ kind: "value", value: widgets() }, 11]]); - s.resolutions.resolutions.clear(); - expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); - }); - - it("keeps tokens without repository or task attribution when a directory is gone", async () => { - const s = setup(); - s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); - expect(await rows(s, "repository")).toEqual([[{ kind: "value", value: widgets() }, 11]]); - s.locator.directories.delete("/work/a"); - expect(await rows(s, "repository")).toEqual([[{ kind: "absent" }, 11]]); - expect(await rows(s, "task")).toEqual([[{ kind: "unattributed", reason: "no-binding" }, 11]]); - }); - it("settles a call held in two partitions under the ledger lock", async () => { const s = setup(); s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); diff --git a/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts index 107b6eede..4b91d1039 100644 --- a/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts @@ -12,7 +12,6 @@ import { storeCalls } from "../../../../../src/contexts/telemetry/application/le import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; import { ReportUsageUseCase } from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; import { SnapshotBindingsUseCase } from "../../../../../src/contexts/telemetry/application/snapshot-bindings-use-case.js"; import type { UsageLedger } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; import { attribute } from "../../../../../src/contexts/telemetry/domain/report/attribution.js"; @@ -247,7 +246,7 @@ describe("a report on a bounded period", () => { await settled.leftOut.add(leftOut); const bounded = await new ReportUsageUseCase( ingestOf(settled), - new ReportLedger(settled), + settled, declared, identity, new FakeLocator(), @@ -257,9 +256,7 @@ describe("a report on a bounded period", () => { const fullLedger = await ledgerWith(files); await fullLedger.leftOut.add(leftOut); const full = await fullLedger.load(); - const kept = full.records - .filter((record) => inPeriod(record.at, period)) - .map((record) => ({ ...record, repository_uncertain: true as const })); + const kept = full.records.filter((record) => inPeriod(record.at, period)); // A session or repository typed by its start names a value among the period's calls. const resolved: { -readonly [K in keyof ReportFilters]: ReportFilters[K] } = { ...filters }; for (const flag of ["session", "repository"] as const) { @@ -267,7 +264,7 @@ describe("a report on a bounded period", () => { if (typed === null) continue; const found = resolveId( typed, - kept.flatMap((record) => (flag === "session" ? [record.session_id] : [])) + kept.map((record) => (flag === "session" ? record.session_id : record.repository_id)) ); if (found.kind === "ambiguous") { expect(bounded).toEqual({ status: "ambiguous", flag, typed, ids: found.ids }); diff --git a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts deleted file mode 100644 index 55fe9a45b..000000000 --- a/cli/tests/contexts/telemetry/application/report/session-consent.unit.test.ts +++ /dev/null @@ -1,273 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { DirectoryResolver } from "../../../../../src/contexts/telemetry/application/directory-resolver.js"; -import { SessionConsent } from "../../../../../src/contexts/telemetry/application/report/session-consent.js"; -import { cloneKey } from "../../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; -import { resolutionKey } from "../../../../../src/contexts/telemetry/domain/repository-resolution.js"; -import { consentValue } from "../../../../../src/contexts/telemetry/domain/telemetry-consent.js"; -import { - cloneOf, - FakeConsents, - FakeLocator, - grantedBy, - InMemoryConsentHistory, - InMemoryLedger, - InMemoryResolutions, -} from "../../../../helpers/ports/in-memory-telemetry.js"; - -const cwd = "/work/repo"; -const repo = "repository-one"; -const at = "2026-10-07T10:30:00.000Z"; -const start = "2026-10-07T10:00:00.000Z"; -const admission = { feature: { state: "unbound" as const }, cwd, repositoryId: repo }; - -function setup(history = new InMemoryConsentHistory()) { - const ledger = new InMemoryLedger(); - const locator = new FakeLocator(); - const source = new FakeConsents(); - const resolutions = new InMemoryResolutions(); - const clone = cloneOf(`${cwd}/.git`); - locator.directories.set(cwd, { - status: "repository", - root: cwd, - mainRoot: cwd, - clone, - remote: null, - rootCommit: repo, - }); - source.cloneSays(clone, grantedBy(clone)); - history.consented(clone, "2026-10-07T09:00:00.000Z"); - const directories = new DirectoryResolver(locator, source, resolutions, history, { - caseInsensitiveFileSystem: false, - now: () => new Date("2026-10-07T12:00:00.000Z"), - }); - return { - checker: new SessionConsent(ledger, directories, history), - ledger, - locator, - source, - resolutions, - history, - clone, - }; -} - -describe("consent for an observed session", () => { - it("proves the consented clone and refuses a replacement at the same path", async () => { - const s = setup(); - const original = await s.checker.proof(admission, at, start, "2026-10-07T11:00:00.000Z"); - expect(original).toMatch(/^[a-f0-9]{64}$/); - expect(await s.checker.proof(admission, at, start, "2026-10-07T11:00:00.000Z")).toBe(original); - const replacement = cloneOf(`${cwd}/.git`, { - ino: "9", - birthtimeMs: Date.parse("2026-10-07T10:45:00.000Z"), - }); - s.locator.directories.set(cwd, { - status: "repository", - root: cwd, - mainRoot: cwd, - clone: replacement, - remote: null, - rootCommit: repo, - }); - s.history.consented(replacement, "2026-10-07T10:45:00.000Z"); - s.source.cloneSays(replacement, grantedBy(replacement)); - expect(await s.checker.proof(admission, at, start, "2026-10-07T11:00:00.000Z")).toBeNull(); - }); - - it("allows an instant or whole interval only inside one clone's consent", async () => { - const s = setup(); - expect(await s.checker.permits(admission, at)).toBe("allow"); - expect(await s.checker.permits(admission, at, start)).toBe("allow"); - expect(s.ledger.events).toEqual(["lock", "unlock", "lock", "unlock"]); - }); - - it("rejects a cumulative interval crossing an opt-out even after opt-in resumes", async () => { - const s = setup(); - const first = s.history.written[0]; - if (first?.kind !== "open") throw new Error("Missing initial consent"); - s.history.written.push( - { kind: "close", token: first.token, at: "2026-10-07T10:15:00.000Z" }, - { kind: "open", token: "new-consent", clone: s.clone, at: "2026-10-07T10:20:00.000Z" } - ); - s.source.cloneSays(s.clone, consentValue("new-consent")); - expect(await s.checker.permits(admission, at, start)).toBe("deny"); - expect(await s.checker.permits(admission, start, undefined, at)).toBe("allow"); - expect(await s.checker.permits(admission, at, "2026-10-07T10:20:00.000Z")).toBe("allow"); - expect(await s.checker.permits(admission, "2026-10-07T10:15:00.000Z", start)).toBe("deny"); - }); - - it("denies a window spanning two different clones at the same path", async () => { - const s = setup(); - const newClone = cloneOf(`${cwd}/.git`, { - ino: "8", - birthtimeMs: Date.parse("2026-10-07T10:15:00.000Z"), - }); - s.resolutions.resolutions.set(resolutionKey(cwd, s.clone), { - dir: cwd, - root: cwd, - repository_id: repo, - clone: s.clone, - seen_at: "2026-10-07T09:00:00.000Z", - }); - s.locator.directories.set(cwd, { - status: "repository", - root: cwd, - mainRoot: cwd, - clone: newClone, - remote: null, - rootCommit: repo, - }); - s.history.consented(newClone, "2026-10-07T10:15:00.000Z"); - s.source.cloneSays(newClone, grantedBy(newClone)); - expect(await s.checker.permits(admission, at, start)).toBe("deny"); - expect(await s.checker.permits(admission, at)).toBe("allow"); - expect(await s.checker.permits(admission, start, undefined, at)).toBe("deny"); - }); - - it("denies a repository other than the session's admitted repository", async () => { - const s = setup(); - expect(await s.checker.permits({ ...admission, repositoryId: "other" }, at)).toBe("deny"); - }); - - it("allows unlinked token observations without a repository only for the same consented clone", async () => { - const s = setup(); - const uncertain = { ...admission, repositoryId: null }; - expect(await s.checker.permits(uncertain, at, start)).toBe("allow"); - expect( - await s.checker.permits( - { ...uncertain, feature: { state: "confirmed", feature: "owner/repo#1", branch: "work" } }, - at - ) - ).toBe("unknown"); - const replacement = cloneOf(`${cwd}/.git`, { - ino: "9", - birthtimeMs: Date.parse("2026-10-07T10:45:00.000Z"), - }); - s.resolutions.resolutions.set(resolutionKey(cwd, s.clone), { - dir: cwd, - root: cwd, - repository_id: repo, - clone: s.clone, - seen_at: "2026-10-07T09:00:00.000Z", - }); - s.locator.directories.set(cwd, { - status: "repository", - root: cwd, - mainRoot: cwd, - clone: replacement, - remote: null, - rootCommit: repo, - }); - s.history.consented(replacement, "2026-10-07T10:45:00.000Z"); - s.source.cloneSays(replacement, grantedBy(replacement)); - expect(await s.checker.permits(uncertain, at, undefined, "2026-10-07T11:00:00.000Z")).toBe( - "deny" - ); - }); - - it.each(["start", "end"] as const)( - "keeps a window unknown when its %s is unreadable and its other end is denied", - async (unreadable) => { - const s = setup(); - const newClone = cloneOf(`${cwd}/.git`, { - ino: "8", - birthtimeMs: Date.parse("2026-10-07T10:15:00.000Z"), - }); - s.resolutions.resolutions.set(resolutionKey(cwd, s.clone), { - dir: cwd, - root: cwd, - repository_id: repo, - clone: s.clone, - seen_at: "2026-10-07T09:00:00.000Z", - }); - s.locator.directories.set(cwd, { - status: "repository", - root: cwd, - mainRoot: cwd, - clone: newClone, - remote: null, - rootCommit: repo, - }); - const initial = s.history.written[0]; - if (initial?.kind !== "open") throw new Error("Missing initial consent"); - if (unreadable === "end") - s.history.written[0] = { ...initial, at: "2026-10-07T10:10:00.000Z" }; - s.history.consented( - newClone, - unreadable === "start" ? "2026-10-07T10:45:00.000Z" : "2026-10-07T10:20:00.000Z" - ); - s.source.cloneSays(newClone, grantedBy(newClone)); - const unavailableClone = unreadable === "start" ? s.clone : newClone; - s.source.unreadableClones.add(cloneKey(unavailableClone)); - expect(await s.checker.permits(admission, at, start)).toBe("unknown"); - expect(await s.checker.permits(admission, start)).toBe( - unreadable === "start" ? "unknown" : "deny" - ); - expect(await s.checker.permits(admission, at)).toBe( - unreadable === "end" ? "unknown" : "deny" - ); - expect(s.ledger.events).toEqual(["lock", "unlock", "lock", "unlock", "lock", "unlock"]); - } - ); - - it.each([ - [{ ...admission, cwd: null }, at, undefined], - [{ ...admission, cwd: "" }, at, undefined], - [admission, "invalid", undefined], - [admission, at, "invalid"], - [admission, start, at], - ])( - "leaves incomplete or inconsistent evidence unknown without reading the ledger", - async (a, end, from) => { - const s = setup(); - expect(await s.checker.permits(a, end, from)).toBe("unknown"); - expect(s.ledger.exclusiveRuns).toBe(0); - } - ); - - it.each([ - ["gone", "unknown"], - ["unreadable", "unknown"], - ["outside-repository", "deny"], - ] as const)("does not permit an unresolved %s directory", async (status, expected) => { - const s = setup(); - s.locator.directories.set(cwd, { status }); - expect(await s.checker.permits(admission, at)).toBe(expected); - }); - - it("denies a clone without consent, and treats damaged history as unknown", async () => { - const s = setup(); - s.history.written.length = 0; - expect(await s.checker.permits(admission, at)).toBe("deny"); - s.history.damaged = true; - expect(await s.checker.permits(admission, at)).toBe("unknown"); - }); - - it("does not authorize when the history becomes damaged after directory resolution", async () => { - class BecomesDamaged extends InMemoryConsentHistory { - calls = 0; - override async read() { - const result = await super.read(); - return { ...result, damaged: ++this.calls > 1 }; - } - } - const s = setup(new BecomesDamaged()); - expect(await s.checker.permits(admission, at)).toBe("unknown"); - expect(s.resolutions.saves).toBe(1); - expect(s.ledger.events).toEqual(["lock", "unlock"]); - }); - - it("closes directory resolution and releases the lock when the history read fails", async () => { - class FailsOnRecheck extends InMemoryConsentHistory { - calls = 0; - override async read() { - if (++this.calls > 1) throw new Error("history unavailable"); - return super.read(); - } - } - const s = setup(new FailsOnRecheck()); - await expect(s.checker.permits(admission, at)).rejects.toThrow("history unavailable"); - expect(s.resolutions.saves).toBe(1); - expect(s.ledger.events).toEqual(["lock", "unlock"]); - }); -}); diff --git a/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts deleted file mode 100644 index 2dbfe7546..000000000 --- a/cli/tests/contexts/telemetry/application/report/session-feature-use-case.unit.test.ts +++ /dev/null @@ -1,144 +0,0 @@ -import { expect, it, vi } from "vitest"; -import { DirectoryResolver } from "../../../../../src/contexts/telemetry/application/directory-resolver.js"; -import type { IngestResult } from "../../../../../src/contexts/telemetry/application/ingest-usage-use-case.js"; -import { ReportLedger } from "../../../../../src/contexts/telemetry/application/report/repository-uncertainty.js"; -import { SessionConsent } from "../../../../../src/contexts/telemetry/application/report/session-consent.js"; -import { SessionFeatureUseCase } from "../../../../../src/contexts/telemetry/application/report/session-feature-use-case.js"; -import type { UsageLedger } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; -import { - cloneOf, - FakeConsents, - FakeLocator, - grantedBy, - InMemoryConsentHistory, - InMemoryLedger, - InMemoryResolutions, -} from "../../../../helpers/ports/in-memory-telemetry.js"; - -it("refuses synchronization before reading the ledger when telemetry is disabled", async () => { - const unexpected = (): never => { - throw new Error("Unexpected read"); - }; - const exclusively = vi.fn(unexpected); - const ledger: UsageLedger = { - exclusively, - leftOut: { add: unexpected, counts: unexpected }, - load: unexpected, - save: unexpected, - leaveOutRequestless: unexpected, - positions: unexpected, - savePositions: unexpected, - }; - const refused: IngestResult = { - refused: true, - filesRead: 0, - added: 0, - updated: 0, - unrecognised: 0, - skippedLedgerLines: 0, - snapshots: 0, - notStored: { - "outside-repo": 0, - "never-seen-alive": 0, - "no-consent": 0, - "consent-closed": 0, - "unreadable-consent": 0, - "no-cwd": 0, - undated: 0, - }, - consentLogDamaged: false, - oldestTranscriptAt: null, - }; - const useCase = new SessionFeatureUseCase( - { execute: async () => refused }, - new ReportLedger(ledger), - { read: unexpected }, - { permits: unexpected } - ); - - await expect(useCase.snapshot("codex")).rejects.toThrow("telemetry is disabled"); - expect(exclusively).not.toHaveBeenCalled(); -}); - -it("withdraws a feature after an observed origin change without losing consented usage", async () => { - const cwd = "/work/repo"; - const clone = cloneOf(`${cwd}/.git`); - const locator = new FakeLocator(); - const source = new FakeConsents(); - const history = new InMemoryConsentHistory(); - const ledger = new InMemoryLedger(); - let now = "2026-10-07T09:30:00.000Z"; - const repository = { - status: "repository" as const, - root: cwd, - mainRoot: cwd, - clone, - remote: null, - rootCommit: "repo-a", - }; - locator.directories.set(cwd, repository); - source.cloneSays(clone, grantedBy(clone)); - history.consented(clone, "2026-10-07T09:00:00.000Z"); - const directories = new DirectoryResolver(locator, source, new InMemoryResolutions(), history, { - caseInsensitiveFileSystem: false, - now: () => new Date(now), - }); - const remembered = await directories.open(); - await remembered.resolve(cwd, now); - await remembered.close(); - now = "2026-10-07T11:00:00.000Z"; - ledger.records = [ - { - tool: "codex", - tool_version: null, - key: "request", - session_id: "session", - agent: "main", - at: "2026-10-07T10:00:00.000Z", - model: null, - input: 42, - output: 9, - cache_read: null, - cache_write: null, - cache_write_1h: null, - reasoning: null, - cwd, - git_branch: "work", - repository_id: "repo-a", - }, - ]; - const facts = { - declarations: [ - { - session_id: "session", - task: "work", - ticket: "owner/repo#7", - none: false, - declared_at: "2026-10-07T09:00:00.000Z", - by: "command" as const, - }, - ], - carries: [], - branches: new Map(), - }; - const useCase = new SessionFeatureUseCase( - { execute: async () => ({ refused: false }) as IngestResult }, - new ReportLedger(ledger, directories), - { read: async () => facts }, - new SessionConsent(ledger, directories, history) - ); - expect((await useCase.snapshot("codex")).get("session")).toMatchObject({ - repositoryId: "repo-a", - feature: { state: "confirmed", feature: "owner/repo#7" }, - }); - - now = "2026-10-07T12:00:00.000Z"; - locator.directories.set(cwd, { ...repository, rootCommit: "repo-b" }); - const admission = (await useCase.snapshot("codex")).get("session"); - expect(admission).toEqual({ cwd, repositoryId: null, feature: { state: "ambiguous" } }); - if (!admission) throw new Error("Missing session admission"); - expect(await useCase.permits(admission, "2026-10-07T10:00:00.000Z", undefined, now)).toBe( - "allow" - ); - expect(ledger.records[0]).toMatchObject({ input: 42, output: 9 }); -}); diff --git a/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts b/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts index 48ca3cc7f..59f57aeeb 100644 --- a/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/switch/telemetry-switch-use-cases.unit.test.ts @@ -549,7 +549,7 @@ describe("aidd telemetry off, kept as the end of an interval", () => { await s.on.execute("/w/repo", "keep"); s.consents.values.set("/w/repo", "off"); s.clock.now = new Date(T2); - expect(await s.off.execute("/w/repo")).toEqual({ status: "off", changed: true, closedAt: T2 }); + expect(await s.off.execute("/w/repo")).toEqual({ status: "off", changed: true }); expect(s.history.written.at(-1)).toEqual({ kind: "close", token: "tok-1", at: T2 }); }); diff --git a/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts b/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts index cb3ab83fe..52edf4a8c 100644 --- a/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts +++ b/cli/tests/contexts/telemetry/domain/repository-resolution.unit.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest"; import type { CloneIdentity } from "../../../../src/contexts/telemetry/domain/consent/clone-identity.js"; import { cwdKey, - ownersAt, + ownerAt, parseResolutions, type RepositoryResolution, renderResolutions, @@ -119,18 +119,18 @@ describe("the clone that answers for a call", () => { const next = { ...resolution, clone: { ...CLONE, ino: "2", birthtimeMs: 5_000 } }; it("is the only clone there is, whenever the call was made", () => { - expect(ownersAt([old], 0)).toEqual([old]); - expect(ownersAt([old], 1e12)).toEqual([old]); + expect(ownerAt([old], 0)).toBe(old); + expect(ownerAt([old], 1e12)).toBe(old); }); it("is the latest clone born by then", () => { - expect(ownersAt([next, old], 4_999)).toEqual([old]); - expect(ownersAt([next, old], 5_000)).toEqual([next]); - expect(ownersAt([old, next], 9_000)).toEqual([next]); + expect(ownerAt([next, old], 4_999)).toBe(old); + expect(ownerAt([next, old], 5_000)).toBe(next); + expect(ownerAt([old, next], 9_000)).toBe(next); }); it("is the first when the call is older than every clone", () => { - expect(ownersAt([next, old], 10)).toEqual([old]); + expect(ownerAt([next, old], 10)).toBe(old); }); it("goes by the time ingest first saw the directory, where a clone has no birth time", () => { @@ -144,13 +144,12 @@ describe("the clone that answers for a call", () => { clone: { ...next.clone, birthtimeMs: 0 }, seen_at: "2026-10-05T00:00:00.000Z", }; - expect(ownersAt([late, early], Date.parse("2026-10-04T00:00:00.000Z"))).toEqual([early]); - expect(ownersAt([early, late], Date.parse("2026-10-05T00:00:00.000Z"))).toEqual([late]); + expect(ownerAt([late, early], Date.parse("2026-10-04T00:00:00.000Z"))).toBe(early); + expect(ownerAt([early, late], Date.parse("2026-10-05T00:00:00.000Z"))).toBe(late); }); - it("leaves tied birth times unresolved", () => { + it("is the same whichever order the clones are given in", () => { const tie = { ...old, clone: { ...old.clone, ino: "3" } }; - expect(ownersAt([old, tie], 1_000)).toEqual([old, tie]); - expect(ownersAt([tie, old], 1_000)).toEqual([tie, old]); + expect(ownerAt([old, tie], 1_000)).toBe(ownerAt([tie, old], 1_000)); }); }); diff --git a/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts b/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts deleted file mode 100644 index b450503da..000000000 --- a/cli/tests/contexts/telemetry/domain/session-feature.unit.test.ts +++ /dev/null @@ -1,86 +0,0 @@ -import { describe, expect, it } from "vitest"; -import type { AttributionFacts } from "../../../../src/contexts/telemetry/domain/report/attribution.js"; -import { - coversConsentWindow, - sameSessionAdmission, - sessionFeatureOf, -} from "../../../../src/contexts/telemetry/domain/report/session-feature.js"; -import type { StoredUsage } from "../../../../src/contexts/telemetry/domain/stored-usage.js"; - -const facts: AttributionFacts = { declarations: [], carries: [], branches: new Map() }; -const call: StoredUsage = { - tool: "codex", - tool_version: null, - key: "request", - session_id: "session", - agent: "main", - at: "2026-11-04T00:00:00Z", - model: null, - input: 1, - output: 1, - cache_read: null, - cache_write: null, - cache_write_1h: null, - reasoning: null, - cwd: null, - git_branch: "feature/work", - repository_id: "repo", -}; - -describe("session feature binding", () => { - it("leaves initially unbound sessions unbound", () => { - expect(sessionFeatureOf([call], facts)).toEqual({ state: "unbound" }); - }); - it("confirms only the same declared ticket and branch for every call", () => { - const declared: AttributionFacts = { - ...facts, - declarations: [ - { - session_id: "session", - task: "work", - ticket: "owner/repo#7", - none: false, - declared_at: "2026-11-03T00:00:00Z", - by: "command", - }, - ], - }; - expect(sessionFeatureOf([call], declared)).toEqual({ - state: "confirmed", - feature: "owner/repo#7", - branch: "feature/work", - }); - expect(sessionFeatureOf([call, { ...call, git_branch: "feature/other" }], declared)).toEqual({ - state: "ambiguous", - }); - expect(sessionFeatureOf([call, { ...call, git_branch: null }], declared)).toEqual({ - state: "ambiguous", - }); - expect(sessionFeatureOf([call, { ...call, repository_uncertain: true }], declared)).toEqual({ - state: "ambiguous", - }); - }); -}); - -it("keeps an uncertain repository unlinked while allowing the same session to continue", () => { - const uncertain = { feature: { state: "ambiguous" as const }, cwd: "/work", repositoryId: null }; - expect(sameSessionAdmission(uncertain, uncertain)).toBe(true); - expect(sameSessionAdmission(uncertain, { ...uncertain, cwd: "/elsewhere" })).toBe(false); - const confirmed = { - ...uncertain, - feature: { state: "confirmed" as const, feature: "owner/repo#7", branch: "work" }, - }; - expect(sameSessionAdmission(confirmed, confirmed)).toBe(false); -}); - -it("never exports a cumulative metric across an off/on consent gap", () => { - const clone = { path: "/repo/.git", dev: "1", ino: "2", birthtimeMs: 0 }; - const intervals = [ - { token: "one", clone, from: 100, to: 200 }, - { token: "two", clone, from: 300, to: null }, - ]; - expect(coversConsentWindow(intervals, clone, 120, 190)).toBe(true); - expect(coversConsentWindow(intervals, clone, 120, 350)).toBe(false); - expect(coversConsentWindow(intervals, clone, 300, 350)).toBe(true); - expect(coversConsentWindow(intervals, clone, 190, 200)).toBe(false); -}); diff --git a/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts new file mode 100644 index 000000000..bece11e52 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts @@ -0,0 +1,58 @@ +import { expect, it } from "vitest"; +import { openCodePromptSpans } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; + +const text = (key: string, stringValue: string) => ({ key, value: { stringValue } }); + +function capture(overrides: Record<string, unknown> = {}) { + return JSON.stringify({ + signal: "traces", + payload: { + resourceSpans: [ + { + resource: { + attributes: [ + text("service.name", "opencode"), + text("service.version", "2.2.5"), + text("host.name", "private-host"), + ], + }, + scopeSpans: [ + { + spans: [ + { + traceId: "00000000000000000000000000000001", + spanId: "0000000000000002", + name: "POST", + kind: 2, + startTimeUnixNano: "1700000000000000000", + endTimeUnixNano: "1700000000000000001", + attributes: [ + text("http.route", "/api/session/:sessionID/prompt"), + text("url.path", "/api/session/ses_probe/prompt"), + text("http.request.method", "POST"), + { key: "http.response.status_code", value: { intValue: "200" } }, + text("url.full", "http://localhost/private?secret=value"), + ], + ...overrides, + }, + ], + }, + ], + }, + ], + }, + }); +} + +it("forwards only allowlisted OpenCode prompt span data", () => { + const output = openCodePromptSpans(capture()); + expect(output).toHaveLength(1); + expect(JSON.stringify(output)).toContain('"url.path"'); + expect(JSON.stringify(output)).toContain('"spanId":"0000000000000002"'); + expect(JSON.stringify(output)).not.toMatch(/url.full|host.name|secret/); +}); + +it("rejects client spans and invalid timestamps", () => { + expect(openCodePromptSpans(capture({ kind: 3 }))).toEqual([]); + expect(openCodePromptSpans(capture({ startTimeUnixNano: "1700000000000000002" }))).toEqual([]); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts deleted file mode 100644 index c06989243..000000000 --- a/cli/tests/contexts/telemetry/otel/sync-all-otel.integration.test.ts +++ /dev/null @@ -1,1029 +0,0 @@ -import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; -import { createServer, type Server } from "node:http"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { afterEach, expect, it as test } from "vitest"; -import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; - -const it = test.skipIf(process.platform === "win32"); -const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); -function line(session: string) { - return `${JSON.stringify({ - signal: "logs", - payload: { - resourceLogs: [ - { - scopeLogs: [ - { - logRecords: [ - { - timeUnixNano: "1793763900000000000", - attributes: [ - attribute("conversation.id", session), - attribute("user.email", "developer@example.test"), - attribute("event.name", "codex.sse_event"), - attribute("event.kind", "response.completed"), - attribute("input_token_count", "42"), - ], - }, - ], - }, - ], - }, - ], - }, - })}\n`; -} -let server: Server | undefined; -const dirs: string[] = []; -afterEach(async () => { - if (server) await new Promise<void>((resolve) => server?.close(() => resolve())); - await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); - server = undefined; -}); - -it("discovers a newly consented session without transmitting a different session", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - let snapshotCount = 0; - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => { - snapshotCount++; - return snapshotCount > 1 ? new Map([["allowed", admission]]) : new Map(); - }, - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed") + line("not-consented")); - const result = await running; - expect(result.sent).toBe(1); - expect(received).toHaveLength(1); - expect(received[0]).toContain("allowed"); - expect(received[0]).not.toContain("not-consented"); -}); - -it("forwards token observations for a consented session whose repository is uncertain", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-uncertain-repo-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "ambiguous" as const }, - cwd: "/project", - repositoryId: null, - }; - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["allowed", admission]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - const result = await running; - expect(result.sent).toBe(1); - expect(received[0]).toContain("input_token_count"); - expect(received[0]).not.toContain("aidd.session_feature"); -}); - -it("retries a denial caused by an admission changing during the consent check", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-stale-admission-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const known = { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "repo" }; - const uncertain = { - feature: { state: "ambiguous" as const }, - cwd: "/project", - repositoryId: null, - }; - let changed = false; - const timeout = setTimeout(() => abort.abort(), 3000); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - refreshMs: 20, - }, - async () => new Map([["allowed", changed ? uncertain : known]]), - async (admission) => { - if (admission.repositoryId !== null) { - changed = true; - return "deny"; - } - return "allow"; - } - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - const result = await running; - clearTimeout(timeout); - expect(result.denied).toBe(0); - expect(received.some((body) => body.includes("input_token_count"))).toBe(true); -}); - -it("forwards consented observations when the same capture line also contains denied or undated records", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-mixed-consent-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const validAt = new Date(1793763900000).toISOString(); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["allowed", admission]]), - async (_admission, at) => (at === validAt ? "allow" : "deny"), - async (_admission, at) => (at === validAt ? "a".repeat(64) : null) - ); - const payload = JSON.parse(line("allowed")); - const records = payload.payload.resourceLogs[0].scopeLogs[0].logRecords; - const valid = records[0]; - const denied = structuredClone(valid); - denied.timeUnixNano = "1793763800000000000"; - denied.attributes.find( - (entry: { key: string }) => entry.key === "input_token_count" - ).value.stringValue = "17"; - const undated = structuredClone(valid); - delete undated.timeUnixNano; - undated.attributes.find( - (entry: { key: string }) => entry.key === "input_token_count" - ).value.stringValue = "28"; - records.push(denied, undated); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, `${JSON.stringify(payload)}\n`); - const result = await running; - expect(result.denied).toBe(2); - expect(result.pending).toBe(0); - expect(received).toHaveLength(1); - expect(received[0]).toContain('"42"'); - expect(received[0]).not.toContain('"17"'); - expect(received[0]).not.toContain('"28"'); -}); - -it("waits for every consent verdict before sending a capture line once", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-unknown-batch-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const firstAt = new Date(1793763900000).toISOString(); - let ready = false; - const timeout = setTimeout(() => abort.abort(), 3000); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - refreshMs: 20, - }, - async () => new Map([["allowed", admission]]), - async (_admission, at) => (at === firstAt || ready ? "allow" : "unknown"), - async () => "a".repeat(64) - ); - const payload = JSON.parse(line("allowed")); - const records = payload.payload.resourceLogs[0].scopeLogs[0].logRecords; - const second = structuredClone(records[0]); - second.timeUnixNano = "1793763901000000000"; - records.push(second); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, `${JSON.stringify(payload)}\n`); - await new Promise((resolve) => setTimeout(resolve, 100)); - expect(received).toHaveLength(0); - ready = true; - const result = await running; - clearTimeout(timeout); - expect(result.pending).toBe(0); - expect(received).toHaveLength(1); - expect(received[0]).toContain("1793763900000000000"); - expect(received[0]).toContain("1793763901000000000"); -}); - -it("keeps an oversized line pending without sending a partial batch", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-oversize-batch-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let received = 0; - server = createServer((request, response) => { - request.resume(); - received++; - response.writeHead(200).end("{}"); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const abort = new AbortController(); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["allowed", admission]]), - async () => "allow", - async () => "a".repeat(64) - ); - const payload = JSON.parse(line("allowed")); - const records = payload.payload.resourceLogs[0].scopeLogs[0].logRecords; - records[0].attributes.push(attribute("model", "x".repeat(1_100_000))); - records.push(structuredClone(records[0])); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, `${JSON.stringify(payload)}\n`); - await new Promise((resolve) => setTimeout(resolve, 200)); - abort.abort(); - const result = await running; - expect(result.pending).toBe(1); - expect(received).toBe(0); -}); - -it("keeps one oversized observation pending and continues with the next line", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-single-oversize-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - const abort = new AbortController(); - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const timeout = setTimeout(() => abort.abort(), 3000); - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["allowed", admission]]), - async () => "allow", - async () => "a".repeat(64) - ); - const payload = JSON.parse(line("allowed")); - payload.payload.resourceLogs[0].scopeLogs[0].logRecords[0].attributes.push( - attribute("model", "x".repeat(2_100_000)) - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, `${JSON.stringify(payload)}\n${line("allowed")}`); - const result = await running; - clearTimeout(timeout); - expect(result.pending).toBe(1); - expect(received).toHaveLength(1); - expect(received[0]).toContain("input_token_count"); - expect(received[0]).not.toContain("xxxxxxxxxxxxxxxxxxxxxxxx"); -}); - -it("replays a held v2 range after restart only for the original consented clone", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-pending-v2-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - refreshMs: 20, - }; - const known = { feature: { state: "unbound" as const }, cwd: "/project", repositoryId: "repo" }; - const uncertain = { - feature: { state: "ambiguous" as const }, - cwd: "/project", - repositoryId: null, - }; - const firstAbort = new AbortController(); - const first = syncAllOtel( - { ...options, signal: firstAbort.signal }, - async () => new Map([["allowed", known]]), - async (_admission, _at, _start, current) => (current ? "unknown" : "allow"), - async () => "a".repeat(64) - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 100)); - firstAbort.abort(); - expect((await first).pending).toBe(1); - const secondAbort = new AbortController(); - const second = syncAllOtel( - { ...options, signal: secondAbort.signal }, - async () => new Map([["allowed", uncertain]]), - async () => "allow", - async () => "b".repeat(64) - ); - await new Promise((resolve) => setTimeout(resolve, 100)); - secondAbort.abort(); - expect((await second).pending).toBe(1); - expect(received).toHaveLength(0); - const thirdAbort = new AbortController(); - const third = syncAllOtel( - { ...options, signal: thirdAbort.signal }, - async () => new Map([["allowed", uncertain]]), - async () => "allow", - async () => "a".repeat(64) - ); - await new Promise((resolve) => setTimeout(resolve, 100)); - thirdAbort.abort(); - expect((await third).pending).toBe(0); - expect(received.some((body) => body.includes("input_token_count"))).toBe(true); -}); - -it("replays a failed post from its last durable checkpoint", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-retry-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let fail = true; - let calls = 0; - const abort = new AbortController(); - server = createServer((request, response) => { - request.resume(); - request.on("end", () => { - calls++; - response.writeHead(fail ? 403 : 200).end("{}"); - if (!fail) abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const snapshot = async () => new Map([["allowed", admission]]); - const first = syncAllOtel(options, snapshot, async () => "allow"); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await expect(first).rejects.toThrow("HTTP 403"); - fail = false; - const result = await syncAllOtel( - { ...options, signal: abort.signal }, - snapshot, - async () => "allow" - ); - expect(result.sent).toBe(1); - expect(calls).toBe(2); -}); - -it("retries a transient HTTP failure without losing its checkpoint", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-retry-live-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const abort = new AbortController(); - let calls = 0; - server = createServer((request, response) => { - request.resume(); - request.on("end", () => { - calls++; - response.writeHead(calls === 1 ? 503 : 200).end("{}"); - if (calls > 1) abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["allowed", admission]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - const result = await running; - expect(calls).toBe(2); - expect(result.sent).toBe(1); -}); - -it("retains an unsent capture across consent withdrawal and resumes only after consent returns", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-withdrawn-all-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let consenting = true; - let calls = 0; - const abort = new AbortController(); - const firstAbort = new AbortController(); - server = createServer((request, response) => { - request.resume(); - request.on("end", () => { - calls++; - if (calls === 1) { - consenting = false; - response.writeHead(503).end("{}"); - setTimeout(() => firstAbort.abort(), 100); - } else { - response.writeHead(200).end("{}"); - abort.abort(); - } - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const snapshot = async () => new Map([["allowed", admission]]); - const historical = new Date(1_793_763_900_000).toISOString(); - const permits = async (_admission: unknown, at: string, _start?: string, current?: string) => - at === historical && (!current || consenting) ? ("allow" as const) : ("deny" as const); - const first = syncAllOtel({ ...options, signal: firstAbort.signal }, snapshot, permits); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - expect(await first).toMatchObject({ sent: 0, pending: 1 }); - expect(calls).toBe(1); - consenting = true; - const result = await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - expect(result.sent).toBe(1); - expect(calls).toBe(2); -}); - -it("isolates a revoked session across restarts while another session continues", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-two-sessions-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let consenting = true; - let historicalAllowed = true; - let firstFailure = true; - let abort = new AbortController(); - const accepted: string[] = []; - const attempts: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - const id = body.includes('"stringValue":"one"') ? "one" : "two"; - attempts.push(id); - if (id === "one" && firstFailure) { - firstFailure = false; - consenting = false; - response.writeHead(503).end("{}"); - } else { - accepted.push(id); - response.writeHead(200).end("{}"); - abort.abort(); - } - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - refreshMs: 20, - }; - const one = { feature: { state: "unbound" as const }, cwd: "/one", repositoryId: "one-repo" }; - const two = { feature: { state: "unbound" as const }, cwd: "/two", repositoryId: "two-repo" }; - const snapshot = async () => - new Map([ - ["one", one], - ["two", two], - ]); - const permits = async ( - admission: { repositoryId: string | null }, - _at: string, - _start?: string, - current?: string - ) => - admission.repositoryId === "one-repo" && (!historicalAllowed || (!!current && !consenting)) - ? ("deny" as const) - : ("allow" as const); - const initial = JSON.parse(line("one")); - initial.payload.resourceLogs[0].scopeLogs[0].logRecords.push( - JSON.parse(line("two")).payload.resourceLogs[0].scopeLogs[0].logRecords[0] - ); - const first = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, `${JSON.stringify(initial)}\n`); - expect(await first).toMatchObject({ sent: 1, pending: 1 }); - expect(attempts).toEqual(["one", "two"]); - expect(accepted).toEqual(["two"]); - - historicalAllowed = false; - abort = new AbortController(); - const second = syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("two")); - expect(await second).toMatchObject({ sent: 1, pending: 1 }); - expect(attempts).toEqual(["one", "two", "two"]); - - historicalAllowed = true; - consenting = true; - abort = new AbortController(); - const third = await syncAllOtel({ ...options, signal: abort.signal }, snapshot, permits); - expect(third).toMatchObject({ sent: 1, pending: 0 }); - expect(accepted).toEqual(["two", "two", "one"]); -}); - -it.each(["missing", "cwd", "repository", "unavailable"] as const)( - "does not retry if the session admission is %s after a failed post", - async (change) => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-admission-retry-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let changed = false; - let calls = 0; - const abort = new AbortController(); - server = createServer((request, response) => { - calls++; - request.resume(); - request.on("end", () => { - changed = true; - response.writeHead(503).end("{}"); - setTimeout(() => abort.abort(), 100); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (!address || typeof address === "string") throw new Error("No test port"); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - signal: abort.signal, - }, - async () => { - if (changed && change === "unavailable") throw new Error("Admission unavailable"); - if (changed && change === "missing") return new Map(); - return new Map([ - [ - "allowed", - { - ...admission, - cwd: changed && change === "cwd" ? "/elsewhere" : admission.cwd, - repositoryId: changed && change === "repository" ? "other" : admission.repositoryId, - }, - ], - ]); - }, - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - if (change === "unavailable") await expect(running).rejects.toThrow(/Admission unavailable/); - else expect(await running).toMatchObject({ sent: 0, pending: 1 }); - expect(calls).toBe(1); - } -); - -it("does not send an event outside the clone's consent interval", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-consent-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - let calls = 0; - server = createServer((request, response) => { - calls++; - request.resume(); - response.writeHead(200).end("{}"); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const abort = new AbortController(); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const running = syncAllOtel( - { - config: "codex", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - signal: abort.signal, - intervalMs: 10, - }, - async () => new Map([["allowed", admission]]), - async () => "deny" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 100)); - abort.abort(); - const result = await running; - expect(result.denied).toBe(1); - expect(calls).toBe(0); -}); - -it("recovers an interrupted pending append without losing either session on the same line", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-all-pending-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - const firstAbort = new AbortController(); - const first = syncAllOtel( - { ...options, signal: firstAbort.signal }, - async () => new Map([["allowed", admission]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await expect( - syncAllOtel( - options, - async () => new Map(), - async () => "allow" - ) - ).rejects.toThrow(/lock/i); - const combined = JSON.parse(line("allowed")); - combined.payload.resourceLogs[0].scopeLogs[0].logRecords.push( - JSON.parse(line("waiting")).payload.resourceLogs[0].scopeLogs[0].logRecords[0] - ); - await appendFile(capture, `${JSON.stringify(combined)}\n`); - await new Promise((resolve) => setTimeout(resolve, 100)); - firstAbort.abort(); - await first; - expect(received).toHaveLength(1); - expect(received[0]).toContain("allowed"); - const stateFiles = await readdir(join(home, "state", "otel-sync")); - const pendingDirectory = join( - home, - "state", - "otel-sync", - stateFiles.find((name) => name.endsWith(".pending")) ?? "missing" - ); - const pending = join(pendingDirectory, (await readdir(pendingDirectory))[0]); - await appendFile(pending, "123"); // interrupted append before its checkpoint - await appendFile(capture, line("waiting")); - const secondAbort = new AbortController(); - const second = syncAllOtel( - { ...options, signal: secondAbort.signal }, - async () => new Map(), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 100)); - secondAbort.abort(); - await second; - expect((await readFile(pending, "utf8")).trim().split("\n")).toHaveLength(2); - const thirdAbort = new AbortController(); - const third = syncAllOtel( - { ...options, signal: thirdAbort.signal }, - async () => new Map([["waiting", admission]]), - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 100)); - thirdAbort.abort(); - await third; - expect(received).toHaveLength(3); - expect( - received.slice(1).every((body) => body.includes("waiting") && !body.includes('"allowed"')) - ).toBe(true); -}); - -it("only sends provider events for a confirmed feature, including after restart", async () => { - const home = await mkdtemp(join(tmpdir(), "aidd-sync-first-conflict-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const capture = join(captureDir, "codex.jsonl"); - await writeFile(capture, "", { mode: 0o600 }); - const received: string[] = []; - server = createServer((request, response) => { - let body = ""; - request.on("data", (chunk: Buffer) => { - body += chunk.toString(); - }); - request.on("end", () => { - received.push(body); - response.writeHead(200).end("{}"); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const options = { - config: "codex" as const, - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test-token", - home, - stateDir: join(home, "state"), - intervalMs: 10, - }; - const admission = { - feature: { state: "confirmed" as const, branch: "work", feature: "owner/repo#1" }, - cwd: "/project", - repositoryId: "repo", - }; - const snapshot = async () => new Map([["allowed", admission]]); - const firstAbort = new AbortController(); - const first = syncAllOtel( - { ...options, signal: firstAbort.signal }, - snapshot, - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 120)); - firstAbort.abort(); - await first; - const secondAbort = new AbortController(); - const second = syncAllOtel( - { ...options, signal: secondAbort.signal }, - snapshot, - async () => "allow" - ); - await new Promise((resolve) => setTimeout(resolve, 80)); - await appendFile(capture, line("allowed")); - await new Promise((resolve) => setTimeout(resolve, 120)); - secondAbort.abort(); - await second; - expect(received).toHaveLength(2); - expect(received.every((body) => body.includes("codex.sse_event"))).toBe(true); -}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts index 6a117301b..0bfeffdcb 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-attributes.integration.test.ts @@ -1,10 +1,5 @@ -import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; -import { createServer, type Server } from "node:http"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { afterEach, expect, test } from "vitest"; +import { expect, test } from "vitest"; import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; -import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; const attribute = (key: string, value: unknown) => ({ key, value }); const text = (key: string, value: string) => attribute(key, { stringValue: value }); @@ -401,15 +396,7 @@ test.each(["user.email", "user.account_id", "organization.id"])( } ); -const dirs: string[] = []; -let server: Server | undefined; -afterEach(async () => { - if (server) await new Promise<void>((resolve) => server?.close(() => resolve())); - server = undefined; - await Promise.all(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); -}); - -test.skipIf(process.platform === "win32").each([ +test.each([ { values: [text("input_token_count", "100"), text("input_token_count", "5")], error: "Conflicting duplicate OTLP attributes.", @@ -435,56 +422,6 @@ test.skipIf(process.platform === "win32").each([ error: "Invalid OTLP attribute value.", body: { stringValue: ["codex.sse_event"], intValue: "9" }, }, -])( - "invalid attributes stop sync before sending or advancing its checkpoint", - async ({ values, error, body }) => { - const home = await mkdtemp(join(tmpdir(), "aidd-conflicting-attributes-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const file = join(captureDir, "codex.jsonl"); - await writeFile(file, "", { mode: 0o600 }); - let calls = 0; - const abort = new AbortController(); - server = createServer((request, response) => { - request.resume(); - request.on("end", () => { - calls++; - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const stateDir = join(home, "state"); - const running = syncAllOtel( - { - ...options, - home, - stateDir, - endpoint: `http://127.0.0.1:${address.port}`, - intervalMs: 10, - signal: abort.signal, - }, - async () => - new Map([ - ["session", { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }], - ]), - async () => "allow" - ); - const directory = join(stateDir, "otel-sync"); - let checkpoint: string | undefined; - for (let attempt = 0; attempt < 100 && !checkpoint; attempt++) { - checkpoint = (await readdir(directory).catch(() => [])).find((entry) => - entry.endsWith(".checkpoint.json") - ); - if (!checkpoint) await new Promise((resolve) => setTimeout(resolve, 10)); - } - if (!checkpoint) throw new Error("No initial checkpoint"); - await appendFile(file, `${capture(values, [], [], body)}\n`); - await expect(running).rejects.toThrow(error); - expect(calls).toBe(0); - expect(JSON.parse(await readFile(join(directory, checkpoint), "utf8")).offset).toBe(0); - } -); +])("rejects malformed attributes before transfer selection", ({ values, error, body }) => { + expect(() => selectTransfers(capture(values, [], [], body), options)).toThrow(error); +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts index 834972749..66877a0f5 100644 --- a/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/sync-otel-integrity.integration.test.ts @@ -1,4 +1,4 @@ -import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { createServer, type Server } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -8,7 +8,6 @@ import { send, syncOtel, } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; -import { syncAllOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js"; const posixTest = test.skipIf(process.platform === "win32"); const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); @@ -217,67 +216,15 @@ test("preserves exact integer strings for signed and unsigned metric values", () expect(payload).toContain(`"${value}"`); }); -posixTest.each([ +test.each([ claudePoint({ asInt: 1785044858922210000 }), claudePoint({ asDouble: 5 }), claudePoint({ flags: 1 }), metricData({ sum: {}, summary: {} }), metricData({ summary: {} }), -])( - "an invalid metric stops all-session sync before delivery or checkpoint advance", - async (line) => { - const home = await mkdtemp(join(tmpdir(), "aidd-inexact-metric-")); - dirs.push(home); - const captureDir = join(home, ".local/share/aidd-otel-observation"); - const stateDir = join(home, "state"); - await mkdir(captureDir, { recursive: true, mode: 0o700 }); - const file = join(captureDir, "claude.jsonl"); - await writeFile(file, "", { mode: 0o600 }); - let calls = 0; - const abort = new AbortController(); - server = createServer((request, response) => { - request.resume(); - request.on("end", () => { - calls++; - response.writeHead(200).end("{}"); - abort.abort(); - }); - }); - await new Promise<void>((resolve) => server?.listen(0, "127.0.0.1", resolve)); - const address = server.address(); - if (address === null || typeof address === "string") throw new Error("No test port"); - const running = syncAllOtel( - { - config: "claude", - tenant: "tenant", - endpoint: `http://127.0.0.1:${address.port}`, - token: "test", - home, - stateDir, - intervalMs: 10, - signal: abort.signal, - }, - async () => - new Map([ - ["session", { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }], - ]), - async () => "allow" - ); - const directory = join(stateDir, "otel-sync"); - let checkpoint: string | undefined; - for (let attempt = 0; attempt < 100 && !checkpoint; attempt++) { - const entries = await readdir(directory).catch(() => []); - checkpoint = entries.find((entry) => entry.endsWith(".checkpoint.json")); - if (!checkpoint) await new Promise((resolve) => setTimeout(resolve, 10)); - } - if (!checkpoint) throw new Error("No initial checkpoint"); - await appendFile(file, `${line}\n`); - await expect(running).rejects.toThrow(/OTLP/); - const saved = JSON.parse(await readFile(join(directory, checkpoint), "utf8")); - expect(saved.offset).toBe(0); - expect(calls).toBe(0); - } -); +])("rejects malformed metrics before transfer selection", (line) => { + expect(() => selectTransfers(line, { ...options, config: "claude" })).toThrow(/OTLP/); +}); function metricWithMetadata(unit: string, temporality: number, monotonic: boolean): string { return JSON.stringify({ diff --git a/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts b/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts index c70a7eaa8..46f4b4b8e 100644 --- a/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts +++ b/cli/tests/contexts/telemetry/telemetry-lifecycle.integration.test.ts @@ -257,11 +257,7 @@ describe("opting in to a repository the previous version measured", () => { it("turns off with the git config value off, and leaves .aidd/config.json alone", async () => { await configure('{"keep":1}'); await deps.telemetryOnUseCase.execute(repo, "keep"); - expect(await deps.telemetryOffUseCase.execute(repo)).toMatchObject({ - status: "off", - changed: true, - closedAt: expect.any(String), - }); + expect(await deps.telemetryOffUseCase.execute(repo)).toEqual({ status: "off", changed: true }); expect(consentOf()).toBe("off"); expect(read(join(repo, ".aidd", "config.json"))).toBe('{"keep":1}'); }); diff --git a/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts b/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts index 0a92f0733..350b99a19 100644 --- a/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts +++ b/cli/tests/e2e/telemetry-lifecycle.e2e.test.ts @@ -183,20 +183,9 @@ describe("aidd telemetry off", () => { const run = await cli(["off"]); expect(run.exitCode).toBe(0); expect(run.stdout).toContain("Measurement is off"); - const history = readFileSync(join(box.telemetry, "ledger", "consents.jsonl"), "utf8") - .trim() - .split("\n") - .map((line) => JSON.parse(line) as { close?: string }); - const close = history.find((event) => event.close !== undefined); - expect(close?.close).toBeDefined(); - expect(run.stdout).toContain(`Consent close recorded at ${close?.close}`); - expect(run.stdout).toContain("Native OTLP configuration and collector are unchanged"); expect(git(repo, box.gitEnv, "config", "--local", "--get", "aidd.telemetry").trim()).toBe( "off" ); - const repeated = await cli(["off"]); - expect(repeated.stdout).not.toContain("Consent close recorded at"); - expect(repeated.stdout).toContain("Native OTLP configuration and collector are unchanged"); }); }); diff --git a/cli/tests/e2e/telemetry-report-sandbox.ts b/cli/tests/e2e/telemetry-report-sandbox.ts index 9da1d5908..772c3e519 100644 --- a/cli/tests/e2e/telemetry-report-sandbox.ts +++ b/cli/tests/e2e/telemetry-report-sandbox.ts @@ -265,25 +265,12 @@ export function useReportSandbox(): void { // Opting in is `on`: a key set by hand opens no consent interval, and nothing is stored. expect((await aidd(["on", "--yes", "--anonymous"])).exitCode).toBe(0); await backdateConsent(telemetry); - // The synthetic calls also need a repository observation from before they ran. - await dateRootObservation("2026-09-01T00:00:00.000Z"); }); afterEach(async () => { await env.cleanup(); }); } -/** Simulate when the sandbox's clone was first observed, alongside its backdated consent. */ -export async function dateRootObservation(at: string): Promise<void> { - const path = join(telemetry, "ledger", "roots.json"); - const roots = JSON.parse(await readFile(path, "utf8")) as { - directories: Array<{ seen_at: string }>; - }; - if (roots.directories.length !== 1) throw new Error("Expected one remembered clone"); - roots.directories[0].seen_at = at; - await writeFile(path, `${JSON.stringify(roots, null, 2)}\n`); -} - /** Appends a call to the sandbox's main transcript, as Claude would write it later. */ export async function addCall(call: Call): Promise<void> { await appendFile(join(claude, "projects", "sandbox", "main.jsonl"), `${usageLine(call, repo)}\n`); diff --git a/cli/tests/e2e/telemetry-report.e2e.test.ts b/cli/tests/e2e/telemetry-report.e2e.test.ts index dba34813b..60f03c3a9 100644 --- a/cli/tests/e2e/telemetry-report.e2e.test.ts +++ b/cli/tests/e2e/telemetry-report.e2e.test.ts @@ -5,7 +5,6 @@ import { CALLS, type Call, COUNTERS, - dateRootObservation, declareBranch, type Envelope, held, @@ -20,21 +19,6 @@ import { useReportSandbox(); describe("aidd telemetry report", () => { - it("keeps tokens without repository or task attribution before the first clone observation", async () => { - await dateRootObservation("2026-10-09T00:00:00.000Z"); - const repository = await json("repository"); - expect(repository.totals.records).toBe(CALLS.length); - expect(repository.rows).toHaveLength(1); - expect(repository.rows[0]).toMatchObject({ kind: "absent", records: CALLS.length }); - const tasks = await json("task"); - expect(tasks.rows).toHaveLength(1); - expect(tasks.rows[0]).toMatchObject({ - kind: "unattributed", - reason: "no-binding", - records: CALLS.length, - }); - }); - it("answers on every axis, and every axis adds up to the same totals", async () => { await declareBranch(); const known = (pick: (call: Call) => number | null) => diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index d6cd40d01..fb794822b 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -167,7 +167,7 @@ { "invocation": "aidd telemetry otel", "exitCode": 0, - "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local capture\n without changing files\n collect [options] Capture authenticated OTLP on a local loopback port\n sync [options] Send one consented session's local telemetry to a loopback\n Gouvernail\n help [command] display help for command" + "help": "Usage: aidd telemetry otel [options] [command]\n\nConfigure native OpenTelemetry exporters\n\nOptions:\n -h, --help display help for command\n\nCommands:\n init [options] Enable detailed local OTLP export in one tool's user\n configuration\n verify [options] Check native OTLP configuration and existing local\n capture without changing files\n collect [options] Capture authenticated OTLP on a local loopback port\n run [options] <prompt> Run one fresh provider session and send its allowed\n OTLP observations\n help [command] display help for command" }, { "invocation": "aidd telemetry otel collect", @@ -180,9 +180,9 @@ "help": "Usage: aidd telemetry otel init [options]\n\nEnable detailed local OTLP export in one tool's user configuration\n\nOptions:\n --config <tool> User configuration to merge (choices: \"claude\",\n \"codex\", \"opencode\")\n --endpoint <environment> Collector destination (choices: \"dev\", \"prod\")\n -h, --help display help for command" }, { - "invocation": "aidd telemetry otel sync", + "invocation": "aidd telemetry otel run", "exitCode": 0, - "help": "Usage: aidd telemetry otel sync [options]\n\nSend one consented session's local telemetry to a loopback Gouvernail\n\nOptions:\n --config <tool> Tool to sync (choices: \"claude\", \"codex\")\n --session <id> Exact provider session ID; omit with --follow to discover\n sessions\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --follow Follow new events for this session (default: false)\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel run [options] <prompt>\n\nRun one fresh provider session and send its allowed OTLP observations\n\nOptions:\n --config <tool> Tool to run (choices: \"claude\", \"codex\", \"opencode\")\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --yes Consent to capture and send telemetry from this execution\n --model <model> Provider model to use\n --keep-capture Retain the private local OTLP capture\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index 90fb8d9ae..7daae1c0a 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -23,8 +23,6 @@ const input = vi.fn(); const propose = vi.fn(); const ensureInstallationId = vi.fn(); const checkInstallationId = vi.fn(); -const sessionSnapshot = vi.fn(); -const sessionPermits = vi.fn(); vi.mock("../../../src/runtime/wiring/framework.js", () => ({ createDeps: vi.fn(async () => ({ @@ -41,7 +39,6 @@ vi.mock("../../../src/runtime/wiring/framework.js", () => ({ reportUsageUseCase: { execute: report }, manageIdentityUseCase: identity, proposeIdentityUseCase: { execute: propose }, - sessionFeatureUseCase: { snapshot: sessionSnapshot, permits: sessionPermits }, }, })), createMenuDeps: vi.fn(), @@ -776,131 +773,33 @@ describe("native telemetry commands", () => { expect(check).toHaveBeenCalledWith("claude", "dev"); }); - it("forwards one consented session without printing its token", async () => { - vi.stubEnv("AIDD_OTEL_TOKEN", "private-test-token"); - const admission = { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }; - sessionSnapshot.mockResolvedValue(new Map([["s1", admission]])); - sessionPermits.mockResolvedValueOnce("allow").mockResolvedValueOnce("deny"); - const sync = vi.spyOn(otel, "syncCapturedSession").mockResolvedValue({ - sent: 2, - skipped: 0, - offset: 12, - partialLine: true, - }); - expect( - await run( - "otel", - "sync", - "--config=claude", - "--session=s1", - "--tenant=acme", - "--endpoint=http://127.0.0.1:3000" - ) - ).toBeNull(); - expect(sessionSnapshot).toHaveBeenCalledWith("claude-code"); - const call = sync.mock.calls[0]; - if (!call) throw new Error("the session was not passed to the exporter"); - expect(call[0]).toMatchObject({ - config: "claude", - session: "s1", - tenant: "acme", - endpoint: "http://127.0.0.1:3000", - token: "private-test-token", - follow: false, - }); - await expect(call[1]()).resolves.toBeUndefined(); - const transfer = { signal: "logs" as const, payload: {}, at: "2026-11-04T00:00:00Z" }; - expect(await call[2](transfer)).toBe(true); - expect(await call[2](transfer)).toBe(false); - expect(await call[2]({ signal: "logs", payload: {} })).toBe(false); - expect(stdout()).toContain("Sent 2 observations"); - expect(stdout()).toContain("last incomplete capture line"); - expect(stdout()).not.toContain("private-test-token"); - }); - - it("never sends a session that lacks consent", async () => { - sessionSnapshot.mockResolvedValue(new Map()); - const sync = vi.spyOn(otel, "syncCapturedSession"); - expect( - await run( - "otel", - "sync", - "--config=codex", - "--session=s2", - "--tenant=acme", - "--endpoint=http://127.0.0.1:3000" - ) - ).toBe(1); - expect(sessionSnapshot).toHaveBeenCalledWith("codex"); - expect(stderr()).toContain("No consented usage found"); - expect(sync).not.toHaveBeenCalled(); - }); - - it("stops admitting events when a followed session loses consent", async () => { - vi.stubEnv("AIDD_OTEL_TOKEN", ""); - const admission = { feature: { state: "unbound" }, cwd: "/project", repositoryId: "repo" }; - sessionSnapshot - .mockResolvedValueOnce(new Map([["s1", admission]])) - .mockResolvedValueOnce(new Map([["s1", admission]])) - .mockResolvedValueOnce(new Map()); - sessionPermits.mockResolvedValue("allow"); - const sync = vi.spyOn(otel, "syncCapturedSession").mockResolvedValue({ - sent: 1, - skipped: 0, - offset: 1, - partialLine: false, - }); - expect( - await run( - "otel", - "sync", - "--config=codex", - "--session=s1", - "--tenant=acme", - "--endpoint=http://127.0.0.1:3000", - "--follow" - ) - ).toBeNull(); - expect(stdout()).toContain("Following the selected session"); - expect(stdout()).not.toContain("last incomplete capture line"); - const call = sync.mock.calls[0]; - if (!call) throw new Error("the session was not passed to the exporter"); - expect(call[0]).toMatchObject({ follow: true, token: "" }); - await expect(call[1]()).rejects.toThrow("No consented usage found"); - expect(await call[2]({ signal: "logs", payload: {}, at: "2026-11-04T00:00:00Z" })).toBe(false); - }); - - it("requires a session for one-shot sync and discovers sessions only with follow", async () => { - vi.stubEnv("AIDD_OTEL_TOKEN", "private-test-token"); - const follow = vi - .spyOn(otel, "followCapturedSessions") - .mockResolvedValue({ sent: 3, denied: 1, pending: 2 }); + it("requires explicit execution consent before launching a provider", async () => { + const runOtel = vi.spyOn(otel, "runOtelSession"); const args = [ "otel", - "sync", - "--config=codex", + "run", + "--config=claude", "--tenant=acme", "--endpoint=http://127.0.0.1:3000", ]; - expect(await run(...args)).toBeNull(); - expect(process.exitCode).toBe(1); - expect(stderr()).toContain("Pass --session"); - expect(follow).not.toHaveBeenCalled(); - process.exitCode = previousExitCode; - sessionSnapshot.mockResolvedValue(new Map()); - sessionPermits.mockResolvedValue("unknown"); - expect(await run(...args, "--follow")).toBeNull(); - const call = follow.mock.calls[0]; - if (!call) throw new Error("the session follower was not started"); - expect(call[0]).toMatchObject({ config: "codex", tenant: "acme", token: "private-test-token" }); - expect(await call[1]()).toEqual(new Map()); - const admission = { - feature: { state: "unbound" as const }, - cwd: "/project", - repositoryId: "repo", - }; - expect(await call[2](admission, "2026-11-04T00:00:00Z")).toBe("unknown"); - expect(stdout()).toContain("Sent 3 observations; 1 were outside consent; 2 await confirmation"); - expect(stdout()).not.toContain("private-test-token"); + expect(await run(...args, "--", "reply OK")).toBe(1); + expect(stderr()).toContain("Pass --yes to consent"); + expect(runOtel).not.toHaveBeenCalled(); + runOtel.mockImplementation(async ({ retainCapture }) => { + retainCapture?.("/private/claude.jsonl"); + throw new Error("Provider exited early."); + }); + expect(await run(...args, "--yes", "--keep-capture", "--model=haiku", "--", "reply OK")).toBe( + 1 + ); + expect(runOtel).toHaveBeenCalledWith( + expect.objectContaining({ + config: "claude", + model: "haiku", + prompt: "reply OK", + retainCapture: expect.any(Function), + }) + ); + expect(stdout()).toContain("Private OTLP capture: /private/claude.jsonl"); }); }); diff --git a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts index 57055ea98..60cc3483f 100644 --- a/cli/tests/runtime/wiring/otel-wiring.unit.test.ts +++ b/cli/tests/runtime/wiring/otel-wiring.unit.test.ts @@ -26,20 +26,13 @@ vi.mock("../../../src/runtime/filesystem/installation-identity.js", () => ({ vi.mock("../../../src/runtime/user-config-dir.js", () => ({ userConfigDirOf: () => "/profile/aidd", })); -vi.mock("../../../src/runtime/wiring/telemetry.js", () => ({ - telemetryDir: () => "/profile/telemetry", -})); -const { collectOtel, configureOtel, checkOtel, syncCapturedSession, followCapturedSessions } = - await import("../../../src/runtime/wiring/telemetry-otel.js"); +const { collectOtel, configureOtel, checkOtel } = await import( + "../../../src/runtime/wiring/telemetry-otel.js" +); const collector = await import( "../../../src/contexts/telemetry/infrastructure/otel/actions/collect.js" ); -const sync = await import("../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"); -const follow = await import( - "../../../src/contexts/telemetry/infrastructure/otel/actions/sync-all.js" -); - beforeEach(() => { vi.clearAllMocks(); mocks.checkId.mockReset().mockResolvedValue(null); @@ -102,41 +95,4 @@ describe("OTLP runtime wiring", () => { env: process.env, }); }); - - it("uses the private runtime profile for one session and automatic following", async () => { - const bounded = vi - .spyOn(sync, "syncOtel") - .mockResolvedValue({ sent: 1, skipped: 0, offset: 5, partialLine: false }); - const automatic = vi - .spyOn(follow, "syncAllOtel") - .mockResolvedValue({ sent: 2, denied: 0, pending: 0 }); - const options = { - config: "codex" as const, - tenant: "acme", - endpoint: "http://127.0.0.1:3000", - token: "test-token", - }; - const resolve = async () => {}; - const allow = async () => true; - const snapshot = async () => new Map(); - const permits = async () => "allow" as const; - await syncCapturedSession({ ...options, session: "s1", follow: false }, resolve, allow); - await followCapturedSessions(options, snapshot, permits); - expect(bounded).toHaveBeenCalledWith( - { - ...options, - session: "s1", - follow: false, - home: "/profile", - }, - resolve, - allow - ); - expect(automatic).toHaveBeenCalledWith( - { ...options, home: "/profile", stateDir: "/profile/telemetry" }, - snapshot, - permits, - undefined - ); - }); }); diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index d1096c3bf..c7622014e 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -11,9 +11,7 @@ Gouvernail's AI KPI view uses provider OpenTelemetry (OTel) observations only. T Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. -OTLP collection, verification and development forwarding are separate steps. The collector keeps provider OTLP locally; the consent-aware sync forwards only supported signals and allowlisted attributes to a loopback Gouvernail endpoint. The local collector does not check per-clone consent, so its private files may contain other sessions and sensitive data. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). - -The current sync still reads the framework's local usage ledger to establish session-to-clone consent. It does not forward OpenCode traces. Consequently, it is not yet an OTel-only end-to-end pipeline for the three tools. +`otel run --yes` captures one provider execution in isolation and forwards supported observations and allowlisted attributes to loopback Gouvernail. It neither reads the local usage ledger nor asserts a clone or feature association. With `--keep-capture`, raw private files can contain sensitive content. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). In a local OpenCode v2.2.5 capture, two prompt spans included a session ID in their HTTP path; no model or token counts were observed. This observation does not establish that every OpenCode signal includes a session ID. OpenCode session-event feeds and session exports are not provider OTLP and are not used in this view. Production export remains disabled until authenticated user identity and the destination are implemented. diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 2a3fdb545..a82fa5ee5 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -24,25 +24,23 @@ flowchart LR Tasks -->|"attribute"| Report ``` -The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The framework's local ledger powers `aidd telemetry report` and remains separate. The development sync currently relies on the ledger to associate a session with a clone before checking consent. Do not add their totals: both sources can cover the same calls. See the [field inventory](OBSERVABILITY.md). +The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The framework's local ledger powers `aidd telemetry report` and remains separate. Do not add their totals: both sources can cover the same calls. See the [field inventory](OBSERVABILITY.md). ## Choose a mode | Mode | Opt-in and destination | Identity | What is kept or sent | | --- | --- | --- | --- | | Local usage | `aidd telemetry on` explicitly enables measurement for the current clone from that time. The ledger and report stay on this machine. | An optional name labels the local report; anonymous use is supported. | Per-call usage read from Claude Code and Codex session files: session, time, model, agent, token counts and local attribution context. The ledger does not store prompts or responses. | -| OTLP development | Explicitly configure each provider with `otel init --endpoint=dev` and start the local collector. A separate `otel sync` sends supported data only to loopback Gouvernail for sessions covered by the clone's consent. | No local report name is required. Provider identifiers may still be sent when present; attribution in Gouvernail needs a matched active member. The connector token is not a person identity. | The collector stores raw provider events, which may include prompts, responses, commands and tool output. The sync excludes fields for those contents and traces, but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | +| OTLP development | `otel run --yes` explicitly opts in one isolated execution and sends supported observations to loopback Gouvernail. | Provider identifiers may be sent when present; attribution requires a matched active member. The connector token is not a person identity. | With `--keep-capture`, the private capture can contain prompts, responses, commands and tool output. Forwarding filters fields but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | | OTLP production | **Not available:** `--endpoint=prod` is refused. `https://gouvernail.eu` is an intended future destination, not an active exporter. | **Required before production is enabled:** an authenticated user identity. This is a product requirement, not yet implemented or enforced. | No production payload is sent by this feature today. Consent, authentication, retention and deletion behavior must be implemented before enabling it. | ### Consent and scope -- `on` is an explicit opt-in for one clone, not a team-wide setting. The local ledger and OTLP sync use its consent intervals; an earlier consent does not cover later work after `off`. -- `otel init` is a separate, explicit action to configure a provider's user-level exporter. The CLI does not show a separate consent dialog or record separate OTLP consent; the init skill asks for authorization before running it. When that provider and the collector are running, raw OTLP may be captured from other projects too: **the collector does not check clone consent**. The sync checks consent before forwarding. -- Claude metric sync also sends a deterministic fingerprint derived from the original series attributes; it can expose equality or allow guessing low-entropy values. -- `aidd telemetry off` stops local measurement for the current clone; it does **not** disable provider exporters or stop the OTLP collector. A process started with `AIDD_TELEMETRY=0` refuses local measurement and sync, not raw collection; changing an environment variable does not change an already running process. -- `aidd telemetry forget` previews or deletes local usage measurement. It does **not** delete OTLP capture files, sync state, or data already received by Gouvernail. Production deletion is not available; the requested production policy is to disallow it. -- Raw captures are private local files, but have no automatic retention limit. They can contain sensitive content. The local collector and sync processes do not restart automatically. -- Automatic sync sends each session's capture line as one request. A line over 2 MiB remains pending and is reported; later lines continue. A timeout after receiver acceptance cannot guarantee exactly-once delivery without receiver-side idempotence. +- `on` opts in the separate local usage ledger for one clone. `otel run --yes` opts in the single provider execution independently; it does not attribute observations to a clone or feature. +- `otel init` separately configures a user-level exporter. Its global collector can capture other projects and does not check clone consent. It is not required for `otel run`. +- `aidd telemetry off` and `forget` affect local usage measurement; they do not disable provider exporters or delete retained OTLP captures or data already received by Gouvernail. +- Retained captures are private local files without automatic deletion. They can contain sensitive content. Claude metric forwarding also sends a deterministic series fingerprint, which can expose equality or allow guesses of low-entropy values. +- Delivery after a timeout cannot guarantee exactly-once behavior without receiver-side idempotence. ### Three different identifiers @@ -50,7 +48,7 @@ The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The | --- | --- | --- | | Local report name | Optional label in `~/.config/aidd/telemetry/identity.json`; the current label is applied when reporting, not recorded as proof per call. | No. | | Installation UUID | Stable ID in `~/.config/aidd/installation/id.json`; not a login or credential. | Not currently. | -| Provider identity | Native email, account or organization ID when emitted. Gouvernail must match an approved active member to attribute a session. | The allowlisted identifiers may be sent by the dev sync. | +| Provider identity | Email, account or organization ID when emitted. Gouvernail must match an approved active member to attribute a session. | Allowlisted identifiers may be sent by `otel run`. | The default paths can change with the configured user directory. A connector bearer token authorizes the destination connection; it does not identify the person. diff --git a/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md index 3a5a9220a..13e12141d 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/04-stop.md @@ -10,7 +10,7 @@ The project directory. 1. Say what stopping does: new Claude Code or Codex work in this clone is no longer counted, nor asked about where it was asked (Claude Code, the Codex TUI), and what was measured stays. It does not disable provider OTLP exporters or the raw collector. Wait for a yes. 2. Run `aidd telemetry off` and relay its message. -3. Run `aidd telemetry status` and verify that measurement is off. Report the result by scope in the table below. The CLI does not report the exact stop timestamp; do not invent one. Previously consented observations can still be processed or sent after `off`. +3. Run `aidd telemetry status` and verify that measurement is off. Report the result by scope in the table below. The CLI does not report the exact stop timestamp; do not invent one. An independently consented `otel run` is unaffected. 4. Say that deleting what was measured is a separate step, and that it needs its own confirmation. If the person wants raw OTLP to stop too, explain that `off` does not do that: the exporter and collector must be stopped separately, and there is no automatic restore command in this feature. ## Output @@ -19,5 +19,5 @@ The project directory. | --- | --- | --- | --- | | Local usage for this clone | Stopped if the command and status confirm it | Exact time not reported by the CLI | Retained | | OTLP export and raw local collection | Not stopped by `off`; current process state is unknown unless separately checked | Unknown | Captures retained | -| OTLP sync to local Gouvernail | New events outside consent are not eligible; previously consented events may still be sent | Consent interval closes on `off`; exact time not reported | Already received data retained | +| OTLP execution to local Gouvernail | Independent; `otel run --yes` consents to each execution | Not controlled by `off` | Already received data retained | | Gouvernail production | Not available in this feature | Not applicable | Not managed by this command | diff --git a/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md index 7c8184d13..f65b0ea43 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/05-forget.md @@ -1,6 +1,6 @@ # 05 - Forget -Delete local usage measurement after a preview and explicit confirmation. This command does not delete provider OTLP captures, OTLP sync state, installation UUID or data already received by Gouvernail. +Delete local usage measurement after a preview and explicit confirmation. This command does not delete provider OTLP captures, the installation UUID or data already received by Gouvernail. ## Process diff --git a/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md index 7be16eb10..f4627e4cc 100644 --- a/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md +++ b/plugins/aidd-telemetry/skills/00-init/actions/06-otel-init.md @@ -9,13 +9,13 @@ The provider (`claude`, `codex` or `opencode`) and destination (`dev` or `prod`) ## Process 1. Run `aidd telemetry otel init --help` and report an unavailable command as requiring an update of `@ai-driven-dev/cli`. -2. Obtain both choices and explain the separate opt-in: native telemetry goes to the selected collector and can include prompts, responses, commands and tool output. The provider's user-level export can cover other projects; the raw collector does not check clone consent. The separate sync checks consent before forwarding. +2. Obtain both choices and explain the separate opt-in: provider telemetry goes to the selected collector and can include prompts, responses, commands and tool output. The provider's user-level export can cover other projects; the raw collector does not check clone consent. - The CLI's development endpoint is `http://127.0.0.1:14318`. Start the collector with `aidd telemetry otel collect`; it creates the private token. Initialization does not start it. - Production is refused. `https://gouvernail.eu` is an intended future destination, not a configured endpoint; requiring an authenticated person there is a future requirement, not enforced today. 3. Run `aidd telemetry otel init --config=<provider> --endpoint=<destination>` after an explicit request enabling that export. - A request naming both choices is sufficient authorization. - Let the CLI validate configuration and credentials without requesting a token in chat. -4. Report whether configuration changed and relay the CLI's restart or launch instructions. OpenCode major version 1 needs the launch command printed by the CLI to supply the endpoint and token; major version 2 needs its service restarted. +4. Report whether configuration changed and relay the CLI's restart or launch instructions. OpenCode v2 needs its service restarted. 5. Continue to [verify receipt](07-otel-verify.md) after the person runs a short session with the selected provider. ## Output diff --git a/scripts/__tests__/aidd-telemetry-hooks.test.js b/scripts/__tests__/aidd-telemetry-hooks.test.js index 57b5ca959..f1e1c578b 100644 --- a/scripts/__tests__/aidd-telemetry-hooks.test.js +++ b/scripts/__tests__/aidd-telemetry-hooks.test.js @@ -51,10 +51,9 @@ const GIT_DIR_ON_PATH = gitDirectory(); /** git as a test sets a repository up: never through a variable a commit hook exported. */ function git(cwd, ...args) { const env = Object.fromEntries(Object.entries(process.env).filter(([k]) => !k.startsWith("GIT_"))); - // A commit can detach maintenance and change object files while the copy tests read them. const run = spawnSync( "git", - ["-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", "-c", "maintenance.auto=false", ...args], + ["-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", ...args], { cwd, env, encoding: "utf8" } ); assert.equal(run.status, 0, `git ${args.join(" ")}: ${run.stderr}`); @@ -663,13 +662,6 @@ test("consent: a key with no interval open for the clone grants nothing, asks no } }); -test("sandbox: disables automatic Git maintenance before copying a repository", () => { - withBox({}, (box) => { - git(box.repo, "config", "--local", "maintenance.auto", "true"); - assert.equal(git(box.repo, "config", "--get", "maintenance.auto").trim(), "false"); - }); -}); - test("consent: a cp -R copy carries the key and not the clone, so it asks nothing and closes nothing", () => { withBox({}, (box) => { git(box.repo, "commit", "--allow-empty", "-q", "-m", "x"); From 205719195243cb764d30280c7cd000bf81165402 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 10:47:37 +0800 Subject: [PATCH 32/56] docs(telemetry): explain OpenCode OTLP coverage --- plugins/aidd-telemetry/OBSERVABILITY.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index c7622014e..ee110bd7f 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -13,7 +13,7 @@ Token fields are provider-specific. Claude's input excludes cache reads and writ `otel run --yes` captures one provider execution in isolation and forwards supported observations and allowlisted attributes to loopback Gouvernail. It neither reads the local usage ledger nor asserts a clone or feature association. With `--keep-capture`, raw private files can contain sensitive content. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). -In a local OpenCode v2.2.5 capture, two prompt spans included a session ID in their HTTP path; no model or token counts were observed. This observation does not establish that every OpenCode signal includes a session ID. OpenCode session-event feeds and session exports are not provider OTLP and are not used in this view. Production export remains disabled until authenticated user identity and the destination are implemented. +In two local OpenCode v2.2.5 executions, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. A startup log included the requested model, but no session ID or proof of the model actually used. The framework forwards only the validated prompt span, not the other traces or logs. OpenCode computes usage on an internal event bus, which is not provider OTLP and is not used in this view. These observations do not prove what other versions or configurations emit. Production export remains disabled until authenticated user identity and the destination are implemented. ## Sources @@ -22,6 +22,7 @@ Read 2026-11-06: - [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) - [Codex telemetry configuration](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted) - [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts) +- [OpenCode v2.2.5 usage projection](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/projector.ts#L81-L110) - [OpenCode v2 HTTP API audit](https://github.com/anomalyco/opencode/blob/dev/V2_HTTP_API_AUDIT.md) (current development branch; the v2.2.5 path was observed locally) - [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) - [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) From 4907069de52eb474f32e6b51699099e1e8830e2c Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 10:49:43 +0800 Subject: [PATCH 33/56] docs(telemetry): clarify OpenCode trace correlation --- plugins/aidd-telemetry/OBSERVABILITY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index ee110bd7f..8172f66c3 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -13,7 +13,7 @@ Token fields are provider-specific. Claude's input excludes cache reads and writ `otel run --yes` captures one provider execution in isolation and forwards supported observations and allowlisted attributes to loopback Gouvernail. It neither reads the local usage ledger nor asserts a clone or feature association. With `--keep-capture`, raw private files can contain sensitive content. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). -In two local OpenCode v2.2.5 executions, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. A startup log included the requested model, but no session ID or proof of the model actually used. The framework forwards only the validated prompt span, not the other traces or logs. OpenCode computes usage on an internal event bus, which is not provider OTLP and is not used in this view. These observations do not prove what other versions or configurations emit. Production export remains disabled until authenticated user identity and the destination are implemented. +In two local OpenCode v2.2.5 executions, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. Five other HTTP spans per execution contained that session ID but are not forwarded. The prompt spans had no trace-linked descendants in these captures; the model and usage spans belonged to a separate server trace without a session ID. A startup log included the requested model, but no session ID or proof of the model actually used. The framework forwards only the validated prompt span, not the other traces or logs. OpenCode computes usage on an internal event bus, which is not provider OTLP and is not used in this view. These observations do not prove what other versions or configurations emit. Production export remains disabled until authenticated user identity and the destination are implemented. ## Sources From 643bef363b263124bd6ca843881be79776ff562c Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 11:08:57 +0800 Subject: [PATCH 34/56] docs(telemetry): map provider OTLP coverage --- plugins/aidd-telemetry/OBSERVABILITY.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 8172f66c3..34b0bb041 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -15,14 +15,26 @@ Token fields are provider-specific. Claude's input excludes cache reads and writ In two local OpenCode v2.2.5 executions, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. Five other HTTP spans per execution contained that session ID but are not forwarded. The prompt spans had no trace-linked descendants in these captures; the model and usage spans belonged to a separate server trace without a session ID. A startup log included the requested model, but no session ID or proof of the model actually used. The framework forwards only the validated prompt span, not the other traces or logs. OpenCode computes usage on an internal event bus, which is not provider OTLP and is not used in this view. These observations do not prove what other versions or configurations emit. Production export remains disabled until authenticated user identity and the destination are implemented. +## Additional coverage investigated + +| Provider | Available or observed in OTLP | Current limit and next step | +| --- | --- | --- | +| Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort, latency and separate token and cache counters. These fields appeared in both local captures. | The framework does not forward these events today. An allowlist of technical fields could expose them; avoid counting the same request again through metrics. Repository attributes require a separate opt-in. | +| Codex | Logs, metrics and traces are independently configurable. Completed response events can include model, effort and usage. | The observed turn-usage histogram had no session ID and the response events had no turn or request ID. Neither timestamps nor a previous session's effort establish a safe join. Forward additional documented, session-linked events only after validating their fields. | +| OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | Standard emitted OTLP did not establish session-linked model, effort or usage. Instrumenting the producer before its usage normalization could emit explicit session-linked OTLP, but is not implemented or validated. Its internal plugin event bus is a different source, and normalizes missing counters to zero; a model variant is not necessarily reasoning effort. | + +An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. + ## Sources Read 2026-11-06: - [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) +- [Claude Code API request event](https://code.claude.com/docs/en/monitoring-usage#api-request-event) - [Codex telemetry configuration](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted) - [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts) - [OpenCode v2.2.5 usage projection](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/projector.ts#L81-L110) +- [OpenCode v2.2.5 session events](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts#L333-L389) and [usage normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts#L8-L19) - [OpenCode v2 HTTP API audit](https://github.com/anomalyco/opencode/blob/dev/V2_HTTP_API_AUDIT.md) (current development branch; the v2.2.5 path was observed locally) - [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) - [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) From ec0b2f500d871bf271a9dc538d3c20515fedf469 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 12:01:49 +0800 Subject: [PATCH 35/56] feat(telemetry): export OpenCode session events as OTLP --- aidd_docs/recipes/observe-agent-telemetry.md | 8 +- .../configs/otel-opencode-plugin.js.txt | 125 ++++++++++++++ .../infrastructure/otel/actions/init.ts | 10 +- .../otel/actions/opencode-plugin-file.ts | 41 +++++ .../otel/actions/opencode-plugin-logs.ts | 162 ++++++++++++++++++ .../infrastructure/otel/actions/run.ts | 89 +++++++++- .../infrastructure/otel/actions/verify.ts | 14 +- .../infrastructure/otel/config/opencode.json | 4 +- .../telemetry/infrastructure/otel/settings.ts | 6 +- .../otel/init-otel.integration.test.ts | 11 +- .../telemetry/otel/run-otel.unit.test.ts | 79 +++++++++ .../otel/verify-otel.integration.test.ts | 12 ++ plugins/aidd-telemetry/OBSERVABILITY.md | 15 +- plugins/aidd-telemetry/README.md | 2 +- 14 files changed, 560 insertions(+), 18 deletions(-) create mode 100644 cli/assets/configs/otel-opencode-plugin.js.txt create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index eae21f6c2..63c932ca5 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -1,6 +1,6 @@ # Observe agent telemetry locally -Run Claude Code, Codex and OpenCode v2 with provider OpenTelemetry (OTel), then inspect the local Gouvernail dashboard. The framework's separate local usage ledger is not the source for this view. +Run Claude Code, Codex and OpenCode v2 with OpenTelemetry (OTel), then inspect the local Gouvernail dashboard. For OpenCode, an opted-in AIDD plugin converts public session events to OTLP; the standard exporter does not supply the session-linked usage observed by this plugin. The framework's separate local usage ledger is not the source for this view. ## Steps to observe agent telemetry locally @@ -40,14 +40,14 @@ for i in 1 2; do done ``` -Each command launches a separate session with an isolated collector. It does not read the framework usage ledger or establish a project or feature association. `otel init` and the global development collector are a separate configuration path; they are not required for these commands. +Each command launches a separate session with an isolated collector. The OpenCode command loads the AIDD plugin for that run. It does not read the framework usage ledger or establish a repository or feature association. `otel init` and the global development collector are a separate configuration path; they are not required for these commands. ### 3) โœ… Check receipt -Open the local AI KPI page and check the session IDs and observed fields. Delivery does not prove that every provider event was emitted or received. In the OpenCode v2.2.5 capture, HTTP prompt spans included session IDs but no model or token counts. Missing or uncertain values remain unavailable. See the [field inventory](../../plugins/aidd-telemetry/OBSERVABILITY.md). +Open the local AI KPI page and check the session IDs and observed fields. Delivery does not prove that every event was emitted or received. For OpenCode, the plugin reports the selected model and positive normalized token counts in separate events; an ambiguous zero stays unavailable. Missing or uncertain values remain unavailable. See the [field inventory](../../plugins/aidd-telemetry/OBSERVABILITY.md). ```sh -open 'http://localhost:55857/enterprise/ai-driven-dev/kpi/ai' +open 'http://localhost:55857/enterprise/ai-driven-dev/kpi/sessions' ``` Sources accessed 2026-11-06: [Claude monitoring](https://code.claude.com/docs/en/monitoring-usage), [Codex telemetry](https://learn.chatgpt.com/docs/config-file/config-advanced#what-gets-emitted), [OpenCode v2 CLI](https://dev.opencode.ai/v2/docs/cli) and [OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts). diff --git a/cli/assets/configs/otel-opencode-plugin.js.txt b/cli/assets/configs/otel-opencode-plugin.js.txt new file mode 100644 index 000000000..648f3acb5 --- /dev/null +++ b/cli/assets/configs/otel-opencode-plugin.js.txt @@ -0,0 +1,125 @@ +// OpenCode server plugin v2.2.5+. The framework enables it only for an opted-in run. +// Public event stream: https://dev.opencode.ai/v2/docs/build/plugins/server (2026-11-06). +const types = ["session.created", "session.step.started", "session.step.ended", "session.step.failed"]; + +const text = (key, value) => ({ key, value: { stringValue: value } }); +const integer = (key, value) => ({ key, value: { intValue: String(value) } }); +const id = (value, prefix) => + typeof value === "string" && new RegExp(`^${prefix}_[A-Za-z0-9]{1,64}$`).test(value); +const label = (value) => + typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/@+#-]{0,199}$/.test(value); + +function destination() { + const endpoint = process.env.AIDD_OPENCODE_OTEL_LOGS_ENDPOINT; + const authorization = /^Authorization=Bearer ([A-Za-z0-9_-]+)$/.exec( + process.env.AIDD_OPENCODE_OTEL_HEADERS ?? "", + ); + if (!endpoint || !authorization) return; + try { + const url = new URL(endpoint); + const loopback = ["127.0.0.1", "localhost", "[::1]"].includes(url.hostname); + if ((url.protocol !== "https:" && !(url.protocol === "http:" && loopback)) || + url.username || url.password || url.search || url.hash || !url.pathname.endsWith("/v1/logs")) return; + return { url: url.href, authorization: `Bearer ${authorization[1]}` }; + } catch { + return; + } +} + +// Source: https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts +// Usage's absent-to-zero normalization: https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts (2026-11-06). +function record(event) { + const data = event?.data; + const durable = event?.durable; + if (!types.includes(event?.type) || !id(event.id, "evt") || !id(data?.sessionID, "ses") || + !Number.isSafeInteger(event.created) || event.created <= 0 || + !Number.isSafeInteger(durable?.seq) || durable.seq < 0 || + !Number.isSafeInteger(durable?.version) || durable.version < 1 || + durable.aggregateID !== data.sessionID) return; + if (event.type !== "session.created" && !id(data.assistantMessageID, "msg")) return; + const nanos = BigInt(event.created) * 1000000n; + if (nanos > 18446744073709551615n) return; + const attributes = [ + text("event.name", event.type), + text("event.id", event.id), + text("session.id", data.sessionID), + integer("opencode.event.sequence", durable.seq), + integer("opencode.event.version", durable.version), + ]; + if (event.type === "session.created") { + if (label(data.projectID)) attributes.push(text("opencode.project.id", data.projectID)); + if (id(data.parentID, "ses")) attributes.push(text("opencode.parent.session.id", data.parentID)); + } else { + attributes.push(text("assistant.message.id", data.assistantMessageID)); + if (event.type === "session.step.started") { + if (label(data.model?.providerID)) attributes.push(text("opencode.model.provider", data.model.providerID)); + if (label(data.model?.id)) attributes.push(text("opencode.model.name", data.model.id)); + if (label(data.model?.variant)) attributes.push(text("opencode.model.variant", data.model.variant)); + if (label(data.agent)) attributes.push(text("opencode.agent.name", data.agent)); + } else { + const counters = [ + ["opencode.usage.input", data.tokens?.input], + ["opencode.usage.output", data.tokens?.output], + ["opencode.usage.reasoning", data.tokens?.reasoning], + ["opencode.usage.cache_read", data.tokens?.cache?.read], + ["opencode.usage.cache_write", data.tokens?.cache?.write], + ]; + for (const [key, value] of counters) { + // A zero may mean the upstream provider omitted the value. + if (Number.isSafeInteger(value) && value > 0) attributes.push(integer(key, value)); + } + } + } + return { timeUnixNano: String(nanos), attributes }; +} + +export default { + id: "aidd.otel", + setup(ctx) { + const target = destination(); + if (!target) { + if (process.env.AIDD_OPENCODE_OTEL_LOGS_ENDPOINT || process.env.AIDD_OPENCODE_OTEL_HEADERS) + console.warn("AIDD OpenTelemetry: the OpenCode exporter configuration is invalid."); + return; + } + const controller = new AbortController(); + const pending = new Set(); + let warned = false; + const warn = () => { + if (!warned) console.warn("AIDD OpenTelemetry: an OpenCode observation could not be exported."); + warned = true; + }; + const version = label(ctx.app?.version) ? ctx.app.version : undefined; + const resource = [text("service.name", "opencode")]; + if (version) resource.push(text("service.version", version)); + void (async () => { + try { + for await (const event of ctx.event.subscribe({ types, signal: controller.signal })) { + if (controller.signal.aborted) break; + const log = record(event); + if (!log) continue; + if (pending.size >= 64) { warn(); continue; } + const body = JSON.stringify({ resourceLogs: [{ + resource: { attributes: resource }, + scopeLogs: [{ scope: { name: "aidd.opencode.otel", version: "1" }, logRecords: [log] }], + }] }); + const work = fetch(target.url, { + method: "POST", + headers: { "content-type": "application/json", authorization: target.authorization }, + body, + redirect: "error", + signal: AbortSignal.timeout(1500), + }).then((response) => { if (!response.ok) warn(); }).catch(warn); + pending.add(work); + void work.finally(() => pending.delete(work)); + } + } catch { + if (!controller.signal.aborted) warn(); + } + })(); + return async () => { + controller.abort(); + await Promise.allSettled([...pending]); + }; + }, +}; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts index 04d6818b1..93c107f6d 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/init.ts @@ -8,7 +8,9 @@ import { type Edit, type Endpoint, type Host, + OPENCODE_CONFIG_ID, } from "../settings.js"; +import { installOpenCodePlugin } from "./opencode-plugin-file.js"; async function applyEdit(edit: Edit): Promise<boolean> { const before = await readSettings(edit.path, edit.initial); @@ -43,5 +45,11 @@ async function applyEdit(edit: Edit): Promise<boolean> { export async function initializeOtel(config: Config, endpoint: Endpoint, host: Host) { const edit = await configurationEdit(config, endpoint, host); - return { path: edit.path, launch: edit.launch, changed: await applyEdit(edit) }; + if (config === OPENCODE_CONFIG_ID) { + const before = await readSettings(edit.path, edit.initial); + edit.update(before.content); + } + const pluginChanged = config === OPENCODE_CONFIG_ID ? await installOpenCodePlugin(host) : false; + const settingsChanged = await applyEdit(edit); + return { path: edit.path, launch: edit.launch, changed: pluginChanged || settingsChanged }; } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts new file mode 100644 index 000000000..c32b66833 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts @@ -0,0 +1,41 @@ +import { constants } from "node:fs"; +import { lstat, mkdir, open, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import openCodePlugin from "../../../../../../assets/configs/otel-opencode-plugin.js.txt"; +import { type Host, openCodeConfigDir } from "../settings.js"; + +export const openCodePluginDirectory = (host: Host) => + join(openCodeConfigDir(host), "plugins", "aidd-otel"); + +export async function hasOpenCodePlugin(host: Host): Promise<boolean> { + const path = join(openCodePluginDirectory(host), "index.js"); + try { + const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const info = await file.stat(); + if ( + !info.isFile() || + (info.mode & 0o022) !== 0 || + (await file.readFile("utf8")) !== openCodePlugin + ) + throw new Error("OpenCode telemetry plugin is unsafe or has different content."); + return true; + } finally { + await file.close(); + } + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return false; + throw error; + } +} + +export async function installOpenCodePlugin(host: Host): Promise<boolean> { + if (await hasOpenCodePlugin(host)) return false; + const directory = openCodePluginDirectory(host); + await mkdir(directory, { recursive: true, mode: 0o700 }); + const info = await lstat(directory); + if (!info.isDirectory() || info.isSymbolicLink()) + throw new Error("Invalid OpenCode plugin directory."); + await writeFile(join(directory, "index.js"), openCodePlugin, { flag: "wx", mode: 0o600 }); + return true; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts new file mode 100644 index 000000000..a126a250e --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.ts @@ -0,0 +1,162 @@ +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; +import { OPENCODE_CONFIG_ID } from "../settings.js"; + +type Fields = Map<string, string | number>; +const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); +const object = (value: unknown) => asPlainObject(value) ?? {}; +const string = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const integer = (key: string, value: number) => ({ key, value: { intValue: String(value) } }); +const label = (value: unknown): value is string => + typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/@+#-]{0,199}$/u.test(value); +const id = (value: unknown, prefix: string): value is string => + typeof value === "string" && new RegExp(`^${prefix}_[A-Za-z0-9]{1,64}$`).test(value); + +function attributes(value: unknown): Fields { + const result: Fields = new Map(); + const seen = new Set<string>(); + for (const item of list(value)) { + const entry = object(item); + const data = object(entry.value); + if (typeof entry.key !== "string" || seen.has(entry.key)) return new Map(); + seen.add(entry.key); + const raw = data.stringValue ?? data.intValue; + if (typeof raw === "string" || typeof raw === "number") result.set(entry.key, raw); + } + return result; +} + +function numeric(value: unknown, minimum: number): number | undefined { + if (typeof value !== "number" && (typeof value !== "string" || !/^\d+$/.test(value))) return; + const result = Number(value); + if (Number.isSafeInteger(result) && result >= minimum) return result; + return undefined; +} + +/** Select only the plugin's documented technical fields; never forward raw provider logs. */ +export type OpenCodePluginLog = { session: string; parent?: string; payload: unknown }; + +export function openCodePluginLogs(line: string): OpenCodePluginLog[] { + const envelope = object(JSON.parse(line)); + if (envelope.signal !== "logs") return []; + const selected: OpenCodePluginLog[] = []; + for (const rawResource of list(object(envelope.payload).resourceLogs)) { + const resource = object(rawResource); + const resourceAttributes = attributes(object(resource.resource).attributes); + if (resourceAttributes.get("service.name") !== OPENCODE_CONFIG_ID) continue; + const version = resourceAttributes.get("service.version"); + for (const rawScope of list(resource.scopeLogs)) { + const scope = object(rawScope); + if (object(scope.scope).name !== "aidd.opencode.otel" || object(scope.scope).version !== "1") + continue; + for (const rawRecord of list(scope.logRecords)) { + const record = object(rawRecord); + const values = attributes(record.attributes); + const name = values.get("event.name"); + const eventId = values.get("event.id"); + const session = values.get("session.id"); + const message = values.get("assistant.message.id"); + const seq = numeric(values.get("opencode.event.sequence"), 0); + const eventVersion = numeric(values.get("opencode.event.version"), 1); + const time = record.timeUnixNano; + if ( + typeof name !== "string" || + ![ + "session.created", + "session.step.started", + "session.step.ended", + "session.step.failed", + ].includes(name) || + !id(eventId, "evt") || + !id(session, "ses") || + (name !== "session.created" && !id(message, "msg")) || + seq === undefined || + eventVersion !== 1 || + typeof time !== "string" || + !/^[1-9]\d{0,19}$/.test(time) || + BigInt(time) > 18446744073709551615n + ) + continue; + const safe = [ + string("event.name", name), + string("event.id", eventId), + string("session.id", session), + integer("opencode.event.sequence", seq), + integer("opencode.event.version", eventVersion), + ]; + if (typeof message === "string" && name !== "session.created") + safe.push(string("assistant.message.id", message)); + const strings = + name === "session.created" + ? ["opencode.project.id", "opencode.parent.session.id"] + : name === "session.step.started" + ? [ + "opencode.model.provider", + "opencode.model.name", + "opencode.model.variant", + "opencode.agent.name", + ] + : []; + for (const key of strings) { + const value = values.get(key); + if (label(value) && (key !== "opencode.parent.session.id" || id(value, "ses"))) + safe.push(string(key, value)); + } + if (name === "session.step.ended" || name === "session.step.failed") { + for (const key of [ + "opencode.usage.input", + "opencode.usage.output", + "opencode.usage.reasoning", + "opencode.usage.cache_read", + "opencode.usage.cache_write", + ]) { + const value = numeric(values.get(key), 1); + if (value !== undefined) safe.push(integer(key, value)); + } + } + const resourceAttrs = [string("service.name", OPENCODE_CONFIG_ID)]; + if (label(version)) resourceAttrs.push(string("service.version", version)); + const parent = values.get("opencode.parent.session.id"); + selected.push({ + session, + ...(name === "session.created" && id(parent, "ses") ? { parent } : {}), + payload: { + resourceLogs: [ + { + resource: { attributes: resourceAttrs }, + scopeLogs: [ + { + scope: { name: "aidd.opencode.otel", version: "1" }, + logRecords: [{ timeUnixNano: time, attributes: safe }], + }, + ], + }, + ], + }, + }); + } + } + } + return selected; +} + +/** Keep only descendants whose parent is explicitly present in a session.created observation. */ +export function relatedOpenCodeLogs(root: string, logs: OpenCodePluginLog[]): OpenCodePluginLog[] { + const parents = new Map<string, Set<string>>(); + for (const log of logs) { + if (!log.parent) continue; + const candidates = parents.get(log.session) ?? new Set<string>(); + candidates.add(log.parent); + parents.set(log.session, candidates); + } + const known = new Set([root]); + let changed = true; + while (changed) { + changed = false; + for (const [session, candidates] of parents) { + if (known.has(session) || candidates.size !== 1 || !known.has([...candidates][0])) continue; + known.add(session); + changed = true; + } + } + return logs.filter((log) => known.has(log.session)); +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts index 8975d0515..bee5ed343 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -1,9 +1,10 @@ import { spawn } from "node:child_process"; import { createReadStream } from "node:fs"; -import { mkdtemp, rename, rm, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; +import { mkdir, mkdtemp, rename, rm, writeFile } from "node:fs/promises"; +import { homedir, tmpdir } from "node:os"; import { join } from "node:path"; import { createInterface } from "node:readline"; +import openCodePlugin from "../../../../../../assets/configs/otel-opencode-plugin.js.txt"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import claude from "../config/claude.json" with { type: "json" }; import codex from "../config/codex.json" with { type: "json" }; @@ -15,6 +16,12 @@ import { readToken, } from "../settings.js"; import { startOtelCollector } from "./collect.js"; +import { hasOpenCodePlugin } from "./opencode-plugin-file.js"; +import { + type OpenCodePluginLog, + openCodePluginLogs, + relatedOpenCodeLogs, +} from "./opencode-plugin-logs.js"; import { destination, sessionIdsInCapture, syncOtel } from "./sync.js"; type RunOptions = { @@ -83,11 +90,37 @@ async function provider(options: RunOptions, collector: string, token: string, d input: options.prompt, }; } + const hasInstalledPlugin = await hasOpenCodePlugin({ home: homedir(), env: process.env }); + if (!hasInstalledPlugin) { + const pluginDir = join(directory, "opencode-plugin"); + await mkdir(pluginDir, { mode: 0o700 }); + await writeFile(join(pluginDir, "index.js"), openCodePlugin, { mode: 0o600, flag: "wx" }); + let inline: Record<string, unknown> = {}; + if (env.OPENCODE_CONFIG_CONTENT) { + try { + const parsed = asPlainObject(JSON.parse(env.OPENCODE_CONFIG_CONTENT)); + if (!parsed) throw new Error(); + inline = parsed; + } catch { + throw new Error( + "OPENCODE_CONFIG_CONTENT must be a JSON object to enable OpenCode telemetry for this run." + ); + } + } + if (inline.plugins !== undefined && !Array.isArray(inline.plugins)) + throw new Error("OpenCode inline plugins must be an array."); + env.OPENCODE_CONFIG_CONTENT = JSON.stringify({ + ...inline, + plugins: [...(inline.plugins ?? []), pluginDir], + }); + } return { env: { ...env, OTEL_EXPORTER_OTLP_ENDPOINT: `${collector}/${OPENCODE_CONFIG_ID}`, OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${token}`, + AIDD_OPENCODE_OTEL_LOGS_ENDPOINT: `${collector}/${OPENCODE_CONFIG_ID}/v1/logs`, + AIDD_OPENCODE_OTEL_HEADERS: `Authorization=Bearer ${token}`, }, args: ["run", "--standalone", ...model, "--", options.prompt], }; @@ -203,6 +236,7 @@ async function retainCapture(home: string, capture: string, config: Config): Pro try { await rename(capture, retained); await rm(join(home, ".local"), { recursive: true }); + await rm(join(home, "opencode-plugin"), { recursive: true, force: true }); await rm(join(home, "claude-settings.json"), { force: true }); return retained; } catch (error) { @@ -259,14 +293,43 @@ export async function runOtel(options: RunOptions): Promise<number> { origin ); let selected: unknown; + const logs: OpenCodePluginLog[] = []; + let rootSession: string | undefined; for await (const line of lines) { if (!line.trim()) continue; for (const payload of openCodePromptSpans(line)) { if (selected) throw new Error("The isolated OpenCode run emitted more than one prompt span."); selected = payload; + const resource = asPlainObject(list(asPlainObject(payload)?.resourceSpans)[0]); + const scope = asPlainObject(list(resource?.scopeSpans)[0]); + const span = asPlainObject(list(scope?.spans)[0]); + const path = attributes(span?.attributes).get("url.path"); + if (typeof path !== "string") throw new Error("OpenCode prompt session is missing."); + rootSession = path.split("/")[3]; + } + for (const log of openCodePluginLogs(line)) { + logs.push(log); } } + if (!rootSession || !selected) + throw new Error( + "OpenCode prompt trace is missing; session attribution cannot be verified." + ); + const related = relatedOpenCodeLogs(rootSession, logs); + if (related.length !== logs.length) + console.warn( + "OpenCode observations from sessions without a proven parent were not forwarded." + ); + if (!related.some((log) => log.session === rootSession)) + throw new Error("OpenCode telemetry plugin did not emit any session observations."); + const logsBody = JSON.stringify({ + resourceLogs: related.flatMap((log) => list(asPlainObject(log.payload)?.resourceLogs)), + }); + if (Buffer.byteLength(logsBody, "utf8") > 2 * 1024 * 1024) + throw new Error( + "OpenCode observations exceed the receiver's 2 MiB limit; nothing was sent." + ); if (selected) { const response = await fetch(url, { method: "POST", @@ -285,6 +348,28 @@ export async function runOtel(options: RunOptions): Promise<number> { throw new Error("Gouvernail did not retain the OpenCode span."); sent++; } + const logsUrl = new URL( + `/api/kpi/otel/${OPENCODE_CONFIG_ID}/${options.tenant}/v1/logs`, + origin + ); + try { + const response = await fetch(logsUrl, { + method: "POST", + redirect: "manual", + signal: AbortSignal.any([options.signal, AbortSignal.timeout(15000)]), + headers: { authorization: `Bearer ${options.token}`, "content-type": "application/json" }, + body: logsBody, + }); + if (!response.ok) + throw new Error(`Gouvernail rejected OpenCode logs: HTTP ${response.status}.`); + const receipt = asPlainObject(await response.json()); + if (receipt?.accepted !== related.length || Object.keys(receipt).length !== 1) + throw new Error("Gouvernail did not retain all OpenCode observations."); + } catch (error) { + const message = error instanceof Error ? error.message : "OpenCode logs could not be sent."; + throw new Error(`${message} The prompt trace may already have been retained.`); + } + sent += related.length; } else { const sessions = new Set<string>(); for await (const line of lines) { diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts index 08d223d81..865a59530 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/verify.ts @@ -3,7 +3,14 @@ import { open } from "node:fs/promises"; import { join } from "node:path"; import { readSettings } from "../files.js"; import { transformCapturedLine } from "../observations.js"; -import { type Config, configurationEdit, type Endpoint, type Host } from "../settings.js"; +import { + type Config, + configurationEdit, + type Endpoint, + type Host, + OPENCODE_CONFIG_ID, +} from "../settings.js"; +import { hasOpenCodePlugin } from "./opencode-plugin-file.js"; export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) { const edit = await configurationEdit(config, endpoint, host); @@ -14,6 +21,11 @@ export async function verifyOtel(config: Config, endpoint: Endpoint, host: Host) if (!current.private) { throw new Error("OTLP configuration must be private (mode 0600)."); } + if (config === OPENCODE_CONFIG_ID && !(await hasOpenCodePlugin(host))) { + throw new Error( + "OpenCode telemetry plugin is not installed. Run aidd telemetry otel init first." + ); + } const capture = join(host.home, ".local/share/aidd-otel-observation", `${config}.jsonl`); const handle = await open( capture, diff --git a/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json index 30fafb505..3782e5cef 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json +++ b/cli/src/contexts/telemetry/infrastructure/otel/config/opencode.json @@ -1,6 +1,8 @@ { "env": { "OTEL_EXPORTER_OTLP_ENDPOINT": "{collector}/opencode", - "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}" + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer {token}", + "AIDD_OPENCODE_OTEL_LOGS_ENDPOINT": "{collector}/opencode/v1/logs", + "AIDD_OPENCODE_OTEL_HEADERS": "Authorization=Bearer {token}" } } diff --git a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts index a1329eda5..a160bd81b 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/settings.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/settings.ts @@ -122,9 +122,13 @@ function openCodeGlobalDir(host: Host): string { return join(host.env.XDG_CONFIG_HOME || join(host.home, ".config"), OPENCODE_CONFIG_ID); } +export function openCodeConfigDir(host: Host): string { + return host.env.OPENCODE_CONFIG_DIR || openCodeGlobalDir(host); +} + async function openCodeEdit(host: Host, token: string): Promise<Edit> { await requireOpenCodeV2(host); - const directory = host.env.OPENCODE_CONFIG_DIR || openCodeGlobalDir(host); + const directory = openCodeConfigDir(host); const additions = interpolate(opencodeTemplate.env, token); return { path: join(directory, "service.json"), diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index d3ed68d2b..9a856c18a 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -138,9 +138,18 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: true }); expect(JSON.parse(await readFile(path, "utf8"))).toMatchObject({ port: 1234, - env: { EXISTING: "keep", OTEL_EXPORTER_OTLP_ENDPOINT: "http://127.0.0.1:14318/opencode" }, + env: { + EXISTING: "keep", + OTEL_EXPORTER_OTLP_ENDPOINT: "http://127.0.0.1:14318/opencode", + AIDD_OPENCODE_OTEL_LOGS_ENDPOINT: "http://127.0.0.1:14318/opencode/v1/logs", + }, }); + const plugin = join(dir, "plugins/aidd-otel/index.js"); + expect(await readFile(plugin, "utf8")).toContain('id: "aidd.otel"'); expect(await initializeOtel("opencode", "dev", v2)).toMatchObject({ changed: false }); + await writeFile(plugin, "user plugin\n"); + await expect(initializeOtel("opencode", "dev", v2)).rejects.toThrow(/plugin.*different/); + expect(await readFile(plugin, "utf8")).toBe("user plugin\n"); }); it("rejects OpenCode 1 and versions before 2.2.5", async () => { diff --git a/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts index bece11e52..e625d1ac4 100644 --- a/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts +++ b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts @@ -1,4 +1,8 @@ import { expect, it } from "vitest"; +import { + openCodePluginLogs, + relatedOpenCodeLogs, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.js"; import { openCodePromptSpans } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; const text = (key: string, stringValue: string) => ({ key, value: { stringValue } }); @@ -52,7 +56,82 @@ it("forwards only allowlisted OpenCode prompt span data", () => { expect(JSON.stringify(output)).not.toMatch(/url.full|host.name|secret/); }); +it("only keeps child sessions linked by an unambiguous explicit parent", () => { + const logs = [ + { session: "ses_root", payload: "root" }, + { session: "ses_child", parent: "ses_root", payload: "child" }, + { session: "ses_grandchild", parent: "ses_child", payload: "grandchild" }, + { session: "ses_unknown", payload: "unrelated" }, + { session: "ses_conflict", parent: "ses_root", payload: "conflict one" }, + { session: "ses_conflict", parent: "ses_unknown", payload: "conflict two" }, + ]; + expect(relatedOpenCodeLogs("ses_root", logs).map((item) => item.payload)).toEqual([ + "root", + "child", + "grandchild", + ]); +}); + it("rejects client spans and invalid timestamps", () => { expect(openCodePromptSpans(capture({ kind: 3 }))).toEqual([]); expect(openCodePromptSpans(capture({ startTimeUnixNano: "1700000000000000002" }))).toEqual([]); }); + +it("forwards only allowlisted plugin observations and never turns an ambiguous zero into usage", () => { + const value = (key: string, intValue: string) => ({ key, value: { intValue } }); + const attributes = [ + text("event.name", "session.step.ended"), + text("event.id", "evt_probe"), + text("session.id", "ses_probe"), + text("assistant.message.id", "msg_probe"), + value("opencode.event.sequence", "12"), + value("opencode.event.version", "1"), + value("opencode.usage.input", "15"), + value("opencode.usage.cache_read", "0"), + text("prompt", "private-prompt"), + ]; + const scope = { + scope: { name: "aidd.opencode.otel", version: "1" }, + logRecords: [ + { + timeUnixNano: "1700000000000000000", + body: { stringValue: "private-response" }, + attributes, + }, + ], + }; + const capture = JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { + attributes: [text("service.name", "opencode"), text("host.name", "private-host")], + }, + scopeLogs: [scope], + }, + ], + }, + }); + const output = openCodePluginLogs(capture); + expect(output).toHaveLength(1); + expect(output[0].session).toBe("ses_probe"); + expect(JSON.stringify(output)).toContain('"intValue":"15"'); + expect(JSON.stringify(output)).not.toMatch(/private|prompt|cache_read/); + scope.scope.version = "2"; + expect( + openCodePluginLogs( + JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: [text("service.name", "opencode")] }, + scopeLogs: [scope], + }, + ], + }, + }) + ) + ).toEqual([]); +}); diff --git a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts index 585751b5a..3cbd5ee40 100644 --- a/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/verify-otel.integration.test.ts @@ -19,6 +19,18 @@ afterEach(async () => { }); describe.skipIf(process.platform === "win32")("OTLP verification", () => { + it("requires the installed OpenCode plugin before reporting a capture", async () => { + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + await initializeOtel("opencode", "dev", host); + const plugin = join(home, ".config/opencode/plugins/aidd-otel/index.js"); + await writeFile( + join(home, ".local/share/aidd-otel-observation/opencode.jsonl"), + '{"signal":"logs"}\n' + ); + expect(await verifyOtel("opencode", "dev", host)).toMatchObject({ lines: 1 }); + await rm(plugin); + await expect(verifyOtel("opencode", "dev", host)).rejects.toThrow(/plugin is not installed/); + }); it("requires initialization without creating settings", async () => { await expect(verifyOtel("claude", "dev", { home, env: {} })).rejects.toThrow(/not initialized/); await expect(readFile(join(home, ".claude/settings.json"))).rejects.toMatchObject({ diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 34b0bb041..33546cc3e 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -1,19 +1,21 @@ # Provider OpenTelemetry -Gouvernail's AI KPI view uses provider OpenTelemetry (OTel) observations only. The framework's existing local usage ledger is separate and is not sent as OTel. A provider observation is not proof that every event was delivered; missing or ambiguous data remains unavailable. +Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2, an opted-in AIDD plugin converts public OpenCode server events into OTLP; these are not emitted by OpenCode's standard OTLP exporter. The framework's existing local usage ledger is separate and is not sent as OTel. An observation is not proof that every event was delivered; missing or ambiguous data remains unavailable. | Field | Claude Code OTel | Codex OTel | OpenCode v2 OTel | | --- | --- | --- | --- | -| Session | `session.id`, when present | `conversation.id`, when present | In the `url.path` of observed `/api/session/{id}/prompt` HTTP spans | -| Model | `model`, when present | `model`, when present | Not yet established for all signals | -| Usage | `claude_code.token.usage` and `api_request` | `codex.sse_event` and turn traces | Not established in the observed OTLP capture | +| Session | `session.id`, when present | `conversation.id`, when present | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | +| Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | +| Usage | `claude_code.token.usage` and `api_request` | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | | Identity | Provider attributes, when emitted | Provider attributes, when emitted | Not yet established | Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. `otel run --yes` captures one provider execution in isolation and forwards supported observations and allowlisted attributes to loopback Gouvernail. It neither reads the local usage ledger nor asserts a clone or feature association. With `--keep-capture`, raw private files can contain sensitive content. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). -In two local OpenCode v2.2.5 executions, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. Five other HTTP spans per execution contained that session ID but are not forwarded. The prompt spans had no trace-linked descendants in these captures; the model and usage spans belonged to a separate server trace without a session ID. A startup log included the requested model, but no session ID or proof of the model actually used. The framework forwards only the validated prompt span, not the other traces or logs. OpenCode computes usage on an internal event bus, which is not provider OTLP and is not used in this view. These observations do not prove what other versions or configurations emit. Production export remains disabled until authenticated user identity and the destination are implemented. +In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. Five other HTTP spans per execution contained that session ID but were not forwarded. The prompt spans had no trace-linked descendants in these captures; the model and usage spans belonged to a separate server trace without a session ID. A startup log included the requested model, but no session ID or proof of the model actually used. Those runs forwarded only the validated prompt span. These observations do not prove what other versions or configurations emit. + +The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.started`, `session.step.ended` and `session.step.failed`, and exports each event separately as OTLP logs. It includes the original event and session IDs, message ID when present, selected model and variant at step start, and positive OpenCode-normalized counters at step end. A creation event can also carry an OpenCode project ID and parent session ID; neither is a repository name. It does not export prompts, responses, tool contents or cost. The plugin stream can miss events; event IDs allow deduplication of observations received, not proof of complete delivery. Do not match retries solely by assistant message ID. Production export remains disabled. ## Additional coverage investigated @@ -21,7 +23,7 @@ In two local OpenCode v2.2.5 executions, the raw OTLP contained 1,335 spans, 276 | --- | --- | --- | | Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort, latency and separate token and cache counters. These fields appeared in both local captures. | The framework does not forward these events today. An allowlist of technical fields could expose them; avoid counting the same request again through metrics. Repository attributes require a separate opt-in. | | Codex | Logs, metrics and traces are independently configurable. Completed response events can include model, effort and usage. | The observed turn-usage histogram had no session ID and the response events had no turn or request ID. Neither timestamps nor a previous session's effort establish a safe join. Forward additional documented, session-linked events only after validating their fields. | -| OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | Standard emitted OTLP did not establish session-linked model, effort or usage. Instrumenting the producer before its usage normalization could emit explicit session-linked OTLP, but is not implemented or validated. Its internal plugin event bus is a different source, and normalizes missing counters to zero; a model variant is not necessarily reasoning effort. | +| OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | The opted-in plugin sends session-linked OpenCode events as OTLP. Missing counters are normalized to zero upstream, so zeros cannot establish actual usage. A model variant is not necessarily reasoning effort. | An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. @@ -35,6 +37,7 @@ Read 2026-11-06: - [OpenCode v2.2.5 OTLP exporter](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/util/src/observability/otlp.ts) - [OpenCode v2.2.5 usage projection](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/projector.ts#L81-L110) - [OpenCode v2.2.5 session events](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/schema/src/session-event.ts#L333-L389) and [usage normalization](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/usage.ts#L8-L19) +- [OpenCode v2 server plugins](https://dev.opencode.ai/v2/docs/build/plugins/server) and [retry handling](https://github.com/anomalyco/opencode/blob/v2.2.5/packages/core/src/session/runner/llm.ts#L229-L334) - [OpenCode v2 HTTP API audit](https://github.com/anomalyco/opencode/blob/dev/V2_HTTP_API_AUDIT.md) (current development branch; the v2.2.5 path was observed locally) - [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) - [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index a82fa5ee5..1a1bb9070 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -89,4 +89,4 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. Configure an exporter with `aidd telemetry otel init --config=<claude|codex|opencode> --endpoint=dev`, follow its launch instructions, then run `aidd telemetry otel verify` with the same options after a test session. The collector must already be running. See the [setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md) for destination requirements and verification limits, and the [provider field mapping](OBSERVABILITY.md) for available signals and their limits. -Restart the OpenCode v2 service after initialization. Receipt of OTLP logs or traces does not by itself establish that session identifiers or token usage were exported. +OpenCode initialization installs an opt-in server plugin. It sends session and event IDs, selected model and variant, optional OpenCode project and parent IDs, and positive OpenCode-normalized token counts to the configured local collector. It does not send prompts, responses, tool contents or cost. A missing or zero counter remains unavailable; delivery is not guaranteed. Restart the OpenCode v2 service after initialization. The one-execution `otel run --yes` enables this plugin for that run; it does not require global initialization. From 775594dd0c2545e16d7d7d63d2d77d74f26d287d Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 12:06:02 +0800 Subject: [PATCH 36/56] fix(telemetry): reject special plugin files without blocking --- .../otel/actions/opencode-plugin-file.ts | 2 +- .../otel/init-otel.integration.test.ts | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts index c32b66833..264d644b8 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts @@ -10,7 +10,7 @@ export const openCodePluginDirectory = (host: Host) => export async function hasOpenCodePlugin(host: Host): Promise<boolean> { const path = join(openCodePluginDirectory(host), "index.js"); try { - const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); + const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); try { const info = await file.stat(); if ( diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 9a856c18a..16d76f644 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -1,8 +1,11 @@ +import { execFileSync } from "node:child_process"; +import { constants } from "node:fs"; import { mkdir, mkdtemp, open, readFile, rm, stat, symlink, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; +import { hasOpenCodePlugin } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.js"; const TOKEN = "example-private-token"; let home: string; @@ -24,6 +27,30 @@ function seedToken() { } describe.skipIf(process.platform === "win32")("OTLP initialization", () => { + it("rejects a FIFO plugin without blocking", async () => { + const path = join(home, ".config/opencode/plugins/aidd-otel/index.js"); + await mkdir(dirname(path), { recursive: true }); + execFileSync("mkfifo", [path]); + const check = hasOpenCodePlugin({ home, env: {} }).then( + () => "accepted", + (error: Error) => error.message + ); + let timer: ReturnType<typeof setTimeout> | undefined; + const result = await Promise.race([ + check, + new Promise<string>((resolve) => { + timer = setTimeout(() => resolve("blocked"), 1000); + }), + ]); + if (timer) clearTimeout(timer); + if (result === "blocked") { + // Release a regressed blocking read so this behavioral check can fail cleanly. + const writer = await open(path, constants.O_WRONLY | constants.O_NONBLOCK); + await check; + await writer.close(); + } + expect(result).toBe("OpenCode telemetry plugin is unsafe or has different content."); + }); it("refuses production before even needing the local token or executable", async () => { await expect(initializeOtel("opencode", "prod", { home, env: {} })).rejects.toThrow( /production.*Gouvernail/ From 564533118076a41f082588ce62d2b0ef779c4e0c Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 12:10:17 +0800 Subject: [PATCH 37/56] fix(telemetry): reject conflicting OpenCode plugin entries --- .../otel/actions/opencode-plugin-file.ts | 35 ++++++++++++++++- .../otel/init-otel.integration.test.ts | 38 ++++++++++++++++++- 2 files changed, 70 insertions(+), 3 deletions(-) diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts index 264d644b8..c7e11ba0f 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts @@ -7,10 +7,40 @@ import { type Host, openCodeConfigDir } from "../settings.js"; export const openCodePluginDirectory = (host: Host) => join(openCodeConfigDir(host), "plugins", "aidd-otel"); +// OpenCode v2 resolves package exports first, then server.*, then index.*. +const competingEntries = [ + "package.json", + "server.ts", + "server.tsx", + "server.js", + "index.ts", + "index.tsx", +]; + export async function hasOpenCodePlugin(host: Host): Promise<boolean> { - const path = join(openCodePluginDirectory(host), "index.js"); + const directory = openCodePluginDirectory(host); try { - const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + const info = await lstat(directory); + if (!info.isDirectory() || info.isSymbolicLink()) + throw new Error("Invalid OpenCode plugin directory."); + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return false; + throw error; + } + for (const name of competingEntries) { + try { + await lstat(join(directory, name)); + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") continue; + throw error; + } + throw new Error("OpenCode telemetry plugin has a competing entry point."); + } + try { + const file = await open( + join(directory, "index.js"), + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); try { const info = await file.stat(); if ( @@ -36,6 +66,7 @@ export async function installOpenCodePlugin(host: Host): Promise<boolean> { const info = await lstat(directory); if (!info.isDirectory() || info.isSymbolicLink()) throw new Error("Invalid OpenCode plugin directory."); + if (await hasOpenCodePlugin(host)) return false; await writeFile(join(directory, "index.js"), openCodePlugin, { flag: "wx", mode: 0o600 }); return true; } diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 16d76f644..87f60fef8 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -6,6 +6,7 @@ import { dirname, join } from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { initializeOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/init.js"; import { hasOpenCodePlugin } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.js"; +import { verifyOtel } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/verify.js"; const TOKEN = "example-private-token"; let home: string; @@ -27,10 +28,45 @@ function seedToken() { } describe.skipIf(process.platform === "win32")("OTLP initialization", () => { + it.each(["package.json", "server.ts", "server.tsx", "server.js", "index.ts", "index.tsx"])( + "refuses competing %s without changing it", + async (name) => { + await seedToken(); + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + const entry = join(home, ".config/opencode/plugins/aidd-otel", name); + await seed(entry, "user content"); + await expect(initializeOtel("opencode", "dev", host)).rejects.toThrow( + /competing entry point/ + ); + expect(await readFile(entry, "utf8")).toBe("user content"); + await expect(stat(join(dirname(entry), "index.js"))).rejects.toMatchObject({ + code: "ENOENT", + }); + } + ); + it("rejects a competing symlink and FIFO without reading either", async () => { + const directory = join(home, ".config/opencode/plugins/aidd-otel"); + await mkdir(directory, { recursive: true }); + await symlink(join(home, "missing"), join(directory, "package.json")); + await expect(hasOpenCodePlugin({ home, env: {} })).rejects.toThrow(/competing entry point/); + await rm(join(directory, "package.json")); + execFileSync("/usr/bin/mkfifo", [join(directory, "server.js")]); + await expect(hasOpenCodePlugin({ home, env: {} })).rejects.toThrow(/competing entry point/); + }); + it("refuses a competing entry added after installation", async () => { + await seedToken(); + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + await initializeOtel("opencode", "dev", host); + const entry = join(home, ".config/opencode/plugins/aidd-otel/server.js"); + await writeFile(entry, "user plugin\n"); + await expect(hasOpenCodePlugin(host)).rejects.toThrow(/competing entry point/); + await expect(verifyOtel("opencode", "dev", host)).rejects.toThrow(/competing entry point/); + expect(await readFile(entry, "utf8")).toBe("user plugin\n"); + }); it("rejects a FIFO plugin without blocking", async () => { const path = join(home, ".config/opencode/plugins/aidd-otel/index.js"); await mkdir(dirname(path), { recursive: true }); - execFileSync("mkfifo", [path]); + execFileSync("/usr/bin/mkfifo", [path]); const check = hasOpenCodePlugin({ home, env: {} }).then( () => "accepted", (error: Error) => error.message From 83b7ccf9526b74f02c2eb3dc61f2526e744424ef Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 12:12:55 +0800 Subject: [PATCH 38/56] fix(telemetry): guard all OpenCode plugin entry variants --- .../otel/actions/opencode-plugin-file.ts | 28 ++++--------- .../otel/init-otel.integration.test.ts | 41 +++++++++++-------- 2 files changed, 34 insertions(+), 35 deletions(-) diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts index c7e11ba0f..03e4e4263 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-file.ts @@ -1,5 +1,5 @@ import { constants } from "node:fs"; -import { lstat, mkdir, open, writeFile } from "node:fs/promises"; +import { lstat, mkdir, open, readdir, writeFile } from "node:fs/promises"; import { join } from "node:path"; import openCodePlugin from "../../../../../../assets/configs/otel-opencode-plugin.js.txt"; import { type Host, openCodeConfigDir } from "../settings.js"; @@ -7,16 +7,6 @@ import { type Host, openCodeConfigDir } from "../settings.js"; export const openCodePluginDirectory = (host: Host) => join(openCodeConfigDir(host), "plugins", "aidd-otel"); -// OpenCode v2 resolves package exports first, then server.*, then index.*. -const competingEntries = [ - "package.json", - "server.ts", - "server.tsx", - "server.js", - "index.ts", - "index.tsx", -]; - export async function hasOpenCodePlugin(host: Host): Promise<boolean> { const directory = openCodePluginDirectory(host); try { @@ -27,14 +17,14 @@ export async function hasOpenCodePlugin(host: Host): Promise<boolean> { if (error instanceof Error && "code" in error && error.code === "ENOENT") return false; throw error; } - for (const name of competingEntries) { - try { - await lstat(join(directory, name)); - } catch (error) { - if (error instanceof Error && "code" in error && error.code === "ENOENT") continue; - throw error; - } - throw new Error("OpenCode telemetry plugin has a competing entry point."); + for (const name of await readdir(directory)) { + if ( + name === "package.json" || + name === "server" || + name.startsWith("server.") || + (name !== "index.js" && (name === "index" || name.startsWith("index."))) + ) + throw new Error("OpenCode telemetry plugin has a competing entry point."); } try { const file = await open( diff --git a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts index 87f60fef8..0409125d6 100644 --- a/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts +++ b/cli/tests/contexts/telemetry/otel/init-otel.integration.test.ts @@ -28,22 +28,28 @@ function seedToken() { } describe.skipIf(process.platform === "win32")("OTLP initialization", () => { - it.each(["package.json", "server.ts", "server.tsx", "server.js", "index.ts", "index.tsx"])( - "refuses competing %s without changing it", - async (name) => { - await seedToken(); - const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; - const entry = join(home, ".config/opencode/plugins/aidd-otel", name); - await seed(entry, "user content"); - await expect(initializeOtel("opencode", "dev", host)).rejects.toThrow( - /competing entry point/ - ); - expect(await readFile(entry, "utf8")).toBe("user content"); - await expect(stat(join(dirname(entry), "index.js"))).rejects.toMatchObject({ - code: "ENOENT", - }); - } - ); + it.each([ + "package.json", + "server", + "server.ts", + "server.tsx", + "server.js", + "server.mjs", + "index", + "index.ts", + "index.tsx", + "index.mts", + ])("refuses competing %s without changing it", async (name) => { + await seedToken(); + const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; + const entry = join(home, ".config/opencode/plugins/aidd-otel", name); + await seed(entry, "user content"); + await expect(initializeOtel("opencode", "dev", host)).rejects.toThrow(/competing entry point/); + expect(await readFile(entry, "utf8")).toBe("user content"); + await expect(stat(join(dirname(entry), "index.js"))).rejects.toMatchObject({ + code: "ENOENT", + }); + }); it("rejects a competing symlink and FIFO without reading either", async () => { const directory = join(home, ".config/opencode/plugins/aidd-otel"); await mkdir(directory, { recursive: true }); @@ -57,6 +63,9 @@ describe.skipIf(process.platform === "win32")("OTLP initialization", () => { await seedToken(); const host = { home, env: {}, opencodeVersion: async () => "2.2.5" }; await initializeOtel("opencode", "dev", host); + const notes = join(home, ".config/opencode/plugins/aidd-otel/notes.txt"); + await writeFile(notes, "unrelated\n"); + await expect(hasOpenCodePlugin(host)).resolves.toBe(true); const entry = join(home, ".config/opencode/plugins/aidd-otel/server.js"); await writeFile(entry, "user plugin\n"); await expect(hasOpenCodePlugin(host)).rejects.toThrow(/competing entry point/); From ea5c7277a2ea50b66a474385dbef39bb9879b0a5 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Fri, 6 Nov 2026 15:51:19 +0800 Subject: [PATCH 39/56] docs(telemetry): clarify OpenCode token attribution --- aidd_docs/recipes/observe-agent-telemetry.md | 2 +- plugins/aidd-telemetry/OBSERVABILITY.md | 2 +- plugins/aidd-telemetry/README.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index 63c932ca5..045477400 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -6,7 +6,7 @@ Run Claude Code, Codex and OpenCode v2 with OpenTelemetry (OTel), then inspect t ### 1) โš™๏ธ Prepare the collector destination -Start local Gouvernail and configure each tool's connector. OpenCode's connector is generated in the Sources page for an explicit team. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. +Start local Gouvernail and configure each tool's connector. Generate OpenCode's token in Sources for an explicit team and member; issuing a new token replaces the previous OpenCode token for that organization. The member is the token's declared assignee, not proof of the person using it. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. | Tool | Connector token | | --- | --- | diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 33546cc3e..9dad3fd24 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -7,7 +7,7 @@ Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2 | Session | `session.id`, when present | `conversation.id`, when present | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | | Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | | Usage | `claude_code.token.usage` and `api_request` | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | -| Identity | Provider attributes, when emitted | Provider attributes, when emitted | Not yet established | +| Identity | Provider attributes, when emitted | Provider attributes, when emitted | Member explicitly assigned to the connector token in local Gouvernail; not a provider identity or proof of the person holding the token | Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 1a1bb9070..1ab51f95c 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -31,7 +31,7 @@ The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The | Mode | Opt-in and destination | Identity | What is kept or sent | | --- | --- | --- | --- | | Local usage | `aidd telemetry on` explicitly enables measurement for the current clone from that time. The ledger and report stay on this machine. | An optional name labels the local report; anonymous use is supported. | Per-call usage read from Claude Code and Codex session files: session, time, model, agent, token counts and local attribution context. The ledger does not store prompts or responses. | -| OTLP development | `otel run --yes` explicitly opts in one isolated execution and sends supported observations to loopback Gouvernail. | Provider identifiers may be sent when present; attribution requires a matched active member. The connector token is not a person identity. | With `--keep-capture`, the private capture can contain prompts, responses, commands and tool output. Forwarding filters fields but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | +| OTLP development | `otel run --yes` explicitly opts in one isolated execution and sends supported observations to loopback Gouvernail. | Provider identifiers may be sent when present; attribution requires a matched active member. For OpenCode, a token can be explicitly assigned to a member in local Gouvernail; this does not prove who holds it. | With `--keep-capture`, the private capture can contain prompts, responses, commands and tool output. Forwarding filters fields but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | | OTLP production | **Not available:** `--endpoint=prod` is refused. `https://gouvernail.eu` is an intended future destination, not an active exporter. | **Required before production is enabled:** an authenticated user identity. This is a product requirement, not yet implemented or enforced. | No production payload is sent by this feature today. Consent, authentication, retention and deletion behavior must be implemented before enabling it. | ### Consent and scope From fbeddb997802616d6dfdc9ee5994bbd7d689a806 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 08:33:37 +0800 Subject: [PATCH 40/56] feat(telemetry): observe two turns in one session Make local comparisons reproducible and preserve explicitly enabled repository metadata. --- aidd_docs/recipes/observe-agent-telemetry.md | 39 ++- .../infrastructure/otel/actions/run.ts | 296 ++++++++++++++++-- .../infrastructure/otel/actions/sync.ts | 20 +- cli/src/presentation/commands/telemetry.ts | 7 + .../telemetry/otel/run-otel.unit.test.ts | 26 +- .../otel/sync-repository.integration.test.ts | 107 +++++++ 6 files changed, 444 insertions(+), 51 deletions(-) create mode 100644 cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index 045477400..684ca6277 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -16,31 +16,30 @@ Start local Gouvernail and configure each tool's connector. Generate OpenCode's `--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. -### 2) ๐Ÿงช Run two sessions per tool +### 2) ๐Ÿงช Send two messages in each session -Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each pair. Choose models available to your accounts. +Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each command. Choose models available to your accounts. ```sh -# Claude connector token must be set before this pair. -for i in 1 2; do - aidd telemetry otel run --config claude --model haiku --tenant ai-driven-dev \ - --endpoint http://127.0.0.1:55857 --yes --keep-capture -- "Reply with only CLAUDE-$i." -done - -# Codex connector token must be set before this pair. -for i in 1 2; do - aidd telemetry otel run --config codex --model gpt-5.6-luna --tenant ai-driven-dev \ - --endpoint http://127.0.0.1:55857 --yes --keep-capture -- "Reply with only CODEX-$i." -done - -# OpenCode connector token must be set before this pair. -for i in 1 2; do - aidd telemetry otel run --config opencode --model claude-code/claude-haiku-4-5-20251001 \ - --tenant ai-driven-dev --endpoint http://127.0.0.1:55857 --yes --keep-capture -- "Reply with only OPENCODE-$i." -done +# Claude connector token +aidd telemetry otel run --config claude --model haiku --tenant ai-driven-dev \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture \ + --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." + +# Codex connector token +aidd telemetry otel run --config codex --model gpt-6-luna --tenant ai-driven-dev \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture \ + --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." + +# OpenCode connector token +aidd telemetry otel run --config opencode --model claude-code/claude-haiku-4-5-20251001 \ + --tenant ai-driven-dev --endpoint http://127.0.0.1:55857 --yes --keep-capture \ + --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." ``` -Each command launches a separate session with an isolated collector. The OpenCode command loads the AIDD plugin for that run. It does not read the framework usage ledger or establish a repository or feature association. `otel init` and the global development collector are a separate configuration path; they are not required for these commands. +Each command uses an isolated collector and resumes the first session for the follow-up. Claude and Codex persist these conversations in their usual local session history; OpenCode uses a temporary database removed after the run. This history is used only by the tools to resume; Gouvernail receives OTLP, not the tools' session files. Verify both responses when checking continuity: a shared session ID alone does not prove the model used the first message. + +The command waits for an OTLP usage observation after each execution. Claude identifies the matching prompt explicitly; OpenCode supplies a completed step and message ID. Codex supplies session-linked usage but no reliable per-prompt join; observations can include additional requests, so do not attribute their totals to a specific prompt. An emitted observation does not guarantee that every requested field is available. `otel init` and the global development collector are a separate configuration path. ### 3) โœ… Check receipt diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts index bee5ed343..a5271ef9c 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -1,6 +1,7 @@ import { spawn } from "node:child_process"; +import { randomBytes, randomUUID } from "node:crypto"; import { createReadStream } from "node:fs"; -import { mkdir, mkdtemp, rename, rm, writeFile } from "node:fs/promises"; +import { mkdir, mkdtemp, open, rename, rm, writeFile } from "node:fs/promises"; import { homedir, tmpdir } from "node:os"; import { join } from "node:path"; import { createInterface } from "node:readline"; @@ -31,6 +32,8 @@ type RunOptions = { token: string; cwd: string; prompt: string; + followUp?: string; + includeRepository?: boolean; model?: string; retainCapture?: (path: string) => void; signal: AbortSignal; @@ -55,14 +58,14 @@ async function provider(options: RunOptions, collector: string, token: string, d settings.OTEL_TRACES_EXPORTER = "none"; settings.OTEL_METRIC_EXPORT_INTERVAL = "1000"; settings.OTEL_LOGS_EXPORT_INTERVAL = "1000"; - settings.OTEL_METRICS_INCLUDE_REPOSITORY = "false"; + settings.OTEL_METRICS_INCLUDE_REPOSITORY = options.includeRepository ? "true" : "false"; const file = join(directory, "claude-settings.json"); await writeFile(file, JSON.stringify({ env: settings }), { mode: 0o600, flag: "wx" }); return { env: { ...env, ...settings }, args: [ "--print", - "--no-session-persistence", + ...(options.followUp ? ["--session-id", randomUUID()] : ["--no-session-persistence"]), "--settings", file, ...model, @@ -80,7 +83,7 @@ async function provider(options: RunOptions, collector: string, token: string, d env, args: [ "exec", - "--ephemeral", + ...(options.followUp ? ["--json"] : ["--ephemeral"]), ...model, ...overrides, "-c", @@ -90,6 +93,11 @@ async function provider(options: RunOptions, collector: string, token: string, d input: options.prompt, }; } + if (options.followUp) { + const databaseDir = join(directory, "opencode-data"); + await mkdir(databaseDir, { mode: 0o700 }); + env.OPENCODE_DB = join(databaseDir, "session.db"); + } const hasInstalledPlugin = await hasOpenCodePlugin({ home: homedir(), env: process.env }); if (!hasInstalledPlugin) { const pluginDir = join(directory, "opencode-plugin"); @@ -122,7 +130,14 @@ async function provider(options: RunOptions, collector: string, token: string, d AIDD_OPENCODE_OTEL_LOGS_ENDPOINT: `${collector}/${OPENCODE_CONFIG_ID}/v1/logs`, AIDD_OPENCODE_OTEL_HEADERS: `Authorization=Bearer ${token}`, }, - args: ["run", "--standalone", ...model, "--", options.prompt], + args: [ + "run", + "--standalone", + ...(options.followUp ? ["--session", `ses_${randomBytes(16).toString("hex")}`] : []), + ...model, + "--", + options.prompt, + ], }; } @@ -237,6 +252,7 @@ async function retainCapture(home: string, capture: string, config: Config): Pro await rename(capture, retained); await rm(join(home, ".local"), { recursive: true }); await rm(join(home, "opencode-plugin"), { recursive: true, force: true }); + await rm(join(home, "opencode-data"), { recursive: true, force: true }); await rm(join(home, "claude-settings.json"), { force: true }); return retained; } catch (error) { @@ -245,10 +261,197 @@ async function retainCapture(home: string, capture: string, config: Config): Pro } } +function promptSession(payload: unknown): string | undefined { + const resource = asPlainObject(list(asPlainObject(payload)?.resourceSpans)[0]); + const scope = asPlainObject(list(resource?.scopeSpans)[0]); + const span = asPlainObject(list(scope?.spans)[0]); + const path = attributes(span?.attributes).get("url.path"); + return typeof path === "string" ? path.split("/")[3] : undefined; +} + +/** Find the last complete JSONL record without consuming an in-progress append. */ +export async function completeCaptureEnd(capture: string, start: number): Promise<number> { + const file = await open(capture, "r"); + try { + let end = (await file.stat()).size; + const buffer = Buffer.alloc(4096); + while (end > start) { + const length = Math.min(buffer.length, end - start); + const offset = end - length; + const { bytesRead } = await file.read(buffer, 0, length, offset); + const newline = buffer.subarray(0, bytesRead).lastIndexOf(10); + if (newline >= 0) return offset + newline + 1; + end = offset; + } + return start; + } finally { + await file.close(); + } +} + +async function captureSessions(capture: string, config: Config, start: number) { + const end = await completeCaptureEnd(capture, start); + const sessions = new Set<string>(); + const prompts = new Set<string>(); + const usage = new Set<string>(); + if (end <= start) return { end, sessions, usage }; + const lines = createInterface({ + input: createReadStream(capture, { start, end: end - 1 }), + crlfDelay: Infinity, + }); + for await (const line of lines) { + if (!line.trim()) continue; + if (config === OPENCODE_CONFIG_ID) { + for (const payload of openCodePromptSpans(line)) { + const session = promptSession(payload); + if (session) sessions.add(session); + } + for (const log of openCodePluginLogs(line)) { + const resource = asPlainObject(list(asPlainObject(log.payload)?.resourceLogs)[0]); + const scope = asPlainObject(list(resource?.scopeLogs)[0]); + const record = asPlainObject(list(scope?.logRecords)[0]); + const fields = attributes(record?.attributes); + const message = fields.get("assistant.message.id"); + if (fields.get("event.name") === "session.step.ended" && typeof message === "string") + usage.add(`${log.session}:${message}`); + } + } else { + for (const session of sessionIdsInCapture(line, config)) sessions.add(session); + const envelope = asPlainObject(JSON.parse(line)); + for (const resource of list(asPlainObject(envelope?.payload)?.resourceLogs)) { + for (const scope of list(asPlainObject(resource)?.scopeLogs)) { + for (const record of list(asPlainObject(scope)?.logRecords)) { + const fields = attributes(asPlainObject(record)?.attributes); + const name = fields.get("event.name"); + const prompt = fields.get("prompt.id"); + if (config === CLAUDE_CONFIG_ID && typeof prompt === "string") { + if (name === "user_prompt") prompts.add(prompt); + if (name === "api_request") usage.add(prompt); + } + if ( + config === CODEX_CONFIG_ID && + name === "codex.sse_event" && + fields.get("event.kind") === "response.completed" && + fields.has("input_token_count") && + fields.has("output_token_count") + ) + usage.add(String(fields.get("event.timestamp") ?? "completed")); + } + } + } + } + } + if (config === CLAUDE_CONFIG_ID) + for (const item of usage) if (!prompts.has(item)) usage.delete(item); + return { end, sessions, usage }; +} + +async function waitForUsage( + capture: string, + config: Config, + start: number, + expected: string, + signal: AbortSignal +) { + const deadline = Date.now() + 10_000; + while (true) { + const observation = await captureSessions(capture, config, start); + if (config === OPENCODE_CONFIG_ID) + for (const item of observation.usage) + if (!item.startsWith(`${expected}:`)) observation.usage.delete(item); + if ( + observation.sessions.size === 1 && + observation.sessions.has(expected) && + observation.usage.size > 0 + ) + return observation; + if (Date.now() >= deadline || signal.aborted) + throw new Error( + "The provider did not emit a completed usage observation for the expected session." + ); + await new Promise<void>((resolve) => setTimeout(resolve, 100)); + } +} + +function requireSession(sessions: Set<string>, expected: string): void { + if (sessions.size !== 1 || !sessions.has(expected)) + throw new Error("The execution did not emit exactly the expected OTLP session."); +} + +async function executeProvider(input: { + config: Config; + args: string[]; + env: NodeJS.ProcessEnv; + cwd: string; + prompt?: string; + signal: AbortSignal; + codexJson: boolean; +}): Promise<string | undefined> { + let thread: string | undefined; + let invalidThread = false; + const code = await new Promise<number | null>((resolve, reject) => { + const child = spawn(input.config, input.args, { + cwd: input.cwd, + env: input.env, + signal: input.signal, + stdio: [input.prompt ? "pipe" : "ignore", input.codexJson ? "pipe" : "inherit", "inherit"], + }); + if (child.stdout) { + const lines = createInterface({ input: child.stdout, crlfDelay: Infinity }); + lines.on("line", (line) => { + let event: Record<string, unknown> | null = null; + try { + event = asPlainObject(JSON.parse(line)); + } catch { + invalidThread = true; + return; + } + if (event?.type === "thread.started") { + const value = event.thread_id; + if ( + thread || + typeof value !== "string" || + !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value) + ) + invalidThread = true; + else thread = value; + } + const item = asPlainObject(event?.item); + if ( + event?.type === "item.completed" && + item?.type === "agent_message" && + typeof item.text === "string" + ) + process.stdout.write(`${item.text}\n`); + }); + } + child.on("error", reject); + child.on("close", resolve); + child.stdin?.on("error", reject); + child.stdin?.end(input.prompt); + }); + if (code !== 0) throw new Error(`${input.config} execution did not complete successfully.`); + if (input.codexJson && (!thread || invalidThread)) + throw new Error("Codex did not report exactly one valid session ID."); + return thread; +} + +function resumeArgs(config: Config, first: string[], session: string, prompt: string): string[] { + if (config === CODEX_CONFIG_ID) return ["exec", "resume", ...first.slice(1, -1), session, "-"]; + const args = [...first]; + if (config === CLAUDE_CONFIG_ID) args[args.indexOf("--session-id")] = "--resume"; + args[args.length - 1] = prompt; + return args; +} + /** Consent covers this one invocation and its isolated collector only. */ export async function runOtel(options: RunOptions): Promise<number> { if (process.env.AIDD_TELEMETRY === "0") throw new Error("AIDD telemetry is disabled."); if (!options.prompt.trim()) throw new Error("Pass a nonempty prompt."); + if (options.followUp !== undefined && !options.followUp.trim()) + throw new Error("Pass a nonempty follow-up prompt."); + if (options.includeRepository && options.config !== CLAUDE_CONFIG_ID) + throw new Error("Repository metadata is only supported for Claude Code."); if (options.model && !/^[A-Za-z0-9][A-Za-z0-9._:/#-]{0,127}$/.test(options.model)) { throw new Error("Provider model ID is invalid."); } @@ -269,19 +472,58 @@ export async function runOtel(options: RunOptions): Promise<number> { await readToken(home), home ); - const code = await new Promise<number | null>((resolve, reject) => { - const child = spawn(options.config, launch.args, { - cwd: options.cwd, + const first = await executeProvider({ + config: options.config, + args: launch.args, + env: launch.env, + cwd: options.cwd, + prompt: launch.input, + signal: options.signal, + codexJson: !!options.followUp && options.config === CODEX_CONFIG_ID, + }); + if (options.followUp) { + const session = + options.config === CODEX_CONFIG_ID + ? first + : launch.args[ + launch.args.indexOf( + options.config === CLAUDE_CONFIG_ID ? "--session-id" : "--session" + ) + 1 + ]; + if (!session) throw new Error("The provider session is missing."); + const firstCapture = await waitForUsage( + capture, + options.config, + 0, + session, + options.signal + ); + requireSession(firstCapture.sessions, session); + const second = await executeProvider({ + config: options.config, + args: resumeArgs(options.config, launch.args, session, options.followUp), env: launch.env, + cwd: options.cwd, + prompt: options.config === CODEX_CONFIG_ID ? options.followUp : undefined, signal: options.signal, - stdio: [launch.input ? "pipe" : "ignore", "inherit", "inherit"], + codexJson: options.config === CODEX_CONFIG_ID, }); - child.on("error", reject); - child.on("close", resolve); - child.stdin?.on("error", reject); - child.stdin?.end(launch.input); - }); - if (code !== 0) throw new Error(`${options.config} execution did not complete successfully.`); + if (options.config === CODEX_CONFIG_ID && second !== session) + throw new Error("Codex resumed a different session."); + const secondCapture = await waitForUsage( + capture, + options.config, + firstCapture.end, + session, + options.signal + ); + requireSession(secondCapture.sessions, session); + if ( + options.config !== CODEX_CONFIG_ID && + [...secondCapture.usage].some((id) => firstCapture.usage.has(id)) + ) + throw new Error("The follow-up did not emit a distinct usage observation."); + } } finally { await collector.close(); } @@ -292,27 +534,25 @@ export async function runOtel(options: RunOptions): Promise<number> { `/api/kpi/otel/${OPENCODE_CONFIG_ID}/${options.tenant}/v1/traces`, origin ); - let selected: unknown; + const selected: unknown[] = []; const logs: OpenCodePluginLog[] = []; let rootSession: string | undefined; for await (const line of lines) { if (!line.trim()) continue; for (const payload of openCodePromptSpans(line)) { - if (selected) - throw new Error("The isolated OpenCode run emitted more than one prompt span."); - selected = payload; - const resource = asPlainObject(list(asPlainObject(payload)?.resourceSpans)[0]); - const scope = asPlainObject(list(resource?.scopeSpans)[0]); - const span = asPlainObject(list(scope?.spans)[0]); - const path = attributes(span?.attributes).get("url.path"); - if (typeof path !== "string") throw new Error("OpenCode prompt session is missing."); - rootSession = path.split("/")[3]; + const session = promptSession(payload); + if (!session || (rootSession && rootSession !== session)) + throw new Error("OpenCode prompt spans do not identify the same session."); + rootSession = session; + selected.push(payload); + if (selected.length > (options.followUp ? 2 : 1)) + throw new Error("The isolated OpenCode run emitted more prompt spans than expected."); } for (const log of openCodePluginLogs(line)) { logs.push(log); } } - if (!rootSession || !selected) + if (!rootSession || selected.length !== (options.followUp ? 2 : 1)) throw new Error( "OpenCode prompt trace is missing; session attribution cannot be verified." ); @@ -330,7 +570,7 @@ export async function runOtel(options: RunOptions): Promise<number> { throw new Error( "OpenCode observations exceed the receiver's 2 MiB limit; nothing was sent." ); - if (selected) { + for (const payload of selected) { const response = await fetch(url, { method: "POST", redirect: "manual", @@ -339,7 +579,7 @@ export async function runOtel(options: RunOptions): Promise<number> { authorization: `Bearer ${options.token}`, "content-type": "application/json", }, - body: JSON.stringify(selected), + body: JSON.stringify(payload), }); if (!response.ok) throw new Error(`Gouvernail rejected OpenCode traces: HTTP ${response.status}.`); diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index c5ca82f87..ff1496b7e 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -81,6 +81,7 @@ const SAFE_ATTRIBUTES = new Set([ "session.id", "conversation.id", "organization.id", + "vcs.repository.url.full", "event.name", "event.kind", "event.sequence", @@ -142,7 +143,11 @@ function merge(...maps: Map<string, string>[]): Map<string, string> { for (const map of maps) { for (const [key, value] of map) { const previous = out.get(key); - if (IDENTITY_ATTRIBUTES.has(key) && previous !== undefined && previous !== value) + if ( + (IDENTITY_ATTRIBUTES.has(key) || key === "vcs.repository.url.full") && + previous !== undefined && + previous !== value + ) throw new Error("Conflicting OTLP identity attributes."); out.set(key, value); } @@ -152,13 +157,24 @@ function merge(...maps: Map<string, string>[]): Map<string, string> { function safeAttributes(values: Map<string, string>) { return [...values] - .filter(([key]) => SAFE_ATTRIBUTES.has(key)) + .filter( + ([key, value]) => + SAFE_ATTRIBUTES.has(key) && + (key !== "vcs.repository.url.full" || isSafeRepositoryUrl(value)) + ) .map(([key, value]) => ({ key, value: { stringValue: value }, })); } +function isSafeRepositoryUrl(value: string): boolean { + return ( + /^https:\/\/(?:github\.com|gitlab\.com)\/[A-Za-z0-9_.-]+(?:\/[A-Za-z0-9_.-]+)+$/.test(value) && + !value.split("/").some((part) => part === "." || part === "..") + ); +} + function eventNameFromBody(body: unknown): string { if (body === undefined) return ""; if (typeof object(body).stringValue !== "string") diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index 459e0e886..b0b8e0dc5 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -219,6 +219,11 @@ export function registerTelemetryCommand(program: Command): void { .requiredOption("--endpoint <origin>", "Loopback HTTP origin") .option("--yes", "Consent to capture and send telemetry from this execution") .option("--model <model>", "Provider model to use") + .option( + "--follow-up <prompt>", + "Send a second prompt in the same session (Claude and Codex retain local history)" + ) + .option("--include-repository", "Include repository metadata for Claude Code") .option("--keep-capture", "Retain the private local OTLP capture") .action( async ( @@ -229,6 +234,8 @@ export function registerTelemetryCommand(program: Command): void { endpoint: string; yes?: boolean; model?: string; + followUp?: string; + includeRepository?: boolean; keepCapture?: boolean; } ) => { diff --git a/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts index e625d1ac4..9bf2ea8fd 100644 --- a/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts +++ b/cli/tests/contexts/telemetry/otel/run-otel.unit.test.ts @@ -1,9 +1,15 @@ +import { appendFile, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { expect, it } from "vitest"; import { openCodePluginLogs, relatedOpenCodeLogs, } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/opencode-plugin-logs.js"; -import { openCodePromptSpans } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; +import { + completeCaptureEnd, + openCodePromptSpans, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/run.js"; const text = (key: string, stringValue: string) => ({ key, value: { stringValue } }); @@ -135,3 +141,21 @@ it("forwards only allowlisted plugin observations and never turns an ambiguous z ) ).toEqual([]); }); + +it("only advances the capture boundary after a complete JSONL record", async () => { + const directory = await mkdtemp(join(tmpdir(), "aidd-otel-boundary-")); + const file = join(directory, "capture.jsonl"); + try { + const first = '{"value":"รฉ"}\n'; + await writeFile(file, `${first}{"value":`); + const firstEnd = Buffer.byteLength(first); + expect(await completeCaptureEnd(file, 0)).toBe(firstEnd); + expect(await completeCaptureEnd(file, firstEnd)).toBe(firstEnd); + await appendFile(file, '"complete"}\n'); + expect(await completeCaptureEnd(file, firstEnd)).toBe( + firstEnd + Buffer.byteLength('{"value":"complete"}\n') + ); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); diff --git a/cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts b/cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts new file mode 100644 index 000000000..1ba95a2e5 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/sync-repository.integration.test.ts @@ -0,0 +1,107 @@ +import { describe, expect, it } from "vitest"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "claude" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:3000", + token: "test", + home: "/tmp", + follow: false, +}; +const payload = (signal: "logs" | "metrics", url: string, conflictingUrl?: string) => + JSON.stringify({ + signal, + payload: + signal === "logs" + ? { + resourceLogs: [ + { + resource: { + attributes: [attr("session.id", "session"), attr("vcs.repository.url.full", url)], + }, + scopeLogs: [ + { + logRecords: [ + { + timeUnixNano: "1793763900000000000", + attributes: [ + attr("event.name", "tool_result"), + ...(conflictingUrl + ? [attr("vcs.repository.url.full", conflictingUrl)] + : []), + ], + }, + ], + }, + ], + }, + ], + } + : { + resourceMetrics: [ + { + resource: { + attributes: [attr("session.id", "session"), attr("vcs.repository.url.full", url)], + }, + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { + dataPoints: [ + { + timeUnixNano: "1793763900000000000", + asInt: "42", + attributes: [ + attr("type", "input"), + ...(conflictingUrl + ? [attr("vcs.repository.url.full", conflictingUrl)] + : []), + ], + }, + ], + }, + }, + ], + }, + ], + }, + ], + }, + }); + +describe("optional OTLP repository", () => { + it.each(["logs", "metrics"] as const)("forwards a safe repository in %s", (signal) => { + const transfers = selectTransfers(payload(signal, "https://github.com/org/repo"), options); + expect(transfers).toHaveLength(1); + expect(JSON.stringify(transfers)).toContain("https://github.com/org/repo"); + }); + it.each([ + "https://user:secret@github.com/org/repo", + "https://github.com/org/repo?token=secret", + "https://github.com/org/repo#secret", + "https://evil.example/org/repo", + ])("does not forward unsafe URL %s", (url) => { + for (const signal of ["logs", "metrics"] as const) { + const transfers = selectTransfers(payload(signal, url), options); + expect(transfers).toHaveLength(1); + expect(JSON.stringify(transfers)).not.toContain("vcs.repository.url.full"); + expect(JSON.stringify(transfers)).not.toContain("secret"); + } + }); + it.each(["logs", "metrics"] as const)( + "rejects conflicting repository attributes in %s", + (signal) => { + expect(() => + selectTransfers( + payload(signal, "https://github.com/org/one", "https://github.com/org/two"), + options + ) + ).toThrow(); + } + ); +}); From 99da51f6f1c306cf5bd7732ccfd1c5d53a6e5647 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 08:36:01 +0800 Subject: [PATCH 41/56] test(telemetry): refresh command help snapshot Keep the command reference aligned with the two-turn runner. --- cli/tests/golden/snapshots/help/surface.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index fb794822b..7d112125f 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -182,7 +182,7 @@ { "invocation": "aidd telemetry otel run", "exitCode": 0, - "help": "Usage: aidd telemetry otel run [options] <prompt>\n\nRun one fresh provider session and send its allowed OTLP observations\n\nOptions:\n --config <tool> Tool to run (choices: \"claude\", \"codex\", \"opencode\")\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --yes Consent to capture and send telemetry from this execution\n --model <model> Provider model to use\n --keep-capture Retain the private local OTLP capture\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel run [options] <prompt>\n\nRun one fresh provider session and send its allowed OTLP observations\n\nOptions:\n --config <tool> Tool to run (choices: \"claude\", \"codex\", \"opencode\")\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --yes Consent to capture and send telemetry from this\n execution\n --model <model> Provider model to use\n --follow-up <prompt> Send a second prompt in the same session (Claude and\n Codex retain local history)\n --include-repository Include repository metadata for Claude Code\n --keep-capture Retain the private local OTLP capture\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", From 0c38c27e397c0fe4bd2a222bf90c6a433d53061d Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 08:43:12 +0800 Subject: [PATCH 42/56] feat(telemetry): opt in to installation attribution Allow session observations to carry a declared installation without treating it as a user identity. --- aidd_docs/recipes/observe-agent-telemetry.md | 2 + .../otel/actions/installation-attribute.ts | 50 +++++++++ .../infrastructure/otel/actions/run.ts | 19 +++- .../infrastructure/otel/actions/sync.ts | 15 ++- cli/src/presentation/commands/telemetry.ts | 2 + cli/src/runtime/wiring/telemetry-otel.ts | 11 +- .../otel/installation-attribute.unit.test.ts | 104 ++++++++++++++++++ cli/tests/golden/snapshots/help/surface.json | 2 +- plugins/aidd-telemetry/README.md | 4 +- 9 files changed, 200 insertions(+), 9 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts create mode 100644 cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index 684ca6277..733b830e7 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -16,6 +16,8 @@ Start local Gouvernail and configure each tool's connector. Generate OpenCode's `--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. +Add `--include-installation-id` to send the existing AIDD installation UUID as an AIDD-added `aidd.installation.id` resource attribute for that execution. It is an installation identifier, not an authenticated person; member association requires a separate authorized mapping. This flag fails if no installation UUID exists. Captures remain provider data and do not contain this added attribute. + ### 2) ๐Ÿงช Send two messages in each session Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each command. Choose models available to your accounts. diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts new file mode 100644 index 000000000..fbc32773c --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.ts @@ -0,0 +1,50 @@ +import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; + +const KEY = "aidd.installation.id"; +const UUID_V4 = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +/** A provider must not override the installation declared by the AIDD runner. */ +export function checkInstallationClaim(value: unknown, id: string): void { + if (!UUID_V4.test(id)) throw new Error("Installation ID is invalid."); + if (Array.isArray(value)) { + for (const item of value) checkInstallationClaim(item, id); + return; + } + const object = asPlainObject(value); + if (!object) return; + if (object.key === KEY && asPlainObject(object.value)?.stringValue !== id) + throw new Error("Conflicting OTLP installation ID."); + for (const item of Object.values(object)) checkInstallationClaim(item, id); +} + +/** Label an already selected transfer; it is an installation claim, not a user identity. */ +export function attachInstallationId( + payload: unknown, + field: "resourceLogs" | "resourceMetrics" | "resourceSpans", + id: string +): Record<string, unknown> { + checkInstallationClaim(payload, id); + const source = asPlainObject(payload); + const resources = source?.[field]; + if (!source || !Array.isArray(resources)) throw new Error("Invalid OTLP resource envelope."); + return { + ...source, + [field]: resources.map((value) => { + const entry = asPlainObject(value); + if (!entry) throw new Error("Invalid OTLP resource."); + const resource = asPlainObject(entry.resource) ?? {}; + const attrs = resource.attributes; + if (attrs !== undefined && !Array.isArray(attrs)) throw new Error("Invalid OTLP attributes."); + return { + ...entry, + resource: { + ...resource, + attributes: [ + ...(attrs ?? []).filter((item) => asPlainObject(item)?.key !== KEY), + { key: KEY, value: { stringValue: id } }, + ], + }, + }; + }), + }; +} diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts index a5271ef9c..92c2749f6 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -17,6 +17,7 @@ import { readToken, } from "../settings.js"; import { startOtelCollector } from "./collect.js"; +import { attachInstallationId, checkInstallationClaim } from "./installation-attribute.js"; import { hasOpenCodePlugin } from "./opencode-plugin-file.js"; import { type OpenCodePluginLog, @@ -34,6 +35,7 @@ type RunOptions = { prompt: string; followUp?: string; includeRepository?: boolean; + installationId?: string; model?: string; retainCapture?: (path: string) => void; signal: AbortSignal; @@ -539,6 +541,8 @@ export async function runOtel(options: RunOptions): Promise<number> { let rootSession: string | undefined; for await (const line of lines) { if (!line.trim()) continue; + if (options.installationId) + checkInstallationClaim(JSON.parse(line), options.installationId); for (const payload of openCodePromptSpans(line)) { const session = promptSession(payload); if (!session || (rootSession && rootSession !== session)) @@ -563,9 +567,14 @@ export async function runOtel(options: RunOptions): Promise<number> { ); if (!related.some((log) => log.session === rootSession)) throw new Error("OpenCode telemetry plugin did not emit any session observations."); - const logsBody = JSON.stringify({ + const logsPayload = { resourceLogs: related.flatMap((log) => list(asPlainObject(log.payload)?.resourceLogs)), - }); + }; + const logsBody = JSON.stringify( + options.installationId + ? attachInstallationId(logsPayload, "resourceLogs", options.installationId) + : logsPayload + ); if (Buffer.byteLength(logsBody, "utf8") > 2 * 1024 * 1024) throw new Error( "OpenCode observations exceed the receiver's 2 MiB limit; nothing was sent." @@ -579,7 +588,11 @@ export async function runOtel(options: RunOptions): Promise<number> { authorization: `Bearer ${options.token}`, "content-type": "application/json", }, - body: JSON.stringify(payload), + body: JSON.stringify( + options.installationId + ? attachInstallationId(payload, "resourceSpans", options.installationId) + : payload + ), }); if (!response.ok) throw new Error(`Gouvernail rejected OpenCode traces: HTTP ${response.status}.`); diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index ff1496b7e..9ab94c772 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -5,6 +5,7 @@ import { setTimeout as delay } from "node:timers/promises"; import { isDeepStrictEqual } from "node:util"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; +import { attachInstallationId, checkInstallationClaim } from "./installation-attribute.js"; import { readAttributeValue } from "./read-attribute-value.js"; import { seriesFingerprint } from "./series-fingerprint.js"; @@ -18,6 +19,7 @@ export interface SyncOptions { readonly follow: boolean; readonly signal?: AbortSignal; readonly intervalMs?: number; + readonly installationId?: string; } type RecordObject = Record<string, unknown>; @@ -396,6 +398,7 @@ export function selectTransfers(line: string, options: SyncOptions): Transfer[] } const envelope = object(parsed); const payload = object(envelope.payload); + if (options.installationId) checkInstallationClaim(payload, options.installationId); const out: Transfer[] = []; const sessionKey = options.config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id"; if (envelope.signal === "logs") { @@ -549,7 +552,17 @@ export function selectTransfers(line: string, options: SyncOptions): Transfer[] } } } - return out; + const installationId = options.installationId; + return installationId + ? out.map((transfer) => ({ + ...transfer, + payload: attachInstallationId( + transfer.payload, + transfer.signal === "logs" ? "resourceLogs" : "resourceMetrics", + installationId + ), + })) + : out; } export function destination(options: SyncOptions, signal: "logs" | "metrics"): URL { diff --git a/cli/src/presentation/commands/telemetry.ts b/cli/src/presentation/commands/telemetry.ts index b0b8e0dc5..e18752c9f 100644 --- a/cli/src/presentation/commands/telemetry.ts +++ b/cli/src/presentation/commands/telemetry.ts @@ -224,6 +224,7 @@ export function registerTelemetryCommand(program: Command): void { "Send a second prompt in the same session (Claude and Codex retain local history)" ) .option("--include-repository", "Include repository metadata for Claude Code") + .option("--include-installation-id", "Send the AIDD installation ID (not a user identity)") .option("--keep-capture", "Retain the private local OTLP capture") .action( async ( @@ -236,6 +237,7 @@ export function registerTelemetryCommand(program: Command): void { model?: string; followUp?: string; includeRepository?: boolean; + includeInstallationId?: boolean; keepCapture?: boolean; } ) => { diff --git a/cli/src/runtime/wiring/telemetry-otel.ts b/cli/src/runtime/wiring/telemetry-otel.ts index b11e86838..bdff8de8f 100644 --- a/cli/src/runtime/wiring/telemetry-otel.ts +++ b/cli/src/runtime/wiring/telemetry-otel.ts @@ -14,8 +14,15 @@ import { } from "../filesystem/installation-identity.js"; import { userConfigDirOf } from "../user-config-dir.js"; -export function runOtelSession(options: Parameters<typeof runOtel>[0]) { - return runOtel(options); +export async function runOtelSession( + options: Parameters<typeof runOtel>[0] & { includeInstallationId?: boolean } +) { + const installationId = options.includeInstallationId + ? await checkInstallationId(userConfigDirOf(process.env, homedir())) + : undefined; + if (options.includeInstallationId && !installationId) + throw new Error("No AIDD installation ID exists. Run framework setup before including it."); + return runOtel({ ...options, installationId: installationId ?? undefined }); } export function collectOtel(port: number) { diff --git a/cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts b/cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts new file mode 100644 index 000000000..6f5e525c7 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/installation-attribute.unit.test.ts @@ -0,0 +1,104 @@ +import { describe, expect, it } from "vitest"; +import { + attachInstallationId, + checkInstallationClaim, +} from "../../../../src/contexts/telemetry/infrastructure/otel/actions/installation-attribute.js"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; + +const id = "6ba7b810-9dad-41d1-80b4-00c04fd430c8"; +const attr = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const options = { + config: "codex" as const, + session: "session", + tenant: "tenant", + endpoint: "http://127.0.0.1:3000", + token: "test", + home: "/tmp", + follow: false, +}; +const codex = (extra: unknown[] = []) => + JSON.stringify({ + signal: "logs", + payload: { + resourceLogs: [ + { + resource: { attributes: [attr("conversation.id", "session"), ...extra] }, + scopeLogs: [ + { logRecords: [{ attributes: [attr("event.name", "codex.conversation_starts")] }] }, + ], + }, + ], + }, + }); + +describe("installation ID opt-in", () => { + it("adds the ID to selected Codex resources only when requested", () => { + expect(JSON.stringify(selectTransfers(codex(), options))).not.toContain("aidd.installation.id"); + const output = selectTransfers(codex(), { ...options, installationId: id }); + expect(output).toHaveLength(1); + expect(JSON.stringify(output)).toContain(JSON.stringify(attr("aidd.installation.id", id))); + }); + it("refuses a conflicting provider claim", () => { + expect(() => + selectTransfers(codex([attr("aidd.installation.id", "wrong")]), { + ...options, + installationId: id, + }) + ).toThrow("Conflicting OTLP installation ID"); + expect(() => + checkInstallationClaim( + { resourceLogs: [{ attributes: [attr("aidd.installation.id", "wrong")] }] }, + id + ) + ).toThrow(); + }); + it.each(["logs", "metrics"] as const)("adds the ID to Claude %s resources", (signal) => { + const resource = { attributes: [attr("session.id", "session")] }; + const payload = + signal === "logs" + ? { + resourceLogs: [ + { + resource, + scopeLogs: [{ logRecords: [{ attributes: [attr("event.name", "tool_result")] }] }], + }, + ], + } + : { + resourceMetrics: [ + { + resource, + scopeMetrics: [ + { + metrics: [ + { + name: "claude_code.token.usage", + sum: { dataPoints: [{ asInt: "7", attributes: [attr("type", "input")] }] }, + }, + ], + }, + ], + }, + ], + }; + const output = selectTransfers(JSON.stringify({ signal, payload }), { + ...options, + config: "claude", + installationId: id, + }); + expect(output).toHaveLength(1); + expect(JSON.stringify(output)).toContain(JSON.stringify(attr("aidd.installation.id", id))); + }); + it.each(["resourceLogs", "resourceMetrics", "resourceSpans"] as const)( + "labels %s without overwriting other attributes", + (field) => { + const result = attachInstallationId( + { [field]: [{ resource: { attributes: [attr("service.name", "opencode")] } }] }, + field, + id + ); + expect(JSON.stringify(result)).toContain(JSON.stringify(attr("service.name", "opencode"))); + expect(JSON.stringify(result)).toContain(JSON.stringify(attr("aidd.installation.id", id))); + } + ); +}); diff --git a/cli/tests/golden/snapshots/help/surface.json b/cli/tests/golden/snapshots/help/surface.json index 7d112125f..4a2ff22d7 100644 --- a/cli/tests/golden/snapshots/help/surface.json +++ b/cli/tests/golden/snapshots/help/surface.json @@ -182,7 +182,7 @@ { "invocation": "aidd telemetry otel run", "exitCode": 0, - "help": "Usage: aidd telemetry otel run [options] <prompt>\n\nRun one fresh provider session and send its allowed OTLP observations\n\nOptions:\n --config <tool> Tool to run (choices: \"claude\", \"codex\", \"opencode\")\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --yes Consent to capture and send telemetry from this\n execution\n --model <model> Provider model to use\n --follow-up <prompt> Send a second prompt in the same session (Claude and\n Codex retain local history)\n --include-repository Include repository metadata for Claude Code\n --keep-capture Retain the private local OTLP capture\n -h, --help display help for command" + "help": "Usage: aidd telemetry otel run [options] <prompt>\n\nRun one fresh provider session and send its allowed OTLP observations\n\nOptions:\n --config <tool> Tool to run (choices: \"claude\", \"codex\",\n \"opencode\")\n --tenant <slug> Gouvernail tenant slug\n --endpoint <origin> Loopback HTTP origin\n --yes Consent to capture and send telemetry from this\n execution\n --model <model> Provider model to use\n --follow-up <prompt> Send a second prompt in the same session (Claude\n and Codex retain local history)\n --include-repository Include repository metadata for Claude Code\n --include-installation-id Send the AIDD installation ID (not a user identity)\n --keep-capture Retain the private local OTLP capture\n -h, --help display help for command" }, { "invocation": "aidd telemetry otel verify", diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 1ab51f95c..9d54974ca 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -47,7 +47,7 @@ The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The | Identifier | Purpose and storage | Sent to Gouvernail? | | --- | --- | --- | | Local report name | Optional label in `~/.config/aidd/telemetry/identity.json`; the current label is applied when reporting, not recorded as proof per call. | No. | -| Installation UUID | Stable ID in `~/.config/aidd/installation/id.json`; not a login or credential. | Not currently. | +| Installation UUID | Stable ID in `~/.config/aidd/installation/id.json`; not a login or credential. | Only with `otel run --include-installation-id`, as `aidd.installation.id`. | | Provider identity | Email, account or organization ID when emitted. Gouvernail must match an approved active member to attribute a session. | Allowlisted identifiers may be sent by `otel run`. | The default paths can change with the configured user directory. A connector bearer token authorizes the destination connection; it does not identify the person. @@ -83,7 +83,7 @@ Codex: trust the plugin's hooks once, with `/hooks` in the TUI. - Data lives in `~/.config/aidd/telemetry` and the clone's git config. - You are named only if you choose to be. - `AIDD_TELEMETRY=0` refuses AIDD usage measurement everywhere. -- Framework setup creates a stable installation UUID at `~/.config/aidd/installation/id.json` (or under the configured user directory), independently of the optional person label. It is not a credential or a login; telemetry setup creates it for older installations. It is retained when telemetry is forgotten. It is not yet attached to usage records or linked to an account. +- Framework setup creates a stable installation UUID at `~/.config/aidd/installation/id.json` (or under the configured user directory), independently of the optional person label. It is not a credential or a login; telemetry setup creates it for older installations. It is retained when telemetry is forgotten. It is not linked to an account or attached to local usage records. The optional `otel run --include-installation-id` adds it to that execution's outgoing OTLP resources; it does not authenticate a person. ## Provider OpenTelemetry From f189cbab97954260d44ab8e40c2e37a78594af1e Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 08:49:50 +0800 Subject: [PATCH 43/56] docs(telemetry): include installation in local examples --- aidd_docs/recipes/observe-agent-telemetry.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index 733b830e7..c6adfaccd 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -20,22 +20,22 @@ Add `--include-installation-id` to send the existing AIDD installation UUID as a ### 2) ๐Ÿงช Send two messages in each session -Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each command. Choose models available to your accounts. +Set `AIDD_OTEL_TOKEN` to the corresponding connector token before each command. Choose models available to your accounts. These examples opt in to sending the installation UUID. ```sh # Claude connector token aidd telemetry otel run --config claude --model haiku --tenant ai-driven-dev \ - --endpoint http://127.0.0.1:55857 --yes --keep-capture \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture --include-installation-id \ --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." # Codex connector token aidd telemetry otel run --config codex --model gpt-6-luna --tenant ai-driven-dev \ - --endpoint http://127.0.0.1:55857 --yes --keep-capture \ + --endpoint http://127.0.0.1:55857 --yes --keep-capture --include-installation-id \ --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." # OpenCode connector token aidd telemetry otel run --config opencode --model claude-code/claude-haiku-4-5-20251001 \ - --tenant ai-driven-dev --endpoint http://127.0.0.1:55857 --yes --keep-capture \ + --tenant ai-driven-dev --endpoint http://127.0.0.1:55857 --yes --keep-capture --include-installation-id \ --follow-up "What word did I ask you to remember?" -- "Remember ORION72. Reply READY." ``` From f6f749dae8cd293e5ec8d260b95fbeccde42e22b Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 09:17:09 +0800 Subject: [PATCH 44/56] feat(telemetry): forward Codex turn traces Preserve session-linked turn observations without forwarding trace content. --- aidd_docs/recipes/observe-agent-telemetry.md | 2 +- .../infrastructure/otel/actions/run.ts | 18 +++ .../infrastructure/otel/actions/sync.ts | 39 ++++- .../infrastructure/otel/turn-traces.ts | 138 ++++++++++++++++++ .../otel/codex-turn-traces.unit.test.ts | 87 +++++++++++ plugins/aidd-telemetry/OBSERVABILITY.md | 6 +- 6 files changed, 282 insertions(+), 8 deletions(-) create mode 100644 cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts create mode 100644 cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index c6adfaccd..664756f43 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -41,7 +41,7 @@ aidd telemetry otel run --config opencode --model claude-code/claude-haiku-4-5-2 Each command uses an isolated collector and resumes the first session for the follow-up. Claude and Codex persist these conversations in their usual local session history; OpenCode uses a temporary database removed after the run. This history is used only by the tools to resume; Gouvernail receives OTLP, not the tools' session files. Verify both responses when checking continuity: a shared session ID alone does not prove the model used the first message. -The command waits for an OTLP usage observation after each execution. Claude identifies the matching prompt explicitly; OpenCode supplies a completed step and message ID. Codex supplies session-linked usage but no reliable per-prompt join; observations can include additional requests, so do not attribute their totals to a specific prompt. An emitted observation does not guarantee that every requested field is available. `otel init` and the global development collector are a separate configuration path. +The command waits for an OTLP usage observation after each execution. Claude identifies the matching prompt explicitly; OpenCode supplies a completed step and message ID. Codex turn spans observed with CLI 0.166.0 carry session, turn and usage together. Turn usage can cover multiple requests or models; zero cache and output values can be provider defaults. Do not add the same turn's span and response-event counters. The run sends logs before spans and excludes spans from a different session without inferring a parent relationship. An emitted observation does not guarantee that every requested field is available. `otel init` and the global development collector are a separate configuration path; generic `otel sync` does not yet forward Codex turn spans. ### 3) โœ… Check receipt diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts index 92c2749f6..601cb5c62 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/run.ts @@ -640,6 +640,24 @@ export async function runOtel(options: RunOptions): Promise<number> { async () => !options.signal.aborted ); sent += result.sent; + if (options.config === CODEX_CONFIG_ID) { + if (result.sent === 0) + throw new Error("Codex session logs were not retained; turn traces were not sent."); + try { + const traces = await syncOtel( + { ...base, session, home, traceOnly: true }, + async () => { + if (options.signal.aborted) throw new Error("OTLP execution was cancelled."); + }, + async () => !options.signal.aborted + ); + sent += traces.sent; + } catch (error) { + const message = + error instanceof Error ? error.message : "Codex traces could not be sent."; + throw new Error(`${message} Logs may already have been retained.`); + } + } } } return sent; diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index 9ab94c772..eb9929bd3 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -5,6 +5,7 @@ import { setTimeout as delay } from "node:timers/promises"; import { isDeepStrictEqual } from "node:util"; import { asPlainObject } from "../../../../../kernel/reading/plain-object.js"; import { CLAUDE_CONFIG_ID, CODEX_CONFIG_ID, type SyncConfig } from "../settings.js"; +import { codexTurnTransfers } from "../turn-traces.js"; import { attachInstallationId, checkInstallationClaim } from "./installation-attribute.js"; import { readAttributeValue } from "./read-attribute-value.js"; import { seriesFingerprint } from "./series-fingerprint.js"; @@ -20,6 +21,8 @@ export interface SyncOptions { readonly signal?: AbortSignal; readonly intervalMs?: number; readonly installationId?: string; + /** Used by isolated Codex runs after logs have been delivered. */ + readonly traceOnly?: boolean; } type RecordObject = Record<string, unknown>; @@ -276,7 +279,7 @@ function timeOf(entry: RecordObject) { } export interface Transfer { - readonly signal: "logs" | "metrics"; + readonly signal: "logs" | "metrics" | "traces"; readonly payload: RecordObject; readonly at?: string; readonly start?: string; @@ -292,7 +295,12 @@ export function packTransfers( const groups = new Map<string, Array<{ transfer: Transfer; count: number; items: Transfer[] }>>(); for (const item of transfers) { const key = groupByTimestamp ? `${item.signal}\0${item.at ?? ""}` : item.signal; - const root = item.signal === "logs" ? "resourceLogs" : "resourceMetrics"; + const root = + item.signal === "logs" + ? "resourceLogs" + : item.signal === "metrics" + ? "resourceMetrics" + : "resourceSpans"; const resources = array(item.payload[root]); const batches = groups.get(key) ?? []; const last = batches.at(-1); @@ -399,6 +407,17 @@ export function selectTransfers(line: string, options: SyncOptions): Transfer[] const envelope = object(parsed); const payload = object(envelope.payload); if (options.installationId) checkInstallationClaim(payload, options.installationId); + if (options.traceOnly) { + if (options.config !== CODEX_CONFIG_ID || envelope.signal !== "traces") return []; + const transfers = codexTurnTransfers(payload, options.session); + const installationId = options.installationId; + return installationId + ? transfers.map((transfer) => ({ + ...transfer, + payload: attachInstallationId(transfer.payload, "resourceSpans", installationId), + })) + : transfers; + } const out: Transfer[] = []; const sessionKey = options.config === CLAUDE_CONFIG_ID ? "session.id" : "conversation.id"; if (envelope.signal === "logs") { @@ -565,7 +584,7 @@ export function selectTransfers(line: string, options: SyncOptions): Transfer[] : out; } -export function destination(options: SyncOptions, signal: "logs" | "metrics"): URL { +export function destination(options: SyncOptions, signal: "logs" | "metrics" | "traces"): URL { let endpoint: URL; try { endpoint = new URL(options.endpoint); @@ -618,8 +637,18 @@ async function assertFullDelivery(response: Response, signal: Transfer["signal"] if (raw === undefined || raw === null) return; const partial = asPlainObject(raw); if (!partial) throw new Error("Gouvernail returned an invalid OTLP response."); - const camel = signal === "logs" ? "rejectedLogRecords" : "rejectedDataPoints"; - const snake = signal === "logs" ? "rejected_log_records" : "rejected_data_points"; + const camel = + signal === "logs" + ? "rejectedLogRecords" + : signal === "metrics" + ? "rejectedDataPoints" + : "rejectedSpans"; + const snake = + signal === "logs" + ? "rejected_log_records" + : signal === "metrics" + ? "rejected_data_points" + : "rejected_spans"; if ( Object.keys(partial).some( (key) => ![camel, snake, "errorMessage", "error_message"].includes(key) diff --git a/cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts b/cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts new file mode 100644 index 000000000..47fa40b17 --- /dev/null +++ b/cli/src/contexts/telemetry/infrastructure/otel/turn-traces.ts @@ -0,0 +1,138 @@ +import { asPlainObject } from "../../../../kernel/reading/plain-object.js"; +import type { Transfer } from "./actions/sync.js"; + +const UUID = /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/i; +const TEXT_KEYS = new Set(["thread.id", "turn.id", "model", "codex.turn.reasoning_effort"]); +const COUNTER_KEYS = new Set([ + "codex.turn.token_usage.input_tokens", + "codex.turn.token_usage.cached_input_tokens", + "codex.turn.token_usage.cache_write_input_tokens", + "codex.turn.token_usage.non_cached_input_tokens", + "codex.turn.token_usage.output_tokens", + "codex.turn.token_usage.reasoning_output_tokens", + "codex.turn.token_usage.total_tokens", +]); +const obj = (value: unknown) => asPlainObject(value) ?? {}; +const list = (value: unknown) => (Array.isArray(value) ? value : []); + +function safeAttributes( + value: unknown +): Map<string, { stringValue: string } | { intValue: string | number }> { + const result = new Map<string, { stringValue: string } | { intValue: string | number }>(); + for (const raw of list(value)) { + const entry = obj(raw); + const key = entry.key; + if (typeof key !== "string" || (!TEXT_KEYS.has(key) && !COUNTER_KEYS.has(key))) continue; + const content = obj(entry.value); + // tracing adds a runtime integer thread.id alongside Codex's string UUID thread.id. + if (key === "thread.id" && content.intValue !== undefined && content.stringValue === undefined) + continue; + let safe: { stringValue: string } | { intValue: string | number }; + if (TEXT_KEYS.has(key)) { + if (typeof content.stringValue !== "string" || Object.keys(content).length !== 1) + throw new Error("Invalid Codex turn attribute."); + const text = content.stringValue; + if ( + key === "thread.id" || key === "turn.id" + ? !UUID.test(text) + : !/^[A-Za-z0-9][A-Za-z0-9._:/#-]{0,127}$/.test(text) + ) + throw new Error("Invalid Codex turn attribute."); + safe = { stringValue: text }; + } else { + const number = content.intValue; + if ( + Object.keys(content).length !== 1 || + !( + (typeof number === "string" && + /^\d+$/.test(number) && + Number.isSafeInteger(Number(number))) || + (typeof number === "number" && Number.isSafeInteger(number) && number >= 0) + ) + ) + throw new Error("Invalid Codex turn counter."); + safe = { intValue: number as string | number }; + } + const previous = result.get(key); + if (previous && JSON.stringify(previous) !== JSON.stringify(safe)) + throw new Error("Conflicting Codex turn attribute."); + result.set(key, safe); + } + return result; +} + +function time(value: unknown): string | undefined { + if (typeof value !== "string" || !/^\d{1,20}$/.test(value)) return undefined; + const nanos = BigInt(value); + if (nanos <= 0n || nanos > 18446744073709551615n) return undefined; + try { + return new Date(Number(nanos / 1_000_000n)).toISOString(); + } catch { + return undefined; + } +} + +/** Only provider turn spans are exported: no events, links, paths or prompt content. */ +export function codexTurnTransfers(payload: unknown, session: string): Transfer[] { + const transfers: Transfer[] = []; + for (const rawResource of list(obj(payload).resourceSpans)) { + for (const rawScope of list(obj(rawResource).scopeSpans)) { + for (const rawSpan of list(obj(rawScope).spans)) { + const span = obj(rawSpan); + if (span.name !== "session_task.turn") continue; + const identity = safeAttributes( + list(span.attributes).filter((item) => obj(item).key === "thread.id") + ); + const thread = obj(identity.get("thread.id")).stringValue; + if (typeof thread !== "string") throw new Error("Codex turn session ID is missing."); + if (thread.toLowerCase() !== session.toLowerCase()) continue; + const attrs = safeAttributes(span.attributes); + if (!attrs.has("turn.id")) throw new Error("Codex turn ID is missing."); + if ( + typeof span.traceId !== "string" || + !/^[a-f0-9]{32}$/i.test(span.traceId) || + /^0+$/.test(span.traceId) || + typeof span.spanId !== "string" || + !/^[a-f0-9]{16}$/i.test(span.spanId) || + /^0+$/.test(span.spanId) + ) + throw new Error("Invalid Codex trace identifier."); + const start = time(span.startTimeUnixNano); + const at = time(span.endTimeUnixNano); + if ( + !start || + !at || + BigInt(String(span.startTimeUnixNano)) > BigInt(String(span.endTimeUnixNano)) + ) + throw new Error("Invalid Codex turn interval."); + const safe = [...attrs].map(([key, value]) => ({ key, value })); + transfers.push({ + signal: "traces", + start, + at, + payload: { + resourceSpans: [ + { + scopeSpans: [ + { + spans: [ + { + name: "session_task.turn", + traceId: span.traceId, + spanId: span.spanId, + startTimeUnixNano: span.startTimeUnixNano, + endTimeUnixNano: span.endTimeUnixNano, + attributes: safe, + }, + ], + }, + ], + }, + ], + }, + }); + } + } + } + return transfers; +} diff --git a/cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts b/cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts new file mode 100644 index 000000000..bae9a4456 --- /dev/null +++ b/cli/tests/contexts/telemetry/otel/codex-turn-traces.unit.test.ts @@ -0,0 +1,87 @@ +import { describe, expect, it } from "vitest"; +import { selectTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/actions/sync.js"; +import { codexTurnTransfers } from "../../../../src/contexts/telemetry/infrastructure/otel/turn-traces.js"; + +const session = "00000000-0000-4000-8000-000000000001"; +const turn = "00000000-0000-4000-8000-000000000002"; +const attr = (key: string, value: Record<string, string>) => ({ key, value }); +function span() { + return { + name: "session_task.turn", + traceId: "1234567890abcdef1234567890abcdef", + spanId: "1234567890abcdef", + startTimeUnixNano: "1794000000000000000", + endTimeUnixNano: "1794000001000000000", + events: [{ name: "private prompt" }], + attributes: [ + attr("thread.id", { intValue: "7" }), + attr("thread.id", { stringValue: session }), + attr("turn.id", { stringValue: turn }), + attr("model", { stringValue: "gpt-6-luna" }), + attr("codex.turn.token_usage.input_tokens", { intValue: "42" }), + attr("code.file.path", { stringValue: "/private/path" }), + ], + }; +} +function payload(spans: unknown[]) { + return { + resourceSpans: [ + { + resource: { attributes: [attr("secret", { stringValue: "private" })] }, + scopeSpans: [{ spans }], + }, + ], + }; +} + +describe("Codex turn trace filtering", () => { + it("keeps a session's typed turn data and omits other trace content", () => { + const out = codexTurnTransfers(payload([span()]), session); + expect(out).toHaveLength(1); + expect(JSON.stringify(out)).not.toMatch(/private|code.file.path|events/); + const selected = out[0].payload.resourceSpans as Array<{ + scopeSpans: Array<{ spans: Array<{ attributes: Array<{ key: string; value: unknown }> }> }>; + }>; + const attrs = selected[0].scopeSpans[0].spans[0].attributes; + expect(attrs.filter((item) => item.key === "thread.id")).toEqual([ + attr("thread.id", { stringValue: session }), + ]); + expect(attrs).toContainEqual(attr("codex.turn.token_usage.input_tokens", { intValue: "42" })); + expect(codexTurnTransfers(payload([span()]), turn)).toEqual([]); + }); + it("rejects a missing or invalid session ID", () => { + const missing = span(); + missing.attributes = missing.attributes.filter((item) => item.key !== "thread.id"); + expect(() => codexTurnTransfers(payload([missing]), session)).toThrow("missing"); + const invalid = span(); + invalid.attributes = invalid.attributes.filter((item) => item.key !== "thread.id"); + invalid.attributes.push(attr("thread.id", { stringValue: "invalid" })); + expect(() => codexTurnTransfers(payload([invalid]), session)).toThrow("Invalid"); + }); + it("rejects conflicting session IDs", () => { + const item = span(); + item.attributes.push(attr("thread.id", { stringValue: turn })); + expect(() => codexTurnTransfers(payload([item]), session)).toThrow("Conflicting"); + }); + it("only selects traces in the second isolated-run pass", () => { + const line = JSON.stringify({ signal: "traces", payload: payload([span()]) }); + const options = { + config: "codex" as const, + session, + tenant: "test", + endpoint: "http://127.0.0.1:1", + token: "token", + home: "/tmp", + follow: false, + }; + expect(selectTransfers(line, options)).toEqual([]); + expect(selectTransfers(line, { ...options, traceOnly: true })).toHaveLength(1); + expect( + selectTransfers(line, { + ...options, + traceOnly: true, + installationId: "00000000-0000-4000-8000-000000000003", + })[0].payload.resourceSpans + ).toBeDefined(); + }); +}); diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 9dad3fd24..a5932676a 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -4,7 +4,7 @@ Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2 | Field | Claude Code OTel | Codex OTel | OpenCode v2 OTel | | --- | --- | --- | --- | -| Session | `session.id`, when present | `conversation.id`, when present | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | +| Session | `session.id`, when present | `conversation.id` in logs; `thread.id` in observed turn spans | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | | Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | | Usage | `claude_code.token.usage` and `api_request` | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | | Identity | Provider attributes, when emitted | Provider attributes, when emitted | Member explicitly assigned to the connector token in local Gouvernail; not a provider identity or proof of the person holding the token | @@ -22,7 +22,7 @@ The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.sta | Provider | Available or observed in OTLP | Current limit and next step | | --- | --- | --- | | Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort, latency and separate token and cache counters. These fields appeared in both local captures. | The framework does not forward these events today. An allowlist of technical fields could expose them; avoid counting the same request again through metrics. Repository attributes require a separate opt-in. | -| Codex | Logs, metrics and traces are independently configurable. Completed response events can include model, effort and usage. | The observed turn-usage histogram had no session ID and the response events had no turn or request ID. Neither timestamps nor a previous session's effort establish a safe join. Forward additional documented, session-linked events only after validating their fields. | +| Codex | In two local turns with CLI 0.166.0, a `session_task.turn` span contained `thread.id`, `turn.id`, selected model, effort and turn usage together. | The span counts the session usage delta during the turn and may cover several requests or models. The model is selected, not confirmed for every request. Cache and output zeros can be normalized defaults. A turn ID can recur during recovery; retain trace and span IDs. `otel run` sends allowlisted turn spans after session logs; generic `otel sync` does not yet forward these spans. Never add those counters to response-event counters. | | OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | The opted-in plugin sends session-linked OpenCode events as OTLP. Missing counters are normalized to zero upstream, so zeros cannot establish actual usage. A model variant is not necessarily reasoning effort. | An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. @@ -41,3 +41,5 @@ Read 2026-11-06: - [OpenCode v2 HTTP API audit](https://github.com/anomalyco/opencode/blob/dev/V2_HTTP_API_AUDIT.md) (current development branch; the v2.2.5 path was observed locally) - [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) - [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) + +Read 2026-11-07: [Codex 0.166.0 turn span and counters](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L369-L382), [turn counter capture](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L760-L810) and [cache defaulting](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/codex-api/src/sse/responses.rs#L131-L153). From 691e013423b1935fe0dd66767513ddc40316d6b5 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 10:10:40 +0800 Subject: [PATCH 45/56] docs(telemetry): explain assigned local tokens Describe installation binding and legacy attribution for each tool. --- aidd_docs/recipes/observe-agent-telemetry.md | 4 ++-- plugins/aidd-telemetry/README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index 664756f43..b5a5963d8 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -6,7 +6,7 @@ Run Claude Code, Codex and OpenCode v2 with OpenTelemetry (OTel), then inspect t ### 1) โš™๏ธ Prepare the collector destination -Start local Gouvernail and configure each tool's connector. Generate OpenCode's token in Sources for an explicit team and member; issuing a new token replaces the previous OpenCode token for that organization. The member is the token's declared assignee, not proof of the person using it. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. +Start local Gouvernail and configure each tool's connector. In Sources, generate a token for an explicit team, member and installation UUID. Rotating a token replaces the previous token for that tool and organization. The member is the token's declared assignee, not proof of the person using it. Existing shared tokens retain their previous declarative attribution. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. | Tool | Connector token | | --- | --- | @@ -16,7 +16,7 @@ Start local Gouvernail and configure each tool's connector. Generate OpenCode's `--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. -Add `--include-installation-id` to send the existing AIDD installation UUID as an AIDD-added `aidd.installation.id` resource attribute for that execution. It is an installation identifier, not an authenticated person; member association requires a separate authorized mapping. This flag fails if no installation UUID exists. Captures remain provider data and do not contain this added attribute. +The new assigned tokens require `--include-installation-id`, which sends the existing AIDD installation UUID as an AIDD-added `aidd.installation.id` resource attribute for that execution. It is an installation identifier, not an authenticated person. This flag fails if no installation UUID exists. Captures remain provider data and do not contain this added attribute; the global `otel init` configuration does not add it. ### 2) ๐Ÿงช Send two messages in each session diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 9d54974ca..395c9f87e 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -31,7 +31,7 @@ The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The | Mode | Opt-in and destination | Identity | What is kept or sent | | --- | --- | --- | --- | | Local usage | `aidd telemetry on` explicitly enables measurement for the current clone from that time. The ledger and report stay on this machine. | An optional name labels the local report; anonymous use is supported. | Per-call usage read from Claude Code and Codex session files: session, time, model, agent, token counts and local attribution context. The ledger does not store prompts or responses. | -| OTLP development | `otel run --yes` explicitly opts in one isolated execution and sends supported observations to loopback Gouvernail. | Provider identifiers may be sent when present; attribution requires a matched active member. For OpenCode, a token can be explicitly assigned to a member in local Gouvernail; this does not prove who holds it. | With `--keep-capture`, the private capture can contain prompts, responses, commands and tool output. Forwarding filters fields but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | +| OTLP development | `otel run --yes` explicitly opts in one isolated execution and sends supported observations to loopback Gouvernail. | A token can be explicitly assigned to a team, member and installation for each tool in local Gouvernail. It does not prove who holds it. Older shared tokens retain their declarative attribution. | With `--keep-capture`, the private capture can contain prompts, responses, commands and tool output. Forwarding filters fields but does not inspect allowed attribute values for secrets. See the [field inventory](OBSERVABILITY.md). | | OTLP production | **Not available:** `--endpoint=prod` is refused. `https://gouvernail.eu` is an intended future destination, not an active exporter. | **Required before production is enabled:** an authenticated user identity. This is a product requirement, not yet implemented or enforced. | No production payload is sent by this feature today. Consent, authentication, retention and deletion behavior must be implemented before enabling it. | ### Consent and scope From b354b1786ab0bee079e8de25c43758c15dca8f9d Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 10:34:41 +0800 Subject: [PATCH 46/56] docs(telemetry): clarify forwarded usage and identity Keep the field inventory aligned with the local OTLP forwarding and token attribution behavior. --- plugins/aidd-telemetry/OBSERVABILITY.md | 6 ++++-- plugins/aidd-telemetry/README.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index a5932676a..cd81a3f5d 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -6,11 +6,13 @@ Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2 | --- | --- | --- | --- | | Session | `session.id`, when present | `conversation.id` in logs; `thread.id` in observed turn spans | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | | Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | -| Usage | `claude_code.token.usage` and `api_request` | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | -| Identity | Provider attributes, when emitted | Provider attributes, when emitted | Member explicitly assigned to the connector token in local Gouvernail; not a provider identity or proof of the person holding the token | +| Usage | Forwarded `claude_code.token.usage` metrics; `api_request` is captured but not forwarded | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | +| Identity | Provider attributes when emitted, or an explicit connector-token assignment | Provider attributes when emitted, or an explicit connector-token assignment | An explicit connector-token assignment | Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. +In local Gouvernail, an assigned connector token binds a tool to a declared member, team and installation UUID. This is not proof of the person holding the token. Older shared tokens retain declarative attribution. + `otel run --yes` captures one provider execution in isolation and forwards supported observations and allowlisted attributes to loopback Gouvernail. It neither reads the local usage ledger nor asserts a clone or feature association. With `--keep-capture`, raw private files can contain sensitive content. See [the local setup recipe](../../aidd_docs/recipes/observe-agent-telemetry.md). In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTLP contained 1,335 spans, 276 logs and no metric batches. A prompt span in each execution included a session ID in its HTTP path; no session-linked model, effort or token attributes were observed. Five other HTTP spans per execution contained that session ID but were not forwarded. The prompt spans had no trace-linked descendants in these captures; the model and usage spans belonged to a separate server trace without a session ID. A startup log included the requested model, but no session ID or proof of the model actually used. Those runs forwarded only the validated prompt span. These observations do not prove what other versions or configurations emit. diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 395c9f87e..219e8e2bf 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -50,7 +50,7 @@ The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The | Installation UUID | Stable ID in `~/.config/aidd/installation/id.json`; not a login or credential. | Only with `otel run --include-installation-id`, as `aidd.installation.id`. | | Provider identity | Email, account or organization ID when emitted. Gouvernail must match an approved active member to attribute a session. | Allowlisted identifiers may be sent by `otel run`. | -The default paths can change with the configured user directory. A connector bearer token authorizes the destination connection; it does not identify the person. +The default paths can change with the configured user directory. A connector bearer token authorizes the destination connection. In local Gouvernail, an assigned token also identifies its declared member, team and installation; it does not prove who holds the token. ## Start From e75b9cc55f733cca0b7b9accb9241120d4a387d8 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 10:36:55 +0800 Subject: [PATCH 47/56] docs(telemetry): record Claude metric interval gap Preserve the evidence needed to improve request-level attribution without guessing. --- plugins/aidd-telemetry/OBSERVABILITY.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index cd81a3f5d..b1af46d0b 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -29,6 +29,8 @@ The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.sta An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. +In one two-prompt Claude capture, the second request's input and cache-read metric series started one millisecond before the cache-creation series, although the captured `api_request` reported all three counts together. The dashboard cannot safely join metric windows by temporal proximity, so their combined input remains unavailable. Forwarding request events with their request and prompt IDs could provide request-level evidence, but must include deduplication and must not add those counts to the metrics. Event delivery can still be incomplete. [Claude event correlation](https://code.claude.com/docs/en/monitoring-usage#event-correlation-attributes) (read 2026-11-07). + ## Sources Read 2026-11-06: From ea429ac1dbb408bd20cedc8d45c3fe2eeecff397 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 10:48:04 +0800 Subject: [PATCH 48/56] feat(telemetry): forward Claude request evidence Allow request-level correlation and usage reconciliation without forwarding message contents. --- .../infrastructure/otel/actions/sync.ts | 40 ++++++++++++++++++- plugins/aidd-telemetry/OBSERVABILITY.md | 6 +-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts index eb9929bd3..55a763320 100644 --- a/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts +++ b/cli/src/contexts/telemetry/infrastructure/otel/actions/sync.ts @@ -173,6 +173,31 @@ function safeAttributes(values: Map<string, string>) { })); } +// Request fields are allowed only on Claude API request events, never on other logs. +function claudeRequestAttributes(raw: unknown) { + const result: { key: string; value: { stringValue: string } }[] = []; + const entries = array(raw).map(object); + for (const key of ["request_id", "prompt.id", "effort", "model"]) { + const value = object(entries.find((entry) => entry.key === key)?.value).stringValue; + if (typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9_.:-]{0,199}$/.test(value)) + result.push({ key, value: { stringValue: value } }); + } + for (const key of [ + "input_tokens", + "output_tokens", + "cache_read_tokens", + "cache_creation_tokens", + ]) { + const rawValue = object(entries.find((entry) => entry.key === key)?.value); + const rawNumber = rawValue.intValue ?? rawValue.stringValue; + if (typeof rawNumber !== "number" && typeof rawNumber !== "string") continue; + const value = String(rawNumber); + if (/^(0|[1-9]\d*)$/.test(value) && Number.isSafeInteger(Number(value))) + result.push({ key, value: { stringValue: value } }); + } + return result; +} + function isSafeRepositoryUrl(value: string): boolean { return ( /^https:\/\/(?:github\.com|gitlab\.com)\/[A-Za-z0-9_.-]+(?:\/[A-Za-z0-9_.-]+)+$/.test(value) && @@ -433,7 +458,12 @@ export function selectTransfers(line: string, options: SyncOptions): Transfer[] const event = a.get("event.name") ?? eventNameFromBody(record.body); const wanted = options.config === CLAUDE_CONFIG_ID - ? event === "tool_result" || event === "claude_code.tool_result" + ? [ + "tool_result", + "claude_code.tool_result", + "api_request", + "claude_code.api_request", + ].includes(event) : CODEX_EVENTS.has(event) && (event !== "codex.sse_event" || a.get("event.kind") === "response.completed"); if (!wanted) continue; @@ -451,6 +481,14 @@ export function selectTransfers(line: string, options: SyncOptions): Transfer[] else a.delete("event.timestamp"); } const safe = safeAttributes(a); + if ( + options.config === CLAUDE_CONFIG_ID && + (event === "api_request" || event === "claude_code.api_request") + ) { + const model = safe.findIndex((attribute) => attribute.key === "model"); + if (model !== -1) safe.splice(model, 1); + safe.push(...claudeRequestAttributes(record.attributes)); + } if (options.config === CODEX_CONFIG_ID && event !== "codex.tool_result") { const key = event === "codex.conversation_starts" ? "reasoning_effort" : "model_reasoning_effort"; diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index b1af46d0b..95e52149b 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -6,7 +6,7 @@ Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2 | --- | --- | --- | --- | | Session | `session.id`, when present | `conversation.id` in logs; `thread.id` in observed turn spans | In observed `/api/session/{id}/prompt` HTTP spans and plugin session events | | Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | -| Usage | Forwarded `claude_code.token.usage` metrics; `api_request` is captured but not forwarded | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | +| Usage | Forwarded `claude_code.token.usage` metrics and technical fields from `api_request` events | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | | Identity | Provider attributes when emitted, or an explicit connector-token assignment | Provider attributes when emitted, or an explicit connector-token assignment | An explicit connector-token assignment | Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. @@ -23,13 +23,13 @@ The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.sta | Provider | Available or observed in OTLP | Current limit and next step | | --- | --- | --- | -| Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort, latency and separate token and cache counters. These fields appeared in both local captures. | The framework does not forward these events today. An allowlist of technical fields could expose them; avoid counting the same request again through metrics. Repository attributes require a separate opt-in. | +| Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort and separate token and cache counters. These fields appeared in both local captures and are forwarded when valid. | The receiver compares request and metric counts; it does not add the two signals. Missing or conflicting observations remain unavailable. Repository attributes require a separate opt-in. | | Codex | In two local turns with CLI 0.166.0, a `session_task.turn` span contained `thread.id`, `turn.id`, selected model, effort and turn usage together. | The span counts the session usage delta during the turn and may cover several requests or models. The model is selected, not confirmed for every request. Cache and output zeros can be normalized defaults. A turn ID can recur during recovery; retain trace and span IDs. `otel run` sends allowlisted turn spans after session logs; generic `otel sync` does not yet forward these spans. Never add those counters to response-event counters. | | OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | The opted-in plugin sends session-linked OpenCode events as OTLP. Missing counters are normalized to zero upstream, so zeros cannot establish actual usage. A model variant is not necessarily reasoning effort. | An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. -In one two-prompt Claude capture, the second request's input and cache-read metric series started one millisecond before the cache-creation series, although the captured `api_request` reported all three counts together. The dashboard cannot safely join metric windows by temporal proximity, so their combined input remains unavailable. Forwarding request events with their request and prompt IDs could provide request-level evidence, but must include deduplication and must not add those counts to the metrics. Event delivery can still be incomplete. [Claude event correlation](https://code.claude.com/docs/en/monitoring-usage#event-correlation-attributes) (read 2026-11-07). +In one two-prompt Claude capture, the second request's input and cache-read metric series started one millisecond before the cache-creation series, although `api_request` reported all three counts together. The dashboard does not join metric windows by temporal proximity. It can display the observed input including cache when the deduplicated request counts agree with each corresponding metric count. Event delivery can still be incomplete. [Claude event correlation](https://code.claude.com/docs/en/monitoring-usage#event-correlation-attributes) (read 2026-11-07). ## Sources From 34551476c0f2af6dce8cd0728ec73b13e8d99325 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 11:08:10 +0800 Subject: [PATCH 49/56] docs(telemetry): document administrator token setup --- aidd_docs/recipes/observe-agent-telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index b5a5963d8..eb5d3a591 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -6,7 +6,7 @@ Run Claude Code, Codex and OpenCode v2 with OpenTelemetry (OTel), then inspect t ### 1) โš™๏ธ Prepare the collector destination -Start local Gouvernail and configure each tool's connector. In Sources, generate a token for an explicit team, member and installation UUID. Rotating a token replaces the previous token for that tool and organization. The member is the token's declared assignee, not proof of the person using it. Existing shared tokens retain their previous declarative attribution. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. +Start local Gouvernail and configure each tool's connector. A passkey-verified administrator generates tokens in Sources for an explicit team, member and installation UUID. Rotating a token replaces the previous token for that tool and organization. The member is the token's declared assignee, not proof of the person using it. Existing shared tokens retain their previous declarative attribution. For each tool, set `AIDD_OTEL_TOKEN` to its connector token in your shell without publishing it. The examples use the local tenant `ai-driven-dev` and loopback port `55857`; adjust them for your setup. | Tool | Connector token | | --- | --- | From 022c3b0207854f9ef513b6a66a4b3bb4cdc44db2 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 11:12:06 +0800 Subject: [PATCH 50/56] docs(telemetry): clarify temporary capture retention Set accurate expectations for reproducing local observations. --- aidd_docs/recipes/observe-agent-telemetry.md | 2 +- plugins/aidd-telemetry/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/aidd_docs/recipes/observe-agent-telemetry.md b/aidd_docs/recipes/observe-agent-telemetry.md index eb5d3a591..3f8c7464f 100644 --- a/aidd_docs/recipes/observe-agent-telemetry.md +++ b/aidd_docs/recipes/observe-agent-telemetry.md @@ -14,7 +14,7 @@ Start local Gouvernail and configure each tool's connector. A passkey-verified a | Codex | Codex OTLP connector | | OpenCode v2 | OpenCode OTLP connector for the selected team | -`--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. +`--yes` opts in to capturing and forwarding telemetry from that one execution. `--keep-capture` retains a private local file whose path is printed by the command, including when an execution fails after capture. The file remains in the system temporary directory and can be removed by system cleanup; copy it to a protected location if it must be retained. Raw captures can contain prompts, responses, commands, tool output and secrets: restrict access and delete them when no longer needed. Without this option the temporary capture is removed. The new assigned tokens require `--include-installation-id`, which sends the existing AIDD installation UUID as an AIDD-added `aidd.installation.id` resource attribute for that execution. It is an installation identifier, not an authenticated person. This flag fails if no installation UUID exists. Captures remain provider data and do not contain this added attribute; the global `otel init` configuration does not add it. diff --git a/plugins/aidd-telemetry/README.md b/plugins/aidd-telemetry/README.md index 219e8e2bf..79742e476 100644 --- a/plugins/aidd-telemetry/README.md +++ b/plugins/aidd-telemetry/README.md @@ -39,7 +39,7 @@ The Gouvernail AI KPI view uses provider OpenTelemetry (OTel) signals only. The - `on` opts in the separate local usage ledger for one clone. `otel run --yes` opts in the single provider execution independently; it does not attribute observations to a clone or feature. - `otel init` separately configures a user-level exporter. Its global collector can capture other projects and does not check clone consent. It is not required for `otel run`. - `aidd telemetry off` and `forget` affect local usage measurement; they do not disable provider exporters or delete retained OTLP captures or data already received by Gouvernail. -- Retained captures are private local files without automatic deletion. They can contain sensitive content. Claude metric forwarding also sends a deterministic series fingerprint, which can expose equality or allow guesses of low-entropy values. +- Retained captures are private local files in the system temporary directory. The CLI does not delete them, but system cleanup can. They can contain sensitive content. Claude metric forwarding also sends a deterministic series fingerprint, which can expose equality or allow guesses of low-entropy values. - Delivery after a timeout cannot guarantee exactly-once behavior without receiver-side idempotence. ### Three different identifiers From cc02e5ef7bae894d88c9b1a915dacebd0c15be82 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 11:14:48 +0800 Subject: [PATCH 51/56] docs(telemetry): clarify OpenCode Claude hook limits --- plugins/aidd-telemetry/OBSERVABILITY.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 95e52149b..0b7b17bcf 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -19,6 +19,8 @@ In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTL The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.started`, `session.step.ended` and `session.step.failed`, and exports each event separately as OTLP logs. It includes the original event and session IDs, message ID when present, selected model and variant at step start, and positive OpenCode-normalized counters at step end. A creation event can also carry an OpenCode project ID and parent session ID; neither is a repository name. It does not export prompts, responses, tool contents or cost. The plugin stream can miss events; event IDs allow deduplication of observations received, not proof of complete delivery. Do not match retries solely by assistant message ID. Production export remains disabled. +For the locally tested `claude-code` provider, OpenCode v2.2.5 documents that its HTTP request and response hooks do not run because Claude Code makes the API calls. Those hooks therefore cannot recover an ambiguous cache zero for this provider. [OpenCode Claude Agent limitations](https://github.com/anomalyco/opencode/blob/v2.2.5/specs/v2/claude-agent-provider.md#limitations) (read 2026-11-07). + ## Additional coverage investigated | Provider | Available or observed in OTLP | Current limit and next step | From 415c7f34f95ac5dcb32cb948919b26e2fddbce03 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 11:16:16 +0800 Subject: [PATCH 52/56] docs(telemetry): remove unconfirmed provider assumption --- plugins/aidd-telemetry/OBSERVABILITY.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 0b7b17bcf..95e52149b 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -19,8 +19,6 @@ In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTL The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.started`, `session.step.ended` and `session.step.failed`, and exports each event separately as OTLP logs. It includes the original event and session IDs, message ID when present, selected model and variant at step start, and positive OpenCode-normalized counters at step end. A creation event can also carry an OpenCode project ID and parent session ID; neither is a repository name. It does not export prompts, responses, tool contents or cost. The plugin stream can miss events; event IDs allow deduplication of observations received, not proof of complete delivery. Do not match retries solely by assistant message ID. Production export remains disabled. -For the locally tested `claude-code` provider, OpenCode v2.2.5 documents that its HTTP request and response hooks do not run because Claude Code makes the API calls. Those hooks therefore cannot recover an ambiguous cache zero for this provider. [OpenCode Claude Agent limitations](https://github.com/anomalyco/opencode/blob/v2.2.5/specs/v2/claude-agent-provider.md#limitations) (read 2026-11-07). - ## Additional coverage investigated | Provider | Available or observed in OTLP | Current limit and next step | From 96c6ffa4265b89573734c3cc5b486961836de842 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 11:18:40 +0800 Subject: [PATCH 53/56] docs(telemetry): explain proxy cache ambiguity --- plugins/aidd-telemetry/OBSERVABILITY.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index 95e52149b..eef88b641 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -19,6 +19,8 @@ In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTL The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.started`, `session.step.ended` and `session.step.failed`, and exports each event separately as OTLP logs. It includes the original event and session IDs, message ID when present, selected model and variant at step start, and positive OpenCode-normalized counters at step end. A creation event can also carry an OpenCode project ID and parent session ID; neither is a repository name. It does not export prompts, responses, tool contents or cost. The plugin stream can miss events; event IDs allow deduplication of observations received, not proof of complete delivery. Do not match retries solely by assistant message ID. Production export remains disabled. +One local setup uses `@openchamber/opencode-claude` v1.7.0 behind an OpenAI-compatible provider named `claude-code`. Its per-turn usage conversion maps a missing cache-write count and an explicit zero to the same output, before OpenCode receives it. An OpenCode HTTP hook cannot distinguish those cases in that conversion path. This is specific to that provider and version; it does not establish the behavior of every OpenCode provider. [Provider usage conversion](https://github.com/openchamber/opencode-claude/blob/v1.7.0/src/usage.ts) (read 2026-11-07). + ## Additional coverage investigated | Provider | Available or observed in OTLP | Current limit and next step | From a9c7bdee6e72996bb045e14f434b1a38a63bd2a3 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 11:21:19 +0800 Subject: [PATCH 54/56] docs(telemetry): qualify proxy input counts --- plugins/aidd-telemetry/OBSERVABILITY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index eef88b641..f16b88e87 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -19,7 +19,7 @@ In two local OpenCode v2.2.5 executions before the plugin was added, the raw OTL The opt-in OpenCode v2 plugin subscribes to `session.created`, `session.step.started`, `session.step.ended` and `session.step.failed`, and exports each event separately as OTLP logs. It includes the original event and session IDs, message ID when present, selected model and variant at step start, and positive OpenCode-normalized counters at step end. A creation event can also carry an OpenCode project ID and parent session ID; neither is a repository name. It does not export prompts, responses, tool contents or cost. The plugin stream can miss events; event IDs allow deduplication of observations received, not proof of complete delivery. Do not match retries solely by assistant message ID. Production export remains disabled. -One local setup uses `@openchamber/opencode-claude` v1.7.0 behind an OpenAI-compatible provider named `claude-code`. Its per-turn usage conversion maps a missing cache-write count and an explicit zero to the same output, before OpenCode receives it. An OpenCode HTTP hook cannot distinguish those cases in that conversion path. This is specific to that provider and version; it does not establish the behavior of every OpenCode provider. [Provider usage conversion](https://github.com/openchamber/opencode-claude/blob/v1.7.0/src/usage.ts) (read 2026-11-07). +One local setup uses `@openchamber/opencode-claude` v1.7.0 behind an OpenAI-compatible provider named `claude-code`. Its per-turn usage conversion maps a missing cache-write count and an explicit zero to the same output, before OpenCode receives it. An OpenCode HTTP hook cannot distinguish those cases in that conversion path. For a turn with multiple model calls, the proxy can report the last call's prompt context rather than the sum of their inputs. This is specific to that provider and version; it does not establish the behavior of every OpenCode provider. [Provider usage conversion](https://github.com/openchamber/opencode-claude/blob/v1.7.0/src/usage.ts) (read 2026-11-07). ## Additional coverage investigated From 9bbda0a660b97a08ebf0a44a7e66ecfd8146c686 Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 14:34:15 +0800 Subject: [PATCH 55/56] docs(telemetry): record repository attribute coverage --- plugins/aidd-telemetry/OBSERVABILITY.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index f16b88e87..db9032ee2 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -8,6 +8,7 @@ Gouvernail's AI KPI view uses OpenTelemetry (OTel) observations. For OpenCode v2 | Model | `model`, when present | `model`, when present | Selected model and variant on plugin step-start events, when emitted; not proof of the served model or effective effort | | Usage | Forwarded `claude_code.token.usage` metrics and technical fields from `api_request` events | `codex.sse_event` and turn traces | OpenCode-normalized input, output, reasoning and cache counts from plugin step-end events; ambiguous zero stays unavailable | | Identity | Provider attributes when emitted, or an explicit connector-token assignment | Provider attributes when emitted, or an explicit connector-token assignment | An explicit connector-token assignment | +| Repository and branch | Repository URL when explicitly enabled; no branch attribute in the local two-turn capture | No repository or branch attribute observed in the local two-turn capture | No repository or branch attribute observed in the local two-turn capture | Token fields are provider-specific. Claude's input excludes cache reads and writes; Codex's input includes cached input. A Codex cache zero can mean the provider omitted cache details. Never add a log, metric and trace describing the same usage, or treat an absent value as zero. @@ -49,3 +50,5 @@ Read 2026-11-06: - [OTLP protocol](https://opentelemetry.io/docs/specs/otlp/) Read 2026-11-07: [Codex 0.166.0 turn span and counters](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L369-L382), [turn counter capture](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L760-L810) and [cache defaulting](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/codex-api/src/sse/responses.rs#L131-L153). + +Read 2026-11-07: [Claude Code repository attributes](https://code.claude.com/docs/en/monitoring-usage#repository-attributes). From 8b2eb05d1290eab038f9fd598f6e16286e79e04d Mon Sep 17 00:00:00 2001 From: alexsoyes <contact.alexsoyes@gmail.com> Date: Sat, 7 Nov 2026 14:39:08 +0800 Subject: [PATCH 56/56] docs(telemetry): qualify Codex cache counters --- plugins/aidd-telemetry/OBSERVABILITY.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/plugins/aidd-telemetry/OBSERVABILITY.md b/plugins/aidd-telemetry/OBSERVABILITY.md index db9032ee2..d140b402d 100644 --- a/plugins/aidd-telemetry/OBSERVABILITY.md +++ b/plugins/aidd-telemetry/OBSERVABILITY.md @@ -27,7 +27,7 @@ One local setup uses `@openchamber/opencode-claude` v1.7.0 behind an OpenAI-comp | Provider | Available or observed in OTLP | Current limit and next step | | --- | --- | --- | | Claude Code | The documented `api_request` event can carry request and prompt IDs, model, effort and separate token and cache counters. These fields appeared in both local captures and are forwarded when valid. | The receiver compares request and metric counts; it does not add the two signals. Missing or conflicting observations remain unavailable. Repository attributes require a separate opt-in. | -| Codex | In two local turns with CLI 0.166.0, a `session_task.turn` span contained `thread.id`, `turn.id`, selected model, effort and turn usage together. | The span counts the session usage delta during the turn and may cover several requests or models. The model is selected, not confirmed for every request. Cache and output zeros can be normalized defaults. A turn ID can recur during recovery; retain trace and span IDs. `otel run` sends allowlisted turn spans after session logs; generic `otel sync` does not yet forward these spans. Never add those counters to response-event counters. | +| Codex | In local two-turn captures with CLI 0.166.0 and 0.167.0, a `session_task.turn` span contained `thread.id`, `turn.id`, selected model, effort and turn usage together. | The span counts the session usage delta during the turn and may cover several requests or models. The model is selected, not confirmed for every request. Cache and output zeros can be normalized defaults. A turn ID can recur during recovery; retain trace and span IDs. `otel run` sends allowlisted turn spans after session logs; generic `otel sync` does not yet forward these spans. Never add those counters to response-event counters. | | OpenCode v2 | The v2.2.5 exporter emits logs and traces. In our captures, session HTTP spans and model/usage spans did not share a usable correlation. | The opted-in plugin sends session-linked OpenCode events as OTLP. Missing counters are normalized to zero upstream, so zeros cannot establish actual usage. A model variant is not necessarily reasoning effort. | An observed effort is evidence only for its own observation. Do not carry it into a later session or an unlinked event, and do not equate a requested model with a confirmed response model. @@ -51,4 +51,6 @@ Read 2026-11-06: Read 2026-11-07: [Codex 0.166.0 turn span and counters](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L369-L382), [turn counter capture](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/core/src/tasks/mod.rs#L760-L810) and [cache defaulting](https://github.com/openai/codex/blob/rust-v0.166.0/codex-rs/codex-api/src/sse/responses.rs#L131-L153). +In the 0.167.0 capture, two turn spans and four response events report zero cache writes, but they share [the same upstream defaulting](https://github.com/openai/codex/blob/rust-v0.167.0/codex-rs/codex-api/src/sse/responses.rs#L124-L145), so they do not independently establish a zero. Codex's [`non_cached_input_tokens`](https://github.com/openai/codex/blob/rust-v0.167.0/codex-rs/protocol/src/protocol.rs#L2487-L2492) subtracts cache reads but can still include cache writes; it is not the dashboard's cache-excluding input. Sources read 2026-11-07. + Read 2026-11-07: [Claude Code repository attributes](https://code.claude.com/docs/en/monitoring-usage#repository-attributes).