diff --git a/.agents/work/telemetry-coverage-gaps.md b/.agents/work/telemetry-coverage-gaps.md new file mode 100644 index 00000000..666afe56 --- /dev/null +++ b/.agents/work/telemetry-coverage-gaps.md @@ -0,0 +1,315 @@ +# Telemetry coverage gaps: signup to durable session usage + +Status: working review, 2026-06-29. + +Scope reviewed: + +- `sessions-api`: v3 API, internal worker endpoints, runtime, delivery, sources, Slack/GitHub, telemetry worker. +- `opencomputer`: new dashboard, `api-edge`, `events-ingest`, product/billing workers, TypeScript SDK. + +Goal: reconstruct a user's journey with enough precision to answer: who tried what, from which surface, what system accepted it, where it failed or slowed down, which retry/reconcile path ran, and what the user saw next. This must work when a customer mixes dashboard, REST, SDK, webhooks, Slack, GitHub, and session client tokens. + +## Current state + +The sessions-api telemetry spine is the strongest part: + +- `src/v3/telemetry/index.ts` builds bounded, redacted ClickHouse rows and deliberately does **not** gate event names behind a central taxonomy. Keep that. Adding telemetry must not require editing an enum. +- Public `/v3` requests get trace/request ids via `traceMiddleware`, with `api.request.start` / `api.request.end`. +- Runtime, queue, delivery, SSE, source materialization, Slack ingress, GitHub manifest, credentials, and managed model provisioning all have some domain events. +- `agent-telemetry` CF Queue -> telemetry worker -> ClickHouse is best-effort, batched, self-instrumented, and not on the product path. + +The weak side is product journey coverage: + +- Browser/dashboard currently initializes PostHog and identifies the logged-in user, but does not send a durable event stream to our ClickHouse telemetry spine. +- `api-edge` owns signup/auth, dashboard API, billing, API keys, dashboard -> `/v3` proxying, and sandbox webhook management, but most flows emit only console logs or D1 state changes. +- The TypeScript SDK sends no SDK/version/surface/correlation headers and does not expose response trace ids on thrown errors. +- `/v3` product-level events are uneven: session create/steer/SSE are rich; agent/repo/credential/destination CRUD and read-side activation paths mostly collapse to generic HTTP request telemetry. + +## Invariants + +1. One correlation vocabulary across dashboard, REST, SDK, edge workers, sessions-api, queues, runtime, and delivery: + - `trace_id`, `span_id`, `parent_span_id`, `request_id` + - `client_request_id` + - `owner_id_hash`, `org_id_hash`, `user_id_hash` + - `session_id`, `agent_id`, `turn_id`, `event_id`, `delivery_id`, `sandbox_id` + - `surface`: `dashboard`, `sdk`, `rest`, `client_token`, `slack`, `github`, `worker`, `internal`, `runtime` + - `sdk_name`, `sdk_version`, `ui_build`, `ui_route` as attrs until they deserve columns +2. No secrets, raw prompts, raw completions, raw repo contents, raw provider keys, bearer tokens, cookies, GitHub tokens, webhook secrets, or long user text in telemetry. +3. Telemetry is never product-correctness critical. It can retry, drop after exhaustion, and self-report loss, but cannot block user work. +4. Product events are free-form dotted names by convention. Do not recreate a central taxonomy gate. +5. Every event should make the next operational question easier. Avoid vanity page events that cannot explain a state transition, failure, or activation drop-off. + +## P0: shared correlation across API, SDK, and dashboard + +Today, dashboard calls, SDK calls, and raw REST calls can hit the same `/v3` operation but are hard to distinguish and join. + +Add client-origin headers: + +- Dashboard `apiFetch`: + - generate `X-OC-Client-Request-Id` per logical request + - send `X-OC-Surface: dashboard` + - send `X-OC-UI-Build` when available + - send `X-OC-UI-Route` as a route template, not concrete ids + - forward/record returned `x-trace-id` and `x-request-id` +- Dashboard `/api/dashboard/v3/*` proxy: + - forward `X-OC-Client-Request-Id`, `X-OC-Surface`, `X-OC-UI-Build`, `X-OC-UI-Route` + - forward an inbound `X-Trace-Id` if present, or mint one at the edge and return it + - emit edge proxy start/end with upstream status, upstream trace id, duration, caller org/user, and route template +- TypeScript SDK: + - send `User-Agent: @opencomputer/sdk/` + - send `X-OC-SDK: typescript` + - send `X-OC-SDK-Version` + - send `X-OC-Surface: sdk` + - generate `X-OC-Client-Request-Id` per request + - send `X-OC-Client-Attempt` on retries + - optionally accept an inbound trace id for advanced callers + - expose `traceId` and `requestId` on `OpenComputerError` from response headers as well as JSON body +- Raw REST: + - document that callers may pass `X-OC-Client-Request-Id` + - echo trace/request ids consistently + +Server changes: + +- `traceMiddleware` should capture these safe headers into telemetry attrs. +- Auth middleware should emit one auth outcome per request, not only selected handlers. +- `api.request.end` should include error envelope `type/code` when available, not only HTTP status. + +This is the highest-leverage work because it lets one org's mixed dashboard/SDK/API journey become one traceable graph. + +## P0: browser telemetry collector + +PostHog alone is not enough for failure reconstruction because it is outside the ClickHouse flight recorder and does not naturally join to backend traces. + +Add a small browser telemetry path, preferably a separate CF Worker or a narrow `api-edge` route that can later split out: + +- Browser sends to `POST /api/telemetry` with `navigator.sendBeacon` and fetch fallback. +- The collector validates origin/CORS, rate-limits, bounds payload size, redacts, enriches from `oc_session` when authenticated, and sends to a queue. +- The queue lands in ClickHouse, either the existing `agent_telemetry_events` table with `service="web-ui"` / `surface="dashboard"`, or a sibling `product_telemetry_events` table with the same trace fields. +- Unauthenticated events carry an anonymous browser id only. After login, link the anonymous id to hashed user/org in the collector. +- Keep PostHog for product analytics, cohorts, and UI funnels, but use ClickHouse for reconstructing user-visible failures and backend joins. + +Minimum browser events: + +- `ui.page.view`: route template, org/user when known, build id. +- `ui.action.start` / `ui.action.end`: operation name, entity type, outcome, duration. +- `ui.form.invalid`: field names/error classes only, no values. +- `ui.toast.error`: user-visible error title + safe error class. +- `ui.error.boundary`: component stack hash, route, error name/message head. +- `ui.chunk.reload`: stale chunk reload attempted/succeeded/failed. +- `ui.sse.open` / `ui.sse.close` / `ui.sse.error`: session id, start cursor, last seq, reconnect count. +- `ui.copy`: API key copied, docs/API hint copied/opened; no copied value. + +Do not make frontend telemetry depend on PostHog being enabled. The product flight recorder should be first-party. + +## P0: edge auth, signup, and activation funnel + +`api-edge` owns the first product-critical steps, but most are not in ClickHouse. + +Instrument: + +- `auth.login.start`: provider, redirect origin. +- `auth.callback.start`: has_code, has_error. +- `auth.workos.exchange`: status, duration, WorkOS error class. +- `auth.user.resolve`: created vs existing. +- `auth.org.resolve`: created vs existing, selected org, personal vs org. +- `auth.session.mint`: success/failure. +- `auth.signup.complete`: new user/org, home cell, billing provider, first org id hash. +- `auth.logout`: result before `posthog.reset()`. +- `auth.refresh`: success/failure, expiry class. + +Activation events: + +- `api_key.create`: dashboard/API, name_present, scopes, org/user, result. +- `api_key.first_used`: first successful `/api/whoami` or `/v3` call for that key. +- `sdk.first_seen`: first request from an SDK version for an org/key. +- `dashboard.first_v3_action`: first dashboard-proxied v3 mutation. +- `agent.create`, `credential.create`, `session.create`, `session.first_event`, `session.first_completion`. + +These should power a view like: + +`signup -> API key created -> SDK request -> agent created -> session created -> first turn started -> first user-visible completion`. + +## P1: edge and billing system events + +Billing and model access are activation-critical. Today `model_billing`, `model_meter`, Autumn/Stripe paths mostly log to console or mutate D1. + +Instrument at the edge: + +- `billing.customer.create`: provider, new/existing, duration, result. +- `billing.checkout.create`: plan/topup/setup/portal, result, redirect vs direct charge. +- `billing.webhook.received`: provider, event type, signature result. +- `billing.project`: org, halted transition, balance bucket, concurrency tier, duration. +- `billing.self_heal`: triggered, still_halted, provider status. +- `model_billing.enable.start/end`: org, previous status, attempts, OR key created, sessions-api handoff result. +- `model_billing.handoff`: sessions-api status, managed credential id present, duration. +- `model_meter.run.start/end`: org count, billed org count, debited amount bucket, failures. +- `model_meter.org`: per-org result, cap patch result, projected halt state. + +Reason: a user can fail to create a managed agent because model billing did not provision, because Autumn failed, because a cap patch failed, or because sessions-api rejected handoff. Those are currently hard to reconstruct end to end. + +## P1: `/v3` auth and product-level API events + +Public `/v3` generic request telemetry is useful but too low-level for product analysis. + +Add or fill gaps: + +- Global `auth.check` in org auth middleware: + - auth kind: `oc_org_key`, `oc_org_token`, `client_token`, `turn_token`, `dev_key`, `missing`, `invalid` + - resolution: `org`, `per_key`, `invalid`, `unavailable`, `skipped_self_auth` + - never emit token/key material +- Agent API: + - `agent.create`, `agent.create.error`, `agent.update`, `agent.read`, `agent.list` + - attrs: runtime, model, credential mode (`managed`, `byo`, `default`, `inline_key`), created/idempotent, revision +- Credential API: + - route-level create/rotate/delete/default/list with provider, default flag, backend result + - core credential emits exist, but route telemetry should capture user-visible validation and HTTP outcomes +- Repo/GitHub API: + - repos CRUD currently has no emits; add `repo.create/update/list/get` + - GitHub app emits exist; add read/list and installation preflight outcomes if setup UI depends on them +- Destinations/deliveries API: + - destination create/update/delete/list, SSRF rejection reason, secret rotated, enabled toggled + - redelivery requested/accepted/rejected +- Session read/lifecycle: + - `session.get`, `session.list`, `session.cancel`, `session.archive`, `session.client_token.create`, `session.result.get`, `session.turns.list` + - session create should emit mapped 422 errors (`runtime_unavailable`, `no_credential`, `managed_unavailable`) before returning; today unknown and conflict paths are richer than these expected failures +- Sources: + - `session.sources.list` + - source materialization already emits runner steps; ensure every source row status transition has source name, auth kind, repo identity hash, ref kind, sha mismatch, duration, bytes, retryable + +Read events can be sampled later if volume is high, but for launch/activation they are useful. Mutations and errors should not be sampled. + +## P1: dashboard action coverage + +The dashboard has many React Query mutations that only show toasts on error. Add operation-level instrumentation around them, preferably through a small helper rather than hand-written `posthog.capture` calls everywhere. + +Cover: + +- Agent create/update/start session from agent detail. +- Session start/steer/cancel/archive/open detail. +- Credential create/rotate/delete/default, including the stepped progress path. +- API key create/revoke/copy once. +- GitHub app setup, repo registration, source status polling once those screens land. +- Slack connect intent/complete/disconnect. +- Webhook create/delete/test/redeliver/reveal secret. +- Billing setup/portal/topup/concurrency/autotopup/promo. +- Org switch, invite, revoke, remove member, custom domain. + +Each operation event should include: + +- operation name +- route template +- entity type/id when safe +- client request id +- backend trace id/request id if a request happened +- duration +- result: `ok`, `error`, `abandoned`, `validation_error` +- error class/http status, not raw response body + +This avoids relying on generic page views to infer intent. + +## P1: SDK observability surface + +The SDK should not phone home independently. Server-side request telemetry is enough, provided the SDK annotates requests. + +Add: + +- SDK/version/surface headers. +- Client request id and retry attempt headers. +- Trace/request ids on thrown errors: + +```ts +try { + await oc.sessions.create(...) +} catch (err) { + if (err instanceof OpenComputerError) { + console.error(err.requestId, err.traceId, err.code) + } +} +``` + +- Optional callback hooks for customer observability, not for our telemetry: + - `onRequest`, `onResponse`, `onRetry` could be useful later. + - Avoid adding them unless there is a real user need; headers solve our own reconstruction first. + +## P1: session stream and client-token experience + +SSE server-side telemetry is better than most areas, but dashboard/client experience is incomplete. + +Add: + +- Browser `ui.sse.*` events as above. +- SDK `ClientSession.events()` should send SDK headers and track reconnect attempts through `X-OC-Client-Attempt` or query attrs if headers are available. +- Server should include reader kind and auth scope in SSE telemetry: org key, client token, query token, turn token. +- Capture stream close reason, frames sent, replay count, last cursor, duration, and parse/schema errors in dashboard. + +This is critical because "the session is stuck" often means "events are flowing but UI dropped the stream" or "the user only saw a stale poll result." + +## P2: ClickHouse schema and views + +The current wide table is good enough. Use `attributes` for new dimensions first. + +Add views once data lands: + +- `activation_funnel_by_org`: signup, key, SDK/dashboard action, agent, credential, session, first turn, first completion. +- `org_journey_timeline`: all events for one hashed org/user, including edge/dashboard. +- `api_request_failures_by_surface`: route + surface + SDK version + status/error. +- `dashboard_errors`: route + UI build + error class. +- `billing_journey`: signup through credits/model billing/halt/resume. +- `github_setup_journey`: app link, manifest, callback, install, repo use, source materialize. + +Also fix stale schema comments that imply `event_name` is backed by an `EVENT_NAMES` enum. The implementation is intentionally free-form. + +## P2: edge worker telemetry helper + +Build one small worker-compatible helper shared by `api-edge`, `events-ingest`, and future telemetry collector: + +- `emitEdgeTelemetry(env, event)` is best-effort and never throws. +- It bounds/redacts attrs using the same rules as sessions-api. +- It sends full rows to the telemetry queue or a dedicated product telemetry queue. +- It accepts explicit org/user/session ids and hashes where needed. +- It has no central event-name enum. + +Do not rely on Tail Workers/Axiom as the product flight recorder. Tail logs are useful for exception backstops, but they are unstructured, console-dependent, and not joined to session traces. + +## P2: internal endpoints and scheduled jobs + +Internal endpoints now manually adopt trace headers in places, and product-async forwards trace headers. That is acceptable, but make it systematic: + +- Add a small internal trace/auth middleware for `/internal/*` that emits request start/end and auth outcome. +- Keep `/internal/managed-credential` separately HMAC-gated, but instrument handoff auth result and replay rejection. +- Product-async scheduled reconciler currently logs scheduled successes/failures; add explicit `reconciler.scheduled.start/end/error` from the worker, plus regional endpoint summaries. +- Include `request_id` in worker-forwarded trace headers, not only trace/span. + +## Data safety notes + +- Hash raw user ids and org ids before ClickHouse unless they are already opaque and policy says they are safe. Existing sessions-api hashes `owner_id`; edge should mirror that. +- Do not emit email to ClickHouse. PostHog identify may include email for product analytics, but the ClickHouse flight recorder should use ids/hashes. +- Do not emit repo URLs if they can include private org/repo names unless we explicitly accept that. Prefer repo id or a stable hash; if clear `owner/repo` is needed for ops, decide deliberately. +- Do not emit webhook URLs, provider keys, GitHub tokens, cookies, WorkOS tokens, OpenRouter keys, prompts, completions, or message bodies. +- Error messages should be bounded and classified. Raw upstream bodies should be avoided unless redacted and capped. + +## First implementation slice + +1. Add `client_request_id`, `surface`, SDK/version, UI build/route propagation through dashboard, SDK, api-edge proxy, and sessions-api telemetry attrs. +2. Add the browser telemetry collector path and wire `apiFetch`, ErrorBoundary, route views, action helper, and SSE events. +3. Add edge auth/signup/billing/model-billing emits through a shared edge telemetry helper. +4. Fill `/v3` product gaps for auth middleware, agent/repo/credential/destination/session lifecycle expected errors. +5. Add ClickHouse views for activation and surface failures. + +This slice is enough to reconstruct: + +- user signs up +- user creates/copies an API key +- SDK or dashboard creates an agent +- credential/model billing resolves or fails +- session starts +- source checkout/runtime starts +- first turn completes, hangs, crashes, or fails to emit +- UI stream renders, disconnects, or errors + +## Open questions + +- Should browser telemetry and backend telemetry share `agent_telemetry_events`, or should browser/product telemetry have a sibling table with identical correlation fields? Sharing is simpler for joins; a sibling table keeps high-volume UI events from crowding runtime ops. +- How much anonymous pre-login UI behavior do we care to keep in ClickHouse versus PostHog only? +- Do we want raw org ids in ClickHouse for internal support convenience, or always hashed ids with a lookup in D1 when needed? +- Should dashboard route/pageview events be sampled after launch, while mutations/errors remain unsampled?