diff --git a/plugins/recce/hooks/scripts/README-telemetry.md b/plugins/recce/hooks/scripts/README-telemetry.md new file mode 100644 index 0000000..b1601f2 --- /dev/null +++ b/plugins/recce/hooks/scripts/README-telemetry.md @@ -0,0 +1,94 @@ +# `/recce-verify` telemetry — opt-in PostHog events + +L3 funnel signal for the `Agent-blind spots: /recce-verify v1` project ([DRC-3597](https://linear.app/recce/issue/DRC-3597)). Lets the project see whether real-world agent users reach for the skill, complete it, and convert downstream — the production complement to the L1 offline eval (DRC-3405) and L2 in-driver trace metrics (DRC-3586). + +**Off by default.** Telemetry fires only when the user opts in. + +## What gets emitted + +| Event | When | Properties | +|---|---|---| +| `recce_verify.skill_invoked` | `/recce-verify` SKILL.md activates | `tier_detected` (0 / 1 / 2), `agent` (claude_code / codex), `recce_version`, `plugin_version` | +| `recce_verify.tier_degraded` | Skill detects degraded capability and falls back | `from_tier`, `to_tier`, `reason` (recce_missing / mcp_unreachable / no_dev_env / …) | +| `recce_verify.tool_call` | Any `mcp__plugin_recce_recce__*` call inside the skill flow | `tool_name`, `success` (true/false), `error_class` (truncated), `duration_ms_bucket` | +| `recce_verify.verdict_emitted` | Skill writes its structured verdict | `verdict` (catch/miss/partial/abstain), `evidence_tier`, `subset` (1a/1b/1c) | +| `recce_verify.session_completed` | Last skill step before agent yields | `cells_invoked` count, `total_duration_ms_bucket` | + +**No PII, no SQL bodies, no model identifiers in event properties** — coarse buckets only. The `verdict` value is event-bound (`catch` / `miss` / `partial`), not free-form. + +## How to opt in (user side) + +Either: + +```bash +export RECCE_TELEMETRY_OPT_IN=1 # per-shell +``` + +Or write to `~/.recce/config.yml`: + +```yaml +telemetry_opt_in: true +``` + +To opt out for one invocation while the env / config is on: + +```bash +RECCE_TELEMETRY_DISABLED=1 claude # bypass for this session +``` + +## How to wire (plugin maintainer side) + +The emitter script is at `plugins/recce/hooks/scripts/telemetry.sh`. Call it with the event name + key=value props: + +```bash +bash "${CLAUDE_PLUGIN_ROOT}/hooks/scripts/telemetry.sh" recce_verify.skill_invoked \ + tier_detected=1 agent=claude_code recce_version=0.42.0 plugin_version=0.2.0 +``` + +Fires in the background. Never blocks. Returns 0 unconditionally. + +### Auto-wired (via hooks.json) + +None yet — the wiring needs design decisions (which events fire from hooks vs from inline SKILL.md calls) that should happen in a follow-up PR. This PR ships the emitter foundation only. + +Recommended next steps (see DRC-3597 acceptance): + +1. Add `PostToolUse` hook matcher `mcp__plugin_recce_recce__.*` → `telemetry.sh recce_verify.tool_call tool_name=$TOOL_NAME success=$SUCCESS`. Requires deciding how to scope to "only when /recce-verify is active" — proposal: SKILL.md sets a marker file at Step 0, hook checks marker before firing. +2. Add `Stop` hook → `telemetry.sh recce_verify.session_completed cells_invoked=…`. Requires aggregating per-session counts (e.g. from the marker file). +3. Inline `bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/telemetry.sh recce_verify.skill_invoked …` in SKILL.md Step 0 (single-line addition, low risk). +4. Inline `… recce_verify.verdict_emitted …` in the SKILL.md verdict-write step. + +### PostHog project configuration + +The plugin maintainer fills in the PostHog public project key during release packaging — either by patching `telemetry.sh` (constant) or by shipping a small wrapper that exports `RECCE_POSTHOG_PROJECT_KEY` before calling the script. Both approaches keep the key in plugin-controlled code, not in user config. + +For development: + +```bash +export RECCE_POSTHOG_PROJECT_KEY=phc_xxxxx +export RECCE_TELEMETRY_OPT_IN=1 +bash plugins/recce/hooks/scripts/test-telemetry.sh +``` + +## Funnel attribution + +Each user's events carry a stable `distinct_id` from `~/.recce/installation-id` (UUID4, lazily created on first opt-in emit, persisted across sessions). The file holds only the UUID — no credentials, no personal info. + +Attribution to Recce Cloud signups requires a join key on the Cloud side. That's not in this PR — coordinate with @Andy when Cloud's signup form picks up the parameter. + +## Failure modes (all silent, all non-blocking) + +- Opt-in unset → script exits 0 before doing anything. +- `~/.recce/` unwriteable → installation-id creation skipped, script exits 0. +- No PostHog key configured → script exits 0 before any HTTP call. +- PostHog endpoint unreachable / 5xx / timeout → `curl --max-time 2`; the request runs in a background subshell, so the script returns immediately regardless of outcome. +- `jq` missing → properties JSON falls back to empty `{}`; event still fires with no props. +- Any other unexpected failure → trailing `|| true` and `set -u` (no `-e`) keep the script returning 0. + +## Audit script + +`test-telemetry.sh` exercises the opt-in paths without firing a real network request, by setting `RECCE_POSTHOG_PROJECT_KEY=""` so the emit short-circuits before the `curl`. Use it after editing `telemetry.sh`: + +```bash +bash plugins/recce/hooks/scripts/test-telemetry.sh +``` diff --git a/plugins/recce/hooks/scripts/telemetry.sh b/plugins/recce/hooks/scripts/telemetry.sh new file mode 100755 index 0000000..af06c93 --- /dev/null +++ b/plugins/recce/hooks/scripts/telemetry.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# telemetry.sh — opt-in PostHog event emitter (DRC-3597, L3 funnel signal) +# +# Fires events for the /recce-verify skill funnel: +# - recce_verify.skill_invoked +# - recce_verify.tier_degraded +# - recce_verify.tool_call +# - recce_verify.verdict_emitted +# - recce_verify.session_completed +# +# Off by default. Activate via one of: +# - export RECCE_TELEMETRY_OPT_IN=1 +# - write `telemetry_opt_in: true` to ~/.recce/config.yml +# +# Failure modes are silent — never blocks the agent flow. +# +# Usage: +# telemetry.sh [key=value] [key=value] ... + +set -u + +# ── Opt-in gate ──────────────────────────────────────────────────────────────── + +if [[ "${RECCE_TELEMETRY_OPT_IN:-}" != "1" ]]; then + cfg="${HOME}/.recce/config.yml" + if [[ ! -f "${cfg}" ]] || ! grep -qE '^telemetry_opt_in:[[:space:]]*true$' "${cfg}" 2>/dev/null; then + exit 0 + fi +fi + +# ── Per-tool one-flag bypass ─────────────────────────────────────────────────── + +if [[ "${RECCE_TELEMETRY_DISABLED:-}" == "1" ]]; then + exit 0 +fi + +# ── Event + properties ───────────────────────────────────────────────────────── + +event="${1:-}" +shift || true + +if [[ -z "${event}" ]]; then + echo "telemetry.sh: missing event name" >&2 + exit 0 +fi + +# ── Anonymous, stable installation ID ───────────────────────────────────────── + +install_id_file="${HOME}/.recce/installation-id" +if [[ ! -f "${install_id_file}" ]]; then + mkdir -p "$(dirname "${install_id_file}")" 2>/dev/null || exit 0 + if command -v uuidgen >/dev/null 2>&1; then + uuidgen | tr '[:upper:]' '[:lower:]' > "${install_id_file}" 2>/dev/null || exit 0 + elif command -v python3 >/dev/null 2>&1; then + python3 -c 'import uuid; print(uuid.uuid4())' > "${install_id_file}" 2>/dev/null || exit 0 + else + exit 0 + fi +fi +install_id="$(cat "${install_id_file}" 2>/dev/null || true)" +[[ -z "${install_id}" ]] && exit 0 + +# ── PostHog destination ──────────────────────────────────────────────────────── + +posthog_host="${POSTHOG_HOST:-https://us.i.posthog.com}" +posthog_key="${RECCE_POSTHOG_PROJECT_KEY:-}" + +# If no key configured, drop the event silently. The plugin maintainer fills +# this in during plugin packaging; users opting in still need a key to fire. +if [[ -z "${posthog_key}" ]]; then + exit 0 +fi + +# ── Build properties JSON from key=value args ───────────────────────────────── + +props='{}' +if command -v jq >/dev/null 2>&1; then + for kv in "$@"; do + k="${kv%%=*}" + v="${kv#*=}" + [[ "${k}" == "${kv}" ]] && continue # no '=' → skip + props=$(jq --arg k "${k}" --arg v "${v}" '. + {($k): $v}' <<< "${props}" 2>/dev/null) || props='{}' + done +fi + +# ── Fire-and-forget POST ─────────────────────────────────────────────────────── + +payload=$(cat </dev/null 2>&1 || true +) & + +exit 0 diff --git a/plugins/recce/hooks/scripts/test-telemetry.sh b/plugins/recce/hooks/scripts/test-telemetry.sh new file mode 100755 index 0000000..c6afeca --- /dev/null +++ b/plugins/recce/hooks/scripts/test-telemetry.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# test-telemetry.sh — verify opt-in / opt-out paths in telemetry.sh +# +# Exercises every short-circuit in the emitter without hitting the network. +# Sets RECCE_POSTHOG_PROJECT_KEY="" so the emit ends before the curl call; +# this isolates opt-in logic from network behaviour. + +set -uo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" + +EMIT="./telemetry.sh" +PASS=0 +FAIL=0 + +run() { + local label="$1" + shift + local want="$1" + shift + # Capture stdout + stderr together + set +e + out=$(env -i HOME="${HOME}" PATH="${PATH}" "$@" "${EMIT}" recce_verify.test prop_a=1 2>&1) + rc=$? + set -e + if [[ "${rc}" == "${want}" ]]; then + PASS=$((PASS + 1)) + printf " pass rc=%d %s\n" "${rc}" "${label}" + else + FAIL=$((FAIL + 1)) + printf " FAIL want=%s got=%d %s\n --- output ---\n%s\n --- end ---\n" \ + "${want}" "${rc}" "${label}" "${out}" + fi +} + +echo "Testing telemetry.sh opt-in paths (no network)..." +echo + +# All cases should rc=0 (silent + non-blocking is the contract). The behaviour +# difference is whether the script proceeds past the opt-in gate, which we +# infer from -x trace if needed; here we just confirm rc=0 across paths. + +run "default off (no env, no config)" 0 +run "RECCE_TELEMETRY_OPT_IN=1 + no key (drops at key check)" 0 \ + RECCE_TELEMETRY_OPT_IN=1 + +run "RECCE_TELEMETRY_OPT_IN=1 + key + DISABLED=1 (bypass)" 0 \ + RECCE_TELEMETRY_OPT_IN=1 RECCE_POSTHOG_PROJECT_KEY=phc_fake RECCE_TELEMETRY_DISABLED=1 + +run "missing event name" 0 + +echo +echo "Summary: ${PASS} pass · ${FAIL} fail" +exit "${FAIL}"