feat(recce): opt-in PostHog telemetry scaffolding for /recce-verify (DRC-3597)

plugins/recce/hooks/scripts/README-telemetry.md

@@ -0,0 +1,94 @@
+# `/recce-verify` telemetry — opt-in PostHog events
+
+L3 funnel signal for the `Agent-blind spots: /recce-verify v1` project ([DRC-3597](https://linear.app/recce/issue/DRC-3597)). Lets the project see whether real-world agent users reach for the skill, complete it, and convert downstream — the production complement to the L1 offline eval (DRC-3405) and L2 in-driver trace metrics (DRC-3586).
+
+**Off by default.** Telemetry fires only when the user opts in.
+
+## What gets emitted
+
+| Event | When | Properties |
+|---|---|---|
+| `recce_verify.skill_invoked` | `/recce-verify` SKILL.md activates | `tier_detected` (0 / 1 / 2), `agent` (claude_code / codex), `recce_version`, `plugin_version` |
+| `recce_verify.tier_degraded` | Skill detects degraded capability and falls back | `from_tier`, `to_tier`, `reason` (recce_missing / mcp_unreachable / no_dev_env / …) |
+| `recce_verify.tool_call` | Any `mcp__plugin_recce_recce__*` call inside the skill flow | `tool_name`, `success` (true/false), `error_class` (truncated), `duration_ms_bucket` |
+| `recce_verify.verdict_emitted` | Skill writes its structured verdict | `verdict` (catch/miss/partial/abstain), `evidence_tier`, `subset` (1a/1b/1c) |
+| `recce_verify.session_completed` | Last skill step before agent yields | `cells_invoked` count, `total_duration_ms_bucket` |
+
+**No PII, no SQL bodies, no model identifiers in event properties** — coarse buckets only. The `verdict` value is event-bound (`catch` / `miss` / `partial`), not free-form.
+
+## How to opt in (user side)
+
+Either:
+
+```bash
+export RECCE_TELEMETRY_OPT_IN=1                # per-shell
+```
+
+Or write to `~/.recce/config.yml`:
+
+```yaml
+telemetry_opt_in: true
+```
+
+To opt out for one invocation while the env / config is on:
+
+```bash
+RECCE_TELEMETRY_DISABLED=1 claude            # bypass for this session
+```
+
+## How to wire (plugin maintainer side)
+
+The emitter script is at `plugins/recce/hooks/scripts/telemetry.sh`. Call it with the event name + key=value props:
+
+```bash
+bash "${CLAUDE_PLUGIN_ROOT}/hooks/scripts/telemetry.sh" recce_verify.skill_invoked \
+    tier_detected=1 agent=claude_code recce_version=0.42.0 plugin_version=0.2.0
+```
+
+Fires in the background. Never blocks. Returns 0 unconditionally.
+
+### Auto-wired (via hooks.json)
+
+None yet — the wiring needs design decisions (which events fire from hooks vs from inline SKILL.md calls) that should happen in a follow-up PR. This PR ships the emitter foundation only.
+
+Recommended next steps (see DRC-3597 acceptance):
+
+1. Add `PostToolUse` hook matcher `mcp__plugin_recce_recce__.*` → `telemetry.sh recce_verify.tool_call tool_name=$TOOL_NAME success=$SUCCESS`. Requires deciding how to scope to "only when /recce-verify is active" — proposal: SKILL.md sets a marker file at Step 0, hook checks marker before firing.
+2. Add `Stop` hook → `telemetry.sh recce_verify.session_completed cells_invoked=…`. Requires aggregating per-session counts (e.g. from the marker file).
+3. Inline `bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/telemetry.sh recce_verify.skill_invoked …` in SKILL.md Step 0 (single-line addition, low risk).
+4. Inline `… recce_verify.verdict_emitted …` in the SKILL.md verdict-write step.
+
+### PostHog project configuration
+
+The plugin maintainer fills in the PostHog public project key during release packaging — either by patching `telemetry.sh` (constant) or by shipping a small wrapper that exports `RECCE_POSTHOG_PROJECT_KEY` before calling the script. Both approaches keep the key in plugin-controlled code, not in user config.
+
+For development:
+
+```bash
+export RECCE_POSTHOG_PROJECT_KEY=phc_xxxxx
+export RECCE_TELEMETRY_OPT_IN=1
+bash plugins/recce/hooks/scripts/test-telemetry.sh
+```
+
+## Funnel attribution
+
+Each user's events carry a stable `distinct_id` from `~/.recce/installation-id` (UUID4, lazily created on first opt-in emit, persisted across sessions). The file holds only the UUID — no credentials, no personal info.
+
+Attribution to Recce Cloud signups requires a join key on the Cloud side. That's not in this PR — coordinate with @Andy when Cloud's signup form picks up the parameter.
+
+## Failure modes (all silent, all non-blocking)
+
+- Opt-in unset → script exits 0 before doing anything.
+- `~/.recce/` unwriteable → installation-id creation skipped, script exits 0.
+- No PostHog key configured → script exits 0 before any HTTP call.
+- PostHog endpoint unreachable / 5xx / timeout → `curl --max-time 2`; the request runs in a background subshell, so the script returns immediately regardless of outcome.
+- `jq` missing → properties JSON falls back to empty `{}`; event still fires with no props.
+- Any other unexpected failure → trailing `|| true` and `set -u` (no `-e`) keep the script returning 0.
+
+## Audit script
+
+`test-telemetry.sh` exercises the opt-in paths without firing a real network request, by setting `RECCE_POSTHOG_PROJECT_KEY=""` so the emit short-circuits before the `curl`. Use it after editing `telemetry.sh`:
+
+```bash
+bash plugins/recce/hooks/scripts/test-telemetry.sh
+```

plugins/recce/hooks/scripts/telemetry.sh

@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+# telemetry.sh — opt-in PostHog event emitter (DRC-3597, L3 funnel signal)
+#
+# Fires events for the /recce-verify skill funnel:
+#   - recce_verify.skill_invoked
+#   - recce_verify.tier_degraded
+#   - recce_verify.tool_call
+#   - recce_verify.verdict_emitted
+#   - recce_verify.session_completed
+#
+# Off by default. Activate via one of:
+#   - export RECCE_TELEMETRY_OPT_IN=1
+#   - write `telemetry_opt_in: true` to ~/.recce/config.yml
+#
+# Failure modes are silent — never blocks the agent flow.
+#
+# Usage:
+#   telemetry.sh <event_name> [key=value] [key=value] ...
+
+set -u
+
+# ── Opt-in gate ────────────────────────────────────────────────────────────────
+
+if [[ "${RECCE_TELEMETRY_OPT_IN:-}" != "1" ]]; then
+    cfg="${HOME}/.recce/config.yml"
+    if [[ ! -f "${cfg}" ]] || ! grep -qE '^telemetry_opt_in:[[:space:]]*true$' "${cfg}" 2>/dev/null; then
+        exit 0
+    fi
+fi
+
+# ── Per-tool one-flag bypass ───────────────────────────────────────────────────
+
+if [[ "${RECCE_TELEMETRY_DISABLED:-}" == "1" ]]; then
+    exit 0
+fi
+
+# ── Event + properties ─────────────────────────────────────────────────────────
+
+event="${1:-}"
+shift || true
+
+if [[ -z "${event}" ]]; then
+    echo "telemetry.sh: missing event name" >&2
+    exit 0
+fi
+
+# ── Anonymous, stable installation ID ─────────────────────────────────────────
+
+install_id_file="${HOME}/.recce/installation-id"
+if [[ ! -f "${install_id_file}" ]]; then
+    mkdir -p "$(dirname "${install_id_file}")" 2>/dev/null || exit 0
+    if command -v uuidgen >/dev/null 2>&1; then
+        uuidgen | tr '[:upper:]' '[:lower:]' > "${install_id_file}" 2>/dev/null || exit 0
+    elif command -v python3 >/dev/null 2>&1; then
+        python3 -c 'import uuid; print(uuid.uuid4())' > "${install_id_file}" 2>/dev/null || exit 0
+    else
+        exit 0
+    fi
+fi
+install_id="$(cat "${install_id_file}" 2>/dev/null || true)"
+[[ -z "${install_id}" ]] && exit 0
+
+# ── PostHog destination ────────────────────────────────────────────────────────
+
+posthog_host="${POSTHOG_HOST:-https://us.i.posthog.com}"
+posthog_key="${RECCE_POSTHOG_PROJECT_KEY:-}"
+
+# If no key configured, drop the event silently. The plugin maintainer fills
+# this in during plugin packaging; users opting in still need a key to fire.
+if [[ -z "${posthog_key}" ]]; then
+    exit 0
+fi
+
+# ── Build properties JSON from key=value args ─────────────────────────────────
+
+props='{}'
+if command -v jq >/dev/null 2>&1; then
+    for kv in "$@"; do
+        k="${kv%%=*}"
+        v="${kv#*=}"
+        [[ "${k}" == "${kv}" ]] && continue   # no '=' → skip
+        props=$(jq --arg k "${k}" --arg v "${v}" '. + {($k): $v}' <<< "${props}" 2>/dev/null) || props='{}'
+    done
+fi
+
+# ── Fire-and-forget POST ───────────────────────────────────────────────────────
+
+payload=$(cat <<EOF
+{"api_key":"${posthog_key}","event":"${event}","distinct_id":"${install_id}","properties":${props}}
+EOF
+)
+
+# Background so we never block; --max-time caps wait at 2s if foregrounded
+(
+    curl -fsS -X POST "${posthog_host}/capture/" \
+        -H "Content-Type: application/json" \
+        -d "${payload}" \
+        --max-time 2 >/dev/null 2>&1 || true
+) &
+
+exit 0

plugins/recce/hooks/scripts/test-telemetry.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# test-telemetry.sh — verify opt-in / opt-out paths in telemetry.sh
+#
+# Exercises every short-circuit in the emitter without hitting the network.
+# Sets RECCE_POSTHOG_PROJECT_KEY="" so the emit ends before the curl call;
+# this isolates opt-in logic from network behaviour.
+
+set -uo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+EMIT="./telemetry.sh"
+PASS=0
+FAIL=0
+
+run() {
+    local label="$1"
+    shift
+    local want="$1"
+    shift
+    # Capture stdout + stderr together
+    set +e
+    out=$(env -i HOME="${HOME}" PATH="${PATH}" "$@" "${EMIT}" recce_verify.test prop_a=1 2>&1)
+    rc=$?
+    set -e
+    if [[ "${rc}" == "${want}" ]]; then
+        PASS=$((PASS + 1))
+        printf "  pass  rc=%d  %s\n" "${rc}" "${label}"
+    else
+        FAIL=$((FAIL + 1))
+        printf "  FAIL  want=%s got=%d  %s\n  --- output ---\n%s\n  --- end ---\n" \
+            "${want}" "${rc}" "${label}" "${out}"
+    fi
+}
+
+echo "Testing telemetry.sh opt-in paths (no network)..."
+echo
+
+# All cases should rc=0 (silent + non-blocking is the contract). The behaviour
+# difference is whether the script proceeds past the opt-in gate, which we
+# infer from -x trace if needed; here we just confirm rc=0 across paths.
+
+run "default off (no env, no config)" 0
+run "RECCE_TELEMETRY_OPT_IN=1 + no key (drops at key check)" 0 \
+    RECCE_TELEMETRY_OPT_IN=1
+
+run "RECCE_TELEMETRY_OPT_IN=1 + key + DISABLED=1 (bypass)" 0 \
+    RECCE_TELEMETRY_OPT_IN=1 RECCE_POSTHOG_PROJECT_KEY=phc_fake RECCE_TELEMETRY_DISABLED=1
+
+run "missing event name" 0
+
+echo
+echo "Summary: ${PASS} pass · ${FAIL} fail"
+exit "${FAIL}"