feat(llma): Add Claude Code LLM Analytics integration#24
Merged
andrewm4894 merged 14 commits intomainfrom Apr 13, 2026
Merged
Conversation
Send Claude Code sessions to PostHog LLM Analytics as $ai_generation, $ai_span, and $ai_trace events. Parses session JSONL on SessionEnd hook and posts to PostHog's /batch endpoint. No-op unless POSTHOG_API_KEY is set. Costs are calculated by PostHog's ingestion pipeline automatically. Supports session-level or per-message trace grouping via POSTHOG_LLMA_TRACE_GROUPING env var (defaults to session).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41d8082d9e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Fix tool result matching (check content[].tool_use_id not just sourceToolUseID) - Fix dedup to keep last entry per message ID (later entries have tool_use blocks) - Use real timestamps from session log instead of send time - Add $ai_trace_name from first user prompt (with XML tag stripping) - Capture slash command invocations that lack promptId - Add /posthog:llma-cc-ingest command for manual session ingestion
/clear, /exit, /help, [Request interrupted] etc. are skipped in favour of the first real user prompt.
PostHog's ingestion pipeline extracts $ai_tools_called from $ai_output_choices content blocks. Include tool_use blocks in Anthropic format so tools are detected automatically.
andrewm4894
changed the title
Users may have POSTHOG_API_KEY set for their app's analytics. Sending Claude Code sessions there silently would be a surprise. Now requires both POSTHOG_LLMA_CC_ENABLED=true and POSTHOG_API_KEY.
45 tests covering: generation/span/trace building, stop reason mapping, privacy mode, timestamps, dedup, tool result matching, prompt ID resolution, trace naming, trace grouping modes, and config loading.
Split monolithic scripts into a proper package: - posthog_llma/events.py — $ai_* event builders - posthog_llma/sender.py — batch sender and status file - posthog_llma/config.py — env var and config file loading - posthog_llma/parser.py — Claude Code JSONL session parser - posthog_llma/event_builder.py — parsed session to PostHog events - posthog_llma/trace_naming.py — trace name selection logic Hooks and scripts are now thin entry points that import from the package. All 45 tests passing.
Drop the custom .local.md config file in favour of standard Claude Code settings.json env blocks. Users can set env vars globally in ~/.claude/settings.json or per-project in .claude/settings.local.json.
When trace_grouping=message and prompt_id is empty, use the generation's span_id as fallback so its child tool spans share the same trace_id instead of each getting independent random UUIDs.
…k trace roots - Replace PEP 604 `X | None` with `Optional[X]` so entrypoints work on stock macOS python3 (3.9.6) instead of requiring 3.10+ - Add $ai_parent_id to $ai_generation events so they link to their trace root in PostHog's span hierarchy - Emit $ai_trace root events for fallback (unresolved prompt) trace IDs in message grouping mode, closing the orphan gap
4 tasks
Users can set a JSON object of custom properties that get merged into every $ai_generation, $ai_span, and $ai_trace event. Useful for tagging events with team, product, or environment metadata for filtering in PostHog.
- Wrap hook main() in top-level try/except so any unexpected error silently exits 0 instead of showing users a traceback - Use safe int parsing for max_attribute_length config to handle invalid env var values gracefully
Tool use blocks in $ai_output_choices were only sending type and name, missing the input args PostHog displays in the trace UI. Now includes args with safe truncation to max_attribute_length and a fallback for unserializable inputs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
$ai_generation,$ai_span, and$ai_traceevents to PostHog LLM AnalyticsPOSTHOG_LLMA_CC_ENABLED=trueplusPOSTHOG_API_KEY-- existing users are unaffected$ai_tools_calledare calculated by PostHog's ingestion pipeline automaticallyPOSTHOG_LLMA_TRACE_GROUPINGPOSTHOG_LLMA_CUSTOM_PROPERTIES(JSON object)Package structure
User experience
Bug fixes
X | NonewithOptional[X]so entrypoints work on stock macOSpython3(3.9.6) instead of requiring 3.10+$ai_parent_idon generations:$ai_generationevents now set$ai_parent_idlinking them to their trace root in PostHog's span hierarchyprompt_idnow get a root$ai_traceevent instead of being orphanedTest plan
$ai_generationevents with model, tokens, cost (auto-enriched by PostHog)$ai_generationevents include$ai_parent_idpointing at trace root$ai_spanevents for tool calls with parent-child linking$ai_traceevents with session-level grouping and trace name from first meaningful prompt$ai_traceevents in message grouping mode$ai_tools_calledauto-extracted by PostHog from output_choices tool_use blocksPOSTHOG_LLMA_CC_ENABLEDis not set to true/posthog:llma-cc-ingest/usr/bin/python3(macOS 3.9.6)