Skip to content

Add ai_trace_sample_rate to sample AI observability events by trace id #965

Description

@jakesciotto

Problem Statement

Teams whose AI observability volume is mostly $ai_* events have no supported way to keep a fraction of traces. The only path today is a hand-written before_send hook that hashes $ai_trace_id. Most teams do not write one. A hook that samples per event splits a trace, so the trace view shows partial trees.

Related: PostHog/posthog-js#4997 for the same option in posthog-node.

Solution Brainstorm

One client option, default off:

Client(api_key, ai_trace_sample_rate=0.1)

Env var POSTHOG_AI_TRACE_SAMPLE_RATE, same precedence rule as POSTHOG_CAPTURE_MODE (kwarg > env > default).

Behavior:

  1. Applies to events named $ai_* that carry $ai_trace_id.
  2. FNV-1a 32-bit hash of the trace id. Keep when hash / 2^32 < rate. Same hash in every SDK, so a mixed Python and Node stack makes one decision per trace.
  3. Runs in Client._enqueue before before_send, so a customer hook sees only kept events.
  4. Adds $ai_trace_sample_rate to kept events, so the server can scale totals later.
  5. Events with no trace id are kept. Rate unset or 1.0 means no sampling. Out-of-range values log a warning and are ignored.
flowchart LR
  W[AI wrappers and agent processors] --> C[_capture_ai_event]
  M[Manual capture of $ai_* events] --> E[Client._enqueue]
  C --> E
  E --> S{ai_trace_sample_rate}
  S -- drop --> X[Dropped]
  S -- keep --> B[before_send] --> Q[Queue]
Loading

Alternatives considered: a helper that returns a before_send function. Rejected because before_send takes one callable, so the helper does not compose with an existing hook.

Impact

  • Gives AI observability users a supported volume lever that keeps traces whole.
  • Removes the need for a custom hook that is easy to get wrong.

Blast radius

  • Default off. No behavior change for any existing caller.
  • Touches Client.__init__ and Client._enqueue only. Wrappers do not change.
  • Affects $ai_* events only. Other events never reach the sampler.
  • Kept $ai_* events gain one property when the option is set.
  • Public API grows by one option and one env var. references/public_api_snapshot.txt changes.

Not in this issue

Tail sampling that keeps errored traces. A per-call override. Server-side upweighting of cost dashboards.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions