Skip to content

Generic OpenTelemetry tracing: api-only instrumentor, GenAI attributes, LLM attempt spans - #324

Merged
Eigenwise merged 40 commits into
mainfrom
feat/generic-otel-tracing
Oct 21, 2026
Merged

Eigenwise merged 40 commits into
mainfrom
feat/generic-otel-tracing

Conversation

@Eigenwise

@Eigenwise Eigenwise commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

What

Tracing no longer depends on Phoenix or the OpenInference packages. AtomicAgentsInstrumentor runs on opentelemetry-api alone and emits OpenTelemetry GenAI semantic-convention spans that any backend reads: a plain OTLP collector, Grafana Tempo, Sentry, Datadog, Honeycomb, PostHog, Jaeger, Arize Phoenix, Langfuse.

  • Spans: invoke_agent {name} (AGENT), execute_tool {tool_name} (TOOL) and one chat {model} CLIENT span per Instructor attempt, driven by Instructor hooks: per-attempt and total usage (input, output, cached, reasoning tokens) for both chat-completion and Responses shapes, gen_ai.response.model, attempt count, validation errors as events, error.type on failures. Spans always attach to the current context.
  • Attributes: gen_ai.* primary; atomic_agents.* for framework detail; OpenInference keys only when openinference-instrumentation is importable (new phoenix extra) or TracingConfig(openinference_attributes=True).
  • Config: TracingConfig(hide_inputs, hide_outputs, max_attribute_length=4096, openinference_attributes, llm_spans); env ATOMIC_AGENTS_TRACE_HIDE_INPUTS/_OUTPUTS (fallback OPENINFERENCE_HIDE_*), ATOMIC_AGENTS_TRACE_MAX_ATTRIBUTE_LENGTH, ATOMIC_AGENTS_TRACE_LLM_SPANS. Hide wins over truncation and is applied before serialization; under hide_outputs every exception is reduced to <Type>: <N> attempts (__REDACTED__).
  • API: AgentConfig(name=...) for stable span names; using_conversation(conversation_id) sets gen_ai.conversation.id; a second instrument() with different explicit values replaces in place with a WARNING, None keeps the installed value; AtomicAgent.clear_hooks() removes only its own handlers; cancellation is recorded as atomic_agents.cancelled, never ERROR.
  • Packaging: atomic-agents[observability] is now opentelemetry-api only; atomic-agents[phoenix] adds the OpenInference packages. Phoenix users on register(auto_instrument=True) keep working; set ATOMIC_AGENTS_TRACE_LLM_SPANS=false when a provider instrumentor also emits LLM spans.
  • Package layout: observability/ split into tracing_config, span_payloads, span_attributes, agent_spans, tool_spans, installation, llm_attempt_spans, with tracing.py as a thin facade.
  • Docs: guide rewritten OTLP-first with per-backend recipes (marked untested where nobody ran them); README section; phoenix-tracing example on the phoenix extra.

Design reviewed and probed by the Contractify consumer (Ada general chat instrumentation, Sentry and Grafana Tempo targets): verdict adoptable on d1c1bc8, 59 of 60 checks passing with real Instructor 1.17 + openai 2.53 and no OpenInference installed. Two should-fix items from that run (tool cancellation status, TraceConfig path honoring the env switch) land in the last commit before merge.

Verification

  • Merged-tree gate: 800 tests passed, pyright 0 errors on the package, Black/Flake8 clean, Sphinx builds, no function above radon's A band.
  • Observability suite: 176 tests including 24 real-Instructor cases over httpx.MockTransport across TOOLS, JSON, JSON_SCHEMA and RESPONSES_TOOLS, a two-thread registration race, suppression, redaction on retry exhaustion.
  • CI: fresh-install job installs the wheel with [observability] + opentelemetry-sdk, proves openinference is absent, runs a real Instructor client over a mock transport and asserts GenAI keys and no OpenInference keys; lowest-versions job compiles both extras.
  • Internal adversarial clean-code review (SQ-151) and the consumer's full-diff review (19 findings) both addressed with regression tests.

🤖 Generated with Claude Code

Kenny Vaneetvelde and others added 30 commits October 21, 2026 11:10
Co-Authored-By: Claude Code <noreply@anthropic.com>
Observability suite: 57 passed after updating imports.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Add typed tracing configuration, optional OpenInference attributes,
privacy controls, payload truncation, suppression and error types.
Focused observability regressions pass, including 12 assertion-failing
negative controls. Changed-body complexity stays below 6; incomplete
LCOV inventory measurements are tracked in SQ-145.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Replace tracer provider and config on re-instrumentation with a warning.
Cover context isolation, plain-tracer parenting and provider replacement.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The lowest-versions job compiled only the observability extra, which no
longer includes the OpenInference packages after the extras split, so
test_tracing.py and test_opentelemetry_discovery.py failed at collection
with ModuleNotFoundError: No module named 'openinference' (run
39577808157). Compile the phoenix extra too so those floors stay tested.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Use model-visible tool names, collect descriptive attributes independently,
filter absent values, cap error/schema text, map provider names and honor
Atomic Agents visibility settings. Warn about unknown launcher options.

Verified 112 observability tests, Pyright, Black and Flake8.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep tracing.py as the compatibility facade and move configuration, payloads,
attributes, agent spans, tool spans, and installation into their own modules.
Update test module references without changing assertions.

Co-Authored-By: Claude Code <noreply@anthropic.com>
… change

SQ-149 made gen_ai.tool.name and the execute_tool span name use
tool.tool_name (the input schema title), so the smoke script's lookup of
"execute_tool SmokeTool" raised KeyError in run 39582225418. Look spans
up by gen_ai.operation.name, assert the name against SmokeTool().tool_name
and check the class under atomic_agents.tool.class. The extracted script
passes locally with OpenInference keys forced off.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Exercise retry parentage, shared-client isolation and stream cleanup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Both style checks pass for the hook module.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Phoenix becomes one backend section. Adds a backends list, attribute tables (GenAI, atomic_agents, optional OpenInference), LLM attempt spans, naming and conversations, privacy and size, and re-instrumentation. README gets an Observability section and the phoenix example README an install line.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Defer run annotations to allow direct attempt-module imports.
Remove unused method names from wrapper factories.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Add ATOMIC_AGENTS_TRACE_LLM_SPANS and use it in the Phoenix example instead of replacing the instrumentor configuration. Cover false, mixed-case false, zero and enabled defaults.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Exercise retry exhaustion, Responses usage, validation attempt status,
and bounded traceback payloads through offline Instructor transports.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Kenny Vaneetvelde and others added 8 commits October 21, 2026 13:01
…limits

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep provider-default resolution at the installation boundary.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The observability package is now plain OpenTelemetry with GenAI attributes
primary and OpenInference optional, split into nine modules, with the
observability/phoenix extras, TracingConfig, using_conversation,
AgentConfig.name, LLM attempt spans and the ATOMIC_AGENTS_TRACE_* variables.
Eight map documents and the state manifest updated; CLAUDE.md untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Eigenwise
Eigenwise marked this pull request as ready for review October 21, 2026 11:26
Kenny Vaneetvelde and others added 2 commits October 21, 2026 13:30
Keep cancelled tools unset, inherit environment settings with visibility configs,
and exercise conversation attributes in the API-only smoke. Document the
installed Instructor retry logger's privacy limits.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Eigenwise
Eigenwise merged commit b8506eb into main Oct 21, 2026
3 checks passed
Eigenwise added a commit that referenced this pull request Oct 21, 2026
Bump the package and lockfile root version to 2.15.0, add the release
notes for the generic OpenTelemetry tracing shipped in #324, and update
the codebase map's package version. No production function bodies change.

Co-authored-by: Kenny Vaneetvelde <kenny@eigenwise.io>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant