Repository navigation
.NET: Python: Add OpenTelemetry spans for agent, chat, and function middleware #8559
Description
Summary
Add first-class OpenTelemetry telemetry for middleware execution in the Python Agent Framework.
Current behavior
Agent, chat, and function middleware execute inside the existing middleware pipelines, but MAF does not emit dedicated middleware spans or middleware-specific span attributes. Their execution is therefore only visible indirectly within the enclosing agent, chat, or function/tool span.
Requested behavior
Emit a dedicated span for each middleware invocation, while preserving the existing parent span relationships:
- Agent middleware spans under the
invoke_agentspan. - Chat middleware spans under the
chat/ inference span. - Function middleware spans under the function/tool execution span.
Suggested span attributes include:
- Middleware category:
agent,chat, orfunction. - Middleware implementation name.
- Execution outcome: completed, terminated, or failed.
- Duration.
- Exception/error type when applicable.
Middleware telemetry should follow the existing observability configuration, exporter setup, sensitive-data controls, and OpenTelemetry semantic-convention guidance. Middleware arguments, messages, and results must not be captured unless sensitive telemetry is explicitly enabled and the data follows the existing redaction policy.
Acceptance criteria
- Dedicated middleware spans are emitted for agent, chat, and function middleware.
- Span names and attributes are documented and covered by unit tests.
- Middleware spans preserve the correct parent-child relationships for streamed and non-streamed execution.
- Normal middleware termination is distinguishable from middleware failure.
- Middleware telemetry does not change execution behavior or break async stream cleanup.
- Exceptions and cancellation are recorded according to existing MAF telemetry conventions.
- The implementation works for class-based and callable middleware where the middleware name can be determined.
Related issues
- .NET: Workflow Middleware to Support Customizing Telemetry #3075 requests customizable workflow telemetry in .NET.
- Improve native GenAI semantic convention coverage #6626 tracks broader native GenAI semantic-convention coverage.
- Python: [Bug]: Graph workflow stream abandonment logs an OpenTelemetry context detach error #8397 tracked a Python workflow stream OpenTelemetry cleanup problem and is now closed.
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
[-]Python: Add OpenTelemetry spans for agent, chat, and function middleware[/-][+].NET: Python: Add OpenTelemetry spans for agent, chat, and function middleware[/+]droideronline commentedon Sep 22, 2026
Tao Chen (@TaoChenOSU) - kindly take a look when you get a chance
rksharma-owg commentedon Sep 27, 2026
I traced the Python middleware and telemetry layers on
main.AgentMiddlewareLayerruns beforeAgentTelemetryLayer; chat middleware likewise sits outsideChatTelemetryLayer; and function middleware wraps the tool invocation where the function span starts. A span around each middleware callback in the current pipelines would therefore be a parent of the existing agent/chat/tool span, rather than its child as requested here. Middleware that ends the call beforecall_next()never enters the inner telemetry layer.For short-circuited calls, should the agent/chat/tool parent span still be created outside the middleware pipeline, or should the middleware span attach to the ambient parent without an operation span? That choice seems necessary before changing the layering while preserving streaming cleanup and existing trace semantics.
eavanvalkenburg commentedon Sep 29, 2026
Dineshsuriya D (@droideronline) thanks for this, and RKS (@rksharma-owg) is correct, that we indeed run middleware outside the current spans, because measuring that as part of the inner would conflate things, having said that, I do agree that it might make sense for telemetry spans around the middleware parts, so couple of questions:
sivakumar-mahalingam commentedon Sep 29, 2026
Starting the invoke_agent / chat / execute_tool span before the middleware pipeline (instead of inside the telemetry layer) would make each middleware span a natural child of the operation span, which matches the requested hierarchy. It also covers the short-circuit case cleanly - the operation span still exists and can carry an outcome like
terminatedwhen a middleware returns without callingcall_next(), so traces don't lose the call entirely.droideronline commentedon Sep 29, 2026
I’d suggest emitting no span when there’s no middleware to run, and one span per middleware invocation rather than one span for the whole pipeline. That gives us useful per-middleware names and outcomes without adding empty or redundant spans.
For the hierarchy, I’d prefer each middleware span to be a child within its corresponding operation span (agent, model/chat, or tool call), rather than an ancestor of that span. That keeps middleware execution visible as part of the operation it affects. We should arrange the telemetry boundary/layering to make that nesting work, including for middleware that short-circuits before
call_next(), and preserve the existing streaming cleanup behavior.eavanvalkenburg commentedon Oct 27, 2026
Dineshsuriya D (@droideronline) RKS (@rksharma-owg) Sivakumar Mahalingam (@sivakumar-mahalingam) thanks for the discussion so far. Before we decide on implementation, could you work with the issue owner on a detailed telemetry plan?
Please build on the answers already given: one span per middleware invocation, and no spans for empty pipelines. The plan should spell out:
The body currently proposes Python behavior; call out any .NET parity questions for separate maintainer coordination. This is a request for a design proposal, not implementation approval, and does not change the existing vNext scheduling.
rksharma-owg commentedon Oct 28, 2026
The requested per-middleware spans and empty-pipeline behavior are clear. One hierarchy decision remains: the existing inner operation spans intentionally exclude middleware time. Making middleware temporal children of those same spans would expand their boundaries. I suggest preserving those spans and adding an outer scope only when a category has middleware. This needs the issue owner's agreement because it changes the exact hierarchy requested in the issue body.
Hierarchy and alternatives
agent_framework.agent.middleware_scope,agent_framework.chat.middleware_scope, andagent_framework.function.middleware_scope. Each covers the middleware chain and any stream consumption/cleanup. Middleware spans nest as the chain executes; the existinginvoke_agent,chat, andexecute_toolspans remain inside the innermost callback and retain their present boundaries and duration metrics. Short-circuiting creates no synthetic inner operation span.Ordinary call with two agent middleware, under the recommended option:
Short circuit:
Apply the same rule independently for chat and function categories, including nested calls. These envelope and attribute names are proposals, not OpenTelemetry standard names or existing APIs.
Span names and attributes
Proposed callback span name:
middleware <category> <qualified-name>, categoryagent,chat, orfunction, kind INTERNAL. Proposed attributes:agent_framework.middleware.category,.name,.streaming(boolean), and.outcome. Outcome is one ofcompleted,terminated,failed,cancelled, orclosed(early explicit stream close without an exception). Optionalerror.typeuses the exception class name, consistent with existingcapture_exceptionbehavior. Duration comes from span timestamps; no duplicate duration metric is needed initially.Resolve a function/bound method using its module and qualified name; for class-based middleware use the class module and qualified name; unwrap the framework's
MiddlewareWrapperso every callable is not named after the wrapper. For partials use the underlying callable's name without argument values; for anonymous callables retain the stable qualified name or a bounded type-name fallback. Never userepr, object IDs, captured values, dynamic context data, or arguments. Repeated registrations may have the same name; distinct span IDs identify invocations. Repeatedcall_next()calls produce separate downstream invocations, rather than reusing a previously ended span.Lifecycle
process(context, call_next), keep it active for callback pre/post work and awaited downstream work, and end infinally. A return withoutcall_next, orMiddlewareTermination, isterminated, not an error. Callback exceptions arefailed; rethrow the identical exception without changing suppression rules. Cancellation iscancelledand must propagate unchanged, including BaseException paths.ResponseStream.from_awaitable; constructing an unconsumed stream must create no span. Once invoked, keep one callback span alive through that invocation's resulting stream, registered transforms/gates, final-result handling and cleanup, ending exactly once at terminal cleanup. Returning the stream wrapper is not completion.get_final_response()finalize the same spans once. ExplicitResponseStream.close()before exhaustion ends them asclosed; cancellation/error close preserves the corresponding outcome. Cleanup failures becomefailedonly when they are the terminal error; they must not replace an already propagating primary error. An entirely unconsumed stream has no started spans. A started stream abandoned without closing follows the existing ownership contract; do not add telemetry-only draining, background tasks, finalizer-dependent correctness, or fabricated completion.Configuration and privacy
I propose using existing
OBSERVABILITY_SETTINGS.ENABLED/disable_instrumentation()controls and the configured tracer provider/sampler, with no new per-category switches. Within the agreed vNext feature, emit only for nonempty executed pipelines when instrumentation is enabled. The feature changes trace shape, so document that change and the extra envelope cost; if maintainers require a staged opt-in, introduce one feature-wide control with default off, rather than several independent settings. Its public spelling/scope should be agreed before implementation.Use the existing sensitive-data gate for exception messages/stacks and status descriptions. With sensitive data disabled, record only bounded category/name/outcome and error class, no prompt, messages, function arguments/results, credentials, session state, instance attributes or exception text. No sensitive context serialization is needed even when sensitive telemetry is enabled. Reuse current instrumentation scope and semantic-convention selection for existing spans; do not label middleware-specific attributes as standard
gen_aiconventions.Focused regression plan
Use the repository's in-memory span exporter/fake clients and existing middleware/observability fixtures. Cover all three categories: empty/single/multiple pipelines; callable/class/partial naming; constructor and per-call registration; repeated and concurrent
call_next; normal return, short circuit, MiddlewareTermination, exception and cancellation. Assert exact counts, parent IDs, existing inner timing boundaries and metric behavior with controlled barriers, plus unchanged return values, exception identity and downstream invocation counts.For agent/chat streams cover construction without consumption, normal iteration, final-response-only consumption, early close, error/cancellation before and after first update, transforms/gates/cleanup failures, repeated close/final-result calls, and creation/consumption in different tasks. Assert no leaked ambient span/context, exactly-once span finalization, and unchanged update order, buffering and cleanup. Add instrumentation-disabled, unsampled and sensitive-data-off canary checks including exception text and callable captures. Run focused middleware/observability suites plus required formatting/type checks, core suite and cloud validation before any eventual implementation PR. No tests were executed for this design-only response.
.NET parity should be coordinated separately: map its delegating middleware and ActivitySource names, cancellation and async-enumerable disposal boundaries, existing operation metrics, and any future option. Python names or implementation details should not be imposed on .NET without its maintainers' agreement.
The remaining design decision is whether to accept the envelope hierarchy while preserving inner timing, or explicitly change operation boundaries to match the original requested hierarchy. Implementation should wait for that direction and the existing vNext scheduling gate.
AI assistance was used for the proposed design and source analysis. No implementation was prepared and no tests were executed for this design proposal.
Source references