.NET: Workflow Middleware to Support Customizing Telemetry

[skylarbpayne]

Hey folks,

Love a lot of the functionality of Agent Framework. Right now, I see a lot of use cases we have that could benefit from Workflows. However, I would like to customize the spans, particularly to include inputs/outputs. This makes it easier to construct evals from traces for compound AI systems.

Is there some way to do that today?

[moonbox3]

Hi Skylar Payne (@skylarbpayne),

Thanks for the interest in Agent Framework workflows. There are several ways to customize telemetry today:

Current Approaches

1. Agent-Level Middleware for Input/Output Telemetry

For agents within workflows (or standalone), you can use middleware to intercept invocations and enrich spans with inputs/outputs:

from agent_framework import AgentMiddleware, AgentRunContext
from opentelemetry import trace

class TelemetryMiddleware(AgentMiddleware):
    async def process(self, context: AgentRunContext, next):
        span = trace.get_current_span()
        # Record inputs
        span.set_attribute("agent.input", str(context.messages[-1].text) if context.messages else "")
        
        await next(context)
        
        # Record outputs
        if context.result:
            span.set_attribute("agent.output", str(context.result.text))

See the middleware samples for patterns including class_based_middleware.py and function_based_middleware.py.

2. Custom Spans Within Executors

Inside any executor handler, you can create custom spans with inputs/outputs:

from agent_framework import Executor, WorkflowContext, handler
from agent_framework.observability import get_tracer

class MyExecutor(Executor):
    @handler
    async def process(self, message: str, ctx: WorkflowContext[str]) -> None:
        tracer = get_tracer()
        with tracer.start_as_current_span("my_custom_operation") as span:
            span.set_attribute("input", message)
            result = do_something(message)
            span.set_attribute("output", result)
            await ctx.send_message(result)

3. Workflow-Level Observability

Workflows automatically emit spans for:

• workflow.run - overall workflow execution
• executor.process {executor_id} - each executor invocation
• message.send - messages between executors

You can access the current span context and add custom attributes/events within executor handlers.

What's Not Available Today

There's currently no dedicated workflow-level middleware hook that intercepts all executor invocations in a centralized way to add custom span attributes. The telemetry enrichment happens either:

• At the agent level (via AgentMiddleware)
• At the chat client level (via ChatMiddleware)
• Within individual executors (manual span creation)

If you need a workflow-scoped middleware that can:

• Intercept every executor invocation
• Access inputs/outputs for all executors uniformly
• Enrich spans with custom attributes for evals

This would be a new feature. Could you share more about your specific use case? For example:

• Are you primarily using WorkflowAgent (agent executors) or custom Executor classes?
• What specific span attributes would be most valuable for your eval construction?

This would help us understand if the existing middleware patterns can meet your needs or if we should consider a workflow-level middleware abstraction.

[skylarbpayne]

When building complex agentic systems, it's helpful to be able to inspect "what happened" at multiple levels. As I "zoom in" (ie looking at a single executor rather than the whole workflow), I need to know what was passed to that executor to meaningfully understand it.

By capture executor level inputs/outputs you also enable a clearer path to constructing evals around different scales of the system (ie at workflow => executor => agent levels). This enables testability which in turn provides quick feedback loops for testing new implementations of a particular component (e.g. what if I want to swap an 'agent' / llm call with a simple classifier and want to evaluate quality / cost trade-off?)

Maybe a more concrete example will help. Let's say we want to build a voice agentLet's say we have a workflow like the following to run customer support chatbot(s):

1. Given company info about brand voice, desired tone/personality, we dynamically generate a personality prompt section
2. Given what information we already have about the user, we dynamically generate a personalized greeting
3. Given the user's query we retrieve potentially relevant help documents or former cases that might be related
4. All this goes into the main chatbot, which also can search directly for relevant help docs / cases through agentic tool use. through the conversation

I want to build evals around everything here: each individual step, as well as the whole thing.

I wouldn't want to have to explicitly implement this for every single workflow/executor.

For what it's worth, this is "standard out of the box" with most other frameworks that support graphs/workflows. E.g. see LangGraph traces in LangSmith.

[moonbox3]

Thanks for the detailed explanation, Skylar Payne (@skylarbpayne), this helps clarify the need.

To make sure I understand what you're asking for:

Automatic span enrichment with:

• Input: The message payload passed to each executor
• Output: What the executor sent downstream (via ctx.send_message) or yielded (via ctx.yield_output)

This would enable:

1. Full trace inspection at any zoom level without manual instrumentation
2. Eval construction at workflow, executor, and agent granularity
3. A/B testing component implementations (agent vs classifier, etc.)

We should add opt-in automatic input/output capture to executor spans. Something like:

# At workflow level
workflow = (
    WorkflowBuilder()
    .add_edge(executor1, executor2)
    .with_telemetry(capture_inputs=True, capture_outputs=True)  # opt-in
    .build()
)

Or via observability settings:

from agent_framework.observability import OBSERVABILITY_SETTINGS

OBSERVABILITY_SETTINGS.CAPTURE_EXECUTOR_IO = True

The implementation would:

1. Serialize inputs before handler invocation and add as span attribute
2. Track send_message/yield_output calls and add outputs as span attributes
3. Support configurable serialization (for custom types, redaction, size limits)

Does this capture what you're looking for? If so, we will track this feature request. Any preferences on the API shape (workflow-level config vs global setting)?

[skylarbpayne]

That captures most of it, though I would prefer something that lets me add attributes to the span myself (eg middleware). The reason is that Executors may be parametrized in someway and I might want to capture that in the span.

But capturing inputs/outputs works is a good 80% solution (and I can always stuff the parameters into the input). So yes, I think this captures what I'm looking for.

I think global with workflow level overrides: the reality is that the common case for me will be "all track input/output", but I can also imagine there might be some particularly sensitive use case where we explicitly do not want to track it.

(Also, fwiw I'm using dotnet!)

[moonbox3]

Peter Ibekwe (@peibekwe) any thoughts on how this would be handled in .NET? Thanks.