[MEAI] Support OpenTelemetry GenAI semantic convention version selection

[rogerbarreto]

Background

Microsoft.Extensions.AI currently emits the latest GenAI semantic conventions supported by the package. Version 10.9.0 emits the v1.41 representation. It does not read OTEL_SEMCONV_STABILITY_OPT_IN.

OpenTelemetry introduced OTEL_SEMCONV_STABILITY_OPT_IN in the v1.37 GenAI documentation for instrumentations that were already emitting v1.36 or earlier:

• A list containing gen_ai_latest_experimental selects the latest experimental GenAI conventions supported by the instrumentation.
• A list that does not contain that token retains the v1.36 representation.
• The upstream recommendation also retains v1.36 when the variable is not defined.

Microsoft.Extensions.AI migrated from v1.36 to v1.37 in #6767 and has since advanced through v1.41. Reverting to v1.36 whenever the variable is absent would now change the default telemetry contract for existing users.

Microsoft Agent Framework depends on OpenTelemetryChatClient for agent, chat, and tool execution telemetry. microsoft/agent-framework#7673 is adding the same version selection to the Python implementation. The .NET implementation cannot provide equivalent behavior without support from Microsoft.Extensions.AI.

The current draft #7642 tracks newer experimental convention changes. It does not add version selection.

Current behavior

Configuration	Microsoft.Extensions.AI 10.9.0
Variable absent	Latest experimental representation
gen_ai_latest_experimental present	Latest experimental representation
Variable defined without the token	Latest experimental representation

The package therefore cannot emit a consistent v1.36 representation for users and frameworks that need that compatibility contract.

Proposed behavior

Implement both the v1.36 compatibility representation and the latest experimental representation. Keep the decision about an absent environment variable isolated from the rest of the implementation.

Configuration	Recommended non-breaking behavior	Literal upstream alternative
Variable absent	Preserve the current latest experimental representation	Use the v1.36 compatibility representation
gen_ai_latest_experimental present	Use the latest experimental representation	Use the latest experimental representation
Variable defined without the token	Use the v1.36 compatibility representation	Use the v1.36 compatibility representation
Explicit API configuration	Use the explicitly selected representation	Use the explicitly selected representation

The recommended behavior matches microsoft/agent-framework#7673. It preserves the current default while allowing users to select v1.36 by defining OTEL_SEMCONV_STABILITY_OPT_IN without the GenAI token.

The alternative follows the v1.37 OpenTelemetry transition text literally, but changes the default telemetry emitted by current Microsoft.Extensions.AI applications. That option should be treated as a breaking behavioral change.

Representation requirements

v1.36 compatibility mode

• Emit gen_ai.system on applicable spans and metrics.
• Emit message content through the structured events gen_ai.system.message, gen_ai.user.message, gen_ai.assistant.message, gen_ai.tool.message, and gen_ai.choice.
• Omit attributes, metrics, and values introduced after v1.36.
• Preserve the v1.36 event body formats for text, roles, function calls, and function results.

Latest experimental mode

• Preserve the current gen_ai.provider.name behavior.
• Preserve gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.
• Continue tracking the latest experimental conventions supported by the package.
• Do not emit the old v1.36 message events by default.

Both modes

• Continue using OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to control sensitive content.
• Never emit both provider attribute names or both message representations for one operation.
• Apply one resolved mode consistently to spans, metrics, message content, and tool execution.
• Resolve the mode once per instrumentation instance so an environment change cannot mix representations during that instance's lifetime.

The v1.36 GenAI conventions also have Development status. The implementation and documentation should call this a v1.36 compatibility mode rather than a stable mode.

Proposed API direction

The exact public API needs owner review. A local prototype validates the following shape:

1. A way to select v1.36 compatibility or latest experimental explicitly for an instrumentation instance.
2. A public, read-only way for wrappers to inspect the resolved mode.
3. Clear precedence where explicit API configuration overrides the environment variable.

namespace Microsoft.Extensions.AI;

public enum OpenTelemetryGenAISemanticConvention
{
    Version1_36,
    LatestExperimental,
}

public sealed partial class OpenTelemetryChatClient
{
    public OpenTelemetryGenAISemanticConvention SemanticConvention { get; set; }
}

The enum and properties are marked experimental. The property is initialized from the environment when the client is created. The builder callback can override it before the client is used. Wrappers such as Microsoft Agent Framework can read the final value and apply the same provider attribute name to their agent spans. Future OpenTelemetry selections can be represented by additional enum values without overloading the meaning of Version1_36.

Alternative API shapes are acceptable if they support explicit selection and expose the resolved value without requiring reflection.

Initial implementation scope

The first change should cover the components that existed during the v1.36 to v1.37 migration and directly affect Microsoft Agent Framework:

• OpenTelemetryChatClient
• FunctionInvokingChatClient and FunctionInvocationProcessor
• OpenTelemetryEmbeddingGenerator
• Shared constants, serializers, and metric helpers used by those components

Image, audio, hosted file, and realtime instrumentation were added or expanded after v1.36. We should decide whether they remain latest experimental only or whether this work should define a compatibility policy for all GenAI instrumentation in one change.

Validation

Tests should cover:

• An absent variable
• An empty variable
• A comma-separated list containing gen_ai_latest_experimental
• A list without the GenAI token
• Whitespace and unrelated category tokens
• Explicit API precedence over the environment
• Sensitive content enabled and disabled
• Non-streaming and streaming chat
• Success, failure, and cancellation
• Tool success and tool failure
• Chat and embedding spans and metrics
• Structured v1.36 event bodies
• Presence of fields from the selected mode
• Absence of fields from the other mode
• No duplicate spans, events, metrics, or attributes

The local prototype passes the complete filtered AI build and test flow across the repository's supported test frameworks.

Open questions

1. When OTEL_SEMCONV_STABILITY_OPT_IN is absent, should the package preserve its current latest experimental behavior or return to v1.36 as described by the upstream transition guidance?
2. What public API shape should select and expose the resolved mode?
3. Should the first implementation cover only chat, tool invocation, and embeddings, or all current GenAI instrumentation?
4. Which package version and release channel can make this available to Microsoft Agent Framework?

Related work

• Python: Consolidate OTel GenAI Semantic Conventions verisioning microsoft/agent-framework#7657
• [BREAKING] Python: Consolidate OTel GenAI Semantic Conventions versions microsoft/agent-framework#7673
• Update OpenTelemetry semantic conventions version from 1.35 to 1.36 #6579
• Update to genai standard convention 1.37 #6767
• [MEAI] Update Aspire to recognize telemetry sources from both Experimental.Microsoft.Extensions.AI and Microsoft.Extensions.AI #6736
• Update OpenTelemetry gen-ai conventions through v1.41 #7497
• Update open-telemetry/semantic-conventions-genai to latest #7642
• OpenTelemetry v1.37 GenAI transition guidance

[rogerbarreto]

Closing this issue for now as it is not clear if this OTEL_SEMCONV_STABILITY_OPT_IN flag is really something we strive to support. Will come back to this issue if it turns out to be necessary.