[bot] OpenAI fallback-path instrumentation silently drops multimodal input content (images, audio) from `braintrust.input_json`

[braintrust-bot]

Summary

When using the WithBraintrust() extension method or OpenAITelemetry.Wrap() (the fallback instrumentation path — no BraintrustPipelineTransport injected), braintrust.input_json silently drops all non-text message content parts. Any ChatMessageContentPart created via CreateImagePart() or CreateInputAudioPart() has .Text == null and is filtered out, so multimodal messages record only text portions — or nothing at all if the message contains only images.

What is missing

InstrumentedChatClient.GetTextContent() (lines 385–392) extracts only parts where .Text != null:

private static string? GetTextContent(ChatMessageContent? content)
{
    if (content == null || content.Count == 0) return null;

    var textParts = content.Where(p => p.Text != null).Select(p => p.Text).ToList();
    if (textParts.Count == 0) return null;
    return textParts.Count == 1 ? textParts[0] : string.Join("", textParts);
}

SerializeChatMessage() (lines 309–341) calls GetTextContent() for all message roles except assistant-with-tool-calls and tool results:

// For UserChatMessage, SystemChatMessage, and AssistantChatMessage without tool calls:
return new { role, content = GetTextContent(message.Content) };

The OpenAI .NET SDK v2.9.1 defines the following ChatMessageContentPart factory methods, all of which produce a part with .Text == null:

Factory method	.Text	Captured?
CreateTextPart(string)	non-null	✅
CreateImagePart(Uri, detail)	null	❌
CreateImagePart(BinaryData, mediaType, detail)	null	❌
CreateInputAudioPart(BinaryData, format)	null	❌
CreateRefusalPart(string)	null	❌

For a user message containing only an image (a common vision workflow), GetTextContent returns null and braintrust.input_json records "content": null for that message.

Which path is affected

Path	Multimodal content captured?
BraintrustOpenAI.WrapOpenAI() (primary path — BraintrustPipelineTransport active)	✅ raw HTTP request body includes image URLs or base64 data
client.WithBraintrust() / OpenAITelemetry.Wrap() (fallback path — no transport)	❌ non-text parts filtered by GetTextContent

This is the same asymmetry documented for reasoning_effort in issue #65.

Impact

Vision and multimodal workflows are now mainstream for GPT-4o and similar models. Users on the fallback instrumentation path who send image or audio inputs will have incomplete span inputs — the model's visual context is invisible in Braintrust traces. This prevents:

• Reproducing a call from trace data alone
• Evaluating model responses relative to the actual input content
• Debugging unexpected model outputs for vision tasks

The repo's own example at examples/OpenAIInstrumentation/Program.cs (lines 59–78) demonstrates multimodal image usage as an explicitly supported scenario:

var multimodalMessages = new ChatMessage[]
{
    new UserChatMessage(new[]
    {
        ChatMessageContentPart.CreateTextPart("Summarize what you see in this image."),
        ChatMessageContentPart.CreateImagePart(BinaryData.FromBytes(demoImageBytes), "image/png")
    })
};

The image part in this example would be silently dropped from braintrust.input_json on the fallback path.

Braintrust docs status

The Braintrust OpenAI integration page explicitly lists "Multimodal content" as a supported tracing capability. Status: supported (documented as a supported feature; silently dropped for non-text content parts in the fallback instrumentation path).

Upstream sources

• OpenAI Vision guide: https://platform.openai.com/docs/guides/vision — ChatMessageContentPart.CreateImagePart() for image inputs
• OpenAI Audio inputs: https://platform.openai.com/docs/guides/audio — ChatMessageContentPart.CreateInputAudioPart() for audio inputs
• OpenAI .NET SDK v2.9.1: OpenAI.Chat.ChatMessageContentPart — factory methods for image and audio parts with .Text == null

Local files inspected

• src/Braintrust.Sdk.OpenAI/InstrumentedChatClient.cs — GetTextContent() (lines 385–392): filters p.Text != null, dropping image and audio parts; SerializeChatMessage() (lines 309–341): calls GetTextContent() for all non-tool-call message roles
• src/Braintrust.Sdk.OpenAI/BraintrustPipelineTransport.cs — primary path captures raw HTTP body including all content; not affected
• src/Braintrust.Sdk.OpenAI/BraintrustOpenAI.cs — documents both instrumentation paths
• src/Braintrust.Sdk.OpenAI/Braintrust.Sdk.OpenAI.csproj — confirms OpenAI SDK v2.9.1 dependency
• examples/OpenAIInstrumentation/Program.cs — multimodal image example at lines 59–78