Ark/bra 3203 otel attachments#38
Closed
Andrew Kent (realark) wants to merge 2 commits into
Closed
Conversation
Andrew Kent (realark)
force-pushed
the
ark/BRA-3203-otel-attachments
branch
from
84d9301 to
2078b44
Compare
- Most SDK users won't directly handle attachments because the instrumentation will do the right thing for them
- The SDK will provide an `Attachment` util which serializes attachments in a way Braintrust will recognize
- This util is used to write instrumentation or fill in missing gaps for custom situations
- The SDK will send the full attachment base64-encoded and the backend will upload the file to s3 and replace the message with a pointer to s3
- In the future, we can have the SDK do upload+conversion without any breaking changes
- Provide an `Attachment` util which:
- has convenience functions for creating base64 data
- has a json serializer idiomatic to the SDK's language
- Use this utility to replace vendor-specific attachment messages with Braintrust attachment messages
Attachment Examples
```
Attachment attachment = Attachment.ofFile(Attachment.ContentType.IMAGE_JPEG, testFile.toString());
JsonSerializer<Attachment> jacksonSerializer = Attachment.createSerializer();
```
And the instrumentation will use this util to map vendor-specific message parts. For example, OAI image_url:
```
{
"image_url": {
"url": "data:image/jpeg;base64,SOME_BASE64",
"detail": "high",
"valid": true
},
"type": "image_url",
"valid": true
}
```
--->
`{ type: "base64_attachment", content: "data:image/jpeg;base64,SOME_BASE64" }`
The genai semconv does not actually define a format for attachments or images, so we have to map vendor-specific formats to something Braintrust can understand. We do this by defining our own GenericPart message as defined in the otel input/output message json schemas
- https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/#gen-ai-input-messages
- https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-input-messages.json
- https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-output-messages.json
Braintrust recognizes three attachment types.
- raw base64 data: `{ type: "base64_attachment", content: "SOME_BASE64" }`
- stored in Braintrust s3: `{ type: "braintrust_attachment", filename, key, content_type }`
- stored external to Braintrust: `{ type: "inline_attachment", filename, src, content_type }`
Currently, the SDK will send `base64_attachment` and the backend will convert it to `braintrust_attachment`
In the future, the SDK may do this conversion before sending traces. If we decide to go that route, this will _not _require any code changes for SDK users. It will happen somewhere in the implementation (probably a span processor).
Andrew Kent (realark)
force-pushed
the
ark/BRA-3203-otel-attachments
branch
from
2078b44 to
5530bd7
Compare
Contributor
Author
|
This work will be merged into the new repo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Braintrust-friendly attachments, conforming to otel genai semconv
tl;dr
Attachmentutil which serializes attachments in a way Braintrust will recognizeto implement otel Support in an SDK
Attachmentutil which:Attachment Examples
And the instrumentation will use this util to map vendor-specific message parts. For example, OAI image_url:
--->
{ type: "base64_attachment", content: "data:image/jpeg;base64,SOME_BASE64" }details
Otel genai semconv does not actually define a format for attachments or images, so we have to map vendor-specific formats to something Braintrust can understand. We do this with our own GenericPart message as defined in the otel input/output message json schemas
Braintrust recognizes three attachment types.
{ type: "base64_attachment", content: "SOME_BASE64" }{ type: "braintrust_attachment", filename, key, content_type }{ type: "inline_attachment", filename, src, content_type }Currently, the SDK will send
base64_attachmentand the backend will convert it tobraintrust_attachmentIn the future, the SDK may do this conversion before sending traces. If we decide to go that route, this will not require any code changes for SDK users. It will happen somewhere in the implementation (probably a span processor).