Braintrust Java SDK + LangChain4j Example

This repository demonstrates how to properly configure the Braintrust Java SDK with LangChain4j to ensure traces appear in Braintrust.

The Problem

When using Braintrust Java SDK with LangChain4j, you may encounter:

1. Traces not appearing in Braintrust - The JVM exits before traces are flushed
2. Traces going to wrong project - Using BRAINTRUST_PROJECT_ID instead of BRAINTRUST_DEFAULT_PROJECT_ID

The Solution

This example demonstrates the critical fix: explicitly shutting down OpenTelemetry to flush traces before the JVM exits. This is especially important when using Maven's exec:java plugin.

What's Included

This repo includes a comprehensive multi-turn conversation example with tools and RAG, demonstrating:

• Multi-turn conversations with conversation memory
• Tool execution (weather, calculator, knowledge base search)
• RAG (Retrieval-Augmented Generation) integration with semantic search
• Metadata tracking and scoring across conversation turns

Examples Included

1. MultiTurnWithToolsExample.java - Complete multi-turn conversation with tools and RAG
2. MultiTurnConversationExample.java - Simpler multi-turn conversation without tools
3. Supporting Classes:
   • Tools.java - Tool implementations (weather, calculator, knowledge base)
   • RAGService.java - In-memory vector store with semantic search

Prerequisites

• Java 17 or higher
• Maven 3.6+
• OpenAI API key
• Braintrust API key and project

Setup

1. Clone the repository

git clone <your-repo-url>
cd braintrust-java-langchain4j-example

2. Set environment variables

IMPORTANT: Use BRAINTRUST_DEFAULT_PROJECT_ID (not BRAINTRUST_PROJECT_ID)

export OPENAI_API_KEY="your-openai-api-key"
export BRAINTRUST_API_KEY="your-braintrust-api-key"
export BRAINTRUST_DEFAULT_PROJECT_ID="your-project-id"

3. Install dependencies

mvn clean install

Running the Example

Option 1: Using the Python runner (loads .env automatically)

python3 run.py

Option 2: Using Maven directly

# Set environment variables first
export OPENAI_API_KEY="your-openai-api-key"
export BRAINTRUST_API_KEY="your-braintrust-api-key"
export BRAINTRUST_DEFAULT_PROJECT_ID="your-project-id"

mvn clean compile exec:java

You should see output like:

Braintrust project URI: https://www.braintrust.dev/app/Braintrust%20Demos/p/default-java-project

=== Multi-Turn Conversation with Tools (ID: 9a36872b) ===

User (Turn 1): What's the weather like in San Francisco?
Assistant: The weather in San Francisco is sunny with a temperature of 72°F...

User (Turn 2): Can you calculate the average of these numbers: 10, 20, 30, 40, 50?
Assistant: The average of the numbers 10, 20, 30, 40, and 50 is 30.0.

User (Turn 3): What is RAG and how does it work?
Assistant: RAG, or Retrieval-Augmented Generation, is a framework...

User (Turn 4): Can you summarize what we've discussed so far?
Assistant: Certainly! Here's a summary of our discussion...

✅ Conversation complete! View your data in Braintrust: https://www.braintrust.dev/app/...

What You'll See in Braintrust

The example creates a hierarchical trace structure:

conversation_9a36872b (task)
├── conversation_turn_1 (task)
│   ├── get_weather (tool)
│   └── chat.completions (llm)
├── conversation_turn_2 (task)
│   ├── calculate_metrics (tool)
│   └── chat.completions (llm)
├── conversation_turn_3 (task)
│   ├── search_knowledge_base (tool)
│   │   └── rag_search (tool)
│   └── chat.completions (llm)
└── conversation_turn_4 (task)
    └── chat.completions (llm)

Key Configuration Fixes

1. Explicit OpenTelemetry Shutdown (Critical!)

🔥 This is the critical fix for traces not appearing: The example explicitly calls ((OpenTelemetrySdk) openTelemetry).close() before exiting.

This ensures all traces are flushed to Braintrust before the JVM exits. Without this, Maven's exec:java plugin tears down the classloader before the shutdown hook can complete, causing traces to be lost.

// At the end of main()
if (openTelemetry instanceof OpenTelemetrySdk) {
    ((OpenTelemetrySdk) openTelemetry).close();
}

2. Environment Variable Name

⚠️ Use BRAINTRUST_DEFAULT_PROJECT_ID instead of BRAINTRUST_PROJECT_ID

The Java SDK uses BRAINTRUST_DEFAULT_PROJECT_ID to set the project context. Using the wrong variable name will cause traces to default to default-java-project.

3. Simplified Dependencies

The example uses minimal dependencies:

• braintrust-sdk-java v0.2.1 (includes OpenTelemetry automatically)
• langchain4j v1.9.1
• langchain4j-open-ai v1.9.1
• slf4j-simple v2.0.17 (for logging)

Note: You don't need to explicitly declare OpenTelemetry dependencies - the Braintrust SDK includes them. Adding them manually can cause version conflicts.

4. Span Hierarchy with makeCurrent()

To nest child spans under parent spans, use makeCurrent():

Span conversationSpan = tracer.spanBuilder("conversation").startSpan();
try (Scope ignored = conversationSpan.makeCurrent()) {
    // All spans created here will automatically nest under conversationSpan
    processTurn(...); // Creates turn spans as children
} finally {
    conversationSpan.end();
}

Project Structure

.
├── pom.xml
├── README.md
├── SLACK_RESPONSE.md
├── run.py
├── .env
└── src/main/java/com/braintrust/example/
    ├── MultiTurnWithToolsExample.java    # Main example with tools & RAG
    ├── MultiTurnConversationExample.java # Simpler multi-turn example
    ├── Tools.java                        # Tool implementations
    └── RAGService.java                   # In-memory RAG service

Features Demonstrated

Multi-Turn Conversation

• Conversation memory across turns
• Parent span grouping all turns
• Child turn spans with proper nesting

Tool Execution

1. Weather Tool - Mock external API call
2. Calculator Tool - Statistical metrics computation
3. Knowledge Base Search - RAG-powered semantic search

RAG Integration

• In-memory vector store (production would use ChromaDB/Pinecone)
• Simple keyword-based similarity (production would use embeddings)
• Nested tool spans (search_knowledge_base → rag_search)

Metadata & Scoring

• Tool call paths tracked in metadata
• Online scoring (response_quality, tool_usage, overall)
• Scores attached to turn spans for analysis

Troubleshooting

Issue 1: Traces not appearing in Braintrust (Most Common!)

Symptom: Your code runs successfully but traces never show up in Braintrust

Solution: Add explicit OpenTelemetry shutdown:

// At the end of main()
if (openTelemetry instanceof OpenTelemetrySdk) {
    ((OpenTelemetrySdk) openTelemetry).close();
}

Why: Maven's exec:java plugin tears down the classloader before the shutdown hook can complete, causing traces to be lost.

Issue 2: Traces going to default-java-project

Symptom: Traces appear in the wrong project

Solution: Use BRAINTRUST_DEFAULT_PROJECT_ID (not BRAINTRUST_PROJECT_ID):

export BRAINTRUST_DEFAULT_PROJECT_ID="your-project-id"

Resources

• Braintrust Java SDK
• LangChain4j Documentation
• Braintrust Documentation

Support

For issues or questions:

• Braintrust SDK: GitHub Issues
• LangChain4j: GitHub Issues