feat: trace judge calls in Braintrust

[mattrossman]

Motivation

We want to understand how our scorers are judging eval outputs. Today we capture the final output of our judge, but not the input / usage details. We don't know how much $ we spend on judge inference or where inefficiencies are in our judge context (e.g. #366)

Changes

Scorers now ask for verdicts through ctx.judge() instead of plain judge(), which records each judge call's model, full prompt, output, tokens and timing in result.json. The uploader logs each call as an LLM span under the score span:

passed (score, purpose: scorer)
├─ gpt-6-sol (llm, purpose: scorer)
└─ gpt-6-sol (llm, purpose: scorer)

Note

23 of the changed eval files are just swapping judge() for ctx.judge()

purpose: "scorer" is how Braintrust keeps judge cost out of preset cost charts. Core no longer exports judge() so folks are encouraged to use ctx.judge as the preferred way to access the judge in scorers similar to ctx.query.

As an incidental fix, the root eval span no longer logs agent tokens, since its LLM spans carry them. Braintrust sums metrics across every span in a trace, so with the initial approach in #351 the experiment table double counts tokens (example). Note aiSdkAgent has no per-request usage so its rows now show no tokens, but it's unused and slated for removal anyway.

Verification

See Braintrust experiment of the regression suite and this specific sample trace with LLM judge calls now visible under the scorer span.

Timeline	Messages	Span

Tip

Braintrust hides scorer spans by default in the Timeline view, to see them enable "Include score spans in timeline" from the overflow menu:

Future improvements

I'd like to more formally pair checks with ctx.judge() calls so we can attach nice labels to these spans, e.g. judge: wrote safe RLS policy. Today, we don't have that structural pairing so I've left it out of this PR.

Closes AI-1271

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 6, 2026 2:31pm UTC

[gregnr]

LGTM. In an ideal future world, we lean into OTel more so that our traces are provider-agnostic, but I understand we may be losing some BT-specific functionality by doing that. Not blocking this in any case!

[mattrossman]

@gregnr that's an interesting idea, Braintrust does support OTel via most of the GenAI semantic conventions and then the Braintrust-specific stuff it achieves through its own braintrust attributes.

In my testing, the only parts that still require Braintrust-specific API/SDK are:

• Creating experiments (reference)
• Uploading the raw session-archive.tar.gz to a span (reference)

I'll keep that in mind as we expand evals tracing. I don't think we get much benefit migrating to OTel short term but maybe we can consider it when we want the data funneling from/to more places.

[mattrossman]

Tracking that suggestion in AI-1287!