fix(cli): run trace-only evaluations through the judge pipeline - #345
Draft
liamcrumm wants to merge 8 commits into
Draft
fix(cli): run trace-only evaluations through the judge pipeline#345liamcrumm wants to merge 8 commits into
liamcrumm wants to merge 8 commits into
Conversation
Import OTLP conversations into a new run with stable case IDs, source provenance, and preserved request/tool evidence. Reuse the normal judge and viewer paths without executing a target. Keep parse-only conversion explicit and fail incomplete evaluations without treating them as passes. Reuse the inference-row fingerprint helper from ASSERT PR responsibleai#308 without its Langfuse integration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Gate scoring on reconstructed evidence rather than raw output fields. Retain chain/agent outputs in completion order and deduplicate only explicit ancestor mirrors. Reconstruct structured historical tool calls and correlate receipts with captured actions by identity, including indexed OpenInference messages. Add regressions for both deep-review blockers, root/child ordering, historical call formats, nested histories, and reused call IDs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Normalize GenAI tool_call_response parts and ensure OpenInference agent outputs use one emission path. Process completed tools before tied snapshot inputs, preserve assistant text before actions, and place newly recovered historical context before its matched captured evidence. Cover each reproduced delta-review finding, zero-duration spans, ancestor mirrors, and mixed captured/history context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Order equal-time events using own-span, parent-span, and compatible history-extension dependencies rather than phase priorities alone. Keep recovered context before every matched parallel action and restore receipt correlations across common history prefixes. Add regressions for reversed exports, zero-duration model/tool responses, parent-child ties, parallel completion order, repeated receipt conflicts across traces, and noncausal identical outputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Prefer a matching newly captured occurrence over common-prefix reuse while reserving captures for new suffix requests. Preserve receipt-conflict checks when no fresh execution exists. Compare conversation prefixes independently of system-instruction updates so prior actions are not duplicated. Cover fresh resets and continuing histories across trace IDs, reused call IDs, equal and different receipts, and changed system instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Use actual pending-match state instead of expiring captures at every input snapshot. Unrelated CHAIN, AGENT, or LLM inputs cannot invalidate a captured execution that has not yet been observed in history. Keep suffix reservation and conflicting-receipt checks unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Plan capture assignments for unknown requests before repeated observations. Prefer exact recorded receipts over incomplete candidates, retain incompatible older executions, and keep known-observation context handling separate from new-request reservations. Correct the parse-only follow-up hint to use judge-traces for scored runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
Allocate compatible matches before incompatible fallbacks and protect captures reserved for other requests. Preserve identified history prefixes across unrelated wrappers and trace IDs. Repeated observations may move a captured binding only to a later completed capture with the matching receipt, never to an older or incomplete action. Add regressions for compatible prefix/history-only suffix allocation, repeated receipt conservation, and cross-trace wrapper replays. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6091e46-c1a4-40ca-b207-8f063de2d64b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
judge-tracesnow scores exported OTLP JSON traces through the existing judge pipeline. It writes scores, run metadata, and viewer artifacts without invoking the target or generating a taxonomy.The command requires an enabled judge, provider credentials, and an existing taxonomy. Use
--parse-onlyfor the previous conversion-only behavior without model calls.Changes
Validation
The tests use synthetic traces and a deterministic judge substitute. They cover tool-only disclosure, harmless internal retrieval, blocked attempts, over-refusal context, missing evidence, provider errors, artifact joins, and output preservation. Regression cases also cover root-only outputs, structured history and receipts without earlier spans, parent/child mirrors, repeated call IDs, and message/action ordering. No production traces, live agents, or paid judge calls were used.
Scope
The change is limited to local trace import and judging. It does not add a hosted service, trace-store connector, attack engine, or Foundry publishing path. A recorded tool request alone does not prove that the tool executed.