[SushantGautam]

Found while running real multi-turn traced audits against agent 1 (Support Refund Assistant) on a live model.

Agents keep the conversation across turns

Traced agent runs go through Open WebUI's saved-chat path, which builds the model input from the chat Open WebUI stores. Every turn created a new chat holding only the latest user message, so from turn 2 the agent answered without the conversation. This was confirmed in Open WebUI's database and in the model-call spans.

Each later turn now continues the scenario's chat (chat_id, plus the previous reply as parent_id), keyed by model, W3C trace id and the turns so far. Untraced runs were never affected: they send the full history themselves.

Native knowledge search in Open WebUI

acme_search_knowledge always failed inside Open WebUI ("Docs directory not found"), and the agent then fell back to Open WebUI's own knowledge search. It is removed from the tool pushed to Open WebUI, and the seeded agent prompt asks for the attached knowledge base. The keyword search stays as Studio's local stand-in for untraced runs (infra/fixtures/local_knowledge.py).

Trace content stored once per trace (audits/agentic/compact.py)

GenAI spans repeat the system prompt, the history and earlier tool results on every model call. Studio now stores trace evidence in the same reference format as the Open WebUI fork's compact mode, whichever target sent the spans:

• a repeated message becomes {role, parts: [{type: reference, span_id, attribute, index}]};
• a repeated value becomes {type: reference, …};
• knowledge chunks are kept once, as retrieval documents.

References are expanded before rules and judges read the spans. On 1-turn runs this stores about a third less, and much less on multi-turn runs.

Judges see each piece once, from the trace

• Main judge: reads selected spans through the SimpleAudit engine, which caps each span at 2,000 characters. It now gets a judge view:

  • only gen_ai.* attributes and the error fields (error.type, exception.*);
  • no workflow wrappers;
  • repeats shown as "(same as span …)" notes, and Open WebUI's RAG template dropped from user messages;
  • messages already in the conversation pointed to it;
  • decisions, results and documents placed before the input history.

  On a 5-turn run the judge's input tokens fell 28%, even though the agent now sees full history, and the spans cut by the length limit fell from 10 to 1.

• Semantic agentic judge:

  • whole answers (the 800-character cap cut off conclusions);
  • every turn as the trace records it, with no Studio transcript;
  • no duplicated final answer or RAG template;
  • retrieval source names instead of whole document JSON.

Trajectory timeline

• Repeats are shown as "↩ same as LLM-1 · …".
• Retrieval steps show their query and retrieved documents, and JSON blocks are indented.
• Failed steps get an error badge and an "Error" block ("ValueError: No order found …").
• Model calls are tagged from GenAI output only: amber tool call or green reply. Every turn's reply is highlighted, not only the last model call of the trace.
• 👤 marks the call that received the user's turn.
• Distinct icons: 🧠 tool call, 💬 reply, 🔧 tool run.
• The "agent → tool" rows now read GenAI tool_call parts. They used to read OpenAI's tool_calls, so they never showed on GenAI traces.

Demo scenarios work in multi-turn runs

• D09 depended on SIMPLEAUDIT_DEMO_FAIL_FIRST, which nothing set and which failed only once per machine. It also capped tracking calls at 1–2 over the whole conversation. It now tests reporting tracking status only as observed, and the fail-first hook is removed.
• D02 drops its "at most 1 error step" cap.
• No count rule with an upper bound remains in the pack.

Open WebUI v0.12.4.9-subpath

• 0.12.4.8: agentic knowledge search traced as a GenAI retrieval span (fork Merge dev into main: agent config domain, OTLP tracing, Open WebUI chat #9).
• 0.12.4.9: compact GenAI content by default (fork feat: SIMPLEAUDIT_SQLITE_PATH for local dev DB; clean .env.local.example #10).

Checks

• uv run pytest -m "not embedded_hatchet": 1626 passed, 3 skipped
• uv run ruff check .: clean
• makemigrations --check: no changes
• Live traced audits on Open WebUI 0.12.4.9 (1 and 5 turns), 10 of 10 scenarios each. Run 38 (5 turns): every scenario pass.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.