Skip to content

feat: attach the offered tool definitions to every generation - #34

Open
milanagm wants to merge 2 commits into
mainfrom
feat/tool-definitions-on-generations
Open

milanagm wants to merge 2 commits into
mainfrom
feat/tool-definitions-on-generations

Conversation

@milanagm

@milanagm milanagm commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Closes #23.
LFE-16207

Change

A trace recorded which tools the model calledtool_calls on the generation output, Tool: <name> spans — but not which tools it was offered, so it could not distinguish "not offered" from "not chosen", and the Langfuse playground had no tools to replay a generation with.

Each generation input now carries the definitions of the tools in the request as tools on its first message, which is where the Langfuse UI reads them and dedupes by name.

The set is pi's active tools, which is narrower than the registry. Measured on pi 0.84.2 with a probe on before_provider_request, reading event.payload (the real wire request):

registry (getAllTools):  read, bash, powershell, edit, write, grep, find, ls   ← 8
active  (getActiveTools): read, bash, edit, write                              ← 4
wire    (payload.tools):  read, bash, edit, write                              ← 4

So getAllTools() alone would over-report by four tools. The active names drive both membership and order, since that is what goes on the wire; the registry only supplies each schema. Both accessors are optional-chained and the helper never throws, so a pi without them degrades to no tools.

The generation input is now a ChatML message array

This is the part that makes the feature actually visible, and it is a fix in its own right.

The generation input was a bare message object. Langfuse normalizes generation input with ChatMlArraySchema, and its single-message fallback (langfuse#16565) only accepts a message carrying content or tool_calls. pi's generations after a tool call are {role: "tool", tool_results: [...]}, which match neither — so they fail to parse, fall back to the raw JSON view, and drop the tool definitions entirely.

Verified by replaying the real captured payloads of a three-generation turn through Langfuse's own normalizer at current main, plus the Tools-section logic from useChatMLParser:

generation main #26 this PR
0 — role: "user" chat ✔, no tools chat ✔, tools ✔ chat ✔, tools ✔
1 — role: "tool" chat ✘, raw JSON chat ✘, raw JSON chat ✔, tools ✔
2 — role: "tool" chat ✘, raw JSON chat ✘, raw JSON chat ✔, tools ✔

Wrapping the message in a one-element array fixes all three, and keeps the input composable with any later change that turns it into a real message list.

Tests

  • pnpm typecheck clean; pnpm test 67 of 67 passed, up from 62 on main.
  • Five tests: the two helpers (first-message attach with no mutation, pass-through when there is nothing to attach; active-only narrowing with order and ChatML-field projection; graceful degradation on a pi without the accessors or with throwing ones), plus one integration test that reads the exported generation inputs of a real sandbox run and asserts each is a message array carrying ["read", "bash", "edit", "write"] with their schemas.
  • Reverting only the array wrapping, keeping everything else, turns the integration test red with `generation 0 input must be an array of messages

@milanagm
milanagm requested a review from hassiebp September 11, 2026 10:33
@milanagm
milanagm force-pushed the feat/tool-definitions-on-generations branch from 5e9dc03 to 360228a Compare September 11, 2026 10:42

@hassiebp hassiebp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after resolved conflicts 👍

@milanagm
milanagm force-pushed the feat/tool-definitions-on-generations branch from 360228a to 39e3d09 Compare September 15, 2026 11:20

Copy link
Copy Markdown
Collaborator Author

@hassiebp resolved

Ali-Aleph-Alpha added a commit to Ali-Aleph-Alpha/pi-observability-plugin that referenced this pull request Sep 15, 2026
Generation output now carries the model's reasoning as Langfuse thinking
blocks next to the answer text:

- structured thinking blocks from pi's message content are included in
  the generation output (previously only in the history input)
- reasoning streamed inline as <think> tags (vLLM default, DeepSeek
  distill, and other OpenAI-compatible servers that do not send a
  structured reasoning field) is split out of the answer text; an
  unclosed tag is treated as reasoning until the end of the stream
- structured blocks win over inline tags so a model emitting both is
  not traced twice
- thinking content goes through the secret redactor, since models
  regularly quote file contents (including .env) into their reasoning
- thinkingSignature stays untraced (provider replay data)

Time to first token now counts thinking tokens, which reasoning models
stream before any text.

Each generation also records the tool definitions the model was called
with (name, description, parameter schema) as available_tools metadata,
and the turn root lists the active tool names, so a trace can
distinguish 'tool not offered' from 'tool not chosen'.

Note: the available_tools part overlaps with langfuse#34 (which attaches tool
definitions to the generation input for playground replay). Happy to
drop it in favor of langfuse#34 and keep only the reasoning changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Attach tool definitions to generations

2 participants