Skip to content

feat: trace inline <think> reasoning and time thinking tokens - #39

Draft
milanagm wants to merge 1 commit into
mainfrom
feat/inline-reasoning-ttft
Draft

milanagm wants to merge 1 commit into
mainfrom
feat/inline-reasoning-ttft

Conversation

@milanagm

Copy link
Copy Markdown
Collaborator

Summary

  • Carries the two parts of feat: trace reasoning content and offered tools on generations #37 (by @Ali-Aleph-Alpha) that are genuinely still missing on main. The rest of that PR had already landed independently while it was open: the reasoning-on-output half via feat: carry assistant reasoning on the generation output too #38 (merged 2.5 h after feat: trace reasoning content and offered tools on generations #37 was opened) and the tool definitions via feat: attach the offered tool definitions to every generation #34.
  • Inline <think> reasoning. pi turns a provider's structured reasoning_content delta into a thinking part, but some OpenAI-compatible servers stream the reasoning inline as <think> tags in the text content instead — the vLLM default, the DeepSeek distills, and proxies in front of either. pi keeps that text verbatim (pi@0.84.2 has no occurrence of <think> anywhere in dist), so the reasoning was traced as the answer. splitInlineThinking / extractAnswerAndThinking now split it out and feed it through the existing extractThinking, so inline reasoning travels the same path as structured reasoning. Only a leading tag counts, which keeps an answer that merely mentions a <think> tag — a coding agent explaining one — from being rewritten. Structured parts win, so a model emitting both is not traced twice.
  • Time to first token counts thinking tokens. message_update only set completionStartTime on non-empty extractText(...). A reasoning model streams its thinking before any answer text, so a generation that reasons and then calls a tool without prose got no TTFT at all, and a reason-then-text one reported TTFT ≈ the whole call duration. It now also checks extractThinking(...).

Linear: https://linear.app/clickhouse/issue/LFE-16443/pi-plugin-inline-think-reasoning-is-traced-as-the-answer-and-thinking
Fixes LFE-16443

Verification

Why this is a draft

The inline-<think> split emits {type: "thinking", content}, and that shape currently does not render in the Langfuse trace preview — a regression in the normalized I/O parser, filed as LFE-16438 / langfuse/langfuse#17610. The payload here is correct either way, and TTFT renders today, but the reasoning block only becomes visible once that upstream fix deploys. Keeping this a draft until then so the end-to-end check can be done in the UI rather than only against the API.

Not in this change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant