Skip to content

Record model parameters (max tokens, thinking level) on generations #22

Description

@wierdbytes

Problem

pi-observability-plugin does not record any model parameters on generations, so Langfuse cannot show how a model was called — only which model.

Verified in src/index.ts (v0.1.2):

  • the generation created in before_provider_request gets model: ctx.model?.id and metadata.provider, nothing else — modelParameters is never set
  • rg "modelParameters|thinkingLevel|maxTokens" over src/ returns 0 matches
  • stop_reason is recorded on message_end, but the cap that produced a stop_reason: length is not

This hurts when:

  • comparing runs of the same prompt at different thinking levels (/thinking high vs low) — the traces are indistinguishable
  • diagnosing truncated outputs — you see stop_reason: length but not the max_tokens that was in effect
  • reviewing traces later, once the pi settings that produced them have changed

Both values are already on the extension context, and pi resolves every request from the same two fields:

  • max output tokens come from model.maxTokens (clamped to the remaining context); the coding agent never overrides it per request, so ctx.model.maxTokens is what goes out
  • reasoning comes from the thinking level, or nothing when it is off; each provider maps that level to its own effort/budget field
  • temperature and other sampling params are only sent when a caller sets them explicitly, which the coding agent does not

Expected

Every generation carries modelParameters with max_tokens (from ctx.model.maxTokens) and, for reasoning models with thinking on, thinking_level (from ctx.thinkingLevel), so the Langfuse UI shows them next to the model name.

Suggested implementation

  1. A pure extractModelParameters(model, thinkingLevel) returning Record<string, string | number> | undefined, with unit tests for the reasoning / non-reasoning / off cases. pi clamps the level to off for non-reasoning models before sending, so a stale level on the context must not produce a misleading chip.
  2. In before_provider_request, pass modelParameters: extractModelParameters(ctx.model, ctx.thinkingLevel) to startObservation. @langfuse/tracing drops undefined attributes, so no conditional spread is needed.
  3. Integration test on the sandbox model (maxTokens: 8192, reasoning: false) → { "max_tokens": 8192 }; with reasoning: true and defaultThinkingLevel: "high" in settings.json{ "max_tokens": 8192, "thinking_level": "high" }.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions