Skip to content

fix(assessments): Fix LangGraph install, OpenRouter model names, and Mastra setup - #167

Merged
obostjancic merged 1 commit into
mainfrom
fix/harness-pins-models-mastra
Nov 16, 2026
Merged

obostjancic merged 1 commit into
mainfrom
fix/harness-pins-models-mastra

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

This fixes three harness problems that lower the daily score without saying anything about the SDKs.

JS LangGraph install (node + cloudflare, 0 since Nov 4). The configs pin @langchain/core to 1.2.9, but @langchain/langgraph@1.7.4 requires ^1.3.4, so npm install fails with ERESOLVE on all 4 variants. The pin is now 1, the same as the llm langchain configs. @langchain/openai@1.5.10 and @langchain/anthropic@1.5.8 accept ^1.2.9, so they still resolve.

Response model overrides. Every target calls the model through OpenRouter, and OpenRouter returns names that include the provider: anthropic/claude-haiku-4.5 and openai/gpt-5-nano. The overrides expected claude-haiku-4-5* and gpt-5-nano*, so 11 targets have reported model.response.mismatch (major) since the reports began. The SDKs record the model the API returned, so this mismatch was never a product finding. The overrides now use the served names, the same way the Vercel configs already do.

Mastra. The template registered @mastra/sentry's SentryExporter. Sentry v11 includes mastraIntegration in its default tracing integrations and warns that the two must not run together, because the exporter calls Sentry.init() again. On Next.js that crashed in getDynamicSamplingContextFromSpan, so the target has always scored 1. The template now creates a plain Mastra and lets the SDK instrument it. @mastra/observability stays installed because the SDK uses the app's copy to bootstrap the pipeline.

via Ogi.

--

View Session [Sentry]

…Mastra setup

Allow @langchain/core 1.x for JS LangGraph so npm can satisfy
@langchain/langgraph 1.7's ^1.3.4 peer range.

Expect the provider-qualified model names that OpenRouter returns
(anthropic/claude-haiku-4.5, openai/gpt-5-nano) in response model
overrides.

Drop the @mastra/sentry exporter. Sentry v11 instruments Mastra by
default, and the second exporter re-inits Sentry and crashed the
Next.js run.

Co-Authored-By: Ognjen Bostjancic <ognjen.bostjancic@sentry.io>
@obostjancic
obostjancic marked this pull request as ready for review November 16, 2026 10:14
@obostjancic
obostjancic merged commit 5493120 into main Nov 16, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant