feat(mcp): Add MCP server for Sentry docs with AI-powered evals #16235
+1,792
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add vitest-evals based testing to verify AI assistants correctly select MCP tools. Uses Claude Sonnet 4.5 as the scoring model to predict which tools would be called for given user queries. Test coverage: - search-docs: 6 test cases for documentation search - get-doc: 5 test cases for fetching doc content - list-platforms: 6 test cases for platform discovery - get-doc-tree: 6 test cases for navigation structure All 23 tests pass with scores ranging from 0.85 to 1.0. Usage: - yarn eval: Run evals in watch mode - yarn eval:run: Run evals once Requires ANTHROPIC_API_KEY in .env.local Co-Authored-By: Claude <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Add Model Context Protocol server exposing Sentry documentation to AI assistants. Provides 4 tools: - search_docs: Search documentation with optional platform filtering - get_doc: Fetch full markdown content of a documentation page - list_platforms: List available SDK platforms and their guides - get_doc_tree: Get documentation structure for navigation Endpoint: /mcp (HTTP Streamable transport) Co-Authored-By: Claude <noreply@anthropic.com>
sergical
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…tization Fixes GitHub Advanced Security alert for incomplete multi-character sanitization. The regex `/<[^>]*>/g` can leave partial tags when input contains nested angle brackets (e.g., `<<script>` → `<script>`). The new `stripHtmlTags()` helper applies the regex iteratively until no more tags remain, preventing XSS vulnerabilities from malformed HTML. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ait errors - Add try/catch to search_docs tool matching pattern used by other MCP tools - Fix require-await ESLint errors in eval files by using Promise.resolve() instead of unnecessary async functions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| // Skip trailing slash redirect for MCP and API routes | ||
| if (pathname.startsWith('/mcp') || pathname.startsWith('/api')) { | ||
| return NextResponse.next(); | ||
| } |
There was a problem hiding this comment.
Overly broad path check skips trailing slashes incorrectly
Low Severity
The condition pathname.startsWith('/mcp') is too broad and would match any path beginning with those characters (e.g., /mcp-guide, /mcpfoo), not just the actual MCP endpoint at /mcp. Such paths would incorrectly skip the trailing slash redirect that all other documentation pages receive. The check for pathname.startsWith('/api') is also redundant since the middleware matcher already excludes /api paths via the negative lookahead regex.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Model Context Protocol (MCP) server to expose Sentry documentation to AI assistants, plus
vitest-evalsbased evaluation tests to verify tool selection quality.MCP Server
Endpoint:
/mcp(HTTP Streamable transport)Tools
search_docsget_doclist_platformsget_doc_treeEvaluation Tests
Uses Claude Sonnet 4.5 as the scoring model to verify AI assistants correctly select MCP tools.
Test Coverage
search_docsget_doclist_platformsget_doc_treeScore Results
All 23 tests pass with scores between 0.85 - 1.0:
Threshold is 0.6 for passing.
Usage
Requires
ANTHROPIC_API_KEYin.env.localfor evals.Test plan
yarn lint:tspassesyarn eval:runpasses (23/23 tests)🤖 Generated with Claude Code