Repository navigation
Context Window & Token Overhead Benchmark: REST Tool Explosion vs. Schema-First GraphQL Approach #3891
Replies: 1 comment
Update: Live Agent Trace Reading Discussion #3891To test the practical implications of capability discovery, I asked Claude Sonnet to explain this exact discussion (#3891) without using web search or direct URL fetching. The agent had access to:
The instruction was:
Full trace: https://claude.ai/share/efa71161-dd9b-4df9-927c-61c1ae749380 Reading Discussion #3891github-officialThe agent attempted: Result: The discussion exists, but no discussion-reading capability was available through the exposed tool surface. Transport cost before failure: Useful output: graphql-enhancedThe agent executed: query {
repository(owner: "github", name: "github-mcp-server") {
discussion(number: 3891) {
title
body
url
author { login }
}
}
}Result: Transport cost: Useful output: Same Workload: Repo Metadata + 10 Recent Issuesgithub-officialResponse tokens: Issues observed:
graphql-enhancedSingle query: query {
repository(owner:"github", name:"github-mcp-server") {
nameWithOwner
stargazerCount
forkCount
watchers { totalCount }
open: issues(states: OPEN) { totalCount }
closed: issues(states: CLOSED) { totalCount }
issues(first:10, orderBy:{field: UPDATED_AT, direction: DESC}) {
nodes {
number
title
state
updatedAt
}
}
}
}Response tokens: What surprised meThis experiment started as a token benchmark. It ended up becoming a capability-discovery benchmark. The capability already exists in GitHub: The capability exists in GitHub GraphQL: But the capability is absent from the exposed MCP tool surface: Which raises an interesting architectural question. If the ecosystem is now forming working groups around progressive disclosure, tool grouping, tool filtering, and selective visibility, perhaps the deeper question is:
For me, the most interesting outcome wasn't the token savings. It was watching a generic GraphQL MCP server successfully read a discussion that the official GitHub MCP server could not access. |
Uh oh!
There was an error while loading. Please reload this page.
Hi team,
While integrating
github-mcp-server(v2.2.0) into real-world agentic workflows (Claude Desktop), I ran an empirical transport-level benchmark comparing the official REST-based MCP server against a schema-first GraphQL MCP approach (@letoribo/mcp-graphql-enhanced).Using a custom
stdioJSON-RPC interceptor paired with@anthropic-ai/tokenizer, I measured exact token consumption across initialization, static schema injection, and multi-turn live queries.The results highlight a critical bottleneck in static REST tool definitions and show why the efficiency gap compounds dramatically as conversations grow longer.
For transparency, the complete Claude Desktop session used for validation is available here:
Shared chat:
https://claude.ai/share/ffb7b29e-3f65-435b-a19e-6b1d25681817
1. Static Overhead (
tools/list& Initialization)On connection startup, the official server injects 46 individual tool definitions into the model's context window.
tools/listPayload Sizetools/listToken Costinit+ lists)2. Live Multi-Turn Execution Benchmark
In a single Claude Desktop session, I executed two consecutive queries comparing repository metadata and recent issues for:
letoribo/mcp-graphql-enhancedgithub/github-mcp-server(fetching repository metadata plus the 10 most recently updated issues).
github/github-mcp-server+ 10 IssuesWhy the Efficiency Gap Grows Over Long Conversations
1. Zero Re-Introspection
After the initial schema synchronization is cached, every subsequent GraphQL request sends only a compact query document and receives only the exact fields requested.
2. Compounding Context Tax
In multi-turn LLM sessions, the static tool catalog remains part of the model's working context.
With the official server, ~60K tokens of tool metadata must be carried alongside normal reasoning. With a two-tool GraphQL interface, that recurring overhead is effectively eliminated.
The longer the conversation becomes, the wider the efficiency advantage grows in favor of the schema-first approach.
3. Fewer Round-Trips
Fetching repository metadata together with recent issues requires multiple REST-oriented tool invocations.
GraphQL resolves the same information through a single query document and a single response.
3. Data Accuracy & REST Legacy Artifacts
Side-by-side comparison on
github/github-mcp-serveralso surfaced a number of well-known REST v3 artifacts that can mislead LLM agents.Inflated Watcher Counts
search_repositoriesreturned:which mirrors
stargazers_count.GraphQL's:
returned:
which reflects actual repository watchers/subscribers.
Issues vs. Pull Requests Conflation
REST returned:
combining open issues and open pull requests.
GraphQL separately exposed:
making the distinction explicit.
Inconsistent Field Presence
REST omitted comment counters when the value was zero and returned substantial URL-template and permission boilerplate.
GraphQL returned deterministic field values and only the requested fields.
Relation to Issue #2437
I noticed there is already an open feature request:
The benchmark suggests that a schema-first or hybrid GraphQL execution model offers benefits beyond developer convenience:
Takeaway
The benchmark suggests that the primary scalability bottleneck is not query execution but tool surface area.
A schema-first GraphQL model compresses capability discovery into a small set of generic primitives while preserving access to the full GitHub capability surface through introspection and typed queries.
I'm happy to share raw
stdiotraces, benchmark scripts, and measurement methodology if the data would be useful for further discussion.All reactions