Skip to content

Commit a4b9fa8

Browse files
feat(mcp): improve MCP tool quality — error hints, shared helpers, better messages (#2337)
## What Targeted improvements to MCP tool quality, identified and validated through performance evals against real agent workflows. These changes reduce tool-call failures, improve agent self-correction, and make tool outputs more actionable. ### Query tools: top-level `where` + orderBy resolution **Table & Timeseries** now accept a top-level `where` / `whereLanguage` that scopes the entire query. A shared `mergeWhereIntoSelectItems()` helper injects it into each select item's aggCondition (ANDed with any per-item where). No more duplicated merge logic. **Table** gets an `orderBy` resolver: agents frequently write `orderBy: "count"`, which generates `ORDER BY count ASC` — but ClickHouse can't resolve bare `count` since it's a function. The resolver maps aggFn names to aliases or synthesized expressions (`count()`, `avg(Duration)`, etc.). **Timeseries** now detects single-bucket collapse (1 row returned with explicit granularity) and adds a hint to adjust the time range or granularity. ### Event patterns: topN, trendBuckets, restructured output - `topN` (1-100, default 20) — control how many patterns are returned - `trendBuckets` (0-60, default 24) — set to 0 to disable trend sparklines (smaller response) - Output restructured: top-level `summary` + `patterns` array + `usage` guide - Each pattern now includes `rank` and `shareOfTotal` for easier analysis ### Schema & description improvements - Richer tool descriptions with concrete examples, common mistakes, and expression tips - `MCP_AGG_FN_OPTIONS` extracted as shared constant (single source of truth for aggFn names) - `whereLanguage` description now includes syntax matching guidance and common mistakes - `groupBy` documents multi-column comma-delimited input and arbitrary ClickHouse expressions - `valueExpression` description shows useful patterns (duration conversion, attribute casting, boolean→numeric) ### Infrastructure - **Error hints**: Duplicated ClickHouse catch blocks extracted into `clickHouseErrorResult()` with pattern-matched hints for DateTime64 casting, AS alias quoting, and response size overflow - **`trimToolResponse`**: Now returns `{ data, isTrimmed }` so callers detect trimming without re-serializing - **Dashboard schemas**: `granularity` added to line/stacked_bar external dashboard zod schemas - **`describeSource`**: Switched to compact JSON output (saves context tokens) - **Better not-found messages**: Source/connection not-found errors suggest calling `hyperdx_list_sources` - **Regex fix**: `errorHint` AS-alias regex uses `\bAS\b` word boundary to avoid false matches on words like DATABASE, ALIAS ## Why These changes came out of running MCP performance evals against real agent tasks. The combined effect: - Error hints cut agent confusion loops on ClickHouse-specific errors - orderBy resolution eliminates the most common ClickHouse ORDER BY failure mode - Top-level where reduces multi-step filter workarounds - Restructured event pattern output makes pattern analysis workflows significantly more reliable > **Coming soon:** We're building out a structured eval framework for MCP tool performance. Stay tuned — we'll be sharing the harness, benchmark suite, and results publicly. ## Files changed (12) | File | Change | |------|--------| | `helpers.ts` | `mergeWhereIntoSelectItems()`, `clickHouseErrorResult()`, error hints, better not-found messages | | `table.ts` | Top-level where, orderBy resolver, updated descriptions | | `timeseries.ts` | Top-level where, single-bucket collapse detection | | `schemas.ts` | `MCP_AGG_FN_OPTIONS`, richer descriptions for all fields | | `eventPatterns.ts` | `topN`, `trendBuckets` parameters | | `runEventPatterns.ts` | Restructured output format, shareOfTotal, rank | | `describeSource.ts` | Compact JSON, trimToolResponse API update | | `dashboards.ts` | `granularity` in `convertToInternalTileConfig` | | `trimToolResponse.ts` | Returns `{ data, isTrimmed }` | | `trimToolResponse.test.ts` | Updated tests for new return shape | | `queryTool.test.ts` | Test for `topN` property | | `zod.ts` | `SQLIntervalSchema` + `granularity` on time chart schemas |
1 parent 3123db5 commit a4b9fa8

13 files changed

Lines changed: 848 additions & 158 deletions

File tree

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@hyperdx/api': patch
3+
---
4+
5+
feat(mcp): improve MCP tool quality — error hints, shared helpers, better messages
6+
7+
Extract duplicated ClickHouse error handling into a shared helper with
8+
pattern-matched error hints (DateTime64 casting, AS alias quoting, response
9+
size limits) so agents get actionable guidance on common failures. Add
10+
reusable mergeWhereIntoSelectItems() helper for consistent top-level where
11+
injection. Improve source/connection-not-found messages to suggest calling
12+
hyperdx_list_sources.

‎packages/api/src/mcp/__tests__/query.test.ts‎

Lines changed: 260 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,17 @@
1-
import { parseTimeRange } from '../tools/query/helpers';
1+
// Mock heavy dependencies that break in unit-test context (no ClickHouse/Mongo)
2+
jest.mock('@/models/source', () => ({}));
3+
jest.mock('@/controllers/sources', () => ({}));
4+
jest.mock('@/controllers/connection', () => ({}));
5+
jest.mock('@/utils/trimToolResponse', () => ({
6+
trimToolResponse: (data: unknown) => ({ data, isTrimmed: false }),
7+
}));
8+
9+
import {
10+
errorHint,
11+
mergeWhereIntoSelectItems,
12+
parseTimeRange,
13+
} from '../tools/query/helpers';
14+
import { resolveOrderBy } from '../tools/query/table';
215

316
describe('parseTimeRange', () => {
417
it('should return default range (last 15 minutes) when no arguments provided', () => {
@@ -72,3 +85,249 @@ describe('parseTimeRange', () => {
7285
expect(result).toHaveProperty('error');
7386
});
7487
});
88+
89+
// ─── mergeWhereIntoSelectItems ───────────────────────────────────────────────
90+
91+
describe('mergeWhereIntoSelectItems', () => {
92+
it('should return items unchanged when topWhere is empty', () => {
93+
const items = [{ where: 'level:error', whereLanguage: 'lucene' as const }];
94+
const { items: result, warnings } = mergeWhereIntoSelectItems(
95+
items,
96+
'',
97+
'lucene',
98+
);
99+
expect(result).toBe(items); // same reference, not a copy
100+
expect(warnings).toHaveLength(0);
101+
});
102+
103+
it('should inject top-level where when item has no where', () => {
104+
const items = [{ where: '', whereLanguage: 'lucene' as const }];
105+
const { items: result, warnings } = mergeWhereIntoSelectItems(
106+
items,
107+
'service:api',
108+
'lucene',
109+
);
110+
expect(result[0].where).toBe('service:api');
111+
expect(result[0].whereLanguage).toBe('lucene');
112+
expect(warnings).toHaveLength(0);
113+
});
114+
115+
it('should AND-combine when languages match', () => {
116+
const items = [{ where: 'level:error', whereLanguage: 'lucene' as const }];
117+
const { items: result, warnings } = mergeWhereIntoSelectItems(
118+
items,
119+
'service:api',
120+
'lucene',
121+
);
122+
expect(result[0].where).toBe('(service:api) AND (level:error)');
123+
expect(result[0].whereLanguage).toBe('lucene');
124+
expect(warnings).toHaveLength(0);
125+
});
126+
127+
it('should AND-combine SQL filters when both use sql', () => {
128+
const items = [
129+
{ where: 'StatusCode >= 500', whereLanguage: 'sql' as const },
130+
];
131+
const { items: result, warnings } = mergeWhereIntoSelectItems(
132+
items,
133+
"ServiceName = 'api'",
134+
'sql',
135+
);
136+
expect(result[0].where).toBe(
137+
"(ServiceName = 'api') AND (StatusCode >= 500)",
138+
);
139+
expect(result[0].whereLanguage).toBe('sql');
140+
expect(warnings).toHaveLength(0);
141+
});
142+
143+
it('should skip item and emit warning when languages differ', () => {
144+
const items = [
145+
{ where: 'StatusCode >= 500', whereLanguage: 'sql' as const },
146+
];
147+
const { items: result, warnings } = mergeWhereIntoSelectItems(
148+
items,
149+
'service:api',
150+
'lucene',
151+
);
152+
// Item's own where is kept unchanged
153+
expect(result[0].where).toBe('StatusCode >= 500');
154+
expect(result[0].whereLanguage).toBe('sql');
155+
// A warning is emitted
156+
expect(warnings).toHaveLength(1);
157+
expect(warnings[0]).toContain('select[0]');
158+
expect(warnings[0]).toContain('was NOT applied');
159+
});
160+
161+
it('should handle mixed items — some merged, some skipped', () => {
162+
const items = [
163+
{ where: 'level:error', whereLanguage: 'lucene' as const },
164+
{ where: 'StatusCode >= 500', whereLanguage: 'sql' as const },
165+
{ where: '', whereLanguage: 'lucene' as const },
166+
];
167+
const { items: result, warnings } = mergeWhereIntoSelectItems(
168+
items,
169+
'service:api',
170+
'lucene',
171+
);
172+
// Item 0: merged (both lucene)
173+
expect(result[0].where).toBe('(service:api) AND (level:error)');
174+
// Item 1: skipped (sql vs lucene)
175+
expect(result[1].where).toBe('StatusCode >= 500');
176+
// Item 2: injected (empty where)
177+
expect(result[2].where).toBe('service:api');
178+
expect(warnings).toHaveLength(1);
179+
expect(warnings[0]).toContain('select[1]');
180+
});
181+
});
182+
183+
// ─── errorHint ───────────────────────────────────────────────────────────────
184+
185+
describe('errorHint', () => {
186+
it('should match DateTime64 conversion errors', () => {
187+
const hint = errorHint(
188+
"Cannot convert string '2025-01-01T00:00:00Z' to type DateTime64(9)",
189+
);
190+
expect(hint).not.toBeNull();
191+
expect(hint).toContain('toDateTime64');
192+
});
193+
194+
it('should match DateTime64 parse errors', () => {
195+
const hint = errorHint(
196+
"Cannot parse string '2025-01-01' as type DateTime64",
197+
);
198+
expect(hint).not.toBeNull();
199+
expect(hint).toContain('toDateTime64');
200+
});
201+
202+
it('should match AS alias syntax errors with word boundary', () => {
203+
const hint = errorHint('Syntax error: unexpected token AS');
204+
expect(hint).not.toBeNull();
205+
expect(hint).toContain('alias');
206+
});
207+
208+
it('should NOT match AS embedded in other words like DATABASE', () => {
209+
const hint = errorHint("Syntax error: Unknown database 'FOOBAR'");
210+
expect(hint).toBeNull();
211+
});
212+
213+
it('should NOT match lowercase "as" in cast expressions', () => {
214+
const hint = errorHint(
215+
'Syntax error: cannot cast as Float64 near token...',
216+
);
217+
expect(hint).toBeNull();
218+
});
219+
220+
it('should match V8 string length overflow', () => {
221+
const hint = errorHint(
222+
'response length exceeds the maximum allowed size of V8 String',
223+
);
224+
expect(hint).not.toBeNull();
225+
expect(hint).toContain('LIMIT');
226+
});
227+
228+
it('should return null for unrecognized errors', () => {
229+
const hint = errorHint('Connection refused');
230+
expect(hint).toBeNull();
231+
});
232+
});
233+
234+
// ─── resolveOrderBy ──────────────────────────────────────────────────────────
235+
236+
describe('resolveOrderBy', () => {
237+
it('should return undefined when orderBy is undefined', () => {
238+
expect(resolveOrderBy(undefined, [{ aggFn: 'count' }])).toBeUndefined();
239+
});
240+
241+
it('should pass through non-aggFn values unchanged', () => {
242+
expect(resolveOrderBy('SpanName', [{ aggFn: 'count' }])).toBe('SpanName');
243+
});
244+
245+
it('should keep orderBy when it matches an alias', () => {
246+
expect(resolveOrderBy('Total', [{ aggFn: 'count', alias: 'Total' }])).toBe(
247+
'Total',
248+
);
249+
});
250+
251+
it('should return canonical alias case for case-insensitive match', () => {
252+
expect(resolveOrderBy('total', [{ aggFn: 'count', alias: 'Total' }])).toBe(
253+
'Total',
254+
);
255+
expect(
256+
resolveOrderBy('TOTAL DESC', [{ aggFn: 'count', alias: 'Total' }]),
257+
).toBe('Total DESC');
258+
});
259+
260+
it('should synthesize count()', () => {
261+
expect(resolveOrderBy('count', [{ aggFn: 'count' }])).toBe('count()');
262+
});
263+
264+
it('should synthesize avg(Duration)', () => {
265+
expect(
266+
resolveOrderBy('avg', [{ aggFn: 'avg', valueExpression: 'Duration' }]),
267+
).toBe('avg(Duration)');
268+
});
269+
270+
it('should synthesize quantile with level', () => {
271+
expect(
272+
resolveOrderBy('quantile', [
273+
{ aggFn: 'quantile', valueExpression: 'Duration', level: 0.99 },
274+
]),
275+
).toBe('quantile(0.99)(Duration)');
276+
});
277+
278+
it('should prefer alias over synthesized expression', () => {
279+
expect(
280+
resolveOrderBy('count', [{ aggFn: 'count', alias: 'Total Rows' }]),
281+
).toBe('Total Rows');
282+
});
283+
284+
it('should be case-insensitive for aggFn matching', () => {
285+
expect(resolveOrderBy('Count', [{ aggFn: 'count' }])).toBe('count()');
286+
expect(
287+
resolveOrderBy('AVG', [{ aggFn: 'avg', valueExpression: 'Duration' }]),
288+
).toBe('avg(Duration)');
289+
});
290+
291+
it('should synthesize count(DISTINCT ...) for count_distinct', () => {
292+
expect(
293+
resolveOrderBy('count_distinct', [
294+
{ aggFn: 'count_distinct', valueExpression: 'UserId' },
295+
]),
296+
).toBe('count(DISTINCT UserId)');
297+
});
298+
299+
it('should synthesize count(DISTINCT ...) with direction', () => {
300+
expect(
301+
resolveOrderBy('count_distinct DESC', [
302+
{ aggFn: 'count_distinct', valueExpression: 'UserId' },
303+
]),
304+
).toBe('count(DISTINCT UserId) DESC');
305+
});
306+
307+
it('should NOT synthesize for "none" aggFn', () => {
308+
expect(
309+
resolveOrderBy('none', [
310+
{ aggFn: 'none', valueExpression: 'Duration / 1e6' },
311+
]),
312+
).toBe('none');
313+
});
314+
315+
it('should handle trailing ASC/DESC', () => {
316+
expect(resolveOrderBy('count DESC', [{ aggFn: 'count' }])).toBe(
317+
'count() DESC',
318+
);
319+
expect(
320+
resolveOrderBy('avg ASC', [
321+
{ aggFn: 'avg', valueExpression: 'Duration' },
322+
]),
323+
).toBe('avg(Duration) ASC');
324+
});
325+
326+
it('should pass through quantile without level unchanged', () => {
327+
expect(
328+
resolveOrderBy('quantile', [
329+
{ aggFn: 'quantile', valueExpression: 'Duration' },
330+
]),
331+
).toBe('quantile');
332+
});
333+
});

‎packages/api/src/mcp/__tests__/queryTool.test.ts‎

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,7 @@ describe('MCP Query Tools', () => {
138138
expect(props).toContain('sourceId');
139139
expect(props).toContain('where');
140140
expect(props).toContain('sampleSize');
141+
expect(props).toContain('topN');
141142
expect(props).toContain('bodyExpression');
142143
expect(tool!.inputSchema.required).toContain('sourceId');
143144
});
@@ -370,14 +371,14 @@ describe('MCP Query Tools', () => {
370371
expect(result.isError).toBeFalsy();
371372
expect(result.content).toHaveLength(1);
372373
const output = JSON.parse(getFirstText(result));
373-
expect(output).toHaveProperty('result');
374-
expect(output.result).toHaveProperty('patterns');
375-
expect(output.result).toHaveProperty('totalCount');
376-
expect(output.result).toHaveProperty('sampledRows');
377-
expect(output.result).toHaveProperty('sampleMultiplier');
378-
expect(output.result).toHaveProperty('bodyColumn', 'Body');
379-
expect(output.result).toHaveProperty('timeRange');
380-
expect(Array.isArray(output.result.patterns)).toBe(true);
374+
expect(output).toHaveProperty('summary');
375+
expect(output).toHaveProperty('patterns');
376+
expect(output.summary).toHaveProperty('totalCount');
377+
expect(output.summary).toHaveProperty('sampledCount');
378+
expect(output.summary).toHaveProperty('sampleMultiplier');
379+
expect(output.summary).toHaveProperty('bodyColumn', 'Body');
380+
expect(output.summary).toHaveProperty('timeRange');
381+
expect(Array.isArray(output.patterns)).toBe(true);
381382
});
382383

383384
it('should execute with explicit bodyExpression on trace source', async () => {
@@ -391,9 +392,9 @@ describe('MCP Query Tools', () => {
391392
expect(result.isError).toBeFalsy();
392393
expect(result.content).toHaveLength(1);
393394
const output = JSON.parse(getFirstText(result));
394-
expect(output).toHaveProperty('result');
395-
expect(Array.isArray(output.result.patterns)).toBe(true);
396-
expect(output.result).toHaveProperty('bodyColumn', 'SpanName');
395+
expect(output).toHaveProperty('summary');
396+
expect(Array.isArray(output.patterns)).toBe(true);
397+
expect(output.summary).toHaveProperty('bodyColumn', 'SpanName');
397398
});
398399

399400
it('should accept custom bodyExpression on log source', async () => {
@@ -407,7 +408,7 @@ describe('MCP Query Tools', () => {
407408
expect(result.isError).toBeFalsy();
408409
expect(result.content).toHaveLength(1);
409410
const output = JSON.parse(getFirstText(result));
410-
expect(output.result).toHaveProperty('bodyColumn', 'SeverityText');
411+
expect(output.summary).toHaveProperty('bodyColumn', 'SeverityText');
411412
});
412413

413414
it('should respect sampleSize parameter', async () => {
@@ -503,10 +504,11 @@ describe('MCP Query Tools', () => {
503504

504505
expect(result.isError).toBeFalsy();
505506
const output = JSON.parse(getFirstText(result));
506-
const { patterns, totalCount, sampledRows } = output.result;
507+
const { patterns } = output;
508+
const { totalCount, sampledCount } = output.summary;
507509

508510
expect(totalCount).toBeGreaterThanOrEqual(31);
509-
expect(sampledRows).toBeGreaterThanOrEqual(31);
511+
expect(sampledCount).toBeGreaterThanOrEqual(31);
510512
expect(patterns.length).toBeGreaterThanOrEqual(1);
511513

512514
// The most common pattern should contain <*> placeholders

‎packages/api/src/mcp/tools/query/eventDeltas.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -526,13 +526,20 @@ export function registerEventDeltas(server: McpServer, context: McpContext) {
526526
'are dropped by default — set includeHidden:true to inspect them.',
527527
};
528528

529-
const trimmedOutput = trimToolResponse(output);
529+
const { data: trimmedOutput, isTrimmed } = trimToolResponse(output);
530+
531+
const finalOutput = isTrimmed
532+
? {
533+
...trimmedOutput,
534+
note: 'Result was trimmed for context size. Narrow the time range, add filters, or reduce topN/topValuesPerProperty to reduce data.',
535+
}
536+
: trimmedOutput;
530537

531538
return {
532539
content: [
533540
{
534541
type: 'text' as const,
535-
text: JSON.stringify(trimmedOutput, null, 2),
542+
text: JSON.stringify(finalOutput, null, 2),
536543
},
537544
],
538545
};

0 commit comments

Comments
 (0)