Skip to content

Commit 3571bfa

Browse files
authored
[HDX-3999] Implement hdx query with connections command and Drain pattern mining (#2133)
## Summary Implements [HDX-3999](https://linear.app/clickhouse/issue/HDX-3999/implement-hdx-query-one-shot-clickhouse-sql-execution) — a one-shot `hdx query` for executing raw ClickHouse SQL against a configured connection, plus supporting UX around it for agent-driven workflows. This is pre-launch iteration on the unreleased CLI surface; no released behavior changes. ## What changed ### `hdx connections` (new subcommand) - Lists ClickHouse connections configured for the authenticated team (`id`, `name`, `host`), matching the UX of `hdx sources` and `hdx dashboards`. - Supports `--json` for programmatic consumption (`hdx connections --json | jq -r '.[0].id'`). - Exports `ConnectionResponse` from `api/client.ts` for consistency with other response types. - Registered in `packages/cli/AGENTS.md`. ### `hdx query` reshape - Replaces `--source <nameOrId>` with `--connection-id <id>`. Raw SQL only needs a connection, so the source-to-connection lookup was unnecessary indirection. Agents discover IDs via `hdx connections --json`. - On query error, lazily fetches the connection list to distinguish "unknown connection ID" from "bad SQL" — no happy-path cost. - Default `--format` changes from `JSON` (wrapped envelope) → `JSONEachRow` (NDJSON). Streamable, smaller, trivially consumable from shell pipelines. Envelope is still available via `--format JSON`. - Documents the exit-code contract in `--help` so consumers can disambiguate empty result (exit 0, empty stdout) from failure (exit 1, stderr message). ### `hdx query --patterns` (Drain-based pattern mining) - `--patterns`: clusters the result's text into log patterns via Drain (`@hyperdx/common-utils/dist/drain`) and emits one NDJSON object per pattern sorted by count desc: ```json {"pattern":"<template>","count":<n>,"sample":"<first sample>"} ``` - `--body-column <name>`: cluster only that column's string value. Defaults to clustering the whole row JSON-serialized, which works for any SELECT shape. If the column isn't in the result, exits 1 with the available columns on stderr. - Pattern mining forces `JSON` internally regardless of `--format` so named rows can be iterated; a one-line note is written to stderr if the user passed a conflicting `--format`. - The existing raw-SQL path is unchanged when `--patterns` is not set. ### Help-text polish - Top-level `hdx --help` now mentions `--patterns` in the `query` description so agents can discover it without drilling into `hdx query --help`. - Expanded help block documents `--body-column` in detail and adds a sampling tip with an explicit performance warning that `ORDER BY rand()` scans and sorts all WHERE-matched rows before LIMIT and must be paired with a selective WHERE. ## Why Agents driving HyperDX need a direct, bounded way to run SQL and summarize large result sets. `--patterns` makes it cheap to ask "what are the top log templates in this window" without a separate pipeline step. ## Testing - Manual: `hdx connections`, `hdx connections --json`, `hdx query --connection-id <id> --sql "SELECT 1"`, `hdx query --connection-id <id> --patterns --body-column Body --sql "SELECT Body FROM default.otel_logs LIMIT 1000"`. - `make ci-lint` passes. - No automated tests added — `packages/cli` has no Jest suite for `cli.tsx` today; following the existing convention. ## Breaking changes - `hdx query` no longer accepts `--source`. The command is pre-launch so no downstream consumers yet.
1 parent 2a28aee commit 3571bfa

3 files changed

Lines changed: 245 additions & 33 deletions

File tree

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
"@hyperdx/cli": minor
3+
---
4+
5+
Add `hdx query` and `hdx connections` commands for agentic ClickHouse workflows:
6+
7+
- **`hdx connections`** — list ClickHouse connections (`id`, `name`, `host`) for the authenticated team. Supports `--json` for programmatic consumption.
8+
- **`hdx query --connection-id <id> --sql <query>`** — run raw SQL against a configured ClickHouse connection via the `/clickhouse-proxy`. Default `--format` is `JSONEachRow` (NDJSON); any ClickHouse format is accepted. Exit codes: `0` on success (empty stdout = zero rows), `1` on failure. On error, a lazy connection lookup distinguishes "unknown connection ID" from "bad SQL".
9+
- **`hdx query --patterns`** — post-process the result with the Drain algorithm (`@hyperdx/common-utils/dist/drain`) and emit one NDJSON pattern object per cluster, sorted by count desc: `{"pattern":"<template>","count":<n>,"sample":"<first sample>"}`. Use `--body-column <name>` to cluster a single column's string value; defaults to the whole row JSON-serialized so any SELECT shape works.
10+
- `--help` documents the exit-code contract and includes sampling guidance (`ORDER BY rand()` + selective WHERE warning) for mining over large tables.

‎packages/cli/AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ logic (`@hyperdx/common-utils`) as the web frontend.
1616
hdx tui -s <url> # Interactive TUI (main command)
1717
hdx stream -s <url> --source "Logs" # Non-interactive streaming to stdout
1818
hdx sources -s <url> # List available sources
19+
hdx connections # List ClickHouse connections (id, name, host)
1920
hdx auth login -s <url> # Sign in (interactive or -e/-p flags)
2021
hdx auth status # Show auth status (reads saved session)
2122
hdx auth logout # Clear saved session

‎packages/cli/src/cli.tsx‎

Lines changed: 234 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ import TextInput from 'ink-text-input';
1111
import Spinner from 'ink-spinner';
1212
import { Command } from 'commander';
1313
import chalk from 'chalk';
14+
import {
15+
TemplateMiner,
16+
TemplateMinerConfig,
17+
} from '@hyperdx/common-utils/dist/drain';
1418

1519
import App from '@/App';
1620
import { ApiClient } from '@/api/client';
@@ -664,6 +668,66 @@ Examples:
664668
}
665669
});
666670

671+
// ---- Connections ---------------------------------------------------
672+
673+
program
674+
.command('connections')
675+
.description('List ClickHouse connections (id, name, host)')
676+
.option('-a, --app-url <url>', 'HyperDX app URL')
677+
.option('--json', 'Output as JSON (for programmatic consumption)')
678+
.addHelpText(
679+
'after',
680+
`
681+
About:
682+
Lists ClickHouse connections configured for the authenticated team.
683+
Each connection is a named set of credentials that one or more
684+
sources point at (via the 'connection' field on a source).
685+
686+
Use 'hdx sources --json' to see which connection each source uses.
687+
688+
JSON output schema (--json):
689+
Array of objects, each with:
690+
id - Connection ID (matches source.connection)
691+
name - Human-readable connection name
692+
host - ClickHouse host URL
693+
694+
Examples:
695+
$ hdx connections
696+
$ hdx connections --json
697+
$ hdx connections --json | jq '.[] | select(.name=="Default")'
698+
`,
699+
)
700+
.action(async opts => {
701+
const client = await ensureSession(opts.appUrl);
702+
703+
const connections = await client.getConnections();
704+
if (connections.length === 0) {
705+
if (opts.json) {
706+
process.stdout.write('[]\n');
707+
} else {
708+
process.stdout.write('No connections found.\n');
709+
}
710+
return;
711+
}
712+
713+
if (opts.json) {
714+
const output = connections.map(c => ({
715+
id: c.id,
716+
name: c.name,
717+
host: c.host,
718+
}));
719+
process.stdout.write(JSON.stringify(output, null, 2) + '\n');
720+
return;
721+
}
722+
723+
// Human-readable: one line per connection
724+
for (const c of connections) {
725+
process.stdout.write(
726+
`${chalk.bold.cyan(c.name)} ${chalk.dim(c.host)} ${chalk.dim(`[${c.id}]`)}\n`,
727+
);
728+
}
729+
});
730+
667731
// ---- Dashboards ----------------------------------------------------
668732

669733
program
@@ -784,62 +848,177 @@ Examples:
784848

785849
program
786850
.command('query')
787-
.description('Run a raw SQL query against a ClickHouse source')
788-
.requiredOption('--source <nameOrId>', 'Source name or ID')
851+
.description(
852+
'Run raw SQL against a ClickHouse connection (add --patterns to cluster result into Drain log patterns)',
853+
)
854+
.requiredOption(
855+
'--connection-id <id>',
856+
"Connection ID (from 'hdx connections --json')",
857+
)
789858
.requiredOption('--sql <query>', 'SQL query to execute')
790859
.option('-a, --app-url <url>', 'HyperDX app URL')
791-
.option('--format <format>', 'ClickHouse output format', 'JSON')
860+
.option('--format <format>', 'ClickHouse output format', 'JSONEachRow')
861+
.option(
862+
'--patterns',
863+
'Mine Drain log patterns from the query result instead of emitting rows',
864+
)
865+
.option(
866+
'--body-column <name>',
867+
'Column whose string value is clustered when --patterns is set (default: whole row JSON)',
868+
)
792869
.addHelpText(
793870
'after',
794871
`
795872
About:
796873
Execute a raw ClickHouse SQL query through the HyperDX proxy, using
797-
the connection credentials associated with a source. This is useful
798-
for ad-hoc exploration, debugging, and agent-driven queries.
874+
a configured ClickHouse connection. Designed for ad-hoc exploration,
875+
debugging, and agent-driven queries.
799876
800-
The --source flag accepts either the source name (case-insensitive)
801-
or the source ID (from 'hdx sources --json').
877+
The --connection-id flag takes a connection ID. Use 'hdx connections
878+
--json' to discover available connection IDs. Use 'hdx sources --json'
879+
to see which connection each source uses (the 'connection' field).
802880
803881
The query is sent as-is to ClickHouse — you are responsible for
804-
writing valid SQL. Use 'hdx sources' to discover table names and
805-
column schemas.
806-
807-
Output is written to stdout. Use --format to control the ClickHouse
808-
response format (JSON, JSONEachRow, TabSeparated, CSV, etc.).
882+
writing valid SQL. Output is written to stdout. Use --format to
883+
control the ClickHouse response format (JSONEachRow, JSON,
884+
TabSeparated, CSV, etc.). The default is JSONEachRow (one JSON
885+
object per line) — streamable and easy to consume from agents and
886+
shell pipelines (e.g. \`jq -c\`).
887+
888+
Pattern mining (--patterns):
889+
Cluster the result's text into log patterns using the Drain
890+
algorithm, then emit one JSON object per pattern (sorted by count
891+
desc). Useful for summarizing a large result set into a small
892+
number of templated lines.
893+
894+
Drain runs in-process over whatever rows the SQL returns, so the
895+
result quality depends on the rows you give it:
896+
897+
--body-column <name>
898+
When set, only that column's string value is clustered. Useful
899+
when you SELECT * but only want to mine over (e.g.) Body or
900+
SpanName. If <name> isn't in the result, the command exits 1
901+
and lists available columns on stderr.
902+
903+
When omitted, the whole row is JSON-serialized and clustered as
904+
a single string. This works for any SELECT shape without column
905+
guessing, but produces noisier templates than clustering a
906+
single text column.
907+
908+
Sampling tip:
909+
For very large tables, prefer sampling over a tight LIMIT to
910+
avoid biasing toward a single time slice. Append \`ORDER BY rand()\`
911+
to your SQL, e.g.:
912+
913+
--sql "SELECT Body FROM default.otel_logs
914+
WHERE Timestamp > now() - INTERVAL 1 HOUR
915+
ORDER BY rand() LIMIT 10000"
916+
917+
Be aware: \`ORDER BY rand()\` forces ClickHouse to scan and sort
918+
all rows matched by the WHERE clause before applying LIMIT, which
919+
can be expensive on large tables. Always pair it with a selective
920+
WHERE (time range, service, severity, etc.) to bound the scan.
921+
922+
Output forces JSON internally regardless of --format. Each line is:
923+
{"pattern":"<template>","count":<n>,"sample":"<first sample>"}
924+
925+
Exit codes:
926+
0 Success. Stdout contains the result (empty stdout means zero rows).
927+
1 Failure. Stderr contains the error.
809928
810929
Examples:
811-
$ hdx query --source "Logs" --sql "SELECT count() FROM default.otel_logs"
812-
$ hdx query --source "Traces" --sql "SELECT * FROM default.otel_traces LIMIT 5"
813-
$ hdx query --source "Logs" --sql "SELECT Body FROM default.otel_logs LIMIT 3" --format JSONEachRow
930+
$ CONN=$(hdx connections --json | jq -r '.[0].id')
931+
$ hdx query --connection-id "$CONN" --sql "SELECT count() FROM default.otel_logs"
932+
$ hdx query --connection-id "$CONN" --sql "SELECT * FROM default.otel_traces LIMIT 5"
933+
$ hdx query --connection-id "$CONN" --patterns \\
934+
--sql "SELECT Body FROM default.otel_logs LIMIT 10000" --body-column Body
814935
`,
815936
)
816937
.action(async opts => {
817938
const client = await ensureSession(opts.appUrl);
939+
const chClient = client.createClickHouseClient();
818940

819-
const sources = await client.getSources();
820-
const source = sources.find(
821-
s =>
822-
s.name.toLowerCase() === opts.source.toLowerCase() ||
823-
s.id === opts.source ||
824-
s._id === opts.source,
825-
);
941+
try {
942+
if (opts.patterns) {
943+
// Pattern mining forces JSON internally so we can iterate named rows.
944+
if (
945+
opts.format &&
946+
opts.format !== 'JSON' &&
947+
opts.format !== 'JSONEachRow'
948+
) {
949+
_origError(
950+
chalk.dim('--patterns: ignoring --format, using JSON internally\n'),
951+
);
952+
}
953+
const resultSet = await chClient.query({
954+
query: opts.sql,
955+
format: 'JSON',
956+
connectionId: opts.connectionId,
957+
});
958+
const json = (await resultSet.json()) as {
959+
data?: Array<Record<string, unknown>>;
960+
};
961+
const rows = json.data ?? [];
962+
if (rows.length === 0) {
963+
return; // exit 0, empty stdout
964+
}
826965

827-
if (!source) {
828-
_origError(chalk.red(`Source "${opts.source}" not found.\n`));
829-
_origError('Available sources:');
830-
for (const s of sources) {
831-
_origError(` - ${s.name} (${s.kind}) [${s.id}]`);
832-
}
833-
process.exit(1);
834-
}
966+
// If --body-column was provided, validate it exists in the result.
967+
if (opts.bodyColumn) {
968+
const cols = Object.keys(rows[0]);
969+
if (!cols.includes(opts.bodyColumn)) {
970+
_origError(
971+
chalk.red(
972+
`--body-column '${opts.bodyColumn}' not found in result. ` +
973+
`Available columns: ${cols.join(', ')}\n`,
974+
),
975+
);
976+
process.exit(1);
977+
}
978+
}
835979

836-
const chClient = client.createClickHouseClient();
980+
const flatten = (s: string): string =>
981+
s
982+
.replace(/\n/g, ' ')
983+
.replace(/\s{2,}/g, ' ')
984+
.trim();
985+
986+
const miner = new TemplateMiner(new TemplateMinerConfig());
987+
const groups = new Map<number, { count: number; sample: string }>();
988+
for (const row of rows) {
989+
const raw = opts.bodyColumn
990+
? row[opts.bodyColumn]
991+
: JSON.stringify(row);
992+
const text = raw != null ? flatten(String(raw)) : '';
993+
const { clusterId } = miner.addLogMessage(text);
994+
const existing = groups.get(clusterId);
995+
if (existing) {
996+
existing.count += 1;
997+
} else {
998+
groups.set(clusterId, { count: 1, sample: text });
999+
}
1000+
}
8371001

838-
try {
1002+
const out: Array<{ pattern: string; count: number; sample: string }> =
1003+
[];
1004+
for (const [, g] of groups) {
1005+
const tpl =
1006+
miner.match(g.sample, 'fallback')?.getTemplate() ?? g.sample;
1007+
out.push({ pattern: tpl, count: g.count, sample: g.sample });
1008+
}
1009+
out.sort((a, b) => b.count - a.count);
1010+
1011+
for (const p of out) {
1012+
process.stdout.write(JSON.stringify(p) + '\n');
1013+
}
1014+
return;
1015+
}
1016+
1017+
// Default: raw SQL → stdout in the requested format.
8391018
const resultSet = await chClient.query({
8401019
query: opts.sql,
8411020
format: opts.format,
842-
connectionId: source.connection,
1021+
connectionId: opts.connectionId,
8431022
});
8441023
const text = await resultSet.text();
8451024
process.stdout.write(text);
@@ -849,6 +1028,28 @@ Examples:
8491028
}
8501029
} catch (err) {
8511030
const msg = err instanceof Error ? err.message : String(err);
1031+
1032+
// On error, check whether the connection ID was even valid —
1033+
// gives agents a clear signal of which class of failure occurred.
1034+
try {
1035+
const connections = await client.getConnections();
1036+
const known = connections.some(
1037+
c => c.id === opts.connectionId || c._id === opts.connectionId,
1038+
);
1039+
if (!known) {
1040+
_origError(
1041+
chalk.red(`Connection "${opts.connectionId}" not found.\n`),
1042+
);
1043+
_origError('Available connections:');
1044+
for (const c of connections) {
1045+
_origError(` - ${c.name} [${c.id}]`);
1046+
}
1047+
process.exit(1);
1048+
}
1049+
} catch {
1050+
// Couldn't list connections — fall through to the generic error.
1051+
}
1052+
8521053
_origError(chalk.red(`Query failed: ${msg}\n`));
8531054
process.exit(1);
8541055
}

0 commit comments

Comments
 (0)