Repository navigation
Commit 34d829c
authored
feat: add an LLM dashboard user filter and make presence filters index-eligible (#3104)
## Summary
Two changes to the LLM dashboard, both touching how its `WHERE` clauses are built.
**Performance.** Every tile carries a `WHERE` testing whether attribute keys are present, expressed as `SpanAttributes['key'] != ''` — the one shape that can't be indexed. `!= ''` normalizes to `notEmpty()`, which no skip index supports, so every granule is scanned. Map subscripts are also subcolumn references, so on ClickHouse 26.3+ each distinct key adds a per-part size lookup during PREWHERE planning (HDX-5348). These filters now lead with `mapContains`, which the `mapKeys()` index serves and which costs no per-part lookups.
**Feature.** Adds a user filter beside the existing session filter, so the dashboard can be scoped to one end user.
<details>
<summary>Index and function references</summary>
ClickHouse's text index condition ([`MergeTreeIndexConditionText::isSupportedFunction`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/MergeTreeIndexConditionText.cpp)) serves `mapContainsKey` and `has()` against a `mapKeys()` index; a `bloom_filter` over `mapKeys()` answers the same shape. #3096 moved `idx_span_attr_key` to `text(tokenizer='array')`, which makes this pointed: the index is there and these queries couldn't use it.
`mapContains` is a case-sensitive alias of `mapContainsKey` ([FunctionsMapMiscellaneous.cpp:766](https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/array/FunctionsMapMiscellaneous.cpp)).
</details>
## Measured on staging
SharedMergeTree, 26.6.1, 162 active parts. `EXPLAIN` only — no rows read.
**Planning cost**, 10-key detection predicate (`isLLMSpan`), subcolumn sizes forced on:
| | duration | `S3GetObject` | outcome |
| :--- | ---: | ---: | :--- |
| `!= ''` | 36,111 ms | 1,480 | `TIMEOUT_EXCEEDED` before start |
| `mapContains` | **7 ms** | **0** | ok |
**Granule pruning**, 2-key predicate over 6h (`EXPLAIN indexes = 1`):
| | parts | granules | skip index |
| :--- | ---: | ---: | :--- |
| `!= ''` | 19 | 1,306 | *no `Skip` section* |
| `mapContains` | 3 | **3** | `idx_span_attr_key` |
**Equivalence**: both forms return `66` over 7 days (248M rows scanned).
## How the filters are shaped
Review moved this on from the first version, which was bare `mapContains` everywhere. Presence alone admits a key explicitly set to `''`, and most of these gates are paired with a value expression the dashboard *groups by* — so an empty `user.email` became a blank "Top Users" row whose drill-in link went nowhere.
Gates that feed a group-by therefore keep their value check, and the presence term is pruning only:
```sql
indexHint(mapContains(a,'k1') OR mapContains(a,'k2')) AND coalesce(nullif(a['k1'],''), …) != ''
```
The value term defines the result; the presence term reaches skip-index analysis without being re-evaluated per surviving row. `indexHint` is the existing repo idiom for this (`mapKeyIndexExpression`, queryParser.ts:1728). It costs no extra per-part lookups, because the value term reads the same keys the group-by already reads.
Three positional rules fell out of this, each with a test pinning it:
- **`isLLMSpan` is bare presence, not hinted.** `indexHint` returns true at execution, so hinting a filter that *is* the presence check would match every span. This gate is where the planning win comes from.
- **Select-list gates are unhinted.** Skip-index analysis never reaches the select list, so a hint there folds to a constant while lengthening every query. `hasReportedTokens` is unhinted for this reason; `isToolSpan` is used in both positions and is exposed in both forms.
- **Hints sit above an `OR`, never inside one.** A skip index can only drop a granule when every arm of an `OR` is decidable by it, so `isToolSpan`'s hint is hoisted to cover the span-kind key too — sound because both arms imply one of the hinted keys.
## User filter
Follows the session filter exactly: a dropdown of the distinct values seen in the searched range, applied centrally in `baseLLMChartConfig` so every tab picks it up, and held in the URL rather than the search form so it can be set programmatically without two-way sync.
Users resolve through the same cross-dialect expression "Top Users" groups by (`user.email`, `enduser.id`, `user.id`, `ai.telemetry.metadata.userId`), so a dropdown value always matches the rows that produced it. The Errors tab's correlated **log** query is scoped too, matched against the log source's own attribute column.
`SessionSelect` and `UserSelect` are now thin wrappers over a shared `DistinctValueSelect`. That extraction also fixed a bug pre-existing in `SessionSelect`: an applied value absent from the fetched options fell back to the placeholder, so the control read "All users" while every chart stayed filtered. Absence isn't only staleness — it also happens before the query resolves (deep-linking a filtered URL) and when the value falls off the row limit — so the applied value is kept selectable rather than cleared.
`buildSessionCondition` is now `buildScopeCondition`: it was already generic, and there are two scopes.
## Behavior changes
**LLM span detection is presence-based**: a span carrying `gen_ai.system` at all is an LLM span whatever the value. Nothing groups by that predicate, so it can't produce a blank row. On staging this made no difference (66 = 66 over 248M rows) — OTel SDKs omit attributes rather than emitting empty strings — but it is a real semantic change and is in the changeset.
Every other gate preserves its exact previous semantics.
## Deliberately unchanged
- **Value expressions** (`model`, token counts, `costUsd`, `ttftMs`, …) still read subscripts — they need the values. They remain subject to the planning fan-out, which the client-level setting in #3102 covers.
- **`isToolSpan`'s `openinference.span.kind = 'TOOL'`** stays an equality: it discriminates one span kind from others, and the query builder rewrites equality onto the attribute-items index.
- **`hasTtft`** keeps `> 0` — it gates a p50/p95 and zero-valued rows would drag the percentiles down.
- **JSON attribute columns** are untouched. Their paths are real subcolumns, there is no key index to prune with, and a presence term would only duplicate reads — so the JSON branch emits the original expression, pinned by `toBe` assertions.
- **`buildScopeCondition`'s `coalesce(...) = 'value'`.** The indexable form is `has(SpanAttributeItems, 'key=value')`, but the items column name lives in `renderChartConfig`'s `textIndexInfoLookup` and isn't reachable from the app package. Follow-up.
## Caveat: materialized columns
`fastifySQL` rewrites a `Col['key']` subscript onto a materialized column when an operator has created one, and it matches on the subscript AST node — so it cannot see `mapContains`. Such tables never had the planning problem either, since a rewritten subscript is no longer a subcolumn reference. This trades that rewrite for skip-index pruning, which is the better deal only where those columns don't exist (the default, and what the numbers above were measured on).
Teaching `fastifySQL` to match the `mapContains` shape would fix this and benefit queryParser's existing `indexHint(mapContains(...))` emissions too, but it changes shared query planning for every source. Not in this PR.
## How to test on Vercel preview
**Preview route:** `/llm`
1. Open `/llm`; confirm the Overview tab renders tiles with data, and that values match what the page showed before this change.
2. Latency tab: "Time to First Token" still plots median and p95.
3. Sessions tab: the session dropdown populates and selecting one filters every tile.
4. **User filter**: pick a user from the new dropdown and confirm every tab rescopes, including the Errors tab's log rows. Copy the URL, open it in a new tab, and confirm the dropdown still shows the selected user rather than "All users".
5. Move the time picker to a window with no data for the selected user; the dropdown must keep showing that user (with a clear "×") rather than reading "All users" over empty charts.
## Verification
- `yarn ci:unit` (app): 233 suites, 3,827 tests, pass
- `yarn ci:unit` (common-utils): 36 suites, 2,526 tests, pass
- `make ci-lint`: 0 errors · `tsc --noEmit` clean · `knip` clean
- Staging `EXPLAIN` A/B above; result equivalence confirmed over 7 days of data
## Notes for reviewers
- **The staging numbers were taken on the bare `mapContains` shape**, before review moved most gates to `indexHint(presence) AND value`. `indexHint` feeds the same expression to index analysis, and the `isLLMSpan` headline (36s → 7ms) is directly valid since that gate is still bare presence — but the post-review shape was not re-measured on staging. It is covered by a `renderChartConfig` test asserting the kv-items rewrite still fires through an `indexHint`-bearing condition.
- Staging still runs the **bloom-filter** `idx_span_attr_key`, since #3096 isn't deployed there. `mapContains` is served by both bloom-filter and text indexes on `mapKeys`, so the pruning result holds either way — but the measurements are from the bloom-filter index.
- The two `EXPLAIN indexes = 1` runs report different part counts (16 vs 19) because ingest continued between them; it doesn't affect the granule comparison.
- Two changesets: `patch` for the perf fix, `minor` for the user filter.
## References
- Related: [HDX-5348](https://linear.app/clickhouse/issue/HDX-5348/ch-263-allow-calculating-subcolumns-sizes-for-merge-tree-reading-makes) — the 26.3 planning regression; #3102 handles it at the client level
- Builds on: #3096 (trace schema text indexes)1 parent 41eee7d commit 34d829c
21 files changed
Lines changed: 781 additions & 120 deletions
File tree
- .changeset
- packages
- app/src/llm
- __tests__
- dashboard
- lib
- common-utils/src/__tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
| |||
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
59 | 63 | | |
60 | 64 | | |
61 | 65 | | |
| |||
70 | 74 | | |
71 | 75 | | |
72 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
73 | 83 | | |
74 | 84 | | |
75 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
42 | | - | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
48 | 53 | | |
49 | 54 | | |
| 55 | + | |
50 | 56 | | |
51 | 57 | | |
52 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
49 | 59 | | |
50 | 60 | | |
51 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
33 | 42 | | |
34 | 43 | | |
35 | 44 | | |
| |||
123 | 132 | | |
124 | 133 | | |
125 | 134 | | |
126 | | - | |
| 135 | + | |
127 | 136 | | |
128 | 137 | | |
129 | 138 | | |
| |||
136 | 145 | | |
137 | 146 | | |
138 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
139 | 189 | | |
140 | 190 | | |
141 | 191 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
48 | 51 | | |
49 | 52 | | |
50 | 53 | | |
51 | 54 | | |
52 | 55 | | |
53 | | - | |
| 56 | + | |
54 | 57 | | |
55 | | - | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
56 | 62 | | |
57 | 63 | | |
58 | 64 | | |
59 | 65 | | |
| 66 | + | |
60 | 67 | | |
61 | 68 | | |
62 | 69 | | |
| |||
65 | 72 | | |
66 | 73 | | |
67 | 74 | | |
| 75 | + | |
68 | 76 | | |
69 | 77 | | |
0 commit comments