diff --git a/CHANGELOG.md b/CHANGELOG.md index 21afee7c..0ab5a0ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,14 @@ auto-generated per-PR notes; this file is the curated, human-readable history. ## [Unreleased] +### Changed +- Consolidated the non-Iceberg examples into three responsive flagship + dashboards: On-time flights, Shop analytics, and ClickHouse Operations. + Each uses an authored `grafana-grid@1` layout with a complete `flow@1` + fallback, shared filters, and focused visible tiles; useful secondary + analyses remain untiled Library entries. Removed the superseded KPI, logs, + query-log, System Explorer, and earlier Grafana-port fixtures and generator. + ### Added - **Cross-tab workspace consistency: refresh before write and stale-tab invalidation** (#343). Workbench and editable-Dashboard tabs editing the same diff --git a/README.md b/README.md index 4669066f..9f797c02 100644 --- a/README.md +++ b/README.md @@ -23,15 +23,20 @@ held at **100% test coverage**. See the [**feature tour, deployment guide and screenshots**](https://docs.altinity.com/altinity-sql-browser/) on the project site. Try it live on the Antalya demo cluster: **https://antalya.demo.altinity.cloud/sql**. -The [**ontime chart demo**](docs/ONTIME-CHART-DEMO.md) is a ready-made library of 10 -queries (load [`examples/ontime-charts.json`](examples/ontime-charts.json) via -**File ▾ → Import queries**) that walks through every chart type and feature against the public -`ontime` flight dataset. The [**system explorer demo**](docs/SYSTEM-EXPLORER-DEMO.md) -is a 14-query library (load [`examples/system-explorer-charts.json`](examples/system-explorer-charts.json) -via **File ▾ → Import queries**) that introspects ClickHouse's own `system` database — -running queries, merges/replication health, and historical query/part/error -activity — with a shared From/To filter driving every time-ranged Dashboard tile -at once. +Three flagship bundles cover the main workflows without a pile of one-feature +fixtures: + +- [**On-time flights**](docs/ONTIME-CHART-DEMO.md) — seven analytical tiles, + a KPI band, a shared 2023 date range, and carrier/airport multiselects over + the public `ontime` dataset. +- [**Shop analytics**](docs/SHOP-ANALYTICS-DEMO.md) — seven business tiles over + the schema, sample data, materialized views, and dictionary created by + [`examples/shop-demo.sql`](examples/shop-demo.sql). +- [**ClickHouse Operations**](docs/CLICKHOUSE-OPERATIONS-DEMO.md) — sixteen + operator-first tiles for live health, resources, background work, and + investigation, with the remaining operational queries kept in the Library. + +Load the corresponding portable bundle from `examples/` with **File ▾ → Open…**. The [**Iceberg catalog explorer**](docs/ICEBERG-CATALOG-EXPLORER-DEMO.md) is a distributable installer + two dashboards for Iceberg data-lake catalogs: [`examples/iceberg-install.json`](examples/iceberg-install.json) generates the @@ -92,9 +97,9 @@ ordinary `.kpi-panel` grid, while a **favorited, explicitly-KPI-typed** Dashboar query instead joins a full-width **KPI band** — a flat, wrapping card stream with no per-favorite name, description, or statistics footer, spanning every flow Dashboard layout (Report/2/3 columns). Consecutive explicit KPI -favorites merge into one shared band. The complete -[`kpi-panel.json`](examples/kpi-panel.json) Library example can be opened from -**File ▾ → Import queries** to see both. +favorites merge into one shared band. The On-time, Shop, and Operations +flagship dashboards all include a production +KPI band alongside charts, tables, and logs. When constructing a named tuple from expressions, either enable alias-derived member names for the query: @@ -492,14 +497,10 @@ nested option values, limits each helper to 1,000 options, and falls back to the ordinary parameter field when a source, consumer type, or provider conflicts. Filter sources run and reconcile saved values before any Panel query starts. -The complete [`query-log-explorer.json`](examples/query-log-explorer.json) -Library example (load via **File ▾ → Import queries**) demonstrates every filter -variant against `system.query_log` on any cluster: three Filter sources, one -per option shape (`Array(Tuple(value, label))`, `Map(String, String)`, plain -`Array(T)`), alongside plain auto-detected numeric/text fields — a KPI panel, -four analytical Panels adapted from the Altinity KB's ["Handy queries for -system.query_log"](https://kb.altinity.com/altinity-kb-useful-queries/query_log/), -a Logs panel, and a Text panel explaining the demo. +The flagship bundles demonstrate the same contracts in context: readable +`Array(Tuple(value, label))` airport options in On-time, targeted country and +category filters in Shop, and inferred `Array(T)` multiselects for operational +query-log dimensions in ClickHouse Operations. ```sql SELECT diff --git a/docs/CLICKHOUSE-OPERATIONS-DEMO.md b/docs/CLICKHOUSE-OPERATIONS-DEMO.md new file mode 100644 index 00000000..360807cd --- /dev/null +++ b/docs/CLICKHOUSE-OPERATIONS-DEMO.md @@ -0,0 +1,27 @@ +# ClickHouse Operations dashboard + +[`examples/clickhouse-operations.json`](../examples/clickhouse-operations.json) +is an operator-first dashboard imported and curated from the ClickHouse +Operations bundle. Open it with **File ▾ → Open…** against a cluster where the +signed-in user can inspect the relevant `system` tables. + +Its sixteen visible tiles follow an investigation path: + +- overview: live KPIs, running queries, query rate, active connections; +- resources: CPU, memory, I/O utilization, network traffic; +- background work: merges, mutations, replication checks; +- investigation: errors, largest tables, query-hash metric, query details, and + recent server logs. + +The active time range defaults to `-1h` through `now`. `user`, `query_kind`, +`exception_code`, and `query_hash` are inferred searchable multiselects; their +consumers consistently use `Array(T)` parameters with `has(...)`. `metric` and +`is_initial_query` remain single-select controls, while time and log search are +scalar. The other downloaded operational queries remain untiled in the Library. + +Typical grants include `SELECT` on `system.metric_log`, +`system.asynchronous_metric_log`, `system.query_log`, `system.part_log`, +`system.text_log`, `system.parts`, `system.merges`, `system.mutations`, and +replication-related system tables. Exact availability depends on server +version and deployment policy; missing grants fail individual tiles without +hiding the rest of the dashboard. diff --git a/docs/ONTIME-CHART-DEMO.md b/docs/ONTIME-CHART-DEMO.md index da06463e..fe44cb59 100644 --- a/docs/ONTIME-CHART-DEMO.md +++ b/docs/ONTIME-CHART-DEMO.md @@ -1,65 +1,27 @@ -# Chart demo — the `ontime` flight dataset +# On-time flights dashboard -A ready-made **Library** of 10 analytical queries that show off every chart type and -feature in the Altinity® SQL Browser, running against the public US flight-history -dataset (`ontime`, ~230M rows, 1987–2025) on the Antalya demo cluster. +[`examples/ontime-charts.json`](../examples/ontime-charts.json) is the flagship +analytical dashboard for the public `ontime` flight dataset. Open it on the +[Antalya demo](https://antalya.demo.altinity.cloud/sql) with **File ▾ → Open…**. -- **Live demo:** **https://antalya.demo.altinity.cloud/sql** -- **The library file:** [`examples/ontime-charts.json`](../examples/ontime-charts.json) - ([raw download](https://raw.githubusercontent.com/Altinity/altinity-sql-browser/main/examples/ontime-charts.json)) -- **Reproduce it:** [`examples/mjs/build-ontime-charts.mjs`](../examples/mjs/build-ontime-charts.mjs) - regenerates the JSON (it derives each chart's schema key live with - `clickhouse-client --connection antalya`). +The authored grid contains seven visible tiles: Flight KPIs, Daily flights, +On-time rate, Busiest origins, Monthly carrier volume, Delay causes by carrier, +and Cancellation reasons. Additional chart analyses remain available as +untiled Library entries. -## Load it (≈30 seconds) +The active time range defaults to `2023-01-01` through `2023-12-31`. Carrier +and Origin airport start inactive and become searchable multiselects because +every targeted query declares `Array(String)` and uses `has(...)`. Airport +codes are the bound values; readable airport names are the labels. -1. Open **https://antalya.demo.altinity.cloud/sql** and sign in (**Continue with Google**, - or use the credentials box). -2. Download [`ontime-charts.json`](https://raw.githubusercontent.com/Altinity/altinity-sql-browser/main/examples/ontime-charts.json) - (right-click → Save link as…). -3. In the header, click **File ▾ → Open…** and pick the file. The library is renamed - **ontime-charts** and fills with 10 saved queries (confirm the replace if you already - had queries saved). -4. Click any query in the **Queries** panel — it runs and opens straight into its chart. - Switch **Table / JSON / Chart** at the top of the results, or change the **Type / X / Y / - Series** dropdowns to re-encode any chart live. +The dashboard reads `ontime.fact_ontime` and joins +`ontime.dim_airports` for origin names. To refresh panel schema keys against +the configured connection, run: -## What each query demonstrates +```bash +node examples/mjs/build-ontime-charts.mjs +``` -| # | Query | Chart | Feature | -|---|-------|-------|---------| -| 1 | Busiest origin airports — 2023 | Bar (horizontal) | categorical axis; joined to `dim_airports` for readable names; hover any bar (long or short) for its exact value | -| 2 | Flights by month — 2023 | Column | numeric `month` auto-detected as an ordinal axis; K/M-humanised value ticks | -| 3 | Daily flights — 2023 | Line | `Date` axis auto-detected as a time series (~365 points) | -| 4 | Daily on-time rate — 2023 | Area | filled time series (a percentage measure) | -| 5 | Cancellation reasons — 2023 | Pie | share of a small category set, with a legend | -| 6 | Monthly flights by carrier — 2023 | Grouped bars | a **Series** column (carrier) splits each month into per-carrier bars | -| 7 | Average delay breakdown by carrier — 2023 | Multi-measure columns | four measures plotted together (“All measures”) | -| 8 | Daily flights since 2022 | Line | result exceeds the chart cap → a **“first 500 of 1.5K rows”** note (the table keeps them all) | -| 9 | Flights by day of week — 2023 | Column | ordinal `dayofweek` axis | -| 10 | Worst average departure delay by airport — 2023 | Bar (horizontal) | a non-count measure (avg minutes), joined for names | - -Each saved query stores its panel configuration in the complete query Spec, so it reopens exactly as designed. (Charts -plot the first 500 rows; the full result is always available in the Table view.) - -## Direct links - -Every query is also reachable as a single shareable link — open one and the SQL **and** its -chart configuration are pre-loaded; press **Run**, then the **Chart** tab. (Inside the app, -the **Share** button copies the same kind of link for whatever you're looking at.) - -- **Bar** — [Busiest origin airports — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUXG4gICAgYS5EaXNwbGF5QWlycG9ydE5hbWUgQVMgYWlycG9ydCxcbiAgICBjb3VudCgpIEFTIGZsaWdodHNcbkZST00gb250aW1lLmZhY3Rfb250aW1lIEFTIGZcbklOTkVSIEpPSU4gb250aW1lLmRpbV9haXJwb3J0cyBBUyBhXG4gICAgT04gYS5BaXJwb3J0Q29kZSA9IGYuT3JpZ2luQ29kZSBBTkQgYS5Jc0xhdGVzdCA9IDFcbldIRVJFIGYuWWVhciA9IDIwMjNcbkdST1VQIEJZIGFpcnBvcnRcbk9SREVSIEJZIGZsaWdodHMgREVTQ1xuTElNSVQgMTUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImhiYXIiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6ImFpcnBvcnQ6U3RyaW5nfGZsaWdodHM6VUludDY0In19) -- **Column** — [Flights by month — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIE1vbnRoIEFTIG1vbnRoLCBjb3VudCgpIEFTIGZsaWdodHNcbkZST00gb250aW1lLmZhY3Rfb250aW1lXG5XSEVSRSBZZWFyID0gMjAyM1xuR1JPVVAgQlkgbW9udGhcbk9SREVSIEJZIG1vbnRoIiwiY2hhcnQiOnsiY2ZnIjp7InR5cGUiOiJiYXIiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6Im1vbnRoOlVJbnQ4fGZsaWdodHM6VUludDY0In19) -- **Line** — [Daily flights — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIEZsaWdodERhdGUgQVMgZGF0ZSwgY291bnQoKSBBUyBmbGlnaHRzXG5GUk9NIG9udGltZS5mYWN0X29udGltZVxuV0hFUkUgWWVhciA9IDIwMjNcbkdST1VQIEJZIGRhdGVcbk9SREVSIEJZIGRhdGUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImxpbmUiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6ImRhdGU6RGF0ZXxmbGlnaHRzOlVJbnQ2NCJ9fQ==) -- **Area** — [Daily on-time rate — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUXG4gICAgRmxpZ2h0RGF0ZSBBUyBkYXRlLFxuICAgIHJvdW5kKDEwMCAqIGNvdW50SWYoQXJyRGVsMTUgPSAwKSAvIGNvdW50KCksIDEpIEFTIG9uX3RpbWVfcGN0XG5GUk9NIG9udGltZS5mYWN0X29udGltZVxuV0hFUkUgWWVhciA9IDIwMjNcbkdST1VQIEJZIGRhdGVcbk9SREVSIEJZIGRhdGUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImFyZWEiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6ImRhdGU6RGF0ZXxvbl90aW1lX3BjdDpGbG9hdDY0In19) -- **Pie** — [Cancellation reasons — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUXG4gICAgbXVsdGlJZihDYW5jZWxsYXRpb25Db2RlID0gJ0EnLCAnQ2FycmllcicsXG4gICAgICAgICAgICBDYW5jZWxsYXRpb25Db2RlID0gJ0InLCAnV2VhdGhlcicsXG4gICAgICAgICAgICBDYW5jZWxsYXRpb25Db2RlID0gJ0MnLCAnTmF0aW9uYWwgQWlyIFN5c3RlbScsXG4gICAgICAgICAgICBDYW5jZWxsYXRpb25Db2RlID0gJ0QnLCAnU2VjdXJpdHknLCAnT3RoZXInKSBBUyByZWFzb24sXG4gICAgY291bnQoKSBBUyBjYW5jZWxsYXRpb25zXG5GUk9NIG9udGltZS5mYWN0X29udGltZVxuV0hFUkUgWWVhciA9IDIwMjMgQU5EIENhbmNlbGxlZCA9IDFcbkdST1VQIEJZIHJlYXNvblxuT1JERVIgQlkgY2FuY2VsbGF0aW9ucyBERVNDIiwiY2hhcnQiOnsiY2ZnIjp7InR5cGUiOiJwaWUiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6InJlYXNvbjpTdHJpbmd8Y2FuY2VsbGF0aW9uczpVSW50NjQifX0=) -- **Grouped columns** — [Monthly flights by carrier — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUXG4gICAgTW9udGggQVMgbW9udGgsXG4gICAgQ2FycmllciBBUyBjYXJyaWVyLFxuICAgIGNvdW50KCkgQVMgZmxpZ2h0c1xuRlJPTSBvbnRpbWUuZmFjdF9vbnRpbWVcbldIRVJFIFllYXIgPSAyMDIzIEFORCBDYXJyaWVyIElOICgnV04nLCAnQUEnLCAnREwnLCAnVUEnKVxuR1JPVVAgQlkgbW9udGgsIGNhcnJpZXJcbk9SREVSIEJZIG1vbnRoLCBjYXJyaWVyIiwiY2hhcnQiOnsiY2ZnIjp7InR5cGUiOiJiYXIiLCJ4IjowLCJ5IjpbMl0sInNlcmllcyI6MX0sImtleSI6Im1vbnRoOlVJbnQ4fGNhcnJpZXI6TG93Q2FyZGluYWxpdHkoU3RyaW5nKXxmbGlnaHRzOlVJbnQ2NCJ9fQ==) -- **Multi-measure columns** — [Average delay breakdown by carrier — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUXG4gICAgQ2FycmllciBBUyBjYXJyaWVyLFxuICAgIHJvdW5kKGF2ZyhDYXJyaWVyRGVsYXkpLCAxKSBBUyBjYXJyaWVyX2RlbGF5LFxuICAgIHJvdW5kKGF2ZyhXZWF0aGVyRGVsYXkpLCAxKSBBUyB3ZWF0aGVyX2RlbGF5LFxuICAgIHJvdW5kKGF2ZyhOQVNEZWxheSksIDEpIEFTIG5hc19kZWxheSxcbiAgICByb3VuZChhdmcoTGF0ZUFpcmNyYWZ0RGVsYXkpLCAxKSBBUyBsYXRlX2FpcmNyYWZ0X2RlbGF5XG5GUk9NIG9udGltZS5mYWN0X29udGltZVxuV0hFUkUgWWVhciA9IDIwMjMgQU5EIEFyckRlbDE1ID0gMVxuR1JPVVAgQlkgY2FycmllclxuT1JERVIgQlkgY2Fycmllcl9kZWxheSBERVNDXG5MSU1JVCAxMiIsImNoYXJ0Ijp7ImNmZyI6eyJ0eXBlIjoiYmFyIiwieCI6MCwieSI6WzEsMiwzLDRdLCJzZXJpZXMiOm51bGx9LCJrZXkiOiJjYXJyaWVyOkxvd0NhcmRpbmFsaXR5KFN0cmluZyl8Y2Fycmllcl9kZWxheTpOdWxsYWJsZShGbG9hdDY0KXx3ZWF0aGVyX2RlbGF5Ok51bGxhYmxlKEZsb2F0NjQpfG5hc19kZWxheTpOdWxsYWJsZShGbG9hdDY0KXxsYXRlX2FpcmNyYWZ0X2RlbGF5Ok51bGxhYmxlKEZsb2F0NjQpIn19) -- **Line (capped)** — [Daily flights since 2022](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIEZsaWdodERhdGUgQVMgZGF0ZSwgY291bnQoKSBBUyBmbGlnaHRzXG5GUk9NIG9udGltZS5mYWN0X29udGltZVxuV0hFUkUgRmxpZ2h0RGF0ZSA+PSAnMjAyMi0wMS0wMSdcbkdST1VQIEJZIGRhdGVcbk9SREVSIEJZIGRhdGUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImxpbmUiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6ImRhdGU6RGF0ZXxmbGlnaHRzOlVJbnQ2NCJ9fQ==) -- **Column** — [Flights by day of week — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIERheU9mV2VlayBBUyBkYXlvZndlZWssIGNvdW50KCkgQVMgZmxpZ2h0c1xuRlJPTSBvbnRpbWUuZmFjdF9vbnRpbWVcbldIRVJFIFllYXIgPSAyMDIzXG5HUk9VUCBCWSBkYXlvZndlZWtcbk9SREVSIEJZIGRheW9md2VlayIsImNoYXJ0Ijp7ImNmZyI6eyJ0eXBlIjoiYmFyIiwieCI6MCwieSI6WzFdLCJzZXJpZXMiOm51bGx9LCJrZXkiOiJkYXlvZndlZWs6VUludDh8ZmxpZ2h0czpVSW50NjQifX0=) -- **Bar** — [Worst average departure delay by airport — 2023](https://antalya.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUXG4gICAgYS5EaXNwbGF5QWlycG9ydE5hbWUgQVMgYWlycG9ydCxcbiAgICByb3VuZChhdmcoZi5EZXBEZWxheU1pbnV0ZXMpLCAxKSBBUyBhdmdfZGVwX2RlbGF5XG5GUk9NIG9udGltZS5mYWN0X29udGltZSBBUyBmXG5JTk5FUiBKT0lOIG9udGltZS5kaW1fYWlycG9ydHMgQVMgYVxuICAgIE9OIGEuQWlycG9ydENvZGUgPSBmLk9yaWdpbkNvZGUgQU5EIGEuSXNMYXRlc3QgPSAxXG5XSEVSRSBmLlllYXIgPSAyMDIzXG5HUk9VUCBCWSBhaXJwb3J0XG5IQVZJTkcgY291bnQoKSA+PSAxMDAwMFxuT1JERVIgQlkgYXZnX2RlcF9kZWxheSBERVNDXG5MSU1JVCAxNSIsImNoYXJ0Ijp7ImNmZyI6eyJ0eXBlIjoiaGJhciIsIngiOjAsInkiOlsxXSwic2VyaWVzIjpudWxsfSwia2V5IjoiYWlycG9ydDpTdHJpbmd8YXZnX2RlcF9kZWxheTpOdWxsYWJsZShGbG9hdDY0KSJ9fQ==) - -## Tables used - -- `ontime.fact_ontime` — one row per US domestic flight (dates, carrier, origin/dest, delays, cancellations, …). -- `ontime.dim_airports` — airport reference data; joined on `AirportCode = OriginCode AND IsLatest = 1` for human-readable airport names. +The generator preserves the authored tile order and sizes, filters and +defaults, KPI field configuration, semantic dashboard identity, and complete +`flow@1` fallback. diff --git a/docs/SHOP-ANALYTICS-DEMO.md b/docs/SHOP-ANALYTICS-DEMO.md new file mode 100644 index 00000000..f63235f8 --- /dev/null +++ b/docs/SHOP-ANALYTICS-DEMO.md @@ -0,0 +1,23 @@ +# Shop analytics dashboard + +The Shop example has two deliberately separate artifacts: + +- [`examples/shop-demo.sql`](../examples/shop-demo.sql) creates the `shop` + schema, sample events/products, dictionary, materialized views, aggregate + tables, and lineage relationships. +- [`examples/shop-charts.json`](../examples/shop-charts.json) contains the + reusable queries and authored dashboard, without duplicating setup SQL. + +Load the SQL once with a suitably privileged ClickHouse client, then open the +bundle with **File ▾ → Open…**. The dashboard shows Sales KPIs, Daily revenue, +Revenue by country, Revenue by category, Top products, Daily active users, and +Traffic by hour. Country revenue trends remain as an untiled Library analysis. + +The active range defaults to `-90d` through `now`. Country and Category are +inactive searchable multiselects. Their target lists are explicit because the +aggregate tables intentionally do not all carry both dimensions. + +The setup user needs create/insert privileges for `shop.*`. Dashboard users +need `SELECT ON shop.*`; `SELECT ON system.dictionaries` is also required to +show dictionary lineage. The dictionary's ClickHouse source user must be able +to read `shop.products`. diff --git a/docs/SYSTEM-EXPLORER-DEMO.md b/docs/SYSTEM-EXPLORER-DEMO.md deleted file mode 100644 index c0bf852e..00000000 --- a/docs/SYSTEM-EXPLORER-DEMO.md +++ /dev/null @@ -1,90 +0,0 @@ -# System explorer demo — introspecting ClickHouse® itself - -A ready-made **Library** of 14 queries against ClickHouse's own `system` database — -running queries, merges/mutations/replication health, storage, and historical -query/part/error activity — running on the OSS `github.demo` cluster. Ideas and -query shapes are adapted from Mikhail Filimonov's -[ClickHouse ops Grafana dashboard](https://gist.github.com/filimonov/271e5b27c085356c67db3c1bf2204506) -(68 panels covering `metric_log`, `asynchronous_metric_log`, `query_log`, -`query_views_log`, `part_log`, and `error_log`) — not ported 1:1 (no Grafana -template macros, no per-cluster time-series for every background-pool metric), -just enough to show the shape of "explore your own cluster" as a Library + -Dashboard, not a full monitoring reimplementation. - -The six historical queries (#9–#14) share **one pair of query variables**, -`{from:String}`/`{to:String}` (parsed with `parseDateTimeBestEffort`), instead -of each hardcoding its own `now() - INTERVAL …` window. Same names everywhere -means the Dashboard's global filter bar (#149 D3) renders a single **From / -To** field pair that re-runs all six time-ranged tiles together when you type -a new range — one filter, six charts. - -- **Live demo:** **https://github.demo.altinity.cloud/sql** -- **The library file:** [`examples/system-explorer-charts.json`](../examples/system-explorer-charts.json) - ([raw download](https://raw.githubusercontent.com/Altinity/altinity-sql-browser/main/examples/system-explorer-charts.json)) -- **Reproduce it:** [`examples/mjs/build-system-explorer-charts.mjs`](../examples/mjs/build-system-explorer-charts.mjs) - regenerates the JSON (it derives each chart's schema key live via `DESCRIBE`- - equivalent `FORMAT JSON`, with throwaway `--param_from`/`--param_to` values - bound just so ClickHouse can resolve column types — the shipped SQL keeps - the placeholders unbound for the browser to fill in). - -## Load it (≈30 seconds) - -1. Open **https://github.demo.altinity.cloud/sql** and sign in (**Continue with - GitHub** via Auth0, or use the credentials box — see - [LOGIN-SCREEN.md](LOGIN-SCREEN.md) for what each login path grants). -2. Download [`system-explorer-charts.json`](https://raw.githubusercontent.com/Altinity/altinity-sql-browser/main/examples/system-explorer-charts.json) - (right-click → Save link as…). -3. In the header, click **File ▾ → Append…** and pick the file (Append merges - into whatever's already in your Library, reporting `Added N`; use **Open…** - instead if you'd rather replace the whole Library). Eight of the fourteen - queries import already **favorited**. -4. Click **File ▾ → "Open as dashboard"** (or the Dashboard link in the - sidebar). Two KPI-less tiles (#6–8, live snapshots) render immediately; - the six time-ranged tiles (#9–13, minus the table-only #14) show an "Enter - a value for: from, to" placeholder until you type a range into the - dashboard's **From / To** filter fields — then all six re-run together. - `parseDateTimeBestEffort` accepts most absolute formats; e.g. From - `2026-07-01 00:00:00`, To `2026-07-05 00:00:00` (there's no relative - `now`/`today` shorthand — type a real timestamp for "to" as well). - -## What each query demonstrates - -| # | Query | View | What it shows | -|---|-------|------|----------------| -| 1 | Currently running queries | Table | `system.processes` live snapshot — often empty; that's a real result | -| 2 | Merges in progress | Table | `system.merges` — background merge progress + size | -| 3 | Mutations in progress | Table | `system.mutations WHERE NOT is_done`, with failure reason | -| 4 | Replication status | Table | `system.replicas` — delay, queue depth, leadership | -| 5 | Stuck replication queue entries | Table | `system.replication_queue WHERE num_tries > 0` | -| 6 | Largest tables by disk usage | Bar (horizontal) | `system.parts` summed per table | -| 7 | Active parts by table | Bar (horizontal) | part *count* per table — an early "too many parts" signal | -| 8 | Cumulative error counters | Bar (horizontal) | `system.errors` — every error code hit since restart | -| 9 | Queries per minute | Line | `system.query_log` bucketed per minute over `{from}`/`{to}`; `DateTime` axis auto-detected as time | -| 10 | Slowest query patterns — avg duration | Bar (horizontal) | `query_log` over `{from}`/`{to}`, grouped by `normalized_query_hash`, a non-count measure | -| 11 | Query errors over time | Grouped bars | `query_log` failures over `{from}`/`{to}`, **Series** = error name | -| 12 | Part lifecycle events over time | Grouped bars | `part_log` over `{from}`/`{to}`, **Series** = `event_type` (an `Enum8` column) | -| 13 | Memory usage over time | Line | `system.metric_log`'s `CurrentMetric_MemoryTracking` over `{from}`/`{to}`, averaged per minute | -| 14 | Query cost breakdown — slowest patterns (detail) | Table | the deep-dive version of #10 over `{from}`/`{to}`: executions, rows/bytes read, p99 memory | - -Rows 1–5 and 14 need `SELECT` on the relevant `system.*` table; rows 6–13 also -read `system.query_log`/`system.part_log`/`system.metric_log`, which most -demo/read-only users won't have — sign in with an account that has broader -`system` grants (or run it against your own cluster as an admin) to see all -fourteen populate. - -## Direct links - -Every chartable query is also reachable as a single shareable link — open one -and the SQL **and** its chart configuration are pre-loaded. Rows 6–8 need only -**Run**, then the **Chart** tab; rows 9–13 also need `from`/`to` values typed -into the variable strip below the editor before Run is enabled (the link -itself can't carry a variable *value*, only the query). - -- **Bar** — [Largest tables by disk usage](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIGNvbmNhdChkYXRhYmFzZSwgJy4nLCB0YWJsZSkgQVMgdGFibGUsIHN1bShieXRlc19vbl9kaXNrKSBBUyBkaXNrX2J5dGVzXG5GUk9NIHN5c3RlbS5wYXJ0c1xuV0hFUkUgYWN0aXZlXG5HUk9VUCBCWSBkYXRhYmFzZSwgdGFibGVcbk9SREVSIEJZIGRpc2tfYnl0ZXMgREVTQ1xuTElNSVQgMTUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImhiYXIiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6InRhYmxlOlN0cmluZ3xkaXNrX2J5dGVzOlVJbnQ2NCJ9fQ==) -- **Bar** — [Active parts by table](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIGNvbmNhdChkYXRhYmFzZSwgJy4nLCB0YWJsZSkgQVMgdGFibGUsIGNvdW50KCkgQVMgcGFydHNcbkZST00gc3lzdGVtLnBhcnRzXG5XSEVSRSBhY3RpdmVcbkdST1VQIEJZIGRhdGFiYXNlLCB0YWJsZVxuT1JERVIgQlkgcGFydHMgREVTQ1xuTElNSVQgMTUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImhiYXIiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6InRhYmxlOlN0cmluZ3xwYXJ0czpVSW50NjQifX0=) -- **Bar** — [Cumulative error counters](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIG5hbWUsIHZhbHVlIEFTIHRpbWVzXG5GUk9NIHN5c3RlbS5lcnJvcnNcbldIRVJFIHZhbHVlID4gMFxuT1JERVIgQlkgdmFsdWUgREVTQ1xuTElNSVQgMTUiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImhiYXIiLCJ4IjowLCJ5IjpbMV0sInNlcmllcyI6bnVsbH0sImtleSI6Im5hbWU6U3RyaW5nfHRpbWVzOlVJbnQ2NCJ9fQ==) -- **Line** — [Queries per minute](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIHRvU3RhcnRPZk1pbnV0ZShldmVudF90aW1lKSBBUyB0LCBjb3VudCgpIEFTIHF1ZXJpZXNcbkZST00gc3lzdGVtLnF1ZXJ5X2xvZ1xuV0hFUkUgZXZlbnRfdGltZSBCRVRXRUVOIHBhcnNlRGF0ZVRpbWVCZXN0RWZmb3J0KHtmcm9tOlN0cmluZ30pIEFORCBwYXJzZURhdGVUaW1lQmVzdEVmZm9ydCh7dG86U3RyaW5nfSkgQU5EIHR5cGUgPSAnUXVlcnlGaW5pc2gnXG5HUk9VUCBCWSB0XG5PUkRFUiBCWSB0IiwiY2hhcnQiOnsiY2ZnIjp7InR5cGUiOiJsaW5lIiwieCI6MCwieSI6WzFdLCJzZXJpZXMiOm51bGx9LCJrZXkiOiJ0OkRhdGVUaW1lfHF1ZXJpZXM6VUludDY0In19) -- **Bar** — [Slowest query patterns — avg duration](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIGxlZnQoYW55KHF1ZXJ5KSwgNTApIEFTIHF1ZXJ5LCBhdmcocXVlcnlfZHVyYXRpb25fbXMpIEFTIGF2Z19kdXJhdGlvbl9tc1xuRlJPTSBzeXN0ZW0ucXVlcnlfbG9nXG5XSEVSRSBldmVudF90aW1lIEJFVFdFRU4gcGFyc2VEYXRlVGltZUJlc3RFZmZvcnQoe2Zyb206U3RyaW5nfSkgQU5EIHBhcnNlRGF0ZVRpbWVCZXN0RWZmb3J0KHt0bzpTdHJpbmd9KSBBTkQgdHlwZSA9ICdRdWVyeUZpbmlzaCdcbkdST1VQIEJZIG5vcm1hbGl6ZWRfcXVlcnlfaGFzaFxuT1JERVIgQlkgYXZnX2R1cmF0aW9uX21zIERFU0NcbkxJTUlUIDE1IiwiY2hhcnQiOnsiY2ZnIjp7InR5cGUiOiJoYmFyIiwieCI6MCwieSI6WzFdLCJzZXJpZXMiOm51bGx9LCJrZXkiOiJxdWVyeTpTdHJpbmd8YXZnX2R1cmF0aW9uX21zOkZsb2F0NjQifX0=) -- **Grouped bars** — [Query errors over time](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIHRvU3RhcnRPZkhvdXIoZXZlbnRfdGltZSkgQVMgdCwgZXJyb3JDb2RlVG9OYW1lKGV4Y2VwdGlvbl9jb2RlKSBBUyBlcnJvciwgY291bnQoKSBBUyBuXG5GUk9NIHN5c3RlbS5xdWVyeV9sb2dcbldIRVJFIGV2ZW50X3RpbWUgQkVUV0VFTiBwYXJzZURhdGVUaW1lQmVzdEVmZm9ydCh7ZnJvbTpTdHJpbmd9KSBBTkQgcGFyc2VEYXRlVGltZUJlc3RFZmZvcnQoe3RvOlN0cmluZ30pIEFORCBleGNlcHRpb25fY29kZSAhPSAwXG5HUk9VUCBCWSB0LCBlcnJvclxuT1JERVIgQlkgdCIsImNoYXJ0Ijp7ImNmZyI6eyJ0eXBlIjoiYmFyIiwieCI6MCwieSI6WzJdLCJzZXJpZXMiOjF9LCJrZXkiOiJ0OkRhdGVUaW1lfGVycm9yOkxvd0NhcmRpbmFsaXR5KFN0cmluZyl8bjpVSW50NjQifX0=) -- **Grouped bars** — [Part lifecycle events over time](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIHRvU3RhcnRPZkhvdXIoZXZlbnRfdGltZSkgQVMgdCwgZXZlbnRfdHlwZSwgY291bnQoKSBBUyBuXG5GUk9NIHN5c3RlbS5wYXJ0X2xvZ1xuV0hFUkUgZXZlbnRfdGltZSBCRVRXRUVOIHBhcnNlRGF0ZVRpbWVCZXN0RWZmb3J0KHtmcm9tOlN0cmluZ30pIEFORCBwYXJzZURhdGVUaW1lQmVzdEVmZm9ydCh7dG86U3RyaW5nfSlcbkdST1VQIEJZIHQsIGV2ZW50X3R5cGVcbk9SREVSIEJZIHQiLCJjaGFydCI6eyJjZmciOnsidHlwZSI6ImJhciIsIngiOjAsInkiOlsyXSwic2VyaWVzIjoxfSwia2V5IjoidDpEYXRlVGltZXxldmVudF90eXBlOkVudW04KCdOZXdQYXJ0JyA9IDEsICdNZXJnZVBhcnRzJyA9IDIsICdEb3dubG9hZFBhcnQnID0gMywgJ1JlbW92ZVBhcnQnID0gNCwgJ011dGF0ZVBhcnQnID0gNSwgJ01vdmVQYXJ0JyA9IDYsICdNZXJnZVBhcnRzU3RhcnQnID0gNywgJ011dGF0ZVBhcnRTdGFydCcgPSA4KXxuOlVJbnQ2NCJ9fQ==) -- **Line** — [Memory usage over time](https://github.demo.altinity.cloud/sql#eyJfX2FzYiI6MSwic3FsIjoiU0VMRUNUIHRvU3RhcnRPZk1pbnV0ZShldmVudF90aW1lKSBBUyB0LCBhdmcoQ3VycmVudE1ldHJpY19NZW1vcnlUcmFja2luZykgQVMgbWVtb3J5X2J5dGVzXG5GUk9NIHN5c3RlbS5tZXRyaWNfbG9nXG5XSEVSRSBldmVudF90aW1lIEJFVFdFRU4gcGFyc2VEYXRlVGltZUJlc3RFZmZvcnQoe2Zyb206U3RyaW5nfSkgQU5EIHBhcnNlRGF0ZVRpbWVCZXN0RWZmb3J0KHt0bzpTdHJpbmd9KVxuR1JPVVAgQlkgdFxuT1JERVIgQlkgdCIsImNoYXJ0Ijp7ImNmZyI6eyJ0eXBlIjoibGluZSIsIngiOjAsInkiOlsxXSwic2VyaWVzIjpudWxsfSwia2V5IjoidDpEYXRlVGltZXxtZW1vcnlfYnl0ZXM6RmxvYXQ2NCJ9fQ==) diff --git a/docs/local-app.html b/docs/local-app.html index d1d4459d..8f1a0df9 100644 --- a/docs/local-app.html +++ b/docs/local-app.html @@ -150,7 +150,7 @@

Bundled demo endpoints

Want a guided tour? Load the ontime chart demo — ten ready-made queries that walk through every chart type — or the - system explorer demo, + ClickHouse Operations demo, fourteen queries that introspect ClickHouse's own system database.

diff --git a/examples/grafana-clickhouse-ops-enhanced.json b/examples/clickhouse-operations.json similarity index 86% rename from examples/grafana-clickhouse-ops-enhanced.json rename to examples/clickhouse-operations.json index 95ffdce4..3f548a92 100644 --- a/examples/grafana-clickhouse-ops-enhanced.json +++ b/examples/clickhouse-operations.json @@ -2,58 +2,104 @@ "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", "format": "altinity-sql-browser/portable-bundle", "version": 1, - "exportedAt": "2026-07-16T00:00:00.000Z", + "exportedAt": "2026-07-22T00:00:00.000Z", "metadata": { - "name": "ClickHouse operations", - "description": "Operational ClickHouse dashboard adapted from the Grafana dashboard." + "name": "ClickHouse Operations", + "description": "Operator-first server overview, resources, background work, and investigation views." }, "queries": [ { - "id": "gco-filter", - "sql": "SELECT\n [toUInt8(1), toUInt8(0)] AS is_initial_query,\n (\n SELECT arraySort(groupUniqArray(query_kind))\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND query_kind != ''\n ) AS query_kind,\n (\n SELECT arraySort(groupUniqArray(initial_user))\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND initial_user != ''\n ) AS user,\n (\n SELECT groupArray((exception_code AS value, concat(toString(exception_code), ' · ', errorCodeToName(exception_code)) AS label))\n FROM\n (\n SELECT exception_code\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND exception_code != 0\n GROUP BY exception_code\n ORDER BY count() DESC\n LIMIT 100\n )\n ) AS exception_code,\n (\n SELECT groupArray((normalized_query_hash AS value, concat(toString(normalized_query_hash), ' · ', substring(sample_query, 1, 80)) AS label))\n FROM\n (\n SELECT normalized_query_hash, anyHeavy(replaceAll(query, '\\n', ' ')) AS sample_query\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND normalized_query_hash != 0\n GROUP BY normalized_query_hash\n ORDER BY count() DESC\n LIMIT 80\n )\n ) AS query_hash,\n ['count', 'avg_duration', 'max_duration', 'cpu_time', 'read_bytes', 'written_bytes', 'avg_written_rows', 'result_bytes', 'network_bytes', 'memory', 'max_memory', 'network_wait', 'io_time', 'io_wait', 'zk_txns', 'read_bps', 'write_bps', 'parts_inserted', 'parts_inserted_avg', 'marks_load_time', 'marks_miss_rate', 'selected_parts', 'selected_ranges', 'selected_marks', 'exceptions', 'open_files', 'external_processing_files', 'threads1', 'threads2', 'threads3'] AS metric\nSETTINGS enable_named_columns_in_function_tuple = 1", + "id": "gco-kpi-overview", + "sql": "WITH snap AS (\n SELECT\n (SELECT max(event_time) FROM merge(system, '^metric_log')) - 300 AS t_prev,\n argMax(CurrentMetric_Query, event_time) AS q_now,\n round(argMax(CurrentMetric_MemoryTracking, event_time) / 1048576) AS mem_now,\n round(argMaxIf(CurrentMetric_MemoryTracking, event_time, event_time <= t_prev) / 1048576) AS mem_prev,\n argMax(CurrentMetric_PartsActive, event_time) AS parts_now,\n argMaxIf(CurrentMetric_PartsActive, event_time, event_time <= t_prev) AS parts_prev,\n round(max(CurrentMetric_MemoryTracking) / 1048576) AS mem_peak\n FROM merge(system, '^metric_log')\n)\nSELECT\n q_now AS running_queries,\n CAST((mem_now, mem_now - mem_prev) AS Tuple(value Int64, delta Int64)) AS memory_used,\n CAST((parts_now, parts_now - parts_prev) AS Tuple(value Int64, delta Int64)) AS active_parts,\n mem_peak AS peak_memory\nFROM snap", "specVersion": 1, "spec": { - "name": "Grafana port filters", - "favorite": false, - "description": "Filter source for is_initial_query, query_kind, user, exception_code, query_hash, and metric. Dynamic query-log options use a fixed last-seven-days window. Controls are single-select; blank means Grafana All. Query hashes are capped at the 80 most frequent.", + "name": "Overview · Live KPIs", + "description": "Current running queries, tracked/peak memory (MiB), and active parts from the latest system.metric_log snapshot. Memory and parts carry a delta against five minutes earlier.", + "favorite": true, + "view": "panel", + "panel": { + "cfg": { + "type": "kpi" + }, + "fieldConfig": { + "defaults": { + "noValue": "—" + }, + "columns": { + "active_parts": { + "displayName": "Active parts", + "description": "Active data parts across all tables.", + "decimals": 0, + "delta": { + "decimals": 0, + "positiveIsGood": false + } + }, + "memory_used": { + "displayName": "Memory tracked", + "description": "Server-tracked memory at the latest sample.", + "unit": " MiB", + "decimals": 0, + "delta": { + "unit": " MiB", + "decimals": 0, + "positiveIsGood": false + } + }, + "peak_memory": { + "displayName": "Peak memory", + "description": "Maximum tracked memory over the window.", + "unit": " MiB", + "decimals": 0 + }, + "running_queries": { + "displayName": "Running queries", + "description": "Concurrent queries at the latest sample.", + "decimals": 0, + "color": "#4f8cff" + } + } + } + }, "dashboard": { - "role": "filter" + "role": "panel", + "sizeHints": { + "preferred": "compact", + "minimum": "compact", + "aspectRatio": 2 + } } } }, { - "id": "gco-029-running-queries", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n max(CurrentMetric_Query) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", + "id": "gco-donut-query-kind", + "sql": "SELECT query_kind, count() AS queries\nFROM merge(system, '^query_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\n AND type != 'QueryStart'\n AND query_kind != ''\nGROUP BY query_kind\nORDER BY queries DESC", "specVersion": 1, "spec": { - "name": "Overview · Running Queries (max over interval)", - "favorite": true, - "description": "CurrentMetric_Query from system.metric_log. Single-node port; the original per-host series collapses to one line.", + "name": "Queries · Mix by kind (Donut)", + "description": "Share of query kinds (Select, Insert, …) from system.query_log over the dashboard window. Uses the Donut Style preset.", + "favorite": false, "view": "panel", "panel": { "cfg": { - "type": "line", + "type": "pie", + "style": { + "frame": "normal", + "legend": "show", + "shape": "donut" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { - "value": { - "displayName": "Running queries", + "queries": { + "displayName": "Queries", "unit": " queries", - "decimals": 0, - "description": "Maximum concurrent queries in each bucket." + "decimals": 0 } } } @@ -74,33 +120,80 @@ "specVersion": 1, "spec": { "name": "Other activities · Merges running", - "favorite": true, "description": "CurrentMetric_Merge from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, + "series": null + }, + "fieldConfig": { + "columns": { + "value": { + "displayName": "Merges", + "description": "Average active merge tasks in each bucket.", + "unit": " tasks", + "decimals": 1 + } + } + } + }, + "dashboard": { + "role": "panel", + "sizeHints": { + "preferred": "medium", + "minimum": "compact", + "aspectRatio": 1.5 + } + } + } + }, + { + "id": "gco-029-running-queries", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n max(CurrentMetric_Query) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", + "specVersion": 1, + "spec": { + "name": "Overview · Running Queries (max over interval)", + "description": "CurrentMetric_Query from system.metric_log. Single-node port; the original per-host series collapses to one line.", + "favorite": true, + "view": "panel", + "panel": { + "cfg": { + "type": "line", "style": { + "axes": "show", "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", "grid": "auto", - "axes": "show" - } + "legend": "auto", + "points": "auto", + "scale": "zero" + }, + "x": 0, + "y": [ + 1 + ], + "series": null }, "fieldConfig": { "columns": { "value": { - "displayName": "Merges", - "unit": " tasks", - "decimals": 1, - "description": "Average active merge tasks in each bucket." + "displayName": "Running queries", + "description": "Maximum concurrent queries in each bucket.", + "unit": " queries", + "decimals": 0 } } } @@ -121,33 +214,33 @@ "specVersion": 1, "spec": { "name": "Other activities · Mutations running", - "favorite": true, "description": "CurrentMetric_PartMutation from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Mutations", + "description": "Average active mutation tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active mutation tasks in each bucket." + "decimals": 1 } } } @@ -168,33 +261,33 @@ "specVersion": 1, "spec": { "name": "Other activities · Moves running", - "favorite": true, "description": "CurrentMetric_Move from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Moves", + "description": "Average active move tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active move tasks in each bucket." + "decimals": 1 } } } @@ -215,33 +308,33 @@ "specVersion": 1, "spec": { "name": "Other activities · DistributedSend running", - "favorite": true, "description": "CurrentMetric_DistributedSend from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Distributed sends", + "description": "Average active distributed-send tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active distributed-send tasks in each bucket." + "decimals": 1 } } } @@ -262,33 +355,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ReplicatedChecks running", - "favorite": true, "description": "CurrentMetric_ReplicatedChecks from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Replication checks", + "description": "Average active replication-check tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active replication-check tasks in each bucket." + "decimals": 1 } } } @@ -309,33 +402,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ReplicatedFetch running", - "favorite": true, "description": "CurrentMetric_ReplicatedFetch from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Replication fetches", + "description": "Average active replication-fetch tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active replication-fetch tasks in each bucket." + "decimals": 1 } } } @@ -356,33 +449,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ReplicatedSend running", - "favorite": true, "description": "CurrentMetric_ReplicatedSend from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Replication sends", + "description": "Average active replication-send tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active replication-send tasks in each bucket." + "decimals": 1 } } } @@ -403,33 +496,33 @@ "specVersion": 1, "spec": { "name": "Other activities · KafkaBackgroundReads running", - "favorite": true, "description": "CurrentMetric_KafkaBackgroundReads from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Kafka background reads", + "description": "Average active Kafka background-read tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active Kafka background-read tasks in each bucket." + "decimals": 1 } } } @@ -450,33 +543,33 @@ "specVersion": 1, "spec": { "name": "Other activities · RefreshingViews running", - "favorite": true, "description": "CurrentMetric_RefreshingViews from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Refreshing views", + "description": "Average active refreshable-view tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active refreshable-view tasks in each bucket." + "decimals": 1 } } } @@ -497,33 +590,33 @@ "specVersion": 1, "spec": { "name": "Other activities · KafkaWrites", - "favorite": true, "description": "CurrentMetric_KafkaWrites from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Kafka writes", + "description": "Average active Kafka write tasks in each bucket.", "unit": " tasks", - "decimals": 1, - "description": "Average active Kafka write tasks in each bucket." + "decimals": 1 } } } @@ -544,33 +637,33 @@ "specVersion": 1, "spec": { "name": "Other activities · BackgroundSchedulePoolTask", - "favorite": true, "description": "CurrentMetric_BackgroundSchedulePoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Background schedule tasks", + "description": "Average active BackgroundSchedulePool tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active BackgroundSchedulePool tasks." + "decimals": 1 } } } @@ -591,33 +684,33 @@ "specVersion": 1, "spec": { "name": "Other activities · BackgroundCommonPoolTask", - "favorite": true, "description": "CurrentMetric_BackgroundCommonPoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Background common tasks", + "description": "Average active BackgroundCommonPool tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active BackgroundCommonPool tasks." + "decimals": 1 } } } @@ -638,33 +731,33 @@ "specVersion": 1, "spec": { "name": "Other activities · BackgroundMovePoolTask", - "favorite": true, "description": "CurrentMetric_BackgroundMovePoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Background move tasks", + "description": "Average active BackgroundMovePool tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active BackgroundMovePool tasks." + "decimals": 1 } } } @@ -685,33 +778,33 @@ "specVersion": 1, "spec": { "name": "Other activities · BackgroundFetchesPoolTask", - "favorite": true, "description": "CurrentMetric_BackgroundFetchesPoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Background fetch tasks", + "description": "Average active BackgroundFetchesPool tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active BackgroundFetchesPool tasks." + "decimals": 1 } } } @@ -727,38 +820,38 @@ } }, { - "id": "gco-066-background-merges-mutations", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n avg(CurrentMetric_BackgroundMergesAndMutationsPoolTask) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", + "id": "gco-074-background-distributed", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n avg(CurrentMetric_BackgroundDistributedSchedulePoolTask) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", "specVersion": 1, "spec": { - "name": "Other activities · BackgroundMergesAndMutationsPoolTask", - "favorite": true, - "description": "CurrentMetric_BackgroundMergesAndMutationsPoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "name": "Other activities · BackgroundDistributedSchedulePoolTask", + "description": "CurrentMetric_BackgroundDistributedSchedulePoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { - "displayName": "Background merge/mutation tasks", + "displayName": "Background distributed tasks", + "description": "Average active distributed schedule tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active background merge and mutation tasks." + "decimals": 1 } } } @@ -774,38 +867,38 @@ } }, { - "id": "gco-074-background-distributed", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n avg(CurrentMetric_BackgroundDistributedSchedulePoolTask) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", + "id": "gco-066-background-merges-mutations", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n avg(CurrentMetric_BackgroundMergesAndMutationsPoolTask) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", "specVersion": 1, "spec": { - "name": "Other activities · BackgroundDistributedSchedulePoolTask", - "favorite": true, - "description": "CurrentMetric_BackgroundDistributedSchedulePoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "name": "Other activities · BackgroundMergesAndMutationsPoolTask", + "description": "CurrentMetric_BackgroundMergesAndMutationsPoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { - "displayName": "Background distributed tasks", + "displayName": "Background merge/mutation tasks", + "description": "Average active background merge and mutation tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active distributed schedule tasks." + "decimals": 1 } } } @@ -826,33 +919,33 @@ "specVersion": 1, "spec": { "name": "Other activities · BackgroundMessageBrokerSchedulePoolTask", - "favorite": true, "description": "CurrentMetric_BackgroundMessageBrokerSchedulePoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Background broker tasks", + "description": "Average active message-broker schedule tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active message-broker schedule tasks." + "decimals": 1 } } } @@ -873,33 +966,33 @@ "specVersion": 1, "spec": { "name": "Other activities · BackgroundBufferFlushSchedulePoolTask", - "favorite": true, "description": "CurrentMetric_BackgroundBufferFlushSchedulePoolTask from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Background buffer flush tasks", + "description": "Average active buffer-flush schedule tasks.", "unit": " tasks", - "decimals": 1, - "description": "Average active buffer-flush schedule tasks." + "decimals": 1 } } } @@ -920,33 +1013,33 @@ "specVersion": 1, "spec": { "name": "Other activities · GlobalThreadActive", - "favorite": true, "description": "CurrentMetric_GlobalThreadActive from system.metric_log, averaged into an automatic roughly-300-point time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Active global threads", + "description": "Average active global threads.", "unit": " threads", - "decimals": 1, - "description": "Average active global threads." + "decimals": 1 } } } @@ -967,33 +1060,33 @@ "specVersion": 1, "spec": { "name": "Other activities · Accounted time (ClickHouse, average over interval)", - "favorite": true, "description": "ProfileEvent timing columns normalized to microseconds and expanded into a Series column. This preserves the source dashboard dynamic COLUMNS approach but uses SQL Browser line-series pivoting.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "stacked", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "stacked" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Accounted time", - "unit": " µs", - "description": "ClickHouse timing components normalized to microseconds." + "description": "ClickHouse timing components normalized to microseconds.", + "unit": " µs" } } } @@ -1014,34 +1107,34 @@ "specVersion": 1, "spec": { "name": "Other activities · Active tasks in local task-specific pools (p99.9 over interval)", - "favorite": true, "description": "Dynamic *ThreadsActive metrics expanded to one SQL Browser line series per task pool.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "stacked", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "stacked" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Active pool tasks", + "description": "p99.9 active tasks split by task-specific pool.", "unit": " tasks", - "decimals": 0, - "description": "p99.9 active tasks split by task-specific pool." + "decimals": 0 } } } @@ -1062,33 +1155,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ZooKeeper transactions (average per interval)", - "favorite": true, "description": "ProfileEvent_ZooKeeperTransactions from system.metric_log.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", - "x": 0, - "y": [ - 1 - ], - "series": null, "style": { + "axes": "show", "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", "grid": "auto", - "axes": "show" - } + "legend": "auto", + "points": "auto", + "scale": "zero" + }, + "x": 0, + "y": [ + 1 + ], + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "ZooKeeper transactions", + "description": "Average ZooKeeper transactions in each bucket.", "unit": " txns", - "decimals": 1, - "description": "Average ZooKeeper transactions in each bucket." + "decimals": 1 } } } @@ -1109,33 +1202,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ZooKeeper latency (average per interval)", - "favorite": true, "description": "ZooKeeper wait microseconds divided by transactions for each automatic time bucket.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "hide", + "legend": "hide", + "points": "hide", + "scale": "data" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "hide", - "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "ZooKeeper latency", + "description": "Average ZooKeeper wait per transaction.", "unit": " µs", - "decimals": 1, - "description": "Average ZooKeeper wait per transaction." + "decimals": 1 } } } @@ -1156,33 +1249,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ZooKeeper in-flight requests (p95 per interval)", - "favorite": true, "description": "CurrentMetric_ZooKeeperRequest p95 from system.metric_log.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "ZooKeeper in-flight requests", + "description": "p95 in-flight ZooKeeper requests.", "unit": " requests", - "decimals": 1, - "description": "p95 in-flight ZooKeeper requests." + "decimals": 1 } } } @@ -1203,33 +1296,33 @@ "specVersion": 1, "spec": { "name": "Other activities · ZooKeeper traffic (average per interval)", - "favorite": true, "description": "Received bytes are positive and sent bytes negative, matching the Grafana directional plot.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "overlay", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "ZooKeeper traffic", - "unit": " B/s", - "description": "Received traffic is positive and sent traffic is negative." + "description": "Received traffic is positive and sent traffic is negative.", + "unit": " B/s" } } } @@ -1250,33 +1343,33 @@ "specVersion": 1, "spec": { "name": "System metrics · CPUs loaded (ClickHouse-scoped, average per interval)", - "favorite": true, "description": "ProfileEvent_OSCPUVirtualTimeMicroseconds converted to CPU-seconds per sample.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "hide", + "legend": "hide", + "points": "hide", + "scale": "data" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "hide", - "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "CPU time", + "description": "ClickHouse-scoped CPU-seconds per sample.", "unit": " CPU s", - "decimals": 2, - "description": "ClickHouse-scoped CPU-seconds per sample." + "decimals": 2 } } } @@ -1297,33 +1390,33 @@ "specVersion": 1, "spec": { "name": "System metrics · Memory usage (ClickHouse-scoped, max per interval)", - "favorite": true, "description": "Maximum CurrentMetric_MemoryTracking in bytes.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 1 - ], - "series": null, "style": { + "axes": "show", "curve": "linear", + "grid": "hide", + "legend": "hide", "points": "hide", - "stack": "overlay", "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 1 + ], + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Memory usage", - "unit": " B", - "description": "Maximum ClickHouse memory tracking in each bucket." + "description": "Maximum ClickHouse memory tracking in each bucket.", + "unit": " B" } } } @@ -1344,34 +1437,34 @@ "specVersion": 1, "spec": { "name": "System metrics · CPU usage (system-wide, average per interval)", - "favorite": true, "description": "1 - OSIdleTimeNormalized from system.asynchronous_metric_log.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 1 - ], - "series": null, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "overlay", "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 1 + ], + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "CPU usage", + "description": "System-wide non-idle CPU percentage.", "unit": "%", - "decimals": 1, - "description": "System-wide non-idle CPU percentage." + "decimals": 1 } } } @@ -1392,32 +1485,32 @@ "specVersion": 1, "spec": { "name": "System metrics · Load Average (system-wide, max per interval)", - "favorite": true, "description": "LoadAverage1 from system.asynchronous_metric_log.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "hide", + "legend": "hide", + "points": "hide", + "scale": "data" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "hide", - "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Load average", - "decimals": 2, - "description": "Maximum one-minute system load average." + "description": "Maximum one-minute system load average.", + "decimals": 2 } } } @@ -1438,34 +1531,34 @@ "specVersion": 1, "spec": { "name": "System metrics · IO utilization % (system-wide, average per interval)", - "favorite": true, "description": "BlockActiveTime per asynchronous-metrics update interval, one series per block device. Target is ClickHouse 25.8+, so the post-25.4 unit multiplier is used.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "overlay", "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "I/O utilization", + "description": "Block-device active-time percentage.", "unit": "%", - "decimals": 1, - "description": "Block-device active-time percentage." + "decimals": 1 } } } @@ -1486,33 +1579,33 @@ "specVersion": 1, "spec": { "name": "System metrics · Network In & Out (system-wide, average per interval)", - "favorite": true, "description": "Network receive bits/s positive and send bits/s negative, one series per interface.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "overlay", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Network traffic", - "unit": " bit/s", - "description": "Receive traffic is positive and send traffic is negative, split by interface." + "description": "Receive traffic is positive and send traffic is negative, split by interface.", + "unit": " bit/s" } } } @@ -1533,33 +1626,33 @@ "specVersion": 1, "spec": { "name": "System metrics · IOWait (ClickHouse-scoped, average per interval)", - "favorite": true, "description": "ProfileEvent_OSIOWaitMicroseconds converted to seconds.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "hide", + "legend": "hide", + "points": "hide", + "scale": "data" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "hide", - "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "I/O wait", + "description": "ClickHouse-scoped I/O wait time.", "unit": " s", - "decimals": 3, - "description": "ClickHouse-scoped I/O wait time." + "decimals": 3 } } } @@ -1580,33 +1673,33 @@ "specVersion": 1, "spec": { "name": "System metrics · CPUWait (ClickHouse-scoped, average per interval)", - "favorite": true, "description": "ProfileEvent_OSCPUWaitMicroseconds converted to seconds.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "hide", + "legend": "hide", + "points": "hide", + "scale": "data" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "hide", - "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "CPU wait", + "description": "ClickHouse-scoped CPU wait time.", "unit": " s", - "decimals": 3, - "description": "ClickHouse-scoped CPU wait time." + "decimals": 3 } } } @@ -1627,33 +1720,33 @@ "specVersion": 1, "spec": { "name": "System metrics · Disk reads & writes (ClickHouse-scoped, average per interval)", - "favorite": true, "description": "OS read bytes positive and write bytes negative.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "overlay", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Disk I/O", - "unit": " B/s", - "description": "Reads are positive and writes are negative." + "description": "Reads are positive and writes are negative.", + "unit": " B/s" } } } @@ -1674,33 +1767,33 @@ "specVersion": 1, "spec": { "name": "System metrics · Disk + page cache reads & writes (ClickHouse-scoped, average per interval)", - "favorite": true, "description": "OSReadChars/OSWriteChars include page-cache traffic; writes are plotted negative.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "overlay", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "overlay" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Disk and page-cache I/O", - "unit": " B/s", - "description": "Reads are positive and writes are negative, including page-cache traffic." + "description": "Reads are positive and writes are negative, including page-cache traffic.", + "unit": " B/s" } } } @@ -1721,32 +1814,32 @@ "specVersion": 1, "spec": { "name": "System metrics · IO bytes (system-wide, average per interval)", - "favorite": true, "description": "Block read bytes/s positive and write bytes/s negative, one series per block device.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "data" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "curve": "linear", - "points": "auto", - "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "I/O throughput", - "unit": " B/s", - "description": "Read values are positive and write values are negative, split by device." + "description": "Read values are positive and write values are negative, split by device.", + "unit": " B/s" } } } @@ -1767,33 +1860,33 @@ "specVersion": 1, "spec": { "name": "System metrics · IOPS (system-wide, average per interval)", - "favorite": true, "description": "Block read operations/s positive and write operations/s negative, one series per block device.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "line", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", - "points": "auto", - "scale": "data", - "legend": "auto", "grid": "auto", - "axes": "show" - } + "legend": "auto", + "points": "auto", + "scale": "data" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "I/O operations", + "description": "Read operations are positive and write operations are negative, split by device.", "unit": " ops/s", - "decimals": 1, - "description": "Read operations are positive and write operations are negative, split by device." + "decimals": 1 } } } @@ -1814,33 +1907,33 @@ "specVersion": 1, "spec": { "name": "System metrics · Queries Started (ClickHouse, average per interval)", - "favorite": true, "description": "ProfileEvent_Query from system.metric_log.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "line", + "style": { + "axes": "show", + "curve": "linear", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "zero" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "curve": "linear", - "points": "auto", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": null }, "fieldConfig": { "columns": { "value": { "displayName": "Queries started", + "description": "Average queries started in each bucket.", "unit": " queries", - "decimals": 1, - "description": "Average queries started in each bucket." + "decimals": 1 } } } @@ -1861,34 +1954,34 @@ "specVersion": 1, "spec": { "name": "System metrics · Connections active (ClickHouse, max per interval)", - "favorite": true, "description": "TCP, HTTP, interserver, and MySQL connections from CurrentMetric columns.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "area", - "x": 0, - "y": [ - 2 - ], - "series": 1, "style": { + "axes": "show", "curve": "linear", + "grid": "auto", + "legend": "auto", "points": "auto", - "stack": "stacked", "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "stack": "stacked" + }, + "x": 0, + "y": [ + 2 + ], + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Active connections", + "description": "Active TCP, HTTP, interserver, and MySQL connections.", "unit": " connections", - "decimals": 0, - "description": "Active TCP, HTTP, interserver, and MySQL connections." + "decimals": 0 } } } @@ -1905,37 +1998,37 @@ }, { "id": "gco-099-error-log", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n concat(error, ' (', toString(code), if(remote, ' remote', ''), ')') AS series,\n toFloat64(sum(value)) AS value\nFROM merge(system, '^error_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\n /*[ AND code = {exception_code:Int32} ]*/\nGROUP BY t, series\nORDER BY t, series", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n concat(error, ' (', toString(code), if(remote, ' remote', ''), ')') AS series,\n toFloat64(sum(value)) AS value\nFROM merge(system, '^error_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\n /*[ AND code = {exception_code:Array(Int32)} ]*/\nGROUP BY t, series\nORDER BY t, series", "specVersion": 1, "spec": { "name": "error_log (24.8+) · Errors over interval", - "favorite": true, "description": "Grafana status history converted to grouped columns. Series combine error name, code, and remote flag.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { "value": { "displayName": "Errors", + "description": "Error occurrences in each time bucket.", "unit": " errors", - "decimals": 0, - "description": "Error occurrences in each time bucket." + "decimals": 0 } } } @@ -1952,29 +2045,29 @@ }, { "id": "gco-025-query-hash", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n toString(normalized_query_hash) AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND query_kind = {query_kind:String} ]*/\n /*[ AND exception_code = {exception_code:Int32} ]*/\n /*[ AND initial_user = {user:String} ]*/\n /*[ AND normalized_query_hash = {query_hash:UInt64} ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 100 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n toString(normalized_query_hash) AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND has({query_kind:Array(String)}, query_kind) ]*/\n /*[ AND has({exception_code:Array(Int32)}, exception_code) ]*/\n /*[ AND has({user:Array(String)}, initial_user) ]*/\n /*[ AND has({query_hash:Array(UInt64)}, normalized_query_hash) ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 100 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", "specVersion": 1, "spec": { "name": "query_log by query hash · Selected metric by normalized query hash", - "favorite": true, "description": "Grafana status history converted to grouped columns. Long-running queries are represented in every automatic bucket they overlap. Blank metric means count.", + "favorite": true, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -1997,12 +2090,12 @@ }, { "id": "gco-030-query-hash-details", - "sql": "SELECT\n toString(normalized_query_hash) AS query_hash,\n argMax(query_id, ProfileEvents['OSCPUVirtualTimeMicroseconds']) AS sample_query_id,\n replaceAll(argMax(query, ProfileEvents['OSCPUVirtualTimeMicroseconds']), '\\n', ' ') AS query,\n count() AS count,\n max(query_duration_ms) AS max_duration_ms,\n avg(query_duration_ms) AS avg_duration_ms,\n sum(ProfileEvents['OSCPUVirtualTimeMicroseconds']) / 1000000 AS cpu_time_s,\n quantile(0.99)(memory_usage) AS memory_q99,\n sum(read_rows) AS read_rows,\n sum(read_bytes) AS read_bytes,\n sum(written_rows) AS total_written_rows,\n avg(written_rows) AS avg_written_rows,\n sum(written_bytes) AS written_bytes,\n sum(result_rows) AS result_rows,\n sum(result_bytes) AS result_bytes,\n sum(ProfileEvents['NetworkReceiveBytes']) AS network_receive_bytes,\n sum(ProfileEvents['NetworkSendBytes']) AS network_send_bytes,\n sum(ProfileEvents['OSCPUVirtualTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0) AS cpu_usage,\n sum(ProfileEvents['OSIOWaitMicroseconds']) / nullIf(sum(ProfileEvents['RealTimeMicroseconds']), 0) AS io_wait_ratio,\n sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0) AS concurrency,\n arrayStringConcat(groupUniqArrayIf(5)(errorCodeToName(exception_code), exception_code != 0), ',') AS exceptions,\n arrayStringConcat(groupUniqArray(5)(initial_user), ',') AS users,\n sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000 AS os_io_wait_s,\n sum(ProfileEvents['DiskReadElapsedMicroseconds']) / 1000000 AS disk_read_s,\n sum(ProfileEvents['DiskWriteElapsedMicroseconds']) / 1000000 AS disk_write_s,\n sum(ProfileEvents['RealTimeMicroseconds']) / 1000000 AS real_time_s,\n sum(ProfileEvents['UserTimeMicroseconds']) / 1000000 AS user_time_s,\n sum(ProfileEvents['SystemTimeMicroseconds']) / 1000000 AS system_time_s,\n sum(ProfileEvents['NetworkSendElapsedMicroseconds']) / 1000000 AS network_send_s,\n sum(ProfileEvents['NetworkReceiveElapsedMicroseconds']) / 1000000 AS network_receive_s,\n sum(ProfileEvents['SelectedParts']) AS selected_parts,\n sum(ProfileEvents['SelectedRanges']) AS selected_ranges,\n sum(ProfileEvents['SelectedMarks']) AS selected_marks,\n sum(ProfileEvents['SelectedRows']) AS selected_rows,\n sum(ProfileEvents['SelectedBytes']) AS selected_bytes,\n sum(ProfileEvents['FileOpen']) AS file_open,\n sum(ProfileEvents['ZooKeeperTransactions']) AS zookeeper_transactions,\n sum(ProfileEvents['OSReadBytes']) AS os_read_bytes_excluding_page_cache,\n sum(ProfileEvents['OSWriteBytes']) AS os_write_bytes_excluding_page_cache,\n sum(ProfileEvents['OSReadChars']) AS os_read_chars_including_page_cache,\n sum(ProfileEvents['OSWriteChars']) AS os_write_chars_including_page_cache,\n anyIf(exception, exception != '') AS exception_sample,\n min(event_time) AS min_event_time,\n max(event_time) AS max_event_time\nFROM merge(system, '^query_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND query_kind = {query_kind:String} ]*/\n /*[ AND exception_code = {exception_code:Int32} ]*/\n /*[ AND initial_user = {user:String} ]*/\n /*[ AND normalized_query_hash = {query_hash:UInt64} ]*/\nGROUP BY normalized_query_hash\nORDER BY count DESC\nLIMIT 200", + "sql": "SELECT\n toString(normalized_query_hash) AS query_hash,\n argMax(query_id, ProfileEvents['OSCPUVirtualTimeMicroseconds']) AS sample_query_id,\n replaceAll(argMax(query, ProfileEvents['OSCPUVirtualTimeMicroseconds']), '\\n', ' ') AS query,\n count() AS count,\n max(query_duration_ms) AS max_duration_ms,\n avg(query_duration_ms) AS avg_duration_ms,\n sum(ProfileEvents['OSCPUVirtualTimeMicroseconds']) / 1000000 AS cpu_time_s,\n quantile(0.99)(memory_usage) AS memory_q99,\n sum(read_rows) AS read_rows,\n sum(read_bytes) AS read_bytes,\n sum(written_rows) AS total_written_rows,\n avg(written_rows) AS avg_written_rows,\n sum(written_bytes) AS written_bytes,\n sum(result_rows) AS result_rows,\n sum(result_bytes) AS result_bytes,\n sum(ProfileEvents['NetworkReceiveBytes']) AS network_receive_bytes,\n sum(ProfileEvents['NetworkSendBytes']) AS network_send_bytes,\n sum(ProfileEvents['OSCPUVirtualTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0) AS cpu_usage,\n sum(ProfileEvents['OSIOWaitMicroseconds']) / nullIf(sum(ProfileEvents['RealTimeMicroseconds']), 0) AS io_wait_ratio,\n sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0) AS concurrency,\n arrayStringConcat(groupUniqArrayIf(5)(errorCodeToName(exception_code), exception_code != 0), ',') AS exceptions,\n arrayStringConcat(groupUniqArray(5)(initial_user), ',') AS users,\n sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000 AS os_io_wait_s,\n sum(ProfileEvents['DiskReadElapsedMicroseconds']) / 1000000 AS disk_read_s,\n sum(ProfileEvents['DiskWriteElapsedMicroseconds']) / 1000000 AS disk_write_s,\n sum(ProfileEvents['RealTimeMicroseconds']) / 1000000 AS real_time_s,\n sum(ProfileEvents['UserTimeMicroseconds']) / 1000000 AS user_time_s,\n sum(ProfileEvents['SystemTimeMicroseconds']) / 1000000 AS system_time_s,\n sum(ProfileEvents['NetworkSendElapsedMicroseconds']) / 1000000 AS network_send_s,\n sum(ProfileEvents['NetworkReceiveElapsedMicroseconds']) / 1000000 AS network_receive_s,\n sum(ProfileEvents['SelectedParts']) AS selected_parts,\n sum(ProfileEvents['SelectedRanges']) AS selected_ranges,\n sum(ProfileEvents['SelectedMarks']) AS selected_marks,\n sum(ProfileEvents['SelectedRows']) AS selected_rows,\n sum(ProfileEvents['SelectedBytes']) AS selected_bytes,\n sum(ProfileEvents['FileOpen']) AS file_open,\n sum(ProfileEvents['ZooKeeperTransactions']) AS zookeeper_transactions,\n sum(ProfileEvents['OSReadBytes']) AS os_read_bytes_excluding_page_cache,\n sum(ProfileEvents['OSWriteBytes']) AS os_write_bytes_excluding_page_cache,\n sum(ProfileEvents['OSReadChars']) AS os_read_chars_including_page_cache,\n sum(ProfileEvents['OSWriteChars']) AS os_write_chars_including_page_cache,\n anyIf(exception, exception != '') AS exception_sample,\n min(event_time) AS min_event_time,\n max(event_time) AS max_event_time\nFROM merge(system, '^query_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND has({query_kind:Array(String)}, query_kind) ]*/\n /*[ AND has({exception_code:Array(Int32)}, exception_code) ]*/\n /*[ AND has({user:Array(String)}, initial_user) ]*/\n /*[ AND has({query_hash:Array(UInt64)}, normalized_query_hash) ]*/\nGROUP BY normalized_query_hash\nORDER BY count DESC\nLIMIT 200", "specVersion": 1, "spec": { "name": "query_log by query hash DETAILS · Query hash details", - "favorite": true, "description": "Resource and timing breakdown per normalized query hash, adapted to a single node and SQL Browser parameters.", + "favorite": true, "view": "panel", "panel": { "cfg": { @@ -2021,29 +2114,29 @@ }, { "id": "gco-056-query-table", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n arrayJoin(arrayFilter(name -> name NOT LIKE '%temporary%', tables)) AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND query_kind = {query_kind:String} ]*/\n /*[ AND exception_code = {exception_code:Int32} ]*/\n /*[ AND initial_user = {user:String} ]*/\n /*[ AND normalized_query_hash = {query_hash:UInt64} ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 50 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n arrayJoin(arrayFilter(name -> name NOT LIKE '%temporary%', tables)) AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND has({query_kind:Array(String)}, query_kind) ]*/\n /*[ AND has({exception_code:Array(Int32)}, exception_code) ]*/\n /*[ AND has({user:Array(String)}, initial_user) ]*/\n /*[ AND has({query_hash:Array(UInt64)}, normalized_query_hash) ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 50 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", "specVersion": 1, "spec": { "name": "query_log by table · Selected metric by table", - "favorite": true, "description": "Grafana status history converted to grouped columns. One query touching several tables contributes to each table series.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2066,29 +2159,29 @@ }, { "id": "gco-037-query-user", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n if(initial_user = '', '', initial_user) AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND query_kind = {query_kind:String} ]*/\n /*[ AND exception_code = {exception_code:Int32} ]*/\n /*[ AND initial_user = {user:String} ]*/\n /*[ AND normalized_query_hash = {query_hash:UInt64} ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 100 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n if(initial_user = '', '', initial_user) AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND has({query_kind:Array(String)}, query_kind) ]*/\n /*[ AND has({exception_code:Array(Int32)}, exception_code) ]*/\n /*[ AND has({user:Array(String)}, initial_user) ]*/\n /*[ AND has({query_hash:Array(UInt64)}, normalized_query_hash) ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 100 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", "specVersion": 1, "spec": { "name": "query_log by user · Selected metric by initial user", - "favorite": true, "description": "Grafana status history converted to grouped columns and grouped by initial_user.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2111,29 +2204,29 @@ }, { "id": "gco-026-query-host", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n hostName() AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND query_kind = {query_kind:String} ]*/\n /*[ AND exception_code = {exception_code:Int32} ]*/\n /*[ AND initial_user = {user:String} ]*/\n /*[ AND normalized_query_hash = {query_hash:UInt64} ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 100 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n t,\n series,\n value\nFROM\n(\n SELECT * FROM (\n WITH\n intDiv(toUInt32(event_time), bucket_s) AS finish_bucket,\n intDiv(toUInt32(query_start_time), bucket_s) AS start_bucket,\n arrayMap(i -> toDateTime((start_bucket + i) * bucket_s), range(toUInt32(finish_bucket - start_bucket + 1))) AS buckets\n SELECT\n arrayJoin(buckets) AS t,\n hostName() AS series,\n multiIf(\n /*[ {metric:String} = 'avg_duration', toFloat64(avg(query_duration_ms)), ]*/\n /*[ {metric:String} = 'max_duration', toFloat64(max(query_duration_ms)), ]*/\n /*[ {metric:String} = 'cpu_time', toFloat64(sum(ProfileEvents['UserTimeMicroseconds']) + sum(ProfileEvents['SystemTimeMicroseconds'])), ]*/\n /*[ {metric:String} = 'read_bytes', toFloat64(sum(read_bytes)), ]*/\n /*[ {metric:String} = 'written_bytes', toFloat64(sum(written_bytes)), ]*/\n /*[ {metric:String} = 'avg_written_rows', toFloat64(avg(written_rows)), ]*/\n /*[ {metric:String} = 'result_bytes', toFloat64(sum(result_bytes)), ]*/\n /*[ {metric:String} = 'network_bytes', toFloat64(sum(ProfileEvents['NetworkReceiveBytes']) + sum(ProfileEvents['NetworkSendBytes'])), ]*/\n /*[ {metric:String} = 'memory', toFloat64(sum(memory_usage)), ]*/\n /*[ {metric:String} = 'max_memory', toFloat64(max(memory_usage)), ]*/\n /*[ {metric:String} = 'network_wait', toFloat64((sum(ProfileEvents['NetworkSendElapsedMicroseconds']) + sum(ProfileEvents['NetworkReceiveElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_time', toFloat64((sum(ProfileEvents['DiskReadElapsedMicroseconds']) + sum(ProfileEvents['DiskWriteElapsedMicroseconds'])) / 1000000), ]*/\n /*[ {metric:String} = 'io_wait', toFloat64(sum(ProfileEvents['OSIOWaitMicroseconds']) / 1000000), ]*/\n /*[ {metric:String} = 'zk_txns', toFloat64(sum(ProfileEvents['ZooKeeperTransactions'])), ]*/\n /*[ {metric:String} = 'read_bps', toFloat64(sum(read_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'write_bps', toFloat64(sum(written_bytes) * 1000 / nullIf(sum(query_duration_ms), 0)), ]*/\n /*[ {metric:String} = 'parts_inserted', toFloat64(sum(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'parts_inserted_avg', toFloat64(avg(ProfileEvents['InsertedCompactParts'] + ProfileEvents['InsertedWideParts'])), ]*/\n /*[ {metric:String} = 'marks_load_time', toFloat64(sum(ProfileEvents['WaitMarksLoadMicroseconds'])), ]*/\n /*[ {metric:String} = 'marks_miss_rate', toFloat64(sum(ProfileEvents['MarkCacheMisses']) / nullIf(sum(ProfileEvents['MarkCacheHits']) + sum(ProfileEvents['MarkCacheMisses']), 0)), ]*/\n /*[ {metric:String} = 'selected_parts', toFloat64(sum(ProfileEvents['SelectedParts'])), ]*/\n /*[ {metric:String} = 'selected_ranges', toFloat64(sum(ProfileEvents['SelectedRanges'])), ]*/\n /*[ {metric:String} = 'selected_marks', toFloat64(sum(ProfileEvents['SelectedMarks'])), ]*/\n /*[ {metric:String} = 'exceptions', toFloat64(countIf(exception_code != 0)), ]*/\n /*[ {metric:String} = 'open_files', toFloat64(sum(ProfileEvents['FileOpen'])), ]*/\n /*[ {metric:String} = 'external_processing_files', toFloat64(sum(ProfileEvents['ExternalProcessingFilesTotal'])), ]*/\n /*[ {metric:String} = 'threads1', toFloat64(max(peak_threads_usage)), ]*/\n /*[ {metric:String} = 'threads2', toFloat64(max(length(thread_ids))), ]*/\n /*[ {metric:String} = 'threads3', toFloat64(sum(ProfileEvents['RealTimeMicroseconds']) / nullIf(1000 * sum(query_duration_ms), 0)), ]*/\n 1, toFloat64(count()),\n toFloat64(count())\n ) AS value\n FROM merge(system, '^query_log')\n WHERE event_time >= {from:DateTime} - INTERVAL 20 MINUTE\n AND event_time <= {to:DateTime} + INTERVAL 20 MINUTE\n AND type != 'QueryStart'\n /*[ AND is_initial_query = {is_initial_query:UInt8} ]*/\n /*[ AND has({query_kind:Array(String)}, query_kind) ]*/\n /*[ AND has({exception_code:Array(Int32)}, exception_code) ]*/\n /*[ AND has({user:Array(String)}, initial_user) ]*/\n /*[ AND has({query_hash:Array(UInt64)}, normalized_query_hash) ]*/\n GROUP BY t, series\n )\n ORDER BY count() OVER (PARTITION BY series) DESC, sum(value) OVER (PARTITION BY series) DESC\n LIMIT 100 BY t\n)\nWHERE t >= {from:DateTime} AND t <= {to:DateTime}\nORDER BY t, series", "specVersion": 1, "spec": { "name": "query_log by host · Selected metric by host", - "favorite": true, "description": "Retained for source fidelity. In the single-node port this normally contains one host series.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2160,25 +2253,25 @@ "specVersion": 1, "spec": { "name": "query_views_log · Selected metric by view name", - "favorite": true, "description": "Grafana status history converted to grouped columns. query_views_log fields are remapped to the common metric vocabulary; blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2205,25 +2298,25 @@ "specVersion": 1, "spec": { "name": "part_log: NewPart · part_log metric (NewPart by table)", - "favorite": true, "description": "Approximation of Grafana status history. The selected optional metric is grouped by time bucket and table for system.part_log event_type='NewPart'. Blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2250,25 +2343,25 @@ "specVersion": 1, "spec": { "name": "part_log: DownloadPart · part_log metric (DownloadPart by table)", - "favorite": true, "description": "Approximation of Grafana status history. The selected optional metric is grouped by time bucket and table for system.part_log event_type='DownloadPart'. Blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2295,25 +2388,25 @@ "specVersion": 1, "spec": { "name": "part_log: MergeParts · part_log metric (MergeParts by table)", - "favorite": true, "description": "Approximation of Grafana status history. The selected optional metric is grouped by time bucket and table for system.part_log event_type='MergeParts'. Blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2340,25 +2433,25 @@ "specVersion": 1, "spec": { "name": "part_log: MutatePart · part_log metric (MutatePart by table)", - "favorite": true, "description": "Approximation of Grafana status history. The selected optional metric is grouped by time bucket and table for system.part_log event_type='MutatePart'. Blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2385,25 +2478,25 @@ "specVersion": 1, "spec": { "name": "part_log: MovePart · part_log metric (MovePart by table)", - "favorite": true, "description": "Approximation of Grafana status history. The selected optional metric is grouped by time bucket and table for system.part_log event_type='MovePart'. Blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2430,25 +2523,25 @@ "specVersion": 1, "spec": { "name": "part_log: RemovePart · part_log metric (RemovePart by table)", - "favorite": true, "description": "Approximation of Grafana status history. The selected optional metric is grouped by time bucket and table for system.part_log event_type='RemovePart'. Blank metric means count.", + "favorite": false, "view": "panel", "panel": { "cfg": { "type": "bar", + "style": { + "axes": "show", + "density": "normal", + "grid": "auto", + "legend": "auto", + "mode": "stacked", + "scale": "zero" + }, "x": 0, "y": [ 2 ], - "series": 1, - "style": { - "mode": "stacked", - "density": "normal", - "scale": "zero", - "legend": "auto", - "grid": "auto", - "axes": "show" - } + "series": 1 }, "fieldConfig": { "columns": { @@ -2470,54 +2563,37 @@ } }, { - "id": "gco-kpi-overview", - "sql": "WITH snap AS (\n SELECT\n (SELECT max(event_time) FROM merge(system, '^metric_log')) - 300 AS t_prev,\n argMax(CurrentMetric_Query, event_time) AS q_now,\n round(argMax(CurrentMetric_MemoryTracking, event_time) / 1048576) AS mem_now,\n round(argMaxIf(CurrentMetric_MemoryTracking, event_time, event_time <= t_prev) / 1048576) AS mem_prev,\n argMax(CurrentMetric_PartsActive, event_time) AS parts_now,\n argMaxIf(CurrentMetric_PartsActive, event_time, event_time <= t_prev) AS parts_prev,\n round(max(CurrentMetric_MemoryTracking) / 1048576) AS mem_peak\n FROM merge(system, '^metric_log')\n)\nSELECT\n q_now AS running_queries,\n CAST((mem_now, mem_now - mem_prev) AS Tuple(value Int64, delta Int64)) AS memory_used,\n CAST((parts_now, parts_now - parts_prev) AS Tuple(value Int64, delta Int64)) AS active_parts,\n mem_peak AS peak_memory\nFROM snap", + "id": "gco-spark-memory", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n round(avg(CurrentMetric_MemoryTracking) / 1048576) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", "specVersion": 1, "spec": { - "name": "Overview · Live KPIs", - "favorite": true, - "description": "Current running queries, tracked/peak memory (MiB), and active parts from the latest system.metric_log snapshot. Memory and parts carry a delta against five minutes earlier.", + "name": "Memory · Tracked (Sparkline)", + "description": "Average tracked memory (MiB) per bucket from system.metric_log, drawn with the Sparkline Style preset (no axes, grid, legend, or markers — hover still works).", + "favorite": false, "view": "panel", "panel": { "cfg": { - "type": "kpi" + "type": "line", + "style": { + "axes": "hide", + "curve": "linear", + "grid": "hide", + "legend": "hide", + "points": "hide", + "scale": "data" + }, + "x": 0, + "y": [ + 1 + ], + "series": null }, "fieldConfig": { - "defaults": { - "noValue": "—" - }, "columns": { - "running_queries": { - "displayName": "Running queries", - "decimals": 0, - "color": "#4f8cff", - "description": "Concurrent queries at the latest sample." - }, - "memory_used": { + "value": { "displayName": "Memory tracked", "unit": " MiB", - "decimals": 0, - "description": "Server-tracked memory at the latest sample.", - "delta": { - "unit": " MiB", - "decimals": 0, - "positiveIsGood": false - } - }, - "active_parts": { - "displayName": "Active parts", - "decimals": 0, - "description": "Active data parts across all tables.", - "delta": { - "decimals": 0, - "positiveIsGood": false - } - }, - "peak_memory": { - "displayName": "Peak memory", - "unit": " MiB", - "decimals": 0, - "description": "Maximum tracked memory over the window." + "decimals": 0 } } } @@ -2525,42 +2601,45 @@ "dashboard": { "role": "panel", "sizeHints": { - "preferred": "compact", + "preferred": "medium", "minimum": "compact", - "aspectRatio": 2 + "aspectRatio": 1.5 } } } }, { - "id": "gco-donut-query-kind", - "sql": "SELECT query_kind, count() AS queries\nFROM merge(system, '^query_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\n AND type != 'QueryStart'\n AND query_kind != ''\nGROUP BY query_kind\nORDER BY queries DESC", + "id": "gco-smooth-load", + "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n avg(CurrentMetric_QueryThread) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", "specVersion": 1, "spec": { - "name": "Queries · Mix by kind (Donut)", - "favorite": true, - "description": "Share of query kinds (Select, Insert, …) from system.query_log over the dashboard window. Uses the Donut Style preset.", + "name": "Threads · Query threads (Smooth)", + "description": "Average active query threads per bucket from system.metric_log, drawn with the Smooth Style preset (monotone interpolation — no overshoot past the data).", + "favorite": false, "view": "panel", "panel": { "cfg": { - "type": "pie", + "type": "line", + "style": { + "axes": "show", + "curve": "smooth", + "grid": "auto", + "legend": "auto", + "points": "auto", + "scale": "data" + }, "x": 0, "y": [ 1 ], - "series": null, - "style": { - "shape": "donut", - "legend": "show", - "frame": "normal" - } + "series": null }, "fieldConfig": { "columns": { - "queries": { - "displayName": "Queries", - "unit": " queries", - "decimals": 0 + "value": { + "displayName": "Query threads", + "unit": " threads", + "decimals": 1 } } } @@ -2576,93 +2655,62 @@ } }, { - "id": "gco-spark-memory", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n round(avg(CurrentMetric_MemoryTracking) / 1048576) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", + "id": "gco-filter", + "sql": "SELECT\n [toUInt8(1), toUInt8(0)] AS is_initial_query,\n (\n SELECT arraySort(groupUniqArray(query_kind))\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND query_kind != ''\n ) AS query_kind,\n (\n SELECT arraySort(groupUniqArray(initial_user))\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND initial_user != ''\n ) AS user,\n (\n SELECT groupArray((exception_code AS value, concat(toString(exception_code), ' · ', errorCodeToName(exception_code)) AS label))\n FROM\n (\n SELECT exception_code\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND exception_code != 0\n GROUP BY exception_code\n ORDER BY count() DESC\n LIMIT 100\n )\n ) AS exception_code,\n (\n SELECT groupArray((normalized_query_hash AS value, concat(toString(normalized_query_hash), ' · ', substring(sample_query, 1, 80)) AS label))\n FROM\n (\n SELECT normalized_query_hash, anyHeavy(replaceAll(query, '\\n', ' ')) AS sample_query\n FROM merge(system, '^query_log')\n WHERE event_time >= now() - INTERVAL 7 DAY AND normalized_query_hash != 0\n GROUP BY normalized_query_hash\n ORDER BY count() DESC\n LIMIT 80\n )\n ) AS query_hash,\n ['count', 'avg_duration', 'max_duration', 'cpu_time', 'read_bytes', 'written_bytes', 'avg_written_rows', 'result_bytes', 'network_bytes', 'memory', 'max_memory', 'network_wait', 'io_time', 'io_wait', 'zk_txns', 'read_bps', 'write_bps', 'parts_inserted', 'parts_inserted_avg', 'marks_load_time', 'marks_miss_rate', 'selected_parts', 'selected_ranges', 'selected_marks', 'exceptions', 'open_files', 'external_processing_files', 'threads1', 'threads2', 'threads3'] AS metric\nSETTINGS enable_named_columns_in_function_tuple = 1", "specVersion": 1, "spec": { - "name": "Memory · Tracked (Sparkline)", + "name": "Grafana port filters", + "description": "Filter source for single-select metric/is_initial_query and inferred multiselect query_kind, user, exception_code, and query_hash controls. Blank means all.", + "favorite": false, + "dashboard": { + "role": "filter" + } + } + }, + { + "id": "ops-largest-tables", + "sql": "SELECT concat(database, '.', table) AS table, formatReadableSize(sum(bytes_on_disk)) AS disk, sum(bytes_on_disk) AS disk_bytes FROM system.parts WHERE active GROUP BY database, table ORDER BY disk_bytes DESC LIMIT 15", + "specVersion": 1, + "spec": { + "name": "Largest tables by disk", + "description": "Largest active tables by compressed bytes on disk.", "favorite": true, - "description": "Average tracked memory (MiB) per bucket from system.metric_log, drawn with the Sparkline Style preset (no axes, grid, legend, or markers — hover still works).", "view": "panel", "panel": { "cfg": { - "type": "line", - "x": 0, - "y": [ - 1 - ], - "series": null, - "style": { - "scale": "data", - "legend": "hide", - "grid": "hide", - "axes": "hide", - "curve": "linear", - "points": "hide" - } - }, - "fieldConfig": { - "columns": { - "value": { - "displayName": "Memory tracked", - "unit": " MiB", - "decimals": 0 - } - } + "type": "table" } }, "dashboard": { "role": "panel", "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 + "preferred": "wide", + "minimum": "medium", + "aspectRatio": 1.6 } } } }, { - "id": "gco-smooth-load", - "sql": "WITH\n greatest(1, toUInt32(ceil(greatest(1, dateDiff('second', {from:DateTime}, {to:DateTime})) / 300))) AS bucket_s\nSELECT\n toDateTime(intDiv(toUInt32(event_time), bucket_s) * bucket_s) AS t,\n avg(CurrentMetric_QueryThread) AS value\nFROM merge(system, '^metric_log')\nWHERE event_time >= {from:DateTime}\n AND event_time <= {to:DateTime}\nGROUP BY t\nORDER BY t", + "id": "ops-recent-server-logs", + "sql": "SELECT event_time, level, logger_name, message FROM system.text_log WHERE event_time BETWEEN {from:DateTime} AND {to:DateTime}\n/*[ AND positionCaseInsensitive(message, {search:String}) > 0 ]*/\nORDER BY event_time DESC LIMIT 500", "specVersion": 1, "spec": { - "name": "Threads · Query threads (Smooth)", + "name": "Recent server logs", + "description": "Most recent server log entries, optionally narrowed by message text.", "favorite": true, - "description": "Average active query threads per bucket from system.metric_log, drawn with the Smooth Style preset (monotone interpolation — no overshoot past the data).", "view": "panel", "panel": { "cfg": { - "type": "line", - "x": 0, - "y": [ - 1 - ], - "series": null, - "style": { - "scale": "data", - "legend": "auto", - "grid": "auto", - "axes": "show", - "curve": "smooth", - "points": "auto" - } - }, - "fieldConfig": { - "columns": { - "value": { - "displayName": "Query threads", - "unit": " threads", - "decimals": 1 - } - } + "type": "logs" } }, "dashboard": { "role": "panel", "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 + "preferred": "wide", + "minimum": "medium", + "aspectRatio": 1.6 } } } @@ -2671,389 +2719,270 @@ "dashboards": [ { "documentVersion": 1, - "id": "clickhouse-ops-enhanced", - "title": "ClickHouse operations", - "description": "Operational ClickHouse dashboard adapted from the Grafana dashboard.", + "id": "clickhouse-operations", + "title": "ClickHouse Operations", + "description": "Operator-first server overview, resources, background work, and investigation views.", "revision": 1, "layout": { - "type": "flow", + "type": "grafana-grid", "version": 1, - "preset": "columns-2", "items": { - "tile-gco-029-running-queries": { - "span": 1, - "height": "large" - }, - "tile-gco-047-merges": { - "span": 1, - "height": "large" - }, - "tile-gco-048-mutations": { - "span": 1, - "height": "large" - }, - "tile-gco-049-moves": { - "span": 1, - "height": "large" - }, - "tile-gco-050-distributed-send": { - "span": 1, - "height": "large" - }, - "tile-gco-064-replicated-checks": { - "span": 1, - "height": "large" - }, - "tile-gco-061-replicated-fetch": { - "span": 1, - "height": "large" - }, - "tile-gco-062-replicated-send": { - "span": 1, - "height": "large" - }, - "tile-gco-063-kafka-background-reads": { - "span": 1, - "height": "large" - }, - "tile-gco-065-refreshing-views": { - "span": 1, - "height": "large" - }, - "tile-gco-077-kafka-writes": { - "span": 1, - "height": "large" - }, - "tile-gco-071-background-schedule": { - "span": 1, - "height": "large" - }, - "tile-gco-068-background-common": { - "span": 1, - "height": "large" - }, - "tile-gco-069-background-move": { - "span": 1, - "height": "large" - }, - "tile-gco-067-background-fetches": { - "span": 1, - "height": "large" - }, - "tile-gco-066-background-merges-mutations": { - "span": 1, - "height": "large" - }, - "tile-gco-074-background-distributed": { - "span": 1, - "height": "large" - }, - "tile-gco-075-background-message-broker": { - "span": 1, - "height": "large" - }, - "tile-gco-073-background-buffer-flush": { - "span": 1, - "height": "large" - }, - "tile-gco-078-global-thread-active": { - "span": 1, - "height": "large" - }, - "tile-gco-094-accounted-time": { - "span": 1, - "height": "large" - }, - "tile-gco-076-active-pools": { - "span": 1, - "height": "large" - }, - "tile-gco-095-zk-transactions": { - "span": 1, - "height": "large" + "tile-gco-kpi-overview": { + "span": 12, + "height": 2 }, - "tile-gco-096-zk-latency": { - "span": 1, - "height": "large" + "tile-gco-029-running-queries": { + "span": 4, + "height": 3 }, - "tile-gco-097-zk-inflight": { - "span": 1, - "height": "large" + "tile-gco-040-queries-started": { + "span": 4, + "height": 3 }, - "tile-gco-098-zk-traffic": { - "span": 1, - "height": "large" + "tile-gco-055-connections": { + "span": 4, + "height": 3 }, "tile-gco-041-cpus-loaded": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 }, "tile-gco-053-memory": { - "span": 1, - "height": "large" - }, - "tile-gco-093-cpu-system": { - "span": 1, - "height": "large" - }, - "tile-gco-028-load-average": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 }, "tile-gco-058-io-util": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 }, "tile-gco-046-network": { - "span": 1, - "height": "large" - }, - "tile-gco-042-io-wait": { - "span": 1, - "height": "large" - }, - "tile-gco-043-cpu-wait": { - "span": 1, - "height": "large" - }, - "tile-gco-044-disk-io": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 }, - "tile-gco-045-page-cache-io": { - "span": 1, - "height": "large" - }, - "tile-gco-059-io-bytes": { - "span": 1, - "height": "large" - }, - "tile-gco-060-iops": { - "span": 1, - "height": "large" + "tile-gco-047-merges": { + "span": 6, + "height": 3 }, - "tile-gco-040-queries-started": { - "span": 1, - "height": "large" + "tile-gco-048-mutations": { + "span": 6, + "height": 3 }, - "tile-gco-055-connections": { - "span": 1, - "height": "large" + "tile-gco-064-replicated-checks": { + "span": 6, + "height": 3 }, "tile-gco-099-error-log": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 + }, + "tile-ops-largest-tables": { + "span": 6, + "height": 3 }, "tile-gco-025-query-hash": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 }, "tile-gco-030-query-hash-details": { - "span": 2, - "height": "large" - }, - "tile-gco-056-query-table": { - "span": 1, - "height": "large" - }, - "tile-gco-037-query-user": { - "span": 1, - "height": "large" - }, - "tile-gco-026-query-host": { - "span": 1, - "height": "large" - }, - "tile-gco-079-query-views": { - "span": 1, - "height": "large" - }, - "tile-gco-part-newpart": { - "span": 1, - "height": "large" - }, - "tile-gco-part-downloadpart": { - "span": 1, - "height": "large" - }, - "tile-gco-part-mergeparts": { - "span": 1, - "height": "large" - }, - "tile-gco-part-mutatepart": { - "span": 1, - "height": "large" - }, - "tile-gco-part-movepart": { - "span": 1, - "height": "large" - }, - "tile-gco-part-removepart": { - "span": 1, - "height": "large" - }, - "tile-gco-kpi-overview": { - "span": 1, - "height": "compact" - }, - "tile-gco-donut-query-kind": { - "span": 1, - "height": "large" - }, - "tile-gco-spark-memory": { - "span": 1, - "height": "large" - }, - "tile-gco-smooth-load": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 + }, + "tile-ops-recent-server-logs": { + "span": 6, + "height": 3 + } + }, + "fallback": { + "type": "flow", + "version": 1, + "preset": "columns-2", + "items": { + "tile-gco-kpi-overview": { + "span": 2, + "height": "compact" + }, + "tile-gco-029-running-queries": { + "span": 1, + "height": "large" + }, + "tile-gco-040-queries-started": { + "span": 1, + "height": "large" + }, + "tile-gco-055-connections": { + "span": 1, + "height": "large" + }, + "tile-gco-041-cpus-loaded": { + "span": 1, + "height": "large" + }, + "tile-gco-053-memory": { + "span": 1, + "height": "large" + }, + "tile-gco-058-io-util": { + "span": 1, + "height": "large" + }, + "tile-gco-046-network": { + "span": 1, + "height": "large" + }, + "tile-gco-047-merges": { + "span": 1, + "height": "large" + }, + "tile-gco-048-mutations": { + "span": 1, + "height": "large" + }, + "tile-gco-064-replicated-checks": { + "span": 1, + "height": "large" + }, + "tile-gco-099-error-log": { + "span": 1, + "height": "large" + }, + "tile-ops-largest-tables": { + "span": 2, + "height": "large" + }, + "tile-gco-025-query-hash": { + "span": 1, + "height": "large" + }, + "tile-gco-030-query-hash-details": { + "span": 2, + "height": "large" + }, + "tile-ops-recent-server-logs": { + "span": 2, + "height": "large" + } } } }, "filters": [ { - "id": "filter-from", - "parameter": "from" + "id": "ops-from", + "parameter": "from", + "label": "From", + "defaultValue": "-1h", + "defaultActive": true, + "targets": [ + "tile-gco-029-running-queries", + "tile-gco-040-queries-started", + "tile-gco-055-connections", + "tile-gco-041-cpus-loaded", + "tile-gco-053-memory", + "tile-gco-058-io-util", + "tile-gco-046-network", + "tile-gco-047-merges", + "tile-gco-048-mutations", + "tile-gco-064-replicated-checks", + "tile-gco-099-error-log", + "tile-gco-025-query-hash", + "tile-gco-030-query-hash-details", + "tile-ops-recent-server-logs" + ] + }, + { + "id": "ops-to", + "parameter": "to", + "label": "To", + "defaultValue": "now", + "defaultActive": true, + "targets": [ + "tile-gco-029-running-queries", + "tile-gco-040-queries-started", + "tile-gco-055-connections", + "tile-gco-041-cpus-loaded", + "tile-gco-053-memory", + "tile-gco-058-io-util", + "tile-gco-046-network", + "tile-gco-047-merges", + "tile-gco-048-mutations", + "tile-gco-064-replicated-checks", + "tile-gco-099-error-log", + "tile-gco-025-query-hash", + "tile-gco-030-query-hash-details", + "tile-ops-recent-server-logs" + ] + }, + { + "id": "ops-user", + "parameter": "user", + "label": "user", + "sourceQueryId": "gco-filter", + "defaultValue": [], + "defaultActive": false }, { - "id": "filter-to", - "parameter": "to" + "id": "ops-query_kind", + "parameter": "query_kind", + "label": "query kind", + "sourceQueryId": "gco-filter", + "defaultValue": [], + "defaultActive": false }, { - "id": "filter-exception_code", + "id": "ops-exception_code", "parameter": "exception_code", - "sourceQueryId": "gco-filter" - }, - { - "id": "filter-metric", - "parameter": "metric", - "sourceQueryId": "gco-filter" + "label": "exception code", + "sourceQueryId": "gco-filter", + "defaultValue": [], + "defaultActive": false }, { - "id": "filter-is_initial_query", - "parameter": "is_initial_query", - "sourceQueryId": "gco-filter" + "id": "ops-query_hash", + "parameter": "query_hash", + "label": "query hash", + "sourceQueryId": "gco-filter", + "defaultValue": [], + "defaultActive": false }, { - "id": "filter-query_kind", - "parameter": "query_kind", - "sourceQueryId": "gco-filter" + "id": "ops-metric", + "parameter": "metric", + "label": "Metric", + "sourceQueryId": "gco-filter", + "defaultValue": "count", + "defaultActive": true }, { - "id": "filter-user", - "parameter": "user", - "sourceQueryId": "gco-filter" + "id": "ops-initial", + "parameter": "is_initial_query", + "label": "Initial query", + "sourceQueryId": "gco-filter", + "defaultValue": 1, + "defaultActive": false }, { - "id": "filter-query_hash", - "parameter": "query_hash", - "sourceQueryId": "gco-filter" + "id": "ops-search", + "parameter": "search", + "label": "Log search", + "defaultValue": "", + "defaultActive": false } ], "tiles": [ { - "id": "tile-gco-029-running-queries", - "queryId": "gco-029-running-queries" - }, - { - "id": "tile-gco-047-merges", - "queryId": "gco-047-merges" - }, - { - "id": "tile-gco-048-mutations", - "queryId": "gco-048-mutations" - }, - { - "id": "tile-gco-049-moves", - "queryId": "gco-049-moves" - }, - { - "id": "tile-gco-050-distributed-send", - "queryId": "gco-050-distributed-send" - }, - { - "id": "tile-gco-064-replicated-checks", - "queryId": "gco-064-replicated-checks" - }, - { - "id": "tile-gco-061-replicated-fetch", - "queryId": "gco-061-replicated-fetch" - }, - { - "id": "tile-gco-062-replicated-send", - "queryId": "gco-062-replicated-send" - }, - { - "id": "tile-gco-063-kafka-background-reads", - "queryId": "gco-063-kafka-background-reads" - }, - { - "id": "tile-gco-065-refreshing-views", - "queryId": "gco-065-refreshing-views" - }, - { - "id": "tile-gco-077-kafka-writes", - "queryId": "gco-077-kafka-writes" - }, - { - "id": "tile-gco-071-background-schedule", - "queryId": "gco-071-background-schedule" - }, - { - "id": "tile-gco-068-background-common", - "queryId": "gco-068-background-common" - }, - { - "id": "tile-gco-069-background-move", - "queryId": "gco-069-background-move" - }, - { - "id": "tile-gco-067-background-fetches", - "queryId": "gco-067-background-fetches" - }, - { - "id": "tile-gco-066-background-merges-mutations", - "queryId": "gco-066-background-merges-mutations" - }, - { - "id": "tile-gco-074-background-distributed", - "queryId": "gco-074-background-distributed" - }, - { - "id": "tile-gco-075-background-message-broker", - "queryId": "gco-075-background-message-broker" - }, - { - "id": "tile-gco-073-background-buffer-flush", - "queryId": "gco-073-background-buffer-flush" - }, - { - "id": "tile-gco-078-global-thread-active", - "queryId": "gco-078-global-thread-active" - }, - { - "id": "tile-gco-094-accounted-time", - "queryId": "gco-094-accounted-time" - }, - { - "id": "tile-gco-076-active-pools", - "queryId": "gco-076-active-pools" - }, - { - "id": "tile-gco-095-zk-transactions", - "queryId": "gco-095-zk-transactions" + "id": "tile-gco-kpi-overview", + "queryId": "gco-kpi-overview" }, { - "id": "tile-gco-096-zk-latency", - "queryId": "gco-096-zk-latency" + "id": "tile-gco-029-running-queries", + "queryId": "gco-029-running-queries" }, { - "id": "tile-gco-097-zk-inflight", - "queryId": "gco-097-zk-inflight" + "id": "tile-gco-040-queries-started", + "queryId": "gco-040-queries-started" }, { - "id": "tile-gco-098-zk-traffic", - "queryId": "gco-098-zk-traffic" + "id": "tile-gco-055-connections", + "queryId": "gco-055-connections" }, { "id": "tile-gco-041-cpus-loaded", @@ -3063,14 +2992,6 @@ "id": "tile-gco-053-memory", "queryId": "gco-053-memory" }, - { - "id": "tile-gco-093-cpu-system", - "queryId": "gco-093-cpu-system" - }, - { - "id": "tile-gco-028-load-average", - "queryId": "gco-028-load-average" - }, { "id": "tile-gco-058-io-util", "queryId": "gco-058-io-util" @@ -3080,41 +3001,25 @@ "queryId": "gco-046-network" }, { - "id": "tile-gco-042-io-wait", - "queryId": "gco-042-io-wait" - }, - { - "id": "tile-gco-043-cpu-wait", - "queryId": "gco-043-cpu-wait" - }, - { - "id": "tile-gco-044-disk-io", - "queryId": "gco-044-disk-io" - }, - { - "id": "tile-gco-045-page-cache-io", - "queryId": "gco-045-page-cache-io" - }, - { - "id": "tile-gco-059-io-bytes", - "queryId": "gco-059-io-bytes" - }, - { - "id": "tile-gco-060-iops", - "queryId": "gco-060-iops" + "id": "tile-gco-047-merges", + "queryId": "gco-047-merges" }, { - "id": "tile-gco-040-queries-started", - "queryId": "gco-040-queries-started" + "id": "tile-gco-048-mutations", + "queryId": "gco-048-mutations" }, { - "id": "tile-gco-055-connections", - "queryId": "gco-055-connections" + "id": "tile-gco-064-replicated-checks", + "queryId": "gco-064-replicated-checks" }, { "id": "tile-gco-099-error-log", "queryId": "gco-099-error-log" }, + { + "id": "tile-ops-largest-tables", + "queryId": "ops-largest-tables" + }, { "id": "tile-gco-025-query-hash", "queryId": "gco-025-query-hash" @@ -3124,60 +3029,8 @@ "queryId": "gco-030-query-hash-details" }, { - "id": "tile-gco-056-query-table", - "queryId": "gco-056-query-table" - }, - { - "id": "tile-gco-037-query-user", - "queryId": "gco-037-query-user" - }, - { - "id": "tile-gco-026-query-host", - "queryId": "gco-026-query-host" - }, - { - "id": "tile-gco-079-query-views", - "queryId": "gco-079-query-views" - }, - { - "id": "tile-gco-part-newpart", - "queryId": "gco-part-newpart" - }, - { - "id": "tile-gco-part-downloadpart", - "queryId": "gco-part-downloadpart" - }, - { - "id": "tile-gco-part-mergeparts", - "queryId": "gco-part-mergeparts" - }, - { - "id": "tile-gco-part-mutatepart", - "queryId": "gco-part-mutatepart" - }, - { - "id": "tile-gco-part-movepart", - "queryId": "gco-part-movepart" - }, - { - "id": "tile-gco-part-removepart", - "queryId": "gco-part-removepart" - }, - { - "id": "tile-gco-kpi-overview", - "queryId": "gco-kpi-overview" - }, - { - "id": "tile-gco-donut-query-kind", - "queryId": "gco-donut-query-kind" - }, - { - "id": "tile-gco-spark-memory", - "queryId": "gco-spark-memory" - }, - { - "id": "tile-gco-smooth-load", - "queryId": "gco-smooth-load" + "id": "tile-ops-recent-server-logs", + "queryId": "ops-recent-server-logs" } ] } diff --git a/examples/kpi-panel.json b/examples/kpi-panel.json deleted file mode 100644 index ca839b08..00000000 --- a/examples/kpi-panel.json +++ /dev/null @@ -1,85 +0,0 @@ -{ - "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", - "format": "altinity-sql-browser/portable-bundle", - "version": 1, - "exportedAt": "2026-07-14T00:00:00.000Z", - "metadata": { - "name": "KPI panel example", - "description": "Scalar and named-tuple KPI presentation example." - }, - "queries": [ - { - "id": "kpi-service-health", - "sql": "SELECT count() AS active_users, (99.95 AS value, 0.08 AS delta) AS availability SETTINGS enable_named_columns_in_function_tuple = 1", - "specVersion": 1, - "spec": { - "name": "Service KPIs", - "description": "Scalar and named-tuple KPI cards from one SQL row.", - "favorite": true, - "view": "panel", - "panel": { - "cfg": { - "type": "kpi" - }, - "fieldConfig": { - "defaults": { - "noValue": "—" - }, - "columns": { - "active_users": { - "displayName": "Active users", - "color": "#4f8cff" - }, - "availability": { - "displayName": "Availability", - "description": "Current service availability.", - "unit": "%", - "decimals": 2, - "delta": { - "unit": " pp", - "decimals": 2, - "positiveIsGood": true - } - } - } - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "compact", - "minimum": "compact", - "aspectRatio": 2 - } - } - } - } - ], - "dashboards": [ - { - "documentVersion": 1, - "id": "kpi-panel-example", - "title": "KPI panel example", - "description": "Scalar and named-tuple KPI presentation example.", - "revision": 1, - "layout": { - "type": "flow", - "version": 1, - "preset": "report", - "items": { - "tile-kpi-service-health": { - "span": 1, - "height": "compact" - } - } - }, - "filters": [], - "tiles": [ - { - "id": "tile-kpi-service-health", - "queryId": "kpi-service-health" - } - ] - } - ] -} diff --git a/examples/mjs/README.md b/examples/mjs/README.md index f5526b95..43f41c15 100644 --- a/examples/mjs/README.md +++ b/examples/mjs/README.md @@ -2,8 +2,9 @@ The checked-in JSON files under `examples/` are canonical **portable bundle v1** documents. Query definitions use saved-query **Spec v1** and every -Dashboard example includes an explicit **Dashboard document v1** with tile -membership, flow-layout placement, and filter definitions. +Dashboard example includes an explicit **Dashboard document v1** with semantic +tile order, filter definitions, and either `flow@1` or `grafana-grid@1` layout. +Every grid layout carries a complete `flow@1` fallback. Legacy Library v1/v2 JSON remains importable for compatibility, but it is not an authoring format for new or regenerated examples. @@ -18,14 +19,15 @@ authoring format for new or regenerated examples. ## Generators -- `build-ontime-charts.mjs` regenerates `ontime-charts.json`. -- `build-system-explorer-charts.mjs` regenerates - `system-explorer-charts.json`. +- `build-ontime-charts.mjs` refreshes the live panel schema keys in + `ontime-charts.json` while preserving its authored grid, filters, KPI + configuration, tile order, and flow fallback. - `build-iceberg-install.mjs` regenerates `iceberg-install.json`. - `build-iceberg-dashboards.mjs` regenerates `iceberg-catalog-dashboard.json` and `iceberg-dba-dashboard.json`. - `example-bundle.mjs` owns the shared portable-bundle and Dashboard authoring - helpers used by those generators. + helpers, including explicit grid sizing, filters/defaults/targets, and flow + fallback generation. The dashboard generators that derive live result schema keys require an appropriately privileged ClickHouse client connection. The install generator diff --git a/examples/mjs/build-iceberg-dashboards.mjs b/examples/mjs/build-iceberg-dashboards.mjs index e4318173..b024c358 100644 --- a/examples/mjs/build-iceberg-dashboards.mjs +++ b/examples/mjs/build-iceberg-dashboards.mjs @@ -16,7 +16,7 @@ // `panel.key` exactly equals schemaKey(resultColumns) = "name:type|…"; a // wrong key silently re-derives axes (worse than none), so keys are derived // live from the real cluster through FORMAT JSON — same rationale as -// build-system-explorer-charts.mjs. +// the other example generators. // // Run: node examples/mjs/build-iceberg-dashboards.mjs // (needs the ice_meta_all views installed; ICE_CH_CMD overrides the diff --git a/examples/mjs/build-ontime-charts.mjs b/examples/mjs/build-ontime-charts.mjs index c5ece1ab..cb0de77d 100644 --- a/examples/mjs/build-ontime-charts.mjs +++ b/examples/mjs/build-ontime-charts.mjs @@ -1,208 +1,51 @@ -// Generator for examples/ontime-charts.json — a saved-queries "Library" file for -// the Altinity SQL Browser that demonstrates every chart feature against the -// public `ontime` flights dataset on the antalya cluster. +// Refresh the live schema keys in examples/ontime-charts.json against the +// configured antalya ClickHouse connection. The checked-in bundle is the +// authored source of truth for SQL, semantic tile order, grafana-grid sizing, +// filters/defaults/targets, KPI field configuration, and the flow fallback. // -// Why a generator: the browser only restores a saved chart config when the -// entry's `spec.panel.key` exactly equals schemaKey(resultColumns) = "name:type|…" -// (see src/ui/results.js chartCfgFor / src/core/chart-data.js schemaKey). -// Hand-writing those type strings is error-prone, so we derive each key live -// from `DESCRIBE ()` against the real cluster. -// -// Run: node examples/mjs/build-ontime-charts.mjs (needs `clickhouse-client --connection antalya`) -// Out: examples/ontime-charts.json +// Run: node examples/mjs/build-ontime-charts.mjs import { execFileSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; import { dirname, resolve } from 'node:path'; -import { buildDashboard, writeExampleBundle } from './example-bundle.mjs'; +import { writeExampleBundle } from './example-bundle.mjs'; const here = dirname(fileURLToPath(import.meta.url)); -const CONNECTION = 'antalya'; - -// Each spec: a query + the chart we want it to open with. `cfg` matches the -// app's shape { type, x, y:[...], series }; x/series are column indices, y a -// list of measure-column indices. `view:'panel'` makes a click open the panel. -const SPECS = [ - { - name: 'Busiest origin airports — 2023', - description: 'Top 15 departure airports by flight count (joined to dim_airports for readable names). Horizontal Bar — hover any bar, long or short, to read its exact value.', - cfg: { type: 'hbar', x: 0, y: [1], series: null }, - sql: `SELECT - a.DisplayAirportName AS airport, - count() AS flights -FROM ontime.fact_ontime AS f -INNER JOIN ontime.dim_airports AS a - ON a.AirportCode = f.OriginCode AND a.IsLatest = 1 -WHERE f.Year = 2023 -GROUP BY airport -ORDER BY flights DESC -LIMIT 15`, - }, - { - name: 'Flights by month — 2023', - description: 'Monthly US flight volume. A numeric column named "month" is detected as an ordinal axis → vertical Column chart, with K/M-humanised value ticks.', - cfg: { type: 'bar', x: 0, y: [1], series: null }, - sql: `SELECT Month AS month, count() AS flights -FROM ontime.fact_ontime -WHERE Year = 2023 -GROUP BY month -ORDER BY month`, - }, - { - name: 'Daily flights — 2023', - description: 'One point per day across 2023 (~365 rows). A Date X axis is auto-detected as a time series → Line chart.', - cfg: { type: 'line', x: 0, y: [1], series: null }, - sql: `SELECT FlightDate AS date, count() AS flights -FROM ontime.fact_ontime -WHERE Year = 2023 -GROUP BY date -ORDER BY date`, - }, - { - name: 'Daily on-time rate — 2023', - description: 'Share of flights arriving on time (< 15 min late) per day, as a percentage. Rendered as a filled Area chart.', - cfg: { type: 'area', x: 0, y: [1], series: null }, - sql: `SELECT - FlightDate AS date, - round(100 * countIf(ArrDel15 = 0) / count(), 1) AS on_time_pct -FROM ontime.fact_ontime -WHERE Year = 2023 -GROUP BY date -ORDER BY date`, - }, - { - name: 'Cancellation reasons — 2023', - description: 'Why flights were cancelled in 2023 (carrier / weather / national air system / security). A small categorical breakdown → Pie chart with a legend.', - cfg: { type: 'pie', x: 0, y: [1], series: null }, - sql: `SELECT - multiIf(CancellationCode = 'A', 'Carrier', - CancellationCode = 'B', 'Weather', - CancellationCode = 'C', 'National Air System', - CancellationCode = 'D', 'Security', 'Other') AS reason, - count() AS cancellations -FROM ontime.fact_ontime -WHERE Year = 2023 AND Cancelled = 1 -GROUP BY reason -ORDER BY cancellations DESC`, - }, - { - name: 'Monthly flights by carrier — 2023', - description: 'Flights per month split across four major carriers (WN, AA, DL, UA). The "carrier" column is used as the Series, producing grouped bars with a per-carrier legend.', - cfg: { type: 'bar', x: 0, y: [2], series: 1 }, - sql: `SELECT - Month AS month, - Carrier AS carrier, - count() AS flights -FROM ontime.fact_ontime -WHERE Year = 2023 AND Carrier IN ('WN', 'AA', 'DL', 'UA') -GROUP BY month, carrier -ORDER BY month, carrier`, - }, - { - name: 'Average delay breakdown by carrier — 2023', - description: 'Mean minutes of each delay cause (carrier, weather, NAS, late aircraft) for delayed flights, per carrier. Four measures plotted at once ("All measures") as grouped columns.', - cfg: { type: 'bar', x: 0, y: [1, 2, 3, 4], series: null }, - sql: `SELECT - Carrier AS carrier, - round(avg(CarrierDelay), 1) AS carrier_delay, - round(avg(WeatherDelay), 1) AS weather_delay, - round(avg(NASDelay), 1) AS nas_delay, - round(avg(LateAircraftDelay), 1) AS late_aircraft_delay -FROM ontime.fact_ontime -WHERE Year = 2023 AND ArrDel15 = 1 -GROUP BY carrier -ORDER BY carrier_delay DESC -LIMIT 12`, - }, - { - name: 'Daily flights since 2022', - description: 'Every day from 2022 onward (~1,460 points). The chart plots the first 500 and shows a "first 500 of N rows" note — the table view still has them all.', - cfg: { type: 'line', x: 0, y: [1], series: null }, - sql: `SELECT FlightDate AS date, count() AS flights -FROM ontime.fact_ontime -WHERE FlightDate >= '2022-01-01' -GROUP BY date -ORDER BY date`, - }, - { - name: 'Flights by day of week — 2023', - description: 'Volume by day of week (1 = Monday … 7 = Sunday). "dayofweek" is recognised as an ordinal axis → Column chart.', - cfg: { type: 'bar', x: 0, y: [1], series: null }, - sql: `SELECT DayOfWeek AS dayofweek, count() AS flights -FROM ontime.fact_ontime -WHERE Year = 2023 -GROUP BY dayofweek -ORDER BY dayofweek`, - }, - { - name: 'Worst average departure delay by airport — 2023', - description: 'Airports with the highest mean departure delay (minutes) among those with ≥ 10,000 departures in 2023. Horizontal Bar of a non-count measure, joined for names.', - cfg: { type: 'hbar', x: 0, y: [1], series: null }, - sql: `SELECT - a.DisplayAirportName AS airport, - round(avg(f.DepDelayMinutes), 1) AS avg_dep_delay -FROM ontime.fact_ontime AS f -INNER JOIN ontime.dim_airports AS a - ON a.AirportCode = f.OriginCode AND a.IsLatest = 1 -WHERE f.Year = 2023 -GROUP BY airport -HAVING count() >= 10000 -ORDER BY avg_dep_delay DESC -LIMIT 15`, - }, -]; +const outPath = resolve(here, '..', 'ontime-charts.json'); +const document = JSON.parse(readFileSync(outPath, 'utf8')); + +const ch = (query) => execFileSync( + 'clickhouse-client', [ + '--connection', 'antalya', + '--param_from', '2023-01-01', '--param_to', '2023-12-31', + '--query', query, + ], + { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }, +); -const ch = (query) => - execFileSync('clickhouse-client', ['--connection', CONNECTION, '--query', query], { - encoding: 'utf8', - maxBuffer: 64 * 1024 * 1024, - }); - -// schemaKey == columns.map(c => c.name + ':' + c.type).join('|'), derived from -// DESCRIBE so it matches exactly what the app receives at run time. function schemaKey(sql) { - const out = ch(`DESCRIBE (${sql})`); - return out + return ch(`DESCRIBE (${sql})`) .split('\n') - .filter((l) => l.trim()) - .map((l) => { const [name, type] = l.split('\t'); return `${name}:${type}`; }) + .filter((line) => line.trim()) + .map((line) => { + const [name, type] = line.split('\t'); + return `${name}:${type}`; + }) .join('|'); } -const resultRows = (sql) => Number(ch(`SELECT count() FROM (${sql})`).trim()); - -const queries = SPECS.map((s, i) => { - const key = schemaKey(s.sql); - const rows = resultRows(s.sql); - console.log(`#${i + 1} ${s.cfg.type.padEnd(4)} rows=${String(rows).padStart(5)} key=${key}`); - return { - id: 's' + (i + 1), - sql: s.sql, - specVersion: 1, - spec: { - name: s.name, - favorite: false, - description: s.description, - panel: { cfg: s.cfg, key }, - view: 'panel', - }, - }; -}); - -const dashboard = buildDashboard({ - id: 'ontime-chart-gallery', - title: 'On-time chart gallery', - description: 'Chart gallery over the public ontime flight dataset.', - queries, - tileQueryIds: queries.map((query) => query.id), - preset: 'columns-2', -}); +for (const query of document.queries) { + if (!query.spec?.panel?.cfg?.type) continue; + const key = schemaKey(query.sql); + query.spec.panel.key = key; + console.log(`${query.id.padEnd(14)} ${query.spec.panel.cfg.type.padEnd(5)} ${key}`); +} -const outPath = resolve(here, 'ontime-charts.json'); writeExampleBundle(outPath, { exportedAt: new Date().toISOString(), - metadata: { name: dashboard.title, description: dashboard.description }, - queries, - dashboards: [dashboard], + metadata: document.metadata, + queries: document.queries, + dashboards: document.dashboards, }); -console.log(`\nwrote ${outPath} (${queries.length} queries)`); +console.log(`wrote ${outPath}`); diff --git a/examples/mjs/build-system-explorer-charts.mjs b/examples/mjs/build-system-explorer-charts.mjs deleted file mode 100644 index 5fc117c6..00000000 --- a/examples/mjs/build-system-explorer-charts.mjs +++ /dev/null @@ -1,257 +0,0 @@ -// Generator for examples/system-explorer-charts.json — a saved-queries -// "Library" file for the Altinity SQL Browser that explores ClickHouse's own -// system database: currently-running work, merges/mutations/replication -// health, storage, and historical query/part/error activity from the *_log -// tables. Ideas and query shapes are adapted (not ported 1:1 — no Grafana -// template macros) from Mikhail Filimonov's ClickHouse ops dashboard: -// https://gist.github.com/filimonov/271e5b27c085356c67db3c1bf2204506 -// -// Why a generator: the browser only restores a saved chart config when the -// entry's `spec.panel.key` exactly equals schemaKey(resultColumns) = "name:type|…" -// (see src/ui/results.js chartCfgFor / src/core/chart-data.js schemaKey). -// Hand-writing those type strings is error-prone (Enum8/LowCardinality wrap -// exactly), so we derive each key live from `DESCRIBE ()` against a -// real cluster, read through FORMAT JSON so the type string matches exactly -// what the app's HTTP+JSON interface receives (clickhouse-client's default -// TSV output escapes embedded quotes differently and will silently produce a -// key that never matches at runtime). -// -// A handful of entries are plain live-snapshot tables (system.processes, -// system.merges, …) with no chart — those need SELECT privilege on the -// underlying table but no query history, and are commonly close to empty on -// an idle cluster (that's a legitimate result, not a bug). -// -// Every time-ranged *_log query shares two ClickHouse native query parameters, -// `{from:String}`/`{to:String}` (parsed via parseDateTimeBestEffort), instead -// of a hardcoded `now() - INTERVAL …`. Same param names across every entry -// means the Dashboard's global filter bar (#149 D3) renders ONE From/To pair -// that drives all six time-ranged tiles at once. DESCRIBE can't resolve an -// unbound parameter, so schemaKey() below binds throwaway test values via -// `--param_from`/`--param_to` purely to derive column types — the *shipped* -// SQL keeps the placeholders unbound for the browser to fill in. -// -// Run: node examples/mjs/build-system-explorer-charts.mjs [connection-name] -// Needs a `clickhouse-client` connection with SELECT on system.* — NOT the -// narrow "demo" fixture user some clusters expose by default (it can't read -// system.processes/query_log/etc). Defaults to `github-admin`; this file was -// authored against the github.demo cluster via -// kubectl exec chi-github-github-0-0-0 -c clickhouse-pod -- -// clickhouse-client --user clickhouse_operator --password "$PASS" ... -// since no adequately-privileged named CLI connection existed in that session. -// Out: examples/system-explorer-charts.json - -import { execFileSync } from 'node:child_process'; -import { fileURLToPath } from 'node:url'; -import { dirname, resolve } from 'node:path'; -import { buildDashboard, writeExampleBundle } from './example-bundle.mjs'; - -const here = dirname(fileURLToPath(import.meta.url)); -const CONNECTION = process.argv[2] || 'github-admin'; - -// Each spec: a query + (for chartable ones) the chart we want it to open -// with. `cfg` matches the app's shape { type, x, y:[...], series }; x/series -// are column indices, y a list of measure-column indices. A spec with no -// `cfg` is a live-snapshot table — no chart, opens in Table view. -const SPECS = [ - { - name: 'Currently running queries', - description: 'Live snapshot of system.processes — every query executing right now, slowest first. Empty when the cluster is idle; that\'s a real "nothing running" result, not an error.', - sql: `SELECT query_id, user, elapsed, read_rows, formatReadableSize(memory_usage) AS memory, left(query, 80) AS query -FROM system.processes -ORDER BY elapsed DESC -LIMIT 20`, - }, - { - name: 'Merges in progress', - description: 'Live snapshot of system.merges — background merges currently running, with progress and compressed size. Usually empty between merge cycles on a small cluster.', - sql: `SELECT database, table, elapsed, round(progress * 100, 1) AS pct_done, num_parts, is_mutation, formatReadableSize(total_size_bytes_compressed) AS size -FROM system.merges -ORDER BY elapsed DESC -LIMIT 20`, - }, - { - name: 'Mutations in progress', - description: 'Unfinished ALTER UPDATE/DELETE mutations from system.mutations, with the failure reason if one is stuck retrying.', - sql: `SELECT database, table, mutation_id, command, parts_to_do, latest_fail_reason -FROM system.mutations -WHERE NOT is_done -ORDER BY create_time -LIMIT 20`, - }, - { - name: 'Replication status', - description: 'system.replicas health per table — leadership, read-only state, replication delay, and queue depth. Sorted worst-lag-first.', - sql: `SELECT database, table, is_leader, is_readonly, absolute_delay, queue_size, inserts_in_queue, merges_in_queue -FROM system.replicas -ORDER BY absolute_delay DESC -LIMIT 20`, - }, - { - name: 'Stuck replication queue entries', - description: 'system.replication_queue entries that have already failed and retried at least once, with the last exception — the first place to look when a replica falls behind.', - sql: `SELECT database, table, type, create_time, num_tries, last_exception -FROM system.replication_queue -WHERE num_tries > 0 -ORDER BY num_tries DESC -LIMIT 20`, - }, - { - name: 'Largest tables by disk usage', - description: 'Every active part in system.parts, summed per table, largest first. Horizontal Bar — hover any bar for the exact byte count.', - cfg: { type: 'hbar', x: 0, y: [1], series: null }, - sql: `SELECT concat(database, '.', table) AS table, sum(bytes_on_disk) AS disk_bytes -FROM system.parts -WHERE active -GROUP BY database, table -ORDER BY disk_bytes DESC -LIMIT 15`, - }, - { - name: 'Active parts by table', - description: 'Active part *count* per table (not size) — a table climbing here between refreshes is trending toward "too many parts". Horizontal Bar.', - cfg: { type: 'hbar', x: 0, y: [1], series: null }, - sql: `SELECT concat(database, '.', table) AS table, count() AS parts -FROM system.parts -WHERE active -GROUP BY database, table -ORDER BY parts DESC -LIMIT 15`, - }, - { - name: 'Cumulative error counters', - description: 'system.errors — every error code the server has hit since last restart, most frequent first. A quick "what\'s actually going wrong here" check. Horizontal Bar.', - cfg: { type: 'hbar', x: 0, y: [1], series: null }, - sql: `SELECT name, value AS times -FROM system.errors -WHERE value > 0 -ORDER BY value DESC -LIMIT 15`, - }, - { - name: 'Queries per minute', - description: 'Finished-query volume from system.query_log, bucketed per minute, over a {from:String}/{to:String} range (shared with every other time-ranged query below — the Dashboard filter bar renders one From/To pair that drives them all). A DateTime X axis is auto-detected as a time series → Line chart.', - cfg: { type: 'line', x: 0, y: [1], series: null }, - sql: `SELECT toStartOfMinute(event_time) AS t, count() AS queries -FROM system.query_log -WHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND type = 'QueryFinish' -GROUP BY t -ORDER BY t`, - }, - { - name: 'Slowest query patterns — avg duration', - description: 'Distinct query shapes (system.query_log grouped by normalized_query_hash) ranked by average duration over a {from:String}/{to:String} range. Horizontal Bar of a non-count measure.', - cfg: { type: 'hbar', x: 0, y: [1], series: null }, - sql: `SELECT left(any(query), 50) AS query, avg(query_duration_ms) AS avg_duration_ms -FROM system.query_log -WHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND type = 'QueryFinish' -GROUP BY normalized_query_hash -ORDER BY avg_duration_ms DESC -LIMIT 15`, - }, - { - name: 'Query errors over time', - description: 'Failed queries from system.query_log over a {from:String}/{to:String} range, broken down by ClickHouse error name. The "error" column is used as the Series, producing grouped/stacked bars per error code.', - cfg: { type: 'bar', x: 0, y: [2], series: 1 }, - sql: `SELECT toStartOfHour(event_time) AS t, errorCodeToName(exception_code) AS error, count() AS n -FROM system.query_log -WHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND exception_code != 0 -GROUP BY t, error -ORDER BY t`, - }, - { - name: 'Part lifecycle events over time', - description: 'system.part_log over a {from:String}/{to:String} range — new/merged/mutated/downloaded/removed parts per hour, one query instead of five separate panels. "event_type" is the Series.', - cfg: { type: 'bar', x: 0, y: [2], series: 1 }, - sql: `SELECT toStartOfHour(event_time) AS t, event_type, count() AS n -FROM system.part_log -WHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) -GROUP BY t, event_type -ORDER BY t`, - }, - { - name: 'Memory usage over time', - description: 'Average tracked memory (system.metric_log\'s CurrentMetric_MemoryTracking) per minute over a {from:String}/{to:String} range. Line chart.', - cfg: { type: 'line', x: 0, y: [1], series: null }, - sql: `SELECT toStartOfMinute(event_time) AS t, avg(CurrentMetric_MemoryTracking) AS memory_bytes -FROM system.metric_log -WHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) -GROUP BY t -ORDER BY t`, - }, - { - name: 'Query cost breakdown — slowest patterns (detail)', - description: 'The deep-dive version of "Slowest query patterns": executions, max/avg duration, rows and bytes read, and p99 memory per query shape over a {from:String}/{to:String} range. Table view — too many columns for one chart, but the full picture behind the bar chart above.', - sql: `SELECT - normalized_query_hash, - left(argMax(query, query_duration_ms), 60) AS sample_query, - count() AS executions, - max(query_duration_ms) AS max_ms, - avg(query_duration_ms) AS avg_ms, - sum(read_rows) AS read_rows, - formatReadableSize(sum(read_bytes)) AS read_bytes, - quantile(0.99)(memory_usage) AS p99_memory -FROM system.query_log -WHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND type = 'QueryFinish' -GROUP BY normalized_query_hash -ORDER BY avg_ms DESC -LIMIT 15`, - }, -]; - -// Throwaway values just to let DESCRIBE/FORMAT JSON resolve column types for -// queries that reference {from:String}/{to:String} — never shipped in the -// output; the JSON's `sql` keeps the placeholders unbound. -const TEST_FROM = '2026-07-01 00:00:00'; -const TEST_TO = '2026-07-08 00:00:00'; - -const ch = (query) => - execFileSync('clickhouse-client', [ - '--connection', CONNECTION, - '--param_from', TEST_FROM, - '--param_to', TEST_TO, - '--query', query, - ], { - encoding: 'utf8', - maxBuffer: 64 * 1024 * 1024, - }); - -// schemaKey == columns.map(c => c.name + ':' + c.type).join('|'), derived via -// FORMAT JSON (not clickhouse-client's default TSV, which escapes embedded -// quotes in e.g. an Enum8(...) type string differently from the HTTP+JSON -// interface the app actually uses) so it matches exactly what the browser -// receives at run time. -function schemaKey(sql) { - const out = JSON.parse(ch(`SELECT * FROM (${sql}) LIMIT 1 FORMAT JSON`)); - return out.meta.map((m) => `${m.name}:${m.type}`).join('|'); -} - -const queries = SPECS.map((s, i) => { - const base = { - id: 'sys-' + (i + 1), - sql: s.sql, - specVersion: 1, - spec: { name: s.name, favorite: !!s.cfg, description: s.description }, - }; - if (!s.cfg) return base; - const key = schemaKey(s.sql); - console.log(`#${i + 1} ${s.cfg.type.padEnd(4)} key=${key}`); - return { ...base, spec: { ...base.spec, panel: { cfg: s.cfg, key }, view: 'panel' } }; -}); - -const dashboard = buildDashboard({ - id: 'system-explorer', - title: 'ClickHouse system explorer', - description: 'Operational views over ClickHouse system tables.', - queries, - tileQueryIds: queries.filter((query) => query.spec.favorite).map((query) => query.id), - preset: 'columns-2', -}); - -const outPath = resolve(here, 'system-explorer-charts.json'); -writeExampleBundle(outPath, { - exportedAt: new Date().toISOString(), - metadata: { name: dashboard.title, description: dashboard.description }, - queries, - dashboards: [dashboard], -}); -console.log(`\nwrote ${outPath} (${queries.length} queries, ${queries.filter((q) => q.spec.favorite).length} favorited for the Dashboard)`); diff --git a/examples/mjs/example-bundle.mjs b/examples/mjs/example-bundle.mjs index 7cd3e566..6aa83f0c 100644 --- a/examples/mjs/example-bundle.mjs +++ b/examples/mjs/example-bundle.mjs @@ -47,6 +47,14 @@ function placementFor(query, preset) { return { span: 1, height: 'large' }; } +function flowLayout(selected, tiles, preset, placements = {}) { + const items = Object.fromEntries(selected.map((query, index) => [ + tiles[index].id, + placements[query.id] || placementFor(query, preset), + ])); + return { type: 'flow', version: 1, preset, items }; +} + function scanParameterNames(sql) { const names = []; const seen = new Set(); @@ -68,6 +76,10 @@ export function buildDashboard({ tileQueryIds, sourceByParameter = {}, preset = 'columns-2', + filters: authoredFilters, + grid, + flowPlacements = {}, + revision = 1, }) { if (!id || !title) throw new Error('Dashboard id and title are required'); if (!['report', 'columns-2', 'columns-3'].includes(preset)) { @@ -85,10 +97,7 @@ export function buildDashboard({ }); const tiles = selected.map((query) => ({ id: `tile-${query.id}`, queryId: query.id })); - const items = Object.fromEntries(selected.map((query, index) => [ - tiles[index].id, - placementFor(query, preset), - ])); + const fallback = flowLayout(selected, tiles, preset, flowPlacements); const parameterNames = []; const seenParameters = new Set(); @@ -101,19 +110,32 @@ export function buildDashboard({ } } - const filters = parameterNames.map((parameter) => ({ + const inferredFilters = parameterNames.map((parameter) => ({ id: `filter-${parameter}`, parameter, ...(sourceByParameter[parameter] ? { sourceQueryId: sourceByParameter[parameter] } : {}), })); + const filters = authoredFilters === undefined ? inferredFilters : clone(authoredFilters); + const layout = grid + ? { + type: 'grafana-grid', + version: 1, + items: Object.fromEntries(selected.map((query, index) => [ + tiles[index].id, + grid[query.id] || { span: 6, height: 2 }, + ])), + fallback, + } + : fallback; + return { documentVersion: 1, id, title, ...(description ? { description } : {}), - revision: 1, - layout: { type: 'flow', version: 1, preset, items }, + revision, + layout, filters, tiles, }; @@ -190,8 +212,15 @@ export function assertValidExampleBundle(document) { if (!Array.isArray(dashboard.tiles) || !Array.isArray(dashboard.filters)) { throw new Error(`Dashboard ${JSON.stringify(dashboard.id)} requires tiles and filters arrays`); } - if (dashboard.layout?.type !== 'flow' || dashboard.layout?.version !== 1) { - throw new Error(`Dashboard ${JSON.stringify(dashboard.id)} must use flow@1`); + const layout = dashboard.layout; + const supportedLayout = layout?.version === 1 + && (layout.type === 'flow' || layout.type === 'grafana-grid'); + if (!supportedLayout) { + throw new Error(`Dashboard ${JSON.stringify(dashboard.id)} must use flow@1 or grafana-grid@1`); + } + if (layout.type === 'grafana-grid' + && (layout.fallback?.type !== 'flow' || layout.fallback?.version !== 1)) { + throw new Error(`Dashboard ${JSON.stringify(dashboard.id)} grafana-grid@1 layout requires a flow@1 fallback`); } for (const tile of dashboard.tiles) { if (!queryIds.has(tile.queryId)) { diff --git a/examples/mjs/normalize-examples.mjs b/examples/mjs/normalize-examples.mjs index 4d72897c..ab413f57 100644 --- a/examples/mjs/normalize-examples.mjs +++ b/examples/mjs/normalize-examples.mjs @@ -19,42 +19,17 @@ const examples = resolve(here, '..'); const checkOnly = process.argv.includes('--check'); const CONFIG = { - 'kpi-panel.json': { - id: 'kpi-panel-example', title: 'KPI panel example', preset: 'report', - description: 'Scalar and named-tuple KPI presentation example.', - }, - 'text-log-panel.json': { - id: 'text-log-panel-example', title: 'Server log panel example', preset: 'report', - description: 'Parameterized system.text_log dashboard example.', - }, 'shop-charts.json': { - id: 'shop-charts', title: 'Shop analytics', preset: 'columns-2', - description: 'Chart examples over the sample shop dataset.', + id: 'shop-analytics', title: 'Shop analytics', authoredDashboard: true, + description: 'Revenue, buyers, products, geography, and traffic over the shop-demo.sql dataset.', }, - 'query-log-explorer.json': { - id: 'query-log-explorer', title: 'Query log explorer', preset: 'columns-2', - description: 'ClickHouse query-log health, performance, and usage dashboard.', - sourceByParameter: { user: 'qle-filter', query_kind: 'qle-filter' }, - }, - 'grafana-clickhouse-ops-enhanced.json': { - id: 'clickhouse-ops-enhanced', title: 'ClickHouse operations', preset: 'columns-2', - description: 'Operational ClickHouse dashboard adapted from the Grafana dashboard.', - sourceByParameter: { - is_initial_query: 'gco-filter', - query_kind: 'gco-filter', - user: 'gco-filter', - exception_code: 'gco-filter', - query_hash: 'gco-filter', - metric: 'gco-filter', - }, + 'clickhouse-operations.json': { + id: 'clickhouse-operations', title: 'ClickHouse Operations', authoredDashboard: true, + description: 'Operator-first server overview, resources, background work, and investigation views.', }, 'ontime-charts.json': { - id: 'ontime-chart-gallery', title: 'On-time chart gallery', preset: 'columns-2', allPanels: true, - description: 'Chart gallery over the public ontime flight dataset.', - }, - 'system-explorer-charts.json': { - id: 'system-explorer', title: 'ClickHouse system explorer', preset: 'columns-2', - description: 'Operational views over ClickHouse system tables.', + id: 'ontime-flights', title: 'On-time flights', authoredDashboard: true, + description: 'Flight punctuality, volume, carriers, airports, delays, and cancellations with a shared 2023 slice.', }, 'iceberg-catalog-dashboard.json': { id: 'iceberg-catalog-explorer', title: 'Iceberg catalog explorer — BI', preset: 'columns-2', @@ -111,7 +86,9 @@ function tileQueryIds(queries, config) { function normalizeDocument(name, document, config) { const queries = cleanStaleWording(queriesOf(document, name)); const selectedIds = config.dashboard === false ? [] : tileQueryIds(queries, config); - const dashboards = selectedIds.length ? [buildDashboard({ + const authored = config.authoredDashboard ? cleanStaleWording(document.dashboards?.[0]) : null; + if (config.authoredDashboard && !authored) throw new Error(`${name}: expected an authored Dashboard`); + const dashboards = authored ? [authored] : selectedIds.length ? [buildDashboard({ id: config.id, title: config.title, description: config.description, diff --git a/examples/ontime-charts.json b/examples/ontime-charts.json index 4390ae38..4193aa3a 100644 --- a/examples/ontime-charts.json +++ b/examples/ontime-charts.json @@ -2,18 +2,64 @@ "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", "format": "altinity-sql-browser/portable-bundle", "version": 1, - "exportedAt": "2026-06-24T13:22:11.940Z", + "exportedAt": "2026-07-22T00:00:00.000Z", "metadata": { - "name": "On-time chart gallery", - "description": "Chart gallery over the public ontime flight dataset." + "name": "On-time flights", + "description": "Flight punctuality, volume, carriers, airports, delays, and cancellations with a shared 2023 slice." }, "queries": [ + { + "id": "ontime-kpis", + "sql": "SELECT count() AS flights, round(100 * countIf(f.ArrDel15 = 0) / count(), 1) AS on_time_pct, countIf(f.Cancelled = 1) AS cancellations, round(avg(f.DepDelayMinutes), 1) AS avg_departure_delay\nFROM ontime.fact_ontime AS f\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/", + "specVersion": 1, + "spec": { + "name": "Flight KPIs", + "description": "Flight volume, punctuality, cancellations, and mean departure delay for the selected slice.", + "favorite": true, + "view": "panel", + "panel": { + "cfg": { + "type": "kpi" + }, + "fieldConfig": { + "columns": { + "flights": { + "displayName": "Flights", + "decimals": 0 + }, + "on_time_pct": { + "displayName": "On time", + "unit": "%", + "decimals": 1 + }, + "cancellations": { + "displayName": "Cancellations", + "decimals": 0 + }, + "avg_departure_delay": { + "displayName": "Avg departure delay", + "unit": " min", + "decimals": 1 + } + } + } + }, + "dashboard": { + "role": "panel", + "sizeHints": { + "preferred": "compact", + "minimum": "compact", + "aspectRatio": 2 + } + } + } + }, { "id": "s1", - "sql": "SELECT\n a.DisplayAirportName AS airport,\n count() AS flights\nFROM ontime.fact_ontime AS f\nINNER JOIN ontime.dim_airports AS a\n ON a.AirportCode = f.OriginCode AND a.IsLatest = 1\nWHERE f.Year = 2023\nGROUP BY airport\nORDER BY flights DESC\nLIMIT 15", + "sql": "SELECT a.DisplayAirportName AS airport, count() AS flights\nFROM ontime.fact_ontime AS f\nINNER JOIN ontime.dim_airports AS a ON a.AirportCode = f.OriginCode AND a.IsLatest = 1\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/\nGROUP BY airport ORDER BY flights DESC LIMIT 15", "specVersion": 1, "spec": { - "name": "Busiest origin airports — 2023", + "name": "Busiest origins", "favorite": true, "description": "Top 15 departure airports by flight count (joined to dim_airports for readable names). Horizontal Bar — hover any bar, long or short, to read its exact value.", "view": "panel", @@ -44,7 +90,7 @@ "specVersion": 1, "spec": { "name": "Flights by month — 2023", - "favorite": true, + "favorite": false, "description": "Monthly US flight volume. A numeric column named \"month\" is detected as an ordinal axis → vertical Column chart, with K/M-humanised value ticks.", "view": "panel", "panel": { @@ -70,10 +116,10 @@ }, { "id": "s3", - "sql": "SELECT FlightDate AS date, count() AS flights\nFROM ontime.fact_ontime\nWHERE Year = 2023\nGROUP BY date\nORDER BY date", + "sql": "SELECT f.FlightDate AS date, count() AS flights\nFROM ontime.fact_ontime AS f\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/\nGROUP BY date ORDER BY date", "specVersion": 1, "spec": { - "name": "Daily flights — 2023", + "name": "Daily flights", "favorite": true, "description": "One point per day across 2023 (~365 rows). A Date X axis is auto-detected as a time series → Line chart.", "view": "panel", @@ -100,10 +146,10 @@ }, { "id": "s4", - "sql": "SELECT\n FlightDate AS date,\n round(100 * countIf(ArrDel15 = 0) / count(), 1) AS on_time_pct\nFROM ontime.fact_ontime\nWHERE Year = 2023\nGROUP BY date\nORDER BY date", + "sql": "SELECT f.FlightDate AS date, round(100 * countIf(f.ArrDel15 = 0) / count(), 1) AS on_time_pct\nFROM ontime.fact_ontime AS f\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/\nGROUP BY date ORDER BY date", "specVersion": 1, "spec": { - "name": "Daily on-time rate — 2023", + "name": "On-time rate", "favorite": true, "description": "Share of flights arriving on time (< 15 min late) per day, as a percentage. Rendered as a filled Area chart.", "view": "panel", @@ -130,10 +176,10 @@ }, { "id": "s5", - "sql": "SELECT\n multiIf(CancellationCode = 'A', 'Carrier',\n CancellationCode = 'B', 'Weather',\n CancellationCode = 'C', 'National Air System',\n CancellationCode = 'D', 'Security', 'Other') AS reason,\n count() AS cancellations\nFROM ontime.fact_ontime\nWHERE Year = 2023 AND Cancelled = 1\nGROUP BY reason\nORDER BY cancellations DESC", + "sql": "SELECT multiIf(f.CancellationCode = 'A', 'Carrier', f.CancellationCode = 'B', 'Weather', f.CancellationCode = 'C', 'National Air System', f.CancellationCode = 'D', 'Security', 'Other') AS reason, count() AS cancellations\nFROM ontime.fact_ontime AS f\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/ AND f.Cancelled = 1\nGROUP BY reason ORDER BY cancellations DESC", "specVersion": 1, "spec": { - "name": "Cancellation reasons — 2023", + "name": "Cancellation reasons", "favorite": true, "description": "Why flights were cancelled in 2023 (carrier / weather / national air system / security). A small categorical breakdown → Pie chart with a legend.", "view": "panel", @@ -160,10 +206,10 @@ }, { "id": "s6", - "sql": "SELECT\n Month AS month,\n Carrier AS carrier,\n count() AS flights\nFROM ontime.fact_ontime\nWHERE Year = 2023 AND Carrier IN ('WN', 'AA', 'DL', 'UA')\nGROUP BY month, carrier\nORDER BY month, carrier", + "sql": "SELECT toStartOfMonth(f.FlightDate) AS month, f.Carrier AS carrier, count() AS flights\nFROM ontime.fact_ontime AS f\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/\nGROUP BY month, carrier ORDER BY month, carrier", "specVersion": 1, "spec": { - "name": "Monthly flights by carrier — 2023", + "name": "Monthly carrier volume", "favorite": true, "description": "Flights per month split across four major carriers (WN, AA, DL, UA). The \"carrier\" column is used as the Series, producing grouped bars with a per-carrier legend.", "view": "panel", @@ -176,7 +222,7 @@ ], "series": 1 }, - "key": "month:UInt8|carrier:LowCardinality(String)|flights:UInt64" + "key": "month:Date|carrier:LowCardinality(String)|flights:UInt64" }, "dashboard": { "role": "panel", @@ -190,10 +236,10 @@ }, { "id": "s7", - "sql": "SELECT\n Carrier AS carrier,\n round(avg(CarrierDelay), 1) AS carrier_delay,\n round(avg(WeatherDelay), 1) AS weather_delay,\n round(avg(NASDelay), 1) AS nas_delay,\n round(avg(LateAircraftDelay), 1) AS late_aircraft_delay\nFROM ontime.fact_ontime\nWHERE Year = 2023 AND ArrDel15 = 1\nGROUP BY carrier\nORDER BY carrier_delay DESC\nLIMIT 12", + "sql": "SELECT f.Carrier AS carrier, round(avg(f.CarrierDelay), 1) AS carrier_delay, round(avg(f.WeatherDelay), 1) AS weather_delay, round(avg(f.NASDelay), 1) AS nas_delay, round(avg(f.LateAircraftDelay), 1) AS late_aircraft_delay\nFROM ontime.fact_ontime AS f\nWHERE f.FlightDate BETWEEN {from:Date} AND {to:Date}\n/*[ AND has({carrier:Array(String)}, f.Carrier) ]*/\n/*[ AND has({origin:Array(String)}, f.OriginCode) ]*/ AND f.ArrDel15 = 1\nGROUP BY carrier ORDER BY carrier_delay DESC LIMIT 12", "specVersion": 1, "spec": { - "name": "Average delay breakdown by carrier — 2023", + "name": "Delay causes by carrier", "favorite": true, "description": "Mean minutes of each delay cause (carrier, weather, NAS, late aircraft) for delayed flights, per carrier. Four measures plotted at once (\"All measures\") as grouped columns.", "view": "panel", @@ -227,7 +273,7 @@ "specVersion": 1, "spec": { "name": "Daily flights since 2022", - "favorite": true, + "favorite": false, "description": "Every day from 2022 onward (~1,460 points). The chart plots the first 500 and shows a \"first 500 of N rows\" note — the table view still has them all.", "view": "panel", "panel": { @@ -257,7 +303,7 @@ "specVersion": 1, "spec": { "name": "Flights by day of week — 2023", - "favorite": true, + "favorite": false, "description": "Volume by day of week (1 = Monday … 7 = Sunday). \"dayofweek\" is recognised as an ordinal axis → Column chart.", "view": "panel", "panel": { @@ -287,7 +333,7 @@ "specVersion": 1, "spec": { "name": "Worst average departure delay by airport — 2023", - "favorite": true, + "favorite": false, "description": "Airports with the highest mean departure delay (minutes) among those with ≥ 10,000 departures in 2023. Horizontal Bar of a non-count measure, joined for names.", "view": "panel", "panel": { @@ -310,71 +356,169 @@ } } } + }, + { + "id": "ontime-filter-options", + "sql": "SELECT\n (SELECT arraySort(groupUniqArray(Carrier)) FROM ontime.fact_ontime WHERE Carrier != '') AS carrier,\n (SELECT groupArray((AirportCode AS value, DisplayAirportName AS label)) FROM (SELECT AirportCode, any(DisplayAirportName) AS DisplayAirportName FROM ontime.dim_airports WHERE IsLatest = 1 GROUP BY AirportCode ORDER BY DisplayAirportName)) AS origin", + "specVersion": 1, + "spec": { + "name": "On-time dimensions", + "description": "Carrier codes and airport-code values with readable airport labels.", + "favorite": false, + "dashboard": { + "role": "filter" + } + } } ], "dashboards": [ { "documentVersion": 1, - "id": "ontime-chart-gallery", - "title": "On-time chart gallery", - "description": "Chart gallery over the public ontime flight dataset.", + "id": "ontime-flights", + "title": "On-time flights", + "description": "Flight punctuality, volume, carriers, airports, delays, and cancellations with a shared 2023 slice.", "revision": 1, "layout": { - "type": "flow", + "type": "grafana-grid", "version": 1, - "preset": "columns-2", "items": { - "tile-s1": { - "span": 1, - "height": "large" - }, - "tile-s2": { - "span": 1, - "height": "large" + "tile-ontime-kpis": { + "span": 12, + "height": 2 }, "tile-s3": { - "span": 1, - "height": "large" + "span": 8, + "height": 3 }, "tile-s4": { - "span": 1, - "height": "large" + "span": 4, + "height": 3 }, - "tile-s5": { - "span": 1, - "height": "large" + "tile-s1": { + "span": 6, + "height": 3 }, "tile-s6": { - "span": 1, - "height": "large" + "span": 6, + "height": 3 }, "tile-s7": { - "span": 1, - "height": "large" + "span": 8, + "height": 3 }, - "tile-s8": { - "span": 1, - "height": "large" - }, - "tile-s9": { - "span": 1, - "height": "large" - }, - "tile-s10": { - "span": 1, - "height": "large" + "tile-s5": { + "span": 4, + "height": 3 + } + }, + "fallback": { + "type": "flow", + "version": 1, + "preset": "columns-2", + "items": { + "tile-ontime-kpis": { + "span": 2, + "height": "compact" + }, + "tile-s3": { + "span": 1, + "height": "large" + }, + "tile-s4": { + "span": 1, + "height": "large" + }, + "tile-s1": { + "span": 1, + "height": "large" + }, + "tile-s6": { + "span": 1, + "height": "large" + }, + "tile-s7": { + "span": 1, + "height": "large" + }, + "tile-s5": { + "span": 1, + "height": "large" + } } } }, - "filters": [], - "tiles": [ + "filters": [ { - "id": "tile-s1", - "queryId": "s1" + "id": "ontime-date-from", + "parameter": "from", + "label": "From", + "defaultValue": "2023-01-01", + "defaultActive": true, + "targets": [ + "tile-ontime-kpis", + "tile-s3", + "tile-s4", + "tile-s1", + "tile-s6", + "tile-s7", + "tile-s5" + ] }, { - "id": "tile-s2", - "queryId": "s2" + "id": "ontime-date-to", + "parameter": "to", + "label": "To", + "defaultValue": "2023-12-31", + "defaultActive": true, + "targets": [ + "tile-ontime-kpis", + "tile-s3", + "tile-s4", + "tile-s1", + "tile-s6", + "tile-s7", + "tile-s5" + ] + }, + { + "id": "ontime-carrier", + "parameter": "carrier", + "label": "Carrier", + "sourceQueryId": "ontime-filter-options", + "defaultValue": [], + "defaultActive": false, + "targets": [ + "tile-ontime-kpis", + "tile-s3", + "tile-s4", + "tile-s1", + "tile-s6", + "tile-s7", + "tile-s5" + ] + }, + { + "id": "ontime-origin", + "parameter": "origin", + "label": "Origin airport", + "sourceQueryId": "ontime-filter-options", + "defaultValue": [], + "defaultActive": false, + "targets": [ + "tile-ontime-kpis", + "tile-s3", + "tile-s4", + "tile-s1", + "tile-s6", + "tile-s7", + "tile-s5" + ] + } + ], + "tiles": [ + { + "id": "tile-ontime-kpis", + "queryId": "ontime-kpis" }, { "id": "tile-s3", @@ -385,8 +529,8 @@ "queryId": "s4" }, { - "id": "tile-s5", - "queryId": "s5" + "id": "tile-s1", + "queryId": "s1" }, { "id": "tile-s6", @@ -397,16 +541,8 @@ "queryId": "s7" }, { - "id": "tile-s8", - "queryId": "s8" - }, - { - "id": "tile-s9", - "queryId": "s9" - }, - { - "id": "tile-s10", - "queryId": "s10" + "id": "tile-s5", + "queryId": "s5" } ] } diff --git a/examples/query-log-explorer.json b/examples/query-log-explorer.json deleted file mode 100644 index dafff817..00000000 --- a/examples/query-log-explorer.json +++ /dev/null @@ -1,305 +0,0 @@ -{ - "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", - "format": "altinity-sql-browser/portable-bundle", - "version": 1, - "exportedAt": "2026-07-15T00:00:00.000Z", - "metadata": { - "name": "Query log explorer", - "description": "ClickHouse query-log health, performance, and usage dashboard." - }, - "queries": [ - { - "id": "qle-filter", - "sql": "SELECT\n arraySort(item -> item.label, groupUniqArray((user AS value, splitByChar('@', user)[1] AS label))) AS user,\n arraySort(groupUniqArray(query_kind)) AS query_kind\nFROM system.query_log\nWHERE query_log.user != ''\n AND event_time >= {from:DateTime}\n /*[ AND event_time <= {to:DateTime} ]*/\nSETTINGS enable_named_columns_in_function_tuple = 1", - "specVersion": 1, - "spec": { - "name": "Filter", - "favorite": false, - "description": "Single Dashboard Filter source. `user` is a value/label bundle (Array(Tuple(value, label)) — #160): the value is the full user that binds to a Panel's {user:String}, the label is the name before '@' for display. `query_kind` is a plain Array(String). Upgrades any Panel's {user:String} and {query_kind:String} parameters to searchable single-select dropdowns.", - "dashboard": { - "role": "filter" - } - } - }, - { - "id": "qle-kpi", - "sql": "SELECT\n count() AS total_queries,\n countIf(exception_code != 0) AS errors,\n round(100 * countIf(exception_code != 0) / count(), 2) AS error_rate_pct,\n round(avg(query_duration_ms), 1) AS avg_duration_ms,\n round(quantile(0.99)(query_duration_ms), 1) AS p99_duration_ms,\n uniqExact(user) AS active_users\nFROM system.query_log\nWHERE type IN ('QueryFinish', 'ExceptionWhileProcessing') AND query_kind = 'Select'\n AND event_time >= {from:DateTime}\n/*[ AND event_time <= {to:DateTime} ]*/\n/*[ AND positionCaseInsensitive(query, {search:String}) > 0 ]*/", - "specVersion": 1, - "spec": { - "name": "Query log health (KPI)", - "favorite": true, - "description": "Top-line SELECT-query health for this cluster: volume, error rate, and latency. The time window is a real DateTime range shared by every data Panel: {from:DateTime} is REQUIRED (a Panel stays unfilled until you pick a start), {to:DateTime} is optional (blank means \"up to now\"). Both render as relative-time fields — type an absolute timestamp or a relative expression like -24h / -7d / now. The optional {search:String} field substring-matches the query text across every Panel too.", - "view": "panel", - "panel": { - "cfg": { - "type": "kpi" - }, - "fieldConfig": { - "columns": { - "total_queries": { - "displayName": "SELECT queries" - }, - "errors": { - "displayName": "Errors" - }, - "error_rate_pct": { - "displayName": "Error rate", - "unit": "%", - "decimals": 2 - }, - "avg_duration_ms": { - "displayName": "Avg duration", - "unit": " ms", - "decimals": 1 - }, - "p99_duration_ms": { - "displayName": "P99 duration", - "unit": " ms", - "decimals": 1 - }, - "active_users": { - "displayName": "Active users" - } - } - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "compact", - "minimum": "compact", - "aspectRatio": 2 - } - } - } - }, - { - "id": "qle-text", - "sql": "", - "specVersion": 1, - "spec": { - "name": "About this demo", - "favorite": true, - "description": "Static Text panel explaining the dashboard.", - "view": "panel", - "panel": { - "cfg": { - "type": "text", - "content": "# Dashboard Filter sources demo\n\nEvery bundled Dashboard tile below is a Panel; one separate **Filter**-role source (#160) that populates the filter bar above with `user` and `query_kind` option lists queried live from `system.query_log` on the **otel** demo cluster.\n\n**Shared filters** — plain, type-detected controls (no Filter source), declared by the data Panels:\n\n- `from` — **required** window start (`DateTime`); type an absolute time or a relative expression like `-24h`, `-7d`, `now`\n- `to` — optional window end (`DateTime`); blank means \"up to now\"\n- `search` — optional case-insensitive substring search over the query text (and, in the log panel, the exception message)\n- `minDurationMs` — optional minimum query-duration threshold, in ms (slowest-queries panel)\n\n**Curated filters** — sourced from the single favorited **Filter** query, upgrading matching Panel parameters to searchable dropdowns:\n\n- `user` — real ClickHouse users seen in the query log\n- `query_kind` — the query kinds seen in the query log (Select, Insert, …)\n\nAnalytical panels are adapted from the Altinity KB page [\"Handy queries for system.query_log\"](https://kb.altinity.com/altinity-kb-useful-queries/query_log/)." - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "wide", - "minimum": "medium", - "aspectRatio": 2 - } - } - } - }, - { - "id": "qle-slowest", - "sql": "SELECT\n event_time,\n user,\n round(query_duration_ms) AS duration_ms,\n formatReadableSize(memory_usage) AS memory,\n read_rows,\n substring(query, 1, 100) AS query_preview\nFROM system.query_log\nWHERE type = 'QueryFinish' AND query_kind = 'Select'\n AND event_time >= {from:DateTime}\n/*[ AND event_time <= {to:DateTime} ]*/\n/*[ AND user = {user:String} ]*/\n/*[ AND query_duration_ms >= {minDurationMs:UInt32} ]*/\n/*[ AND positionCaseInsensitive(query, {search:String}) > 0 ]*/\nORDER BY query_duration_ms DESC\nLIMIT 50", - "specVersion": 1, - "spec": { - "name": "Slowest SELECT queries", - "favorite": true, - "description": "Adapted from the Altinity KB's \"Most resource-intensive queries\" — the highest-latency completed SELECTs, with duration/memory/rows-read to spot what's actually expensive. Demonstrates the shared time range ({from:DateTime} required, {to:DateTime} optional), the CURATED {user:String} dropdown (from the Filter source), the AUTO {minDurationMs:UInt32} numeric threshold, and the universal {search:String} query-text search.", - "view": "panel", - "panel": { - "cfg": { - "type": "table" - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "wide", - "minimum": "medium", - "aspectRatio": 1.6 - } - } - } - }, - { - "id": "qle-columns", - "sql": "SELECT arrayJoin(columns) AS column, count() AS hits\nFROM system.query_log\nWHERE type = 'QueryFinish'\n AND event_time >= {from:DateTime}\n AND notEmpty(columns)\n/*[ AND event_time <= {to:DateTime} ]*/\n/*[ AND query_kind = {query_kind:String} ]*/\n/*[ AND positionCaseInsensitive(query, {search:String}) > 0 ]*/\nGROUP BY column\nORDER BY hits DESC\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Most-selected columns", - "favorite": true, - "description": "Adapted from the Altinity KB's \"Most-selected columns\" — which columns have been referenced most often. Demonstrates the CURATED {query_kind:String} dropdown (from the Filter source) alongside the shared time range ({from:DateTime} required, {to:DateTime} optional) and the universal {search:String} search.", - "view": "panel", - "panel": { - "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "qle-functions", - "sql": "SELECT arrayJoin(used_functions) AS function, count() AS hits\nFROM system.query_log\nWHERE type = 'QueryFinish'\n AND event_time >= {from:DateTime}\n AND notEmpty(used_functions)\n/*[ AND event_time <= {to:DateTime} ]*/\n/*[ AND positionCaseInsensitive(query, {search:String}) > 0 ]*/\nGROUP BY function\nORDER BY hits DESC\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Most-used functions", - "favorite": true, - "description": "Adapted from the Altinity KB's \"Most-used functions\" — which SQL functions appear most often across executed queries. Uses the shared time range ({from:DateTime} required, {to:DateTime} optional) and the universal {search:String} query-text search, the same fields every other Panel declares.", - "view": "panel", - "panel": { - "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "qle-logs", - "sql": "SELECT\n event_time,\n multiIf(exception_code != 0, 'error', query_duration_ms > 1000, 'warn', 'info') AS level,\n if(exception_code != 0, exception, query) AS message,\n user,\n query_kind,\n query_duration_ms,\n formatReadableSize(memory_usage) AS memory\nFROM system.query_log\nWHERE type IN ('QueryFinish', 'ExceptionWhileProcessing')\n AND query_kind NOT IN ('Insert', 'AsyncInsertFlush')\n AND event_time >= {from:DateTime}\n/*[ AND event_time <= {to:DateTime} ]*/\n/*[ AND user = {user:String} ]*/\n/*[ AND (positionCaseInsensitive(query, {search:String}) > 0 OR positionCaseInsensitive(exception, {search:String}) > 0) ]*/\nORDER BY event_time DESC\nLIMIT 300", - "specVersion": 1, - "spec": { - "name": "Recent query activity (log)", - "favorite": true, - "description": "Raw query_log rows as a Logs view, level-colored by outcome (error/slow/info) and excluding the noisy internal Insert/AsyncInsertFlush firehose so the feed reads as actual query activity. Uses the shared time range ({from:DateTime} required, {to:DateTime} optional), the CURATED {user:String} dropdown (from the Filter source), and the universal {search:String} search — which here matches both the query text and the exception message.", - "view": "panel", - "panel": { - "cfg": { - "type": "logs", - "time": "event_time", - "msg": "message", - "level": "level" - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "wide", - "minimum": "medium", - "aspectRatio": 1.6 - } - } - } - } - ], - "dashboards": [ - { - "documentVersion": 1, - "id": "query-log-explorer", - "title": "Query log explorer", - "description": "ClickHouse query-log health, performance, and usage dashboard.", - "revision": 1, - "layout": { - "type": "flow", - "version": 1, - "preset": "columns-2", - "items": { - "tile-qle-kpi": { - "span": 1, - "height": "compact" - }, - "tile-qle-text": { - "span": 2, - "height": "large" - }, - "tile-qle-slowest": { - "span": 2, - "height": "large" - }, - "tile-qle-columns": { - "span": 1, - "height": "large" - }, - "tile-qle-functions": { - "span": 1, - "height": "large" - }, - "tile-qle-logs": { - "span": 2, - "height": "large" - } - } - }, - "filters": [ - { - "id": "filter-from", - "parameter": "from" - }, - { - "id": "filter-to", - "parameter": "to" - }, - { - "id": "filter-search", - "parameter": "search" - }, - { - "id": "filter-user", - "parameter": "user", - "sourceQueryId": "qle-filter" - }, - { - "id": "filter-minDurationMs", - "parameter": "minDurationMs" - }, - { - "id": "filter-query_kind", - "parameter": "query_kind", - "sourceQueryId": "qle-filter" - } - ], - "tiles": [ - { - "id": "tile-qle-kpi", - "queryId": "qle-kpi" - }, - { - "id": "tile-qle-text", - "queryId": "qle-text" - }, - { - "id": "tile-qle-slowest", - "queryId": "qle-slowest" - }, - { - "id": "tile-qle-columns", - "queryId": "qle-columns" - }, - { - "id": "tile-qle-functions", - "queryId": "qle-functions" - }, - { - "id": "tile-qle-logs", - "queryId": "qle-logs" - } - ] - } - ] -} diff --git a/examples/shop-charts.json b/examples/shop-charts.json index 5dffcb0e..9a13c397 100644 --- a/examples/shop-charts.json +++ b/examples/shop-charts.json @@ -2,50 +2,66 @@ "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", "format": "altinity-sql-browser/portable-bundle", "version": 1, - "exportedAt": "2026-06-28T00:00:00.000Z", + "exportedAt": "2026-07-22T00:00:00.000Z", "metadata": { "name": "Shop analytics", - "description": "Chart examples over the sample shop dataset." + "description": "Revenue, buyers, products, geography, and traffic over the shop-demo.sql dataset." }, "queries": [ { - "id": "shop-revenue-by-country", - "sql": "SELECT country, sum(revenue) AS revenue\nFROM shop.daily_sales\nGROUP BY country\nORDER BY revenue DESC", + "id": "shop-kpis", + "sql": "SELECT round(sumIf(amount, event_type = 'purchase'), 2) AS revenue, countIf(event_type = 'purchase') AS purchases, uniqIf(user_id, event_type = 'purchase') AS active_buyers, round(100 * countIf(event_type = 'purchase') / count(), 1) AS conversion_rate\nFROM shop.events_raw\nWHERE event_time BETWEEN {from:DateTime} AND {to:DateTime}\n/*[ AND country IN {country:Array(String)} ]*/\n/*[ AND has({category:Array(String)}, dictGet('shop.products_dict', 'category', toUInt64(product_id))) ]*/", "specVersion": 1, "spec": { - "name": "Revenue by country", + "name": "Sales KPIs", + "description": "Revenue, purchases, active buyers, and purchase conversion for the selected slice.", "favorite": true, - "description": "Total revenue per country from the daily_sales aggregate (fed by the mv_daily_sales materialized view). A LowCardinality category on the X axis → horizontal Bar.", "view": "panel", "panel": { "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null + "type": "kpi" }, - "key": "country:LowCardinality(String)|revenue:Decimal(38, 2)" + "fieldConfig": { + "columns": { + "revenue": { + "displayName": "Revenue", + "unit": " USD", + "decimals": 2 + }, + "purchases": { + "displayName": "Purchases", + "decimals": 0 + }, + "active_buyers": { + "displayName": "Active buyers", + "decimals": 0 + }, + "conversion_rate": { + "displayName": "Conversion rate", + "unit": "%", + "decimals": 1 + } + } + } }, "dashboard": { "role": "panel", "sizeHints": { - "preferred": "medium", + "preferred": "compact", "minimum": "compact", - "aspectRatio": 1.5 + "aspectRatio": 2 } } } }, { "id": "shop-daily-revenue", - "sql": "SELECT day, sum(revenue) AS revenue\nFROM shop.daily_sales\nGROUP BY day\nORDER BY day", + "sql": "SELECT day, sum(revenue) AS revenue FROM shop.daily_sales WHERE day BETWEEN toDate({from:DateTime}) AND toDate({to:DateTime})\n/*[ AND country IN {country:Array(String)} ]*/\nGROUP BY day ORDER BY day", "specVersion": 1, "spec": { "name": "Daily revenue", - "favorite": false, - "description": "Revenue per day across the last ~90 days, from daily_sales. A Date X axis is auto-detected as a time series → filled Area chart.", + "description": "", + "favorite": true, "view": "panel", "panel": { "cfg": { @@ -57,28 +73,36 @@ "series": null }, "key": "day:Date|revenue:Decimal(38, 2)" + }, + "dashboard": { + "role": "panel", + "sizeHints": { + "preferred": "medium", + "minimum": "compact", + "aspectRatio": 1.5 + } } } }, { - "id": "shop-revenue-by-category", - "sql": "SELECT category, sum(revenue) AS revenue\nFROM shop.category_revenue\nGROUP BY category\nORDER BY revenue DESC", + "id": "shop-revenue-by-country", + "sql": "SELECT country, sum(revenue) AS revenue FROM shop.daily_sales WHERE day BETWEEN toDate({from:DateTime}) AND toDate({to:DateTime})\n/*[ AND has({country:Array(String)}, country) ]*/\nGROUP BY country ORDER BY revenue DESC", "specVersion": 1, "spec": { - "name": "Revenue by category", + "name": "Revenue by country", + "description": "", "favorite": true, - "description": "Revenue per product category from category_revenue — the mv_category_revenue view derives the category with dictGet() against the products_dict dictionary. A small breakdown → Pie.", "view": "panel", "panel": { "cfg": { - "type": "pie", + "type": "hbar", "x": 0, "y": [ 1 ], "series": null }, - "key": "category:LowCardinality(String)|revenue:Decimal(38, 2)" + "key": "country:LowCardinality(String)|revenue:Decimal(38, 2)" }, "dashboard": { "role": "panel", @@ -91,13 +115,13 @@ } }, { - "id": "shop-event-mix", - "sql": "SELECT event_type, count() AS events\nFROM shop.events_raw\nGROUP BY event_type\nORDER BY events DESC", + "id": "shop-revenue-by-category", + "sql": "SELECT category, sum(revenue) AS revenue FROM shop.category_revenue WHERE day BETWEEN toDate({from:DateTime}) AND toDate({to:DateTime})\n/*[ AND has({category:Array(String)}, category) ]*/\nGROUP BY category ORDER BY revenue DESC", "specVersion": 1, "spec": { - "name": "Event type mix", - "favorite": false, - "description": "Share of raw events by type (purchase / view / cart) straight from events_raw. A small categorical split → Pie with a legend.", + "name": "Revenue by category", + "description": "", + "favorite": true, "view": "panel", "panel": { "cfg": { @@ -108,18 +132,26 @@ ], "series": null }, - "key": "event_type:LowCardinality(String)|events:UInt64" + "key": "category:LowCardinality(String)|revenue:Decimal(38, 2)" + }, + "dashboard": { + "role": "panel", + "sizeHints": { + "preferred": "medium", + "minimum": "compact", + "aspectRatio": 1.5 + } } } }, { "id": "shop-top-products", - "sql": "SELECT p.name AS product, round(sum(e.amount)) AS revenue\nFROM shop.events_raw AS e\nINNER JOIN shop.products AS p ON p.product_id = e.product_id\nWHERE e.event_type = 'purchase'\nGROUP BY product\nORDER BY revenue DESC\nLIMIT 15", + "sql": "SELECT dictGet('shop.products_dict', 'name', toUInt64(product_id)) AS product, round(sum(amount), 2) AS revenue FROM shop.events_raw WHERE event_type = 'purchase' AND event_time BETWEEN {from:DateTime} AND {to:DateTime}\n/*[ AND country IN {country:Array(String)} ]*/\n/*[ AND has({category:Array(String)}, dictGet('shop.products_dict', 'category', toUInt64(product_id))) ]*/\nGROUP BY product ORDER BY revenue DESC LIMIT 12", "specVersion": 1, "spec": { - "name": "Top 15 products by revenue", - "favorite": false, - "description": "Joins events_raw to the products dimension table for readable names, summing purchase amounts. Horizontal Bar of the top 15.", + "name": "Top products", + "description": "", + "favorite": true, "view": "panel", "panel": { "cfg": { @@ -131,17 +163,25 @@ "series": null }, "key": "product:String|revenue:Decimal(38, 2)" + }, + "dashboard": { + "role": "panel", + "sizeHints": { + "preferred": "medium", + "minimum": "compact", + "aspectRatio": 1.5 + } } } }, { "id": "shop-daily-active-users", - "sql": "SELECT toDate(hour) AS day, uniqMerge(users) AS users\nFROM shop.hourly_active_users\nGROUP BY day\nORDER BY day", + "sql": "SELECT toDate(hour) AS day, uniqMerge(users) AS users FROM shop.hourly_active_users WHERE hour BETWEEN {from:DateTime} AND {to:DateTime}\n/*[ AND country IN {country:Array(String)} ]*/\nGROUP BY day ORDER BY day", "specVersion": 1, "spec": { "name": "Daily active users", - "favorite": false, - "description": "Distinct users per day, merged from the hourly_active_users AggregatingMergeTree with uniqMerge() — shows how an AggregateFunction state is finalized. Date axis → Line.", + "description": "", + "favorite": true, "view": "panel", "panel": { "cfg": { @@ -153,28 +193,36 @@ "series": null }, "key": "day:Date|users:UInt64" + }, + "dashboard": { + "role": "panel", + "sizeHints": { + "preferred": "medium", + "minimum": "compact", + "aspectRatio": 1.5 + } } } }, { - "id": "shop-revenue-trend-by-country", - "sql": "SELECT day, country, sum(revenue) AS revenue\nFROM shop.daily_sales\nWHERE country IN ('US', 'GB', 'DE', 'JP')\nGROUP BY day, country\nORDER BY day", + "id": "shop-traffic-by-hour", + "sql": "SELECT toHour(event_time) AS hour, count() AS events FROM shop.events_raw WHERE event_time BETWEEN {from:DateTime} AND {to:DateTime}\n/*[ AND country IN {country:Array(String)} ]*/\n/*[ AND has({category:Array(String)}, dictGet('shop.products_dict', 'category', toUInt64(product_id))) ]*/\nGROUP BY hour ORDER BY hour", "specVersion": 1, "spec": { - "name": "Revenue trend by country", + "name": "Traffic by hour", + "description": "", "favorite": true, - "description": "Daily revenue for four countries at once — the country column drives the Series, producing one Line per country with a legend.", "view": "panel", "panel": { "cfg": { - "type": "line", + "type": "bar", "x": 0, "y": [ - 2 + 1 ], - "series": 1 + "series": null }, - "key": "day:Date|country:LowCardinality(String)|revenue:Decimal(38, 2)" + "key": "hour:UInt8|events:UInt64" }, "dashboard": { "role": "panel", @@ -187,24 +235,37 @@ } }, { - "id": "shop-orders-by-hour", - "sql": "SELECT toHour(event_time) AS hour, count() AS events\nFROM shop.events_raw\nGROUP BY hour\nORDER BY hour", + "id": "shop-country-revenue-trend", + "sql": "SELECT day, country, sum(revenue) AS revenue FROM shop.daily_sales WHERE day BETWEEN toDate({from:DateTime}) AND toDate({to:DateTime}) GROUP BY day, country ORDER BY day, country", "specVersion": 1, "spec": { - "name": "Traffic by hour of day", + "name": "Country revenue trends", + "description": "Additional Library analysis retained outside the flagship grid.", "favorite": false, - "description": "Event volume by hour of day (0–23) from events_raw. A numeric ordinal X axis → vertical Column chart.", "view": "panel", "panel": { "cfg": { - "type": "bar", + "type": "line", "x": 0, "y": [ - 1 + 2 ], - "series": null + "series": 1 }, - "key": "hour:UInt8|events:UInt64" + "key": "day:Date|country:LowCardinality(String)|revenue:Decimal(38, 2)" + } + } + }, + { + "id": "shop-filter-options", + "sql": "SELECT (SELECT arraySort(groupUniqArray(country)) FROM shop.events_raw) AS country, (SELECT arraySort(groupUniqArray(category)) FROM shop.products) AS category", + "specVersion": 1, + "spec": { + "name": "Shop dimensions", + "description": "Country and product-category choices for the Shop dashboard.", + "favorite": false, + "dashboard": { + "role": "filter" } } } @@ -212,31 +273,152 @@ "dashboards": [ { "documentVersion": 1, - "id": "shop-charts", + "id": "shop-analytics", "title": "Shop analytics", - "description": "Chart examples over the sample shop dataset.", + "description": "Revenue, buyers, products, geography, and traffic over the shop-demo.sql dataset.", "revision": 1, "layout": { - "type": "flow", + "type": "grafana-grid", "version": 1, - "preset": "columns-2", "items": { + "tile-shop-kpis": { + "span": 12, + "height": 2 + }, + "tile-shop-daily-revenue": { + "span": 8, + "height": 3 + }, "tile-shop-revenue-by-country": { - "span": 1, - "height": "large" + "span": 4, + "height": 3 }, "tile-shop-revenue-by-category": { - "span": 1, - "height": "large" + "span": 4, + "height": 3 }, - "tile-shop-revenue-trend-by-country": { - "span": 1, - "height": "large" + "tile-shop-top-products": { + "span": 8, + "height": 3 + }, + "tile-shop-daily-active-users": { + "span": 8, + "height": 3 + }, + "tile-shop-traffic-by-hour": { + "span": 4, + "height": 3 + } + }, + "fallback": { + "type": "flow", + "version": 1, + "preset": "columns-2", + "items": { + "tile-shop-kpis": { + "span": 2, + "height": "compact" + }, + "tile-shop-daily-revenue": { + "span": 1, + "height": "large" + }, + "tile-shop-revenue-by-country": { + "span": 1, + "height": "large" + }, + "tile-shop-revenue-by-category": { + "span": 1, + "height": "large" + }, + "tile-shop-top-products": { + "span": 1, + "height": "large" + }, + "tile-shop-daily-active-users": { + "span": 1, + "height": "large" + }, + "tile-shop-traffic-by-hour": { + "span": 1, + "height": "large" + } } } }, - "filters": [], + "filters": [ + { + "id": "shop-from", + "parameter": "from", + "label": "From", + "defaultValue": "-90d", + "defaultActive": true, + "targets": [ + "tile-shop-kpis", + "tile-shop-daily-revenue", + "tile-shop-revenue-by-country", + "tile-shop-revenue-by-category", + "tile-shop-top-products", + "tile-shop-daily-active-users", + "tile-shop-traffic-by-hour" + ] + }, + { + "id": "shop-to", + "parameter": "to", + "label": "To", + "defaultValue": "now", + "defaultActive": true, + "targets": [ + "tile-shop-kpis", + "tile-shop-daily-revenue", + "tile-shop-revenue-by-country", + "tile-shop-revenue-by-category", + "tile-shop-top-products", + "tile-shop-daily-active-users", + "tile-shop-traffic-by-hour" + ] + }, + { + "id": "shop-country", + "parameter": "country", + "label": "Country", + "sourceQueryId": "shop-filter-options", + "defaultValue": [], + "defaultActive": false, + "targets": [ + "tile-shop-kpis", + "tile-shop-daily-revenue", + "tile-shop-revenue-by-country", + "tile-shop-top-products", + "tile-shop-daily-active-users", + "tile-shop-traffic-by-hour" + ] + }, + { + "id": "shop-category", + "parameter": "category", + "label": "Category", + "sourceQueryId": "shop-filter-options", + "defaultValue": [], + "defaultActive": false, + "targets": [ + "tile-shop-kpis", + "tile-shop-revenue-by-category", + "tile-shop-top-products", + "tile-shop-traffic-by-hour" + ] + } + ], "tiles": [ + { + "id": "tile-shop-kpis", + "queryId": "shop-kpis" + }, + { + "id": "tile-shop-daily-revenue", + "queryId": "shop-daily-revenue" + }, { "id": "tile-shop-revenue-by-country", "queryId": "shop-revenue-by-country" @@ -246,8 +428,16 @@ "queryId": "shop-revenue-by-category" }, { - "id": "tile-shop-revenue-trend-by-country", - "queryId": "shop-revenue-trend-by-country" + "id": "tile-shop-top-products", + "queryId": "shop-top-products" + }, + { + "id": "tile-shop-daily-active-users", + "queryId": "shop-daily-active-users" + }, + { + "id": "tile-shop-traffic-by-hour", + "queryId": "shop-traffic-by-hour" } ] } diff --git a/examples/shop-demo.sql b/examples/shop-demo.sql index d9df9ce3..d158a0e5 100644 --- a/examples/shop-demo.sql +++ b/examples/shop-demo.sql @@ -1,7 +1,11 @@ +-- Required setup for examples/shop-charts.json. Load this file once with: +-- clickhouse-client --multiquery < examples/shop-demo.sql +-- The portable dashboard bundle intentionally does not duplicate schema/data. +-- -- shop — a tiny demo schema that EXISTS to show the data-flow graph: -- a raw events table feeding 3 materialized views into aggregate targets, -- a dictionary sourced from a dimension table, and a view on an aggregate. --- Generated data is intentionally small (50 products, 20k events). +-- Generated data is intentionally small (50 products, 300k events / ~90 days). -- -- On a replicated antalya cluster, wrap each statement in ON CLUSTER '{cluster}' -- and use Replicated* engines so every replica has it; shown single-node here. diff --git a/examples/system-explorer-charts.json b/examples/system-explorer-charts.json deleted file mode 100644 index b492bd2a..00000000 --- a/examples/system-explorer-charts.json +++ /dev/null @@ -1,404 +0,0 @@ -{ - "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", - "format": "altinity-sql-browser/portable-bundle", - "version": 1, - "exportedAt": "2026-07-04T19:47:55.982Z", - "metadata": { - "name": "ClickHouse system explorer", - "description": "Operational views over ClickHouse system tables." - }, - "queries": [ - { - "id": "sys-1", - "sql": "SELECT query_id, user, elapsed, read_rows, formatReadableSize(memory_usage) AS memory, left(query, 80) AS query\nFROM system.processes\nORDER BY elapsed DESC\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Currently running queries", - "favorite": false, - "description": "Live snapshot of system.processes — every query executing right now, slowest first. Empty when the cluster is idle; that's a real \"nothing running\" result, not an error." - } - }, - { - "id": "sys-2", - "sql": "SELECT database, table, elapsed, round(progress * 100, 1) AS pct_done, num_parts, is_mutation, formatReadableSize(total_size_bytes_compressed) AS size\nFROM system.merges\nORDER BY elapsed DESC\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Merges in progress", - "favorite": false, - "description": "Live snapshot of system.merges — background merges currently running, with progress and compressed size. Usually empty between merge cycles on a small cluster." - } - }, - { - "id": "sys-3", - "sql": "SELECT database, table, mutation_id, command, parts_to_do, latest_fail_reason\nFROM system.mutations\nWHERE NOT is_done\nORDER BY create_time\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Mutations in progress", - "favorite": false, - "description": "Unfinished ALTER UPDATE/DELETE mutations from system.mutations, with the failure reason if one is stuck retrying." - } - }, - { - "id": "sys-4", - "sql": "SELECT database, table, is_leader, is_readonly, absolute_delay, queue_size, inserts_in_queue, merges_in_queue\nFROM system.replicas\nORDER BY absolute_delay DESC\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Replication status", - "favorite": false, - "description": "system.replicas health per table — leadership, read-only state, replication delay, and queue depth. Sorted worst-lag-first." - } - }, - { - "id": "sys-5", - "sql": "SELECT database, table, type, create_time, num_tries, last_exception\nFROM system.replication_queue\nWHERE num_tries > 0\nORDER BY num_tries DESC\nLIMIT 20", - "specVersion": 1, - "spec": { - "name": "Stuck replication queue entries", - "favorite": false, - "description": "system.replication_queue entries that have already failed and retried at least once, with the last exception — the first place to look when a replica falls behind." - } - }, - { - "id": "sys-6", - "sql": "SELECT concat(database, '.', table) AS table, sum(bytes_on_disk) AS disk_bytes\nFROM system.parts\nWHERE active\nGROUP BY database, table\nORDER BY disk_bytes DESC\nLIMIT 15", - "specVersion": 1, - "spec": { - "name": "Largest tables by disk usage", - "favorite": true, - "description": "Every active part in system.parts, summed per table, largest first. Horizontal Bar — hover any bar for the exact byte count.", - "view": "panel", - "panel": { - "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null - }, - "key": "table:String|disk_bytes:UInt64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-7", - "sql": "SELECT concat(database, '.', table) AS table, count() AS parts\nFROM system.parts\nWHERE active\nGROUP BY database, table\nORDER BY parts DESC\nLIMIT 15", - "specVersion": 1, - "spec": { - "name": "Active parts by table", - "favorite": true, - "description": "Active part *count* per table (not size) — a table climbing here between refreshes is trending toward \"too many parts\". Horizontal Bar.", - "view": "panel", - "panel": { - "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null - }, - "key": "table:String|parts:UInt64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-8", - "sql": "SELECT name, value AS times\nFROM system.errors\nWHERE value > 0\nORDER BY value DESC\nLIMIT 15", - "specVersion": 1, - "spec": { - "name": "Cumulative error counters", - "favorite": true, - "description": "system.errors — every error code the server has hit since last restart, most frequent first. A quick \"what's actually going wrong here\" check. Horizontal Bar.", - "view": "panel", - "panel": { - "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null - }, - "key": "name:String|times:UInt64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-9", - "sql": "SELECT toStartOfMinute(event_time) AS t, count() AS queries\nFROM system.query_log\nWHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND type = 'QueryFinish'\nGROUP BY t\nORDER BY t", - "specVersion": 1, - "spec": { - "name": "Queries per minute", - "favorite": true, - "description": "Finished-query volume from system.query_log, bucketed per minute, over a {from:String}/{to:String} range (shared with every other time-ranged query below — the Dashboard filter bar renders one From/To pair that drives them all). A DateTime X axis is auto-detected as a time series → Line chart.", - "view": "panel", - "panel": { - "cfg": { - "type": "line", - "x": 0, - "y": [ - 1 - ], - "series": null - }, - "key": "t:DateTime|queries:UInt64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-10", - "sql": "SELECT left(any(query), 50) AS query, avg(query_duration_ms) AS avg_duration_ms\nFROM system.query_log\nWHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND type = 'QueryFinish'\nGROUP BY normalized_query_hash\nORDER BY avg_duration_ms DESC\nLIMIT 15", - "specVersion": 1, - "spec": { - "name": "Slowest query patterns — avg duration", - "favorite": true, - "description": "Distinct query shapes (system.query_log grouped by normalized_query_hash) ranked by average duration over a {from:String}/{to:String} range. Horizontal Bar of a non-count measure.", - "view": "panel", - "panel": { - "cfg": { - "type": "hbar", - "x": 0, - "y": [ - 1 - ], - "series": null - }, - "key": "query:String|avg_duration_ms:Float64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-11", - "sql": "SELECT toStartOfHour(event_time) AS t, errorCodeToName(exception_code) AS error, count() AS n\nFROM system.query_log\nWHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND exception_code != 0\nGROUP BY t, error\nORDER BY t", - "specVersion": 1, - "spec": { - "name": "Query errors over time", - "favorite": true, - "description": "Failed queries from system.query_log over a {from:String}/{to:String} range, broken down by ClickHouse error name. The \"error\" column is used as the Series, producing grouped/stacked bars per error code.", - "view": "panel", - "panel": { - "cfg": { - "type": "bar", - "x": 0, - "y": [ - 2 - ], - "series": 1 - }, - "key": "t:DateTime|error:LowCardinality(String)|n:UInt64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-12", - "sql": "SELECT toStartOfHour(event_time) AS t, event_type, count() AS n\nFROM system.part_log\nWHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String})\nGROUP BY t, event_type\nORDER BY t", - "specVersion": 1, - "spec": { - "name": "Part lifecycle events over time", - "favorite": true, - "description": "system.part_log over a {from:String}/{to:String} range — new/merged/mutated/downloaded/removed parts per hour, one query instead of five separate panels. \"event_type\" is the Series.", - "view": "panel", - "panel": { - "cfg": { - "type": "bar", - "x": 0, - "y": [ - 2 - ], - "series": 1 - }, - "key": "t:DateTime|event_type:Enum8('NewPart' = 1, 'MergeParts' = 2, 'DownloadPart' = 3, 'RemovePart' = 4, 'MutatePart' = 5, 'MovePart' = 6, 'MergePartsStart' = 7, 'MutatePartStart' = 8)|n:UInt64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-13", - "sql": "SELECT toStartOfMinute(event_time) AS t, avg(CurrentMetric_MemoryTracking) AS memory_bytes\nFROM system.metric_log\nWHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String})\nGROUP BY t\nORDER BY t", - "specVersion": 1, - "spec": { - "name": "Memory usage over time", - "favorite": true, - "description": "Average tracked memory (system.metric_log's CurrentMetric_MemoryTracking) per minute over a {from:String}/{to:String} range. Line chart.", - "view": "panel", - "panel": { - "cfg": { - "type": "line", - "x": 0, - "y": [ - 1 - ], - "series": null - }, - "key": "t:DateTime|memory_bytes:Float64" - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "medium", - "minimum": "compact", - "aspectRatio": 1.5 - } - } - } - }, - { - "id": "sys-14", - "sql": "SELECT\n normalized_query_hash,\n left(argMax(query, query_duration_ms), 60) AS sample_query,\n count() AS executions,\n max(query_duration_ms) AS max_ms,\n avg(query_duration_ms) AS avg_ms,\n sum(read_rows) AS read_rows,\n formatReadableSize(sum(read_bytes)) AS read_bytes,\n quantile(0.99)(memory_usage) AS p99_memory\nFROM system.query_log\nWHERE event_time BETWEEN parseDateTimeBestEffort({from:String}) AND parseDateTimeBestEffort({to:String}) AND type = 'QueryFinish'\nGROUP BY normalized_query_hash\nORDER BY avg_ms DESC\nLIMIT 15", - "specVersion": 1, - "spec": { - "name": "Query cost breakdown — slowest patterns (detail)", - "favorite": false, - "description": "The deep-dive version of \"Slowest query patterns\": executions, max/avg duration, rows and bytes read, and p99 memory per query shape over a {from:String}/{to:String} range. Table view — too many columns for one chart, but the full picture behind the bar chart above." - } - } - ], - "dashboards": [ - { - "documentVersion": 1, - "id": "system-explorer", - "title": "ClickHouse system explorer", - "description": "Operational views over ClickHouse system tables.", - "revision": 1, - "layout": { - "type": "flow", - "version": 1, - "preset": "columns-2", - "items": { - "tile-sys-6": { - "span": 1, - "height": "large" - }, - "tile-sys-7": { - "span": 1, - "height": "large" - }, - "tile-sys-8": { - "span": 1, - "height": "large" - }, - "tile-sys-9": { - "span": 1, - "height": "large" - }, - "tile-sys-10": { - "span": 1, - "height": "large" - }, - "tile-sys-11": { - "span": 1, - "height": "large" - }, - "tile-sys-12": { - "span": 1, - "height": "large" - }, - "tile-sys-13": { - "span": 1, - "height": "large" - } - } - }, - "filters": [ - { - "id": "filter-from", - "parameter": "from" - }, - { - "id": "filter-to", - "parameter": "to" - } - ], - "tiles": [ - { - "id": "tile-sys-6", - "queryId": "sys-6" - }, - { - "id": "tile-sys-7", - "queryId": "sys-7" - }, - { - "id": "tile-sys-8", - "queryId": "sys-8" - }, - { - "id": "tile-sys-9", - "queryId": "sys-9" - }, - { - "id": "tile-sys-10", - "queryId": "sys-10" - }, - { - "id": "tile-sys-11", - "queryId": "sys-11" - }, - { - "id": "tile-sys-12", - "queryId": "sys-12" - }, - { - "id": "tile-sys-13", - "queryId": "sys-13" - } - ] - } - ] -} diff --git a/examples/text-log-panel.json b/examples/text-log-panel.json deleted file mode 100644 index 8ca6a37b..00000000 --- a/examples/text-log-panel.json +++ /dev/null @@ -1,87 +0,0 @@ -{ - "$schema": "https://altinity.com/schemas/altinity-sql-browser/portable-bundle-v1.schema.json", - "format": "altinity-sql-browser/portable-bundle", - "version": 1, - "exportedAt": "2026-07-13T05:01:01.000Z", - "metadata": { - "name": "Server log panel example", - "description": "Parameterized system.text_log dashboard example." - }, - "queries": [ - { - "id": "log-1", - "sql": "SELECT event_time, level, logger_name, message\nFROM system.text_log\nWHERE event_time > {from:DateTime}\n/*[ AND event_time <= {to:DateTime} ]*/\n/*[ AND level = {level:Enum8('Fatal' = 1, 'Critical' = 2, 'Error' = 3, 'Warning' = 4, 'Notice' = 5, 'Information' = 6, 'Debug' = 7, 'Trace' = 8, 'Test' = 9)} ]*/\n/*[ AND positionCaseInsensitive(logger_name, {logger:String}) > 0 ]*/\n/*[ AND positionCaseInsensitive(message, {message:String}) > 0 ]*/\nORDER BY event_time DESC\nLIMIT 500", - "specVersion": 1, - "spec": { - "name": "Server logs in a time range", - "favorite": true, - "description": "{from:DateTime} is required (drives the Dashboard's global From filter, and accepts a relative expression like -1h/now-7d). to/level/logger/message are wrapped in /*[ ... ]*/ optional blocks, so they're non-required: blank ones drop out of the query entirely, filling one in adds its predicate. level renders as a dropdown (declared Enum8 member names); logger/message use positionCaseInsensitive(...) > 0 for a plain case-insensitive substring search — no ILIKE wildcard footgun if the search term itself contains a literal % or _.", - "view": "panel", - "panel": { - "cfg": { - "type": "logs", - "time": "event_time", - "msg": "message", - "level": "level" - } - }, - "dashboard": { - "role": "panel", - "sizeHints": { - "preferred": "wide", - "minimum": "medium", - "aspectRatio": 1.6 - } - } - } - } - ], - "dashboards": [ - { - "documentVersion": 1, - "id": "text-log-panel-example", - "title": "Server log panel example", - "description": "Parameterized system.text_log dashboard example.", - "revision": 1, - "layout": { - "type": "flow", - "version": 1, - "preset": "report", - "items": { - "tile-log-1": { - "span": 1, - "height": "large" - } - } - }, - "filters": [ - { - "id": "filter-from", - "parameter": "from" - }, - { - "id": "filter-to", - "parameter": "to" - }, - { - "id": "filter-level", - "parameter": "level" - }, - { - "id": "filter-logger", - "parameter": "logger" - }, - { - "id": "filter-message", - "parameter": "message" - } - ], - "tiles": [ - { - "id": "tile-log-1", - "queryId": "log-1" - } - ] - } - ] -} diff --git a/tests/unit/spec-examples.test.js b/tests/unit/spec-examples.test.js index ee8279cb..46497422 100644 --- a/tests/unit/spec-examples.test.js +++ b/tests/unit/spec-examples.test.js @@ -8,6 +8,7 @@ import { decodePortableBundleJson } from '../../src/dashboard/model/portable-bun import { querySpecSchemaService } from '../../src/core/spec-schema.js'; import { filterExecution } from '../../src/core/filter-execution.js'; import { effectiveDashboardRole } from '../../src/core/result-choice.js'; +import { analyzeParameterizedSources } from '../../src/core/param-pipeline.js'; const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..'); @@ -27,7 +28,11 @@ describe('schema artifacts and examples', () => { it('keeps every checked-in JSON example on portable bundle v1 with explicit Dashboard v1 documents', () => { const examples = resolve(root, 'examples'); - for (const name of readdirSync(examples).filter((item) => item.endsWith('.json'))) { + const names = readdirSync(examples).filter((item) => item.endsWith('.json')).sort(); + expect(names.filter((name) => !name.startsWith('iceberg'))).toEqual([ + 'clickhouse-operations.json', 'ontime-charts.json', 'shop-charts.json', + ]); + for (const name of names) { const text = readFileSync(resolve(examples, name), 'utf8'); const bundle = decodeExample(text, name); expect(bundle.format, name).toBe('altinity-sql-browser/portable-bundle'); @@ -36,8 +41,21 @@ describe('schema artifacts and examples', () => { expect(() => assertValidExampleBundle(bundle), name).not.toThrow(); for (const dashboard of bundle.dashboards) { expect(dashboard.documentVersion, name).toBe(1); - expect(dashboard.layout.type, name).toBe('flow'); + expect(['flow', 'grafana-grid'], name).toContain(dashboard.layout.type); expect(dashboard.tiles.length, name).toBeGreaterThan(0); + const tileIds = new Set(dashboard.tiles.map((tile) => tile.id)); + const queryIds = new Set(bundle.queries.map((query) => query.id)); + for (const tile of dashboard.tiles) expect(queryIds.has(tile.queryId), `${name}:${tile.id}`).toBe(true); + for (const filter of dashboard.filters) { + if (filter.sourceQueryId) expect(queryIds.has(filter.sourceQueryId), `${name}:${filter.id}`).toBe(true); + for (const target of filter.targets || []) expect(tileIds.has(target), `${name}:${filter.id}`).toBe(true); + } + if (dashboard.layout.type === 'grafana-grid') { + expect(dashboard.layout.fallback?.type, name).toBe('flow'); + expect(dashboard.layout.fallback?.version, name).toBe(1); + expect(Object.keys(dashboard.layout.items).sort(), name).toEqual([...tileIds].sort()); + expect(Object.keys(dashboard.layout.fallback.items).sort(), name).toEqual([...tileIds].sort()); + } } } expect(() => execFileSync(process.execPath, ['examples/mjs/normalize-examples.mjs', '--check'], { @@ -45,6 +63,48 @@ describe('schema artifacts and examples', () => { })).not.toThrow(); }); + it('keeps every flagship dimensional filter on one inferred multiselect Array(T) contract', () => { + const expected = { + 'ontime-charts.json': { carrier: 'Array(String)', origin: 'Array(String)' }, + 'shop-charts.json': { country: 'Array(String)', category: 'Array(String)' }, + 'clickhouse-operations.json': { + user: 'Array(String)', query_kind: 'Array(String)', + exception_code: 'Array(Int32)', query_hash: 'Array(UInt64)', + }, + }; + for (const [name, contracts] of Object.entries(expected)) { + const bundle = decodeExample(readFileSync(resolve(root, 'examples', name), 'utf8'), name); + const dashboard = bundle.dashboards[0]; + const queryById = new Map(bundle.queries.map((query) => [query.id, query])); + for (const [parameter, type] of Object.entries(contracts)) { + const filter = dashboard.filters.find((item) => item.parameter === parameter); + expect(filter?.sourceQueryId, `${name}:${parameter}`).toBeTruthy(); + expect(filter?.selection, `${name}:${parameter}`).toBeUndefined(); + const targetIds = filter.targets || dashboard.tiles.map((tile) => tile.id); + const sources = targetIds.map((tileId) => { + const tile = dashboard.tiles.find((item) => item.id === tileId); + const query = tile && queryById.get(tile.queryId); + return query ? { id: tileId, sql: query.sql, bindPolicy: 'row-returning' } : null; + }).filter(Boolean); + const analysis = analyzeParameterizedSources(sources); + const declarations = analysis.fields[parameter]?.declarations.filter((item) => item.bound) || []; + expect(declarations.length, `${name}:${parameter}`).toBeGreaterThan(0); + expect([...new Set(declarations.map((item) => item.type))], `${name}:${parameter}`).toEqual([type]); + } + } + }); + + it('keeps authored analytical chart encodings pinned to result schema keys', () => { + for (const name of ['ontime-charts.json', 'shop-charts.json']) { + const bundle = decodeExample(readFileSync(resolve(root, 'examples', name), 'utf8'), name); + const visible = new Set(bundle.dashboards[0].tiles.map((tile) => tile.queryId)); + for (const query of bundle.queries) { + if (!visible.has(query.id) || query.spec.panel?.cfg?.type === 'kpi') continue; + expect(query.spec.panel?.key, `${name}:${query.id}`).toMatch(/^[^:]+:.+/); + } + } + }); + it('validates the generated Iceberg drilldown portable-bundle template', () => { const template = readFileSync(resolve(root, 'examples/iceberg-templates/ice_meta_drilldown.json.tmpl'), 'utf8') .replaceAll('__CATALOG__', 'demo');