labs/managed-rbac: how finely Managed ClickStack splits access

README.md

@@ -26,6 +26,7 @@ Observability on ClickHouse: OpenTelemetry ingestion, the ClickStack/HyperDX UI,
 | [labs/hostmetrics-volume](labs/hostmetrics-volume/) | Linux and Windows EC2: data points and wire bytes per host at 1-minute hostmetrics, four ways to watch a Windows game-server process stop (process scraper, perfcounters, `windows_service`, a check script) and one gateway behind an endpoint service (verified 2026-11-06) |
 | [labs/llm-serving-observability](labs/llm-serving-observability/) | Self-hosted LLM serving (vLLM behind LiteLLM) observed with ClickStack, runnable without a GPU: the OTel Collector configuration, an `nvidia-smi` → OTLP pusher, LiteLLM gateway-metric cardinality with and without the attribute allowlist, and the 32-tile HyperDX dashboard imported onto hand-made sources |
 | [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS audit trail and HA metrics in ClickStack: a three-node Raft cluster's audit files (written by whichever node is active) through `file_log` and OTTL into OTel logs — TraceId per request, issuance joined to later use on HMAC'd accessors, no secret in clear — plus per-node metrics with `unauthenticated_metrics_access`; failover checked (verified locally and on Managed ClickStack, 2026-11-06) |
+| [labs/managed-rbac](labs/managed-rbac/) | How finely Managed ClickStack splits access per game, user and column: ClickStack roles (resource, name/tag/ID, the API's beta row conditions), the per-user JWT identity search uses and the one shared identity alerts use, row and masking policies on each — and where alerts and Map keys leak (verified on Managed ClickStack with an Admin and a test user, 2026-11-07) |
 
 ### 🎓 Workshops (`workshops/`)
 
@@ -117,6 +118,7 @@ ClickHouse 기반 관측성 실습입니다. OpenTelemetry 수집, ClickStack/Hy
 | [labs/hostmetrics-volume](labs/hostmetrics-volume/) | Linux·Windows EC2: 1분 hostmetrics의 호스트당 data point와 전송 바이트, Windows 게임 서버 프로세스 중지를 보는 네 방법(process scraper, perfcounters, `windows_service`, 확인 스크립트), endpoint service 뒤의 gateway 하나 (2026-11-06 검증) |
 | [labs/llm-serving-observability](labs/llm-serving-observability/) | LiteLLM 뒤의 vLLM을 ClickStack으로 관측, GPU 없이 실행: OTel Collector 설정, `nvidia-smi` → OTLP 푸셔, 속성 허용 목록 유무에 따른 LiteLLM 게이트웨이 메트릭 카디널리티, 손으로 만든 소스에 가져온 HyperDX 대시보드(타일 32개) |
 | [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS 감사 기록과 HA 메트릭을 ClickStack에서: Raft 3노드 클러스터의 감사 파일(그때 active인 노드가 씀)을 `file_log`와 OTTL로 OTel 로그로 — 요청마다 TraceId, HMAC 처리된 accessor로 발급과 이후 사용 연결, 평문 시크릿 없음 — 그리고 `unauthenticated_metrics_access`로 노드별 메트릭, 장애 전환 검사(2026-11-06 로컬과 Managed ClickStack에서 검증) |
+| [labs/managed-rbac](labs/managed-rbac/) | Managed ClickStack 접근 권한을 게임·사용자·열 단위로 어디까지 나눌 수 있나: ClickStack 역할(리소스, 이름·태그·ID, API의 beta 행 조건), 검색이 쓰는 사용자별 JWT 신원과 알림이 쓰는 공용 신원 하나, 각각에 대한 row·masking policy, 그리고 알림과 Map 키로 새는 지점 (Admin과 시험 사용자로 Managed ClickStack에서 검증, 2026-11-07) |
 
 ### 🎓 워크숍 (`workshops/`)
 

labs/managed-rbac/.gitignore

@@ -0,0 +1 @@
+results/role-log.jsonl

labs/managed-rbac/lab.yaml

@@ -0,0 +1,5 @@
+# Runner keys (tools/lab.schema.md). Not run by hol: needs a ClickHouse Cloud service with Managed ClickStack.
+target: cloud
+tier: T1
+services: [clickhouse-cloud, clickstack]
+verified_on: "ClickHouse Cloud 26.8.1.608 (one Admin) and 26.6.1.2455 (Admin + test user), Managed ClickStack hyperdx-chc 2.43.1 (alerts 2.42.0), 2026-11-07"

labs/managed-rbac/results/2026-11-07-managed.md

@@ -0,0 +1,109 @@
+# 2026-11-07 run — Managed ClickStack, Seoul
+
+ClickHouse Cloud `26.8.1.608`, ap-northeast-2, 2 replicas. ClickStack UI client
+`hyperdx-chc 2.43.1-sha831caee` (from `http_user_agent`), alert worker
+`hyperdx-chc-alerts 2.42.0-sha91f0392`. One ClickStack user, role Admin. Times UTC.
+Identifiers (IdP subject, the JWT name's hash, object IDs) are left out on purpose.
+
+## Data
+
+`bin/lab.py setup` at 04:36: 172,800 rows over the previous 24 h (alpha / bravo / charlie ≈ 46.5k
+each, prod ≈ 80 %), then `bin/lab.py feed 90` (60 rows every 30 s) from 04:44.
+
+## Identities (query_log, session_log, last 7 days / today)
+
+| Who | Client | Session roles | Seen as |
+|---|---|---|---|
+| `JWT::<idp>_<subject>:<email>::<hash>` | `hyperdx-chc 2.43.1` | `[]`, profile `default`, auth `JWT` | search, dashboards, filters |
+| `hdx-alert-internal` | `hyperdx-chc-alerts 2.42.0` | `['hdx_alert_role']`, auth `SSL_CERTIFICATE` | every alert query |
+| `sql-console` | `sqlui/261106…` | `['clickpipes_system','default_role']` | the console's SQL editor |
+
+`system.users`: the JWT user is in storage `cloud`, auth type `jwt`. `SHOW GRANTS` for it lists
+the full SQL-console-admin privilege set granted directly (including `CREATE ROW POLICY`,
+`CREATE MASKING POLICY`, `ROLE ADMIN`, with grant option). No ClickStack query had
+`additional_table_filters` in `Settings`. One JWT name (one hash) was seen all day.
+
+## ClickStack roles (Cloud API, beta endpoints)
+
+- `GET /clickstack/roles` on both services of the organization: the same three system roles with the
+  same IDs. `GET /clickstack/sources`: 14 on one service, 0 on the other.
+- `POST /clickstack/roles` with `manage User` or `manage all`: HTTP 400 `Roles cannot grant
+  create, update, or delete access to Users. Only the Admin role can invite, remove, or re-role
+  team members.`
+- Eight roles accepted (`lab.py role all`, 04:40–04:41), including three with
+  `integration: clickhouse` row conditions (SQL and Lucene). `system.row_policies` stayed empty —
+  nothing is created on the ClickHouse side while no user holds the role.
+- Team Settings → Members & Roles → Edit on `rbac-lab · team-alpha`: Dashboards / Saved Searches /
+  Sources "Using Fine-Grained Controls", the tag rule shown as *Tag is game:alpha · Read*; no
+  section for the row condition. The dialog was closed without Save.
+- Team members: the runner (Admin) and one Member. No restricted-role session was observed.
+
+## Row policies (each step followed by a fresh search: a new time range)
+
+| Step | Policy | ClickStack search (6 h, group by Game) | Alert queries | query_log |
+|---|---|---|---|---|
+| D0 04:44 | none | 42,368 rows, 3 games | all games | `used_row_policies = []` |
+| D1 04:52 | `USING Game='alpha' TO sql_console_admin, sql_console_read_only` | 1,070 rows / 15 min, **3 games** | — | — |
+| D2 04:45 | `USING Game='alpha' TO ALL EXCEPT default` | 14,117 rows, alpha only | from 04:46 bravo, charlie `count()` = 0; states bravo, charlie OK, alpha ALERT (04:49) | policy listed on JWT and `hdx-alert-internal` queries |
+| D3 04:46 | + `USING Game='bravo' TO <that JWT user>` | 28,242 rows, alpha + bravo; bravo dashboard SQL tile (no WHERE): alpha, bravo rows only | unchanged | both policies listed |
+| D3' | same `TO` a JWT name that never logged in | — | — | `UNKNOWN_ROLE` |
+| D4 04:49 | D2 altered to `EXCEPT default, hdx-alert-internal` | unchanged | 04:50 bravo, charlie ALERT | alert queries without policy |
+
+D1 ran last (after the others were dropped); D2–D4 are in time order.
+
+## Masking policies
+
+| Step | Policy | Result |
+|---|---|---|
+| M1 04:49 | `LogAttributes = mapApply((k, v) -> (k, if(k='client.ip', '***masked***', v)), LogAttributes) TO <JWT user>` | every ClickStack query of that user: `ClickHouseQueryError … There is no column k in table` |
+| — | `CREATE OR REPLACE MASKING POLICY …` | syntax error: not accepted; DROP then CREATE |
+| M2 04:50 | `LogAttributes = mapUpdate(LogAttributes, map('client.ip','***masked***'))` | ClickStack column `LogAttributes['client.ip']` shows the original; row detail shows `***masked***` |
+| M2, SQL user | same policy on a temporary user | `SELECT LogAttributes` masked; `SELECT LogAttributes['client.ip']` original (masked only with `optimize_functions_to_subcolumns = 0`); `WHERE LogAttributes['client.ip'] LIKE '10.%'` → 57,920 rows |
+| M3 04:51 | `Body = replaceRegexpAll(Body, 'player=p-[0-9]+', 'player=***')` | ClickStack Body masked; SQL user: `WHERE Body LIKE '%player=p-0041%'` → 0, `LIKE '%player=***%'` → 57,920 |
+
+All policies and the temporary user dropped at 04:53. The ClickStack objects, the scenario roles
+and `rbac_lab` were kept for a second round with more users.
+
+# Round 2 — a second user, trial organization (same day)
+
+A new trial organization with one service: ClickHouse Cloud `26.6.1.2455`, ap-northeast-2, one
+replica (12 GiB), ClickStack UI `hyperdx-chc 2.43.1-sha831caee`, alert worker
+`hyperdx-chc-alerts 2.42.0-sha91f0392`. Two users: the organization Admin (Google sign-in, ClickStack
+Admin) and a test user (email and password, console roles Member + Service Admin, ClickStack role
+changed per step). Each in its own Chrome profile, so both views were driven side by side.
+Same `sql/schema.sql`, `sql/rows.sql` (172,800 rows), `lab.py setup` and `role all`; feed every 30 s
+from 05:48 to 06:32 UTC. Times UTC.
+
+## Identities
+
+| Who | Seen as | Note |
+|---|---|---|
+| `JWT::ClickHouse::<service-id>::<idp-subject>:<email>::<hash>` | `hyperdx-chc` (search, dashboards) **and** `sql-console/production` (the console SQL editor) | the name carries the service ID here; on the Seoul service it did not |
+| `hyperdx-alert-internal` | `hyperdx-chc-alerts` | `hdx-alert-internal` on the Seoul service |
+
+## ClickStack roles, as the test user saw them
+
+| Step | Role given in Team Settings | Result |
+|---|---|---|
+| R1 06:23 | `rbac-lab · rows-alpha` (row condition) | **refused**: `Not allowed to assign a role with privileges exceeding your own` (the Admin assigning) |
+| R1' 06:31 | `rbac-lab · team-alpha` (row condition + objects + alerts) | refused, same message |
+| R2 06:24 | `rbac-lab · types` as first written (no Team read) | assigned; the test user's ClickStack shows only `Unauthorized — You are not authorized to access this resource` |
+| R3 06:28 | `types` with `read Team` added | dashboards listed (all three) and open read-only ("You have read-only access to this dashboard"); no Alerts or AI notebooks in the menu; `/alerts` says "You do not have permission to view alerts" |
+| R4 06:29 | `objects-alpha` (tag `game:alpha` read, source by name) | only `rbac-lab · alpha` listed; the bravo dashboard by URL: "Dashboard Not Available"; **search on the same source with `Game:"bravo"`: 7,039 rows** |
+| R5 06:31 | `Member` (system) | all three alerts listed with state and history |
+
+`objects-id` and the system `ReadOnly` role were created but not assigned in this round.
+
+## Row policy on one user, two users side by side
+
+`CREATE ROW POLICY rbac_lab_user_alpha ON rbac_lab.otel_logs FOR SELECT USING Game = 'alpha' TO <test user's JWT name>` at 06:30.
+
+| | Test user | Admin |
+|---|---|---|
+| Search, same 3 h, group by Game | 7,036 rows, alpha only | 21,109 rows, three games |
+| `Game:"charlie" SeverityText:"ERROR"`, 1 h | 0 rows, 0 scanned | — |
+| Alerts page (test user as Member) | all three alerts; charlie `ALERT`, bravo and charlie history bars red | same |
+| Console SQL editor, `SELECT Game, count() … GROUP BY Game` | alpha 59,380; `currentUser() LIKE 'JWT::%'` = 1 | — |
+
+query_log 06:23–06:33: 32 queries of the test user carried `rbac_lab_user_alpha`; alert queries
+carried none.

labs/managed-rbac/sql/policies.sql

@@ -0,0 +1,45 @@
+-- labs/managed-rbac: the ClickHouse-side statements the 2026-11-07 run used, in order, on
+-- ClickHouse Cloud 26.8.1.608 (Seoul). Run them as the service's `default` user. Replace
+-- {jwt_user} with the ClickStack user's database name, read after that user has opened
+-- ClickStack once:  SELECT name FROM system.users WHERE name LIKE 'JWT::%'
+-- Each step was followed by a fresh ClickStack search (a new time range, so no cached result)
+-- and by system.query_log (user, used_row_policies, result_rows).
+
+-- D1. A policy on the SQL console roles. No effect on ClickStack: its JWT session has no roles.
+CREATE ROW POLICY rbac_lab_role_based ON rbac_lab.otel_logs FOR SELECT USING Game = 'alpha'
+    TO sql_console_admin, sql_console_read_only;
+DROP ROW POLICY rbac_lab_role_based ON rbac_lab.otel_logs;
+
+-- D2. A blanket policy. Every ClickStack user AND the alert worker see alpha only.
+CREATE ROW POLICY rbac_lab_alpha_all ON rbac_lab.otel_logs FOR SELECT USING Game = 'alpha'
+    TO ALL EXCEPT default;
+
+-- D3. One more policy for one ClickStack user. Permissive policies are OR-ed: alpha + bravo.
+CREATE ROW POLICY rbac_lab_user_bravo ON rbac_lab.otel_logs FOR SELECT USING Game = 'bravo'
+    TO `{jwt_user}`;
+-- A name that has not logged in yet is refused: UNKNOWN_ROLE.
+
+-- D4. The alert worker left out of the blanket policy: alerts see every game again.
+ALTER ROW POLICY rbac_lab_alpha_all ON rbac_lab.otel_logs TO ALL EXCEPT default, `hdx-alert-internal`;
+
+-- M1. Masking with a lambda: every query of that user fails ("There is no column k in table").
+CREATE MASKING POLICY rbac_lab_mask_ip ON rbac_lab.otel_logs
+    UPDATE LogAttributes = mapApply((k, v) -> (k, if(k = 'client.ip', '***masked***', v)), LogAttributes)
+    TO `{jwt_user}`;
+DROP MASKING POLICY rbac_lab_mask_ip ON rbac_lab.otel_logs;   -- OR REPLACE is not accepted
+
+-- M2. Masking a Map key without a lambda: the whole map is masked, LogAttributes['client.ip'] is not.
+CREATE MASKING POLICY rbac_lab_mask_ip ON rbac_lab.otel_logs
+    UPDATE LogAttributes = mapUpdate(LogAttributes, map('client.ip', '***masked***'))
+    TO `{jwt_user}`;
+
+-- M3. Masking a String column: masked in results and in WHERE.
+CREATE MASKING POLICY rbac_lab_mask_body ON rbac_lab.otel_logs
+    UPDATE Body = replaceRegexpAll(Body, 'player=p-[0-9]+', 'player=***')
+    TO `{jwt_user}`;
+
+-- Clean up
+DROP ROW POLICY IF EXISTS rbac_lab_alpha_all ON rbac_lab.otel_logs;
+DROP ROW POLICY IF EXISTS rbac_lab_user_bravo ON rbac_lab.otel_logs;
+DROP MASKING POLICY IF EXISTS rbac_lab_mask_ip ON rbac_lab.otel_logs;
+DROP MASKING POLICY IF EXISTS rbac_lab_mask_body ON rbac_lab.otel_logs;

labs/managed-rbac/sql/rows.sql

@@ -0,0 +1,37 @@
+-- Synthetic rows: {n:UInt64} rows spread evenly from now() - {span_s:UInt64} seconds to now().
+-- Three games (alpha, bravo, charlie), prod 80 % / qa 20 %, three services per game,
+-- 200 players, 5 % ERROR and 10 % WARN. Every value is made up.
+INSERT INTO rbac_lab.otel_logs
+SELECT
+    now64(9) - toIntervalMillisecond(intDiv(({n:UInt64} - number) * {span_s:UInt64} * 1000, {n:UInt64})) AS Timestamp,
+    lower(hex(cityHash64(number, 't'))) AS TraceId,
+    lower(hex(cityHash64(number, 's') % 4294967296)) AS SpanId,
+    1 AS TraceFlags,
+    multiIf(r % 100 < 5, 'ERROR', r % 100 < 15, 'WARN', 'INFO') AS SeverityText,
+    multiIf(r % 100 < 5, 17, r % 100 < 15, 13, 9) AS SeverityNumber,
+    concat(game, '-', svc) AS ServiceName,
+    multiIf(
+        svc = 'billing' AND r % 100 < 5, concat('payment failed player=', player, ' code=PG_TIMEOUT'),
+        svc = 'billing', concat('purchase ok player=', player, ' item=gem_pack_', toString(r % 7)),
+        svc = 'match' AND r % 100 < 5, concat('match server crashed room=', toString(r % 500), ' player=', player),
+        svc = 'match', concat('match finished room=', toString(r % 500), ' player=', player, ' duration_s=', toString(60 + r % 600)),
+        r % 100 < 15, concat('slow login player=', player, ' latency_ms=', toString(800 + r % 2000)),
+        concat('login ok player=', player)) AS Body,
+    '' AS ResourceSchemaUrl,
+    map('service.name', concat(game, '-', svc), 'game', game, 'deployment.environment.name', env,
+        'team', concat(game, '-liveops'), 'cloud.region', if(game = 'charlie', 'us-west-2', 'ap-northeast-1')) AS ResourceAttributes,
+    '' AS ScopeSchemaUrl, 'rbac-lab' AS ScopeName, '' AS ScopeVersion, map() AS ScopeAttributes,
+    map('player.id', player, 'client.ip', concat('10.', toString(r % 256), '.', toString(intDiv(r, 256) % 256), '.', toString(number % 254 + 1)),
+        'guild.id', concat('g-', toString(r % 40))) AS LogAttributes,
+    '' AS EventName,
+    game AS Game,
+    env AS Env
+FROM (
+    SELECT number,
+           cityHash64(number, {seed:String}) AS r,
+           ['alpha', 'bravo', 'charlie'][number % 3 + 1] AS game,
+           if(intDiv(r, 7) % 10 < 8, 'prod', 'qa') AS env,
+           ['gateway', 'match', 'billing'][intDiv(r, 11) % 3 + 1] AS svc,
+           concat('p-', leftPad(toString(intDiv(r, 13) % 200), 4, '0')) AS player
+    FROM numbers({n:UInt64})
+);

labs/managed-rbac/sql/schema.sql

@@ -0,0 +1,31 @@
+-- labs/managed-rbac: one shared log table for three games, the way a game company often lands logs
+-- (game and environment first in the sort key). ClickStack's otel_logs columns plus two top-level
+-- columns, Game and Env, that row conditions and row policies can filter on cheaply.
+CREATE DATABASE IF NOT EXISTS rbac_lab;
+
+CREATE TABLE IF NOT EXISTS rbac_lab.otel_logs
+(
+    `Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+    `TraceId` String CODEC(ZSTD(1)),
+    `SpanId` String CODEC(ZSTD(1)),
+    `TraceFlags` UInt8,
+    `SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
+    `SeverityNumber` UInt8,
+    `ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
+    `Body` String CODEC(ZSTD(1)),
+    `ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
+    `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+    `ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
+    `ScopeName` String CODEC(ZSTD(1)),
+    `ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
+    `ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+    `LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+    `EventName` String CODEC(ZSTD(1)),
+    `Game` LowCardinality(String) CODEC(ZSTD(1)),
+    `Env` LowCardinality(String) CODEC(ZSTD(1)),
+    INDEX idx_lower_body lower(Body) TYPE text(tokenizer = 'splitByNonAlpha') GRANULARITY 100000000
+)
+ENGINE = MergeTree
+PARTITION BY toDate(Timestamp)
+ORDER BY (Game, Env, toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
+TTL toDateTime(Timestamp) + toIntervalDay(7);