[sentry-junior]

Most of ClickHouse's CPU on a backend shard goes to setup. Phase 1 of bootstrap-snuba.py runs snuba bootstrap --force five times at once: snuba-snuba-1's default, plus default_gw0..3. Each one is ~700 DDL statements, mostly ALTERs of wide tables. On master x64 shard 21, ClickHouse used 80 of its 129 CPU-s in that ~37s window, about 2.2 of the runner's 4 cores, while the machine was pegged.

• One bootstrap, then copies. Phase 1 bootstraps default_gw0, then runs BACKUP DATABASE ... TO Memory and RESTORE ... AS default_gwN for the other workers. RESTORE rewrites the database in materialized views and in the Merge table. The copies match a bootstrapped database (SHOW CREATE for every table plus migrations_local) and take under a second. If the copy fails, it falls back to bootstrapping each worker. Locally this cut ClickHouse CPU for the bootstrap from 54 to 11 CPU-s.
• CI ClickHouse config, loaded before any per-worker Snuba connects. A users.d profile turns off log_queries, processor profiles, and the query and memory profilers, and the logger level drops from trace to warning. Both reload without a restart. Profile settings only apply when a session starts, and [experiment] ci(backend): stop ClickHouse logging every test query #129369 loaded them too late. A new ClickHouse queries logged step prints query_log rows per database, and default_gw* should be empty.
• Phases in the resource monitor. Run monitor-resources.py --phase NAME to set the phase. The sampler takes a sample as soon as the phase changes, and the summary's cpu_s_by_phase splits CPU by phase: setup, snuba-bootstrap, tests.

Before/after CI numbers, normalized by runner CPU model, are in a comment below.

Open questions:

• Is it OK for per-worker databases to come from RESTORE rather than migrations? The fallback covers a failed copy, not a schema that silently differs.
• Follow-ups not in this PR: snuba-snuba-1 still bootstraps default, which no test uses (~13–16 CPU-s). Turning off the server-level system logs (metric_log, asynchronous_metric_log, ...) needs a ClickHouse restart and saves only ~2–3 CPU-s per shard.

via @joshuarli.

--

View Session [Sentry]

[sentry-junior]

Results from run 43539870699 (all 30 shards passed, full suite with Trigger: Override Selective Testing).

PR backend shards run on Arm, and every shard here and in the baselines got the same CPU (arm implementer=0x41 part=0xd49), so no CPU-model normalization was needed. The baselines are four concurrent full-suite Arm PR runs on master's bootstrap: 43539157894, 43537880436, 43536092756, 43530563379. Master's x64 runs aren't comparable because they use different runners. For each shard index, I took the median of the four baselines and compared it with the same shard here. Setup and test are split at the Run backend test step's start time.

per shard (median)	baseline	this PR	paired diff
job wall	261.8s	245.5s	−14.8s (−6%)
setup wall (job start → tests)	93.8s	78.0s	−15.5s (−17%)
test step wall	157.8s	155.0s	−1.2s (noise)
ClickHouse CPU-s, setup	81.3	29.8	−52.1 (−64%)
ClickHouse CPU-s, tests	63.3	48.8	−13.1 (−21%)
ClickHouse CPU-s, total	145.6	78.4	−64.5 (−44%)

Across the 30 shards, that's about 8 runner-minutes and 32 ClickHouse CPU-minutes saved per full run.

• Phase split (new cpu_s_by_phase, median per shard): snuba-bootstrap lasts 20.1s and uses 27.6 ClickHouse CPU-s. That covers the gw0 bootstrap plus snuba-snuba-1's own default bootstrap running at the same time. The rest of setup uses 2.3 and tests uses 49.0. Before this PR, phase 1 was a ~37s window with ClickHouse at ~2.2 cores.
• Profile took effect: all 30 shards logged 0 query_log rows for default_gw0..3. Before, it was ~23k rows per shard, and [experiment] ci(backend): stop ClickHouse logging every test query #129369's reload didn't change that. default logged 1–193 rows from snuba-snuba-1 sessions opened before the reload. That's expected, since phase 2 stops snuba-snuba-1. The test-step drop of 13 CPU-s comes from this profile and the logger level.
• Copy: all 30 shards copied default_gw0 to gw1..3, and none fell back to bootstrapping each worker.

Open follow-ups, not in this PR:

• snuba-snuba-1's default bootstrap (~13–15 ClickHouse CPU-s, unused by tests) is now about half of the snuba-bootstrap phase.
• Turning off server-level system logs needs a restart. It saves ~2–3 CPU-s per shard (measured locally).

[linear-code]

DI-2852