Repository navigation
Add scheme workload for creating and altering tables and topics - #58933
stanislav-shchetinin wants to merge 7 commits into
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! 🚀 New features to boost your workflow:
|
Add initial_count for optional preparation and create missing objects during run with independent CREATE and ALTER concurrency limits. Validate existing schemas before mutations and cover partial initialization, restarts, cancellation, and mixed scheduling with unit and SDK integration tests.
Preserve the scheme workload entry and the new release notes from main. Resolve the conflict that blocked the PR check-running-allowed gate.
|
⚪ ⚪
🟢 |
|
⚪
🟢 |
Request partition boundaries during preflight and reject tables whose actual partition count differs from the YAML configuration. Add an integration regression covering rejection before CREATE or ALTER, and align private member names with the project style.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
|
⚪
🟢 |
There was a problem hiding this comment.
AI Review Summary
Verdict: ✅ No Critical or Major issues found
Critical and Major issues
No Critical or Major issues found.
Other findings
- Minor | Medium: Worker-count model is unbounded and inconsistent with config validation:
Runsums create+alter worker counts, which can exceed the executor'sintlimit for configs thatParseConfigaccepts —ydb/library/workload/scheme/runner.cpp:101 - Minor | Medium:
Executespawns one native thread per concurrent object viaRunAdditionalThreads, so large-but-valid concurrency values exhaust threads/memory mid-run instead of failing fast —ydb/library/workload/scheme/runner.cpp:55 - Nit | High: The
--seconds must be positivecheck also runs for theinitsubcommand where the option does not exist and the value is a constant —ydb/public/lib/ydb_cli/commands/scheme_workload/scheme_workload.cpp:37
This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.
| for (ui64 index = 0; index < Config_.ObjectCount(); ++index) { | ||
| (states[index] == EObjectState::Missing ? missing : ready).push_back(index); | ||
| } | ||
| const ui64 concurrency = std::min<ui64>(Config_.CreateConcurrency, missing.size()) |
There was a problem hiding this comment.
Severity: Minor
Confidence: Medium
Run computes the total worker count as min(CreateConcurrency, missing) + min(AlterConcurrency, ObjectCount). ParseConfig validates each concurrency field against std::numeric_limits<int>::max() (with the comment "The executor uses signed worker indices"), but this sum is not re-validated against that limit, so a config with e.g. create_concurrency: 2147483647 and enough objects passes parsing and then fails later in Execute with the generic "Invalid worker count" error. Additionally, for such large count values the states vector (ObjectCount() entries) and the thread pool below are sized from unvalidated counts, so the failure surfaces only after a multi-GB allocation.
Suggested fix: validate the combined create+alter worker bound (and a practical upper bound on count) in ParseConfig, or clamp the worker count in Run instead of asserting.
| std::exception_ptr error; | ||
| const TCancelled stop = [&] { return failed.load() || Cancelled_(); }; | ||
| NPar::TLocalExecutor executor; | ||
| executor.RunAdditionalThreads(workers - 1); |
There was a problem hiding this comment.
Severity: Minor
Confidence: Medium
Execute creates a native thread per concurrent object (RunAdditionalThreads(workers - 1)), and workers can be as large as min(concurrency, objects). Since the whole purpose of this tool is load testing, users are likely to raise create_concurrency/alter_concurrency together with object counts; a config like create_concurrency: 100000 with 100000+ objects would attempt to spawn ~100k OS threads, exhausting memory/handle limits or hanging thread creation mid-run, rather than failing fast with a clear error. The work is scheme-RPC-bound, so parallelism beyond a few hundred lanes adds little.
Suggested fix: either cap the executor thread count (process objects from a bounded worker pool) or reject impractically large concurrency values in ParseConfig with an explicit message.
| void Parse(TConfig& config) override { | ||
| TClientCommand::Parse(config); | ||
| WorkloadConfig_ = NYdbWorkload::NScheme::ParseConfig(TFileInput(ConfigPath_).ReadAll()); | ||
| Y_ENSURE(Seconds_ > 0, "--seconds must be positive"); |
There was a problem hiding this comment.
Severity: Nit
Confidence: High
Y_ENSURE(Seconds_ > 0, "--seconds must be positive") in Parse also executes for the init subcommand, where --seconds is not registered and Seconds_ is the constant 60, so the check is dead code there and may confuse readers into thinking init honors a duration.
Suggested fix: move the check under the if (!Init_) branch (e.g. into Config/Parse next to the option definition).
KIKIMR-27705
Changelog entry
...
Description for reviewers
...
Local-ydb OOM runs
The configurations below produced confirmed OOM kills with workload commit
d08d1af65a9bf7896fb08a172dda49babd0ce840. Each run used a fresh single-nodelocal_ydb, a 4 GiB server cgroup memory limit, CPU affinity 0–3, no swap, tiny mode and disk-backed PDisks. The CLI ran outside the server cgroup.runstarted from an empty database, withoutinit(initial_count: 0), and mixed CREATE with ALTER. The duration limit was--seconds 180for rows 1 and 4 and--seconds 60for the other rows. Each listed configuration was run once; load was increased in preceding runs until OOM was observed.All nine runs reached the 4 GiB cgroup limit. OOM was verified using the kernel
CONSTRAINT_MEMCG/Killed process <PID> (ydbd)records and systemdResult=oom-kill. Time is measured from starting theruncommand, withmemory.oom_controlsampled every 0.1 s.Target counts are YAML
countvalues. Successful operation counts include only responses acknowledged to the client; requests with lost responses may have completed on the server. Batch sizes are configured values: rows 7 and 9 had no acknowledged consumer ALTERs before OOM.These are observed OOM configurations under the stated memory limit, not minimum failure thresholds. Memory is shared by SchemeShard, DataShard and PQ in the same
ydbdprocess; these runs do not attribute the exhaustion or a leak specifically to SchemeShard.