Skip to content

Add scheme workload for creating and altering tables and topics - #58933

Open
stanislav-shchetinin wants to merge 7 commits into
ydb-platform:mainfrom
stanislav-shchetinin:feat/scheme-workload-v1
Open

stanislav-shchetinin wants to merge 7 commits into
ydb-platform:mainfrom
stanislav-shchetinin:feat/scheme-workload-v1

Conversation

@stanislav-shchetinin

@stanislav-shchetinin stanislav-shchetinin commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

KIKIMR-27705

Changelog entry

...

Description for reviewers

...

Local-ydb OOM runs

The configurations below produced confirmed OOM kills with workload commit d08d1af65a9bf7896fb08a172dda49babd0ce840. Each run used a fresh single-node local_ydb, a 4 GiB server cgroup memory limit, CPU affinity 0–3, no swap, tiny mode and disk-backed PDisks. The CLI ran outside the server cgroup.

run started from an empty database, without init (initial_count: 0), and mixed CREATE with ALTER. The duration limit was --seconds 180 for rows 1 and 4 and --seconds 60 for the other rows. Each listed configuration was run once; load was increased in preceding runs until OOM was observed.

# Target tables / topics Partitions per object CREATE / ALTER concurrency ALTER batch: columns / consumers Successful CREATE: tables / topics Successful ALTER Time to OOM, s
1 20000 / 0 1 16 / 16 1 / — 7079 / 0 9102 53.9
2 100 / 0 64 16 / 16 8 / — 100 / 0 1062 34.8
3 500 / 0 64 64 / 64 128 / — 72 / 0 72 10.6
4 0 / 20000 1 16 / 16 — / 1 0 / 5052 6473 39.3
5 0 / 100 64 16 / 16 — / 8 0 / 67 144 9.9
6 0 / 500 64 64 / 64 — / 64 0 / 64 24 7.5
7 50 / 50 64 64 / 64 128 / 64 50 / 19 35 8.3
8 200 / 200 16 32 / 32 64 / 16 200 / 186 481 11.8
9 500 / 500 64 128 / 128 128 / 64 12 / 0 50 10.2

All nine runs reached the 4 GiB cgroup limit. OOM was verified using the kernel CONSTRAINT_MEMCG / Killed process <PID> (ydbd) records and systemd Result=oom-kill. Time is measured from starting the run command, with memory.oom_control sampled every 0.1 s.

Target counts are YAML count values. Successful operation counts include only responses acknowledged to the client; requests with lost responses may have completed on the server. Batch sizes are configured values: rows 7 and 9 had no acknowledged consumer ALTERs before OOM.

These are observed OOM configurations under the stated memory limit, not minimum failure thresholds. Memory is shared by SchemeShard, DataShard and PQ in the same ydbd process; these runs do not attribute the exhaustion or a leak specifically to SchemeShard.

@codecov-commenter

codecov-commenter commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.00583% with 48 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ic/lib/ydb_cli/commands/scheme_workload/client.cpp 84.16% 3 Missing and 16 partials ⚠️
ydb/library/workload/scheme/config.cpp 75.71% 0 Missing and 17 partials ⚠️
ydb/library/workload/scheme/runner.cpp 89.71% 3 Missing and 8 partials ⚠️
...b_cli/commands/scheme_workload/scheme_workload.cpp 97.67% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add initial_count for optional preparation and create missing objects during run with independent CREATE and ALTER concurrency limits. Validate existing schemas before mutations and cover partial initialization, restarts, cancellation, and mixed scheduling with unit and SDK integration tests.
Preserve the scheme workload entry and the new release notes from main. Resolve the conflict that blocked the PR check-running-allowed gate.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-13 09:39:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for 77cac5a has started.
⚪ 2026-11-13 09:39:59 UTC Artifacts will be uploaded here
⚪ 2026-11-13 09:41:25 UTC ya make is running...
🟡 2026-11-13 10:28:33 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10344 10224 0 2 115 3

⚪ 2026-11-13 10:28:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-11-13 10:35:43 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
51 (only retried tests) 51 0 0 0 0

🟢 2026-11-13 10:35:48 UTC Build successful.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-18 09:31:20 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6731ff4 has started.
⚪ 2026-11-18 09:31:43 UTC Artifacts will be uploaded here
⚪ 2026-11-18 09:33:11 UTC ya make is running...
🟢 2026-11-18 10:05:26 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
7930 7854 0 0 71 5

🟢 2026-11-18 10:05:33 UTC Build successful.

Request partition boundaries during preflight and reject tables whose actual partition count differs from the YAML configuration. Add an integration regression covering rejection before CREATE or ALTER, and align private member names with the project style.
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-18 10:19:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for 94bdde0 has started.
⚪ 2026-11-18 10:19:28 UTC Artifacts will be uploaded here
⚪ 2026-11-18 10:20:57 UTC ya make is running...
⚫ 2026-11-18 10:37:32 UTC Check cancelled

@stanislav-shchetinin
stanislav-shchetinin marked this pull request as ready for review November 18, 2026 10:43
@stanislav-shchetinin
stanislav-shchetinin requested a review from a team as a code owner November 18, 2026 10:43
Copilot AI balanced review requested due to automatic review settings November 18, 2026 10:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-18 11:14:34 UTC Pre-commit check linux-x86_64-relwithdebinfo for 09edbac has started.
⚪ 2026-11-18 11:15:06 UTC Artifacts will be uploaded here
⚪ 2026-11-18 11:16:36 UTC ya make is running...
🟢 2026-11-18 11:48:02 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
7961 7885 0 0 71 5

🟢 2026-11-18 11:48:09 UTC Build successful.

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ✅ No Critical or Major issues found

Critical and Major issues

No Critical or Major issues found.

Other findings

  • Minor | Medium: Worker-count model is unbounded and inconsistent with config validation: Run sums create+alter worker counts, which can exceed the executor's int limit for configs that ParseConfig accepts — ydb/library/workload/scheme/runner.cpp:101
  • Minor | Medium: Execute spawns one native thread per concurrent object via RunAdditionalThreads, so large-but-valid concurrency values exhaust threads/memory mid-run instead of failing fast — ydb/library/workload/scheme/runner.cpp:55
  • Nit | High: The --seconds must be positive check also runs for the init subcommand where the option does not exist and the value is a constant — ydb/public/lib/ydb_cli/commands/scheme_workload/scheme_workload.cpp:37

This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.

for (ui64 index = 0; index < Config_.ObjectCount(); ++index) {
(states[index] == EObjectState::Missing ? missing : ready).push_back(index);
}
const ui64 concurrency = std::min<ui64>(Config_.CreateConcurrency, missing.size())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

Run computes the total worker count as min(CreateConcurrency, missing) + min(AlterConcurrency, ObjectCount). ParseConfig validates each concurrency field against std::numeric_limits<int>::max() (with the comment "The executor uses signed worker indices"), but this sum is not re-validated against that limit, so a config with e.g. create_concurrency: 2147483647 and enough objects passes parsing and then fails later in Execute with the generic "Invalid worker count" error. Additionally, for such large count values the states vector (ObjectCount() entries) and the thread pool below are sized from unvalidated counts, so the failure surfaces only after a multi-GB allocation.

Suggested fix: validate the combined create+alter worker bound (and a practical upper bound on count) in ParseConfig, or clamp the worker count in Run instead of asserting.

std::exception_ptr error;
const TCancelled stop = [&] { return failed.load() || Cancelled_(); };
NPar::TLocalExecutor executor;
executor.RunAdditionalThreads(workers - 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

Execute creates a native thread per concurrent object (RunAdditionalThreads(workers - 1)), and workers can be as large as min(concurrency, objects). Since the whole purpose of this tool is load testing, users are likely to raise create_concurrency/alter_concurrency together with object counts; a config like create_concurrency: 100000 with 100000+ objects would attempt to spawn ~100k OS threads, exhausting memory/handle limits or hanging thread creation mid-run, rather than failing fast with a clear error. The work is scheme-RPC-bound, so parallelism beyond a few hundred lanes adds little.

Suggested fix: either cap the executor thread count (process objects from a bounded worker pool) or reject impractically large concurrency values in ParseConfig with an explicit message.

void Parse(TConfig& config) override {
TClientCommand::Parse(config);
WorkloadConfig_ = NYdbWorkload::NScheme::ParseConfig(TFileInput(ConfigPath_).ReadAll());
Y_ENSURE(Seconds_ > 0, "--seconds must be positive");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Nit
Confidence: High

Y_ENSURE(Seconds_ > 0, "--seconds must be positive") in Parse also executes for the init subcommand, where --seconds is not registered and Seconds_ is the constant 60, so the check is dead code there and may confuse readers into thinking init honors a duration.

Suggested fix: move the check under the if (!Init_) branch (e.g. into Config/Parse next to the option definition).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants