Skip to content

kqp: pass and use PRAGMA ydb.MaxTasksPerStage - #59980

Draft
yumkam wants to merge 1 commit into
ydb-platform:mainfrom
yumkam:fix-max-tasks-per-stage
Draft

yumkam wants to merge 1 commit into
ydb-platform:mainfrom
yumkam:fix-max-tasks-per-stage

Conversation

@yumkam

@yumkam yumkam commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

It is currently only used for source tasks and HashShuffle ignores it. One can override it with ydb.OverridePlanner, but it is quite fragile and user-unfriendly (they must guess tx and stage, which may wildly variate depending on query)

Changelog entry

...

Description for reviewers

...

It is currently only used for source tasks and HashShuffle ignores it.
One can override it with ydb.OverridePlanner, but it is quite fragile
and user-unfriendly.
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-19 01:13:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for f1042b3 has started.
⚪ 2026-11-19 01:14:18 UTC Artifacts will be uploaded here
⚪ 2026-11-19 01:15:48 UTC ya make is running...
🟡 2026-11-19 02:30:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
55067 53262 0 4 1793 8

⚪ 2026-11-19 02:31:13 UTC ya make is running... (failed tests rerun, try 2)
🟡 2026-11-19 02:32:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
425 (only retried tests) 422 0 2 0 1

⚪ 2026-11-19 02:32:31 UTC ya make is running... (failed tests rerun, try 3)
🔴 2026-11-19 02:33:26 UTC Some tests failed, follow the links below.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
403 (only retried tests) 400 0 2 0 1

🟢 2026-11-19 02:33:30 UTC Build successful.
🟡 2026-11-19 02:33:54 UTC ydbd size 2.9 GiB changed* by +30.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7c1bcec merge: f1042b3 diff diff %
ydbd size 3 045 473 272 Bytes 3 077 703 920 Bytes +30.7 MiB +1.058%
ydbd stripped size 643 672 192 Bytes 647 376 640 Bytes +3.5 MiB +0.576%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ❌ Critical: 0; Major: 1

Critical and Major issues

  • Major | Medium: MaxTasksPerStage is not propagated to the per-partition requests created by TKqpPartitionedExecuter (FillRequestFrom()), so BATCH UPDATE/DELETE executions run with the pragma cap silently disabled — ydb/core/kqp/executer_actor/kqp_partitioned_executer.cpp:56

Other findings

  • Minor | Medium: stages with a forced task count (PRAGMA ydb.OverridePlanner) take the FIXED branch and bypass the new MaxTasksPerStage clamp — ydb/core/kqp/executer_actor/kqp_tasks_graph.cpp:4442
  • Minor | Medium: the clamp added to CountComputeTasks() has no test coverage; existing pragma tests only exercise external-source stages — ydb/core/kqp/executer_actor/kqp_tasks_graph.cpp:4448

This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.

} else {
auto [newPartitionCount, _] = GetMaxTasksAggregation(stageInfo, inputTasks, nodesCount);
partitionsCount = std::max(newPartitionCount, partitionsCount);
if (MaxTasksPerStage != 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

The clamp added here has no test coverage. The existing PRAGMA ydb.MaxTasksPerStage tests (kqp_federated_query_ut.cpp, kqp_s3_plan_ut.cpp, streaming_*) only exercise external-source stages, where the limit was already honored via the compiler before this change; nothing asserts that a hash-shuffle stage with the pragma set produces at most MaxTasksPerStage tasks in CountComputeTasks(). A test over a multi-shard table (e.g. a group-by/join query with PRAGMA ydb.MaxTasksPerStage = "2" verifying the produced task count via stats/plan) is needed to keep the changed behavior safe.

@robot-vibe-db

robot-vibe-db Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Finding outside the inline diff

ydb/core/kqp/executer_actor/kqp_partitioned_executer.cpp:56

Severity: Major
Confidence: Medium

FillRequestFrom() copies per-request execution settings into the per-partition requests that TKqpPartitionedExecuter creates for every BATCH UPDATE/DELETE partition (CreateExecuterWithBuffer -> CreateKqpExecuter), but the new MaxTasksPerStage field added to TExecPhysicalRequest in this PR is not in the list. The original request carries the pragma value set by PrepareBaseRequest() (kqp_session_actor.cpp:1890), yet every partition request is rebuilt with MaxTasksPerStage = 0, so its tasks graph runs with the limit disabled and hash-shuffle stages inside a batch operation can spawn an unbounded number of tasks — the exact growth the new clamp in TKqpTasksGraph::CountComputeTasks() is meant to bound. A large batch operation on a heavily partitioned table therefore does not honor PRAGMA ydb.MaxTasksPerStage at all.

Suggested fix: add request.MaxTasksPerStage = from.MaxTasksPerStage; to FillRequestFrom(). Note that OutputChunkMaxSize was already missed by this copy helper in the same way, so consider making the propagation exhaustive (e.g. copying a dedicated settings sub-struct) so the next added request field cannot be silently dropped.

@robot-vibe-db

robot-vibe-db Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Finding outside the inline diff

ydb/core/kqp/executer_actor/kqp_tasks_graph.cpp:4442

Severity: Minor
Confidence: Medium

The new MaxTasksPerStage clamp is only applied in the else branch below: a stage with a forced TaskCount (set by PRAGMA ydb.OverridePlanner via stage.SetTaskCount() in ApplyOverridePlannerSettings, kqp_query_compiler.cpp:697) takes this FIXED branch and ignores the pragma entirely. A user who combines PRAGMA ydb.MaxTasksPerStage = "N" with an OverridePlanner entry larger than N gets more than N tasks in that stage, which contradicts the pragma's intent of being a per-stage cap.

If the pragma is meant to be an absolute cap, apply it to the FIXED branch as well (or validate/intersect the two limits at compile time in the compiler); otherwise the precedence between the two pragmas is worth documenting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant