Skip to content

Benchmark parsed YQL requests at the KQP logical-root boundary - #59987

Open
asmyasnikov wants to merge 7 commits into
ydb-platform:mainfrom
asmyasnikov:codex/parsed-query-main-benchmark
Open

asmyasnikov wants to merge 7 commits into
ydb-platform:mainfrom
asmyasnikov:codex/parsed-query-main-benchmark

Conversation

@asmyasnikov

@asmyasnikov asmyasnikov commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Measure four ways to supply a read-only query to the legacy KQP SQL frontend on main with EnableNewRBO. This is the main-branch counterpart to the NewParser benchmark.

The text baseline enters the stock PrepareGenericQuery(SQL). The grammar stage supplies the generated TSQLv1ParserAST and its original tokens to SqlASTToYql. A translated YQL AST resumes before CompileExpr. The late kql_query_list input restores the expression DAG produced by BuildKqlQuery, rebinds table metadata against the supplied catalog and enters the same RBO pipeline. The host entry points let the tool resume at these actual compiler boundaries and the callbacks expose intermediate results that the ordinary prepare API consumes internally.

Each server sample starts immediately before decoding a complete in-memory ExecuteQueryRequest and stops in TKqpNewRBOTransformer::DoTransform(), just before if (Roots.size() > 1). At this point the compiler has set up the logical roots and called ComputeParents(). A compilation-local observer records the timestamp on the calling thread and terminates the replay before statistics or optimizer rules.

The benchmark warms the offline catalog, verifies the full request envelope and the logical outputs against SQL, rotates input formats and records wall and thread CPU time for every iteration. A separate producer run starts with rendered SQL and ends after serialization of the complete request. The experimental messages live under the test tool. Field 4 holds the public YQL text input. Field 5 holds ParsedQuery, whose oneof stage selects a protobuf representation of an intermediate compiler result. Timer endpoint

Public corpus results

Public query Samples per format text antlr_ast yql_ast kql_query_list
ClickBench Q1 300 3.901 3.864 3.785 0.926
TPC-C warehouse tax 300 2.127 2.055 1.922 0.613
TPC-C Stock Level 300 4.085 3.954 3.684 1.168
KQP five-table join 300 4.435 4.292 4.071 1.387
KQP ManyJoinsWithDecimalRanges, 90 LEFT JOINs 100 176.719 168.965 160.686 55.163
Controlled extension of KQP ManyJoinsWithDecimalRanges, 96 LEFT JOINs 100 191.267 183.810 175.185 59.736
ClickBench Q30 75 59.602 57.906 56.360 17.976
TPC-H Q8 — — — — —
TPC-DS Q88 75 43.974 42.005 38.758 9.060
TPC-DS Q14 — — — — —

Cells give median wall milliseconds per request. A dash means the series has no verified measurement for that input and format.

On the 90-join query, text took 184.713 ms at p95. Mean CPU was 178.447 ms on the server and 0.004 ms on the producer. Its full request used 12,241 bytes.

On the 90-join query, kql_query_list took 57.248 ms at p95. Mean CPU was 55.447 ms on the server and 141.425 ms on the producer. Its full request used 123,564 bytes.

The table below compares the stock PrepareGenericQuery entry points for the same public text request. Pairing requires matching ordered output names and types and rejects NewParser fallback. 6 of 10 public inputs qualified.

Public query Main p50 / p95 (ms) NewParser p50 / p95 (ms)
ClickBench Q1 3.901 / 4.276 0.215 / 0.229
TPC-C warehouse tax 2.127 / 2.486 0.184 / 0.198
TPC-C Stock Level 4.085 / 4.403 0.375 / 0.416
KQP ManyJoinsWithDecimalRanges, 90 LEFT JOINs 176.719 / 184.713 16.098 / 16.504
Controlled extension of KQP ManyJoinsWithDecimalRanges, 96 LEFT JOINs 191.267 / 226.640 17.276 / 17.620
ClickBench Q30 59.602 / 63.727 4.678 / 4.941

These measurements used one Apple M3 Pro with 36 GiB RAM and macOS 26.6.2 arm64. The two executables ran sequentially. A cross-branch pair was excluded for the five-table join (7 versus 11 output columns) and for TPC-DS Q88 (different first output column). The main replay could not annotate DateTime2.GetYear for TPC-H Q8. Its forced YqlSelect translation rejected ONLY/SEMI for TPC-DS Q14.

Reproduction

./ya make --threads=5 --link-threads=1 --build relwithdebinfo -DNO_DEBUGINFO=yes -tA ydb/tests/tools/parsed_query_bench

The build also guards a GNU-only linker flag in the transitive Bison host tool on Darwin. Each public result uses the schema and query fixtures for the selected workload. Commands, pipeline and input formats

The public 96-join control uses the same SQL generator as the upstream 90-join test, with six more LEFT JOINs and six more tables. From the build checkout, run:

set -e
dir=$(mktemp -d)
bench=$PWD/ydb/tests/tools/parsed_query_bench/parsed_query_legacy_bench
curl -fsSL "https://raw.githubusercontent.com/asmyasnikov/ydb/ae1f7515d54ac171171695e563d44d2f55d0feac/ydb/tests/tools/parsed_query_bench/native/corpus_catalog.py" -o "$dir/generate90.py"
sed 's/91)/97)/g' "$dir/generate90.py" > "$dir/generate96.py"
cd "$dir"
python3 generate96.py many-joins --query-out query.yql > catalog.json
set -- text antlr_ast yql_ast kql_query_list
"$bench" --catalog catalog.json --export query.yql "$@"
"$bench" --catalog catalog.json --verify "$@"
"$bench" --catalog catalog.json --bench 10 100 "$@"
"$bench" --catalog catalog.json --client 10 100 query.yql "$@"

The complete text request has SHA-256 46c0af5998103d3da27a11ca528c6bf7436c1aecfccafc188de91eb8fb3ed9c2.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-19 03:53:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4d064c3 has started.
⚪ 2026-11-19 03:53:29 UTC Artifacts will be uploaded here
⚪ 2026-11-19 03:54:58 UTC ya make is running...
⚫ 2026-11-19 04:13:36 UTC Check cancelled

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-19 04:16:18 UTC Pre-commit check linux-x86_64-relwithdebinfo for dc5a1d4 has started.
⚪ 2026-11-19 04:16:40 UTC Artifacts will be uploaded here
⚪ 2026-11-19 04:18:10 UTC ya make is running...
🔴 2026-11-19 05:11:04 UTC Build failed, see the logs. Also see fail summary
🟡 2026-11-19 05:11:36 UTC ydbd size 2.9 GiB changed* by +30.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7c1bcec merge: dc5a1d4 diff diff %
ydbd size 3 045 473 272 Bytes 3 077 714 608 Bytes +30.7 MiB +1.059%
ydbd stripped size 643 672 192 Bytes 647 382 528 Bytes +3.5 MiB +0.576%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-19 05:26:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2ee4a5e has started.
⚪ 2026-11-19 05:26:58 UTC Artifacts will be uploaded here
⚪ 2026-11-19 05:28:27 UTC ya make is running...

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@asmyasnikov
asmyasnikov marked this pull request as ready for review November 19, 2026 05:43
@asmyasnikov
asmyasnikov requested review from a team as code owners November 19, 2026 05:43
Copilot AI balanced review requested due to automatic review settings November 19, 2026 05:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The compiler-resume APIs reconstruct state outside normal frontend setup, requiring a KQP compiler maintainer to validate imported-KQL context parity beyond what static inspection and covered fixtures establish.

No open issues 🎉

What changed in this PR

Adds an offline benchmark comparing four query representations through the legacy KQP frontend, stopping at logical-root construction in the new rule-based optimizer. It measures server compilation and producer serialization costs while checking request and logical-output equivalence.

Changes:

  • Adds text, grammar AST, translated YQL AST, and KQL expression-graph inputs, with timing, verification, and regression tests.
  • Exposes compiler entry points and callbacks for replaying and capturing intermediate stages.
  • Guards Bison’s GNU-only linker flag on Darwin.
File Description
ydb/​tests/​tools/​ya.make Registers the benchmark tool.
ydb/​tests/​tools/​parsed_query_bench/​ya.make Defines the executable and fingerprint generation.
ydb/​tests/​tools/​parsed_query_bench/​ut/​ya.make Registers Python integration tests.
ydb/​tests/​tools/​parsed_query_bench/​ut/​test_legacy.py Tests equivalence, timing output, and invalid inputs.
ydb/​tests/​tools/​parsed_query_bench/​syntax.h Declares syntax conversion interfaces.
ydb/​tests/​tools/​parsed_query_bench/​syntax.cpp Implements grammar and YQL AST conversion.
ydb/​tests/​tools/​parsed_query_bench/​README.md Documents stages, timing boundaries, and usage.
ydb/​tests/​tools/​parsed_query_bench/​proto/​ya.make Builds experimental protobuf schemas.
ydb/​tests/​tools/​parsed_query_bench/​proto/​parsed_query.proto Defines grammar and YQL AST payloads.
ydb/​tests/​tools/​parsed_query_bench/​proto/​kql_query_list.proto Defines serialized KQL expression graphs.
ydb/​tests/​tools/​parsed_query_bench/​proto/​execute_query.proto Defines the benchmark request envelope.
ydb/​tests/​tools/​parsed_query_bench/​measurement.h Provides bounded decoding and timing helpers.
ydb/​tests/​tools/​parsed_query_bench/​main.cpp Implements export, verification, and measurement commands.
ydb/​tests/​tools/​parsed_query_bench/​kql_codec.h Declares KQL graph serialization interfaces.
ydb/​tests/​tools/​parsed_query_bench/​kql_codec.cpp Encodes, restores, and validates KQL graphs.
ydb/​tests/​tools/​parsed_query_bench/​grammar_fingerprint.py Generates a parser-schema fingerprint.
ydb/​tests/​tools/​parsed_query_bench/​engine.h Defines benchmark engine and result interfaces.
ydb/​tests/​tools/​parsed_query_bench/​engine.cpp Implements offline catalog binding and compiler replay.
ydb/​core/​kqp/​tracing/​kqp_compile_trace.h Adds intermediate-stage capture callbacks.
ydb/​core/​kqp/​opt/​rbo/​kqp_rbo_transformer.cpp Captures logical roots and stops benchmark compilation.
ydb/​core/​kqp/​host/​kqp_runner.cpp Exposes optimizer entry and KQL capture boundaries.
ydb/​core/​kqp/​host/​kqp_host.h Declares the KQL inspection entry point.
ydb/​core/​kqp/​host/​kqp_host.cpp Supports imported inputs and translation timing callbacks.
ydb/​core/​kqp/​host/​kqp_host_impl.h Declares the runner’s new optimizer entry point.
ydb/​core/​kqp/​common/​simple/​query_ref.h Adds an optional translated-AST reference.
contrib/​tools/​bison/​ya.make Excludes the GNU linker flag on Darwin.
contrib/​tools/​bison/​.yandex_meta/​__init__.py Preserves the Darwin guard during regeneration.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants