Repository navigation
[VARKA-613] Leave residual-heavy partial projections to Spark for row consumers - #1619
Merged
Merged
Conversation
added 9 commits
November 20, 2026 09:47
…a row consumer A projection that fuses in part under a row consumer runs its kernel, then evaluates the residual entries per row in a merge projection outside whole-stage code. When its residual entries outnumber its fused ones, that plan is slower than Spark's own projection: the day-time baseline's four extracts, of which only the day fuses, read 0.7X at 256 bits and 0.5X at 128 on the EPYC 7763 runners. VarkaColumnarRule's post stage now asks VarkaPartialDecline in both of its row-consumer projection arms and, where it declines, plans Spark's ProjectExec over a plain ColumnarToRowExec. A columnar consumer keeps every partial projection, and a test's forced residual (forceResidualAt) is never declined. The rule was chosen by VarkaPartialProjectionProbe over the day-time baseline's table on the laptop, committed as sql/varka/plans/m8/results/VARKA-613-probe.txt: the rule by shared input columns declined a shape that wins and kept one that loses. The blend's class in VarkaDayTimeIntervalBaselineBenchmark is NONE, and VarkaPartialDeclineSuite holds the four probe shapes under a row and a columnar consumer at three emit levels. Generated-by: Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Opus 5.5)
…w path does The decline planned Spark's projection over a ColumnarToRowExec of the child, but Spark inserts no to-row transition under a row parent for a child that produces rows too, as the Arrow-cached scan does: the row path reads that cache as rows. The declined projection now reads its child as the transition rule would give it, a Varka filter through its to-row node (transformUp does not revisit what an arm returns, so a compacting filter would otherwise stay in the plan), and any other columnar child through a ColumnarToRowExec. The suite compares the declined plan with the Varka-off session's again and adds a shape over a fused filter. Generated-by: Claude Code (Claude Opus 5.5)
…ection The decline counted entries only, so one expensive fused entry beside two cheap residual ones lost its kernel. It now also requires the residual entries to outweigh the fused ones by operators. A declined projection's Varka projection child is asked the same question. Tests cover both, and the G34 case runs through a noop write. Generated-by: Claude Code (Claude Sonnet 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
…tests Generated-by: Claude Code (Claude Sonnet 5.5)
Author
|
Option matrix run: https://github.com/MaxGekk/spark/actions/runs/45019774322 (success) |
added 3 commits
November 22, 2026 12:12
Generated-by: Claude Code (Claude Sonnet 5.5)
…rojection Generated-by: Claude Code (Claude Sonnet 5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Varka can fuse a projection in part. The other entries, the residual entries, run one row at a time.
VarkaColumnarRulenow leaves a partly fused projection to Spark when the consumer wants rows and its residual entries are more in number and heavier (by operator count) than its fused entries.ProjectExecover the rows of its child.VarkaPartialDecline.reason. It logs its reason at DEBUG.How the decline works and how the rule was chosen
Varka can fuse a projection in part.
The entries that its compiler lowers run in a SIMD kernel over each Arrow batch. The other entries, the residual entries, run one row at a time.
When the consumer wants rows,
VarkaColumnarToRowExecruns the kernel and then builds each output row in a mergeUnsafeProjectionover aJoinedRowof the input row and the kernel row.That merge runs the residual entries per row, outside whole-stage code.
Spark's own plan runs every entry in one whole-stage loop over the vectors of the batch.
With this change, the post stage of
VarkaColumnarRuleleaves a projection whole to Spark when its residual entries are more in number and also heavier than its fused entries. The weight is the count of operators in the expression of each entry.The post stage is the only place that knows the consumer wants rows.
The new plan is Spark's own
ProjectExec, over the rows of its child:InMemoryTableScanExecmakes rows and batches, so Spark adds noColumnarToRowunder a row parent. The declined plan is then the plan of a Varka-off session, operator for operator:*(1) ProjectoverScan In-memory table.VarkaFilterColumnarToRowExec. It reads the selection bitmap and does not compact. A row consumer of the filter alone gets the same node. The code builds it inside the declining arm, becausetransformUpdoes not visit again what an arm returns.ColumnarToRowExec.The rule and its use:
VarkaPartialDecline.reason, a small Java method over the sealed per-entry classification of the compiler. It counts fused entries (from every kernel) against residual entries, and weighs each by the operators of its expression. Forwarded columns count as neither. It declines only when the residual entries are more in number and in weight. It returns the reason, or nothing.ColumnarToRowExec(VarkaProjectExec)plans Spark's projection on a decline.ProjectExec ... if isVarkaEligibleleaves the projection as it is on a decline, so a declined plan that reaches the rule again stays Spark's projection.EXPLAIN.supportsColumnarWrite, for projections that fuse whole, and for partial projections that have no more residual entries than fused entries.forceResidualAtis set, nothing is declined. The injector forces an entry to be residual, so that the Spark differential and the composition fuzzer test the merge path. A decline would remove that path.How I chose the rule: R1 declines when every column that the fused entries read is also read by a residual entry. R2 declines when the residual entries outnumber the fused entries.
The probe
VarkaPartialProjectionProbetimed four shapes over the 2,000,000 Arrow-cached rows of the day-time baseline, on the laptop under-XX:UseAVX=2at 256 and 128 bits (output:sql/varka/plans/m8/results/VARKA-613-probe.txt). The shapes:DAYfused besideHOUR,MINUTEandSECONDof the same column.extract(HOUR)of their column.make_dt_interval(i, i2, i3).The blend section of
VarkaDayTimeIntervalBaselineBenchmarkis nowNONEat both densities, and its queries suite lists it so.x9 < y9, -x38 < y38inVarkaDecimalBaselineQueriesSuitehas one fused and one residual entry, so it staysPARTIAL.The laptop had a load of 30 to 60. I read a pass only when its control,
extract(DAY FROM dt)alone, wins as on the runners. Only the passes with one row in ten null did.Best times in ms, row path / Varka on:
R1 would decline c, which wins, and keep b, which loses at 128 bits.
R2 declines a and b, and keeps c and d.
R2 fits every counted reading, and also the committed loss of the blend at both widths on the runners.
VARKA-613.mdsection 9 has the full reading and the scored predictions.Why are the changes needed?
A fused shape must not be slower than the row path.
The day-time baseline
extract DAY, HOUR, MINUTE and SECOND togetherfuses only theDAYfield.On the EPYC 7763 runners it took 66.1 ns a row against 47.2 for the row path at 256 bits, and 94.4 against 47.2 at 128 bits.
The fused entry adds a batch pass and a merge for each row.
More on the reason
A fused shape must not be slower than the row path.
The day-time baseline
extract DAY, HOUR, MINUTE and SECOND togetherfuses only theDAYfield.On the EPYC 7763 runners it took 66.1 ns a row against 47.2 for the row path at 256 bits, and 94.4 against 47.2 at 128 bits (
VarkaDayTimeIntervalBaselineBenchmark-jdk25-runner-epyc7763*-results.txt).extract(DAY FROM dt)alone wins 3.9X, so the kernel is fast. The loss comes from the partial plan: one fused entry saves one extract for each row, but it adds a batch pass and the indirection of the merge for each row.In milestone 2, a residual-heavy projection through a row consumer was measured at 0.7x for the same reason.
Fusing the other three extracts would also fix the blend, but that work depends on three other rows that are not close to merging.
Does this PR introduce any user-facing change?
Yes, in plans only: such a projection is now Spark's
ProjectoverColumnarToRow. Answers do not change. Varka is not in any released Spark.More on the user-facing change
With
spark.sql.codegen.varka.enabled, a projection under a row consumer whose residual entries outnumber its fused entries is now planned as Spark'sProjectoverColumnarToRow, not asVarkaColumnarToRow.Answers do not change.
How was this patch tested?
On the laptop, on the head after the merge of
origin/master(6f6c9db), with the default emit options:--lint), Error Prone and both lints passed.dev/varka_matrix.sh --defaultsover 34 suites: 456 tests, 0 failed. Its first run failed 4 tests. I fixed them.VarkaPartialDeclineSuiteruns atuseAVX-1, 2 and 3.9b4a82786dc: the PR Build (run 45016277036) and the full option matrix (run 45019774322; 174 jobs passed, 1 skipped, 0 failed) are green.Full test record, suite checks and the list of what was not run
All runs are on the laptop, on this branch after the merge of
origin/master(6f6c9db), with the default emit options, each under the shared heavy-job lock.dev/varka_regen_generated.sh --allran to the end and changed no file. So no generated table records the plan class that this change moves.dev/varka_push_check.pypassed: precommit range check, census check, size check, and its matrix ofVarkaCastMatrixSuite,VarkaCoverageSuite,VarkaEmittedBytesSuite,VarkaFuzzTypesSuite,VarkaTypesSuiteandVarkaPartialDeclineSuite.dev/varka_errorprone.sh,dev/lint-scalaanddev/lint-javapassed.dev/varka_push_check.py --lintpassed after the changes of the review round: precommit, plan lint, quote check, census, size (800 counted lines), Error Prone, scalastyle, checkstyle, and its matrix (46 ok, 0 failed, 3 canceled by the suites themselves).dev/varka_matrix.sh --defaultsover 34 suites, one JVM, under the heavy lock: 456 tests, 0 failed. The suites areVarkaPartialDeclineSuite,VarkaDayTimeIntervalBaselineQueriesSuite,VarkaDecimalBaselineQueriesSuite, every other*BaselineQueriesSuite,VarkaEndToEndSuite,VarkaSparkFuzzSuite,VarkaFloatingLaneSuite,VarkaGoldenCorpusSuite,VarkaDifferentialSuite,VarkaColumnarWriteSuite,VarkaProjectExecSuite,VarkaColumnarToRowExecSuite,VarkaCodegenGiveUpSuite, and the other suites that nameVarkaProjectExec,VarkaColumnarToRowExecorPARTIAL.VarkaDecimalBaselineQueriesSuite: a rule that only weighed declined the decimalPARTIALshape. I fixed it by requiring the count as well. One wasVarkaCodegenGiveUpSuiteG34: its projection with one fused and sixty residual entries is now declined under a row consumer, so the test runs its measured windows through anoopwrite.What
VarkaPartialDeclineSuitechecks, atuseAVX-1, 2 and 3:VarkaColumnarToRowExec.WHERE dt > dt2) plans Spark's projection over the to-row node of the filter, with no compacting filter left.noopwrite, every shape keepsVarkaProjectExec.forceResidualAtexemption.Not run:
varka-option-matrix); it later passed (see above). The rule readsVarkaEmitOptions.forceResidualAt, and the configurations of the matrix should not move it.VarkaPartialDeclineSuiteruns atuseAVX-1, 2 and 3 on the laptop, which is AVX-512. It expects shapes c and d to keep their kernel at every level. If an AVX2 runner declines one of their entries, c would have one fused entry against three residual entries and be declined, and the suite would fail there.VarkaDayTimeIntervalBaselineBenchmarkat 256 and 128 bits.VarkaPartialProjectionProbethere, to confirm R2 on a quiet machine or to show that it is too narrow. The rule rests on the laptop probe at a load of 30 to 60, so R2 is provisional until that run. The blend declines under every candidate rule, so the reading of the baseline does not depend on the probe.Test time: the new suite and the sql/core Varka suites above ran in 2 min 44 s together. The new suite alone is about 40 s of that on the laptop, by an earlier run of the author.
Known trade-offs:
ProjectExec ... if isVarkaEligiblecompiles the projection a second time to ask for the decline. The arm handles projections that the pre stage did not see. The compile runs once for each plan and has no side effects.projectionToRows). A fused filter child becomes its to-row node.VarkaPartialProjectionProbeshows no entry lines for them. To read the split, you need the pre-stage plan.VarkaSparkDifferential.forcedDifferencereads its residual count from the unforced plan, which the decline may already have made a plain projection. Such a run counts as not applied, never as passed. The applied-share check of the fuzz suite passed here.Review
/code-review high(second run): 3 important findings, all fixed. The first high run raised 5 important findings and 4 nits. I fixed or accepted all 5 and left the 4 nits.Review history and the nit list
/code-review highran a first time on the branch. It raised five important findings and four nits. I fixed or accepted all five important findings (see the trade-offs). I left the four nits./code-review mediumthen ran on the merged branch. It found no important finding.A second
/code-review highfound three important findings. I fixed all three in that round:VARKA-613.md9.5 records them. They found two real consequences: the decimalPARTIALshape and the G34 test.abs(-dt * 3)beside two residual extracts. Plan section 9.2 says why this is the smaller design.declinedChildnever checked a nested Varka projection again. It stayed a kernel under a plainColumnarToRowExec. The arm anddeclinedChildnow shareprojectionToRows, which asks the rule of the inner projection too. A test covers two stacked declined projections.The five nits of the second review are left as they are:
dev/varka_bench_regen.shregenerates them.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Sonnet 5.5)