Skip to content

[VARKA-613] Leave residual-heavy partial projections to Spark for row consumers - #1619

Merged
MaxGekk merged 12 commits into
vecbricks:masterfrom
MaxGekk:varka-613-partial-projection
Nov 22, 2026
Merged

MaxGekk merged 12 commits into
vecbricks:masterfrom
MaxGekk:varka-613-partial-projection

Conversation

@MaxGekk

@MaxGekk MaxGekk commented Nov 22, 2026 •

Copy link
Copy Markdown

What changes were proposed in this pull request?

Varka can fuse a projection in part. The other entries, the residual entries, run one row at a time.

  • VarkaColumnarRule now leaves a partly fused projection to Spark when the consumer wants rows and its residual entries are more in number and heavier (by operator count) than its fused entries.
  • The new plan is Spark's own ProjectExec over the rows of its child.
  • The rule is VarkaPartialDecline.reason. It logs its reason at DEBUG.
  • A kernel stays for a columnar consumer and for other projections.
  • The rule rests on a laptop probe, so its choice (R2) is provisional.
How the decline works and how the rule was chosen

Varka can fuse a projection in part.
The entries that its compiler lowers run in a SIMD kernel over each Arrow batch. The other entries, the residual entries, run one row at a time.
When the consumer wants rows, VarkaColumnarToRowExec runs the kernel and then builds each output row in a merge UnsafeProjection over a JoinedRow of the input row and the kernel row.
That merge runs the residual entries per row, outside whole-stage code.
Spark's own plan runs every entry in one whole-stage loop over the vectors of the batch.

With this change, the post stage of VarkaColumnarRule leaves a projection whole to Spark when its residual entries are more in number and also heavier than its fused entries. The weight is the count of operators in the expression of each entry.
The post stage is the only place that knows the consumer wants rows.
The new plan is Spark's own ProjectExec, over the rows of its child:

  • A child that makes rows itself stays as it is. The Arrow-cached InMemoryTableScanExec makes rows and batches, so Spark adds no ColumnarToRow under a row parent. The declined plan is then the plan of a Varka-off session, operator for operator: *(1) Project over Scan In-memory table.
  • A fused Varka filter becomes its to-row node, VarkaFilterColumnarToRowExec. It reads the selection bitmap and does not compact. A row consumer of the filter alone gets the same node. The code builds it inside the declining arm, because transformUp does not visit again what an arm returns.
  • Any other columnar child gets a ColumnarToRowExec.

The rule and its use:

  • The rule is VarkaPartialDecline.reason, a small Java method over the sealed per-entry classification of the compiler. It counts fused entries (from every kernel) against residual entries, and weighs each by the operators of its expression. Forwarded columns count as neither. It declines only when the residual entries are more in number and in weight. It returns the reason, or nothing.
  • Both row-consumer projection arms of the post stage ask it. ColumnarToRowExec(VarkaProjectExec) plans Spark's projection on a decline. ProjectExec ... if isVarkaEligible leaves the projection as it is on a decline, so a declined plan that reaches the rule again stays Spark's projection.
  • The reason goes to the DEBUG log. A declined plan has no Varka node to show it in EXPLAIN.
  • A kernel stays for a columnar consumer, such as a DSv2 write with supportsColumnarWrite, for projections that fuse whole, and for partial projections that have no more residual entries than fused entries.
  • The test fault injector. While forceResidualAt is set, nothing is declined. The injector forces an entry to be residual, so that the Spark differential and the composition fuzzer test the merge path. A decline would remove that path.

How I chose the rule: R1 declines when every column that the fused entries read is also read by a residual entry. R2 declines when the residual entries outnumber the fused entries.
The probe VarkaPartialProjectionProbe timed four shapes over the 2,000,000 Arrow-cached rows of the day-time baseline, on the laptop under -XX:UseAVX=2 at 256 and 128 bits (output: sql/varka/plans/m8/results/VARKA-613-probe.txt). The shapes:

  • a: DAY fused beside HOUR, MINUTE and SECOND of the same column.
  • b: the same, with the three over another column.
  • c: three fused arithmetic entries beside a residual extract(HOUR) of their column.
  • d: the same three beside a residual make_dt_interval(i, i2, i3).

The blend section of VarkaDayTimeIntervalBaselineBenchmark is now NONE at both densities, and its queries suite lists it so. x9 < y9, -x38 < y38 in VarkaDecimalBaselineQueriesSuite has one fused and one residual entry, so it stays PARTIAL.

The laptop had a load of 30 to 60. I read a pass only when its control, extract(DAY FROM dt) alone, wins as on the runners. Only the passes with one row in ten null did.
Best times in ms, row path / Varka on:

shape run 1, 256 bits run 2, 256 bits run 1, 128 bits
a 103 / 153 395 / 426 305 / 394
b 128 / 136 289 / 351 281 / 402
c 118 / 81 328 / 296 not read (checked interval arithmetic loses at 128 bits on its own)
d 285 / 183 464 / 515 not read (same)

R1 would decline c, which wins, and keep b, which loses at 128 bits.
R2 declines a and b, and keeps c and d.
R2 fits every counted reading, and also the committed loss of the blend at both widths on the runners.
VARKA-613.md section 9 has the full reading and the scored predictions.

Why are the changes needed?

A fused shape must not be slower than the row path.
The day-time baseline extract DAY, HOUR, MINUTE and SECOND together fuses only the DAY field.
On the EPYC 7763 runners it took 66.1 ns a row against 47.2 for the row path at 256 bits, and 94.4 against 47.2 at 128 bits.
The fused entry adds a batch pass and a merge for each row.

More on the reason

A fused shape must not be slower than the row path.
The day-time baseline extract DAY, HOUR, MINUTE and SECOND together fuses only the DAY field.
On the EPYC 7763 runners it took 66.1 ns a row against 47.2 for the row path at 256 bits, and 94.4 against 47.2 at 128 bits (VarkaDayTimeIntervalBaselineBenchmark-jdk25-runner-epyc7763*-results.txt).
extract(DAY FROM dt) alone wins 3.9X, so the kernel is fast. The loss comes from the partial plan: one fused entry saves one extract for each row, but it adds a batch pass and the indirection of the merge for each row.
In milestone 2, a residual-heavy projection through a row consumer was measured at 0.7x for the same reason.
Fusing the other three extracts would also fix the blend, but that work depends on three other rows that are not close to merging.

Does this PR introduce any user-facing change?

Yes, in plans only: such a projection is now Spark's Project over ColumnarToRow. Answers do not change. Varka is not in any released Spark.

More on the user-facing change

With spark.sql.codegen.varka.enabled, a projection under a row consumer whose residual entries outnumber its fused entries is now planned as Spark's Project over ColumnarToRow, not as VarkaColumnarToRow.
Answers do not change.

How was this patch tested?

On the laptop, on the head after the merge of origin/master (6f6c9db), with the default emit options:

  • Push check (also --lint), Error Prone and both lints passed.
  • dev/varka_matrix.sh --defaults over 34 suites: 456 tests, 0 failed. Its first run failed 4 tests. I fixed them.
  • New VarkaPartialDeclineSuite runs at useAVX -1, 2 and 3.
  • On CI, on the final head 9b4a82786dc: the PR Build (run 45016277036) and the full option matrix (run 45019774322; 174 jobs passed, 1 skipped, 0 failed) are green.
  • Not run: any AVX2 machine, the benchmark dry runs and runner measurements. Known trade-offs are in the block below.
Full test record, suite checks and the list of what was not run

All runs are on the laptop, on this branch after the merge of origin/master (6f6c9db), with the default emit options, each under the shared heavy-job lock.

  • dev/varka_regen_generated.sh --all ran to the end and changed no file. So no generated table records the plan class that this change moves.
  • dev/varka_push_check.py passed: precommit range check, census check, size check, and its matrix of VarkaCastMatrixSuite, VarkaCoverageSuite, VarkaEmittedBytesSuite, VarkaFuzzTypesSuite, VarkaTypesSuite and VarkaPartialDeclineSuite.
  • dev/varka_errorprone.sh, dev/lint-scala and dev/lint-java passed.
  • dev/varka_push_check.py --lint passed after the changes of the review round: precommit, plan lint, quote check, census, size (800 counted lines), Error Prone, scalastyle, checkstyle, and its matrix (46 ok, 0 failed, 3 canceled by the suites themselves).
  • dev/varka_matrix.sh --defaults over 34 suites, one JVM, under the heavy lock: 456 tests, 0 failed. The suites are VarkaPartialDeclineSuite, VarkaDayTimeIntervalBaselineQueriesSuite, VarkaDecimalBaselineQueriesSuite, every other *BaselineQueriesSuite, VarkaEndToEndSuite, VarkaSparkFuzzSuite, VarkaFloatingLaneSuite, VarkaGoldenCorpusSuite, VarkaDifferentialSuite, VarkaColumnarWriteSuite, VarkaProjectExecSuite, VarkaColumnarToRowExecSuite, VarkaCodegenGiveUpSuite, and the other suites that name VarkaProjectExec, VarkaColumnarToRowExec or PARTIAL.
  • The first run of that matrix failed four tests. Three were in VarkaDecimalBaselineQueriesSuite: a rule that only weighed declined the decimal PARTIAL shape. I fixed it by requiring the count as well. One was VarkaCodegenGiveUpSuite G34: its projection with one fused and sixty residual entries is now declined under a row consumer, so the test runs its measured windows through a noop write.
  • Size: 800 counted lines, under the cap of 1000.

What VarkaPartialDeclineSuite checks, at useAVX -1, 2 and 3:

  • Under a row consumer, shapes a and b plan Spark's projection, operator for operator the plan of the Varka-off session. Shapes c and d keep VarkaColumnarToRowExec.
  • The blend over a fused filter (WHERE dt > dt2) plans Spark's projection over the to-row node of the filter, with no compacting filter left.
  • Under a noop write, every shape keeps VarkaProjectExec.
  • One expensive fused entry beside two cheap residual entries keeps its kernel.
  • A declined projection over a declined projection leaves both to Spark.
  • A declined shape that goes through the post stage twice stays Spark's projection.
  • A whole-fused projection is unchanged.
  • The reason of the rule, and the forceResidualAt exemption.

Not run:

  • Before the CI run, the option matrix had not run on the branch (varka-option-matrix); it later passed (see above). The rule reads VarkaEmitOptions.forceResidualAt, and the configurations of the matrix should not move it.
  • The suites on an AVX2 machine. VarkaPartialDeclineSuite runs at useAVX -1, 2 and 3 on the laptop, which is AVX-512. It expects shapes c and d to keep their kernel at every level. If an AVX2 runner declines one of their entries, c would have one fused entry against three residual entries and be declined, and the suite would fail there.
  • The plan-only dry runs of VarkaDayTimeIntervalBaselineBenchmark at 256 and 128 bits.
  • Runner measurements. One is the day-time baseline on the EPYC 7763 at both widths. With this change its blend section should read 1.0X within 5%, because both arms now plan alike. The other is VarkaPartialProjectionProbe there, to confirm R2 on a quiet machine or to show that it is too narrow. The rule rests on the laptop probe at a load of 30 to 60, so R2 is provisional until that run. The blend declines under every candidate rule, so the reading of the baseline does not depend on the probe.

Test time: the new suite and the sql/core Varka suites above ran in 2 min 44 s together. The new suite alone is about 40 s of that on the laptop, by an earlier run of the author.

Known trade-offs:

  • The post-stage arm ProjectExec ... if isVarkaEligible compiles the projection a second time to ask for the decline. The arm handles projections that the pre stage did not see. The compile runs once for each plan and has no side effects.
  • The decline gives up the kernel batches where a later consumer turns the plan columnar again. One example is the Arrow cache builder that reads a cached view whose top is a declined projection.
  • The two decline sites read their emit options from different places: the frozen options of the pre-stage node, and the session conf at the post stage. They agree while the conf does not change between the two passes.
  • A declined projection over another Varka projection asks the rule of that projection too (projectionToRows). A fused filter child becomes its to-row node.
  • The weight is the operator count of the expression of an entry. Only the probe shapes calibrate this proxy. The runner probe is where to widen it.
  • With the decline on, shapes a and b are planned as Spark's projection. So the per-entry description of VarkaPartialProjectionProbe shows no entry lines for them. To read the split, you need the pre-stage plan.
  • VarkaSparkDifferential.forcedDifference reads its residual count from the unforced plan, which the decline may already have made a plain projection. Such a run counts as not applied, never as passed. The applied-share check of the fuzz suite passed here.

Review

/code-review high (second run): 3 important findings, all fixed. The first high run raised 5 important findings and 4 nits. I fixed or accepted all 5 and left the 4 nits.

Review history and the nit list

/code-review high ran a first time on the branch. It raised five important findings and four nits. I fixed or accepted all five important findings (see the trade-offs). I left the four nits.
/code-review medium then ran on the merged branch. It found no important finding.

A second /code-review high found three important findings. I fixed all three in that round:

  1. The final shape of the decline had never run in full. The owed suites ran (456 tests, 0 failed), and VARKA-613.md 9.5 records them. They found two real consequences: the decimal PARTIAL shape and the G34 test.
  2. The rule declined on entry counts alone, so one expensive fused entry beside two cheap residual entries lost its kernel. The rule now also weighs each entry by the operators of its expression. It declines only when the residual entries are more in number and in weight, so it declines no shape that the count kept. A new test keeps abs(-dt * 3) beside two residual extracts. Plan section 9.2 says why this is the smaller design.
  3. declinedChild never checked a nested Varka projection again. It stayed a kernel under a plain ColumnarToRowExec. The arm and declinedChild now share projectionToRows, which asks the rule of the inner projection too. A test covers two stacked declined projections.

The five nits of the second review are left as they are:

  • The post-stage arm compiles the partial projection once more.
  • The two arms use different emit options.
  • The probe cannot measure shapes a and b after the decline.
  • A decline leaves no trace that a user can see.
  • The committed day-time results files still say the blend fuses in part. dev/varka_bench_regen.sh regenerates them.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Sonnet 5.5)

t added 9 commits November 20, 2026 09:47
…a row consumer

A projection that fuses in part under a row consumer runs its kernel, then
evaluates the residual entries per row in a merge projection outside
whole-stage code. When its residual entries outnumber its fused ones, that
plan is slower than Spark's own projection: the day-time baseline's four
extracts, of which only the day fuses, read 0.7X at 256 bits and 0.5X at
128 on the EPYC 7763 runners.

VarkaColumnarRule's post stage now asks VarkaPartialDecline in both of its
row-consumer projection arms and, where it declines, plans Spark's
ProjectExec over a plain ColumnarToRowExec. A columnar consumer keeps every
partial projection, and a test's forced residual (forceResidualAt) is never
declined. The rule was chosen by VarkaPartialProjectionProbe over the
day-time baseline's table on the laptop, committed as
sql/varka/plans/m8/results/VARKA-613-probe.txt: the rule by shared input
columns declined a shape that wins and kept one that loses.

The blend's class in VarkaDayTimeIntervalBaselineBenchmark is NONE, and
VarkaPartialDeclineSuite holds the four probe shapes under a row and a
columnar consumer at three emit levels.

Generated-by: Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Opus 5.5)
…w path does

The decline planned Spark's projection over a ColumnarToRowExec of the
child, but Spark inserts no to-row transition under a row parent for a
child that produces rows too, as the Arrow-cached scan does: the row path
reads that cache as rows. The declined projection now reads its child
as the transition rule would give it, a Varka filter through its to-row
node (transformUp does not revisit what an arm returns, so a compacting
filter would otherwise stay in the plan), and any other columnar child
through a ColumnarToRowExec. The suite compares the declined plan with
the Varka-off session's again and adds a shape over a fused filter.

Generated-by: Claude Code (Claude Opus 5.5)
…ection

The decline counted entries only, so one expensive fused entry beside two cheap residual
ones lost its kernel. It now also requires the residual entries to outweigh the fused ones
by operators. A declined projection's Varka projection child is asked the same question.
Tests cover both, and the G34 case runs through a noop write.

Generated-by: Claude Code (Claude Sonnet 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
…tests

Generated-by: Claude Code (Claude Sonnet 5.5)
@MaxGekk

MaxGekk commented Nov 22, 2026 •

Copy link
Copy Markdown
Author

Option matrix run: https://github.com/MaxGekk/spark/actions/runs/45019774322 (success)

@MaxGekk
MaxGekk merged commit b4cbee9 into vecbricks:master Nov 22, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant