Skip to content

[VARKA-348] Fuse every 64-bit decimal value as a projection output - #1628

Open
MaxGekk wants to merge 17 commits into
vecbricks:masterfrom
MaxGekk:varka-348-decimal-compare-r
Open

MaxGekk wants to merge 17 commits into
vecbricks:masterfrom
MaxGekk:varka-348-decimal-compare-r

Conversation

@MaxGekk

@MaxGekk MaxGekk commented Nov 22, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

A decimal of precision 18 or less is an unscaled long. Varka already fuses comparisons over such decimals. A decimal value as a projection output (if(c, a, b), coalesce(a, b), abs(a), a sum) was declined, so the row engine ran it.

  • VarkaDecimalCompiler.output compiles any value() and wraps it in WidenDecimal. This node stores a long as an Arrow decimal slot.
  • The census marks fifteen value entries as dec^ (fused in part).
  • coverage.json gains the family "DECIMAL values". decimal_refinement.smt2 proves each one against Spark's specification.
  • New suite VarkaDecimalOutputSuite checks each value against Spark's eval.
  • The decimal baseline benchmark marks twelve value sections as fused.
Design details, the two repairs and the repository lessons

VarkaDecimalCompiler.output checks the output's type as before, then compiles any value() and wraps it in WidenDecimal, the node that stores a long as a decimal slot of the output's type.
The call is bracketed by enterDecimal and exitDecimal, so the cached choice conditions are dropped before an entry's tables can roll back.
The two decimal-valued TIME functions become decimal operands through a value() arm instead of a special case.

The census marks dec^ on fifteen value entries, among them the choices, coalesce, nvl, nullif, abs, the negation, the greatest and the arithmetic operators.
least and try_subtract are listed as owed until coverage.json holds their rows. The pair's conformance suite lists them in declinesInPart.

coverage.json gains the family "DECIMAL values": a CASE, an IF, coalesce, greatest, least, a negation, abs and try_subtract over decimals within 18 digits.
decimal_refinement.smt2 renders each of them.
Its generator now compiles a decimal-valued row as a projection's one output. It refines the WidenDecimal root against the same Spark specification that a predicate row has.

VarkaDecimalOutputSuite runs each value under both decimalStore arms at one, two, four and eight long lanes, with nulls and the ends of the precision.
It compares with Spark's eval and reads back what ArrowColumnVector holds.
It also covers abs under both lowerings, sums and differences, a constant and a typed null beside a column's value, and the decline past 18 digits.

The decimal baseline benchmark marks twelve value sections as fused whole in both sessions: the choice at precisions 9 and 18, the cast from DECIMAL(9, 2) to DECIMAL(18, 4), and the three arithmetic projections at precision 9, each at both null densities.
A value root is no condition, so the two-lane condition refusal does not take them.

Two repository lessons are added to skills/working-in-this-repo.md:

  • The census check refuses a claim that coverage.json cannot show, until the coverage suite regenerates it.
  • The quote check reads a two-digit section number as a measurement.

Two repairs ride with the change:

  • The conformance harness's decimal slot lane refused a WidenDecimal root. So VarkaKernelChecks now writes, sizes and reads a decimal store output as a slot. VarkaTypeConformance.overflowReachable lets a suite say that the sum, difference, negation and abs over DECIMAL(10, 2) cannot overflow.
  • The refinement printer now gives a WidenDecimal root's data as a signed value, because the specification's decimal value is signed.

The value outputs of the 128-bit decimal pair still decline. They need a two-word store, which is separate work not yet done. So VarkaDecimalConformanceSuite keeps its declinesInPart list.
The long suite's overflow check stays cancelled by the owed x * y, a DECIMAL(21, 4) output of that same work.

Why are the changes needed?

A projection that returns a decimal is as common as one that compares it. Until then, the decimal baseline's value sections can only run on the row engine.

Does this PR introduce any user-facing change?

No new setting. Projections that return a decimal of precision 18 or less through these forms now fuse, with Spark's answers.

How was this patch tested?

On the laptop, on head a2061571f1e: the branch merged with master e982a4dbd32, with no conflict. Master then moved to b4cbee926d1. I merged it with no conflict. The merge left sql/varka/emitted_bytes.json stale, so I regenerated it (final head f80de834aaf). On that head the push check and extra-gate.sh pass. The other gates ran on a2061571f1e only.

  • Push check, both lints and the proof suites passed.
  • Catalyst suites: 215 passed, 8 cancelled, 0 failed. The sql baseline suite: 11 passed.
  • decimal_refinement.smt2 holds under Z3 and cvc5.
  • cvc5 timed out on range_calendar.smt2. This PR does not touch it. It passes alone.
  • Not rerun after the merge: Error Prone, the full regeneration, the proof regeneration and the width audit.
  • Not run: any benchmark, CI Build, option matrix, and the audit under UseAVX=2.
Full test record and what was not run

On the laptop (AVX-512, 256-bit datapath), one sbt at a time, SPARK_LOCAL_IP=127.0.0.1, on 22 November 2026 at head a2061571f1e, which is the branch merged with origin/master at e982a4dbd32 (the merge had no conflict).
Passed there:

  • dev/varka_regen_generated.sh --all. It left nothing to commit.
  • python3 dev/varka_push_check.py.
  • dev/lint-java, dev/lint-scala, and catalyst/testOnly *VarkaInternalErrorSuite *VarkaProofFilesSuite.
  • build/sbt 'catalyst/testOnly *VarkaCoverageSuite *VarkaDecimalLongConformanceSuite *VarkaDecimalConformanceSuite *VarkaDecimalOutputSuite *VarkaExpressionCompilerSuite *VarkaFamilySwitchSuite': 215 passed, 8 cancelled, 0 failed.
  • build/sbt 'sql/testOnly *VarkaDecimalBaselineQueriesSuite': 11 passed.
  • dev/varka_prove.sh --solver both. decimal_refinement.smt2 holds under Z3 and cvc5 with 370 checks.

In that run cvc5 timed out on range_calendar.smt2, a file this pull request does not touch, beside other load.
The file alone, and dev/varka_prove.sh --lint, then hold under both solvers.

Run on the earlier head fcb8202ff9c and not repeated after the merge:

  • dev/varka_regen_generated.sh --all --full.
  • dev/varka_errorprone.sh.
  • VARKA_PROOFS_REGEN=true for decimal_refinement.smt2.
  • The width audit. It took about five minutes on a loaded laptop, close to its 600 s budget.

The merge changed none of the files this pull request changes. The regeneration above found the generated files unchanged.
No option-matrix configuration exists for the decimal store arm. VarkaDecimalOutputSuite runs both decimalStore arms itself.

Not run: the benchmark on any host (the runners' readings are owed), a CI Build or option matrix, and the width audit under UseAVX=2.

Review

/code-review high found no important defect. It ran before the merge, and I did not rerun it. The merge had no conflict and the patch id is the same. I could not confirm the SHA it reviewed. Five nits are open. The size is 925 counted lines.

Earlier reviews and the nit list

/code-review high ran on varka-348-decimal-compare-r before the merge of origin/master e982a4dbd32 (merge base 96f7ff5402b) and found no important defect.
Two earlier /code-review medium runs had found five important items, all fixed or resolved.
The merge was conflict-free. The hand-written diff (catalyst, core, docs) has the same patch id before and after it, so the review was not repeated.
The size is 925 counted lines of 2197 changed, under the cap of 1000.

Five nits are left as they are:

  • A null test over a decimal TIME producer's value (time_to_seconds(t) IS NULL) passes value() and then declines in valid() for want of an arm.
  • widened now declines where it threw on a scale or precision mismatch. A future arm's bug would show as a quiet fall back to the row engine.
  • The refinement generator no longer refuses a predicate row with a store root, or a decimal row without one.
  • The whole-root store test of VarkaDecimalOutputSuite runs five of its shapes through ArrowColumnVector and the rest only on the long under the store.
  • The contract that each value arm guards its own overflow is stated in a comment only.

The one markdown line in working-in-this-repo.md over 100 columns is left.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Sonnet 5.5)

t added 17 commits November 21, 2026 23:46
…ue compiler

VarkaDecimalCompiler.output checks the output's type as before, then compiles
any value() and wraps it in WidenDecimal. The call is bracketed by
enterDecimal/exitDecimal so the cached choice conditions are dropped before an
entry's tables can roll back. As a result, every decimal value within 18
digits is a projection output: rescaling casts, negation, abs, the choices,
the picks, and 349's sums, differences and products. The two decimal-valued
TIME functions become decimal operands through a value() arm.

The census marks dec^ on fifteen value entries; least and try_subtract wait
for coverage.json to hold their rows. The pair's
conformance suite lists them in declinesInPart. VarkaExpressionCompilerSuite's
test of the TIME producers below a root now checks that they fuse there.

Generated-by: Claude Code (Claude Opus 5.5)
…and suite

The coverage family "DECIMAL values" adds eight projection rows: a CASE, an
IF, coalesce, greatest, least, a negation, abs and try_subtract over decimals
within 18 digits. Each row is rendered by decimal_refinement.smt2. That file's
generator now compiles a decimal-valued row as a projection's one output and
refines its WidenDecimal root against the same Spark specification as a
predicate row's.

VarkaDecimalOutputSuite runs each value under the store at one, two, four and
eight long lanes, with nulls and the precision's ends, against Spark's eval.
It covers abs under both lowerings, the sums and differences, and the decline
past 18 digits. VARKA-348.md section 10, Step 5, records the part and the decisions taken
while the owner was away, and PLAN.md row 348 records its state.

Generated-by: Claude Code (Claude Opus 5.5)
…uite

VarkaDecimalOutputSuite now runs the whole root, WidenDecimal over the
decimal columns' leaves, into a DecimalVector of the output's type under both
decimalStore arms at every width, and holds what ArrowColumnVector reads back
to Spark's eval. A second new test fuses a constant and a typed null as
outputs beside a column's value. VARKA-348.md scores the first run, which
passed, and records why the laptop's abs reading waits.

Generated-by: Claude Code (Claude Opus 5.5)
VarkaDecimalBaselineBenchmark checks each section's plan against its fusion
class before timing. With every 64-bit decimal value now an output through
the decimal store, twelve sections fuse whole in both sessions: the choice at
precisions 9 and 18, the cast from DECIMAL(9, 2) to DECIMAL(18, 4), and the
three arithmetic projections at precision 9, each at both null densities. A
value root is no condition, so row 503's two-lane refusal does not take them.
VARKA-348.md and PLAN.md row 348 record the change.

Generated-by: Claude Code (Claude Opus 5.5)
Two lessons from building 348b go into working-in-this-repo.md, and SKILLS.md
is regenerated:
- the census pre-commit check refuses a claim that coverage.json cannot show
  until the coverage suite regenerates it;
- the quote check reads a two-digit section number as a measurement.

VarkaDecimalOutputSuite gains the bare rescaling cast as an output root.
VARKA-348.md names the measurement rule that kept the laptop's abs reading
out of this window.

Generated-by: Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
…type, trim the expected declines

The review found the census without least and try_subtract marked although coverage.json now
holds their rows, the pair's conformance suite not listing them among the forms that decline
in part, the expected-decline map still naming the TIME decimal producers that now fuse as
operands, and widened throwing where a value's type differs from the output's, which now
declines with both types named.

Generated-by: Claude Code (Claude Sonnet 5.5)
…values by their signed reading

The conformance harness's decimal slot lane refused a WidenDecimal root, and the
decimal refinement compared a value root's unsigned printed word with the
specification's signed value.

Generated-by: Claude Code (Claude Sonnet 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
…s for the value rows

Generated-by: Claude Code (Claude Sonnet 5.5)
…imal overflows, fix a clue

Generated-by: Claude Code (Claude Sonnet 5.5)
…overflow

Generated-by: Claude Code (Claude Sonnet 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant