Repository navigation
Conversation
added 17 commits
November 21, 2026 23:46
…ue compiler VarkaDecimalCompiler.output checks the output's type as before, then compiles any value() and wraps it in WidenDecimal. The call is bracketed by enterDecimal/exitDecimal so the cached choice conditions are dropped before an entry's tables can roll back. As a result, every decimal value within 18 digits is a projection output: rescaling casts, negation, abs, the choices, the picks, and 349's sums, differences and products. The two decimal-valued TIME functions become decimal operands through a value() arm. The census marks dec^ on fifteen value entries; least and try_subtract wait for coverage.json to hold their rows. The pair's conformance suite lists them in declinesInPart. VarkaExpressionCompilerSuite's test of the TIME producers below a root now checks that they fuse there. Generated-by: Claude Code (Claude Opus 5.5)
…and suite The coverage family "DECIMAL values" adds eight projection rows: a CASE, an IF, coalesce, greatest, least, a negation, abs and try_subtract over decimals within 18 digits. Each row is rendered by decimal_refinement.smt2. That file's generator now compiles a decimal-valued row as a projection's one output and refines its WidenDecimal root against the same Spark specification as a predicate row's. VarkaDecimalOutputSuite runs each value under the store at one, two, four and eight long lanes, with nulls and the precision's ends, against Spark's eval. It covers abs under both lowerings, the sums and differences, and the decline past 18 digits. VARKA-348.md section 10, Step 5, records the part and the decisions taken while the owner was away, and PLAN.md row 348 records its state. Generated-by: Claude Code (Claude Opus 5.5)
…uite VarkaDecimalOutputSuite now runs the whole root, WidenDecimal over the decimal columns' leaves, into a DecimalVector of the output's type under both decimalStore arms at every width, and holds what ArrowColumnVector reads back to Spark's eval. A second new test fuses a constant and a typed null as outputs beside a column's value. VARKA-348.md scores the first run, which passed, and records why the laptop's abs reading waits. Generated-by: Claude Code (Claude Opus 5.5)
VarkaDecimalBaselineBenchmark checks each section's plan against its fusion class before timing. With every 64-bit decimal value now an output through the decimal store, twelve sections fuse whole in both sessions: the choice at precisions 9 and 18, the cast from DECIMAL(9, 2) to DECIMAL(18, 4), and the three arithmetic projections at precision 9, each at both null densities. A value root is no condition, so row 503's two-lane refusal does not take them. VARKA-348.md and PLAN.md row 348 record the change. Generated-by: Claude Code (Claude Opus 5.5)
Two lessons from building 348b go into working-in-this-repo.md, and SKILLS.md is regenerated: - the census pre-commit check refuses a claim that coverage.json cannot show until the coverage suite regenerates it; - the quote check reads a two-digit section number as a measurement. VarkaDecimalOutputSuite gains the bare rescaling cast as an output root. VARKA-348.md names the measurement rule that kept the laptop's abs reading out of this window. Generated-by: Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
…type, trim the expected declines The review found the census without least and try_subtract marked although coverage.json now holds their rows, the pair's conformance suite not listing them among the forms that decline in part, the expected-decline map still naming the TIME decimal producers that now fuse as operands, and widened throwing where a value's type differs from the output's, which now declines with both types named. Generated-by: Claude Code (Claude Sonnet 5.5)
…values by their signed reading The conformance harness's decimal slot lane refused a WidenDecimal root, and the decimal refinement compared a value root's unsigned printed word with the specification's signed value. Generated-by: Claude Code (Claude Sonnet 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
…s for the value rows Generated-by: Claude Code (Claude Sonnet 5.5)
…imal overflows, fix a clue Generated-by: Claude Code (Claude Sonnet 5.5)
…overflow Generated-by: Claude Code (Claude Sonnet 5.5)
Generated-by: Claude Code (Claude Sonnet 5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A decimal of precision 18 or less is an unscaled long. Varka already fuses comparisons over such decimals. A decimal value as a projection output (
if(c, a, b),coalesce(a, b),abs(a), a sum) was declined, so the row engine ran it.VarkaDecimalCompiler.outputcompiles anyvalue()and wraps it inWidenDecimal. This node stores a long as an Arrow decimal slot.dec^(fused in part).coverage.jsongains the family "DECIMAL values".decimal_refinement.smt2proves each one against Spark's specification.VarkaDecimalOutputSuitechecks each value against Spark's eval.Design details, the two repairs and the repository lessons
VarkaDecimalCompiler.outputchecks the output's type as before, then compiles anyvalue()and wraps it inWidenDecimal, the node that stores a long as a decimal slot of the output's type.The call is bracketed by
enterDecimalandexitDecimal, so the cached choice conditions are dropped before an entry's tables can roll back.The two decimal-valued TIME functions become decimal operands through a
value()arm instead of a special case.The census marks
dec^on fifteen value entries, among them the choices,coalesce,nvl,nullif,abs, the negation, the greatest and the arithmetic operators.leastandtry_subtractare listed as owed untilcoverage.jsonholds their rows. The pair's conformance suite lists them indeclinesInPart.coverage.jsongains the family "DECIMAL values": aCASE, anIF,coalesce,greatest,least, a negation,absandtry_subtractover decimals within 18 digits.decimal_refinement.smt2renders each of them.Its generator now compiles a decimal-valued row as a projection's one output. It refines the
WidenDecimalroot against the same Spark specification that a predicate row has.VarkaDecimalOutputSuiteruns each value under bothdecimalStorearms at one, two, four and eight long lanes, with nulls and the ends of the precision.It compares with Spark's eval and reads back what
ArrowColumnVectorholds.It also covers
absunder both lowerings, sums and differences, a constant and a typed null beside a column's value, and the decline past 18 digits.The decimal baseline benchmark marks twelve value sections as fused whole in both sessions: the choice at precisions 9 and 18, the cast from
DECIMAL(9, 2)toDECIMAL(18, 4), and the three arithmetic projections at precision 9, each at both null densities.A value root is no condition, so the two-lane condition refusal does not take them.
Two repository lessons are added to
skills/working-in-this-repo.md:coverage.jsoncannot show, until the coverage suite regenerates it.Two repairs ride with the change:
WidenDecimalroot. SoVarkaKernelChecksnow writes, sizes and reads a decimal store output as a slot.VarkaTypeConformance.overflowReachablelets a suite say that the sum, difference, negation andabsoverDECIMAL(10, 2)cannot overflow.WidenDecimalroot's data as a signed value, because the specification's decimal value is signed.The value outputs of the 128-bit decimal pair still decline. They need a two-word store, which is separate work not yet done. So
VarkaDecimalConformanceSuitekeeps itsdeclinesInPartlist.The long suite's overflow check stays cancelled by the owed
x * y, aDECIMAL(21, 4)output of that same work.Why are the changes needed?
A projection that returns a decimal is as common as one that compares it. Until then, the decimal baseline's value sections can only run on the row engine.
Does this PR introduce any user-facing change?
No new setting. Projections that return a decimal of precision 18 or less through these forms now fuse, with Spark's answers.
How was this patch tested?
On the laptop, on head
a2061571f1e: the branch merged with mastere982a4dbd32, with no conflict. Master then moved tob4cbee926d1. I merged it with no conflict. The merge leftsql/varka/emitted_bytes.jsonstale, so I regenerated it (final headf80de834aaf). On that head the push check andextra-gate.shpass. The other gates ran ona2061571f1eonly.decimal_refinement.smt2holds under Z3 and cvc5.range_calendar.smt2. This PR does not touch it. It passes alone.UseAVX=2.Full test record and what was not run
On the laptop (AVX-512, 256-bit datapath), one sbt at a time,
SPARK_LOCAL_IP=127.0.0.1, on 22 November 2026 at heada2061571f1e, which is the branch merged withorigin/masterate982a4dbd32(the merge had no conflict).Passed there:
dev/varka_regen_generated.sh --all. It left nothing to commit.python3 dev/varka_push_check.py.dev/lint-java,dev/lint-scala, andcatalyst/testOnly *VarkaInternalErrorSuite *VarkaProofFilesSuite.build/sbt 'catalyst/testOnly *VarkaCoverageSuite *VarkaDecimalLongConformanceSuite *VarkaDecimalConformanceSuite *VarkaDecimalOutputSuite *VarkaExpressionCompilerSuite *VarkaFamilySwitchSuite': 215 passed, 8 cancelled, 0 failed.build/sbt 'sql/testOnly *VarkaDecimalBaselineQueriesSuite': 11 passed.dev/varka_prove.sh --solver both.decimal_refinement.smt2holds under Z3 and cvc5 with 370 checks.In that run cvc5 timed out on
range_calendar.smt2, a file this pull request does not touch, beside other load.The file alone, and
dev/varka_prove.sh --lint, then hold under both solvers.Run on the earlier head
fcb8202ff9cand not repeated after the merge:dev/varka_regen_generated.sh --all --full.dev/varka_errorprone.sh.VARKA_PROOFS_REGEN=truefordecimal_refinement.smt2.The merge changed none of the files this pull request changes. The regeneration above found the generated files unchanged.
No option-matrix configuration exists for the decimal store arm.
VarkaDecimalOutputSuiteruns bothdecimalStorearms itself.Not run: the benchmark on any host (the runners' readings are owed), a CI Build or option matrix, and the width audit under
UseAVX=2.Review
/code-review highfound no important defect. It ran before the merge, and I did not rerun it. The merge had no conflict and the patch id is the same. I could not confirm the SHA it reviewed. Five nits are open. The size is 925 counted lines.Earlier reviews and the nit list
/code-review highran onvarka-348-decimal-compare-rbefore the merge oforigin/mastere982a4dbd32(merge base96f7ff5402b) and found no important defect.Two earlier
/code-review mediumruns had found five important items, all fixed or resolved.The merge was conflict-free. The hand-written diff (catalyst, core, docs) has the same patch id before and after it, so the review was not repeated.
The size is 925 counted lines of 2197 changed, under the cap of 1000.
Five nits are left as they are:
time_to_seconds(t) IS NULL) passesvalue()and then declines invalid()for want of an arm.widenednow declines where it threw on a scale or precision mismatch. A future arm's bug would show as a quiet fall back to the row engine.VarkaDecimalOutputSuiteruns five of its shapes throughArrowColumnVectorand the rest only on the long under the store.valuearm guards its own overflow is stated in a comment only.The one markdown line in
working-in-this-repo.mdover 100 columns is left.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Sonnet 5.5)