Skip to content

Keep the cast when folding json_get, plus a property-test harness for the rewriter - #6

Open
adriangb wants to merge 5 commits into
claude/base-pr-127from
claude/json-functions-testing-5d2biz
Open

adriangb wants to merge 5 commits into
claude/base-pr-127from
claude/json-functions-testing-5d2biz

Conversation

@adriangb

@adriangb adriangb commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

Base branch note. This session can only push to pydantic/datafusion-functions-json, so claude/base-pr-127 is a pinned copy of PR datafusion-contrib#127's head (5ff1fd6) on this fork. The diff shown is exactly this change, stacked on datafusion-contrib#127. Retarget at datafusion-contrib/datafusion-functions-json when merging.

The bug

JsonFunctionRewriter replaced CAST(json_get(x,'k') AS T) with the bare accessor for T. The accessors return one Arrow type per JSON type — json_get_int is always Int64, json_get_float always Float64, json_get_str always Utf8 — so whenever T was not that type, the expression silently came out as the accessor's type and the narrowing the cast promised never happened:

CAST(json_get(x,'k') AS INT)           -> Int64,   not Int32
CAST(json_get(x,'k') AS REAL)          -> Float64, not Float32
CAST(json_get(x,'k') AS DECIMAL(10,2)) -> Float64, not Decimal128(10,2)
CAST(json_get(x,'k') AS VARCHAR)       -> Utf8,    not Utf8View

Values escaped the declared type too. On {"a": 3000000000}:

before after
CAST(json_get(x,'a') AS INT) 3000000000 (Int64) overflow error, as a real cast does
TRY_CAST(json_get(x,'a') AS INT) 3000000000 (Int64) NULL (Int32), as TRY_CAST promises
CAST(json_get(x,'a') AS DECIMAL(10,2)) 3000000000.0 (Float64) overflow error

The CAST half predates datafusion-contrib#127; datafusion-contrib#127 routes TRY_CAST through the same type table, which is how the TRY_CAST row above arises.

The fix

Replace only the cast's input, and drop the cast only where the accessor already returns the type that was asked for:

CAST(json_get(x,'k') AS BIGINT)  ->  json_get_int(x,'k')                      (unchanged)
CAST(json_get(x,'k') AS INT)     ->  CAST(json_get_int(x,'k') AS Int32)       (new)

::bigint, ::double and ::boolean plan exactly as before. Where the cast stays it costs one primitive-to-primitive cast, and the JSON union is still never materialized — which was the whole point of the rewrite.

That removes the reason arrow_cast folded only the four types an accessor returns exactly: arrow_cast(x,'Int32') now folds to CAST(json_get_int(x) AS Int32) and keeps its Int32, so it stops materializing the union for no reason. Both paths share one type table now, with the arrow_* type argument parsed the way DataFusion parses it itself in ArrowCastFunc::return_field_from_args. ArrowCastFunc::simplify already drops the cast when the accessor's type matches, so the exact arrow_cast cases keep their old plans as well.

The set of Arrow types that fold is unchanged — smallint and friends are still left alone.

Pushdown is unaffected

JsonGet::placement moves a json_get over a column with literal path arguments towards the leaf nodes, and a cast sitting on top of the accessor is exactly the shape that could have stranded it higher up the plan. It does not: the accessor still lands in the leaf projection for every folding target, and unwrap_cast_in_comparison removes the added cast again for an integer comparison, so where (json_data->'foo')::int = 5 plans identically to before. narrowing_cast_still_reaches_the_leaf_nodes pins this.

Tests

tests/main.rs expectations move from the accessor's type to the type the query asked for (::int → Int32, ::float → Float32, ::numeric → Decimal128(38,10), ::string → Utf8View), and the plan tests show the retained cast. test_plan_arrow_cast_fn_inexact_type_not_folded is now test_plan_arrow_cast_fn_narrowing_type_keeps_cast — its premise is inverted by this change.

tests/rewrite_differential.rs (new)

Property tests over generated JSON documents × 5 cast spellings (CAST, ::, TRY_CAST, arrow_cast, arrow_try_cast) × 8 target types.

prop_fold_is_invisible — folding must equal applying the same cast to the typed accessor: CAST(json_get(x,'k') AS T) == CAST(json_get_<T>(x,'k') AS T). The right-hand side is not itself folded (the rewriter only folds casts over json_get), so it works as a reference implementation. This is the property that found the bug above; it now covers every folding pair rather than only the type-exact ones.

prop_unnest_is_invisible — flattening json_get(json_get(j,'a'),'b') must not change the answer, checked against the genuinely un-rewritten plan.

Getting that un-rewritten plan needs no second SessionContext. Function rewrites run in the analyzer, before the optimizer merges projections, so a cast over a subquery column is never folded — and merge_projection then inlines the json_get back underneath. barrier_really_blocks_folding asserts the trick still works, so the differential cannot quietly become a tautology.

For cast folding that plain "same with and without the rewrite" equality is false by design, and unfolded_union_cast_is_weaker records why: casting the JSON union is far weaker than the accessors (CAST(union AS Float64) is NULL even for 42), so the rewrite is deliberately better than what it replaces. Hence the typed-accessor reference instead.

Also every_cast_spelling_folds_as_documented (plan-shape table over the whole matrix), narrowing_cast_preserves_type_and_narrows, narrowing_cast_still_avoids_the_union and narrowing_cast_still_reaches_the_leaf_nodes.

tests/json_robustness.rs (new)

Three properties across all 11 path-taking UDFs, so a change to any one of them or to the shared scanning code in common.rs has to keep them true:

  • prop_no_panic_on_arbitrary_json — valid JSON, truncated JSON, JSON with a byte removed, and arbitrary text must come back as NULL, never a panic or an error. This is the shape of the bug fixed in Fix panic on JSON integers outside the i64 fast path datafusion-contrib/datafusion-functions-json#124 (a JSON integer too wide for jiter's fast path reached a todo!()); the generator emits those integers, so it would have caught it.
  • prop_scalar_and_array_paths_agree — a literal document is const-evaluated through each UDF's ScalarValue path, a column goes through the array path.
  • prop_input_encoding_does_not_change_result — Utf8, LargeUtf8, Utf8View and dictionary-encoded input must agree.

That last one found a separate, unrelated bug — json_get_array fails on every dictionary-encoded column — which is pinned here by json_get_array_errors_on_dictionary_input and fixed in its own PR. Delete that test and the json_get_array exclusion when the fix lands.

Case counts default low so the suite stays fast (~14s); PROPTEST_CASES overrides, which is how these get run as a fuzzer:

PROPTEST_CASES=50000 cargo test --release --test rewrite_differential

Verification

🤖 Generated with Claude Code

https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr

adriangb and others added 5 commits September 10, 2026 03:03
The rewriter's cast folding and nested-call flattening are optimizations, so
both are only correct if they are invisible to the caller. `tests/main.rs`
checks that with hand-written examples, one per spelling; these generate input
instead.

Function rewrites run in the analyzer, before the optimizer merges projections,
so a cast applied to a subquery column never sees the `json_get` underneath it
and is not folded — and `merge_projection` then inlines the `json_get` back
under the cast. That yields the un-rewritten plan without needing a second
`SessionContext`, and `barrier_really_blocks_folding` keeps it honest.

For nested `json_get` flattening that gives the plain with/without-optimizer
differential, `prop_unnest_is_invisible`.

For cast folding it does not: casting the JSON union is far weaker than the
typed accessors (`CAST(union AS Float64)` is NULL even for `42`), so the fold
is deliberately better than what it replaces. `prop_fold_is_invisible` instead
checks the equivalence the fold actually promises, that folding equals using
the typed accessor and then applying the cast that was written.

That equivalence holds only where the accessor returns exactly the type asked
for. `narrowing_cast_fold_is_not_type_preserving` and
`narrowing_try_cast_fold_returns_value_instead_of_null` pin down where it does
not: the `CAST`/`TRY_CAST` table folds `int`, `real`, `decimal` and `varchar`
to accessors returning `Int64`, `Float64` and `Utf8`, so the expression takes
the accessor's type and the narrowing never happens. Both tests characterize
current behaviour so they stay green, and say what to do when it is fixed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
Three properties that hold for every accessor, so a change to any one of them
or to the shared scanning code in `common.rs` has to keep them true.

`prop_no_panic_on_arbitrary_json` feeds each UDF valid JSON, truncated JSON,
JSON with a byte removed, and arbitrary text. Malformed input has to come back
as NULL rather than a panic or an error. That is the shape of the bug fixed in
774c570 (datafusion-contrib#124): a JSON integer too wide for jiter's fast path reached a
`todo!()`. The generator emits those integers, so this would have caught it.

`prop_scalar_and_array_paths_agree` checks a literal document against the same
document in a column. A literal is const-evaluated through each UDF's
`ScalarValue` path while a column goes through the array path; they are
separate code in every UDF here.

`prop_input_encoding_does_not_change_result` checks `Utf8`, `LargeUtf8`,
`Utf8View` and dictionary-encoded input against each other — four more separate
paths, currently covered only by hand-written cases via `for_all_json_datatypes`.

That last one found `json_get_array_errors_on_dictionary_input`: every
`json_get_array` call on a dictionary-encoded JSON column fails, whatever the
document and whatever the path. `return_type_check` promises
`Dictionary(Int64, List(Utf8))` because the value type is not primitive, but
`BuildArrayList` sets `ACCEPT_DICT_RETURN = false` so `invoke_array_array`
returns a bare `List`. It is the only UDF with that combination. Debug builds
trip DataFusion's own assertion and release builds reach Arrow's record batch
schema check, so the test matches on the promised type that appears in both.
It characterizes the failure and says what to delete when it is fixed.

Case counts default low so the suite stays fast, and `PROPTEST_CASES` still
wins, which is how these get run as a fuzzer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
The fold replaced `CAST(json_get(x,'k') AS T)` with the bare accessor for `T`.
The accessors return one Arrow type per JSON type — `json_get_int` is always
`Int64` — so whenever `T` was not that type, the expression silently came out
as the accessor's type and the narrowing the cast promised never happened:

    CAST(json_get(x,'k') AS INT)           -> Int64,   not Int32
    CAST(json_get(x,'k') AS REAL)          -> Float64, not Float32
    CAST(json_get(x,'k') AS DECIMAL(10,2)) -> Float64, not Decimal128(10,2)
    CAST(json_get(x,'k') AS VARCHAR)       -> Utf8,    not Utf8View

and on `{"a": 3000000000}`, `CAST(.. AS INT)` returned 3000000000 where a real
cast raises an overflow error, while `TRY_CAST(.. AS INT)` returned it instead
of NULL.

Now only the cast's input is replaced. The cast stays unless the accessor
already returns the type asked for, in which case it is dropped and this is the
substitution it has always been — so `::bigint`, `::double` and `::boolean`
plan exactly as before. Where it stays it costs one primitive-to-primitive
cast and the JSON union is still never materialized, which was the whole point.

That also removes the reason `arrow_cast` folded only the four types an
accessor returns exactly: `arrow_cast(x,'Int32')` now folds to
`CAST(json_get_int(x) AS Int32)` and keeps its `Int32`. Both paths share one
type table, with the `arrow_*` type argument parsed the way DataFusion parses
it itself in `ArrowCastFunc::return_field_from_args`.

`tests/main.rs` expectations change from the accessor's type to the type the
query asked for. `prop_fold_is_invisible` now covers every folding
(spelling, type) pair rather than only the type-exact ones, and the two tests
that characterized the old behaviour are replaced by
`narrowing_cast_preserves_type_and_narrows` and
`narrowing_cast_still_avoids_the_union`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
`JsonGet::placement` moves a `json_get` over a column with literal path
arguments towards the leaf nodes, which is where a filter wants the work to
happen. A cast now sits on top of the accessor for a narrowing target, which is
exactly the shape that could have stranded it higher up the plan. It does not —
the accessor still lands in the leaf projection and `unwrap_cast_in_comparison`
even removes the cast again for an integer comparison — so pin it before some
later change quietly gives it up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
CI runs `cargo clippy --all-targets -- -D warnings`, which promotes the
warn-by-default `clippy::filter_next` that the crate's own pedantic-only
lint config does not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant