Skip to content

Fix json_get_array on dictionary-encoded JSON columns - #7

Closed
adriangb wants to merge 7 commits into
claude/base-upstream-mainfrom
claude/fix-json-get-array-dict
Closed

adriangb wants to merge 7 commits into
claude/base-upstream-mainfrom
claude/fix-json-get-array-dict

Conversation

@adriangb

Copy link
Copy Markdown
Member

Note on the base branch: this session cannot push to datafusion-contrib, so the PR is opened against claude/base-upstream-main on the pydantic fork — a pinned copy of datafusion-contrib/datafusion-functions-json main (c52f83d). It is intended to be retargeted at datafusion-contrib/datafusion-functions-json main when merging.

Symptom

json_get_array is completely unusable against a dictionary-encoded JSON column. Every call fails, whatever the document and whatever the path. Against a table whose j column is Dictionary(Int32, Utf8):

select json_get_array(j) from t       -- fails
select json_get_array(j, 'a') from t  -- fails

with [1, 2, 3], {"a": [1, 2]}, null and not json all failing identically. In a debug build:

Internal error: Assertion failed: result_data_type == *expected_type: Function 'json_get_array'
returned value of type 'List(Utf8, metadata: {...})' while the following type was promised at
planning time and expected: 'Dictionary(Int64, List(Utf8, metadata: {...}))'.

and in release, the equivalent from the RecordBatch schema check:

Arrow error: Invalid argument error: column types must match schema types,
expected Dictionary(Int64, List(Utf8, ...)) but found List(Utf8, ...) at column index 0

The same documents in a Utf8 / Utf8View / LargeUtf8 column work fine, and no other UDF in the crate is affected.

Root cause

Two independent pieces of code decide whether a dictionary input produces a dictionary output, and they disagreed.

return_type_check in src/common.rs promised a dictionary whenever the first argument was a dictionary and the value type was non-primitive:

if first_dict_key_type.is_some() && !value_type.is_primitive() {
    Ok(DataType::Dictionary(Box::new(DataType::Int64), Box::new(value_type)))
} else {
    Ok(value_type)
}

But at execution time, invoke_array_array and invoke_array_scalars only wrap the result back into a dictionary if R::ACCEPT_DICT_RETURN, the per-UDF associated const on InvokeResult.

json_get_array's BuildArrayList sets ACCEPT_DICT_RETURN = false while its value type is List(Utf8) — non-primitive. It is the only UDF in the crate with that combination, which is why it is the only one broken: return_type promised Dictionary(Int64, List(..)) and invoke delivered a bare List(..).

For the record, the full matrix of InvokeResult impls:

impl UDF(s) ACCEPT_DICT_RETURN value type primitive?
UInt64Array json_length false UInt64 yes
Int64Array json_get_int false Int64 yes
Float64Array json_get_float false Float64 yes
BooleanArray json_contains, json_get_bool false Boolean no (both discard the result via .map(|_| DataType::Boolean))
BuildArrayList json_get_array false List(Utf8) no ← the bug
StringArray json_as_text, json_get_str, json_get_json true Utf8 no
BuildListArray json_object_keys true List(Utf8) no
JsonUnion json_get true union no

The fix

return_type_check is now generic over the InvokeResult implementation the UDF passes to invoke, and the dictionary wrapping is keyed off R::ACCEPT_DICT_RETURN:

pub fn return_type_check<R: InvokeResult>(
    args: &[DataType],
    fn_name: &str,
    value_type: DataType,
) -> DataFusionResult<DataType> {
    ...
    // this must mirror the dictionary handling in `invoke_array_array` and `invoke_array_scalars`,
    // which wrap the result back into a dictionary if and only if `R::ACCEPT_DICT_RETURN` is set
    if first_dict_key_type.is_some() && R::ACCEPT_DICT_RETURN {

Each return_type call site is updated to name the same type it already passes to invoke::<R> — e.g. return_type_check::<BuildArrayList>(..) alongside invoke::<BuildArrayList>(..).

Why this shape

The alternative — special-casing JsonGetArray::return_type to unwrap the dictionary afterwards — fixes today's symptom but leaves the underlying hazard in place: return_type and invoke would still derive the dictionary decision from two unrelated facts, and the next UDF added with a non-primitive value type and ACCEPT_DICT_RETURN = false would hit exactly the same failure. Threading the const through means both sides now read the same fact, so they cannot drift.

The !value_type.is_primitive() condition is dropped rather than kept alongside the new one, because ACCEPT_DICT_RETURN on its own is precisely the predicate invoke uses — keeping is_primitive would leave a way to drift in the other direction (a primitive-valued UDF with ACCEPT_DICT_RETURN = true would have invoke wrap a dictionary that return_type did not promise). Dropping it changes no declared return type today: no impl combines ACCEPT_DICT_RETURN = true with a primitive value type, as the table above shows.

Behaviour change

json_get_array is the only UDF whose declared return type changes: with a dictionary-encoded first argument it is now List(Utf8 {json metadata}) instead of the unsatisfiable Dictionary(Int64, List(..)). Every other caller was checked against the table above and is unchanged — including json_contains and json_get_bool, which pass Boolean (non-primitive in arrow) but already discard the checked type via .map(|_| DataType::Boolean).

src/rewrite.rs is untouched.

Tests

Added test_json_get_array_all_datatypes to tests/main.rs, driven through the existing for_all_json_datatypes helper so it covers Utf8, LargeUtf8, Utf8View, DICT_TYPE and LARGE_DICT_TYPE — the two dictionary cases are the ones that failed before this change. It asserts actual values, not just the absence of an error:

  • no path argument, ["foo"] → ["foo"], type List(_)
  • with a path argument, {"foo": [1]} / 'foo' → [1], type List(_)
  • a full assert_batches_eq! over the whole test table, checking that non-array and non-JSON documents come back as null rather than erroring

Verified against the pre-fix tree: the new test reproduces the reported error for the dictionary datatypes and passes for the string ones.

All green on c52f83d + this commit:

  • cargo fmt --all --check — clean
  • cargo clippy --all-targets --all-features — zero warnings (the crate denies clippy::pedantic)
  • cargo test — 173 passed, 0 failed
  • cargo test --release — 173 passed, 0 failed (release matters here: the failure surfaces as an Arrow schema-mismatch error rather than a debug assertion)

🤖 Generated with Claude Code

https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr


Generated by Claude Code

adriangb and others added 7 commits September 10, 2026 03:56
`return_type_check` promised a `Dictionary(Int64, <value type>)` return type
whenever the first argument was a dictionary and the value type was
non-primitive, but `invoke_array_array` / `invoke_array_scalars` only wrap the
result back into a dictionary when the function's `InvokeResult::ACCEPT_DICT_RETURN`
is set. `json_get_array` is the only UDF in the crate with a non-primitive value
type (`List(Utf8)`) and `ACCEPT_DICT_RETURN = false`, so every call against a
dictionary-encoded column failed the planned-vs-actual return type check.

Make `return_type_check` generic over the `InvokeResult` implementation and key
the dictionary wrapping off `R::ACCEPT_DICT_RETURN`, so the declared return type
and the array `invoke` actually builds are derived from the same fact and cannot
drift apart again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…iter (datafusion-contrib#130)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`json_get_array_errors_on_dictionary_input` asserted the bug fixed in this
branch and asked to be deleted once the fix landed. Remove it and stop
excluding `json_get_array` from `prop_input_encoding_does_not_change_result`,
which now covers dictionary input for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `json_get_bool` and `json_contains` no longer need `.map(|_| DataType::Boolean)`:
  `return_type_check::<BooleanArray>` already declares a bare `Boolean` now that
  the dictionary decision reads `ACCEPT_DICT_RETURN`.
- Document `ACCEPT_DICT_RETURN` properly (fixing a typo), and note on
  `return_type_check` that dictionary *scalar* inputs are still not re-wrapped
  (the existing FIXMEs in the scalar invoke paths).
- Add `test_dict_get_array`, covering `json_get_array` over a dictionary JSON
  argument with a column path, i.e. the `invoke_array_array` branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread src/common.rs
return None;
}
(JsonPath::Index(index), Peek::Array) => peek = jiter_array_step(&mut jiter, *index)?,
// as in postgres's path operators, a text element which reaches an array is

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High src/common.rs:586

json_get('[1,2]', '0') and json -> '0' now return the first array element instead of treating '0' as an object-field key, regressing text-key lookup semantics. The JsonPath::Key/Peek::Array arm applies PostgreSQL's text-to-array-index conversion to jiter_json_get_union callers, but that conversion is reserved for #>/#>> text-array paths; restrict it to those path operations rather than applying it in this shared function.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/common.rs around line 586:

`json_get('[1,2]', '0')` and `json -> '0'` now return the first array element instead of treating `'0'` as an object-field key, regressing text-key lookup semantics. The `JsonPath::Key`/`Peek::Array` arm applies PostgreSQL's text-to-array-index conversion to `jiter_json_get_union` callers, but that conversion is reserved for `#>`/`#>>` text-array paths; restrict it to those path operations rather than applying it in this shared function.

Comment thread src/common.rs
// one row, and `value(0)` unwraps that row into the array of its elements
if list.len() != 1 {
return exec_err!("Expected a scalar list as a JSON path, got {} rows.", list.len());
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium src/common.rs:150

A null scalar list path is treated as an empty path, so json_as_text returns the entire JSON instead of null when its nullable list path argument is null. list.value(0) is arbitrary for null rows and commonly yields an empty child slice; check list.is_null(0) and append JsonPath::None before reading the value.

-                    let elements = list.value(0);
+                    if list.is_null(0) {
+                        path.push(JsonPath::None);
+                        continue;
+                    }
+                    let elements = list.value(0);
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/common.rs around line 150:

A null scalar list path is treated as an empty path, so `json_as_text` returns the entire JSON instead of null when its nullable list path argument is null. `list.value(0)` is arbitrary for null rows and commonly yields an empty child slice; check `list.is_null(0)` and append `JsonPath::None` before reading the value.

Comment thread src/common.rs
(JsonPath::Index(index), Peek::Array) => peek = jiter_array_step(&mut jiter, *index)?,
// as in postgres's path operators, a text element which reaches an array is
// used as an index if it spells one, e.g. `j #> '{a,0}'`
(JsonPath::Key(key), Peek::Array) => peek = jiter_array_step(&mut jiter, key.parse().ok()?)?,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High src/common.rs:588

Textual numeric paths still return null for JsonUnion inputs such as json_get_int(json_get('[10]', 0), '0'), while the equivalent integer path 0 succeeds. is_object_lookup classifies every JsonPath::Key as an object lookup before the union is unpacked, so '0' selects the union's null object child instead of the array element; update that classification to recognize numeric keys as array indexes.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/common.rs around line 588:

Textual numeric paths still return `null` for `JsonUnion` inputs such as `json_get_int(json_get('[10]', 0), '0')`, while the equivalent integer path `0` succeeds. `is_object_lookup` classifies every `JsonPath::Key` as an object lookup before the union is unpacked, so `'0'` selects the union's null object child instead of the array element; update that classification to recognize numeric keys as array indexes.

@adriangb adriangb closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants