Repository navigation
Conversation
`return_type_check` promised a `Dictionary(Int64, <value type>)` return type whenever the first argument was a dictionary and the value type was non-primitive, but `invoke_array_array` / `invoke_array_scalars` only wrap the result back into a dictionary when the function's `InvokeResult::ACCEPT_DICT_RETURN` is set. `json_get_array` is the only UDF in the crate with a non-primitive value type (`List(Utf8)`) and `ACCEPT_DICT_RETURN = false`, so every call against a dictionary-encoded column failed the planned-vs-actual return type check. Make `return_type_check` generic over the `InvokeResult` implementation and key the dictionary wrapping off `R::ACCEPT_DICT_RETURN`, so the declared return type and the array `invoke` actually builds are derived from the same fact and cannot drift apart again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…iter (datafusion-contrib#130) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`json_get_array_errors_on_dictionary_input` asserted the bug fixed in this branch and asked to be deleted once the fix landed. Remove it and stop excluding `json_get_array` from `prop_input_encoding_does_not_change_result`, which now covers dictionary input for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `json_get_bool` and `json_contains` no longer need `.map(|_| DataType::Boolean)`: `return_type_check::<BooleanArray>` already declares a bare `Boolean` now that the dictionary decision reads `ACCEPT_DICT_RETURN`. - Document `ACCEPT_DICT_RETURN` properly (fixing a typo), and note on `return_type_check` that dictionary *scalar* inputs are still not re-wrapped (the existing FIXMEs in the scalar invoke paths). - Add `test_dict_get_array`, covering `json_get_array` over a dictionary JSON argument with a column path, i.e. the `invoke_array_array` branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| return None; | ||
| } | ||
| (JsonPath::Index(index), Peek::Array) => peek = jiter_array_step(&mut jiter, *index)?, | ||
| // as in postgres's path operators, a text element which reaches an array is |
There was a problem hiding this comment.
🟠 High src/common.rs:586
json_get('[1,2]', '0') and json -> '0' now return the first array element instead of treating '0' as an object-field key, regressing text-key lookup semantics. The JsonPath::Key/Peek::Array arm applies PostgreSQL's text-to-array-index conversion to jiter_json_get_union callers, but that conversion is reserved for #>/#>> text-array paths; restrict it to those path operations rather than applying it in this shared function.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/common.rs around line 586:
`json_get('[1,2]', '0')` and `json -> '0'` now return the first array element instead of treating `'0'` as an object-field key, regressing text-key lookup semantics. The `JsonPath::Key`/`Peek::Array` arm applies PostgreSQL's text-to-array-index conversion to `jiter_json_get_union` callers, but that conversion is reserved for `#>`/`#>>` text-array paths; restrict it to those path operations rather than applying it in this shared function.
| // one row, and `value(0)` unwraps that row into the array of its elements | ||
| if list.len() != 1 { | ||
| return exec_err!("Expected a scalar list as a JSON path, got {} rows.", list.len()); | ||
| } |
There was a problem hiding this comment.
🟡 Medium src/common.rs:150
A null scalar list path is treated as an empty path, so json_as_text returns the entire JSON instead of null when its nullable list path argument is null. list.value(0) is arbitrary for null rows and commonly yields an empty child slice; check list.is_null(0) and append JsonPath::None before reading the value.
- let elements = list.value(0);
+ if list.is_null(0) {
+ path.push(JsonPath::None);
+ continue;
+ }
+ let elements = list.value(0);🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/common.rs around line 150:
A null scalar list path is treated as an empty path, so `json_as_text` returns the entire JSON instead of null when its nullable list path argument is null. `list.value(0)` is arbitrary for null rows and commonly yields an empty child slice; check `list.is_null(0)` and append `JsonPath::None` before reading the value.
| (JsonPath::Index(index), Peek::Array) => peek = jiter_array_step(&mut jiter, *index)?, | ||
| // as in postgres's path operators, a text element which reaches an array is | ||
| // used as an index if it spells one, e.g. `j #> '{a,0}'` | ||
| (JsonPath::Key(key), Peek::Array) => peek = jiter_array_step(&mut jiter, key.parse().ok()?)?, |
There was a problem hiding this comment.
🟠 High src/common.rs:588
Textual numeric paths still return null for JsonUnion inputs such as json_get_int(json_get('[10]', 0), '0'), while the equivalent integer path 0 succeeds. is_object_lookup classifies every JsonPath::Key as an object lookup before the union is unpacked, so '0' selects the union's null object child instead of the array element; update that classification to recognize numeric keys as array indexes.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/common.rs around line 588:
Textual numeric paths still return `null` for `JsonUnion` inputs such as `json_get_int(json_get('[10]', 0), '0')`, while the equivalent integer path `0` succeeds. `is_object_lookup` classifies every `JsonPath::Key` as an object lookup before the union is unpacked, so `'0'` selects the union's null object child instead of the array element; update that classification to recognize numeric keys as array indexes.
Symptom
json_get_arrayis completely unusable against a dictionary-encoded JSON column. Every call fails, whatever the document and whatever the path. Against a table whosejcolumn isDictionary(Int32, Utf8):with
[1, 2, 3],{"a": [1, 2]},nullandnot jsonall failing identically. In a debug build:and in release, the equivalent from the RecordBatch schema check:
The same documents in a
Utf8/Utf8View/LargeUtf8column work fine, and no other UDF in the crate is affected.Root cause
Two independent pieces of code decide whether a dictionary input produces a dictionary output, and they disagreed.
return_type_checkinsrc/common.rspromised a dictionary whenever the first argument was a dictionary and the value type was non-primitive:But at execution time,
invoke_array_arrayandinvoke_array_scalarsonly wrap the result back into a dictionaryif R::ACCEPT_DICT_RETURN, the per-UDF associated const onInvokeResult.json_get_array'sBuildArrayListsetsACCEPT_DICT_RETURN = falsewhile its value type isList(Utf8)— non-primitive. It is the only UDF in the crate with that combination, which is why it is the only one broken:return_typepromisedDictionary(Int64, List(..))andinvokedelivered a bareList(..).For the record, the full matrix of
InvokeResultimpls:ACCEPT_DICT_RETURNUInt64Arrayjson_lengthUInt64Int64Arrayjson_get_intInt64Float64Arrayjson_get_floatFloat64BooleanArrayjson_contains,json_get_boolBoolean.map(|_| DataType::Boolean))BuildArrayListjson_get_arrayList(Utf8)StringArrayjson_as_text,json_get_str,json_get_jsonUtf8BuildListArrayjson_object_keysList(Utf8)JsonUnionjson_getThe fix
return_type_checkis now generic over theInvokeResultimplementation the UDF passes toinvoke, and the dictionary wrapping is keyed offR::ACCEPT_DICT_RETURN:Each
return_typecall site is updated to name the same type it already passes toinvoke::<R>— e.g.return_type_check::<BuildArrayList>(..)alongsideinvoke::<BuildArrayList>(..).Why this shape
The alternative — special-casing
JsonGetArray::return_typeto unwrap the dictionary afterwards — fixes today's symptom but leaves the underlying hazard in place:return_typeandinvokewould still derive the dictionary decision from two unrelated facts, and the next UDF added with a non-primitive value type andACCEPT_DICT_RETURN = falsewould hit exactly the same failure. Threading the const through means both sides now read the same fact, so they cannot drift.The
!value_type.is_primitive()condition is dropped rather than kept alongside the new one, becauseACCEPT_DICT_RETURNon its own is precisely the predicateinvokeuses — keepingis_primitivewould leave a way to drift in the other direction (a primitive-valued UDF withACCEPT_DICT_RETURN = truewould haveinvokewrap a dictionary thatreturn_typedid not promise). Dropping it changes no declared return type today: no impl combinesACCEPT_DICT_RETURN = truewith a primitive value type, as the table above shows.Behaviour change
json_get_arrayis the only UDF whose declared return type changes: with a dictionary-encoded first argument it is nowList(Utf8 {json metadata})instead of the unsatisfiableDictionary(Int64, List(..)). Every other caller was checked against the table above and is unchanged — includingjson_containsandjson_get_bool, which passBoolean(non-primitive in arrow) but already discard the checked type via.map(|_| DataType::Boolean).src/rewrite.rsis untouched.Tests
Added
test_json_get_array_all_datatypestotests/main.rs, driven through the existingfor_all_json_datatypeshelper so it coversUtf8,LargeUtf8,Utf8View,DICT_TYPEandLARGE_DICT_TYPE— the two dictionary cases are the ones that failed before this change. It asserts actual values, not just the absence of an error:["foo"]→["foo"], typeList(_){"foo": [1]}/'foo'→[1], typeList(_)assert_batches_eq!over the wholetesttable, checking that non-array and non-JSON documents come back as null rather than erroringVerified against the pre-fix tree: the new test reproduces the reported error for the dictionary datatypes and passes for the string ones.
All green on
c52f83d+ this commit:cargo fmt --all --check— cleancargo clippy --all-targets --all-features— zero warnings (the crate deniesclippy::pedantic)cargo test— 173 passed, 0 failedcargo test --release— 173 passed, 0 failed (release matters here: the failure surfaces as an Arrow schema-mismatch error rather than a debug assertion)🤖 Generated with Claude Code
https://claude.ai/code/session_01VQbTD3AHrpLgGodFEDhJvr
Generated by Claude Code