Repository navigation
Conversation
…_str` and `json_get_json`
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #151 +/- ##
==========================================
+ Coverage 93.79% 94.03% +0.24%
==========================================
Files 18 19 +1
Lines 1950 2013 +63
Branches 1950 2013 +63
==========================================
+ Hits 1829 1893 +64
Misses 66 66
+ Partials 55 54 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
adriangb
left a comment
There was a problem hiding this comment.
I found one issue : nested and flattened forms return different string types
With j as Utf8View:
json_get_str(json_get(j, 'a'), 'b') -- Utf8 (reads the union's Utf8 `str` child)
json_get_str(j, 'a', 'b') -- Utf8View (reads j)The flattened result is the one we want. But the type of the expression now depends on whether the unnest rewrite runs. with_string_type_as_utf8 hides this in the test.
I suggest that we fix this in a follow-up PR, not here. json_get builds the union with string children (str, array, object) of the input's string type, so Utf8View in gives Utf8View children. Then string_result_type reads the type from the union's str child, and we remove with_string_type_as_utf8. This also stops a Utf8View batch of more than 2 GiB from overflowing the i32 offsets of the union's Utf8 children.
The cost would be that JSON_UNION_DATA_TYPE is no longer one constant, so code that compares the exact type must change, maybe calling a function. This is also a visible type change for downstream users.
|
Yeah |
|
lgtm |
…_str` and `json_get_json` Upstream: datafusion-contrib#151
json_as_text,json_get_strandjson_get_jsonalways returnUtf8, whatever the string type of the JSON argument.Utf8hasi32offsets, so one batch holds at most 2 GiB of text, but aUtf8VieworLargeUtf8argument has no such limit. When the extracted values in one batch exceed 2 GiB, the builder panics:For example, 512 rows of
{"inputs": "<5 MiB string>"}asUtf8View(about 2.7 GB) panic injson_as_text(j, 'inputs').With this change, these functions return the string type of their JSON argument:
Utf8givesUtf8,LargeUtf8givesLargeUtf8, andUtf8ViewgivesUtf8View.Dictionary(Int64, <its value type>).json_get) givesUtf8, as before, because the union holds its strings asUtf8.Each value is a slice of the JSON text or a decoded JSON string, and a decoded string is never longer than its escaped form. So the result never needs more offset space than the argument, and the overflow cannot occur. The example above now returns a
Utf8Viewof about 2.7 GB.Results for a
Utf8argument do not change. For aLargeUtf8orUtf8Viewargument the values are the same, but the type changes:(j -> 'k')::textover aUtf8Viewcolumn now drops the cast, becausejson_as_textalready returnsUtf8View.arrow_cast(json_get(j, 'k'), 'Utf8')over the same column now keeps it.json_as_text(json_get(j, 'a'), 'b')tojson_as_text(j, 'a', 'b')can change the string type. The nested call reads the union'sUtf8strings, but the flattened call returns the type ofj.Tests cover each function and
->>over every string and dictionary encoding. The direct-builder check runs for all three result types, and the property tests compare string types as one family where the input encoding differs. A test over 2 GiB needs several GB of memory, so it is not included. I ran it locally: it panics onmainand passes with this change.Not changed: the strings inside the
json_getunion and the items ofjson_get_arrayare stillUtf8.