Skip to content

Return the JSON argument's string type from json_as_text, json_get_str and json_get_json - #151

Open
cetra3 wants to merge 1 commit into
datafusion-contrib:mainfrom
cetra3:output-follows-input-string-type
Open

cetra3 wants to merge 1 commit into
datafusion-contrib:mainfrom
cetra3:output-follows-input-string-type

Conversation

@cetra3

@cetra3 cetra3 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

json_as_text, json_get_str and json_get_json always return Utf8, whatever the string type of the JSON argument. Utf8 has i32 offsets, so one batch holds at most 2 GiB of text, but a Utf8View or LargeUtf8 argument has no such limit. When the extracted values in one batch exceed 2 GiB, the builder panics:

panicked at arrow-array/src/builder/generic_bytes_builder.rs:94:57:
byte array offset overflow

For example, 512 rows of {"inputs": "<5 MiB string>"} as Utf8View (about 2.7 GB) panic in json_as_text(j, 'inputs').

With this change, these functions return the string type of their JSON argument:

  • Utf8 gives Utf8, LargeUtf8 gives LargeUtf8, and Utf8View gives Utf8View.
  • A dictionary argument gives Dictionary(Int64, <its value type>).
  • A JSON union (the result of json_get) gives Utf8, as before, because the union holds its strings as Utf8.

Each value is a slice of the JSON text or a decoded JSON string, and a decoded string is never longer than its escaped form. So the result never needs more offset space than the argument, and the overflow cannot occur. The example above now returns a Utf8View of about 2.7 GB.

Results for a Utf8 argument do not change. For a LargeUtf8 or Utf8View argument the values are the same, but the type changes:

  • (j -> 'k')::text over a Utf8View column now drops the cast, because json_as_text already returns Utf8View. arrow_cast(json_get(j, 'k'), 'Utf8') over the same column now keeps it.
  • Flattening json_as_text(json_get(j, 'a'), 'b') to json_as_text(j, 'a', 'b') can change the string type. The nested call reads the union's Utf8 strings, but the flattened call returns the type of j.

Tests cover each function and ->> over every string and dictionary encoding. The direct-builder check runs for all three result types, and the property tests compare string types as one family where the input encoding differs. A test over 2 GiB needs several GB of memory, so it is not included. I ran it locally: it panics on main and passes with this change.

Not changed: the strings inside the json_get union and the items of json_get_array are still Utf8.

@codecov-commenter

codecov-commenter commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.20000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 94.03%. Comparing base (aea0da0) to head (75d23be).

Files with missing lines Patch % Lines
src/common_string.rs 98.80% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #151      +/-   ##
==========================================
+ Coverage   93.79%   94.03%   +0.24%     
==========================================
  Files          18       19       +1     
  Lines        1950     2013      +63     
  Branches     1950     2013      +63     
==========================================
+ Hits         1829     1893      +64     
  Misses         66       66              
+ Partials       55       54       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@adriangb adriangb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one issue : nested and flattened forms return different string types

With j as Utf8View:

json_get_str(json_get(j, 'a'), 'b')  -- Utf8 (reads the union's Utf8 `str` child)
json_get_str(j, 'a', 'b')            -- Utf8View (reads j)

The flattened result is the one we want. But the type of the expression now depends on whether the unnest rewrite runs. with_string_type_as_utf8 hides this in the test.

I suggest that we fix this in a follow-up PR, not here. json_get builds the union with string children (str, array, object) of the input's string type, so Utf8View in gives Utf8View children. Then string_result_type reads the type from the union's str child, and we remove with_string_type_as_utf8. This also stops a Utf8View batch of more than 2 GiB from overflowing the i32 offsets of the union's Utf8 children.

The cost would be that JSON_UNION_DATA_TYPE is no longer one constant, so code that compares the exact type must change, maybe calling a function. This is also a visible type change for downstream users.

@cetra3

cetra3 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Yeah json_get is a known caveat of this PR, but is a bit more of a "breaking" change. I will work on a followup to support it

@adriangb

Copy link
Copy Markdown
Collaborator

lgtm

cetra3 added a commit to pydantic/datafusion-functions-json that referenced this pull request Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants