Repository navigation
Conversation
Points the datafusion dependencies at apache/datafusion main (currently 54.1.0, rev 4a40101) so breakage from unreleased DataFusion changes shows up here rather than during the upgrade after a release. One behaviour change needed fixing: comparing a JSON union to a string, e.g. `json_get(json_data, 'foo')='abc'`, used to fail to plan (apache/datafusion#10180). DataFusion now coerces the union to Utf8, so the query plans and runs, yielding null for values that aren't JSON strings. test_json_get_equals asserted the old planning error and has been rewritten to assert the new results, with test_plan_json_get_equals pinning the plan the coercion produces. Also raises rust-version to 1.94.0 to match DataFusion main's MSRV, which the CI MSRV job resolves from Cargo.toml. This branch cannot be published as-is: cargo publish rejects git dependencies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CShJj2bykoAodw9WnMQuRY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canary branch that builds and tests against unreleased DataFusion, so the work needed for the next DataFusion release is already done when it lands.
Not mergeable/releasable as-is —
cargo publishrejects git dependencies. The intended lifecycle is: keep this branch around, let CI tell us when DataFusion main breaks us, and when the next DataFusion is released swap the git deps back toversion = "<next>"and merge.Stacked on #4 (the upstream sync + 0.54.3 bump); base retargets to
mainonce that merges.Changes
Cargo.toml: bothdatafusionentries (normal + dev) now point athttps://github.com/apache/datafusion.git,branch = "main". Tested against4a40101(54.1.0, arrow 59.1.0).Cargo.toml:rust-versionraised1.88.0→1.94.0, matching DataFusion main's MSRV. Without this the CIresolve/MSRV job builds DataFusion main on 1.88 and fails.tests/main.rs: updatedtest_json_get_equalsand addedtest_plan_json_get_equalsfor the one real behavior change (below).Why a git dependency instead of
[patch.crates-io]The manifest previously carried a commented-out
[patch.crates-io]block for this purpose. A patch only applies while the git version still satisfies theversion = "..."requirement, so the moment main bumps its major, cargo drops the patch with a warning and quietly resolves the released version again — CI goes green while testing exactly the thing this branch exists to not test. A git dependency has no such failure mode.The one breakage found
json_get(json_data, 'foo')='abc'— comparing the JSON union against a string — used to fail to plan, per apache/datafusion#10180, andtest_json_get_equalsasserted that error:On DataFusion main it plans and runs. Type coercion inserts a cast to Utf8:
giving
truefor the row whosefoois"abc"and null for every value that isn't a JSON string (arrays, objects, missing keys, invalid JSON). That's the same result an explicit::stringcast would produce, so the test now asserts those rows, and a second test pins the plan.Worth noting for later: the implicit cast is not rewritten to
json_get_strthe way an explicit::stringcast is (test_plan_arrow_cast_intand friends). Extending the rewrite to coercion-inserted casts would make this path cheaper — deliberately left out of scope here.Testing
Against DataFusion main:
cargo testpasses (169 tests),cargo fmt --checkclean,cargo clippy --all-targets -- -D warningsclean.Generated by Claude Code