Skip to content

Prep for the next DataFusion: build against apache/datafusion main - #5

Open
adriangb wants to merge 1 commit into
claude/release-version-prep-ii17k2from
claude/prep-df-55
Open

adriangb wants to merge 1 commit into
claude/release-version-prep-ii17k2from
claude/prep-df-55

Conversation

@adriangb

Copy link
Copy Markdown
Member

Canary branch that builds and tests against unreleased DataFusion, so the work needed for the next DataFusion release is already done when it lands.

Not mergeable/releasable as-is — cargo publish rejects git dependencies. The intended lifecycle is: keep this branch around, let CI tell us when DataFusion main breaks us, and when the next DataFusion is released swap the git deps back to version = "<next>" and merge.

Stacked on #4 (the upstream sync + 0.54.3 bump); base retargets to main once that merges.

Changes

  • Cargo.toml: both datafusion entries (normal + dev) now point at https://github.com/apache/datafusion.git, branch = "main". Tested against 4a40101 (54.1.0, arrow 59.1.0).
  • Cargo.toml: rust-version raised 1.88.0 → 1.94.0, matching DataFusion main's MSRV. Without this the CI resolve/MSRV job builds DataFusion main on 1.88 and fails.
  • tests/main.rs: updated test_json_get_equals and added test_plan_json_get_equals for the one real behavior change (below).

Why a git dependency instead of [patch.crates-io]

The manifest previously carried a commented-out [patch.crates-io] block for this purpose. A patch only applies while the git version still satisfies the version = "..." requirement, so the moment main bumps its major, cargo drops the patch with a warning and quietly resolves the released version again — CI goes green while testing exactly the thing this branch exists to not test. A git dependency has no such failure mode.

The one breakage found

json_get(json_data, 'foo')='abc' — comparing the JSON union against a string — used to fail to plan, per apache/datafusion#10180, and test_json_get_equals asserted that error:

Error during planning: Cannot infer common argument type for comparison operation Union...

On DataFusion main it plans and runs. Type coercion inserts a cast to Utf8:

Projection: test.name, CAST(json_get(test.json_data, Utf8("foo")) AS Utf8) = Utf8("abc")
  TableScan: test projection=[name, json_data]

giving true for the row whose foo is "abc" and null for every value that isn't a JSON string (arrays, objects, missing keys, invalid JSON). That's the same result an explicit ::string cast would produce, so the test now asserts those rows, and a second test pins the plan.

Worth noting for later: the implicit cast is not rewritten to json_get_str the way an explicit ::string cast is (test_plan_arrow_cast_int and friends). Extending the rewrite to coercion-inserted casts would make this path cheaper — deliberately left out of scope here.

Testing

Against DataFusion main: cargo test passes (169 tests), cargo fmt --check clean, cargo clippy --all-targets -- -D warnings clean.


Generated by Claude Code

Points the datafusion dependencies at apache/datafusion main (currently
54.1.0, rev 4a40101) so breakage from unreleased DataFusion changes shows
up here rather than during the upgrade after a release.

One behaviour change needed fixing: comparing a JSON union to a string,
e.g. `json_get(json_data, 'foo')='abc'`, used to fail to plan
(apache/datafusion#10180). DataFusion now coerces the union to Utf8, so
the query plans and runs, yielding null for values that aren't JSON
strings. test_json_get_equals asserted the old planning error and has
been rewritten to assert the new results, with test_plan_json_get_equals
pinning the plan the coercion produces.

Also raises rust-version to 1.94.0 to match DataFusion main's MSRV,
which the CI MSRV job resolves from Cargo.toml.

This branch cannot be published as-is: cargo publish rejects git
dependencies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CShJj2bykoAodw9WnMQuRY
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants