GH-50339: [R] read_ipc_stream fails to unify nested uint64 fields inside a Struct array across record batches#50374
Merged
Conversation
|
|
github-actions
Bot
added
Component: R
Component: Documentation
awaiting committer review
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses inconsistent type unification for nested uint64 fields when converting Arrow data (e.g., from read_ipc_stream()) into R, by making uint64 always convert to R numeric (double) to ensure consistent nested/list-column element types across record batches.
Changes:
- Update the R conversion path to always map Arrow
uint64todouble(no downcasting tointeger). - Add/adjust unit tests to cover both top-level and nested/list
uint64conversions (GH-50339). - Update user-facing docs and release notes to reflect the breaking behavior change.
Reviewed changes
Copilot reviewed 6 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| r/src/array_to_vector.cpp | Always uses the double converter for UINT64. |
| r/tests/testthat/test-Array.R | Updates existing uint32 expectations and adds new coverage for uint64 always converting to double, including inside list columns. |
| r/vignettes/data_types.Rmd | Updates vignette text/table to document uint64 → double behavior. |
| r/R/type.R | Updates roxygen documentation for integer type translations, reflecting uint64 always as double. |
| r/man/data-type.Rd | Mirrors the updated documentation in the generated Rd. |
| r/NEWS.md | Adds a breaking-change note for uint64 conversion behavior (#50339). |
| r/src/arrowExports.cpp | Reorders generated export glue for Field helpers (no functional change intended). |
| r/man/acero.Rd | Updates the documented compute-function count. |
Files not reviewed (2)
- r/man/acero.Rd: Generated file
- r/man/data-type.Rd: Generated file
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| When translating from Arrow to R, integer types always translate to R integers unless one of the following exceptions applies: | ||
|
|
||
| - If the value of an Arrow uint32 or uint64 falls outside the range allowed for R integers, the result will be a numeric vector in R | ||
| - Arrow uint64 types are always converted to numeric (double) vectors in R |
Comment on lines
+359
to
+363
| #' `uint64` (64 bit unsigned integer) is always converted to `double` | ||
| #' ("numeric") in R. `uint32` (32 bit unsigned integer) and `int64` (64-bit | ||
| #' signed integer) types may contain values that exceed the range of R's | ||
| #' `integer` type (32-bit signed integer). When they do, `uint32` is converted | ||
| #' to `double` ("numeric") and `int64` is converted to `bit64::integer64`. For |
Member
There was a problem hiding this comment.
I agree with this comment, though it might make this a bit too verbose? But having some detail that there is a loss of precision here is good to call out
Member
Author
There was a problem hiding this comment.
I added something hopefully brief enough to be less verbose
jonkeane
approved these changes
Jul 8, 2026
jonkeane
left a comment
jonkeane
left a comment
Member
There was a problem hiding this comment.
I'm ok with this if the tests are happy. I'm not sure I 100% understand the usecase that is causing this, but the logic makes sense.
Comment on lines
+359
to
+363
| #' `uint64` (64 bit unsigned integer) is always converted to `double` | ||
| #' ("numeric") in R. `uint32` (32 bit unsigned integer) and `int64` (64-bit | ||
| #' signed integer) types may contain values that exceed the range of R's | ||
| #' `integer` type (32-bit signed integer). When they do, `uint32` is converted | ||
| #' to `double` ("numeric") and `int64` is converted to `bit64::integer64`. For |
Member
There was a problem hiding this comment.
I agree with this comment, though it might make this a bit too verbose? But having some detail that there is a loss of precision here is good to call out
github-actions
Bot
added
awaiting merge
Comment on lines
104
to
+108
| - If the user sets `options(arrow.int64_downcast = FALSE)`, the Arrow int64 type always yields a `bit64::integer64` vector in R | ||
| regardless of the value | ||
| - Arrow uint64 types are always converted to numeric (double) vectors in R. | ||
| Note that doubles cannot exactly represent all uint64 values; precision may | ||
| be lost for values above 2^53. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Attempting downcasting on UINT64 Fields when converting them from Arrow to R works fine on individual arrays, but fails on list arrays, because we only use the individual row to work out whether to downcast.
I considered a few different solutions but I concluded that this one is the least complex while presenting a good solution - having UINT64 always be converted to doubles and never attempt downcasting.
The impact here is that people who are expecting integers via downcasting will see a change in behaviour, but I think there's a strong argument here that if you are using UINT64 to store values that can be downcast, you probably need to be using UINT32 anyway.
Other options considered: fix this in
Converter_Listand check values to see if they can be downcast: significantly more complex and only fixes one level of nesting.I checked duckdb and they always convert UINT64 to doubles.
What changes are included in this PR?
Never try to downcast UNIT64 values when converting to R
Are these changes tested?
Yes
Are there any user-facing changes?
Breaking change as mentioned in rationale above.