Skip to content

GH-34319: [R] Batch simple attribute restoration for wide tables - #55668

Draft
Aurantium-47 wants to merge 1 commit into
apache:mainfrom
Aurantium-47:perf/r-metadata-bulk-restore
Draft

Aurantium-47 wants to merge 1 commit into
apache:mainfrom
Aurantium-47:perf/r-metadata-bulk-restore

Conversation

@Aurantium-47

Copy link
Copy Markdown

Rationale for this change

#34319 still exhibits substantial overhead when restoring per-column R attributes on wide tables. The current restoration loop repeatedly replaces individual data-frame columns and repeatedly looks up column names. Reading only a few columns also incurs the separate cost of sanitizing the file's full R metadata; this PR does not alter that safety check.

What changes are included in this PR?

  • Match selected column and metadata positions once, and apply one bulk column replacement for ordinary data frames/tibbles containing plain atomic columns and simple attribute metadata.
  • Keep the original ordered path for unusual names, subclasses, nested/custom objects and class-changing metadata.
  • Fall back to the original ordered restoration if the fast path encounters a warning or error, preserving partial recovery from invalid metadata.
  • Cover ordering, missing and duplicate names, empty rows, subclass dispatch, invalid metadata and early termination before custom nested-column methods.

Are these changes tested?

  • The current-main test-metadata.R file passes all 105 expectations with the modified R function injected into a conda-forge Arrow R 25.0.1 runtime (R 4.5.3, Linux x86_64).
  • A separate differential review exercised 1,352 original/patched function cases and compared returned values, warning text/counts and uncaught errors, with no differences. An additional custom nested-data-frame method invocation counter verifies early termination.
  • git diff --check passes.
  • End-to-end Parquet benchmark: 16,000 columns × 10 rows, per-column text and levels attributes, same file for both implementations, warmup plus three alternating original/patched repetitions and explicit garbage collection.
    | Selected columns (of 16,000) | Original median | Patched median | Ratio |
    | --- | ---: | ---: | ---: |
    | 200 | 1.806 s | 1.767 s | 1.02× |
    | 16000 | 22.227 s | 2.674 s | 8.31× |

All-column raw seconds: original [22.227, 22.876, 21.933], patched [2.768, 2.508, 2.674]. The 200-column difference is small and should be treated as noise, not a demonstrated speedup. This is a synthetic reproduction of the issue, not a general Arrow performance claim.

The original and patched R functions are both taken from main at 21156bccc5f3b2eaaa21b9045cccd02ba332f435. The native Arrow library was held constant at 25.0.1; this is not a local build of current-main C++, and a complete current-main package build/test remains for CI. No C++ code changes are included.

Are there any user-facing changes?

Faster restoration of simple per-column attributes on wide tables. No API or file-format change. The separate metadata deserialization/sanitization cost remains, so narrow projections should not be expected to receive the same improvement.

Was AI used for this PR?

AI was used for implementation, differential review, tests, local benchmark execution and this description. No human review is claimed.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants