Skip to content

[BUG] Parquet reader reads shredded VARIANT value leaves as strings instead of binary #25425

Open

Description

@vuule

Describe the bug
The Parquet reader only treats the direct metadata and value children of a VARIANT-annotated group as binary. With shredding, more value leaves sit below typed_value: under each field group of a shredded object (typed_value.<field>.value), and under the element of a shredded array (typed_value.list.element.value). Their parents have no VARIANT annotation, so aggregate_reader_metadata::is_binary_leaf (added in #25339) returns false for them. As a result:

  • With convert_strings_to_categories off, these leaves are read as STRING columns instead of list<uint8>.
  • With it on, they're hashed to INT32, which loses the variant-encoded bytes.

Steps/Code to reproduce bug
python/cudf/cudf/tests/data/parquet/duckdb_variant_sample.parquet has a shredded object:

optional group v (Variant(1)) {
  required binary metadata;
  optional binary value;
  optional group typed_value {
    optional group n { optional binary value; optional int32 typed_value; }
    optional group k { optional binary value; optional binary typed_value (String); }
  }
}
import pylibcudf as plc

source = plc.io.SourceInfo(["duckdb_variant_sample.parquet"])
for categories in (False, True):
    options = (
        plc.io.parquet.ParquetReaderOptions.builder(source)
        .convert_strings_to_categories(categories)
        .build()
    )
    print(plc.io.parquet.read_parquet(options).tbl.to_arrow().schema)

In both reads, v.metadata and v.value come back as list<uint8>. v.typed_value.n.value and v.typed_value.k.value come back as string without categories and int32 with them. This is based on the reader's code path; I haven't run it.

Expected behavior
Every value leaf that belongs to a VARIANT group, including those under shredded typed_value objects and arrays, is read as list<uint8> and never converted to categories. typed_value leaves keep their normal type handling, so k.typed_value stays a string.

Environment overview (please complete the following information)
Not environment-specific; this is reader logic on main.

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcuIOcuIO issuelibcudfAffects libcudf (C++/CUDA) code.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions