Repository navigation
Description
Describe the bug
The Parquet reader only treats the direct metadata and value children of a VARIANT-annotated group as binary. With shredding, more value leaves sit below typed_value: under each field group of a shredded object (typed_value.<field>.value), and under the element of a shredded array (typed_value.list.element.value). Their parents have no VARIANT annotation, so aggregate_reader_metadata::is_binary_leaf (added in #25339) returns false for them. As a result:
- With
convert_strings_to_categoriesoff, these leaves are read as STRING columns instead oflist<uint8>. - With it on, they're hashed to INT32, which loses the variant-encoded bytes.
Steps/Code to reproduce bug
python/cudf/cudf/tests/data/parquet/duckdb_variant_sample.parquet has a shredded object:
optional group v (Variant(1)) {
required binary metadata;
optional binary value;
optional group typed_value {
optional group n { optional binary value; optional int32 typed_value; }
optional group k { optional binary value; optional binary typed_value (String); }
}
}
import pylibcudf as plc
source = plc.io.SourceInfo(["duckdb_variant_sample.parquet"])
for categories in (False, True):
options = (
plc.io.parquet.ParquetReaderOptions.builder(source)
.convert_strings_to_categories(categories)
.build()
)
print(plc.io.parquet.read_parquet(options).tbl.to_arrow().schema)In both reads, v.metadata and v.value come back as list<uint8>. v.typed_value.n.value and v.typed_value.k.value come back as string without categories and int32 with them. This is based on the reader's code path; I haven't run it.
Expected behavior
Every value leaf that belongs to a VARIANT group, including those under shredded typed_value objects and arrays, is read as list<uint8> and never converted to categories. typed_value leaves keep their normal type handling, so k.typed_value stays a string.
Environment overview (please complete the following information)
Not environment-specific; this is reader logic on main.
Additional context
- Every categorization and binary-flag decision goes through
is_binary_leafafter Skip strings-to-categories for FILE and VARIANT binaries #25339, so the fix is mostly an ancestor walk inside that predicate: from avalueleaf, up through object field groups and arraylist/elementlevels oftyped_value, to a VARIANT group. The cross-source checks would then also reject mixed shredded/non-shredded classifications. - This changes output with categories off too (STRING to
list<uint8>), so it needs its own tests. Array shredding currently has no test fixture. - Skip strings-to-categories for FILE and VARIANT binaries #25339 adds a Python test that reads
duckdb_variant_sample.parquetwith categories on; it should stop excluding these leaves once this is fixed. - Related: [FEA][Story] Support ingest of Parquet VARIANT columns #22312.
Activity