Skip to content

[Variant] unshred_variant panics when a shredded field name is absent from metadata #11069

Description

@peterxcli

Describe the bug

unshred_variant still panics when a present shredded object field has no entry in the row's metadata dictionary. VariantArray::try_new accepts the array, but the read-only metadata builder later returns an error that ObjectBuilder::insert unwraps.

Reproduced on arrow-rs 4cd8be954f6bc6b6dd265140207365b59a9900ec, after #9741. This is malformed input under the current Variant metadata requirement, which requires shredded field names in metadata. The bug is the panic from a fallible kernel consuming file data.

To Reproduce

use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray, Int32Array, StructArray};
use arrow::datatypes::Field;
use parquet_variant_compute::{VariantArray, unshred_variant};

fn structure(fields: Vec<(&str, ArrayRef, bool)>) -> StructArray {
    let schema = fields.iter().map(|(name, array, nullable)|
        Arc::new(Field::new(*name, array.data_type().clone(), *nullable)))
        .collect::<Vec<_>>();
    StructArray::new(schema.into(), fields.into_iter().map(|(_, a, _)| a).collect(), None)
}

fn main() {
    let missing: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>]));
    let a: ArrayRef = Arc::new(structure(vec![
        ("value", missing.clone(), true),
        ("typed_value", Arc::new(Int32Array::from(vec![1])), true),
    ]));
    let typed: ArrayRef = Arc::new(structure(vec![("a", a, false)]));
    let input = structure(vec![
        ("metadata", Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]])), false),
        ("value", missing, true),
        ("typed_value", typed, true),
    ]);
    let input = VariantArray::try_new(&input).unwrap();
    let _ = unshred_variant(&input); // panics instead of returning Err
}

Panic: called Result::unwrap() on an Err value: InvalidArgumentError("Field name 'a' not found in metadata dictionary") at parquet-variant/src/builder/object.rs:104.

Expected behavior

Return ArrowError for a present field whose name is absent from metadata, including nested objects and list elements. Absent object fields and rows masked by parent nulls should not trigger this error. Supporting permissive metadata repair would be a separate compatibility choice, not a requirement of this bug fix.

Additional context

Related prior panic report: #9740 / #9741. Comet encounters this through Spark-compatible input handling and currently extends metadata and remaps residual field IDs before unshredding in apache/datafusion-comet#5868.

Downstream tracking: apache/datafusion-comet#5477. Returning an error fixes the Arrow panic but does not by itself replace Comet's permissive Spark compatibility behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions