[Variant] Support dictionary-encoded value and typed_value inputs

[peterxcli]

Is your feature request related to a problem or challenge?

Comet decodes dictionary-encoded Variant value and typed_value children before calling VariantArray::try_new. Such arrays can arise when a Parquet reader restores dictionary types from an embedded Arrow schema. #10802 / #10810 cover encoded metadata only.

This is a compatibility feature request: the canonical Variant extension explicitly permits dictionary/run-end encoding for metadata, but does not currently permit these encodings for value or typed_value.

On arrow-rs 4cd8be954f6bc6b6dd265140207365b59a9900ec, this reproducer rejects both inputs:

use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray, DictionaryArray, Int8Array, Int32Array, StructArray};
use arrow::datatypes::{Field, Int8Type};
use parquet_variant_compute::VariantArray;

fn main() {
    let value: ArrayRef = Arc::new(DictionaryArray::<Int8Type>::new(
        Int8Array::from(vec![0]),
        Arc::new(BinaryArray::from_vec(vec![&[12, 1]])),
    ));
    let typed: ArrayRef = Arc::new(DictionaryArray::<Int8Type>::new(
        Int8Array::from(vec![0]), Arc::new(Int32Array::from(vec![1])),
    ));
    for (name, encoded) in [("value", value), ("typed_value", typed)] {
        let metadata: ArrayRef = Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]]));
        let mut fields = vec![Arc::new(Field::new("metadata", metadata.data_type().clone(), false))];
        let mut columns = vec![metadata];
        if name == "typed_value" {
            let value: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>]));
            fields.push(Arc::new(Field::new("value", value.data_type().clone(), true)));
            columns.push(value);
        }
        fields.push(Arc::new(Field::new(name, encoded.data_type().clone(), true)));
        columns.push(encoded);
        let input = StructArray::new(fields.into(), columns, None);
        println!("{}", VariantArray::try_new(&input).unwrap_err());
    }
}

Errors: Illegal shredded value type: Dictionary(Int8, Binary) and Illegal shredded value type: Dictionary(Int8, Int32).

Describe the solution you'd like

Support dictionary-encoded value and supported primitive typed_value inputs, including nested shredding states, through the shared Variant input path. Preserve logical nulls, parent nulls and sliced dictionary indices. If canonical storage remains strict, a supported normalization entry point would also let callers avoid maintaining their own recursive decoder.

Describe alternatives you've considered

Comet currently decodes dictionaries recursively. For Parquet scans it also suppresses advisory Arrow schema hints to match Spark's physical interpretation; that can avoid these arrays at the reader boundary. Rejecting noncanonical encoded inputs is another valid policy, but encoded-metadata support alone does not justify removing the existing compatibility branch.

Additional context

Tracked downstream in apache/datafusion-comet#5477; direct projection work is apache/datafusion-comet#5868. This concerns encoded inputs, distinct from dictionary output support in #10013 / #10014. Run-end-encoded value/typed_value is outside this request's immediate scope.

[peterxcli]

take