Allow deriving a Parquet file Arrow schema from metadata during lazy opening

[peterxcli]

Is your feature request related to a problem or challenge?

Comet's projected Variant scans need to infer each file's Arrow schema from its physical Parquet schema, ignore advisory ARROW:schema, and map ENUM leaves to Utf8 while retaining raw BINARY. The shredded layout can differ between files and is only available after loading that file's footer.

#22360 already supports an explicit PartitionedFile.arrow_schema. That solves this when the complete physical file schema is known before opening. Arrow's ArrowReaderOptions::with_schema also supports nested Binary-to-Utf8 hints, so a new Arrow ENUM conversion API is unnecessary for this case.

The remaining gap is configuring that schema during lazy file opening. On DataFusion 9082d6b10c29b72d56bede3d8e353d9d61fde542, the opener creates options from the already populated PartitionedFile, then calls ArrowReaderMetadata::load_async. ParquetFileReaderFactory returns an AsyncFileReader, whose get_metadata receives borrowed options and returns ParquetMetaData; it cannot return a newly derived Arrow schema or revised reader options to the opener.

Comet currently rewrites the returned footer's Arrow schema hint. Rebuilding public FileMetaData cannot retain its private decryptor state, so this path requires an encrypted Variant fallback.

Describe the solution you'd like

Allow a ParquetSource reader policy to derive a supplied Arrow schema from the loaded physical SchemaDescriptor during file opening, before advisory Arrow schema parsing and filter/schema preparation. Reuse Arrow's existing with_schema support and the original Arc<ParquetMetaData>, preserving decryption properties, page indexes and existing metadata-cache behavior.

The policy must be optional per source, leave ordinary scans unchanged, and support files with different nested schemas without fetching their footers during planning. A specific trait or callback design is not prescribed here.

Describe alternatives you've considered

• Populate PartitionedFile.arrow_schema through Allow specifying an arrow schema for PartitionedFile #22360 when the full per-file schema is already available. For Comet's current path, this requires moving footer loading ahead of lazy opening.
• Rewrite the footer hint: works for unencrypted files but rebuilds metadata and couples schema policy to storage metadata.
• with_skip_arrow_metadata(true) handles hint suppression alone; the ENUM policy still needs the physical annotation.
• Parquet schema file type coercion functionality is not applied on all fields in nested schemas, only on the top-level fields #25192 addresses recursion in existing table-schema coercions. The table's logical Variant schema does not contain the per-file shredded leaves or distinguish physical ENUM from BINARY, so recursion alone does not solve this case.

Additional context

Tracked in apache/datafusion-comet#5477, implemented as a workaround in apache/datafusion-comet#5868. This requests a reader integration point, not a change to DataFusion's default ENUM interpretation or an expansion of the canonical Variant format.

A local check against arrow-rs 4cd8be954f6bc6b6dd265140207365b59a9900ec passed a supplied schema for nested { e: ENUM, b: BINARY }: e became Utf8, b stayed Binary, an invalid advisory ARROW:schema was ignored, and Arc::ptr_eq(reader_metadata.metadata(), original_metadata) remained true. No encrypted scan was run; preserving decryption behavior needs coverage when implementing this integration.

[peterxcli]

take