Skip to content

feat(parquet): allow updating cached file key-value metadata - #12117

Merged
Rich-T-kid merged 1 commit into
apache:mainfrom
manuzhang:parquet-metadata-key-values
Nov 14, 2026
Merged

Rich-T-kid merged 1 commit into
apache:mainfrom
manuzhang:parquet-metadata-key-values

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

Relates to apache/datafusion-comet#6246.

Rationale for this change

Applications that adapt cached Parquet metadata need to replace file-level key/value entries, for example an Arrow schema hint, before constructing a reader. The existing public constructors require rebuilding FileMetaData and ParquetMetaData to make this change. That loses private encryption metadata and the file decryptor, so this approach cannot safely handle encrypted files.

A setter on the existing metadata builder lets callers update those entries while preserving the state required to read the file. A downstream use is Comet's per-file schema adaptation, which currently has to exclude encrypted files from its metadata rewrite.

What changes are included in this PR?

Add ParquetMetaDataBuilder::set_key_value_metadata(Option<Vec<KeyValue>>). It replaces the complete list, with None removing the entries. Arc::make_mut gives shared file metadata copy-on-write behavior and preserves the other file metadata, row groups, page indexes and decryptor. The update affects only in-memory metadata.

Are these changes tested?

A unit test covers adding, replacing and removing entries, including an explicitly empty list. It checks that cached metadata remains unchanged, the other file metadata (including encryption fields when enabled) is preserved, and row groups and page indexes remain shared.

An encrypted-read regression covers encrypted and plaintext footers. It updates already-loaded metadata, verifies its page indexes, and reads the encrypted data without supplying decryption keys again. This checks that the cached decryptor survives the update.

Are there any significant user-facing changes?

This is an additive public API. Existing method signatures and behavior are unchanged.

AI assistance: Codex drafted the implementation and regression tests and ran local validation.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Nov 7, 2026
@manuzhang
manuzhang marked this pull request as ready for review November 7, 2026 16:44

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me -- thanks @manuzhang

}
}

/// Replaces the file-level key-value metadata.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is consistent with the other APIs to set the metadata like

/// Sets "key_value_metadata" property (defaults to `None`).
pub fn set_key_value_metadata(mut self, value: Option<Vec<KeyValue>>) -> Self {
self.key_value_metadata = value;
self
}

/// it does not modify the Parquet file.
///
/// Other metadata, including row groups, page indexes, and decryption state,
/// is preserved. Shared file metadata is cloned before modification.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

technically speaking I think the shared file metadata is "cloned if necessary" (if there there are no other copied it won't be cloned)

@Rich-T-kid
Rich-T-kid added this pull request to the merge queue Nov 13, 2026
@Rich-T-kid

Copy link
Copy Markdown
Contributor

thanks @manuzhang @alamb

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026
@Rich-T-kid
Rich-T-kid added this pull request to the merge queue Nov 14, 2026
Merged via the queue into apache:main with commit 2553a2b Nov 14, 2026
24 checks passed
@Rich-T-kid Rich-T-kid added the enhancement New feature/addition label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature/addition parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants