Skip to content

bench(variant): add per-row metadata variant_get and shred_variant benchmarks - #11512

Closed
cetra3 wants to merge 1 commit into
apache:mainfrom
pydantic:variant-per-row-metadata-bench
Closed

cetra3 wants to merge 1 commit into
apache:mainfrom
pydantic:variant-per-row-metadata-bench

Conversation

@cetra3

@cetra3 cetra3 commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • N/A

Rationale for this change

The existing unshredded object benchmarks (variant_get_unshredded_object_path_262k_rows, shred_variant_*_object_8k_rows) repeat one metadata and value buffer for every row through BinaryView. Data written one record at a time has its own metadata dictionary in each row, with different field names and different bytes. Writers that build each row independently, such as VariantArrayBuilder, produce this shape. A benchmark with shared metadata cannot show how a change behaves on that shape. For example, it favors caches keyed on the metadata pointer.

What changes are included in this PR?

  • A generator for objects in which every row has its own metadata dictionary. Each row holds a field that every row has, a field that about 10% of rows have, and about 16 other fields drawn from a pool of 300 names. The fields are inserted in name order, so each dictionary is sorted. The values alternate between integers and short strings.
  • variant_get_per_row_metadata_{present,sparse,absent}_64k_rows: reads one field as Int32 that every row, about 10% of rows, or no row has.
  • shred_variant_per_row_metadata_8k_rows: shreds 5 fields from the same shape.

Are these changes tested?

The benchmarks check the number of non-null results for each case before timing.

Are there any significant user-facing changes?

No.

@github-actions github-actions Bot added the parquet-variant parquet-variant* crates label Oct 11, 2026
@cetra3

cetra3 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

superceded by #11624

@cetra3 cetra3 closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant