Skip to content

feat(file_index): integrate pruning into raw data-file reads - #783

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:file-index-read
Open

feat(file_index): integrate pruning into raw data-file reads#783
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:file-index-read

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Closes #780

FileIndex outer-format support, Bitmap/Bloom readers, and predicate composition already existed, but production raw scans did not consume embedded indexes or .index sidecars. Therefore, Rust table reads could not use FileIndexes written by Java Paimon.

Brief change log

  • Add file-index.read.enabled, defaulting to true, and enable it for ordinary, incremental, and audit raw reads.
  • Read embedded or aligned sidecar FileIndexes with Java-compatible precedence.
  • Evaluate indexes against conservatively remapped file-schema predicates while ignoring unsupported index identifiers.
  • Apply Skip and Selection before opening data files, intersecting selections with split row ranges and deletion vectors.
  • Preserve residual predicates and conservatively fall back for unsafe schema evolution or overly fragmented selections.
  • Add coverage for embedded and sidecar indexes, schema evolution, Bitmap/Bloom pruning, deletion vectors, row ranges, unsupported indexes, and enabled/disabled result equivalence.

Tests

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo test --locked -p paimon --all-targets --features fulltext,vortex
  • cargo test --locked -p paimon-rest-server --all-targets

API and Format

No public API or storage-format changes. This reads existing Java-compatible FileIndex containers and Bitmap/Bloom payloads.

Documentation

No documentation changes.

Use embedded and sidecar indexes to prune raw scans while preserving residual predicates and schema-evolution safety.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate FileIndex pruning into raw data-file reads

1 participant