[perf](catalog) two-level cross-query cache for external partition derived views#65829

Merged

morningman merged 19 commits into

apache:branch-catalog-spifrom

morningman:catalog-spi-2-lvl-cache

Jul 21, 2026

Commits on Jul 20, 2026

[doc](catalog) design: external partition derived-view second-level cache (connector-view cache + NereidsSortedPartitionsCacheManager wiring)

morningman
and
claude
committed
[doc](catalog) partition-cache design: drop use_meta_cache=false handling (always true under new SPI)

morningman
and
claude
committed
[doc](catalog) impl plan: Cache B — external SortedPartitionRanges reuse (iceberg+paimon)

morningman
and
claude
committed
[feat](catalog) external MVCC tables implement SupportBinarySearchFilteringPartitions (snapshot-keyed ranges)

morningman
and
claude
committed
[feat](catalog) revive ExternalTable.getSortedPartitionRanges to delegate to NereidsSortedPartitionsCacheManager

morningman
and
claude
committed
[feat](catalog) PruneFileScanPartition: consult NereidsSortedPartitionsCacheManager before building ranges

morningman
and
claude
committed
[feat](catalog) drop external SortedPartitionRanges cache on table invalidation

morningman
and
claude
committed
[test](catalog) external SortedPartitionRanges cache: hit, version-rebuild, origin-map consistency

morningman
and
claude
committed
[test](catalog) close review gap: drive real PluginDrivenMvccExternalTable + ExternalMetaCacheMgr wiring

morningman
and
claude
committed
[fix](catalog) gate Cache B partition-ranges reuse on a real (non-sentinel) MVCC snapshot id
morningman
committed
[feature](catalog) add generic ConnectorPartitionViewCache (cache A) framework
morningman
committed
[perf](catalog) wire cross-query derived partition-view cache A into the iceberg connector (S3)

morningman
and
claude
committed
[perf](catalog) wire cross-query derived partition-view cache A into the paimon connector (S4)

morningman
and
claude
committed
[feat](catalog) enable Cache B for snapshot-less connectors via a partition-name-set version token

morningman
and
claude
committed
[perf](catalog) wire cross-query derived partition-view cache A into the hive connector (S6)

morningman
and
claude
committed
[fix](catalog) make Cache B version the frozen partition name-set for all engines + drop Cache B on db/catalog invalidation

morningman
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[perf](catalog) two-level cross-query cache for external partition derived views#65829

[perf](catalog) two-level cross-query cache for external partition derived views#65829
morningman merged 19 commits into
apache:branch-catalog-spifrom
morningman:catalog-spi-2-lvl-cache

Commits on Jul 20, 2026

[doc](catalog) design: external partition derived-view second-level cache (connector-view cache + NereidsSortedPartitionsCacheManager wiring)

[doc](catalog) partition-cache design: drop use_meta_cache=false handling (always true under new SPI)

[doc](catalog) impl plan: Cache B — external SortedPartitionRanges reuse (iceberg+paimon)

[feat](catalog) external MVCC tables implement SupportBinarySearchFilteringPartitions (snapshot-keyed ranges)

[feat](catalog) revive ExternalTable.getSortedPartitionRanges to delegate to NereidsSortedPartitionsCacheManager

[feat](catalog) PruneFileScanPartition: consult NereidsSortedPartitionsCacheManager before building ranges

[feat](catalog) drop external SortedPartitionRanges cache on table invalidation

[test](catalog) external SortedPartitionRanges cache: hit, version-rebuild, origin-map consistency

[test](catalog) close review gap: drive real PluginDrivenMvccExternalTable + ExternalMetaCacheMgr wiring

[fix](catalog) gate Cache B partition-ranges reuse on a real (non-sentinel) MVCC snapshot id

[feature](catalog) add generic ConnectorPartitionViewCache (cache A) framework

[perf](catalog) wire cross-query derived partition-view cache A into the iceberg connector (S3)

[perf](catalog) wire cross-query derived partition-view cache A into the paimon connector (S4)

[feat](catalog) enable Cache B for snapshot-less connectors via a partition-name-set version token

[perf](catalog) wire cross-query derived partition-view cache A into the hive connector (S6)

[fix](catalog) make Cache B version the frozen partition name-set for all engines + drop Cache B on db/catalog invalidation

Commits on Jul 21, 2026

[test](catalog) stub getStatementScope in PhysicalIcebergMergeSinkTest connector mock

[fix](ci) allowlist gitleaks generic-api-key false positive on "token: iceberg/paimon"

[fix](catalog) remove superseded file-scoped parquet position deletes on iceberg v3 DELETE