You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: configure DataFusion's built-in CacheManager from Java #74
Is your feature request related to a problem or challenge?
The Rust API to enable any of these is RuntimeEnvBuilder::with_cache_manager(CacheManagerConfig). The Java binding exposes none of it — every SessionContext ends up with the no-op defaults (metadata-cache enabled but at 0 size, list-files and stats caches disabled). For a parquet workload reading the same footer thousands of times across queries, that means every footer read goes back to the object store. For a long-lived statistics-driven planner, no stats persist across queries.
There is no Java surface today to plug in or even just turn on the built-in caches.
Describe the solution you'd like
Add a cacheManager(CacheManagerOptions) setter to SessionContextBuilder. The options object mirrors upstream CacheManagerConfig 1:1 — three independent toggles, each with the same knobs upstream exposes:
Java SPI for cache implementations — org.apache.datafusion.cache.{FileMetadataCache, ListFilesCache, FileStatisticsCache} interfaces that callers implement in Java, with arbitrary upcalls into Java for every cache get/put. Rejected for v1. A FileMetadataCache::get runs once per parquet file in a scan; routing those through JNI upcalls turns a hot path into a slow path. The upstream Rust traits are easy to implement; embedders that want a custom cache (LRU on disk, Foyer-backed, network-replicated) can build it Rust-side and ship a fork until the day the cost-benefit of a Java SPI flips. The Out of scope note in CONTRIB_ISSUES.md for #13 ("Pluggable CacheManagerimplementations written in Java") makes this explicit.
Activity