Repository navigation
perf(engine): cache Parquet footers on read connections - #497
Merged
Merged
Conversation
The engine binds an explicit per-file Parquet list on every read statement and a panel issues several, so with parquet_metadata_cache off DuckDB re-read each file's footer on every statement. parquet_metadata_cache is a GLOBAL-scoped option that the duckdb-go driver applies only to the connection that issues the SET, so enabling it once at boot does not reach the pooled read connections. It is now set on each connection in the init hook, next to the UTC setting, and kept in the policy allowed_configs so it still applies on connections opened after lock_configuration. The lockdown is unchanged: the cache only holds immutable Parquet footers and cannot widen file or external access. Measured with an engine-internal A/B over immutable fixtures (same files, one pinned read transaction, AB/BA alternation, 100 samples per arm). Median read latency drops with the cache on: 128 files: 11.83 -> 10.69 ms (-9.6%) 512 files: 33.02 -> 31.02 ms (-6.1%) 1024 files: 62.67 -> 58.61 ms (-6.5%) p95 and p99 improve similarly at 512 and 1024 files. The footer cache costs bounded memory (about 150 MB of OBJECT_CACHE at 1024 fixture files). It keys on path, mtime and size, and our batch paths are unique and immutable (uuid ingest ids and compact-<nanos> compaction ids; retired paths are deleted, never overwritten in place), so a cached footer can never be stale. Closes #484
vishr
enabled auto-merge
November 16, 2026 01:45
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
parquet_metadata_cachewas never set, so it kept its engine default offalse. Every read statement binds an explicit per-file Parquet list, and a panel issues several statements, so DuckDB re-read each file's footer per statement instead of reusing it.Closes #484.
Measurement first
This issue called for measurement before any change. Engine-internal A/B benchmark, same immutable fixtures, one pinned read transaction, AB/BA alternation to control the OS page cache, 100 samples per arm — median read latency, cache OFF → ON:
p95/p99 also improve at the larger file counts (1024 p99 70.5 → 62.0 ms). The win is consistent and monotonic and robust to ordering, matching the setting's purpose (reading the same files repeatedly). Memory cost is noted: the footer cache grows to ~150 MB at 1024 fixture files; it is bounded footer caching under DuckDB's memory accounting (not unbounded read-ahead), and active file counts are bounded by compaction.
Scope finding (verified on the pinned DuckDB 2.0.0 engine, not assumed)
parquet_metadata_cacheis global-scoped, default false. But on the shared on-disk DSN the engine uses across its read/write/memory/parse pools, aSETon one connection does not propagate to other pooled connections, and the DSN/instance-config route is rejected because the parquet extension loads later. So it must be set per connection, in the same init hook that setsSET TimeZone='UTC', and added toallowed_configsso theSETstill succeeds on connections opened afterlock_configuration=true.Change
internal/engine/duck.go— the per-connection init hook now runsSET parquet_metadata_cache=trueright after the UTC set.internal/duckdb/policy.go—allowed_configsgainsparquet_metadata_cache;lock_configuration=trueand the file/external-access lockdown are unchanged.Correctness — no stale-cache risk
The cache keys on path + mtime + size. Batch paths are unique and never reused: ingest IDs are UUIDs, compaction output IDs are
compact-<nanos>; files are immutable once published, compaction writes new paths, and retired inputs are deleted, never overwritten in place. So a cached footer can never serve a different file.Tests
TestReadConnectionsEnableParquetMetadataCacheasserts the setting istrueon a freshly opened (post-lock) connection of all four pools and that the lock still refusesSET enable_external_access(lockdown intact). A//go:build filebenchA/B harness (metadata_cache_bench_test.go) is the measurement tool.Verification
just test "./internal/engine/..."and"./internal/query/...",just vet, andjust check-static(gofmt, deadcode, behavioral test names) all green; public arbitrary SQL still cannot change file or external access.