feat: On disk cache for split footers by Darkheir · Pull Request #78 · SekoiaLab/quickwit

Darkheir · 2026-07-08T15:27:13Z

Description

Describe the proposed changes made in this PR.

How was this PR tested?

Describe how you tested this PR.

Copilot

Pull request overview

This PR introduces an optional persistent (on-disk) cache layer for split footers, backing the existing in-memory “hotcache” footer cache so footers can survive searcher restarts and avoid refetching from object storage.

Changes:

Add a disk-backed size-bounded cache (DiskSizedCache) and a two-tier cache wrapper (TieredSizedCache) to combine memory + optional disk tiers.
Wire the new disk footer cache into SearcherContext construction and searcher startup via a new split_footer_disk_cache_capacity config option.
Extend metrics and documentation/config examples to reflect the new persistent cache.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
quickwit/quickwit-storage/src/metrics.rs	Adds metrics bucket for the new disk footer cache.
quickwit/quickwit-storage/src/lib.rs	Re-exports new cache types for downstream crates.
quickwit/quickwit-storage/src/cache/mod.rs	Registers and re-exports the new cache modules.
quickwit/quickwit-storage/src/cache/disk_sized_cache.rs	Implements the on-disk, LRU, size-bounded cache with tests.
quickwit/quickwit-storage/src/cache/tiered_sized_cache.rs	Implements the memory+disk tiered cache wrapper with tests.
quickwit/quickwit-config/src/node_config/mod.rs	Adds `split_footer_disk_cache_capacity` to `SearcherConfig`.
quickwit/quickwit-config/src/node_config/serialize.rs	Updates config serialization tests/defaults for new field.
quickwit/quickwit-serve/src/lib.rs	Opens disk cache under data dir and passes it into `SearcherContext`.
quickwit/quickwit-search/src/service.rs	Updates `SearcherContext` to use `TieredSizedCache` and new constructor arg.
quickwit/quickwit-search/src/leaf.rs	Switches footer cache parameter type to the tiered cache.
quickwit/quickwit-search/src/lib.rs	Updates context construction to new `SearcherContext::new` signature.
quickwit/quickwit-search/src/tests.rs	Updates tests to new `SearcherContext::new` signature.
docs/overview/concepts/querying.md	Documents the new split footer disk cache conceptually.
docs/configuration/node-config.md	Documents the new searcher configuration key.
config/quickwit.yaml	Adds commented example for enabling the new setting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Darkheir · 2026-07-08T15:36:17Z

+
+    /// Returns the cached payload for the given key, if present on disk.
+    pub fn get(&self, key: &K) -> Option<OwnedBytes> {
+        let file_name = key.to_string();


We are not concerned by this issue in the current PR. Should I still add this defensive check in case the cache is re-used later ?

Darkheir · 2026-07-08T15:36:24Z

+    /// recency is refreshed.
+    pub fn put(&self, key: K, bytes: OwnedBytes) {
+        let num_bytes = bytes.len() as u64;
+        let file_name = key.to_string();


We are not concerned by this issue in the current PR. Should I still add this defensive check in case the cache is re-used later ?

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

rdettai-sk · 2026-07-08T16:26:07Z

+        for dir_entry_res in std::fs::read_dir(&root_path)? {
+            let dir_entry = dir_entry_res?;
+            let metadata = match dir_entry.metadata() {


This is calling stat on all files. It should be fast on an ssd (tried it locally, 150ms). I do have some concerns on EBS. Also this will also hold up the actor loop (never yelds). I assume it's okay on a fast disk as it's called only once. I think it can have bad side effects if the disk is low.

We have the same logic for the splits cache: https://github.com/SekoiaLab/quickwit/blob/sekoia/quickwit/quickwit-storage/src/split_cache/mod.rs#L63-L66

The test I performed was on 100k files. Actually looking a little bit further back in time, the number of entries in the current cache goes up to 220k per searcher. Given that this cache could be configured bigger and that it will never get wiped until the limit is reached, I think we can expect it to go up to millions.

To avoid blocking the loop we could make open async and iterate inside spawn_blocking.

Regarding all the metadata calls I'm not sure we should remove them. The alternative of having a single manifest file feels a bit tricky because it adds write overhead and is problematic for crash consistency while the metadata calls are made only on startup

If we want to avoid making all the stat calls we could store the size of the files in their name so we would have all the required info.

We would also lose the seed order for the LRUCache that is using mtime, but this is flawed anyway because it doesn't track the last time the footer was accessed, only the time it was created.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

rdettai-sk

Looks good! The tiered approach is just a bit unfortunate in that it uses 2 locks / 2 states when theoretically only 1 should be enough. But I think contention should remain small as the runtime doesn't have many threads. Also eviction on a two tier state would not be easy.

rdettai-sk · 2026-07-09T13:34:25Z

+        };
+        if !victims.is_empty() {
+            let root_path = self.root_path.clone();
+            let _ = tokio::task::spawn_blocking(move || remove_files(&root_path, &victims)).await;


I think we don't need to await this

I think we could end in a strange state if the key is added again before being removed

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

Copilot AI review requested due to automatic review settings July 8, 2026 15:27

Copilot started reviewing on behalf of Darkheir July 8, 2026 15:27 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Darkheir force-pushed the feat/split_footer_on_disk_cache branch from c6e7050 to 7440c8f Compare July 8, 2026 15:35

feat: On disk cache for split footers

ecd01e9

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

Darkheir force-pushed the feat/split_footer_on_disk_cache branch from 7440c8f to ecd01e9 Compare July 8, 2026 15:38

feat: Async disk cache access

303dec6

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

Darkheir requested a review from Copilot July 8, 2026 15:52

Copilot started reviewing on behalf of Darkheir July 8, 2026 15:52 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread quickwit/quickwit-storage/src/cache/disk_sized_cache.rs

Comment thread quickwit/quickwit-storage/src/cache/disk_sized_cache.rs

Comment thread quickwit/quickwit-storage/src/cache/tiered_sized_cache.rs

rdettai-sk reviewed Jul 8, 2026

View reviewed changes

Darkheir force-pushed the feat/split_footer_on_disk_cache branch from 6e10f50 to bf2d309 Compare July 8, 2026 16:27

Apply suggestions from code review

aa723c2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Darkheir force-pushed the feat/split_footer_on_disk_cache branch from bf2d309 to aa723c2 Compare July 8, 2026 17:02

Darkheir added 2 commits July 9, 2026 10:08

feat: Async disk cache open

9a1d10f

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

feat: Sub-folder sharding

44dffd9

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

Darkheir requested a review from rdettai-sk July 9, 2026 12:44

rdettai-sk approved these changes Jul 9, 2026

View reviewed changes

feat: Apply PR review suggestion

0b228a7

Signed-off-by: Darkheir <raphael.cohen@sekoia.io>

Darkheir merged commit 1f226bd into sekoia Jul 10, 2026
3 checks passed

Uh oh!

Conversation

Darkheir commented Jul 8, 2026

Description

How was this PR tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdettai-sk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants