Skip to content

zarr: limit sharded write cache to prevent unbounded memory growth#14086

Closed
ClayWarren wants to merge 2 commits intoOSGeo:masterfrom
ClayWarren:master
Closed

zarr: limit sharded write cache to prevent unbounded memory growth#14086
ClayWarren wants to merge 2 commits intoOSGeo:masterfrom
ClayWarren:master

Conversation

@ClayWarren
Copy link
Copy Markdown

Motivation

  • The sharded write path previously retained a full decoded buffer per touched shard in m_oShardWriteCache, allowing sparse writes across many shards to grow memory unbounded and enabling a denial-of-service via memory exhaustion.
  • The intent of the change is to bound in-memory decoded shards during sharded writes while preserving existing semantics that encode each shard once on Flush().

Description

  • Enforce a configurable cap on the number of decoded shard entries kept in m_oShardWriteCache inside ZarrV3Array::FlushDirtyBlockSharded() and flush+evict one entry when the cap is reached to limit memory usage before Flush() is called.
  • Make the cap configurable via the GDAL_ZARR_V3_MAX_SHARD_CACHE_ENTRIES environment/config option, with a default of 128 and a minimum floor of 1 to preserve functionality.
  • The eviction path calls the existing FlushSingleShard() to encode and write the evicted shard before erasing it from the cache to avoid losing dirty data.
  • Change is localized to frmts/zarr/zarr_v3_array.cpp and keeps the original behavior of flushing a shard immediately when all inner chunks are written.

Testing

  • Ran git diff --check to ensure no trivial whitespace or patch errors and it succeeded.
  • The change is a minimal patch designed to be low-risk and ready for CI validation.

AI tool usage

  • AI (Codex Security in research preview) supported my development of this PR.

…-shard-write-cache

zarr: limit sharded write cache to prevent unbounded memory growth
@rouault
Copy link
Copy Markdown
Member

rouault commented Mar 7, 2026

A cleaner fix would be to use a lru11::Cache instead of a std::map

CC @wietzesuijker

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 5, 2026

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 28 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

  • that all unit tests are passing

  • that all comments by reviewers have been addressed

  • that there is enough information for reviewers, in particular link
    to any issues which this pull request fixes

  • that you have written unit tests where possible
    In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
    If there is no further activity on this pull request, it will be closed in 2 weeks.

@github-actions github-actions Bot added the stale label Apr 5, 2026
@github-actions
Copy link
Copy Markdown

While we hate to see this happen, this PR has been automatically closed because it has not had any activity in the last 6 weeks. If this pull request should be reconsidered, please follow the guidelines in the previous comment and reopen this pull request. Or, if you have any further questions, just ask! We love to help, and if there's anything the GDAL project can do to help push this PR forward please let us know how we can assist.

@github-actions github-actions Bot closed this Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants