zarr: limit sharded write cache to prevent unbounded memory growth#14086
zarr: limit sharded write cache to prevent unbounded memory growth#14086ClayWarren wants to merge 2 commits intoOSGeo:masterfrom
Conversation
…-shard-write-cache zarr: limit sharded write cache to prevent unbounded memory growth
|
A cleaner fix would be to use a lru11::Cache instead of a std::map |
|
The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 28 days and is being automatically marked as "stale". If you think this pull request should be merged, please check
|
|
While we hate to see this happen, this PR has been automatically closed because it has not had any activity in the last 6 weeks. If this pull request should be reconsidered, please follow the guidelines in the previous comment and reopen this pull request. Or, if you have any further questions, just ask! We love to help, and if there's anything the GDAL project can do to help push this PR forward please let us know how we can assist. |
Motivation
m_oShardWriteCache, allowing sparse writes across many shards to grow memory unbounded and enabling a denial-of-service via memory exhaustion.Flush().Description
m_oShardWriteCacheinsideZarrV3Array::FlushDirtyBlockSharded()and flush+evict one entry when the cap is reached to limit memory usage beforeFlush()is called.GDAL_ZARR_V3_MAX_SHARD_CACHE_ENTRIESenvironment/config option, with a default of128and a minimum floor of1to preserve functionality.FlushSingleShard()to encode and write the evicted shard before erasing it from the cache to avoid losing dirty data.frmts/zarr/zarr_v3_array.cppand keeps the original behavior of flushing a shard immediately when all inner chunks are written.Testing
git diff --checkto ensure no trivial whitespace or patch errors and it succeeded.AI tool usage