fix(config): warn when a pinned DuckDB budget cannot fit the machine - #261
Merged
Merged
Conversation
Setting storage.duckdb.memory skips detection entirely -- resolveSizing only fills a value that is empty -- so nothing ever compared the figure to the machine it would run on. Both production deployments pinned 4GB on 8 GiB hosts and were OOM-killed repeatedly, and the startup log recorded the pin without comment: duckdb_memory 4GB, detected_memory_bytes 0, memory_source "". DuckDB's budget is not the process's budget. The Go heap, the ingest appender and compaction's merges all allocate on top of it, which is why duckDBMemoryPercent leaves 40% of the machine alone when it sizes itself. A pinned value is now judged by that same standard, and an unparseable one is called out rather than passed through to DuckDB to interpret. This warns rather than refuses. An operator who has measured their workload may legitimately know better than the default share, and failing startup on a configuration that has been running would trade a survivable problem for an outage. Adds parseByteSize for the forms an operator is likely to write -- a bare count, or a decimal with a GB/GiB/MB/MiB/KB/KiB suffix. Note on applyMemoryHeadroom, which this does not touch: it is reachable, contrary to what the ingest-memory issue currently claims. resolveSizing leaves DuckDBMemory empty when detection declines, and on that path DuckDB keeps its own cgroup-aware default and applyMemoryHeadroom clamps it -- exactly the case its comment describes. The issue will be corrected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth item of #254.
Problem
Setting
storage.duckdb.memoryskips detection entirely —resolveSizingonly fills a value that is empty — so nothing ever compared the figure to the machine it would run on.Both production deployments pin
4GBon 8 GiB hosts and have been OOM-killed repeatedly. The startup log recorded the pin without comment:Every field that could have flagged the mismatch reads zero, because pinning the value is what stopped them being populated.
Change
DuckDB's budget is not the process's budget — the Go heap, the ingest appender and compaction's merges all allocate on top of it. That is why
duckDBMemoryPercentleaves 40% of the machine alone when sizing itself. A pinned value is now judged by the same standard, and an unparseable one is called out rather than handed to DuckDB to interpret.It warns, it does not refuse. An operator who has measured their workload may legitimately know better than the default share, and failing startup on a configuration that has been running would trade a survivable problem for an outage.
Adds
parseByteSizefor the forms an operator is likely to write: a bare count, or a decimal with aGB/GiB/MB/MiB/KB/KiBsuffix.Correction to #254
The issue claims
applyMemoryHeadroomis "effectively dead code". That is wrong, and I wrote it. I took it from a review without checking the branch myself.resolveDuckDBMemoryreturns""when detection declines, soDuckDBMemorystays empty on that path, thecfg.DuckDBMemory == ""guard atduck.go:267opens, andapplyMemoryHeadroomclamps DuckDB's own cgroup-aware default. That is exactly the case its comment describes. This PR leaves it alone; I am commenting the correction on the issue.Tests
TestPinnedMemoryConcern— table across pins inside the automatic share, at half the machine, above the share, above the machine, unparseable, absent, and an undetectable machine that cannot be judged either way.TestParseByteSize— the suffix forms, a bare count, a decimal, surrounding whitespace, and the negative and unparseable cases.Both written first and verified failing.
One gap, stated plainly: the
slog.Warncall site itself is not covered. The decision function is fully tested; the wiring is a single call to it, and there is no log-capture helper in this package to lean on.just checkpasses.