Repository navigation
Description
Symptom
The process is OOM-killed by the kernel at ~7.4-7.5 GiB anon-rss. Both deployments are affected; they run the same image digest.
Production, 8 kills in 17 days, accelerating to roughly daily:
[Wed Sep 2 07:45:31 2026] Out of memory: Killed process 1980535 (fanout) anon-rss:7379468kB
[Fri Sep 11 12:49:19 2026] Out of memory: Killed process 3246221 (fanout) anon-rss:7519036kB
[Sun Sep 13 06:00:41 2026] Out of memory: Killed process 164900 (fanout) anon-rss:7447816kB
[Mon Sep 14 14:23:22 2026] Out of memory: Killed process 987248 (fanout) anon-rss:7456236kB
[Tue Sep 15 14:32:06 2026] Out of memory: Killed process 1632330 (fanout) anon-rss:7502852kB
[Wed Sep 16 17:57:43 2026] Out of memory: Killed process 2115459 (fanout) anon-rss:7508416kB
[Fri Sep 18 01:35:37 2026] Out of memory: Killed process 2664784 (fanout) anon-rss:7417464kB
[Sat Sep 19 13:57:10 2026] Out of memory: Killed process 3292339 (fanout) anon-rss:7445156kB
The kill is constraint=CONSTRAINT_NONE ... global_oom — no cgroup limit on the container, so the process takes the whole VM with it. On the demo host the proxy was killed alongside it and sshd could not fork, leaving the box unreachable behind a Cloudflare 524 rather than simply restarting.
Each kill is a SIGKILL, not the configured 30s TERM drain, so anything buffered between the OTLP listener and the lake writer is lost.
Shape
RSS sampled once a minute for 90 minutes across both hosts:
production peak 6.78 GiB, avg 2.79 GiB (7.68 GiB VM)
demo peak 10.24 GiB, avg 7.59 GiB (11.6 GiB VM, raised from 7.68 during the run)
This is a large-amplitude sawtooth, not a leak and not monotonic growth. Memory is released each cycle. Peak amplitude scales with ingest rate: the heavily-ingesting demo peaks at 10.2 GiB, production at 6.8 GiB. An OOM happens when one peak clears the box.
The excursions occur with no query traffic, which places them on the write path.
Raising the demo VM to 12 GiB kept it alive through peaks that would have killed an 8 GiB box several times over, but it still runs at ~88% of the larger VM. Capacity buys headroom; it does not reduce the peak.
Root causes
Measured with a throwaway in-package probe (50k synthetic spans, 41.8 MiB raw JSON and strings, peak HeapInuse sampled every 2ms). These figures are not currently reproducible in-tree — see Verification.
1. Parquet dictionary encoding on near-unique columns. attributes_json, events_json and links_json carry ,dict (internal/telemetry/parquet_rows.go:20-22), and are near-unique per row. parquet-go holds the column-chunk dictionary for the entire row group, falling back to PLAIN only past 1 GiB, and a row group is not flushed until MaxRowsPerRowGroup (50,000, parquet.go:29). So each ingest file is one fully-materialised row group whose dictionary is effectively a third copy of the payload.
| Configuration | Encoder peak | Encode time |
|---|---|---|
50k rows, ,dict |
136 MiB (3.25x raw) | 178 ms |
| 50k rows, PLAIN json columns | 44 MiB (1.06x raw) | 95 ms |
2. Compaction's sorted merge, at least as large as ingest. mergeTypedParquet (parquet.go:1181-1245) appends every row group of every input into one k-way merge; parquet-go opens a cursor per row group and reads from all of them during initialisation, so every input's dictionary is resident simultaneously. Measured at ~40-48 MiB per 50k-span input, linear — 622 MiB for 16 inputs. Selection admits up to 128 inputs (internal/query/duck.go:104) and 25M rows, so a generation-0 pass can reach ~5 GiB live. It runs every 10s with an 8s budget, and the budget is only checked between compactions (compaction.go:255-260), so an oversized merge always starts.
3. No Go memory limit. There is no GOMEMLIMIT or debug.SetMemoryLimit anywhere in the repository. Whole-commit peak for 50k spans measures 196 MiB at GOGC=5 versus 373 MiB at GOGC=100 — under production GOGC roughly half the Go RSS is GC slack.
4. Conversion copy. makeSpanParquetRow (parquet_rows.go:50) does string(r.ResourceJSON) and the same for attributes, events and links, while the originals stay live until durable ack. ~62 MiB live per 50k spans, of which ~38 MiB is string copies.
5. Bytes are unbounded even though rows are not. Rows are capped at 50,000 (store/writer.go:22), but nothing caps a batch by size, and nothing bounds in-flight handlers — concurrency is HTTP/2 streams times connections, each holding its decoded batch and proto until ack.
Related defects found alongside
Compaction restart loop. An OOM mid-merge writes no marker — the marker is written after PrepareReplacement (compaction.go:134-141) — so a clean restart reselects the same group and dies identically. This fits production's repeating daily kill better than ingest does, since production's 20ms admission windows produce small batches.
applyMemoryHeadroom is dead code. It runs only when cfg.DuckDBMemory == "" (internal/query/duck.go:267), but resolveSizing (internal/config/sizing.go:79-88) always fills that value when detection succeeds. The guard never opens.
A pinned FANOUT_DUCKDB_MEMORY is never validated. Setting it skips detectMemory() entirely (sizing.go:78); nothing checks the value against machine size. Both deployments pin 4GB on 8 GiB hosts, which is roughly the maximum that could fit even before the Go side is counted. Confirmed in production logs: "duckdb_memory":"4GB","duckdb_memory_auto":false,"detected_memory_bytes":0,"memory_source":"".
The sizing constant is calibrated against the wrong figure. duckDBMemoryPercent = 60 reserves 40% for Go based on a stated "~1.3 GB of Go runtime" (sizing.go:18-29). Under sustained ingest the Go side measures 3-4 GiB.
Proposed work, in order
- Drop
,dictfrom the near-unique JSON columns (parquet_rows.go:20-22,:39,:91,:125-127). Keep it onresource_json,scope_*,http_*andhist_*, which do repeat. Tag-only; DuckDB reads either encoding, so no migration. Costs roughly 24% ingest file size, which compaction re-encodes anyway. debug.SetMemoryLimit(detected - DuckDB budget - reserve)at startup, reusingdetectMemory. Removes the GOGC doubling.- Cap compaction input by cumulative on-disk bytes in
selectBoundedCompactionGroup(compaction.go:188-230), which currently admits by row and file count only. Re-measure the expansion factor after (1); it drops from ~13x file size to ~3.3x. - Change the JSON fields on
Span/Log/Metric(internal/telemetry/rows.go) from[]bytetostringso conversion stops copying. - Add a byte-weighted batch limit alongside
maxGroupBatchRows. - Admission control on wire bytes in the Export handlers, rejecting with
RESOURCE_EXHAUSTED/503 when over budget. This is the only item that makes the Go side bounded rather than merely smaller, and it changes behaviour under load — worth deciding separately.
Then recalibrate duckDBMemoryPercent against the new figures, and fix the three sizing defects above.
Considered and rejected: moving encoding under the publish gate, which would contradict the design note at parquet.go:171-174; and reducing commit workers, which costs throughput without addressing the per-batch multiplier.
Verification
There is no in-tree benchmark for any of this. just stress throughput is not in this checkout — config.go:44-48 refers to an external fanout-bench harness. The measurements above came from a probe that no longer exists.
First task should be a permanent peak-heap benchmark in internal/telemetry (sampler over HeapInuse plus b.ReportMetric), which serves as both the failing test for item (1) and the regression gate for the rest. In production, /-/metrics already exposes go_memstats_heap_inuse_bytes, go_memstats_heap_sys_bytes and process_resident_memory_bytes on the default registry; sampling those at a spike turns the "RSS minus DuckDB budget equals Go heap" inference into a measured fact. FANOUT_PPROF_ENABLED enables the heap profile.
Activity
vishr commentedon Sep 20, 2026
Correction to this issue, under Related defects found alongside.
The claim that
applyMemoryHeadroomis "effectively dead code" is wrong. I carried it over from a review without verifying it myself.resolveSizingfillsDuckDBMemoryonly when detection succeeds. When detection declines,resolveDuckDBMemoryreturns"",DuckDBMemorystays empty, thecfg.DuckDBMemory == ""guard atinternal/query/duck.go:267opens, andapplyMemoryHeadroomclamps DuckDB's own cgroup-aware default. That is precisely the case its comment describes, and it is reachable.The other two sizing defects in that section stand:
FANOUT_DUCKDB_MEMORYis never validated against the machine — fixed in fix(config): warn when a pinned DuckDB budget cannot fit the machine #261duckDBMemoryPercentreserves 40% against a stated "~1.3 GB of Go runtime" while the measured Go side is 3-4 GiB — still open, and deliberately left until the working-set reductions land, since they change the number