Symptom
The process is OOM-killed by the kernel at ~7.4-7.5 GiB anon-rss. Both deployments are affected; they run the same image digest.
Production, 8 kills in 17 days, accelerating to roughly daily:
[Wed Sep 2 07:45:31 2026] Out of memory: Killed process 1980535 (fanout) anon-rss:7379468kB
[Fri Sep 11 12:49:19 2026] Out of memory: Killed process 3246221 (fanout) anon-rss:7519036kB
[Sun Sep 13 06:00:41 2026] Out of memory: Killed process 164900 (fanout) anon-rss:7447816kB
[Mon Sep 14 14:23:22 2026] Out of memory: Killed process 987248 (fanout) anon-rss:7456236kB
[Tue Sep 15 14:32:06 2026] Out of memory: Killed process 1632330 (fanout) anon-rss:7502852kB
[Wed Sep 16 17:57:43 2026] Out of memory: Killed process 2115459 (fanout) anon-rss:7508416kB
[Fri Sep 18 01:35:37 2026] Out of memory: Killed process 2664784 (fanout) anon-rss:7417464kB
[Sat Sep 19 13:57:10 2026] Out of memory: Killed process 3292339 (fanout) anon-rss:7445156kB
The kill is constraint=CONSTRAINT_NONE ... global_oom — no cgroup limit on the container, so the process takes the whole VM with it. On the demo host the proxy was killed alongside it and sshd could not fork, leaving the box unreachable behind a Cloudflare 524 rather than simply restarting.
Each kill is a SIGKILL, not the configured 30s TERM drain, so anything buffered between the OTLP listener and the lake writer is lost.
Shape
RSS sampled once a minute for 90 minutes across both hosts:
production peak 6.78 GiB, avg 2.79 GiB (7.68 GiB VM)
demo peak 10.24 GiB, avg 7.59 GiB (11.6 GiB VM, raised from 7.68 during the run)
This is a large-amplitude sawtooth, not a leak and not monotonic growth. Memory is released each cycle. Peak amplitude scales with ingest rate: the heavily-ingesting demo peaks at 10.2 GiB, production at 6.8 GiB. An OOM happens when one peak clears the box.
The excursions occur with no query traffic, which places them on the write path.
Raising the demo VM to 12 GiB kept it alive through peaks that would have killed an 8 GiB box several times over, but it still runs at ~88% of the larger VM. Capacity buys headroom; it does not reduce the peak.
Root causes
Measured with a throwaway in-package probe (50k synthetic spans, 41.8 MiB raw JSON and strings, peak HeapInuse sampled every 2ms). These figures are not currently reproducible in-tree — see Verification.
1. Parquet dictionary encoding on near-unique columns. attributes_json, events_json and links_json carry ,dict (internal/telemetry/parquet_rows.go:20-22), and are near-unique per row. parquet-go holds the column-chunk dictionary for the entire row group, falling back to PLAIN only past 1 GiB, and a row group is not flushed until MaxRowsPerRowGroup (50,000, parquet.go:29). So each ingest file is one fully-materialised row group whose dictionary is effectively a third copy of the payload.
| Configuration |
Encoder peak |
Encode time |
50k rows, ,dict |
136 MiB (3.25x raw) |
178 ms |
| 50k rows, PLAIN json columns |
44 MiB (1.06x raw) |
95 ms |
2. Compaction's sorted merge, at least as large as ingest. mergeTypedParquet (parquet.go:1181-1245) appends every row group of every input into one k-way merge; parquet-go opens a cursor per row group and reads from all of them during initialisation, so every input's dictionary is resident simultaneously. Measured at ~40-48 MiB per 50k-span input, linear — 622 MiB for 16 inputs. Selection admits up to 128 inputs (internal/query/duck.go:104) and 25M rows, so a generation-0 pass can reach ~5 GiB live. It runs every 10s with an 8s budget, and the budget is only checked between compactions (compaction.go:255-260), so an oversized merge always starts.
3. No Go memory limit. There is no GOMEMLIMIT or debug.SetMemoryLimit anywhere in the repository. Whole-commit peak for 50k spans measures 196 MiB at GOGC=5 versus 373 MiB at GOGC=100 — under production GOGC roughly half the Go RSS is GC slack.
4. Conversion copy. makeSpanParquetRow (parquet_rows.go:50) does string(r.ResourceJSON) and the same for attributes, events and links, while the originals stay live until durable ack. ~62 MiB live per 50k spans, of which ~38 MiB is string copies.
5. Bytes are unbounded even though rows are not. Rows are capped at 50,000 (store/writer.go:22), but nothing caps a batch by size, and nothing bounds in-flight handlers — concurrency is HTTP/2 streams times connections, each holding its decoded batch and proto until ack.
Related defects found alongside
Compaction restart loop. An OOM mid-merge writes no marker — the marker is written after PrepareReplacement (compaction.go:134-141) — so a clean restart reselects the same group and dies identically. This fits production's repeating daily kill better than ingest does, since production's 20ms admission windows produce small batches.
applyMemoryHeadroom is dead code. It runs only when cfg.DuckDBMemory == "" (internal/query/duck.go:267), but resolveSizing (internal/config/sizing.go:79-88) always fills that value when detection succeeds. The guard never opens.
A pinned FANOUT_DUCKDB_MEMORY is never validated. Setting it skips detectMemory() entirely (sizing.go:78); nothing checks the value against machine size. Both deployments pin 4GB on 8 GiB hosts, which is roughly the maximum that could fit even before the Go side is counted. Confirmed in production logs: "duckdb_memory":"4GB","duckdb_memory_auto":false,"detected_memory_bytes":0,"memory_source":"".
The sizing constant is calibrated against the wrong figure. duckDBMemoryPercent = 60 reserves 40% for Go based on a stated "~1.3 GB of Go runtime" (sizing.go:18-29). Under sustained ingest the Go side measures 3-4 GiB.
Proposed work, in order
- Drop
,dict from the near-unique JSON columns (parquet_rows.go:20-22, :39, :91, :125-127). Keep it on resource_json, scope_*, http_* and hist_*, which do repeat. Tag-only; DuckDB reads either encoding, so no migration. Costs roughly 24% ingest file size, which compaction re-encodes anyway.
debug.SetMemoryLimit(detected - DuckDB budget - reserve) at startup, reusing detectMemory. Removes the GOGC doubling.- Cap compaction input by cumulative on-disk bytes in
selectBoundedCompactionGroup (compaction.go:188-230), which currently admits by row and file count only. Re-measure the expansion factor after (1); it drops from ~13x file size to ~3.3x.
- Change the JSON fields on
Span/Log/Metric (internal/telemetry/rows.go) from []byte to string so conversion stops copying.
- Add a byte-weighted batch limit alongside
maxGroupBatchRows.
- Admission control on wire bytes in the Export handlers, rejecting with
RESOURCE_EXHAUSTED/503 when over budget. This is the only item that makes the Go side bounded rather than merely smaller, and it changes behaviour under load — worth deciding separately.
Then recalibrate duckDBMemoryPercent against the new figures, and fix the three sizing defects above.
Considered and rejected: moving encoding under the publish gate, which would contradict the design note at parquet.go:171-174; and reducing commit workers, which costs throughput without addressing the per-batch multiplier.
Verification
There is no in-tree benchmark for any of this. just stress throughput is not in this checkout — config.go:44-48 refers to an external fanout-bench harness. The measurements above came from a probe that no longer exists.
First task should be a permanent peak-heap benchmark in internal/telemetry (sampler over HeapInuse plus b.ReportMetric), which serves as both the failing test for item (1) and the regression gate for the rest. In production, /-/metrics already exposes go_memstats_heap_inuse_bytes, go_memstats_heap_sys_bytes and process_resident_memory_bytes on the default registry; sampling those at a spike turns the "RSS minus DuckDB budget equals Go heap" inference into a measured fact. FANOUT_PPROF_ENABLED enables the heap profile.
Symptom
The process is OOM-killed by the kernel at ~7.4-7.5 GiB
anon-rss. Both deployments are affected; they run the same image digest.Production, 8 kills in 17 days, accelerating to roughly daily:
The kill is
constraint=CONSTRAINT_NONE ... global_oom— no cgroup limit on the container, so the process takes the whole VM with it. On the demo host the proxy was killed alongside it and sshd could not fork, leaving the box unreachable behind a Cloudflare 524 rather than simply restarting.Each kill is a
SIGKILL, not the configured 30sTERMdrain, so anything buffered between the OTLP listener and the lake writer is lost.Shape
RSS sampled once a minute for 90 minutes across both hosts:
This is a large-amplitude sawtooth, not a leak and not monotonic growth. Memory is released each cycle. Peak amplitude scales with ingest rate: the heavily-ingesting demo peaks at 10.2 GiB, production at 6.8 GiB. An OOM happens when one peak clears the box.
The excursions occur with no query traffic, which places them on the write path.
Raising the demo VM to 12 GiB kept it alive through peaks that would have killed an 8 GiB box several times over, but it still runs at ~88% of the larger VM. Capacity buys headroom; it does not reduce the peak.
Root causes
Measured with a throwaway in-package probe (50k synthetic spans, 41.8 MiB raw JSON and strings, peak
HeapInusesampled every 2ms). These figures are not currently reproducible in-tree — see Verification.1. Parquet dictionary encoding on near-unique columns.
attributes_json,events_jsonandlinks_jsoncarry,dict(internal/telemetry/parquet_rows.go:20-22), and are near-unique per row. parquet-go holds the column-chunk dictionary for the entire row group, falling back to PLAIN only past 1 GiB, and a row group is not flushed untilMaxRowsPerRowGroup(50,000,parquet.go:29). So each ingest file is one fully-materialised row group whose dictionary is effectively a third copy of the payload.,dict2. Compaction's sorted merge, at least as large as ingest.
mergeTypedParquet(parquet.go:1181-1245) appends every row group of every input into one k-way merge; parquet-go opens a cursor per row group and reads from all of them during initialisation, so every input's dictionary is resident simultaneously. Measured at ~40-48 MiB per 50k-span input, linear — 622 MiB for 16 inputs. Selection admits up to 128 inputs (internal/query/duck.go:104) and 25M rows, so a generation-0 pass can reach ~5 GiB live. It runs every 10s with an 8s budget, and the budget is only checked between compactions (compaction.go:255-260), so an oversized merge always starts.3. No Go memory limit. There is no
GOMEMLIMITordebug.SetMemoryLimitanywhere in the repository. Whole-commit peak for 50k spans measures 196 MiB at GOGC=5 versus 373 MiB at GOGC=100 — under production GOGC roughly half the Go RSS is GC slack.4. Conversion copy.
makeSpanParquetRow(parquet_rows.go:50) doesstring(r.ResourceJSON)and the same for attributes, events and links, while the originals stay live until durable ack. ~62 MiB live per 50k spans, of which ~38 MiB is string copies.5. Bytes are unbounded even though rows are not. Rows are capped at 50,000 (
store/writer.go:22), but nothing caps a batch by size, and nothing bounds in-flight handlers — concurrency is HTTP/2 streams times connections, each holding its decoded batch and proto until ack.Related defects found alongside
Compaction restart loop. An OOM mid-merge writes no marker — the marker is written after
PrepareReplacement(compaction.go:134-141) — so a clean restart reselects the same group and dies identically. This fits production's repeating daily kill better than ingest does, since production's 20ms admission windows produce small batches.applyMemoryHeadroomis dead code. It runs only whencfg.DuckDBMemory == ""(internal/query/duck.go:267), butresolveSizing(internal/config/sizing.go:79-88) always fills that value when detection succeeds. The guard never opens.A pinned
FANOUT_DUCKDB_MEMORYis never validated. Setting it skipsdetectMemory()entirely (sizing.go:78); nothing checks the value against machine size. Both deployments pin4GBon 8 GiB hosts, which is roughly the maximum that could fit even before the Go side is counted. Confirmed in production logs:"duckdb_memory":"4GB","duckdb_memory_auto":false,"detected_memory_bytes":0,"memory_source":"".The sizing constant is calibrated against the wrong figure.
duckDBMemoryPercent = 60reserves 40% for Go based on a stated "~1.3 GB of Go runtime" (sizing.go:18-29). Under sustained ingest the Go side measures 3-4 GiB.Proposed work, in order
,dictfrom the near-unique JSON columns (parquet_rows.go:20-22,:39,:91,:125-127). Keep it onresource_json,scope_*,http_*andhist_*, which do repeat. Tag-only; DuckDB reads either encoding, so no migration. Costs roughly 24% ingest file size, which compaction re-encodes anyway.debug.SetMemoryLimit(detected - DuckDB budget - reserve)at startup, reusingdetectMemory. Removes the GOGC doubling.selectBoundedCompactionGroup(compaction.go:188-230), which currently admits by row and file count only. Re-measure the expansion factor after (1); it drops from ~13x file size to ~3.3x.Span/Log/Metric(internal/telemetry/rows.go) from[]bytetostringso conversion stops copying.maxGroupBatchRows.RESOURCE_EXHAUSTED/503 when over budget. This is the only item that makes the Go side bounded rather than merely smaller, and it changes behaviour under load — worth deciding separately.Then recalibrate
duckDBMemoryPercentagainst the new figures, and fix the three sizing defects above.Considered and rejected: moving encoding under the publish gate, which would contradict the design note at
parquet.go:171-174; and reducing commit workers, which costs throughput without addressing the per-batch multiplier.Verification
There is no in-tree benchmark for any of this.
just stress throughputis not in this checkout —config.go:44-48refers to an externalfanout-benchharness. The measurements above came from a probe that no longer exists.First task should be a permanent peak-heap benchmark in
internal/telemetry(sampler overHeapInuseplusb.ReportMetric), which serves as both the failing test for item (1) and the regression gate for the rest. In production,/-/metricsalready exposesgo_memstats_heap_inuse_bytes,go_memstats_heap_sys_bytesandprocess_resident_memory_byteson the default registry; sampling those at a spike turns the "RSS minus DuckDB budget equals Go heap" inference into a measured fact.FANOUT_PPROF_ENABLEDenables the heap profile.