Skip to content

perf(telemetry): stop copying JSON payloads when converting to Parquet rows - #262

Merged
vishr merged 1 commit into
mainfrom
perf/zero-copy-json-columns
Sep 20, 2026
Merged

vishr merged 1 commit into
mainfrom
perf/zero-copy-json-columns

Conversation

@vishr

@vishr vishr commented Sep 20, 2026

Copy link
Copy Markdown
Member

Fifth item of #254.

Problem

Span, Log and Metric held their JSON columns as []byte, and makeSpanParquetRow and its siblings converted each with string(...):

ResourceJSON: string(r.ResourceJSON), AttributesJSON: string(r.AttributesJSON),
EventsJSON:   string(r.EventsJSON),   LinksJSON:      string(r.LinksJSON),

The originals stay live until the commit is durably acknowledged, so every batch held its JSON twice — and four commit workers run at once, so it was four duplicate copies resident at exactly the moment the process is most likely to be killed.

Change

The fields and the ingest producers that fill them are strings, so conversion assigns rather than copies.

attrsJSON already copied out of its pooled buffer before returning it (append([]byte(nil), buf.Bytes()...)), and buf.String() is the same single allocation — so the producers are no worse. The second copy downstream is what goes away.

Measurement

20,000 spans carrying 8.1 MiB of JSON, measuring the conversion alone:

allocated
before 18.2 MiB
after 9.6 MiB

9.6 MiB is the row-struct headers and nothing else. The JSON is no longer duplicated.

Being straight about the end-to-end number

The commit-level gate does not move outside its own noise. It spans 2.8x-5.6x payload run to run, because parquet-go and zstd pool their buffers, and a saving this size hides inside that spread. Before: 3.24x-5.31x. After: 2.79x-5.64x. Those are the same distribution.

That is why this PR adds TestSpanConversionDoesNotCopyJSONPayloads, which measures the conversion in isolation, where the signal is clean. commitAllocRatioLimit stays at 7.0 - I am not ratcheting it on a change I cannot show through it.

The saving is real and it compounds in production in a way the synthetic test cannot show: the duplicate was held per in-flight commit, across four workers, with more batches queued behind them.

Scope

[]byte to string across internal/telemetry/rows.go, the seven producers in internal/ingest/server.go, parquet_rows.go, their tests, and the experiments/ fixtures that share the module.

No behaviour change. makeSpanParquetRow already turned a nil slice into an empty string, so an absent column is written exactly as before.

just check passes, go vet ./... clean.

…t rows

Span, Log and Metric held their JSON columns as []byte, and
makeSpanParquetRow and its siblings converted each one with string(...)
while the originals stayed live until the commit was durably acknowledged.
Every batch therefore held its JSON twice, and four commit workers run at
once, so it was four duplicate copies resident at the moment the process
is most likely to be killed.

The fields and the ingest producers that fill them are strings now, so the
conversion assigns instead of copying. attrsJSON already copied out of its
pooled buffer before returning, and buf.String() is the same single
allocation, so the producers are no worse; the second copy downstream is
what goes away.

Measured over 20k spans carrying 8.1 MiB of JSON, the conversion allocated
18.2 MiB before and 9.6 MiB after -- the row-struct headers and nothing
else.

The end-to-end commit figure does not move outside its own noise: it
spans 2.8x-5.6x payload run to run because parquet-go and zstd pool their
buffers, and a saving this size hides inside that spread. The new test
measures the conversion alone for that reason. The saving is real, it is
just not visible through the coarser gate, which stays where it is.

No behaviour change: makeSpanParquetRow already turned a nil slice into an
empty string, so a column that was absent is written exactly as before.
@vishr
vishr merged commit 4709aab into main Sep 20, 2026
8 checks passed
@vishr
vishr deleted the perf/zero-copy-json-columns branch September 20, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant