perf(telemetry): stop copying JSON payloads when converting to Parquet rows - #262
Merged
Merged
Conversation
…t rows Span, Log and Metric held their JSON columns as []byte, and makeSpanParquetRow and its siblings converted each one with string(...) while the originals stayed live until the commit was durably acknowledged. Every batch therefore held its JSON twice, and four commit workers run at once, so it was four duplicate copies resident at the moment the process is most likely to be killed. The fields and the ingest producers that fill them are strings now, so the conversion assigns instead of copying. attrsJSON already copied out of its pooled buffer before returning, and buf.String() is the same single allocation, so the producers are no worse; the second copy downstream is what goes away. Measured over 20k spans carrying 8.1 MiB of JSON, the conversion allocated 18.2 MiB before and 9.6 MiB after -- the row-struct headers and nothing else. The end-to-end commit figure does not move outside its own noise: it spans 2.8x-5.6x payload run to run because parquet-go and zstd pool their buffers, and a saving this size hides inside that spread. The new test measures the conversion alone for that reason. The saving is real, it is just not visible through the coarser gate, which stays where it is. No behaviour change: makeSpanParquetRow already turned a nil slice into an empty string, so a column that was absent is written exactly as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifth item of #254.
Problem
Span,LogandMetricheld their JSON columns as[]byte, andmakeSpanParquetRowand its siblings converted each withstring(...):The originals stay live until the commit is durably acknowledged, so every batch held its JSON twice — and four commit workers run at once, so it was four duplicate copies resident at exactly the moment the process is most likely to be killed.
Change
The fields and the ingest producers that fill them are strings, so conversion assigns rather than copies.
attrsJSONalready copied out of its pooled buffer before returning it (append([]byte(nil), buf.Bytes()...)), andbuf.String()is the same single allocation — so the producers are no worse. The second copy downstream is what goes away.Measurement
20,000 spans carrying 8.1 MiB of JSON, measuring the conversion alone:
9.6 MiB is the row-struct headers and nothing else. The JSON is no longer duplicated.
Being straight about the end-to-end number
The commit-level gate does not move outside its own noise. It spans 2.8x-5.6x payload run to run, because parquet-go and zstd pool their buffers, and a saving this size hides inside that spread. Before: 3.24x-5.31x. After: 2.79x-5.64x. Those are the same distribution.
That is why this PR adds
TestSpanConversionDoesNotCopyJSONPayloads, which measures the conversion in isolation, where the signal is clean.commitAllocRatioLimitstays at 7.0 - I am not ratcheting it on a change I cannot show through it.The saving is real and it compounds in production in a way the synthetic test cannot show: the duplicate was held per in-flight commit, across four workers, with more batches queued behind them.
Scope
[]bytetostringacrossinternal/telemetry/rows.go, the seven producers ininternal/ingest/server.go,parquet_rows.go, their tests, and theexperiments/fixtures that share the module.No behaviour change.
makeSpanParquetRowalready turned a nil slice into an empty string, so an absent column is written exactly as before.just checkpasses,go vet ./...clean.