perf(telemetry): stop dictionary-encoding near-unique JSON columns

[vishr]

First item of #254.

Problem

Parquet dictionary encoding holds every distinct value of a column chunk for the whole row group, and ingest flushes a row group only at parquetRowGroupRows (50,000, parquet.go:29). That is a saving on a column whose values repeat. It is a second full copy of the payload on a column whose values do not.

Five columns are near-unique per row by construction and were dictionary-encoded anyway:

• attributes_json (spans, logs, metrics)
• events_json, links_json (spans)
• exemplars_json (metrics)
• exception_message (spans)

Columns that genuinely repeat — resource_json, scope_*, http_*, hist_*, severity, service — keep their dictionaries.

Measurement

Peak HeapInuse during one CommitBatch, 50,000 synthetic spans, 23.7 MiB payload, sampled every 2ms with GOGC pinned low, repeated runs under -race:

Encoding	Peak / payload
dictionary (before)	4.09x – 4.92x
plain (after)	2.79x – 3.49x

Roughly 40 MiB less live heap per in-flight commit. Four commit workers run concurrently with more batches queued behind them, so this multiplies across the process.

What this does not fix

It makes one commit cheaper, not cheap. What remains above 1x:

• the conversion copy in makeSpanParquetRow — string(r.ResourceJSON) and friends, while the originals stay live until durable ack
• ~24 MiB of row-struct headers per 50k spans
• the writer's own page and compression buffers

Each is tracked separately in #254. Compaction's k-way merge, which is likely the larger allocator, is untouched here.

Test

TestCommitBatchPeakHeapStaysNearPayloadSize samples heap during a commit and fails above a ratio placed in the gap between the two measured distributions above, rather than at a guessed number. It was written first and verified to fail against the dictionary encoding — restoring the old tags reproduces ratio 4.92x and a red test.

The constant is documented as a ratchet: lower it as each remaining item in #254 lands.

BenchmarkCommitBatch reports peak-heap-MiB and peak/payload so a regression shows up as a number. There was no in-tree memory benchmark before this — the note in config.go:44-48 points at an external fanout-bench harness that is not in this checkout.

Compatibility

None needed. DuckDB reads either encoding, and parquet-go compares schema by node rather than by encoding, so existing files and in-flight merges are unaffected. Ingest files grow modestly; compaction re-encodes them anyway.

just check passes.