perf(store): bound a compaction pass by the bytes it admits

[vishr]

Second item of #254. Follows #255.

Problem

mergeTypedParquet appends every row group of every input into one k-way merge, and parquet-go opens a cursor per row group and reads from all of them during initialisation. Every input's dictionaries are resident simultaneously, so the live cost of a pass tracks the bytes it admits.

selectBoundedCompactionGroup bounded rows (25M) and file count. Neither predicts bytes. A batch of wide spans carrying large attribute payloads and a batch of bare log lines can hold identical row counts and differ by an order of magnitude on disk.

With up to 128 inputs admissible (internal/query/duck.go:104), a generation-0 pass could hold several gigabytes live. That makes compaction a strong candidate for the dominant allocator in the process — plausibly larger than ingest, and notably it fits the production kill pattern better, since production's 20ms admission windows produce small ingest batches but the same compaction passes.

Change

A 256 MiB ceiling on admitted bytes, alongside the existing row and count ceilings.

A pair is always admitted, however large. A merge of two inputs is bounded by those two, and refusing it would strand files that are individually over budget instead of ever shrinking them. That is the same stranding the row ceiling's bounded-group behaviour already exists to avoid — the comment on selectBoundedCompactionGroup calls it out. Past a pair, the ceiling binds.

I got this wrong on the first attempt: skipping any batch over the ceiling stranded large files permanently, and two existing tests caught it.

Measuring batch size

BatchMetadata.Bytes is populated in loadStoredBatch from the os.Stat that validation already performs — no extra syscalls, and no metadata version bump, because the field is derived on load rather than persisted (json:"-").

A batch that was never measured is charged a deliberately pessimistic per-row estimate rather than treated as weightless. Over-charging costs smaller groups and more passes; under-charging costs the merge the ceiling exists to prevent.

Tests

• TestSelectBoundedCompactionGroupStopsAtTheByteBudget — sixteen 64 MiB inputs, well inside the row and count ceilings. Asserts the byte ceiling binds and that a group is still produced.
• TestSelectBoundedCompactionGroupChargesUnsizedBatchesAnEstimate — unmeasured batches are priced, not free.

Both written first and verified failing.

Two existing tests changed

TestSelectCompactionBatchesBuildsRowBoundedGroup and TestSelectCompactionBatchesCombinesSaturatedLargeFiles now declare explicit small Bytes.

Both exist to pin the row ceiling. Their fixtures carry no size, so the new estimate became their binding constraint and they began asserting the byte ceiling instead — passing or failing for a reason unrelated to their names. Declaring a small size keeps each test exercising the ceiling it is about.

Flagging this explicitly: their fixtures changed, their assertions did not.

Still open in #254

The restart loop is untouched here — an OOM mid-merge writes no marker (compaction.go:134-141), so a clean restart reselects the same group and dies identically. This change makes that far less likely by shrinking the group; it does not stop the loop. Separate fix.

just check passes.