feat(ingest)!: bound the decoded telemetry held in flight

[vishr]

Sixth item of #254, and the one that changes behaviour. Also carries the byte-weighted batch limit (fifth item).

Why this is the one that matters

Every other change in this series makes one commit cost less. None of them makes the process bounded.

Ingest concurrency is HTTP/2 streams times connections — net/http allows 250 streams per connection by default and nothing in this repo lowers it — and each handler holds its decoded batch and its proto until the commit is durably acknowledged. Nothing counted how much was in flight at once. A large enough burst was still an out-of-memory kill however cheap an individual batch became.

Change

Submit charges a request against a byte ceiling on entry and releases it on return.

That is the right seam: both transports funnel through it, and it already blocks until durable acknowledgement, so the ledger measures exactly what the process is holding rather than what arrived on the wire.

A refusal returns RESOURCE_EXHAUSTED over gRPC and the existing 503 over HTTP. Both are what the OTLP specification tells an exporter to back off and retry on, so a shed request is delayed, not lost. fanout_ingest_shed_total makes it visible before it becomes a support question.

An empty ledger admits any single request however large. One request is already one atomic batch, and refusing the only thing in flight would shed load the process is not short of. The ceiling exists to stop requests accumulating, not to reject a large one.

Also: byte-weighted group batches

maxGroupBatchRows caps rows, but a row is whatever the sender put in it. 50,000 bare log lines and 50,000 spans carrying kilobytes of attributes are the same count and differ by orders of magnitude in what the commit holds. batchBytes counts the JSON columns and free-form strings, which carry effectively all of the variable size.

Breaking change — please read

A deployment under sustained overload now receives retryable refusals where it previously received acceptance and, eventually, a killed process.

ingest.max_in_flight_bytes defaults to 256 MiB. Setting it to 0 restores the previous unbounded behaviour exactly.

I want a second opinion on the default. 256 MiB is chosen to sit well above any plausible steady state while still being a small fraction of an 8 GiB host — but it is a judgement, not a measurement, and it is the number that decides when a healthy deployment starts shedding.

Tests

• TestSubmitRefusesWorkBeyondTheInFlightBudget — charge, refuse, release, admit again, ledger returns to zero.
• TestSubmitBudgetOfZeroAdmitsEverything — the unconfigured path is unchanged.
• TestSubmitOverBudgetIsReportedToTheCaller — refusal reaches the caller as ErrIngestOverBudget, not a hang.
• TestBatchBytesMeasuresThePayloadNotTheRowCount — equal row counts, wildly different payloads.
• TestGroupBatchBytesAdmitsALoneOversizedRequest — the lone-request exemption.

All written first and verified failing.

just check passes, including the config-schema and generated-docs gates, which caught the new key being undocumented.