[Trace] Add bounded export queue/backpressure instead of spawning one goroutine per completed span

[cssbruno]

Priority

Medium-high — runtime availability under load and predictable observability overhead.

Context

runtime/trace/span.go exports a completed span by starting a new goroutine for every span:

func recordSpanAsync(tracer *Tracer, sink Sink, snapshot Snapshot) {
    go func() {
        ctx, cancel := context.WithTimeout(context.Background(), defaultSinkTimeout)
        defer cancel()
        _ = sink.RecordSpan(ctx, snapshot)
        ...
    }()
}

Each goroutine has a timeout, but there is no global queue size, concurrency limit, or drop/backpressure policy around sink export.

Problem

A slow or stuck sink can cause unbounded goroutine growth when many spans complete.

Consequences:

1. High request volume can create many concurrent export goroutines.
2. A slow sink can consume memory and scheduler resources even though each individual export eventually times out.
3. Runtime overhead becomes hard to reason about for generated applications.
4. There is no explicit dropped-span counter or health signal for export backpressure.
5. Tests may cover sink timeout behavior but not sustained high-throughput pressure.

Proposed direction

Move span export behind a bounded worker/queue owned by the tracer.

Suggested behavior:

• fixed maximum queue length;
• fixed maximum concurrent sink exports;
• configurable or documented drop policy when full;
• counters for exported, failed, timed out, and dropped spans;
• graceful shutdown/drain hook where useful;
• synchronous fallback only for tests, if needed.

Acceptance criteria

• Completing spans cannot spawn unbounded goroutines.
• A slow sink is isolated by a bounded queue and worker/concurrency limit.
• Dropped or timed-out spans are counted in tracer export health.
• Tests cover slow sink, blocked sink, burst traffic, queue overflow, and normal export.
• Default behavior remains dependency-free and safe for generated apps.
• Docs describe the trace export budget and drop policy.

Related

• Define and enforce the v1 core versus optional product boundaries #677 — v1 core versus optional product boundaries
• [Compiler] Introduce one bounded subprocess supervisor for all external tools #685 — bounded external/process supervision; this issue applies the same bounded-resource principle to in-process trace export