Repository navigation
Description
How do you use Sentry?
Self-hosted/on-premise
Version
2.75.0 (Python 3.15.2, only sentry-sdk installed); also reproduces on 2.72.0, and master at e9c62e5 has the same code
Steps to Reproduce
Client.flush() can return before a batch is delivered to the transport. The batcher's flusher thread removes the batch from the buffer under _lock and calls _capture_func only after releasing it (Batcher._flush, SpanBatcher._flush). A flush() from another thread that runs in this window finds the buffer empty and returns at once, while the batch is still on its way to the transport.
The same window applies to logs and metrics (Batcher._flush) and to streamed spans (SpanBatcher._flush). For spans it opens right after a segment ends, since the segment end wakes the flusher thread immediately.
The transport below only widens the window so the race reproduces every time:
import threading
import time
import sentry_sdk
from sentry_sdk.transport import Transport
received = []
taken = threading.Event()
class SlowTransport(Transport):
def capture_envelope(self, envelope):
if threading.current_thread() is not threading.main_thread():
taken.set()
time.sleep(0.5) # widen the window: batch already removed from the buffer
received.extend(item.type for item in envelope.items)
sentry_sdk.init(
dsn="https://public@example.invalid/1",
transport=SlowTransport,
trace_lifecycle="stream",
traces_sample_rate=1.0,
)
with sentry_sdk.traces.start_span(name="segment"):
pass
taken.wait() # the flusher thread has taken the batch and is delivering it
sentry_sdk.flush()
print("after flush():", received)Expected Result
after flush(): ['span'] — flush() is documented as "Wait for the current events to be sent".
Actual Result
after flush(): []; the span arrives later.
Two consequences:
- Tests that capture envelopes with an in-memory transport and assert after
flush()are flaky. Client.close()callsflush()and then setsself.transport = None. A batch in the window then reaches_capture_envelopewith no transport and is dropped silently, so the last spans or logs of a shutting-down process can be lost.
The SDK's own test suite already works around this: _install_flush_completion_handshake in tests/conftest.py, added in #7170, makes flush() wait for the flusher thread, and #7170 describes the race. User code and user test suites have no such hook without patching private batcher internals.
A possible fix: a separate flush lock held for the whole _flush, including the _capture_func call. _lock stays short, so it doesn't interfere with the deadlock fix in #5684, the open PR #7776, or the change proposed in #8023. A flush() from the user's thread then waits for an in-flight flush of the flusher thread before draining the rest. On its own this lock isn't enough, though: a reentrant flush() from a transport would deadlock on it (see the comments below).
Metadata
Metadata
Assignees
Projects
- StatusShow more project fieldsWaiting for: Product Owner
Activity
linear-code commentedon Nov 16, 2026
PY-3082
alexander-alderman-webb commentedon Nov 16, 2026
I would presume that if a batch is removed in the flusher thread, that it reaches the transport in a bounded time (since this is all runs under a lock, and we should not be holding the lock for a long time).
Do you have a concrete case where the behavior causes a problem for you, or can we update the documentation of
flush()instead of changing behavior?realicon23 commentedon Nov 17, 2026
Hello, a local regression check on
e9c62e5reproduced the ordering with a fake capture callback. The background flusher drains the buffer, then pauses at the callback before the transport sees the batch. An explicitBatcher.flush()returns during that pause. The buffer lock has already been released at that point.This is a synthetic test. I don't have a production incident to add to the reporter's case. Would you prefer to document the handoff window, or should an explicit flush wait for an in-progress handoff within its existing timeout budget?
Flamefork commentedon Nov 17, 2026
Short answer: the concrete case is the one the Sentry SDK's own test suite already ran into. #7170 reports span-test failures on free-threading because the background flusher takes items before the synchronous flush, and
tests/conftest.pyhandles it with_install_flush_completion_handshake, which makesflush()wait for the flusher thread. That's exactly the guarantee we need: our test suite uses the real Sentry SDK with an in-memory transport and asserts right afterflush(). Outside the Sentry SDK's tests that guarantee is only available by patching private batcher internals, which is what we do now.On the lock: the handoff isn't under it.
Batcher._flushandSpanBatcher._flushrelease_lockbefore calling_capture_func. WithHttpTransportthe handoff is just an enqueue, but nothing orders it beforeClient.flush()returns.There is also a shutdown path that drops data:
close()flushes, kills the batchers without waiting for their threads, and setsself.transport = None. A batch the flusher thread is still handing off then finds no transport and is dropped. With Spotlight enabled the window is a whole synchronous HTTP request. The script below reproduces it on 2.75.0 with the realHttpTransportand a local server that holds the Spotlight response:close()returns immediately and the span never reaches the Sentry endpoint.Documenting the current behavior would leave users where we are now, patching internals. So I'd suggest moving the handshake from
tests/conftest.pyinto the batcher, with the wait bounded by the flush timeout, so thatflush()gives users the same guarantee the Sentry SDK's tests rely on. Shutdown can probably be fixed separately and more simply: in a quick monkeypatch experiment, havingclose()stop and join the batcher threads before flushing the transport delivered the span, including on interpreter exit. Joining inkill()alone wasn't enough on exit, because the batch then lands in the transport queue after the transport flush. The separate lock I proposed in the issue isn't safe on its own: a reentrantflush()from a transport would deadlock on it. I'm happy to send a PR for either part.Repro script (Spotlight + close)