Skip to content

flush() can return before a batch taken by the flusher thread reaches the transport #8300

Open

Description

@Flamefork

How do you use Sentry?

Self-hosted/on-premise

Version

2.75.0 (Python 3.15.2, only sentry-sdk installed); also reproduces on 2.72.0, and master at e9c62e5 has the same code

Steps to Reproduce

Client.flush() can return before a batch is delivered to the transport. The batcher's flusher thread removes the batch from the buffer under _lock and calls _capture_func only after releasing it (Batcher._flush, SpanBatcher._flush). A flush() from another thread that runs in this window finds the buffer empty and returns at once, while the batch is still on its way to the transport.

The same window applies to logs and metrics (Batcher._flush) and to streamed spans (SpanBatcher._flush). For spans it opens right after a segment ends, since the segment end wakes the flusher thread immediately.

The transport below only widens the window so the race reproduces every time:

import threading
import time

import sentry_sdk
from sentry_sdk.transport import Transport

received = []
taken = threading.Event()


class SlowTransport(Transport):
    def capture_envelope(self, envelope):
        if threading.current_thread() is not threading.main_thread():
            taken.set()
            time.sleep(0.5)  # widen the window: batch already removed from the buffer
        received.extend(item.type for item in envelope.items)


sentry_sdk.init(
    dsn="https://public@example.invalid/1",
    transport=SlowTransport,
    trace_lifecycle="stream",
    traces_sample_rate=1.0,
)

with sentry_sdk.traces.start_span(name="segment"):
    pass

taken.wait()  # the flusher thread has taken the batch and is delivering it
sentry_sdk.flush()
print("after flush():", received)

Expected Result

after flush(): ['span'] — flush() is documented as "Wait for the current events to be sent".

Actual Result

after flush(): []; the span arrives later.

Two consequences:

  • Tests that capture envelopes with an in-memory transport and assert after flush() are flaky.
  • Client.close() calls flush() and then sets self.transport = None. A batch in the window then reaches _capture_envelope with no transport and is dropped silently, so the last spans or logs of a shutting-down process can be lost.

The SDK's own test suite already works around this: _install_flush_completion_handshake in tests/conftest.py, added in #7170, makes flush() wait for the flusher thread, and #7170 describes the race. User code and user test suites have no such hook without patching private batcher internals.

A possible fix: a separate flush lock held for the whole _flush, including the _capture_func call. _lock stays short, so it doesn't interfere with the deadlock fix in #5684, the open PR #7776, or the change proposed in #8023. A flush() from the user's thread then waits for an in-flight flush of the flusher thread before draining the rest. On its own this lock isn't enough, though: a reentrant flush() from a transport would deadlock on it (see the comments below).

Activity

  1. linear-code commented on Nov 16, 2026

    @linear-code
  2. moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Nov 16, 2026
  3. alexander-alderman-webb commented on Nov 16, 2026

    @alexander-alderman-webb
    Collaborator

    I would presume that if a batch is removed in the flusher thread, that it reaches the transport in a bounded time (since this is all runs under a lock, and we should not be holding the lock for a long time).

    Do you have a concrete case where the behavior causes a problem for you, or can we update the documentation of flush() instead of changing behavior?

  4. moved this from Waiting for: Product Owner to No status in GitHub Issues with 👀 3on Nov 16, 2026
  5. realicon23 commented on Nov 17, 2026

    @realicon23

    Hello, a local regression check on e9c62e5 reproduced the ordering with a fake capture callback. The background flusher drains the buffer, then pauses at the callback before the transport sees the batch. An explicit Batcher.flush() returns during that pause. The buffer lock has already been released at that point.

    This is a synthetic test. I don't have a production incident to add to the reporter's case. Would you prefer to document the handoff window, or should an explicit flush wait for an in-progress handoff within its existing timeout budget?

  6. moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Nov 17, 2026
  7. Flamefork commented on Nov 17, 2026

    @Flamefork
    Author

    Short answer: the concrete case is the one the Sentry SDK's own test suite already ran into. #7170 reports span-test failures on free-threading because the background flusher takes items before the synchronous flush, and tests/conftest.py handles it with _install_flush_completion_handshake, which makes flush() wait for the flusher thread. That's exactly the guarantee we need: our test suite uses the real Sentry SDK with an in-memory transport and asserts right after flush(). Outside the Sentry SDK's tests that guarantee is only available by patching private batcher internals, which is what we do now.

    On the lock: the handoff isn't under it. Batcher._flush and SpanBatcher._flush release _lock before calling _capture_func. With HttpTransport the handoff is just an enqueue, but nothing orders it before Client.flush() returns.

    There is also a shutdown path that drops data: close() flushes, kills the batchers without waiting for their threads, and sets self.transport = None. A batch the flusher thread is still handing off then finds no transport and is dropped. With Spotlight enabled the window is a whole synchronous HTTP request. The script below reproduces it on 2.75.0 with the real HttpTransport and a local server that holds the Spotlight response: close() returns immediately and the span never reaches the Sentry endpoint.

    Documenting the current behavior would leave users where we are now, patching internals. So I'd suggest moving the handshake from tests/conftest.py into the batcher, with the wait bounded by the flush timeout, so that flush() gives users the same guarantee the Sentry SDK's tests rely on. Shutdown can probably be fixed separately and more simply: in a quick monkeypatch experiment, having close() stop and join the batcher threads before flushing the transport delivered the span, including on interpreter exit. Joining in kill() alone wasn't enough on exit, because the batch then lands in the transport queue after the transport flush. The separate lock I proposed in the issue isn't safe on its own: a reentrant flush() from a transport would deadlock on it. I'm happy to send a PR for either part.

    Repro script (Spotlight + close)
    # /// script
    # dependencies = ["sentry-sdk==2.75.0"]
    # ///
    import gzip
    import threading
    import time
    from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
    
    import sentry_sdk
    from sentry_sdk.envelope import Envelope
    
    
    class Server(ThreadingHTTPServer):
        daemon_threads = True
    
        def __init__(self):
            super().__init__(("127.0.0.1", 0), Handler)
            self.spotlight = threading.Event()
            self.release = threading.Event()
            self.span = threading.Event()
            self.items = []
            self.thread = threading.Thread(target=self.serve_forever, daemon=True)
            self.thread.start()
    
        @property
        def port(self):
            return self.server_address[1]
    
    
    class Handler(BaseHTTPRequestHandler):
        def do_POST(self):
            body = self.rfile.read(int(self.headers['Content-Length']))
            if self.path == '/stream':
                self.server.spotlight.set()
                if not self.server.release.wait(10):
                    raise RuntimeError('Spotlight release timed out')
            elif self.path == '/api/1/envelope/':
                if self.headers.get('Content-Encoding') == 'gzip':
                    body = gzip.decompress(body)
                envelope = Envelope.deserialize(body)
                types = [item.type for item in envelope.items]
                self.server.items.extend(types)
                if 'span' in types:
                    self.server.span.set()
            else:
                raise RuntimeError(f'unexpected path: {self.path}')
            self.send_response(200)
            self.send_header('Content-Length', '0')
            self.end_headers()
    
        def log_message(self, format, *args):
            pass
    
    
    def init(port):
        sentry_sdk.init(
            dsn=f'http://public@127.0.0.1:{port}/1',
            spotlight=f'http://127.0.0.1:{port}/stream',
            trace_lifecycle='stream', traces_sample_rate=1.0,
            default_integrations=False, enable_backpressure_handling=False,
        )
        return sentry_sdk.get_client()
    
    
    def segment(client):
        with sentry_sdk.traces.start_span(name='repro'):
            pass
        thread = client.span_batcher._flusher
        assert thread is not None
        return thread
    
    
    def run(action):
        server = Server()
        client = init(server.port)
        flusher = segment(client)
        assert server.spotlight.wait(5), 'Spotlight was not reached'
        start = time.monotonic()
        getattr(client, action)()
        elapsed = time.monotonic() - start
        before_release = server.span.is_set()
        server.release.set()
        flusher.join(5) if action == 'close' else None
        if action == 'flush':
            assert server.span.wait(5), 'span did not arrive after release'
            client.close()
            flusher.join(5)
        received = server.span.wait(1)
        assert not flusher.is_alive(), 'batcher thread did not stop'
        print(f'{action}: returned={elapsed:.3f}s, span_before_release={before_release}, '
              f'span_after_release={received}, items={server.items}', flush=True)
        server.shutdown()
        server.server_close()
        return before_release, received
    
    
    if __name__ == '__main__':
        assert run('close') == (False, False)
        assert run('flush') == (False, True)
        print('VERDICT: close loses the in-flight span; flush returns before delivery.', flush=True)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions