fix: reconcile big segment data with polling endpoint when stream is quiet

[kinyoklion]

Ticket: SDK-2840

Root cause of the big_segments integration flake

TestEndToEnd/big_segments/{Redis,DynamoDB}/another_big_segment_is_created_after_synchronizer_has_started is the most frequent remaining CI flake (12+ occurrences May–July). Log forensics on two recent failures (run 30562656155, v9, 07-30; run 29779343829, v8, 07-20) show the same signature:

• The synchronizer's poll → stream → poll sequence ran as designed, including the bridging poll after the stream was established.
• The revisions that went missing were published after that bridging poll completed — e.g. the second half of an include list landed within ~1s of the bridging poll, and the second segment's revisions a few seconds later.
• The /big-segments SSE stream had connected successfully but delivered nothing for the entire wait (60 seconds in the v8 run, so test(integrationtests): increase SDK key expiry margin and big-segment sync timeout #711's timeout bump demonstrably does not fix this), on both environments' connections.

The bridging poll closes the gap up to the moment it completes, but it treats "stream response established" as "subscribed from this instant". If the upstream subscription is not yet fully effective at that point (the CI test environments are created seconds before connecting), events published in that window are never delivered — and the protocol has no redelivery, so the data is stale until some later unrelated event for that environment arrives (heartbeats keep the connection alive, so the 5-minute read timeout doesn't rescue it).

Changes

• consumeStream performs one follow-up reconciliation poll per stream connection, at the first quiet-stream tick (30s) after the stream is established, covering the delayed-subscription window. After that single poll the stream is trusted and the quiet-stream timer goes back to what it did before: refreshing the store's synchronizedOn timestamp only. Steady-state behavior is pure streaming — no periodic polling.
• Ports test(integrationtests): increase SDK key expiry margin and big-segment sync timeout #711's evaluation-wait bump (20s → 60s) to the v9 integration test — it was only ever applied to v8. 60s covers the follow-up reconciliation with margin.
• The supervisor no longer logs the will retry warning when it is shutting down cleanly — that warning appears at the tail of every failed-test log dump and reads as a stream failure when it's actually teardown.
• New unit test TestSyncReconcilesWhenStreamIsQuiet: stream connects but stays silent, a missed revision exists only on the poll endpoint; the follow-up poll applies it without a stream reconnect, and no further poll requests happen after it.

Verification

• go test -race -count=3 ./internal/bigsegments/ green; the reconcile test passes 50 consecutive -race runs under full CPU load.
• go vet -tags integrationtests ./integrationtests/ clean.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit adbb250. Configure here.}

[cursor]

Reconcile races with stream events

Medium Severity

reconcile can advance the store cursor while a matching stream event is already queued or in flight. When that event is processed next, applyPatches fails the PreviousVersion check and the existing out-of-order path returns nil, forcing a full stream restart and retry backoff even though the revision was already applied. The connection-time poll avoids this by reconnecting when it catches up (!done), but quiet-period reconcile does not.

Additional Locations (1)

• internal/bigsegments/sync.go#L400-L403

 

^{Reviewed by Cursor Bugbot for commit adbb250. Configure here.}