[BUG] Wait on a completion state in the Elasticsearch exporter#4298
Draft
thc1006 wants to merge 4 commits into
Draft
[BUG] Wait on a completion state in the Elasticsearch exporter#4298thc1006 wants to merge 4 commits into
thc1006 wants to merge 4 commits into
Conversation
waitForResponse() waited on a condition variable with no predicate and then returned a bool that nothing on the error paths ever set. OnResponse() set the flag under the mutex and notified after releasing it, so a notification delivered before the exporting thread reached cv_.wait() was lost and that thread blocked with nothing left to wake it. The OnEvent failure cases were worse: they notified without taking the mutex and without recording anything at all. A spurious wakeup also returned false, reporting a request that was still in flight as failed. The handler now records an explicit completion state under the mutex on both the success and the failure paths, first writer wins so a session destroyed after a good response does not overwrite it, and the wait is on a predicate over that state. SessionState::Destroyed also records a failure if nothing was recorded yet, so a session that ends without a response no longer leaves the waiter blocked forever. Checked with a two-thread harness: with the notification delivered before the waiter parks, the old shape blocks until the test is killed while the new one returns immediately. Fixes open-telemetry#4296 Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4298 +/- ##
==========================================
- Coverage 81.31% 81.30% -0.01%
==========================================
Files 445 445
Lines 18859 18859
==========================================
- Hits 15333 15331 -2
- Misses 3526 3528 +2
🚀 New features to boost your workflow:
|
IWYU reports stdint.h as unused once cstdint is there for the enum base type. The only fixed-width type in the file is uint8_t, which cstdint provides. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4296
Draft while CI runs. Ready for review from my side.
The problem
OnResponse()setresponse_received_under the mutex and callednotify_all()after releasing it. If that landed before the exporting thread reachedcv_.wait(lk), the notification was gone and the thread blocked with nothing left to wake it. TheOnEvent()failure cases were worse: they callednotify_all()without taking the mutex and without recording anything, so an early failure left the waiter with no state to observe even if it did wake.A spurious wakeup also returned
response_received_, still false, so a request that was still in flight was reported as a failure.SessionState::Destroyedneither notified nor recorded anything, so a session that ended without a response or an error event left the waiter blocked indefinitely.Two threads, one recording the outcome and one waiting, with the ordering flipped between runs:
The last run had to be killed by a timeout.
The fix
The handler records an explicit completion state under the mutex on both the success and the failure paths, and the wait is a predicate over that state:
Because the predicate is evaluated under the same mutex that publishes the state, an outcome recorded before the waiter arrives is simply observed rather than missed, and a spurious wakeup goes back to waiting instead of reporting a failure.
Recording is first writer wins. That matters for
Destroyed, which now records a failure too: it fires at the end of every session, including successful ones, and must not overwrite a result that has already been recorded. Its purpose here is to release a waiter when a session ends without ever producing a response.What this does not change
The wait still has no timeout of its own. It is bounded by the request timeout arriving as a
TimedOutsession event, plus theDestroyedfallback added here. Giving the wait its own deadline would be a behavior change beyond this fix, and I would rather do it separately if you want it.Verification
std::uint8_tbase type so it does not tripperformance-enum-size, and<cstdint>is added for it. CI is the authority onwarning_limit.Scope
Synchronization only. The bulk response parsing is #4295 with its own PR, and the internal linkage of
ResponseHandlerbelongs to the #4196 cleanup.