Skip to content

[BUG] Wait on a completion state in the Elasticsearch exporter#4298

Draft
thc1006 wants to merge 4 commits into
open-telemetry:mainfrom
thc1006:bugfix/elasticsearch-completion-state
Draft

[BUG] Wait on a completion state in the Elasticsearch exporter#4298
thc1006 wants to merge 4 commits into
open-telemetry:mainfrom
thc1006:bugfix/elasticsearch-completion-state

Conversation

@thc1006

@thc1006 thc1006 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #4296

Draft while CI runs. Ready for review from my side.

The problem

bool waitForResponse()
{
  std::unique_lock<std::mutex> lk(mutex_);
  cv_.wait(lk);
  return response_received_;
}

OnResponse() set response_received_ under the mutex and called notify_all() after releasing it. If that landed before the exporting thread reached cv_.wait(lk), the notification was gone and the thread blocked with nothing left to wake it. The OnEvent() failure cases were worse: they called notify_all() without taking the mutex and without recording anything, so an early failure left the waiter with no state to observe even if it did wake.

A spurious wakeup also returned response_received_, still false, so a request that was still in flight was reported as a failure.

SessionState::Destroyed neither notified nor recorded anything, so a session that ended without a response or an error event left the waiter blocked indefinitely.

Two threads, one recording the outcome and one waiting, with the ordering flipped between runs:

=== notification arrives AFTER the waiter parks ===
  old: response after wait                 returned success
  new: response after wait                 returned success
=== notification arrives BEFORE the waiter parks ===
  new: response before wait                returned success
  old: response before wait                HUNG, no wakeup left to receive

The last run had to be killed by a timeout.

The fix

The handler records an explicit completion state under the mutex on both the success and the failure paths, and the wait is a predicate over that state:

cv_.wait(lk, [this] { return completion_ != CompletionState::Pending; });
return completion_ == CompletionState::Success;

Because the predicate is evaluated under the same mutex that publishes the state, an outcome recorded before the waiter arrives is simply observed rather than missed, and a spurious wakeup goes back to waiting instead of reporting a failure.

Recording is first writer wins. That matters for Destroyed, which now records a failure too: it fires at the end of every session, including successful ones, and must not overwrite a result that has already been recorded. Its purpose here is to release a waiter when a session ends without ever producing a response.

What this does not change

The wait still has no timeout of its own. It is bounded by the request timeout arriving as a TimedOut session event, plus the Destroyed fallback added here. Giving the wait its own deadline would be a behavior change beyond this fix, and I would rather do it separately if you want it.

Verification

  • The comparison above was run, not reasoned about.
  • clang-format 18 clean.
  • I could not run the repo's clang-tidy against this file locally, since it needs the exporter's full include set. The enum carries an explicit std::uint8_t base type so it does not trip performance-enum-size, and <cstdint> is added for it. CI is the authority on warning_limit.

Scope

Synchronization only. The bulk response parsing is #4295 with its own PR, and the internal linkage of ResponseHandler belongs to the #4196 cleanup.

thc1006 added 2 commits July 25, 2026 01:23
waitForResponse() waited on a condition variable with no predicate and
then returned a bool that nothing on the error paths ever set.

OnResponse() set the flag under the mutex and notified after releasing
it, so a notification delivered before the exporting thread reached
cv_.wait() was lost and that thread blocked with nothing left to wake
it. The OnEvent failure cases were worse: they notified without taking
the mutex and without recording anything at all. A spurious wakeup also
returned false, reporting a request that was still in flight as failed.

The handler now records an explicit completion state under the mutex on
both the success and the failure paths, first writer wins so a session
destroyed after a good response does not overwrite it, and the wait is
on a predicate over that state. SessionState::Destroyed also records a
failure if nothing was recorded yet, so a session that ends without a
response no longer leaves the waiter blocked forever.

Checked with a two-thread harness: with the notification delivered
before the waiter parks, the old shape blocks until the test is killed
while the new one returns immediately.

Fixes open-telemetry#4296

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.30%. Comparing base (d3c7a47) to head (2129088).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4298      +/-   ##
==========================================
- Coverage   81.31%   81.30%   -0.01%     
==========================================
  Files         445      445              
  Lines       18859    18859              
==========================================
- Hits        15333    15331       -2     
- Misses       3526     3528       +2     
Files with missing lines Coverage Δ
...orters/elasticsearch/src/es_log_record_exporter.cc 12.22% <ø> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

thc1006 added 2 commits July 25, 2026 02:11
IWYU reports stdint.h as unused once cstdint is there for the enum base
type. The only fixed-width type in the file is uint8_t, which cstdint
provides.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The gcc 14 abiv2 run failed only on ext.http.curl.BasicCurlHttpTests.ElegantQuitQuick,
which this change does not touch, so re-running to confirm it is a flake.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Elasticsearch synchronous export can block forever on a lost notification

1 participant