# ThreadSanitizer reports concurrent writes in `AsyncMethodInvocation::Notify`

[gorundebug]

Add a description

ThreadSanitizer reports concurrent writes in AsyncMethodInvocation::Notify

Summary

With more than one server gRPC completion queue, a small native userver service
reliably triggers a ThreadSanitizer data-race report in
userver::ugrpc::impl::AsyncMethodInvocation::Notify(bool).

The reproducer does not use ServiceLib or another application framework. It is
a direct userver application derived from cppnativeexample and contains only
a userver gRPC server, a userver gRPC client and a small HTTP handler that
invokes a unary RPC.

Changing only grpc-server.completion-queue-count produces the following
result:

Completion queues	Workload	Result
1	4 concurrent HTTP-to-unary-gRPC workers for 10 seconds	2652 successful requests, 0 errors, no TSan report
4	the same workload and binaries	3585 successful requests before the report; gRPC server exits with code 66

Environment

• userver: v3.1, commit c9f77729c0edce7e423def2d4a4450aa7fc9d259
• gRPC: 1.71.0
• Protobuf: 5.29.3
• compiler: Clang 18.1.3
• C++ standard: C++20
• build type: optimized Release with debug information and frame pointers
• sanitizer: ThreadSanitizer
• container OS: Ubuntu 24.04, Linux arm64
• host: Apple Silicon macOS with Docker Desktop

The same AsyncMethodInvocation::Notify implementation is still present on
userver develop at commit
815d358e065d0896ad23831b96c6f1c8f471cf30 (checked 2026-08-30).

Build configuration

userver and the application are both instrumented with TSan. The relevant
userver configuration is:

-DUSERVER_FEATURE_UBOOST_CORO=ON
-DUSERVER_SANITIZE=thread
-DUSERVER_LTO=OFF

The application is compiled and linked with:

-fsanitize=thread
-fno-omit-frame-pointer

Only userver's official runtime suppression file is enabled:

TSAN_OPTIONS=halt_on_error=1:\
ignore_interceptors_accesses=1:\
suppressions=/opt/userver/cmake/tsan.suppressions.txt:\
allow_addr2line=1:\
external_symbolizer_path=/usr/bin/addr2line

There is no application-side suppression for this report and no modified
third-party source.

Reproducer topology

The inventory process registers a direct userver unary gRPC service:

class InventoryService final
    : public inventoryserviceapi::InventoryServiceApiBase::Component {
 public:
  ProcessOrderItemResult ProcessOrderItem(
      CallContext&,
      processorderitem::ProcessOrderItemRequest&& request) override {
    processorderitem::ProcessOrderItemResponse response;
    response.set_available_qty(0);
    response.set_reserved(false);
    response.set_status("OUT_OF_STOCK");
    return response;
  }
};

int main(int argc, char* argv[]) {
  auto components = userver::components::MinimalServerComponentList()
      .AppendComponentList(userver::ugrpc::server::MinimalComponentList())
      .Append<InventoryService>();
  return userver::utils::DaemonMain(argc, argv, components);
}

The relevant static configuration is:

components_manager:
  task_processors:
    main-task-processor:
      worker_threads: 4
    grpc-blocking-task-processor:
      worker_threads: 1
      thread_name: grpc-worker
  default_task_processor: main-task-processor
  components:
    grpc-server:
      port: 9202
      completion-queue-count: 4
    grpc-server-middlewares-pipeline:
    inventory-service-api:
      task-processor: main-task-processor
      disable-all-pipeline-middlewares: true

A separate direct userver HTTP process uses
userver::ugrpc::client::SimpleClientComponent and performs one unary
ProcessOrderItem call per HTTP request. Four client connections and four
concurrent HTTP workers are used for ten seconds. All calls use ordinary unary
request/response semantics; the test does not intentionally cancel calls or
stop either process while the load is running.

To verify that the failure is controlled by the queue count, the exact same
binaries and load are first run with:

completion-queue-count: 1

That control run remains clean. The processes are then restarted after changing
only the value to 4, which reproduces the report.

ThreadSanitizer report

WARNING: ThreadSanitizer: data race (pid=1)
  Write of size 1 by thread T1012:
    #0 userver::v3_1::ugrpc::impl::AsyncMethodInvocation::Notify(bool)
       grpc/src/ugrpc/impl/async_method_invocation.cpp:18
    #1 userver::v3_1::ugrpc::impl::QueueRunner::QueueRunner(...)::$_0
       grpc/src/ugrpc/impl/queue_runner.cpp:25

  Previous write of size 1 by thread T1015:
    #0 userver::v3_1::ugrpc::impl::AsyncMethodInvocation::Notify(bool)
       grpc/src/ugrpc/impl/async_method_invocation.cpp:18
    #1 userver::v3_1::ugrpc::impl::QueueRunner::QueueRunner(...)::$_0
       grpc/src/ugrpc/impl/queue_runner.cpp:25

  Thread T1012 'grpc-queue' created in:
    userver::v3_1::ugrpc::impl::CompletionQueuePoolBase::CompletionQueuePoolBase
    grpc/src/ugrpc/impl/completion_queue_pool_base.cpp:19

  Thread T1015 'grpc-queue' created in:
    userver::v3_1::ugrpc::impl::CompletionQueuePoolBase::CompletionQueuePoolBase
    grpc/src/ugrpc/impl/completion_queue_pool_base.cpp:19

SUMMARY: ThreadSanitizer: data race
  grpc/src/ugrpc/impl/async_method_invocation.cpp:18
  in userver::v3_1::ugrpc::impl::AsyncMethodInvocation::Notify(bool)

Both reports reference the same byte in AsyncMethodInvocation::ok_. The
relevant implementation is:

void AsyncMethodInvocation::Notify(bool ok) noexcept {
    ok_ = ok;
    event_.Send();
}

Expected behavior

If grpc-server.completion-queue-count: 4 is a supported configuration, unary
RPC processing should not produce a data race when several queue threads are
active.

Questions

1. Is a server completion-queue-count greater than one supported with the
   current AsyncMethodInvocation implementation?
2. Can one completion tag legitimately reach Notify from two queue threads,
   or does this indicate a tag lifetime/reuse problem?
3. Is this a known TSan false positive with an official suppression, or a real
   race with an existing fix?
4. Is there an upstream configuration or newer revision that should be used to
   exercise multiple gRPC completion queues under TSan?

The application cannot safely add a mutex around this code because it neither
owns AsyncMethodInvocation nor knows whether two notifications are valid.
For the same reason, the reproducer intentionally does not propose a local
source patch or suppression.