fix(client): Avoid missing final send CQE

[nchild-cornelis-networks]

Hey all!

So we kept hitting odd timeouts on our small RDMA deployment. We would run elbencho to get a filesystem benchmark and every 50 or so runs the ~10s test would take 5 minutes and eventually result in some messages like:

beegfs: elb-wrk-6(163505): writefile (communication): Failed to send message to node_storage_1
beegfs: elb-wrk-6(163505): writefile (communication): SocketError. ErrCode: -70
beegfs: elb-wrk-6(163505): writefile (communication): Communication error in SENDDATA stage. Node: node_storage_2
beegfs: elb-wrk-6(163505): writefile (communication): Communication error. Node: node_storage_1
beegfs: elb-wrk-6(163505): writefile (communication): Communication error. Node: node_storage_2
beegfs: elb-wrk-1(163500): writefile (communication): Failed to send message to node_storage_1
beegfs: elb-wrk-1(163500): writefile (communication): Communication error. Node: node_storage_1

After much bpftracing, the finding was that there was a CQE just waiting in the CQ with no one polling it for the full timeout window. At the time the CQE was enqueued, IB_CQ_NEXT_COMP was not set but it was set sometime after the enqueue..

From RDMA providers perspective everything is valid. This is the exact race that IB_CQ_REPORT_MISSED_EVENTS was added to catch. Beegfs does not use IB_CQ_REPORT_MISSED_EVENTS so this is not caught.

Option 1 is to add beegfs support for IB_CQ_REPORT_MISSED_EVENTS. I chose not to do this because I do not know if other vendors support this and that would require more code overhaul than just the simple two line change.

In the end I could avoid the issue by swapping the ordering of the atomic inc and the req_notify call. This makes sense to me because we don't want to notify about a new CQE UNTIL we are re-armed to capture the next CQE, otherwise there IS a race window and we may miss the next CQE, perhaps I explain this better in the commit message. But essentially, the pseudocode of previous implementation and order of execution was the following:

__IBVSocket_sendCompletionHandler:
     atomic_inc(cnt)                 <- 1. CQE X callback we bump
     ib_req_notify_cq()             <- 5. now set notify bit


__IBVSocket_waitForTotalSendCompletion:
  do {
    saved_cnt = atomic_read(cnt)  <-  2. save after CQE X bumps
    if (ib_poll_cq() = 0)                    <- 3. empty poll, we polled X in prev iteration 
        wait_until  atomic_read(cnt) != saved_cnt  <- 6. wait for CQE which is already enqueued
  } while (outstanding ops)

<RDMA vendor enqueues new CQE>    <- 4. notify bit is not set so silently enqueue