Repository navigation
storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403
Description
What happened
Running the full Cassandra pytest suite (devtools/run-cassandra-tests -- --pytest --comprehensive --parallel, 2 workers) against a single-node cassandra:4.1 with a 60,000-item backup-restore fixture in the mix, the server lost its connection to Cassandra about 25 minutes in and never recovered. Every request after that failed with No nodes available in query plan! (surfaced to clients as InternalServerError, including Internal error during authentication), until the process was restarted. 90 tests failed and 314 errored in that run.
Server log at the moment it happened:
ERROR extenddb_engine::create_table: storage internal error internal_error=Internal error: Failed to drop data table: General error: Connection closed while waiting for response!
ERROR cdrs_tokio::transport: Transport error! error=General error: Unmatched stream id: 7914
WARN cdrs_tokio::cluster::connection_pool: Connection down. Starting reconnection. broadcast_rpc_address=Some(240.10.0.6:7000)
ERROR extenddb_storage_cassandra::cassandra_util: account_exists query failed: General error: No nodes available in query plan!
... (every request from here on)
Reading of the driver (cdrs-tokio 9.0.2, src/transport.rs)
ResponseHandlerMap::next_stream_id hands out ids 1..=i16::MAX and wraps to 1 without checking that the id's previous request has been answered. A slow request (here, a DROP TABLE under load) can still be outstanding when its id is reused; when the late response arrives the handler is already gone, send_response returns Unmatched stream id, the ? in start_reading_normal_frames ends the read loop, and the connection is closed. The reconnection then targets broadcast_rpc_address (240.10.0.6:7000 in this container), not the contact point the server was configured with, and never succeeds, so the pool stays empty.
This is from reading the code and the log, not a reproduction in isolation.
Why it matters
A busy production server with one Cassandra node (or one connection per node) can hit this after ~32k requests with any one slow statement in flight, and then serves nothing until restarted.
Options
- Open more than one connection per node (cdrs-tokio supports a per-node pool size), which divides the id space.
- Make
next_stream_idskip ids that still have a handler. - Reconnect to the configured contact point rather than
broadcast_rpc_address, or make reconnection retry. - Upstream fix or driver change.
Workaround in #384
devtools/run-cassandra-tests now sets EXTENDDB_TEST_BACKUP_ITEMS=10000 unless it is already set, which keeps the test suite under the threshold. Nothing changes for production.
Activity
jcshepherd commentedon Oct 23, 2026
#457
LeeroyHannigan commentedon Oct 24, 2026
While verifying #527 on a single
cassandra:4.1node: after adocker restartof the node mid-restore, the server'scdrs-tokiosession did not come back once the node accepted CQL again (about 50 s later). Every request for the rest of a 240 s window answeredInternalServerError(Internal error during authentication,No nodes available in query plan!), and the restore's own cleanup (abandon_restore_target) and the in-process restore sweep could not run, so the failed target stayedCREATINGuntil the server was restarted, whose startup sweep then removed it in 4 s. Same session, same function onmain; noted here since the reconnect behaviour is what this issue and #457 are about. Measurement details are in the #527 description under Recovery.LeeroyHannigan commentedon Oct 28, 2026
I traced the restart case from my last comment while reviewing #457. It's in cdrs-tokio's metadata refresh, and it happens on main as well as on #457.
After the node comes back, the control connection reconnects and refreshes the cluster metadata. On an incremental refresh the driver rebuilds each node with
Node::clone_with_node_info(metadata_builder.rs:78-81 in 9.0.2), which marks the address as changed when the old gossip address differs from the new client address (node.rs:280-289). On Cassandra 4.x the gossip address isbroadcast_addressplusbroadcast_portfromsystem.local, here<container ip>:7000, and the client address is the contact point,127.0.0.1:<port>. They never match, so the node is rebuilt asUnknownwith no distance and an empty pool. The load balancer skips it from then on, and on a single node nothing moves it back because the node sends no status event about itself. Every query after that getsNo nodes available in query plan!.A full refresh doesn't have this problem, it copies the contact point and keeps it
Up(metadata_builder.rs:20-28). The incremental path runs when the control connection reconnects while the node is still in the plan. Behind a published Docker port that's the usual case. Docker's proxy accepts the TCP connection while the node's native transport is down and then closes it, so a failed dial leaves a stale error in the control connection's one-slot error channel (transport.rs:394-405, control_connection.rs:43). When the control connection gets back and does a full refresh, it reads that stale error, drops the good connection, reconnects, and the second refresh is incremental:Queries succeed from 21.095 and fail from 21.667 to the end of the run. There's a second trigger that needs no stale error. If the pool reconnects first and marks the node
Up, the control connection's next refresh is incremental too.Measured on one cassandra:4.1 node with
nodetool disablebinary, 4s, thenenablebinary, counting a run as recovered when queries then succeed for 30s without a failure:devtools/run-cassandra-testsexpose the node, it recovered in 1 of 10 runs on fix(storage-cassandra): apply max_connections and override CDRS default reconnect policy #457's head and 1 of 10 on main. In most of the other runs the session ended onNo nodes available in query plan!and didn't recover within 3 minutes. Withdocker restartinstead it was 2 of 10 and 3 of 10.The reconnection policy only changes how far apart the attempts are, so #457's 5s cap doesn't change this. 9.0.2 is the latest release and master has the same comparison, so it needs a driver patch or a workaround on our side. Comparing the old client address with the new one, instead of the gossip address, looks like the fix in the driver, but I haven't tried it.
I haven't reproduced the sustained-load case from the original report. Any reconnect of the control connection while the node is in the plan goes down the same path, so I expect it's the same cause. From the code, the comparison also fails without NAT, since the gossip port is 7000 and the client port is 9042, but I've only run it against a single node in Docker.