Skip to content

net/tls: keep the first handshake error for queued handshake waiters - #331

Open
travisdowns wants to merge 1 commit into
redpanda-data:v27.1.x-prefrom
travisdowns:td-tls-handshake-keep-error-v27.1.x-pre
Open

travisdowns wants to merge 1 commit into
redpanda-data:v27.1.x-prefrom
travisdowns:td-tls-handshake-keep-error-v27.1.x-pre

Conversation

@travisdowns

Copy link
Copy Markdown
Member

A read and a write on a TLS socket that find the session not yet connected both
call handshake(), which serializes them on the input and output semaphores. If
the first handshake (driven by the winner of the race) failed, the second ran it
again on the failed session. The library then reported a different error, often
a generic one ("System error during handshake" in OpenSSL, "The specified
session has been invalidated" in GnuTLS), and it replaced the recorded error, so
the second waiter and every later operation saw it instead of the real cause,
such as a certificate verification failure.

Instead, check the error after acquiring the semaphore, and if set return the
recorded error instead of retrying the handshake, in both backends.

(cherry picked from commit 46f0a47)

Backport of scylladb#3855, which is still in review upstream.

Additional details

Why it is needed here

A Redpanda change that moves the RPC client's background read loop into an explicit scheduling group makes the read loop start one task later than before, so a request's write now usually starts the TLS handshake and the read loop queues behind it. With a revoked peer certificate, the write gets certificate revoked, but the read loop gets System error during handshake: error:400C0107:lib(128)::operation fail and logs it at ERROR level. The certificate revocation integration test, which allows only the certificate errors, fails on every debug-build run. With this fix the read loop reports certificate revoked and the test passes: 3/3 failures before and 3/3 passes after, on a local debug build with TRACE logging.

Branch

This is the first commit on v27.1.x-pre, created from the v26.3.x tip (dfcbf1c05) now that Redpanda has branched for 26.3, following the release-branching process. Redpanda dev already pins dfcbf1c05, so bumping dev to this branch adds only this fix.

Tests

Two new cases in tests/unit/tls_test.cc. Each starts a write and flush on a client whose handshake will fail, then a read that queues behind it:

Test Requires Before this change
test_read_queued_behind_handshake_gets_verification_error: the client does not trust the server's CA both fail with tls::verification_error fails on OpenSSL (the read gets System error during handshake); passes on GnuTLS
test_read_queued_behind_handshake_gets_handshake_error: the peer answers with plain HTTP both report the same error fails on both: OpenSSL System error during handshake vs Wrong SSL Version number; GnuTLS The specified session has been invalidated vs An unexpected TLS packet was received

On this branch, both pass on both backends, and the full tls_test passes in dev mode with the GnuTLS and OpenSSL backends.

A read and a write on a TLS socket that find the session not yet connected both
call handshake(), which serializes them on the input and output semaphores. If
the first handshake (driven by the winner of the race) failed, the second ran it
again on the failed session. The library then reported a different error, often
a generic one ("System error during handshake" in OpenSSL, "The specified
session has been invalidated" in GnuTLS), and it replaced the recorded error, so
the second waiter and every later operation saw it instead of the real cause,
such as a certificate verification failure.

Instead, check the error after acquiring the semaphore, and if set return the
recorded error instead of retrying the handshake, in both backends.

(cherry picked from commit 46f0a47)
Copilot AI balanced review requested due to automatic review settings November 19, 2026 13:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The TLS handshake error-preservation change has no active review issues.

No open issues 🎉

What changed in this PR

This PR preserves the original TLS handshake failure for queued callers, preventing later reads or writes from retrying a failed handshake and reporting a less useful error.

Changes:

  • Recheck the recorded error after acquiring both handshake semaphores in the OpenSSL and GnuTLS backends.
  • Add regression tests for queued reads after certificate verification failures and invalid TLS responses.
File Description
tests/​unit/​tls_test.cc Adds concurrent flush/read tests that verify handshake errors are preserved.
src/​net/​tls_openssl.cc Returns the recorded error instead of retrying a failed handshake.
src/​net/​tls_gnutls.cc Applies the same error check to the GnuTLS backend.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants