Retry poll on EINTR in the vendored nats-io client

[alexey-milovidov]

Closes: #120201
Related: #119867
Related: #119848

natsSock_WaitReady in the vendored nats-io client treated a poll interrupted by a signal (EINTR) as a socket error. The query profiler signals every thread, so a connection attempt of a NATS table failed at random with Cannot connect to Nats last error: (unix/sock.c:56): poll error: 4. The credentials-rotation integration tests use nats_startup_connect_tries = 1, so the one interrupted poll failed their CREATE TABLE (6 times on master and unrelated PRs since 2026-09-14).

The fix in the fork (ClickHouse/nats.c@b88704df, src/unix/sock.c) waits again for whatever is left of the deadline. The submodule bump also brings the two fork changes already merged there, which #119867 bumps to: the libuv attach ordering and the JetStream fetch lock-order backport.

Example failure: https://s3.amazonaws.com/clickhouse-test-reports/praktika.html?PR=119848&sha=ce21881b746a07dc79574410b18f5c5b57e38551&name_0=PR&name_1=Integration%20tests%20(amd_asan_ubsan,%20db%20disk,%204/8)

Changelog category (leave one):

• Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix spurious NATS connection failures when the vendored nats-io client gets EINTR while waiting for a socket, and include the upstream JetStream pull-subscription deadlock fix in the vendored client.

🤖 Generated with Claude Code

Version info

• Merged into: 26.10.1.384-master (included in 26.10 and later)
• Backported to: 26.9.2.2, 26.8.11.10, 26.7.14.12, 26.3.33.127

[clickhouse-gh]

Workflow [PR], commit [5f9101d]

Summary: ❌

• Performance Comparison: Performance dashboard

job_name	test_name	status	info	comment
Build profile diff		ERROR

---

AI Review

Summary

This PR bumps contrib/nats-io from 3e3a3f10 to b88704df, bringing the poll EINTR retry plus the already-merged libuv attach-ordering and JetStream pull-subscription lock-order fixes. I reviewed the current vendored range, the prior review thread, and ClickHouse's actual NATS / JetStream usage, and I did not find a remaining correctness, compatibility, or test-evidence issue that still needs action in the current code.

Final Verdict

• Status: ✅ Approve

LLVM Coverage Report

Measured on commit 5f9101d.

Metric	Baseline	Current	Δ
Lines	88.50%	88.40%	-0.10%
Functions	91.80%	91.80%	+0.00%
Branches	80.70%	80.70%	+0.00%

Changed lines: Uncovered code analysis did not run: No coverable C/C++ source files changed (contrib/ is excluded from coverage).

Newly covered: +184 lines in 50 files (-99 lines lost coverage) · Details

Full report

[PedroTadim]

Backporting to fix CI

[alexey-milovidov]

🕵 Merged master (the branch was 2021 commits behind) and pushed 5f9101d8; _nats_io builds clean at the bumped submodule, and master's own nats-io pointer (3e3a3f10) is an ancestor of b88704df, so the bump is still a strict superset.

Adopted the AI review: the category is now Bug Fix with the suggested changelog entry.

Both reds on b0f8bd81 are unrelated to a submodule bump:

• Stress test (amd_debug) — Logical error: GroupingAggregatedTransform pushed bucket A twice (STID 3994-4c1d): fleet-wide, 100+ hits in the last two weeks including master runs (pull_request_number = 0). Issue [CI crash] GroupingAggregatedTransform pushData failure #115986, fix in progress in Keep the bucket-order promise of a shard aggregation in the Cascades optimizer #119814 (not merged yet, so merging master does not clear it).
• Build profile diff — fleet-wide infrastructure failure No connection to LogCluster, Build profile diff fails fleet-wide with No connection to LogCluster #120462.

[groeneai]

Verified the bump and measured what it is worth, since a gitlink bump shows nothing in the diff.

The fix works, measured. I built the vendored client twice, at master's pointer 3e3a3f10 and at the bumped b88704df, and called the socket layer directly (natsSock_ConnectTcp to a blackholed address, 2 s deadline), with and without a 1 kHz SIGALRM storm on the calling thread:

pointer	signals	result	runs
3e3a3f10 (master)	1 kHz	gives up after 0 to 1 ms: (unix/sock.c:56): poll error: 4	3/3
3e3a3f10 (control)	none	waits the full 2002 ms, then (unix/sock.c:58): Timeout	3/3
b88704df (this PR)	1 kHz, 2000 delivered	waits 2000 ms, then Timeout	3/3
b88704df (control)	none	2002 ms, Timeout	3/3

That is the CI string byte for byte, reproduced at master's pointer and gone at yours, with the deadline intact (the loop recomputes natsDeadline_GetTimeout per iteration). The first row reports signals=1: one delivered signal is enough. Line 56 at master's pointer is that nats_setError and errno 4 is EINTR, so the file and line in the report pin the failure to that single unretried poll(). Reachable outside CI too: system.stack_trace signals every thread (STACK_TRACE_SERVICE_SIGNAL, src/Common/StackTraceServiceSignal.h).

What it costs today. The poll error: 4 signature starts 2026-09-14 and has not stopped: 65 failures across 56 distinct pull requests plus 4 master and 2 release-branch runs in 30 days, 15 pull requests today. Three tests, all NATS credentials rotation (test_jetstream_unacked_messages_survive_rotation, test_jetstream_credentials_rejected_after_rotation, test_stopped_nats_table_does_not_resubscribe_after_rotation).

The bump carries three fixes. 3e3a3f10..b88704df is 5 commits: the EINTR retry, plus nats.c#5 (libuv attach ordering) and nats.c#6 (JetStream fetch lock-order backport), which you merged into the fork on 2026-09-13 and which have never reached master. My bump for those two, #119867, is closed, so this PR is now the only open path for all three.

Ordering. #119507 moves the submodule to the v3.13.0 branch, which is divergent from ClickHouse/v3.9.2 (merge base 233ca8ef) and does not carry this fix: zero EINTR anywhere in its src/ tree, and its natsSock_WaitReady still has the bare poll() with the same "poll error: %d". Whichever of the two lands second has to carry the other's commits forward, and if that is #119507 the EINTR fix goes away again.

Its own CI on 5f9101d8: 148 green, two reds, one of which needs you.

• Build profile diff: the fleet-wide CI logs cluster outage, 270 distinct pull requests in the last 24 hours, fixed by your CI: "Build profile diff" is green when the CI logs cluster does not answer #121048. Not this PR.
• Mergeable Check and Finish Workflow: new_tests_check.py prints "No new tests have been added". The Bug Fix category put pr-bugfix on the PR, that hook then requires a new functional, integration or unit test or a passing Bugfix Validation job, and a one-line gitlink has none. Mergeable Check reports it as Failed: Workflow Post Hook, so it blocks the merge.

Two ways out, your call:

1. Changelog category CI Fix or Improvement. check_labels swaps pr-bugfix for pr-ci (it removes the previous category label) and the hook returns at "Not a bug fix PR". v26.3-must-backport is not a category label, so it survives.
2. Keep Bug Fix and add a test. I would not: the bug needs a signal delivered to the connecting thread while it sits in poll(), and I see no supported setting that does that deterministically from an integration test. My C harness does it in three lines of setitimer, so the natural home for a regression test is the fork's own suite. Say the word and I will send it there.