Skip to content

e2e: TestXoverStalledClientDoesNotHoldGroupLock covers the whole stall - #375

Merged
go-while merged 3 commits into
mainfrom
fix/xover-stall-test-wait
Nov 11, 2026
Merged

go-while merged 3 commits into
mainfrom
fix/xover-stall-test-wait

Conversation

@go-while

@go-while go-while commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Old (#274): after A's XOVER 1-40000 status line the test slept 700 ms
"for A's reply to fill the socket", then measured one store by B
(bound 1.5 s). If A's session had not reached the records yet (slow or
loaded machine), B took the lock first and passed whatever the code did
with the lock.

New: after A's 224, B stores one uncached article after the other (each
store needs the group's exclusive lock) until A's session has ended. A
reads nothing, so its session ends only when a send times out,
ClientTimeout after it blocked: B's stores cover the whole stall, and
with the lock held across the write one of them waits until A's session
ends. The longest wait must stay within 1.5 s, as before. A's session is
found with daemonChildren (/proc, or pgrep -P without /proc, from #369),
so the test runs on FreeBSD and macOS too; it skips only where neither
exists. The stores are 2 ms apart (pacing only: at most about 1500 of
the group's 40000 articles in A's 3 s), and the test fails clearly if it
runs out of uncached articles.

An earlier head of this PR waited for /proc//syscall to show A's
session in send() and skipped elsewhere (Terra's finding: FreeBSD, a
required check, lost the test). A client-side check (A's receive queue
via FIONREAD stops growing) was tried as the fallback and rejected: it
only shows that A's TCP window closed, while the server still fills its
own send buffer, and mutant 2 below passed with it 3/3.

Measured (12 cores), 3 runs each:

build result B's longest wait
main pass 10-90 ms
main, -race pass 20-220 ms
mutant 1 FAIL 3/3 3.01-3.04 s
mutant 2 FAIL 3/3 3.01-3.05 s (also with -race)
main, /proc hidden (pgrep) pass 2/2 0 s
mutant 1, /proc hidden (pgrep) FAIL 2/2 2.89-2.9 s

Mutants (scratch only, not committed), NVNewsgroup::printoverdb_stored:

  1. flush_chunk() before guard.release(): each chunk written with the
    shared lock held.
  2. the same plus usleep(1 s) before the first chunk (a slow machine):
    the old test passed with it ("second session waited 0s").

Part of the review of weakened tests that go-while asked for. Shown to fail with the guarded defect (scratch mutant; not committed).

NEWS: changes/fix-xover-stall-test-wait.md.

🤖 Generated with Claude Code

go-while and others added 3 commits November 11, 2026 13:25
Old (#274): after A's XOVER 1-40000 status line the test slept 700 ms
"for A's reply to fill the socket", then measured B's ARTICLE store
(bound 1.5 s). If A's session had not reached the records yet (slow or
loaded machine), B took the lock first and passed whatever the code
did with the lock.

New: after the warm-up, wait until the daemon has no child, so that
A's session is its only child; after A's 224, poll /proc/<pid>/syscall
until the session sleeps in sendto/sendmsg/write (state S), then start
B. Linux only now (skips elsewhere, and if /proc/<pid>/syscall can't be
read); the code under test is the same on every system. The warm cache
and the 1.5 s bound stay.

Measured (12 cores): A blocked in send() 5-27 ms after the request,
128-142 ms with -race; B waited 0 s in all 5 + 5 runs.

Mutants (scratch only), NVNewsgroup::printoverdb_stored:
1. flush_chunk() before guard.release() (each chunk written with the
   shared lock held): new test fails, 3/3:
     the second session waited 3.042975562s for the group lock while
     the first client stopped reading; ...
   (the old test fails too, with 2.31 s, as the stall had begun 0.7 s
   before B).
2. the same plus usleep(1 s) before the first chunk (a slow machine):
   the old test passes ("second session waited 0s"); the new one waits
   for the block (1.032 s) and fails with 2.99 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous version waited for /proc/<pid>/syscall to show the first
session in send() and skipped where that file doesn't exist, FreeBSD
included, where main still ran the test. Now the second client stores
one uncached article after the other (each needs the group's exclusive
lock) from the first client's 224 until the first session has ended.
That session reads nothing and ends only when a send times out,
ClientTimeout after it blocked, so the stores cover the whole stall on
any system: with the lock held across the write one of them waits until
the session ends. The session is found with daemonChildren (/proc or
pgrep).

Mutants (scratch builds, not committed), NVNewsgroup::printoverdb_stored:
flush_chunk() before guard.release() fails 3/3 (waits 3.01-3.04 s); the
same plus usleep(1 s) first, a slow machine, fails 3/3 (3.01-3.05 s), also
with -race and with /proc hidden (pgrep). main passes, longest wait
10-90 ms (20-220 ms with -race).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@go-while go-while changed the title e2e: TestXoverStalledClientDoesNotHoldGroupLock waits for A to block e2e: TestXoverStalledClientDoesNotHoldGroupLock covers the whole stall Nov 11, 2026
@go-while
go-while merged commit 524e3e2 into main Nov 11, 2026
31 checks passed
@go-while
go-while deleted the fix/xover-stall-test-wait branch November 11, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant