Skip to content

feat(core/transport): keep one SSH connection per transport - #214

Merged
safl merged 1 commit into
mainfrom
ssh-keep-connection
Oct 23, 2026
Merged

safl merged 1 commit into
mainfrom
ssh-keep-connection

Conversation

@safl

@safl safl commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

The SSH transport now keeps one connection per transport instead of connecting and disconnecting around every run(), get() and put(). Against a lab host at 43 ms ping a command cost about 0.55 s with a connection each and about 0.175 s over a kept one; oXIO's test harness had to override _SSH__connect/_SSH__disconnect to get this, which this change makes unnecessary.

What to look at carefully:

  • SSH.__connect() reuses the connection while get_transport().is_active(), otherwise closes and connects again with a 30 s keepalive. A connection that died silently but still reports active blocks the next command until the kernel gives up on it, which with the 30 s keepalive writing is tcp_retries2, about 15 minutes on Linux defaults (a fresh connect gave up after the SYN retries, about 2 minutes); that command then fails once and the one after reconnects. There is deliberately no retry, since a retry could run a command twice. A connect that fails after paramiko started the transport, as on refused authentication, is closed rather than kept.
  • wait_for_transport now closes the transport before each probe, so "up" and "down" are still decided by a fresh connect rather than by a kept session that can outlive a remote no longer accepting new ones.
  • There is no option to turn the kept connection off: [cijoe.transport.<name>] is passed straight to paramiko's connect(), so a cijoe key there would have to be stripped first. Say if you want one anyway.
  • New Cijoe.close(), which takes an optional transport name to close only that one (as wait_for_transport does), and Transport.close(); the pytest plugin calls Cijoe.close() in pytest_unconfigure, and a weakref.finalize closes the client at exit.

Verification gap: the new tests in tests/core/test_transport_ssh.py drive a real Cijoe over a fake paramiko client (one connection for many commands and transfers, a reconnect after a drop, close(), a failed connect not being kept, a command's SSH error dropping the connection, a failed transfer dropping it, closing a single transport, and the fresh-connect probe in wait_for_transport). Each fails without the code it covers; on main, test_wait_for_transport_probes_a_fresh_connection passes its assertion and fails only in the fixture, since Cijoe.close() does not exist there. They ran locally on Python 3.14 with paramiko 5.0.0, and CI runs them on 3.10 to 3.13 on Linux and macOS with whatever paramiko and scp resolve, since pyproject.toml pins neither. Nothing ran against a real SSH host from this branch: keepalive, reconnect after a real reboot, SCP over a reused connection and wait_for_transport timing are unexercised. The same keep-connection approach, as an override in oXIO's harness, has run full test sessions against four lab hosts. Two existing tests in tests/core/test_collector.py fail locally with and without this change, from stray files in /tmp and the templates directory.

Assisted-by: Claude Code:claude-opus-5-5

@coveralls

coveralls commented Oct 15, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 39939074938

Coverage decreased (-4.8%) to 71.088%

Details

  • Coverage decreased (-4.8%) from the base build.
  • Patch coverage: 3 uncovered changes across 1 file (36 of 39 lines covered, 92.31%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
src/cijoe/core/transport.py 32 29 90.63%
Total (3 files) 39 36 92.31%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 2249
Covered Lines: 1675
Line Coverage: 74.48%
Relevant Branches: 812
Covered Branches: 501
Branch Coverage: 61.7%
Branches in Coverage %: Yes
Coverage Strength: 0.74 hits per line

💛 - Coveralls

@safl
safl force-pushed the ssh-keep-connection branch 2 times, most recently from 9faccb2 to 44079a9 Compare October 23, 2026 10:04
@safl
safl requested a review from naddinadja October 23, 2026 10:17
The SSH transport connected before and disconnected after every run(),
get() and put(), so each command paid a key exchange. Measured from a
test harness against a host at 43 ms ping, a command cost about 0.55 s
that way and about 0.175 s over a kept connection, which made cheap
remote checks expensive enough to be left out. Per-command connections
existed to cope with a remote that reboots and to not leave connections
open at exit; a kept connection handles both by connecting again on
next use when the old one is no longer active, and by being closed by
Cijoe.close(), at pytest unconfigure, or by a finalizer as Python exits.
A connect that fails part way, as when authentication is refused, is
closed rather than kept. A 30 s keepalive keeps an idle connection's
NAT and firewall state. wait_for_transport closes the connection before
each probe, since a kept session can outlive a remote that no longer
accepts new ones, and only a fresh connect tells whether it is up or
down.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Simon A. F. Lund <os@safl.dk>
@safl
safl force-pushed the ssh-keep-connection branch from 44079a9 to 40d9721 Compare October 23, 2026 10:26

@naddinadja naddinadja left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! :)

Comment thread CHANGELOG.md
HEAD can be subject to a git rebase.

## 0.9.61 (Upcoming)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to remember to remove the "(Upcoming)" once we do the new tag

@safl

safl commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

xNVMe exercises this together with #215 and #216 in xnvme/xnvme#825, which pins cijoe to the preview branch ci-preview-overlay-ssh (01aedad). Its verify jobs are green, and across 13 of them the guest provision-and-test step takes 2734 s less than on xNVMe main, a mean of 210 s per job. The largest saving, 948 s on test_os, is in the job that runs the most commands, which is where the kept connection matters. The per-job table is in xnvme/xnvme#825. Each side is a single run on GitHub runners, so small per-job differences may be noise.

@safl
safl merged commit 25e5776 into main Oct 23, 2026
30 checks passed
@safl safl mentioned this pull request Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants