Skip to content

feat: expose curl's connection-cache size as the maxconnects pool option - #188

Closed
puzza007 wants to merge 1 commit into
masterfrom
maxconnects-option
Closed

puzza007 wants to merge 1 commit into
masterfrom
maxconnects-option

Conversation

@puzza007

@puzza007 puzza007 commented Jul 18, 2026 •

Copy link
Copy Markdown
Owner

Local soak testing — sustained mixed h1/h2/h3 load against nginx over tens of minutes — surfaced a slow fd climb in the C ports (~1 fd/minute at ~185 req/s). Per-state lsof breakdowns identified the mechanism: the server closes keepalive connections after its keepalive_requests limit, curl detects dead cached connections only lazily on reuse attempts, and the corpses sit in the idle connection cache in CLOSE_WAIT/CLOSED still holding their fds. It isn't a true leak — production is bounded by server recycling and curl's default cache sizing — but katipo offered no way to bound the cache explicitly. This adds {maxconnects, N}, passed through to CURLMOPT_MAXCONNECTS exactly like the other curl-multi pool options.

The sizing guidance documented on the option comes from measurement, and the middle data point is the instructive one. Over identical 10-minute soak runs: the unbounded default drifted 165→182 fds; maxconnects=8 — below the workload's ~15 concurrent connections — exploded 320→8,517, because undersizing defeats connection reuse entirely and the resulting churn produces dead connections ~500× faster than lazy reaping retires them (hundreds of CLOSE_WAIT sockets within a minute); maxconnects=64, sized above peak concurrency, produced the best curve of all at 162→173. Hence the rule documented in the README: size it above peak concurrent connections, or leave it unset.

The soak harness and fd-probe tooling that produced these numbers live on a local branch by design; this PR carries only the option itself, its suite test, and the docs. 203 tests, dialyzer, xref, and lint pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vf2gxZLpxT4HvBfZKCgP3g

@codecov

codecov Bot commented Jul 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.58%. Comparing base (587e0a5) to head (bf270b7).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #188      +/-   ##
==========================================
+ Coverage   78.37%   78.58%   +0.20%     
==========================================
  Files           9        9              
  Lines        1369     1373       +4     
==========================================
+ Hits         1073     1079       +6     
+ Misses        296      294       -2     
Flag Coverage Δ
c 78.58% <100.00%> (+0.20%) ⬆️
erlang 78.58% <100.00%> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Local soak testing (sustained mixed h1/h2/h3 load against nginx)
surfaced a slow fd climb in the C ports: dead keepalive connections --
closed by the server after its keepalive_requests limit, but detected
only lazily by curl on reuse attempts -- accumulate in the idle
connection cache in CLOSE_WAIT/CLOSED with their fds held. Not a leak
(production is bounded by server recycling and curl's default cache
sizing), but katipo offered no way to bound the cache explicitly.
{maxconnects, N} now passes through to CURLMOPT_MAXCONNECTS like the
other multi options.

The measurements behind the sizing guidance documented on the option:
over identical 10-minute soak runs, the unbounded default drifted
165->182 fds; maxconnects=8 -- below the workload's ~15 concurrent
connections -- exploded 320->8517, because undersizing defeats reuse
entirely and the connection churn produces dead connections ~500x
faster than the lazy reaping retires them; maxconnects=64, sized above
peak concurrency, gave the best curve of all at 162->173. Hence the
rule stated in the README, the C handler, and the CHANGELOG: size it
above peak concurrent connections or leave it unset.

203 tests, dialyzer, xref, and lint pass.
@puzza007
puzza007 force-pushed the maxconnects-option branch from cb17d3a to bf270b7 Compare July 18, 2026 23:59
@puzza007

Copy link
Copy Markdown
Owner Author

Closing unmerged: the soak data shows the correctly-sized option only flattens fd growth marginally over the unbounded default (162→173 vs 165→182 over 10 min), while an undersized value is catastrophic (320→8,517). curl's default cache sizing (4× easy handles) is already adaptive, and max_total_connections covers the rare genuine need to cap connection fds. A knob this hard to reason about isn't worth its footgun.

@puzza007 puzza007 closed this Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant