remote: use deques for the RPC call queues, fixes #10450

[ThomasWaldmann]

Fixes #10450 (1.4-maint).

RemoteRepository.call_many took the next call from the front of a list (calls.pop(0), self.preload_ids.pop(0)). Each of these moves all remaining entries. For a single item with millions of chunks (preload of all its chunk ids + get_many over them), extract from a remote repository was therefore quadratic in the chunk count of that item.

This PR uses collections.deque with popleft() for calls, waiting_for and preload_ids. preload_ids is fed by preload() for every item, so reversing the list (to pop from the end) would need a prepend per item, which is quadratic too. A deque avoids that.

Measurements

Measured on a Ryzen 5 8500GE (Zen 4), Python 3.13, repokey-blake2, lz4. The archive has one stdin item of N unique 4 KiB chunks (fixed,4096). The data compresses to almost nothing, so the times show per-chunk cost. The command is borg extract --stdout ::a big. For remote runs the repo is ssh://__testsuite__/... (borg serve over pipes).

chunks	local	remote, this PR	remote, 1.4-maint
250 k	2.7 s	4.2 s	9.4 s
500 k	5.1 s	8.2 s	28.2 s
1 M	10.2 s	15.6 s	115.0 s
2 M	20.0 s	31.3 s	862.3 s
4 M	40.2 s	62.0 s	–
12.3 M	122.9 s	190.8 s	– (extrapolated: 8+ h)

With this PR, remote extract is linear at about 15.5 µs per chunk up to 12.3 M chunks, the size from the issue. Peak RSS does not change.

On master, borg2's own repository access has no such lists. The calls.pop(0) in legacy/remote.py is only reached via borg transfer --from-borg1 --chunker-params from a remote borg 1.x repo. This PR does not cover that path.

Tests

pytest src/borg/testsuite/remote.py src/borg/testsuite/archiver.py -k "Remote or remote": 255 passed, 22 skipped.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.09%. Comparing base (8e13924) to head (94d840a).
⚠️ Report is 1 commits behind head on 1.4-maint.

Additional details and impacted files

@@              Coverage Diff              @@
##           1.4-maint   #10454      +/-   ##
=============================================
- Coverage      82.13%   82.09%   -0.04%     
=============================================
  Files             38       38              
  Lines          11478    11479       +1     
  Branches        1808     1808              
=============================================
- Hits            9427     9424       -3     
- Misses          1466     1469       +3     
- Partials         585      586       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.