Repository navigation
cli: reduce mirror clone latency - #68
Conversation
leolannenmaki
left a comment
There was a problem hiding this comment.
Scope: both commits at 31830c3 (base ccf52e8; main has since taken #66, unrelated). Read every hunk in repo.py, git_mirror.py, the tests and the docs, plus the server side on avrea-core main: /tokens/git (claims keyed by the requested casing, unresolvable repositories silently omitted), /repos/{owner}/{repository}/mirror/session (case-insensitive lookup), candidate_endpoints, and avrea-git's claim lookup (exact-case map, 404 for a missing claim). That confirms the discard-on-rename branch is both necessary and correct.
Ran at head in a scratch clone: make test (1551 passed), make lint, make docs-check, all clean. CI 10/10 green.
Source-informed probes with git 2.56:
-creset followed by--config-envappend collect in that order forhttp.<url>.extraHeader(git config --show-scope --get-allshows the empty entry, then the Authorization entry).- The clone argv from
clone_mirroragainst a local mirror answering 401, 404 and 503, compared withnearest_mirroron the same single endpoint. - The environment of a
core.hooksPathpost-checkout hook during a--config-envclone.
Not exercised: a live clone against staging or prod mirrors, and Git older than 2.31.
Findings are inline: 1 Medium (nonblocking), 2 Low.
jtiensuu-avrea
left a comment
There was a problem hiding this comment.
Hold for one required security fix: the initial clone token can enter
configuration-based Git Trace2 logs. Details and the verified mitigation
are inline.
Reviewed 31830c3 against ccf52e8, including an independent second pass.
Two source-informed native-Git probes reproduced the disclosure at head
and not at base. Nine focused tests pass, and all ten CI checks passed on
the reviewed revision. The existing tracing test also passes at base and
head because it checks environment cleanup rather than the trace output.
The probes use a dummy token and establish disclosure before network I/O;
they do not establish a successful clone or subsequent token misuse.
Performance claims were not rebenchmarked.
Probe directly selected mirrors after clone failures while preserving Git output and exit codes. Document Git 2.31+, clarify child-process token visibility, and cover recovery hints with native Git tests.
A clone with one available mirror previously performed an unnecessary probe, minted a second token through Git's credential helper, and started the CLI for a credential-store no-op. Discovery and token issuance also ran sequentially. This change removes that overhead and overlaps the independent API requests for
owner/repoclones.storeanderasein the shell wrapper without starting the CLI. Update command documentation and generated references.The usual single-mirror path drops from 8 to 5 HTTP requests and from 3 to 1 CLI process starts. Parallel issuance can mint a token even if discovery or placement validation subsequently fails; no probe or clone receives it in those cases.
Five alternating before/after pairs measured the additional parallelization against commit
9d499e0(which already has the other optimizations). With the same Python runtime, mirrored repository revision, and Finland endpointgit.eu-fi1.a.avrea.com, median clone time fell from 0.617 s to 0.493 s (~20%). Tests were finished before this benchmark. Separate traces took 0.508 s sequentially / 0.428 s concurrently and confirmed both API requests start together. These are local measurements; network latency varies.Validation:
make test(1,551 passed),make lint,make docs-check, and whitespace checks passed. A synchronization barrier verifies request overlap without relying on elapsed-time thresholds. Regression coverage includes canonical-name changes, discovery/token error precedence, scoped credentials, unsafe placements, and sequential ID resolution. Native Git integration tests verify fresh tokens on later fetches, no persisted credentials, and blocked redirects.