Skip to content

cli: reduce mirror clone latency - #68

Merged
leolannenmaki merged 3 commits into
mainfrom
cli/faster-mirror-clone
Oct 30, 2026
Merged

leolannenmaki merged 3 commits into
mainfrom
cli/faster-mirror-clone

Conversation

@Ormod

@Ormod Ormod commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

A clone with one available mirror previously performed an unnecessary probe, minted a second token through Git's credential helper, and started the CLI for a credential-store no-op. Discovery and token issuance also ran sequentially. This change removes that overhead and overlaps the independent API requests for owner/repo clones.

  • Use a single advertised mirror directly; keep racing when multiple mirrors are available.
  • Request mirror discovery and the Git token concurrently for repository-name inputs. Repository-ID inputs still resolve the canonical name first.
  • Validate discovery and the selected placement before using credentials. Discovery errors take precedence over speculative token errors. If discovery returns a different name, including a case change, discard the speculative result and request credentials for the exact returned name.
  • Pass the initial token to Git through a URL-scoped, process-only authorization header. Keep credentials out of argv and repository configuration, disable redirects, and retain the helper for fresh tokens on later fetches.
  • Handle credential store and erase in the shell wrapper without starting the CLI. Update command documentation and generated references.

The usual single-mirror path drops from 8 to 5 HTTP requests and from 3 to 1 CLI process starts. Parallel issuance can mint a token even if discovery or placement validation subsequently fails; no probe or clone receives it in those cases.

Five alternating before/after pairs measured the additional parallelization against commit 9d499e0 (which already has the other optimizations). With the same Python runtime, mirrored repository revision, and Finland endpoint git.eu-fi1.a.avrea.com, median clone time fell from 0.617 s to 0.493 s (~20%). Tests were finished before this benchmark. Separate traces took 0.508 s sequentially / 0.428 s concurrently and confirmed both API requests start together. These are local measurements; network latency varies.

Validation: make test (1,551 passed), make lint, make docs-check, and whitespace checks passed. A synchronization barrier verifies request overlap without relying on elapsed-time thresholds. Regression coverage includes canonical-name changes, discovery/token error precedence, scoped credentials, unsafe placements, and sequential ID resolution. Native Git integration tests verify fresh tokens on later fetches, no persisted credentials, and blocked redirects.

@leolannenmaki leolannenmaki left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scope: both commits at 31830c3 (base ccf52e8; main has since taken #66, unrelated). Read every hunk in repo.py, git_mirror.py, the tests and the docs, plus the server side on avrea-core main: /tokens/git (claims keyed by the requested casing, unresolvable repositories silently omitted), /repos/{owner}/{repository}/mirror/session (case-insensitive lookup), candidate_endpoints, and avrea-git's claim lookup (exact-case map, 404 for a missing claim). That confirms the discard-on-rename branch is both necessary and correct.

Ran at head in a scratch clone: make test (1551 passed), make lint, make docs-check, all clean. CI 10/10 green.

Source-informed probes with git 2.56:

  • -c reset followed by --config-env append collect in that order for http.<url>.extraHeader (git config --show-scope --get-all shows the empty entry, then the Authorization entry).
  • The clone argv from clone_mirror against a local mirror answering 401, 404 and 503, compared with nearest_mirror on the same single endpoint.
  • The environment of a core.hooksPath post-checkout hook during a --config-env clone.

Not exercised: a live clone against staging or prod mirrors, and Git older than 2.31.

Findings are inline: 1 Medium (nonblocking), 2 Low.

Comment thread avrea_cli/commands/repo.py Resolved
Comment thread avrea_cli/git_mirror.py Resolved
Comment thread avrea_cli/git_mirror.py Resolved

@jtiensuu-avrea jtiensuu-avrea left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold for one required security fix: the initial clone token can enter
configuration-based Git Trace2 logs. Details and the verified mitigation
are inline.

Reviewed 31830c3 against ccf52e8, including an independent second pass.
Two source-informed native-Git probes reproduced the disclosure at head
and not at base. Nine focused tests pass, and all ten CI checks passed on
the reviewed revision. The existing tracing test also passes at base and
head because it checks environment cleanup rather than the trace output.

The probes use a dummy token and establish disclosure before network I/O;
they do not establish a successful clone or subsequent token misuse.
Performance claims were not rebenchmarked.

Comment thread avrea_cli/git_mirror.py Resolved
Probe directly selected mirrors after clone failures while preserving Git output and exit codes. Document Git 2.31+, clarify child-process token visibility, and cover recovery hints with native Git tests.
@leolannenmaki
leolannenmaki merged commit df78a0d into main Oct 30, 2026
10 checks passed
@leolannenmaki
leolannenmaki deleted the cli/faster-mirror-clone branch October 30, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants