Benchmark large-repo performance and publish the numbers - #475
Merged
Merged
Conversation
"Slow on big repositories" is the most consistent structural complaint about every established GUI client, and being fast is one of our two strongest claims. We had no numbers, so it was an adjective. Four fixtures. Three generate from a fast-import stream in seconds and are deterministic — seeded content, a fixed epoch, a fixed author — so the same parameters produce the same object ids on any machine, and each isolates one dimension: deep (50,000 commits), wide (50,000 files all modified, 5,000 untracked), refs (5,001 branches, 2,000 tags). Breadth hurts differently from depth, and one combined fixture would give a number that cannot say which dimension moved. The fourth is a real torvalds/linux clone, opt-in, because a synthetic repository cannot stand in for 1.5 million real commits and nobody re-runs a benchmark that starts with a multi-gigabyte download. The harness drives the real Libgit2Backend through the real GitBackend trait. It is behind required-features so the Rust CI gate never builds or links it. Why: three decisions carry the rest. The composite is the point. open_screen issues the ELEVEN reads refreshAll issues, simultaneously, behind a barrier — the only shape that can catch "a slow status blocks everything else on that repo", which an op-at-a-time benchmark is structurally blind to. The barrier is load-bearing: spawn eleven threads without one and the first read finishes before the last starts. Baselines ask the same question, not the cheapest one sharing a name. status returns per-file line counts, so its baseline is git status plus both --numstat diffs; the log baselines are --topo-order because the commit graph's lanes depend on that ordering. The first draft used a plain git log and made our first page look fourteen times slower than git. It is not — it is at parity. Repeats are time-boxed rather than counted, floored at three, because ten repeats of an eight-second log page is thirteen minutes for one table row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The method, what each number does and does not include, and the results. The good. The whole first screen — eleven concurrent reads — costs 255 ms on a 50,000-commit repository and 219 ms on one with 7,001 refs. Opening a repository is 0.11 ms. On the wide fixture, open_screen (5.42 s) equals status alone (5.42 s): eleven concurrent reads cost what the slowest costs rather than the sum, which is repo_locks.rs doing its job and the single thing most worth not regressing. A ten-minute soak ran 2,344 fan-outs with memory flat at 67 to 69 MB and the median identical in both halves, which is the "jank after a while" complaint answered with data. The bad. Opening torvalds/linux costs 15.8 seconds, and ten pages into its history costs two minutes and thirty-eight seconds. Three findings, each with the measurement attached. We are at parity with git log --topo-order on the first page, so the topological sort is the cost and git pays it too — but git pays it once and then skips, while we re-pay it per page, and the per-page cost is flat in depth. collect_ref_map is rebuilt per page as well, which is 16x git's work on a 2,000-commit repository with 7,001 refs. And the obvious cheap fix is ruled out: writing a commit-graph takes git from 9.51 s to 21 ms on the kernel and does nothing for us, because libgit2's revwalk does not read it. Why publish the bad ones: the user who opens a 1.4-million-commit repository and waits is precisely the user who went looking for an alternative. Learning that here is the entire point, and a document that lists only the flattering half is an adjective with extra steps. The published record sits beside the document rather than under site/. #257 asks for a figure on the marketing site and the block for it is written, but the honest headline today is fifteen point eight seconds, and the right response to that number is to fix it rather than to sell with it. Keeping the data out of site/ also means a re-measurement can never trigger the site deployment, which fires on any push to main touching that path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The way an adjective grows back is not malice. It is somebody with a release to cut, a figure that reads badly, and a two-character edit to a JSON file nobody diffs. So the document's table is RENDERED from the published record by the same module that writes it, and this re-renders it and compares byte for byte. The two can only agree if both came out of one pnpm bench. It also rejects a row whose median came from fewer than three samples, a ratio printed against a baseline that only measured process start-up, and an operation key the renderer does not know how to order — which would otherwise sort silently to the end of the table. It cannot check that the numbers are TRUE; nothing short of re-running the benchmark could. It checks that they are consistent and that they cover what the document claims. Both of its inputs live under docs/dev/, which is already in the js path filter in tests.yml, and the last assertion here is that it still is. A guard skippable by exactly the change it polices is the #210 failure mode, and it has shipped here once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jonassaa
force-pushed
the
feat/large-repo-benchmark
branch
from
September 17, 2026 11:45
f2de135 to
4817065
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #257 — the benchmark, the numbers and the guard. The marketing-site half
is deliberately held back; see the last section.
"Slow on big repositories" is the most consistent structural complaint about
every established GUI client, and being fast is one of our two strongest claims.
We had no numbers, so it was an adjective.
What landed
scripts/bench-fixtures.mjs— four fixtures. Three generate fromgit fast-importin seconds, deterministically (seeded content, fixed epoch,fixed author), each isolating one dimension:
deep(50,000 commits),wide(50,000 files, all modified, 5,000 untracked),
refs(5,001 branches, 2,000tags). The fourth is a real
torvalds/linuxclone, opt-in.src-tauri/benches/repo_bench.rs— drives the realLibgit2Backendthrough the real
GitBackendtrait. Behindrequired-features = ["bench"],so
cargo checkandcargo testnever build or link it.pnpm bench— fixtures → measure → publish, end to end.docs/dev/performance.md+docs/dev/benchmark.json— the method,what each number does and does not include, and the results.
test/benchmark.test.ts— re-renders the document's table from thepublished record and fails when they disagree.
Nothing here changes app behaviour, and nothing under
site/**is touched,so merging cannot trigger the site deployment.
What it measures, and what it does not
The backend, not the app: there is no webview in it, so no number includes
React.
open_screennarrows that gap by issuing the eleven readsrefreshAllissues, simultaneously — the only shape that can catch "a slowstatus blocks everything else on that repo" — and
open_screen_ipcadds theserde_jsonencoding that happens before the frontend sees a byte.Every operation a single
gitinvocation can answer carries that invocation asa baseline, and the ratio is against git's work — its wall clock minus
process start-up (12.4 ms per invocation, measured), because we pay none of it.
That is deliberately the comparison that makes us look worse.
What the first run found
The good. The whole first screen costs 255 ms on a 50,000-commit
repository and 219 ms on one with 7,001 refs. Opening a repository is
0.11 ms. On
wide,open_screen(5.42 s) equalsstatusalone (5.42 s) —eleven concurrent reads cost what the slowest costs, not the sum, which is
git/repo_locks.rsdoing its job. A ten-minute soak ran 2,344 fan-outs withmemory flat at 67→69 MB and the median identical in both halves.
The bad. Opening
torvalds/linuxcosts 15.8 s, and ten pages into itshistory costs 2 m 38 s. Filed as #473 with the evidence:
git log --topo-orderon the first page — thetopological sort is the cost, and git pays it too;
cost is flat in depth);
collect_ref_mapis rebuilt per page too — 16× git's work on therefsfixture, with twenty-five times less history than
deep;to 21 ms on the kernel and does nothing for us (63.4 s vs 64.2 s),
because libgit2's revwalk does not read it.
Also filed: #474 —
file_historywalks to the root of history on any file withfewer than
limitcommits, holding the exclusive lock while it does. It is whythe harness picks the most-changed path.
A near miss worth reading
The first draft compared
log_pageagainst a plaingit log -500and made thefirst page look 14× slower than git. It is not — that baseline was asking an
easier question. The correction is in the spec, and it is why the module doc
insists baselines ask the same question as the thing they measure.
Why the site block is not here
#257's last bullet asks for a measured figure on the marketing site. The block
is written, but it is not shipping in this PR: the honest headline today is
15.8 seconds to open the kernel, and the right response to that number is to fix
it rather than to sell with it. It follows #473.
Keeping the record under
docs/dev/rather thansite/src/data/has a secondbenefit worth keeping regardless —
site.ymldeploys on any push tomaintouching
site/**, so with the data there a routinepnpm benchwould haveredeployed the public website as a side effect.
#257 therefore stays open on that last bullet, which is why this says "Part
of" rather than "Closes".
Verification
pnpm test— 4,157 tests, 405 files, all passing (includes the newguard).
cargo test— 1,334 passing, 0 failing;cargo test --no-runbuildszero bench targets, and
cargo build --features bench --bench repo_benchstill builds it.
tsc --noEmitandtsc -p e2e/tsconfig.json --noEmit— clean.passing: doctoring one figure in the published record fails the re-render
check, naming
pnpm benchas the fix. Restored afterwards.required
e2e-linux; this revision only removes the site files and moves therecord, and re-runs the full JS and Rust suites above.
pnpm-lock.yamlandCargo.lockuntouched.🤖 Generated with Claude Code