Use PGO for Cargo

[Kobzol]

View all comments

I downloaded the built Cargo and compared it with the parent commit locally on bors (which is of course not in the PGO training set). The results are quite nice:

Benchmark 1 (26 runs): cargo +babf6ea4aab9ed78ea9400b25b0e7e79a6870f29 test --no-run
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           192ms ± 6.95ms     188ms …  226ms          1 ( 4%)        0%
  peak_rss           96.3MB ±  354KB    95.6MB … 96.9MB          0 ( 0%)        0%
  cpu_cycles          454M  ± 5.92M      448M  …  480M           1 ( 4%)        0%
  instructions        797M  ± 5.68M      796M  …  825M           1 ( 4%)        0%
  cache_references   32.7M  ±  352K     32.0M  … 34.0M           1 ( 4%)        0%
  cache_misses       8.82M  ± 86.9K     8.67M  … 9.10M           1 ( 4%)        0%
  branch_misses      3.76M  ± 41.3K     3.72M  … 3.93M           1 ( 4%)        0%

Benchmark 2 (28 runs): cargo +b2aa8c0474895b96828df9c09182952caf6f6450 test --no-run
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           185ms ± 7.07ms     175ms …  208ms          1 ( 4%)        ⚡-  3.9% ±  2.0%
  peak_rss           90.9MB ±  207KB    90.4MB … 91.3MB          1 ( 4%)        ⚡-  5.6% ±  0.2%
  cpu_cycles          420M  ± 9.31M      406M  …  440M           0 ( 0%)        ⚡-  7.7% ±  0.9%
  instructions        736M  ± 5.47M      735M  …  764M           1 ( 4%)        ⚡-  7.6% ±  0.4%
  cache_references   26.2M  ±  406K     25.7M  … 28.0M           2 ( 7%)        ⚡- 19.9% ±  0.6%
  cache_misses       6.01M  ± 92.7K     5.85M  … 6.28M           1 ( 4%)        ⚡- 31.8% ±  0.6%
  branch_misses      3.47M  ± 55.9K     3.40M  … 3.69M           1 ( 4%)        ⚡-  7.6% ±  0.7%

~7.5% icount improvement, that's pretty cool.

[Kobzol]

@bors try @rust-timer queue

[rust-bors]

💔 Test for 0109abd failed: CI. Failed job:

• try - dist-x86_64-linux-quick (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[rust-bors]

💔 Test for 7353424 failed: CI. Failed job:

• try - dist-x86_64-linux-quick (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[rust-bors]

💔 Test for 1f59755 failed: CI. Failed job:

• try - dist-x86_64-linux-quick (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[rust-bors]

💔 Test for cf8ac50 failed: CI. Failed job:

• try - dist-x86_64-linux-quick (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[rust-bors]

⚠️ A new commit 28266f3021ce80e4e7279cd49ee76b04eedb3f20 was pushed.

This pull request was unapproved.

[Kobzol]

@bors r-

Forgot to normalize paths again (this time before Windows CI fails).

@bors try jobs=dist-x86_64-msvc

[rust-bors]

This pull request was not previously approved.

[rust-bors]

☀️ Try build successful (CI)
Build commit: a4dfced (a4dfced7df570b754aeef0a2bcdbb39c2a8cb6b1)
Base parent: 14ead7d (14ead7d3d509f50efe9afb84673e65885dfd4d07)

[Kobzol]

Good.

@bors r=jieyouxu

[rust-bors]

📌 Commit 28266f3 has been approved by jieyouxu

It is now in the queue for this repository.

[jhpratt]

@bors p=2 scheduling,perf

[rust-bors]

☀️ Test successful - CI
Approved by: jieyouxu
Duration: 3h 12m 54s
Pushing 5a8cd23 to main...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 87e5904 (parent) -> 5a8cd23 (this PR)

Test differences

Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 5a8cd237d4fad99d862aed51fab8cb5345c97013 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. i686-gnu-1: 1h 34m -> 2h 25m (+53.1%)
2. x86_64-rust-for-linux: 54m 22s -> 33m 51s (-37.7%)
3. x86_64-gnu-llvm-22-3: 1h 27m -> 1h 54m (+31.0%)
4. arm-android: 1h 45m -> 1h 13m (-29.9%)
5. dist-arm-linux-gnueabi: 1h 9m -> 1h 29m (+29.6%)
6. dist-x86_64-linux-alt: 1h 55m -> 2h 28m (+28.6%)
7. x86_64-gnu-llvm-22-2: 1h 44m -> 1h 19m (-23.8%)
8. dist-loongarch64-musl: 1h 49m -> 1h 23m (-23.6%)
9. dist-armhf-linux: 1h 9m -> 1h 25m (+22.2%)
10. x86_64-gnu-gcc-core-tests: 14m 17s -> 11m 15s (-21.3%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (5a8cd23): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.2%, 0.3%]	7
Regressions ❌ (secondary)	0.3%	[0.2%, 0.4%]	6
Improvements ✅ (primary)	-0.2%	[-0.5%, -0.1%]	52
Improvements ✅ (secondary)	-0.3%	[-0.5%, -0.1%]	51
All ❌✅ (primary)	-0.2%	[-0.5%, 0.3%]	59

Max RSS (memory usage)

Results (secondary -0.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.5%	[2.5%, 2.5%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.8%	[-3.8%, -3.8%]	1
All ❌✅ (primary)	-	-	0

Cycles

Results (primary -2.1%, secondary -4.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.1%	[-2.1%, -2.1%]	1
Improvements ✅ (secondary)	-4.0%	[-4.4%, -3.8%]	3
All ❌✅ (primary)	-2.1%	[-2.1%, -2.1%]	1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 486.983s -> 487.476s (0.10%)
Artifact size: 391.82 MiB -> 387.96 MiB (-0.99%)

[panstromek]

How come that this is perf improvement on rustc-perf? I though we don't measure cargo overhead.

[Kobzol]

Good question. In theory, we shouldn't be, but I know that often when the Cargo submodule is updated, it tends to produce slight changes in perf. results. The only explanation I have for that is that slight changes in Cargo and rustc invocations made by it somehow perturb the collection of PGO profiles 🤷‍♂️