Concurrency scaling is sub-linear: 4x jobs (8->32) only buys ~1.5x wall-clock

[awto-au]

Observation

Two full-corpus baseline runs (510 .c TUs, same c2rust rev 67df081, same corpus rev) back to back on the same host:

jobs	wall-clock	per-file p50	per-file p90	per-file max	peak RSS p50/p90/max
8	~5m40s	5.1s	8.5s	23.4s	532MB / 649MB / 767MB
32	~3m43s	12.1s	22.3s	71.3s	similar range

Peak RSS per process barely changed between the two runs, and host memory never dropped below ~30GB available (of 62GB total) at 32 jobs — this isn't a memory-pressure story. But per-file wall-clock roughly tripled going from 8 to 32 concurrent jobs, so a 4x increase in concurrency only bought a ~1.5x reduction in total wall-clock. That's well short of linear scaling and suggests real contention somewhere other than RAM.

What to find out

• Is this just CPU oversubscription (32 jobs on 32 logical cores, plus OS/desktop overhead, is a reasonable ceiling and this is simply expected diminishing returns)? Or is there contention specific to c2rust's own process — e.g. Clang's -nostdinc/header search re-parsing the same kernel headers independently in every process (no shared PCH/module cache across invocations), lock contention in a shared resource, or filesystem I/O contention writing overlapping AST-export intermediate state?
• Would a -fmodules/precompiled-header approach, or some other shared-state mechanism between concurrent c2rust transpile invocations, meaningfully cut the per-file cost under concurrency?

Context

Measured via linux-rs's scripts/run_c2rust_baseline.py (adaptive --jobs, now computed from real peak-RSS history instead of the RLIMIT_AS ceiling) against this fork's corpus of kernel .c files. Not urgent — current throughput is usable — but worth understanding before assuming higher concurrency is free.

[awto-au]

Diagnosis: plain CPU oversubscription, not a hidden contention source

Built the fork at 67df081 in an isolated worktree and ran solo-vs-contended experiments on kernel/pid.c (one of the slower files in the original run).

Solo (idle host, perf stat): 17.16s wall, 17.10s task-clock, cache-miss ratio ~18% (177.8M/973.2M, perf-core PMU).

Contended (30 other real kernel .c files launched concurrently, 31-way total on 32 cores): 22.4-25.8s wall (perf stat and /usr/bin/time -v agreed), task-clock ≈ elapsed in both cases, 883 involuntary context switches, cache-misses +45% (260.3M).

The tell: task-clock tracks wall-clock almost exactly in both runs. If this were I/O or lock contention, task-clock would sit well below elapsed (process blocked, not running). Instead the process gets essentially all the CPU-seconds it needs, just spread over more wall-clock because 30+ equally CPU-bound peers are fair-sharing 32 cores. ps//proc/<pid>/status sampling mid-run showed every c2rust-transpile process in R (running), none in D or blocked S — real CPU contention, not disk I/O or a lock.

Header re-parsing is real but isn't the mechanism of the slowdown. Every process independently #includes kernel headers with zero sharing — pid.c's 974-line source expands to a 95,267-line preprocessed TU (~98x), fully lexed/parsed per process (c2rust-ast-exporter has no -fmodules/PCH/module-cache support; AST export runs in-process via FFI, not a subprocess). That's a large fixed per-process cost, but it doesn't grow under contention — it's simply multiplied by process count, same as always. It explains why total CPU demand is high, not why scaling is sub-linear past ~8 jobs.

Tried -include-pch as an experiment: not a safe fix as-is. A PCH built from the exact per-TU flags let transpile run, but produced a different (and eventually panicking) AST vs plain -include — extra "Missing child" warnings then a TagTypeUnknown panic in conversion.rs:1024. Matches an earlier PCH-divergence experiment in linux-rs (5/104 lib/*.c files diverged nopch vs pch), likely from interaction with GNU label-address (&&label/_THIS_IP_) handling. Needs real validation work before it's trustworthy — not a quick win.

No shared filesystem/lock contention found. create_temp_compile_commands already uses a unique tempfile::Builder dir per invocation (parallel-safe by design); no global cache path under $HOME//tmp; no cross-process mutexes/locks in c2rust-ast-exporter/c2rust-transpile. rustfmt runs per output file on a small .rs, not a shared resource.

Bottom line: 32 jobs on 32 logical cores is oversubscription, and each process is fully CPU-bound for ~17-70s with a large, non-shareable per-process parse cost. No separate lock/IO/cache-contention bug underneath — expected diminishing returns from more CPU-bound processes than cores. ~8-16 jobs looked like a more efficient operating point than 32 on this host. The only lever that changes the fixed per-process cost itself (not just reshuffles it) would be a real header-sharing mechanism — validated -fmodules, or a PCH built carefully enough not to change AST shape — and that needs real engineering, not a quick config flag.