Observed
On the #228 branch, just focus-test assets after a change to one file, guest/artifacts/diagnostics/test_sandbox.py (the in-VM doctor's guest-binary list), took:
| step |
duration |
assets.build.x86_64 |
11.0 min |
assets.build.arm64 |
10.9 min |
assets.asset-dependencies |
0.5 min (dependency image reused) |
| everything else |
about 1 min each or less |
The run before it had real image input changes (Cargo.lock, capsem-init, the guest agents, the Dockerfiles). It took 17.4 and 13.7 min. A one-file test change costs almost as much as a full rebuild.
Why (from reading the pipeline, not yet profiled step by step)
The layer order in config/docker/Dockerfile.rootfs.j2 is fine: COPY diagnostics/ /usr/local/lib/capsem_tests/ comes late, and the dependency image is reused. The time goes to work every rootfs build repeats over the entire filesystem, whatever changed:
find / -xdev ... -exec chmod u-s,g-s and its re-scan;- exporting the container filesystem;
- erofs packing, hashing and OBOM validation;
- for x86_64, all of it under emulation on an arm64 host.
In-VM test tooling (diagnostics/ → /usr/local/lib/capsem_tests, capsem_bench/, capsem-doctor, capsem-bench) changes far more often than the rootfs, and every edit pays the whole-image cost on both architectures.
Fix sketch
- Profile inside
assets.build.<arch> first, so the fix targets the real split: layer build, the setuid scan, export, erofs pack, hash/OBOM.
- Take the test tooling out of the rootfs identity. Ship
capsem_tests / capsem_bench and the doctor/bench entry points as a separate small artifact (their own erofs, or the initrd, which already has a fast repack path via /build-initrd). Mount it read-only at the same paths, with its own hash in the manifest. Then a doctor edit only repacks that small artifact.
- Whatever stays in the rootfs: keep the setuid strip/verify in the layer that installs packages, not as a whole-filesystem pass after the last
COPY.
Keep the invariants: guest test tooling stays read-only and hash-verified, and a changed test file must still invalidate what ships it. The cached rootfs must not be reused with stale diagnostics.
Observed
On the #228 branch,
just focus-test assetsafter a change to one file,guest/artifacts/diagnostics/test_sandbox.py(the in-VM doctor's guest-binary list), took:assets.build.x86_64assets.build.arm64assets.asset-dependenciesThe run before it had real image input changes (
Cargo.lock,capsem-init, the guest agents, the Dockerfiles). It took 17.4 and 13.7 min. A one-file test change costs almost as much as a full rebuild.Why (from reading the pipeline, not yet profiled step by step)
The layer order in
config/docker/Dockerfile.rootfs.j2is fine:COPY diagnostics/ /usr/local/lib/capsem_tests/comes late, and the dependency image is reused. The time goes to work every rootfs build repeats over the entire filesystem, whatever changed:find / -xdev ... -exec chmod u-s,g-sand its re-scan;In-VM test tooling (
diagnostics/→/usr/local/lib/capsem_tests,capsem_bench/,capsem-doctor,capsem-bench) changes far more often than the rootfs, and every edit pays the whole-image cost on both architectures.Fix sketch
assets.build.<arch>first, so the fix targets the real split: layer build, the setuid scan, export, erofs pack, hash/OBOM.capsem_tests/capsem_benchand the doctor/bench entry points as a separate small artifact (their own erofs, or the initrd, which already has a fast repack path via/build-initrd). Mount it read-only at the same paths, with its own hash in the manifest. Then a doctor edit only repacks that small artifact.COPY.Keep the invariants: guest test tooling stays read-only and hash-verified, and a changed test file must still invalidate what ships it. The cached rootfs must not be reused with stale diagnostics.