Skip to content

atelet: keep image file mtimes so Python reuses the image's .pyc files - #1

Open
jiayimeow wants to merge 1 commit into
mainfrom
imagecache-keep-mtimes
Open

jiayimeow wants to merge 1 commit into
mainfrom
imagecache-keep-mtimes

Conversation

@jiayimeow

Copy link
Copy Markdown
Owner

unpackLayer (internal/imagecache/unpack.go) never applies the mtime recorded in a layer's tar headers, so every file and directory in an unpacked image carries the time the node unpacked it.

Python trusts a .pyc only while its source still has the mtime recorded in the .pyc, so every .pyc shipped in an image reads as stale. Each actor recompiles every module it imports, writes the new .pyc files into its rootfs upper, and carries them in every snapshot from then on. The unpacked tree also differs from node to node, while cmd/ateom-microvm/restore.go notes that cross-node consistency relies on a deterministic unpack of the same image.

Changes

  • Regular files get the atime and mtime of their tar entry as soon as they are written.
  • Directories get theirs in the final pass that already restores directory modes, after all of their children are written, so read-only directories are covered too.
  • Symlinks keep the unpack time, because Chtimes follows the link. Hard links share their target's inode.
  • New test: TestUnpackLayer_RestoresModTimes.

Impact (measured)

A/B on the same atelet build, with and without this change. The build is release-0.1 at fa6d949, whose unpack.go is identical to main's.

  • Setup: microVM sandbox class (1 vCPU, 1920 MiB guest) on a c3-highmem-192-metal node.
  • Workload: a recorded 21-step coding-agent trace (SWE-bench astropy-7336, conda-based image), replayed in 4 cycles.
  • Rounds: 3, alternating between the two builds. Each round runs the trace once with pause and once with suspend/resume between cycles.
  • Exec time is the wall time to replay one cycle's steps.
  • In every row, the ranges of the two builds across the rounds don't overlap.
Metric Without With Change
Exec time, cycle 1 (pause / suspend path) 2,128 / 2,169 ms 1,462 / 1,403 ms −31% / −35%
Exec time, all 4 cycles (pause / suspend path) 9,475 / 9,468 ms 8,457 / 8,360 ms −11% / −12%
Resume, median (suspend path) 1,051 ms 939 ms −11%
rootfs upper, sum over 5 suspends 79.9 MB 30.9 MB −61%
Last snapshot, stored in GCS 111.3 MB 101.8 MB −9%

Without the change, the actor rewrote 663 of the image's .pyc files (11.9 MB). 662 of them hold the same bytecode as the shipped copy, apart from the embedded source path. The other one is a module the trace edits.

Cold start from the golden snapshot showed no significant difference.

Testing

  • gofmt, go vet, golangci-lint and go test -race ./internal/imagecache/... are clean. The new test fails without the fix.

  • This package's root-gated tests weren't run locally; CI runs them under sudo.

  • The A/B above ran this change on a microVM cluster.

  • Tests pass

  • Appropriate changes to documentation are included in the PR (none needed)

@jiayimeow
jiayimeow changed the base branch from main to bperf-e2e October 5, 2026 18:21
@jiayimeow
jiayimeow force-pushed the imagecache-keep-mtimes branch from bfa6b47 to 7978174 Compare October 5, 2026 18:21
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🚀 baseline-perf (PR · micro-VM · swebench-astropy-7336)

══════════════════════════════════════════════════════════════
 baseline-perf · swebench-astropy-7336 · 87 s
 substrate 60fd555b · commit 60fd555b
 k8s 1.37
 disk  unknown (not a Compute Engine node)
 arm  B · atelet=atelet@6d61d8b8 (stock)
 node idle: 0.3 busy cores · disk 0% · PSI io 0%, cpu 2%
 node during the run: 1.4 busy cores · disk 1% · PSI io 0%, cpu 13%
══════════════════════════════════════════════════════════════
 Median of 4 cycles       Resume      Exec      Park
 Pause (on the node)      434 ms   5548 ms    529 ms
 Suspend (to GCS)         952 ms   5806 ms   1144 ms
 Cold start (golden)      685 ms         –         –

 ── Pause path (checkpoint kept on the node) ──
 Resume  434 ms
   ateom_restore            424 ms   98%  ████████████████████
   ateapi (outside atelet)    7 ms    2%
   other                      3 ms    1%

 Pause  529 ms
   ateom_checkpoint            501 ms   95%  ███████████████████
   other in atelet Checkpoint   19 ms    4%  █
   other                         9 ms    2%

 ── Suspend path (snapshot to GCS) ──
 Resume  952 ms
   download         497 ms   52%  ██████████
   vm_restore       343 ms   36%  ███████
   other            111 ms   12%  ██

 Suspend  1144 ms
   persist          602 ms   53%  ███████████
   snapshot         402 ms   35%  ███████
   other            140 ms   12%  ██

 Snapshot  2,148 MB image · 235 MB populated (234 MB memory, 0 MB rootfs upper) · – in GCS

 Note: snapshot size in GCS unknown: to select an already authenticated account to use.

📊 baseline-perf --compare (main vs PR)

════════════════════════════════════════════════════════════════════════════════════════
 baseline-perf A/B · swebench-astropy-7336 · 2 rounds · 8 min
 substrate 60fd555b
 k8s 1.37
 disk  unknown (not a Compute Engine node)
 A  atelet=atelet:bperf-a@1dbb08b9 (.@origin/bperf-e2e) · atelet commit 9653564c
 B  atelet=atelet@6d61d8b8 (stock) · atelet commit 60fd555b
════════════════════════════════════════════════════════════════════════════════════════
 Mean (range) of 2 runs             A                       B                   B - A
 Pause path (checkpoint kept on the node)
   Cold start (golden)         763 ms 740-787          679 ms 673-685          -84 ms  -11% *
   Resume, median              591 ms 576-605          434 ms 434-434         -157 ms  -27% *
   Exec, median              6,774 ms 6,582-6,967    5,624 ms 5,548-5,700   -1,150 ms  -17% *
   Park, median                587 ms 575-598          573 ms 529-618          -13 ms   -2%
   Exec, 4 cycles           25,870 ms 25,732-26,009 22,300 ms 21,966-22,634 -3,570 ms  -14% *
   Exec, cycle 1             5,806 ms 5,610-6,003    3,907 ms 3,906-3,907   -1,899 ms  -33% *
 Suspend path (snapshot to GCS)
   Cold start (golden)         730 ms 726-735          676 ms 659-693          -55 ms   -7% *
   Resume, median            1,200 ms 1,187-1,213      974 ms 952-996         -226 ms  -19% *
   Exec, median              6,426 ms 6,415-6,438    5,792 ms 5,778-5,806     -634 ms  -10% *
   Park, median              1,272 ms 1,260-1,284    1,154 ms 1,144-1,163     -119 ms   -9% *
   Exec, 4 cycles           24,812 ms 24,569-25,055 22,962 ms 22,942-22,981 -1,851 ms   -7% *
   Exec, cycle 1             5,513 ms 5,476-5,549    3,826 ms 3,787-3,866   -1,686 ms  -31% *
 Suspend path snapshots
   rootfs upper, total        80.0 MB 80.0-80.0       31.0 MB 31.0-31.0      -49.0 MB  -61% *
   Last memory, populated    270.3 MB 270.2-270.4    252.0 MB 251.8-252.2    -18.3 MB   -7% *
   Last rootfs upper          43.0 MB 43.0-43.0       30.3 MB 30.3-30.3      -12.7 MB  -30% *
   Last snapshot, populated  313.8 MB 313.7-313.9    282.8 MB 282.6-283.0    -31.0 MB  -10% *
   rootfs upper per suspend, A: 2.4 · 10.6 · 10.7 · 13.4 · 43.0 MB
   rootfs upper per suspend, B: 0.0 · 0.0 · 0.1 · 0.7 · 30.3 MB

 Run  Arm       pause exec  suspend exec    cold start   upper total  node
   1  A          25,732 ms     25,055 ms        740 ms       80.0 MB  idle
   2  B          22,634 ms     22,942 ms        673 ms       31.0 MB  idle
   3  A          26,009 ms     24,569 ms        787 ms       80.0 MB  idle
   4  B          21,966 ms     22,981 ms        685 ms       31.0 MB  idle

 Note: * the arms' ranges don't overlap: the difference is bigger than the spread between runs

@jiayimeow
jiayimeow changed the base branch from bperf-e2e to main October 5, 2026 20:13
@jiayimeow
jiayimeow force-pushed the imagecache-keep-mtimes branch from 7978174 to 8f60213 Compare October 5, 2026 20:13
@jiayimeow
jiayimeow force-pushed the imagecache-keep-mtimes branch from 8f60213 to 3e6103f Compare October 12, 2026 22:49
unpackLayer never applied the mtime recorded in a layer's tar headers,
so every unpacked file and directory carried the time the node unpacked
it. Python trusts a .pyc only while its source still has the mtime
recorded in the .pyc, so every .pyc shipped in an image read as stale:
each actor recompiled every module it imported and wrote the new .pyc
files into its rootfs upper, which then went into every snapshot. The
unpacked tree also differed from node to node, while restore relies on
a deterministic unpack of the same image.

Set each regular file's atime and mtime to its entry's mtime once the
file is written. Do the same for each directory in the final pass that
already restores directory modes, after all of its children are written.
Symlinks keep the unpack time, since Chtimes would follow the link, and
hard links share their target's inode.
@jiayimeow
jiayimeow force-pushed the imagecache-keep-mtimes branch from 3e6103f to 65d6c86 Compare October 12, 2026 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant