Skip to content

test(hosts): copy the Codex native cache without fs::copy - #4673

Merged
ScriptedAlchemy merged 2 commits into
masterfrom
fix/codex-relocated-bundle-cache
Oct 31, 2026
Merged

ScriptedAlchemy merged 2 commits into
masterfrom
fix/codex-relocated-bundle-cache

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Problem

//crates/tracedecay-agent-hosts:unit_test failed twice on unrelated PRs (Linux CI, 2026-10-31) in
agents::codex::tests::native_cache_content_drift_and_binary_relocation_require_refresh: after the test refreshed
the Codex plugin cache, exactly one cache file differed by content only (same length, different sha256), so the
activation probe correctly refused to call the plugin active.

Verified root cause

  • No size/mtime shortcut exists. The probe hashes bytes: codex_loaded_cache_matches_rendered_bundle compares
    rendered, source, and cache content digests (crates/tracedecay-agent-hosts/src/agents/bundle_identity.rs:26-58,
    agents/codex.rs:1454-1495), and every refresh write publishes bytes
    (agents/codex.rs:775-786, agents/text_file_transaction.rs:473-511). The corrupted skill text is static
    (plugin/skills/assessing-impact/SKILL.md is embedded at build time), so "refresh skipped an unchanged-size file"
    cannot explain it.
  • The divergence is introduced by the fixture's copy. The Codex fixture materialized the host cache with
    std::fs::copy (crates/tracedecay-agent-hosts/src/agents/codex/tests.rs:773 at b989d1f6e5), while the
    otherwise equivalent Claude fixture copies explicit bytes.
  • Controlled comparison across the Linux CI fleet (7 runs): the Claude fixture passed 7/7, the Codex fixture
    failed 2/7 (/tmp/td-ci/4668, /tmp/td-ci/4669). Each failure named exactly one file the fixture had just
    copied — cache/skills/assessing-impact/SKILL.md and, in the other run, cache/.mcp.json — with identical
    length and a different digest. The skill's bytes were identical before the refresh, so the copy itself produced
    the wrong bytes, not a stale render.
  • Why Linux-only: std::fs::copy is not an in-process byte copy. On Linux io::copy moves data between the
    file descriptors with copy_file_range(2)/sendfile(2)/splice(2) (pinned std 1.99.1,
    library/alloc/src/io/copy.rs:73-84); macOS clones or copyfile(3)s
    (library/std/src/sys/fs/unix.rs:2202-2249). The same fixture therefore passes on macOS.
  • This already happened to the sibling fixture: the Claude fixture was fixed in 03212da800
    ("test(hosts): avoid Linux native cache copy offload", merged in Bound storage-report sizing to a shared filesystem budget #4353). The Codex fixture kept the crate's last
    std::fs::copy.

Not reproduced locally. macOS is green before and after (pre-fix run, then --runs_per_test=3); no Linux host
was available. The evidence is the 7-run cross-host comparison plus the repository's earlier identical fix.

Fix

  • New agents/native_cache_fixture.rs is the single byte-level authority both host fixtures use to materialize a
    native cache (std::fs::read + std::fs::write, never std::fs::copy), with the reason recorded so a future
    fixture cannot reintroduce the offload path.
  • assert_rendered_bundle_bytes now runs inside the Codex test immediately after the relocation refresh, so a cache
    divergence fails at the refresh site naming the file and both len/sha256 values instead of surfacing as an
    opaque activation failure. The Claude test uses the same assertion.
  • New native_cache_same_size_unchanged_mtime_content_drift_deactivates pins that activation is content-based: a
    same-length byte change with the original mtime restored still deactivates the plugin.
  • No product code changed: the probe already compares content digests and truthfully reported the diverged cache.

Evidence

  • bazel test //crates/tracedecay-agent-hosts:unit_test --runs_per_test=3 --nocache_test_results → 3/3 PASSED
    (561 passed, 0 failed, 1 ignored per run), including both refresh fixtures and the new drift test.
  • bazel build --config=clippy //crates/tracedecay-agent-hosts:unit_test → clean with -Dwarnings.
  • bazel build //crates/tracedecay-agent-hosts:rustfmt_check_2024 → clean.
  • bazel run //scripts:gen_builds -- --check → "all 61 generated BUILD.bazel files are current".

🤖 Generated with Claude Code


Devin Review

The Codex native-cache fixture materialized its cache with
`std::fs::copy`, while the byte-identical Claude fixture copies explicit
bytes. Across the seven Linux CI runs in this fleet, the Claude fixture
passed 7/7 and the Codex fixture failed 2/7, always with exactly one
cache file differing by content only (same length, different sha256) at
a file the fixture had just copied.

`std::fs::copy` does not copy bytes in-process: on Linux `io::copy`
moves data between the file descriptors with `copy_file_range(2)`,
`sendfile(2)`, or `splice(2)` (std 1.99.1,
`library/alloc/src/io/copy.rs`), while macOS clones or `copyfile(3)`s
(`library/std/src/sys/fs/unix.rs`). A byte-level defect in the CI
filesystem's offload path is therefore invisible on macOS. The Claude
fixture already hit this and was fixed in 03212da ("test(hosts):
avoid Linux native cache copy offload"); the Codex fixture kept the
crate's last `std::fs::copy`.

Materialize both hosts' caches through one byte-level fixture authority,
assert the refreshed source and cache bytes at the refresh site, and pin
that a same-size byte change with an unchanged mtime still deactivates
the plugin. The activation probe was already content-based
(`bundle_identity.rs`), so no product code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a879898

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@ScriptedAlchemy
ScriptedAlchemy enabled auto-merge (squash) October 31, 2026 21:01
@ScriptedAlchemy
ScriptedAlchemy merged commit 9f8a565 into master Oct 31, 2026
16 checks passed
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

Independent audit correction: the CI artifacts demonstrate same-length cache-byte divergence, and the explicit read/write fixture retains exact-byte assertions. They do not establish Linux copy-offload or a kernel implementation as the cause; no controlled Linux reproduction isolated that mechanism. Treat the change as a byte-fidelity workaround with an unestablished underlying cause. PR #4679 corrects the maintained source comment; the production activation probes were not changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant