Skip to content

fix(storage): prevent unpublished ready-pack entries from causing false dependency cycles (#723) - #737

Open
markankamaransheogarat-sys wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
markankamaransheogarat-sys:fix/concurrent-first-installs-cycle-723
Open

markankamaransheogarat-sys wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
markankamaransheogarat-sys:fix/concurrent-first-installs-cycle-723

Conversation

@markankamaransheogarat-sys

Copy link
Copy Markdown

Fixes #723.

Root Cause

During concurrent first-wins saves, an unpublished candidate in Packer::ready can collide by ObjectId with a record already committed by another writer in Fetch, producing two false-positive Integrity("dependency cycle") paths:

  1. WaveSource shadowing committed records (core/crates/layerfs-storage/src/save/source.rs): WaveSource::base_location (and WaveSource::location outside lost-candidate reconstruction) checked self.packer.location(id) before self.fetch. If a committed object's base chain walked into an ObjectId that also had an unpublished tentative copy in Packer::ready pointing in the opposite direction, DepthCache::cost_of / Resolver mixed the committed and unpublished graphs and reported a cycle.
  2. Wave::resolve walking from an unpublished root to the committed winner (core/crates/layerfs-storage/src/encoding/delta/read.rs): When acknowledge() loses first-wins on id and calls Wave::resolve(id) to reconstruct its own candidate bytes from Packer::ready, the root (chain[0]) has pack_id_a (unpublished) while its committed base chain in Fetch can legitimately reference the winner's committed id at pack_id_b. Checking only object_id in Resolver::read_with_caches treated pack_id_a and pack_id_b as the same node.

Changes

  • WaveSource::base_location now queries self.fetch.base_location first and falls back to self.packer.location only if absent.
  • WaveSource::location queries self.fetch.location first by default, and sets prefer_packer: true only in Wave::resolve (reconstructing this Save's own lost candidate).
  • Resolver::read_with_caches cycle detection compares full record locators (object_id, pack_id, group_number, record_number) instead of object_id alone.
  • Added deterministic regression tests in core/crates/layerfs-storage/tests/concurrent_winner_chain.rs.

Verification

  • Reproduced the exact Integrity("dependency cycle") failure with a 2-thread concurrent_first_installs stress harness over MemoryMetadata (failed within 1–27 iterations before the fix; passed 100/100 iterations after the fix).
  • cargo test -p layerfs-storage (all test targets pass).
  • cargo clippy -p layerfs-storage --tests --no-deps -- -D warnings and rustfmt --check on modified files pass.
  • Note: layerfs-persistence integration tests (concurrent_first_installs) are #![cfg(target_os = "linux")] and were not run locally on Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: concurrent_first_installs fails with Integrity("dependency cycle") (~4%)

2 participants