Skip to content

fix(acp): release owner lock on failed claims; stabilize ACP run test - #257

Closed
kumanday wants to merge 1 commit into
mainfrom
claude/fix-acp-flaky-tests
Closed

kumanday wants to merge 1 commit into
mainfrom
claude/fix-acp-flaky-tests

Conversation

@kumanday

@kumanday kumanday commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Two rust CI tests failed intermittently on unrelated PRs (e.g. #253). Both come from races; neither was a slow runner hitting a timeout.

1. opensymphony_acp::durable::tests::reuse_checks_profile_scope_workspace_repository_and_checkout_binding (real product bug)

Durability::claim takes an exclusive flock on acp-owner.lock and then validates bindings. When a check fails (Binding("profile"), Binding("profile fingerprint"), InvalidMetadata, …), the locked File was simply dropped. Only Durability::drop called unlock(), and the claim had failed before a Durability existed.

A flock belongs to the open file description. If another thread forks a child at that moment (the lib test binary runs about 1,480 tests in parallel, and many spawn processes), the child inherits the descriptor until exec closes it. A plain close in the parent then leaves the lock held, so the next claim's try_lock gets WouldBlock and returns DurabilityError::Owned instead of the expected Binding(..). That is exactly what CI showed: the "credential scope" iteration failed right after two Binding rejections (run 42456882698, lib binary at 118s under load). In production, a rejected claim could also briefly make a legitimate concurrent claim see a spurious Owned.

Fix: wrap the lock in an OwnerLock guard right after try_lock succeeds. The guard unlocks explicitly on every drop path, and Durability holds the guard in place of its own Drop impl.

2. run::run_dispatches_acp_with_exact_cwd_hooks_and_no_openhands (test race)

The mock Linear issue stays active. After each successful ACP run the scheduler queues a continuation retry, and persist_retry_pending rewrites run.json to status: preparation_failed with pending_retry: true and retry_reason: continuation. The test polled every 25 ms for status == "succeeded", a state that lasts only a few milliseconds. In the failing CI run (42457696381, attempt 1) the after_run hook finished at 07:40:31.643 and run.json was overwritten at .648. The test missed every window for 17 attempts and timed out with run.json at attempt 17.

Fix: the test now accepts either succeeded or a pending continuation retry with no retry_error, as long as that same manifest recorded a successful after_run hook. Continuation retries are queued only after a normal worker exit; failures use failure/stalled, so the check is still strict. The timeout is unchanged.

Evidence

  • Lock mechanism, reproduced directly: a Python fcntl.flock + os.fork experiment with the same kernel semantics. The lock is held, a child is forked and sleeps before exiting, the parent closes the fd, then the lock file is reopened and locked:
    • close only: next claim WouldBlock -> DurabilityError::Owned
    • explicit LOCK_UN before close: next claim acquired lock
  • Repetition, macOS, 16 cores: each iteration ran the run test, the full opensymphony_acp:: lib suite (53 tests, --test-threads 64) and the durable test concurrently.
    • 25/25 iterations passed for all three.
    • 20/20 more passed with 16 CPU busy-loop processes for extra load. The slowest run-test pass was 4.76 s, against the 20 s deadline.
    • I could not reproduce the original failure on this machine before the fix, so these runs show the fixed tests pass reliably, not a before/after difference. The root-cause evidence is the CI logs plus the flock experiment above.
  • cargo fmt --all --check: clean
  • cargo clippy-dev (--all-targets -D warnings): clean

Durability::claim dropped its locked File without an explicit unlock on
binding and metadata rejection paths. A child forked concurrently (before
exec closes CLOEXEC descriptors) shares the open file description, so the
flock survived the failed claim and the next claim saw a spurious Owned.
Wrap the lock in an OwnerLock guard that unlocks on every drop path.

run_dispatches_acp_with_exact_cwd_hooks_and_no_openhands polled for
run.json status `succeeded`, which the continuation retry rewrites to
`preparation_failed` + pending_retry within milliseconds because the mock
issue stays active. Accept a pending continuation retry on a manifest that
recorded a successful after_run hook as equivalent normal-exit evidence.

Co-Authored-By: Claude <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T18:05:09.330647Z 7cd279e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kumanday kumanday closed this Nov 8, 2026
@kumanday
kumanday deleted the claude/fix-acp-flaky-tests branch November 8, 2026 18:20
@kumanday

kumanday commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

Superseded by #258 (same commit, head branch fix/acp-flaky-tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant