Skip to content

fix(automation): publish scheduler pause state atomically - #4357

Merged
ScriptedAlchemy merged 31 commits into
masterfrom
copilot/fix-scheduler-pause-state-issue
Oct 23, 2026
Merged

ScriptedAlchemy merged 31 commits into
masterfrom
copilot/fix-scheduler-pause-state-issue

Conversation

Copilot AI commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Persist scheduler pause changes through the private filesystem authority so readers observe complete control states and a successful save follows namespace durability barriers. Create missing dashboard ancestors durably before atomic publication, retain typed corruption errors, and preserve staging cleanup and prior state on pre-publication failure.

Windows publication uses the shared write-through MoveFileExW authority for replacement and no-replace operations. Open readers allow deletion sharing, so existing readers cannot prevent replacement. Coverage includes denied-delete cleanup, long Unicode paths, invalid NUL paths, and no-replace semantics. Rebuildable accelerator writes retain their lighter path.

Ready pull requests retain master's native Windows workspace build and the required Test Windows check name. Manual native tests can select macOS or Windows and explicit Bazel target patterns. Draft pull requests skip heavy builds. Codex exact-session lookup fixtures force retained continuation across date buckets independently of filesystem entry ordering.

Validation:

  • Integration 1afb526432ec3bc7dcd8c099fd16f6440d67bdd4 merges master 12e828ec84. The only manual conflict was in private-fs/tests/framed_log.rs: retain all Windows publication tests and master's Linux barrier-observation regression, including exact member/unrelated barrier counts, byte readback, and missing-member refusal. Scheduler production source is unchanged.
  • On that merged tree, all six Linux framed-log tests passed uncached remotely, and both private-fs and automation-runtime Bazel formatting checks passed (00f74eaf-d255-45b2-97c8-3cb4aa794f86). The Windows-only atomic-publication target compiled with zero Linux cases; this is not Windows runtime evidence.
  • All five scheduler-control tests passed uncached remotely, with zero failures/ignored and 427 unrelated tests filtered, in 0.19 seconds (df6f87e4-2961-46bb-8e9f-3c24a4bbeeb5). The wrapper initially expected one test; the corrected count check against the Bazel log confirmed five. Rustc/TestRunner were remote, local fallback disabled, priority 10, minimal output downloads. Frozen pnpm install, whitespace, and the commit hook passed. The normal push passed a fresh ancestry guard. Full hosted CI on the updated head remains required before merge.

Historical evidence:

  • Earlier head 55d45866e895dcf5b400d4379a0a7d584a307c01 only moves the new scheduler test imports to module scope after c98ee67d71. All five scheduler-control tests passed remotely, with no ignored cases (427 unrelated tests filtered): atomic read/write, missing ancestors, ancestor sync failure, concurrent readers, and failed staging. Invocation: 5c9b7ce9-821a-434d-8355-d99295bc751e. Rust compilation and tests used remote workers with local fallback disabled; the Bazel formatting action passed locally in a sandbox (d098d60f-4866-458c-bf63-7b00bec8f103). Whitespace and commitlint passed. Required hosted CI must pass on this updated head before merge.
  • On parent c98ee67d71, hosted Linux fleet and aggregate, Clippy, shipped CLI, native Windows workspace build, repository gates, dashboard, feature gates, and benchmark checks passed. Optional manual macOS and stock-host jobs were skipped by workflow policy. CI run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/39810344331.
  • On 614e3e3461, full remote agent-host and sessions unit suites passed: 579 and 622 tests, with one pre-existing ignored agent-host test and no filtered cases; invocation 66c68ac0-1a6a-43d8-8661-be00fdbda6e1.
  • Earlier native Windows CI on 20e215fee7 passed all 47 tests across the three filesystem targets, with none filtered: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/39718197724/job/119635047155. The platform publication source is unchanged. Physical power-loss recovery has not been exercised.

Fixes #4208.

Daemon admission correction at 69e260430c0e9c7afe0664e9ba03cf160ea561a7:

  • Fleet run 39884685215 exposed a non-Git alias admission race: bounded repository preflight had already resolved the identity, but gate selection performed a second discovery and could falsely return typed repository-discovery deferral. Gate selection now consumes the preflight common directory (or decided non-repository state), preserving linked-worktree single flight and scope separation.
  • A controlled regression reproduced that exact deferral before the fix: 1 passed, 1 failed (invocation). After the fix, 7 focused tests passed, 0 failed, 931 filtered, in 10.20s (invocation). Formatting, commitlint, and whitespace checks passed. Remote Rustc/TestRunner were required with local fallback disabled.
  • Root source review found no blocker in the three-file daemon change. Separate dashboard request timeouts from the same fleet run remain under investigation. Current-head full hosted CI remains required before merge; earlier passing checks do not validate this new head.

Ledger read correction at d76b2899e98668ce5dec6f3d824d603b5144ea24 (preserves daemon fix 69e260430c):

  • Concurrent dashboard execution reproduced the manual-launcher GET timeout. The slow run measured ledger acquisition at 7.10s and a directory-sync span at 2.30s, while ledger body work stayed at or below 40.1ms. The precise acquisition breakdown remains unproven.
  • Existing-lock reads now reuse the canonical no-follow open path and exclusively lock the same sidecar inode, avoiding repeated directory permission changes and namespace sync. Only genuine absence falls back to creation. Writer/first-creation barriers and exact-append recovery guards remain in place.
  • The controlled regression failed before and passed after. Four focused checks include first-creation barrier preservation, independent-process lock exclusion, and symlink refusal.
  • All 699 relevant tests passed, 0 failed on the final head: automation runtime 435, automation runner 177, dashboard 87 (invocation). Required remote execution flags, Bazel formatting, commitlint, and committed whitespace checks passed. No investigative instrumentation remains.
  • Root reviewed the read/write boundary and found no blocker. Required publisher/recovery sync and genuine lock contention remain possible latency sources; these passing targets do not establish full-fleet readiness. Fresh hosted CI on this head is queued and must pass before merge.

@changeset-bot

changeset-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d76b289

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copilot AI linked an issue Oct 19, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Fix scheduler pause state atomicity issue across readers and crashes Publish scheduler pause state atomically Oct 19, 2026
Copilot AI requested a review from ScriptedAlchemy October 19, 2026 04:01
@ScriptedAlchemy
ScriptedAlchemy force-pushed the copilot/fix-scheduler-pause-state-issue branch from 3809057 to 47c61f2 Compare October 20, 2026 22:42
@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review October 21, 2026 06:27

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed 2119d79 to close the behavioral coverage gap for #4208. Production readers overlap 64 pause/unpause publications, and failures injected through the existing publication primitive must preserve committed pause bytes, report the error and remove staging files. Corrupt JSON must retain the typed Config error. No deadline, budget or production fault hook changed. Pinned rustfmt, whitespace and commitlint pass; native execution is with CI.

devin-ai-integration[bot]

This comment was marked as resolved.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed the durability follow-up. Scheduler pause publication now creates missing profile/dashboard ancestors through PrivateStoreIo::create_dir_all_durable before atomic replacement. The canonical Windows replacement path uses the existing validated Unicode/long-path encoding with MoveFileExW(REPLACE_EXISTING | WRITE_THROUGH); no cross-volume copy fallback is allowed. Rebuildable accelerator writes keep their existing non-durable behavior. Added ancestor-barrier failure/retry and private hierarchy regressions plus Windows held-reader, delete-sharing denial, staging cleanup, long-path, embedded-NUL and no-replace checks. Pinned rustfmt, diff checks, and commitlint pass; native/Windows execution awaits CI.

ScriptedAlchemy and others added 6 commits October 21, 2026 01:49
Reuse the ingest probe's WAL restart alignment before settling each
retention fixture. Preserve the64-message window and1.2x scaling bound,
and assert that neither measured tick includes periodic checkpoint work.
Warm-up retention remains free to settle and checkpoint its backlog.
@ScriptedAlchemy
ScriptedAlchemy enabled auto-merge (squash) October 22, 2026 04:36
@ScriptedAlchemy ScriptedAlchemy changed the title Publish scheduler pause state atomically fix(automation): publish scheduler pause state atomically Oct 22, 2026
ScriptedAlchemy and others added 7 commits October 22, 2026 12:43
Open existing sidecars through the shared no-follow capability path and
acquire the same exclusive lock. This removes repeated durable directory
setup from existing-lock reads while preserving first-creation and writer
publication barriers.

Use the canonical namespace-sync fault injector to prove settled reads
avoid namespace publication. Cover first-creation and writer failures,
parent aliases, final symlink refusal, and cross-process inode exclusion.
@ScriptedAlchemy
ScriptedAlchemy merged commit e72ff87 into master Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Persist scheduler pause state atomically across readers and crashes

2 participants