fix(sync): eliminate the tz-sync registration race that aborts scheduled runs - #5
Merged
Merged
Conversation
writeFileAtomic writes to a temp file in the target's directory and renames it into place, so a concurrent reader never observes a truncated or half-written file. Constraint: temp file must share the target's directory to keep rename on one filesystem Confidence: high Scope-risk: narrow Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The daily tz-sync rewrites the shared executor shims and re-registers task plists. Rewriting these in place lets launchd exec a shim (or launchctl read a plist) mid-write, yielding a torn read. Atomic write+rename removes that hazard regardless of timing — the reader always sees a whole file. Constraint: launchd may exec the shim at the exact instant sync rewrites it Rejected: skip rewriting unchanged files | reduces frequency but not the torn-read hazard Confidence: high Scope-risk: narrow Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports whether a live, identity-verified executor currently holds a task's lock, reusing the existing readLockPid + verifyProcessIdentity helpers. A dead or unverifiable lock reports false so a stale lock can't block callers. Constraint: must not treat a PID-reused or legacy (no startTime) lock as running Confidence: high Scope-risk: narrow Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The daily tz-sync fires at the same 3am slot as scheduled tasks. When it re-registers a task, registerDarwin does launchctl unload+load on that task's plist — which tears down the job's process tree if it is mid-run, surfacing as a sub-second exit-1 with empty logs. Sync now checks isTaskRunning before re-registering and defers a live task to the next cycle. The predicate is injectable so the sync loop stays unit-testable. Constraint: a task's executor acquires its lock only after launchd exec's it, leaving a small launch-window sliver the lock check can't see Rejected: also query launchctl running-state to close the sliver | not worth the complexity; the lock check shrinks the window from the full run to node startup, and a missed task self-heals next sync Rejected: shift the task off the 3am slot | probabilistic band-aid; sleep/wake coalescing refires missed jobs together anyway Confidence: high Scope-risk: narrow Not-tested: the launch-window sliver between launchd exec and lock acquisition Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dortort
marked this pull request as ready for review
August 24, 2026 20:02
writeFileAtomic derived its temp path from the pid alone, so two concurrent writes to the same target in one process shared a temp file — one rename could leave the other hitting ENOENT or installing unexpected content. Add a per-call counter so concurrent writes never collide. Not currently triggerable in-tree (init writes distinct targets; sync's loop is sequential), but the helper is general-purpose and the guarantee should hold unconditionally. Constraint: temp file must stay in the target's directory to keep rename atomic Confidence: high Scope-risk: narrow Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Scheduled tasks intermittently fail at the very start of a run: a sub-second exit-1 with empty stdout/stderr and no session ever created. On the affected machine this recurred roughly monthly against an otherwise-healthy task (mostly successful runs), always with the same signature.
Root cause
The daily
tz-syncjob (claude-scheduler-cli sync, which keeps localStartCalendarIntervals aligned to UTC across DST) is scheduled for 3am local — the same slot many tasks run at. When it fires alongside a task,syncmutates the very things that task's launch depends on:syncrewrites the shared executor shims (claude-scheduler-run,claude-scheduler-cli) in place. launchd canexeca shim mid-write and read a truncated file.registerDarwindoeslaunchctl unload→loadon the task's plist. If the task's job is mid-run, launchd tears down its process tree; the executor records the killed child ascode ?? 1→ exit 1 with empty logs.Shifting a task off the 3am slot only lowers the probability — and sleep/wake coalescing refires missed jobs together anyway, defeating the offset. The durable fix addresses the coordination, at both layers.
Fix
1. Skip re-registering a task whose job is running (
fix(sync), Hazard B). Beforesyncre-registers a task, it checks a newisTaskRunningpredicate (reusing the executor's existing lock:readLockPid+verifyProcessIdentity+ a liveness probe). A live task is deferred to the next sync cycle, so its already-loaded registration is never torn down mid-run. The predicate is injectable to keep the sync loop unit-testable.2. Write plists and shims atomically (
fix(platform,init), Hazard A). A newwriteFileAtomichelper writes to a temp file in the target's directory andrenames it into place. A concurrent reader always sees a whole file, regardless of timing — this closes the torn-read hazard that the per-task lock structurally can't reach (the shims are shared, and they're read before any lock exists).The two layers are complementary: the lock guards the unload/reload path; atomic writes are an unconditional invariant for the file rewrites.
Deliberately out of scope (YAGNI)
syncreloads, not a correctness fix; once reloads are lock-guarded and writes are atomic, reducing their frequency buys nothing.execand the executor acquiring its lock) via alaunchctlrunning-state query. The lock check already shrinks the vulnerable window from the whole run down to node startup, and a task missed by one sync self-heals on the next. Not worth the added complexity until evidence shows the sliver actually bites.Testing
writeFileAtomic: new-file, overwrite, mode preservation, no temp-file leak on success or failure.isTaskRunning: false for no-lock / unverifiable (nostartTime) / dead-pid; true for a live, identity-verified process, flipping to false once it's killed.sync: a running task is not re-registered and lands inskipped; all tasks register when none are running.