Skip to content

fix(sync): eliminate the tz-sync registration race that aborts scheduled runs - #5

Merged
dortort merged 5 commits into
mainfrom
fix/sync-registration-race
Aug 26, 2026
Merged

dortort merged 5 commits into
mainfrom
fix/sync-registration-race

Conversation

@dortort

@dortort dortort commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Problem

Scheduled tasks intermittently fail at the very start of a run: a sub-second exit-1 with empty stdout/stderr and no session ever created. On the affected machine this recurred roughly monthly against an otherwise-healthy task (mostly successful runs), always with the same signature.

Root cause

The daily tz-sync job (claude-scheduler-cli sync, which keeps local StartCalendarIntervals aligned to UTC across DST) is scheduled for 3am local — the same slot many tasks run at. When it fires alongside a task, sync mutates the very things that task's launch depends on:

  • Hazard A — torn shim read. sync rewrites the shared executor shims (claude-scheduler-run, claude-scheduler-cli) in place. launchd can exec a shim mid-write and read a truncated file.
  • Hazard B — reloading a live service. registerDarwin does launchctl unload → load on the task's plist. If the task's job is mid-run, launchd tears down its process tree; the executor records the killed child as code ?? 1 → exit 1 with empty logs.

Shifting a task off the 3am slot only lowers the probability — and sleep/wake coalescing refires missed jobs together anyway, defeating the offset. The durable fix addresses the coordination, at both layers.

Fix

1. Skip re-registering a task whose job is running (fix(sync), Hazard B). Before sync re-registers a task, it checks a new isTaskRunning predicate (reusing the executor's existing lock: readLockPid + verifyProcessIdentity + a liveness probe). A live task is deferred to the next sync cycle, so its already-loaded registration is never torn down mid-run. The predicate is injectable to keep the sync loop unit-testable.

2. Write plists and shims atomically (fix(platform,init), Hazard A). A new writeFileAtomic helper writes to a temp file in the target's directory and renames it into place. A concurrent reader always sees a whole file, regardless of timing — this closes the torn-read hazard that the per-task lock structurally can't reach (the shims are shared, and they're read before any lock exists).

The two layers are complementary: the lock guards the unload/reload path; atomic writes are an unconditional invariant for the file rewrites.

Deliberately out of scope (YAGNI)

  • Reload-only-on-change — an optimization of how often sync reloads, not a correctness fix; once reloads are lock-guarded and writes are atomic, reducing their frequency buys nothing.
  • Closing the launch-window sliver (the gap between launchd exec and the executor acquiring its lock) via a launchctl running-state query. The lock check already shrinks the vulnerable window from the whole run down to node startup, and a task missed by one sync self-heals on the next. Not worth the added complexity until evidence shows the sliver actually bites.

Testing

  • writeFileAtomic: new-file, overwrite, mode preservation, no temp-file leak on success or failure.
  • isTaskRunning: false for no-lock / unverifiable (no startTime) / dead-pid; true for a live, identity-verified process, flipping to false once it's killed.
  • sync: a running task is not re-registered and lands in skipped; all tasks register when none are running.
  • Full suite green (352 tests), typecheck and lint clean.

dortort and others added 4 commits August 24, 2026 13:21
writeFileAtomic writes to a temp file in the target's directory and renames
it into place, so a concurrent reader never observes a truncated or
half-written file.

Constraint: temp file must share the target's directory to keep rename on one filesystem
Confidence: high
Scope-risk: narrow

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The daily tz-sync rewrites the shared executor shims and re-registers task
plists. Rewriting these in place lets launchd exec a shim (or launchctl read
a plist) mid-write, yielding a torn read. Atomic write+rename removes that
hazard regardless of timing — the reader always sees a whole file.

Constraint: launchd may exec the shim at the exact instant sync rewrites it
Rejected: skip rewriting unchanged files | reduces frequency but not the torn-read hazard
Confidence: high
Scope-risk: narrow

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports whether a live, identity-verified executor currently holds a task's
lock, reusing the existing readLockPid + verifyProcessIdentity helpers. A dead
or unverifiable lock reports false so a stale lock can't block callers.

Constraint: must not treat a PID-reused or legacy (no startTime) lock as running
Confidence: high
Scope-risk: narrow

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The daily tz-sync fires at the same 3am slot as scheduled tasks. When it
re-registers a task, registerDarwin does launchctl unload+load on that task's
plist — which tears down the job's process tree if it is mid-run, surfacing as
a sub-second exit-1 with empty logs. Sync now checks isTaskRunning before
re-registering and defers a live task to the next cycle.

The predicate is injectable so the sync loop stays unit-testable.

Constraint: a task's executor acquires its lock only after launchd exec's it, leaving a small launch-window sliver the lock check can't see
Rejected: also query launchctl running-state to close the sliver | not worth the complexity; the lock check shrinks the window from the full run to node startup, and a missed task self-heals next sync
Rejected: shift the task off the 3am slot | probabilistic band-aid; sleep/wake coalescing refires missed jobs together anyway
Confidence: high
Scope-risk: narrow
Not-tested: the launch-window sliver between launchd exec and lock acquisition

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dortort dortort self-assigned this Aug 24, 2026
@dortort
dortort marked this pull request as ready for review August 24, 2026 20:02
writeFileAtomic derived its temp path from the pid alone, so two concurrent
writes to the same target in one process shared a temp file — one rename could
leave the other hitting ENOENT or installing unexpected content. Add a
per-call counter so concurrent writes never collide.

Not currently triggerable in-tree (init writes distinct targets; sync's loop
is sequential), but the helper is general-purpose and the guarantee should hold
unconditionally.

Constraint: temp file must stay in the target's directory to keep rename atomic
Confidence: high
Scope-risk: narrow

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dortort
dortort merged commit cf20e80 into main Aug 26, 2026
5 checks passed
@dortort
dortort deleted the fix/sync-registration-race branch August 26, 2026 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant