Make the 01:20 UTC detector safe: gate entries on a confirmed bar, and stop losing the day-stamp

[eaitbrahim]

Follow-up to #172, from an adversarial review of it. #172 verified the clock-side invariant correctly — 9624 simulated firings, 402 runs on 402 distinct UTC dates, both DST transitions — but it traded a ~13-hour data-publication margin for 20 minutes, and nothing measured whether the data had actually arrived. This PR closes that, plus three ways the day-stamp could be lost.

Nothing is deployed. ~/keel still runs the old local-anchored script, so there is no live exposure today — but a release cut from main as it stands would create it.

---

Finding 1 (HIGH) — a late candle re-evaluates yesterday's bar, placing a duplicate order

turtle_breakout._completed_days withholds the just-closed daily bar until the 00:00–01:00 UTC hourly bar closes. Verified against the real function, at 01:20 UTC on UTC date X:

feed state	effective newest daily bar
both series current	X-1 — correct
ONE_HOUR one bar late	X-2 — regressed
ONE_DAY not yet stored	X-2 — regressed
(at the old 13:05 UTC slot, even 12 bars of hourly lag)	X-1 — harmless

X-2 is the bar yesterday's 01:20 UTC cycle already evaluated. Nothing on the live path dedupes an entry — get_open_positions gates exits/reconcile/status but never entry, the signals table is never read back, client_order_id is a fresh uuid4, and the rails are dollar caps, not per-day counters. So that re-evaluation is a duplicate real-money order, not a delayed one.

Why nothing caught it: market_feed.poll_once writes nothing and raises nothing when the venue has no new bar; run_once set last_feed_ts = now_ts unconditionally, so rail 12 — which measures whether the poller ran, not whether data arrived — was blind; and market_feed.is_fresh is measured against the finest configured granularity (FIFTEEN_MINUTE live), never ONE_DAY. The cycle exited 0, the runner stamped the day, and there was no retry.

The fix

data/freshness.py already had expected_last_ts and the bars_behind arithmetic and zero references from agent.py, execution/ or strategy/. New entry_bar_ready() uses them: the gated series must be bars_behind == 0, and every finer configured series must have advanced past that bar's close.

That second condition is _completed_days's own condition stated generically, so the gate cannot read "ready" while _completed_days would still withhold the bar. It is also no stricter than it must be — it only asks that the finer series crossed the boundary, not that it is at its own newest bar, so an hourly series five bars late at 14:20 UTC still confirms fine. A bars_behind == 0 requirement on FIFTEEN_MINUTE would have left only 5 minutes of margin at the 01:20 trigger and blocked entries routinely.

Deliberately not reusing assess()/DEFAULT_TOLERANCE_BARS: that 2-bar tolerance exists so an operator-facing staleness alert does not fire on the normal forming-bar lag. A one-bar-late hourly series is exactly what duplicates an order, and the alert tolerance would wave it through.

Where the gate lives, and why not rail 12

In run_once, not as a rail. Two reasons:

1. A rail cannot fail the cycle. Rails yield placed=False and a zero exit. The runner would still stamp the day, and the fresh bar would never be evaluated at all — trading a duplicate order for a silently missed day. The gate has to fail the cycle so the runner declines to stamp.
2. A rail only runs when a signal fires. On the ~99% of days no signal fires, a stale bar would still be stamped as "done", and the real bar would go unevaluated.

Fixing rail 12 instead would be actively dangerous. Its threshold is interval_sec * 3 and it answers "did the poller run". Making last_feed_ts advance only on new data would, in a once-daily deployment, leave it ~24h stale every cycle and veto everything.

Semantics

• Entries only. Exits still run for every rule, unfiltered — an open position's rule-driven channel exit runs in-process (the protective stop rests at the broker, the channel exit does not), and holding a losing position an extra cycle is strictly worse than a delayed entry.
• Gated on the granularity the rule trades on. _entry_gate_granularity falls back to the coarsest configured granularity, diverging from engine._trading_granularity's finest-fallback on purpose: Dca declares no timeframe yet reads candles_by_tf[ONE_DAY] directly and keys its cadence off that bar's timestamp, so gating it on FIFTEEN_MINUTE would miss the same hazard entirely.
• Nonzero exit. Single-cycle keel agent exits DATA_NOT_READY_EXIT (4), so the runner declines to stamp and one of the remaining 23 triggers retries an hour later — duplicate order → ≤60 minutes of delay. --loop is unchanged: it skips the cycle and tries again next interval.
• Structured log. agent.entry_bar_not_ready names the granularity, expected ts, stored ts, bars_behind, reason and blocking series.

---

Findings 2/3/4 — three ways the day-stamp could be lost

Finding 2 (HIGH) — a failed stamp write was swallowed. Reproduced: with a read-only logs dir the cycle ran, the stamp was absent, rc was 0, and the next trigger ran a second full cycle. Two layers now:

• Pre-flight, before keel is invoked at all — probe that the stamp is persistable, and refuse to run a cycle if not. This is the layer that matters. Exiting nonzero after a cycle has run does not prevent the duplicate: the order is already placed. The only way to turn "duplicate real order" into "no trading plus a loud alert" is to refuse to trade when we cannot record that we traded.
• Atomic write + read-back afterwards, as belt and braces. > "$STAMP" truncates first, so a torn write left an empty stamp — which reads as "never ran" and re-runs the day.

Finding 3 (MED) — empty date -u output disabled the detector permanently. On this hardware $((10#$(date -u '+%H'))) on empty input evaluates to 0, not an error, so an empty clock gave TODAY="", HOUR=0; a missing stamp also reads "", so [ "" = "" ] was true — "already ran", forever, with no alert. The unmodified script literally printed detector already ran this UTC day () -- skipping. Now validated before use (exit 64, nothing run, stamp untouched).

Finding 4 (MED) — the stamp compare was =, so a clock rollback re-evaluated a bar. A Mac booting with a bad RTC before NTP settles (RunAtLoad fires immediately) reads a past date, which != treats as "not today". Now strictly-less-than, making the stamp monotonic. A malformed stamp is refused (exit 65) rather than compared — "garbage" < "2026-08-06" is false and would read as "already ran" forever.

Also: the "N signal(s) PENDING — run the agent interactively" notification now fires only on a clean cycle. Per the script's own COROLLARY, running the agent by hand bypasses the stamp, so prompting for it off a failed cycle's partially-parsed output pointed the operator straight at a duplicate entry. An ordinary nonzero keel exit deliberately does not notify — it is expected and self-healing, and alerting 23 times a day would train the operator to ignore the alerts that do need a human. The policy is written into the header, along with exit codes 64/65/66.

Finding 7 — "no missed day" was stated flatly in the header and plist; it is conditional on the machine being powered on for at least one eligible trigger that UTC day. Both now say so.

Finding 10 — the PENDLOG line logged local, unlabelled time next to lines that are all UTC.

Finding 8 (flock) — documented, not fixed. flock is not installed on macOS (verified), launchd will not start a job already running, and the race that actually matters is a manual run, which bypasses the stamp regardless of any lock.

---

Finding 6 — the tests were testing a model, not the artifact

_run_gate re-implemented the shell gate in Python, so the year-long simulation proved nothing about the shipped script. Worse, _skip_before_sched_hour made the only two real-shell tests silently skip whenever CI ran between 00:00 and 01:00 UTC — on the sole barrier to duplicate orders.

• The skip is gone. Every real-script test now shims date on PATH and injects its own instant.
• The model is pinned to the artifact. Driving the real script over all 9624 triggers would add ~2.5 minutes to a 10-second suite, so the model is kept — but test_the_simulated_gate_matches_the_real_script replays the real script over a curated adversarial sequence (normal days, the sub-SCHED_HOUR trigger, both DST transitions including the twice-fired local 01:20, and the boot-after-outage catch-up) and requires it to agree with the model at every trigger.
• _stored_series no longer hardcodes instantaneous, never-failing publication — the assumption that made this whole bug class invisible. It takes hourly_lag_bars/daily_lag_bars, and a new test pins the premise against the real _completed_days.

Verification

Every new test was confirmed to fail for the right reason before the fix.

• Neutering only the entry gate (engine.evaluate(ready_rules, …) → product_rules) makes all 5 new agent/CLI tests fail, with the turtle rule emitting a real ENTER Signal on ts=777600 — bar X-2, the already-traded bar.
• Running the new schedule suite against the unmodified script: 10 failed, 21 passed, including rc=0 despite a "Permission denied" stamp write, and detector already ran this UTC day () -- skipping on an empty clock.

Independently re-verified end-to-end in a sandboxed copy of the script (live path asserted absent):

scenario	cycles run	rc	notified
empty date -u	0	64	yes
read-only logs dir	0 (was 2, rc=0)	66	yes
happy path	1	0	—
second trigger, same UTC day	1	0	—
clock rolled back 200 days	1	0	—
clock corrected	1	0	—
next UTC day	2	0	—
corrupt stamp	0	65	yes
failed cycle printing signals=2	1	7	no PENDING prompt, no stamp

Stamp never truncated: forced write failure left yesterday's 2026-06-14 intact.

Gate: ruff clean · mypy clean (94 files) · 1953 passed (from 1924 on main).

Suite time went 10.3s → 19.5s. ~2.4s of that is the differential test; the rest is the 11 per-property real-script tests. That is the honest price of Finding 6 — the alternative is deleting artifact-level coverage of the only barrier against duplicate real-money orders.

What this does not do

It is not an entry dedupe. The day-stamp remains the only thing preventing two cycles in one UTC day from entering twice, and the two characterization tests in tests/test_agent.py still pin that hazard (they are unaffected — their config declares only ONE_DAY, so there is no finer series to confirm against). This PR removes the specific way a late candle could make the stamp's "one run per UTC day" guarantee re-evaluate an already-traded bar.

🤖 Generated with Claude Code

[eaitbrahim]

Review round 2 — CRITICAL + regression fixed (bdef825)

Both defects confirmed and reproduced before fixing. The critical one was mine, and the review's diagnosis was exactly right.

CRITICAL — the gate placed orders and then failed the cycle

Confirmed in code: ready_rules was built per-product inside the loop, and engine.evaluate/executor.execute ran in that same iteration, so a product examined earlier had already placed real orders by the time a later product's rule was found not ready. blocked_entries was recorded, never consulted before execution.

run_once is now two passes: a pre-pass over every (product, rule) pair computes entries_allowed = not blocked_entries before any execution; the main pass runs exits for every non-stale product unconditionally and evaluates entries only when entries_allowed.

The stale-product skip deliberately still runs first, so a dead venue or delisting is skipped as stale before the gate sees it and cannot halt the cycle — that ordering is pinned by a new test.

Semantic change, documented in the code and stated plainly: one lagging product now delays every entry that cycle, including entries whose own data was fresh. That is the intended trade — a delayed entry is recoverable within ≤60 minutes, a duplicate live entry is not — and it is what makes the retry idempotent, which is what gives "≤60 minutes of delay" its meaning.

Verified by neutering only the admission bit (entries_allowed = True): 8 tests go red, including

AssertionError: BTC-USD's DCA entry was evaluated even though XLM-USD's was blocked this cycle:
  [Signal(rule_name='dca', product_id='BTC-USD', action=ENTER, side=BUY, ..., ts=864000),
   Signal(rule_name='dca', product_id='XLM-USD', action=ENTER, side=BUY, ..., ts=777600)]

— a real BTC BUY evaluated alongside a non-empty blocked_entries, i.e. the exact XLM/BTC-DCA shape from the review. All 8 restore green.

MED — forward clock excursion (a regression this PR introduced)

Reproduced through this PR's own sandbox harness: stamp 2035-01-01, then five consecutive real days at rc=0 with zero notifications. The old = compare self-healed the next day; the monotonic compare did not.

A stamp strictly ahead of today is corrupt state, not "already ran" → notify + exit 67 (distinct from 65 so an operator can tell "future stamp" from "not a date"). A stamp equal to today stays a silent rc=0 so the common path doesn't start over-alerting.

Independently re-verified:

sequence	rc	alerts
2026-08-06 01:20	0	0 (stamps 2026-08-06)
2035-01-01 03:20	0	0 (stamps 2035-01-01)
2026-08-07/08/09	67 each	1 each
same-UTC-day repeat	0	0 — control
rollback to 2026-02-01	67	1, no new cycle, stamp stays 2026-08-06

The stamp is never overwritten on the 67 path.

The false prose claim that < and = "diverge only under a clock rollback" is corrected — that claim is why nothing caught this — and the differential test's sequence now includes a forward jump.

MED — escalation on a stuck detector

This matters more now that any unconfirmed product withholds the whole cycle. Counter next to the stamp, guarded so a bug in it degrades to "stops escalating", never "stops trading":

failure 1 -> rc=4  alerts=0  counter=1
failure 2 -> rc=4  alerts=0  counter=2
failure 3 -> rc=4  alerts=1  counter=3     <- escalates
failure 4,5 -> rc=4 alerts=1  counter=4,5
failure 6 -> rc=4  alerts=2  counter=6     <- and every 3 after
clean cycle -> counter reset; next single failure silent again

Stamp never written during failures.

Also

Trimmed the manifest test's overstated claim: it asserts the committed file, not a deployment DB, so it catches a reseeded box's state being committed, not the reseed itself. rule_manifest.py apply is what catches that live.

Not addressed, deliberately

_entry_gate_granularity returning an unpolled granularity is now bounded rather than closed: it is unreachable in today's config (every rule gates on ONE_DAY, which is polled), and the escalation alert above means it can no longer fail silently — it would alert after 3 triggers instead of burning all 23 in silence. A startup validation would close it properly; I did not add one because it is a config-shape check that belongs with config loading, not in the cycle, and I did not want to widen this PR further after a critical miss.

Gate

ruff clean · mypy clean (94 files) · 1961 passed · bash -n clean · plist parses under strict XML. Live deployment untouched.

🤖 Generated with Claude Code

[eaitbrahim]

Review round 3 — all four items closed (68c2db6)

Both substantive claims reproduced independently before fixing. One item's stated mechanism turned out to be wrong, and I've corrected it rather than confirmed it — details below.

1 (MEDIUM) — pre-flight now proves $STAMP is REPLACEABLE

Reproduced exactly as reported, with $STAMP made a directory:

exit codes: 66 66 66 66 66
CYCLES RUN: 5      <- each one places real orders

Two layers:

• Round-trip through $STAMP itself — write a temp file, mv -f it onto $STAMP, read back. Payload is the stamp's own validated contents, so a pass is a semantic no-op; when no stamp exists the probe is removed so the day stays unstamped. Same reproduction now gives 66 ×5 with 0 cycles run. Mode-000 still works (a rename needs directory permission, not permission on the target inode) and is pinned.
• Halt sentinel for the residual window where the stamp breaks between a passing pre-flight and the write. A cycle has already run there, so every later trigger refuses with exit 68 until a human clears it — bounding damage to the one cycle that ran instead of 23.

2 (MEDIUM) — the atomicity test now discriminates

Confirmed the mutation myself: collapsing the whole temp+mv -f+readback block to a single truncating printf > "$STAMP" left all 34 tests green. Fixed on both sides — the behavioural test's docstring now claims only what it proves and explains why a behavioural test can't discriminate here (uchg fails at open(2), before truncation), plus a new source-level test pins the design.

4 (LOW) — exit codes and the three unkilled fixes

Every returncode != 0 assertion is now an exact code. The exit-65 point was real: the old test used a malformed stamp sorting above today, which lands in the 67 branch anyway. The discriminating case is one sorting below today (1999-1-1) — without the 65 branch that reads as "older than today" and runs a cycle.

Mutation ledger — every fix verified red

mutation	result
plain > instead of temp+mv	red
readback dropped	red
mv -f → cp -f	red
hour-range -gt 23 → if false	red
read_failcount validation removed	red (5/6 params)
exit-65 branch removed	red
pre-flight round-trip removed	red
halt-sentinel gate removed	red
both pre-flight guards removed	red

One deliberate survivor, disclosed: the "exists but is not a regular file" check is not individually killable — I verified the round-trip alone already detects a directory (exit 66, zero cycles). It's kept for a clearer operator message and to stop mv littering a probe file inside that directory, and is documented as belt-and-braces rather than a sole detector. I chose that over inventing a contrived test to manufacture a red.

3 (LOW-MED) — comment fixed, and the stated mechanism was wrong

The comment is corrected: a withheld cycle withholds entries only and can place a real SELL.

But the claim that position_rule:<product> clearing is what makes the exit retry idempotent does not survive mutation. Removing that clear alone leaves the suite green. The real guard is the audit-log qty netting in _held_position, implemented independently in both agent.py and execution/executor.py — an exit is a market order, so its SELL is recorded filled immediately and the position already reads as closed on retry, before position_rule is consulted.

Only breaking all three together produces the duplicate — verified, a second SELL (order id 3, same qty and fill). So the retry is safe, and more redundantly than believed, but not for the stated reason. position_rule clearing is load-bearing for something else (stale bracket state poisoning the next position on that product), documented at its own call site. The test records the corrected mechanism rather than the assumed one.

Scope

Dropped the unrelated test_rule_manifest.py comment from 10 lines to 3.

Gate

ruff clean · mypy clean (94 files) · 1976 passed · bash -n clean · live deployment untouched.

Suite 10.3s → 33s. The growth is real-script subprocess invocations; the alternative is deleting artifact-level coverage of the barrier against duplicate real-money orders.

🤖 Generated with Claude Code