From 48d2d8aa5b5505520ef8c38b027ba2b2656fb265 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 18:36:11 -0400 Subject: [PATCH 1/9] fix(infra): survive the two silent-outage modes that took the stack down Everything reported "running" while nothing worked. Two independent supervision gaps, both invisible because the process stayed alive: 1. Infra was down for 13 days. dev.lantern.infra is RunAtLoad-only and the containers had no restart policy, so when Docker Desktop lost the boot race on Aug 3 the wrapper logged "docker NEVER became ready", exited 1, and nothing ever retried. Postgres/Redis/MinIO stayed down while every service looped on connection-refused. - compose: restart: unless-stopped, so Docker's own supervisor brings them back whenever the daemon does and launchd is out of the loop. - infra.plist: KeepAlive{SuccessfulExit:false} + 300s throttle, so a Docker-not-ready exit retries instead of giving up forever. A successful one-shot run still exits 0 and is not restarted. 2. The dashboard served a build it never made. Next.js content-hashes chunk filenames per build and the wrapper deliberately skips rebuilding when a bundle exists, so a server left running across a rebuild answers HTML referencing chunks it doesn't have: every /_next/static/chunks/*.js 400s and the page dies with "Application error: a client-side exception has occurred". The running process dated from Jul 19; the build from Jul 24. - dashboard.plist: WatchPaths on .next/BUILD_ID, so a rebuild actually takes effect. Verified: infra containers healthy and carrying the restart policy; API /healthz ok and serving agents/runs from Postgres; every dashboard chunk 200 both on localhost and over the tailnet. --- infra/docker/docker-compose.yml | 5 +++++ scripts/launchd/dev.lantern.dashboard.plist | 11 +++++++++++ scripts/launchd/dev.lantern.infra.plist | 15 +++++++++++++-- 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/infra/docker/docker-compose.yml b/infra/docker/docker-compose.yml index 42325114..10745e45 100644 --- a/infra/docker/docker-compose.yml +++ b/infra/docker/docker-compose.yml @@ -10,6 +10,9 @@ x-common-env: &common-env services: postgres: image: pgvector/pgvector:pg16 + # ponytail: Docker's own restart policy is the durable fix — containers + # come back whenever the daemon does, with no launchd job in the loop. + restart: unless-stopped ports: - "5432:5432" environment: @@ -26,6 +29,7 @@ services: redis: image: redis:7-alpine + restart: unless-stopped ports: - "6379:6379" healthcheck: @@ -36,6 +40,7 @@ services: minio: image: minio/minio:latest + restart: unless-stopped ports: - "9000:9000" - "9001:9001" diff --git a/scripts/launchd/dev.lantern.dashboard.plist b/scripts/launchd/dev.lantern.dashboard.plist index 51ae6a72..3fb20dd5 100644 --- a/scripts/launchd/dev.lantern.dashboard.plist +++ b/scripts/launchd/dev.lantern.dashboard.plist @@ -31,6 +31,17 @@ Logs: ~/Library/Logs/Lantern/dashboard.{out,err}.log ThrottleInterval 15 + + WatchPaths + + __REPO_ROOT__/apps/web/.next/BUILD_ID + + StandardOutPath __HOME__/Library/Logs/Lantern/dashboard.out.log StandardErrorPath diff --git a/scripts/launchd/dev.lantern.infra.plist b/scripts/launchd/dev.lantern.infra.plist index 3eb979c9..fedd2065 100644 --- a/scripts/launchd/dev.lantern.infra.plist +++ b/scripts/launchd/dev.lantern.infra.plist @@ -25,10 +25,21 @@ Logs: ~/Library/Logs/Lantern/infra.{out,err}.log WorkingDirectory __REPO_ROOT__ - + RunAtLoad + KeepAlive + + SuccessfulExit + + + ThrottleInterval + 300 StandardOutPath __HOME__/Library/Logs/Lantern/infra.out.log From 91896b82c70b6553cad1886687bd64086fbe4046 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 19:08:05 -0400 Subject: [PATCH 2/9] fix(bridge): stop the duplicate-send storms and the "undefined" OTP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit of the owner's real chat.db found the "spammy / repetitive" report is two concrete defects, not a tone problem. 1. Duplicate-send storms — 88 duplicate pairs sent <=300s apart. One fraud alert went out 22x, another 27x, and one reply 15x inside 32 seconds, several within the same second. Cause: autoActLifeEvent has always had a hasActed() guard, but the ping and digest routes that surface a life event to the owner had none, so a re-delivered or re-classified inbound re-sent the same DM with nothing to stop it. The fraud pair alone is 49 of ~1,899 self-chat messages. Fix: dedup where all three routes converge — right after proactiveDecision, reusing the idempotency key already computed for auto-act — rather than bolting a separate guard onto each route. Applied to both bridges. 2. "your code is undefined" — shipped 11x over 36 days. A codeless OTP interpolated f.code straight into the template. Fixed at the source in buildOwnerMessage so no caller can ship it: a codeless OTP now says a code arrived and to check the original message. Audited the sibling templates — f.time and f.merchant were already ternary-guarded; f.code was the only unguarded interpolation. Tests: 1489/1489 pass (2 new regression tests pinning the undefined case and that a real code still renders). Both bridges tsc --noEmit clean. Not fixed here: the self-narrated capability apologies (F3) and the WhatsApp sendSelf outage, both still under audit. --- packages/bridge-core/src/life-events.test.ts | 14 ++++++++++++++ packages/bridge-core/src/life-events.ts | 5 +++++ services/imessage-bridge/src/session.ts | 20 ++++++++++++++++++++ services/whatsapp-bridge/src/session.ts | 16 ++++++++++++++++ 4 files changed, 55 insertions(+) diff --git a/packages/bridge-core/src/life-events.test.ts b/packages/bridge-core/src/life-events.test.ts index 6a23551b..49a09d0a 100644 --- a/packages/bridge-core/src/life-events.test.ts +++ b/packages/bridge-core/src/life-events.test.ts @@ -64,6 +64,20 @@ describe("life-events — undated travel/appointment must NOT nudge (placeholder const e = ev({ kind: "otp", urgency: "now", fields: { code: "123456" }, rawText: "Your verification code is 123456" }); assert.notEqual(proactiveDecision(e, {}, NOW).route, "suppress", "otp must still surface"); }); + + // REGRESSION: a codeless OTP interpolated `undefined` into the owner DM and + // shipped "🔑 your code is undefined" 11 times over 36 days in production. + test("otp with no code never renders the literal string undefined", () => { + const e = ev({ kind: "otp", urgency: "now", fields: {}, rawText: "Your verification code is expiring" }); + const msg = proactiveDecision(e, {}, NOW).ownerMessage; + assert.ok(!/undefined/.test(msg), `ownerMessage leaked "undefined": ${msg}`); + assert.ok(msg.length > 0, "codeless otp must still say something"); + }); + + test("otp WITH a code still reports it", () => { + const e = ev({ kind: "otp", urgency: "now", fields: { code: "611586" }, rawText: "code 611586" }); + assert.match(proactiveDecision(e, {}, NOW).ownerMessage, /611586/); + }); }); describe("life-events — recency gate: only surface RECENT + actionable (stale-Gmail-nudge bug)", () => { diff --git a/packages/bridge-core/src/life-events.ts b/packages/bridge-core/src/life-events.ts index 704952ee..2bc5f4b0 100644 --- a/packages/bridge-core/src/life-events.ts +++ b/packages/bridge-core/src/life-events.ts @@ -716,6 +716,11 @@ function buildOwnerMessage(event: LifeEvent, actions: ProactiveAction[]): string return `⚠️ ${who} flagged a declined/suspicious charge — might be fraud.${phone ? ` want the number (${phone})?` : " want the number?"}`; } case "otp": { + // A missing code interpolated as the literal string "undefined" and + // shipped "🔑 your code is undefined" 11 times over 36 days. An OTP + // ping with no code carries no information, so say what we do know + // rather than inventing a value we never extracted. + if (!f.code) return `🔑 a login code arrived${f.merchant ? ` from ${f.merchant}` : ""} — check the original message.`; return `🔑 your code is ${f.code} — (i won't share this with anyone).`; } case "appointment": { diff --git a/services/imessage-bridge/src/session.ts b/services/imessage-bridge/src/session.ts index de2ccf65..d452d87f 100644 --- a/services/imessage-bridge/src/session.ts +++ b/services/imessage-bridge/src/session.ts @@ -4443,6 +4443,26 @@ export class IMessageSession { if (decision.route === "suppress") return true; // owned + dropped — do NOT emit + // Dedup EVERY owner-facing emission, not just auto-act. autoActLifeEvent + // has always had a hasActed guard, but the ping/digest routes below did + // not — so a re-delivered or re-classified inbound fired the same DM again + // with nothing to stop it. Observed in production: one fraud alert sent + // 22×, another 27×, and one reply 15× inside 32 seconds (several in the + // same second). That is the "spammy/repetitive" report. The key is already + // computed above for auto-act; reuse it so all three routes share one + // guard rather than each growing its own. + if (auto.idempotencyKey) { + const { hasActed, markActed } = await import("@lantern/bridge-core/life-events-store"); + if (hasActed(auto.idempotencyKey)) { + this.logger.info( + { kind: event.kind, idempotencyKey: auto.idempotencyKey, route: decision.route }, + "life-event emit skipped — already surfaced (idempotent)", + ); + return true; + } + markActed(auto.idempotencyKey); + } + if (!owner) { // No self-chat target — surface to the dashboard feed so it isn't invisible. this.broadcast({ type: "activity", data: { kind: "system", summary: decision.ownerMessage, timestamp: Date.now() } }); diff --git a/services/whatsapp-bridge/src/session.ts b/services/whatsapp-bridge/src/session.ts index 0abf4c01..83fed221 100644 --- a/services/whatsapp-bridge/src/session.ts +++ b/services/whatsapp-bridge/src/session.ts @@ -3596,6 +3596,22 @@ export class WhatsAppSession { const decision = proactiveDecision(event, prefs); if (decision.route === "suppress") return true; // owned + dropped — do NOT emit + + // Same gap as the iMessage bridge: auto-act was guarded by hasActed but + // the ping/digest routes below were not, so a re-delivered inbound re-sent + // the same owner DM unbounded. Reuse the key already computed for auto-act + // so all routes share one guard. + if (auto.idempotencyKey) { + const { hasActed, markActed } = await import("@lantern/bridge-core/life-events-store"); + if (hasActed(auto.idempotencyKey)) { + this.logger.info( + { kind: event.kind, idempotencyKey: auto.idempotencyKey, route: decision.route }, + "life-event emit skipped — already surfaced (idempotent)", + ); + return true; + } + markActed(auto.idempotencyKey); + } if (!ownJid) { this.logActivity("attention_dm", decision.ownerMessage, { scope: "self" }); return true; From 2e758610ef28b19db2fc1d37a295feff56b662e8 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 19:33:19 -0400 Subject: [PATCH 3/9] fix(bridge): throttle the "still working on it" apology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The agent-null fallback fired on every failed round-trip with no throttle, so it apologised on a loop: 11× in production, 8 of them inside 55 seconds. The eighth "give me another minute" tells the owner nothing the first didn't — it is just the noise they reported. Once per 5 minutes per chat now; suppressed repeats log instead of sending. Same fix on both bridges (WhatsApp uses a scalar rather than a per-chat map because that path only ever writes to the owner's own thread). Deliberately NOT changed: the sibling "couldn't generate a reply to " notice. It already dedups on a 5-minute window (DROP_NOTIFY_DEDUP_MS), so its 4 occurrences across a 36-day window are 4 genuine incidents, not a storm. Silencing real operational drops would trade noise for blindness. Both bridges tsc --noEmit clean; 1489/1489 bridge-core tests pass. --- services/imessage-bridge/src/session.ts | 17 ++++++++++++++++- services/whatsapp-bridge/src/session.ts | 15 ++++++++++++++- 2 files changed, 30 insertions(+), 2 deletions(-) diff --git a/services/imessage-bridge/src/session.ts b/services/imessage-bridge/src/session.ts index d452d87f..8cb8302d 100644 --- a/services/imessage-bridge/src/session.ts +++ b/services/imessage-bridge/src/session.ts @@ -6474,6 +6474,12 @@ export class IMessageSession { // chat_identifier == own handle) and remember. private selfChatRowIds: Set = new Set(); + // Per-chat timestamp of the last "I'm still working on it" fallback. The + // fallback fires whenever the agent returns nothing, and with no throttle it + // went out 11× in production — 8 of them inside 55 seconds. Repeating + // "give me another minute" eight times says nothing the first one didn't. + private lastAgentStallNoticeAt: Map = new Map(); + // Per-chat timestamp of the last doc query. Lets us recognize // short follow-ups ("send it", "yes", "the first one") as // continuations within a 5-minute window. @@ -10283,7 +10289,16 @@ export class IMessageSession { if (!draft) { // Graceful fallback — never "couldn't reach the agent — try // again". The bot OWNS the retry; the user should not have to. - void this.send(jid, "hmm, that one took longer than I'd like. give me another minute and ask again"); + // Say it once per 5 minutes per chat. Beyond that the owner already + // knows the agent is struggling, and repeating it is the noise they + // reported — better to stay quiet than to apologise on a loop. + const lastStall = this.lastAgentStallNoticeAt.get(jid) ?? 0; + if (Date.now() - lastStall > 5 * 60_000) { + this.lastAgentStallNoticeAt.set(jid, Date.now()); + void this.send(jid, "hmm, that one took longer than I'd like. give me another minute and ask again"); + } else { + this.logger.info({ jid }, "agent stall notice suppressed — already sent within 5m"); + } return; } diff --git a/services/whatsapp-bridge/src/session.ts b/services/whatsapp-bridge/src/session.ts index 83fed221..2120409e 100644 --- a/services/whatsapp-bridge/src/session.ts +++ b/services/whatsapp-bridge/src/session.ts @@ -1225,6 +1225,11 @@ export class WhatsAppSession { private pendingOffers: Map = new Map(); private static readonly OFFER_TTL_MS = 10 * 60_000; + // Timestamp of the last "still working on it" fallback to self-chat. Scalar + // rather than a per-chat map because this path only ever writes to the + // owner's own thread. See the throttle at the agent-null fallback. + private lastAgentStallNoticeAt = 0; + // Typed owner-facts (spouse/anniversary/kids/naming-rules) extracted from a // self-chat teach, HELD pending the owner's "yes" before they're written as // authoritative "never contradict" ground truth. Without this a mis-extracted @@ -6400,7 +6405,15 @@ export class WhatsAppSession { this.logger.info({ totalMs: Date.now() - startedAt, hadDraft: !!draft, thinkingSent }, "doc query done (whatsapp)"); if (!draft) { - await this.confirmToSelf("hmm, that one took longer than I'd like. give me another minute and ask again"); + // Throttled like the iMessage twin: the fallback fires on every + // agent-null, and unthrottled it apologises on a loop (11× observed, + // 8 inside 55s). Once per 5 minutes carries the same information. + if (Date.now() - this.lastAgentStallNoticeAt > 5 * 60_000) { + this.lastAgentStallNoticeAt = Date.now(); + await this.confirmToSelf("hmm, that one took longer than I'd like. give me another minute and ask again"); + } else { + this.logger.info("agent stall notice suppressed — already sent within 5m"); + } return; } From d3d69531be00a9746338981f0d29d8ea70f345da Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 19:50:51 -0400 Subject: [PATCH 4/9] =?UTF-8?q?fix(wa):=20end=20the=2023-day=20news/digest?= =?UTF-8?q?=20outage=20=E2=80=94=20tick=20guard=20didn't=20match=20sendSel?= =?UTF-8?q?f?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit runNewsProactiveTick guarded on `!this.socket`, but sendSelf requires `!this.socket || !this.connected`. On `connection === "close"` the socket OBJECT survives while `connected` flips false, so every 12 minutes the tick cleared its own guard and then threw "not connected" inside sendSelf. Result: 3,238 of 3,246 sendSelf calls failed since Jul 24 — the WhatsApp daily digest and news alerts have been ~100% dead for 23 days while the bridge reported healthy. Contact replies use a different path and were unaffected, which is why it stayed invisible. Two changes: - The fix: make the tick's guard match what sendSelf actually requires. The timer intentionally keeps running, so the feed resumes on its own when the socket returns. - Hygiene: disconnect() cleared eleven timers but missed newsTimer. That alone would NOT have fixed this (the production path is the close handler, which never calls disconnect()), but a leaked interval on teardown is worth closing while here. Checked the siblings: runCommuteTick, runEnergyTick and runHealthCoachTick already test `connected` — news was the lone outlier, matching the stack traces. runFlywheelTick omits it too but fires every 8h with a conditional send, so it is 3/day worst case rather than 120/day; left alone deliberately. tsc --noEmit clean. --- services/whatsapp-bridge/src/session.ts | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/services/whatsapp-bridge/src/session.ts b/services/whatsapp-bridge/src/session.ts index 2120409e..51f754b3 100644 --- a/services/whatsapp-bridge/src/session.ts +++ b/services/whatsapp-bridge/src/session.ts @@ -11166,7 +11166,14 @@ export class WhatsAppSession { /** Push only high-signal NEW AI news to self-chat. Deduped + quiet-hours aware. */ private async runNewsProactiveTick(): Promise { try { - if (this.killSwitch || !this.socket) return; + // Must match what sendSelf actually requires (!socket || !connected). + // Guarding on `socket` alone was the 23-day outage: on `connection === + // "close"` the socket OBJECT survives while `connected` flips false, so + // the tick cleared this guard every 12 minutes and then threw "not + // connected" inside sendSelf — 3,238 failures, taking the daily digest + // and news alerts with it. The timer intentionally keeps running so the + // feed resumes on its own once the socket is back. + if (this.killSwitch || !this.socket || !this.connected) return; if (this.proactivePaused()) return; const r = await authedFetch("/v1/news?sort=popular&limit=20"); if (!r.ok) return; @@ -12072,6 +12079,16 @@ export class WhatsAppSession { clearTimeout(this.reconnectTimer); this.reconnectTimer = null; } + // newsTimer was the one interval disconnect() forgot, and it fires + // sendSelf. Left running across a disconnect it retried every 12 minutes + // against a dead socket: 3,238 of 3,246 sendSelf calls failed with "not + // connected" over 23 days, taking the daily digest and news alerts with + // them. Every sibling timer here was already cleared; this one just got + // missed. connect() re-arms it, so clearing is safe. + if (this.newsTimer) { + clearInterval(this.newsTimer); + this.newsTimer = null; + } if (this.gcTimer) { clearInterval(this.gcTimer); this.gcTimer = null; From d2173286e1ccf00a3cb425f25d5a34199a0aba5d Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 21:26:34 -0400 Subject: [PATCH 5/9] fix(bridge): release the dedup claim when the owner DM never lands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review caught a real regression in the dedup I added: markActed() ran before the send, and the send swallows errors (.catch(() => {})). A transient failure would therefore mark the event surfaced forever — the retry sees hasActed() and stays silent. That traded "fraud alert sent 27 times" for "fraud alert possibly sent zero times", which is the worse failure. Keeping the claim BEFORE the emit is deliberate: two concurrent classifications of the same inbound must not both pass the check and both send — that race is the storm. So claim up front, and RELEASE the claim on every path where nothing actually reached the owner: - owner DM send failed (both bridges) - no owner self-chat target / no own JID (both bridges) Digest keeps its claim: the enqueue is synchronous and cannot fail. releaseLifeEventClaim() wraps the existing unmarkActed() store primitive, whose round-trip is already pinned by "unmarkActed lets a key be re-acted later" in auto-act-ladder.test.ts. The WhatsApp case is not hypothetical — sendSelf throws whenever the socket is down, the exact failure that took the digest out for 23 days, so without this the first outage would have permanently silenced every event it touched. 1489/1489 bridge-core tests pass; both bridges tsc --noEmit clean. No unit test at the session layer — session.ts has no test harness in this repo; the store primitive it relies on is covered. --- services/imessage-bridge/src/session.ts | 29 ++++++++++++++++++++++++- services/whatsapp-bridge/src/session.ts | 25 ++++++++++++++++++++- 2 files changed, 52 insertions(+), 2 deletions(-) diff --git a/services/imessage-bridge/src/session.ts b/services/imessage-bridge/src/session.ts index 8cb8302d..fd2e41c2 100644 --- a/services/imessage-bridge/src/session.ts +++ b/services/imessage-bridge/src/session.ts @@ -1383,6 +1383,21 @@ export class IMessageSession { // This must NOT be called for NORMAL non-responses — "not the owner", // "not addressed in a group", "low-confidence draft held", "trivial // chatter ack". Those are by-design silence, not drops. + // Release a life-event idempotency claim taken before an emission that then + // failed to deliver. Without this, a claim-then-fail permanently suppresses + // the event: the retry would see hasActed() and stay silent forever. Only + // ever called on paths where nothing reached the owner. + private async releaseLifeEventClaim(key: string | undefined, why: string): Promise { + if (!key) return; + try { + const { unmarkActed } = await import("@lantern/bridge-core/life-events-store"); + unmarkActed(key); + this.logger.warn({ idempotencyKey: key, why }, "life-event claim released — not delivered, will retry"); + } catch (err) { + this.logger.warn({ err, idempotencyKey: key }, "failed to release life-event claim"); + } + } + private notifyOwnerOfDrop(message: string, dedupeKey: string): void { try { const now = Date.now(); @@ -4451,6 +4466,12 @@ export class IMessageSession { // same second). That is the "spammy/repetitive" report. The key is already // computed above for auto-act; reuse it so all three routes share one // guard rather than each growing its own. + // Claim the key BEFORE emitting so two concurrent classifications of the + // same inbound can't both pass the check and both send — that race is the + // storm. But a claim that is never delivered must be RELEASED: marking + // up-front and then failing the send would suppress the alert forever, and + // a fraud notice silently dropped is worse than one sent twice. Every + // non-delivering path below calls releaseLifeEventClaim(). if (auto.idempotencyKey) { const { hasActed, markActed } = await import("@lantern/bridge-core/life-events-store"); if (hasActed(auto.idempotencyKey)) { @@ -4465,6 +4486,9 @@ export class IMessageSession { if (!owner) { // No self-chat target — surface to the dashboard feed so it isn't invisible. + // The owner DM never happened, so release the claim: once a self-chat + // target resolves, this event should still be able to reach them. + await this.releaseLifeEventClaim(auto.idempotencyKey, "no owner self-chat target"); this.broadcast({ type: "activity", data: { kind: "system", summary: decision.ownerMessage, timestamp: Date.now() } }); return true; } @@ -4488,7 +4512,10 @@ export class IMessageSession { } // ping — DM the owner now + arm the one-tap offer for the top action. - await this.send(owner, decision.ownerMessage).catch(() => {}); + // Release the claim if the DM did not land, so a transient send error + // cannot suppress this event forever. + const delivered = await this.send(owner, decision.ownerMessage).then(() => true).catch(() => false); + if (!delivered) await this.releaseLifeEventClaim(auto.idempotencyKey, "owner DM send failed"); this.lastLifeEventKind = event.kind; const top = decision.actions.find((a) => a.kind !== "snooze" && a.kind !== "none"); if (top && top.offerAction) { diff --git a/services/whatsapp-bridge/src/session.ts b/services/whatsapp-bridge/src/session.ts index 51f754b3..34f44f07 100644 --- a/services/whatsapp-bridge/src/session.ts +++ b/services/whatsapp-bridge/src/session.ts @@ -3618,6 +3618,9 @@ export class WhatsAppSession { markActed(auto.idempotencyKey); } if (!ownJid) { + // Nothing reached the owner — release the claim so this can retry once + // a self JID resolves, instead of being suppressed forever. + await this.releaseLifeEventClaim(auto.idempotencyKey, "no own JID"); this.logActivity("attention_dm", decision.ownerMessage, { scope: "self" }); return true; } @@ -3639,7 +3642,12 @@ export class WhatsAppSession { } // ping — DM the owner now + arm the one-tap offer for the top action. - await this.confirmToSelf(decision.ownerMessage).catch(() => {}); + // Release the claim if the DM did not land — a transient send failure must + // not suppress the event permanently. Especially relevant here: sendSelf + // throws whenever the socket is down, which is exactly the failure mode + // that took the digest out for 23 days. + const delivered = await this.confirmToSelf(decision.ownerMessage).then(() => true).catch(() => false); + if (!delivered) await this.releaseLifeEventClaim(auto.idempotencyKey, "owner DM send failed"); this.lastLifeEventKind = event.kind; const top = decision.actions.find((a) => a.kind !== "snooze" && a.kind !== "none"); if (top && top.offerAction) { @@ -3803,6 +3811,21 @@ export class WhatsAppSession { } // Auto-act couldn't run cleanly — surface a suggest ping instead of dropping. + // Release a life-event idempotency claim taken before an emission that then + // failed to deliver. Without it a claim-then-fail suppresses the event + // forever: the retry sees hasActed() and stays silent. Mirror of the + // iMessage helper; only called where nothing reached the owner. + private async releaseLifeEventClaim(key: string | undefined, why: string): Promise { + if (!key) return; + try { + const { unmarkActed } = await import("@lantern/bridge-core/life-events-store"); + unmarkActed(key); + this.logger.warn({ idempotencyKey: key, why }, "life-event claim released — not delivered, will retry"); + } catch (err) { + this.logger.warn({ err, idempotencyKey: key }, "failed to release life-event claim"); + } + } + private async autoActFallbackToSuggest( ownJid: string, event: import("@lantern/bridge-core/life-events").LifeEvent, From f7410e5e6ff7154dc40527a76d35dfc2b87e6e12 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 21:58:43 -0400 Subject: [PATCH 6/9] =?UTF-8?q?fix(infra):=20make=20restart-on-rebuild=20a?= =?UTF-8?q?ctually=20work=20=E2=80=94=20WatchPaths=20was=20a=20no-op?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The WatchPaths I added to the dashboard plist does not do what I claimed. launchd's WatchPaths only LAUNCHES a job that is not running; it does not restart a running one. The dashboard sets KeepAlive=true, so it is always running and the key was dead weight. Verified rather than assumed: changed BUILD_ID, PID did not move. Replaced with a separate dev.lantern.dashboard-reload job — not KeepAlive, so a BUILD_ID change starts it, it kickstarts the dashboard, and it exits. Verified: touching BUILD_ID moved the dashboard PID 88764 -> 99422 and the tailnet page plus its chunks came back 200. RunAtLoad is false on purpose (otherwise it would kick the dashboard on every login) and ThrottleInterval 30 coalesces a build's writes into one restart. This closes a second symptom of the same stale-build defect, not just the first. NEXT_PUBLIC_API_URL is inlined into the client bundle at BUILD time, so a server left running across a rebuild also serves the OLD API host: the owner's phone was posting login to http://localhost:8080 — itself — and the page reported it as "invalid credentials". Rebuilding with the tailnet host and restarting fixed it; this job stops it recurring. --- .../dev.lantern.dashboard-reload.plist | 52 +++++++++++++++++++ scripts/launchd/dev.lantern.dashboard.plist | 14 ++--- 2 files changed, 56 insertions(+), 10 deletions(-) create mode 100644 scripts/launchd/dev.lantern.dashboard-reload.plist diff --git a/scripts/launchd/dev.lantern.dashboard-reload.plist b/scripts/launchd/dev.lantern.dashboard-reload.plist new file mode 100644 index 00000000..3769f69d --- /dev/null +++ b/scripts/launchd/dev.lantern.dashboard-reload.plist @@ -0,0 +1,52 @@ + + + + + Label + dev.lantern.dashboard-reload + + + ProgramArguments + + /bin/launchctl + kickstart + -k + gui/__UID__/dev.lantern.dashboard + + + WatchPaths + + __REPO_ROOT__/apps/web/.next/BUILD_ID + + + + RunAtLoad + + + + ThrottleInterval + 30 + + StandardOutPath + __HOME__/Library/Logs/Lantern/dashboard-reload.out.log + StandardErrorPath + __HOME__/Library/Logs/Lantern/dashboard-reload.err.log + + diff --git a/scripts/launchd/dev.lantern.dashboard.plist b/scripts/launchd/dev.lantern.dashboard.plist index 3fb20dd5..1f8c472a 100644 --- a/scripts/launchd/dev.lantern.dashboard.plist +++ b/scripts/launchd/dev.lantern.dashboard.plist @@ -31,16 +31,10 @@ Logs: ~/Library/Logs/Lantern/dashboard.{out,err}.log ThrottleInterval 15 - - WatchPaths - - __REPO_ROOT__/apps/web/.next/BUILD_ID - + StandardOutPath __HOME__/Library/Logs/Lantern/dashboard.out.log From 6a46da56ea3deef0433e91f12af2bcf4d5dd2542 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 22:20:15 -0400 Subject: [PATCH 7/9] =?UTF-8?q?fix(security):=20bump=20Go=20toolchain=20to?= =?UTF-8?q?=201.26.6=20=E2=80=94=20clears=207=20stdlib=20CVEs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit govulncheck failed the PR on seven stdlib advisories, all reachable from real call paths in the control-plane (TLS handshake via grpc.Server.Serve and http.Server.ListenAndServe, bytes.Buffer.ReadFrom in EmbedText): GO-2026-6088/6089/6090/6091 and GO-2026-5026 and friends, spanning crypto/tls, net/http, net/url, html/template, encoding/asn1, encoding/xml Every one is "Fixed in: @go1.26.6" — no dependency is implicated, so the whole fix is the toolchain directive. Bumped 1.26.5 -> 1.26.6 across all 13 modules so they stay consistent; CI resolves its Go via go-version-file: services/control-plane/go.mod and picks this up. Verified: all 13 modules build, and govulncheck on the control-plane (the module whose call paths were flagged) now reports "No vulnerabilities found". Pre-existing on master, not introduced by this branch — fixed here because the branch is what CI gates. --- e2e/quickstart/go.mod | 2 +- e2e/runtime/go.mod | 2 +- gen/go/go.mod | 2 +- packages/cli/go.mod | 2 +- packages/sdk-go/go.mod | 2 +- services/billing/go.mod | 2 +- services/control-plane/go.mod | 2 +- services/data-plane-agent/go.mod | 2 +- services/memory/go.mod | 2 +- services/notifier/go.mod | 2 +- services/runtime-scheduler/go.mod | 2 +- services/scheduler/go.mod | 2 +- services/workflow-engine/go.mod | 2 +- 13 files changed, 13 insertions(+), 13 deletions(-) diff --git a/e2e/quickstart/go.mod b/e2e/quickstart/go.mod index 5b85b7b4..5d35af80 100644 --- a/e2e/quickstart/go.mod +++ b/e2e/quickstart/go.mod @@ -2,4 +2,4 @@ module github.com/dshakes/lantern/e2e/quickstart go 1.23 -toolchain go1.26.5 +toolchain go1.26.6 diff --git a/e2e/runtime/go.mod b/e2e/runtime/go.mod index 64cd2abc..fcfad11d 100644 --- a/e2e/runtime/go.mod +++ b/e2e/runtime/go.mod @@ -2,4 +2,4 @@ module github.com/dshakes/lantern/e2e/runtime go 1.23 -toolchain go1.26.5 +toolchain go1.26.6 diff --git a/gen/go/go.mod b/gen/go/go.mod index 24d71703..475beafb 100644 --- a/gen/go/go.mod +++ b/gen/go/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/gen/go go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( google.golang.org/grpc v1.82.1 diff --git a/packages/cli/go.mod b/packages/cli/go.mod index 78409ae8..e1bf6449 100644 --- a/packages/cli/go.mod +++ b/packages/cli/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/packages/cli go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/dshakes/lantern/gen/go v0.0.0 diff --git a/packages/sdk-go/go.mod b/packages/sdk-go/go.mod index c3e1e933..682b17c9 100644 --- a/packages/sdk-go/go.mod +++ b/packages/sdk-go/go.mod @@ -2,4 +2,4 @@ module github.com/dshakes/lantern/packages/sdk-go go 1.23 -toolchain go1.26.5 +toolchain go1.26.6 diff --git a/services/billing/go.mod b/services/billing/go.mod index 92528942..dab28814 100644 --- a/services/billing/go.mod +++ b/services/billing/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/billing go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/dshakes/lantern/gen/go v0.0.0 diff --git a/services/control-plane/go.mod b/services/control-plane/go.mod index 59fdb06a..8d7e14e6 100644 --- a/services/control-plane/go.mod +++ b/services/control-plane/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/control-plane go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/golang-jwt/jwt/v5 v5.3.0 diff --git a/services/data-plane-agent/go.mod b/services/data-plane-agent/go.mod index 5b21e605..e745a8df 100644 --- a/services/data-plane-agent/go.mod +++ b/services/data-plane-agent/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/data-plane-agent go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/dshakes/lantern/gen/go v0.0.0 diff --git a/services/memory/go.mod b/services/memory/go.mod index 705fad94..5fd8f639 100644 --- a/services/memory/go.mod +++ b/services/memory/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/memory go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/jackc/pgx/v5 v5.7.4 diff --git a/services/notifier/go.mod b/services/notifier/go.mod index 1230252f..bf4d95ac 100644 --- a/services/notifier/go.mod +++ b/services/notifier/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/notifier go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/jackc/pgx/v5 v5.7.4 diff --git a/services/runtime-scheduler/go.mod b/services/runtime-scheduler/go.mod index 1b886a4a..c87423a7 100644 --- a/services/runtime-scheduler/go.mod +++ b/services/runtime-scheduler/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/runtime-scheduler go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/dshakes/lantern/gen/go v0.0.0 diff --git a/services/scheduler/go.mod b/services/scheduler/go.mod index 676c9d17..b3b58a6c 100644 --- a/services/scheduler/go.mod +++ b/services/scheduler/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/scheduler go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/dshakes/lantern/gen/go v0.0.0 diff --git a/services/workflow-engine/go.mod b/services/workflow-engine/go.mod index 775f0704..f3705ccc 100644 --- a/services/workflow-engine/go.mod +++ b/services/workflow-engine/go.mod @@ -2,7 +2,7 @@ module github.com/dshakes/lantern/services/workflow-engine go 1.25.0 -toolchain go1.26.5 +toolchain go1.26.6 require ( github.com/dshakes/lantern/gen/go v0.0.0 From 7bc9156177b78b6ac16cecdde9670ca74bb735a5 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Sun, 16 Aug 2026 23:42:52 -0400 Subject: [PATCH 8/9] =?UTF-8?q?fix(ci):=20bump=20the=20GOTOOLCHAIN=20pins?= =?UTF-8?q?=20too=20=E2=80=94=20go.mod=20alone=20was=20being=20overridden?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The toolchain bump in the previous commit did not take: the vuln job ran on that exact SHA and still reported "Found in: html/template@go1.26.5". Both workflows set GOTOOLCHAIN: go1.26.5 as job env, and that overrides the go.mod toolchain directive, so CI kept building with the vulnerable stdlib no matter what the modules asked for. Bumped both pins to go1.26.6 and refreshed the vuln comment, which still named the older advisory it was pinned for. Verified locally on the control-plane (the module whose call paths were flagged): govulncheck reports "No vulnerabilities found" on 1.26.6. --- .github/workflows/sdlc-lint.yml | 2 +- .github/workflows/sdlc-vuln.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/sdlc-lint.yml b/.github/workflows/sdlc-lint.yml index 5d7a9444..ed35cb3e 100644 --- a/.github/workflows/sdlc-lint.yml +++ b/.github/workflows/sdlc-lint.yml @@ -24,7 +24,7 @@ jobs: name: lint # stable name — referenced by branch-protection required checks runs-on: ubuntu-latest env: - GOTOOLCHAIN: go1.26.5 # pin to the same toolchain as the QA job + GOTOOLCHAIN: go1.26.6 # pin to the same toolchain as the QA job steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 diff --git a/.github/workflows/sdlc-vuln.yml b/.github/workflows/sdlc-vuln.yml index ab57e3c4..ac3fd60a 100644 --- a/.github/workflows/sdlc-vuln.yml +++ b/.github/workflows/sdlc-vuln.yml @@ -33,7 +33,7 @@ jobs: name: vuln # stable check name — referenced by branch-protection required checks runs-on: ubuntu-latest env: - GOTOOLCHAIN: go1.26.5 # pin to the toolchain that fixes GO-2026-5856 (crypto/tls ECH leak) and all stdlib CVEs through go1.26.5 + GOTOOLCHAIN: go1.26.6 # pin to the toolchain that fixes GO-2026-6088/6089/6090/6091 and all stdlib CVEs through go1.26.6 steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 From 9275da32b480bacc2632f4052a865f9e44c65c40 Mon Sep 17 00:00:00 2001 From: Shekhar Mudarapu Date: Mon, 17 Aug 2026 00:36:04 -0400 Subject: [PATCH 9/9] fix(infra): install.sh must actually install dashboard-reload MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review caught that the new job was dead on arrival for anyone but me: I installed it by hand with manual placeholder substitution, so it works on this machine while a fresh `install.sh` run would never create it — and if it somehow did, the plist would still carry a literal __UID__ in its kickstart target and fail. Two one-line gaps, both real: - __UID__ was not in the sed substitution list (only __NODE__, __REPO_ROOT__, __HOME__) - "dashboard-reload" was not in ALL_SERVICES Nothing else was needed: SRC/DST are derived from the service name, and the node_modules case-arm correctly skips a job that has no package. Verified by running the installer's exact substitution into a temp file: the result lints clean, has zero remaining placeholders, resolves to gui/501/dev.lantern.dashboard, and diffs IDENTICAL to the hand-installed plist already proven to restart the dashboard on a BUILD_ID change. --- scripts/launchd/install.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/launchd/install.sh b/scripts/launchd/install.sh index ef41fb2f..13e24830 100755 --- a/scripts/launchd/install.sh +++ b/scripts/launchd/install.sh @@ -25,7 +25,8 @@ LOG_DIR="$HOME/Library/Logs/Lantern" # Order matters — infra brings up docker, api waits for postgres, # dashboard waits for api. LaunchAgents run in parallel but each # wrapper waits for its upstream dependency before launching. -ALL_SERVICES=( "infra" "api" "dashboard" "whatsapp-bridge" "imessage-bridge" \ +ALL_SERVICES=( "infra" "api" "dashboard" "dashboard-reload" \ + "whatsapp-bridge" "imessage-bridge" \ "model-router" "runtime-manager" "runtime-scheduler" \ "workflow-engine" "gateway" "surface-gateway" ) @@ -108,6 +109,7 @@ for s in "${ALL_SERVICES[@]}"; do -e "s|__NODE__|$NODE_BIN|g" \ -e "s|__REPO_ROOT__|$REPO_ROOT|g" \ -e "s|__HOME__|$HOME|g" \ + -e "s|__UID__|$(id -u)|g" \ "$SRC" > "$TMP" mv "$TMP" "$DST"