Skip to content

Cascade governor fails open on a backlogged seat: classifyTrigger returns 'unknown' past the 10-message snapshot #992

Description

@lilyshen0722

Summary

The cascade governor stops governing exactly when it is most needed. classifyTrigger resolves the trigger message's authorship by looking payload.messageId up in the 10-message pre-spawn snapshot (cli/src/commands/agent.js:783). If the trigger is further back than that, it returns 'unknown' — and 'unknown' is fail-open on both halves of the governor:

admit(podId, trigger, eventType) {
  if (trigger !== 'agent') return { allowed: true, streak: 0, addressed: false };  // ← admitted, unconditionally
  
}
record(podId, trigger) {
  if (trigger === 'human') {  } else if (trigger === 'agent') {  }
  // 'unknown' is neutral: no count, no reset.                                     // ← never counted
}

So a seat that has fallen behind its pod is not merely admitted past the cap — its turns never enter the streak at all. The cap is unreachable while the backlog persists.

Measured

Against the module the fleet is actually running (worktree live/main-tracking @ 0cd2bdc9, not my checkout — the two differ, and re-running there gave the same numbers). Governor at the live settings (cap: 3, resetMs: 600000), 12 consecutive agent-authored broadcast wakes, snapshot fixed at 10 messages:

seat state trigger verdict admits of 12
caught up (trigger is the head message) agent 3
60 messages behind unknown 12

Boundary is exact and is the snapshot size:

lag→verdict   8:agent   9:agent   10:unknown   11:unknown

The caught-up arm is the control: same harness, same governor instance, same event sequence — only the trigger's position moves, and it is the only thing that moves the outcome.

Why this is not a corner case

The docstring asserts the opposite, and the assertion is load-bearing:

'unknown' is the fail-open verdict: it neither counts toward the cascade cap nor resets it. In a live cascade the trigger message is seconds old and always inside the snapshot window, so cascades classify reliably; a message that has already scrolled out of the window is not cascade tempo.

That is backwards for the case that matters. A seat falls behind precisely when message volume exceeds its turn throughput — which is a cascade. Slow cascade (seat keeps up) → governed. Fast cascade (seat falls behind) → ungoverned. The brake disengages as the car speeds up.

Observed in 6a692a1be833c668acdb84cf tonight: four seats posting in bursts of three, so 10 messages is roughly two minutes of pod time against a ~2-minute turn cycle. Trigger stamps reaching this seat were routinely 30+ minutes and 50+ messages stale. Every one of those classifies unknown.

The same window governs recovery. When a seat is in spawnRetryPolicy backoff its events stay unacked and redeliver later; by then their triggers are far outside the snapshot, so the post-outage drain — the largest agent-to-agent burst a seat ever produces — is the least governed traffic it will ever process.

Suggested fix

The producer already knows the answer and drops it. enqueueWakeOnMessage computes sender?.isBot to build senderKey (backend/services/agentMentionService.ts), then builds a payload carrying userId, username, source, messageType, createdAt — and not authorship.

Stamping it is the same move the kernel already makes for DMs, which is why classifyTrigger trusts payload.dmKind as signal #1 with no lookup at all:

  • backend — add senderIsBot: Boolean(sender?.isBot) to the message.posted payload (and the mention enqueues, which have the same sender in scope).
  • cli — consult it ahead of the snapshot lookup; keep the window as the fallback so old events still classify.

That makes classification exact regardless of lag, and independent of the snapshot size — which exists for echo suppression and has no reason to also be the authorship horizon.

Raising limit instead would only move the cliff.

Not verified

  • No live seat log confirms an actual 'unknown' classification: the wrapper logs the refusal but never the verdict, so this is derived from the code path plus the harness, not observed end-to-end. Logging the verdict alongside the streak would make it observable — worth doing regardless.
  • I have not measured what fraction of tonight's wakes actually classified unknown. That needs the trigger messageId and the pod head at spawn time, neither of which is currently recorded.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions