docs: incident response on Switch — hub, responder agent and PagerDuty (CHOO-2721) - #443
Open
christian-mcdermott wants to merge 6 commits into
Open
christian-mcdermott wants to merge 6 commits into
christian-mcdermott wants to merge 6 commits into
Conversation
First slice of the incident-response design doc: what the SOP actually asks of Switch (one step of seven), and the room mapping — three rooms, what belongs in a thread, and the three things that look like rooms and must not be. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…CHOO-2721) Names the eleven parameters and gives the template twice: once in the single-room format that works on main today, once in the group format it wants once that branch lands. The merged-format document was run through the shipped parser rather than written by eye — it resolves with no placeholders left over. Records the two parameters that cannot exist (a list of responders, and who is on call), and what a template still cannot set on a room. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reads the flint-tracker setup off the live instance rather than assuming it, then says which parts a rotating on-call group should copy and which six break. States the recommendation: one shared agent per product, owned by a dedicated non-admin service user, auto-session on shared infrastructure, open addressing, never run from a laptop. User-agnostic means owned by a non-person, not owned by nobody — an ownerless agent cannot attach or create resources at all. Argues roles out: a lease is held per agent globally, so a shared agent can hold one in one incident at a time, and humans cannot hold roles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…CHOO-2721) Eighteen gaps grouped by what they block, each with a proposed ticket: knowing who is on call, making the template usable, getting the room made at all, the shared agent, and closing the incident out. Ends with what is usable on day one with no Switch change, and the order to remove the manual steps in. Two small template changes and a service account buy most of it; resolving a platform group to room members is the one the SOP has been asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g (CHOO-2721) A fact-check against the source found thirteen claims that were wrong or overstated. The substantive ones: - A template is not the only way to reach aliases, links, the group or join listeners — create_room and update_room already accept all of them. The finding is that a template is strictly less capable than the room creation it wraps, which makes the gap a cheap pass-through rather than a feature. - An agent can already open a war room; what it cannot do is open one from a template. - The gateway policy editor disables Save on an unmatchable rule, so it cannot brick an agent outright. The reachable damage is quieter: widening the default policy drops the owner rule. - A bootstrap-owned agent is manageable by an admin; it is its credential that can never be revealed. - Switch does run periodic work and does verify an inbound signed webhook on one bridge. The true claims are narrower: no scheduling primitive a room can use, and no generic alert ingress. - write_visibility: public grants write to any principal in the tenant, not just participants, and that covers archiving the room. The template now sets it private, and says why. Also fixes two quoted strings that were not faithful, and the operation and command counts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reworked after review: the war room is built by a responder agent calling create_room, not instantiated from a room template. The spine is now the workstream-hub pattern this repo already runs — a standing hub whose instructions carry the SOP plus a bindings block, an exclusive role that is a thin shell over the lease and @-addressing, an agent that builds one room per item with a full instruction card, and the banner protocol. An incident is a work item with a clock on it. Three things the rework changed: - A template cannot look anything up, and the most valuable value here is a lookup. An agent can go and find who is on call. - PagerDuty is reached the way Jira already is: an MCP server on the host, a reference type carrying the prose, and instance values in the hub's bindings. That closes what had been the hardest gap — Switch does not need to model a rotation, because the agent can ask. - Roles were wrongly ruled out. A lease is per agent globally, which rules out a role per war room, not a role in the hub. The hub role is exactly "reach whoever is coordinating without knowing which agent that is". Adds the Slack ingress findings (a third-party app's post does reach a room and can wake an agent; its rich blocks and its edits largely do not), and the two MCP constraints: per-machine scope, and no agent-scoped secret storage. Net: the design needs no Switch changes to run, one small change to be safe, and two more to be pleasant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
christian-mcdermott
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design document for running an on-call/incident-response SOP on Switch. Design only — nothing is built here. No product code, no template registered, no agent created. Anything that needs code is a proposed ticket, not work started.
docs/old/incident-response-sop.md, written againstmainat514d5ba4.The shape
A standing incident hub per product — the existing alert channel, adopted — whose
instructionscarry the SOP plus a bindings block of instance values. An exclusiveresponderrole in that hub: a thin shell granting the lease and@responderaddressing, so anyone reaches whoever is coordinating without knowing which agent that is. A shared responder agent that assumes it on connect, and on a declaration looks up who is on call, builds the war room withcreate_room, and posts a banner in the hub whose thread is the incident's whole record.That is the workstream-hub pattern this repo already runs, applied to incidents. An incident is a work item with a clock on it.
Three findings that shaped it
A room template cannot look anything up. Every value has to be supplied by whoever instantiates it, and the most valuable value here — who is on call — is a lookup. Separately, a room template is strictly less capable than the room creation it wraps:
aliases,linked_rooms,group_idandjoin_event_listenersare all already accepted bycreate_roomand simply not carried by the template format. So the agent builds the room; the YAML is registered as the reviewable spec, not as the mechanism.PagerDuty is reached the way Jira already is — and it is worth being precise that the Jira "integration" is a reference type whose instructions say "use the Atlassian MCP connector your operator installed". Switch ships the pointer and the prose. Copying that for PagerDuty closes what had been the hardest gap in the document: Switch never models a rotation, because the agent asks PagerDuty and passes the names to
create_room.Roles were wrongly ruled out in the first draft. A lease is unique per agent, globally — which rules out a role per war room, not a role in the hub. Corrected.
What it needs from Switch
Nothing, to run. The design is configuration plus one agent definition. One small change (a service account, G11) to be safe about ownership, and two more to be pleasant.
Twenty-one gaps, each with a ticket
Including two live bugs found on the way, neither fixed here:
ownerrule from any rule it saves, so widening an agent's default policy silently locks its owner out.bridgefield is documented as "omit for an internal-only room". Omitting it gets the default bridge, and separately makesusers:fail on a room that would have been bridged.And two constraints worth knowing before anyone plans around them: MCP scope is per-machine, not per-agent, and there is no agent-scoped secret storage anywhere in Switch.
On accuracy
The whole document is claims about how Switch behaves today. It was fact-checked adversarially against the source, which found thirteen wrong or overstated claims — all corrected, three of them conclusions that had to change. The room YAML is run through the shipped parser rather than written by eye. Anything still wrong is worth flagging hard; this is meant to be handed to a team as ground truth.
No names, emails or internal channel names: the team's specifics are bindings, not content.
🤖 Generated with Claude Code