From 30b2e2ddce69027f547eb054d8d39590812267ec Mon Sep 17 00:00:00 2001 From: Christian McDermott Date: Fri, 11 Sep 2026 15:26:00 -0400 Subject: [PATCH 1/6] docs(incident-response): the SOP, and how it maps onto rooms (CHOO-2721) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First slice of the incident-response design doc: what the SOP actually asks of Switch (one step of seven), and the room mapping — three rooms, what belongs in a thread, and the three things that look like rooms and must not be. Co-Authored-By: Claude Opus 5 (1M context) --- docs/old/incident-response-sop.md | 214 ++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 docs/old/incident-response-sop.md diff --git a/docs/old/incident-response-sop.md b/docs/old/incident-response-sop.md new file mode 100644 index 000000000..085a14830 --- /dev/null +++ b/docs/old/incident-response-sop.md @@ -0,0 +1,214 @@ +# Incident response on Switch + +How an on-call/incident-response SOP runs on Switch: the room shape, a reusable +room template that provisions a war room, and a responder agent that a rotating +on-call group can share without any one engineer owning it. + +This is a design, not an implementation. Nothing here has been built. Where +Switch cannot do what the SOP needs, the gap is named and a ticket proposed +rather than designed around — [Gaps](#gaps) is the part to read if you read +only one section. + +Written against `main` at `514d5ba4` (the template registry). Every claim about +current behaviour was checked against the code, and the file it lives in is +cited so a reader can confirm it rather than trust it. + +## Scope + +The subject is a specific SOP: a lightweight, post-launch, business-hours +rotation that pages through PagerDuty and coordinates in Slack. It is +deliberately temporary — its own text says it will be replaced once a 24/7 +rotation and real tooling exist. So the design optimises for *reuse and +disposal*, not for permanence: a template a team instantiates per incident and +throws away, not a standing structure to maintain. + +The SOP belongs to one product team. This document does not reproduce it. The +team's channel names, service owners and escalation contacts are inputs, not +content — they arrive as template parameters at instantiation time. That keeps +this document reusable across products, and keeps a public repository free of +one team's internal routing. + +**Out of scope.** Designing the PagerDuty or Datadog integrations themselves; +Switch Console's side of any of this; anything that needs code to exist before +it can be described. Where the SOP depends on such a thing, it appears in +[Gaps](#gaps). + +## The SOP, and the one place Switch appears in it + +The flow, compressed: + +1. A Datadog monitor breaches and pages the on-call primary through PagerDuty, + repeating up to three times. The same alerts also land in a standing alert + channel. +2. On-call acknowledges in PagerDuty — "I'm investigating." +3. On-call triages against logs, dashboards and runbooks, and decides the one + question that matters: **is a customer affected?** No means it is a routine + alert, resolved in PagerDuty with notes, and nothing else happens. Yes means + a customer incident is declared. +4. Declaring sets a severity, and **the declaration is what triggers + coordination**. At the top severity a dedicated war-room channel is created + and the responders are pulled into it. +5. Updates go out on a clock — hourly at the top severity, every four hours at + the next — as a five-field situation report posted to both the alert channel + and the stakeholder channel. +6. On-call acts on their own authority (roll back, roll forward, emergency fix) + and escalates if it is not resolved in about an hour, up a four-tier ladder. +7. Recovery is confirmed in PagerDuty, a postmortem is written, and a review is + scheduled within five business days for the top severity. + +Read that list again and notice how little of it is Switch's. Paging, ack, +severity, the resolve, the audit trail of the incident record — all PagerDuty. +Diagnosis is Datadog and the runbooks. Switch appears at exactly one point, step +4: + +> Switch will auto-create a dedicated Slack channel and invite on-call engineers +> to the war room. + +That is the whole ask, and it is worth being blunt about the size of it: **the +SOP does not need Switch to run incident response. It needs Switch to +manufacture a correctly-shaped, correctly-populated room in the seconds after a +declaration, and then to be useful inside it.** A design that tries to move +severity, paging or the incident record into Switch is designing a competitor to +PagerDuty that nobody asked for. + +The value Switch adds is not the channel — Slack can make a channel. It is that +the room arrives *already furnished*: the runbook attached, the service-owner +map attached, the situation-report shape attached, the responder agent already +in it and already briefed on which service is broken and how severe it is. A +human doing this by hand at 03:00 does it badly or not at all. + +### The questions the SOP has not answered + +The source document carries open comments, and three of them are load-bearing +for this design. They are not oversights in the SOP; they are places where the +SOP is waiting on Switch: + +- **"How will Switch pull in who's on-call from PagerDuty?"** Nobody has + answered this. It is the single hardest requirement in the document, and + [Gaps](#gaps) treats it as such. +- **"Can we automate mirroring updates from the alert channel into the + stakeholder channel?"** Two rooms, one message, no relay. +- **"A scheduled Slack workflow could mention the Switch agent to kick off + updates."** This one is already the right answer, and it works today — + see [Cadence](#cadence-and-the-thing-that-nudges). + +A fourth comment observes that postmortems are missing from the SOP entirely. +The template can help there, and does. + +## Mapping the SOP onto rooms + +Switch has exactly one structural primitive that matters here — the room — plus +threads inside it and links between rooms. Getting the mapping right is mostly a +matter of refusing to over-model. + +### What is a room + +**Three, and only three.** + +**The alert hub** — standing, long-lived, one per product. Datadog's alerts land +here, and this is where on-call acknowledges and posts situation reports. It +already exists as a Slack channel; adopting it into Switch is a matter of adding +the Switch app to it, not creating anything. High volume, low signal, and nobody +should be expected to have read it. + +**The stakeholder channel** — standing, long-lived, one per product. High-level +status only, for people who need to know that something is wrong and not how. +Also already exists. + +**The war room** — one per declared incident, created at declaration, dead after +the postmortem. This is the room the template makes. Public, per the SOP's own +resolution of that question: a war room that stakeholders cannot read generates +a second, worse war room in DMs. + +The two standing rooms are not the template's business. They are pre-existing +channels the template *points at*, and their names are parameters. + +### What is a thread + +Everything that would otherwise fragment the war room. In particular: + +- **A workstream inside the incident.** Two people chasing two hypotheses thread + separately and the room stays readable. +- **A situation report and its follow-ups.** The SITREP goes at the root; the + "what does that mean for the API?" questions hang off it. +- **A tool's noisy output.** Log dumps and query results belong under the + message that asked for them. + +Switch threads bridge natively to Mattermost and to Telegram forum topics. On a +Slack-bridged room — which this is — a threaded reply shows in the channel as a +reply count under the original post, so **anything the room must not miss goes at +the root**. That is not a Switch limitation to work around; it is a rule for +whoever writes in the room, agent or human. The responder agent's standing +instructions should say so, and the template's `instructions` field is where +that lives. + +### What is neither + +Three things that look like they want to be rooms and must not be. + +**The incident itself.** The incident is a PagerDuty record with an id, a +severity, a timeline and a resolution. The war room is a *conversation about* it. +Modelling the incident in Switch means two systems disagreeing about severity at +the worst possible moment. The room carries the incident id in its name and a +link to the record in its description; that is the whole of the relationship. + +**The on-call rotation.** A rotation is a schedule — who is responsible between +which hours. Switch has no schedule, no rotation and no concept of duty (see +[Gaps](#gaps)). A "rotation room" would be a room whose membership someone has to +remember to edit every Monday, which is a worse rotation than the one PagerDuty +already runs. The rotation stays in PagerDuty and reaches Switch, if at all, as a +mention group. + +**A per-service standing room.** Tempting, because the SOP's severity table is +organised by service and each service has an owner. But a room per service is a +room per service to keep alive, and the thing that is actually needed — "who owns +the ingestion pipeline, and what does its runbook say" — is a lookup, not a +conversation. It belongs in the war room as an attached document, which is +exactly what the template does with it. + +### The lifecycle, end to end + +| Moment | What happens in Switch | +| --- | --- | +| Alert fires | Nothing. Datadog → PagerDuty → the alert hub channel. | +| Ack, triage | Nothing. On-call works in PagerDuty, Datadog and the runbooks. | +| Routine alert, no customer impact | Nothing, ever. Most alerts end here and must cost zero Switch overhead. | +| **Customer incident declared** | **The war room is instantiated from the template**, named for the incident, furnished with runbook, owner map and SITREP shape, with the responder agent already in it and briefed. | +| Responders assemble | Invitees are added: on-call, the service owner, the stream lead, support. Public channel, so anyone else can walk in. | +| Investigation | Threads per hypothesis. The responder agent answers lookups, drafts SITREPs, and keeps the timeline. | +| Situation report due | A scheduled nudge addresses the responder agent; it drafts from the room and a human posts or corrects it. | +| Escalation at ~1h | A human decision. Switch's part is that the escalation ladder is *in the room* as a document, so nobody has to find it. | +| Recovery confirmed | The room stays open — the postmortem is written from it. | +| Postmortem written | The seeded postmortem document is filled in from the room's own timeline. | +| Review scheduled and held | Out of Switch. | +| Done | The room is archived. Archive is not deletion; the transcript survives. | + +Two properties of that table are the design: + +- **Nothing happens until a customer incident is declared.** The overwhelmingly + common path — an alert that resolves itself — never touches Switch. Any design + that provisions a room per alert will be switched off within a week. +- **The room outlives the incident.** It closes at the postmortem, not at + recovery. The postmortem is written from the room's own record, which is the + main argument for having conducted the incident in a room at all. + +### Cadence, and the thing that nudges + +The SOP puts situation reports on a clock: hourly at the top severity, every four +hours below it. Something has to remember. + +**Switch cannot.** There is no scheduler, no cron, no timer, and no deferred +action anywhere in `core/switch_core` — the only `schedule` symbols in the tree +are `asyncio` call-soon helpers inside the transport and bridge loops. An agent +in a room cannot ask to be woken in an hour. + +The SOP's own comment thread already has the answer, and it is the right one: a +scheduled Slack workflow that posts into the war room mentioning the responder +agent. The mention arrives as an addressed event, the agent drafts the SITREP +from the room's context, a human checks and posts it. This works today, needs no +Switch change, and keeps the clock in the tool that is good at clocks. + +It is worth being honest that this is a workaround, not a design: the schedule +lives in a Slack workflow that nobody will remember to delete when the incident +closes, and it has to be created per incident. That is a real cost, and it is +[Gaps](#gaps) item G7. From af09d475c6258e0337e517cb6a90f7b0320cb28c Mon Sep 17 00:00:00 2001 From: Christian McDermott Date: Fri, 11 Sep 2026 15:29:39 -0400 Subject: [PATCH 2/6] docs(incident-response): the war-room template, parameters and YAML (CHOO-2721) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Names the eleven parameters and gives the template twice: once in the single-room format that works on main today, once in the group format it wants once that branch lands. The merged-format document was run through the shipped parser rather than written by eye — it resolves with no placeholders left over. Records the two parameters that cannot exist (a list of responders, and who is on call), and what a template still cannot set on a room. Co-Authored-By: Claude Opus 5 (1M context) --- docs/old/incident-response-sop.md | 451 ++++++++++++++++++++++++++++++ 1 file changed, 451 insertions(+) diff --git a/docs/old/incident-response-sop.md b/docs/old/incident-response-sop.md index 085a14830..768ede9da 100644 --- a/docs/old/incident-response-sop.md +++ b/docs/old/incident-response-sop.md @@ -212,3 +212,454 @@ It is worth being honest that this is a workaround, not a design: the schedule lives in a Slack workflow that nobody will remember to delete when the incident closes, and it has to be created per incident. That is a real cost, and it is [Gaps](#gaps) item G7. + +## The war-room template + +The template is the deliverable that makes this reusable. One document, stored +in the registry, instantiated per incident with the incident's particulars as +inputs, and reusable across products because everything product-specific is a +parameter. + +### What the template feature actually is today + +Before the YAML, the state of the feature it is built on, because two of the +three pieces named on the ticket are not on `main` and the design has to be +honest about which half exists. + +**Merged and working** (`core/switch_core/rooms_yaml.py`): + +- A single-room YAML document with exactly three top-level keys — `room:` + (required), `params:` and `version:`. `version:` is parsed, type-checked and + then ignored. An unrecognised top-level key is a hard error. +- Typed parameters: `string`, `number`, `boolean`, `enum`, each with an optional + `description` and `default`. There is no `required:` key — a parameter is + required exactly when it has no default. +- `{name}` interpolation over the `room:` block. Two modes: if a field is + *entirely* one placeholder, the parameter's typed value is substituted whole, + which is how an `enum` can fill `channel_type:`; otherwise each placeholder is + stringified in place. +- A registry (`core/switch_core/gateway/templates.py`): store, list, fetch, + patch, delete and lint a template, with names unique per owner, listing + visible tenant-wide and mutation restricted to the owner or an admin. + +**Not merged.** Group templates — `group:` / `rooms:` / `links:`, per-room +agent aliases in YAML, and interpolation into dict *keys* — are on +`origin/work/group-templates`, along with `create_room_from_yaml`, the operation +that would let an agent instantiate a template at all. Template built-ins like +`{$creator}` are on a different branch again. + +**The two halves do not meet.** The registry stores documents; `POST +/rooms/from-yaml` provisions from a document supplied in the request body. +Nothing fetches a stored template by id and provisions it. Instantiating a +registered template today means downloading its content and posting it back — +and the dashboard's create-from-YAML page posts raw YAML with no `inputs`, so +*from the UI, only a template whose parameters all have defaults can be used at +all*. Passing inputs requires an API client sending the JSON body form. For a +template whose entire purpose is per-incident particulars, that is disqualifying +on its own; it is [Gaps](#gaps) G2 and G3. + +The template below is therefore written twice: once in the merged format, so it +can be built and used now, and once in the group format, as the target. + +### Parameters + +Eleven, in three groups. + +**The incident** — supplied per instantiation, no defaults, all required: + +| Parameter | Type | What it is | +| --- | --- | --- | +| `incident_id` | string | The incident record's id, e.g. `1287`. Goes in the room name so the channel is greppable against PagerDuty. | +| `severity` | enum | `sev0` \| `sev1` \| `sev2`. Drives the update cadence named in the room instructions. | +| `service` | string | The affected service. Drives the owner lookup and the runbook section. | +| `summary` | string | One line: what is broken. Becomes the room description. | +| `incident_url` | string | Link to the incident record. The room's pointer at the system of record. | + +**The product** — supplied per product, and the reason this is reusable rather +than one team's room: + +| Parameter | Type | Default | What it is | +| --- | --- | --- | --- | +| `product` | string | — | Short product name. Prefixes the room name. | +| `alert_channel` | string | — | Where alerts land and situation reports are posted. | +| `comms_channel` | string | — | Where stakeholder updates go. | +| `runbook_reference` | string | — | Name of an existing Switch reference pointing at the product's runbooks. | +| `responder_agent` | string | — | The shared responder agent's name. | + +**The deployment** — sane defaults, rarely overridden: + +| Parameter | Type | Default | What it is | +| --- | --- | --- | --- | +| `bridge` | string | — | Collaboration bridge display name. Required; see the note on omitting it below. | +| `visibility` | enum | `channel_public` | `channel_public` \| `channel_private`. Public by default, per the SOP. | + +Two parameters that are conspicuously *not* here, because they cannot be: + +- **The responders.** `users:` takes a list, and a parameter cannot hold one — + `ParamSpec.type` is `string | number | boolean | enum` and nothing else. A + parameter set to `"alice,bob"` interpolates into a single username + `alice,bob`, which resolves to nobody. So the invitee list is either + hard-coded in the template or spread across one parameter per seat. Neither is + acceptable for a rotation. [Gaps](#gaps) G4. +- **Who is on call.** Nothing in Switch knows. [Gaps](#gaps) G1, and the + hardest problem in this document. + +### The template, in the format that works on `main` today + +```yaml +version: 0 + +params: + # ── the incident ────────────────────────────────────────────────────────── + incident_id: + type: string + description: Incident record id, e.g. 1287 + severity: + type: enum + enum: [sev0, sev1, sev2] + description: Declared severity; sets the update cadence + service: + type: string + description: The affected service, as named in the severity guidelines + summary: + type: string + description: One line — what is broken, in a stakeholder's words + incident_url: + type: string + description: Link to the incident record in the paging system + + # ── the product ─────────────────────────────────────────────────────────── + product: + type: string + description: Short product name; prefixes the room and channel name + alert_channel: + type: string + description: Channel where alerts land and situation reports are posted + comms_channel: + type: string + description: Channel where stakeholder updates go + runbook_reference: + type: string + description: Name of the Switch reference holding this product's runbooks + responder_agent: + type: string + description: The shared incident responder agent + + # ── the deployment ──────────────────────────────────────────────────────── + bridge: + type: string + description: Collaboration bridge display name + visibility: + type: enum + enum: [channel_public, channel_private] + default: channel_public + description: War rooms are public by default so stakeholders can read along + +room: + name: "{product} incident {incident_id}" + description: "{severity} · {service} · {summary} · {incident_url}" + bridge: "{bridge}" + channel_type: "{visibility}" + read_visibility: public + write_visibility: public + + agents: ["{responder_agent}"] + + instructions: | + This is the war room for {product} incident {incident_id} — {severity} on + {service}. + + What is broken: {summary} + The incident record is the system of record: {incident_url} + + ## For everyone in this room + + Post at the ROOT for anything the room must not miss. This room is bridged + to Slack, where a threaded reply shows only as a reply count under the + original post — a status change buried in a thread will be missed. Use + threads for a single line of investigation, for follow-up questions under a + situation report, and for tool output. + + Severity, ack and resolution live in the paging system, not here. If they + disagree, the paging system is right. Update it there and say so here. + + ## For the responder agent + + You are here to remove lookups and paperwork from the responders, not to + run the incident. A human decides what to do. + + - Answer from the attached documents first. The service-owner map says who + owns {service}; the runbook reference points at how to diagnose it. Cite + what you used. + - When asked for a situation report, draft it in the shape the attached + SITREP document gives and post the draft here. A human posts it onward to + {alert_channel} and {comms_channel} — you do not post to those channels. + - Keep a running timeline as the incident moves: what changed, when, who + did it. The postmortem is written from it. + - Say what you do not know. During an incident a confident wrong answer + costs more than silence. + - Cadence for {severity}: sev0 updates hourly, sev1 every four hours, sev2 + on change only. You will be nudged; if a nudge is late, say so. + + ## Escalation + + Escalate if this is not resolved in about an hour. The ladder is in the + attached escalation document. Escalating is not an admission of failure and + does not need sign-off. + + roles: + - name: scribe + exclusive: true + instructions: | + You are keeping this incident's timeline. Record what changed, when, + and who did it, as it happens — one line per event, at the room root. + Do not editorialise and do not diagnose; the timeline is evidence for + the postmortem, not an analysis. + + Read the room's history before you start, so the timeline begins at the + declaration and not at the moment you arrived. + + references: + - name: "{runbook_reference}" + + docs: + - name: "Service owners" + description: "Which service belongs to whom, and in which timezone" + instructions: | + Consult before asking the room who owns something. Answer from this and + cite it. If {service} is not listed, say so plainly rather than + guessing — an unlisted service is a real gap in the SOP, not a lookup + failure. + content: | + # Service owners + + Filled in per product at instantiation. One row per service: + service, owning team, primary contact role, timezone. + + A service with no owner is an escalation to the incident coordinator, + not a dead end. + + - name: "Situation report" + description: "The five fields a situation report must carry" + instructions: | + Use this shape verbatim when drafting a situation report. Do not add + fields, do not drop the Ask — an update with no Ask reads as "no help + needed" and that is rarely true. + content: | + # Situation report + + - **Summary** — one line: what is broken, plus the incident id. + - **Severity** — the declared severity, and impact: who and what, and + the blast radius. + - **Started** — time, and the suspected trigger (a deploy, a config + change, unknown). + - **Progress** — diagnostics run, actions tried, current state. + - **Ask** — what help is needed, from whom. Say "none" explicitly if + none. + + - name: "Escalation ladder" + description: "Who to pull in, and when" + instructions: | + Consult when an incident has run about an hour without resolution, or + when someone asks who to escalate to. Name the tier, not a person — + the room will know who currently holds it. + content: | + # Escalation ladder + + 1. **On-call primary** — every declared incident. + 2. **Workstream lead or service owner** — stuck about an hour, or the + problem needs depth in one service. Have a situation report ready + before escalating. + 3. **Engineering and product decision pair** — the fix needs a business + call quickly. + 4. **Incident coordinator and leads** — org-wide impact, or all hands. + + Escalation is time-based, not judgement-based. An hour without + resolution escalates whether or not it feels close. + + - name: "Postmortem" + description: "The write-up this incident owes, and the shape of it" + instructions: | + Fill this in from the room's own timeline once recovery is confirmed. + Draft it here; a human owns it. Blameless: name systems and decisions, + never people. + content: | + # Postmortem — {product} incident {incident_id} + + - **What happened** — the customer-visible failure, in one paragraph. + - **Impact** — who was affected, how many, for how long. + - **Timeline** — from the first signal to recovery confirmed. + - **Root cause** — the five whys, not the first why. + - **What went well** — including anything that limited the blast radius. + - **What did not** — detection gaps, missing runbooks, wrong owners. + - **Actions** — each with an owner and a ticket. An action with neither + is a wish. + + Review within five business days for the top severity. +``` + +Notes on choices in that document that are not arbitrary: + +- **`channel_type: "{visibility}"` is the whole-field form.** When a field is + the *entire* placeholder, the parameter's typed value is substituted rather + than stringified. For an `enum` the two are the same, but the distinction is + the only way a `number` or `boolean` parameter can ever fill a non-string + field, and it is worth knowing because a partial placeholder degrades to a + string silently. +- **The room name slugifies cleanly.** Switch derives the Slack channel name with + `re.sub(r"[^a-z0-9_-]", "-", name.lower()).strip("-")[:80]` + (`core/switch_core/bridges/collaboration/slack/adapter.py:1366`). The SOP's + bracketed convention — `[Product] [Incident #]` — would produce + `product---incident-42`, with runs of hyphens where the punctuation was. + `{product} incident {incident_id}` gives `product-incident-42`. Cosmetic, but + channel names are what responders type under pressure. +- **`bridge:` is a required parameter and not omitted.** The field's own comment + says to omit it for an internal-only room, and that is wrong: omitting it + falls through to the instance default bridge. Worse, the guard that rejects + `users:` on an unbridged room tests the template's resolved bridge id, so a + template with `users:` and no `bridge:` is rejected even though the room would + in fact have been bridged. Naming the bridge explicitly sidesteps both. + [Gaps](#gaps) G10. +- **The `scribe` role is defined but nobody is told to take it.** See + [Roles are the wrong tool for the on-call rotation](#roles-are-the-wrong-tool-for-the-on-call-rotation) + — a role is held per *agent*, globally, so the shared responder can hold it in + one incident at a time. It is there for a responder's own coding agent to + assume, and the template does not assume it on anyone's behalf. +- **`write_visibility: public`** means any participant's owner can restructure + the room — pull in another agent, attach a reference. During an incident that + is the behaviour you want; it is also worth knowing you chose it. + +This document is not illustrative. It was run through the shipped parser — +`ParamSpec`, `resolve_params`, `interpolate`, `RoomSpec` — with the inputs below, +and it resolves: `visibility` defaults to `channel_public`, every placeholder +substitutes with none left over, the room comes out named `flint incident 1287`, +and the Slack channel it would create is `flint-incident-1287`. Anything in this +section that turns out to be wrong is a bug in the design, not a typo in the +YAML. + +### Instantiating it + +Today, one call, with the inputs in a JSON body: + +``` +POST /rooms/from-yaml +Content-Type: application/json + +{ + "yaml": "", + "inputs": { + "incident_id": "1287", + "severity": "sev0", + "service": "ingestion", + "summary": "Ingestion stalled; no new findings for 40 minutes", + "incident_url": "https:///incidents/1287", + "product": "flint", + "alert_channel": "", + "comms_channel": "", + "runbook_reference": "Flint runbooks", + "responder_agent": "flint-responder", + "bridge": "" + } +} +``` + +A missing required parameter, an undeclared input, or an `enum` value outside its +list fails with a 400 *before* anything is provisioned — there is no half-made +room to clean up. One trap to know: a `{word}` in the body that no parameter +declares is **left verbatim**, deliberately, so that JSON braces in a document's +content survive. A typo in a placeholder name does not error; it ships. Lint the +template (`POST /templates/validate`) before registering it, and read the +findings — the registry itself only refuses three of them. + +### The target-state template, once group templates land + +The merged format makes one room. The SOP wants the postmortem written after +recovery, and a comment in the source document observes there is no postmortem +process at all yet. A second, linked room is the better shape: the war room +closes when the incident does, and the postmortem room outlives it, holding the +review and the actions. + +That needs `origin/work/group-templates`. In its format: + +```yaml +version: 0 + +params: + incident_id: { type: string } + severity: { type: enum, enum: [sev0, sev1, sev2] } + service: { type: string } + summary: { type: string } + product: { type: string } + bridge: { type: string } + responder_agent: { type: string } + +group: + name: "{product} incident {incident_id}" + description: "War room and postmortem for {product} incident {incident_id}" + color: "#dc2626" + +rooms: + - name: "{product} incident {incident_id}" + description: "{severity} · {service} · {summary}" + bridge: "{bridge}" + channel_type: channel_public + agents: ["{responder_agent}"] + aliases: + "{responder_agent}": responder + instructions: | + ...as above... + + - name: "{product} incident {incident_id} postmortem" + description: "Postmortem and review actions for {product} incident {incident_id}" + bridge: "{bridge}" + channel_type: channel_public + agents: ["{responder_agent}"] + aliases: + "{responder_agent}": scribe + instructions: | + The incident is over. This room exists to produce the write-up and to + track the actions out of it. Draft from the war room's timeline; a human + owns the document. Blameless — name systems and decisions, never people. + +links: + - from: "{product} incident {incident_id}" + to: "{product} incident {incident_id} postmortem" + label: postmortem + - from: "{product} incident {incident_id} postmortem" + to: "{product} incident {incident_id}" + label: incident +``` + +Three things that branch buys and `main` cannot express: + +- **The group** — both rooms filed together, so an incident is one thing in the + room tree rather than two rooms that happen to share a name prefix. +- **The links** — the postmortem room points back at the war room and vice + versa, both directions, because a link is one-way. +- **The aliases** — `@responder` in the war room and `@scribe` in the postmortem + room, both the same agent. This is the per-room identity the responder needs, + and it is worth dwelling on: a rotating group should address a *function*, not + an agent's name. Aliases also require interpolation into a dict *key*, which + only that branch supports. + +### What a template still cannot set + +Measured against what a room can hold, the merged template reaches +`name`, `description`, `instructions`, `bridge`, `channel_type`, the two +visibilities, `agents`, `users`, `roles`, `references` and `docs`. It cannot +reach: + +| Cannot set | Why it matters here | +| --- | --- | +| `aliases` | No `@responder` handle; responders must know the agent's real name. On the group branch. | +| `linked_rooms` | The war room cannot point at the alert hub, the stakeholder channel or the postmortem room. On the group branch. | +| `group_id` | Incidents cannot be filed under a product's group. On the group branch. | +| `join_event_listeners` | The responder cannot greet an arriving responder and orient them — the single highest-value automation in a war room, and it is unreachable. | +| `internal_only` | Not needed here (war rooms are bridged), but the field's documentation is actively misleading. | +| `package_ids` | No packaged tooling attached at creation. | +| room metadata | Nowhere structured to record severity or the incident id; both live in prose. | + +`join_event_listeners` is the one to feel bad about. A war room's worst moment is +the fourth person arriving twenty minutes in and asking "what's the state?" — a +question the responder agent could answer automatically the instant they join, +and cannot, because the template cannot opt it into join events. From 091e9415974bdcbca157fbf3a54f38ef5cf515b2 Mon Sep 17 00:00:00 2001 From: Christian McDermott Date: Fri, 11 Sep 2026 15:31:57 -0400 Subject: [PATCH 3/6] docs(incident-response): the shared responder agent (CHOO-2721) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reads the flint-tracker setup off the live instance rather than assuming it, then says which parts a rotating on-call group should copy and which six break. States the recommendation: one shared agent per product, owned by a dedicated non-admin service user, auto-session on shared infrastructure, open addressing, never run from a laptop. User-agnostic means owned by a non-person, not owned by nobody — an ownerless agent cannot attach or create resources at all. Argues roles out: a lease is held per agent globally, so a shared agent can hold one in one incident at a time, and humans cannot hold roles. Co-Authored-By: Claude Opus 5 (1M context) --- docs/old/incident-response-sop.md | 273 ++++++++++++++++++++++++++++++ 1 file changed, 273 insertions(+) diff --git a/docs/old/incident-response-sop.md b/docs/old/incident-response-sop.md index 768ede9da..a19b3770e 100644 --- a/docs/old/incident-response-sop.md +++ b/docs/old/incident-response-sop.md @@ -663,3 +663,276 @@ reach: the fourth person arriving twenty minutes in and asking "what's the state?" — a question the responder agent could answer automatically the instant they join, and cannot, because the template cannot opt it into join events. + +## The responder agent + +The rotation is the whole problem. Six engineers take the pager in turn; the +agent has to be the same agent for all of them, reachable by whoever is on duty, +and not degraded by the fact that the person who set it up is on holiday. + +### What the SOP needs it to do + +Modest, deliberately. In the war room: + +- **Answer lookups.** Who owns this service. What the runbook says. What the + escalation ladder is. These are the questions that cost minutes at 03:00 and + they are all document reads. +- **Draft the situation report** in the right shape when nudged, from what the + room has said, for a human to check and post onward. +- **Keep the timeline**, so the postmortem is written from a record rather than + from memory. +- **Orient arrivals** — say what is known so far when someone joins. + +Note what is absent: it does not page, does not set severity, does not decide, +and does not act on production. The SOP grants *humans* the authority to roll +back and to push emergency fixes; extending that to a shared agent that six +people can address and nobody can attribute would be the single worst decision +available here. See [The rule that makes it safe](#the-rule-that-makes-it-safe). + +### What `flint-tracker` actually is + +Read from the live instance rather than assumed, because it is the prior art and +being wrong about it would poison the recommendation: + +- **Name: `flint-tracker`.** No owner suffix. Every other Claude Code agent on + the instance registered through Switch Console carries one — + `claude-code..`. This one reads as a service, and that is not + cosmetic: the name is the routing key for everything. Mentions, the Slack user + group the bridge mints, room aliases and `target_names` all resolve `name`, + and `display_name` routes nothing at all. +- **Owner: the deployment's `Admin` account.** Not a person. +- **`connection_model: auto_session`, `channels_enabled: true`**, with a working + directory on a shared always-on host rather than on anyone's laptop. Something + runs there continuously, watching for the agent to be addressed and spawning a + session on demand. +- **`addressing_policy: null`** — wide open. Anyone in any room it is in can + address it. +- **Six room memberships across three platforms** — Slack, Discord and + Mattermost. It is designed to be invited around, not to live in one room. +- **A description written at the reader**, ending "Invite it into any room and + ask what was decided, what changed, or who owns something." It tells a + stranger what to do with it. + +That is a coherent design and most of it is exactly right for a responder. + +### What carries over + +**The name.** `flint-responder`, not `claude-code.oncall.`. It is the +handle six people will type under pressure and it must not encode whose agent it +is. + +**Shared infrastructure, not a laptop.** This is the load-bearing one. An +`auto_session` agent is brought online by a watcher process; put that watcher on +an always-on host and the agent is online regardless of who is on duty, whether +their machine is asleep, or whether they have ever installed Switch Console. +A responder that only works when a particular laptop is open is not a responder. + +**An open addressing policy.** A rotating group cannot be enumerated, so the +policy cannot enumerate it. Open within the rooms it is in is the correct +setting, and it is what `flint-tracker` runs. + +**Membership by invitation.** The agent belongs to rooms, not to a room. A war +room is created and the agent is added; nothing about the agent changes per +incident. + +**A description that tells a stranger what to ask.** Half the value of a war-room +agent is discovered by someone who has never used it, mid-incident, from the +member list. + +### What breaks + +Six things, in rough order of how much they will hurt. + +**1. Admin ownership hands the agent the whole deployment.** An agent inherits +*exactly* its owner's permissions — the authorization module says so in its +opening lines — and `User.role == "admin"` is a global bypass on every read, +write and delete. So an Admin-owned agent can modify or delete any reference, +document, package or room in the tenant. For an agent that reads Slack and +summarises, that is an over-grant you can live with. For a responder that runs +during an incident, with tool access, addressed by six people under time +pressure, it is not: the moment its blast radius is widest is exactly the moment +it is unbounded. **Do not copy this part.** + +**2. There is nothing good to own it instead.** Switch has exactly one +shared-owner construct: the synthetic bootstrap account that owns every agent +registered with the deployment-wide token. It is deliberately non-admin — right +— and it is password-less and cannot be logged into — fatal. Nobody can manage +its agents, and nobody can ever reveal their credentials, because credential +reveal is strict owner equality with no admin bypass. So the correct answer, +"own it with a non-person account that is not an admin", requires a service user +that someone can actually authenticate as, and there is no supported way to make +one. [Gaps](#gaps) G11. + +Note also that "user-agnostic" cannot mean *ownerless*. An agent with +`owner_id IS NULL` cannot create a reference, attach one, list references, or +attach resources when creating a room — every one of those paths resolves the +agent to its owner and fails loudly without one. It cannot even edit itself over +MCP, because that guard compares two `None`s and refuses. Ownerless is a broken +agent, not a neutral one. **User-agnostic means owned by a non-person, not owned +by nobody.** + +And ownership is permanent: `owner_id` is set at registration and there is no +endpoint anywhere that changes it. Registering the responder under a person "just +for now" means it is theirs until someone runs an `UPDATE`. + +**3. The default addressing policy locks the rotation out, and the UI that fixes +it breaks it.** Every agent registered through any HTTP path is created +owner-only with an empty allowed-agents list. The `register_agent` function takes +an `owner_only=False` parameter, but no wire path passes it — the sole caller is +the server-side connector registration, whose comment is worth quoting because it +is this design's precedent: + +> A server-side connector agent is a service the deployment offers everyone, not +> one person's assistant; it is owned by whoever holds the registration token +> only in the bookkeeping sense. Owner-only would make it answer to that account +> alone. + +So the responder is born locked and must be widened afterwards through +`PUT /agents/{id}/addressing-policy`. And here is the landmine: the gateway's +React policy editor models only the four id-shaped dimensions and drops the +symbolic `owner` and `owner_agents` rules when it saves. Open an owner-only agent +in the dashboard, change anything, save — and the policy becomes one that admits +nobody. The agent then answers every responder with "You're not permitted to +direct messages to me in this room." Mid-incident, that reads as an outage. +Switch Console's editor handles the symbolic rules correctly; the gateway's does +not. [Gaps](#gaps) G12. + +**4. The offline nudge wakes the wrong person.** When an `auto_session` agent is +addressed in a room where nothing can start it, Switch posts on its behalf. The +message names the *owner*: + +> `@owner` — I'm not online in this room, and `@asker` needs me. Open Switch +> Console to bring me online here. + +and, when there is no owner account on that platform to mention: + +> I'm not online in this room. **My owner needs to open Switch Console** to bring +> me online here. + +The code's own comment explains the reasoning — "the fix is for the OWNER to open +it, and nobody else in the room can act" — which is sound for a personal agent +and exactly wrong for a shared one. At 03:00 the war room will either name a +service account nobody watches, or a dead end. What it should name is whoever is +on call. [Gaps](#gaps) G13. + +Running the watcher on an always-on host makes this rare rather than fixing it. + +**5. One credential, no rotation, no per-holder revocation.** One agent has +exactly one API key row. There is no rotation endpoint: the only way to change +the key is re-registration with overwrite, which deletes the old row, so every +holder breaks at once. Reveal is restricted to the owning user with no admin +bypass. And the token is a bearer credential in a plaintext file in the agent's +working directory — the repository's own documentation calls that a known +exposure. + +The practical consequence is a rule rather than a fix: **the responder's +credential lives in exactly one place, on the shared host, and is never +distributed to responders.** Handing it to six laptops means six copies of a +token nobody can individually revoke, on machines that leave with their owners. +[Gaps](#gaps) G14. + +This also settles a mechanical question. Two people *can* run sessions as the +same agent — identity is per directory, not per machine, and an agent may hold up +to 32 connections. But at most one session of an agent may act in a given room, +and `connect_to_room` always takes over: the newcomer wins, the incumbent is +disconnected from that room and told it lost, and whatever it was doing there +stops. Two responders each starting a session during one incident would evict +each other in turn. One process, on one host, is the only sane operating mode. + +**6. Nothing records which human drove it.** No actor is stored on connections, +sessions, runtime state, role leases or messages; a message is attributed to the +agent, not to whoever prompted it. For most agents that is a shrug. For incident +response it is not, because the postmortem's second question is always "who did +what, when". [Gaps](#gaps) G15. + +There is a mitigation, and it is a design rule rather than a feature — see below. + +### Roles are the wrong tool for the on-call rotation + +Room roles look purpose-built for this: named, assumable instruction bundles; +`@role` reaches whoever currently holds it; an exclusive role admits one holder +and auto-releases about six seconds after that holder dies. "Address whoever is +currently the incident commander" is precisely the sentence roles exist for. + +They still do not work here, for three reasons. + +**A lease is held per agent, globally.** The lease table is unique on the agent, +not on the session and not on the room. One shared responder can therefore hold +one role, in one room, across the entire instance. Two concurrent incidents and +it can be the scribe in only one of them. Worse, two sessions of the same agent +assuming the same role is treated as an idempotent re-assume — the second simply +overwrites the first's session pointer — so roles provide no arbitration at all +between two people running the shared agent, which is the one thing you might +have hoped they would provide. [Gaps](#gaps) G16. + +**Humans cannot hold roles.** Assuming a role is an agent operation. The incident +commander is a person, so the role cannot be theirs. + +**There is no eligibility control.** The role model carries an `eligibility` +field that is declared, documented as a forward-looking hook, and read by +nothing. Any room member may assume any role. "Only the on-call primary may take +incident commander" is not expressible. [Gaps](#gaps) G17. + +Where roles *do* work is between distinct agents. If responders bring their own +coding agents into the war room — which they will, because that is how anyone +investigates — then an exclusive `scribe` role is genuinely good: one holder at a +time, real handoff by release-and-assume, automatic release within seconds if +that engineer's session dies. That is why the template defines the role and +assigns it to nobody. + +So: **incident commander and scribe stay human conventions, written in the room's +instructions. The `scribe` role exists for a responder's own agent to pick up, +not for the shared responder.** + +### The recommendation + +**One shared responder agent per product, owned by a dedicated non-admin service +user, running `auto_session` on shared always-on infrastructure, with an open +addressing policy, invited into each war room by the template, and never run from +an engineer's machine.** + +Concretely, on top of what `flint-tracker` already gets right: + +| Setting | Value | Why | +| --- | --- | --- | +| `name` | `-responder` | The routing key. No person in it. | +| owner | a dedicated service user, **not** an admin | The agent inherits its owner's permissions exactly. | +| `connection_model` | `auto_session` | Comes online when addressed; nobody has to remember to start it. | +| watcher | one, on an always-on host | Online regardless of whose turn it is. | +| credential | one copy, on that host | Cannot be revoked per holder, so do not spread it. | +| addressing policy | open | A rotation cannot be enumerated. | +| room membership | per incident, via the template | Nothing about the agent changes per incident. | +| roles held | none | A lease is per agent; holding one breaks the second concurrent incident. | + +Two alternatives, and why not: + +- **One responder agent per engineer.** Real per-human attribution, real role + arbitration, and each agent already exists in some form. But it is six + registrations, six addressing policies and six credentials to keep consistent, + it churns on every rotation change, and each agent is still personally owned — + so the day someone leaves, their responder leaves with them. It solves + attribution by giving up shared identity, which is the thing that was asked + for. +- **Own the shared agent with the Admin account, like `flint-tracker`.** One + fewer problem today, in exchange for an agent with unbounded authority over + every room and resource in the deployment, addressable by anyone, during the + worst hour of the quarter. If G11 cannot be closed before the first incident, + this is the compromise to take *knowingly and temporarily* — and the mitigation + is that the agent has no production access and no write path outside its rooms. + +### The rule that makes it safe + +Because nothing records which human drove the agent, the room transcript has to +carry the attribution instead. That is achievable, but only if the agent is +constrained: + +> **The responder takes no consequential action that a human did not ask for, in +> the room, in writing.** Everything it does is either a read, or a draft posted +> back to the room for a human to act on. It never posts to the stakeholder +> channel, never touches the incident record, and never runs anything against +> production. + +Under that rule the room *is* the audit log: every action the agent took has a +message above it from the person who asked. Relax the rule and the attribution +hole in G15 becomes a real one. The rule belongs in the room instructions, where +the template puts it, and in the agent's own configuration. From 2557a989d3851fe8e83819c37be885d364ac21d0 Mon Sep 17 00:00:00 2001 From: Christian McDermott Date: Fri, 11 Sep 2026 15:34:43 -0400 Subject: [PATCH 4/6] docs(incident-response): the gap list, and the order to close it in (CHOO-2721) Eighteen gaps grouped by what they block, each with a proposed ticket: knowing who is on call, making the template usable, getting the room made at all, the shared agent, and closing the incident out. Ends with what is usable on day one with no Switch change, and the order to remove the manual steps in. Two small template changes and a service account buy most of it; resolving a platform group to room members is the one the SOP has been asking for. Co-Authored-By: Claude Opus 5 (1M context) --- docs/old/incident-response-sop.md | 307 +++++++++++++++++++++++++++++- 1 file changed, 304 insertions(+), 3 deletions(-) diff --git a/docs/old/incident-response-sop.md b/docs/old/incident-response-sop.md index a19b3770e..47295f885 100644 --- a/docs/old/incident-response-sop.md +++ b/docs/old/incident-response-sop.md @@ -13,6 +13,14 @@ Written against `main` at `514d5ba4` (the template registry). Every claim about current behaviour was checked against the code, and the file it lives in is cited so a reader can confirm it rather than trust it. +- [Scope](#scope) +- [The SOP, and the one place Switch appears in it](#the-sop-and-the-one-place-switch-appears-in-it) +- [Mapping the SOP onto rooms](#mapping-the-sop-onto-rooms) +- [The war-room template](#the-war-room-template) +- [The responder agent](#the-responder-agent) +- [Gaps](#gaps) +- [What to build first](#what-to-build-first) + ## Scope The subject is a specific SOP: a lightweight, post-launch, business-hours @@ -135,9 +143,9 @@ Everything that would otherwise fragment the war room. In particular: message that asked for them. Switch threads bridge natively to Mattermost and to Telegram forum topics. On a -Slack-bridged room — which this is — a threaded reply shows in the channel as a -reply count under the original post, so **anything the room must not miss goes at -the root**. That is not a Switch limitation to work around; it is a rule for +Slack-bridged room — which a war room under this SOP is — a threaded reply shows +in the channel as a reply count under the original post, so **anything the room +must not miss goes at the root**. That is not a Switch limitation to work around; it is a rule for whoever writes in the room, agent or human. The responder agent's standing instructions should say so, and the template's `instructions` field is where that lives. @@ -936,3 +944,296 @@ Under that rule the room *is* the audit log: every action the agent took has a message above it from the person who asked. Relax the rule and the attribution hole in G15 becomes a real one. The rule belongs in the room instructions, where the template puts it, and in the agent's own configuration. + +## Gaps + +Eighteen, grouped by what they block. Each names what is missing, why it matters +for this SOP specifically, and a ticket to file. Sizes are rough: **S** is days, +**M** is a sprint, **L** is a project. + +Read G1 first. It is the one the SOP itself has flagged and nobody has answered. + +### A. Knowing who is on call + +**G1 — Switch has no concept of duty, and cannot learn one.** +There is no rotation, schedule, team or on-call anything in the codebase. The +tenant membership role field exists but is documented as recording a value that +nothing yet reads. The SOP's own comment thread asks "how will Switch pull in +who's on-call from PagerDuty?" and answers itself with the right idea — a mention +group whose membership auto-rotates. Switch cannot consume that either: +`add_users_to_room` resolves individual usernames against the people the bridge +has already seen, and there is no call anywhere that expands a Slack user group +into its members. Note the asymmetry — Switch *creates* a Slack user group per +agent, so `@` works, but it never sets that group's membership and cannot +read anyone else's. + +Until this is closed, "invite the on-call engineers" is a human action, and the +template's invitee list is hard-coded or empty. + +> **Proposed ticket:** *Resolve a platform group to room members* — accept a +> bridge group handle wherever `user_names` is accepted, expand it through the +> platform (Slack user groups, Discord roles, Mattermost groups) at the moment +> of use, and add the members. Deliberately no schedule in Switch: the rotation +> stays in PagerDuty, which syncs the group. **M** + +> **Proposed follow-up:** *Room escalation target* — a room-level setting naming +> who to reach when an agent cannot be brought online or nobody has responded, +> resolvable to a group. Feeds G13. **S** + +### B. Making the template usable + +**G2 — The registry cannot instantiate what it stores.** +`POST /rooms/from-yaml` provisions from a document in the request body. The +registry stores documents. Nothing joins them: there is no "instantiate template +`` with these inputs". Using a registered template today means fetching its +content and posting it back, which makes the registry a filing cabinet rather +than a feature. + +> **Proposed ticket:** *Instantiate a stored template by id* — +> `POST /templates/{id}/instantiate` taking `inputs`, resolving the stored +> content and provisioning through the existing path. **S** + +**G3 — The dashboard cannot supply parameter inputs.** +The create-from-YAML page posts raw YAML with no `inputs` field, so from the UI +only a template whose every parameter has a default can be used. A template +built for per-incident particulars has no useful defaults. Parameters are +therefore reachable only from an API client — which, for a feature whose whole +point is that a human instantiates it, is the same as unreachable. + +> **Proposed ticket:** *Parameter form for template instantiation* — render the +> declared `params:` as a form (using `description`, which is currently stored +> and never displayed), collect values, post the JSON body form. **S** + +**G4 — A parameter cannot hold a list.** +`ParamSpec.type` is `string`, `number`, `boolean` or `enum`. A room's `agents:` +and `users:` are lists, so a variable-length membership cannot be parameterised +at all: `"alice,bob"` interpolates into one username that resolves to nobody. For +an incident template whose responders differ every time, this is the difference +between a template that works and one that has to be edited before each use. + +> **Proposed ticket:** *List-typed template parameters* — add a `list` parameter +> type whose whole-field substitution splices into the surrounding list rather +> than stringifying. **M** + +**G9 — A template cannot set aliases, links, group or join events.** +`RoomCreateConfig` carries `aliases`, `linked_rooms`, `group_id`, `package_ids` +and `join_event_listeners`; the template provisioner populates none of them. The +first three land with group templates. `join_event_listeners` lands nowhere, and +it is the one this SOP most wants: without it the responder cannot greet an +arriving responder and tell them the state, which is the highest-value automation +a war room has. + +> **Proposed ticket:** *Land group templates* — merge +> `origin/work/group-templates`: `group:`/`rooms:`/`links:`, per-room `aliases:`, +> and dict-key interpolation. **M** + +> **Proposed ticket:** *`join_event_listeners` in a room template* — a per-agent +> opt-in in the room spec. **S** + +**G10 — Omitting `bridge:` silently means "the default bridge", and breaks +`users:`.** +The field's own comment says to omit it for an internal-only room. That is wrong: +omitting it falls through to the instance default bridge. And the guard that +rejects `users:` on an unbridged room tests the template's own resolved bridge +id, so a template with `users:` and no `bridge:` is refused even though the room +would have been bridged. Two authoring traps in one field. + +> **Proposed ticket:** *Fix `bridge:` omission semantics in a room template* — +> resolve the default bridge before the `users:` guard, and add an explicit +> `internal_only:` key so "no channel" is stated rather than inferred. **S** + +**G-trap — a mistyped placeholder ships.** +Not a gap so much as a hazard worth writing down: a `{word}` that no parameter +declares is left verbatim, on purpose, so JSON braces in document content +survive. A typo in a placeholder name therefore does not error — it appears in +the created room. The linter catches some of this; the registry blocks only three +findings and treats the rest as advice. Lint before registering, and read the +output. + +### C. Getting the room made at all + +**G5 — No agent can instantiate a template, and no one can from a channel.** +The agent operation surface has fifty-odd operations and none of them touch +templates. The in-room command set has twenty commands and none of them creates a +room. So the responder agent cannot open a war room when asked, and an on-call +engineer in Slack cannot declare an incident from the channel they are already +in. Both are exactly the moments this feature exists for. + +> **Proposed ticket:** *`create_room_from_yaml` agent operation* — land the +> operation already written on `origin/work/group-templates`, extended to take a +> template id (needs G2). **S** + +> **Proposed ticket:** *`!declare-incident` in-room command* — instantiate a +> configured template from a bridged channel with positional inputs, and post +> the new room's link back. **M** + +**G6 — There is no inbound alert ingress.** +Nothing in Switch listens for an external event. No webhook endpoint, no +signature verification, no mapping from a payload to an action. So the SOP's +promise — "Switch will auto-create a dedicated channel on declaration" — cannot +be kept by Switch alone; something outside has to hold a credential and call the +API. That is workable and probably correct for a first version, but it should be +a decision rather than a discovery. + +> **Proposed ticket:** *Incident intake webhook* — a signed inbound endpoint that +> maps an alerting payload to a template instantiation, with the field mapping +> configured per source. Depends on G2. **L** + +**G7 — There is no scheduler.** +No cron, no timers, no deferred actions. An agent cannot ask to be woken. The +SOP's hourly and four-hourly update cadence therefore lives in an external +scheduled workflow that mentions the agent — which works, and is the right +short-term answer, but has to be created per incident and nobody will remember to +delete it. + +> **Proposed ticket:** *Scheduled room actions* — a room-scoped recurring +> trigger that posts a message or addresses an agent, created with the room and +> disposed of with it. **L** + +**G8 — There is no relay between rooms.** +Linked rooms are metadata: a pointer with a label. There is no mechanism that +mirrors a message from one room into another. The SOP wants situation reports to +land in both the alert channel and the stakeholder channel, and the source +document asks directly whether that can be automated. Today an agent can read +another room without connecting to it, but posting requires connecting, which +means leaving the war room mid-incident. That is not a workaround anyone should +adopt. + +> **Proposed ticket:** *Mirror a message to a linked room* — an operation that +> posts to a room the agent is a member of without moving its connection, +> attributed and marked as a mirror. **M** + +### D. The shared agent + +**G11 — There is no provisionable service account.** +The recommendation in this document rests on owning the responder with a +non-person, non-admin user. Switch has exactly one shared-owner construct — the +synthetic bootstrap account — and it cannot be logged into, so nobody can manage +its agents or reveal their credentials, and credential reveal has no admin +bypass. The alternatives are to own the responder with a real person (defeats the +purpose) or with the Admin account (hands it a global bypass over every room and +resource). **This is the gap the whole responder design depends on.** + +> **Proposed ticket:** *Service accounts* — a non-interactive user that can own +> agents and resources, with authentication a team can hold jointly, and no +> admin role. **M** + +> **Proposed ticket:** *Transfer agent ownership* — an owner-or-admin endpoint +> setting `owner_id`. There is none today, so an agent registered under the wrong +> account stays there. **S** + +**G12 — The gateway's addressing-policy editor silently deletes owner rules.** +The React editor models only the four id-shaped dimensions and drops the symbolic +`owner` / `owner_agents` rules on save. Since every agent is created owner-only, +opening one in the dashboard and saving any change converts it to a policy that +admits nobody — and the agent then answers every request with a refusal. Switch +Console's editor is correct. This is a live bug and it will be hit by exactly the +person trying to widen a responder's policy. + +> **Proposed ticket:** *Preserve symbolic rules in the gateway policy editor* — +> represent `owner` and `owner_agents`, round-trip them, and warn when a saved +> policy admits nobody. **S** + +**G13 — The offline nudge names the owner, not whoever can act.** +When an `auto_session` agent is addressed with nothing to start it, the room is +told to go and wake the owner. For a shared responder that is a service account +nobody watches, or a person who is not on call. The wording is right for a +personal agent and wrong for a shared one, and there is no way to override it. + +> **Proposed ticket:** *Escalation target for an offline shared agent* — when an +> agent has no personal owner, address the nudge to the room's escalation target +> (see G1's follow-up) instead of to `owner_id`. **S** + +**G14 — One credential per agent; no rotation, no per-holder revocation.** +One agent, one API key row. No rotation endpoint — the only rotation is +re-registration with overwrite, which breaks every holder simultaneously. Reveal +is strict owner equality with no admin bypass. A shared agent therefore has a +credential that cannot be issued per person, cannot be revoked per person, and +cannot be recovered by anyone but its owner. + +> **Proposed ticket:** *Per-holder agent credentials* — several named, +> independently revocable keys per agent, each attributable, with a rotation +> endpoint that does not break the others. **M** + +**G15 — Nothing records which human drove a session.** +No actor field on connections, sessions, runtime state, leases or messages. A +shared agent's actions are attributable to the agent and to nobody else. The +mitigation in this design is the rule that the agent acts only on a written +request in the room — which makes the transcript the audit log — but that is a +convention, and conventions are not enforcement. + +> **Proposed ticket:** *Record the operator behind a session* — capture an actor +> on session registration and carry it onto messages the session sends. **M** + +**G16 — A role lease is held per agent, globally.** +Unique on the agent, not on the room and not on the session. One agent can hold +one role across the whole instance, so a shared agent in two concurrent incidents +can be the scribe in only one. And two sessions of the same agent assuming the +same role is an idempotent re-assume, so roles arbitrate nothing between them. + +> **Proposed ticket:** *Scope a role lease to (agent, room)* — allow one agent to +> hold a role in each of several rooms, and decide explicitly what two sessions +> of one agent assuming one role should mean. **M** + +**G17 — Role eligibility is declared and unused; humans cannot hold roles.** +`RoomRole.eligibility` exists, is documented as a forward-looking hook, and is +read by nothing — any room member may assume any role. And roles are assumable +only by agents, so "incident commander" cannot be a role at all. + +> **Proposed ticket:** *Enforce role eligibility* — implement the declared field +> so a role can be restricted. **S** + +> **Proposed ticket:** *Human-holdable roles* — let a person claim a room role +> from the bridged channel, so `@incident-commander` reaches a human. **L** + +### E. Closing the incident out + +**G18 — There is no transcript export.** +The postmortem is written from the room, but there is no endpoint that produces a +room's history: the gateway exposes a room's *configuration* as YAML and nothing +else, and reading messages is an agent-only operation. In practice the responder +agent can page back through the room and post a timeline as an attachment, which +is good enough — so this is the cheapest gap on the list and the least urgent. + +> **Proposed ticket:** *Export a room transcript* — a downloadable, paginated +> history export for a room a user can read. **S** + +## What to build first + +Nothing on that list blocks a first incident. Two things are worth being explicit +about: + +**Usable on day one, with no Switch change at all.** Write the template, register +it, and have one person instantiate it through the API when an incident is +declared, pasting the incident's particulars as inputs. Invite responders by +hand. Put the update cadence in a scheduled Slack workflow that mentions the +responder. Register the responder agent, widen its addressing policy through the +API — not the dashboard, see G12 — and run its watcher on an always-on host. That +is a working SOP on Switch, with two manual steps. + +**The order to remove the manual steps in**, by value per unit of work: + +1. **G2 + G3 — instantiate a stored template, with a form.** Two small changes + that together turn the registry from a filing cabinet into the feature. Until + these land, every other template improvement is invisible to the people who + would use it. Start here. +2. **G11 — a service account.** Small in scope, and the recommendation for the + responder agent is unsound without it. Every day it is missing is a day the + responder is either one person's agent or an admin. +3. **G12 — the policy editor bug.** A few hours' work, and it will otherwise be + discovered by someone widening the responder's policy during an incident. +4. **G1 — resolve a platform group to room members.** The largest single + reduction in manual work: it turns "invite the on-call engineers" from a + human step into a template line, and it is the question the SOP has been + asking. Deliberately without building a rotation in Switch. +5. **G5 — declare an incident from the channel.** Once the template instantiates + cleanly, letting an engineer trigger it from Slack removes the last manual + step in the critical path. +6. **G9 — group templates, and join-event listeners.** The postmortem room, the + links, the `@responder` alias, and the ability to greet an arrival. All + quality, none of it blocking. +7. Everything else, as it starts to hurt. + +The honest summary: **Switch can host this SOP today, badly, with two manual +steps and one unsafe compromise on agent ownership. Items 1 to 3 make it +respectable, and they are small. Item 4 is the one the team actually asked for.** From 1df3d09da8aac1eac5fdf309c53c1116c2dc29bc Mon Sep 17 00:00:00 2001 From: Christian McDermott Date: Fri, 11 Sep 2026 15:59:52 -0400 Subject: [PATCH 5/6] docs(incident-response): correct what an adversarial re-read got wrong (CHOO-2721) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fact-check against the source found thirteen claims that were wrong or overstated. The substantive ones: - A template is not the only way to reach aliases, links, the group or join listeners — create_room and update_room already accept all of them. The finding is that a template is strictly less capable than the room creation it wraps, which makes the gap a cheap pass-through rather than a feature. - An agent can already open a war room; what it cannot do is open one from a template. - The gateway policy editor disables Save on an unmatchable rule, so it cannot brick an agent outright. The reachable damage is quieter: widening the default policy drops the owner rule. - A bootstrap-owned agent is manageable by an admin; it is its credential that can never be revealed. - Switch does run periodic work and does verify an inbound signed webhook on one bridge. The true claims are narrower: no scheduling primitive a room can use, and no generic alert ingress. - write_visibility: public grants write to any principal in the tenant, not just participants, and that covers archiving the room. The template now sets it private, and says why. Also fixes two quoted strings that were not faithful, and the operation and command counts. Co-Authored-By: Claude Opus 5 (1M context) --- docs/old/incident-response-sop.md | 290 +++++++++++++++++++----------- 1 file changed, 188 insertions(+), 102 deletions(-) diff --git a/docs/old/incident-response-sop.md b/docs/old/incident-response-sop.md index 47295f885..538fb9c83 100644 --- a/docs/old/incident-response-sop.md +++ b/docs/old/incident-response-sop.md @@ -10,8 +10,10 @@ rather than designed around — [Gaps](#gaps) is the part to read if you read only one section. Written against `main` at `514d5ba4` (the template registry). Every claim about -current behaviour was checked against the code, and the file it lives in is -cited so a reader can confirm it rather than trust it. +how Switch behaves today was checked against the code on that commit rather than +recalled, and where a behaviour is surprising enough to be worth confirming, the +file it lives in is named. If a claim here has since gone stale, the commit is +the thing to diff against. - [Scope](#scope) - [The SOP, and the one place Switch appears in it](#the-sop-and-the-one-place-switch-appears-in-it) @@ -142,10 +144,12 @@ Everything that would otherwise fragment the war room. In particular: - **A tool's noisy output.** Log dumps and query results belong under the message that asked for them. -Switch threads bridge natively to Mattermost and to Telegram forum topics. On a -Slack-bridged room — which a war room under this SOP is — a threaded reply shows -in the channel as a reply count under the original post, so **anything the room -must not miss goes at the root**. That is not a Switch limitation to work around; it is a rule for +Switch threads bridge to real platform threads everywhere it can — Slack replies +go out with a `thread_ts`, Discord gets a real thread, Telegram gets a forum +topic. The Slack difference is one of *rendering*: a threaded reply appears in +the channel only as a reply count under the original post rather than in the +main flow. So on a Slack-bridged room — which a war room under this SOP is — +**anything the room must not miss goes at the root**. That is not a Switch limitation to work around; it is a rule for whoever writes in the room, agent or human. The responder agent's standing instructions should say so, and the template's `instructions` field is where that lives. @@ -205,10 +209,12 @@ Two properties of that table are the design: The SOP puts situation reports on a clock: hourly at the top severity, every four hours below it. Something has to remember. -**Switch cannot.** There is no scheduler, no cron, no timer, and no deferred -action anywhere in `core/switch_core` — the only `schedule` symbols in the tree -are `asyncio` call-soon helpers inside the transport and bridge loops. An agent -in a room cannot ask to be woken in an hour. +**Switch cannot.** There is no scheduling primitive exposed to a room or an +agent: no cron, no timers, no deferred actions, nothing an agent can ask to be +woken by. Switch does run periodic work internally — connection and +runtime-state sweeps at boot, bridge-level renewal loops, `call_later` timers +that batch attachments — but none of it is reachable from a room, and none of it +is scheduling in the sense the SOP means. The SOP's own comment thread already has the answer, and it is the right one: a scheduled Slack workflow that posts into the war room mentioning the responder @@ -240,8 +246,9 @@ honest about which half exists. (required), `params:` and `version:`. `version:` is parsed, type-checked and then ignored. An unrecognised top-level key is a hard error. - Typed parameters: `string`, `number`, `boolean`, `enum`, each with an optional - `description` and `default`. There is no `required:` key — a parameter is - required exactly when it has no default. + `description`, `default` and — for `enum` — the list of permitted values. + There is no `required:` key: a parameter is required exactly when it has no + default. - `{name}` interpolation over the `room:` block. Two modes: if a field is *entirely* one placeholder, the parameter's typed value is substituted whole, which is how an `enum` can fill `channel_type:`; otherwise each placeholder is @@ -369,7 +376,7 @@ room: bridge: "{bridge}" channel_type: "{visibility}" read_visibility: public - write_visibility: public + write_visibility: private agents: ["{responder_agent}"] @@ -521,29 +528,42 @@ Notes on choices in that document that are not arbitrary: `product---incident-42`, with runs of hyphens where the punctuation was. `{product} incident {incident_id}` gives `product-incident-42`. Cosmetic, but channel names are what responders type under pressure. -- **`bridge:` is a required parameter and not omitted.** The field's own comment - says to omit it for an internal-only room, and that is wrong: omitting it - falls through to the instance default bridge. Worse, the guard that rejects - `users:` on an unbridged room tests the template's resolved bridge id, so a - template with `users:` and no `bridge:` is rejected even though the room would - in fact have been bridged. Naming the bridge explicitly sidesteps both. +- **`bridge:` is a required parameter and not omitted.** The comment on the + template's own `bridge` field says to omit it for an internal-only room, and + that is wrong: omitting it falls through to the instance default bridge — or, + with no default configured, to no bridge at all, and with a default configured + but not running, to a hard failure. Worse, the guard that rejects `users:` on + an unbridged room tests the template's resolved bridge id, so a template with + `users:` and no `bridge:` is rejected even though the room would in fact have + been bridged. Naming the bridge explicitly sidesteps all of it. [Gaps](#gaps) G10. - **The `scribe` role is defined but nobody is told to take it.** See [Roles are the wrong tool for the on-call rotation](#roles-are-the-wrong-tool-for-the-on-call-rotation) — a role is held per *agent*, globally, so the shared responder can hold it in one incident at a time. It is there for a responder's own coding agent to assume, and the template does not assume it on anyone's behalf. -- **`write_visibility: public`** means any participant's owner can restructure - the room — pull in another agent, attach a reference. During an incident that - is the behaviour you want; it is also worth knowing you chose it. +- **`write_visibility: private`, and this is the one to argue about.** Public + write on a room does not mean "participants may restructure it" — it grants + write to *any* principal in the tenant, member or not, and write on a room is + what governs attaching a reference, defining and deleting roles, updating the + room and archiving it. A war room that any agent's owner in the deployment can + archive mid-incident is not a trade worth making, so this template narrows it: + readable by everyone, restructured only by the room's owner and admins. + + The cost is real but small. Attaching a reference mid-incident becomes the + instantiating user's job. Adding agents and users is *not* affected — the + roster path is governed separately and admits existing members regardless of + visibility — so pulling another agent into the war room still works for anyone + already in it, which is the operation that actually matters under pressure. This document is not illustrative. It was run through the shipped parser — -`ParamSpec`, `resolve_params`, `interpolate`, `RoomSpec` — with the inputs below, -and it resolves: `visibility` defaults to `channel_public`, every placeholder -substitutes with none left over, the room comes out named `flint incident 1287`, -and the Slack channel it would create is `flint-incident-1287`. Anything in this -section that turns out to be wrong is a bug in the design, not a typo in the -YAML. +`ParamSpec`, `resolve_params`, `interpolate`, `RoomSpec`, and the visibility-pair +validator — with the inputs below, and it resolves: `visibility` defaults to +`channel_public`, the `public` / `private` visibility pair is accepted, every +placeholder substitutes with none left over anywhere including inside the seeded +documents, the room comes out named `flint incident 1287`, and the Slack channel +it would create is `flint-incident-1287`. Anything in this section that turns out +to be wrong is a bug in the design, not a typo in the YAML. ### Instantiating it @@ -650,27 +670,46 @@ Three things that branch buys and `main` cannot express: an agent's name. Aliases also require interpolation into a dict *key*, which only that branch supports. -### What a template still cannot set +### What a template still cannot set — and why that is the real finding Measured against what a room can hold, the merged template reaches `name`, `description`, `instructions`, `bridge`, `channel_type`, the two visibilities, `agents`, `users`, `roles`, `references` and `docs`. It cannot reach: -| Cannot set | Why it matters here | -| --- | --- | -| `aliases` | No `@responder` handle; responders must know the agent's real name. On the group branch. | -| `linked_rooms` | The war room cannot point at the alert hub, the stakeholder channel or the postmortem room. On the group branch. | -| `group_id` | Incidents cannot be filed under a product's group. On the group branch. | -| `join_event_listeners` | The responder cannot greet an arriving responder and orient them — the single highest-value automation in a war room, and it is unreachable. | -| `internal_only` | Not needed here (war rooms are bridged), but the field's documentation is actively misleading. | -| `package_ids` | No packaged tooling attached at creation. | -| room metadata | Nowhere structured to record severity or the incident id; both live in prose. | - -`join_event_listeners` is the one to feel bad about. A war room's worst moment is -the fourth person arriving twenty minutes in and asking "what's the state?" — a -question the responder agent could answer automatically the instant they join, -and cannot, because the template cannot opt it into join events. +| Cannot set from a template | Why it matters here | Reachable another way? | +| --- | --- | --- | +| `aliases` | No `@responder` handle; responders must know the agent's real name. | **Yes** — `create_room`, `update_room`, `PATCH`, or `!set-alias` in the room. | +| `linked_rooms` | The war room cannot point at the alert hub, the stakeholder channel or the postmortem room. | **Yes** — `create_room`, or `link_rooms` after. | +| `group_id` | Incidents cannot be filed under a product's group. | **Yes** — `group_name` on `create_room`. | +| `join_event_listeners` | Without it the responder never learns that someone joined, so it cannot greet an arrival and tell them the state. | **Yes** — `create_room` or `update_room`. | +| `internal_only` | Not needed here — war rooms are bridged. | Yes, on `create_room`. | +| `package_ids` | No packaged tooling attached at creation. | Yes, on `create_room`. | +| room metadata | Nowhere structured to record severity or the incident id; both live in prose. | No. | + +That last column is the finding, and it is more useful than the list itself: +**a room template is strictly less capable than the room creation it wraps.** +Every one of those fields is already accepted by `create_room` — the HTTP +endpoint, and the agent operation of the same name — and most can be set +afterwards with `update_room`. The template format simply does not carry them. + +Two consequences worth acting on: + +- **The gap is a format gap, not a platform gap**, so it is cheap. Nothing needs + designing; the provisioner needs to pass through fields the config object + already has. +- **Until it closes, an agent that creates the war room by calling `create_room` + directly can do everything the template can and more** — aliases, links, the + group, and join events included. That is a genuine fork in the road: the + template is the reusable, reviewable, version-controlled artifact, and + `create_room` is the capable one. Choosing the template means accepting a + follow-up call to set what it could not, or accepting that the war room has no + `@responder` alias and cannot greet arrivals. + +The greeting is the one worth wanting. A war room's worst recurring moment is the +fourth person arriving twenty minutes in and asking "what's the state?" — a +question the responder could answer the instant they join, if something opted it +into join events. A template cannot; one extra `update_room` call can. ## The responder agent @@ -763,13 +802,22 @@ it is unbounded. **Do not copy this part.** **2. There is nothing good to own it instead.** Switch has exactly one shared-owner construct: the synthetic bootstrap account that owns every agent -registered with the deployment-wide token. It is deliberately non-admin — right -— and it is password-less and cannot be logged into — fatal. Nobody can manage -its agents, and nobody can ever reveal their credentials, because credential -reveal is strict owner equality with no admin bypass. So the correct answer, -"own it with a non-person account that is not an admin", requires a service user -that someone can actually authenticate as, and there is no supported way to make -one. [Gaps](#gaps) G11. +registered with the deployment-wide token. It is deliberately non-admin, which is +right. It is also password-less, so on a password deployment nobody can sign in +as it. (On an OIDC deployment this is softer than it sounds — an identity +provider that asserts that address would link to the existing account — but that +is a deployment accident, not a supported way to hold a shared identity.) + +The consequence is narrower than "unmanageable" and still bad. An admin *can* +manage a bootstrap-owned agent: edit its options, set its addressing policy, +delete it. What nobody can do is **reveal its credential**, because credential +reveal is the one check in the system with strict owner equality and no admin +bypass. So a bootstrap-owned responder is an agent whose token can never be +recovered — you can rotate it by re-registering, and you can never read it. + +So the correct answer, "own it with a non-person account that is not an admin", +needs a service user someone can actually authenticate as, and there is no +supported way to make one. [Gaps](#gaps) G11. Note also that "user-agnostic" cannot mean *ownerless*. An agent with `owner_id IS NULL` cannot create a reference, attach one, list references, or @@ -797,13 +845,22 @@ is this design's precedent: So the responder is born locked and must be widened afterwards through `PUT /agents/{id}/addressing-policy`. And here is the landmine: the gateway's -React policy editor models only the four id-shaped dimensions and drops the -symbolic `owner` and `owner_agents` rules when it saves. Open an owner-only agent -in the dashboard, change anything, save — and the policy becomes one that admits -nobody. The agent then answers every responder with "You're not permitted to -direct messages to me in this room." Mid-incident, that reads as an outage. -Switch Console's editor handles the symbolic rules correctly; the gateway's does -not. [Gaps](#gaps) G12. +React policy editor models only the four id-shaped dimensions, so the symbolic +`owner` and `owner_agents` rules are dropped from any rule it saves. + +The dashboard does catch the worst case — a rule with every sender dimension +empty is flagged "This rule can never match" and Save is disabled — so you +cannot brick the agent outright. What you *can* do is the ordinary thing: open +the default owner-only policy, add an agent to the allowed list, save, and +silently lose `owner: true` in the process. The policy that comes back admits +that one agent and locks out the human owner, who then gets + +> You're not permitted to direct messages to me in this room — my operator has +> restricted who can address me here. + +from their own agent. Mid-incident that reads as an outage. Switch Console's +editor round-trips the symbolic rules correctly; the gateway's does not. +[Gaps](#gaps) G12. **4. The offline nudge wakes the wrong person.** When an `auto_session` agent is addressed in a room where nothing can start it, Switch posts on its behalf. The @@ -812,10 +869,12 @@ message names the *owner*: > `@owner` — I'm not online in this room, and `@asker` needs me. Open Switch > Console to bring me online here. -and, when there is no owner account on that platform to mention: +and, when the owner has no account on that platform to mention: + +> I'm not online in this room, and `@asker` needs me. **My owner needs to open +> Switch Console** to bring me online here. -> I'm not online in this room. **My owner needs to open Switch Console** to bring -> me online here. +Both may carry a terminal command underneath. The code's own comment explains the reasoning — "the fix is for the OWNER to open it, and nobody else in the room can act" — which is sound for a personal agent @@ -842,10 +901,12 @@ token nobody can individually revoke, on machines that leave with their owners. This also settles a mechanical question. Two people *can* run sessions as the same agent — identity is per directory, not per machine, and an agent may hold up to 32 connections. But at most one session of an agent may act in a given room, -and `connect_to_room` always takes over: the newcomer wins, the incumbent is -disconnected from that room and told it lost, and whatever it was doing there -stops. Two responders each starting a session during one incident would evict -each other in turn. One process, on one host, is the only sane operating mode. +and `connect_to_room` always takes over: the newcomer wins and is warned what it +displaced, and the incumbent stops receiving that room's events. The incumbent's +notification is a bare subscription change with no reason attached, so in +practice one responder's session goes quiet without explaining why. Two +responders each starting a session during one incident would evict each other in +turn. One process, on one host, is the only sane operating mode. **6. Nothing records which human drove it.** No actor is stored on connections, sessions, runtime state, role leases or messages; a message is attributed to the @@ -1000,6 +1061,13 @@ built for per-incident particulars has no useful defaults. Parameters are therefore reachable only from an API client — which, for a feature whose whole point is that a human instantiates it, is the same as unreachable. +One caveat to check before acting on this: the template linter's own comments +refer to "a document the Console wizard renders happily", implying a client that +does collect inputs. Nothing under `console/` on this branch posts to +`/rooms/from-yaml` or renders a `params:` block, so either that wizard is +unmerged or it lives somewhere this tree cannot see. If it ships, this gap +narrows to "the gateway cannot", which is much less serious. + > **Proposed ticket:** *Parameter form for template instantiation* — render the > declared `params:` as a form (using `description`, which is currently stored > and never displayed), collect values, post the JSON body form. **S** @@ -1007,21 +1075,23 @@ point is that a human instantiates it, is the same as unreachable. **G4 — A parameter cannot hold a list.** `ParamSpec.type` is `string`, `number`, `boolean` or `enum`. A room's `agents:` and `users:` are lists, so a variable-length membership cannot be parameterised -at all: `"alice,bob"` interpolates into one username that resolves to nobody. For -an incident template whose responders differ every time, this is the difference -between a template that works and one that has to be edited before each use. +at all: `"alice,bob"` interpolates into one entry. In `users:` that entry +silently resolves to nobody and is reported as unresolved; in `agents:` it is a +hard `Unknown agents:` failure that aborts provisioning. For an incident template +whose responders differ every time, this is the difference between a template +that works and one that has to be edited before each use. > **Proposed ticket:** *List-typed template parameters* — add a `list` parameter > type whose whole-field substitution splices into the surrounding list rather > than stringifying. **M** -**G9 — A template cannot set aliases, links, group or join events.** +**G9 — A room template is strictly less capable than the room creation it wraps.** `RoomCreateConfig` carries `aliases`, `linked_rooms`, `group_id`, `package_ids` -and `join_event_listeners`; the template provisioner populates none of them. The -first three land with group templates. `join_event_listeners` lands nowhere, and -it is the one this SOP most wants: without it the responder cannot greet an -arriving responder and tell them the state, which is the highest-value automation -a war room has. +and `join_event_listeners`, and `create_room` accepts every one of them. The +template provisioner populates none. The first three arrive with group templates; +`join_event_listeners` arrives nowhere. Nothing here needs designing — the fields +already exist on the config object and are already validated — so this is a +pass-through, not a feature. > **Proposed ticket:** *Land group templates* — merge > `origin/work/group-templates`: `group:`/`rooms:`/`links:`, per-room `aliases:`, @@ -1032,11 +1102,14 @@ a war room has. **G10 — Omitting `bridge:` silently means "the default bridge", and breaks `users:`.** -The field's own comment says to omit it for an internal-only room. That is wrong: -omitting it falls through to the instance default bridge. And the guard that -rejects `users:` on an unbridged room tests the template's own resolved bridge -id, so a template with `users:` and no `bridge:` is refused even though the room -would have been bridged. Two authoring traps in one field. +The comment on the template's `bridge` field says to omit it for an internal-only +room. That is wrong: omitting it falls through to the instance default bridge — +and, less obviously, to no bridge when there is no default, or to a hard failure +when the default is configured but not running. (The `internal_only` field's own +documentation is accurate and says exactly this; the misleading comment is on the +template side.) Compounding it, the guard that rejects `users:` on an unbridged +room tests the template's own resolved bridge id, so a template with `users:` and +no `bridge:` is refused even though the room would have been bridged. > **Proposed ticket:** *Fix `bridge:` omission semantics in a room template* — > resolve the default bridge before the `users:` guard, and add an explicit @@ -1053,11 +1126,16 @@ output. ### C. Getting the room made at all **G5 — No agent can instantiate a template, and no one can from a channel.** -The agent operation surface has fifty-odd operations and none of them touch -templates. The in-room command set has twenty commands and none of them creates a -room. So the responder agent cannot open a war room when asked, and an on-call -engineer in Slack cannot declare an incident from the channel they are already -in. Both are exactly the moments this feature exists for. +The agent operation surface has 46 operations and not one of them touches +templates. The in-room command set has 21 and not one creates a room. + +Be precise about what this does and does not mean. An agent *can* open a war room +— `create_room` is an agent operation and takes agents, users, roles, references, +links, a group, aliases and join listeners. What it cannot do is open the room +*from the reviewed, version-controlled template*, which is the whole point of +having one. And an on-call engineer in the alert channel cannot declare an +incident from the channel they are already in; they have to leave Slack for an +API client or the dashboard, at the moment they least want to. > **Proposed ticket:** *`create_room_from_yaml` agent operation* — land the > operation already written on `origin/work/group-templates`, extended to take a @@ -1067,20 +1145,24 @@ in. Both are exactly the moments this feature exists for. > configured template from a bridged channel with positional inputs, and post > the new room's link back. **M** -**G6 — There is no inbound alert ingress.** -Nothing in Switch listens for an external event. No webhook endpoint, no -signature verification, no mapping from a payload to an action. So the SOP's -promise — "Switch will auto-create a dedicated channel on declaration" — cannot -be kept by Switch alone; something outside has to hold a credential and call the -API. That is workable and probably correct for a first version, but it should be -a decision rather than a discovery. +**G6 — There is no generic alert ingress.** +Switch does listen for inbound HTTP from a platform — the Teams bridge runs its +own endpoint and verifies the caller's signed token against the published keys, +so the machinery for authenticating an inbound webhook exists and is proven. +What does not exist is anything generic: no endpoint that accepts a third-party +alert payload and maps it to a Switch action. So the SOP's promise — "Switch will +auto-create a dedicated channel on declaration" — cannot be kept by Switch alone; +something outside has to hold a credential and call the API. That is workable and +probably correct for a first version, but it should be a decision rather than a +discovery. > **Proposed ticket:** *Incident intake webhook* — a signed inbound endpoint that > maps an alerting payload to a template instantiation, with the field mapping > configured per source. Depends on G2. **L** -**G7 — There is no scheduler.** -No cron, no timers, no deferred actions. An agent cannot ask to be woken. The +**G7 — There is no scheduling primitive a room or an agent can use.** +Switch runs periodic work internally — sweeps, renewals, batching timers — but +none of it is reachable from a room, and an agent cannot ask to be woken. The SOP's hourly and four-hourly update cadence therefore lives in an external scheduled workflow that mentions the agent — which works, and is the right short-term answer, but has to be created per incident and nobody will remember to @@ -1108,11 +1190,13 @@ adopt. **G11 — There is no provisionable service account.** The recommendation in this document rests on owning the responder with a non-person, non-admin user. Switch has exactly one shared-owner construct — the -synthetic bootstrap account — and it cannot be logged into, so nobody can manage -its agents or reveal their credentials, and credential reveal has no admin -bypass. The alternatives are to own the responder with a real person (defeats the -purpose) or with the Admin account (hands it a global bypass over every room and -resource). **This is the gap the whole responder design depends on.** +synthetic bootstrap account — and on a password deployment nobody can sign in as +it. An admin can still *manage* its agents; what nobody can do is reveal their +credentials, because credential reveal is the one check with strict owner +equality and no admin bypass. The alternatives are to own the responder with a +real person (defeats the purpose) or with the Admin account (hands it a global +bypass over every room and resource in the tenant). **This is the gap the whole +responder design depends on.** > **Proposed ticket:** *Service accounts* — a non-interactive user that can own > agents and resources, with authentication a team can hold jointly, and no @@ -1124,11 +1208,13 @@ resource). **This is the gap the whole responder design depends on.** **G12 — The gateway's addressing-policy editor silently deletes owner rules.** The React editor models only the four id-shaped dimensions and drops the symbolic -`owner` / `owner_agents` rules on save. Since every agent is created owner-only, -opening one in the dashboard and saving any change converts it to a policy that -admits nobody — and the agent then answers every request with a refusal. Switch -Console's editor is correct. This is a live bug and it will be hit by exactly the -person trying to widen a responder's policy. +`owner` / `owner_agents` rules from any rule it saves. It does guard the extreme +case — an all-empty rule is flagged as unmatchable and Save is disabled — so the +agent cannot be bricked outright. The reachable damage is quieter: widening the +default owner-only policy by adding an allowed agent saves a policy that admits +that agent and no longer admits the owner. Switch Console's editor round-trips +the symbolic rules correctly. This is a live bug, and it sits directly on the +path of anyone widening a shared responder's policy. > **Proposed ticket:** *Preserve symbolic rules in the gateway policy editor* — > represent `owner` and `owner_agents`, round-trip them, and warn when a saved From 67a46299eb34b6dfd0559b4d8007b66ce34a6343 Mon Sep 17 00:00:00 2001 From: Christian McDermott Date: Fri, 11 Sep 2026 17:10:21 -0400 Subject: [PATCH 6/6] docs(incident-response): rebuild the design around the agent (CHOO-2721) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reworked after review: the war room is built by a responder agent calling create_room, not instantiated from a room template. The spine is now the workstream-hub pattern this repo already runs — a standing hub whose instructions carry the SOP plus a bindings block, an exclusive role that is a thin shell over the lease and @-addressing, an agent that builds one room per item with a full instruction card, and the banner protocol. An incident is a work item with a clock on it. Three things the rework changed: - A template cannot look anything up, and the most valuable value here is a lookup. An agent can go and find who is on call. - PagerDuty is reached the way Jira already is: an MCP server on the host, a reference type carrying the prose, and instance values in the hub's bindings. That closes what had been the hardest gap — Switch does not need to model a rotation, because the agent can ask. - Roles were wrongly ruled out. A lease is per agent globally, which rules out a role per war room, not a role in the hub. The hub role is exactly "reach whoever is coordinating without knowing which agent that is". Adds the Slack ingress findings (a third-party app's post does reach a room and can wake an agent; its rich blocks and its edits largely do not), and the two MCP constraints: per-machine scope, and no agent-scoped secret storage. Net: the design needs no Switch changes to run, one small change to be safe, and two more to be pleasant. Co-Authored-By: Claude Opus 5 (1M context) --- docs/old/incident-response-sop.md | 1748 ++++++++++++++--------------- 1 file changed, 871 insertions(+), 877 deletions(-) diff --git a/docs/old/incident-response-sop.md b/docs/old/incident-response-sop.md index 538fb9c83..b58f47491 100644 --- a/docs/old/incident-response-sop.md +++ b/docs/old/incident-response-sop.md @@ -1,13 +1,13 @@ # Incident response on Switch -How an on-call/incident-response SOP runs on Switch: the room shape, a reusable -room template that provisions a war room, and a responder agent that a rotating -on-call group can share without any one engineer owning it. +How an on-call/incident-response SOP runs on Switch: a standing incident hub, a +shared responder agent that builds a war room per incident, and PagerDuty reached +the same way Jira already is. This is a design, not an implementation. Nothing here has been built. Where Switch cannot do what the SOP needs, the gap is named and a ticket proposed -rather than designed around — [Gaps](#gaps) is the part to read if you read -only one section. +rather than designed around — [Gaps](#gaps) is the part to read if you read only +one section. Written against `main` at `514d5ba4` (the template registry). Every claim about how Switch behaves today was checked against the code on that commit rather than @@ -18,8 +18,9 @@ the thing to diff against. - [Scope](#scope) - [The SOP, and the one place Switch appears in it](#the-sop-and-the-one-place-switch-appears-in-it) - [Mapping the SOP onto rooms](#mapping-the-sop-onto-rooms) -- [The war-room template](#the-war-room-template) -- [The responder agent](#the-responder-agent) +- [The incident-response agent](#the-incident-response-agent) +- [Reaching PagerDuty](#reaching-pagerduty) +- [Making the agent user-agnostic](#making-the-agent-user-agnostic) - [Gaps](#gaps) - [What to build first](#what-to-build-first) @@ -29,19 +30,19 @@ The subject is a specific SOP: a lightweight, post-launch, business-hours rotation that pages through PagerDuty and coordinates in Slack. It is deliberately temporary — its own text says it will be replaced once a 24/7 rotation and real tooling exist. So the design optimises for *reuse and -disposal*, not for permanence: a template a team instantiates per incident and -throws away, not a standing structure to maintain. +disposal*: a shape a team can stand up per product and throw away, not a standing +structure to maintain. The SOP belongs to one product team. This document does not reproduce it. The -team's channel names, service owners and escalation contacts are inputs, not -content — they arrive as template parameters at instantiation time. That keeps -this document reusable across products, and keeps a public repository free of -one team's internal routing. +team's channel names, service owners and escalation contacts are configuration, +not content — they live in one bindings block, supplied per product. That keeps +this design reusable across products, and keeps a public repository free of one +team's internal routing. -**Out of scope.** Designing the PagerDuty or Datadog integrations themselves; -Switch Console's side of any of this; anything that needs code to exist before -it can be described. Where the SOP depends on such a thing, it appears in -[Gaps](#gaps). +**Out of scope.** Designing the PagerDuty or Datadog products' own +configuration; Switch Console's side of any of this; anything that needs code to +exist before it can be described. Where the SOP depends on such a thing, it +appears in [Gaps](#gaps). ## The SOP, and the one place Switch appears in it @@ -67,9 +68,8 @@ The flow, compressed: scheduled within five business days for the top severity. Read that list again and notice how little of it is Switch's. Paging, ack, -severity, the resolve, the audit trail of the incident record — all PagerDuty. -Diagnosis is Datadog and the runbooks. Switch appears at exactly one point, step -4: +severity, the resolve, the incident record — all PagerDuty. Diagnosis is Datadog +and the runbooks. Switch appears at exactly one point, step 4: > Switch will auto-create a dedicated Slack channel and invite on-call engineers > to the war room. @@ -77,132 +77,118 @@ Diagnosis is Datadog and the runbooks. Switch appears at exactly one point, step That is the whole ask, and it is worth being blunt about the size of it: **the SOP does not need Switch to run incident response. It needs Switch to manufacture a correctly-shaped, correctly-populated room in the seconds after a -declaration, and then to be useful inside it.** A design that tries to move -severity, paging or the incident record into Switch is designing a competitor to -PagerDuty that nobody asked for. +declaration, and then to be useful inside it.** A design that moves severity, +paging or the incident record into Switch is building a competitor to PagerDuty +that nobody asked for. The value Switch adds is not the channel — Slack can make a channel. It is that -the room arrives *already furnished*: the runbook attached, the service-owner -map attached, the situation-report shape attached, the responder agent already -in it and already briefed on which service is broken and how severe it is. A -human doing this by hand at 03:00 does it badly or not at all. +the room arrives *already furnished*: the runbook attached, the service-owner map +attached, the situation-report shape attached, the right people already invited +because something looked up who was on call, and a responder agent already in the +room and already briefed on which service is broken and how severe it is. A human +doing this by hand at 03:00 does it badly or not at all. ### The questions the SOP has not answered -The source document carries open comments, and three of them are load-bearing -for this design. They are not oversights in the SOP; they are places where the -SOP is waiting on Switch: +The source document carries open comments, and three are load-bearing. They are +not oversights in the SOP; they are places where it is waiting on Switch: -- **"How will Switch pull in who's on-call from PagerDuty?"** Nobody has - answered this. It is the single hardest requirement in the document, and - [Gaps](#gaps) treats it as such. +- **"How will Switch pull in who's on-call from PagerDuty?"** This turns out to + have a clean answer, and it is not the one anybody expected — see + [Reaching PagerDuty](#reaching-pagerduty). - **"Can we automate mirroring updates from the alert channel into the - stakeholder channel?"** Two rooms, one message, no relay. + stakeholder channel?"** Two rooms, one message, no relay. [Gaps](#gaps) G8. - **"A scheduled Slack workflow could mention the Switch agent to kick off - updates."** This one is already the right answer, and it works today — - see [Cadence](#cadence-and-the-thing-that-nudges). + updates."** Already the right answer, and it works today — see + [Cadence](#cadence-and-the-thing-that-nudges). A fourth comment observes that postmortems are missing from the SOP entirely. -The template can help there, and does. +The design gives them a home. ## Mapping the SOP onto rooms -Switch has exactly one structural primitive that matters here — the room — plus -threads inside it and links between rooms. Getting the mapping right is mostly a -matter of refusing to over-model. +Switch has one structural primitive that matters here — the room — plus threads +inside it and links between rooms. Getting the mapping right is mostly a matter +of refusing to over-model. ### What is a room **Three, and only three.** -**The alert hub** — standing, long-lived, one per product. Datadog's alerts land -here, and this is where on-call acknowledges and posts situation reports. It -already exists as a Slack channel; adopting it into Switch is a matter of adding -the Switch app to it, not creating anything. High volume, low signal, and nobody -should be expected to have read it. +**The incident hub** — standing, long-lived, one per product. This is the +existing alert channel, adopted into Switch rather than created: Datadog's alerts +land here, on-call acknowledges here, situation reports are posted here, and the +responder agent lives here permanently. It is also where an incident is +*declared*. Its `instructions` carry the SOP and the product's bindings, which is +what makes one design work for several products. **The stakeholder channel** — standing, long-lived, one per product. High-level status only, for people who need to know that something is wrong and not how. -Also already exists. +Already exists; adopted, not created. -**The war room** — one per declared incident, created at declaration, dead after -the postmortem. This is the room the template makes. Public, per the SOP's own -resolution of that question: a war room that stakeholders cannot read generates -a second, worse war room in DMs. - -The two standing rooms are not the template's business. They are pre-existing -channels the template *points at*, and their names are parameters. +**The war room** — one per declared incident, built by the responder agent at +declaration, dead after the postmortem. Public, per the SOP's own resolution of +that question: a war room stakeholders cannot read generates a second, worse war +room in DMs. ### What is a thread -Everything that would otherwise fragment the war room. In particular: - -- **A workstream inside the incident.** Two people chasing two hypotheses thread - separately and the room stays readable. -- **A situation report and its follow-ups.** The SITREP goes at the root; the - "what does that mean for the API?" questions hang off it. -- **A tool's noisy output.** Log dumps and query results belong under the - message that asked for them. +Everything that would otherwise fragment a room. In the war room: one thread per +line of investigation, one per situation report and its follow-ups, one for a +tool's noisy output. In the hub: **one thread per incident**, which is the whole +of the banner protocol below. Switch threads bridge to real platform threads everywhere it can — Slack replies go out with a `thread_ts`, Discord gets a real thread, Telegram gets a forum topic. The Slack difference is one of *rendering*: a threaded reply appears in -the channel only as a reply count under the original post rather than in the -main flow. So on a Slack-bridged room — which a war room under this SOP is — -**anything the room must not miss goes at the root**. That is not a Switch limitation to work around; it is a rule for -whoever writes in the room, agent or human. The responder agent's standing -instructions should say so, and the template's `instructions` field is where -that lives. +the channel only as a reply count under the original post rather than in the main +flow. So on a Slack-bridged room — which both of these are — **anything the room +must not miss goes at the root**. That is a rule for whoever writes in the room, +agent or human, and it belongs in the room's instructions. ### What is neither -Three things that look like they want to be rooms and must not be. - **The incident itself.** The incident is a PagerDuty record with an id, a severity, a timeline and a resolution. The war room is a *conversation about* it. Modelling the incident in Switch means two systems disagreeing about severity at the worst possible moment. The room carries the incident id in its name and a -link to the record in its description; that is the whole of the relationship. +link to the record in its description; that is the whole relationship. -**The on-call rotation.** A rotation is a schedule — who is responsible between -which hours. Switch has no schedule, no rotation and no concept of duty (see -[Gaps](#gaps)). A "rotation room" would be a room whose membership someone has to +**The on-call rotation.** A rotation is a schedule. Switch has no schedule and no +concept of duty, and — importantly — it does not need one, because the agent can +ask PagerDuty. A "rotation room" would be a room whose membership someone has to remember to edit every Monday, which is a worse rotation than the one PagerDuty -already runs. The rotation stays in PagerDuty and reaches Switch, if at all, as a -mention group. +already runs. **A per-service standing room.** Tempting, because the SOP's severity table is -organised by service and each service has an owner. But a room per service is a -room per service to keep alive, and the thing that is actually needed — "who owns -the ingestion pipeline, and what does its runbook say" — is a lookup, not a -conversation. It belongs in the war room as an attached document, which is -exactly what the template does with it. +organised by service and each service has an owner. But the thing actually needed +— "who owns the ingestion pipeline, and what does its runbook say" — is a lookup, +not a conversation. It belongs in the war room as an attached document. ### The lifecycle, end to end | Moment | What happens in Switch | | --- | --- | -| Alert fires | Nothing. Datadog → PagerDuty → the alert hub channel. | +| Alert fires | Nothing. Datadog → PagerDuty → the hub channel, as context. | | Ack, triage | Nothing. On-call works in PagerDuty, Datadog and the runbooks. | | Routine alert, no customer impact | Nothing, ever. Most alerts end here and must cost zero Switch overhead. | -| **Customer incident declared** | **The war room is instantiated from the template**, named for the incident, furnished with runbook, owner map and SITREP shape, with the responder agent already in it and briefed. | -| Responders assemble | Invitees are added: on-call, the service owner, the stream lead, support. Public channel, so anyone else can walk in. | -| Investigation | Threads per hypothesis. The responder agent answers lookups, drafts SITREPs, and keeps the timeline. | -| Situation report due | A scheduled nudge addresses the responder agent; it drafts from the room and a human posts or corrects it. | -| Escalation at ~1h | A human decision. Switch's part is that the escalation ladder is *in the room* as a document, so nobody has to find it. | +| **Customer incident declared** | On-call addresses the responder agent in the hub. The agent looks up who is on call, **builds the war room**, and posts the incident banner in the hub. | +| Responders assemble | Already done — the agent invited them when it built the room. Public channel, so anyone else can walk in. | +| Investigation | Threads per hypothesis. The agent answers lookups, drafts SITREPs, keeps the timeline. | +| Situation report due | A scheduled nudge addresses the agent; it drafts from the room and a human posts it. The SITREP goes in the hub, under the incident's banner thread. | +| Escalation at ~1h | A human decision. Switch's part is that the ladder is *in the room* as a document, so nobody has to find it. | | Recovery confirmed | The room stays open — the postmortem is written from it. | -| Postmortem written | The seeded postmortem document is filled in from the room's own timeline. | -| Review scheduled and held | Out of Switch. | -| Done | The room is archived. Archive is not deletion; the transcript survives. | +| Postmortem written | Drafted from the room's own timeline into the seeded postmortem document. | +| Done | The agent archives the war room. Archive is not deletion; the transcript survives. | Two properties of that table are the design: - **Nothing happens until a customer incident is declared.** The overwhelmingly - common path — an alert that resolves itself — never touches Switch. Any design - that provisions a room per alert will be switched off within a week. + common path — an alert that resolves itself — never touches Switch. A design + that provisions a room per alert gets switched off within a week. - **The room outlives the incident.** It closes at the postmortem, not at - recovery. The postmortem is written from the room's own record, which is the - main argument for having conducted the incident in a room at all. + recovery. That is the main argument for having conducted the incident in a + room at all. ### Cadence, and the thing that nudges @@ -210,116 +196,125 @@ The SOP puts situation reports on a clock: hourly at the top severity, every fou hours below it. Something has to remember. **Switch cannot.** There is no scheduling primitive exposed to a room or an -agent: no cron, no timers, no deferred actions, nothing an agent can ask to be -woken by. Switch does run periodic work internally — connection and -runtime-state sweeps at boot, bridge-level renewal loops, `call_later` timers -that batch attachments — but none of it is reachable from a room, and none of it -is scheduling in the sense the SOP means. - -The SOP's own comment thread already has the answer, and it is the right one: a -scheduled Slack workflow that posts into the war room mentioning the responder -agent. The mention arrives as an addressed event, the agent drafts the SITREP -from the room's context, a human checks and posts it. This works today, needs no +agent: no cron, no timers, nothing an agent can ask to be woken by. Switch runs +periodic work internally — connection and runtime-state sweeps, bridge renewal +loops, timers that batch attachments — but none of it is reachable from a room. + +The SOP's own comment thread already has the answer: a scheduled Slack workflow +that posts into the hub mentioning the responder agent. That wakes the agent (see +[the push direction](#the-push-direction-and-why-not-to-rely-on-it)), it drafts +the SITREP from the room, a human checks and posts it. This works today, needs no Switch change, and keeps the clock in the tool that is good at clocks. -It is worth being honest that this is a workaround, not a design: the schedule -lives in a Slack workflow that nobody will remember to delete when the incident -closes, and it has to be created per incident. That is a real cost, and it is -[Gaps](#gaps) item G7. - -## The war-room template - -The template is the deliverable that makes this reusable. One document, stored -in the registry, instantiated per incident with the incident's particulars as -inputs, and reusable across products because everything product-specific is a -parameter. - -### What the template feature actually is today - -Before the YAML, the state of the feature it is built on, because two of the -three pieces named on the ticket are not on `main` and the design has to be -honest about which half exists. - -**Merged and working** (`core/switch_core/rooms_yaml.py`): - -- A single-room YAML document with exactly three top-level keys — `room:` - (required), `params:` and `version:`. `version:` is parsed, type-checked and - then ignored. An unrecognised top-level key is a hard error. -- Typed parameters: `string`, `number`, `boolean`, `enum`, each with an optional - `description`, `default` and — for `enum` — the list of permitted values. - There is no `required:` key: a parameter is required exactly when it has no - default. -- `{name}` interpolation over the `room:` block. Two modes: if a field is - *entirely* one placeholder, the parameter's typed value is substituted whole, - which is how an `enum` can fill `channel_type:`; otherwise each placeholder is - stringified in place. -- A registry (`core/switch_core/gateway/templates.py`): store, list, fetch, - patch, delete and lint a template, with names unique per owner, listing - visible tenant-wide and mutation restricted to the owner or an admin. - -**Not merged.** Group templates — `group:` / `rooms:` / `links:`, per-room -agent aliases in YAML, and interpolation into dict *keys* — are on -`origin/work/group-templates`, along with `create_room_from_yaml`, the operation -that would let an agent instantiate a template at all. Template built-ins like -`{$creator}` are on a different branch again. - -**The two halves do not meet.** The registry stores documents; `POST -/rooms/from-yaml` provisions from a document supplied in the request body. -Nothing fetches a stored template by id and provisions it. Instantiating a -registered template today means downloading its content and posting it back — -and the dashboard's create-from-YAML page posts raw YAML with no `inputs`, so -*from the UI, only a template whose parameters all have defaults can be used at -all*. Passing inputs requires an API client sending the JSON body form. For a -template whose entire purpose is per-incident particulars, that is disqualifying -on its own; it is [Gaps](#gaps) G2 and G3. - -The template below is therefore written twice: once in the merged format, so it -can be built and used now, and once in the group format, as the target. - -### Parameters - -Eleven, in three groups. - -**The incident** — supplied per instantiation, no defaults, all required: - -| Parameter | Type | What it is | -| --- | --- | --- | -| `incident_id` | string | The incident record's id, e.g. `1287`. Goes in the room name so the channel is greppable against PagerDuty. | -| `severity` | enum | `sev0` \| `sev1` \| `sev2`. Drives the update cadence named in the room instructions. | -| `service` | string | The affected service. Drives the owner lookup and the runbook section. | -| `summary` | string | One line: what is broken. Becomes the room description. | -| `incident_url` | string | Link to the incident record. The room's pointer at the system of record. | - -**The product** — supplied per product, and the reason this is reusable rather -than one team's room: - -| Parameter | Type | Default | What it is | -| --- | --- | --- | --- | -| `product` | string | — | Short product name. Prefixes the room name. | -| `alert_channel` | string | — | Where alerts land and situation reports are posted. | -| `comms_channel` | string | — | Where stakeholder updates go. | -| `runbook_reference` | string | — | Name of an existing Switch reference pointing at the product's runbooks. | -| `responder_agent` | string | — | The shared responder agent's name. | - -**The deployment** — sane defaults, rarely overridden: - -| Parameter | Type | Default | What it is | -| --- | --- | --- | --- | -| `bridge` | string | — | Collaboration bridge display name. Required; see the note on omitting it below. | -| `visibility` | enum | `channel_public` | `channel_public` \| `channel_private`. Public by default, per the SOP. | - -Two parameters that are conspicuously *not* here, because they cannot be: - -- **The responders.** `users:` takes a list, and a parameter cannot hold one — - `ParamSpec.type` is `string | number | boolean | enum` and nothing else. A - parameter set to `"alice,bob"` interpolates into a single username - `alice,bob`, which resolves to nobody. So the invitee list is either - hard-coded in the template or spread across one parameter per seat. Neither is - acceptable for a rotation. [Gaps](#gaps) G4. -- **Who is on call.** Nothing in Switch knows. [Gaps](#gaps) G1, and the - hardest problem in this document. - -### The template, in the format that works on `main` today +Be honest that it is a workaround: the schedule lives in a Slack workflow created +per incident that nobody will remember to delete. [Gaps](#gaps) G7. + +## The incident-response agent + +The room is built by an agent, not instantiated from a room template. That is the +central decision in this document, and it is worth setting out why before the +detail. + +### The prior art: the workstream hub + +Switch already runs a production pattern with exactly this shape — the workstream +hubs that drive this repository's own development. Read from the inside, it is: + +- **A standing hub room**, bridged to a channel, whose `instructions` are a + complete operating manual: the workstream's scope, how work is requested, and + a **bindings block** of instance data — the Jira cloudId, project key, label, + transition ids and assignee account ids; the bridge id and channel type to + create work rooms on; the room-group name; the shared reference id to + propagate into every work room. +- **An exclusive room role**, described in its own instructions as a *thin + shell*. It grants two things and nothing else: the exclusive coordination lease + (at most one live holder) and `@manager` addressing, "so anyone can reach + whoever is currently coordinating without knowing which agent that is." The + procedure is deliberately **not** in the role — it lives in the agent. +- **A manager agent** that assumes the role automatically on connecting, every + session, silently. +- **One room per work item**, created by the agent — private, named to a + convention, filed in a group, linked back to the hub, with the shared + reference propagated and the room's `instructions` set to a full task card + written for whoever picks it up. +- **A banner protocol**: exactly one root-level message per item in the hub, and + its thread is that item's entire conversation — status changes and the final + summary included. +- **A pure-pull model**, stated explicitly: nothing happens automatically; work + proceeds when someone requests it and the manager is online. A hub with nobody + holding the role is idle, not broken. + +Every one of those transfers. An incident is a work item with a clock on it. + +### The incident hub + +One per product. The existing alert channel, adopted into Switch. + +Its `instructions` carry three things: + +1. **The SOP** — the declaration criteria, the severity table, the update + cadence, the escalation ladder, and the authority the on-call already has. +2. **The room-writing rules** — root versus thread, and the standing note that + severity and resolution live in PagerDuty, not here. +3. **The bindings block** — everything instance-specific, in one place, so the + same agent definition serves every product: + +``` +## Responder bindings + +**PagerDuty** (MCP) +- Service ids: +- Escalation policy id: <...> +- Severity map: sev0 → P1, sev1 → P2, sev2 → P3 +- On-call lookup: the schedule attached to the escalation policy above + +**Rooms / bridge** +- War rooms: new PUBLIC Slack channel — bridge_id=<...>, + channel_type="channel_public", named ` incident ` +- Room group: incidents +- Every war room is linked back to this hub +- Runbook reference id to propagate: <...> +- Stakeholder channel for status: <...> + +**People** +- Always invite: the on-call primary (from PagerDuty), the owning service's + owner, the stream lead, support +- Escalation contacts by tier: +``` + +And one **exclusive `responder` role**, the same thin shell: the lease plus +`@responder` addressing, so anyone in the channel reaches whoever is currently +coordinating without knowing which agent that is. The agent assumes it on +connect. + +### What the agent does when an incident is declared + +On-call posts in the hub: + +> `@responder` declare sev0 on ingestion — no new findings for 40 minutes, PD 1287 + +The agent then, in order: + +1. **Reads the bindings** from the hub's instructions. +2. **Asks PagerDuty who is on call** for the escalation policy, and reads the + incident record for the id, title and current severity. If PagerDuty and the + human disagree about severity, it says so in the room and takes PagerDuty's. +3. **Resolves the invitee list** — the on-call primary from PagerDuty, plus the + service owner, stream lead and support from the bindings. +4. **Builds the war room** with a single `create_room` call (below). +5. **Posts the banner** in the hub — one root-level message, the incident's + entire thread from here on. +6. **Greets the room** with what it already knows: severity, service, the + incident link, and the fact that the runbook and owner map are attached. + +Step 4 is the one that has to be an agent rather than a template, and step 2 is +the one that closes the SOP's hardest open question. + +### The room it builds + +Expressed as YAML, because a reviewable artifact beats prose and because the +parts of it that *are* expressible as a room template can be registered as one — +as documentation of the shape, and as a fallback when no agent is online. ```yaml version: 0 @@ -343,13 +338,13 @@ params: type: string description: Link to the incident record in the paging system - # ── the product ─────────────────────────────────────────────────────────── + # ── the product (from the hub's bindings) ───────────────────────────────── product: type: string description: Short product name; prefixes the room and channel name - alert_channel: + hub_channel: type: string - description: Channel where alerts land and situation reports are posted + description: The incident hub, where situation reports are posted comms_channel: type: string description: Channel where stakeholder updates go @@ -368,7 +363,7 @@ params: type: enum enum: [channel_public, channel_private] default: channel_public - description: War rooms are public by default so stakeholders can read along + description: War rooms are public by default, per the SOP room: name: "{product} incident {incident_id}" @@ -408,7 +403,7 @@ room: what you used. - When asked for a situation report, draft it in the shape the attached SITREP document gives and post the draft here. A human posts it onward to - {alert_channel} and {comms_channel} — you do not post to those channels. + {hub_channel} and {comms_channel} — you do not post to those channels. - Keep a running timeline as the incident moves: what changed, when, who did it. The postmortem is written from it. - Say what you do not know. During an incident a confident wrong answer @@ -448,7 +443,7 @@ room: content: | # Service owners - Filled in per product at instantiation. One row per service: + Filled in per product from the hub's bindings. One row per service: service, owning team, primary contact role, timezone. A service with no owner is an escalation to the incident coordinator, @@ -513,330 +508,352 @@ room: Review within five business days for the top severity. ``` -Notes on choices in that document that are not arbitrary: - -- **`channel_type: "{visibility}"` is the whole-field form.** When a field is - the *entire* placeholder, the parameter's typed value is substituted rather - than stringified. For an `enum` the two are the same, but the distinction is - the only way a `number` or `boolean` parameter can ever fill a non-string - field, and it is worth knowing because a partial placeholder degrades to a - string silently. +That document is not illustrative. It was run through the shipped parser — +`ParamSpec`, `resolve_params`, `interpolate`, `RoomSpec`, and the visibility-pair +validator — and it resolves: `visibility` defaults to `channel_public`, the +`public` / `private` visibility pair is accepted, every placeholder substitutes +with none left over anywhere including inside the seeded documents, the room +comes out named `flint incident 1287`, and the Slack channel it would create is +`flint-incident-1287`. Anything wrong here is a bug in the design, not a typo. + +Four choices in it that are not arbitrary: + +- **`channel_type: "{visibility}"` is the whole-field form.** When a field is the + *entire* placeholder, the parameter's typed value is substituted rather than + stringified. For an `enum` the two coincide, but it is the only way a `number` + or `boolean` parameter can fill a non-string field, and a partial placeholder + degrades to a string silently. - **The room name slugifies cleanly.** Switch derives the Slack channel name with `re.sub(r"[^a-z0-9_-]", "-", name.lower()).strip("-")[:80]` (`core/switch_core/bridges/collaboration/slack/adapter.py:1366`). The SOP's - bracketed convention — `[Product] [Incident #]` — would produce - `product---incident-42`, with runs of hyphens where the punctuation was. - `{product} incident {incident_id}` gives `product-incident-42`. Cosmetic, but - channel names are what responders type under pressure. -- **`bridge:` is a required parameter and not omitted.** The comment on the - template's own `bridge` field says to omit it for an internal-only room, and - that is wrong: omitting it falls through to the instance default bridge — or, - with no default configured, to no bridge at all, and with a default configured - but not running, to a hard failure. Worse, the guard that rejects `users:` on - an unbridged room tests the template's resolved bridge id, so a template with - `users:` and no `bridge:` is rejected even though the room would in fact have - been bridged. Naming the bridge explicitly sidesteps all of it. - [Gaps](#gaps) G10. -- **The `scribe` role is defined but nobody is told to take it.** See - [Roles are the wrong tool for the on-call rotation](#roles-are-the-wrong-tool-for-the-on-call-rotation) - — a role is held per *agent*, globally, so the shared responder can hold it in - one incident at a time. It is there for a responder's own coding agent to - assume, and the template does not assume it on anyone's behalf. + bracketed convention — `[Product] [Incident #]` — yields + `product---incident-42`. `{product} incident {incident_id}` yields + `product-incident-42`. Cosmetic, but channel names are what responders type + under pressure. - **`write_visibility: private`, and this is the one to argue about.** Public write on a room does not mean "participants may restructure it" — it grants write to *any* principal in the tenant, member or not, and write on a room is what governs attaching a reference, defining and deleting roles, updating the - room and archiving it. A war room that any agent's owner in the deployment can - archive mid-incident is not a trade worth making, so this template narrows it: - readable by everyone, restructured only by the room's owner and admins. - - The cost is real but small. Attaching a reference mid-incident becomes the - instantiating user's job. Adding agents and users is *not* affected — the - roster path is governed separately and admits existing members regardless of - visibility — so pulling another agent into the war room still works for anyone - already in it, which is the operation that actually matters under pressure. - -This document is not illustrative. It was run through the shipped parser — -`ParamSpec`, `resolve_params`, `interpolate`, `RoomSpec`, and the visibility-pair -validator — with the inputs below, and it resolves: `visibility` defaults to -`channel_public`, the `public` / `private` visibility pair is accepted, every -placeholder substitutes with none left over anywhere including inside the seeded -documents, the room comes out named `flint incident 1287`, and the Slack channel -it would create is `flint-incident-1287`. Anything in this section that turns out -to be wrong is a bug in the design, not a typo in the YAML. - -### Instantiating it - -Today, one call, with the inputs in a JSON body: - -``` -POST /rooms/from-yaml -Content-Type: application/json - -{ - "yaml": "", - "inputs": { - "incident_id": "1287", - "severity": "sev0", - "service": "ingestion", - "summary": "Ingestion stalled; no new findings for 40 minutes", - "incident_url": "https:///incidents/1287", - "product": "flint", - "alert_channel": "", - "comms_channel": "", - "runbook_reference": "Flint runbooks", - "responder_agent": "flint-responder", - "bridge": "" - } -} -``` - -A missing required parameter, an undeclared input, or an `enum` value outside its -list fails with a 400 *before* anything is provisioned — there is no half-made -room to clean up. One trap to know: a `{word}` in the body that no parameter -declares is **left verbatim**, deliberately, so that JSON braces in a document's -content survive. A typo in a placeholder name does not error; it ships. Lint the -template (`POST /templates/validate`) before registering it, and read the -findings — the registry itself only refuses three of them. - -### The target-state template, once group templates land + room and archiving it. A war room any agent's owner in the deployment can + archive mid-incident is not a trade worth making. Adding agents and users is + governed separately and admits existing members regardless, so pulling another + agent in still works for anyone already in the room. +- **The `scribe` role is defined and assigned to nobody.** See + [Roles, correctly scoped](#roles-correctly-scoped). -The merged format makes one room. The SOP wants the postmortem written after -recovery, and a comment in the source document observes there is no postmortem -process at all yet. A second, linked room is the better shape: the war room -closes when the incident does, and the postmortem room outlives it, holding the -review and the actions. +### What only the agent can do -That needs `origin/work/group-templates`. In its format: +The YAML above is the room's *shape*. Four things the agent adds that a room +template provisioned from that document cannot, because the template format does +not carry the fields — even though `create_room` accepts every one of them: -```yaml -version: 0 - -params: - incident_id: { type: string } - severity: { type: enum, enum: [sev0, sev1, sev2] } - service: { type: string } - summary: { type: string } - product: { type: string } - bridge: { type: string } - responder_agent: { type: string } - -group: - name: "{product} incident {incident_id}" - description: "War room and postmortem for {product} incident {incident_id}" - color: "#dc2626" - -rooms: - - name: "{product} incident {incident_id}" - description: "{severity} · {service} · {summary}" - bridge: "{bridge}" - channel_type: channel_public - agents: ["{responder_agent}"] - aliases: - "{responder_agent}": responder - instructions: | - ...as above... - - - name: "{product} incident {incident_id} postmortem" - description: "Postmortem and review actions for {product} incident {incident_id}" - bridge: "{bridge}" - channel_type: channel_public - agents: ["{responder_agent}"] - aliases: - "{responder_agent}": scribe - instructions: | - The incident is over. This room exists to produce the write-up and to - track the actions out of it. Draft from the war room's timeline; a human - owns the document. Blameless — name systems and decisions, never people. - -links: - - from: "{product} incident {incident_id}" - to: "{product} incident {incident_id} postmortem" - label: postmortem - - from: "{product} incident {incident_id} postmortem" - to: "{product} incident {incident_id}" - label: incident -``` - -Three things that branch buys and `main` cannot express: - -- **The group** — both rooms filed together, so an incident is one thing in the - room tree rather than two rooms that happen to share a name prefix. -- **The links** — the postmortem room points back at the war room and vice - versa, both directions, because a link is one-way. -- **The aliases** — `@responder` in the war room and `@scribe` in the postmortem - room, both the same agent. This is the per-room identity the responder needs, - and it is worth dwelling on: a rotating group should address a *function*, not - an agent's name. Aliases also require interpolation into a dict *key*, which - only that branch supports. - -### What a template still cannot set — and why that is the real finding - -Measured against what a room can hold, the merged template reaches -`name`, `description`, `instructions`, `bridge`, `channel_type`, the two -visibilities, `agents`, `users`, `roles`, `references` and `docs`. It cannot -reach: - -| Cannot set from a template | Why it matters here | Reachable another way? | +| The agent sets | A template cannot | Why it matters | | --- | --- | --- | -| `aliases` | No `@responder` handle; responders must know the agent's real name. | **Yes** — `create_room`, `update_room`, `PATCH`, or `!set-alias` in the room. | -| `linked_rooms` | The war room cannot point at the alert hub, the stakeholder channel or the postmortem room. | **Yes** — `create_room`, or `link_rooms` after. | -| `group_id` | Incidents cannot be filed under a product's group. | **Yes** — `group_name` on `create_room`. | -| `join_event_listeners` | Without it the responder never learns that someone joined, so it cannot greet an arrival and tell them the state. | **Yes** — `create_room` or `update_room`. | -| `internal_only` | Not needed here — war rooms are bridged. | Yes, on `create_room`. | -| `package_ids` | No packaged tooling attached at creation. | Yes, on `create_room`. | -| room metadata | Nowhere structured to record severity or the incident id; both live in prose. | No. | - -That last column is the finding, and it is more useful than the list itself: -**a room template is strictly less capable than the room creation it wraps.** -Every one of those fields is already accepted by `create_room` — the HTTP -endpoint, and the agent operation of the same name — and most can be set -afterwards with `update_room`. The template format simply does not carry them. - -Two consequences worth acting on: - -- **The gap is a format gap, not a platform gap**, so it is cheap. Nothing needs - designing; the provisioner needs to pass through fields the config object - already has. -- **Until it closes, an agent that creates the war room by calling `create_room` - directly can do everything the template can and more** — aliases, links, the - group, and join events included. That is a genuine fork in the road: the - template is the reusable, reviewable, version-controlled artifact, and - `create_room` is the capable one. Choosing the template means accepting a - follow-up call to set what it could not, or accepting that the war room has no - `@responder` alias and cannot greet arrivals. - -The greeting is the one worth wanting. A war room's worst recurring moment is the -fourth person arriving twenty minutes in and asking "what's the state?" — a -question the responder could answer the instant they join, if something opted it -into join events. A template cannot; one extra `update_room` call can. - -## The responder agent - -The rotation is the whole problem. Six engineers take the pager in turn; the -agent has to be the same agent for all of them, reachable by whoever is on duty, -and not degraded by the fact that the person who set it up is on holiday. - -### What the SOP needs it to do - -Modest, deliberately. In the war room: - -- **Answer lookups.** Who owns this service. What the runbook says. What the - escalation ladder is. These are the questions that cost minutes at 03:00 and - they are all document reads. -- **Draft the situation report** in the right shape when nudged, from what the - room has said, for a human to check and post onward. -- **Keep the timeline**, so the postmortem is written from a record rather than - from memory. -- **Orient arrivals** — say what is known so far when someone joins. - -Note what is absent: it does not page, does not set severity, does not decide, -and does not act on production. The SOP grants *humans* the authority to roll -back and to push emergency fixes; extending that to a shared agent that six -people can address and nobody can attribute would be the single worst decision -available here. See [The rule that makes it safe](#the-rule-that-makes-it-safe). - -### What `flint-tracker` actually is - -Read from the live instance rather than assumed, because it is the prior art and -being wrong about it would poison the recommendation: - -- **Name: `flint-tracker`.** No owner suffix. Every other Claude Code agent on - the instance registered through Switch Console carries one — - `claude-code..`. This one reads as a service, and that is not - cosmetic: the name is the routing key for everything. Mentions, the Slack user - group the bridge mints, room aliases and `target_names` all resolve `name`, - and `display_name` routes nothing at all. -- **Owner: the deployment's `Admin` account.** Not a person. -- **`connection_model: auto_session`, `channels_enabled: true`**, with a working - directory on a shared always-on host rather than on anyone's laptop. Something - runs there continuously, watching for the agent to be addressed and spawning a - session on demand. -- **`addressing_policy: null`** — wide open. Anyone in any room it is in can - address it. -- **Six room memberships across three platforms** — Slack, Discord and - Mattermost. It is designed to be invited around, not to live in one room. -- **A description written at the reader**, ending "Invite it into any room and - ask what was decided, what changed, or who owns something." It tells a - stranger what to do with it. - -That is a coherent design and most of it is exactly right for a responder. - -### What carries over - -**The name.** `flint-responder`, not `claude-code.oncall.`. It is the -handle six people will type under pressure and it must not encode whose agent it -is. - -**Shared infrastructure, not a laptop.** This is the load-bearing one. An -`auto_session` agent is brought online by a watcher process; put that watcher on -an always-on host and the agent is online regardless of who is on duty, whether -their machine is asleep, or whether they have ever installed Switch Console. -A responder that only works when a particular laptop is open is not a responder. - -**An open addressing policy.** A rotating group cannot be enumerated, so the -policy cannot enumerate it. Open within the rooms it is in is the correct -setting, and it is what `flint-tracker` runs. - -**Membership by invitation.** The agent belongs to rooms, not to a room. A war -room is created and the agent is added; nothing about the agent changes per -incident. - -**A description that tells a stranger what to ask.** Half the value of a war-room -agent is discovered by someone who has never used it, mid-incident, from the -member list. +| `aliases` | ✗ | `@responder` in the war room, so responders address a function, not an agent's name. | +| `linked_rooms` | ✗ | The war room points back at the hub, and the hub at it. | +| `group_name` | ✗ | Incidents filed under one product group instead of loose rooms sharing a prefix. | +| `join_event_listeners` | ✗ | Without it the agent never learns someone joined, so it cannot greet the fourth person arriving twenty minutes in and tell them the state. | +| `user_names` **computed at declaration** | ✗ | The template would need the invitee list as input; the agent *derives* it by asking PagerDuty. | + +The last row is the real argument. A template is a function of its inputs, and +somebody has to supply them. An agent can go and find them. "Who is on call right +now" is not something a human should be typing into a form at 03:00, and it is +the exact question the SOP has an open comment about. + +The first four are a cheaper argument but a sharper one: **a room template is +strictly less capable than the room creation it wraps.** Those fields already +exist on the config object and are already validated; the template format simply +does not pass them through. That makes [Gaps](#gaps) G9 a pass-through fix rather +than a feature — worth doing, and not worth waiting for. + +### The banner protocol + +One root-level message per incident in the hub, posted by the agent immediately +after the war room exists. Its thread is the incident's entire record in the hub: +every situation report, every severity change, the resolution, and a link to the +postmortem. + +This is lifted unchanged from the workstream hubs, and it earns its place three +times over: + +- The SOP already requires situation reports in the alert channel. Under the + banner they are threaded under the incident they belong to instead of + interleaved with unrelated alerts. +- Someone scrolling the hub sees one line per incident, not forty. +- The postmortem is written from one thread. + +### Why an agent and not a room template + +Stated plainly, because it is the decision everything else follows from: + +- **A template cannot look anything up.** Every value must be supplied by whoever + instantiates it. The single most valuable thing here — who is on call — is a + lookup. +- **A template is less capable than `create_room`.** Aliases, links, group and + join listeners are unreachable from the format today. +- **Nothing instantiates a stored template anyway.** The registry stores + documents; `POST /rooms/from-yaml` provisions from a document in the request + body; nothing joins them. And the dashboard's create-from-YAML page posts raw + YAML with no `inputs`, so from the UI only a template whose every parameter has + a default works at all — which an incident template, by definition, is not. +- **The room is only half the job.** Somebody has to post the banner, greet + arrivals, draft the SITREP and archive the room at the end. That is an agent + with a procedure, and once it exists, having it also make the room costs + nothing. + +The template is not useless — register the YAML above so the shape is reviewable, +diffable and available when no agent is online. But it is documentation of the +design, not the mechanism. + +### What is actually reusable + +Not a room template. Three things, together — which is what an **agent template** +would have to mean if the concept is going to earn its name: + +1. **The agent definition** — the responder's procedure, identical for every + product: read the bindings, look up on call, build the room, post the banner, + draft SITREPs, keep the timeline, archive at the end. +2. **The hub room's instruction card** — the SOP text plus the bindings block, + with the product-specific values filled in. This is the only thing that + changes per product. +3. **The room shape** — the YAML above, as the specification the agent builds to. + +Standing up incident response for a second product means writing one bindings +block. That is the reuse the ticket asked for, and it is a stronger form of it +than a room template can offer, because the varying part is one block of +configuration rather than a fork of the artifact. + +## Reaching PagerDuty + +The question was whether an agent can use an API or MCP for PagerDuty, "kinda +like what we do with Jira". The answer is yes — and it is worth being precise +about what the Jira pattern actually is, because it is not an integration. + +### What the Jira pattern actually is + +Switch's entire Jira presence is a **reference type** +(`core/switch_core/bridges/resource/registry.py:83-100`) whose agent-facing +instructions say: + +> To access this Jira resource you need (1) access to the linked project, +> issue(s), or board, and (2) an agent connector that can fetch Jira content on +> your behalf — typically the Atlassian MCP connector… + +Read what that is doing. Switch ships **the pointer and the prose**. It ships no +connector. The capability comes from an MCP server a human installed on the host +the agent runs on, and the instance specifics — cloudId, project key, transition +ids, account ids — live in the hub room's bindings block. Three parts, and only +one of them is Switch's. + +That is the pattern to copy, and copying it is mostly configuration: + +1. **A PagerDuty MCP server on the responder's host** (or, equivalently, a token + in the host environment and the REST API through the agent's own shell). This + is where the capability comes from. +2. **A `pagerduty` reference type**, user-defined — the type registry is open, + any slug matching `^[a-z][a-z0-9_]{1,62}$` that is not a built-in — with + instructions saying what the agent may do with it and what it must never do + (change severity, resolve, acknowledge on someone's behalf). Attach the + references to the hub and to each war room, or bundle them with the runbook + in a package so it is one attach. +3. **A PagerDuty bindings block** in the hub's instructions: service ids, + escalation policy id, the severity map, and which schedule to read for on + call. + +Be clear about the limits of step 2, because the Jira precedent has the same +ones: a reference type gives an agent a display name, a paragraph of +instructions, a value hint and a list of URLs. Every reference type — built-in or +custom — has the same value shape. **No credential, no client, no tool, no +network call.** If nobody installs the MCP server, the agent reads a paragraph +telling it to do something it cannot do. + +### What this closes + +The SOP's hardest open question — how Switch learns who is on call — stops being +a Switch question. The agent asks PagerDuty for the on-call for the escalation +policy, gets names, and passes them to `create_room(user_names=[...])`. Switch +never models a rotation, never syncs a schedule, and never goes stale. + +This is a much better outcome than the alternative that suggested itself first +(teach Switch to expand a Slack user group into members). That would have been a +real feature with real maintenance, and it would still have been a second copy of +a rotation PagerDuty already owns. + +Two residual constraints, neither fatal: + +- The people it names must already be known to the bridge — `add_users_to_room` + resolves usernames against the external users Switch has seen on that bridge, + and an unknown name comes back unresolved rather than failing loudly. The + agent should report unresolved names in the room rather than quietly + inviting four of five people. +- PagerDuty's names are not Slack's. The bindings block needs a mapping, or the + deployment needs PagerDuty users' Slack handles populated on their profiles. + +### The two real constraints on MCP + +**MCP is per-machine, not per-agent.** Every connector plugin bundles exactly one +MCP server — the Switch runtime — and every provider declares MCP scope as +`global`; the capability schema does not admit any other value. Switch Console +writes a per-agent *launch profile* (model, reasoning effort, instructions) and +deliberately registers no MCP server in it. The MCP management UI was removed and +the config adapters that remain have no live callers. + +So giving the responder a PagerDuty MCP server means editing the host's global +config, and every agent session on that machine gets it. That is an argument for +the dedicated responder host this design already wanted — not against the +approach. [Gaps](#gaps) G19. + +**There is no agent-scoped secret storage.** Switch encrypts its own API keys and +bridge tokens, and stores a server-side connector's config as plain JSONB. There +is nothing for a third-party credential belonging to one agent. A PagerDuty token +lives in the host's environment, or in Switch Console's per-provider environment +map — which is plaintext and applies to every agent of that provider. Switch +neither scopes it, rotates it, nor audits its use. [Gaps](#gaps) G20. + +### The alternative: a server-side connector + +Worth naming because it is the only Switch-managed, server-held, +credential-carrying integration point that exists. A server-side connector runs +in Switch's own process, discovers agents on an external platform, registers them +as Switch agents — deliberately **not** owner-only, with the comment that such an +agent "is a service the deployment offers everyone, not one person's assistant" — +and keeps them permanently online. One type exists today. + +A PagerDuty connector would bend the shape: PagerDuty has no agents, so it would +discover one synthetic agent whose job is to answer PagerDuty questions in a +room. The cost is roughly one module implementing five methods plus a +registration line. The result is a *conversational* PagerDuty agent the responder +talks to, not a *tool* the responder calls — which is worse for this use case, +and it would put the PagerDuty token in cleartext in Postgres. + +Recommend against it here. It is the right shape for a future where PagerDuty +access should be a deployment-wide service rather than one host's configuration, +and it is worth remembering then. + +### The push direction, and why not to rely on it + +Everything above is *pull*: the agent, while running, calls out to PagerDuty. +The other direction — PagerDuty causing something to happen in Switch — is +weaker, and the details matter because the failure is silent. + +**What works.** A third-party app posting into a bridged Slack channel is not +filtered out. `bot_message` is explicitly on the adapter's allow-list, with a +comment naming Datadog alerts, and only the Switch app's *own* messages are +suppressed as echoes. The app gets a puppet identity named after it, and if its +message contains `@responder` — as literal text, or as the agent's Slack +user-group pill, which is what Workflow Builder inserts when you pick an agent +from the `@` menu — the agent is addressed, and an `auto_session` agent is +spawned to handle it. + +**What does not.** + +- **An alert with no mention wakes nothing.** Only *addressed* events are + notifiable, and only a notifiable event spawns a session. The alert lands in + the room as context and the responder never moves. This is the likely + surprise. +- **Most of PagerDuty's message content is dropped.** Rich blocks are read only + when the message has no plain-text body, and even then only `section`, + `header` and `rich_text` blocks — `context` and `actions` blocks, where + PagerDuty puts service, urgency, assignee and its buttons, are discarded. + Attachments are read only if no block yielded anything. Since PagerDuty + normally sets a text fallback for the notification preview, Switch usually + sees that one line and nothing else. +- **Edits never arrive.** `message_changed` and `message_deleted` are dropped, so + a PagerDuty message edited in place to "Resolved" leaves Switch's copy saying + the incident is open. +- **An owner-scoped addressing policy turns this into noise.** A bot has no + Switch account, so under a restricted policy every app-triggered mention is + refused with a message telling PagerDuty to link its account in Switch + Console — posted into the channel. One more reason the responder's policy must + be open. +- **`!commands` from a workflow never wake a dormant agent**, though they work + against a live session. Native slash commands are human-only. + +**So: use pull, and declare by hand.** A human addresses the responder in the hub +to declare. This matches the pull model the workstream hubs adopted deliberately +— "nothing happens automatically; work proceeds when someone requests it" — it +needs no Slack plumbing, and the SOP already has a human in exactly that spot +making exactly that decision. The auto-wake path is a legitimate option for the +*cadence nudge*, where the content does not matter and only the mention does. It +is a poor foundation for declaration, where the content is the whole point. + +## Making the agent user-agnostic + +The rotation is the problem the agent has to survive. Six engineers take the +pager in turn; the agent must be the same agent for all of them, reachable by +whoever is on duty, and not degraded because the person who set it up is on +holiday. + +### What the agent does, and does not, do + +In the hub and the war room it answers lookups, drafts situation reports, keeps +the timeline, greets arrivals, and builds and archives the room. It does not +page, does not set severity, does not resolve, does not decide, and does not +touch production. The SOP grants *humans* the authority to roll back and push +emergency fixes; extending that to a shared agent that six people can address and +nobody can attribute would be the worst decision available here. See +[the rule that makes it safe](#the-rule-that-makes-it-safe). + +### What the existing shared agents get right + +`flint-tracker` and the workforce managers are the same shape, read from the live +instance rather than assumed: + +- **A name with no owner suffix** — `flint-tracker`, not + `claude-code..`. The name is the routing key for everything: + mentions, the Slack user group the bridge mints, room aliases, `target_names`. + `display_name` routes nothing. +- **`auto_session` with a watcher on a shared always-on host**, not a laptop. The + agent is online regardless of whose turn it is, whether their machine is + asleep, or whether they have ever installed Switch Console. This is the + load-bearing one, and it is also what makes the PagerDuty MCP install + tractable — one host to configure. +- **An open addressing policy.** A rotating group cannot be enumerated, so the + policy must not try. +- **Membership by invitation.** The agent belongs to rooms, not to a room. +- **A description written at the reader**, telling a stranger what to ask it. + +Copy all of that. ### What breaks -Six things, in rough order of how much they will hurt. - -**1. Admin ownership hands the agent the whole deployment.** An agent inherits -*exactly* its owner's permissions — the authorization module says so in its -opening lines — and `User.role == "admin"` is a global bypass on every read, -write and delete. So an Admin-owned agent can modify or delete any reference, -document, package or room in the tenant. For an agent that reads Slack and -summarises, that is an over-grant you can live with. For a responder that runs -during an incident, with tool access, addressed by six people under time -pressure, it is not: the moment its blast radius is widest is exactly the moment -it is unbounded. **Do not copy this part.** - -**2. There is nothing good to own it instead.** Switch has exactly one -shared-owner construct: the synthetic bootstrap account that owns every agent -registered with the deployment-wide token. It is deliberately non-admin, which is -right. It is also password-less, so on a password deployment nobody can sign in -as it. (On an OIDC deployment this is softer than it sounds — an identity -provider that asserts that address would link to the existing account — but that -is a deployment accident, not a supported way to hold a shared identity.) +**1. Owner permissions.** An agent inherits *exactly* its owner's permissions, +and `User.role == "admin"` is a global bypass on every read, write and delete. The +existing shared agents are owned by the deployment's `Admin` account, so each has +unbounded authority over every reference, document, package and room in the +tenant. That is a deliberate house pattern and it is tolerable for agents that +read and summarise. It is worse for a responder: the moment its blast radius is +widest is exactly the moment it is unbounded, addressed by six people under time +pressure. Own it with a dedicated **non-admin** service user instead. + +**2. There is nothing good to own it with.** Switch has one shared-owner +construct — the synthetic bootstrap account that owns agents registered with the +deployment-wide token. It is deliberately non-admin, which is right, and on a +password deployment nobody can sign in as it. (On an OIDC deployment an identity +provider asserting that address would link to it, but that is an accident, not a +supported way to hold a shared identity.) The consequence is narrower than "unmanageable" and still bad. An admin *can* -manage a bootstrap-owned agent: edit its options, set its addressing policy, -delete it. What nobody can do is **reveal its credential**, because credential -reveal is the one check in the system with strict owner equality and no admin -bypass. So a bootstrap-owned responder is an agent whose token can never be -recovered — you can rotate it by re-registering, and you can never read it. - -So the correct answer, "own it with a non-person account that is not an admin", -needs a service user someone can actually authenticate as, and there is no -supported way to make one. [Gaps](#gaps) G11. - -Note also that "user-agnostic" cannot mean *ownerless*. An agent with -`owner_id IS NULL` cannot create a reference, attach one, list references, or -attach resources when creating a room — every one of those paths resolves the -agent to its owner and fails loudly without one. It cannot even edit itself over -MCP, because that guard compares two `None`s and refuses. Ownerless is a broken -agent, not a neutral one. **User-agnostic means owned by a non-person, not owned -by nobody.** - -And ownership is permanent: `owner_id` is set at registration and there is no -endpoint anywhere that changes it. Registering the responder under a person "just -for now" means it is theirs until someone runs an `UPDATE`. - -**3. The default addressing policy locks the rotation out, and the UI that fixes -it breaks it.** Every agent registered through any HTTP path is created -owner-only with an empty allowed-agents list. The `register_agent` function takes -an `owner_only=False` parameter, but no wire path passes it — the sole caller is -the server-side connector registration, whose comment is worth quoting because it -is this design's precedent: +manage a bootstrap-owned agent — options, addressing policy, deletion. What +nobody can do is **reveal its credential**, because credential reveal is the one +check with strict owner equality and no admin bypass. So a bootstrap-owned +responder has a token that can be rotated and never read. [Gaps](#gaps) G11. + +Ownership is also permanent: `owner_id` is set at registration and no endpoint +changes it. Registering the responder under a person "just for now" means it is +theirs until someone runs an `UPDATE`. + +And note that user-agnostic cannot mean *ownerless*. An agent with no owner +cannot create a reference, attach one, list references, or attach resources when +creating a room — every one of those paths resolves the agent to its owner and +fails. It cannot even edit itself over MCP, where the guard compares two `None`s +and refuses. **User-agnostic means owned by a non-person, not owned by nobody.** + +**3. The default addressing policy locks the rotation out.** Every agent +registered through any HTTP path is created owner-only with an empty +allowed-agents list. `register_agent` takes an `owner_only=False` parameter and no +wire path passes it — the sole caller is server-side connector registration, +whose comment is this design's precedent: > A server-side connector agent is a service the deployment offers everyone, not > one person's assistant; it is owned by whoever holds the registration token @@ -844,482 +861,459 @@ is this design's precedent: > alone. So the responder is born locked and must be widened afterwards through -`PUT /agents/{id}/addressing-policy`. And here is the landmine: the gateway's -React policy editor models only the four id-shaped dimensions, so the symbolic -`owner` and `owner_agents` rules are dropped from any rule it saves. - -The dashboard does catch the worst case — a rule with every sender dimension -empty is flagged "This rule can never match" and Save is disabled — so you -cannot brick the agent outright. What you *can* do is the ordinary thing: open -the default owner-only policy, add an agent to the allowed list, save, and -silently lose `owner: true` in the process. The policy that comes back admits -that one agent and locks out the human owner, who then gets - -> You're not permitted to direct messages to me in this room — my operator has -> restricted who can address me here. - -from their own agent. Mid-incident that reads as an outage. Switch Console's -editor round-trips the symbolic rules correctly; the gateway's does not. -[Gaps](#gaps) G12. +`PUT /agents/{id}/addressing-policy`. The landmine: the gateway's React policy +editor models only the four id-shaped dimensions, so the symbolic `owner` and +`owner_agents` rules are dropped from any rule it saves. It does disable Save on +a rule that can never match, so the agent cannot be bricked outright — but the +ordinary action, adding an allowed agent to the default policy, silently drops +`owner: true` and locks the human owner out. Switch Console's editor round-trips +them correctly; the gateway's does not. [Gaps](#gaps) G12. -**4. The offline nudge wakes the wrong person.** When an `auto_session` agent is -addressed in a room where nothing can start it, Switch posts on its behalf. The -message names the *owner*: +**4. The offline nudge wakes the wrong person.** Addressed with nothing to start +it, an `auto_session` agent posts on its own behalf, naming its *owner*: > `@owner` — I'm not online in this room, and `@asker` needs me. Open Switch > Console to bring me online here. -and, when the owner has no account on that platform to mention: +The code's comment explains the reasoning — "the fix is for the OWNER to open it, +and nobody else in the room can act" — which is right for a personal agent and +exactly wrong for a shared one. At 03:00 the hub names a service account nobody +watches. Running the watcher on an always-on host makes this rare rather than +fixing it. [Gaps](#gaps) G13. -> I'm not online in this room, and `@asker` needs me. **My owner needs to open -> Switch Console** to bring me online here. +**5. One credential, no rotation, no per-holder revocation.** One agent has one +API key row. No rotation endpoint — the only rotation is re-registration with +overwrite, which deletes the old row and breaks every holder at once. Reveal is +owner-only with no admin bypass. And the token is a bearer credential in a +plaintext file in the agent's working directory, which the repository's own +documentation calls a known exposure. -Both may carry a terminal command underneath. - -The code's own comment explains the reasoning — "the fix is for the OWNER to open -it, and nobody else in the room can act" — which is sound for a personal agent -and exactly wrong for a shared one. At 03:00 the war room will either name a -service account nobody watches, or a dead end. What it should name is whoever is -on call. [Gaps](#gaps) G13. - -Running the watcher on an always-on host makes this rare rather than fixing it. - -**5. One credential, no rotation, no per-holder revocation.** One agent has -exactly one API key row. There is no rotation endpoint: the only way to change -the key is re-registration with overwrite, which deletes the old row, so every -holder breaks at once. Reveal is restricted to the owning user with no admin -bypass. And the token is a bearer credential in a plaintext file in the agent's -working directory — the repository's own documentation calls that a known -exposure. - -The practical consequence is a rule rather than a fix: **the responder's -credential lives in exactly one place, on the shared host, and is never -distributed to responders.** Handing it to six laptops means six copies of a -token nobody can individually revoke, on machines that leave with their owners. -[Gaps](#gaps) G14. +The consequence is a rule, not a fix: **the responder's credential lives in +exactly one place, on the shared host, and is never distributed.** Handing it to +six laptops means six copies of a token nobody can individually revoke, on +machines that leave with their owners. [Gaps](#gaps) G14. This also settles a mechanical question. Two people *can* run sessions as the same agent — identity is per directory, not per machine, and an agent may hold up -to 32 connections. But at most one session of an agent may act in a given room, -and `connect_to_room` always takes over: the newcomer wins and is warned what it -displaced, and the incumbent stops receiving that room's events. The incumbent's -notification is a bare subscription change with no reason attached, so in -practice one responder's session goes quiet without explaining why. Two -responders each starting a session during one incident would evict each other in -turn. One process, on one host, is the only sane operating mode. - -**6. Nothing records which human drove it.** No actor is stored on connections, -sessions, runtime state, role leases or messages; a message is attributed to the -agent, not to whoever prompted it. For most agents that is a shrug. For incident -response it is not, because the postmortem's second question is always "who did -what, when". [Gaps](#gaps) G15. - -There is a mitigation, and it is a design rule rather than a feature — see below. - -### Roles are the wrong tool for the on-call rotation - -Room roles look purpose-built for this: named, assumable instruction bundles; -`@role` reaches whoever currently holds it; an exclusive role admits one holder -and auto-releases about six seconds after that holder dies. "Address whoever is -currently the incident commander" is precisely the sentence roles exist for. - -They still do not work here, for three reasons. - -**A lease is held per agent, globally.** The lease table is unique on the agent, -not on the session and not on the room. One shared responder can therefore hold -one role, in one room, across the entire instance. Two concurrent incidents and -it can be the scribe in only one of them. Worse, two sessions of the same agent -assuming the same role is treated as an idempotent re-assume — the second simply -overwrites the first's session pointer — so roles provide no arbitration at all -between two people running the shared agent, which is the one thing you might -have hoped they would provide. [Gaps](#gaps) G16. - -**Humans cannot hold roles.** Assuming a role is an agent operation. The incident -commander is a person, so the role cannot be theirs. - -**There is no eligibility control.** The role model carries an `eligibility` -field that is declared, documented as a forward-looking hook, and read by -nothing. Any room member may assume any role. "Only the on-call primary may take -incident commander" is not expressible. [Gaps](#gaps) G17. - -Where roles *do* work is between distinct agents. If responders bring their own -coding agents into the war room — which they will, because that is how anyone -investigates — then an exclusive `scribe` role is genuinely good: one holder at a -time, real handoff by release-and-assume, automatic release within seconds if -that engineer's session dies. That is why the template defines the role and -assigns it to nobody. - -So: **incident commander and scribe stay human conventions, written in the room's -instructions. The `scribe` role exists for a responder's own agent to pick up, -not for the shared responder.** +to 32 connections. But at most one session may act in a given room, and +`connect_to_room` always takes over: the newcomer wins and is warned what it +displaced, while the incumbent simply stops receiving that room's events, with a +bare subscription change and no reason attached. Two responders starting sessions +during one incident would evict each other in turn. One process, one host. + +**6. Nothing records which human drove it.** No actor on connections, sessions, +runtime state, leases or messages; a message is attributed to the agent. For most +agents that is a shrug. For incident response it is not, because the postmortem's +second question is always "who did what, when". [Gaps](#gaps) G15. + +### Roles, correctly scoped + +Room roles are the right tool here, but only at the hub, and it is worth being +exact about why — the constraint is easy to design past and expensive to discover +late. + +**In the hub: use one.** An exclusive `responder` role is the same thin shell the +workstream hubs use — the lease plus `@responder` addressing, so anyone reaches +whoever is currently coordinating without knowing which agent that is. It +auto-releases within about six seconds of a holder's session dying, so another +agent can take over with no manual handoff. That is genuine failover, and it is +exactly the "address the duty, not the person" semantics the SOP wants. + +**In a war room: do not.** A role lease is unique per *agent*, globally — not per +room, not per session. One shared responder can therefore hold one role across +the whole instance. If it holds `responder` in the hub, it cannot also hold +`scribe` in a war room, and with two concurrent incidents it could be scribe in +only one of them anyway. Two sessions of the same agent assuming the same role is +treated as an idempotent re-assume, so roles arbitrate nothing between them. +[Gaps](#gaps) G16. + +**Humans cannot hold roles at all** — assuming a role is an agent operation — so +"incident commander" cannot be a role. And there is no eligibility control: the +role model carries an `eligibility` field documented as a forward-looking hook +and read by nothing, so any room member may assume any role. [Gaps](#gaps) G17. + +Hence the shape: `responder` in the hub, held by the shared agent; `scribe` +defined in each war room and assigned to nobody, there for a responder's own +coding agent to pick up; incident commander a human convention written into the +room's instructions. ### The recommendation **One shared responder agent per product, owned by a dedicated non-admin service -user, running `auto_session` on shared always-on infrastructure, with an open -addressing policy, invited into each war room by the template, and never run from -an engineer's machine.** - -Concretely, on top of what `flint-tracker` already gets right: +user, running `auto_session` on shared always-on infrastructure with the +PagerDuty MCP server installed, holding an exclusive `responder` role in the +product's incident hub, with an open addressing policy, and never run from an +engineer's machine.** | Setting | Value | Why | | --- | --- | --- | | `name` | `-responder` | The routing key. No person in it. | | owner | a dedicated service user, **not** an admin | The agent inherits its owner's permissions exactly. | | `connection_model` | `auto_session` | Comes online when addressed; nobody has to remember to start it. | -| watcher | one, on an always-on host | Online regardless of whose turn it is. | +| host | one always-on machine, PagerDuty MCP installed | Online regardless of whose turn it is; one place to configure the integration. | | credential | one copy, on that host | Cannot be revoked per holder, so do not spread it. | -| addressing policy | open | A rotation cannot be enumerated. | -| room membership | per incident, via the template | Nothing about the agent changes per incident. | -| roles held | none | A lease is per agent; holding one breaks the second concurrent incident. | +| addressing policy | open | A rotation cannot be enumerated — and a bot-posted mention is refused under any restricted policy. | +| role | exclusive `responder`, in the hub only | Address the duty, not the agent. One lease per agent, so the hub gets it. | +| war rooms | built per incident, archived after the postmortem | Nothing about the agent changes per incident. | Two alternatives, and why not: -- **One responder agent per engineer.** Real per-human attribution, real role - arbitration, and each agent already exists in some form. But it is six - registrations, six addressing policies and six credentials to keep consistent, - it churns on every rotation change, and each agent is still personally owned — - so the day someone leaves, their responder leaves with them. It solves - attribution by giving up shared identity, which is the thing that was asked - for. -- **Own the shared agent with the Admin account, like `flint-tracker`.** One - fewer problem today, in exchange for an agent with unbounded authority over - every room and resource in the deployment, addressable by anyone, during the - worst hour of the quarter. If G11 cannot be closed before the first incident, - this is the compromise to take *knowingly and temporarily* — and the mitigation - is that the agent has no production access and no write path outside its rooms. +- **One responder agent per engineer.** Real per-human attribution and real role + arbitration. But six registrations, six addressing policies, six credentials + and six PagerDuty MCP installs to keep consistent; it churns on every rotation + change; and each agent is still personally owned, so the day someone leaves, + their responder leaves too. It solves attribution by giving up the shared + identity that was the requirement. +- **Own the shared agent with the Admin account**, as the existing shared agents + do. One fewer problem today, in exchange for an agent with unbounded authority + over every room and resource, addressable by anyone, during the worst hour of + the quarter. If G11 cannot be closed before the first incident, take this + *knowingly and temporarily* — the mitigation being that the agent has no + production access and no write path outside its rooms. ### The rule that makes it safe Because nothing records which human drove the agent, the room transcript has to -carry the attribution instead. That is achievable, but only if the agent is -constrained: +carry the attribution instead: > **The responder takes no consequential action that a human did not ask for, in -> the room, in writing.** Everything it does is either a read, or a draft posted -> back to the room for a human to act on. It never posts to the stakeholder -> channel, never touches the incident record, and never runs anything against -> production. +> the room, in writing.** Everything it does is a read, or a draft posted back to +> the room for a human to act on. It never posts to the stakeholder channel, +> never changes the incident record, and never runs anything against production. + +Under that rule the room *is* the audit log: every action has a message above it +from the person who asked. Relax the rule and G15 becomes a real hole. The rule +belongs in the hub's instructions and in the agent's own definition. -Under that rule the room *is* the audit log: every action the agent took has a -message above it from the person who asked. Relax the rule and the attribution -hole in G15 becomes a real one. The rule belongs in the room instructions, where -the template puts it, and in the agent's own configuration. +Note that this rule is what makes PagerDuty access safe to grant. Reading +schedules and incidents is a lookup. Acknowledging, changing severity or +resolving is a decision, and those stay with the human even though the MCP server +would happily let the agent do them — so the `pagerduty` reference type's +instructions must say so explicitly, because the tool surface will not. ## Gaps -Eighteen, grouped by what they block. Each names what is missing, why it matters -for this SOP specifically, and a ticket to file. Sizes are rough: **S** is days, -**M** is a sprint, **L** is a project. +Twenty-one, grouped by what they block. Each says what is missing, why it matters +here, and a ticket to file. Sizes are rough: **S** is days, **M** is a sprint, +**L** is a project. + +**Read the header of group A first.** The gap that looked hardest is largely not +a gap. -Read G1 first. It is the one the SOP itself has flagged and nobody has answered. +### A. Knowing who is on call — mostly closed -### A. Knowing who is on call +Switch has no rotation, schedule or concept of duty, and after working the design +through, **it should not acquire one**. The agent asks PagerDuty and passes the +answer to `create_room`. What remains is smaller: -**G1 — Switch has no concept of duty, and cannot learn one.** -There is no rotation, schedule, team or on-call anything in the codebase. The -tenant membership role field exists but is documented as recording a value that -nothing yet reads. The SOP's own comment thread asks "how will Switch pull in -who's on-call from PagerDuty?" and answers itself with the right idea — a mention -group whose membership auto-rotates. Switch cannot consume that either: -`add_users_to_room` resolves individual usernames against the people the bridge -has already seen, and there is no call anywhere that expands a Slack user group -into its members. Note the asymmetry — Switch *creates* a Slack user group per -agent, so `@` works, but it never sets that group's membership and cannot -read anyone else's. +**G1 — There is no identity mapping across systems.** +PagerDuty knows a person by one name, Slack by another, Switch by a third. +`add_users_to_room` resolves usernames against external users the bridge has +already seen; a name it cannot place is returned as unresolved rather than +raising, so a war room can quietly come up with four of the five people it should +have. There is also no way at all to invite someone the bridge has never seen. -Until this is closed, "invite the on-call engineers" is a human action, and the -template's invitee list is hard-coded or empty. +> **Proposed ticket:** *Surface unresolved invitees as a first-class result* — +> so a caller must handle "these three could not be added" rather than reading it +> out of a list. **S** -> **Proposed ticket:** *Resolve a platform group to room members* — accept a -> bridge group handle wherever `user_names` is accepted, expand it through the -> platform (Slack user groups, Discord roles, Mattermost groups) at the moment -> of use, and add the members. Deliberately no schedule in Switch: the rotation -> stays in PagerDuty, which syncs the group. **M** +> **Proposed follow-up:** *Cross-system identity mapping* — a per-bridge map from +> an external identity to a Switch user, so an agent holding a PagerDuty +> user can find the Slack account. Until then the mapping lives in the hub's +> bindings block, by hand. **M** -> **Proposed follow-up:** *Room escalation target* — a room-level setting naming -> who to reach when an agent cannot be brought online or nobody has responded, -> resolvable to a group. Feeds G13. **S** +### B. The template format -### B. Making the template usable +Only relevant if the template path is taken. The agent path needs none of it. **G2 — The registry cannot instantiate what it stores.** -`POST /rooms/from-yaml` provisions from a document in the request body. The -registry stores documents. Nothing joins them: there is no "instantiate template -`` with these inputs". Using a registered template today means fetching its -content and posting it back, which makes the registry a filing cabinet rather -than a feature. +`POST /rooms/from-yaml` provisions from a document in the request body; the +registry stores documents; nothing joins them. Using a registered template means +fetching its content and posting it back. > **Proposed ticket:** *Instantiate a stored template by id* — -> `POST /templates/{id}/instantiate` taking `inputs`, resolving the stored -> content and provisioning through the existing path. **S** +> `POST /templates/{id}/instantiate` taking `inputs`. **S** **G3 — The dashboard cannot supply parameter inputs.** -The create-from-YAML page posts raw YAML with no `inputs` field, so from the UI -only a template whose every parameter has a default can be used. A template -built for per-incident particulars has no useful defaults. Parameters are -therefore reachable only from an API client — which, for a feature whose whole -point is that a human instantiates it, is the same as unreachable. - -One caveat to check before acting on this: the template linter's own comments -refer to "a document the Console wizard renders happily", implying a client that -does collect inputs. Nothing under `console/` on this branch posts to -`/rooms/from-yaml` or renders a `params:` block, so either that wizard is -unmerged or it lives somewhere this tree cannot see. If it ships, this gap -narrows to "the gateway cannot", which is much less serious. +The create-from-YAML page posts raw YAML with no `inputs`, so from the UI only a +template whose every parameter has a default works. An incident template has no +useful defaults. Caveat: the linter's own comments refer to "a document the +Console wizard renders happily", implying a client that does collect inputs; +nothing in this tree posts to `/rooms/from-yaml` or renders a `params:` block, so +either that wizard is unmerged or lives elsewhere. If it ships, this narrows to +"the gateway cannot". > **Proposed ticket:** *Parameter form for template instantiation* — render the -> declared `params:` as a form (using `description`, which is currently stored -> and never displayed), collect values, post the JSON body form. **S** +> declared `params:` as a form, using the `description` field that is currently +> stored and never displayed. **S** **G4 — A parameter cannot hold a list.** -`ParamSpec.type` is `string`, `number`, `boolean` or `enum`. A room's `agents:` -and `users:` are lists, so a variable-length membership cannot be parameterised -at all: `"alice,bob"` interpolates into one entry. In `users:` that entry -silently resolves to nobody and is reported as unresolved; in `agents:` it is a -hard `Unknown agents:` failure that aborts provisioning. For an incident template -whose responders differ every time, this is the difference between a template -that works and one that has to be edited before each use. - -> **Proposed ticket:** *List-typed template parameters* — add a `list` parameter -> type whose whole-field substitution splices into the surrounding list rather -> than stringifying. **M** - -**G9 — A room template is strictly less capable than the room creation it wraps.** +Types are `string`, `number`, `boolean`, `enum`. `agents:` and `users:` are +lists, so membership cannot be parameterised: `"alice,bob"` becomes one entry, +which in `users:` resolves to nobody and in `agents:` is a hard `Unknown agents:` +failure that aborts provisioning. + +> **Proposed ticket:** *List-typed template parameters* — whole-field +> substitution that splices into the surrounding list. **M** + +**G9 — A room template is strictly less capable than the room creation it +wraps.** `RoomCreateConfig` carries `aliases`, `linked_rooms`, `group_id`, `package_ids` -and `join_event_listeners`, and `create_room` accepts every one of them. The -template provisioner populates none. The first three arrive with group templates; -`join_event_listeners` arrives nowhere. Nothing here needs designing — the fields -already exist on the config object and are already validated — so this is a -pass-through, not a feature. +and `join_event_listeners`, and `create_room` accepts every one. The template +provisioner populates none. The first three arrive with group templates; +`join_event_listeners` arrives nowhere. Nothing needs designing — the fields +already exist and are already validated. + +> **Proposed ticket:** *Pass the remaining room fields through the template +> provisioner*, `join_event_listeners` first. **S** > **Proposed ticket:** *Land group templates* — merge > `origin/work/group-templates`: `group:`/`rooms:`/`links:`, per-room `aliases:`, > and dict-key interpolation. **M** -> **Proposed ticket:** *`join_event_listeners` in a room template* — a per-agent -> opt-in in the room spec. **S** - **G10 — Omitting `bridge:` silently means "the default bridge", and breaks `users:`.** The comment on the template's `bridge` field says to omit it for an internal-only -room. That is wrong: omitting it falls through to the instance default bridge — -and, less obviously, to no bridge when there is no default, or to a hard failure -when the default is configured but not running. (The `internal_only` field's own -documentation is accurate and says exactly this; the misleading comment is on the -template side.) Compounding it, the guard that rejects `users:` on an unbridged -room tests the template's own resolved bridge id, so a template with `users:` and -no `bridge:` is refused even though the room would have been bridged. - -> **Proposed ticket:** *Fix `bridge:` omission semantics in a room template* — -> resolve the default bridge before the `users:` guard, and add an explicit -> `internal_only:` key so "no channel" is stated rather than inferred. **S** - -**G-trap — a mistyped placeholder ships.** -Not a gap so much as a hazard worth writing down: a `{word}` that no parameter -declares is left verbatim, on purpose, so JSON braces in document content -survive. A typo in a placeholder name therefore does not error — it appears in -the created room. The linter catches some of this; the registry blocks only three -findings and treats the rest as advice. Lint before registering, and read the -output. - -### C. Getting the room made at all - -**G5 — No agent can instantiate a template, and no one can from a channel.** -The agent operation surface has 46 operations and not one of them touches -templates. The in-room command set has 21 and not one creates a room. - -Be precise about what this does and does not mean. An agent *can* open a war room -— `create_room` is an agent operation and takes agents, users, roles, references, -links, a group, aliases and join listeners. What it cannot do is open the room -*from the reviewed, version-controlled template*, which is the whole point of -having one. And an on-call engineer in the alert channel cannot declare an -incident from the channel they are already in; they have to leave Slack for an -API client or the dashboard, at the moment they least want to. +room. Omitting it falls through to the instance default — or to no bridge with no +default configured, or to a hard failure with a default configured but not +running. (`internal_only`'s own documentation is accurate; the misleading comment +is on the template side.) And the guard rejecting `users:` on an unbridged room +tests the template's resolved bridge id, so `users:` with no `bridge:` is refused +even though the room would have been bridged. + +> **Proposed ticket:** *Fix `bridge:` omission semantics* — resolve the default +> before the `users:` guard, and add an explicit `internal_only:` key. **S** + +**A hazard, not a gap.** A `{word}` no parameter declares is left verbatim, on +purpose, so JSON braces in document content survive. A typo in a placeholder name +does not error — it ships into the created room. Lint before registering; the +registry blocks only three findings and treats the rest as advice. + +### C. Driving the flow + +**G5 — No agent operation touches templates, and no channel command declares an +incident.** +The agent surface has 46 operations, none template-related; the in-room command +set has 21, none creating a room. An agent *can* open a war room — `create_room` +is an operation, which is what this design uses — but it cannot open one from the +reviewed, version-controlled template. And an on-call engineer in the hub cannot +declare from the channel with a command; they address the agent in prose, which +works, but is less discoverable than `/declare-incident`. > **Proposed ticket:** *`create_room_from_yaml` agent operation* — land the -> operation already written on `origin/work/group-templates`, extended to take a -> template id (needs G2). **S** +> operation on `origin/work/group-templates`, extended to take a template id +> (needs G2). **S** -> **Proposed ticket:** *`!declare-incident` in-room command* — instantiate a -> configured template from a bridged channel with positional inputs, and post -> the new room's link back. **M** +> **Proposed ticket:** *`!declare-incident` in-room command* — positional inputs, +> posts the new room's link back. **M** **G6 — There is no generic alert ingress.** -Switch does listen for inbound HTTP from a platform — the Teams bridge runs its -own endpoint and verifies the caller's signed token against the published keys, -so the machinery for authenticating an inbound webhook exists and is proven. -What does not exist is anything generic: no endpoint that accepts a third-party -alert payload and maps it to a Switch action. So the SOP's promise — "Switch will -auto-create a dedicated channel on declaration" — cannot be kept by Switch alone; -something outside has to hold a credential and call the API. That is workable and -probably correct for a first version, but it should be a decision rather than a -discovery. - -> **Proposed ticket:** *Incident intake webhook* — a signed inbound endpoint that -> maps an alerting payload to a template instantiation, with the field mapping -> configured per source. Depends on G2. **L** +Switch does listen for inbound HTTP from a platform and verify a signed caller — +the Teams bridge does exactly that — so the machinery exists. What does not is +anything generic: no endpoint accepting a third-party alert payload and mapping +it to a Switch action. This design does not need one (it declares by pull), but a +team wanting fully automatic room creation does. + +> **Proposed ticket:** *Incident intake webhook* — a signed inbound endpoint +> mapping an alerting payload to a room build, field mapping configured per +> source. **L** **G7 — There is no scheduling primitive a room or an agent can use.** -Switch runs periodic work internally — sweeps, renewals, batching timers — but -none of it is reachable from a room, and an agent cannot ask to be woken. The -SOP's hourly and four-hourly update cadence therefore lives in an external -scheduled workflow that mentions the agent — which works, and is the right -short-term answer, but has to be created per incident and nobody will remember to -delete it. +Switch runs periodic work internally; none of it is reachable from a room, and an +agent cannot ask to be woken. The SITREP cadence therefore lives in an external +scheduled workflow, created per incident, that nobody will remember to delete. -> **Proposed ticket:** *Scheduled room actions* — a room-scoped recurring -> trigger that posts a message or addresses an agent, created with the room and -> disposed of with it. **L** +> **Proposed ticket:** *Scheduled room actions* — a room-scoped recurring trigger +> that posts a message or addresses an agent, created with the room and disposed +> of with it. **L** **G8 — There is no relay between rooms.** -Linked rooms are metadata: a pointer with a label. There is no mechanism that -mirrors a message from one room into another. The SOP wants situation reports to -land in both the alert channel and the stakeholder channel, and the source -document asks directly whether that can be automated. Today an agent can read -another room without connecting to it, but posting requires connecting, which -means leaving the war room mid-incident. That is not a workaround anyone should -adopt. - -> **Proposed ticket:** *Mirror a message to a linked room* — an operation that -> posts to a room the agent is a member of without moving its connection, -> attributed and marked as a mirror. **M** +Linked rooms are metadata: a pointer with a label. The SOP wants situation reports +in both the hub and the stakeholder channel. An agent can read another room +without connecting, but posting requires connecting, which means leaving the war +room mid-incident. + +> **Proposed ticket:** *Mirror a message to a linked room* — post to a room the +> agent is a member of without moving its connection, attributed and marked as a +> mirror. **M** ### D. The shared agent **G11 — There is no provisionable service account.** -The recommendation in this document rests on owning the responder with a -non-person, non-admin user. Switch has exactly one shared-owner construct — the -synthetic bootstrap account — and on a password deployment nobody can sign in as -it. An admin can still *manage* its agents; what nobody can do is reveal their -credentials, because credential reveal is the one check with strict owner -equality and no admin bypass. The alternatives are to own the responder with a -real person (defeats the purpose) or with the Admin account (hands it a global -bypass over every room and resource in the tenant). **This is the gap the whole -responder design depends on.** +The recommendation rests on owning the responder with a non-person, non-admin +user. The only shared-owner construct is the synthetic bootstrap account, which +on a password deployment nobody can sign in as. An admin can still manage its +agents; nobody can reveal their credentials, because credential reveal is the one +check with strict owner equality and no admin bypass. The alternatives are a real +person (defeats the purpose) or the Admin account (a global bypass over every +room and resource). **This is the gap the responder design depends on.** > **Proposed ticket:** *Service accounts* — a non-interactive user that can own -> agents and resources, with authentication a team can hold jointly, and no -> admin role. **M** +> agents and resources, with authentication a team can hold jointly, and no admin +> role. **M** > **Proposed ticket:** *Transfer agent ownership* — an owner-or-admin endpoint -> setting `owner_id`. There is none today, so an agent registered under the wrong +> setting `owner_id`. There is none, so an agent registered under the wrong > account stays there. **S** -**G12 — The gateway's addressing-policy editor silently deletes owner rules.** -The React editor models only the four id-shaped dimensions and drops the symbolic -`owner` / `owner_agents` rules from any rule it saves. It does guard the extreme -case — an all-empty rule is flagged as unmatchable and Save is disabled — so the -agent cannot be bricked outright. The reachable damage is quieter: widening the -default owner-only policy by adding an allowed agent saves a policy that admits -that agent and no longer admits the owner. Switch Console's editor round-trips -the symbolic rules correctly. This is a live bug, and it sits directly on the -path of anyone widening a shared responder's policy. +**G12 — The gateway's addressing-policy editor drops owner rules.** +It models only the four id-shaped dimensions, so `owner` / `owner_agents` are +dropped from any rule it saves. It disables Save on an unmatchable rule, so the +agent cannot be bricked outright; the reachable damage is quieter — widening the +default owner-only policy by adding an allowed agent saves a policy that no +longer admits the owner. Switch Console's editor is correct. -> **Proposed ticket:** *Preserve symbolic rules in the gateway policy editor* — -> represent `owner` and `owner_agents`, round-trip them, and warn when a saved -> policy admits nobody. **S** +> **Proposed ticket:** *Preserve symbolic rules in the gateway policy editor*, +> and warn when a saved policy admits nobody. **S** **G13 — The offline nudge names the owner, not whoever can act.** -When an `auto_session` agent is addressed with nothing to start it, the room is -told to go and wake the owner. For a shared responder that is a service account -nobody watches, or a person who is not on call. The wording is right for a -personal agent and wrong for a shared one, and there is no way to override it. +An `auto_session` agent addressed with nothing to start it tells the room to go +and wake its owner. For a shared responder that is a service account nobody +watches. -> **Proposed ticket:** *Escalation target for an offline shared agent* — when an -> agent has no personal owner, address the nudge to the room's escalation target -> (see G1's follow-up) instead of to `owner_id`. **S** +> **Proposed ticket:** *Escalation target for an offline shared agent* — address +> the nudge to a room-configured target when the agent has no personal owner. +> **S** **G14 — One credential per agent; no rotation, no per-holder revocation.** -One agent, one API key row. No rotation endpoint — the only rotation is -re-registration with overwrite, which breaks every holder simultaneously. Reveal -is strict owner equality with no admin bypass. A shared agent therefore has a -credential that cannot be issued per person, cannot be revoked per person, and -cannot be recovered by anyone but its owner. +One key row per agent. No rotation endpoint — only re-registration with +overwrite, which breaks every holder at once. Reveal is owner-only. The token is a +bearer credential in a plaintext file in the working directory. > **Proposed ticket:** *Per-holder agent credentials* — several named, > independently revocable keys per agent, each attributable, with a rotation > endpoint that does not break the others. **M** **G15 — Nothing records which human drove a session.** -No actor field on connections, sessions, runtime state, leases or messages. A -shared agent's actions are attributable to the agent and to nobody else. The -mitigation in this design is the rule that the agent acts only on a written -request in the room — which makes the transcript the audit log — but that is a +No actor on connections, sessions, runtime state, leases or messages. A shared +agent's actions are attributable to the agent and nobody else. Mitigated here by convention, and conventions are not enforcement. > **Proposed ticket:** *Record the operator behind a session* — capture an actor -> on session registration and carry it onto messages the session sends. **M** +> at session registration and carry it onto messages that session sends. **M** **G16 — A role lease is held per agent, globally.** -Unique on the agent, not on the room and not on the session. One agent can hold -one role across the whole instance, so a shared agent in two concurrent incidents -can be the scribe in only one. And two sessions of the same agent assuming the -same role is an idempotent re-assume, so roles arbitrate nothing between them. +Unique on the agent, not the room and not the session. One agent holds one role +across the whole instance, so a responder holding `responder` in the hub cannot +also hold `scribe` in a war room. Two sessions of one agent assuming the same +role is an idempotent re-assume, so roles arbitrate nothing between them. -> **Proposed ticket:** *Scope a role lease to (agent, room)* — allow one agent to -> hold a role in each of several rooms, and decide explicitly what two sessions -> of one agent assuming one role should mean. **M** +> **Proposed ticket:** *Scope a role lease to (agent, room)* — and decide +> explicitly what two sessions of one agent assuming one role should mean. **M** **G17 — Role eligibility is declared and unused; humans cannot hold roles.** `RoomRole.eligibility` exists, is documented as a forward-looking hook, and is read by nothing — any room member may assume any role. And roles are assumable -only by agents, so "incident commander" cannot be a role at all. +only by agents, so "incident commander" cannot be one. -> **Proposed ticket:** *Enforce role eligibility* — implement the declared field -> so a role can be restricted. **S** +> **Proposed ticket:** *Enforce role eligibility.* **S** > **Proposed ticket:** *Human-holdable roles* — let a person claim a room role > from the bridged channel, so `@incident-commander` reaches a human. **L** -### E. Closing the incident out +### E. Third-party capability + +**G19 — MCP is per-machine, not per-agent.** +Every connector plugin bundles one MCP server; every provider declares MCP scope +as `global` and the capability schema admits no other value. Switch Console +writes a per-agent launch profile carrying model, effort and instructions, and +deliberately registers no MCP server; the MCP management UI was removed and the +remaining config adapters have no live callers. So giving the responder PagerDuty +means giving it to every agent on that host. Workable — run the responder on its +own host — but it is why "give this one agent a tool" is a machine-provisioning +task rather than a Switch setting. + +> **Proposed ticket:** *Per-agent MCP servers* — a per-agent scope in the +> capability schema and a writer per provider. Note Codex refuses to load a +> config that layers a base entry onto a plugin-provided server, so this is not +> uniform across providers. **L** + +**G20 — There is no agent-scoped secret storage.** +Switch encrypts its own API keys and bridge tokens; a server-side connector's +config sits in plain JSONB. There is nothing for a third-party credential +belonging to one agent. A PagerDuty token lives in the host environment, or in +Switch Console's per-provider environment map, which is plaintext and shared +across every agent of that provider. Switch neither scopes, rotates nor audits +it. + +> **Proposed ticket:** *Agent-scoped third-party credentials* — encrypted at +> rest, injected into that agent's sessions only, revocable independently of the +> agent's own key. **M** + +### F. Fidelity of app-posted messages + +**G21 — A third-party app's message reaches Switch lossily, and its edits not at +all.** +Rich Slack blocks are read only when the message has no plain-text body, and even +then only `section`, `header` and `rich_text` — `context` and `actions` blocks +are discarded, which is where PagerDuty puts service, urgency, assignee and its +buttons. Attachments are read only if no block yielded text. And +`message_changed` / `message_deleted` are dropped entirely, so an alert edited in +place to "Resolved" leaves Switch's copy saying it is open. A message whose +readable body comes out empty is still relayed, as an empty post. + +This design routes around it by declaring through a human rather than parsing +alerts, but any team that wires alerts straight into a room will hit it. + +> **Proposed ticket:** *Extend Block Kit extraction* — read `context` blocks and +> merge attachments rather than treating them as a fallback; drop a message whose +> extracted body is empty rather than relaying it. **S** + +> **Proposed ticket:** *Bridge message edits* — relay `message_changed` as an +> edit, or at minimum as a new message noting the original was amended. **M** + +### G. Closing the incident out **G18 — There is no transcript export.** -The postmortem is written from the room, but there is no endpoint that produces a -room's history: the gateway exposes a room's *configuration* as YAML and nothing -else, and reading messages is an agent-only operation. In practice the responder -agent can page back through the room and post a timeline as an attachment, which -is good enough — so this is the cheapest gap on the list and the least urgent. +The postmortem is written from the room, but no endpoint produces a room's +history: the gateway exposes a room's *configuration* as YAML and nothing else, +and reading messages is an agent-only operation. In practice the responder can +page back through the room and post a timeline as an attachment, which is good +enough — the cheapest gap here and the least urgent. > **Proposed ticket:** *Export a room transcript* — a downloadable, paginated > history export for a room a user can read. **S** ## What to build first -Nothing on that list blocks a first incident. Two things are worth being explicit -about: - -**Usable on day one, with no Switch change at all.** Write the template, register -it, and have one person instantiate it through the API when an incident is -declared, pasting the incident's particulars as inputs. Invite responders by -hand. Put the update cadence in a scheduled Slack workflow that mentions the -responder. Register the responder agent, widen its addressing policy through the -API — not the dashboard, see G12 — and run its watcher on an always-on host. That -is a working SOP on Switch, with two manual steps. - -**The order to remove the manual steps in**, by value per unit of work: - -1. **G2 + G3 — instantiate a stored template, with a form.** Two small changes - that together turn the registry from a filing cabinet into the feature. Until - these land, every other template improvement is invisible to the people who - would use it. Start here. -2. **G11 — a service account.** Small in scope, and the recommendation for the - responder agent is unsound without it. Every day it is missing is a day the - responder is either one person's agent or an admin. -3. **G12 — the policy editor bug.** A few hours' work, and it will otherwise be - discovered by someone widening the responder's policy during an incident. -4. **G1 — resolve a platform group to room members.** The largest single - reduction in manual work: it turns "invite the on-call engineers" from a - human step into a template line, and it is the question the SOP has been - asking. Deliberately without building a rotation in Switch. -5. **G5 — declare an incident from the channel.** Once the template instantiates - cleanly, letting an engineer trigger it from Slack removes the last manual - step in the critical path. -6. **G9 — group templates, and join-event listeners.** The postmortem room, the - links, the `@responder` alias, and the ability to greet an arrival. All - quality, none of it blocking. +**Nothing.** That is the main finding, and it changed during the design. + +Because the room is built by an agent calling `create_room`, and because +PagerDuty is reached the way Jira already is, **this SOP can run on Switch today +with no change to Switch at all.** Standing it up is configuration and one agent +definition: + +1. Adopt the product's alert channel as the incident hub. Write its instruction + card: the SOP, the room-writing rules, the bindings block. +2. Define the exclusive `responder` role in the hub. +3. Register the responder agent, widen its addressing policy **through the API, + not the dashboard** (G12), and run its watcher on an always-on host. +4. Install a PagerDuty MCP server on that host and put a `pagerduty` reference + type and its references in the hub. +5. Register the room YAML as a template so the shape is reviewable — as + documentation, not as the mechanism. +6. Put the SITREP cadence in a scheduled Slack workflow that mentions the agent. + +The one compromise in that list is agent ownership: until G11 exists, the +responder is owned by a person or by Admin, and neither is right. + +**Then, in order of value per unit of work:** + +1. **G11 — a service account.** Small, and the recommendation is unsound without + it. Every day it is missing is a day the responder is either one person's + agent or an admin. +2. **G12 — the policy editor bug.** Hours of work, and it otherwise gets + discovered by whoever widens the responder's policy, during an incident. +3. **G9 — pass the remaining room fields through the template provisioner**, + `join_event_listeners` first. Small, and it is what closes the distance + between the reviewable artifact and the capable one. +4. **G1 — surface unresolved invitees.** Small, and it turns "the war room quietly + came up one person short" into something someone notices. +5. **G8 — mirror to a linked room.** Removes the SOP's most tedious manual step, + posting the same situation report into two channels. +6. **G7 — scheduled room actions.** Retires the per-incident Slack workflow. 7. Everything else, as it starts to hurt. -The honest summary: **Switch can host this SOP today, badly, with two manual -steps and one unsafe compromise on agent ownership. Items 1 to 3 make it -respectable, and they are small. Item 4 is the one the team actually asked for.** +The honest summary: **the design needs no Switch changes to run, one small change +to be safe, and two more to be pleasant.** The template work is worth doing on its +own merits, and this SOP is not blocked on any of it.