From 2c000ae9c302a02b5a06f8521507b1e21beb11b5 Mon Sep 17 00:00:00 2001 From: Josh Dougall Date: Mon, 24 Aug 2026 11:11:57 -0700 Subject: [PATCH 1/4] Refresh the infra pointer pages against upstream The three pointer pages into ChainSafe/infrastructure-general had drifted from the repo they contract with. Two of them carried a status note, live since 20 July, saying heading anchors were pending confirmation. The runbooks already had per-alert headings, so this was confirm-and-link. infrastructure-and-devops.md - Add terragrunt/ as the live home for all Terraform (#1400, closed) and mark terraform/ legacy (#1416). Previously terraform/ was presented as the live answer, so an agent following the map wrote new stacks into the tree being decommissioned. - Correct the stale enumerations. The terraform/ list named data-analytics, Gossamer, Sygma and infra-prod, none of which exist at that path any more; the ansible/ list named Gossamer and Polkadot, which are gone, Forest, which moved to ansible/_OLD/, and omitted ssv. Both are now described structurally rather than enumerated, because a directory snapshot rots monthly (#1238). - Fix the images/ entry to the real upstream directory name. - Link the five observability docs that existed upstream but were unlinked: monitoring-overview, alerting-and-oncall, metrics, logging, tracing. - Drop the status note. incident-response.md - Add a by-alert-name table: 29 alert anchors across infrastructure, lodestar, filecoin, optimism and ipfs-gateway runbooks. Paged operators match the alert name and land on its section rather than a file top. Anchors were generated from the upstream headings, not hand-written. - Add rds-bastion.md, the eleventh runbook, which was missing. The page tells readers to escalate unlisted scenarios, so the omission generated false gap reports. - Deepen the Forest upgrade row to its silence/rollout/verify steps. - Link docs/observability/alerting-and-oncall.md. This page decided when to page without ever linking how paging is wired. - Drop the status note. release-and-deploy.md - Add infra-kubernetes, where Canton production deploys actually ship from (ArgoCD + Helm). The page mapped Canton only to the procedure docs. - Add DNS-Management, Cloudflare DNS as Terraform for every ChainSafe domain, previously absent from the handbook entirely. - Point the IaC row at terragrunt/ for the same reason as above. lychee.toml - Exclude infra-kubernetes and DNS-Management. Both are private, so a repo-scoped GITHUB_TOKEN gets a 404 and link-check would fail on the two links added above. Same reason infrastructure-general is already excluded. Verified against origin/main rather than a local checkout: 90 upstream file and directory targets, 35 heading anchors, and 30 handbook-local links all resolve. Note that link-check cannot cover the infrastructure-general links because that repo is private and excluded, so this was checked by hand. Leaves the post-incident review location alone; that needs a decision first. --- lychee.toml | 4 +++ workflows/incident-response.md | 46 +++++++++++++++++++++++--- workflows/infrastructure-and-devops.md | 14 +++++--- workflows/release-and-deploy.md | 6 ++-- 4 files changed, 59 insertions(+), 11 deletions(-) diff --git a/lychee.toml b/lychee.toml index 895c16a..cc8cce2 100644 --- a/lychee.toml +++ b/lychee.toml @@ -55,6 +55,10 @@ exclude = [ '(^|//)([^/]+\.)?chainsafe\.tech\b', '^https://forms\.gle', '^https://github.com/ChainSafe/infrastructure-general', + # infra-kubernetes and DNS-Management are private too, so an unauthenticated + # (or repo-scoped GITHUB_TOKEN) probe gets a 404. Same reason as the line above. + '^https://github.com/ChainSafe/infra-kubernetes', + '^https://github.com/ChainSafe/DNS-Management', ] # No excluded paths — all markdown is in scope. diff --git a/workflows/incident-response.md b/workflows/incident-response.md index d7564c0..5f673ea 100644 --- a/workflows/incident-response.md +++ b/workflows/incident-response.md @@ -4,8 +4,6 @@ This handbook holds only the **operator decision policy** layer for incidents. T > **In one line:** When to page, when to roll back, who approves a recovery action. The *how* defers to the runbooks. -> **Status note.** The operator-decision-policy section below is complete. The [runbook deep-link map](#runbook-deep-link-map) is confirmed at file level; heading anchors *within* the runbooks are pending [@joshdougall](https://github.com/joshdougall)'s confirmation. Per the deep-link convention, where the upstream lacks an anchor we need for clean linking, the convention is to add the anchor upstream rather than work around it here. - ## Operator decision policy These are the calls a human makes during an incident. The runbooks tell you the mechanics; this page tells you the decisions. @@ -23,6 +21,8 @@ Page the on-call (and yourself) when any of these are true: Do not page for transient blips that auto-recover within the runbook's threshold. The runbooks define those thresholds; defer to them. +For how paging is *wired* — Alertmanager routing, PagerDuty services, escalation policies, and schedules — see [`docs/observability/alerting-and-oncall.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/alerting-and-oncall.md). This page decides *when* to page; that one describes the mechanism that delivers it. + ### When to roll back Roll back when: @@ -52,6 +52,42 @@ Rollback decisions are owned by the on-call operator in consultation with the ch ## Runbook deep-link map +### By alert name + +If you were paged, match the alert name here and land on its section, not the top of a file. + +| Alert fired | Runbook section | +|---|---| +| `BeaconNodeMemoryLeakDetected` | [`lodestar-alerts.md#beaconnodememoryleakdetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#beaconnodememoryleakdetected) | +| `FilecoinForestSyncingFail` | [`filecoin-alerts.md#filecoinforestsyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinforestsyncingfail) | +| `FilecoinLotusSyncingFail` | [`filecoin-alerts.md#filecoinlotussyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinlotussyncingfail) | +| `FilecoinPeerConnected` | [`filecoin-alerts.md#filecoinpeerconnected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinpeerconnected) | +| `FilecoinSnapshotAgeOld` | [`filecoin-alerts.md#filecoinsnapshotageold`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinsnapshotageold) | +| `ForestTipsetsValidatedPerMinute` | [`filecoin-alerts.md#foresttipsetsvalidatedperminute`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#foresttipsetsvalidatedperminute) | +| `HostDiskWillFillIn24Hours` | [`infrastructure-alerts.md#hostdiskwillfillin24hours`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostdiskwillfillin24hours) | +| `HostOomKillDetected` | [`infrastructure-alerts.md#hostoomkilldetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoomkilldetected) | +| `HostOutOfDiskSpace` | [`infrastructure-alerts.md#hostoutofdiskspace`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofdiskspace) | +| `HostOutOfDiskSpaceCritical` | [`infrastructure-alerts.md#hostoutofdiskspacecritical`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofdiskspacecritical) | +| `HostOutOfInodes` | [`infrastructure-alerts.md#hostoutofinodes`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofinodes) | +| `HostOutOfMemory` | [`infrastructure-alerts.md#hostoutofmemory`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofmemory) | +| `HostRequiresReboot` | [`infrastructure-alerts.md#hostrequiresreboot`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostrequiresreboot) | +| `individual_validator_losing_balance` | [`lodestar-alerts.md#individual_validator_losing_balance`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#individual_validator_losing_balance) | +| `InstanceDown` | [`infrastructure-alerts.md#instancedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#instancedown) | +| `IpfsGatewayDown` | [`ipfs-gateway-operations.md#ipfsgatewaydown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewaydown) | +| `IpfsGatewayHighErrorRate` | [`ipfs-gateway-operations.md#ipfsgatewayhigherrorrate`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewayhigherrorrate) | +| `IpfsGatewayHighLatency` | [`ipfs-gateway-operations.md#ipfsgatewayhighlatency`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewayhighlatency) | +| `IpfsKuboDiskUsageHigh` | [`ipfs-gateway-operations.md#ipfskubodiskusagehigh`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubodiskusagehigh) | +| `IpfsKuboNodeDown` | [`ipfs-gateway-operations.md#ipfskubonodedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubonodedown) | +| `IpfsKuboPeerCountLow` | [`ipfs-gateway-operations.md#ipfskubopeercountlow`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubopeercountlow) | +| `LowExitMessagesLeft` | [`lodestar-alerts.md#lowexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#lowexitmessagesleft) | +| `missed_attestations_in_mass` | [`lodestar-alerts.md#missed_attestations_in_mass`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#missed_attestations_in_mass) | +| `NoExitMessagesLeft` | [`lodestar-alerts.md#noexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#noexitmessagesleft) | +| `SnapshotServiceDown` | [`filecoin-alerts.md#snapshotservicedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#snapshotservicedown) | +| `StuckBeaconNode` | [`lodestar-alerts.md#stuckbeaconnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#stuckbeaconnode) | +| `StuckOPNode` | [`optimism-alerts.md#stuckopnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md#stuckopnode) | +| `SystemMemoryLeakDetected` | [`infrastructure-alerts.md#systemmemoryleakdetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#systemmemoryleakdetected) | +| `ValidatorMissedBlock` | [`lodestar-alerts.md#validatormissedblock`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#validatormissedblock) | + ### By chain / product | Scenario | Runbook | @@ -59,15 +95,16 @@ Rollback decisions are owned by the on-call operator in consultation with the ch | General infrastructure alerts (cross-product) | [`infrastructure-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md) | | Ethereum / Lodestar alerts | [`lodestar-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md) | | Filecoin alerts | [`filecoin-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md) | -| Forest upgrade procedures | [`forest-upgrade-procedures.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md) | +| Forest upgrade procedures | [`forest-upgrade-procedures.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md) — [silence alerts first](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-1-silence-alerts), [production rollout](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-3-production-rollout), [health verification](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-4-health-verification) | | Polkadot alerts | [`polkadot-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/polkadot-alerts.md) | | Optimism alerts | [`optimism-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md) | | Lido validator operations | [`lido-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-validator-operations.md) | | Rocketpool node operations | [`rocketpool-node-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rocketpool-node-operations.md) | | IPFS gateway operations | [`ipfs-gateway-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md) | | Canton unclaimed rewards | [`canton-unclaimed-rewards.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/canton-unclaimed-rewards.md) | +| RDS bastion (infra-dev): deploy, tunnel, teardown | [`rds-bastion.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md) — [deploy](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#deploy), [DB tunnel](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#db-tunnel), [teardown](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#teardown) | -If your scenario isn't listed: the runbook may not exist yet. Surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call; do not improvise from this page. +All eleven runbooks upstream are mapped above. If your scenario isn't listed, the runbook does not exist yet: surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. ## Agent role during an incident @@ -98,3 +135,4 @@ Agents can draft the document from chat logs, runbook executions, and PR history - [`../operating-model/gates-and-escalation.md`](../operating-model/gates-and-escalation.md) — the gates incidents put under stress. - [`../operating-model/collaborator-statement.md`](../operating-model/collaborator-statement.md) — operator-first applies under time pressure too. - Upstream: [`ChainSafe/infrastructure-general/docs/runbooks/`](https://github.com/ChainSafe/infrastructure-general/tree/main/docs/runbooks) — the runbooks themselves. +- Upstream: [`docs/observability/alerting-and-oncall.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/alerting-and-oncall.md) — how paging is wired: Alertmanager routing, PagerDuty services, schedules. diff --git a/workflows/infrastructure-and-devops.md b/workflows/infrastructure-and-devops.md index f4041f2..76c27ef 100644 --- a/workflows/infrastructure-and-devops.md +++ b/workflows/infrastructure-and-devops.md @@ -4,8 +4,6 @@ For all infrastructure, IaC, deployment topology, observability, on-call, and De > **In one line:** This page is a navigation surface, not a tutorial. Deep links by intent into the canonical repo; no "see also" gestures. -> **Status note.** File-level deep-link targets on this page are confirmed against the current state of `infrastructure-general/docs/`. Heading anchors *within* those files are pending [@joshdougall](https://github.com/joshdougall)'s confirmation. Where the upstream lacks an anchor we need for clean linking, the convention is to add the anchor upstream rather than work around it here. - ## Why we defer here `infrastructure-general` is the authoritative artifact for ChainSafe infrastructure. It already ships its own [`AGENTS.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/AGENTS.md) and [`CLAUDE.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/CLAUDE.md), confirming the agent-native posture. The handbook treats it as canonical for any question whose answer involves how production systems are built, run, observed, or recovered. @@ -25,14 +23,20 @@ Maintained by [@joshdougall](https://github.com/joshdougall) (Head of Infra). Co | How do we triage infra backlog? | [`docs/backlog-triage.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/backlog-triage.md) | | What's our monitoring footprint? | [`docs/observability/monitoring-inventory.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/monitoring-inventory.md) | | How do we profile production services? | [`docs/observability/profiling.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/profiling.md) | +| How is our monitoring stack put together? | [`docs/observability/monitoring-overview.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/monitoring-overview.md) | +| How is alerting and on-call actually wired? | [`docs/observability/alerting-and-oncall.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/alerting-and-oncall.md) | +| How are metrics collected and stored? | [`docs/observability/metrics.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/metrics.md) | +| How is logging wired? | [`docs/observability/logging.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/logging.md) | +| How is tracing wired? | [`docs/observability/tracing.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/tracing.md) | ### By configuration domain | Need to change… | Go to (upstream) | |---|---| -| Ansible roles / config management | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) — 9 execution directories (Ethereum, Filecoin, Forest, Gossamer, IPFS, OP, Polkadot, zkVerify, general) | -| Terraform / cloud provisioning | [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) — Auth0, data-analytics, Forest, Gossamer, Grafana Cloud, infra-dev, infra-prod, k8s, Sygma | -| Docker images we build | [`images/`](https://github.com/ChainSafe/infrastructure-general/tree/main/images) — filecoin-bootnode-monitor, grafana-alloy, nebula, polkadot-crunch, snapshot-service | +| Ansible roles / config management | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) — one self-contained execution directory per project, each with its own `ansible.cfg`, inventory, `group_vars/`, and `Makefile`. Being collapsed onto the consolidated `ansible/general/` pattern with YAML inventory ([#1238](https://github.com/ChainSafe/infrastructure-general/issues/1238)); superseded directories move to `ansible/_OLD/`. Read the tree, not a list here — it changes monthly. | +| Terraform / cloud provisioning | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — **the live home for all Terraform.** `_modules/` holds the actual `.tf`; `stacks/` holds leaf `terragrunt.hcl` only, grouped `aws//` and `saas//`. Per-account S3 state bootstrapped from `_bootstrap/tf-state/`. Consolidation epic [#1400](https://github.com/ChainSafe/infrastructure-general/issues/1400) is closed. | +| Legacy Terraform (do not add to) | [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) — **legacy, being decommissioned** ([#1416](https://github.com/ChainSafe/infrastructure-general/issues/1416)). New stacks go in `terragrunt/`. | +| Docker images we build | [`images/`](https://github.com/ChainSafe/infrastructure-general/tree/main/images) — `filecoin-boonode-monitor` (spelling is upstream's), `grafana-alloy`, `nebula`, `polkadot-crunch`, `snapshot-service` | | Internal tooling and scripts | [`tools/`](https://github.com/ChainSafe/infrastructure-general/tree/main/tools) | ### By product diff --git a/workflows/release-and-deploy.md b/workflows/release-and-deploy.md index 6b47ac0..1111a8a 100644 --- a/workflows/release-and-deploy.md +++ b/workflows/release-and-deploy.md @@ -57,14 +57,16 @@ Deploy procedures are product-specific and live in `infrastructure-general`: | Product / area | Procedure location | |---|---| -| Canton (k8s) | [`docs/projects/canton/canton-k8s-deployment.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-k8s-deployment.md), [`canton-deploy-new-app.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-deploy-new-app.md) | +| Canton (k8s) — procedure | [`docs/projects/canton/canton-k8s-deployment.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-k8s-deployment.md), [`canton-deploy-new-app.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-deploy-new-app.md) | +| Canton (k8s) — manifests | [`ChainSafe/infra-kubernetes`](https://github.com/ChainSafe/infra-kubernetes) — Canton **production** deploys land here (ArgoCD + Helm). The procedure docs above describe the change; this repo is where it ships from. | | Forest staging environment | [`docs/projects/filecoin/forest-staging-environment.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/forest-staging-environment.md) | | Lodestar production operations | [`docs/projects/ethereum/lodestar-production-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ethereum/lodestar-production-operations.md) | | Lodestar public services | [`docs/projects/ethereum/lodestar-public-services.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ethereum/lodestar-public-services.md) | | SSV operator onboarding | [`docs/projects/ssv/onboarding-validator.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ssv/onboarding-validator.md), [`operator-registration.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ssv/operator-registration.md) | | Faucet operations | [`docs/projects/filecoin/faucet-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/faucet-operations.md) | | Snapshot service | [`docs/projects/filecoin/snapshot-service.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/snapshot-service.md) | -| IaC / Terraform changes | [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) — each subdirectory has its own README | +| IaC / Terraform changes | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — the live home for all Terraform. Legacy [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) is being decommissioned ([#1416](https://github.com/ChainSafe/infrastructure-general/issues/1416)); do not add stacks there. | +| DNS changes (all ChainSafe domains) | [`ChainSafe/DNS-Management`](https://github.com/ChainSafe/DNS-Management) — Cloudflare DNS as Terraform. Slated to fold into `infrastructure-general/terragrunt/` per that tree's README. | | Ansible-driven deploys | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) | If a procedure isn't listed: it may not exist yet. Surface to [@joshdougall](https://github.com/joshdougall) before improvising. From 0f65a99371538b2be9aa183843825a68282a6de3 Mon Sep 17 00:00:00 2001 From: Josh Dougall Date: Tue, 1 Sep 2026 09:33:37 -0700 Subject: [PATCH 2/4] Re-verify against upstream: two new runbooks, DNS-Management scope infrastructure-general moved by ~25 commits between drafting this and re-checking it. Two claims had gone stale. - Map besu-blob-gc-disk-fill.md and celestia-validator-operations.md, both added upstream in the last week, and correct "eleven runbooks" to thirteen. The page tells readers to escalate unlisted scenarios, so a missing runbook produces false gap reports. - DNS-Management is explicitly out of scope for the repo consolidation (#1110), settled upstream in #1504 on 1 September. The earlier wording here said it was slated to fold into terragrunt/, which is the exact claim that PR removed for contradicting #1110. Re-verified against origin/main: 97 file and directory targets, 40 heading anchors, 30 handbook-local links, 0 failures. --- workflows/incident-response.md | 4 +++- workflows/release-and-deploy.md | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/workflows/incident-response.md b/workflows/incident-response.md index 5f673ea..fd2a523 100644 --- a/workflows/incident-response.md +++ b/workflows/incident-response.md @@ -103,8 +103,10 @@ If you were paged, match the alert name here and land on its section, not the to | IPFS gateway operations | [`ipfs-gateway-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md) | | Canton unclaimed rewards | [`canton-unclaimed-rewards.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/canton-unclaimed-rewards.md) | | RDS bastion (infra-dev): deploy, tunnel, teardown | [`rds-bastion.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md) — [deploy](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#deploy), [DB tunnel](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#db-tunnel), [teardown](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#teardown) | +| Besu RocksDB blob-GC disk fill (Lido EL fleet) | [`besu-blob-gc-disk-fill.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md) — [recognising it](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#recognising-it), [remediation](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#remediation), [do not do these](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#do-not-do-these) | +| Celestia validator operations | [`celestia-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md) — [sync status](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-node-sync-status), [commission](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-unclaimed-commission) | -All eleven runbooks upstream are mapped above. If your scenario isn't listed, the runbook does not exist yet: surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. +All thirteen runbooks upstream are mapped above. If your scenario isn't listed, the runbook does not exist yet: surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. ## Agent role during an incident diff --git a/workflows/release-and-deploy.md b/workflows/release-and-deploy.md index 1111a8a..1bbf218 100644 --- a/workflows/release-and-deploy.md +++ b/workflows/release-and-deploy.md @@ -66,7 +66,7 @@ Deploy procedures are product-specific and live in `infrastructure-general`: | Faucet operations | [`docs/projects/filecoin/faucet-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/faucet-operations.md) | | Snapshot service | [`docs/projects/filecoin/snapshot-service.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/snapshot-service.md) | | IaC / Terraform changes | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — the live home for all Terraform. Legacy [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) is being decommissioned ([#1416](https://github.com/ChainSafe/infrastructure-general/issues/1416)); do not add stacks there. | -| DNS changes (all ChainSafe domains) | [`ChainSafe/DNS-Management`](https://github.com/ChainSafe/DNS-Management) — Cloudflare DNS as Terraform. Slated to fold into `infrastructure-general/terragrunt/` per that tree's README. | +| DNS changes (all ChainSafe domains) | [`ChainSafe/DNS-Management`](https://github.com/ChainSafe/DNS-Management) — Cloudflare DNS as Terraform, 7 zones. Org-wide rather than Infra-owned: Infra maintains it, engineering across the org uses it. Explicitly **out of scope** for the infra repo consolidation ([#1110](https://github.com/ChainSafe/infrastructure-general/issues/1110)); it does not fold into `terragrunt/`. | | Ansible-driven deploys | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) | If a procedure isn't listed: it may not exist yet. Surface to [@joshdougall](https://github.com/joshdougall) before improvising. From 7889fe034df735c9724a7947436cedfbb55a315f Mon Sep 17 00:00:00 2001 From: Josh Dougall Date: Mon, 21 Sep 2026 08:05:18 -0700 Subject: [PATCH 3/4] Re-verify against upstream after 136 commits, and complete the runbook map The previous verification was 2026-09-01. infrastructure-general has had 136 commits to main since, and four references had broken. That repo is private and excluded in lychee.toml, so link-check cannot catch any of this; manual re-verification is the only defence and it has a short shelf life. Broken and now fixed: - filecoin-alerts.md#filecoinsnapshotageold. The heading upstream is the combined "FilecoinSnapshotAgeWarning / FilecoinSnapshotAgeOld", so the anchor never resolved. Both alert names now share the one correct anchor. - filecoin-alerts.md#snapshotservicedown. No such heading, and no SnapshotServiceDown alert anywhere on main. Row removed rather than pointed somewhere plausible. - polkadot-alerts.md. Deleted upstream by #1649. There is no polkadot execution directory and no polkadot alert rule left, so the row is gone rather than repointed. - terraform/ as a tree link on two pages. The directory has been removed from main. infrastructure-and-devops.md loses the legacy row entirely; release-and-deploy.md keeps the row and states the tree is gone. The map was also incomplete, which matters more than the dead links. The page tells a paged operator that an unlisted scenario means the runbook does not exist, so a missing entry sends them to improvise. Eighteen runbooks exist upstream and only twelve were mapped. Added the six that were missing, including dirk-alerts.md, vouch-alerts.md and lido-nom-phone-escalation.md, which cover Lido production signing and the NOM phone path. The by-alert table gains the sixteen Dirk and Vouch alert anchors so those pages are reachable by alert name like the rest. Corrected the count claim from thirteen to eighteen. Re-verified against origin/main: 79 file references, 40 heading anchors, 15 directory references, 0 failures, and the runbook map now matches the directory exactly in both directions. --- workflows/incident-response.md | 28 ++++++++++++++++++++++---- workflows/infrastructure-and-devops.md | 1 - workflows/release-and-deploy.md | 2 +- 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/workflows/incident-response.md b/workflows/incident-response.md index fd2a523..a9d45e8 100644 --- a/workflows/incident-response.md +++ b/workflows/incident-response.md @@ -59,10 +59,16 @@ If you were paged, match the alert name here and land on its section, not the to | Alert fired | Runbook section | |---|---| | `BeaconNodeMemoryLeakDetected` | [`lodestar-alerts.md#beaconnodememoryleakdetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#beaconnodememoryleakdetected) | +| `DirkAccountUnlockFailed` | [`dirk-alerts.md#dirkaccountunlockfailed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkaccountunlockfailed) | +| `DirkClientPermissionsMissing` | [`dirk-alerts.md#dirkclientpermissionsmissing`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkclientpermissionsmissing) | +| `DirkDeniedRatioHigh` | [`dirk-alerts.md#dirkdeniedratiohigh`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkdeniedratiohigh) | +| `DirkQuorumLost` | [`dirk-alerts.md#dirkquorumlost`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkquorumlost) | +| `DirkSignerDegraded` | [`dirk-alerts.md#dirksignerdegraded`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirksignerdegraded) | +| `DirkSignerErrors` | [`dirk-alerts.md#dirksignererrors`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirksignererrors) | | `FilecoinForestSyncingFail` | [`filecoin-alerts.md#filecoinforestsyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinforestsyncingfail) | | `FilecoinLotusSyncingFail` | [`filecoin-alerts.md#filecoinlotussyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinlotussyncingfail) | | `FilecoinPeerConnected` | [`filecoin-alerts.md#filecoinpeerconnected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinpeerconnected) | -| `FilecoinSnapshotAgeOld` | [`filecoin-alerts.md#filecoinsnapshotageold`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinsnapshotageold) | +| `FilecoinSnapshotAgeWarning`, `FilecoinSnapshotAgeOld` | [`filecoin-alerts.md#filecoinsnapshotagewarning--filecoinsnapshotageold`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinsnapshotagewarning--filecoinsnapshotageold) | | `ForestTipsetsValidatedPerMinute` | [`filecoin-alerts.md#foresttipsetsvalidatedperminute`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#foresttipsetsvalidatedperminute) | | `HostDiskWillFillIn24Hours` | [`infrastructure-alerts.md#hostdiskwillfillin24hours`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostdiskwillfillin24hours) | | `HostOomKillDetected` | [`infrastructure-alerts.md#hostoomkilldetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoomkilldetected) | @@ -82,11 +88,20 @@ If you were paged, match the alert name here and land on its section, not the to | `LowExitMessagesLeft` | [`lodestar-alerts.md#lowexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#lowexitmessagesleft) | | `missed_attestations_in_mass` | [`lodestar-alerts.md#missed_attestations_in_mass`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#missed_attestations_in_mass) | | `NoExitMessagesLeft` | [`lodestar-alerts.md#noexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#noexitmessagesleft) | -| `SnapshotServiceDown` | [`filecoin-alerts.md#snapshotservicedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#snapshotservicedown) | | `StuckBeaconNode` | [`lodestar-alerts.md#stuckbeaconnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#stuckbeaconnode) | | `StuckOPNode` | [`optimism-alerts.md#stuckopnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md#stuckopnode) | | `SystemMemoryLeakDetected` | [`infrastructure-alerts.md#systemmemoryleakdetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#systemmemoryleakdetected) | | `ValidatorMissedBlock` | [`lodestar-alerts.md#validatormissedblock`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#validatormissedblock) | +| `VouchAccountsUnknownAfterActivation` | [`vouch-alerts.md#vouchaccountsunknownafteractivation`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchaccountsunknownafteractivation) | +| `VouchAccountViewDiverged` | [`vouch-alerts.md#vouchaccountviewdiverged`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchaccountviewdiverged) | +| `VouchAttestationsLate` | [`vouch-alerts.md#vouchattestationslate`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchattestationslate) | +| `VouchBeaconNodeFailing` | [`vouch-alerts.md#vouchbeaconnodefailing`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchbeaconnodefailing) | +| `VouchNoAttestationsSigned` | [`vouch-alerts.md#vouchnoattestationssigned`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchnoattestationssigned) | +| `VouchNoEpochsProcessed` | [`vouch-alerts.md#vouchnoepochsprocessed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchnoepochsprocessed) | +| `VouchNotReady` | [`vouch-alerts.md#vouchnotready`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchnotready) | +| `VouchProposalMissed` | [`vouch-alerts.md#vouchproposalmissed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchproposalmissed) | +| `VouchValidatorExiting` | [`vouch-alerts.md#vouchvalidatorexiting`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchvalidatorexiting) | +| `VouchValidatorSlashed` | [`vouch-alerts.md#vouchvalidatorslashed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchvalidatorslashed) | ### By chain / product @@ -96,17 +111,22 @@ If you were paged, match the alert name here and land on its section, not the to | Ethereum / Lodestar alerts | [`lodestar-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md) | | Filecoin alerts | [`filecoin-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md) | | Forest upgrade procedures | [`forest-upgrade-procedures.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md) — [silence alerts first](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-1-silence-alerts), [production rollout](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-3-production-rollout), [health verification](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-4-health-verification) | -| Polkadot alerts | [`polkadot-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/polkadot-alerts.md) | | Optimism alerts | [`optimism-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md) | | Lido validator operations | [`lido-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-validator-operations.md) | | Rocketpool node operations | [`rocketpool-node-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rocketpool-node-operations.md) | | IPFS gateway operations | [`ipfs-gateway-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md) | | Canton unclaimed rewards | [`canton-unclaimed-rewards.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/canton-unclaimed-rewards.md) | +| Canton DAR proxy identity | [`canton-dar-proxy-identity.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/canton-dar-proxy-identity.md) | +| Lido Dirk/Vouch signing operations | [`lido-dirk-vouch-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-dirk-vouch-operations.md) | +| Lido NOM phone escalation | [`lido-nom-phone-escalation.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-nom-phone-escalation.md) | +| Dirk signer alerts | [`dirk-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md) | +| Vouch validator-client alerts | [`vouch-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md) | +| Nethermind full-pruning disk fill | [`nethermind-fullpruning-disk-fill.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/nethermind-fullpruning-disk-fill.md) | | RDS bastion (infra-dev): deploy, tunnel, teardown | [`rds-bastion.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md) — [deploy](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#deploy), [DB tunnel](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#db-tunnel), [teardown](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#teardown) | | Besu RocksDB blob-GC disk fill (Lido EL fleet) | [`besu-blob-gc-disk-fill.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md) — [recognising it](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#recognising-it), [remediation](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#remediation), [do not do these](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#do-not-do-these) | | Celestia validator operations | [`celestia-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md) — [sync status](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-node-sync-status), [commission](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-unclaimed-commission) | -All thirteen runbooks upstream are mapped above. If your scenario isn't listed, the runbook does not exist yet: surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. +All eighteen runbooks upstream are mapped above. If your scenario isn't listed, the runbook does not exist yet: surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. ## Agent role during an incident diff --git a/workflows/infrastructure-and-devops.md b/workflows/infrastructure-and-devops.md index 76c27ef..13466e5 100644 --- a/workflows/infrastructure-and-devops.md +++ b/workflows/infrastructure-and-devops.md @@ -35,7 +35,6 @@ Maintained by [@joshdougall](https://github.com/joshdougall) (Head of Infra). Co |---|---| | Ansible roles / config management | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) — one self-contained execution directory per project, each with its own `ansible.cfg`, inventory, `group_vars/`, and `Makefile`. Being collapsed onto the consolidated `ansible/general/` pattern with YAML inventory ([#1238](https://github.com/ChainSafe/infrastructure-general/issues/1238)); superseded directories move to `ansible/_OLD/`. Read the tree, not a list here — it changes monthly. | | Terraform / cloud provisioning | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — **the live home for all Terraform.** `_modules/` holds the actual `.tf`; `stacks/` holds leaf `terragrunt.hcl` only, grouped `aws//` and `saas//`. Per-account S3 state bootstrapped from `_bootstrap/tf-state/`. Consolidation epic [#1400](https://github.com/ChainSafe/infrastructure-general/issues/1400) is closed. | -| Legacy Terraform (do not add to) | [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) — **legacy, being decommissioned** ([#1416](https://github.com/ChainSafe/infrastructure-general/issues/1416)). New stacks go in `terragrunt/`. | | Docker images we build | [`images/`](https://github.com/ChainSafe/infrastructure-general/tree/main/images) — `filecoin-boonode-monitor` (spelling is upstream's), `grafana-alloy`, `nebula`, `polkadot-crunch`, `snapshot-service` | | Internal tooling and scripts | [`tools/`](https://github.com/ChainSafe/infrastructure-general/tree/main/tools) | diff --git a/workflows/release-and-deploy.md b/workflows/release-and-deploy.md index 1bbf218..798e9cb 100644 --- a/workflows/release-and-deploy.md +++ b/workflows/release-and-deploy.md @@ -65,7 +65,7 @@ Deploy procedures are product-specific and live in `infrastructure-general`: | SSV operator onboarding | [`docs/projects/ssv/onboarding-validator.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ssv/onboarding-validator.md), [`operator-registration.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ssv/operator-registration.md) | | Faucet operations | [`docs/projects/filecoin/faucet-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/faucet-operations.md) | | Snapshot service | [`docs/projects/filecoin/snapshot-service.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/snapshot-service.md) | -| IaC / Terraform changes | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — the live home for all Terraform. Legacy [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) is being decommissioned ([#1416](https://github.com/ChainSafe/infrastructure-general/issues/1416)); do not add stacks there. | +| IaC / Terraform changes | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — the live home for all Terraform. The legacy `terraform/` tree has been removed from `main`. | | DNS changes (all ChainSafe domains) | [`ChainSafe/DNS-Management`](https://github.com/ChainSafe/DNS-Management) — Cloudflare DNS as Terraform, 7 zones. Org-wide rather than Infra-owned: Infra maintains it, engineering across the org uses it. Explicitly **out of scope** for the infra repo consolidation ([#1110](https://github.com/ChainSafe/infrastructure-general/issues/1110)); it does not fold into `terragrunt/`. | | Ansible-driven deploys | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) | From f73caab7d00605ce2823219c6f827539b21e1cb7 Mon Sep 17 00:00:00 2001 From: Josh Dougall Date: Mon, 21 Sep 2026 08:57:49 -0700 Subject: [PATCH 4/4] Use the real alert names, and stop claiming the alert index is complete Review findings from an independent codex pass on gpt-6-astra, each verified against upstream before applying. The left column of the by-alert table is what an operator reads off their page, so it has to be the alertname Prometheus fires. Three rows carried the runbook heading text instead: - FilecoinLotusSyncingFail is really FilecoinlotusSyncingFail, lowercase l. The typo is upstream in prom_rules.yml:162 and the page had silently corrected it into something nobody will ever be paged with. - individual_validator_losing_balance is really IndividualValidatorLosingBalance (prom_rules.yml:260). - missed_attestations_in_mass is really MissedAttestationsInMass (prom_rules.yml:247). All three are live critical rules with high-urgency PagerDuty routes on Filecoin and Ethereum/Lido mainnet. The runbook anchors were correct and are unchanged; only the names an operator matches on have been fixed. The closing line also asserted that an unlisted scenario means the runbook does not exist. That is true of the scenario table, which is now 18 of 18, and false of the by-alert index, which has never been exhaustive: Aztec, SSV, Celestia and several Filecoin rules page without a row here. Telling a paged operator that no runbook exists when one does is worse than saying nothing, so the claim is now scoped to the table it actually holds for, and points at the chain runbook before the escalation. Polkadot dropped from the chain example list, since its runbook and rules were removed with the chain. The two lychee exclusions were unanchored and left the dot in github.com unescaped, so they also suppressed any repository whose name merely starts with infra-kubernetes or DNS-Management. Both now match the exact repository and its sub-paths only. Full alert-index coverage is deliberately not in this change. Reconciling the index against the Prometheus rules, Alertmanager routes and the Grafana Terraform is a separate audit, roughly forty rows across Aztec, SSV, Celestia and Canton. --- lychee.toml | 4 ++-- workflows/incident-response.md | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/lychee.toml b/lychee.toml index cc8cce2..255095c 100644 --- a/lychee.toml +++ b/lychee.toml @@ -57,8 +57,8 @@ exclude = [ '^https://github.com/ChainSafe/infrastructure-general', # infra-kubernetes and DNS-Management are private too, so an unauthenticated # (or repo-scoped GITHUB_TOKEN) probe gets a 404. Same reason as the line above. - '^https://github.com/ChainSafe/infra-kubernetes', - '^https://github.com/ChainSafe/DNS-Management', + '^https://github\.com/ChainSafe/infra-kubernetes([/?#]|$)', + '^https://github\.com/ChainSafe/DNS-Management([/?#]|$)', ] # No excluded paths — all markdown is in scope. diff --git a/workflows/incident-response.md b/workflows/incident-response.md index a9d45e8..82967bb 100644 --- a/workflows/incident-response.md +++ b/workflows/incident-response.md @@ -17,7 +17,7 @@ Page the on-call (and yourself) when any of these are true: - A monitoring alert fires that the [`infrastructure-alerts`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md) runbook classifies as page-worthy. - Funds, assets, or signing keys are at risk. - A security event is suspected (suspicious access, leaked credential, exploited vulnerability). -- The chain you're operating against is in a degraded state and the runbook for that chain (Polkadot, Ethereum, Filecoin, etc.) calls for it. +- The chain you're operating against is in a degraded state and the runbook for that chain (Ethereum, Filecoin, Celestia, etc.) calls for it. Do not page for transient blips that auto-recover within the runbook's threshold. The runbooks define those thresholds; defer to them. @@ -66,7 +66,7 @@ If you were paged, match the alert name here and land on its section, not the to | `DirkSignerDegraded` | [`dirk-alerts.md#dirksignerdegraded`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirksignerdegraded) | | `DirkSignerErrors` | [`dirk-alerts.md#dirksignererrors`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirksignererrors) | | `FilecoinForestSyncingFail` | [`filecoin-alerts.md#filecoinforestsyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinforestsyncingfail) | -| `FilecoinLotusSyncingFail` | [`filecoin-alerts.md#filecoinlotussyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinlotussyncingfail) | +| `FilecoinlotusSyncingFail` | [`filecoin-alerts.md#filecoinlotussyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinlotussyncingfail) | | `FilecoinPeerConnected` | [`filecoin-alerts.md#filecoinpeerconnected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinpeerconnected) | | `FilecoinSnapshotAgeWarning`, `FilecoinSnapshotAgeOld` | [`filecoin-alerts.md#filecoinsnapshotagewarning--filecoinsnapshotageold`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinsnapshotagewarning--filecoinsnapshotageold) | | `ForestTipsetsValidatedPerMinute` | [`filecoin-alerts.md#foresttipsetsvalidatedperminute`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#foresttipsetsvalidatedperminute) | @@ -77,7 +77,7 @@ If you were paged, match the alert name here and land on its section, not the to | `HostOutOfInodes` | [`infrastructure-alerts.md#hostoutofinodes`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofinodes) | | `HostOutOfMemory` | [`infrastructure-alerts.md#hostoutofmemory`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofmemory) | | `HostRequiresReboot` | [`infrastructure-alerts.md#hostrequiresreboot`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostrequiresreboot) | -| `individual_validator_losing_balance` | [`lodestar-alerts.md#individual_validator_losing_balance`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#individual_validator_losing_balance) | +| `IndividualValidatorLosingBalance` | [`lodestar-alerts.md#individual_validator_losing_balance`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#individual_validator_losing_balance) | | `InstanceDown` | [`infrastructure-alerts.md#instancedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#instancedown) | | `IpfsGatewayDown` | [`ipfs-gateway-operations.md#ipfsgatewaydown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewaydown) | | `IpfsGatewayHighErrorRate` | [`ipfs-gateway-operations.md#ipfsgatewayhigherrorrate`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewayhigherrorrate) | @@ -86,7 +86,7 @@ If you were paged, match the alert name here and land on its section, not the to | `IpfsKuboNodeDown` | [`ipfs-gateway-operations.md#ipfskubonodedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubonodedown) | | `IpfsKuboPeerCountLow` | [`ipfs-gateway-operations.md#ipfskubopeercountlow`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubopeercountlow) | | `LowExitMessagesLeft` | [`lodestar-alerts.md#lowexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#lowexitmessagesleft) | -| `missed_attestations_in_mass` | [`lodestar-alerts.md#missed_attestations_in_mass`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#missed_attestations_in_mass) | +| `MissedAttestationsInMass` | [`lodestar-alerts.md#missed_attestations_in_mass`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#missed_attestations_in_mass) | | `NoExitMessagesLeft` | [`lodestar-alerts.md#noexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#noexitmessagesleft) | | `StuckBeaconNode` | [`lodestar-alerts.md#stuckbeaconnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#stuckbeaconnode) | | `StuckOPNode` | [`optimism-alerts.md#stuckopnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md#stuckopnode) | @@ -126,7 +126,7 @@ If you were paged, match the alert name here and land on its section, not the to | Besu RocksDB blob-GC disk fill (Lido EL fleet) | [`besu-blob-gc-disk-fill.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md) — [recognising it](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#recognising-it), [remediation](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#remediation), [do not do these](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#do-not-do-these) | | Celestia validator operations | [`celestia-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md) — [sync status](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-node-sync-status), [commission](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-unclaimed-commission) | -All eighteen runbooks upstream are mapped above. If your scenario isn't listed, the runbook does not exist yet: surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. +All eighteen runbooks upstream are mapped above, so the **scenario** table is complete. The **by alert name** index above it is not: alerts exist that have no row here, including the Aztec, SSV, Celestia and several Filecoin rules. So a missing alert row does not mean the runbook is missing. Check the runbook for that chain or product first, then surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. ## Agent role during an incident