diff --git a/lychee.toml b/lychee.toml index 895c16a..255095c 100644 --- a/lychee.toml +++ b/lychee.toml @@ -55,6 +55,10 @@ exclude = [ '(^|//)([^/]+\.)?chainsafe\.tech\b', '^https://forms\.gle', '^https://github.com/ChainSafe/infrastructure-general', + # infra-kubernetes and DNS-Management are private too, so an unauthenticated + # (or repo-scoped GITHUB_TOKEN) probe gets a 404. Same reason as the line above. + '^https://github\.com/ChainSafe/infra-kubernetes([/?#]|$)', + '^https://github\.com/ChainSafe/DNS-Management([/?#]|$)', ] # No excluded paths — all markdown is in scope. diff --git a/workflows/incident-response.md b/workflows/incident-response.md index d7564c0..82967bb 100644 --- a/workflows/incident-response.md +++ b/workflows/incident-response.md @@ -4,8 +4,6 @@ This handbook holds only the **operator decision policy** layer for incidents. T > **In one line:** When to page, when to roll back, who approves a recovery action. The *how* defers to the runbooks. -> **Status note.** The operator-decision-policy section below is complete. The [runbook deep-link map](#runbook-deep-link-map) is confirmed at file level; heading anchors *within* the runbooks are pending [@joshdougall](https://github.com/joshdougall)'s confirmation. Per the deep-link convention, where the upstream lacks an anchor we need for clean linking, the convention is to add the anchor upstream rather than work around it here. - ## Operator decision policy These are the calls a human makes during an incident. The runbooks tell you the mechanics; this page tells you the decisions. @@ -19,10 +17,12 @@ Page the on-call (and yourself) when any of these are true: - A monitoring alert fires that the [`infrastructure-alerts`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md) runbook classifies as page-worthy. - Funds, assets, or signing keys are at risk. - A security event is suspected (suspicious access, leaked credential, exploited vulnerability). -- The chain you're operating against is in a degraded state and the runbook for that chain (Polkadot, Ethereum, Filecoin, etc.) calls for it. +- The chain you're operating against is in a degraded state and the runbook for that chain (Ethereum, Filecoin, Celestia, etc.) calls for it. Do not page for transient blips that auto-recover within the runbook's threshold. The runbooks define those thresholds; defer to them. +For how paging is *wired* — Alertmanager routing, PagerDuty services, escalation policies, and schedules — see [`docs/observability/alerting-and-oncall.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/alerting-and-oncall.md). This page decides *when* to page; that one describes the mechanism that delivers it. + ### When to roll back Roll back when: @@ -52,6 +52,57 @@ Rollback decisions are owned by the on-call operator in consultation with the ch ## Runbook deep-link map +### By alert name + +If you were paged, match the alert name here and land on its section, not the top of a file. + +| Alert fired | Runbook section | +|---|---| +| `BeaconNodeMemoryLeakDetected` | [`lodestar-alerts.md#beaconnodememoryleakdetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#beaconnodememoryleakdetected) | +| `DirkAccountUnlockFailed` | [`dirk-alerts.md#dirkaccountunlockfailed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkaccountunlockfailed) | +| `DirkClientPermissionsMissing` | [`dirk-alerts.md#dirkclientpermissionsmissing`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkclientpermissionsmissing) | +| `DirkDeniedRatioHigh` | [`dirk-alerts.md#dirkdeniedratiohigh`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkdeniedratiohigh) | +| `DirkQuorumLost` | [`dirk-alerts.md#dirkquorumlost`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirkquorumlost) | +| `DirkSignerDegraded` | [`dirk-alerts.md#dirksignerdegraded`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirksignerdegraded) | +| `DirkSignerErrors` | [`dirk-alerts.md#dirksignererrors`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md#dirksignererrors) | +| `FilecoinForestSyncingFail` | [`filecoin-alerts.md#filecoinforestsyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinforestsyncingfail) | +| `FilecoinlotusSyncingFail` | [`filecoin-alerts.md#filecoinlotussyncingfail`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinlotussyncingfail) | +| `FilecoinPeerConnected` | [`filecoin-alerts.md#filecoinpeerconnected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinpeerconnected) | +| `FilecoinSnapshotAgeWarning`, `FilecoinSnapshotAgeOld` | [`filecoin-alerts.md#filecoinsnapshotagewarning--filecoinsnapshotageold`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#filecoinsnapshotagewarning--filecoinsnapshotageold) | +| `ForestTipsetsValidatedPerMinute` | [`filecoin-alerts.md#foresttipsetsvalidatedperminute`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md#foresttipsetsvalidatedperminute) | +| `HostDiskWillFillIn24Hours` | [`infrastructure-alerts.md#hostdiskwillfillin24hours`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostdiskwillfillin24hours) | +| `HostOomKillDetected` | [`infrastructure-alerts.md#hostoomkilldetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoomkilldetected) | +| `HostOutOfDiskSpace` | [`infrastructure-alerts.md#hostoutofdiskspace`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofdiskspace) | +| `HostOutOfDiskSpaceCritical` | [`infrastructure-alerts.md#hostoutofdiskspacecritical`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofdiskspacecritical) | +| `HostOutOfInodes` | [`infrastructure-alerts.md#hostoutofinodes`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofinodes) | +| `HostOutOfMemory` | [`infrastructure-alerts.md#hostoutofmemory`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostoutofmemory) | +| `HostRequiresReboot` | [`infrastructure-alerts.md#hostrequiresreboot`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#hostrequiresreboot) | +| `IndividualValidatorLosingBalance` | [`lodestar-alerts.md#individual_validator_losing_balance`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#individual_validator_losing_balance) | +| `InstanceDown` | [`infrastructure-alerts.md#instancedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#instancedown) | +| `IpfsGatewayDown` | [`ipfs-gateway-operations.md#ipfsgatewaydown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewaydown) | +| `IpfsGatewayHighErrorRate` | [`ipfs-gateway-operations.md#ipfsgatewayhigherrorrate`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewayhigherrorrate) | +| `IpfsGatewayHighLatency` | [`ipfs-gateway-operations.md#ipfsgatewayhighlatency`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfsgatewayhighlatency) | +| `IpfsKuboDiskUsageHigh` | [`ipfs-gateway-operations.md#ipfskubodiskusagehigh`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubodiskusagehigh) | +| `IpfsKuboNodeDown` | [`ipfs-gateway-operations.md#ipfskubonodedown`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubonodedown) | +| `IpfsKuboPeerCountLow` | [`ipfs-gateway-operations.md#ipfskubopeercountlow`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md#ipfskubopeercountlow) | +| `LowExitMessagesLeft` | [`lodestar-alerts.md#lowexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#lowexitmessagesleft) | +| `MissedAttestationsInMass` | [`lodestar-alerts.md#missed_attestations_in_mass`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#missed_attestations_in_mass) | +| `NoExitMessagesLeft` | [`lodestar-alerts.md#noexitmessagesleft`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#noexitmessagesleft) | +| `StuckBeaconNode` | [`lodestar-alerts.md#stuckbeaconnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#stuckbeaconnode) | +| `StuckOPNode` | [`optimism-alerts.md#stuckopnode`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md#stuckopnode) | +| `SystemMemoryLeakDetected` | [`infrastructure-alerts.md#systemmemoryleakdetected`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md#systemmemoryleakdetected) | +| `ValidatorMissedBlock` | [`lodestar-alerts.md#validatormissedblock`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md#validatormissedblock) | +| `VouchAccountsUnknownAfterActivation` | [`vouch-alerts.md#vouchaccountsunknownafteractivation`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchaccountsunknownafteractivation) | +| `VouchAccountViewDiverged` | [`vouch-alerts.md#vouchaccountviewdiverged`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchaccountviewdiverged) | +| `VouchAttestationsLate` | [`vouch-alerts.md#vouchattestationslate`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchattestationslate) | +| `VouchBeaconNodeFailing` | [`vouch-alerts.md#vouchbeaconnodefailing`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchbeaconnodefailing) | +| `VouchNoAttestationsSigned` | [`vouch-alerts.md#vouchnoattestationssigned`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchnoattestationssigned) | +| `VouchNoEpochsProcessed` | [`vouch-alerts.md#vouchnoepochsprocessed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchnoepochsprocessed) | +| `VouchNotReady` | [`vouch-alerts.md#vouchnotready`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchnotready) | +| `VouchProposalMissed` | [`vouch-alerts.md#vouchproposalmissed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchproposalmissed) | +| `VouchValidatorExiting` | [`vouch-alerts.md#vouchvalidatorexiting`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchvalidatorexiting) | +| `VouchValidatorSlashed` | [`vouch-alerts.md#vouchvalidatorslashed`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md#vouchvalidatorslashed) | + ### By chain / product | Scenario | Runbook | @@ -59,15 +110,23 @@ Rollback decisions are owned by the on-call operator in consultation with the ch | General infrastructure alerts (cross-product) | [`infrastructure-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/infrastructure-alerts.md) | | Ethereum / Lodestar alerts | [`lodestar-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lodestar-alerts.md) | | Filecoin alerts | [`filecoin-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/filecoin-alerts.md) | -| Forest upgrade procedures | [`forest-upgrade-procedures.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md) | -| Polkadot alerts | [`polkadot-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/polkadot-alerts.md) | +| Forest upgrade procedures | [`forest-upgrade-procedures.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md) — [silence alerts first](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-1-silence-alerts), [production rollout](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-3-production-rollout), [health verification](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/forest-upgrade-procedures.md#step-4-health-verification) | | Optimism alerts | [`optimism-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/optimism-alerts.md) | | Lido validator operations | [`lido-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-validator-operations.md) | | Rocketpool node operations | [`rocketpool-node-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rocketpool-node-operations.md) | | IPFS gateway operations | [`ipfs-gateway-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/ipfs-gateway-operations.md) | | Canton unclaimed rewards | [`canton-unclaimed-rewards.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/canton-unclaimed-rewards.md) | - -If your scenario isn't listed: the runbook may not exist yet. Surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call; do not improvise from this page. +| Canton DAR proxy identity | [`canton-dar-proxy-identity.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/canton-dar-proxy-identity.md) | +| Lido Dirk/Vouch signing operations | [`lido-dirk-vouch-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-dirk-vouch-operations.md) | +| Lido NOM phone escalation | [`lido-nom-phone-escalation.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/lido-nom-phone-escalation.md) | +| Dirk signer alerts | [`dirk-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/dirk-alerts.md) | +| Vouch validator-client alerts | [`vouch-alerts.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/vouch-alerts.md) | +| Nethermind full-pruning disk fill | [`nethermind-fullpruning-disk-fill.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/nethermind-fullpruning-disk-fill.md) | +| RDS bastion (infra-dev): deploy, tunnel, teardown | [`rds-bastion.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md) — [deploy](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#deploy), [DB tunnel](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#db-tunnel), [teardown](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/rds-bastion.md#teardown) | +| Besu RocksDB blob-GC disk fill (Lido EL fleet) | [`besu-blob-gc-disk-fill.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md) — [recognising it](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#recognising-it), [remediation](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#remediation), [do not do these](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/besu-blob-gc-disk-fill.md#do-not-do-these) | +| Celestia validator operations | [`celestia-validator-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md) — [sync status](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-node-sync-status), [commission](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/runbooks/celestia-validator-operations.md#check-unclaimed-commission) | + +All eighteen runbooks upstream are mapped above, so the **scenario** table is complete. The **by alert name** index above it is not: alerts exist that have no row here, including the Aztec, SSV, Celestia and several Filecoin rules. So a missing alert row does not mean the runbook is missing. Check the runbook for that chain or product first, then surface the gap to [@joshdougall](https://github.com/joshdougall) and the on-call, and do not improvise from this page. ## Agent role during an incident @@ -98,3 +157,4 @@ Agents can draft the document from chat logs, runbook executions, and PR history - [`../operating-model/gates-and-escalation.md`](../operating-model/gates-and-escalation.md) — the gates incidents put under stress. - [`../operating-model/collaborator-statement.md`](../operating-model/collaborator-statement.md) — operator-first applies under time pressure too. - Upstream: [`ChainSafe/infrastructure-general/docs/runbooks/`](https://github.com/ChainSafe/infrastructure-general/tree/main/docs/runbooks) — the runbooks themselves. +- Upstream: [`docs/observability/alerting-and-oncall.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/alerting-and-oncall.md) — how paging is wired: Alertmanager routing, PagerDuty services, schedules. diff --git a/workflows/infrastructure-and-devops.md b/workflows/infrastructure-and-devops.md index f4041f2..13466e5 100644 --- a/workflows/infrastructure-and-devops.md +++ b/workflows/infrastructure-and-devops.md @@ -4,8 +4,6 @@ For all infrastructure, IaC, deployment topology, observability, on-call, and De > **In one line:** This page is a navigation surface, not a tutorial. Deep links by intent into the canonical repo; no "see also" gestures. -> **Status note.** File-level deep-link targets on this page are confirmed against the current state of `infrastructure-general/docs/`. Heading anchors *within* those files are pending [@joshdougall](https://github.com/joshdougall)'s confirmation. Where the upstream lacks an anchor we need for clean linking, the convention is to add the anchor upstream rather than work around it here. - ## Why we defer here `infrastructure-general` is the authoritative artifact for ChainSafe infrastructure. It already ships its own [`AGENTS.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/AGENTS.md) and [`CLAUDE.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/CLAUDE.md), confirming the agent-native posture. The handbook treats it as canonical for any question whose answer involves how production systems are built, run, observed, or recovered. @@ -25,14 +23,19 @@ Maintained by [@joshdougall](https://github.com/joshdougall) (Head of Infra). Co | How do we triage infra backlog? | [`docs/backlog-triage.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/backlog-triage.md) | | What's our monitoring footprint? | [`docs/observability/monitoring-inventory.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/monitoring-inventory.md) | | How do we profile production services? | [`docs/observability/profiling.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/profiling.md) | +| How is our monitoring stack put together? | [`docs/observability/monitoring-overview.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/monitoring-overview.md) | +| How is alerting and on-call actually wired? | [`docs/observability/alerting-and-oncall.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/alerting-and-oncall.md) | +| How are metrics collected and stored? | [`docs/observability/metrics.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/metrics.md) | +| How is logging wired? | [`docs/observability/logging.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/logging.md) | +| How is tracing wired? | [`docs/observability/tracing.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/observability/tracing.md) | ### By configuration domain | Need to change… | Go to (upstream) | |---|---| -| Ansible roles / config management | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) — 9 execution directories (Ethereum, Filecoin, Forest, Gossamer, IPFS, OP, Polkadot, zkVerify, general) | -| Terraform / cloud provisioning | [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) — Auth0, data-analytics, Forest, Gossamer, Grafana Cloud, infra-dev, infra-prod, k8s, Sygma | -| Docker images we build | [`images/`](https://github.com/ChainSafe/infrastructure-general/tree/main/images) — filecoin-bootnode-monitor, grafana-alloy, nebula, polkadot-crunch, snapshot-service | +| Ansible roles / config management | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) — one self-contained execution directory per project, each with its own `ansible.cfg`, inventory, `group_vars/`, and `Makefile`. Being collapsed onto the consolidated `ansible/general/` pattern with YAML inventory ([#1238](https://github.com/ChainSafe/infrastructure-general/issues/1238)); superseded directories move to `ansible/_OLD/`. Read the tree, not a list here — it changes monthly. | +| Terraform / cloud provisioning | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — **the live home for all Terraform.** `_modules/` holds the actual `.tf`; `stacks/` holds leaf `terragrunt.hcl` only, grouped `aws//` and `saas//`. Per-account S3 state bootstrapped from `_bootstrap/tf-state/`. Consolidation epic [#1400](https://github.com/ChainSafe/infrastructure-general/issues/1400) is closed. | +| Docker images we build | [`images/`](https://github.com/ChainSafe/infrastructure-general/tree/main/images) — `filecoin-boonode-monitor` (spelling is upstream's), `grafana-alloy`, `nebula`, `polkadot-crunch`, `snapshot-service` | | Internal tooling and scripts | [`tools/`](https://github.com/ChainSafe/infrastructure-general/tree/main/tools) | ### By product diff --git a/workflows/release-and-deploy.md b/workflows/release-and-deploy.md index 6b47ac0..798e9cb 100644 --- a/workflows/release-and-deploy.md +++ b/workflows/release-and-deploy.md @@ -57,14 +57,16 @@ Deploy procedures are product-specific and live in `infrastructure-general`: | Product / area | Procedure location | |---|---| -| Canton (k8s) | [`docs/projects/canton/canton-k8s-deployment.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-k8s-deployment.md), [`canton-deploy-new-app.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-deploy-new-app.md) | +| Canton (k8s) — procedure | [`docs/projects/canton/canton-k8s-deployment.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-k8s-deployment.md), [`canton-deploy-new-app.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/canton/canton-deploy-new-app.md) | +| Canton (k8s) — manifests | [`ChainSafe/infra-kubernetes`](https://github.com/ChainSafe/infra-kubernetes) — Canton **production** deploys land here (ArgoCD + Helm). The procedure docs above describe the change; this repo is where it ships from. | | Forest staging environment | [`docs/projects/filecoin/forest-staging-environment.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/forest-staging-environment.md) | | Lodestar production operations | [`docs/projects/ethereum/lodestar-production-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ethereum/lodestar-production-operations.md) | | Lodestar public services | [`docs/projects/ethereum/lodestar-public-services.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ethereum/lodestar-public-services.md) | | SSV operator onboarding | [`docs/projects/ssv/onboarding-validator.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ssv/onboarding-validator.md), [`operator-registration.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/ssv/operator-registration.md) | | Faucet operations | [`docs/projects/filecoin/faucet-operations.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/faucet-operations.md) | | Snapshot service | [`docs/projects/filecoin/snapshot-service.md`](https://github.com/ChainSafe/infrastructure-general/blob/main/docs/projects/filecoin/snapshot-service.md) | -| IaC / Terraform changes | [`terraform/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terraform) — each subdirectory has its own README | +| IaC / Terraform changes | [`terragrunt/`](https://github.com/ChainSafe/infrastructure-general/tree/main/terragrunt) — the live home for all Terraform. The legacy `terraform/` tree has been removed from `main`. | +| DNS changes (all ChainSafe domains) | [`ChainSafe/DNS-Management`](https://github.com/ChainSafe/DNS-Management) — Cloudflare DNS as Terraform, 7 zones. Org-wide rather than Infra-owned: Infra maintains it, engineering across the org uses it. Explicitly **out of scope** for the infra repo consolidation ([#1110](https://github.com/ChainSafe/infrastructure-general/issues/1110)); it does not fold into `terragrunt/`. | | Ansible-driven deploys | [`ansible/`](https://github.com/ChainSafe/infrastructure-general/tree/main/ansible) | If a procedure isn't listed: it may not exist yet. Surface to [@joshdougall](https://github.com/joshdougall) before improvising.