Refresh the infra pointer pages and clear the pending-confirmation notes - #127
Open
joshdougall wants to merge 4 commits into
Open
joshdougall wants to merge 4 commits into
joshdougall wants to merge 4 commits into
Conversation
The three pointer pages into ChainSafe/infrastructure-general had drifted from the repo they contract with. Two of them carried a status note, live since 20 July, saying heading anchors were pending confirmation. The runbooks already had per-alert headings, so this was confirm-and-link. infrastructure-and-devops.md - Add terragrunt/ as the live home for all Terraform (#1400, closed) and mark terraform/ legacy (#1416). Previously terraform/ was presented as the live answer, so an agent following the map wrote new stacks into the tree being decommissioned. - Correct the stale enumerations. The terraform/ list named data-analytics, Gossamer, Sygma and infra-prod, none of which exist at that path any more; the ansible/ list named Gossamer and Polkadot, which are gone, Forest, which moved to ansible/_OLD/, and omitted ssv. Both are now described structurally rather than enumerated, because a directory snapshot rots monthly (#1238). - Fix the images/ entry to the real upstream directory name. - Link the five observability docs that existed upstream but were unlinked: monitoring-overview, alerting-and-oncall, metrics, logging, tracing. - Drop the status note. incident-response.md - Add a by-alert-name table: 29 alert anchors across infrastructure, lodestar, filecoin, optimism and ipfs-gateway runbooks. Paged operators match the alert name and land on its section rather than a file top. Anchors were generated from the upstream headings, not hand-written. - Add rds-bastion.md, the eleventh runbook, which was missing. The page tells readers to escalate unlisted scenarios, so the omission generated false gap reports. - Deepen the Forest upgrade row to its silence/rollout/verify steps. - Link docs/observability/alerting-and-oncall.md. This page decided when to page without ever linking how paging is wired. - Drop the status note. release-and-deploy.md - Add infra-kubernetes, where Canton production deploys actually ship from (ArgoCD + Helm). The page mapped Canton only to the procedure docs. - Add DNS-Management, Cloudflare DNS as Terraform for every ChainSafe domain, previously absent from the handbook entirely. - Point the IaC row at terragrunt/ for the same reason as above. lychee.toml - Exclude infra-kubernetes and DNS-Management. Both are private, so a repo-scoped GITHUB_TOKEN gets a 404 and link-check would fail on the two links added above. Same reason infrastructure-general is already excluded. Verified against origin/main rather than a local checkout: 90 upstream file and directory targets, 35 heading anchors, and 30 handbook-local links all resolve. Note that link-check cannot cover the infrastructure-general links because that repo is private and excluded, so this was checked by hand. Leaves the post-incident review location alone; that needs a decision first.
infrastructure-general moved by ~25 commits between drafting this and re-checking it. Two claims had gone stale. - Map besu-blob-gc-disk-fill.md and celestia-validator-operations.md, both added upstream in the last week, and correct "eleven runbooks" to thirteen. The page tells readers to escalate unlisted scenarios, so a missing runbook produces false gap reports. - DNS-Management is explicitly out of scope for the repo consolidation (#1110), settled upstream in #1504 on 1 September. The earlier wording here said it was slated to fold into terragrunt/, which is the exact claim that PR removed for contradicting #1110. Re-verified against origin/main: 97 file and directory targets, 40 heading anchors, 30 handbook-local links, 0 failures.
…k map The previous verification was 2026-09-01. infrastructure-general has had 136 commits to main since, and four references had broken. That repo is private and excluded in lychee.toml, so link-check cannot catch any of this; manual re-verification is the only defence and it has a short shelf life. Broken and now fixed: - filecoin-alerts.md#filecoinsnapshotageold. The heading upstream is the combined "FilecoinSnapshotAgeWarning / FilecoinSnapshotAgeOld", so the anchor never resolved. Both alert names now share the one correct anchor. - filecoin-alerts.md#snapshotservicedown. No such heading, and no SnapshotServiceDown alert anywhere on main. Row removed rather than pointed somewhere plausible. - polkadot-alerts.md. Deleted upstream by #1649. There is no polkadot execution directory and no polkadot alert rule left, so the row is gone rather than repointed. - terraform/ as a tree link on two pages. The directory has been removed from main. infrastructure-and-devops.md loses the legacy row entirely; release-and-deploy.md keeps the row and states the tree is gone. The map was also incomplete, which matters more than the dead links. The page tells a paged operator that an unlisted scenario means the runbook does not exist, so a missing entry sends them to improvise. Eighteen runbooks exist upstream and only twelve were mapped. Added the six that were missing, including dirk-alerts.md, vouch-alerts.md and lido-nom-phone-escalation.md, which cover Lido production signing and the NOM phone path. The by-alert table gains the sixteen Dirk and Vouch alert anchors so those pages are reachable by alert name like the rest. Corrected the count claim from thirteen to eighteen. Re-verified against origin/main: 79 file references, 40 heading anchors, 15 directory references, 0 failures, and the runbook map now matches the directory exactly in both directions.
Review findings from an independent codex pass on gpt-6-astra, each verified against upstream before applying. The left column of the by-alert table is what an operator reads off their page, so it has to be the alertname Prometheus fires. Three rows carried the runbook heading text instead: - FilecoinLotusSyncingFail is really FilecoinlotusSyncingFail, lowercase l. The typo is upstream in prom_rules.yml:162 and the page had silently corrected it into something nobody will ever be paged with. - individual_validator_losing_balance is really IndividualValidatorLosingBalance (prom_rules.yml:260). - missed_attestations_in_mass is really MissedAttestationsInMass (prom_rules.yml:247). All three are live critical rules with high-urgency PagerDuty routes on Filecoin and Ethereum/Lido mainnet. The runbook anchors were correct and are unchanged; only the names an operator matches on have been fixed. The closing line also asserted that an unlisted scenario means the runbook does not exist. That is true of the scenario table, which is now 18 of 18, and false of the by-alert index, which has never been exhaustive: Aztec, SSV, Celestia and several Filecoin rules page without a row here. Telling a paged operator that no runbook exists when one does is worse than saying nothing, so the claim is now scoped to the table it actually holds for, and points at the chain runbook before the escalation. Polkadot dropped from the chain example list, since its runbook and rules were removed with the chain. The two lychee exclusions were unanchored and left the dot in github.com unescaped, so they also suppressed any repository whose name merely starts with infra-kubernetes or DNS-Management. Both now match the exact repository and its sub-paths only. Full alert-index coverage is deliberately not in this change. Reconciling the index against the Prometheus rules, Alertmanager routes and the Grafana Terraform is a separate audit, roughly forty rows across Aztec, SSV, Celestia and Canton.
Deploying engineering-handbook with
|
| Latest commit: |
f73caab
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://77a038b2.engineering-handbook-8f2.pages.dev |
| Branch Preview URL: | https://josh-infra-pointers.engineering-handbook-8f2.pages.dev |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Refreshes the three infra pointer pages against
infrastructure-generaland deletes the two"pending @joshdougall's confirmation" status notes that have been live on two of them since July.
Four references had broken and are fixed: a combined heading in
filecoin-alerts.md, an alertthat no longer exists,
polkadot-alerts.md(deleted upstream, chain decommissioned), andterraform/(directory removed frommain).The map was also incomplete, which matters more. The page tells a paged operator that an unlisted
scenario means the runbook does not exist. Eighteen exist upstream and twelve were mapped, so the
six missing ones are added, including Dirk, Vouch and the Lido NOM phone escalation. The by-alert
table gains the Dirk and Vouch alert anchors.
infrastructure-generalis private and excluded inlychee.toml, so link-check cannot cover anyof these. Verified by hand against
origin/main: 79 file references, 40 heading anchors, 15directory references, 0 failures, and the runbook map matches the directory in both directions.
The commit message carries the per-item detail.
Issues
Closes: none. Relates to the infra conformance work in
ChainSafe/infrastructure-general.