From afb9fef1eebd54385e12c91663a2fee74bae4a9e Mon Sep 17 00:00:00 2001 From: johnnymatthews <9611008+johnnymatthews@users.noreply.github.com> Date: Thu, 27 Aug 2026 13:37:33 +0100 Subject: [PATCH 1/4] Fixes borked links. --- content/run/get-started/index.md | 2 +- content/run/operations/troubleshooting/index.md | 2 +- content/run/run-a-generator/guides/quicknode-setup/index.md | 2 +- .../deployment-examples/snapshot-bootstrap/index.md | 4 ++-- content/run/run-a-host/install/index.md | 2 +- content/understand/core-concepts/privacy/index.md | 2 +- content/understand/how-it-works/index.md | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/content/run/get-started/index.md b/content/run/get-started/index.md index 4f46646..fc5a418 100644 --- a/content/run/get-started/index.md +++ b/content/run/get-started/index.md @@ -167,7 +167,7 @@ There are a few things in here worth calling out: - `defradb.url: localhost:9181` is the Host client's _internal_ DefraDB API, not the Generator client's. Inside the Host client container, DefraDB binds to `9181` on `localhost`. The Generator client's API happens to use the same number because they're both DefraDB; we'll remap the published ports in the next step so they don't collide. - `bootstrap_peers` is the only Generator-specific value. The Host client learns everything else (schemas, signed data) from the Generator client over P2P once it connects. -- `minimum_attestations: 1` means the Host client will serve data as soon as it has one signature on it. Production setups use higher values to require independent confirmation from multiple Generator clients. See the [Host overview](/run/run-a-host/) for more on attestations. +- `minimum_attestations: 1` means the Host client will serve data as soon as it has one signature on it. Production setups use higher values to require independent confirmation from multiple Generator clients. See the [Host overview](/run/run-a-host/hardware-requirements/) for more on attestations. - `hub_base_url` points to ShinzoHub. We're not registering anything here, but the Host client expects the field to be present. ## Start the Host client diff --git a/content/run/operations/troubleshooting/index.md b/content/run/operations/troubleshooting/index.md index 5911301..5acc828 100644 --- a/content/run/operations/troubleshooting/index.md +++ b/content/run/operations/troubleshooting/index.md @@ -107,7 +107,7 @@ The Generator client falls back to HTTP polling. Check that `GETH_WS_URL` is cor ### What is the difference between a Host and a Generator? -Generator clients are the write side: they read raw blocks from an execution node, sign the data, gossip it over P2P, and provide attestations to prove the data is correct. Host clients are the read side: they receive those signed primitives, verify them, run WebAssembly lens transforms to build Views, and serve the results to applications over GraphQL. A Generator client talks to a source-chain node, but a Host client never does. See the [Run a Host](/run/run-a-host/) overview for more. +Generator clients are the write side: they read raw blocks from an execution node, sign the data, gossip it over P2P, and provide attestations to prove the data is correct. Host clients are the read side: they receive those signed primitives, verify them, run WebAssembly lens transforms to build Views, and serve the results to applications over GraphQL. A Generator client talks to a source-chain node, but a Host client never does. See the [Run a Host](/run/run-a-host/hardware-requirements/) overview for more. ### Do I need to run my own Generator client to run a Host client? diff --git a/content/run/run-a-generator/guides/quicknode-setup/index.md b/content/run/run-a-generator/guides/quicknode-setup/index.md index 11fffde..c1bedac 100644 --- a/content/run/run-a-generator/guides/quicknode-setup/index.md +++ b/content/run/run-a-generator/guides/quicknode-setup/index.md @@ -239,7 +239,7 @@ chown -R 1001:1001 /root/shinzo-data | `GOMEMLIMIT=5GiB` | Go runtime soft memory limit; keep below `mem_limit`. Scale both up on bigger hosts. | {% admonition(type="info") %} -See the [Generator client config reference](../../../run-a-generator/config-reference/index.md) for a detailed list of available configuration options. +See the [Generator client config reference](../../../run-a-generator/config-reference/) for a detailed list of available configuration options. {% end %} ## Start the client diff --git a/content/run/run-a-host/deployment-examples/snapshot-bootstrap/index.md b/content/run/run-a-host/deployment-examples/snapshot-bootstrap/index.md index 5579446..b49664d 100644 --- a/content/run/run-a-host/deployment-examples/snapshot-bootstrap/index.md +++ b/content/run/run-a-host/deployment-examples/snapshot-bootstrap/index.md @@ -35,7 +35,7 @@ On first startup, the Host downloads signed snapshot files from a trustless inde ## Prerequisites - Docker installed on the VM. -- A trustless indexer serving snapshots over HTTP. The trustless indexer must have `SNAPSHOT_ENABLED=true` and be reachable over HTTPS or HTTP. See the [nginx with TLS scenario](/run/run-a-generator/deployment-examples/nginx-tls-snapshots/) for how to set up a trustless indexer that serves snapshots. +- A trustless indexer serving snapshots over HTTP. The trustless indexer must have `SNAPSHOT_ENABLED=true` and be reachable over HTTPS or HTTP. See the [nginx with TLS scenario](/run/run-a-generator/deployment-examples/nginx-with-snapshots/) for how to set up a trustless indexer that serves snapshots. - The block range you want to bootstrap. The trustless indexer must have snapshot files covering that range. ## Config file @@ -139,7 +139,7 @@ You should see log lines indicating snapshot downloads and imports, followed by - Snapshot bootstrap only runs on first startup when DefraDB has no existing data. If the Host already has data for the requested block range, bootstrap is skipped. - The `indexer_url` in the shipped `config.yaml` is `http://35.206.105.60:8080`. This is a development trustless indexer. Replace it with your own trustless indexer URL or a production trustless indexer that has snapshots enabled. -- The trustless indexer must have `SNAPSHOT_ENABLED=true` on the Generator side. If the trustless indexer is not producing snapshot files, the `/snapshots` endpoint will return nothing and bootstrap will fail. See the [nginx with TLS scenario](/run/run-a-generator/deployment-examples/nginx-tls-snapshots/) for setting up a trustless indexer that serves snapshots. +- The trustless indexer must have `SNAPSHOT_ENABLED=true` on the Generator side. If the trustless indexer is not producing snapshot files, the `/snapshots` endpoint will return nothing and bootstrap will fail. See the [nginx with TLS scenario](/run/run-a-generator/deployment-examples/nginx-with-snapshots/) for setting up a trustless indexer that serves snapshots. - The `DEFRA_URL` env var overrides `defradb.url` at runtime and is read by the Host client (`config/config.go`). The `docker run` above does not set it, so the DefraDB URL comes from `defradb.url` in the YAML config. See [environment variables](/run/run-a-host/config-reference#environment-variables). - The `DEFRA_KEYRING_SECRET` env var uses the `DEFRA_` prefix. The Generator client uses `DEFRADB_KEYRING_SECRET` with the `DEFRADB_` prefix. The two clients use different env var names for the same concept. See [environment variables](/run/run-a-host/config-reference#environment-variables). - `LOG_LEVEL`, `LOG_SOURCE`, and `LOG_STACKTRACE` env vars appear in some deployment scripts but are not read by the Host client. They have been omitted from the `docker run` above. See [env vars that are not read](/run/run-a-host/config-reference#env-vars-that-are-not-read). diff --git a/content/run/run-a-host/install/index.md b/content/run/run-a-host/install/index.md index c0d6b10..a35684a 100644 --- a/content/run/run-a-host/install/index.md +++ b/content/run/run-a-host/install/index.md @@ -13,7 +13,7 @@ A Host client pulls primitive blockchain data from Generator clients, runs Lens - **Docker** (for the Docker path), or **[Go 1.25+](https://go.dev/dl/) and Make** (to build from source). - **Access to a running Generator client.** The Host client doesn't produce data itself. It receives primitive block data from a Generator client over libp2p and transforms it, so it needs at least one Generator client to sync from. The Generator client doesn't have to be your own, you just need one you can reach and its libp2p multiaddr. A public Generator client you can point at is coming; this page will link it here once it's live. Until then, run your own. The [Operator Quickstart](/run/run-a-host/quickstart/) walks through standing up a Generator client and a Host client together. -- **Hardware** that meets the [recommendations](/run/run-a-host/). +- **Hardware** that meets the [recommendations](/run/run-a-host/hardware-requirements/). There are two ways to install: [Docker](#use-docker) (recommended) or [build from source](#build-from-source). diff --git a/content/understand/core-concepts/privacy/index.md b/content/understand/core-concepts/privacy/index.md index df1fe97..de83e06 100644 --- a/content/understand/core-concepts/privacy/index.md +++ b/content/understand/core-concepts/privacy/index.md @@ -19,7 +19,7 @@ A standard Host client ships configured for the public network. On startup it do - Dials the public bootstrap peers shipped in `config.yaml` to reach Generator clients it doesn't control. - Contacts ShinzoHub (`shinzo.hub_base_url`) to fetch every registered View and subscribe to live registration events. -A third step is opt-in: registering on the [Host registry](../register) so other nodes can discover and replicate from it. This is a manual, wallet-signed action from the registration app, not something the Host client does on startup. +A third step is opt-in: registering on the [Host registry](/run/run-a-host/register/) so other nodes can discover and replicate from it. This is a manual, wallet-signed action from the registration app, not something the Host client does on startup. A private Host client connection closes some or all of these. How many you close decides how private the setup is. diff --git a/content/understand/how-it-works/index.md b/content/understand/how-it-works/index.md index 06270df..6b40bd1 100644 --- a/content/understand/how-it-works/index.md +++ b/content/understand/how-it-works/index.md @@ -169,7 +169,7 @@ Generator clients are the entry point. Reserved for validators at mainnet launch #### Hosts -Hosts are the workhorses. They receive primitives, maintain attestation records, run Views, and serve the resulting view documents to subscribers. Anyone can run a Host. See [Run a Host](/run/run-a-host/) for operational details and [Host Client reference](/reference/components/host-client/) for internals. +Hosts are the workhorses. They receive primitives, maintain attestation records, run Views, and serve the resulting view documents to subscribers. Anyone can run a Host. See [Run a Host](/run/run-a-host/hardware-requirements/) for operational details and [Host Client reference](/reference/components/host-client/) for internals. #### Developers From 1388c6dc8b4967d3c07cdbb0f0824a3cb67408a8 Mon Sep 17 00:00:00 2001 From: johnnymatthews <9611008+johnnymatthews@users.noreply.github.com> Date: Thu, 27 Aug 2026 13:37:44 +0100 Subject: [PATCH 2/4] Points footer links at actual pages. --- templates/macros/footer.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/templates/macros/footer.html b/templates/macros/footer.html index 5db5dfb..f5caadf 100644 --- a/templates/macros/footer.html +++ b/templates/macros/footer.html @@ -14,10 +14,10 @@

Get started building with Shinzo

  • Who it's for
    • From f38e14c37b6c5b715cbfe82de349968159b185d3 Mon Sep 17 00:00:00 2001 From: johnnymatthews <9611008+johnnymatthews@users.noreply.github.com> Date: Thu, 27 Aug 2026 13:37:56 +0100 Subject: [PATCH 3/4] Adds new check-links workflow. --- .github/workflows/build.yml | 6 ++ scripts/README.md | 25 ++++++ scripts/build.sh | 23 +++-- scripts/check-links.py | 175 ++++++++++++++++++++++++++++++++++++ scripts/link-allowlist.txt | 7 ++ 5 files changed, 231 insertions(+), 5 deletions(-) create mode 100644 scripts/check-links.py create mode 100644 scripts/link-allowlist.txt diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 89173a5..ba3c42f 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -13,5 +13,11 @@ jobs: - name: Checkout uses: actions/checkout@v4 + - name: Check site + run: ./scripts/build.sh check + - name: Build site run: ./scripts/build.sh + + - name: Check internal links + run: python3 scripts/check-links.py public scripts/link-allowlist.txt diff --git a/scripts/README.md b/scripts/README.md index b6e0379..0148e9a 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -28,6 +28,31 @@ Zola installation behavior: - Otherwise the script downloads the Zola release matching `ZOLA_VERSION`. - `ZOLA_VERSION` is read from the environment first, then from `wrangler.toml`. +#### Subcommands + +- `./scripts/build.sh` (or `./scripts/build.sh build`): production build. This is + what Cloudflare Pages runs. +- `./scripts/build.sh check`: runs `zola check` (validates `@/` internal links + as errors per `[link_checker]` in `config.toml`, and external links as + warnings). Used by CI; does not write output. + +### check-links.py + +Checks internal links in the built site output (`public/`). Unlike +`zola check`, which only validates `@/`-prefixed links, this script checks +**all** internal links regardless of syntax — absolute (`/path/`), relative +(`../path/`), and `@/`-prefixed — by inspecting the built HTML. + +Run it after `zola build`: + +```shell +python3 scripts/check-links.py public +``` + +It verifies that every internal `` resolves to an existing file and +that fragment anchors (`#section`) match an `id` in the target page. Exits +non-zero if any broken links are found. Used by CI. + ### generate-llms.sh Generates `llms.txt` and `llms-full.txt` into the build output directory (defaults to `./public`). diff --git a/scripts/build.sh b/scripts/build.sh index 5e55419..8b7a18c 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -72,11 +72,24 @@ main() { ensure_zola - echo "Building site with Zola..." >&2 - zola build - - echo "Generating llms.txt outputs..." >&2 - "$SCRIPT_DIR/generate-llms.sh" + local cmd="${1:-build}" + case "$cmd" in + build) + echo "Building site with Zola..." >&2 + zola build + + echo "Generating llms.txt outputs..." >&2 + "$SCRIPT_DIR/generate-llms.sh" + ;; + check) + echo "Checking site with Zola..." >&2 + zola check + ;; + *) + echo "error: unknown command '$cmd' (expected: build|check)" >&2 + return 1 + ;; + esac } main "$@" diff --git a/scripts/check-links.py b/scripts/check-links.py new file mode 100644 index 0000000..4d106d6 --- /dev/null +++ b/scripts/check-links.py @@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +"""Check internal links in built Zola site output. + +Walks all .html files in the output directory, extracts href attributes from + tags, and verifies that internal links resolve to existing files. Handles +Zola's directory-style URL scheme (e.g. /path/ -> path/index.html) and +verifies fragment anchors (#section) against id attributes in the target page. + +Usage: check-links.py [allowlist] +Exit: 0 if all links resolve, 1 if broken links are found. + +The optional allowlist file lists hrefs to suppress (one per line, # comments). +Use it to keep CI green while known-broken links are being fixed; remove +entries as they are fixed. +""" + +import sys +from html.parser import HTMLParser +from pathlib import Path + + +class LinkExtractor(HTMLParser): + """Collect href values from tags with their source line numbers.""" + + def __init__(self): + super().__init__() + self.hrefs = [] + + def handle_starttag(self, tag, attrs): + if tag == "a": + for name, value in attrs: + if name == "href" and value: + self.hrefs.append((value, self.getpos()[0])) + + +def is_internal(href): + """True for links that point to pages on this site.""" + return not href.startswith(( + "http://", "https://", "mailto:", "tel:", + "javascript:", "data:", "ftp://", "file:", + )) + + +def resolve_path(href, current_file, output_dir): + """Resolve an internal href to a (Path, fragment) in the output directory. + + Returns (None, fragment) for pure same-page anchor links. + """ + path_part, _, fragment = href.partition("#") + if not path_part: + return None, fragment + + if path_part.startswith("/"): + target = output_dir / path_part.lstrip("/") + else: + target = (current_file.parent / path_part).resolve() + + return target, fragment + + +def find_html_file(target): + """Find the actual file for a resolved target path. + + Zola outputs pages as directory/index.html. Also handles direct files + (images, etc.) and extension-less paths. + """ + if target.is_file(): + return target + if target.is_dir(): + index = target / "index.html" + if index.is_file(): + return index + if not target.suffix: + html_candidate = target.with_suffix(".html") + if html_candidate.is_file(): + return html_candidate + return None + + +def fragment_exists(html_file, fragment): + """Check that an id matching the fragment exists in the HTML file.""" + if not fragment: + return True + try: + text = html_file.read_text(encoding="utf-8") + except OSError: + return True + return f'id="{fragment}"' in text or f"id='{fragment}'" in text + + +def load_allowlist(path): + """Load hrefs to suppress from a file (one per line, # comments).""" + if not path: + return set() + allowed = set() + for line in Path(path).read_text(encoding="utf-8").splitlines(): + line = line.strip() + if line and not line.startswith("#"): + allowed.add(line) + return allowed + + +def main(): + if len(sys.argv) < 2 or len(sys.argv) > 3: + print("usage: check-links.py [allowlist]", file=sys.stderr) + return 2 + + output_dir = Path(sys.argv[1]).resolve() + if not output_dir.is_dir(): + print(f"error: {output_dir} is not a directory", file=sys.stderr) + return 2 + + allowlist = load_allowlist(sys.argv[2] if len(sys.argv) > 2 else None) + + broken = [] + skipped = 0 + checked = 0 + files_scanned = 0 + + for html_file in sorted(output_dir.rglob("*.html")): + files_scanned += 1 + extractor = LinkExtractor() + try: + extractor.feed(html_file.read_text(encoding="utf-8")) + except Exception as exc: + print(f"warning: could not parse {html_file}: {exc}", file=sys.stderr) + continue + + for href, line in extractor.hrefs: + if not is_internal(href): + continue + + if href in allowlist: + skipped += 1 + continue + + target, fragment = resolve_path(href, html_file, output_dir) + + if target is None: + if fragment and not fragment_exists(html_file, fragment): + rel = html_file.relative_to(output_dir) + broken.append(f" {rel}:{line}: {href} (missing #{fragment})") + continue + + checked += 1 + resolved = find_html_file(target) + + if resolved is None: + rel = html_file.relative_to(output_dir) + try: + target_rel = target.relative_to(output_dir) + except ValueError: + target_rel = target + broken.append(f" {rel}:{line}: {href} (not found: {target_rel})") + elif fragment and not fragment_exists(resolved, fragment): + rel = html_file.relative_to(output_dir) + broken.append(f" {rel}:{line}: {href} (missing #{fragment})") + + summary = f"Checked {checked} internal links across {files_scanned} HTML files." + if skipped: + summary += f" Skipped {skipped} allowlisted." + print(summary) + + if broken: + print(f"\nBroken link(s) ({len(broken)}):", file=sys.stderr) + for entry in broken: + print(entry, file=sys.stderr) + return 1 + + print("All internal links resolve.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/link-allowlist.txt b/scripts/link-allowlist.txt new file mode 100644 index 0000000..c00e26c --- /dev/null +++ b/scripts/link-allowlist.txt @@ -0,0 +1,7 @@ +# Known broken internal links — fix and remove from this file. +# +# Currently empty. Add hrefs to suppress (one per line, # comments) to keep CI +# green while known-broken links are being fixed. Remove entries as they are +# fixed so the checker re-enforces them. +# +# See scripts/check-links.py for the checker that consumes this file. From bd205be92da3d76afa13c7b23761cf71f92aa39b Mon Sep 17 00:00:00 2001 From: johnnymatthews <9611008+johnnymatthews@users.noreply.github.com> Date: Mon, 31 Aug 2026 11:17:51 +0100 Subject: [PATCH 4/4] Rewrites script in bash instead. Nobody likes Python. --- .github/workflows/build.yml | 2 +- scripts/README.md | 4 +- scripts/check-links.py | 175 ---------------------------- scripts/check-links.sh | 220 ++++++++++++++++++++++++++++++++++++ scripts/link-allowlist.txt | 2 +- 5 files changed, 224 insertions(+), 179 deletions(-) delete mode 100644 scripts/check-links.py create mode 100755 scripts/check-links.sh diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ba3c42f..6c708ad 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -20,4 +20,4 @@ jobs: run: ./scripts/build.sh - name: Check internal links - run: python3 scripts/check-links.py public scripts/link-allowlist.txt + run: ./scripts/check-links.sh public scripts/link-allowlist.txt diff --git a/scripts/README.md b/scripts/README.md index 0148e9a..4ac86e8 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -36,7 +36,7 @@ Zola installation behavior: as errors per `[link_checker]` in `config.toml`, and external links as warnings). Used by CI; does not write output. -### check-links.py +### check-links.sh Checks internal links in the built site output (`public/`). Unlike `zola check`, which only validates `@/`-prefixed links, this script checks @@ -46,7 +46,7 @@ Checks internal links in the built site output (`public/`). Unlike Run it after `zola build`: ```shell -python3 scripts/check-links.py public +./scripts/check-links.sh public ``` It verifies that every internal `` resolves to an existing file and diff --git a/scripts/check-links.py b/scripts/check-links.py deleted file mode 100644 index 4d106d6..0000000 --- a/scripts/check-links.py +++ /dev/null @@ -1,175 +0,0 @@ -#!/usr/bin/env python3 -"""Check internal links in built Zola site output. - -Walks all .html files in the output directory, extracts href attributes from - tags, and verifies that internal links resolve to existing files. Handles -Zola's directory-style URL scheme (e.g. /path/ -> path/index.html) and -verifies fragment anchors (#section) against id attributes in the target page. - -Usage: check-links.py [allowlist] -Exit: 0 if all links resolve, 1 if broken links are found. - -The optional allowlist file lists hrefs to suppress (one per line, # comments). -Use it to keep CI green while known-broken links are being fixed; remove -entries as they are fixed. -""" - -import sys -from html.parser import HTMLParser -from pathlib import Path - - -class LinkExtractor(HTMLParser): - """Collect href values from tags with their source line numbers.""" - - def __init__(self): - super().__init__() - self.hrefs = [] - - def handle_starttag(self, tag, attrs): - if tag == "a": - for name, value in attrs: - if name == "href" and value: - self.hrefs.append((value, self.getpos()[0])) - - -def is_internal(href): - """True for links that point to pages on this site.""" - return not href.startswith(( - "http://", "https://", "mailto:", "tel:", - "javascript:", "data:", "ftp://", "file:", - )) - - -def resolve_path(href, current_file, output_dir): - """Resolve an internal href to a (Path, fragment) in the output directory. - - Returns (None, fragment) for pure same-page anchor links. - """ - path_part, _, fragment = href.partition("#") - if not path_part: - return None, fragment - - if path_part.startswith("/"): - target = output_dir / path_part.lstrip("/") - else: - target = (current_file.parent / path_part).resolve() - - return target, fragment - - -def find_html_file(target): - """Find the actual file for a resolved target path. - - Zola outputs pages as directory/index.html. Also handles direct files - (images, etc.) and extension-less paths. - """ - if target.is_file(): - return target - if target.is_dir(): - index = target / "index.html" - if index.is_file(): - return index - if not target.suffix: - html_candidate = target.with_suffix(".html") - if html_candidate.is_file(): - return html_candidate - return None - - -def fragment_exists(html_file, fragment): - """Check that an id matching the fragment exists in the HTML file.""" - if not fragment: - return True - try: - text = html_file.read_text(encoding="utf-8") - except OSError: - return True - return f'id="{fragment}"' in text or f"id='{fragment}'" in text - - -def load_allowlist(path): - """Load hrefs to suppress from a file (one per line, # comments).""" - if not path: - return set() - allowed = set() - for line in Path(path).read_text(encoding="utf-8").splitlines(): - line = line.strip() - if line and not line.startswith("#"): - allowed.add(line) - return allowed - - -def main(): - if len(sys.argv) < 2 or len(sys.argv) > 3: - print("usage: check-links.py [allowlist]", file=sys.stderr) - return 2 - - output_dir = Path(sys.argv[1]).resolve() - if not output_dir.is_dir(): - print(f"error: {output_dir} is not a directory", file=sys.stderr) - return 2 - - allowlist = load_allowlist(sys.argv[2] if len(sys.argv) > 2 else None) - - broken = [] - skipped = 0 - checked = 0 - files_scanned = 0 - - for html_file in sorted(output_dir.rglob("*.html")): - files_scanned += 1 - extractor = LinkExtractor() - try: - extractor.feed(html_file.read_text(encoding="utf-8")) - except Exception as exc: - print(f"warning: could not parse {html_file}: {exc}", file=sys.stderr) - continue - - for href, line in extractor.hrefs: - if not is_internal(href): - continue - - if href in allowlist: - skipped += 1 - continue - - target, fragment = resolve_path(href, html_file, output_dir) - - if target is None: - if fragment and not fragment_exists(html_file, fragment): - rel = html_file.relative_to(output_dir) - broken.append(f" {rel}:{line}: {href} (missing #{fragment})") - continue - - checked += 1 - resolved = find_html_file(target) - - if resolved is None: - rel = html_file.relative_to(output_dir) - try: - target_rel = target.relative_to(output_dir) - except ValueError: - target_rel = target - broken.append(f" {rel}:{line}: {href} (not found: {target_rel})") - elif fragment and not fragment_exists(resolved, fragment): - rel = html_file.relative_to(output_dir) - broken.append(f" {rel}:{line}: {href} (missing #{fragment})") - - summary = f"Checked {checked} internal links across {files_scanned} HTML files." - if skipped: - summary += f" Skipped {skipped} allowlisted." - print(summary) - - if broken: - print(f"\nBroken link(s) ({len(broken)}):", file=sys.stderr) - for entry in broken: - print(entry, file=sys.stderr) - return 1 - - print("All internal links resolve.") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/check-links.sh b/scripts/check-links.sh new file mode 100755 index 0000000..2dd1284 --- /dev/null +++ b/scripts/check-links.sh @@ -0,0 +1,220 @@ +#!/usr/bin/env bash +# Check internal links in built Zola site output. +# +# Walks all .html files in the output directory, extracts href attributes from +# tags, and verifies that internal links resolve to existing files. Handles +# Zola's directory-style URL scheme (e.g. /path/ -> path/index.html) and +# verifies fragment anchors (#section) against id attributes in the target page. +# +# Usage: check-links.sh [allowlist] +# Exit: 0 if all links resolve, 1 if broken links are found, 2 on usage error. +# +# The optional allowlist file lists hrefs to suppress (one per line, # comments). +# Use it to keep CI green while known-broken links are being fixed; remove +# entries as they are fixed. + +set -euo pipefail + +export LC_ALL=C + +readonly ANCHOR_RE='<[aA][[:space:]][^>]*href=("[^"]*"|'"'"'[^'"'"']*'"'"')' +readonly ENTITY_RE='&#(x[0-9a-fA-F]+|[0-9]+);' + +usage() { + echo "usage: check-links.sh [allowlist]" >&2 + exit 2 +} + +# Print $1 with numeric (&#NN; / &#xHH;) and the common named HTML entities +# decoded, mirroring what an HTML parser hands to the link checker. +decode_entities() { + local s="$1" ent num code oct ch + while [[ $s =~ $ENTITY_RE ]]; do + ent="${BASH_REMATCH[0]}" + num="${BASH_REMATCH[1]}" + if [[ $num == x* ]]; then + code=$((16#${num#x})) + else + code=$((10#$num)) + fi + if ((code >= 0x20 && code <= 0x7e)); then + printf -v oct '%03o' "$code" + printf -v ch "\\$oct" + s="${s//"$ent"/$ch}" + else + break + fi + done + s="${s//</<}" + s="${s//>/>}" + s="${s//"/\"}" + s="${s//'/\'}" + s="${s//&/&}" + printf '%s' "$s" +} + +# True if the file has an id attribute matching the fragment. +fragment_exists() { + grep -qF "id=\"$2\"" -- "$1" || grep -qF "id='$2'" -- "$1" +} + +# Print $1 lexically normalized (resolves ../ segments, like Path.resolve()). +# Falls back to the input as-is when no parent directory exists to cd into. +normalize_path() { + local p="$1" base out + while [[ $p == */ && $p != "/" ]]; do p="${p%/}"; done + out="$(cd "$p" 2>/dev/null && pwd -P)" || out="" + if [[ -z $out ]]; then + base="${p##*/}" + p="${p%/*}" + if [[ -n $p ]]; then + out="$(cd "$p" 2>/dev/null && pwd -P)" || out="" + [[ -n $out ]] && out="$out/$base" + fi + fi + [[ -n $out ]] || out="$1" + printf '%s\n' "$out" +} + +# Set RESOLVED to the file an internal target maps to, or leave it empty. +# Directory-style URLs map to /index.html; extension-less paths also +# try a sibling .html file. +resolve_target() { + RESOLVED="" + if [[ -f $1 ]]; then + RESOLVED="$1" + elif [[ -d $1 && -f $1/index.html ]]; then + RESOLVED="$1/index.html" + else + local base="${1##*/}" + if [[ $base != ?*.* && -f "$1.html" ]]; then + RESOLVED="$1.html" + fi + fi +} + +main() { + if [[ $# -lt 1 || $# -gt 2 ]]; then + usage + fi + + OUTPUT_DIR="$(cd "$1" 2>/dev/null && pwd -P)" || { + echo "error: $1 is not a directory" >&2 + exit 2 + } + + local ALLOW=() + if [[ $# -eq 2 ]]; then + if [[ ! -f $2 ]]; then + echo "error: allowlist file not found: $2" >&2 + exit 2 + fi + local line + while IFS= read -r line || [[ -n $line ]]; do + line="${line#"${line%%[![:space:]]*}"}" + line="${line%"${line##*[![:space:]]}"}" + [[ -z $line || $line == \#* ]] && continue + ALLOW+=("$line") + done < "$2" + fi + + local BROKEN=() + local checked=0 skipped=0 files_scanned=0 + local html_file rel_src match line_no href path_part fragment + local target resolved norm show + + while IFS= read -r html_file; do + files_scanned=$((files_scanned + 1)) + if [[ ! -r $html_file ]]; then + echo "warning: could not read $html_file" >&2 + continue + fi + rel_src="${html_file#"$OUTPUT_DIR"/}" + + while IFS= read -r match; do + line_no="${match%%:*}" + href="${match#*:}" + href="${href#*href=?}" + href="${href%?}" + [[ -n $href ]] || continue + href="$(decode_entities "$href")" + + case $href in + http://*|https://*|mailto:*|tel:*|javascript:*|data:*|ftp://*|file:*) continue ;; + esac + + if [[ ${#ALLOW[@]} -gt 0 ]]; then + local allowed=0 entry + for entry in "${ALLOW[@]}"; do + [[ $entry == "$href" ]] && { allowed=1; break; } + done + if [[ $allowed -eq 1 ]]; then + skipped=$((skipped + 1)) + continue + fi + fi + + path_part="${href%%#*}" + if [[ $href == *"#"* ]]; then fragment="${href#*#}"; else fragment=""; fi + + if [[ -z $path_part ]]; then + if [[ -n $fragment ]] && ! fragment_exists "$html_file" "$fragment"; then + BROKEN+=(" $rel_src:$line_no: $href (missing #$fragment)") + fi + continue + fi + + checked=$((checked + 1)) + + if [[ $path_part == /* ]]; then + while [[ $path_part == /* ]]; do path_part="${path_part#/}"; done + if [[ -n $path_part ]]; then + target="$OUTPUT_DIR/$path_part" + else + target="$OUTPUT_DIR" + fi + else + target="${html_file%/*}/$path_part" + fi + while [[ $target == */ && $target != "/" ]]; do target="${target%/}"; done + + resolve_target "$target" + + if [[ -z $RESOLVED ]]; then + if [[ $path_part == /* || $href == /* ]]; then + norm="$target" + else + norm="$(normalize_path "$target")" + fi + if [[ $norm == "$OUTPUT_DIR" ]]; then + show="." + elif [[ $norm == "$OUTPUT_DIR"/* ]]; then + show="${norm#"$OUTPUT_DIR"/}" + else + show="$norm" + fi + BROKEN+=(" $rel_src:$line_no: $href (not found: $show)") + elif [[ -n $fragment ]] && ! fragment_exists "$RESOLVED" "$fragment"; then + BROKEN+=(" $rel_src:$line_no: $href (missing #$fragment)") + fi + done < <(grep -noE "$ANCHOR_RE" -- "$html_file" || true) + done < <(find "$OUTPUT_DIR" -type f -name '*.html' | sort) + + local summary="Checked $checked internal links across $files_scanned HTML files." + [[ $skipped -gt 0 ]] && summary="$summary Skipped $skipped allowlisted." + echo "$summary" + + if [[ ${#BROKEN[@]} -gt 0 ]]; then + echo "" >&2 + echo "Broken link(s) (${#BROKEN[@]}):" >&2 + local entry + for entry in "${BROKEN[@]}"; do + echo "$entry" >&2 + done + exit 1 + fi + + echo "All internal links resolve." +} + +main "$@" diff --git a/scripts/link-allowlist.txt b/scripts/link-allowlist.txt index c00e26c..eb86a9f 100644 --- a/scripts/link-allowlist.txt +++ b/scripts/link-allowlist.txt @@ -4,4 +4,4 @@ # green while known-broken links are being fixed. Remove entries as they are # fixed so the checker re-enforces them. # -# See scripts/check-links.py for the checker that consumes this file. +# See scripts/check-links.sh for the checker that consumes this file.