From 3f3dad6c266efcfa156dd3747b6eaa286cc95637 Mon Sep 17 00:00:00 2001 From: robfrank Date: Thu, 12 Nov 2026 16:13:23 +0100 Subject: [PATCH] docs(seo): JSON-LD, Open Graph, git-dated sitemap, llms.txt, robots From the 2026-11-12 SEO audit of docs.arcadedb.com. - head-meta.hbs: Open Graph and Twitter tags and a JSON-LD @graph per page (WebSite, BreadcrumbList, TechArticle; WebPage on a component home). about/publisher/author point at the arcadedb.com #software and #organization nodes by @id. The graph is built in a jsonld helper and serialized with JSON.stringify; @id, url and og:url are the extensionless canonical URL. - head-title.hbs: the docs home is "ArcadeDB Documentation" instead of "ArcadeDB Documentation :: ArcadeDB Documentation". - sitemap-lastmod.py: each sitemap URL gets the last commit date of its source instead of the build timestamp; the deploy checkout now fetches full history. arcadedb-python URLs get no lastmod (their source is in another repository). - generate-llms-txt.py: /llms.txt from the nav, linking the HTML pages (Cloudflare blocks *.md paths for now). - robots.txt: drop the comment claiming Cloudflare prepends a content-signal block (it does not) and add the same Content-Signal line arcadedb.com serves. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/cloudflare-deploy.yml | 4 + CLAUDE.md | 8 +- docs/ui/supplemental/helpers/jsonld.js | 105 +++++++++++++++ docs/ui/supplemental/helpers/plain.js | 28 ++++ docs/ui/supplemental/partials/head-meta.hbs | 19 +++ docs/ui/supplemental/partials/head-title.hbs | 2 + package.json | 5 +- scripts/generate-llms-txt.py | 127 +++++++++++++++++++ scripts/sitemap-lastmod.py | 95 ++++++++++++++ scripts/templates/robots.txt | 10 +- 10 files changed, 397 insertions(+), 6 deletions(-) create mode 100644 docs/ui/supplemental/helpers/jsonld.js create mode 100644 docs/ui/supplemental/helpers/plain.js create mode 100644 docs/ui/supplemental/partials/head-meta.hbs create mode 100644 docs/ui/supplemental/partials/head-title.hbs create mode 100644 scripts/generate-llms-txt.py create mode 100644 scripts/sitemap-lastmod.py diff --git a/.github/workflows/cloudflare-deploy.yml b/.github/workflows/cloudflare-deploy.yml index dcac2f38f..2ee31a7d8 100644 --- a/.github/workflows/cloudflare-deploy.yml +++ b/.github/workflows/cloudflare-deploy.yml @@ -23,6 +23,10 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # Full history: scripts/sitemap-lastmod.py dates each sitemap URL + # from the last commit that touched the page's source. + fetch-depth: 0 - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 with: diff --git a/CLAUDE.md b/CLAUDE.md index 443bbfc9c..6f434ea31 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -129,9 +129,15 @@ Every page gets `` from its `:description:` header attr Bing flags descriptions much shorter than 150-160 characters, so the generator keeps adding intro text up to ~120 characters and tops up a short intro with the section titles. Pages with no usable intro (chapter aggregators, SQL reference pages that open on a syntax block) get a generic "ArcadeDB X: A, B, C." summary; give those a hand-written `:description:` of 140-160 characters. For a chapter file with no heading in the source, put it on the first line. +### Structured data, social tags, sitemap dates, llms.txt + +- `docs/ui/supplemental/partials/head-meta.hbs` writes Open Graph and Twitter tags plus one JSON-LD `@graph` per page (WebSite, BreadcrumbList, TechArticle, or WebPage on a component home). The object is built in `docs/ui/supplemental/helpers/jsonld.js` and serialized with `JSON.stringify`, never templated by hand; Organization and SoftwareApplication are referenced by `@id` on arcadedb.com only. `@id`/`url`/`og:url` use `page.canonicalUrl`, so they stay extensionless. Antora loads each helper from a string, so a helper cannot `require()` a sibling helper. +- `head-title.hbs` drops the ` :: ArcadeDB Documentation` suffix when the page title equals the site title (the docs home). +- `npm run seo` (part of `npm run build`) runs `scripts/sitemap-lastmod.py`, which replaces Antora's build-time `` with each page source's last git commit date (it needs full history, hence `fetch-depth: 0` in the deploy workflow, and skips itself on a shallow clone), and `scripts/generate-llms-txt.py`, which writes `/llms.txt` from the nav. It links the HTML pages: Cloudflare's WAF currently 403s every `*.md` path. + ### URLs (no .html) -Cloudflare Pages serves pages without `.html` and 308s `/page.html` to `/page`. The playbook sets `urls.html_extension_style: drop` so canonical tags, the sitemap and xrefs use the extensionless URL, and `scripts/drop-html-links.py` (part of `npm run build`) strips `.html` from relative links written by hand in passthrough HTML. Keep writing `page.html` in hand-written links; the build fixes them. `scripts/templates/robots.txt` adds the `Sitemap:` line (Cloudflare prepends its content-signal block). +Cloudflare Pages serves pages without `.html` and 308s `/page.html` to `/page`. The playbook sets `urls.html_extension_style: drop` so canonical tags, the sitemap and xrefs use the extensionless URL, and `scripts/drop-html-links.py` (part of `npm run build`) strips `.html` from relative links written by hand in passthrough HTML. Keep writing `page.html` in hand-written links; the build fixes them. `scripts/templates/robots.txt` adds the `Sitemap:` line and the `Content-Signal:` line (Cloudflare serves the file as is; it does not prepend a managed block). ### Full-text search (Pagefind) diff --git a/docs/ui/supplemental/helpers/jsonld.js b/docs/ui/supplemental/helpers/jsonld.js new file mode 100644 index 000000000..a5629a632 --- /dev/null +++ b/docs/ui/supplemental/helpers/jsonld.js @@ -0,0 +1,105 @@ +'use strict' + +// Schema.org JSON-LD for every page: the docs WebSite, a BreadcrumbList built +// from the nav, and a TechArticle (WebPage on a component home). The +// Organization and SoftwareApplication nodes live on arcadedb.com and are +// referenced by @id only. Building the object in JS and serialising it with +// JSON.stringify keeps quotes and entities in titles from breaking the script +// block. Use as {{{jsonld}}} inside + {{/if}} diff --git a/docs/ui/supplemental/partials/head-title.hbs b/docs/ui/supplemental/partials/head-title.hbs new file mode 100644 index 000000000..f9809db45 --- /dev/null +++ b/docs/ui/supplemental/partials/head-title.hbs @@ -0,0 +1,2 @@ + {{!-- The docs home is titled after the site; don't repeat it as a suffix. --}} + {{{detag (or page.title defaultPageTitle)}}}{{#with site.title}}{{#unless (eq (detag ../page.title) this)}} :: {{this}}{{/unless}}{{/with}} diff --git a/package.json b/package.json index 97686cf25..b54058df3 100644 --- a/package.json +++ b/package.json @@ -3,9 +3,10 @@ "private": true, "description": "ArcadeDB documentation built with Antora", "scripts": { - "build": "antora --fetch antora-playbook.yml && npm run links && npm run redirects && npm run index", - "build:local": "antora antora-playbook.yml && npm run links && npm run redirects && npm run index", + "build": "antora --fetch antora-playbook.yml && npm run links && npm run redirects && npm run seo && npm run index", + "build:local": "antora antora-playbook.yml && npm run links && npm run redirects && npm run seo && npm run index", "links": "python3 scripts/drop-html-links.py build/site", + "seo": "python3 scripts/sitemap-lastmod.py build/site && python3 scripts/generate-llms-txt.py build/site", "index": "pagefind --site build/site", "redirects": "cp scripts/templates/_redirects scripts/templates/robots.txt build/site/", "serve": "npx -y http-server build/site -p 8080 -c-1" diff --git a/scripts/generate-llms-txt.py b/scripts/generate-llms-txt.py new file mode 100644 index 000000000..86f0a65d6 --- /dev/null +++ b/scripts/generate-llms-txt.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +""" +Write build/site/llms.txt (https://llmstxt.org) from the Antora nav. + +Sections follow the sidebar: one H2 per nav title and per top-level nav +group, a link per page with the page's meta description. Links point at the +HTML pages: Markdown copies of the pages are not published yet. + +Run after Antora, so the built pages (for descriptions) and the generated +nav files (docs/modules/ROOT/nav*.adoc, written by migrate.sh) both exist. + +Usage: scripts/generate-llms-txt.py build/site +""" +from __future__ import annotations + +import html +import re +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +NAV_DIR = REPO / "docs" / "modules" / "ROOT" +NAV_FILES = ["nav.adoc", "nav-query.adoc"] +SITE_URL = "https://docs.arcadedb.com" +COMPONENT = "arcadedb" + +NAV_TITLE_RE = re.compile(r"^\.(\S.*)$") +ITEM_RE = re.compile(r"^(\*+)\s+(.*)$") +XREF_RE = re.compile(r"^xref:([^\[#]+)\.adoc(#[^\[]*)?\[(.*)\]\s*$") +DESC_RE = re.compile(r'(.*?)(?: :: [^<]*)?", re.S) + +HEADER = """# ArcadeDB Documentation + +> ArcadeDB is an open-source multi-model database: graph, document, key/value, time-series, vector and geospatial data in one engine, queried with SQL, Cypher, Gremlin, GraphQL, MongoDB QL or Redis commands, embedded in Java or run as a server. + +The documentation is organized like its sidebar: getting started and examples, language drivers, data modeling and operations guides, then the query language and API reference. The current release is listed on the home page. +""" + + +def page_url(path: str) -> str: + if path == "index": + return f"{SITE_URL}/{COMPONENT}/" + return f"{SITE_URL}/{COMPONENT}/{path}" + + +def description(site: Path, path: str) -> str: + f = site / COMPONENT / (path + ".html") + if not f.exists(): + return "" + m = DESC_RE.search(f.read_text(encoding="utf-8")) + return html.unescape(m.group(1)).strip() if m else "" + + +def link(label: str, url: str, desc: str) -> str: + label = label.replace("[", "(").replace("]", ")") + return f"- [{label}]({url})" + (f": {desc}" if desc else "") + + +def main() -> int: + site = Path(sys.argv[1] if len(sys.argv) > 1 else "build/site") + out = [HEADER] + seen: set[str] = set() + section: list[str] = [] + + def flush(title: str | None) -> None: + if title and section: + out.append(f"## {title}\n") + out.extend(section) + out.append("") + section.clear() + + title = None + groups: list[str] = [] # label path of unlinked parent groups + for nav in NAV_FILES: + for raw in (NAV_DIR / nav).read_text(encoding="utf-8").splitlines(): + if m := NAV_TITLE_RE.match(raw): + flush(title) + title, groups = m.group(1), [] + continue + m = ITEM_RE.match(raw) + if not m: + continue + depth, text = len(m.group(1)), m.group(2).strip() + groups = groups[: depth - 1] + x = XREF_RE.match(text) + if depth == 1: + flush(title) + title = x.group(3) if x else text + if not x: + groups.append(text if depth > 1 else "") + continue + groups.append("") + path, frag, label = x.group(1), x.group(2) or "", x.group(3) + url = page_url(path) + frag + if url in seen: + continue + seen.add(url) + prefix = ": ".join(g for g in groups[1:-1] if g) + section.append(link(f"{prefix}: {label}" if prefix else label, url, + "" if frag else description(site, path))) + flush(title) + + # The Python bindings component has its own nav in another repository; + # list its pages from the sitemap Antora wrote for it. + py_sitemap = site / "sitemap-arcadedb-python.xml" + if py_sitemap.exists(): + out.append("## Python Bindings (arcadedb-python)\n") + for loc in re.findall(r"([^<]+)", py_sitemap.read_text(encoding="utf-8")): + rel = loc[len(SITE_URL) + 1:] + f = site / (rel + "index.html" if rel.endswith("/") else rel + ".html") + if not f.exists(): + continue + text = f.read_text(encoding="utf-8") + t = TITLE_RE.search(text) + d = DESC_RE.search(text) + out.append(link(html.unescape(t.group(1)).strip() if t else rel, loc, + html.unescape(d.group(1)).strip() if d else "")) + out.append("") + + (site / "llms.txt").write_text("\n".join(out).rstrip() + "\n", encoding="utf-8") + print(f"llms.txt: {sum(1 for l in out if l.startswith('- ['))} links") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/sitemap-lastmod.py b/scripts/sitemap-lastmod.py new file mode 100644 index 000000000..93fd2e7f4 --- /dev/null +++ b/scripts/sitemap-lastmod.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +""" +Replace the build timestamp Antora writes into every sitemap with +the date of the last git commit that touched the page's source. + +Antora stamps every entry with the time of the build, so all URLs look +changed on every deploy and search engines ignore the field. The docs pages +are generated by migrate.sh from src/main/asciidoc/ (and a few templates in +scripts/templates/), so the commit history of those files is the real +change date. + +URLs whose source is not in this repository (the arcadedb-python component) +get no rather than a fake one. On a shallow clone every file would +share one commit date, so the script leaves the sitemaps untouched; CI +checks out the full history (fetch-depth: 0) for this. + +Usage: scripts/sitemap-lastmod.py build/site +""" +from __future__ import annotations + +import re +import subprocess +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +SITE_URL = "https://docs.arcadedb.com" +COMPONENT = "arcadedb" + +# Antora page path (relative to docs/modules/ROOT/pages) -> source in this repo. +TEMPLATE_SOURCES = { + "index.adoc": "scripts/templates/index.adoc", + "get-started.adoc": "scripts/templates/get-started.adoc", +} + +URL_RE = re.compile(r"(.*?)", re.S) +LOC_RE = re.compile(r"([^<]+)") +LASTMOD_RE = re.compile(r"\s*[^<]*") + + +def git(*args: str) -> str: + return subprocess.run(["git", *args], cwd=REPO, check=True, capture_output=True, text=True).stdout + + +def last_commit_dates() -> dict[str, str]: + """Path -> ISO date of the newest commit touching it, in one git log pass.""" + out = git("log", "--format=@%cI", "--name-only", "--", "src/main/asciidoc", "scripts/templates") + dates: dict[str, str] = {} + current = None + for line in out.splitlines(): + if line.startswith("@"): + current = line[1:] + elif line and current: + dates.setdefault(line, current) + return dates + + +def source_for(url: str) -> str | None: + prefix = f"{SITE_URL}/{COMPONENT}/" + if not url.startswith(prefix): + return None + path = url[len(prefix):] + page = (path + "index" if path == "" or path.endswith("/") else path) + ".adoc" + return TEMPLATE_SOURCES.get(page, f"src/main/asciidoc/{page}") + + +def main() -> int: + site = Path(sys.argv[1] if len(sys.argv) > 1 else "build/site") + if git("rev-parse", "--is-shallow-repository").strip() == "true": + print("sitemap-lastmod: shallow clone, keeping Antora's lastmod values", file=sys.stderr) + return 0 + dates = last_commit_dates() + dated = undated = 0 + for sitemap in sorted(site.glob("sitemap-*.xml")): + def fix(m: re.Match[str]) -> str: + nonlocal dated, undated + body = m.group(1) + loc = LOC_RE.search(body) + src = source_for(loc.group(1)) if loc else None + body = LASTMOD_RE.sub("", body) + if src and src in dates: + dated += 1 + body = body.replace("", f"\n{dates[src]}", 1) + else: + undated += 1 + return f"{body}" + + text = sitemap.read_text(encoding="utf-8") + sitemap.write_text(URL_RE.sub(fix, text), encoding="utf-8") + print(f"sitemap-lastmod: {dated} URLs dated from git, {undated} without lastmod") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/templates/robots.txt b/scripts/templates/robots.txt index 18624bd82..d0957b2c8 100644 --- a/scripts/templates/robots.txt +++ b/scripts/templates/robots.txt @@ -1,7 +1,11 @@ -# Copied into build/site/ by `npm run build`. Cloudflare prepends its managed -# content-signal block to this file; the Sitemap line lets crawlers find the -# docs sitemap without it being submitted by hand. +# Copied into build/site/ by `npm run build`. Cloudflare serves this file as +# is (no managed robots.txt block is added), so the Content-Signal line below +# is the one crawlers see; it matches the policy on arcadedb.com. The Sitemap +# line lets crawlers find the docs sitemap without it being submitted by hand. User-agent: * Allow: / +# Content Signals (https://contentsignals.org/) +Content-Signal: search=yes, ai-train=yes, ai-input=yes + Sitemap: https://docs.arcadedb.com/sitemap.xml