[robfrank]

After each production deploy, this PR submits the changed docs URLs to IndexNow, the protocol Bing, Yandex, Seznam and Naver use to learn about page changes (Google ignores it).

How it works

• Key file. scripts/templates/d83c04689e7d448be13934efdf3c9f4d.txt holds only the key. npm run redirects copies it to the site root, as it already does for robots.txt. The key is public by design.
  • The file has no trailing newline, so end-of-file-fixer now excludes it.
  • Root .txt files are served as text/plain; charset=utf-8 (checked on /llms.txt).
• Before the deploy. indexnow.py snapshot saves the live sitemap-*.xml files to $RUNNER_TEMP.
• After a successful deploy. indexnow.py submit compares the new sitemaps with that snapshot and submits a URL when it:
  • is new,
  • has a different lastmod (now the git date of its source, from docs(seo): JSON-LD, Open Graph, git-dated sitemap lastmod, llms.txt, robots.txt #532), or
  • is no longer in the sitemap (a removed page, so engines see its 404 or 301).
• No snapshot, nothing sent. If the snapshot is missing, nothing is submitted, rather than every URL. Batches are capped at 10,000 URLs.
• Never blocks the deploy. Both steps are continue-on-error with timeouts.
  • The submit step first waits for the key file to be live (up to 10 × 15 s).
  • It retries 429 and 5xx responses with backoff, and doesn't retry 400, 403 or 422.
  • It logs each HTTP status and every URL it submits.
• Main repo only. Both steps run only on pushes to main, scheduled runs and manual dispatch in ArcadeData/arcadedb-docs, never on PRs or forks.

Not verified

• Cloudflare Crawler Hints. I couldn't check it: that's a zone setting and this session has no Cloudflare API access. If Crawler Hints is on for arcadedb.com, Cloudflare already sends IndexNow pings for docs.arcadedb.com and these steps are redundant: close this PR, or drop the two steps and the key file.
• Live behavior. Tested locally: the snapshot of the live sitemaps (246 URLs), the diff logic (no change, a changed lastmod, a removed URL, a new URL) and the key lookup. I sent no POST, so the first real submission happens on the first deploy after merge. That deploy is also when /d83c04689e7d448be13934efdf3c9f4d.txt first goes live.
• Earlier removal. core-concepts/chapter was removed by docs(seo): page titles, release version attribute, thin index pages #531, before this exists, so it won't be submitted automatically. Later removals will be.

🤖 Generated with Claude Code

[mergify]

This pull request is merged.