[LIghtJUNction]

Public /robots.txt and /sitemap.xml currently return 404: the September 27 rewrite removed the legacy crawler routes without replacements. Restore anonymous GET/HEAD discovery endpoints using the configured canonical service origin.

The sitemap lists only /. It performs no resource enumeration and exposes no account names, private posts, or mutable resource paths. Robots advertises the sitemap and excludes the operation namespace /-/ from crawling. Existing host validation, read-only methods and security headers apply; HEAD preserves the GET byte length without a response body.

Validation:

• New crawler integration test passes, covering XML contents, canonical origin, public homepage, GET/HEAD, host rejection, query rejection, method rejection and security headers.
• Branding, crawler and request-target regression selection: 146 passed, 7 failed. All seven failures also reproduce on unmodified base f20c024: the existing homepage request-target test stub has no executor following the homepage summary change.
• Ruff check and formatting check pass for changed files; git diff --check passes.

No production service or DNS changes. Existing Search Console sitemap submission can resume after this change is deployed; this PR does not guarantee indexing.

[chatgpt-codex-connector]

You have reached your Codex usage limits for security reviews. Please try again later.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[LIghtJUNction]

CI for head b1ec28b is finished: 8 workflows passed and 4 failed on inherited base issues.

• Deployment rehearsal fails when scripts/check_installed_server.py:186 unlinks the already-removed recovery-drill.json: https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36748441894/job/110000532187
• Integration closure (both locales), Architecture boundaries, and Rewrite contracts hit published_operation_changed after home_summary was added to the published discovery.read_query@1 schema: https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36748441662, https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36748441733, https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36748441686
• Rewrite also hits the stale homepage request-target/nginx fixtures. Preserve homepage read contracts and align installed restore checks #206 addresses those fixtures; the dictionary compatibility and rehearsal cleanup fixes are separate dependencies.

Please refresh this branch after those fixes and rerun CI before treating the crawler restoration as fully validated.