fix(web): let AI search bots crawl the site

web/public/robots.txt

@@ -1,27 +1,24 @@
 # Everything here is public documentation for a free, open-source app.
+# AI search and assistant bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot,
+# Claude-User, PerplexityBot, Perplexity-User) fall into this group on purpose:
+# they fetch pages to cite them in answers, which is how the site gets found.
 User-agent: *
 Allow: /
 
-# AI crawlers are blocked at the edge, with a WAF skip for the two surfaces
+# Training crawlers are blocked at the edge, with a WAF skip for the two surfaces
 # built for them: the llms.txt index and the markdown behind every docs page.
 # Saying so here spends a crawler's budget on the two paths that answer instead
 # of on HTML that will 403. This list is advisory and safe to let drift — the
-# WAF is the enforcement, and a crawler missing from it falls back to the group
-# above and is stopped at the edge anyway, which is the same answer, just later.
+# WAF is the enforcement. Keep it in step with the WAF rule: a search bot listed
+# here or matched there disappears from AI answers.
 #
 # The Allow lines come first on purpose. An RFC 9309 parser takes the longest
 # match and doesn't care about order, but a first-match parser reads `Disallow: /`
 # and stops there — leading with it denies the two paths this group exists to
 # open. The .md line needs `*` and `$`, which are Google extensions: the crawlers
 # named here support them, and one that doesn't just won't open the .md files.
 User-agent: GPTBot
-User-agent: OAI-SearchBot
-User-agent: ChatGPT-User
 User-agent: ClaudeBot
-User-agent: Claude-User
-User-agent: Claude-SearchBot
-User-agent: PerplexityBot
-User-agent: Perplexity-User
 User-agent: meta-externalagent
 User-agent: Bytespider
 User-agent: CCBot