From 8f7698e6c77b5354e19ca4d8c13d435f6d895eae Mon Sep 17 00:00:00 2001
From: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
Date: Thu, 4 Jun 2026 19:30:26 +0200
Subject: [PATCH] feat: add repo-controlled robots.txt allowing all AI crawlers

Allow search engines and AI agents to crawl the public documentation: wildcard
Allow: / with a positive content signal (no ai-train=no) and explicit Allow
records for the major AI crawlers (ClaudeBot, GPTBot, Google-Extended, CCBot,
Bytespider, Amazonbot, Applebot-Extended, meta-externalagent).

VuePress copies src/.vuepress/public/ verbatim to the published site root, so
this is served at /robots.txt.
---
 src/.vuepress/public/robots.txt | 38 +++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 src/.vuepress/public/robots.txt

diff --git a/src/.vuepress/public/robots.txt b/src/.vuepress/public/robots.txt
new file mode 100644
index 0000000..07a40ed
--- /dev/null
+++ b/src/.vuepress/public/robots.txt
@@ -0,0 +1,38 @@
+# robots.txt — dEURO documentation (docs.deuro.com)
+#
+# Public documentation. We explicitly WANT both search engines and AI agents to
+# crawl, index, and learn from this content. This file is version-controlled in
+# this repository and is the authoritative crawl policy for this site.
+#
+# Content signals: all uses are granted — search, AI input / retrieval-augmented
+# generation, and AI training. We deliberately do NOT signal ai-train=no.
+
+User-agent: *
+Allow: /
+Content-Signal: search=yes, ai-input=yes, ai-train=yes
+
+# Major AI crawlers are explicitly welcome. Some honor only their own named
+# record, so each is listed in addition to the wildcard group above.
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: GPTBot
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: CCBot
+Allow: /
+
+User-agent: Bytespider
+Allow: /
+
+User-agent: Amazonbot
+Allow: /
+
+User-agent: Applebot-Extended
+Allow: /
+
+User-agent: meta-externalagent
+Allow: /