Add additional metadata

[olliston]

Pull Request

Description

• Added /llms.txt route - a structured Markdown index following the https://llmstxt.org/, giving LLMs a navigable table of contents for the site's documentation
• Enhanced /llms-full.txt with a site header, page separators (---), and source URLs linking back to each article
• Added tags to advertise llmstxt support to future LLM crawlers - Since most LLM tools don't yet discover /llms.txt automatically, I've added these tags (the same mechanism browsers use to discover RSS feeds), hinting to crawlers that this site supports the standard.
• Extracted baseUrl into src/lib/config.ts to deduplicate NEXT_PUBLIC_BASE_URL usage across layout, sitemap, and LLM routes
• Filtered both llms routes to English-only pages (previously llms-full.txt included all 21 languages, reducing output from ~6MB to ~1MB, most models use English under the hood and just run a translation layer on top of that, so other languages are not really helpful)

Some other details

• /llms.txt index generation uses fumadocs' existing source.pageTree, ensuring that the index mirrors the actual sidebar navigation, including ordering from meta.json files, folder groupings, and section separators. The localizePageTree helper (already used by the docs layout) resolves i18n template strings like {meta.established_information.title} into readable English labels. Separators (---Section Name--- in meta.json) are treated as group headers in the output, with their following siblings nested underneath, matching how other sites structure their llms.txt with hierarchical indented lists.
• /llms-full.txt now includes a --- Title separator and Source: line before each page's content, making it easier for LLMs to identify article boundaries and link back to the original page.

Type of Change

• Documentation fix (typo, grammar, clarification)
• New documentation (guide, tutorial, page)
• Bug fix
• New feature
• Other

Screenshots

Checklist

• Tested locally with bun run dev
• Formatted code to adhere Styleguide with bun format
• Ran bun audit (no critical vulnerabilities)
• Checked spelling and grammar
• Verified all links work
• Followed Contributing Guidelines

---

As there are several changes in this, let me know if you want to drop something.

[olliston]

Btw, bun format produces 267 changed files for me, maybe I'm missing something in my setup?

[olliston]

Hey, have you had the chance to look at this? If you have any feedback, I'd be happy to address it.

[okunamayanad]

Hi, sorry about the late review. I'll get to this today, I'm positive that it'll probably get merged.

[okunamayanad]

After optimizations done on main branch, works without causing a ram spike.
The usability of llms-full.txt is a bit of a question as it's 40+ mb and may cause llms to work less effectively as it fills their context but that's a topic of it's own and the implementation in the PR is better than what it was before.
LGTM!