feat(website): generate llms.txt and llms-full.txt#3179
Open
vpetersson wants to merge 1 commit into
Open
Conversation
Add a post-build generator that produces llms.txt (a curated index) and llms-full.txt (the full text of every page) for the marketing site, following the llmstxt.org convention and served from the site root. The generator extracts clean Markdown from the rendered public/ HTML rather than the Markdown/data sources, so it captures the marketing copy that lives in Hugo layouts (home, features, get-started) and never drifts from what visitors actually see. Wired into `bun run build` and the deploy-website workflow. Also fixes four pre-existing `sort $map` calls in the /api templates: in Hugo, sorting a map collapses it to a slice, so the keys were replaced by numeric indices — API paths, HTTP status codes, request body content-types, and schema field names all rendered as 0, 1, 2… on the live page. Hugo already ranges maps in sorted-key order, so dropping `sort` fixes it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| for (const tr of el.querySelectorAll('tr')) { | ||
| const cells = tr | ||
| .querySelectorAll('th,td') | ||
| .map((c) => inline(c).trim().replace(/\|/g, '\\|')) |
There was a problem hiding this comment.
Pull request overview
Adds automated generation of llms.txt and llms-full.txt for the marketing site (from the built public/ HTML), and fixes Hugo /api template rendering where sort on maps was collapsing keys to numeric indices.
Changes:
- Add a Bun post-build script (
website/scripts/generate-llms.ts) to extract Markdown-like text from rendered HTML and emit/llms.txt+/llms-full.txt. - Wire generation into
bun run buildand the Pages deploy workflow; addnode-html-parseras a devDependency. - Remove incorrect
sortusage on Hugo maps in/apitemplates so API paths/status codes/schema field names render correctly.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
website/scripts/generate-llms.ts |
New generator that crawls public/**/index.html, extracts cleaned Markdown, and writes llms.txt/llms-full.txt. |
website/package.json |
Adds llms:generate and runs it as part of build; adds node-html-parser. |
website/layouts/partials/api-schema.html |
Stops sorting schema properties map so field names don’t become numeric indices. |
website/layouts/_default/api.html |
Stops sorting paths, responses, and request-body content maps so keys render correctly. |
website/bun.lock |
Locks transitive deps for node-html-parser. |
.github/workflows/deploy-website.yaml |
Runs bun run llms:generate after Hugo build so the files ship with Pages artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+161
to
+165
| const flush = () => { | ||
| const t = buf.trim() | ||
| if (t) out.push(t) | ||
| buf = '' | ||
| } |
| ) | ||
| } | ||
|
|
||
| main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues Fixed
No associated issue. Adds LLM-discovery files to the marketing site and fixes a rendering bug on the
/apipage found along the way.Description
Adds a post-build generator (
website/scripts/generate-llms.ts) that produces two files served from the site root, following the llmstxt.org convention:/llms.txt— a curated index: H1, a>summary, and annotated links to every page./llms-full.txt— the full text of all pages concatenated into one Markdown document, each section headed by its source URL.It extracts clean Markdown from the rendered
public/HTML rather than the Markdown/data sources, so it also captures the marketing copy that lives in Hugo layouts (home, features, get-started) and never drifts from what visitors see. Handles headings, lists (incl. code blocks nested in list items), tables, blockquotes, code fences and links; strips nav/footer/scripts; skips Hugo redirect stubs; decodes HTML entities. Wired intobun run buildand a new step indeploy-website.yaml(output ships in the existing Pages artifact). Addsnode-html-parseras a build-time devDependency.Also fixes four pre-existing
sort $mapcalls in the/apitemplates. In Hugo,sorton a map collapses it to a slice, replacing the map keys with numeric indices. On the live page that made API paths, HTTP status codes, request-body content-types, and every schema field name render as0, 1, 2…instead of/api/v2/assets,200,asset_id, etc. Hugo already ranges maps in sorted-key order, so droppingsortfixes it.Checklist
Website-only change (no device runtime impact). Validated end-to-end: clean
bun run build, stricttscpasses, output verified on both minified (CI) and non-minified HTML, and the/apipage confirmed rendering real paths/status codes/field names.