Skip to content

fix(crawler): crawl llms.txt index links instead of ingesting them as content - #2

Merged
AdamRussak merged 2 commits into
mainfrom
fix/llms-txt-index-vs-full
Jul 22, 2026
Merged

fix(crawler): crawl llms.txt index links instead of ingesting them as content#2
AdamRussak merged 2 commits into
mainfrom
fix/llms-txt-index-vs-full

perf(crawler): stream llms.txt download and abort once over the size cap

aa5580a
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar