fix(crawler): crawl llms.txt index links instead of ingesting them as content - #2

fix(crawler): crawl llms.txt index links instead of ingesting them as content#2

AdamRussak merged 2 commits into

mainAdamRussak/self-doc:mainfrom

fix/llms-txt-index-vs-fullAdamRussak/self-doc:fix/llms-txt-index-vs-fullCopy head branch name to clipboard