New doc-collector

[DenysKuchma]

#24

Summary

Adds a new doc-collector boat that crawls website pages and generates lightweight user-facing documentation in spec form.

The new flow is exposed as:

• explorbot docs collect
• explorbot docs init

Generated output includes:

• output/docs/spec.md
• output/docs/pages/*.md
• output/research/*.md

Each documented page is summarized as:

• Purpose
• User Can
• User Might

What’s Included

• standalone boat/doc-collector subproject with its own CLI, config loader, crawl loop, renderer, and Documentarian
• page traversal with visited-state tracking and dead-loop protection via the existing state manager
• crawl target filtering and dynamic URL collapsing
• optional crawl scope modes:
  • site
  • section
  • subtree
• navigation discovery from both regular links and research-derived navigation targets for hash-based/OpenAPI-style docs
• fallback document generation path when structured JSON generation fails on noisy research input

Notes

Most of the feature lives inside boat/doc-collector.

The non-boat changes are either:

• integration changes required to expose the feature through the main CLI, or - small core fixes discovered while running the collector against real sites.