xweb

Internet access CLI for AI agents and humans. Converts unstructured web content into LLM-friendly data streams.

Features

• Web search via multiple providers (Brave, Tavily, Serper, built-in fallback — no API key required)
• Fetch and clean web pages to Markdown, text, HTML, or JSON
• Discover site structure via sitemap or link extraction
• Auto-detect TTY for human-readable vs JSON output
• All subcommands support --json

Install

From npm

npm install -g @theclawlab/xweb

From source

npm install
npm run build
npm link

Quick Start

# Search the web (no API key needed)
xweb search "rust vs go 2026"

# Fetch a page as clean Markdown
xweb fetch "https://example.com/article"

# Discover internal links of a site
xweb explore "https://docs.example.com"

Commands

Command	Description
xweb search <query>	Search the web, returns structured results
xweb fetch <url>	Fetch and clean a web page
xweb explore <url>	Discover internal links (sitemap or crawl)
xweb config	Manage search provider configuration

Search Providers

Provider	API Key Required
simple	No (default fallback)
brave	Yes (BSA-...)
tavily	Yes (tvly-...)
serper	Yes

Provider selection: explicit --provider > configured default > simple fallback.

Configuration

Config file: ~/.config/xweb/default.json

{
  "default_provider": "brave",
  "providers": {
    "brave": { "api_key": "BSA-xxx" },
    "tavily": { "api_key": "tvly-xxx" },
    "serper": { "api_key": "xxx" }
  },
  "fetch_settings": {
    "timeout": 30,
    "max_length": 50000
  }
}

Documentation

• USAGE.md — Full usage guide with all options and examples