From dea43f28a0833ce999cbb5d6532996b1d66c9dd2 Mon Sep 17 00:00:00 2001 From: pauliusminialga-dc Date: Wed, 28 Oct 2026 10:31:30 +0200 Subject: [PATCH 1/4] docs: improve README discoverability for search The repo page is absent from Google's index, and the README body is the only part of it search engines read for topical signals. Add the vocabulary developers actually search for and answer the decision-stage questions on the page itself. - Name the command line, terminal, and Go binary in the intro, alongside the runtimes pobear does not require. - Add a documentation nav line above Quick start. - Add three FAQ entries: scraping from the terminal, the Go runtime question, and how pobear differs from the Decodo CLI. The FAQ comparison also separates pobear from Decodo/cli, which targets the same "web scraping CLI" queries from the opposite default. --- README.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 92eeca2..8228dea 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,17 @@ [![CI](https://github.com/Decodo/pobear/actions/workflows/ci.yml/badge.svg)](https://github.com/Decodo/pobear/actions/workflows/ci.yml) -pobear is an open-source web scraping CLI for extracting structured data and crawling websites. Enable `--ladder` to automatically fall back from direct requests to your configured proxy or scraping API when requests are blocked or rate-limited. +pobear is an open-source web scraping CLI for extracting structured data and crawling websites from the command line. Enable `--ladder` to automatically fall back from direct requests to your configured proxy or scraping API when requests are blocked or rate-limited. -It runs as a single binary, with built-in extraction and reusable scraping jobs. Start without installing a language runtime or creating an account, and keep the same extraction rules as you change fetching methods. +It runs in your terminal as a single Go binary, with built-in extraction and reusable scraping jobs. Start without installing a language runtime such as Python or Node.js, and without creating an account, then keep the same extraction rules as you change fetching methods. - **Collect the fields you need**. Extract text and attributes with CSS selectors, or convert HTML to Markdown. - **Build repeatable workflows**. Save commands as JSON job specs, edit them, and run them from scripts or an AI agent. - **Control your crawl**. Set depth, page limits, scope, and request rates, with sitemap discovery and `robots.txt` support. - **Use additional fetching services when needed**. Keep direct requests for accessible pages and add a proxy or Decodo for more demanding targets. +Documentation: [CLI reference](docs/CLI_REFERENCE.md) · [Job spec reference](docs/JOB_SPEC_REFERENCE.md) · [Provider reference](docs/PROVIDER_REFERENCE.md) + ## Quick start Install on Linux or macOS: @@ -207,10 +209,24 @@ pobear scrape https://books.toscrape.com/ --css 'price=.price_color' \ ## FAQ +### How do I scrape a website from the terminal? + +Install the binary, then run `pobear scrape ` with the fields you want. For example, `pobear scrape https://books.toscrape.com/ --css 'price=.price_color'` prints one NDJSON record containing the extracted prices. See [Quick start](#quick-start) for a crawl example, or run `pobear` with no arguments to open the interactive terminal wizard. + ### Do I need a Decodo account or proxy to use pobear? No. Direct fetching works without an external account. You need proxy access or provider credentials only when you configure those fetching methods. +### Do I need Go installed to run pobear? + +No. pobear ships as a precompiled Go binary for Linux, macOS, and Windows, so there is no Python, Node.js, or browser runtime to install. Go is required only to build from source, as described under [Development](#development). + +### How is pobear different from the Decodo CLI? + +Both scrape from the command line, but they start from opposite defaults. [Decodo CLI](https://github.com/Decodo/cli) is a Node.js client for Decodo's scraping API: every request goes through Decodo, and the API's targets for search engines, eCommerce platforms, and social media are exposed as subcommands. pobear is a standalone Go binary that fetches directly by default and treats a proxy or Decodo as optional tiers you add with `--proxy`, `--provider`, or `--ladder`. + +Use the Decodo CLI when you want the full API surface and its ready-made targets. Use pobear when you want one binary that runs without an account and escalates only when a page blocks you. + ### Can pobear scrape JavaScript-rendered websites? Direct fetching retrieves HTML without executing JavaScript. Use `--provider decodo` for rendered content; the included Decodo configuration requests rendered HTML by default. From bb5c42afbf54117c84ec10198af6c6c0c3329e9a Mon Sep 17 00:00:00 2001 From: pauliusminialga-dc Date: Wed, 28 Oct 2026 10:38:10 +0200 Subject: [PATCH 2/4] Clarify pobear rendering option in README Updated instructions for using the Decodo provider with pobear to include rendering option. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8228dea..c5f1ca3 100644 --- a/README.md +++ b/README.md @@ -229,7 +229,7 @@ Use the Decodo CLI when you want the full API surface and its ready-made targets ### Can pobear scrape JavaScript-rendered websites? -Direct fetching retrieves HTML without executing JavaScript. Use `--provider decodo` for rendered content; the included Decodo configuration requests rendered HTML by default. +Direct fetching retrieves HTML without executing JavaScript. Use `--provider decodo --render` for rendered content. Rendering is off by default. ### What output formats does pobear support? From 7a08e48d2787742a127e8a5e82d47207dfbf3bc4 Mon Sep 17 00:00:00 2001 From: pauliusminialga-dc Date: Wed, 28 Oct 2026 10:44:20 +0200 Subject: [PATCH 3/4] Add hyperlink to pobear in README Updated README to include a hyperlink for pobear. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c5f1ca3..49bd274 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CI](https://github.com/Decodo/pobear/actions/workflows/ci.yml/badge.svg)](https://github.com/Decodo/pobear/actions/workflows/ci.yml) -pobear is an open-source web scraping CLI for extracting structured data and crawling websites from the command line. Enable `--ladder` to automatically fall back from direct requests to your configured proxy or scraping API when requests are blocked or rate-limited. +[pobear](https://help.decodo.com/docs/pobear) is an open-source web scraping CLI for extracting structured data and crawling websites from the command line. Enable `--ladder` to automatically fall back from direct requests to your configured proxy or scraping API when requests are blocked or rate-limited. It runs in your terminal as a single Go binary, with built-in extraction and reusable scraping jobs. Start without installing a language runtime such as Python or Node.js, and without creating an account, then keep the same extraction rules as you change fetching methods. From 73581d9fb33f6f283719bd234ca09e678d203fb0 Mon Sep 17 00:00:00 2001 From: pauliusminialga-dc Date: Wed, 28 Oct 2026 12:07:42 +0200 Subject: [PATCH 4/4] docs: keep the sniff-out tagline above the fold The repo description now carries the search keywords, so the original line moves into the README, where it reads as a tagline rather than as the only thing Google knows about pobear. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 49bd274..3a0341f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # pobear · Open-source web scraping CLI with automatic fallback +*Sniff out your data with any tool you want.* + [![CI](https://github.com/Decodo/pobear/actions/workflows/ci.yml/badge.svg)](https://github.com/Decodo/pobear/actions/workflows/ci.yml) [pobear](https://help.decodo.com/docs/pobear) is an open-source web scraping CLI for extracting structured data and crawling websites from the command line. Enable `--ladder` to automatically fall back from direct requests to your configured proxy or scraping API when requests are blocked or rate-limited.