From 88310fc3eeace29597cde423e6c8245ccf848915 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Wed, 1 Jul 2026 10:52:28 +0200 Subject: [PATCH] Add crawlberg to Web Crawling --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 3fbbab36..d4a6f5db 100644 --- a/README.md +++ b/README.md @@ -1443,6 +1443,7 @@ Online tools, services and APIs to simplify development. ## Web Crawling +* [crawlberg](https://github.com/xberg-io/crawlberg) - High-performance web crawling and scraping engine (Ruby binding over a Rust core) with HTML-to-Markdown conversion and a headless-Chrome fallback. * [LinkThumbnailer](https://github.com/gottfrois/link_thumbnailer) - Ruby gem that generates thumbnail images and videos from a given URL. Much like popular social website with link preview. * [Kimurai](https://github.com/vifreefly/kimuraframework) - A modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites. * [Mechanize](https://github.com/sparklemotion/mechanize) - Mechanize is a ruby library that makes automated web interaction easy.