🕷️ EvilSpider

EvilSpider is a fast, asynchronous attack-surface crawler and reconnaissance engine designed for bug bounty hunters, penetration testers, and security researchers.

Unlike traditional generic crawlers (e.g., Scrapy or Crawlee), EvilSpider focuses strictly on offensive security use cases: uncovering hidden endpoints, hunting leaked secrets and API keys, extracting parameterized URLs, fingerprinting technologies, mapping forms, and generating actionable reconnaissance dashboards.

---

📥 Installation

📦 Via pip

pip install evilspider

⚡ Via pipx (Recommended for isolated CLI)

pipx install evilspider

🛠️ From Source (Development)

git clone https://github.com/Baba01hacker666/evilspider.git
cd evilspider
pip install -e .

---

🚀 Quick Start & Smart Defaults

EvilSpider uses smart defaults — you don't even need to type crawl! Simply run EvilSpider directly with your target URL or flags:

1. Instant Recon Crawl (Smart Default)

evilspider https://example.com
# or with flags:
evilspider -u https://example.com -d 3

2. Multi-Target Crawl from File (Save findings to JSON)

evilspider -l targets.txt -t 20 -o results.json

3. Pipeline URLs to Nuclei / FFUF / HTTPX

evilspider https://example.com --urls-only | nuclei -t http/cves/

4. Generate Interactive Dark-Mode HTML Dashboard

evilspider https://example.com --report-html report.html --report-md report.md

💡 Note on Output Saving: EvilSpider keeps your filesystem clean. It only saves files to disk when you explicitly tell it to (using -o, --output, --csv, --report-html, --report-md, etc.). Otherwise, findings are streamed cleanly in the terminal.

---

⚡ Key Features

• 🧠 Smart Defaults: No boilerplate required. Run evilspider https://example.com directly without typing crawl.
• 🛡️ WAF & Anti-Bot Hunter (25+ Protections): Automatic signature detection for Cloudflare, AWS WAF, Akamai Kona / Bot Manager, Imperva Incapsula, Fastly Signal Sciences, F5 BIG-IP ASM, ModSecurity / OWASP CRS, Sucuri CloudProxy, Barracuda, Fortinet FortiWeb, Datadome, PerimeterX, and more. Features adaptive evasion & backoff to prevent IP bans.
• ⚡ Smart SPA & JS-Heavy App Engine: Automatically detects Single Page Applications (React, Vue, Angular, Next.js, Nuxt, Svelte, Vite) and performs multi-layer JS decoding & deobfuscation:
  • Decodes Base64-encoded hidden API endpoints and secret tokens
  • Unescapes Hex (\x..), Unicode (\u...., \u{...}), and URL-encoded strings
  • Deobfuscates JS Obfuscator string array tables (e.g. _0x1234 = ['/api/...', ...])
  • Mines Client-side Routes (React Router <Route path="...">, Vue Router, Angular Router, Next.js dynamic routes)
  • Mines Webpack & Vite dynamic chunk bundles and fetches/analyzes them recursively
  • Extracts embedded GraphQL queries & mutations
• 💾 Clean Output Saving: Only writes to disk when explicitly asked via -o / --output or report flags.
• 🚀 Async Concurrency Engine: Ultra-fast asynchronous crawler built on asyncio and aiohttp with connection pooling, rate limiting, and configurable worker threads.
• 🚨 Secret & Sensitive Data Hunter: Built-in regex rule engine scanning for 30+ types of credentials across responses (AWS keys, Google Cloud API keys, JWT tokens, Stripe keys, GitHub PATs, Slack webhooks, Private keys, Database URIs, and leaked authorization headers).
• 📝 HTML Form & Input Extractor: Detects and categorizes HTML forms (Login, Registration, File Uploads, Password Reset, Search) along with hidden CSRF tokens and field names.
• 🛠️ Tech Stack Fingerprinter: Detects CMS (WordPress, Drupal, Joomla, Shopify), Backend Frameworks (Django, Laravel, Spring Boot, Express, Rails, FastAPI), Web Servers (Nginx, Apache, Caddy, Cloudflare, IIS), and Frontend Frameworks.
• 🛡️ Security Headers & Reflection Auditor: Identifies missing security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options), dangerous CORS misconfigurations (* with credentials), and reflected URL parameters (potential XSS sinks).
• 📖 API Documentation & Schema Parsing: Automatically parses Swagger 2.0 / OpenAPI 3.x specifications and GraphQL schemas to extract all declared endpoints, parameters, and routes.
• 🎯 Active Sensitive File Prober (probe): Fast active probing module hunting for VCS leaks (.git/HEAD, .svn), exposed configs (.env, web.config), backup archives (dump.sql, backup.zip), and debug endpoints (actuator/health, pprof) with smart soft-404 detection.
• 🌐 Scope & Filter Controls: Scope by domain, subdomains, regex whitelist/blacklist, extension filters, path exclusions, status codes, and loop prevention algorithms.
• 🎭 Browser Impersonation Profiles: Modern User-Agent and Client Hints / header profiles for Chrome, Firefox, Edge, Safari, Chrome Android, and Safari iOS.
• 📊 Multi-Format Reporting: Export findings to JSON, JSON Lines (NDJSON), CSV, Plain text URLs, Parameter wordlists, Markdown reports, and a standalone responsive Interactive Dark-Mode HTML Dashboard.

---

🔥 Recon Workflows

🕵️ 1) Full Hunter Mode (Secrets + Uploads + Sensitive Probing)

evilspider crawl \
  -u https://target.tld \
  --robots --sitemaps \
  --detect-uploads \
  --probe-sensitive \
  -d 4 -t 15 \
  --report-html report.html

🔑 2) Sensitive Endpoint & Backup Prober (probe)

evilspider probe \
  -u https://target.tld \
  --categories vcs,env_config,backups,api_docs,actuators_debug \
  -o probe_findings.json

📁 3) Offline JavaScript & File Analysis (extract)

evilspider extract -f app.bundle.js -o extracted_assets.json

🔐 4) Authenticated Crawl with Session Cookies & Bearer Token

evilspider crawl \
  -u https://target.tld \
  -C 'session=abc12345; role=admin' \
  --bearer-token 'eyJhbGciOiJIUzI1Ni...' \
  -H 'X-Forwarded-For: 127.0.0.1'

🌐 5) Proxied Crawl through Burp Suite / OWASP ZAP

evilspider crawl \
  -u https://target.tld \
  -x http://127.0.0.1:8080 \
  --insecure \
  --report-redirects

🎯 6) Parameter Wordlist Harvesting for Fuzzing

evilspider crawl \
  -u https://target.tld \
  --params-file params_wordlist.txt \
  --urls-only > discovered_urls.txt

# Pipe discovered parameters into ffuf
ffuf -u "https://target.tld/endpoint?FUZZ=1" -w params_wordlist.txt

---

🛠️ CLI Options Reference

evilspider crawl (Main Crawler)

Flag	Description
-u, --url	Target URL to crawl (e.g. https://example.com)
-l, --list	File containing target URLs (or - for stdin)
-c, --config	Path to JSON configuration file
-t, --threads	Number of concurrent workers (default: 10)
-d, --max-depth	Maximum crawl depth (default: 3)
-m, --max-links	Maximum total links to visit (default: 5000)
--max-time	Maximum crawl duration in seconds
--rate-limit	Maximum requests per second
--delay	Delay between requests in seconds
-s, --status	Interesting status codes (default: 200)
--exclude-status	Status codes to ignore (e.g. 404,500)
-e, --exts	Extension focus list (e.g. php,bak,env,json)
-k, --keywords	Keyword filter for response bodies
-p, --params-only	Only report URLs with query parameters
--include-subdomains / --no-include-subdomains	Toggle subdomain in-scope crawling (default: true)
--scope-regex	Whitelist regex pattern for allowed URLs
--exclude-regex	Blacklist regex pattern for ignored URLs
--exclude-paths	Comma-separated paths to ignore (/logout,/signout)
--robots / --sitemaps	Parse robots.txt and sitemap.xml
--detect-uploads	Detect forms with file uploads
--probe-sensitive	Actively probe target for sensitive files (.git, .env, backups, etc.)
--probe-categories	Prober categories (vcs,env_config,backups,api_docs,actuators_debug,graphql,admin_auth)
--no-secrets	Disable secrets and API keys scanner
--no-forms	Disable HTML forms extractor
--no-tech	Disable technology stack fingerprinter
--no-security-audit	Disable security headers auditor
-A, --user-agent	Custom User-Agent string
-i, --impersonate	Browser profile (chrome, firefox, edge, safari, chrome-android, safari-ios)
-C, --cookies	Cookie string or path to cookie file
-H, --headers	Custom header (repeatable, e.g. -H 'X-Header: value')
-x, --proxy	Proxy URL (HTTP/SOCKS5 e.g. http://127.0.0.1:8080)
--bearer-token	Authorization Bearer token
--auth-basic	HTTP Basic Auth in format user:pass
-T, --timeout	Request timeout in seconds (default: 5)
--retries	Number of request retries on failure (default: 2)
--follow-redirects	Follow HTTP redirects (default: true)
--report-redirects	Include redirect chains in results
--insecure	Disable SSL certificate verification
-o, --output	Primary output JSON file path
-j, --json	Stream findings as JSON Lines to stdout
--csv	Export findings as CSV
--report-html	Generate interactive standalone HTML dashboard
--report-md	Generate Markdown summary report
--urls-only	Clean stdout output containing only discovered URLs
--params-file	Save discovered query parameter wordlist
--secrets-file	Save discovered secrets to separate JSON
--forms-file	Save extracted forms to separate JSON
-q, --quiet	Suppress banner and info logs
--silent	Completely silent mode for UNIX piping
-v, --verbose	Enable debug logging
--no-color	Disable ANSI terminal colors

---

📊 Interactive HTML Dashboard

EvilSpider generates a standalone, self-contained HTML report with:

• Summary metric cards (Endpoints, Secrets, Forms, Subdomains, Technologies, Security Issues)
• Dynamic table filtering and instant keyword search
• Dedicated tabs for Secrets, Forms, Technologies, Security Headers, API Docs, and Subdomains
• Zero external CDN dependencies (works completely offline)

---

🏗️ Architecture

evilspider/
├── analyzer/
│   ├── waf.py            # 25+ WAF & Anti-Bot signature detector & adaptive evader
│   ├── js_analyzer.py    # SPA detector, Base64/Hex/Unicode decoder & client route miner
│   ├── secrets.py        # 30+ regex rules for API keys, tokens & credentials
│   ├── tech.py           # Web server, framework & CMS fingerprinter
│   ├── security.py       # Security headers, CORS & reflected param auditor
│   └── api_parser.py     # Swagger 2.0 / OpenAPI 3.x & GraphQL parser
├── extractors/
│   ├── html_extractor.py # Fast HTML parser for links, scripts, frames & comments
│   ├── js_extractor.py   # JavaScript REST endpoint & webpack chunk extractor
│   ├── form_extractor.py # HTML form, input & CSRF token parser
│   └── sourcemap.py      # Source map (.js.map) unminified route extractor
├── prober/
│   ├── probe.py          # Active sensitive file prober with soft-404 detection
│   └── wordlists.py      # Curated recon path lists (VCS, configs, backups, APIs)
├── output/
│   ├── formatter.py      # ANSI terminal formatting, logo & summary tables
│   ├── reporter.py       # Multi-format reporter (JSON, CSV, MD, HTML, TXT)
│   └── templates/
│       └── report_template.html # Responsive dark-mode dashboard
├── tests/                # Comprehensive unit and integration test suite
├── _version.py           # Single source of truth for version and author metadata
├── config.py             # Config parser and CLI argument validator
├── crawler.py            # Async crawl orchestrator and queue worker pool
└── main.py               # CLI entrypoint (crawl, probe, extract, report)

---

⚖️ Legal Disclaimer

EvilSpider is developed for authorized security assessments, penetration testing, bug bounty reconnaissance, and educational purposes only. Do not use this tool against targets without explicit authorization. The authors assume no liability for misuse.