OSINTsweep

Async OSINT recon across 145+ sources — email registration discovery, username enumeration, name lookup, and domain intel. One command, one schema, all results. Built for prospect research, personal footprint audits, and authorized pentesting.

# B2B prospect research mode — defaults that won't cause harm
osint email founder@acme.example --mode prospect
osint username octocat --mode prospect
osint name "Linus Torvalds" --mode prospect --hint github

# Personal footprint audit — unlocks HIBP, prompts to confirm ownership
osint email me@mydomain.example --mode selfcheck

# Authorized pentest recon — audit-logged for client reporting
osint domain target.example --mode pentest

# Scan through proxies — rotate IPs to avoid rate limits
osint email target@example.com --mode prospect --proxy-file proxies.txt
osint username target --proxy socks5://1.2.3.4:1080 --proxy http://5.6.7.8:8080

Why this exists

Existing OSINT tools force you to chain 6 CLIs (sherlock + holehe + gravatar-check + curl + jq + grep) to answer the simplest question: "is this email registered on Twitter, GitHub, and Strava — and is the username taken?" OSINTsweep answers that in one async fan-out with one schema across all sources.

Install

git clone https://github.com/lev-corrupted/OSINTsweep
cd OSINTsweep
uv sync                          # installs deps incl. dev
uv run osint --help

Python 3.13+. Zero mandatory API keys — optional HUNTER_API_KEY, HIBP_API_KEY, EMAILREP_API_KEY, GITHUB_TOKEN unlock higher-quality lookups.

Three modes — gated at runtime

Mode	What's on	What's off
prospect	Public profiles, gravatar, DNS MX, Hunter.io pattern, Sherlock-style username search	HIBP breach lookup, password-reset registration discovery against non-owned emails
selfcheck	Everything, including HIBP	Confirmation required for emails not in ~/.osint-toolkit/owned_emails.txt
pentest	Everything + audit log	(Tool will refuse to run in pentest without --engagement <id>)

There is no default mode. The flag is required. Read INSTRUCTIONS.md for the why.

Source coverage (v0.5)

Category	Count	Sources
Email — registration check	29	Twitter/X, Microsoft, GitHub, Spotify, Instagram, Firefox, Duolingo, Adobe, Chess.com, Atlassian, devRant, Replit, HubSpot, Freelancer, HackerRank, MyFitnessPal, Neocities, Notion, Disney/ESPN, Stremio, HuggingFace, Nextdoor, Coursera, Plurk, Any.do, Etsy, WordPress, Kommo, Insightly
Email — metadata	4	Gravatar, DNS MX, Hunter.io pattern, EmailRep, HIBP (selfcheck/pentest)
Username — presence	105	GitHub, GitLab, Reddit, Twitter/X, YouTube, Steam, Twitch, Keybase, HackerNews, Mastodon, Bluesky, Stack Overflow, Chess.com, Lichess, LeetCode, CodeWars, HackerRank, HuggingFace, Behance, Dribbble, Medium, DEV.to, Substack, Pinterest, Snapchat, Telegram, Spotify, npm, Docker Hub, Last.fm, Redbubble, FurAffinity, 9GAG, BuzzFeed, Bandcamp, Reverbnation, JustGiving, e621, Coderwall, and 60+ more
Name — public profiles	7	Wikipedia, Wikidata, ORCID, Crossref, OpenSanctions, GitHub search, username permutator

All sites are defined declaratively in JSON — add a source by appending one object to holehe_sites.json or username_sites.json. No code changes needed.

Known issues

These are the current bugs and limitations that need fixing or help:

• Instagram — IP-level rate limiting kicks in fast; needs proxy rotation or smarter backoff
• WordPress — Aggressive throttling (login_auth_options_throttled), 30-min IP bans after a few queries
• Chess.com — Cloudflare blocks automated requests aggressively; low success rate in concurrent scans
• EmailRep — Free tier rate limit is very low; set EMAILREP_API_KEY for reliable results
• Cloudflare-protected username sites — Some sites (Substack, CodePen, Kickstarter, Depop, etc.) intermittently return CF challenges during concurrent bursts
• Adobe — Intermittent fingerprint mismatches during high-volume scans (works fine in isolation)

Proxy rotation

Built-in proxy rotation engine to avoid IP-level rate limits and Cloudflare blocks. Supports HTTP, HTTPS, SOCKS4, and SOCKS5 proxies.

# From a file (one proxy per line, # comments supported)
osint email target@example.com --proxy-file proxies.txt

# Inline (repeatable)
osint username target --proxy socks5://user:pass@1.2.3.4:1080 --proxy http://5.6.7.8:8080

# Via environment variable (comma-separated)
export OSINT_PROXIES="http://1.2.3.4:8080,socks5://5.6.7.8:1080"
osint email target@example.com

# Health-check your proxies
osint proxy-check --proxy-file proxies.txt

How it works:

• Proxies are health-checked on startup (tests against httpbin.org)
• Dead proxies are excluded automatically
• On rate-limit (429) or Cloudflare block, the proxy is cooled down for 2 minutes and the request retries on a different proxy
• Proxies with high error rates are deprioritized
• Falls back to direct connection if all proxies fail

Architecture (60-second tour)

target → dispatcher → [N modules in parallel, rate-limited, cached]
                    ↓
              proxy manager → rotate on 429/CF block, cooldown burned IPs
                    ↓
              normalize to Finding schema
                    ↓
              Report → JSON / CSV / Markdown / Rich table

• All HTTP via one shared httpx.AsyncClient.
• Per-host concurrency cap (default 4) — won't hammer a single source.
• 24h SQLite cache — re-querying the same target is free.
• Retries with exponential backoff + Retry-After honor.
• Cloudflare challenge detection with automatic retry.
• Modules declare modes_allowed = {"prospect", "selfcheck", "pentest"} — dispatcher filters before running.
• JSONL scan logging with monthly rotation for audit trails.

Test + run

uv run pytest                    # full suite (no real network calls — all mocked via respx)
uv run pytest --cov              # with coverage
uv run ruff check src tests      # lint

Credits & inspiration

This project builds on the techniques and research from:

• Holehe — email registration discovery via signup/login/password-reset endpoint fingerprinting. Our holehe-style module uses the same core idea with a declarative JSON config.
• Sherlock — username enumeration across social networks. Our sherlock-style module follows the same pattern with status code, body marker, and JSON field detection.
• Maigret — advanced username checking with site-specific parsing. Inspired our metadata extraction approach.
• EmailRep — email reputation API used as a module for risk scoring.
• Hunter.io — email pattern discovery API integrated as an optional module.
• HIBP — Troy Hunt's breach database, integrated for selfcheck/pentest modes.
• httpx — async HTTP client powering all requests.
• Rich — terminal UI rendering.

License

MIT. See LICENSE.

Ethics line

This tool will not add: stalkerware, mass-PII aggregation for sale, facial recognition, ID-document harvesting, or anything that targets a specific individual for harassment. Pull requests in those directions will be rejected. See INSTRUCTIONS.md "Ethics + the line I will not cross".