Kapital Web Exposure Scanner — a configuration-driven orchestrator for external web scanning of assets you are explicitly authorized to test.
KWES runs seven established security CLIs in a pipeline you declare in configuration, preserves their raw output verbatim, and normalizes everything into one finding model with evidence attached.
It was built for, and named after, the estate it was first written to assess — but nothing in it is specific to that estate. The scope file is the only thing that decides what gets touched.
Every hostname, URL, and address a tool discovers is inventory. Only an allowlist file grants permission, and that check happens at the adapter boundary before each stage — not once at startup.
KWES sends real traffic to real systems. Scanning infrastructure you do not own or have written permission to test is unlawful in most jurisdictions.
It is built so that the safe path is the only path:
- There is no
--targetflag. The allowlist is a file with anauthorizationblock naming who approved the work and when. Without it the run does not start. - A discovered hostname is never automatically in scope. Subfinder invents names, dnsx returns third-party CNAMEs, and crawlers find off-host URLs — all of it is re-checked.
- Hosts fronted by a CDN or cloud provider are excluded from active stages even when the hostname is in your allowlist, because permission for a hostname is not permission for the third party serving it.
You are responsible for the authorization behind your scope file. See SECURITY.md.
It is an orchestrator. It handles scope enforcement, one shared traffic budget, process-group cancellation, secret redaction, raw-evidence preservation, resumable runs, and normalization into a single schema.
It is not a vulnerability-detection engine, and not a substitute for a penetration
test. It discovers and reports; a human validates. Every automated result is recorded
unverified until a person says otherwise, and the severity a tool claimed is kept in a
separate field from the severity an analyst assigns. Conflating those two is what makes
automated security reports untrustworthy.
subfinder → dnsx → httpx → katana → jsluice → ffuf → nuclei → normalization
| # | Stage | Tool | What it adds | How it is wrong |
|---|---|---|---|---|
| 1 | subdomains |
subfinder | Hostnames under a root domain, from certificate transparency and passive DNS. Sends nothing to the target. | Returns dead hostnames and names owned by other people. |
| 2 | dns |
dnsx | A/AAAA/CNAME records, plus wildcard-DNS detection. | A wildcard record makes every invented name look alive; without the probe one record inflates the inventory by hundreds. |
| 3 | http |
httpx | Which hosts actually answer, with status, title, server, TLS, and technology guesses. | Technology detection is a header-and-body guess. It routes later checks; it never proves a version. |
| 4 | crawl |
katana | The routes behind each live URL — pages, API paths, form parameters. | Reports links it found, including 404s and routes retired years ago. |
| 5 | js_analysis |
jsluice | API paths parsed out of JavaScript bundles, which no page links to. | Every high-entropy string looks like a key; build hashes dominate the noise. |
| 6 | content_discovery |
ffuf | Paths nobody advertised — the forgotten admin panel, the exposed .git. |
Soft-404s above all. A site answering 200 for missing pages makes every guess a hit. |
| 7 | findings |
nuclei | Template matches for known exposures and misconfigurations. | A template match is a pattern match, not a proof. |
The order lives in config/pipeline.yaml, not in code. No module encodes "after httpx
comes katana". Stages connect by declared input/output kind, so any stage can be
disabled or reordered without touching Python.
Stage 6 is the only one that needs a capability grant. Guessing at unadvertised paths is
more intrusive than following links, so ffuf declares
requires_capabilities = {directory_fuzzing} and the orchestrator refuses the stage
unless both the profile and the scope document grant it. The shipped safe and
fast profiles leave it off.
KWES never installs anything on your behalf. The seven scanners are operator-installed prerequisites. If one is missing, KWES names the binary and fails the run — it will not silently skip a stage or substitute a different tool.
1. The Python package
git clone https://github.com/aykhan019/kwes.git
cd kwes
uv tool install --editable .That puts a kwes command on your PATH, so every example below works from any
directory — kwes scan ... rather than uv run kwes scan .... --editable means the
command tracks this checkout, so a git pull updates the installed tool and the bundled
configuration together.
If kwes: command not found, uv installed it to ~/.local/bin, which is not on your
PATH:
uv tool update-shell # then restart your shellTo develop rather than use, uv sync is enough and uv run kwes ... works without
installing anything globally.
2. The scanners (ProjectDiscovery tools, ffuf, jsluice)
go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
go install github.com/projectdiscovery/dnsx/cmd/dnsx@latest
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
go install github.com/projectdiscovery/katana/cmd/katana@latest
go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
go install github.com/ffuf/ffuf/v2@latest
go install github.com/BishopFox/jsluice/cmd/jsluice@latest3. The Nuclei template set — also not vendored, for the same reason:
nuclei -update-templates4. Confirm the environment
kwes doctordoctor runs each binary's version flag — a local call that sends no network traffic —
and checks the template set is present. Running it first means you find out nuclei is
missing before three stages have already sent traffic.
Every tool's --version output is recorded into each run's manifest. Go-based tools
drift between installs, and results are not reproducible without knowing which build
produced them.
# 1. Write your authorization document.
cp targets/scope.example.yaml targets/scope.yaml
$EDITOR targets/scope.yaml
# 2. Check it loads and see exactly what it would permit. Sends nothing.
kwes validate --scope targets/scope.yaml
# 3. Print the exact command every stage would run. Still sends nothing.
kwes scan --scope targets/scope.yaml --dry-run
# 4. Run it.
kwes scan --scope targets/scope.yaml --profile safe --verbosevalidate and --dry-run are the two commands worth building a habit around. Between
them they answer "would this start, and what exactly would it be allowed to do" without
a single packet leaving the machine.
| Command | Purpose |
|---|---|
kwes validate --scope <file> |
Load and check all configuration. Sends nothing. |
kwes doctor |
Verify every tool an enabled stage needs is installed. Sends nothing. |
kwes scan --scope <file> |
Run the pipeline. |
kwes version |
Print the version. |
| Flag | Effect |
|---|---|
--profile <name> |
safe (default), fast, or deep. See below. |
--dry-run |
Print each stage's exact argv and send nothing. |
--severity <list> |
Narrow templates, e.g. critical. Can only narrow a profile, never widen it. The single biggest lever on runtime. |
--rate <n> |
Cap requests/second for this run. Can only lower the configured budget, never raise it. |
--resume <run_id> |
Replay finished stages from preserved raw output. Sends no traffic for those stages. |
--incremental |
Give the full check set only to hosts that need it. Opt-in; every skipped host is named in the manifest and summary. |
--verbose |
Live trace: target names, per-second throughput, running totals. |
This is the authorization document expressed as configuration, and it is the only thing
that grants KWES permission to send traffic anywhere. See
targets/scope.example.yaml — it is heavily annotated.
authorization:
authorized_by: "Name, Title, Organization"
reference: "Engagement or ticket reference"
valid_until: "2026-12-31"
root_domains: # domains to ENUMERATE UNDER (subfinder is handed these)
- example.com
subdomains: # specific approved hosts; nothing is enumerated under them
- www.example.com
include_subdomains: false # false: only names written here are in scope
exclusions: # never touched, even if discovery finds them
- legacy.example.comTwo things decide how much traffic a run produces:
root_domainsvssubdomainsanswer different questions. A root domain is enumerated under. A subdomain is a single approved host. Puttingwww.example.cominroot_domainsasks subfinder to find subdomains of that host, which returns nothing and spends minutes doing so.include_subdomainsdecides whether discovery can widen the run. Withtrue, anything under an approved root is in scope — so a host nobody reviewed becomes a target the moment a certificate transparency log mentions it.falseis the right setting for most engagements.
An empty key is a deliberate load error. subdomains: with nothing after it is null in
YAML, not an empty list, and "deliberately none" and "not filled in yet" are different
statements. Write subdomains: [] for the first. KWES refuses to guess.
Real scope files are authorization documents, not source code.
targets/is gitignored except for the example.
safe (default) |
fast |
deep |
|
|---|---|---|---|
| For | Conservative default | Estate-wide triage | Short vetted host list |
| Severities | low → critical | high, critical | info → critical |
| Content discovery | off | off | on |
| Crawl | — | — | depth 3, 10 min/host |
| Relative cost | baseline | ~4× faster | ~4× slower |
All three exclude the intrusive, dos, brute-force, fuzzing, headless, and
code template tags, and all three run HTTP templates only.
A host that passes fast has not been fully checked — it has been checked for
high-and-critical findings only. The summary says so.
These are enforced in code, by types and tests, rather than by documentation and reviewer discipline.
Scope is a type, not a check. Adapters accept only AuthorizedTarget values, which
scope.authorize() alone can construct. Passing a raw string is a type error caught by
mypy, not a review miss. A single upfront check would run before the risky data exists —
every stage generates new targets.
One traffic budget, not one per tool. Five tools each capped at 50 requests/second is 250 requests/second arriving at the target. Every tool derives its rate flag from one configured number, and no two active stages run against the same host concurrently. httpx and nuclei both default to 150 on their own, which is why KWES always passes the flag explicitly.
One subprocess site. tools/runner.py is the only place a process is spawned. It is
the single place that enforces timeouts, process-group cancellation, and the no-shell
rule — a second spawn site would silently opt out of all three. A test asserts no other
module can spawn.
No shell=True, ever, and no command strings built by interpolation. Argument lists
only. A hostname is attacker-influenced input; the difference between
subprocess.run(argv) and shell=True is the difference between a hostname and command
execution on your own scanning host.
Cancellation kills the process group. Tools launch in a new session. Signalling only the direct child leaves the tool's own children sending traffic after you believe the run stopped.
Secrets are redacted at ingestion, before anything is written — including before raw output is persisted. Authorization headers, cookies, tokens, and credential values are never logged or stored, and a discovered credential is never tested. Using one is unauthorized access whatever the intent.
Third-party infrastructure is refused. CDN-fronted assets are tagged during DNS and HTTP inventory, IP-level findings are suppressed for CDN ranges, and no IP whose ownership is unconfirmed is actively scanned.
Skipping work is never skipping scope. --incremental may leave a host out of a
stage, but only a stage whose adapter declares itself skippable, only after the host
passed scope.authorize() like any other, and only if the manifest and summary name it
with the date it is relying on. Every uncertainty — unreadable state, a missing
fingerprint, a run that did not finish — resolves to scan. An unnecessary scan costs
requests; a wrongly skipped host costs a finding nothing in the report reveals.
Each run writes to runs/<run_id>/:
runs/<run_id>/
├── raw/<tool>/ # original tool output, unmodified except redaction
├── normalized/
│ ├── assets.jsonl # hosts, and what each stage learned
│ ├── endpoints.jsonl # URLs and API routes
│ └── findings.jsonl # normalized findings
├── reports/summary.md # coverage, counts, affected assets
└── manifest.json # scope, authorization, tool versions,
# per-stage status, refused targets
Raw output is preserved because normalization is lossy, and a finding without its source evidence cannot be validated — which makes it worthless in a report.
Records are appended as they arrive, so a crash in a later stage never destroys
earlier results. assets.jsonl is an append log during the run and is collapsed to one
record per host, atomically, at the end. If the run dies first the uncollapsed log
survives, which loses nothing.
manifest.json is the resume point. Parsing and normalization are pure functions of
preserved raw output, so kwes scan --resume <run_id> rebuilds the same state and sends
no traffic for stages already marked done.
runs/incremental-state.json sits beside run directories because it describes what is
true across runs. It is derived, never authoritative: it records what happened and grants
nothing. Deleting it is always safe and costs one full scan.
runs/is gitignored. It contains findings, raw output, and evidence from real infrastructure.
Nothing is hardcoded — not targets, credentials, binary paths, pipeline order, timeouts, rate limits, concurrency, wordlist paths, or template selections. Configuration is validated on load, and the run aborts on an invalid value rather than silently substituting a default. Untyped configuration fails at request time, against a live target; typed configuration fails at load time, before anything is touched.
| File | Holds |
|---|---|
config/pipeline.yaml |
Stage order and which stages are enabled |
config/tools.yaml |
Binary names, timeouts, and the global traffic budget |
config/profiles/<name>.yaml |
Severities, capabilities, crawl and fuzzing policy |
config/template-routing.yaml |
Which templates run against which asset kinds |
config/cdn-ranges.yaml |
Provider address ranges for third-party detection |
config/incremental.yaml |
Opt-in incremental scanning policy |
config/tools.yaml ships target_rate_limit: 10. It defaults low on purpose.
global:
target_rate_limit: 10 # requests/second reaching the target (httpx, nuclei)
discovery_rate_limit: 25 # requests/second to third-party data sources
concurrency: 3510 is slow, and that is the point — it is the rate you can run against an estate before you know which host is a marketing page and which is a payment gateway. Nothing in KWES can tell those apart, so the default assumes the worst one.
No stated rate limit is not permission to scan fast. Web application firewalls commonly rate-limit a single source address somewhere in the 50–100 range; tripping one blocks your address and ends the run — a self-inflicted outage, plus an alert somebody has to explain.
Raise it per engagement, in stages, and only as far as the authorization you hold. A critical-only sweep is roughly 1,560 requests per live URL, so 500 URLs is about 21.7 hours at 10/s and 4.3 at 50/s. If your window does not fit, split the scope into batches rather than raising the rate — two runs at a reviewed rate are safe; one run at double the rate is not.
--rate can lower the budget for one run but can never raise it: the approved figure has
to be the reviewed one, not whatever the last person typed.
Dependencies point inward. A layer may import from layers above it, never below.
domain/ plain data; imports nothing from KWES
config/ typed settings
scope/ allowlist decisions, the AuthorizedTarget type
tools/runner.py process execution; knows no specific tool
tools/adapters/ one per tool; must NOT import pipeline/
normalize/ storage/ reporting/ consume domain/; no subprocess execution
incremental/ how much work is worth doing, never what is permitted
pipeline/ the only layer that knows stage order
cli.py wiring only
incremental/ can only remove hosts from a list scope.authorize() already approved.
It has no route to add one.
These rules are tested, not just documented — tests/unit/test_architecture.py fails the
build if an adapter imports the pipeline or a module outside the runner spawns a process.
Each adapter implements five separated phases: validate availability and version, build arguments, execute, parse, normalize. Fused phases cannot be unit tested without live traffic, and there is no authorization to generate live traffic for a test.
uv sync --extra dev
uv run pytest # 735 tests, ~20s
uv run ruff check .
uv run mypyDo not run real scans during development. Adapters are tested against recorded output
fixtures in tests/fixtures/<tool>/, never live traffic. Use RFC 2606 reserved names
(example.com, *.test, *.invalid) and RFC 5737 reserved addresses (192.0.2.0/24,
198.51.100.0/24, 203.0.113.0/24).
Nothing in CI sends network traffic. The scanners are deliberately not installed there — a CI job that installed them would be a CI job that could run them.
uv run python scripts/smoke.py # offline confidence check, ~30sNew adapter code requires parser fixtures, unit tests, and failure tests for: missing binary, unexpected version, non-zero exit, malformed or truncated output, timeout, and cancellation.
See CONTRIBUTING.md and
docs/design/discovery-stages.md.
Apache-2.0. KWES orchestrates tools it does not vendor; each retains its own license.