Repository navigation
Conversation
Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.51.0 to 0.52.0. - [Release notes](https://github.com/Kludex/uvicorn/releases) - [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md) - [Commits](Kludex/uvicorn@0.51.0...0.52.0) --- updated-dependencies: - dependency-name: uvicorn dependency-version: 0.52.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [fastapi](https://github.com/fastapi/fastapi) from 0.140.0 to 0.141.1. - [Release notes](https://github.com/fastapi/fastapi/releases) - [Commits](fastapi/fastapi@0.140.0...0.141.1) --- updated-dependencies: - dependency-name: fastapi dependency-version: 0.141.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pip](https://github.com/pypa/pip) from 26.2 to 26.2.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](pypa/pip@26.2...26.2.1) --- updated-dependencies: - dependency-name: pip dependency-version: 26.2.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [packaging](https://github.com/pypa/packaging) from 26.2 to 26.3. - [Release notes](https://github.com/pypa/packaging/releases) - [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst) - [Commits](pypa/packaging@26.2...26.3) --- updated-dependencies: - dependency-name: packaging dependency-version: '26.3' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* test: parallelize suite with pytest-xdist
Test servers bound fixed ports (8888/9999/5556) and 58 test files hardcoded
those in URLs, so the suite could only ever run serially. CI spent ~58 min in
a single pytest call while using one of four cores.
- bbot/test/ports.py: resolve server ports per xdist worker
- conftest.py: tolerate listener=None (workers inherit _BBOT_LOGGING_SETUP)
- 57 test files: hardcoded host:port -> constants from bbot.test.ports
localhost and 127.0.0.1 spellings are kept distinct; virtualhost tests assert
on the Host header, where they are not interchangeable. test_regexes.py is
untouched, its literals are regex fixtures, not references to the test server.
* test: isolate per-worker state for parallel runs
Beyond the server ports, three things were process-global and would corrupt
each other under xdist:
- BBOT home (/tmp/.bbot_test): shared caches and scan output, and
pytest_sessionfinish rmtree's it. The first worker to finish would delete
the directory out from under every worker still running.
- uvicorn (8978) and websockets (8765) servers: bound fixed ports.
- docker-backed tests (kafka, elastic, mongo, mysql, nats, postgres,
rabbitmq): fixed container names and fixed host port bindings, so they
genuinely cannot run concurrently. They are now pinned to a single worker
via xdist_group, keyed off the skip_distro_tests flag they already set.
Renamed ports.py -> worker.py since it now covers more than ports. Port bases
were chosen with distinct mod-100 residues; verified collision-free and clear
of the ephemeral range for up to 64 workers.
* ci: run tests in parallel
pytest-xdist added to the dev group; workflow runs with -n logical
--dist loadgroup. loadgroup (not loadfile) is required so the xdist_group
marker on the docker-backed tests is honored.
* test: bound docker service waits and size worker count by memory
Two problems surfaced running the suite 16-way on a 16-core/15GB box.
Elasticsearch was OOM-killed (exit 137): 16 workers hold ~10GB, leaving
under 3GB, and it wants ~2GB of heap. The test then waited for it in an
unbounded 'while True', so a container that was already dead hung the run
at 99% until the 1200s global timeout. Same unbounded loop in the mongo
test. Both now stop after CONTAINER_READY_TIMEOUT (180s, overridable) and
say what happened, including the exit-137 hint.
'-n logical' is the wrong worker count when cores outnumber GB. worker_count.py
bounds by cores AND by available memory, keeping 5GB free for the docker-backed
services. A 4-core/16GB github runner is unaffected (still 4); this box goes
16 -> 15. Override with BBOT_TEST_WORKERS.
* test: fix parallel-run regressions missed by the first port pass
Running the suite under xdist surfaced failures that were my own, not
pre-existing. Four separate gaps:
Logging. Workers inherit _BBOT_LOGGING_SETUP from the pytest parent, so
BBOTLogger skipped setup entirely and debug.log was never written. The
previous 'if listener is not None' guard hid that instead of fixing it,
and log-reading tests failed with 'assert -retries 1 in ""'. Workers now
clear the flag so each sets up its own logging, and conftest raises if a
process somehow still has no listener.
Port literals the first pass did not match, because it only looked for
127.0.0.1:PORT and localhost:PORT:
* synthetic hostnames carrying the port (test.example:8888, bad.dns:8888)
* the bbot_other_httpservers fixture, six servers hardcoded to 8888/8889
* config values like {"ports": "8888"}, which are not URLs
* 127.0.0.0:9999 (the .0 address, excluded along with .1)
* filename/slug spellings: 127.0.0.1.8888 and http-127-0-0-1-8888
* mocked masscan JSON, whose port is what the module 'discovers'
Hardcoded /tmp/.bbot_test paths, which my own per-worker home broke.
Adds BBOT_TEST_PORT_OFFSET so a second copy of the suite can run beside
one that already owns the base ports, instead of the two colliding and
looking like a worker-isolation bug.
* test: remove ordering dependency and fix integer port comparisons
Full-suite A/B surfaced the last regressions from parallelization.
asset_inventory: TestAsset_InventoryEmitPrevious runs with use_previous=True,
which reads the asset-inventory.csv left behind by an earlier scan of the same
name. It was relying on TestAsset_Inventory having already run. That is
order-dependent, and xdist guarantees neither order nor worker affinity, so it
failed whenever it was scheduled first or landed on another worker (each worker
has its own BBOT home). It now seeds the CSV itself in setup_before_prep, which
makes it independent and drops its runtime from ~20s to ~4s since it no longer
needs the portscan.
Port comparisons written as bare integers (e.port == 8888) in test_scope.py and
test_module_fingerprintx.py. The earlier passes only rewrote string literals, so
these were missed.
Also groups tests that share a scan_name onto one worker, alongside the existing
docker grouping. Not sufficient on its own for the ordering problem above, but
correct for co-locating tests that share a scan home.
* test: stop hardcoding the BBOT home directory name in path assertions
git_clone writes cloned repos to git_repos/<bbot home name>/<repo>, so the
home directory's *basename* shows up inside the paths these tests assert on.
They spelled it '.bbot_test', which stopped matching once each xdist worker
got its own home ('.bbot_test_gw0').
Export BBOT_TEST_DIR_NAME from worker.py and use it instead.
Worth noting these reproduce with '-n 1', a single worker with no concurrency
at all. They were never race conditions, just the per-worker home rename, and
checking '-n 1' before '-n 4' would have found that much faster.
* test: fix port in a regex literal and a self-poisoning cli precondition
test_module_robots matched the server URL with a raw regex string, where the
port sits behind escaped dots (r"http://127\.0\.0\.1:8888/..."). The earlier
rewrites looked for plain literals and skipped it, so the test failed whenever
it landed on a worker whose port was not 8888.
test_cli_presets asserts a scan output dir does not exist yet, and removes it
at the end of the test. Any earlier failure or interrupted run leaves it in
place, and every later run against that BBOT home then fails on the assertion
rather than on the real problem. It now clears the directory up front, so the
precondition is established instead of assumed.
Both showed up as intermittent (~25% of runs) rather than deterministic, which
is what made them the last two to be found.
* test: cut dead code and de-duplicate the container wait
Net removal pass over the parallelization work.
Removed:
- BBOT_TEST_PORT_OFFSET and port_offset(). Added for running two copies of the
suite side by side while benchmarking. Nothing outside worker.py used it.
- LOCALHOST_SSL_HOSTPORT, the BASE_* port aliases, and the worker_index()
wrapper. All unused or single-use indirection.
- OTHER_HTTPSERVER_PORT, which was literally 'OTHER_HTTPSERVER_PORT =
HTTPSERVER_PORT'. Renamed to HTTPSERVER_PORT/HTTPSERVER_PORT_ALT at its one
call site.
- The scan_name xdist grouping. It compensated for asset_inventory's ordering
dependency, which the previous commit removed by having the test seed its own
input, so the grouping no longer does anything. Verified: 3x green at 4
workers without it.
- worker_count's -v flag and MIN_WORKERS floor. The floor guaranteed 2 workers
even when the memory arithmetic said there was room for none, which is an
assumption I never validated; a 2-core/4GB box now correctly gets 1.
- A dead 'import time' the refactor exposed.
De-duplicated:
- The container readiness loop was copy-pasted between elastic (sync) and mongo
(async), each with its own deadline handling and OOM hint. Now one
wait_for_container() helper that takes either kind of callable.
worker.py 154 -> 100 lines, worker_count.py 64 -> 38, conftest hook 40 -> 13.
* test: fix gowitness hardcoded ports and shared home directory
Caught by CI on this PR, not locally: gowitness needs its binary installed, so
both classes error out on my machine and never reached the assertions.
Two bugs:
The multiport test asserted on bare port substrings ('8888' in url) rather than
URL-shaped literals, so the earlier rewrites did not match them. On any worker
but the first the ports are 9188/10299 and the assertion could never pass.
Both classes also pinned an explicit home directory under /tmp and rmtree'd it
at class-definition time, which every worker does to the same path. Workers
deleted each other's dependency installs mid-run, which is where the
FileNotFoundError on the depsinstaller artifact came from. Now worker-scoped.
Worth noting these run with deps.behavior=force_install, so each worker
installs its own copy of chrome, ~650MB. That is fine on a runner where /tmp is
disk-backed, but it multiplies with worker count.
* test: fix a hardcoded port in the virtualhost canary regex
The force_basehost wildcard-canary check still matched a literal 8888, so on
any worker but gw0 it never matched. Both that branch and the fallthrough
return 404, which is why it did not show up as a failure.
* test: give the remaining fixed /tmp paths a per-worker name
Same class of bug as the gowitness home directory that CI caught: paths that
are fine serially but shared once workers run concurrently.
- test_python_api used one BBOT home across two tests, so the dep installer
wrote and read the same artifacts directory from two workers
- test_presets wrote a fixed custom preset dir
- test_command and test_web wrote fixed scratch files, then read them back
* ci: skip the benchmark PR comment on fork pull requests
A pull_request run from a fork gets a read-only GITHUB_TOKEN regardless of the
permissions block, so github-script fails with
HttpError: Resource not accessible by integration (403)
POST /repos/.../issues/{n}/comments
and takes the whole job red. Every fork PR has a failing Performance Benchmarks
check for a reason that has nothing to do with performance, which trains people
to ignore a red check.
Skip the comment step when the head repo is not this repo. The benchmark still
runs and the report is still uploaded as an artifact, so nothing is lost except
a comment that could never have been posted.
* test: cover the worker count and isolation helpers
Codecov flagged worker_count.py at 0 percent: the workflow runs it as a script,
so pytest never imported it. worker.py was only covered incidentally.
These two modules decide how many workers run and how each one gets its own
ports and directories, so a silent regression there either overloads the runner
or lets workers collide.
Verified by mutation: removing the max(1, ...) clamp, dropping the memory cap,
and swapping sched_getaffinity for os.cpu_count each fail these tests.
Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.52.0 to 0.52.1. - [Release notes](https://github.com/Kludex/uvicorn/releases) - [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md) - [Commits](Kludex/uvicorn@0.52.0...0.52.1) --- updated-dependencies: - dependency-name: uvicorn dependency-version: 0.52.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shane Engelman <haroboy876@gmail.com>
Bumps [cachetools](https://github.com/tkem/cachetools) from 7.1.6 to 7.1.7. - [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst) - [Commits](tkem/cachetools@v7.1.6...v7.1.7) --- updated-dependencies: - dependency-name: cachetools dependency-version: 7.1.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shane Engelman <haroboy876@gmail.com>
Bumps [websockets](https://github.com/python-websockets/websockets) from 16.0 to 16.1.1. - [Release notes](https://github.com/python-websockets/websockets/releases) - [Commits](python-websockets/websockets@16.0...16.1.1) --- updated-dependencies: - dependency-name: websockets dependency-version: 16.1.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shane Engelman <haroboy876@gmail.com>
* Bump ruff from 0.15.22 to 0.16.1 Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.22 to 0.16.1. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.22...0.16.1) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.16.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * style: ruff format for ruff 0.16 ruff 0.16 formats python code blocks inside markdown by default, so the existing docs fail 'ruff format --check' under the bumped version. Apply the formatter (10 files, docs only) to make the lint step pass. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shane Engelman <haroboy876@gmail.com>
* fix custom YARA rule emission and rule splitting CustomExtractor.process() called .get() on YaraRuleSettings, which has no get(), so every custom rule match raised AttributeError before the event was emitted. The scan logged one error per match and reported zero findings. Replace the regex-based rule splitter with a brace-depth-aware parser that skips comments, string literals, and regex literals, so braces inside them no longer truncate or drop rules. Rule names now tolerate arbitrary whitespace, import statements are collected and preserved, and unparseable input fails loudly instead of silently dropping a rule. * test: use per-worker HTTPSERVER_URL in custom YARA tests The two custom-YARA module tests hardcoded http://127.0.0.1:8888/ as their target. Under pytest-xdist the httpserver port is offset per worker (worker_port(8888) in bbot/test/worker), so 8888 is only correct on gw0. On any other worker the scan hit the wrong (or no) server, so no HTTP_RESPONSE was produced, the custom rule never matched, and the test failed with 'produced no FINDING, got: []'. TestExcavateYaraCustomEdgeCases passed only because it happened to land on gw0. Point both at HTTPSERVER_URL like every other test in this file. Verified green under -n 2 --dist loadgroup, which reproduced the original failure. --------- Co-authored-by: Shane Engelman <shane.engelman@blacklanternsecurity.com>
Under spawn, retiring a worker every 25 tasks forces a cold re-import of the callback module (~742ms for badsecrets). badsecrets runs on every HTTP_RESPONSE and leaks ~0 memory (RSS flat to 5000 tasks), so 25 costs a ~25x throughput drop and ~130x p95 blowup for no benefit. 250 recovers full throughput while keeping the safety net for the heavy infrequent tenants (kreuzberg, bevigil, waf_bypass).
Bumps [pip](https://github.com/pypa/pip) from 26.2 to 26.2.1. - [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst) - [Commits](pypa/pip@26.2...26.2.1) --- updated-dependencies: - dependency-name: pip dependency-version: 26.2.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [tldextract](https://github.com/john-kurkowski/tldextract) from 5.3.1 to 5.3.2. - [Release notes](https://github.com/john-kurkowski/tldextract/releases) - [Changelog](https://github.com/john-kurkowski/tldextract/blob/master/CHANGELOG.md) - [Commits](john-kurkowski/tldextract@5.3.1...5.3.2) --- updated-dependencies: - dependency-name: tldextract dependency-version: 5.3.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [packaging](https://github.com/pypa/packaging) from 26.2 to 26.3. - [Release notes](https://github.com/pypa/packaging/releases) - [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst) - [Commits](pypa/packaging@26.2...26.3) --- updated-dependencies: - dependency-name: packaging dependency-version: '26.3' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.16.1 to 0.16.2. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.16.1...0.16.2) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.16.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 4.6.1 to 4.6.2. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v4.6.1...v4.6.2) --- updated-dependencies: - dependency-name: pre-commit dependency-version: 4.6.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [xxhash](https://github.com/ifduyue/python-xxhash) from 3.8.1 to 4.0.0. - [Release notes](https://github.com/ifduyue/python-xxhash/releases) - [Changelog](https://github.com/ifduyue/python-xxhash/blob/master/CHANGELOG.rst) - [Commits](ifduyue/python-xxhash@v3.8.1...v4.0.0) --- updated-dependencies: - dependency-name: xxhash dependency-version: 4.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Surface Response.decode_error on HTTP_RESPONSE events and tag them decode-error, so nothing treats an undecoded body as content. Pass redirect_cookies through WebHelper.request().
A body that fails to decode is no longer offered as `body` or parsed for a `title`; `decode_error` carries the reason instead. Drops the `decode-error` tag, which nothing read. Also bounds the blasthttp pin at <0.11, matching the other dependencies.
Bumps [lxml](https://github.com/lxml/lxml) from 6.1.1 to 6.1.2. - [Release notes](https://github.com/lxml/lxml/releases) - [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt) - [Commits](lxml/lxml@lxml-6.1.1...lxml-6.1.2) --- updated-dependencies: - dependency-name: lxml dependency-version: 6.1.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.16.2 to 0.16.3. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.16.2...0.16.3) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.16.3 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 2.0.5 to 2.0.7. - [Release notes](https://github.com/mkdocstrings/python/releases) - [Changelog](https://github.com/mkdocstrings/python/blob/main/CHANGELOG.md) - [Commits](mkdocstrings/python@2.0.5...2.0.7) --- updated-dependencies: - dependency-name: mkdocstrings-python dependency-version: 2.0.7 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.52.1 to 0.52.4. - [Release notes](https://github.com/Kludex/uvicorn/releases) - [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md) - [Commits](Kludex/uvicorn@0.52.1...0.52.4) --- updated-dependencies: - dependency-name: uvicorn dependency-version: 0.52.4 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* run compute_simhash and collapse_urls in the thread executor waf_bypass was submitting compute_simhash on full response bodies through the process pool. xxhash releases the GIL and simhash truncates its input to 3 KB internally, so the pickle-to-worker cost dominated the actual work, and any wedged worker could stall the whole scan for 300 s before helper._reset_process_pool recovered. bevigil.collapse_urls had the same profile at smaller scale. Neither operation needs process isolation. Move both to run_in_executor_cpu so a hung callable can no longer trigger a pool reset. Kreuzberg PDF extraction and badsecrets retain run_in_executor_mp where crash isolation is load-bearing. * Tally simhash bits with Counter instead of a per-bit Python loop * Move long GIL-bound work to the process executor * Document what each executor is for, drop dead extract_params_html
* Require positive confirmation for lightfuzz code-change findings Each of these asserted a vulnerability from a status-code flip or a coincidental statistical signal, with no content-level evidence that the vuln class was present. sqli: the '/'' status flip alone now proves nothing. Suppress any transition into 403, verify the flip is quote-specific with a benign one-vs-two-character control, and require a TRUE/FALSE boolean pair to produce a reproducible content differential before emitting. serial: Error Resolution now sends a same-encoding, same-length twin of the payload with a scrambled magic header. If that also resolves the error, the outcome doesn't depend on the payload's content, so the value is being parsed (e.g. as a URI host) rather than deserialized. crypto: run the structured-identifier discrimination on every candidate pair rather than only those with a leading-zero run, and reject pairs whose XOR zeros out at scattered offsets, which is a shared field template rather than two ciphertexts under one keystream. Also surface archived provenance on findings: FINDING now renders the snapshot URL in output.txt and as a top-level archive_url in output.json, so an archived severity is never read as a claim about the live host. * Strip HTML-escaped payload reflections in the sqli boolean gate * Restore the WAF-signature gate on 403 quote probes * Drop the interior-zeros structured-ID rule * Name the sqli and serial detection thresholds * Trim explanatory comments in the new detection code * Narrow the corrupt_payload decode catch to ValueError
Bumps [pycryptodome](https://github.com/Legrandin/pycryptodome) from 4.1.0 to 4.2.0. - [Release notes](https://github.com/Legrandin/pycryptodome/releases) - [Changelog](https://github.com/Legrandin/pycryptodome/blob/master/Changelog.rst) - [Commits](Legrandin/pycryptodome@v4.1.0...v4.2.0) --- updated-dependencies: - dependency-name: pycryptodome dependency-version: 4.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cachetools](https://github.com/tkem/cachetools) from 7.2.4 to 7.2.7. - [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst) - [Commits](tkem/cachetools@v7.2.4...v7.2.7) --- updated-dependencies: - dependency-name: cachetools dependency-version: 7.2.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [fastapi](https://github.com/fastapi/fastapi) from 0.159.1 to 0.162.0. - [Release notes](https://github.com/fastapi/fastapi/releases) - [Commits](fastapi/fastapi@0.159.1...0.162.0) --- updated-dependencies: - dependency-name: fastapi dependency-version: 0.162.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.17.1 to 0.17.3. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.17.1...0.17.3) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.17.3 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…ruff-0.17.3 Bump ruff from 0.17.1 to 0.17.3
* docs: make AGENTS.md an index of the org standards AGENTS.md loads into every bot context on every turn and had grown to 32KB. The principles, toolchain, git, and AI disclosure sections now live once in blacklanternsecurity/.github/standards and CONTRIBUTING.md, so AGENTS.md links to them instead of restating them. The module authoring reference moves verbatim to docs/dev/module_reference.md and joins the mkdocs nav. The local issue templates void every org default template, including config.yml and the RFC form, so they are removed. Refs blacklanternsecurity/bbot-enterprise#157 * docs(agents): index existing dev docs instead of adding module_reference module_reference.md copied the old AGENTS.md body into docs/dev, where module_howto.md, tests.md, helpers/, architecture.md, and the BaseModule API page already cover the same ground. It also retyped every attribute default from bbot/modules/base.py, so two sources would drift. Drop it and its nav entry, and point AGENTS.md at the existing pages and at base.py as the canonical attribute list. The two bullets restating modularity and per-module tests are org principles.md and testing.md, already indexed. Test commands now use the uv flow from docs/dev/tests.md, and toolchain rows reference pyproject.toml rather than retyping versions. * docs(agents): use a path for the single module test example `uv run pytest -k test_module_robots` printed no result line locally. Selecting the module test file by path ran and passed: 1 passed in 2.96s * docs(agents): restore core principles, architecture, and helpers Also document the per-worker test port constants in bbot/test/worker.py, and drop the hardcoded ports from docs/dev/tests.md. * fix(agents): make the CLAUDE.md import resolve Claude Code reads an @ import up to the next whitespace, so the trailing period made it look for AGENTS.md. and skip it silently. --------- Co-authored-by: liquidsec <paul.mueller08@gmail.com>
…fastapi-0.162.0 Bump fastapi from 0.159.1 to 0.162.0
Automated Docs Update
…cachetools-7.2.7 Bump cachetools from 7.2.4 to 7.2.7
Bumps [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) from 1.4.0 to 1.4.1. - [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases) - [Commits](pytest-dev/pytest-asyncio@v1.4.0...v1.4.1) --- updated-dependencies: - dependency-name: pytest-asyncio dependency-version: 1.4.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.1.1 to 9.2.0. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@9.1.1...9.2.0) --- updated-dependencies: - dependency-name: pytest dependency-version: 9.2.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Remove deepdiff dependency
…pytest-9.2.0 Bump pytest from 9.1.1 to 9.2.0
…pycryptodome-4.2.0 Bump pycryptodome from 4.1.0 to 4.2.0
Bump version to 3.1.0
Drop Python 3.10, add Python 3.15
Update trufflehog to 3.99.8
Replace massdns with blastdns 2.0
Add scanmalware subdomain module
dns.threads goes from 10 to 50 queries in flight per resolver, with adaptive pacing as the backstop. The new dns.max_concurrency (256) is passed to blastdns explicitly and caps the total across resolvers. dnsresolve now runs half as many workers as the client has slots, at least 10, instead of a fixed 25.
…faults Raise DNS concurrency defaults and size dnsresolve from them
en0f
requested changes
Nov 3, 2026
en0f
left a comment
en0f
left a comment
There was a problem hiding this comment.
One thing broken.
Broken: lightfuzz never fuzzes GET, COOKIE or HEADER parameters on WAF-tagged hosts, even when the WAF isn’t blocking anything (bbot/modules/lightfuzz/lightfuzz.py:365, :376, :385)
The rule: filter_event (around line 456) asks nowafpls whether the WAF is in the way. When the answer is STATUS_NO_INTERFERENCE, it lets the event through (“nothing is gating the payload, so fuzz normally”).
The bug: handle_event then ignores that answer and decides from the waf tag alone:
python
in_bypass_mode = "waf" in event.tags and self.avoid_wafs == "try_bypasses"
skip_native_pass = in_bypass_mode and original_type not in ("POSTPARAM", "BODYJSON")
How it fails: a host gets the waf tag just for sitting behind a CDN or WAF provider, and the probe finds nothing blocking. A GETPARAM, COOKIE or HEADER parameter on that host passes the filter. The native pass is skipped, so only the baseline request goes out and no submodule runs. try_post_as_get is also turned off for POST parameters on that host.
Impact: try_bypasses is the default, so under default config lightfuzz checks nothing but POST bodies on any WAF-tagged host. Nothing logs it.
Why CI is green: the existing tests only check POST bodies, and that the filter accepts the event. Nothing asserts that the GET-side submodules run.
Fix: set in_bypass_mode only when (await self.helpers.nowafpls.is_bypassable(event)).bypassed is true. The result is cached per host, so this adds no requests.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 3.1.0
These notes cover everything since 3.0.2. The full migration guide is in
docs/migration/3.1_breaking_changes.md(#3594).Breaking changes
requires-pythonis now>=3.11,<3.16; CI tests 3.11 through 3.15. 3.10 reaches EOL this month.portfiltermodule removed (Filter CDN/WAF ports by default and remove the portfilter module #3534). CDN/WAF port filtering now happens in core and is on by default; opt out with--allow-cdn-ports.config.module_dirsis deprecated and ignored;module_dirsis honored only as a top-level preset key. BBOT asks before loading each custom module folder (-yanswers yes,-Sno longer implies-y), refuses world-writable folders and files, and refuses custom modules that reuse an existing module name. Presetconditionsare read-only, andversion/homeare validated.dns.threadsgoes from 10 to 50 queries in flight per resolver, capped by the newdns.max_concurrency(256). Scans send up to 5x more concurrent ordinary DNS queries by default; lowerdns.threadsfor a fragile resolver.affiliatetag on in-scope events (Don't tag in-scope events as affiliate #3524) can affect existing presets and output consumers.dns.nameserversnow aborts the scan (Replace massdns with blastdns 2.0 #3457). The DNS helper is built during scan prep, so a typo fails fast instead of producing a "completed" scan with zero results.Highlights
DNS (#3457)
--custom-nameserversflag, replacing the system resolvers for normal resolution.dns.threads10 to 50, newdns.max_concurrency(256), and the dnsresolve worker count derived from the client's total slots. On a large wildcardsubdomain-enumscan this halved wall time compared with 3.0.2, with the same subdomains found.is_wildcard(Fix nested wildcard detection in is_wildcard #3536).baddns 2.5.0 (#3457)
DELEGATIONsubmodule, enabled by default and in thebaddns/baddns-heavypresets. Checks_acme-challenge,_dmarcand 15 common DKIM selectors for dangling delegations, with wildcard and generic-finding filtering to keep volume down.NSgained a lame-delegation finding (ceiling raised to HIGH/CONFIRMED).WordMatcheris passed through, matching all signatures in a single pass.blastdnscapped at<2.1to match baddns's own ceiling.New modules
favicon: technology detection from favicon hashes (Add favicon module for technology detection #3532).js_unpacker: detects and unpacks obfuscated JavaScript (Add js_unpacker module for detecting and unpacking obfuscated JS #3335).nowafpls: Cloudflare WAF bypass detector (Add nowafpls Cloudflare WAF bypass detector #3336).otilabs: subdomain enumeration (Add otilabs subdomain module #3496).scanmalware: subdomains from ScanMalware's public scan archive (Add scanmalware subdomain module #3528).Modules
/path, which crt.sh requires (crt: request an explicit / path (blasthttp sends an empty path, which crt.sh rejects with 400) #3428).excavate
<base href>(Resolve excavated relative URLs against <base href> #3522).Core
api_requestrate limiting, plus config deprecation support viaField(deprecated=True)(Overhaul api_request rate limiting and add config deprecation support #3461).affiliate(Don't tag in-scope events as affiliate #3524).HTTP_RESPONSEevents carryurl_extension(Set url_extension on HTTP_RESPONSE events #3406).ClosestHostEventkeeps its inherited port, and the port is restored from JSON (Keep inherited port on ClosestHostEvent and restore port from JSON #3462).download()respectscache_hrsin its freshness check (Respectcache_hrsindownload()freshness check #3563).Target.add()(fix TypeError when a hostless target is sorted against a /32 #3391).Performance
compute_simhash, with corrected executor placements (Speed up compute_simhash and correct executor placements #3331).response_to_jsonremoved in favor ofresponse_to_event_dict(Delete duplicate response_to_json, unify on response_to_event_dict #3473).Dependencies
deepdiffremoved, along withcacheboxandorderly-set(Remove deepdiff dependency #3604). ansible-core resolves to 2.22 on Python 3.13+, which is what makes 3.15 installable.Maintenance
Refs #3555