Repository navigation
Python reachables: remove quadratic hotspots and resolve plain imports in nested files - #189
Merged
Merged
Conversation
…bleByFlows reachableByInternal tested every result against the list of sources, and reachableByFlows tested every path element against the list of starting points. Both were linear scans, so the cost grew with sources times results. On a large Python project (django-DefectDojo) that scan was 59% of the CPU samples of a reachables run. Node equality is by id with a matching hash, so the sets answer exactly what List.contains did.
membersMatchingImports ran a regex over every type decl of the graph for every import. On a 400-package fixture that was 54% of the CPU samples of a whole reachables run. Only module-scope decls can contribute members, so they are collected once per pass and filtered with String.contains, which is what the quoted .*path.* regex matched, memoised per namespace.
Every task that starts at a parameter looked up the method references of its method by filtering all METHOD_REF nodes, as the old TODO there suggested caching. The per-graph index stores node ids (so it never pins a graph in memory) and is rebuilt when the METHOD_REF count changes.
The purl, the component type and every description tag each re-ran the same regex over all calls, identifiers, parameters and methods. The matches are now computed once per component and reused, with tags added in the same order.
`import X` is absolute, but the resolver prefixed the importing file's directory, so `import pickle` in introduction/views.py became calls to introduction.pickle.loads and `import subprocess` in a package became pkg.subprocess.getoutput. Sink rules match full names, so those calls were never tagged, and every flow into them was cut short or lost. This also changes which module a plain import resolves to, by policy: a module X ingested in the importer's own directory is preferred over a top-level X. That is Python's answer when the file runs as a script (its directory is on sys.path) and not under `python -m`, where a same-named installed package would win; for reachables, following the code that is in the graph is the safer choice. Among siblings, a regular package X/ is preferred over a module file X.py, as CPython's finder does. Anything else resolves as the top-level X, as before for files at the input root. On a fixture of 40 copies of the framework and flow probes, every copy now reports exactly the flows of the single-directory run; before, none did.
This was referenced Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speeds up reachables on large Python projects without changing results, and fixes how plain imports resolve in nested files. Part of cdxgen/cdxgen#4419.
Performance (hotspots found with JFR on django-DefectDojo and on a 400-copy Python fixture):
reachableByFlowstested each result against a list of sources. This was 59% of CPU on DefectDojo; it is now a set.CdxPass.tagGenericnow matches each node set once per component.Import resolution (changes results):
import pickleresolved to<dir>.pickle.loads, so those sinks were never tagged. Plain imports now resolve to the top-level module.python -mPython would pick the installed module instead.Testing
DottedFullNamesTestscases; fullsbt testpasses.Related: cdxgen/cdxgen#4433 (cdxgen side), AppThreat/atom#267, AppThreat/atom-parsetools#14.