Skip to content

Analyzer modules dropped at import time are invisible to analysis_completeness #590

Description

@udsy19

What's wrong

_discover_analyzers() in src/skillspector/nodes/analyzers/__init__.py dynamically imports every module under src/skillspector/nodes/analyzers/ and registers it as an analyzer node. If a module fails to import (ImportError from a missing optional dependency, or any other exception raised at import time), the loader logs it at ERROR and moves on:

try:
    mod = importlib.import_module(full_module_name)
except ImportError as exc:
    logger.error("Failed to import analyzer module %s: %s", module_name, exc)
    continue
except Exception as exc:
    logger.error("Error loading analyzer module %s: %s", module_name, exc)
    continue

The module is then simply absent from ANALYZER_NODE_IDS. graph.py:59 only ever builds a graph node for IDs in that list, so the dropped analyzer:

  • gets no graph node,
  • never runs node(),
  • never emits an inspection-ledger event of its own.

analysis_completeness (inspection_ledger.py) is derived entirely from ledger events and analyzer_status_events — both of which require an analyzer to have actually run. A module that never registers leaves no trace anywhere for completeness to key off: the report comes back status: "complete", is_complete: True, and the recommendation stays SAFE, --fail-on-incomplete exits 0.

Concretely: if static_patterns_data_exfiltration (or any other analyzer) fails to import because of a missing/broken optional dependency, a scan silently runs 26 of 27 analyzers and reports a complete, SAFE result — having never looked for exfiltration at all.

Precedent

This is the module-load-time twin of #554/#557: #554 reported that a single custom YARA rule file that fails to compile is dropped with only debug logging while static_yara still reports completed/SAFE, and #557 fixes it by recording the drop as a PARTIAL ledger event so completeness can see it. _discover_analyzers() has the identical fail-open shape, one level up — for a whole analyzer module rather than one rule file — and #557 does not touch it (it's scoped to static_yara.py).

Reproduction

$ python -c "
from skillspector.nodes import analyzers
print(len(analyzers.ANALYZER_NODE_IDS))  # 27 on a clean tree
"

Injecting a single ImportError for static_patterns_data_exfiltration and re-running discovery in-process drops it to 26 registered analyzers with no surface anywhere (ANALYZER_LOAD_ERRORS/FAILED_ANALYZERS/etc. do not exist) for a consumer to detect the difference between 27 and 26.

I have a fix ready (adds an ANALYZER_LOAD_ERRORS registry and folds it into analysis_completeness via a PARTIAL SYSTEM ledger event, matching #557's approach) and will open a PR shortly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions