fix: [ismalicious] count threat listings only, read categories per listing, add docs

README.md

@@ -94,6 +94,7 @@ For further Information see the [license file](https://misp.github.io/misp-modul
 * [IPInfo.io Lookup](https://misp.github.io/misp-modules/expansion/#ipinfo.io-lookup) - An expansion module to query ipinfo.io to gather more information on a given IP address.
 * [IPQualityScore Lookup](https://misp.github.io/misp-modules/expansion/#ipqualityscore-lookup) - IPQualityScore MISP Expansion Module for IP reputation,        Email Validation, Phone Number Validation, Malicious Domain,Malicious URL Scanner,Malicious File Scanner & Compromised username, Password, Email  
 * [IPRep Lookup](https://misp.github.io/misp-modules/expansion/#iprep-lookup) - Module to query IPRep data for IP addresses.
+* [isMalicious Lookup](https://misp.github.io/misp-modules/expansion/#ismalicious-lookup) - Query isMalicious for IP, domain, hostname, and URL reputation.
 * [Ninja Template Rendering](https://misp.github.io/misp-modules/expansion/#ninja-template-rendering) - Render the template with the data passed
 * [Joe Sandbox Import](https://misp.github.io/misp-modules/expansion/#joe-sandbox-import) - Query Joe Sandbox API with a submission url to get the json report and extract its data that is parsed and converted into MISP attributes and objects.
 * [Joe Sandbox Submit](https://misp.github.io/misp-modules/expansion/#joe-sandbox-submit) - A module to submit files or URLs to Joe Sandbox for an advanced analysis, and return the link of the submission.

documentation/README.md

@@ -1456,6 +1456,35 @@ Module to query IPRep data for IP addresses.
 
 -----
 
+#### [isMalicious Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ismalicious.py)
+
+<img src=logos/ismalicious.png height=60>
+
+Query isMalicious for IP, domain, hostname, and URL reputation.
+[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ismalicious.py)]
+
+- **features**:
+>The module takes an IP, domain, hostname or URL attribute (the domain side of a domain|ip) and queries GET /check on the isMalicious API. It returns a text summary with the malicious flag, risk score, categories and source count. Only threat listings are counted: listings that describe what an indicator is (cloud and CDN ranges, Tor exits, DoH resolvers, ad lists, allowlists) are left out of the categories and of the source count. Hover and expansion share the same handler. The queried indicator is sent over TLS; nothing else leaves the MISP instance besides the configured API key.
+
+- **config**:
+> - api_key
+> - api_url
+
+- **input**:
+>An IP address, domain, hostname or URL.
+
+- **output**:
+>Text attributes with the isMalicious reputation summary.
+
+- **references**:
+> - https://ismalicious.com/integrations/misp
+> - https://ismalicious.com/api-docs
+
+- **requirements**:
+>An isMalicious API key (free accounts are available): the X-API-KEY value, Base64 of apiKey:apiSecret, shown under API credentials on the isMalicious account page.
+
+-----
+
 #### [Ninja Template Rendering](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/jinja_template_rendering.py)
 
 Render the template with the data passed

documentation/mkdocs/expansion.md

@@ -1452,6 +1452,35 @@ Module to query IPRep data for IP addresses.
 
 -----
 
+#### [isMalicious Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ismalicious.py)
+
+<img src=../logos/ismalicious.png height=60>
+
+Query isMalicious for IP, domain, hostname, and URL reputation.
+[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ismalicious.py)]
+
+- **features**:
+>The module takes an IP, domain, hostname or URL attribute (the domain side of a domain|ip) and queries GET /check on the isMalicious API. It returns a text summary with the malicious flag, risk score, categories and source count. Only threat listings are counted: listings that describe what an indicator is (cloud and CDN ranges, Tor exits, DoH resolvers, ad lists, allowlists) are left out of the categories and of the source count. Hover and expansion share the same handler. The queried indicator is sent over TLS; nothing else leaves the MISP instance besides the configured API key.
+
+- **config**:
+> - api_key
+> - api_url
+
+- **input**:
+>An IP address, domain, hostname or URL.
+
+- **output**:
+>Text attributes with the isMalicious reputation summary.
+
+- **references**:
+> - https://ismalicious.com/integrations/misp
+> - https://ismalicious.com/api-docs
+
+- **requirements**:
+>An isMalicious API key (free accounts are available): the X-API-KEY value, Base64 of apiKey:apiSecret, shown under API credentials on the isMalicious account page.
+
+-----
+
 #### [Ninja Template Rendering](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/jinja_template_rendering.py)
 
 Render the template with the data passed

documentation/mkdocs/index.md

@@ -64,6 +64,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
 * [IPInfo.io Lookup](https://misp.github.io/misp-modules/expansion/#ipinfo.io-lookup) - An expansion module to query ipinfo.io to gather more information on a given IP address.
 * [IPQualityScore Lookup](https://misp.github.io/misp-modules/expansion/#ipqualityscore-lookup) - IPQualityScore MISP Expansion Module for IP reputation, Email Validation, Phone Number Validation, Malicious Domain and Malicious URL Scanner.
 * [IPRep Lookup](https://misp.github.io/misp-modules/expansion/#iprep-lookup) - Module to query IPRep data for IP addresses.
+* [isMalicious Lookup](https://misp.github.io/misp-modules/expansion/#ismalicious-lookup) - Query isMalicious for IP, domain, hostname, and URL reputation.
 * [Ninja Template Rendering](https://misp.github.io/misp-modules/expansion/#ninja-template-rendering) - Render the template with the data passed
 * [Joe Sandbox Import](https://misp.github.io/misp-modules/expansion/#joe-sandbox-import) - Query Joe Sandbox API with a submission url to get the json report and extract its data that is parsed and converted into MISP attributes and objects.
 * [Joe Sandbox Submit](https://misp.github.io/misp-modules/expansion/#joe-sandbox-submit) - A module to submit files or URLs to Joe Sandbox for an advanced analysis, and return the link of the submission.

misp_modules/modules/expansion/ismalicious.py

@@ -12,26 +12,33 @@
     "format": "misp_standard",
 }
 moduleinfo = {
-    "version": "1.0",
+    "version": "1.1",
     "author": "isMalicious",
     "description": "Query isMalicious for IP, domain, hostname, and URL reputation.",
     "module-type": ["expansion", "hover"],
     "name": "isMalicious Lookup",
-    "logo": "",
-    "requirements": ["An isMalicious API key."],
+    "logo": "ismalicious.png",
+    "requirements": [
+        "An isMalicious API key (free accounts are available): the X-API-KEY value, Base64 of apiKey:apiSecret,"
+        " shown under API credentials on the isMalicious account page."
+    ],
     "features": (
-        "The module takes an IP, domain, hostname or URL attribute and queries GET /check on the isMalicious API."
-        " It returns a text summary with the malicious flag, risk score, categories and source count. Hover and"
-        " expansion share the same handler. The queried indicator is sent over TLS; nothing else leaves the MISP"
-        " instance besides the configured API key."
+        "The module takes an IP, domain, hostname or URL attribute (the domain side of a domain|ip) and queries"
+        " GET /check on the isMalicious API. It returns a text summary with the malicious flag, risk score,"
+        " categories and source count. Only threat listings are counted: listings that describe what an indicator"
+        " is (cloud and CDN ranges, Tor exits, DoH resolvers, ad lists, allowlists) are left out of the categories"
+        " and of the source count. Hover and expansion share the same handler. The queried indicator is sent over"
+        " TLS; nothing else leaves the MISP instance besides the configured API key."
     ),
-    "references": ["https://ismalicious.com/integrations/misp"],
+    "references": ["https://ismalicious.com/integrations/misp", "https://ismalicious.com/api-docs"],
     "input": "An IP address, domain, hostname or URL.",
     "output": "Text attributes with the isMalicious reputation summary.",
 }
 moduleconfig = ["api_key", "api_url"]
 
 DEFAULT_API_URL = "https://api.ismalicious.com"
+# Identifies the module (and its version) to the API instead of a generic client string.
+USER_AGENT = f"ismalicious-misp/{moduleinfo['version']} (+https://ismalicious.com)"
 
 
 def _query_from_attribute(attribute):
@@ -51,17 +58,40 @@ def _risk_score(payload):
     return None
 
 
+def _is_threat_source(source):
+    # Each listing in `sources` may carry a threatClass: "threat" (the default when absent),
+    # "infrastructure" (cloud ranges, Tor exits, DoH resolvers), "policy" (ads, tracking) or
+    # "allowlist". Only threat listings say that an indicator was seen doing something malicious.
+    if not isinstance(source, dict):
+        return True
+    return source.get("threatClass") in (None, "", "threat")
+
+
+def _threat_sources(payload):
+    sources = payload.get("sources") or []
+    if not isinstance(sources, list):
+        return []
+    return [source for source in sources if _is_threat_source(source)]
+
+
 def _categories(payload):
-    categories = payload.get("categories")
+    # The /check response has no top-level categories for IPs, domains and URLs: they are carried by
+    # each listing. Fall back to the primary classification when no threat listing names one.
+    categories = []
+    for source in _threat_sources(payload):
+        if not isinstance(source, dict):
+            continue
+        for value in [source.get("category"), *(source.get("categories") or [])]:
+            if value and value not in categories:
+                categories.append(value)
     if categories:
         return categories
     classification = payload.get("classification") or {}
     return classification.get("primary")
 
 
 def _source_count(payload):
-    sources = payload.get("sources") or []
-    return len(sources) if isinstance(sources, list) else sources
+    return len(_threat_sources(payload))
 
 
 class IsMaliciousParser:
@@ -114,16 +144,24 @@ def handler(q=False):
 
     api_url = str(config.get("api_url") or DEFAULT_API_URL).rstrip("/")
     try:
+        # Redirects are not followed: requests would resend the X-API-KEY header to the new location.
         response = requests.get(
             f"{api_url}/check",
             params={"query": query, "enrichment": "standard"},
-            headers={"User-Agent": "misp-modules", "X-API-KEY": api_key, "Accept": "application/json"},
+            headers={"User-Agent": USER_AGENT, "X-API-KEY": api_key, "Accept": "application/json"},
             timeout=30,
+            allow_redirects=False,
         )
+        if 300 <= response.status_code < 400:
+            return {"error": "isMalicious API answered with a redirect; check that api_url is the HTTPS API URL."}
         response.raise_for_status()
         payload = response.json()
     except requests.exceptions.HTTPError as http_error:
         status = http_error.response.status_code if http_error.response is not None else "unknown"
+        if status == 401:
+            return {"error": "isMalicious API rejected the API key (HTTP 401)."}
+        if status == 429:
+            return {"error": "isMalicious API rate limit or quota exceeded (HTTP 429)."}
         return {"error": f"isMalicious API returned HTTP status {status}."}
     except requests.exceptions.RequestException as request_error:
         return {"error": f"isMalicious API request failed: {request_error}."}

tests/test_ismalicious.py

@@ -5,9 +5,18 @@
 
 HIT_PAYLOAD = {
     "malicious": True,
-    "riskScore": {"score": 91},
-    "categories": ["c2"],
-    "sources": [{"name": "feed-a"}, {"name": "feed-b"}],
+    "riskScore": {"score": 91, "level": "critical"},
+    "sources": [
+        {"name": "feed-a", "category": "c2"},
+        {"name": "feed-b", "category": "botnet", "threatClass": "threat"},
+        {"name": "feed-c", "category": "c2"},
+    ],
+}
+
+EXPECTED_HEADERS = {
+    "User-Agent": ismalicious.USER_AGENT,
+    "X-API-KEY": "k",
+    "Accept": "application/json",
 }
 
 
@@ -31,21 +40,53 @@ def _query(value="1.2.3.4", type_="ip-src", config=None):
     return {"module": "ismalicious", "attribute": attribute, "config": config}
 
 
+def _summary(payload):
+    with patch.object(ismalicious.requests, "get", return_value=MockResponse(payload)):
+        result = ismalicious.handler(json.dumps(_query()))
+    return [attribute["value"] for attribute in result["results"]["Attribute"]]
+
+
 def test_ismalicious_returns_summary_attribute():
     with patch.object(ismalicious.requests, "get", return_value=MockResponse(HIT_PAYLOAD)) as mocked_get:
         result = ismalicious.handler(json.dumps(_query()))
 
     mocked_get.assert_called_once_with(
         "https://api.ismalicious.com/check",
         params={"query": "1.2.3.4", "enrichment": "standard"},
-        headers={"User-Agent": "misp-modules", "X-API-KEY": "k", "Accept": "application/json"},
+        headers=EXPECTED_HEADERS,
         timeout=30,
+        allow_redirects=False,
     )
     values = [attribute["value"] for attribute in result["results"]["Attribute"]]
-    assert values == ["malicious=True score=91 categories=['c2'] sources=2"]
+    assert values == ["malicious=True score=91 categories=['c2', 'botnet'] sources=3"]
     assert result["results"]["Attribute"][0]["comment"] == "isMalicious reputation summary"
 
 
+def test_ismalicious_counts_threat_listings_only():
+    payload = {
+        "malicious": True,
+        "riskScore": {"score": 70},
+        "sources": [
+            {"name": "honeypot", "category": "attack"},
+            {"name": "azure-ip-ranges", "category": "infrastructure", "threatClass": "infrastructure"},
+            {"name": "ad hosts", "category": "ads", "threatClass": "policy"},
+            {"name": "known good", "category": "allowlist", "threatClass": "allowlist"},
+            {"name": "feed-b", "categories": ["botnet", "attack"], "threatClass": "threat"},
+        ],
+    }
+    assert _summary(payload) == ["malicious=True score=70 categories=['attack', 'botnet'] sources=2"]
+
+
+def test_ismalicious_falls_back_to_primary_classification():
+    payload = {
+        "malicious": False,
+        "riskScore": {"score": 12},
+        "classification": {"primary": "unknown"},
+        "sources": [{"name": "azure-ip-ranges", "category": "infrastructure", "threatClass": "infrastructure"}],
+    }
+    assert _summary(payload) == ["malicious=False score=12 categories=unknown sources=0"]
+
+
 def test_ismalicious_uses_domain_side_of_composite_and_custom_api_url():
     query = _query(
         value="evil.example|1.2.3.4",
@@ -58,11 +99,16 @@ def test_ismalicious_uses_domain_side_of_composite_and_custom_api_url():
     mocked_get.assert_called_once_with(
         "https://example.test/check",
         params={"query": "evil.example", "enrichment": "standard"},
-        headers={"User-Agent": "misp-modules", "X-API-KEY": "k", "Accept": "application/json"},
+        headers=EXPECTED_HEADERS,
         timeout=30,
+        allow_redirects=False,
     )
 
 
+def test_ismalicious_user_agent_names_the_module_version():
+    assert ismalicious.USER_AGENT == f"ismalicious-misp/{ismalicious.version()['version']} (+https://ismalicious.com)"
+
+
 def test_ismalicious_missing_api_key():
     with patch.object(ismalicious.requests, "get") as mocked_get:
         result = ismalicious.handler(json.dumps(_query(config={})))
@@ -79,6 +125,30 @@ def test_ismalicious_reports_http_error():
     assert result == {"error": "isMalicious API returned HTTP status 500."}
 
 
+def test_ismalicious_reports_rejected_key_and_quota():
+    with patch.object(ismalicious.requests, "get", return_value=MockResponse({"error": "Unauthorized"}, 401)):
+        assert ismalicious.handler(json.dumps(_query())) == {
+            "error": "isMalicious API rejected the API key (HTTP 401)."
+        }
+    with patch.object(ismalicious.requests, "get", return_value=MockResponse({"error": "Too Many Requests"}, 429)):
+        assert ismalicious.handler(json.dumps(_query())) == {
+            "error": "isMalicious API rate limit or quota exceeded (HTTP 429)."
+        }
+
+
+def test_ismalicious_does_not_follow_redirects():
+    with patch.object(ismalicious.requests, "get", return_value=MockResponse(None, 301)):
+        result = ismalicious.handler(json.dumps(_query(config={"api_key": "k", "api_url": "http://example.test"})))
+    assert "redirect" in result["error"]
+
+
+def test_ismalicious_reports_transport_failure():
+    error = ismalicious.requests.exceptions.ConnectionError("refused")
+    with patch.object(ismalicious.requests, "get", side_effect=error):
+        result = ismalicious.handler(json.dumps(_query()))
+    assert result["error"].startswith("isMalicious API request failed")
+
+
 def test_ismalicious_rejects_invalid_input():
     assert ismalicious.handler(json.dumps({"module": "ismalicious"}))["error"].startswith(
         'This module requires an "attribute" field'
@@ -90,6 +160,7 @@ def test_ismalicious_introspection_and_version():
     assert ismalicious.introspection() == ismalicious.mispattributes
     info = ismalicious.version()
     assert info["name"] == "isMalicious Lookup"
+    assert info["logo"] == "ismalicious.png"
     assert "api_key" in info["config"]
     assert "expansion" in info["module-type"]
     assert "hover" in info["module-type"]