SOCrates 🦉 — An Explainable, Autonomous SOC Triage Agent

"An AI analyst that must always explain itself."

SOCrates ingests real Suricata IDS alerts (eve.json), correlates them into campaigns, reasons over a local MITRE ATT&CK knowledge base (RAG), and produces structured, auditable triage verdicts using a fully local LLM (Ollama) — no security telemetry ever leaves your machine. Containment actions are only ever proposed, never executed without human approval.

Origin: at a CTF competition, an industry tech lead challenged me: "Could you build an autonomous SOC agent?" Building it taught me the real problem isn't autonomy — it's trust. SOCrates is designed around three constraints: it must explain every decision, it must never act without a human, and data must stay local. Everything below, including the measured failure rates and the troubleshooting guide, comes from real test runs on my own machine.

---

Table of Contents

1. What it does
2. Architecture & how the pipeline works
3. Worked example: one alert, end to end
4. Step-by-step setup guide (WSL / Ubuntu / Linux)
5. Configuration reference
6. The explainability contract (real output)
7. Evaluation: 8B vs 3B vs deterministic baseline
8. Observed behavior — what testing actually revealed
9. Security design decisions (OWASP LLM Top 10 mapping)
10. Troubleshooting & FAQ
11. Adding a Web UI (making it feel autonomous)
12. Limitations
13. Future enhancements
14. Project layout, testing & CI

---

1. What it does

Capability	How
Ingests real IDS telemetry	Parses Suricata eve.json (JSON-lines), tolerating corrupt/partial lines
Campaign context	Correlates alerts by source IP so the agent judges behavior, not isolated events
Grounded reasoning (RAG)	Retrieves MITRE ATT&CK notes + a triage playbook from a local knowledge base and cites what it used
Local LLM inference	Ollama (llama3.1:8b by default) — SOC data never touches a third-party API
Structured, validated output	The LLM must answer in a strict JSON schema; anything invalid is rejected at a code-level trust boundary
Graceful degradation	If the LLM is down, missing, times out, or emits invalid JSON, a transparent rule-based analyst takes over — and says so
Human-in-the-loop	The agent proposes containment actions; a human approves or rejects each one
Tamper-evident audit	Every verdict and every human decision is appended to a SHA-256 hash-chained log; verify-audit detects any modification
Analyst-ready output	Markdown incident report + machine-readable verdicts.json

---

2. Architecture & how the pipeline works

flowchart LR
    A[Suricata eve.json] --> B[Ingest & Normalize]
    B --> C[Correlation<br/>group by source IP]
    C --> D{Triage Agent}
    K[(Knowledge Base<br/>MITRE ATT&CK + Playbooks)] -- RAG retrieve --> D
    L[Ollama<br/>local LLM] -- reason --> D
    H[Heuristic Analyst<br/>deterministic fallback] -. LLM down / invalid / timeout .-> D
    D --> V[Validated JSON Verdict<br/>evidence · confidence · MITRE · reasoning]
    V --> R[Markdown Triage Report]
    V --> G[Human-in-the-Loop<br/>approve / reject actions]
    V --> AU[(Hash-chained<br/>Audit Log)]
    G --> AU

Loading

The pipeline for every alert:

1. Perceive — ingest.py normalizes the raw eve.json record into an Alert (source/dest, signature, category, Suricata severity).
2. Correlate — the alert is placed in the context of all other alerts from the same source IP in the batch ("3 alerts from 203.0.113.45 in 4 minutes" reads very differently from one).
3. Retrieve (RAG) — rag.py pulls the most relevant chunks from knowledge/ (MITRE technique notes, severity-calibration rules, false-positive heuristics) by keyword overlap. The retrieved chunk names travel with the verdict as citations.
4. Reason — llm.py sends alert + correlation context + retrieved knowledge to the local Ollama model with a prompt that demands only a JSON object in a fixed schema, at temperature 0 for consistency.
5. Validate — agent.py parses and schema-checks the response. This is the trust boundary: unknown verdict values, malformed confidence, or broken JSON are rejected. Any failure (including timeout) prints a visible [warn] and triggers…
6. Fall back — heuristic.py, a fully deterministic keyword→MITRE analyst, regenerates the verdict and honestly labels it "engine": "heuristic".
7. Explain & record — the verdict (evidence, confidence, MITRE mapping, reasoning trace, knowledge citations) is written to the report, verdicts.json, and the hash-chained audit log.
8. Human decides — in --interactive mode, each recommended action requires explicit approval; approvals and rejections are audited. Execution is simulated — wiring a real firewall API is a deliberate, clearly-marked integration point.

---

3. Worked example: one alert, end to end

Input line from data/sample_eve.json — an internal host making a suspicious DNS query:

{"timestamp":"2026-07-10T14:31:55...","src_ip":"10.0.0.23","dest_ip":"8.8.4.4","dest_port":53,"proto":"UDP",
 "alert":{"signature":"ET DNS Tunnel Suspicious Long TXT Query (possible DNS tunneling)","severity":1}}

What happens: ingestion normalizes it → correlation notes it's a lone event from 10.0.0.23 → retrieval surfaces the T1071.004 technique note ("an internal source IP makes this high-to-critical: it implies existing compromise") and the playbook's severity-calibration rules → the LLM reasons over all of it → validation passes → this verdict is produced (real output from llama3.1:8b, see Section 6) → the two proposed actions (sinkhole the domain, isolate the host) wait for human approval → everything lands in the audit chain.

The key moment: the raw Suricata severity was the same as several other alerts, but the agent escalated this one to CRITICAL — because it reasoned that the source being an internal host implies existing compromise, citing the playbook rule it applied. The deterministic engine, which can only keyword-match, rated it HIGH. That delta is the value proposition of LLM triage, and the citation trail is what makes it trustworthy.

---

4. Step-by-step setup guide

Tested on Ubuntu 24.04 under WSL2 (Windows 11) with Python 3.12. Any Linux/macOS with Python 3.10+ works.

Step 1 — Clone and smoke-test (no LLM needed, ~60 seconds)

git clone https://github.com/fa1829/socrates && cd socrates
PYTHONPATH=src python3 -m socrates triage data/sample_eve.json

The core has zero dependencies — this works on a bare Python install. Without Ollama you'll see engine: heuristic (Ollama not detected) and the pipeline still produces all three outputs:

cat triage_report.md          # analyst-readable incident report
cat verdicts.json             # machine-readable verdicts with full reasoning
PYTHONPATH=src python3 -m socrates verify-audit   # -> Audit chain VALID

Try tampering: edit any value inside audit_log.jsonl, rerun verify-audit, and watch the chain break.

Tip: tired of typing PYTHONPATH=src? Add export PYTHONPATH=$PWD/src to the shell session, or your ~/.bashrc.

Step 2 — Dev environment and tests

Ubuntu protects system Python (PEP 668), so use a virtual environment — do not install via snap:

sudo apt update && sudo apt install -y python3-venv
python3 -m venv .venv
source .venv/bin/activate        # prompt now shows (.venv)
pip install pytest ruff
ruff check src tests && pytest -q     # expect: All checks passed! / 5 passed

source .venv/bin/activate must be re-run in every new terminal (the python3 -m venv .venv creation happens only once).

Step 3 — Install Ollama and pull a model

curl -fsSL https://ollama.com/install.sh | sh
ollama list                       # what's installed right now
ollama pull llama3.1:8b           # ~4.9 GB — best quality (needs ~8 GB free RAM)
# or, for slower machines / faster demos:
ollama pull llama3.2:3b           # ~2.0 GB

Model names are exact. llama3:latest ≠ llama3.1:8b. If the configured model isn't pulled, SOCrates warns and runs in heuristic mode (see Troubleshooting — this exact confusion happened during development).

Step 4 — Run with the LLM

SOCRATES_TIMEOUT=600 PYTHONPATH=src python3 -m socrates triage data/sample_eve.json

Expect it to be slow on CPU — roughly 100 seconds per alert with the 8B model (measured: 12m01s for 7 alerts). Silence between verdict lines is the model thinking, not a hang. The 3B model is ~2.5× faster (measured: 4m51s) with quality trade-offs quantified in Section 7:

SOCRATES_MODEL=llama3.2:3b PYTHONPATH=src python3 -m socrates triage data/sample_eve.json

Step 5 — Human-in-the-loop mode

SOCRATES_TIMEOUT=600 PYTHONPATH=src python3 -m socrates triage data/sample_eve.json --interactive

Each recommended action prompts APPROVE action ...? [y/N]. Approvals print a simulated execution line; both approvals and rejections are written to the audit chain. Nothing is ever enforced automatically.

Step 6 — Point it at your own sensor

PYTHONPATH=src python3 -m socrates triage /var/log/suricata/eve.json --report incident.md

Non-alert events (flows, DNS logs, stats) are filtered out automatically; corrupt lines — common in live eve.json files — are tolerated.

---

5. Configuration reference

All configuration is via environment variables — no config files to manage:

Variable	Default	Purpose
SOCRATES_MODEL	llama3.1:8b	Any model pulled into Ollama
SOCRATES_OLLAMA_URL	http://localhost:11434	Ollama server address (see WSL note in Troubleshooting)
SOCRATES_TIMEOUT	300	Seconds allowed per LLM call — raise for large models on CPU

CLI flags: --knowledge (knowledge base dir), --report, --json, --audit-log, --interactive. Inference runs at temperature 0 with a fixed seed for run-to-run reproducibility, and the model is kept loaded between alerts (keep_alive) to avoid reload latency.

---

6. The explainability contract

Every verdict is a validated JSON object. This is actual output from a real run (llama3.1:8b) — the DNS-tunneling alert from the worked example:

{
  "alert_id": "bbe2f0458478",
  "verdict": "true_positive",
  "severity": "critical",
  "confidence": 0.9,
  "summary": "Potential DNS tunneling detected from internal host, indicating possible compromise.",
  "evidence": [
    "ET DNS Tunnel Suspicious Long TXT Query (possible DNS tunneling) alert triggered",
    "Internal source IP 10.0.0.23 indicates existing compromise",
    "Mitre technique T1071.004 - Application Layer Protocol: DNS (C2 / Tunneling) matches"
  ],
  "mitre_techniques": ["T1071.004 - Application Layer Protocol: DNS (C2 / Tunneling)"],
  "recommended_actions": [
    "Sinkhole the domain to prevent further communication",
    "Isolate the internal host for further investigation and forensics"
  ],
  "reasoning": "The alert indicates a suspicious long TXT query from an internal host, which is a known indicator of DNS tunneling. The Mitre technique T1071.004 matches this behavior, indicating potential C2 or exfiltration activity. Given the internal source IP, this is considered a critical severity incident.",
  "engine": "llm:llama3.1:8b",
  "knowledge_used": [
    "mitre_attack.md#T1071.004 - Application Layer Protocol: DNS (C2 / Tunneling)",
    "triage_playbook.md#Severity calibration"
  ]
}

Note the citation chain: the verdict names both the MITRE note and the playbook rule it applied to reach CRITICAL. If the LLM ever returns anything that fails schema validation, the verdict is regenerated by the deterministic engine and labeled "engine": "heuristic" — the system never silently trusts malformed model output, and (as of v0.1) never silently falls back either: every fallback prints a [warn] with the reason.

---

7. Evaluation: model comparison on real runs

Same 7-alert dataset, same machine (CPU-only, WSL2), measured with time:

Engine	Total (7 alerts)	Per alert	Schema-valid verdicts	Caught internal-host → CRITICAL escalation	Notes
llama3.1:8b	12m 01s	~103 s	7/7	✅ every run (conf 0.90–0.95)	Best reasoning; needs SOCRATES_TIMEOUT≥600 on CPU
llama3.2:3b	4m 51s	~42 s	5/7	❌ rated HIGH/0.80	2.5× faster; weaker JSON discipline and shallower reasoning
Heuristic baseline	<1 s	instant	7/7 (by construction)	❌	Deterministic; doubles as the eval baseline

Two lessons in one table: at 3B, format discipline and multi-hop reasoning degrade together — the smaller model both broke the JSON schema twice and missed the inference that an internal source implies compromise. The validation layer meant those failures degraded gracefully (visible [warn], honest heuristic verdict) instead of poisoning the pipeline.

---

8. Observed behavior

Findings from real test runs, kept here because an agent's measured behavior matters more than its claimed behavior:

Reasoning beyond keywords. The 8B model escalated the DNS-tunneling alert to CRITICAL because the source was an internal host, citing the playbook's severity rule — consistently, in every run. The keyword engine structurally cannot make that inference.

Measured fallback rates. llama3.2:3b: 2/7 schema-validation failures. llama3.1:8b at the old 120 s default timeout: 2/7 timeouts on CPU (0/7 at 600 s). Both failure modes are now surfaced with [warn] lines rather than silent fallback — a fix motivated directly by these observations.

Retrieval ≠ obedience. In one run, retrieval correctly surfaced the playbook's false-positive guidance for TLS certificate-policy alerts — and the model read it and overrode it anyway, ruling true_positive/HIGH with host-isolation recommendations (an over-escalation). The citation trail made the disagreement visible: you can see exactly which guidance the agent was shown and chose to contradict. A black-box system would have hidden this entirely. This is also the strongest argument for the human approval gate.

Non-determinism, then determinism. At temperature 0.1, identical inputs produced different verdicts minutes apart (an alert flipped from needs_review/0.40 to true_positive/0.80 between runs). Inference now runs at temperature 0 with a fixed seed.

A reproducible failure lead. One specific alert (the SSH brute-force) failed schema validation across multiple runs and both models; it was also the alert with zero knowledge-base citations. Working hypothesis: poor retrieval → ungrounded prompt → malformed output. Investigating this link between retrieval quality and output validity is roadmap item #1.

---

9. Security design decisions

Decision	Threat addressed
Agent proposes, human approves; enforcement is a clearly-marked, unimplemented integration point	OWASP LLM08 — Excessive Agency. The observed TLS over-escalation is a live example of why: an autonomous version would have isolated a host over a certificate warning
Strict JSON schema validated in code before any verdict is trusted	LLM02 — Insecure Output Handling
Only normalized alert fields and curated local knowledge enter the prompt — never raw packet payloads (where attacker-controlled text lives)	LLM01 — Prompt Injection (surface reduction; adversarial testing is on the roadmap)
100% local inference via Ollama	Data governance — no third-party processing of security telemetry
SHA-256 hash-chained audit log of every verdict and human decision (verify-audit)	Accountability, tamper-evidence, incident review
Deterministic heuristic engine	Availability under LLM failure and a measurable evaluation baseline

---

10. Troubleshooting & FAQ

Every entry below is a problem actually hit during development.

The banner says engine: llm:... but verdicts look suspiciously uniform (all 70%/40%). The requested model isn't pulled into Ollama — the server answered the health check, but every generate() call failed and the agent fell back to heuristics. Fingerprints: instant completion (a real 8B model takes ~100 s/alert on CPU), uniform confidences, and "engine": "heuristic" in verdicts.json. Check with ollama list and remember names are exact: llama3:latest is not llama3.1:8b. Fix: ollama pull llama3.1:8b or set SOCRATES_MODEL to a model you have. (Since v0.1 the startup check also verifies the model, not just the server, and warns explicitly.)

[warn] LLM call failed ...: timed out. CPU inference on an 8B model can exceed the per-call timeout, especially under RAM pressure (an 8B model wants ~6+ GB; swapping makes it crawl) — typically the later alerts in a batch time out first. Fix: SOCRATES_TIMEOUT=600, or use a smaller model (SOCRATES_MODEL=llama3.2:3b). The model is kept loaded between alerts to avoid reload stalls.

[warn] LLM output failed schema validation. The model answered but broke the JSON contract (prose, missing fields, invalid enum values). More common with smaller models (measured 2/7 at 3B vs 0/7 at 8B). The verdict is regenerated by the heuristic engine and honestly labeled. If one alert fails repeatedly, check whether it retrieved any knowledge (knowledge_used empty?) — see Observed behavior.

Run seems frozen after the banner. It isn't — verdict lines appear only when each alert finishes, and each can take 1–2 minutes on CPU. Watch htop in another terminal if you want reassurance.

-bash: .venv/bin/activate: No such file or directory. The venv was never created — activation only works after python3 -m venv .venv has been run once in the project directory. Don't install ruff/pytest via snap; keep tooling in the venv.

Pasting a code snippet gives try:: command not found / syntax error near unexpected token. That snippet was Python meant for a file, not a shell command. Rule of thumb: lines starting with def, try:, import, or indented blocks → into a file via an editor (code . opens VS Code from WSL); commands like pytest, git, ollama → into the terminal.

Ollama runs on Windows but SOCrates runs in WSL and can't reach it. WSL2's localhost doesn't always route to the Windows host. Find the host IP with ip route show default, then SOCRATES_OLLAMA_URL=http://<host-ip>:11434. On newer Windows 11 builds, enabling mirrored networking in .wslconfig makes plain localhost work.

grep '"knowledge_used"' verdicts.json only shows empty [] entries. A grep artifact, not a bug: grep is line-based and populated arrays span multiple lines in pretty-printed JSON. Inspect with Python instead: python3 -c "import json; [print(v['alert_id'], v['knowledge_used']) for v in json.load(open('verdicts.json'))]".

Why did the same input give different verdicts on two runs? LLM sampling non-determinism (observed at temperature 0.1). Inference now uses temperature 0 + fixed seed. Bit-exact reproducibility across hardware/Ollama versions still isn't guaranteed — another reason the audit log records what was decided each time.

No alert events found in file. The file contains no "event_type": "alert" records — Suricata eve.json mixes many event types (flow, dns, stats) and only alerts are triaged. Check with grep -c '"event_type":"alert"' yourfile.json.

Does my alert data go anywhere? No. Inference is local (Ollama), retrieval is local files, outputs are local files. The project makes zero external network calls at runtime.

Old verdicts are mixed into my report/audit log. The audit log appends by design (that's the point of a chain). For a clean demo run: rm -f audit_log.jsonl triage_report.md verdicts.json first.

Can I use Hugging Face models instead of Ollama? Yes — Ollama pulls GGUF builds of most open models (ollama pull qwen2.5:7b, mistral, etc.). Direct transformers-based inference would slot in behind the same client interface in llm.py.

---

11. Adding a Web UI

The CLI proves the pipeline; a small web dashboard makes the "autonomous SOC teammate" experience visceral — alerts streaming in, verdicts appearing with reasoning expanded, and Approve / Reject buttons replacing the [y/N] prompt. Planned design (roadmap):

Fastest path — Streamlit (~1 day): a single Python file with a file-uploader for eve.json, a model selector, a live verdict table with expandable reasoning/evidence/citations, and approve/reject buttons per action writing to the same audit chain. Free, local, zero JavaScript.

Portfolio-grade path — FastAPI + one HTML page (~1 weekend):

POST /triage        upload eve.json -> triage job starts
GET  /verdicts      stream verdicts as they complete (Server-Sent Events)
POST /decision      {alert_id, action, approved} -> audit chain
GET  /audit/verify  chain integrity check, shown as a green/red badge in the UI

The existing modules already fit this shape: TriageAgent.triage() is called per alert (natural streaming unit), AuditLog.record() handles decisions, and Verdict.to_dict() is the API payload. The UI is a table that fills in row-by-row while the model thinks — the waiting itself demonstrates that real inference is happening locally.

Design rules the UI must keep: approve/reject stays per-action (no "approve all" button — that would quietly delete the human from the loop); every verdict row shows engine, confidence, and citations; the audit-chain status is always visible. Security note: bind to 127.0.0.1 only — this dashboard is a local tool, and exposing an unauthenticated triage-and-approve interface to a network would be an irony the project could not survive.

---

12. Limitations

• Keyword retrieval is shallow. RAG uses keyword overlap; alerts whose vocabulary doesn't match the knowledge base retrieve nothing (observed, and correlated with an LLM output failure). Dense/embedding retrieval is the planned fix.
• The LLM can ignore its grounding. Retrieval puts the right knowledge in the prompt; it doesn't force deference — observed with the TLS playbook override. Mitigation today is visibility (citations) + human approval, not prevention.
• CPU inference is slow. ~100 s/alert at 8B. Fine for batch triage and demos; a real SOC volume needs GPU serving or a smaller tuned model.
• Single-batch correlation only. Alerts are grouped by source IP within one file; there is no cross-run memory, time-window logic, or destination-based correlation yet.
• Actions are simulated. Enforcement (firewall API, EDR isolation) is deliberately unimplemented; the integration point is marked in hitl.py.
• Synthetic evaluation set. 7 hand-crafted alerts prove the pipeline; claims about real-world precision/recall require the labeled-dataset harness on the roadmap.
• Prompt-injection surface is reduced, not tested. Normalized fields keep raw payloads out of the prompt, but signatures themselves are semi-attacker-influenced text; adversarial testing hasn't been performed yet.

13. Future enhancements

1. Retrieval → validity study + embedding RAG — instrument the observed link between empty retrieval and schema failures, then swap keyword overlap for sentence-transformers + ChromaDB behind the existing KnowledgeBase interface.
2. Evaluation harness — labeled alert set; precision/recall/F1 of LLM vs. heuristic vs. ground truth; per-verdict latency recorded in the verdict itself.
3. Web dashboard (Section 11).
4. Live SIEM ingestion — pull alerts from Elastic/OpenSearch instead of files.
5. Multi-agent split (LangGraph) — triage agent + threat-intel enrichment agent + reporting agent with a supervisor.
6. Adversarial testing — craft alert signatures containing injection attempts; measure whether the schema boundary holds.
7. MCP server — expose SOCrates as a tool other AI assistants can call.
8. Real enforcement integrations — opt-in, allow-listed, reversible actions only (e.g., single-IP blocks via firewall API), still behind human approval.

---

14. Project layout, testing & CI

src/socrates/
  ingest.py     # eve.json parsing + source-IP correlation
  rag.py        # local knowledge retrieval (embedding-ready interface)
  llm.py        # Ollama client, timeout/keep-alive/seed handling, robust JSON extraction
  agent.py      # triage loop: retrieve -> reason -> validate -> explain (with visible fallback warnings)
  heuristic.py  # transparent deterministic fallback / evaluation baseline
  hitl.py       # human approval gate for proposed actions
  audit.py      # SHA-256 hash-chained tamper-evident log
  report.py     # Markdown incident report
  models.py     # Alert & Verdict dataclasses — the explainability contract
knowledge/      # MITRE ATT&CK notes + triage playbook (the RAG corpus)
data/           # synthetic sample eve.json (safe to publish)
tests/          # pytest suite: ingestion, verdict schema, campaign escalation, audit tamper-detection

source .venv/bin/activate
ruff check src tests && pytest -q     # 5 tests, including one that tampers with the audit log and verifies detection

GitHub Actions runs lint + tests on every push.

License

MIT — free to use, learn from, and build on.