Add security keys, the certificate names they need, and finish the console kit

.env.example

@@ -97,6 +97,39 @@ AGENT_SHARED_SECRET=
 # TLS_CERT=certs/server.crt
 # TLS_KEY=certs/server.key
 # TLS_CN=soc.example.com
+#
+# TLS_SAN is every *other* name the certificate should answer to, comma
+# separated. Hostnames and addresses both; an entry that parses as an address
+# becomes an IP SAN, because a bare address listed as a DNS name matches
+# nothing. localhost, 127.0.0.1 and ::1 are always included.
+#
+# Put the name you actually type in the browser here. A console reached at
+# https://<address> cannot register a security key at all: WebAuthn's RP ID
+# has to be a domain and browsers reject an address outright.
+#
+# Generation is idempotent, so changing these on a deployment that already has
+# a certificate does nothing on its own - the startup log says so and gives
+# the regenerate command.
+# TLS_SAN=soc.example.com,10.0.0.5
+
+
+# ─── Security keys (WebAuthn) ────────────────────────────────────────────
+# Nothing to set, and that is deliberate.
+#
+# The relying-party ID is derived from the request, so a laptop at
+# https://localhost:8000 and an estate at https://soc.example.com both work
+# with no setting - and no setting can disagree with the URL people type,
+# which is the failure this would otherwise have. `My Security` reports
+# whether a key can be registered from where the browser is standing, and why
+# not when it cannot.
+#
+# The two things that decide it are above: TLS_ENABLED (a key needs a secure
+# context) and TLS_CN/TLS_SAN (it needs a hostname - browsers reject an IP
+# address as a relying-party ID). One-time codes work regardless.
+#
+# The `webauthn` package is imported lazily, so a deployment that pulls new
+# code without reinstalling its dependencies loses the key option rather than
+# the console.
 
 # Telemetry transport. 5001 is the original plaintext listener; 5011 carries
 # the same protocol inside TLS, and an agent chooses between them from its

Dockerfile

@@ -1,66 +1,66 @@
-# --- Stage 1: Build the React Frontend ---
-FROM node:20-slim AS frontend-builder
-WORKDIR /build
-
-# Copy frontend source
-COPY frontend/package*.json ./
-RUN npm install --silent
-COPY frontend/ ./
-
-# Build the frontend (creates 'dist' folder)
-RUN npm run build
-
-# --- Stage 2: Final Production Container ---
-FROM python:3.10-slim
-WORKDIR /app
-
-# Install system dependencies (needed for MySQL, PostgreSQL, and LDAP)
-RUN apt-get update && apt-get install -y \
-    default-libmysqlclient-dev \
-    libpq-dev \
-    libldap2-dev \
-    libsasl2-dev \
-    gcc \
-    && rm -rf /var/lib/apt/lists/*
-
-# Install Python backend dependencies
-# The lock, not the loose list. requirements.txt says what we depend on;
-# requirements.lock says which versions this image is built from, so the
-# image a commit produces today is the image it produces next month.
-COPY ./requirements.txt ./requirements.lock /app/
-RUN pip install --no-cache-dir -r /app/requirements.lock
-
-# Copy backend source. See .dockerignore — .env and data/ are excluded so
-# secrets are not baked into the image layers.
-COPY . /app
-
-# Copy the built frontend from the previous stage
-# Note: Since app.py now expects ./frontend/dist
-RUN mkdir -p /app/frontend/dist
-COPY --from=frontend-builder /build/dist /app/frontend/dist
-
-# Environment variables
-ENV PYTHONUNBUFFERED=1
-ENV VITE_API_BASE_URL=""
-# Empty, not "*": the SPA is served from this same container, so requests are
-# same-origin. A wildcard origin is rejected by browsers once the session
-# cookie is attached.
-ENV CORS_ORIGINS=""
-
-# Drop root. The container previously ran as uid 0, so any code execution bug
-# in the API — the HTTP proxy and the playbook engine both reach out to
-# attacker-influenced input — started with full privileges inside the
-# container. /app/data is the one path written at runtime (the Fernet key when
-# FERNET_KEY is unset), so it is the only thing the app user needs to own.
-RUN useradd --system --create-home --uid 10001 sentora \
-    && mkdir -p /app/data \
-    && chown -R sentora:sentora /app/data \
-    && chmod 700 /app/data
-USER sentora
-
-# Expose Sanic port
-EXPOSE 8000
-
-# Start command
-# Default to running the main app (API + UI)
-CMD ["python", "app.py"]
+# --- Stage 1: Build the React Frontend ---
+FROM node:20-slim AS frontend-builder
+WORKDIR /build
+
+# Copy frontend source
+COPY frontend/package*.json ./
+RUN npm install --silent
+COPY frontend/ ./
+
+# Build the frontend (creates 'dist' folder)
+RUN npm run build
+
+# --- Stage 2: Final Production Container ---
+FROM python:3.10-slim
+WORKDIR /app
+
+# Install system dependencies (needed for MySQL, PostgreSQL, and LDAP)
+RUN apt-get update && apt-get install -y \
+    default-libmysqlclient-dev \
+    libpq-dev \
+    libldap2-dev \
+    libsasl2-dev \
+    gcc \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Python backend dependencies
+# The lock, not the loose list. requirements.txt says what we depend on;
+# requirements.lock says which versions this image is built from, so the
+# image a commit produces today is the image it produces next month.
+COPY ./requirements.txt ./requirements.lock /app/
+RUN pip install --no-cache-dir -r /app/requirements.lock
+
+# Copy backend source. See .dockerignore — .env and data/ are excluded so
+# secrets are not baked into the image layers.
+COPY . /app
+
+# Copy the built frontend from the previous stage
+# Note: Since app.py now expects ./frontend/dist
+RUN mkdir -p /app/frontend/dist
+COPY --from=frontend-builder /build/dist /app/frontend/dist
+
+# Environment variables
+ENV PYTHONUNBUFFERED=1
+ENV VITE_API_BASE_URL=""
+# Empty, not "*": the SPA is served from this same container, so requests are
+# same-origin. A wildcard origin is rejected by browsers once the session
+# cookie is attached.
+ENV CORS_ORIGINS=""
+
+# Drop root. The container previously ran as uid 0, so any code execution bug
+# in the API — the HTTP proxy and the playbook engine both reach out to
+# attacker-influenced input — started with full privileges inside the
+# container. /app/data is the one path written at runtime (the Fernet key when
+# FERNET_KEY is unset), so it is the only thing the app user needs to own.
+RUN useradd --system --create-home --uid 10001 sentora \
+    && mkdir -p /app/data \
+    && chown -R sentora:sentora /app/data \
+    && chmod 700 /app/data
+USER sentora
+
+# Expose Sanic port
+EXPOSE 8000
+
+# Start command
+# Default to running the main app (API + UI)
+CMD ["python", "app.py"]

Sentora/build_agent.ps1

@@ -223,7 +223,14 @@ $sha    = (Get-FileHash -Algorithm SHA256 $artifact).Hash.ToLower()
 
 Write-Ok ("Built main.exe  ({0:N1} MB)" -f $sizeMb)
 Write-Host "    sha256: $sha"
-Write-Host "    Ship it via /api/agent/download/windows (restart the server container to pick it up)."
+# `docker compose restart app` and not `up -d --build app` was what this line
+# used to say, and a restart does nothing: Sentora/ is copied into the image
+# by the Dockerfile, not bind-mounted, so the container keeps serving the
+# binary it was built with. A rebuild-and-reinstall cycle was spent on that -
+# the agent came back with the previous build's hash and every fix looked like
+# it had failed.
+Write-Host "    Serve it: docker compose up -d --build app   (a plain restart keeps the old binary)"
+Write-Host "    Then reinstall the agent; /api/agent/download/windows reads it from the image."
 
 # ─────────────────────── post-build cleanup ───────────────────────
 # Drop the intermediate PyInstaller artifacts so the working tree

Sentora/build_agent.sh

@@ -265,7 +265,12 @@ fi
 
 ok "Built main ($SIZE)"
 printf "    sha256: %s\n" "$SHA"
-printf "    Ship it via /api/agent/download/linux (restart the server container to pick it up).\n"
+# A restart does nothing here: Sentora/ is copied into the image by the
+# Dockerfile rather than bind-mounted, so the container keeps serving the
+# binary it was built with. See the note in build_agent.ps1 - a whole
+# rebuild-and-reinstall cycle was spent discovering it.
+printf "    Serve it: docker compose up -d --build app   (a plain restart keeps the old binary)\n"
+printf "    Then reinstall the agent; /api/agent/download/linux reads it from the image.\n"
 
 # ─────────────────────── post-build cleanup ───────────────────────
 # Drop the intermediate PyInstaller artifacts so the working tree

Sentora/docs/MODULES.md

@@ -2,7 +2,16 @@
 
 Every module is launched from `main.py` via `periodic_wrapped(...)` (or
 a dedicated thread for continuous watchers). Findings are written to
-the agent's local DB and shipped to the server by a background flusher.
+the agent's local DB, and shipped to the server by a background flusher
+**only if the table is named in `TABLES` in `main.py`**.
+
+That last part is not a formality. `software_inventory` and
+`network_inventory` were written every cycle by `inventory` and were not
+on that list, so nothing was ever offered to the server: 368,902 rows and
+78,224 rows sat unsent in the agent's own database while both tables read
+as empty in the console. Adding a module means adding its table there, and
+`tests/test_ingest_schema_coverage.py` now fails if a table in the agent
+schema is neither shipped nor listed as deliberately local.
 
 Convention: tables marked (enc) have sensitive fields encrypted at rest
 by `modules/enc_db.py` using the per-tenant Fernet key. See the
@@ -49,36 +58,54 @@ map.
 
 ### `edr_enforcer` (periodic baseline)
 - Path: `modules/edr_enforcer.py`
-- Behavior: Scheduled hash-baseline scan of a small high-value file set
-  (`/etc/passwd`, `/etc/shadow`, SAM hive path, `hosts`, `sshd_config`).
-  First run establishes the baseline; subsequent runs flag any mismatch
-  as a `security_audit` finding. Complements the realtime `fim` watcher,
-  since the baseline scan catches changes that occurred while the agent
-  was offline.
-- Output table: `security_audit` (enc)
+- Behavior: The agent's periodic endpoint sweep. Five passes per cycle:
+  - `check_fim()` — hash-baseline scan of a small high-value file set
+    (`/etc/passwd`, `/etc/shadow`, SAM hive path, `hosts`, `sshd_config`).
+    Complements the realtime `fim` watcher, which cannot see changes made
+    while the agent was offline.
+  - `track_network()` — current sockets and the process behind each.
+  - `monitor_processes()` — new and exiting processes.
+  - `get_hardware_inventory()` — CPU, memory and disk facts.
+  - `monitor_registry()` — Windows autorun and policy keys.
+- Output tables: `fim_data` (enc), `network_connections` (enc),
+  `process_events` (enc), `hardware_inventory` (enc), `registry_logs` (enc)
+- This entry described a module writing `security_audit` and nothing else.
+  It has never written that table — nothing in the agent has — and the four
+  tables it does write beside `fim_data` were undocumented. `security_audit`
+  is the clearest case in the codebase of a claim outliving its
+  implementation: the document promised it, the schema created it, `TABLES`
+  shipped it empty every cycle, and the console showed it as NOT COLLECTED
+  for the life of every install. It is off the shipping list now; the table
+  stays so a collector can be added without a schema change.
 
 ---
 
 ## Inventory and vulnerability
 
 ### `inventory`
 - Path: `modules/inventory.py`
-- Behavior: Hardware snapshot (CPU model, RAM, disk topology) plus
+- Behavior: Hardware snapshot (CPU model, RAM, disk topology),
   installed-software enumeration (`dpkg-query`, then `rpm -qa`, then a
-  Windows registry walk under `Uninstall`). Writes one
-  `hardware_inventory` row and N `installed_software` rows per sweep.
+  Windows registry walk under `Uninstall`), and the set of listening
+  sockets with the process owning each. Writes one `hardware_inventory`
+  row, N `software_inventory` rows and N `network_inventory` rows per
+  sweep.
 - Output tables: `hardware_inventory` (enc) for `name` and
-  `serial_number`; `installed_software`
+  `serial_number`; `software_inventory`; `network_inventory`
+- The doc said `installed_software`, which is not a table and never was.
+  The two inventory tables were also the pair that reached the server for
+  the first time only after `TABLES` was corrected — see the note at the
+  top.
 
 ### `find_vulns`
 - Path: `modules/find_vulns/` (`info_collector.py` and `find_vuln.py`)
 - Behavior:
   - `info_collector` produces the package list (same logic as
-    `inventory` but emits to the OSV-input format).
+    `inventory` but emitting the OSV input format) into `packages`.
   - `find_vuln` queries `https://api.osv.dev` (or the server-proxied
     air-gap mirror) for known vulnerabilities and writes
     `vulnerabilities_report` rows.
-- Output table: `vulnerabilities_report`
+- Output tables: `packages`, `vulnerabilities_report`
 
 ---
 
@@ -92,7 +119,12 @@ map.
   2. Walks `psutil.process_iter()` for known dual-use tooling: `psexec`,
      `winrs`, `wsmprovhost`, `nmap`, `masscan`, `nc` / `ncat` / `netcat`,
      `socat`, `chisel`, `mimikatz`, `rdesktop`.
-- Output table: `security_audit` (enc) with `type=LATERAL_MOVEMENT`
+- Output table: `events_alert` (enc), source `LateralMovement`
+- Findings go through `send_alert()` in `main.py`, which deduplicates on a
+  content fingerprint before writing. These detectors report *state*, not
+  events — an established loopback SMB connection is re-found every 300s —
+  and without the fingerprint one normal connection produced the same alert
+  roughly 288 times a day.
 
 ### `persistence_hunter`
 - Path: `modules/persistence_hunter.py`
@@ -102,7 +134,17 @@ map.
     running `.vbs` / `.ps1`. Inspects the per-user Startup folder.
   - Linux: `/etc/crontab`, `/etc/cron.d/`, user crontabs and suspicious
     systemd unit files.
-- Output table: `security_audit` (enc) with `type=PERSISTENCE`
+- Output table: `events_alert` (enc), source `PersistenceHunter`
+- Same `send_alert()` path and the same deduplication as
+  `lateral_movement` above.
+
+> All three modules in this section, plus `edr_enforcer` above, were
+> documented as writing `security_audit`. None of them ever did — nothing in
+> this agent has written a row to that table. The schema created it, `TABLES`
+> shipped it empty every cycle, and the console showed it as NOT COLLECTED for
+> the life of every install, which reads as a sensor that broke rather than one
+> that was never built. It is off the shipping list; the table stays so a
+> collector can be added later without a schema change.
 
 ### `check_permissions`
 - Path: `modules/check_permissions/check_permissions.py`
@@ -119,17 +161,28 @@ map.
 
 ### `portscanner`
 - Path: `modules/portscanner/portscanner.py`
-- Behavior: Scans local TCP ports, grabs service banners, classifies
-  the service when a fingerprint matches. Used both for surface mapping
-  and to detect new listeners that appear between sweeps.
-- Output table: `portscan_result`
+- Behavior: Scans `127.0.0.1` hourly, grabs a banner from each open
+  port, and classifies the service when a fingerprint matches. Used for
+  surface mapping and to spot listeners that appear between sweeps.
+- Output table: `portscan_result` — `target_ip`, `port`, `protocol`,
+  `state`, `service`, `product`, `version`, `banner`
+- `target_ip`, `state` and `banner` were columns on the *server's* table
+  that the agent's own table did not have and the insert never mentioned,
+  so all 108 rows on the server carried NULL for each. A port list that
+  does not say which host, or whether the port answered, is a list of
+  numbers.
+- The banner is remote input and is truncated to 2 KB. It is also where
+  the NUL bytes come from that used to kill the whole scan — see
+  `db.py` below.
 
 ### `resource_checker`
 - Path: `modules/resource_checker/resource_checker.py` (plus `disks.py`)
-- Behavior: Per-tick CPU and memory snapshot into `resource_log`. The
-  `disks.py` companion writes per-partition usage to `disk_info` and
+- Behavior: Per-tick CPU and memory snapshot into `resource_usage`. The
+  `disks.py` companion writes per-partition usage to `disk_usage` and
   flags any partition over the soft or hard threshold.
-- Output tables: `resource_log`, `disk_info`
+- Output tables: `resource_usage`, `disk_usage`
+- Documented as `resource_log` and `disk_info` until now. Neither name
+  exists in the schema.
 
 ### `docker_monitor`
 - Path: `modules/docker_monitor/docker_monitor.py`
@@ -180,6 +233,21 @@ an equivalent SQLite schema is shipped at
 [`db/init_sqlite.sql`](../db/init_sqlite.sql) for offline or minimal
 deployments.
 
+Two behaviours a module author needs to know about:
+
+- **`scrub_nuls` strips `\x00` from every bound parameter.** PostgreSQL
+  text cannot hold a NUL and psycopg2 refuses the whole statement with
+  `ValueError: A string literal cannot contain NUL (0x00) characters` —
+  note the type, an `except psycopg2.Error` walks straight past it. Any
+  module that stores bytes off a socket or a file will hit this; the port
+  scanner did, every hour, and lost the entire scan each time because the
+  exception left the loop that was writing the other ports.
+- **`prune_unsent` bounds the backlog of rows that never shipped.**
+  `prune_sent` deliberately refuses to delete an unsent row, which is
+  right for a table being delivered and wrong for one that is not. A table
+  missing from `TABLES` would otherwise grow until the disk did, on a
+  machine this product is meant to be watching rather than filling.
+
 ### `enc_db.py`
 Fernet-backed transparent encryption layer over `db.py`. Pulls the
 active key from `/api/agents/bootstrap` and refreshes it every