Skip to content

Fields we said were encrypted were stored in the clear - #43

Merged
d3vhex merged 1 commit into
mainfrom
feat/platform-self-defence-and-encryption-fixes
Sep 7, 2026
Merged

d3vhex merged 1 commit into
mainfrom
feat/platform-self-defence-and-encryption-fixes

Conversation

@d3vhex

@d3vhex d3vhex commented Sep 7, 2026

Copy link
Copy Markdown
Owner

enc_db.ENCRYPT_FIELDS_MAP listed fim_data.path and hash_sha256 as
encrypted. core.telemetry_crypto.ENCRYPTED_FIELDS listed them as fields to
decrypt. And fim.py — which produces almost every fim_data row there is —
called the plain insert_record. On one host: 1,081 file paths stored
unencrypted
, locally and on the server, while both ends of the design said
otherwise.

Nothing could see it. The server's decrypt_value hands anything without the
enc:: prefix straight back, by design — that is what makes a mixed table
readable — so plaintext and ciphertext render identically in the console.
The table looked perfect. The only way to notice was to ask what the stored
bytes actually begin with.

A test now closes the class, and found six more sites: five in inventory.py
writing hardware_inventory, one in disks.py writing events_alert, and a
fallback in main.py that wrote an alert in the clear if insert_record_enc
were ever missing. That last one was dead code, and a fallback to storing a
secret in the clear is not a fallback — it is a bug waiting for its condition.

The port scanner had been dead every cycle

[!] portscanner failed (1 consecutive): SyntaxError: unterminated quoted
    string at or near "'"

No port scan result was ever written. Because it is a table that produces rows
rarely, that was indistinguishable from a host with no open ports.

The cause matters more than the trigger: the query was assembled by
interpolation with a hand-rolled normalize() that doubled single quotes —
and fetch_where has taken a params tuple all along. nmap returns the
product and version strings a service chose to advertise, so the input is
attacker-controlled by definition, and the answer to that is never a better
escaping function. A security product assembling SQL by concatenation is
running the pattern it exists to detect elsewhere.

Parameterised, and a scan over the whole agent confirms it was the only site
interpolating a value — the others interpolate a table name from our own
fixed list, which no driver lets you parameterise.

Nobody watched the platform

The server already detected attacks on itself and told no one. A brute-force
produced login_logs rows and a lockout while the alert view showed nothing.
A revoked agent key retrying printed to a container log, which nobody reads
until they already suspect something — the wrong order for the one signal
that says a key is being used after it was taken away.

core/self_defence.py records these as events of the same shape as everything
else, with three rules it exists to keep: recording never breaks the thing it
records (every writer swallows its own failure — a server that will not reject
a bad login because it could not journal the rejection has turned a monitoring
gap into an outage); a severity is a claim, so the conclusion is stored
rather than every attempt; and no field here ever holds a credential.

Repeats fold into one row whose count climbs. A brute-force writing a row per
attempt makes the view that is supposed to reveal the attack into the thing
that buries it.

The panel is shown even when empty, which is a correction of the first
version. Hiding it read well as an argument — a permanently empty panel is
furniture people learn to skip — and it recreated exactly the failure the rest
of this work removes: an operator who never sees the panel cannot tell
"nothing has attacked us" from "this platform does not watch itself".

How much of the history can this server actually read?

The answer was invisible. A key mismatch printed one line per field name, once
each per process, then returned <decryption failed — key mismatch> per
value — so a table where 86% of the messages were unreadable produced the same
output as one bad row. Measured: 173 of 200 siem_events messages, 197 of 200
events_alert, 172 of 200 critical_files paths.

The Fernet key comes from the server's bootstrap and there is no rotation path,
so this is not fixable — which is exactly why the quantity has to be visible
rather than the fact. The console now reports the proportion.

And recovery stopped making it worse: the agent no longer re-offers rows it can
no longer decrypt itself. The first re-offer on a live host moved about two
thousand permanently unreadable rows onto the server, spending a bounded
recovery budget on data nobody can read.

The console was opening; nobody was waiting

17:25:03  no console in this process; trying the user's session
17:25:18  falling back to a shell behind pipes     (+15.2s)
          shell alive: powershell -NoLogo -NoProfile

The agent tries three strategies and only the last works when it runs as a
service. Reaching it takes ~15s. The server gave up at 15s and told the
operator the agent "would not open a console". It would; it was still trying.

Fixed on both sides, because raising the timeout alone would have covered the
symptom: STREAM_OPEN_TIMEOUT_S is now a named 45s constant with its reasoning
attached, and the agent remembers which strategy worked. Whether a process
can host a pseudoconsole is a property of how it was started and cannot change
while it runs — paying 15s to rediscover that on every console open is 15s
spent proving something already known. A stale memory falls back to the full
chain rather than reporting a failure, so the cache cannot become an outage.

A guard naming a permission that does not exist

require_permission("read_audit") — invented for the platform-events view and
never seeded anywhere. It fails closed, which is the right direction, but
silently: the route is unreachable for every role including admin, and looks
like a bug in the feature rather than a typo in its guard. A test now checks
every guard against the permissions db/init_userdb.sql actually creates.

A pooled connection returned on GC timing

await conn.ensure_closed() if hasattr(conn, "ensure_closed") else conn.close()

await binds tighter than a conditional expression, so the else branch was
never awaited — and _PooledConn.close is a coroutine. Connections went back
to the pool only when __del__ ran. Not a leak, but a pool that hands back its
slots on garbage-collector timing is a pool that runs out under exactly the
load that makes collection late.

close() is tried first, deliberately: on _PooledConn that is the method
that releases the slot, while ensure_closed would reach through __getattr__
to the underlying connection and shut the socket without telling the pool —
turning the warning into the slot leak it was only pretending to be.

Closing the plaintext ingest path

INGEST_TLS_REQUIRED=1 closes the listener inside the container while Docker
goes on publishing the host port, so a batch is still accepted by the proxy and
written into a socket nobody reads. The guide used to say "delete the line from
docker-compose.yaml", which is a bad thing to ask of a tracked file;
INGEST_PLAINTEXT_BIND binds it to loopback instead, the same mechanism
BIND_ADDR already uses. Verified: 5001 refused from the LAN, telemetry still
flowing on 5011.

Tests

2,392 passing. New invariants: every write to a table with declared encrypted
fields goes through the encrypting insert; no agent module interpolates a value
into a SQL predicate; every require_permission names a permission that
exists; a stream-open deadline must exceed the agent's own fallback chain; and
the two encryption maps must agree.

Several of these tests check that their own detector still matches — a scan
that has stopped recognising anything passes silently, which is the same class
of failure as everything above.

Seven writers stored fields declared encrypted in the clear, and plaintext reads back identically to ciphertext so nothing could see it; the portscanner had been dead every cycle on a SQL string it built by hand.
@d3vhex
d3vhex merged commit f469035 into main Sep 7, 2026
11 checks passed
@d3vhex
d3vhex deleted the feat/platform-self-defence-and-encryption-fixes branch September 7, 2026 18:16
d3vhex added a commit that referenced this pull request Sep 8, 2026
…ryption-fixes

Fields we said were encrypted were stored in the clear
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant