Security keys, a table that never had a collector, and the console around them

[d3vhex]

Security keys, a table that never had a collector, and the console around them

Verified against the running deployment rather than only in tests — which
turned out to matter, because four of the bugs below were found by clicking the
button, after the tests for them had passed.

---

1. Security keys (WebAuthn)

A one-time code typed into a convincing copy of this login page works on the
real one, and the operator has no way to tell. A key signs over the origin it
is actually talking to.

There is no configuration, and that was the design question. The
relying-party ID is derived from the request, so a laptop at
https://localhost:8000 and an estate at https://soc.example.com both work
without being told anything, and no setting can disagree with the URL people
type.

The cost is that some origins cannot host a key, and the console says which
rather than showing a button that fails:

Console opened at	Security keys
https://soc.example.com	available
http://localhost:8000	available — localhost is a secure context by definition
https://10.0.0.5:8000	unavailable. A relying-party ID must be a domain; browsers reject an address.
http://soc.example.com	unavailable. Needs a secure context — set TLS_ENABLED=1.
any, webauthn package absent	unavailable. The import is lazy, so the option is lost rather than the console.

One-time codes are unaffected in every row, and the message says so — an
operator who reads "unavailable" as "two-factor is off here" turns the whole
thing off.

Three decisions worth reviewing:

• A credential records its RP ID. A key registered at localhost is one
  the browser will not offer at a hostname. Without the column the console
  lists a factor that cannot work and the prompt times out silently.
• _has_security_key is deliberately not scoped to the RP ID. Whether a
  login needs a second factor is a property of the account; whether this
  browser can offer a key is a property of the origin. Conflating them would
  let somebody with a key registered at a hostname skip the second factor
  entirely by opening the console at an address — turning the origin binding,
  the whole point of WebAuthn, into a bypass.
• The signature counter is checked. A key presenting a lower count than the
  last one seen is refused and raises WEBAUTHN_COUNTER_REGRESSED (CRITICAL).
  Zero means "not implemented" — true of most platform passkeys — and is
  accepted, because refusing it would switch the control off for the hardware
  people actually have.

Verified end to end on a real authenticator:

user_webauthn        rp_id=localhost  transports=hybrid,internal  sign_count=0
                     created 18:17:25   last_used 18:56:31
webauthn_challenges  0 rows            (single-use deletion works)
login_logs           admin  success  "security key"

base64url is not base64

frontend/src/lib/webauthn.ts has its own tests because this is where WebAuthn
fails intermittently. - and _ stand in for + and /, padding is dropped,
and atob accepts neither. The alphabets agree until a byte lands on one of
the four differing characters — so an implementation that forgets to translate
passes a demo, passes review, then throws InvalidCharacterError for one
operator in three.

---

2. Certificate names (TLS_SAN)

The shipped certificate carried DNS:localhost, DNS:localhost — the SAN list
was [cn, "localhost"] with no dedup and no way to add a name. A console at
https://10.0.0.5:8000 failed hostname verification in every browser, on top
of the untrusted-CA warning: two errors that look like one.

TLS_SAN takes hostnames and addresses; anything that parses as an address
becomes an IP SAN, because a bare address in the DNS list matches nothing.

Generation is idempotent, so editing .env on a deployment that already
has a certificate does nothing — correctly, since a new identity per restart
would break every agent that pinned the CA. It now says so at startup with the
command to fix it. That check silently vanished on its first draft
(except: return made "unreadable" and "names match" produce identical
output); it has its own test.

docs/production-deployment.md §3.8 is the six-step version, including that
--force issues a new CA, and what happens if you skip all of it: nothing
breaks, you just do not get a security key.

---

3. security_audit: the table nothing ever wrote to

It existed in both schemas, in ALLOWED_TABLES, in DEDUP_TABLES and in two
encrypted-field maps since the beginning. Three modules were documented as
producing it. None of them ever did. So it shipped empty every cycle and
the console showed it permanently as NOT COLLECTED — a sensor that reads as
broken rather than as never built.

Sentora/modules/security_audit.py is the collector: posture findings, read
only. Windows (registry, services, local accounts, firewall) and Linux
(sshd_config, sudoers, /etc/passwd, file modes).

Running it on a real host immediately produced three bugs worth keeping:

• It accused the anti-malware service of being malware. WinDefend,
  WdNisSvc and MDCoreSvc came out as CRITICAL persistence, because
  \programdata\ was in the writable-directory list. That root is writable,
  which is what makes it the textbook example — but the vendor subdirectories
  under it are not, and Defender lives there. The worst false positive
  available, and one is enough for an operator to stop reading the category.
• It crashed on a localised Windows. subprocess.run(text=True) decodes
  with the console code page — cp1254 here — and raised UnicodeDecodeError
  inside subprocess's own reader thread, where the caller cannot catch it.
• It overstated. "Can authenticate with an empty password" for
  PasswordRequired=False, which means the account is permitted to have a
  blank password, not that it has one. Downgraded and reworded; a finding that
  overstates is one an operator learns to dismiss, along with the next.

The group-membership check queries by SID (S-1-5-32-544), not by name:
Administrators in English, Yöneticiler here, and matching the name is a
check that silently finds nothing on a localised install.

Deduplication is on (category, finding) and deliberately not on
details — details drift ("2 members: …" → "3 members: …") and a fingerprint
over them re-inserts the same finding whenever one changes. send_alert
learned the general version of this: a state-reporting detector without
deduplication produced the identical alert 288 times a day.

And then the chain still ended in a wall. The rows arrived, stored
correctly, decryptable — and there was no route and no UI. Added both:
GET /<agent>/security_audit and a Posture tab. Its empty state is worded
differently from the telemetry tabs on purpose: empty here is a real answer.

Verified live: 6 findings on this host, all independently confirmed, read back
through the real decrypting path.

---

4. Agent fixes

automations/pending had been 401ing for the life of every install.
AutomationsClient sent no headers at all. The 401 is correct — unauthenticated
the route let anyone read the response actions queued for a host and POST
{"task_id": N, "status": "SUCCESS"} against every id, so the actions never ran
while the console showed them green.

What was missed is that this is not dead code. call_agent_soar queues a
pending row precisely "so an agent that missed the push can still poll for it".
Broken, the fallback did not exist. It authenticates now, and a rejection is
said once rather than four times a second.

The vulnerability scan reported a clean result it could not have reached.
scanners/vuln.py mapped Windows to OSV's NuGet ecosystem, but the package
list is registry Uninstall display names — "Google Chrome" — and NuGet is
Newtonsoft.Json. The two name spaces do not overlap and OSV has no ecosystem
for programs installed on Windows, so the query was structurally incapable of
matching while reporting new_findings=0 every cycle. It is a named skip now,
surfaced through the skipped_reason the console already renders.

---

5. The console (C2 + responsive)

All eight remaining pages moved onto the shared kit: 460 inline style objects
→ 171, and eight hand-rolled modal overlays → the kit's Modal (Escape
closes, backdrop closes, portals to document.body because position: fixed
resolves against the nearest containing block — AgentDetail had hit exactly
that).

The styling was the smaller half. Every one of the eight swallowed its load
errors. An empty Identity & Access reads as "nobody has access". Blank
AdminConfig forms read as "nothing is configured" and those forms save.
Playbooks reported a failed run to the browser console only — on a page whose
own editor labels steps "cannot be undone". AIAnalysis had a hardcoded green
"RabbitMQ Worker Active" pill with no request behind it. Deployment lost
enrolment tokens silently, because navigator.clipboard does not exist outside
a secure context and the button still said "Copied".

Then responsiveness, which the kit's own docstring had warned about: six pages
wrote gridTemplateColumns: '300px 1fr' inline, where a breakpoint cannot go.
Added .split-grid. Three places decided layout from window.innerWidth read
during render — which never follows a resize, and looks responsive to anyone
testing by reloading at each size. Sidebar was right only by accident; it
takes isMobile as a prop now.

---

6. Documentation that described a different product

Sentora/docs/MODULES.md was unchecked and substantially wrong:

• Three table names that do not exist: installed_software,
  resource_log, disk_info.
• edr_enforcer's entry was wrong end to end — documented as a
  hash-baseline scan writing security_audit; it is the agent's main periodic
  collector and writes five other tables, four of them undocumented.
• network_inventory and packages appeared nowhere.

tests/test_module_reference.py holds both directions now. While correcting
the paragraph about scrub_nuls, a real NUL byte got into the file from an
escape that collapsed one layer too far — enough for grep to call it binary
and go quiet. Fixed, with a test.

---

7. tsc --noEmit -p tsconfig.json checks nothing

frontend/tsconfig.json is a solution file: "files": [] and project
references. Pointing tsc -p at it type-checks the empty set and exits 0, for
any code. vite build does not close the gap either — rolldown strips types
without checking them.

Two unused imports left by a refactor passed both, and were caught by the
Docker image build (which runs npm run build → tsc -b) — after the
container had spent a day serving an old agent binary, because the failing
build never produced a new image and the failure was masked by a
docker ... | tail pipeline returning tail's exit code.

tests/test_frontend_typecheck.py makes the real gate structural.

---

8. Four bugs the tests could not have caught

Every one was found by using the feature, and in each case the tests written
for it passed:

Bug	Why the tests missed it
security_audit had no route or UI	The tests checked the collector, not the chain
register/begin returned 500	json in app.py is Sanic's response helper, not the module. Valid Python, fails at request time only
login/2fa/webauthn/* returned 401	Added to the test's session-only list, not to the server's public-handler list
The first fix for that went to the wrong list	_PUBLIC_EXACT_PATHS is a path-shaped mirror, consulted only as a backstop; authenticate matches handler names

New invariants for each: no module attribute may be read off a shadowed name
in app.py; any route that authenticates with a pending token must be in
_PUBLIC_HANDLERS; and the mirror must not drift from it. Each detector was
run against the shipped state to confirm it fires.

---

9. CodeQL alerts

Eight open on main. Five were real, two were true-but-misread, and one is the
design.

The search query builder was not faithful (js/incomplete-sanitization).
The label misleads here: the query box beside the builder is editable by
design, so an operator can type raw Lucene whenever they like and there is no
boundary being crossed. What was broken is that clicking a filter produced a
query meaning something else. It escaped the quote and not the backslash, so
every Windows path ran off the end of its own phrase; and it only quoted when
the value held whitespace or a quote, so a)OR(b went in bare and changed the
structure of the query rather than the term. Moved to
frontend/src/lib/luceneQuery.ts with tests — backslash escaped first, because
doing it second escapes the backslash that pass just added.

Two TLS floors stated rather than inherited. create_default_context has
set TLS 1.2 as the minimum since Python 3.10, so Sentora/main.py changes
nothing today — which is why it is worth writing down, since the guarantee
otherwise depends on which Python the agent was frozen against, and the
connection it protects is every byte of telemetry the endpoint produces. The
probe in link.py is different: nothing is sent over it and verification is
already off, so the floor is not protecting the handshake. It bounds what the
message is allowed to claim — a server that only speaks TLS 1.0 should not
produce "this is serving TLS, switch to https://", because switching would be
to something worse than what the agent has now.

TRUSTED_PROXIES is not a secret, but that branch runs precisely when
somebody has put something in it that is not an address, and the commonest way
that happens is a value pasted into the wrong variable. It logs the entry's
position now instead of its content, which is what an operator needs to find it
anyway.

The agent banner carried the MAC address. OS_INFO is
platform.platform() plus |HOST= and |MAC=; the server needs all of it,
because that is how a reinstalled agent is recognised as the same machine. The
log does not. A hostname is unremarkable in a file that lives on that host; a
MAC is a stable hardware identifier that follows the machine across networks
and reinstalls, sitting in a plain-text file that support bundles and
screenshots routinely include.

The rotation script printed the new database password on its failure path.
The recovery has to exist — the account has already been altered at that point,
so a password nobody has is a database nobody can reach. It does not have to
put the secret in terminal scrollback, a CI job log, or the screenshot somebody
takes of the failure to ask about it. It writes .db_password_rescue beside
.env, owner-readable only, and prints the path. Printing survives for the one
case left: the rescue file cannot be written.

One is dismissed rather than fixed. .env holds DB_PASSWORD in clear
text because that is how every service in the stack is configured, and there is
no version of this product where it does not. The addressable half is done —
.env is restricted to its owner after writing. The rest is a true finding,
accepted.

---

10. Deduplication that could never match anything

Reported from the console: the same program listed over and over on the Assets
page. Measured on the live database:

packages             249,980 rows stored,   167 distinct agent fingerprints
software_inventory   161,157 rows stored,   313 distinct programs
network_inventory     37,225 rows stored,    79 distinct ports
ingest_fingerprint   one row per row received, none of them ever matched

compute_fingerprint hashed the row as it arrived. That cannot work, for two
independent reasons: encrypted columns arrive as enc::gAAAA… and Fernet uses
a random IV, so the same plaintext encrypts differently every time; and
timestamp is stamped per collection cycle, so even a plaintext table changes
on every send. Either one makes the hash unique by construction, so a table
could be listed in DEDUP_TABLES and be structurally incapable of recognising
a repeat — while every layer reported success.

Sentora/modules/enc_db.content_fingerprint exists for exactly this and says so
in its own docstring: the agent computes it over plaintext because the server
cannot. The server never read it. Read the packages line above twice — the
agent had been computing a perfectly good fingerprint for it all along, 167
values for a quarter of a million rows, and it was thrown away on arrival.

• compute_fingerprint prefers the agent's dup_fp; the row hash stays as the
  fallback for a producer that sets none.
• software_inventory and network_inventory set one. The network identity
  excludes the PID — it changes when a service restarts, which would make
  the same listener a new row after every reboot, the same bug in a hat.
• FINGERPRINTED_TABLES widened from the two AI-triaged tables to cover
  fim_data, registry_logs and process_events, which had been going
  without: one host held 7,663 fim_data fingerprints and matched none of
  them, re-reporting the same unchanged files for the life of the install.
• portscan_result sets the same identity its local is_duplicate already
  uses, so the server skips what the host skipped.
• soar_actions deliberately sets none, with the reason recorded: an action
  taken is a one-off event and hashing the row is the right identity there.

Fixing ingest does not shrink the pile, and nothing else ever would, so
there is a one-off repair. Rows carrying a fingerprint collapse onto it,
keeping the lowest id so the first-seen date survives; rows carrying none
cannot be told apart and go, which is safe only because those three
collectors re-report their whole list every cycle. The change-log tables are
untouched — the agent marks a row sent once and never offers it again, so a
row deleted from fim_data is gone.

It runs a bounded chunk per ingest rather than all at once, because
create_tables_if_not_exist is on the ingest path: an unbounded DELETE across
400,000 rows times out the send that triggered it, the agent retries, and the
repair restarts from the top — a repair that never finishes while every ingest
pays for the attempt. The marker is written only once every table has
converged.

The Software tab was reading the wrong table

Chasing the duplicates surfaced the reason they were visible at all. The tab
renders Package Name, Version, Vendor and Installed Date, and was served from
packages — which holds a name and a version. Two of its four columns had
been blank on every row since the tab was written, and a blank cell reads as
a fact about the host: this machine reports no vendor. The agent collects
vendor and install date, ships them, and they landed in software_inventory,
which nothing read.

It is also the better inventory on Windows: packages is the MSI product list,
software_inventory is the Uninstall registry — 313 programs against 167 on
this host. packages stays where it is; it is the vulnerability scanner's
input and its shape belongs to OSV.

tests/test_assets_columns.py checks every rendered column against the schema
of the table its route actually serves. It reports exactly ['install_date', 'vendor'] against the old route. The Network tab had the same bug, fixed by
hand earlier in this branch after someone looked at the screen.

tests/test_ingest_dedup.py checks the writers rather than the list, through
the AST. The regex version of that check required a literal table name and was
blind to the five writers that pass a TABLE constant — including both of the
ones this change was about.

network_inventory had no view either

Same collector, same commit, same gap: shipped and stored for the life of every
install, searchable in Hunt because everything is indexed, and with no page
that shows it. A search box is not an inventory — you cannot ask it "what is
listening on this host" without already knowing.

Added an Assets → Ports tab. It is not a second copy of the Network tab:
network_connections is conversations already underway, this is what the host
accepts, which is the attack-surface question. It is also a better answer than
portscan_result gives — a scan infers the surface from outside, this is the
kernel's own list with the owning process named.

State and Remote are deliberately not columns: every row is a listener, so
both are constant, and a column with the same value on every row spends width
to say nothing. In their place the bind address is rendered with what it means
for reach — 0.0.0.0 all interfaces, 127.0.0.1 loopback — because
that distinction is the entire reason one listening port matters more than
another, and making the operator derive it 80 rows at a time is how it gets
skipped. The chart groups by process rather than by state for the same reason:
grouping listeners by state draws one bar.

The tab row wraps now (flexWrap). width: fit-content does not shrink below
its content, so a fourth tab was simply unreachable on a narrow viewport.

---

11. The search index held ciphertext

Found while checking whether network_inventory was read anywhere. It was —
server.py called index_log for every ingested table unconditionally, so
everything reached OpenSearch. With the row exactly as received.

core/telemetry_crypto documents where agent telemetry becomes plaintext and
where it does not:

the database row          stays encrypted
the RabbitMQ message      stays encrypted
the prompt                gets plaintext
correlation fields        get plaintext

The index is a fifth destination that list never mentioned, and it got
ciphertext by default rather than by choice. Measured on the running
deployment, 300-document samples:

siem-events / message            71/71 enc::gAAAA...
registry-logs / value_data     300/300
fim-data / path                300/300
critical-files / path          122/122
network-connections / process  300/300
hardware-inventory / name      300/300
packages / package             300/300

and the consequence, over the same 69 documents:

message:powershell   ->  0 hits
source:PowerShell    -> 52 hits

About 863,000 documents unsearchable on the one field that mattered. Worth
being precise about the failure mode: through the console it returns nothing,
because /api/logs/search sets allow_leading_wildcard: False. Querying
OpenSearch directly, value_data:*exe* returns 34 hits that are coincidental
exe substrings inside base64 — so the underlying data does produce noise that
looks like findings, and only that one setting stands between an operator and
it.

Indexing was also never a decision. A snapshot table is emptied and
rewritten in MySQL on every batch; the index kept every copy and none of the
deletes. 426,768 hardware_inventory documents for a CPU, some RAM and four
disks — not a history, since what makes it a snapshot is exactly the part
that is missing.

So index_log now decrypts, and INDEXED_TABLES is an allowlist. The rule is
what a search is for: an index answers "find this, anywhere in the fleet, in
this window", which suits an append-only event log and does not suit a table
whose answer is "the current list for this host". The state tables have a route
and a view each, and a view gives an exact answer where a time-windowed
full-text search gives a vague one. NOT_INDEXED records the reason per table
beside the allowlist, and tests/test_opensearch_indexing.py asserts the two
partition ALLOWED_TABLES — a table added to ingest and to neither list is a
silent decision, silent in the direction that matters, since index_log
returns early for anything it does not recognise.

This is a real trade and the deployment guide now states it. OpenSearch
becomes the one store in the stack holding readable telemetry: a copy of
mysql_data yields ciphertext and needs data/fernet.key too, a copy of
opensearch_data yields log lines. Against that, a hunting surface that cannot
match a process name is not a security control, only a broken feature that
reports success. §2.2 has the table, and scripts/prune_search_indices.py
removes the ~1.1M stale inventory documents — printing counts and asking first,
because these may back saved visualisations in Dashboards, which it cannot see
and must not assume about.

Verified live after rebuild:

newest indexed siem_event   plaintext
  'docker event: start type=container container=sentora-ingest ...'
message:container    -> 1 hits    (0 before; the field was ciphertext)
dropped indices      no documents newer than the restart

The ~15.5k documents indexed before the change stay ciphertext. There is no
reindex path and this does not add one.

A defect in §10's repair, found by running it

network_inventory went to 0 rows and stayed there. The agent on that host
had not been rebuilt, so it kept sending the table with no dup_fp, and
"delete what has no fingerprint" deleted each fresh batch as it landed: table
permanently empty, console reading "this host has no listening ports", and the
repair never converging so it never stopped. The same failure this branch is
otherwise about, introduced by the fix for it.

The high-water mark is now taken once per table and stored in
ingest_migration.value. Rows that existed when the repair started go; rows
that arrive later stay whatever they look like. Upgrading the agent is what
fixes the duplicates, rather than a condition for seeing any data at all.

---

Testing

2813 passed, 6 skipped (Python) · 53 pass, 0 fail (frontend) ·
npm run build exits 0 (so tsc -b passed — the only real type gate here).

Verified against the running deployment: the security-key ceremony completed on
real hardware, security_audit findings reached the server and read back
decrypted, automations/pending answers 200, and the agent log carries no NUL
errors.

§10 and §11 were verified after a rebuild of app and ingest:

packages            251,706 -> 169 rows      (167 distinct packages)
ingest_fingerprint  497,441 -> 44,082
network_inventory    35,951 -> 0, refilling
newest siem_event   message indexed as plaintext
message:container   1 hit    (impossible before)

What is not verified live: the high-water bound. The repair had already
stamped its done marker under the first build, so the guard short-circuits and
the bounded path has run only against its five unit tests on this host.
software_inventory and network_inventory will carry dup_fp once the agent
binary is rebuilt; until then they accumulate duplicates that are visible
rather than hidden, which is the intended behaviour of the bound.