Skip to content

Say where the telemetry stops, instead of showing an empty table - #40

Merged
d3vhex merged 1 commit into
mainfrom
feat/attack-chain-and-telemetry-health
Sep 3, 2026
Merged

d3vhex merged 1 commit into
mainfrom
feat/attack-chain-and-telemetry-health

Conversation

@d3vhex

@d3vhex d3vhex commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Three ways to render an empty table

The Network and Hardware tabs were blank. The agent's log said

[+] network_connections sent (50 rows)

every cycle, and the server held none of them. Both were telling the truth,
and there turned out to be three unrelated breaks behind one symptom.

A table cannot be both a snapshot and deduplicated. network_connections,
hardware_inventory, critical_files and docker_containers were in both
sets. A snapshot is emptied before each batch; a duplicate row is skipped when
its fingerprint has been seen. Together: batch one writes 50 rows, batch two
deletes them and skips every replacement, and every batch after that does the
same. Full once, permanently empty afterwards. The two lists are now disjoint
and _assert_ingest_modes_are_exclusive refuses the overlap at import, so the
next one is a server that will not start rather than a tab nobody can explain.

An encrypted column has to be wide enough for the ciphertext.
local_addr was VARCHAR(64) and is one of the fields enc_db encrypts. A
Fernet token is over a hundred characters whatever the plaintext, so every
batch failed on its first row with 1406 Data too long — after the snapshot
DELETE had already run. Four columns were too narrow;
tests/test_encrypted_column_widths.py compares the two declarations that have
to agree, so the next one fails in CI.

One failing collector took the four after it. edr_enforcer.main called
five collectors as five bare statements, so the first to raise ended the cycle
— and the caller retries on a timer, so it ended every cycle after that too.
Network is second and hardware is fourth, which is why they went dark together.
They are isolated now and each names itself. periodic_wrapped also reported
failures behind if debug, which nothing sets: a collector that raised on its
first line failed silently for the life of the agent.

Telemetry health

The deeper reason none of that was visible: the agent marks rows sent as soon
as sendall returns. The ingest protocol has no reply, so a server that
accepts a batch and stores none of it is indistinguishable from one that
stored all of it. The agent reported success in all three cases and was not
wrong to.

/telemetry-health/<agent> puts the two halves next to each other — what the
agent holds and believes it shipped, against what this server holds — and
names the break: lost in transit, not collected, send failing, queued,
flowing. All five used to be the same blank list, and each needs a different
person to do a different thing. The page is reached from the host, which is
where the empty tab that prompts the question is.

The proper fix is an acknowledged ingest, so the agent stops deleting rows it
only assumes arrived. That is a wire-protocol change and belongs with the TLS
work, where the ingest protocol is opened anyway.

ATT&CK as a hierarchy

The coverage endpoint compared techniques as opaque strings, so a rule tagged
T1003.001 and an event carrying T1003 never matched: one landed in
"covered but never seen" and the other in "seen with nothing covering it".
Those two lists are the only ones an operator acts on — one sends somebody to
test a detection that already works, the other to write a rule that exists.

The fix is not "compare parents". Detecting T1003.001 (LSASS memory) says
nothing about T1003.003 (the AD database): different action, different
telemetry, and the rule for one will never fire on the other. So coverage
travels down from a parent and only partially up from a child, and
covered_only_by_a_sibling is now its own state — the one most easily
mistaken for coverage, because the rolled-up cell is green while that specific
action would go unnoticed.

/attack-chain/<agent> lays a host's tactics out in kill-chain order with the
stages that did not happen drawn as gaps. The gaps are the point: you cannot
see from a list of the four tactics that fired that nothing was ever
exfiltrated.

A design system, and charts

1291 inline style objects across 24 pages, and StatCard written three
separate times. No page was written carelessly — there was no shared
vocabulary to be careless about, so spacing, radius and colour were decided
again on every page and diverged.

components/ui is that vocabulary: page header, card, stat, table, badge, and
LoadingState / EmptyState / ErrorState as three separate things, because
collapsing them is how an empty alert list and an alert list that failed to
load came to look identical.

The toy affordances went with it. .card:hover lifted two pixels and glowed
over a 12px backdrop blur — on a page that is mostly cards, the whole console
moved when the pointer crossed it. Radius is one small scale now; the old set
had 12px cards beside 4px inputs.

Recharts sits behind wrappers that read the tokens, because it styles itself
with props and left alone draws a white grid on a black page. Series colours
are deliberately not the severity colours: a bar being red should not imply
danger unless danger is what the bar measures. The dashboard gains fleet
health, connection status, version spread and a two-axis threat trend — a
spike in detections against a flat distinct-technique line is one noisy rule,
not an incident, and only both series together show that.

Also

  • The agents list reports channel_connected: whether a command would reach
    the host now, which is not the same question as whether it is reporting.
  • verify_auth_wiring could not resolve websocket handlers, so the boot-time
    auth audit covered none of them — and those three are the agent channel, a
    root shell, and a live view of a screen.
  • Two bash -n tests were failing on a WSL VM that would not start, which
    read as "the build script is broken shell". A syntax error names itself on
    stderr; an unrunnable shell leaves it empty.

Techniques compare through the ATT&CK hierarchy rather than as strings, the console gets a design system and charts, and four separate silent breaks behind one blank inventory tab are fixed and pinned: a snapshot table that was also deduplicated, an encrypted column too narrow for its ciphertext, a collector chain where the first failure disabled the four after it, and a read that never decrypted.
@d3vhex
d3vhex merged commit 17bd48d into main Sep 3, 2026
11 checks passed
@d3vhex
d3vhex deleted the feat/attack-chain-and-telemetry-health branch September 7, 2026 18:16
d3vhex added a commit that referenced this pull request Sep 8, 2026
…alth

Say where the telemetry stops, instead of showing an empty table
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant