[d3vhex]

Why

The agent was rebuilt and reinstalled to pick up the port scanner fix, and the
port table still had no new rows. Its newest was the previous day. The agent
log said why, ten times over:

[+] 60236/tcp - TCP-60236   | Unknown-60236  | vUnknown
[+] 64120/tcp - TCP-64120   | Unknown-64120  | vUnknown
[!] Unexpected error: A string literal cannot contain NUL (0x00) characters.

The scan ran every hour, found its ports, and died on the write.

A NUL byte from a banner

PostgreSQL text cannot hold \x00 — the wire protocol terminates strings with
it — so psycopg2 refuses the statement. And the scanner opens a socket to every
listening port and reads whatever comes back, which for a binary protocol is
binary. decode(errors='ignore') does not save it: that drops bytes which are
not valid, and \x00 is a perfectly valid code point. The banner keeps it,
the service and product strings are cut from the banner, and the insert dies.

Reproduced against the agent's own database:

raw banner  -> ValueError: A string literal cannot contain NUL (0x00) characters.
scrubbed    -> stored

Note the type: ValueError, not psycopg2.Error. An except psycopg2.Error
would have walked straight past it.

db.scrub_nuls now strips NUL from every bound parameter — inserts, updates
and selects, because the scanner's duplicate check passes the same product
string to a SELECT and that raises first, so the row never reached the insert
that looked like the problem. Bytes are left alone; bytea stores NUL happily
and stripping there would corrupt the one case that was already correct.

Second fix, for the shape rather than the input: the insert loop sat inside the
try that reports the scan as failed, so the first unstorable row ended the scan
and took every port after it. One banner cost the whole hour. It is per-row now.

Three columns that were NULL on all 108 rows

target_ip, state and banner exist on the server's portscan_result and
did not exist on the agent's. The server's schema even says so:

-- Superset: product/version VEYA state/banner/target_ip ikisini de destekle

No agent version has ever sent the second half. So the console got a port list
that does not say which host or whether the port answered — a list of numbers.
Added to the agent schema and to the insert; ADD COLUMN IF NOT EXISTS
verified idempotent against the live agent database, which is what
apply_schema requires since it runs the file on every start.

Two tables collected every cycle and never sent

While checking the local database against the server's:

table	local rows	unsent	on server
software_inventory	368,902	368,902	0
network_inventory	78,224	78,224	0

modules/inventory.py has been writing both since it was written — Inventory: Scanned 326 apps and 68 open ports is in the log every twenty minutes. Neither
name was on TABLES in the agent's main.py, so not one row was ever offered.

The server was ready the entire time: both are in its ALLOWED_TABLES and its
DEDUP_TABLES. It has been prepared to receive and deduplicate two tables
nothing ever sent it, and the console showed a machine with no software
installed and nothing listening.

Every layer was internally consistent, which is why it survived this long.

The backlog, and why it could grow

prune_sent refuses to delete an unsent row. That is right for a table being
delivered — deleting it is the data loss the module exists to prevent — and
exactly wrong for one that is not. So the two of them grew for the life of the
install, on an endpoint this product is supposed to be watching rather than
filling.

prune_unsent is the second bound, at 20,000 rows against the sent limit of
5,000. Measured against the live database before committing to the number:

software_inventory   unsent= 368902  would abandon= 348901  keeps=20000
network_inventory    unsent=  78224  would abandon=  58223  keeps=20000
fim_data             unsent=      0  would abandon=      0
siem_events          unsent=      0  would abandon=      0

It bites only what is not being shipped. A healthy table drains fifty rows
every few seconds and never approaches it, and the log line for it is phrased
as a loss because it is one.

The invariants

None of this was caught by a test, so:

• test_the_agent_ships_every_table_it_collects — every table in the
  agent's own schema is either on TABLES or on AGENT_LOCAL_ONLY with a
  written reason. Two entries so far: automations travels the other way,
  security_audit has no collector. This is the test that fails today without
  the fix.
• test_nothing_is_excused_from_shipping_that_no_longer_exists — the
  excuse list rots like any list.
• test_an_unshipped_table_cannot_grow_for_ever — the bound stays wired in.
• test_a_nul_byte_does_not_reach_postgres and
  test_every_bound_parameter_is_scrubbed across all four helpers.
• test_the_scan_survives_a_row_it_cannot_store, and one per column for
  target_ip / state / banner in both the insert and the agent schema.

One more, small: _schema_tables was reading a table named leaves out of the
comment -- Existing agents: CREATE TABLE IF NOT EXISTS leaves the old CHECK in place. Harmless where the result was filtered against ALLOWED_TABLES, not
harmless in a test that asks a question about every name it finds. Comments are
stripped first now, with the stripper the schema code already uses.

Testing

2533 passed, 5 skipped. The NUL failure and its fix were reproduced against
the running agent database, and the schema migration applied and re-applied
there cleanly.

The agent needs rebuilding again for the code half to take effect; the schema
half applies itself on the next start.