sqlsnoop: eBPF SQL statement monitor for Postgres and MySQL

[necco-c]

Adds sqlsnoop, an eBPF SQL statement monitor for Postgres and MySQL. It reads every statement any process on the host sends off the socket, with the parameter values the driver actually bound, and opens on a ranked list of the ones worth looking at.

Reviewer action items

• Apply the About description — applied at repo creation
• Apply the topic tags — applied, all 15, verified unmangled
• Add a LICENSE file — Apache-2.0, byte-identical to exectop's
• Resolve one flagged claim: the README says "Verified on 6.1, 6.6, 6.12 and bpf-next". Only 6.12.105 arm64 was verified directly by hand; the other three are what this PR's CI run covers. If the matrix goes green this claim is grounded and the box can be ticked as-is.
• Decide whether the four GIFs (9.7 MB) stay committed. Currently committed, matching every sibling, so the README renders on GitHub with no external hosting.
• Review the remaining flagged claims in the reviewer notes (10 of 11 are grounding 1, measured directly).

What it does

Three views. findings opens first, because a live feed at forty statements a second is twenty rows of routine lookups with one real problem somewhere in them, and spotting it is work the tool should do rather than delegate:

   WHEN            WHAT                               COST  STATEMENT
 ▸ 11:55:52 check  ORDER BY with no LIMIT — sort…   19.0ms  SELECT o.status, count(*), avg(o.total) FROM orders o…
   11:55:50 n+1    ×19 worst of 5 bursts            10.8ms  SELECT id, status, total FROM orders WHERE customer_id = ?
   11:55:52 check  SELECT * fetches every column    1.37ms  SELECT * FROM inventory LIMIT ?

feed is one line per statement with values inline and repeated shapes folded; top collapses by shape and ranks by total database time.

An N+1 is the bug this exists to find, and it is invisible to almost everything else: each of the two hundred lookups is fast and correctly indexed, so a slow-query log stays empty and an APM shows a slow endpoint with nothing to blame.

How it works

One BPF program covers both wire protocols. They frame messages differently but both are length-prefixed and tag-then-payload with the statement text at a shallow offset, which is the shared seam. The kernel decides only "is this a SQL frame and where does the text start", copies a fixed window, and stops. All parsing, normalization and parameter decoding happens in JS, where it can be unit tested.

Three facts about real client behaviour shaped the design, each found by running against live traffic rather than reading the specs:

• Clients prepare once and execute many times, so most executions carry only values. Attribution goes through the prepared-statement name in the Bind message. A workload issuing 84 lookups sent 36 statements and 99 parameter blocks, and those unattributed executions are the N+1 — a model needing a statement per execution cannot see what it exists to find.
• Clients do not wait for replies, so the kernel keeps a per-socket ring of send timestamps rather than one slot. With one slot, 120 of 135 replies had no timestamp to pair against and were dropped in the kernel.
• Pipelining clients send values before their statement, so orphaned parameter blocks are held briefly, guarded on placeholder count and age.

Verification

• 43 unit tests over the analysis layer, needing no kernel and no database (node --test test/lib.test.mjs). They caught two real bugs on their first run: verbOf classified WITH x AS (…) DELETE FROM t as a read, so a CTE-wrapped write was never counted as a write nor checked for a missing WHERE; and the leading-wildcard LIKE check could never fire, because its condition reduced to a test of the normalized shape where the % has already been replaced.
• All 6 BPF programs verify on 6.12.105 arm64. on_sendmsg sits at 345,443 instructions, 35% of the 1M ceiling — down from 686,652 (69%) after cutting the Postgres frame loop from six frames to four, which is the change this PR's CI matrix is most worth watching.
• make clean && make is silent. No warnings.
• Live capture verified against PostgreSQL 17 and MySQL 8 with psycopg2, psycopg3, MySQLdb and both command-line clients, on plaintext and inside TLS.

Note for anyone reproducing locally: run make veristat with sudo. Unprivileged it reports every program as failure with zero instructions, which is -EPERM from the loading probe rather than a verifier rejection, and it is a convincing false alarm.

Known limits, stated in the README

• Go and Java clients over TLS are invisible: neither exposes a C symbol to hook. Plaintext works for both.
• Clients on a Unix socket are invisible; the probes hook tcp_sendmsg. This is the most common cause of an empty first screen.
• Latency is socket-paired rather than exact, since neither protocol carries a request id.
• Statements are captured to 256 bytes, parameter blocks to 192, and truncation is marked.
• An 8-byte binary parameter is ambiguous (int8/float8/timestamp) because the Bind message does not carry the type. Ambiguous values are marked and the runnable-statement form refuses to build rather than handing you SQL that runs against the wrong row.

Reviewer notes with the full flagged-claims table live in ~/code/yeet-scripts-readmes/sqlsnoop/notes.md.

[necco-c]

Kernel matrix: green on all four

The first run failed on 6.1 while 6.6, 6.12 and bpf-next passed, which turned out to be a real portability bug rather than a flake. Fixed in 30cdba5 and re-run:

kernel	verdict
6.1.187	✅ all 6 programs
6.6	✅
6.12	✅
bpf-next	✅

What was wrong

Two of the six programs were rejected on 6.1, and they were exactly the two that call iter_base(). One of them is 38 instructions long, which is what ruled out complexity: a program that small cannot be failing for a verifier budget.

Both problems live in reading a msghdr's iov_iter, and both were introduced by a 6.4 change:

• iov_iter gained a ubuf member in 6.4. On 6.1 the field does not exist, so BPF_CORE_READ(msg, msg_iter.ubuf) has no relocation target and the verifier rejects the whole program at load. Not a wrong value: a rejected program.
• The iter_type enum values shifted. 6.4+ has ITER_UBUF = 0, ITER_IOVEC = 1; before that ITER_IOVEC = 0. So the constants clang bakes in are wrong on the other side of that boundary even where the fields exist. This half would have been a silent data bug rather than a load failure, and considerably harder to find.

The fix

CO-RE feature detection. bpf_core_field_exists() guards the ubuf read so it is dropped entirely on a kernel lacking the field, and bpf_core_enum_value() resolves the type constants against the running kernel's BTF rather than the one the compiler saw.

One wrinkle worth recording: bpf_core_field_exists() only defers the check to load time, so the field still has to typecheck at compile time, and a 6.12 vmlinux.h has no msg_iter.iov member. The pre-6.4 layout is therefore declared locally as a preserve_access_index struct, which is the standard idiom for referencing a field the compile-time BTF does not carry.

Verified after the change

• All 6 programs verify on all four kernels. on_sendmsg is 328,721 instructions on 6.1, 33% of the 1M ceiling.
• Live capture unchanged: 207 statements, 675 parameter blocks, 842 replies, values and decimals decoding correctly. Worth checking explicitly, because a CO-RE change to the buffer read is exactly the kind that compiles and verifies while capturing nothing.
• 43/43 unit tests pass, make clean && make silent.

This also grounds the README's "Verified on 6.1, 6.6, 6.12 and bpf-next" claim, which was the one open flagged item. It can be ticked as measured rather than aspirational.