Skip to content

fix(cmdfilter): bound selector-miss ledger keys by size, and drop non-text blocks - #55

Merged
OsherElhadad merged 1 commit into
mainfrom
fix/miss-ledger-bound
Aug 10, 2026
Merged

fix(cmdfilter): bound selector-miss ledger keys by size, and drop non-text blocks#55
OsherElhadad merged 1 commit into
mainfrom
fix/miss-ledger-bound

Conversation

@OsherElhadad

Copy link
Copy Markdown
Collaborator

Closes #48. Found by the integration review, replaying real captured traffic through merged main.

Problem

maxMissKeys = 200 bounds how many keys the ledger holds, not how big they are — and selectorKey runs on whatever the tool returned. Replaying capture-tb.jsonl filled the top 25 slots with base64 image payloads:

[{"type":"image","source":{"type":"base64","data":"iVBOR…

Two consequences:

  1. The ledger stops doing its job. It exists to answer "which filter is worth writing next" (after rtk's parse_failures table). On multimodal traffic that answer is 200 near-identical image blobs, crowding out the actionable misses — the real ones observed were Exit code 1 (211×) and source listings.
  2. 200 multi-KB keys sit in the aggregator under its lock and ship in every /stats scrape.

Fix

Bound the key at 120 bytes. A selector is a shape, so a short prefix identifies it; the tail only makes near-identical misses occupy separate slots. Well above any real command banner, well below a payload. Truncation cuts on a rune boundary so a key stays valid UTF-8 in the JSON payload.

Drop non-text blocks entirely rather than truncating them. An image carries no output shape a filter could ever match, so recording it is noise by construction — not a key that merely happens to be too long. A truncated blob would still be a useless entry occupying a slot.

Test

TestMissLedgerKeysAreBoundedAndTextOnly asserts four things, and the fourth is the one that matters most:

  • the key is bounded, and keeps its identifying prefix (docker build … survives the cut);
  • a truncated key is valid UTF-8 (tested with multi-byte runes);
  • non-text blocks are dropped, in both compact and spaced JSON spellings;
  • real command banners survive intactReading package lists..., > Task :app:compileDebugKotlin, make[1]: Entering directory '/src'. The bound must not cost the signal it exists to rank, and a test that only checked the truncation would happily pass a version that mangled every real selector.

Verified non-vacuous — removing the non-text guard reproduces the reported symptom exactly:

--- FAIL: TestMissLedgerKeysAreBoundedAndTextOnly
    non-text block recorded as a miss shape: "[{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"data\":\"iVBORw0KGgo…

Gates

go build -tags cg_skeleton ./... · go test -tags cg_skeleton ./... · gofmt -l — all clean.

Remaining integration findings

#45 (the volatile-tail split is a silent no-op on Bedrock Converse — it reports Changed: true and achieves nothing) and #47 (pin-budget exhaustion reopening the TailOnly fail-open, on the /compact route only) are still open.

…-text blocks

maxMissKeys bounds how MANY keys the ledger holds, not how big they are, and
selectorKey runs on whatever the tool returned. Replaying multimodal traffic
filled the top slots with base64 image payloads:

  [{"type":"image","source":{"type":"base64","data":"iVBOR...

Two consequences. The ledger exists to answer 'which filter is worth writing
next' (after rtk's parse_failures table), and 200 near-identical image blobs
answer nothing while crowding out the actionable misses. And they sit in the
aggregator under its lock and ship in every /stats scrape.

A selector is a SHAPE, so a 120-byte prefix identifies it; the tail only makes
near-identical misses occupy separate slots. Truncation cuts on a rune boundary
so a key stays valid UTF-8 in the JSON payload.

Non-text blocks are dropped rather than truncated: an image carries no output
shape a filter could ever match, so recording it is noise by construction, not a
key that happens to be too long. A truncated blob would still be a useless entry.

The test asserts the bound, rune-safety, that non-text blocks are dropped, AND
that real command banners survive intact -- the bound must not cost the signal it
exists to rank.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(cmdfilter): the selector-miss ledger bounds key count but not key size — multimodal traffic fills it with base64 blobs

2 participants