fix(cmdfilter): bound selector-miss ledger keys by size, and drop non-text blocks - #55
Merged
Merged
Conversation
…-text blocks
maxMissKeys bounds how MANY keys the ledger holds, not how big they are, and
selectorKey runs on whatever the tool returned. Replaying multimodal traffic
filled the top slots with base64 image payloads:
[{"type":"image","source":{"type":"base64","data":"iVBOR...
Two consequences. The ledger exists to answer 'which filter is worth writing
next' (after rtk's parse_failures table), and 200 near-identical image blobs
answer nothing while crowding out the actionable misses. And they sit in the
aggregator under its lock and ship in every /stats scrape.
A selector is a SHAPE, so a 120-byte prefix identifies it; the tail only makes
near-identical misses occupy separate slots. Truncation cuts on a rune boundary
so a key stays valid UTF-8 in the JSON payload.
Non-text blocks are dropped rather than truncated: an image carries no output
shape a filter could ever match, so recording it is noise by construction, not a
key that happens to be too long. A truncated blob would still be a useless entry.
The test asserts the bound, rune-safety, that non-text blocks are dropped, AND
that real command banners survive intact -- the bound must not cost the signal it
exists to rank.
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #48. Found by the integration review, replaying real captured traffic through merged
main.Problem
maxMissKeys = 200bounds how many keys the ledger holds, not how big they are — andselectorKeyruns on whatever the tool returned. Replayingcapture-tb.jsonlfilled the top 25 slots with base64 image payloads:Two consequences:
parse_failurestable). On multimodal traffic that answer is 200 near-identical image blobs, crowding out the actionable misses — the real ones observed wereExit code 1(211×) and source listings./statsscrape.Fix
Bound the key at 120 bytes. A selector is a shape, so a short prefix identifies it; the tail only makes near-identical misses occupy separate slots. Well above any real command banner, well below a payload. Truncation cuts on a rune boundary so a key stays valid UTF-8 in the JSON payload.
Drop non-text blocks entirely rather than truncating them. An image carries no output shape a filter could ever match, so recording it is noise by construction — not a key that merely happens to be too long. A truncated blob would still be a useless entry occupying a slot.
Test
TestMissLedgerKeysAreBoundedAndTextOnlyasserts four things, and the fourth is the one that matters most:docker build …survives the cut);Reading package lists...,> Task :app:compileDebugKotlin,make[1]: Entering directory '/src'. The bound must not cost the signal it exists to rank, and a test that only checked the truncation would happily pass a version that mangled every real selector.Verified non-vacuous — removing the non-text guard reproduces the reported symptom exactly:
Gates
go build -tags cg_skeleton ./...·go test -tags cg_skeleton ./...·gofmt -l— all clean.Remaining integration findings
#45 (the volatile-tail split is a silent no-op on Bedrock Converse — it reports
Changed: trueand achieves nothing) and #47 (pin-budget exhaustion reopening theTailOnlyfail-open, on the/compactroute only) are still open.