[talwgx]

A compact cap file keyed on templateHash now matches: CompactObject.shouldEncode passes the joined key to TenXLookup.get as a string. The lookup hashes a key's characters, but a templateHash field value carries the template's numeric hash and was looked up by that, so with fieldNames: [templateHash] no event ever matched (0 of 662 compact with all 96 template hashes listed). A message_pattern key is already a string, so cap files keyed on it behave as before.

What changed

• pipelines/run/receive/compact/compact-object-cap.js: TenXLookup.get("compactReceiverLookupFile", "" + fieldSetKey). The cap-file format, the true|false[:untilEpochSec[:reason]] values, row expiry, compactReceiverDefault and receiverOptimize are unchanged.
• pipelines/run/receive/compact/config.yaml: a comment on keying by templateHash and quoting each hash in the cap file (a hash can start with #, the CSV comment character, or hold a space; the hash alphabet has no " or ,).

Checked

Engine 1.1.153 (jar) on a Windows Security XML sample (662 events, 96 templates), with the Splunk settings (no back-references, UTC, one timestamp per event) and the forwarder output's field encoded=shouldEncode() ? encode() : fullText("tenx_passthrough"); each event's template known from the run with every hash listed.

cap file	compact	written whole
templateHash, all 96 hashes, main's script	0	662
templateHash, all 96 hashes	662	0
templateHash, every second hash (48)	264, exactly the events of the 48 listed templates	398, each equal to its input event
templateHash, all 96, rows expired an hour ago	0	662
templateHash, all 96, rows that expire in a day	662	0
message_pattern, one row	output byte-identical to main

On the EventID 4663 sample repeated twice (15,298 events, 1,783 templates, 224 of the hashes needing quotes), every hash listed: 15,298 of 15,298 compact.

Time per event: whole runs on the two samples (662 events repeated 30 times, 5 rounds per arm; 4663 repeated twice, 3 rounds), with no cap entry matching (main's script and key), with this change, and with a one-column list tested by lookupMatch (the other way to key on the template, which has no row expiry):

input	arm	wall time, median (range)	CPU time, median (range)
19,860 events, 96 templates	no entry matching	18.6 s (13.9 to 24.0)	31.5 s (29.5 to 35.5)
	this change, every hash listed	13.5 s (12.2 to 16.9)	36.2 s (27.3 to 37.1)
	lookupMatch, every hash listed	15.5 s (10.6 to 21.6)	34.3 s (31.8 to 34.9)
15,298 events, 1,783 templates	no entry matching	30.5 s (22.8 to 32.7)	68.3 s (62.0 to 71.7)
	this change, every hash listed	33.4 s (26.5 to 55.8)	67.2 s (62.6 to 78.0)
	lookupMatch, every hash listed	33.3 s (22.6 to 37.9)	68.2 s (61.6 to 74.8)

The spread within an arm is larger than any difference between arms, so no cost per event was measurable on this machine at these sizes. The lookup keeps 100 rows in memory and reads a listed key's row from the file on a miss (BaseFileCachedLookup, TextFileLookup.readLine); with 1,783 listed templates most lookups take that path, and it did not show either.

node --test tests/: 15 of 15.

Rate receiver: the same key in its mute lookup

The rate receiver's mute file had the same defect: rate-object-lookup.js and rate-object-lookup-cap.js pass fieldSetKey from joinFields to TenXLookup.get, so with rateReceiver.fieldNames: [templateHash] no mute entry ever matched. The second commit passes it as a string the same way.

Checked with the engine (1.1.153 jar), Windows Security sample of 662 events and 96 templates, the rate receiver keyed on templateHash (fieldNames: [templateHash] in its config), output filtered on !isRoute("drop"), each case once with no cap file (rate-object-lookup.js) and once with a cap file of one * row far above the data (rate-object-lookup-cap.js), same counts in both:

mute file	main: events written	this change
every template hash, 0:<tomorrow>	662	0
every second hash, 0:<tomorrow>	662	398, exactly the events of the 48 unlisted templates, in order
every hash, entry expired an hour ago	662	662, byte-identical
message_pattern key (the default), 0:<tomorrow>	0	0, byte-identical
no mute file	662	662, byte-identical

node --test tests/: 15 of 15.

🤖 Generated with Claude Code