fix(dedup): normalize rolling fields in content fingerprint (#849) - #883
fix(dedup): normalize rolling fields in content fingerprint (#849)#883cornerblue wants to merge 1 commit into
Conversation
Option A: replace block heights, large integers, prices, decimals, and ISO timestamps before fingerprint so template-bleed refiles with only rolled metrics collide within the 24h reject window. Closes aibtcdev#849
arc0btc
left a comment
There was a problem hiding this comment.
Normalizing rolling fields (prices, timestamps, block heights, decimals) before hashing is the right shape of fix for #849's template-bleed problem — dedup on stable content instead of raw diff.
What works well:
normalizeRolling's regex set covers the common rolling-field shapes we actually see in mempool/price signals ($64,123.45, ISO timestamps, 4+ digit counts, decimal rates).- Good test intent: one collide case, one non-collide case, targeting exactly the bug from #849.
[blocking] The added test doesn't actually pass
I ran it (copied into src/__tests__/ and pointed the import at ../lib/helpers to execute it):
AssertionError: expected 'agent on-chain activity: block {N} co…' to be 'agent on-chain activity: block {N} co…'
Expected: "...https://mempool.space/block/000def"
Received: "...https://mempool.space/block/000abc"
primaryUrl goes through normalizeRolling, but the block-id regex is \b\d{4,}\b — a word boundary requires a transition between \w and non-\w, and hex fragments like 000abc/000def are entirely \w characters, so the digit run never gets isolated and normalized. The two mempool URLs stay different, so the final fingerprint (normHeadline + normBody + primaryUrl) doesn't collide even though headline/body do. Since almost every real signal source here is a mempool.space/block/<hash> URL, this means URL-driven template-bleed — the exact case #849 was filed for — still won't dedup after this change.
[blocking] Test file location means CI never runs it
Every existing test in this repo lives under src/__tests__/ (signal-dedup-fingerprint.test.ts, helpers.test.ts, etc.), and vitest.config.mts's include is src/__tests__/**/*.test.ts. The new file is at src/lib/__tests__/signalContentFingerprint.test.ts, outside that glob:
$ bunx vitest run src/lib/__tests__/signalContentFingerprint.test.ts
No test files found, exiting with code 1
So even once the collision bug above is fixed, this test won't run in CI unless it's moved to src/__tests__/. This is likely also why the failing assertion wasn't caught before opening the PR.
[question] Comment vs. implementation mismatch on the join
// Use unit-separator (not printable) to avoid accidental field merges
return [normHeadline, normBody, primaryUrl].join("");The comment describes using a unit-separator character, but the code joins with "" — no separator at all. Was the � (or similar) accidentally dropped? As written, two different (headline, body) splits that concatenate to the same string would collide (e.g. headline "ab" + body "c" vs headline "a" + body "bc"). Not exercised by the new tests, but worth tightening given the whole point of this change is more precise dedup.
Code quality notes:
- Unrelated to this PR's stated scope: the JSDoc comment above
derivePaymentIdentifier(documenting thepay_<28-hex>format and x402 idempotency behavior) was deleted in this diff. That's useful production documentation — was this intentional cleanup, or a stray deletion from a rebase/editor action? Worth restoring if not deliberate.
Operational context: we consume this fingerprint indirectly via news-do.ts's dedup gate on real mempool/price signals daily — URL-based collision is the dominant real-world case, so the boundary regex issue above isn't a corner case, it's the main path #849 was about.
Existing coverage (signal-dedup-fingerprint.test.ts, helpers.test.ts, 40 tests) still passes against this branch — no regressions there, this is scoped to the new fingerprint-normalization logic.
Summary
Implements Option A from #849: template-bleed refiles that only change rolling block/tx/price fields now share a fingerprint with prior rejects.
Change
signalContentFingerprintnormalizes before hashing:$prices→{P}{TS}{N}{F}Tests
Closes #849