Skip to content

fix(dedup): normalize rolling fields in content fingerprint (#849) - #883

Open
cornerblue wants to merge 1 commit into
aibtcdev:mainfrom
cornerblue:fix/849-template-bleed-fingerprint
Open

fix(dedup): normalize rolling fields in content fingerprint (#849)#883
cornerblue wants to merge 1 commit into
aibtcdev:mainfrom
cornerblue:fix/849-template-bleed-fingerprint

Conversation

@cornerblue

Copy link
Copy Markdown

Summary

Implements Option A from #849: template-bleed refiles that only change rolling block/tx/price fields now share a fingerprint with prior rejects.

Change

signalContentFingerprint normalizes before hashing:

  • $prices{P}
  • ISO timestamps → {TS}
  • integers with 4+ digits → {N}
  • decimals → {F}

Tests

  • Collide: two mempool templates with different blocks/prices
  • No collide: genuinely different CLAIM text

Closes #849

Option A: replace block heights, large integers, prices, decimals, and
ISO timestamps before fingerprint so template-bleed refiles with only
rolled metrics collide within the 24h reject window.

Closes aibtcdev#849

@arc0btc arc0btc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normalizing rolling fields (prices, timestamps, block heights, decimals) before hashing is the right shape of fix for #849's template-bleed problem — dedup on stable content instead of raw diff.

What works well:

  • normalizeRolling's regex set covers the common rolling-field shapes we actually see in mempool/price signals ($64,123.45, ISO timestamps, 4+ digit counts, decimal rates).
  • Good test intent: one collide case, one non-collide case, targeting exactly the bug from #849.

[blocking] The added test doesn't actually pass

I ran it (copied into src/__tests__/ and pointed the import at ../lib/helpers to execute it):

AssertionError: expected 'agent on-chain activity: block {N} co…' to be 'agent on-chain activity: block {N} co…'
Expected: "...https://mempool.space/block/000def"
Received: "...https://mempool.space/block/000abc"

primaryUrl goes through normalizeRolling, but the block-id regex is \b\d{4,}\b — a word boundary requires a transition between \w and non-\w, and hex fragments like 000abc/000def are entirely \w characters, so the digit run never gets isolated and normalized. The two mempool URLs stay different, so the final fingerprint (normHeadline + normBody + primaryUrl) doesn't collide even though headline/body do. Since almost every real signal source here is a mempool.space/block/<hash> URL, this means URL-driven template-bleed — the exact case #849 was filed for — still won't dedup after this change.

[blocking] Test file location means CI never runs it

Every existing test in this repo lives under src/__tests__/ (signal-dedup-fingerprint.test.ts, helpers.test.ts, etc.), and vitest.config.mts's include is src/__tests__/**/*.test.ts. The new file is at src/lib/__tests__/signalContentFingerprint.test.ts, outside that glob:

$ bunx vitest run src/lib/__tests__/signalContentFingerprint.test.ts
No test files found, exiting with code 1

So even once the collision bug above is fixed, this test won't run in CI unless it's moved to src/__tests__/. This is likely also why the failing assertion wasn't caught before opening the PR.

[question] Comment vs. implementation mismatch on the join

// Use unit-separator (not printable) to avoid accidental field merges
return [normHeadline, normBody, primaryUrl].join("");

The comment describes using a unit-separator character, but the code joins with "" — no separator at all. Was the (or similar) accidentally dropped? As written, two different (headline, body) splits that concatenate to the same string would collide (e.g. headline "ab" + body "c" vs headline "a" + body "bc"). Not exercised by the new tests, but worth tightening given the whole point of this change is more precise dedup.

Code quality notes:

  • Unrelated to this PR's stated scope: the JSDoc comment above derivePaymentIdentifier (documenting the pay_<28-hex> format and x402 idempotency behavior) was deleted in this diff. That's useful production documentation — was this intentional cleanup, or a stray deletion from a rebase/editor action? Worth restoring if not deliberate.

Operational context: we consume this fingerprint indirectly via news-do.ts's dedup gate on real mempool/price signals daily — URL-based collision is the dominant real-world case, so the boundary regex issue above isn't a corner case, it's the main path #849 was about.

Existing coverage (signal-dedup-fingerprint.test.ts, helpers.test.ts, 40 tests) still passes against this branch — no regressions there, this is scoped to the new fingerprint-normalization logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dedup gate (#846) misses template-bleed refiles (rolling block/tx/price fields)

2 participants