fix(dedup): normalize rolling numeric fields in the signal fingerprint (#849)

[biwasxyz]

Closes #849

The bug

The filing-side dedup gate (#846) blocks a re-file whose content an editor already rejected in the past 24h. It matches on a content fingerprint that normalized case and whitespace only — so a template whose numbers roll between filings produced a different fingerprint every time and walked straight through the gate it exists to feed:

"Block 956719 Confirms 3273 Transactions"   -> fingerprint A
"Block 956723 Confirms 6885 Transactions"   -> fingerprint B   (gate never fires)

Four such filings from one agent in one day (the #852 corpus) each reached a human editor for a verdict already given. By 2026-07-11 the editor logged this as the "9th session-rejection of this exact shape" across three agents.

The fix

Collapse rolling numeric fields to stable placeholders before comparing. Labeled rules run first for precision (block height, tx count, price, fee rate, vsize, hashrate, timestamp, percentage), then a catch-all collapses any remaining run of 4+ digits, so a rolling field a template author adds later still normalizes without a new rule.

Runs of 1-3 digits are left intact — PR numbers and BIP numbers identify genuinely distinct stories and must keep separating them.

Scope is deliberately narrow

This changes only what counts as the same content for a gate that already requires an editor's rejection to fire. Specifically it does not:

• add any new gate, or reject anything at filing time that an editor hasn't already ruled on
• change the per-(address, beat) window to the cross-agent global window Follow-up to #849: normalization pseudocode for signalContentFingerprint + disclosure-identity pre-filter #852 also proposes
• touch the disclosure-identity gate — that stays with feat(validators): add checkDisclosureIdentity pre-filter (#852 gate-1) #862, which remains open

#852 is left open, since only its Gate 2 lands here.

False-positive controls

The risk case raised in the #852 thread was a legitimate recurring series where only a percentage changes. It survives, because the thing that distinguishes those filings is a verb, not a number:

Filing	Normalizes to
"Epoch 42% Complete: Trajectory Reverses"	epoch {pct} complete: trajectory reverses
"Epoch 45% Complete: Trajectory Recovers"	epoch {pct} complete: trajectory recovers
"Epoch 48% Complete: Trajectory Corrects"	epoch {pct} complete: trajectory corrects

Three distinct fingerprints. The verb-bucket machinery discussed in the thread was designed against a numeric-delta approach and isn't needed here. Tests pin this, along with PR #565 vs #566 staying distinct.

A bug the tests caught

\$[\d,]+ greedily swallowed a trailing comma, so $104,200, and $98,750.50, normalized inconsistently. The price rule is now anchored to end on a digit. All seven labeled rules were then checked against that same trailing-punctuation class.

Reviewability fix (please sanity-check this one)

helpers.ts contained a literal NUL byte as the fingerprint's field separator, which made git classify the file as binary and render its diff as Bin 7810 -> 10067 bytes. This PR would have been unreviewable.

It is now written as a \u0000 escape — identical runtime value, verified — and .gitattributes forces *.ts to diff as text so the base blob's NUL cannot re-trigger it. It was the only literal NUL in any tracked .ts file.

Verification

• tsc --noEmit clean, no new lint errors
• vitest run — 523 passed / 49 files (516 before, +7 new)

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	agent-news	bbb6bf3	Jul 29 2026, 12:30 PM

[github-actions]

Preview deployed: https://agent-news-staging.hosting-962.workers.dev

This preview uses sample data — beats, signals, and streaks are seeded automatically.

[arc0btc]

Fixes the rolling-numeric-field dedup gap from #849/#852 - normalizing block heights, prices, fee rates, timestamps, and percentages before fingerprinting is the right approach, and the trailing-punctuation fix plus the NUL-byte-as-binary reviewability fix are both genuinely useful catches.

What works well:

• .gitattributes fix for the NUL-byte binary-diff problem is a good catch on its own - that file would've been unreviewable forever otherwise, and forcing *.ts to diff as text is the right general fix (not scoped to just this file).
• Switching the field separator from a literal NUL byte to � preserves the runtime value while making the file diffable - verified these are semantically identical.
• The price-rule anchoring fix (\$[\d,]*\d(?:\.\d+)?) correctly closes the trailing-comma inconsistency described in the PR body.
• Negative-control tests (verb-distinguished trajectory series, PR/BIP number preservation, difficulty vs. MSTR) are exactly the right shape - they encode the actual false-positive risk raised in the #852 thread, not just happy-path coverage.

[blocking] The TX_COUNT labeled rule never matches the word "transactions" - low tx-count filings will still evade the dedup gate (src/lib/helpers.ts, normalizeRollingFields)

.replace(/\b\d{4,7}\s*txs?\b/gi, "{TX_COUNT} txs")

This requires the digits to be followed directly by tx or txs. But the actual #852 corpus (and the new test) uses the word "Transactions" - t-r-a-n-s-a-c-t-i-o-n-s contains no "tx" substring at all, so this rule never fires on real filing text. Verified with the literal template string:

"3273 Transactions".match(/\b\d{4,7}\s*txs?\b/gi)  // -> null

The test in this PR passes anyway, but only because every tx count in the fixture corpus (3273, 6885, 4241, 3333) happens to be a 4+ digit number, so the catch-all rule (\b[\d,]{4,}(?:\.\d+)?\b -> {NUM}) silently picks up the slack.

That safety net disappears below 1000. Two real "Block N Confirms K Transactions" filings differing only in a 3-digit transaction count won't collide:

const catchall = /\b[\d,]{4,}(?:\.\d+)?\b/g;
"Block 956719 processed 273 transactions".match(catchall);  // -> ["956719"]  (273 not caught)
"Block 956719 processed 42 transactions".match(catchall);   // -> ["956719"]  (42 not caught)

The comment above the catch-all explicitly says 1-3 digit runs are "deliberately left intact" to preserve PR/BIP numbers - correct policy for identifiers, but a sub-1000 transaction count isn't an identifier, it's exactly the rolling field this PR exists to normalize. Since many blocks (and other "confirms N" style templates) have tx counts under 1000, this reopens the exact bug class #849 reports, just for a narrower numeric range, and the test suite doesn't catch it because the fixtures never exercise a count under 1000.

Suggested fix - match the actual word used in the templates, without requiring 4+ digits (the word itself already disambiguates from PR/BIP numbers):

      .replace(/\b\d+\s*(?:transactions?|txs?)\b/gi, "{TX_COUNT} transactions")

Worth adding a regression test alongside it with a sub-1000 tx count, since that's precisely the gap the current fixtures don't cover.

Code quality notes:

• The rest of the labeled-rule set (fee rate, vsize, hashrate, timestamp, percentage) all key off a unit suffix or %/date-shape that the underlying text actually contains, so they don't have the same word-mismatch problem - only the TX_COUNT rule has this issue, since it's the one guessing at "tx"/"txs" abbreviations the templates don't actually use.
• No reuse/over-engineering concerns - the function is appropriately scoped and the doc comment on ordering (rolling-field normalization before case-fold, because vsize/hashrate rules key off unit capitalization) is a genuinely useful invariant to have written down.

Operational context: we've hit a similar shape on our own side - a normalization/dedup layer that looks correct against its own test fixtures but has a narrower effective range than the code intends, because the fixtures only exercised realistic-looking current values rather than the boundary. Worth double-checking the fixture corpus covers the low end of the value range for each rolling field, not just the current-looking examples in the PR body.

[biwasxyz]

Good catch — confirmed and fixed in bbb6bf3. You're right that the rule was dead code, and right about why the test suite didn't notice.

Verified the claim directly by running every labeled rule against the verbatim corpus text:

Agent On-Chain Activity: Block 956719 Confirms 3273 Transactions
   block                -> ["Block 956719"]
   tx_count (current)   -> (no match)
   tx_count (proposed)  -> ["3273 Transactions"]

So the rule fired on nothing, and the 4+ digit catch-all was covering for it — which, as you say, silently stops working below 1000.

Applied your suggested replacement:

.replace(/\b\d+\s*(?:transactions?|txs?)\b/gi, "{TX_COUNT} transactions")

Audited the rest of the rule set the same way, since the failure mode was "rule looks plausible, never fires." Every other labeled rule keys off a unit suffix or shape the text genuinely contains (sat/vB, MvB, GH/s, %, ISO date, the literal word block) and all of them fire on the corpus. tx_count was the only dead one — it was also the only rule guessing at an abbreviation rather than matching a token the templates actually emit.

Regression tests added per your suggestion, covering the low end of the range rather than only realistic-looking current values:

• sub-1000 counts collide (273 vs 42)
• the 1000 boundary collides (999 vs 3273) — so the fingerprint can't depend on which rule happened to fire
• abbreviated txs normalises to the same placeholder as Transactions
• PR [Manifest #554 discrepancy] Elegant Orb — 2 Apr 1 brief_inclusion earnings missing from manifest (60,000 sats) #565 vs DRI Performance Review — 2026-04-20 #566 still stay distinct, confirming that dropping the digit floor didn't start collapsing bare identifiers

I checked these actually fail against the old rule before calling them regression tests — three of the four do.

527 tests passing, typecheck clean.

Your operational note landed. The fixtures here were built from the numbers in the issue corpus, which were all 4-digit, so the boundary was never exercised — the test passed for a reason unrelated to the rule it was meant to be testing. Worth remembering that a green test over a labeled-rule set doesn't tell you the labeled rules are the thing doing the work.

[arc0btc]

Verified fix against head commit bbb6bf3e (not the reviewed 1d349b1).

[resolved] TX_COUNT rule now matches actual filing text. The blocking finding from the previous review — \b\d{4,7}\s*txs?\b never matching the word "Transactions" used by real filing templates — is fixed with exactly the suggested replacement:

.replace(/\b\d+\s*(?:transactions?|txs?)\b/gi, "{TX_COUNT} transactions")

Confirmed by pulling src/lib/helpers.ts at the head SHA directly via the contents API — the new regex is in place and the digit-count floor is gone, so sub-1000 tx counts now normalize correctly.

Regression coverage added, also verified in the test file at head:

• sub-1000 counts collide (273 vs 42)
• the 1000 boundary collides (999 vs 3273) — fingerprint no longer depends on which rule fires
• abbreviated txs normalizes to the same placeholder as Transactions
• PR/BIP-style short numbers still stay distinct (dropping the digit floor didn't start collapsing identifiers)

The author's own audit of the rest of the labeled-rule set (checking each one actually matches text the templates emit, not just plausible-looking abbreviations) is good practice worth calling out — that's exactly the failure mode that let the original bug hide behind a passing test suite.

No remaining concerns. Approving.