Skip to content

Commit 165c3db

Browse files
authored
fix(library): attribute sim counts by clause and table row, not capitalized words (#10115)
* fix(library): attribute sim counts by clause and table row, not capitalized words * fix(library): handle colon lists, parenthetical comparisons, relative clauses, prefixes, and linked names in sim-count * fix(library): keep sim relative clauses attached, anchor parenthetical comparisons to sim, and detect pipe-less table rows * fix(library): remove sim-count comparison asides for any subject and keep counted openers * fix(library): keep sim-count comparison spans inside their sentence * refactor(library): read sim-count comparisons as their own clauses instead of deleting them * fix(library): split sim-count contrasts nested in comparisons, keep count lists together, and end sentences before closing quotes * fix(library): start sim-count clauses only at a new capitalized subject and tell count lists from new clauses * fix(library): check sim claims in feature-first table cells and treat digit product names as new subjects * docs(library): describe sim-count table cells and new-subject clauses accurately
1 parent 1b9fdd9 commit 165c3db

2 files changed

Lines changed: 190 additions & 28 deletions

File tree

‎scripts/check-library-content.test.ts‎

Lines changed: 93 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,76 @@ describe('check-library-content', () => {
330330
['Sim listens for events with 300+ triggers.', '300+ triggers', 'stale'],
331331
['Sim has **1,000+** integrations.', '1,000+ integrations', 'overstates'],
332332
['Sim has 1,000+ Integrations.', '1,000+ Integrations', 'overstates'],
333+
['Sim offers: 1,000+ integrations.', '1,000+ integrations', 'overstates'],
334+
['Sim, unlike Zapier, has 1,000+ integrations.', '1,000+ integrations', 'overstates'],
335+
['Sim, which ships 1,000+ integrations, runs agents.', '1,000+ integrations', 'overstates'],
336+
[
337+
'Sim is an AI workspace, which connects to 1,000+ integrations.',
338+
'1,000+ integrations',
339+
'overstates',
340+
],
341+
['Sim, unlike n8n, has 1,000+ integrations.', '1,000+ integrations', 'overstates'],
342+
[
343+
'Zapier lists thousands of apps, while Make lists 3,000 apps. Sim has 1,000+ integrations, and more.',
344+
'1,000+ integrations',
345+
'overstates',
346+
],
347+
[
348+
'Zapier, while Sim has 1,000+ integrations, focuses on simple automations.',
349+
'1,000+ integrations',
350+
'overstates',
351+
],
352+
[
353+
'Zapier is popular, while Sim has 1,000+ integrations, and both support webhooks.',
354+
'1,000+ integrations',
355+
'overstates',
356+
],
357+
['Sim, unlike Make.com, has 1,000+ integrations.', '1,000+ integrations', 'overstates'],
358+
['Sim connects tools like Slack to 1,000+ integrations.', '1,000+ integrations', 'overstates'],
359+
['Sim is open source but has 1,000+ integrations.', '1,000+ integrations', 'overstates'],
360+
['| Catalog | Sim has 1,000+ integrations |', '1,000+ integrations', 'overstates'],
361+
[
362+
'Sim has 280+ integrations while working across 9,000+ actions.',
363+
'9,000+ actions',
364+
'overstates',
365+
],
366+
['Sim is open source, but has 1,000+ integrations.', '1,000+ integrations', 'overstates'],
367+
[
368+
'While Sim is open source but has 1,000+ integrations, Zapier is proprietary.',
369+
'1,000+ integrations',
370+
'overstates',
371+
],
372+
[
373+
'While Sim has 280+ integrations, 9,000+ actions, and 530+ triggers, Zapier focuses on simple automations.',
374+
'9,000+ actions',
375+
'overstates',
376+
],
377+
[
378+
'While Sim has 1,000+ integrations, Zapier focuses on simple automations.',
379+
'1,000+ integrations',
380+
'overstates',
381+
],
382+
['Sim | Governed agents | 1,000+ integrations |', '1,000+ integrations', 'overstates'],
383+
[
384+
'[Sim](https://www.sim.ai) combines [Chat](https://docs.sim.ai/execution/chat) with 1,000+ integrations.',
385+
'1,000+ integrations',
386+
'overstates',
387+
],
388+
[
389+
"Sim's Agent blocks support 28 LLM providers, and the workspace connects to 1,000+ integrations.",
390+
'1,000+ integrations',
391+
'overstates',
392+
],
393+
[
394+
'| Sim | Governed agents across SaaS tools | 1,000+ integrations |',
395+
'1,000+ integrations',
396+
'overstates',
397+
],
398+
[
399+
'| [Sim](https://www.sim.ai) | Governed agents | 1,000+ integrations |',
400+
'1,000+ integrations',
401+
'overstates',
402+
],
333403
])('rejects a Sim count the catalog does not support: %s', async (body, claim, verdict) => {
334404
writePost('library', 'post', body)
335405
expect(await findingsFor('library', 'post')).toEqual([
@@ -361,10 +431,31 @@ describe('check-library-content', () => {
361431
"Zapier lists 9,000+ integrations, while Sim's 280+ integrations focus on agents.",
362432
'Compared with Sim, MindStudio advertises 1,000+ integrations.',
363433
'Sim is open source. Workato cites 14,000+ integrations.',
364-
'Sim can call any tool that the connector marketplaces built on open protocols expose, often 1,000+ integrations.',
434+
'Unlike Sim, Zapier lists 9,000+ integrations.',
435+
'Like Sim, n8n ships 2,000+ integrations.',
436+
'- Unlike Sim, Zapier lists 9,000+ integrations.',
437+
"Sim has fewer than [Zapier](https://zapier.com)'s 9,000+ integrations.",
438+
'Sim has 280+ integrations, unlike Zapier, which has 9,000+ integrations.',
439+
'Zapier, unlike Sim, has 9,000+ integrations.',
440+
'Sim is open source but Zapier has 9,000+ integrations.',
441+
'While Sim has 280+ integrations but Zapier has 9,000+ integrations, both support webhooks.',
442+
'"Sim is open source." Zapier has 9,000+ integrations.',
443+
'While Sim has 280+ integrations, 9,000+ integrations are available in Zapier.',
444+
'Sim has 280+ integrations, and Zapier has 9,000+ integrations.',
445+
'Sim has 280+ integrations, while n8n lists 2,000+ integrations.',
446+
'| Catalog | Sim has 280+ integrations | n8n lists 2,000+ integrations |',
447+
'While Sim has 280+ integrations, Zapier lists 9,000+ integrations.',
448+
'Zapier, like Sim, has 9,000+ integrations.',
449+
'Sim, unlike Zapier, which has 9,000+ integrations, is open source.',
450+
'n8n | Alternative to Sim | 2,000+ integrations |',
451+
'Sim has 280+ integrations, while Zapier lists 9,000+ integrations.',
452+
"Sim has 280+ integrations, fewer than Zapier's 9,000+ integrations.",
453+
'| n8n | Self-hosted, unlike Sim | 2,000+ integrations |',
454+
'Teams that need 20 integrations and five builders often pick Sim.',
365455
'```text\nSim has 1,000+ integrations\n```',
366456
'````md\n```text\nSim has 1,000+ integrations\n````',
367-
].join('\n\n')
457+
].join('\n\n'),
458+
{ faq: '\n - q: "Is Sim like n8n?"\n a: "Like Sim, n8n ships 2,000+ integrations."' }
368459
)
369460
expect(await findingsFor('library', 'post')).toEqual([])
370461
})

‎scripts/check-library-content.ts‎

Lines changed: 97 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,13 @@
3333
* against the generated catalog (`INTEGRATION_TOTALS`): no higher than the total, and no
3434
* lower than 90% of it. Answer engines quote these figures, so an overclaim such as
3535
* "1,000+ integrations" or a figure left from an older catalog spreads to third-party sites.
36-
* A count belongs to Sim when the nearest "Sim" before it on the line is at most 12 words back
37-
* with no capitalized word between, which leaves competitor counts alone.
36+
* A count belongs to Sim when "Sim" comes before it in the same clause and no other product's
37+
* possessive directly precedes it, or when it sits in a table row whose label cell is Sim
38+
* (other rows are read cell by cell). A comparison that opens a sentence or names a new subject
39+
* ("while Zapier", "Zapier, like Sim,") is its own clause up to the next comma, and the main
40+
* clause resumes after it; ", and" before a new subject also starts a clause.
41+
* A count whose owner is ambiguous within its clause is treated as Sim's; when it belongs to
42+
* another product, name that product as its owner ("Zapier's 9,000+") or give it a sentence.
3843
* Blog and changelog posts are dated, so their counts are true as of publication.
3944
*
4045
* Run one post with `--slug <section>/<slug>` or `--slug <slug>`.
@@ -138,35 +143,101 @@ const ROUTE_FILES = ['page.tsx', 'page.ts', 'route.tsx', 'route.ts']
138143
const COUNT_CLAIM = /\b(\d{1,3}(?:,\d{3})+|\d+)\+?\s+(integrations|actions|triggers)\b/gi
139144
/** Inline emphasis and code markers, which would otherwise split "**1,000+** integrations". */
140145
const INLINE_MARKERS = /\*\*|__|[*`]/g
141-
const SIM_NAME = /\bSim\b/g
142-
const SIM_CLAIM_MAX_WORDS = 12
146+
/** A Markdown link, read as its visible text so "[Zapier](https://zapier.com)'s" names Zapier. */
147+
const LINK_TEXT = /\[([^\]\n]*)\]\([^)\n]*\)/g
148+
/** List, heading, and quote markers, or a quoted YAML key such as `a: "`, ahead of the sentence. */
149+
const LINE_PREFIX = /^\s*(?:(?:[-*+>]|\d+[.)]|#{1,6})\s+)*(?:\w+:\s*["'])?/
150+
const SIM_NAME = /\bSim\b/
151+
/** Words that turn a sentence toward another product: "while Zapier lists ...", "unlike Zapier". */
152+
const CONTRAST = String.raw`(?:while|whereas|but|unlike|compared (?:with|to)|versus|vs\.?)`
153+
/** The same words opening a sentence, plus "Like" ("Like Sim, n8n ..."). */
154+
const OPENING_CONTRAST = String.raw`(?:While|Whereas|But|Unlike|Like|Compared (?:with|to)|Versus|Vs\.?)`
155+
/** Sentence punctuation, with any closing quote or bracket, before whitespace. */
156+
const SENTENCE_END = /(?<=[.!?;]['"”’)\]]*)\s+/
157+
/** The start of a new subject: a capitalized word, or a lowercase product name with a digit (n8n). */
158+
const SUBJECT = String.raw`(?:[A-Z]|[a-z]+\d)`
159+
/** ", and" before a new subject, which starts a clause of its own ("..., and Zapier has ..."). */
160+
const NEW_SUBJECT = new RegExp(String.raw`,\s+and\s+(?=${SUBJECT})`)
161+
/**
162+
* Where a comparison clause starts: a sentence-opening contrast ("Unlike Sim, ..."), or a contrast
163+
* followed by a new subject ("..., while Zapier", "Zapier, like Sim, ..."). A contrast that keeps
164+
* the subject ("open source but has ...", "while working") does not start one.
165+
*/
166+
const COMPARISON_START = new RegExp(
167+
String.raw`^(?=${OPENING_CONTRAST}\b)|,\s*(?=(?:like|${CONTRAST})\s+${SUBJECT})|\s+(?=${CONTRAST}\s+${SUBJECT})`
168+
)
169+
/** A count continuing a list ("..., 9,000+ actions, and 530+ triggers, ..."), not a new clause. */
170+
const COUNT_LIST_ITEM = String.raw`(?:and\s+)?\d[\d,]*\+?\s+(?:integrations|actions|triggers)\b\s*(?:,|and\b|$)`
171+
/**
172+
* Where a comparison clause ends: the first comma that is not a thousands separator, not the
173+
* start of a relative clause continuing it ("unlike Zapier, which has 9,000+ integrations"), and
174+
* not the next item in a list of counts.
175+
*/
176+
const COMPARISON_END = new RegExp(
177+
String.raw`,(?!\d)(?!\s*(?:which|that|who)\b)(?!\s*${COUNT_LIST_ITEM})`
178+
)
179+
/** A contrast with a new subject inside a comparison clause ("..., but Zapier has ..."). */
180+
const INNER_CONTRAST = new RegExp(String.raw`,?\s+(?=${CONTRAST}\s+${SUBJECT})`)
181+
/** The product whose possessive directly precedes a count, as in "Zapier's 9,000+". */
182+
const POSSESSIVE_OWNER = /([\w.-]+)['’]s\s*$/
143183
/** A Sim count below this share of the catalog total is left from an older catalog. */
144184
const SIM_COUNT_FLOOR = 0.9
145185

146186
/**
147-
* The counts on one line of text that belong to Sim, with the claim as written. A sentence or
148-
* clause that moves on to another product starts with a capitalized word, which ends Sim's claim.
187+
* The Sim counts in a table row: every count when its label (first non-empty cell) names Sim, and
188+
* otherwise each cell's own Sim claims, read like prose ("| Catalog | Sim has 280+ integrations |").
149189
*/
150-
function simCountClaims(line: string): { claim: string; count: number; unit: SimCountUnit }[] {
151-
const claims: { claim: string; count: number; unit: SimCountUnit }[] = []
152-
const text = line.replace(INLINE_MARKERS, '')
153-
for (const match of text.matchAll(COUNT_CLAIM)) {
154-
const before = text.slice(0, match.index)
155-
const lastSim = [...before.matchAll(SIM_NAME)].at(-1)
156-
if (lastSim?.index === undefined) continue
157-
const words = before
158-
.slice(lastSim.index + lastSim[0].length)
159-
.split(/\s+/)
160-
.map((word) => word.replace(/^[^A-Za-z0-9]+/, ''))
161-
.filter(Boolean)
162-
if (words.length > SIM_CLAIM_MAX_WORDS || words.some((word) => /^[A-Z]/.test(word))) continue
163-
claims.push({
164-
claim: match[0],
165-
count: Number.parseInt(match[1].replaceAll(',', ''), 10),
166-
unit: match[2].toLowerCase() as SimCountUnit,
167-
})
190+
function simTableRowCounts(cells: string[]): RegExpMatchArray[] {
191+
const label = cells.map((cell) => cell.trim()).find((cell) => cell !== '') ?? ''
192+
return SIM_NAME.test(label.split(/\s/)[0])
193+
? cells.flatMap((cell) => [...cell.matchAll(COUNT_CLAIM)])
194+
: cells.flatMap(simProseCounts)
195+
}
196+
197+
/**
198+
* A sentence's clauses: its main clause, with each comparison taken out and read as a clause of
199+
* its own, so "Sim, unlike Zapier, has ..." reads as "Sim has ..." plus "unlike Zapier".
200+
*/
201+
function sentenceClauses(sentence: string): string[] {
202+
const main: string[] = []
203+
const comparisons: string[] = []
204+
let rest = sentence
205+
for (let start = COMPARISON_START.exec(rest); start; start = COMPARISON_START.exec(rest)) {
206+
main.push(rest.slice(0, start.index))
207+
const comparison = rest.slice(start.index + start[0].length)
208+
const end = COMPARISON_END.exec(comparison)
209+
comparisons.push(...(end ? comparison.slice(0, end.index) : comparison).split(INNER_CONTRAST))
210+
rest = end ? comparison.slice(end.index + 1) : ''
168211
}
169-
return claims
212+
return [[...main, rest].join(' '), ...comparisons]
213+
}
214+
215+
/** Each count whose clause names Sim before it, unless another product's possessive owns it. */
216+
function simProseCounts(text: string): RegExpMatchArray[] {
217+
const clauses = text
218+
.replace(LINE_PREFIX, '')
219+
.split(SENTENCE_END)
220+
.flatMap((sentence) => sentence.split(NEW_SUBJECT))
221+
.flatMap(sentenceClauses)
222+
return clauses.flatMap((clause) =>
223+
[...clause.matchAll(COUNT_CLAIM)].filter((match) => {
224+
const before = clause.slice(0, match.index)
225+
const owner = POSSESSIVE_OWNER.exec(before)?.[1]
226+
return SIM_NAME.test(before) && (owner === undefined || owner === 'Sim')
227+
})
228+
)
229+
}
230+
231+
/** The counts on one line of text that belong to Sim, with the claim as written. */
232+
function simCountClaims(line: string): { claim: string; count: number; unit: SimCountUnit }[] {
233+
const text = line.replace(INLINE_MARKERS, '').replace(LINK_TEXT, '$1')
234+
const cells = text.split('|')
235+
const matches = cells.length > 2 ? simTableRowCounts(cells) : simProseCounts(text)
236+
return matches.map((match) => ({
237+
claim: match[0],
238+
count: Number.parseInt(match[1].replaceAll(',', ''), 10),
239+
unit: match[2].toLowerCase() as SimCountUnit,
240+
}))
170241
}
171242

172243
function isSection(value: string): value is Section {
@@ -458,7 +529,7 @@ export async function checkPost(
458529
count > total
459530
? `"${claim}" overstates Sim's catalog, which has ${total} ${unit}.`
460531
: `"${claim}" is stale: Sim's catalog has ${total} ${unit}.`,
461-
`Write ${formatLowerBound(total)} ${unit}, the catalog-derived figure (INTEGRATION_TOTALS in packages/deployment-config/src/integration-totals.ts).`
532+
`Write ${formatLowerBound(total)} ${unit}, the catalog-derived figure (INTEGRATION_TOTALS in packages/deployment-config/src/integration-totals.ts). If the count is another product's, name it as the owner ("Zapier's 9,000+") or give it its own sentence.`
462533
)
463534
}
464535
}

0 commit comments

Comments
 (0)