Skip to content

fix(script): keep comma-grouped numbers whole in hyphenated words - #395

Open
rahul05ranjan wants to merge 1 commit into
hypit-ai:mainfrom
rahul05ranjan:fix/comma-grouped-number-hyphen
Open

rahul05ranjan wants to merge 1 commit into
hypit-ai:mainfrom
rahul05ranjan:fix/comma-grouped-number-hyphen

Conversation

@rahul05ranjan

Copy link
Copy Markdown

What

Prose analysis split a comma-grouped number at the comma when a hyphenated word followed: a 1,000-word essay became 1, + 000-word (two caption/speech words), and an inner Cue break a 1,||000-word essay was accepted. Plain 2020-word and standalone 1,000 were already correct.

Why

In LEXICAL_UNIT, the number alternative matched 1,000 but its trailing lookahead (?!\p{N}|-[\p{L}\p{M}]) failed before -word. The comma-group alternative then died entirely, so the bare \p{N}+ form matched just 1 and the word alternative re-matched 000-word.

Fix

packages/script/src/lexical.ts — split the number alternative into two:

  • a comma-grouped number now keeps a trailing hyphenated word suffix in the same unit (mirroring what the word alternative already does for 2020-word), so 1,000-word, 1,000-word, and 123,456,789-point are one unit;
  • bare numbers keep the existing lookahead, so 2020-word still falls through to the word alternative unchanged.

Standalone 1,000, 1,234.56, and list-like comma behavior are untouched.

Tests

Added to packages/script/test/script.test.ts (tokens + caption words for ASCII comma, full-width comma, and multi-group cases) and packages/script/test/prose-boundaries.test.ts (1,||000-word now throws SCRIPT_CAPTION_BREAK_TOKEN_BOUNDARY like 2020-||word). Both files pass via node --test (23/23); both new cases fail on unfixed main.

Fixes #391

The number alternative in LEXICAL_UNIT matched a comma-grouped number but failed its lookahead when a hyphenated word followed, so "1,000-word" backtracked to the bare number "1" and split at the comma into "1," + "000-word". Split the alternative so a matched comma group keeps a trailing hyphenated word suffix in the same unit (mirroring what the word alternative already does for "2020-word"), while bare numbers still defer letter suffixes to the word alternative. Standalone "1,000" and plain "2020-word" are unchanged.

Fixes hypit-ai#391

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant