diff --git a/CHANGELOG.md b/CHANGELOG.md index 9decd2d..cc09ca4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,7 +30,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Control characters and bidi/format characters are stripped during normalization, so they can no longer survive into name fields and downstream exports. - Nickname delimiters containing a comma, NUL, whitespace, or control characters are ignored instead of silently corrupting the structural-comma split. - The token budget counts the same separator class the parser splits on, and the uniform-uppercase scan is capped, so hostile separator-heavy rows stay within the documented 65,536-token cost. -- The structural-comma shield degrades to a bounded byte-level mask past 4 KB instead of switching off, keeping surname/given assignment stable on long rows. +- Single-byte nickname delimiter configurations use a bounded byte-level structural-comma mask without the 4,096-byte cutoff. Configurations containing any valid multi-character or multibyte delimiter still stop shielding nickname commas above 4,096 bytes; see the README parsing limits. - `Smith, Jr` reads like `Smith, Junior` (Lastname plus given name) instead of a bare suffix with no given name. - `Parser` internals are split into `StructuralCommaSplitter`, `CommaCredentialTail`, and `SegmentParserFactory` with `TokenCredentialClass` replacing the credential classifier ints; `Name` dispatches on `::class` constants. No behavior change. diff --git a/README.md b/README.md index 74c4b17..70c6c2a 100644 --- a/README.md +++ b/README.md @@ -330,6 +330,15 @@ instead of tokenizing the source again. Explicitly supplied token arrays are also limited to 65,536 entries and 1,048,576 aggregate token bytes; supplying tokens does not bypass validation of the original string. +Custom nickname delimiters have a separate comma-splitting limit. If any valid +configured opener or closer is longer than one byte (including ASCII `<<` / `>>` +and Unicode `«` / `»`), nickname commas are not shielded when the normalized +input exceeds 4,096 bytes. This applies even when that longer pair is unused in +the input. A comma inside a nickname can then become a surname/given-name +separator and change the parsed fields. For long rows with nickname commas, +use only single-byte delimiter pairs, such as the defaults; their byte-level +scan has no 4,096-byte cutoff. The overall input limits still apply. + ### Ambiguous inputs Some inputs have no structural signal. A comma followed only by credentials can diff --git a/src/StructuralCommaSplitter.php b/src/StructuralCommaSplitter.php index c3d3561..c0de13b 100644 --- a/src/StructuralCommaSplitter.php +++ b/src/StructuralCommaSplitter.php @@ -96,7 +96,8 @@ public static function mask(string $name, array $nicknameDelimiters): string return self::maskAscii($name, $pairs, $symmetric); } - // Bound character-array allocation for multibyte delimiters; longer input is unshielded. + // Any multi-character or multibyte endpoint selects this bounded character scan. + // Above 4096 bytes, even commas inside single-byte pairs in a mixed config are unshielded. if (strlen($name) > 4096) { return $name; } diff --git a/tests/CommaSegmentTest.php b/tests/CommaSegmentTest.php index 9cc6b25..6069471 100644 --- a/tests/CommaSegmentTest.php +++ b/tests/CommaSegmentTest.php @@ -353,7 +353,7 @@ public function testSurnameSegmentBindsPastMidSegmentNickname(): void /** * Use multibyte delimiters to exercise the 4096-byte shielding limit; - * the ASCII scan has no length cap. + * the scan for exclusively single-byte delimiter pairs has no such cutoff. * * @return array */