Skip to content

capitalize() breaks a decomposed (NFD) word at its combining mark: josé garcía typed NFD gives José GarcíA #542

Description

@derek73

Case repair splits a word at a combining mark, so a name typed in decomposed form (NFD, the normalization macOS file names and some databases hand back) is repaired as if the letters after the accent were a second word.

>>> import unicodedata
>>> HumanName(unicodedata.normalize("NFD", "josé garcía")).capitalize()   # then str()
'José GarcíA'
>>> HumanName("josé garcía").capitalize()
'José García'
>>> parse(unicodedata.normalize("NFD", "renée o’néill")).capitalized()
'RenéE O’NéIll'

(Outputs shown NFC-composed for legibility; the repaired text keeps the input's form.)

Every release does this, 1.4.0 through 2.3.0 and master, plain and forced, on both surfaces. The parse itself is right: roles and tags for the NFD spelling equal the NFC ones, because _lexicon._normalize NFC-composes before every vocabulary lookup. Only the render view is wrong.

Cause. nameparser/_render.py:27: _WORD = re.compile(r"(\w|\.)+") is the sub-word splitter _cap_text runs per token. A combining mark is Unicode category Mn, which \w does not match, so garcía is the words garci and a with a mark between them, and a capitalized alone is A. The same split is why a mask misses an NFD word: pé.x. under a caller's péx → Péx mask gives Pé.X. composed and PÉ.X. decomposed, the pe half finding no key and taking the acronym clause. capitalized()'s docstring already records the output-side twin (ǰ upper-cases to J plus a combining caron, which a second pass reads as two words); this is the input side of the same limit, and it is reached by ordinary Latin names rather than by a Unicode corner.

Reach. No corpus or case-row name is written NFD (the only combining marks in tools/differential/*.jsonl and tests/v2/cases.py are the Bengali vowel signs of one caseless row and an emoji modifier), so the differential gate cannot see it, and capitalized() is not a compared surface in any case (decisions.md#R4).

Options:

  1. Let the splitter carry marks. Python's re has no \p{M}, so either an explicit class of the combining blocks or a splitter that asks unicodedata.category(ch)[0] == "M". The output keeps the input's form, so rules.md#R4's "Repair changes case and nothing else" holds as tests/v2/test_properties.py::test_case_repair_changes_case_and_nothing_else measures it (a casefold comparison, which does not normalize). Recommended.
  2. NFC-compose before repair. Fixes the split but changes the output's normalization form, which that property test would report as a change beyond case, and R4's statement would need a carve-out.

Either way a rules.md#R4 example line in NFD and a property-test row pin it; _apply_mask and _letter_run_ge2 already read a mark as a non-letter consistently and need no change for the mask case once the word reaches them whole.

Noted in #539's description as a follow-up candidate; measured 2026-09-24.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions