Fix same-line rank/PIB corruption; consistent surname stems; ending counts toward the half-limit (v3.0.14–3.0.16) - #56
Merged
Conversation
…her (v3.0.14)
Masks were substituted into the partially masked line with
str.replace(original, mask, 1); when the mask of one rank contained the
form of another rank on the same line, the second substitution hit the
freshly inserted mask ("рядового ... солдата" -> "старшого рядового",
second rank left unmasked, unmask restored the wrong rank). Replacements
are now numbered placeholders in the working copy of the line and are
substituted once at the end.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XT6iUWaQgahXDB9TWX9Bq7
Generated-With: Claude Code 2.1.42
….15) МАЗУРЕНКА / Мазуренка / Мазуренко (one person in upper case, title case and different grammatical cases) got three unrelated masks because the seed was taken from the surface form. All forms now share one synthetic stem; ending and letter case are applied on top. Unmask is mapping-driven and unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XT6iUWaQgahXDB9TWX9Bq7 Generated-With: Claude Code 2.1.42
…rname" limit (v3.0.16) Only the prefix was capped at half of the word, so with a long ending most of the original stayed visible (Мазуренка -> Мазиденка kept 7 of 9 letters). Prefix + preserved ending are now limited to half of the base form of the surname; the budget is computed from the base form so all grammatical cases keep the same prefix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XT6iUWaQgahXDB9TWX9Bq7 Generated-With: Claude Code 2.1.42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three fixes driven by a real counterexample:
v3.0.14 — engine: several ranks / PIBs on one line no longer corrupt each other
Masks were substituted into the partially masked line with
str.replace(original, mask, 1). When the mask of one rank contained the form of another rank on the same line (рядовий → старший солдат,солдат → рядовий), the second substitution hit the freshly inserted mask:рядового МАЗУРЕНКА та солдата КОВАЛЕНКА→старшого рядового МАЗИДЕНКА та солдата КОВИЛЕНКА(non-existent rank, second rank left unmasked, unmask restored the wrong rank). Replacements are now numbered placeholders in the working copy of the line, substituted once at the end. Same fix for full-name replacements. New tests:tests/test_same_line_replacement.py.v3.0.15 — surname: one person, one synthetic stem
The synthetic stem was seeded from the surface form, so
МАЗУРЕНКА/Мазуренка/Мазуренкоgot three unrelated masks. The seed is now the lower-cased stem; all letter-case and grammatical-case forms share one stem, and ending + case are applied on top. Unmask is mapping-driven and unaffected.v3.0.16 — surname prefix: the preserved ending counts toward "at most half"
Only the prefix was capped at half the word, so with a long ending most of the original stayed visible (
Мазуренка → Мазиденка: 7 of 9 letters). Prefix + preserved ending are now limited to half of the base form:Коваль → 3,Ґудзь → 2,Іванов → 1,Бондаренко → 1,Петренко → 0. Computed from the base form so all cases of one surname keep the same prefix. Docs, config template and tests updated.Test plan
pytest tests/— 810 passed, 1 skippedmypy datamasking/— 0 errorsflake8 --select=E9,F63,F7,F82старшого солдата … та рядового …and unmasks to the original🤖 Generated with Claude Code
https://claude.ai/code/session_01XT6iUWaQgahXDB9TWX9Bq7
Generated by Claude Code