Hi Toni — following up on the ma / na / co / sa question you left for "the Polish eye" in #57. I ran the same collision audit we used for Vietnamese, with the real extractor, and it points somewhere different from removing them — so an issue rather than a PR.
What the audit found (live, language="pl")
The tokenizer keeps only letter runs on lowercased text (re.findall(r"\b[a-zA-ZÀ-ỹ]+...\b") over text.lower()), which changes the picture:
| word |
punctuated form |
bare acronym |
Polish function word (noise if removed) |
na |
N/A does NOT collide — the / splits it into n + a, both below min-length |
NA (North America) collides |
na (on/at) — one of the most frequent PL words |
sa |
S.A. does NOT collide — the . splits it into s + a |
SA collides (much rarer than S.A.) |
są/sa (are) — common |
co |
C/O does NOT collide |
CO collides |
co (what) — common |
ma |
(no punctuated form) |
MA (moving average) collides |
ma (has) — very frequent PL verb |
So the specific collisions that motivated na and sa (N/A, S.A.) don't actually happen — the punctuation fragments them below the min-length guard. Only the bare uppercase acronyms collide, and for all four the Polish function word is far more common in real content than the acronym. That's the inverse of the ci ↔ CI case, where the acronym was ubiquitous and the function word rare.
Conclusion: removing ma/na/co/sa from STOP_WORDS_PL would trade a lot of function-word noise for a niche acronym gain. I don't think it's worth it as a straight removal.
The better fix: preserve uppercase acronyms
The root cause is that extract_weighted_keywords lowercases up front (_CLAUSE_BOUNDARY.split(text.lower())), which conflates the acronym MA with the function word ma. A targeted fix would keep a token that was ALL-CAPS (≥2 chars) in the original text even when its lowercased form is a stop-word — so MA, CO, NA, SA (and every other acronym) survive, without letting the Polish function words through.
That needs the tokenizer to carry original case (today it's discarded at .lower()), so it's a bit more than a one-line stop-word edit — hence an issue first. Happy to send a PR if you like the direction.
(The per-word audit script is small and I can share it if useful.)
Robert
Hi Toni — following up on the
ma/na/co/saquestion you left for "the Polish eye" in #57. I ran the same collision audit we used for Vietnamese, with the real extractor, and it points somewhere different from removing them — so an issue rather than a PR.What the audit found (live,
language="pl")The tokenizer keeps only letter runs on lowercased text (
re.findall(r"\b[a-zA-ZÀ-ỹ]+...\b")overtext.lower()), which changes the picture:naN/Adoes NOT collide — the/splits it inton+a, both below min-lengthNA(North America) collidesna(on/at) — one of the most frequent PL wordssaS.A.does NOT collide — the.splits it intos+aSAcollides (much rarer thanS.A.)są/sa(are) — commoncoC/Odoes NOT collideCOcollidesco(what) — commonmaMA(moving average) collidesma(has) — very frequent PL verbSo the specific collisions that motivated
naandsa(N/A,S.A.) don't actually happen — the punctuation fragments them below the min-length guard. Only the bare uppercase acronyms collide, and for all four the Polish function word is far more common in real content than the acronym. That's the inverse of theci↔CIcase, where the acronym was ubiquitous and the function word rare.Conclusion: removing
ma/na/co/safromSTOP_WORDS_PLwould trade a lot of function-word noise for a niche acronym gain. I don't think it's worth it as a straight removal.The better fix: preserve uppercase acronyms
The root cause is that
extract_weighted_keywordslowercases up front (_CLAUSE_BOUNDARY.split(text.lower())), which conflates the acronymMAwith the function wordma. A targeted fix would keep a token that was ALL-CAPS (≥2 chars) in the original text even when its lowercased form is a stop-word — soMA,CO,NA,SA(and every other acronym) survive, without letting the Polish function words through.That needs the tokenizer to carry original case (today it's discarded at
.lower()), so it's a bit more than a one-line stop-word edit — hence an issue first. Happy to send a PR if you like the direction.(The per-word audit script is small and I can share it if useful.)
Robert