Repository navigation
Corpus: Wikimedia's citation clients are ordinary, not bad bots - #104
Merged
fabriziosalmi merged 2 commits intoOct 18, 2026
Merged
Conversation
VDK
force-pushed
the
allow-wikimedia-citation-uas
branch
from
October 8, 2026 18:34
e1c2a85 to
071eef5
Compare
The bad-bot lists are built from public sources, and two of them name Wikimedia's citation services: mitchellkrogza's bad-user-agents list has `Citoid`, and JayBizzle/Crawler-Detect - which answers "is this a crawler", not "is this hostile" - has `Citoid` and `ZoteroTranslationServer`. The generated patterns therefore refuse the Wikimedia citation service for every site behind these rules. It is one page read on demand, when an editor cites a source, and it returns the metadata for the footnote. It is not crawling the site and it is not hostile. `corpus.wanted_user_agents` is what says who must not be refused, and `leave_out_wanted()` leaves out an entry that matches one, naming it in the header of each file. Adding the two clients there takes the `Citoid` and `ZoteroTranslationServer` entries out of all five targets and leaves every other entry in place. Agents from https://wikitech.wikimedia.org/wiki/Citoid/Allowlisting, cut to the part of the user agent that is stable: `Citoid/WMF` and `ZoteroTranslationServer/WMF`. The browser part in front of it is inherited from translation-server and may change, so it is not in the corpus. Nor is a contact address: the domain is not what an entry is matched on, and the corpus holds no real ones. waf_patterns/* is `python3 badbots.py` on this change, and the corpus counts in README.md and docs/*.md follow the two entries added. Both are committed because tests/test_corpus.py and tests/test_badbots.py hold the committed files to the corpus and to the ordinary clients it says must not be refused. The output holds only this change: the three entries Crawler-Detect gained after the last build (JayBizzle/Crawler-Detect#624: IronFountain-Leads, TheWebReport and webshopok-kutatas) are left out of it, and the nightly build picks them up on its own. The Apache target numbers its rules by position, so every id after an entry that is left out shifts, and that diff is larger than the two rules removed. Co-authored-by: openhands <openhands@all-hands.dev>
VDK
force-pushed
the
allow-wikimedia-citation-uas
branch
from
October 8, 2026 18:49
071eef5 to
8d0ee64
Compare
main's nightly update (2026-10-18) rewrote waf_patterns, so apache/bots.conf conflicted on rule ids. The bot lists are regenerated with `python3 badbots.py` on the merged tree; against main they differ only by the Citoid and ZoteroTranslationServer entries this PR leaves out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thank you, this is correct and well argued. Verified: regenerating on today's main with and without this change removes exactly the Citoid and ZoteroTranslationServer entries from all five targets and nothing else; the CI test steps pass, and the README/docs counts are required by test_corpus. The PR conflicted with today's nightly update of the generated files, so I merged main into your branch and regenerated waf_patterns; your change is unchanged. Merging once CI is green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bad-bot lists are built from public sources, and two of them name Wikimedia's citation services: mitchellkrogza's bad-user-agents list has
Citoid, and JayBizzle/Crawler-Detect - which answers "is this a crawler", not "is this hostile" - hasCitoidandZoteroTranslationServer. The generated patterns therefore refuse the Wikimedia citation service for every site behind these rules.It is one page read on demand, when an editor cites a source, and it returns the metadata for the footnote. It is not crawling the site and it is not hostile.
corpus.wanted_user_agentsis what says who must not be refused, andleave_out_wanted()leaves out an entry that matches one, naming it in the header of each file. Adding the two clients there takes theCitoidandZoteroTranslationServerentries out of all five targets and leaves every other entry in place.Agents from https://wikitech.wikimedia.org/wiki/Citoid/Allowlisting, cut to the part of the user agent that is stable:
Citoid/WMFandZoteroTranslationServer/WMF. The browser part in front of it is inherited from translation-server and may change, so it is not in the corpus. Nor is a contact address: the domain is not what an entry is matched on, and the corpus holds no real ones.waf_patterns/* is
python3 badbots.pyon this change, and the corpus counts in README.md and docs/*.md follow the two entries added. Both are committed because tests/test_corpus.py and tests/test_badbots.py hold the files to what is written. The Apache target numbers its rules by position, so every id after an entry that is left out shifts, and that diff is larger than the two rules removed.