Corpus: Wikimedia's citation clients are ordinary, not bad bots

README.md

@@ -46,13 +46,13 @@ matches one regex against one raw request component, and that is all it can do.
 
 So the converted rule set is measured against ordinary traffic and against
 attacks, with `nginx` itself, by [`tests/test_nginx_blocking.py`](tests/test_nginx_blocking.py).
-Against the rules in this repository (the nginx row of the table below), 162
+Against the rules in this repository (the nginx row of the table below), 164
 ordinary requests in twelve categories of what a false positive looks like, and 21
 attacks ([`patterns/corpus.py`](patterns/corpus.py)):
 
 | | |
 |---|---|
-| Ordinary requests refused | **0 of 162** |
+| Ordinary requests refused | **0 of 164** |
 | Attacks refused, sent in clear | **17 of 21** |
 | Attacks refused, percent-encoded | **6 of 21** |
 
@@ -161,7 +161,7 @@ expressions HAProxy's parser takes for unmatched quotes, a fetch it does not hav
 hundreds of names, documented as a pattern file that it was not. It is now what was
 documented: pattern files, which HAProxy reads a regular expression to a line, and a
 `waf.cfg` to paste into a `frontend`. The test runs it in a real HAProxy: of the 21
-attacks, 17 are refused in clear and 14 percent-encoded (`url_dec`), and 0 of the 162
+attacks, 17 are refused in clear and 14 percent-encoded (`url_dec`), and 0 of the 164
 ordinary requests. Its phrase lists (`@pmFromFile`: `/.env`, `/.git/config`, scanners) are the
 `*.data` files, loaded with `-m sub -i -f`.
 
@@ -170,7 +170,7 @@ it wrote the operators CRS names as if they were patterns, after `re.escape` had
 them into other patterns (`Failed to resolve operator: lt\`), and its bad-bot list gave
 every rule the same id ([#80](https://github.com/fabriziosalmi/patterns/issues/80)). It
 now writes `@rx` as CRS wrote it, and the test runs it in a real Apache with ModSecurity:
-none of the 162 ordinary requests is refused, and 17 of the 21 attacks are, in clear
+none of the 164 ordinary requests is refused, and 17 of the 21 attacks are, in clear
 and percent-encoded, because ModSecurity decodes `ARGS` before a rule reads it, which
 `nginx` cannot do. Its phrase lists (`@pmFromFile`, which CRS uses for `/.env`, `/.git/config` and
 scanners) are written as the `*.data` files ModSecurity reads from next to the rules.

docs/apache.md

@@ -11,7 +11,7 @@ Each rule is written as CRS wrote it, with the operator it named (`@rx`, `@pm` o
 - **Only one transformation is applied, `lowercase`.** The others (`urlDecodeUni`, `htmlEntityDecode`, ...) are written as `t:none`, so a pattern that was written to run after one runs on the raw value. The one thing ModSecurity does for you is `ARGS`: its values are already URL-decoded, so an attack percent-encoded in the query string is caught as the same attack in clear. `REQUEST_FILENAME` is not: it holds the path as it arrived (measured: a rule on `%2e` matches `/%2egit/config`), so a path percent-encoded passes a phrase written in clear.
 - **No chain.** A chain matches when every record does. ModSecurity can say so with `chain`; this backend does not write it yet, and a record of a chain on its own is another rule, so no record of one is written (128 of the 749).
 - **No anomaly score.** CRS adds points and refuses over a threshold. Here a rule decides alone: a rule of severity `high` refuses with a 403, and one below it records the match in the log and lets the request through (`pass,log`).
-- **A rule that refuses ordinary traffic is not written.** Without its transformations, some CRS rules mean something else; each is checked against [a corpus of ordinary requests](https://github.com/fabriziosalmi/patterns/blob/main/patterns/corpus.py), as nginx's are, and left out if one matches. The corpus is held to this by [`tests/test_apache_blocking.py`](https://github.com/fabriziosalmi/patterns/blob/main/tests/test_apache_blocking.py), which runs it through a real Apache on every change: 162 ordinary requests, none refused; of 21 attacks, 17 refused in clear and 17 percent-encoded.
+- **A rule that refuses ordinary traffic is not written.** Without its transformations, some CRS rules mean something else; each is checked against [a corpus of ordinary requests](https://github.com/fabriziosalmi/patterns/blob/main/patterns/corpus.py), as nginx's are, and left out if one matches. The corpus is held to this by [`tests/test_apache_blocking.py`](https://github.com/fabriziosalmi/patterns/blob/main/tests/test_apache_blocking.py), which runs it through a real Apache on every change: 164 ordinary requests, none refused; of 21 attacks, 17 refused in clear and 17 percent-encoded.
 
 At CRS v4.29.0 that is 180 rules written (8 in full, 172 with a named loss), and the rest dropped with a reason. It is a useful first filter in front of an application. It is not the Core Rule Set: if you can install that, do.
 

docs/envoy.md

@@ -13,7 +13,7 @@ Each rule is written as CRS wrote it, a regular expression or a list of phrases,
 - **`(?i)` is kept** and a rule CRS gives `t:lowercase` is matched with case ignored. The other transformations are not applied, and the [coverage matrix](/coverage) says which, rule by rule.
 - **The query string is matched together with the path**, on `:path`: a pattern written for one value sees the path and the other values too. The request body is not matched.
 - **No anomaly score.** Every rule decides alone.
-- **A rule that refuses ordinary traffic is not written.** Each is checked against [a corpus of ordinary requests](https://github.com/fabriziosalmi/patterns/blob/main/patterns/corpus.py) and left out if one matches. [`tests/test_envoy_blocking.py`](https://github.com/fabriziosalmi/patterns/blob/main/tests/test_envoy_blocking.py) runs the files in a real Envoy on every change: 162 ordinary requests, none refused; of 21 attacks, 17 refused in clear and 6 percent-encoded.
+- **A rule that refuses ordinary traffic is not written.** Each is checked against [a corpus of ordinary requests](https://github.com/fabriziosalmi/patterns/blob/main/patterns/corpus.py) and left out if one matches. [`tests/test_envoy_blocking.py`](https://github.com/fabriziosalmi/patterns/blob/main/tests/test_envoy_blocking.py) runs the files in a real Envoy on every change: 164 ordinary requests, none refused; of 21 attacks, 17 refused in clear and 6 percent-encoded.
 
 At CRS v4.29.0 that is 184 rules written (7 in full, 177 with a named loss). It is a useful first filter in front of an application, not the Core Rule Set.
 

docs/haproxy.md

@@ -12,7 +12,7 @@ Each rule is written as CRS wrote it, a regular expression or a list of phrases,
 - **The query string is matched whole**, not parameter by parameter: a pattern written for one value sees the others too. The request body and the cookies are not matched.
 - **`(?i)` is kept as CRS wrote it, in an expression.** HAProxy compiles the patterns with PCRE2, so there is no `-i` to approximate it with.
 - **No anomaly score.** Every rule decides alone.
-- **A rule that refuses ordinary traffic is not written.** Each is checked against [a corpus of ordinary requests](https://github.com/fabriziosalmi/patterns/blob/main/patterns/corpus.py), with its converters applied, and left out if one matches. [`tests/test_haproxy_blocking.py`](https://github.com/fabriziosalmi/patterns/blob/main/tests/test_haproxy_blocking.py) runs it through a real HAProxy on every change: 162 ordinary requests, none refused; of 21 attacks, 17 refused in clear and 14 percent-encoded.
+- **A rule that refuses ordinary traffic is not written.** Each is checked against [a corpus of ordinary requests](https://github.com/fabriziosalmi/patterns/blob/main/patterns/corpus.py), with its converters applied, and left out if one matches. [`tests/test_haproxy_blocking.py`](https://github.com/fabriziosalmi/patterns/blob/main/tests/test_haproxy_blocking.py) runs it through a real HAProxy on every change: 164 ordinary requests, none refused; of 21 attacks, 17 refused in clear and 14 percent-encoded.
 
 At CRS v4.29.0 that is 180 rules written (7 in full, 173 with a named loss). It is a useful first filter in front of an application, not the Core Rule Set.
 

patterns/corpus.py

@@ -404,6 +404,12 @@ def benign(category: str, name: str, why: str, path: str, query: str = "",
     benign("bots", "baiduspider", "a crawler that names `spider` and its own domain",
            "/", user_agent="Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
            referer=""),
+    benign("bots", "citoid", "Wikimedia's citation service: it reads one page, on demand, when an editor cites it",
+           "/", user_agent="Citoid/WMF",
+           referer=""),
+    benign("bots", "zotero-translation-server", "the Zotero translator Wikimedia runs beside Citoid",
+           "/", user_agent="ZoteroTranslationServer/WMF",
+           referer=""),
     benign("bots", "yandexbot", "a crawler whose name ends in `bot`",
            "/", user_agent="Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)", referer=""),
     benign("bots", "applebot", "a crawler inside a Safari agent",