diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..8a3f967 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,295 @@ + + + + + +Porter vs Lemmas · Bitext English Analyzer + + + + + +
+
+
Bitext English Analyzer · open source · Elasticsearch · OpenSearch · Solr
+

Porter stemming can't match sold to sell.
Dictionary lemmas can.

+

Ten real queries from the MIRACL English benchmark (32.9M passages, plain BM25). For each one, see exactly which words the query and the relevant passage share under Elasticsearch's Standard analyzer, its built-in English (Porter) analyzer, and the Bitext lemmatizing analyzer.

+
+
+18.8%F1@10 vs Standard
+
+15.7%nDCG@10 vs Standard
+
+7.6–8.5%every top-10 metric vs English (Porter)
+
1 commandto install
+
+

Elasticsearch 9.5.4, BM25, 799 queries, official trec_eval. Fully reproducible: bitext-search-benchmarks.

+
+ + + +
+ +
+ query word found in the passage + link only Bitext makes + Porter merges different words + +
+

Tokens come from running the real analyzers (Lucene 10 Standard and English analyzers, and the released Bitext analyzer v1.0.0) on the exact query and passage text. Ranks are the positions of the judged-relevant passage in the full benchmark run.

+ +
+

Try it on your own cluster

+

Prebuilt packages are in the v1.0.0 release. No build needed. Restart the node after installing.

+
+
+

Elasticsearch 9.5.4

+ 
bin/elasticsearch-plugin install https://github.com/bitext/bitext-lucene-english-analyzer/releases/download/v1.0.0/bitext-java-english-elasticsearch-1.0.0.zip
+
+
+

OpenSearch 3.4+

+ 
bin/opensearch-plugin install https://github.com/bitext/bitext-lucene-english-analyzer/releases/download/v1.0.0/bitext-java-english-opensearch-1.0.0.zip
+
+
+
+

Then use it in your index settings

+ 
{
+  "settings": {
+    "analysis": {
+      "analyzer": {
+        "bitext_english": { "type": "bitext_java_english" }
+      }
+    }
+  }
+}
+

Solr 10: copy bitext-lucene-english-analyzer-1.0.0.jar from the release into Solr's lib directory and use com.bitext.lucene.analysis.en.EnglishLemmatizingAnalyzer as the field analyzer.

+
+
+ Analyzer on GitHub + Reproduce the benchmark + Found a case where it loses? Tell us +
+
+ +
+

Beyond English

+

English is the hardest case for linguistics to prove its value. In German, lemmatization plus linguistic decompounding improved F1@10 by +36.1% over Elasticsearch's default analyzer on MIRACL German. Bitext covers 77 languages. Ask about other languages.

+

German results: german-benchmark.md (SDK-based; a reproducible package is planned).

+
+ + +
+ + + +