Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions components/blog/blog.css
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@
gap: clamp(32px, 4vw, 56px);
align-items: start;
}
/* A post's text fills its column up to the sidebar: a narrower measure left an empty strip that read as missing content. */
.wirecat-blog .blog-article {
--blog-measure: 100%;
}
.wirecat-blog .blog-sidebar {
position: sticky;
top: 20px;
Expand Down
165 changes: 67 additions & 98 deletions content/blog/making-a-reranker-fast/index.es.md

Large diffs are not rendered by default.

117 changes: 48 additions & 69 deletions content/blog/making-a-reranker-fast/index.md
Original file line number Diff line number Diff line change
@@ -1,103 +1,82 @@
---
title: Making a reranker fast on a laptop, in four acts
description: Batching a quantized reranker changed its answers, not its speed. A native runtime made it 2× faster, not 10×. Shipping that runtime took no new npm packages, and on Windows the library we needed was only inside an npm tarball.
date: 2026-10-27
title: How I made a reranker faster on my own machine
description: When I turned the reranker on by default, WireCat's meaning search went from 1.9 to 5.1 seconds. Batching barely helped and changed the scores, native ONNX Runtime made scoring 2x faster, not 10x, and the Windows DLL I needed was only inside an npm tarball.
date: 2026-11-18
tags: [search, benchmarks, cli]
cover: ./cover.jpg
coverAlt: "Fifty printed message cards banded as top 50, one card read under a loupe, five cards ranked 1 to 5 with the first marked green, and a stopwatch at 3.2 seconds"
coverAlt: "A stack of 50 printed messages banded as top 50, one read under a loupe, 5 laid out in order with the first marked green, a stopwatch at 3.2 seconds"
author: viacheslav-ptsarev
series: laptop-search
glance:
- value: "4.6 s → 3.2 s"
label: "a real --discover, WebAssembly against native"
href: "#act-two-native-was-2-faster-not-10"
label: "--discover on a real archive, WebAssembly and native"
href: "#native-made-scoring-2x-faster-not-10x"
- value: "≤ 15%"
label: "saved by batching, and the scores changed"
href: "#act-one-batching-changed-the-answers-not-the-speed"
label: "the most batching saved, and the scores moved"
href: "#batching-saved-15-and-moved-the-scores"
- value: "113.5 MB"
label: "the npm tarball that held the only Windows DLL that worked"
href: "#act-four-the-windows-library-was-only-inside-an-npm-tarball"
label: "the npm tarball I take one Windows DLL out of"
href: "#on-windows-the-dll-had-to-come-out-of-an-npm-tarball"
- value: "5 + Alpine"
label: "systems CI now runs natively; Alpine stays on WebAssembly"
href: "#act-four-the-windows-library-was-only-inside-an-npm-tarball"
draft: true
label: "systems where CI runs the models natively, Alpine stays on WebAssembly"
href: "#on-windows-the-dll-had-to-come-out-of-an-npm-tarball"
draft: false
---

Our CLIs search a messenger archive on the owner's own machine. `search messages --discover` finds
candidates by meaning and by words, then a small reranker model (mMARCO, 8-bit) reads the top 50 and
orders them. When the reranker became the default, `--discover` went from 1.9 s to 5.1 s on a real
archive. These are the four things we did about it, and what each one taught us.
When I turned the reranker on by default in WireCat search, `search messages --discover` on my archive went from 1.9 to 5.1 seconds. I tried the 2 obvious things: batching, and native ONNX Runtime instead of WebAssembly. Batching barely helped and changed the scores on top of that, and native made scoring 2x faster, while my estimate said 4-10x. And on Windows the DLL I needed was only inside a 113.5 MB npm tarball, so I had to take it out of there.

## Act one: batching changed the answers, not the speed
`--discover` finds candidates by meaning and by words, then a small mMARCO model (8-bit) reads the top 50 and puts them in order, that's the reranker. It all runs locally on the user's machine, no cloud.

The obvious fix is to score the 50 pairs in batches: one model run for many pairs instead of one run per
pair. In WebAssembly it saved at most about 15%, and only with pairs sorted by length. Mixed lengths were
slower, because every pair is padded to the longest one and the padding costs as much as the text.
## Batching saved 15% and moved the scores

Worse, the scores moved. The same pair scored 0.04 alone, 0.48 when padded to 200 tokens, and 0.34 next
to one other pair. The 8-bit model probably sets its range from the whole input of each run, so the
neighbours in a batch change a pair's score. A ranking that depends on which hits share a batch is not
a ranking. We kept one pair per run.
First I tried not running the model 50 times, one pair (question and message) at a time, but giving it pairs in batches. In WebAssembly that saved 15% at most, and only with the pairs sorted by length. With pairs of different lengths in one batch it got even slower, because every pair is padded to the longest one and the padding costs as much as real text.

| Scoring 50 mixed pairs | Time | Largest score change |
And the scores changed. The same pair got 0.04 on its own, 0.48 padded to 200 tokens and 0.34 next to one other pair. I think the 8-bit model sets its value range from the whole input of each run (dynamic quantization), so the neighbours in a batch move the score. That's a guess, I didn't dig deeper. But even if the cause is something else, the order of results would depend on which messages happened to share a batch, so I kept one pair per run.

| 50 pairs of different lengths | Time | Largest score change |
| --- | ---: | ---: |
| One by one | 1.54 s | — |
| One by one | 1.54 s | |
| Batches of 5, sorted by length | 1.33 s | 1.01 |
| Batches of 5, in arrival order | 2.12 s | 0.78 |
| One batch of 50 | 2.37 s | 0.85 |

## Act two: native was 2× faster, not 10×
The text in this run is synthetic, scores are in logits. The fastest option also moves the scores more than the others in the table, and batches of 25 moved them by up to 1.34. I measured only in WebAssembly, whether native batches behave the same I didn't check.

## Native made scoring 2x faster, not 10x

Next I ran the same models through ONNX Runtime's native library instead of WebAssembly. I expected 4-10x, because the Python version scored 50 short synthetic pairs in 0.2 seconds, while WebAssembly took 0.9-1.7 on my side. But I was comparing different runtimes from different runs. When I ran Node with the native library and WebAssembly on the same pairs, it was 0.25-0.30 seconds against 0.60, so 2-2.4x.

On a real archive it came out the same. The reranker reads more than the message, it also gets the chat, the author and the message it replies to, so the median was 153 tokens against 48 in the synthetic set, and every tenth pair was longer than 762. Per pair native is 2-2.5x faster, but the whole `--discover` went from 4.58 to 3.20 seconds (median of 18 calls), so 30%. The rest is loading the models and tokenizers (17 MB of JSON, the same in both runtimes), garbage collection and the search itself.

Next we ran the same models through ONNX Runtime's native library instead of its WebAssembly build. A
short-text benchmark promised 4–10×. On the real archive, `--discover` went from 4.58 s to 3.20 s, with
a direct answer in the top five for 6 of 6 known questions on both runtimes.
The answers stayed the same. On my 6 known questions the direct answer was in the top 5 either way, and 29 of the 30 top-5 hits matched. Near-ties sometimes swap places, because the runtimes score the same pair slightly differently. The starting point here is 4.58, not 5.1, because it's another build and another load on the machine, so a comparison with 5.1 is only rough.

The gap is the input length. Real reranker inputs carry the chat, the author, the date and the message
being replied to: a median of 153 tokens, and one in ten longer than 762. The synthetic set's median was
48. Per pair, native was 2–2.5× faster. Benchmarks on short text overstate the gain for long text.
I also noticed that the reranker sometimes never saw the hit at all. I measured this separately, on one chat. There, in 18 of 119 pairs with a reply, the message it replied to took all 512 of the model's tokens. I cut it to 300 characters, I'll write about that separately. With both changes `--discover` in max 0.81.0 took 2.91 seconds natively and 4.10 through WebAssembly.

A later fix cut the replied-to message to 300 characters (that story is its own post). With both
changes, the released CLI ran `--discover` in 2.91 s natively against 4.10 s through WebAssembly.
## Shipping native without new packages

## Act three: no new packages
The usual way to ship a native library through npm is a separate package per platform. I didn't want more packages and a bigger install for people who never download models. `onnxruntime-node` itself is 288 MB unpacked, because it has the libraries of every system in it.

The usual way to ship a native library to npm users is one package per platform. We did not want five
new packages to publish, or a bigger install for people who never download a model.
For each platform it has 2 parts: a 0.3-0.4 MB binding and the library. The binding looks for the library in its own folder, and I used that - the bindings went into `@wirecat/cli-core-onnx`, which is published anyway, and the library is downloaded with the models from Microsoft's official release and checked against a pinned sha256. Search never downloads anything, the library shows up only after `models download`, and if it's missing or doesn't load, everything runs through WebAssembly as before. I checked the scores on 450 pairs, with the library from Microsoft's release they matched the npm one bit for bit.

The npm package `onnxruntime-node` has two parts per platform: a 0.4 MB connector
(`onnxruntime_binding.node`) and the library itself. The connector looks for the library in its own
folder. So the connectors ride inside a package we already publish, and the library is downloaded with
the models, from Microsoft's official release archive, checked against a pinned sha256 hash. A search
never downloads anything; the library arrives only when the user runs `models download`. If it is
missing or does not load, the models run through WebAssembly as before. On 450 pairs the scores matched
the npm package's own library exactly.
## On Windows the DLL had to come out of an npm tarball

## Act four: the Windows library was only inside an npm tarball
On Windows that trick didn't work. The Windows binding takes a function from `onnxruntime.dll` not by name but by number 4, and that's the DirectML function (DirectML is the Windows layer for the graphics card). Microsoft's release has the plain CPU build without it, and Windows answered the load with just "The operating system cannot run %1", not a word about what's missing.

The same trick failed on Windows. The Windows connector imports one function from `onnxruntime.dll` by
number, not by name: ordinal 4, the setup function for DirectML, Windows' GPU layer. Microsoft's release
DLL is the plain build, without that function, so the connector refused to load it.
The only place I found the DirectML build of 1.30.0 was inside `onnxruntime-node`'s own npm tarball, which is 113.5 MB. NuGet has it too, but only up to 1.24.4. So on Windows `models download` fetches that tarball once, takes one DLL out of it and checks it against a pinned sha256. DirectML.dll itself loads only when needed, so the CPU path works without it. A bit hacky, sure, but no new packages.

The DirectML build of version 1.30.0 is not in Microsoft's release archive, and its NuGet package stops
at an older version. We found it in one place: inside `onnxruntime-node`'s own 113.5 MB npm tarball. So on
Windows, `models download` fetches that tarball once, keeps the one DLL, and checks it against its own
pinned hash. The DirectML part loads only if it is used.
Before this, CI ran only a tiny test model through the npm package, and nothing checked `models download` and what actually runs for the user after it. Now a separate workflow downloads the library and runs 2 tiny test models natively on 5 systems: Linux and Windows on x64 and arm64, plus macOS on Apple silicon. On Alpine, which has no build, it checks that native isn't even offered and everything goes through WebAssembly. Intel Macs stay on WebAssembly too, the package has no library for them (and CI doesn't cover them either).

Then we checked it for real. A new CI workflow downloads the library and runs two tiny test models
natively on Linux x64 and arm64, macOS on Apple silicon, Windows x64 and Windows arm64. It also checks
that Alpine Linux, which has no build, is offered nothing and stays on WebAssembly. All six pass; the
Windows jobs take 1.5–2 minutes.
## What I'd do differently

## What we would tell someone doing the same
- I'd compare runtimes on the same pairs and at the real input length. My 4-10x came from comparing different runs.
- I wouldn't batch a quantized model if the scores of different pairs have to be compared with each other.
- A native library can be downloaded on demand, with a pinned hash and WebAssembly for when it doesn't load. I never needed a package per platform.
- Files being in a package doesn't mean the library loads. For me that's now checked by CI, which really runs a model on each of the 5 systems.

- Measure on your real input lengths. Short synthetic text made the native runtime look up to five times
better than it was.
- Do not batch a dynamically quantized model if its scores must be comparable across pairs.
- A native library can be downloaded on demand, pinned by hash, with WebAssembly as the fallback. You
may not need a package per platform.
- Check the library on every system in CI. Reading the package listing is not the same as loading it.
## What I didn't measure

## Notes
- Native speed on Windows, macOS and Linux arm64. CI checks that it works there, but doesn't time it.
- The MCP server, where the models are already loaded and only scoring gets faster.
- A model without quantization (fp32), which shouldn't depend on its batch neighbours but is bigger and slower.

The measurements, the setup and the scripts are public:
[reranker speed](https://github.com/WireCatLabs/cli-testing/blob/main/performance/search/rerank-speed/README.md).
Real messages were used only as totals; the questions and the per-question rows stay private.
The synthetic runs, their setup and scripts are in [cli-testing](https://github.com/WireCatLabs/cli-testing/blob/main/performance/search/rerank-speed/README.md), you can repeat them. From my real archive there are only totals, the questions and per-question rows stay private. If native came out faster for you on long texts, tell me what you measured on and I'll compare with my numbers.
Loading
Loading