You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Real query embeddings and a fixed query set for recall and the CI gate #23
Today the queries are random document vectors held out from the base (own data and DBpedia; GloVe and SIFT use their published query sets). Real user queries are often short and sit in a different part of the space, so a sampled-vector test can look healthy while recall on real queries drops.
1. Real query embeddings. Add --query-file (.npy / .parquet) and, for Postgres, --query-table / --query-column. When given, report recall on those queries and on the held-out sample as separate columns, and let --min-recall apply to either.
2. Sampling noise vs. a real regression. Query sampling is seeded (--seed), so the CI gate already uses a fixed sample while the data doesn't change. But there's no estimate of how much recall moves with a different sample. Add an option to resample the queries N times and report the spread, so a CI drop can be compared against it. A fixed query file (1) also gives a stable set to gate on.
Docs: say clearly in the README and docs/method.md which queries the numbers use.
Label the query source next to every score in the report ("held-out documents" vs. "real queries from "), so the two recalls can't be mixed up.
Keep query IDs fixed across dataset updates: store the chosen query ids (or the query file hash) with the report, and let the CI gate reuse them, so a change in recall after new data is added isn't a different query sample.
Activity
ahmad-codex commentedon Nov 2, 2026
Two more from the same thread: