Skip to content

Real query embeddings and a fixed query set for recall and the CI gate #23

Open

Description

@ahmad-codex

Today the queries are random document vectors held out from the base (own data and DBpedia; GloVe and SIFT use their published query sets). Real user queries are often short and sit in a different part of the space, so a sampled-vector test can look healthy while recall on real queries drops.

1. Real query embeddings. Add --query-file (.npy / .parquet) and, for Postgres, --query-table / --query-column. When given, report recall on those queries and on the held-out sample as separate columns, and let --min-recall apply to either.

2. Sampling noise vs. a real regression. Query sampling is seeded (--seed), so the CI gate already uses a fixed sample while the data doesn't change. But there's no estimate of how much recall moves with a different sample. Add an option to resample the queries N times and report the spread, so a CI drop can be compared against it. A fixed query file (1) also gives a stable set to gate on.

Docs: say clearly in the README and docs/method.md which queries the numbers use.

Suggested in a Reddit comment on the launch post.

Activity

  1. ahmad-codex commented on Nov 2, 2026

    @ahmad-codex
    OwnerAuthor

    Two more from the same thread:

    • Label the query source next to every score in the report ("held-out documents" vs. "real queries from "), so the two recalls can't be mixed up.
    • Keep query IDs fixed across dataset updates: store the chosen query ids (or the query file hash) with the report, and let the CI gate reuse them, so a change in recall after new data is added isn't a different query sample.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: datasetsDatasets and vector sourcesarea: labCodecs and the codec labenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions