Skip to content

way-embed: CLS pooling on the ModernBERT rank path so Ettin rerankers load #556

Description

@aaronsb

Finding

ADR-189 (proposed in #554) names ettin-reranker-17m-v1 and -32m-v1 (Apache-2.0) as the lead gate candidates. The pinned llama.cpp (ec2b787) cannot run them correctly as rerankers yet:

  • In src/llama-graph.cpp, the LLAMA_POOLING_TYPE_RANK branch hard-codes mean pooling for LLM_ARCH_MODERN_BERT, to match gte-reranker-modernbert-base.
  • Ettin rerankers use CLS pooling, then Dense, GELU, LayerNorm and Dense.
  • Their head lives in separate Sentence Transformers module folders, which the converter does not pick up.

Proposal

  1. Patch the rank path to select pooling from GGUF metadata rather than from the architecture.
  2. Repack the Ettin head in convert_hf_to_gguf.py so the classifier tensors land where the rank path expects them.
  3. Offer the change upstream, and carry it on the submodule until it lands.
  4. Verify score parity against the Sentence Transformers reference to within 1e-3 (ADR-189 eligibility).

If the patch cannot be carried, ADR-189 falls back to the baselines (ms-marco-MiniLM-L6-v2, jina-reranker-v1-tiny-en), which need no change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:waysWays CLI, matching, steering layereffort:mediumA few sittings

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions