Skip to content

add milvus + elastic as a load backend#28

Open
Nathan LeRoy (nleroy917) wants to merge 3 commits into
masterfrom
more_backends
Open

add milvus + elastic as a load backend#28
Nathan LeRoy (nleroy917) wants to merge 3 commits into
masterfrom
more_backends

Conversation

@nleroy917

Copy link
Copy Markdown
Collaborator

No description provided.

Ok(())
}

async fn enable_indexing(&self) -> Result<(), StoreError> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not "enable indexing". Milvus flushes open segments based on size thresholds from the get go. This flushes all open segments, which marks them as eligible for indexing. Not sure if this is important or not but worth mentioning

@nleroy917

Copy link
Copy Markdown
Collaborator Author

Tested the milvus backend and the elastic backend -- both look good. Using the following configs:
embedding:

source:
  type: huggingface_parquet
  dataset_name: mteb/tweet_sentiment_extraction
  split: train
  text_field: text

dense_embedder:
  type: sentence_transformer
  model: sentence-transformers/all-MiniLM-L6-v2
  batch_size: 128
  dtype: bfloat16

sparse_embedder:
  type: fastembed
  model: Qdrant/bm25
  batch_size: 256

storage:
  type: local
  output_dir: /tmp/mteb_tweets_all-MiniLM-L6-v2

pipeline:
  chunk_size: 1000
  flush_threshold: 10000

loading:

datasource:
  type: local
  path: /tmp/mteb_tweets_all-MiniLM-L6-v2
  id_expression: "vf_point_id(filename, file_row_number)"
  payload_fields:
    text: text                                  # plain column
    label: label
    label_text: label_text
    tweet_id: id

vectors:
  dense:
    type: dense
    column: dense_embedding
    distance: cosine
  sparse:
    type: sparse
    column: sparse_embedding

vectorstore:
  type: milvus
  url: http://localhost:19530
  collection_name: mteb_tweets
  # username: root
  # password: ${MILVUS_PASSWORD}
  # id_max_length: 128                 # bump if your ids exceed this
  recreate: false

loader:
  batch_size: 256
  concurrency: 4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants