WARNING: This repository is no longer maintained.

This project is archived and will not receive further updates, bug fixes, or security patches. Issues and pull requests may not be reviewed.

Use this software at your own risk.

---

MFAQ: a Multilingual FAQ Dataset

MFAQ is a multilingual FAQ retrieval dataset. We also release a multilingual FAQ retrieval model trained on this dataset.

Dataset

The dataset is hosted on the HuggingFace hub. You can find it here.

Start by installing the dataset package:

pip install datasets

Then import the dataset:

from datasets import load_dataset
en_dataset = load_dataset("clips/mfaq", "en")

You can find more information about the dataset and the available configurations on the description page.

Model

The pre-trained FAQ retrieval model is also hosted on the HuggingFace Hub. You can find it here.

Start by installing sentence-transformers:

pip install sentence-transformers

Load the model:

from sentence_transformers import SentenceTransformer

Each question must be pre-pended with a <Q>, answers with a <A>.

question = "<Q>How many models can I host on HuggingFace?"
answers = [
  "<A>All plans come with unlimited private models and datasets.",
  "<A>AutoNLP is an automatic way to train and deploy state-of-the-art NLP models, seamlessly integrated with the Hugging Face ecosystem.",
  "<A>Based on how much training data and model variants are created, we send you a compute cost and payment link - as low as $10 per job."
]
model = SentenceTransformer('clips/mfaq')
q_embedding, *a_embeddings = model.encode([question] + answers)
best_answer_idx = sorted(enumerate(a_embeddings), key=lambda x: q_embedding.dot(x[1]), reverse=True)[0][0]
print(answers[best_answer_idx])

Training

train.py uses the HuggingFace Trainer to train a FAQ retrieval model on MFAQ. The following configuration reaches an MRR of 89% on the English subset:

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 train.py \
    --languages en de es fr it nl pt tr ru pl id no sv da vi fi ro cs he hu hr  \
    --output_dir output/1024 \
    --do_train \
    --per_device_train_batch_size 256 \
    --per_device_eval_batch_size 256 \
    --distributed_softmax \
    --max_steps 3000 \
    --evaluation_strategy steps \
    --eval_steps 250 \
    --max_seq_len 128 \
    --warmup_steps 1000 \
    --label_names page_id \
    --logging_steps 5 \
    --fp16 \
    --metric_for_best_model eval_global_mrr \
    --load_best_model_at_end \
    --save_total_limit 3 \
    --report_to tensorboard \
    --dataloader_num_workers 1 \
    --single_domain \
    --hidden_dropout_prob 0.25 \
    --learning_rate 0.00005 \
    --weight_decay 0.01 \
    --alpha 1 \
    --gradient_checkpointing \
    --deepspeed config/ds_config_zero_2.json