RAG Drift Detection

[MasemeneMatlakanaBenny]

Summary

This PR introduces topic-based drift and alignment metrics for RAG systems.

The goal is to evaluate whether the topics present in a RAG knowledge base are aligned with the topics appearing in user queries, as well as to detect topic-level changes between reference and analysis content.

What's Included

• Added topic distribution extraction using BERTopic.

• Added TopicContentQueryDrift for measuring topic alignment between:

  • Knowledge-base content
  • User queries

• Added TopicSemanticContentDrift for comparing topic distributions between:

  • Reference content
  • Analysis/current content

• Added topic-word extraction and dataframe representations for downstream analysis.

• Added topic mismatch detection.

• Integrated categorical similarity metrics:

  • Braun
  • Dice
  • Jaccard
  • Overlap
  • Tanimoto

• Added unit tests covering topic distribution, topic extraction, mismatch detection, dataframe generation, drift construction, and similarity metric delegation.

Motivation

Traditional RAG evaluation often focuses on retrieval relevance or answer quality. This PR adds another dimension by looking at the topic space of the system.

This allows us to answer questions such as:

• Are user queries within the topical scope of the knowledge base?
• Are the topics in the current knowledge base drifting from the reference knowledge base?
• Are the topics represented in retrieved/available knowledge aligned with the topics users are asking about?

The intention is to provide a higher-level signal for monitoring RAG topic alignment, coverage, and semantic drift.

Testing

Unit tests were added for the new functionality, with external topic-model behavior mocked to keep the tests deterministic and independent of BERTopic model training.