RAG Evaluation Engine

Python tools for evaluating RAG (Retrieval-Augmented Generation) outputs with RAGAS-style metrics, custom prompts, and either Google Gemini or OpenAI models.

Features

• Five core RAG metrics: faithfulness, answer relevancy, context precision, context recall, and answer correctness.
• Multi-turn JSON evaluation with per-item averages and an AspectCritic-style conversation quality score.
• Custom prompt configuration through JSON files in rag-eval-engine/custom_prompts/.
• Flask API for CSV-based evaluation with OpenAI.
• Standalone JSON scoring scripts for Gemini and OpenAI.
• Optional RAG answer augmentation script for adding generated answers and retrieved contexts before evaluation.

Project Structure

RAGAS/
|-- README.md
|-- API_doc.md
|-- rag_answer_generator.py
|-- rag-eval-engine/
|   |-- api.py
|   |-- test_api.py
|   |-- score_json.py
|   |-- score_json_multiturn.py
|   |-- score_json_multiturn_v2.py
|   |-- score_ragas_singleturns.py
|   |-- score_ragas_multiturns.py
|   |-- requirements.txt
|   |-- .env.example
|   |-- eval_data/              # Local only; ignored by git
|   |-- custom_prompts/
|   |   |-- faithfulness.json
|   |   |-- answer_relevancy.json
|   |   |-- answer_correctness.json
|   |   |-- context_precision.json
|   |   |-- context_recall.json
|   |   |-- aspect_critic.json
|   |   `-- defaults/
|   `-- RAGAS-inspect/

Large JSON files, CSV files, .env, and eval_data/ are intentionally ignored by git so private datasets and API keys are not committed.

Quick Start

cd C:\Users\user\Desktop\RAGAS
python -m venv .venv
.\.venv\Scripts\Activate.ps1
cd rag-eval-engine
pip install -r requirements.txt
Copy-Item .env.example .env

Edit .env and add the API keys and model names you want to use:

OPENAI_API_KEY=your-openai-api-key-here
OPENAI_MODEL_NAME=gpt-4o-2024-08-06
EMBEDDING_MODEL_NAME=text-embedding-3-small
GEMINI_API_KEY=your-gemini-api-key-here
GEMINI_MODEL_NAME=gemini-2.5-flash

Usage

Standalone JSON Evaluation

Run from rag-eval-engine/.

Single-turn evaluation with Gemini:

python score_json.py path\to\data.json

Multi-turn evaluation with Gemini:

python score_json_multiturn.py path\to\data.json

Multi-turn evaluation with OpenAI:

python score_json_multiturn_v2.py path\to\data.json

Expected JSON input shape:

{
  "items": [
    {
      "item_id": 1,
      "turns": [
        {
          "turn_id": "turn_1",
          "question": "...",
          "answer": "ground truth answer",
          "RAG_answer": "answer generated by your RAG system",
          "RAG_context": [
            { "content": "retrieved chunk..." }
          ]
        }
      ]
    }
  ]
}

Scored output is written next to the input file with a _scored(...) suffix.

Flask API Evaluation

Run from rag-eval-engine/.

python api.py

The server loads eval_data/test_sample.csv and exposes:

POST http://127.0.0.1:5001/evaluate

You can call it with:

python test_api.py

or:

curl -X POST http://127.0.0.1:5001/evaluate

The CSV must contain these columns:

question,contents,answer,ground_truth

RAG Answer Augmentation

Run from the repository root:

python rag_answer_generator.py path\to\data.json

This creates a new file with a _RAG_augmented suffix unless you pass --output.

Metrics

Metric	What it checks	Score
Faithfulness	Whether the answer is grounded in retrieved context	0 or 1
Answer Relevancy	Whether the answer addresses the question	0 or 1
Context Precision	Whether retrieved context is useful	0 or 1
Context Recall	Whether context contains the needed ground-truth information	0 or 1
Answer Correctness	Whether the answer matches the ground truth	0.0 to 1.0
AspectCritic	Multi-turn consistency, retention, and redundancy	0 or 1

Custom Prompts

Prompt settings live in rag-eval-engine/custom_prompts/. Each metric has a JSON config file. To disable a custom prompt and fall back to the default RAGAS behavior, set:

{
  "enabled": false
}

See rag-eval-engine/custom_prompts/README.md for the full schema.

RAGAS Inspector

The inspector scripts print RAGAS metric internals such as prompts, examples, and execution flow.

cd rag-eval-engine\RAGAS-inspect
python run_all_inspectors.py
python inspect_faithfulness.py
python inspect_answer_correctness.py
python inspect_context_precision.py
python inspect_context_recall.py
python inspect_answer_relevancy.py

Notes

• Keep real API keys in .env; never commit them.
• Keep private datasets in eval_data/ or ignored JSON/CSV files.
• Use .env.example as the safe template for required environment variables.

License

MIT License