fix: correct reranker score display and add URL content fetching

[suneel944]

Description

This PR fixes the reranker score display issue in the dashboard and adds URL content fetching to improve reranker accuracy for real chat interactions. The reranker score was incorrectly displaying as a percentage (e.g., 61.36) instead of a 0-1 decimal (e.g., 0.61). Additionally, the framework now fetches content from URLs to provide richer context for reranker scoring, significantly improving accuracy when sources are URLs rather than document text.

Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• ✨ New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📝 Documentation update
• ⚡ Performance improvement
• ♻️ Code refactoring (no functional changes)
• 🧪 Test improvements
• 🔧 Build/config changes

Changes Made

• Fixed reranker score display: Removed reranker_score from percentage_metrics list in collectors.py and data_store.py to prevent incorrect conversion to 0-100 percentage range
• Added URL content fetching: New url_fetcher.py utility with parallel fetching, retry logic, and support for HTML, PDF, JSON, and plain text content
• Improved reranker accuracy: Integrated URL content fetching into RAG metrics calculator to use actual document content instead of URLs for scoring
• Fixed RAG context storage: Updated rag_tester.py to prioritize test-provided retrieved_docs over extracted URLs
• Fixed reranker model loading: Corrected _load_model() check to properly detect transformers-based models
• Source extraction: Enhanced chat_page.py to extract sources from both /api/chat/completed and async polling endpoints
• Query retrieval: Fixed query extraction from RAG context for reranker score calculation
• Removed debug logs: Cleaned up all console.log debug statements from dashboard JavaScript
• Updated documentation: Clarified reranker score display format and documented URL fetching feature

Testing

Test Coverage

• Unit tests added/updated
• Integration tests added/updated
• E2E tests added/updated
• UI tests added/updated (if applicable)
• Security tests added/updated (if applicable)
• Manual testing performed

Test Results

# RAG metrics tests pass with URL fetching enabled
pytest tests/e2e/ai/test_rag_metrics.py -v
# Results: All RAG metric tests pass, reranker_score now displays correctly as 0-1 decimal

Manual Testing:

• Verified reranker score displays as 0.61 (not 61.36) in dashboard
• Confirmed URL content fetching works for HTML, PDF, and JSON content
• Tested parallel fetching with multiple URLs
• Verified retry logic handles failed requests gracefully
• Confirmed RAG context properly stores test data and extracted sources

Code Quality

Pre-Submission Checks

• ✅ make check passes (linting + type checking)
• ✅ make format has been run
• ✅ make test-all passes
• ✅ All pre-commit hooks pass

Code Standards

• Code follows PEP 8 style guidelines
• Type hints added/updated for all functions
• Docstrings added/updated (Google style)
• No new warnings or errors generated
• Imports are organized correctly
• Self-review completed

Documentation

• Code comments added for complex logic
• README.md updated (if needed)
• API documentation updated (if needed)
• Architecture Decision Record (ADR) created (if significant change)

Related Issues

Fixes reranker score display issue where values > 1 were appearing (e.g., 61.36 instead of 0.61)

Additional Context

Screenshots/Demo

Before: Reranker score displayed as 61.36 (incorrect percentage)
After: Reranker score displays as 0.61 (correct 0-1 decimal)

Breaking Changes

None. This is a bug fix and feature addition that maintains backward compatibility.

Performance Impact

• URL Content Fetching: Uses parallel fetching with configurable worker pool (default: 5 workers) for improved performance
• Retry Logic: Exponential backoff (1s, 2s, 4s) prevents overwhelming servers while ensuring reliability
• Content Length Limits: Configurable max content length (default: 10,000 chars) prevents memory issues

Security Considerations

• URL fetching includes timeout protection (default: 10 seconds) to prevent hanging requests
• Content length limits prevent DoS attacks via large responses
• Error handling gracefully handles malicious or malformed URLs

Configuration

New environment variables added:

# Enable/disable URL content fetching (default: true)
RAG_FETCH_URL_CONTENT=true

# Timeout per URL in seconds (default: 10)
RAG_URL_FETCH_TIMEOUT=10

# Maximum content length per URL in characters (default: 10000)
RAG_URL_MAX_CONTENT_LENGTH=10000

# Maximum retry attempts per URL if fetch fails (default: 3)
RAG_URL_MAX_RETRIES=3

# Delay between retries in seconds, uses exponential backoff (default: 1.0)
RAG_URL_RETRY_DELAY=1.0

# Maximum parallel workers for URL fetching (default: 5)
RAG_URL_MAX_WORKERS=5

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published
• I have checked that my changes don't break existing functionality