Major Codebase Refactoring and Search Enhancement

[OriPekelman]

Overview

This pull request implements a comprehensive refactoring of the codebase, focusing on search functionality, template organization, and modern JavaScript practices. The changes improve maintainability, performance, and user experience.

Key Changes

• Performance imrpovement across the board both for ingestion and at runtime
• Implemented MeiliSearch integration for full-text search (keeping the existing search engine as fallback)
• A lot of dead code removal (more can be done if we decide to only keep the new search experience).
• Refactored most JS to modern JavaScript standards
• Enhanced faceted search with improved URL handling and UI
• Implemented pagination for search results
• Added virtual keyboard integration using Keyman
• Centralized CSS and removed deprecated visualization files
• Improved text metadata handling
• Ingestion is now a single step robust process.
• Takes much less space on disk
• HTML generation now the responsibility of the app server
• Caching
• Added some unit tests.

Technical Details

Search Implementation

• Integrated MeiliSearch for robust full-text search capabilities both for full text and meta-data.
• Added search in English translation functionality
• Model now has a to_json method used for search engine indexing.
• Indexing step is separate from ingestion step
• Implemented search result highlighting and context
• Enhanced search results explanation with interactive details
• Added support for searching in meta fields
• Optimized indexing performance
• Implemented a "Preferences" modal to control fuzzy search and virtual keyboard

JavaScript Modernization

• Rewritten wikipedia.js for better debugging and HTTPS support
• Removed jQuery dependency across the application
• Improved error handling for empty search results
• Added HTTPS support for all external integrations

Template & UI Improvements

• Enhanced faceted search with improved URL handling
• Implemented consistent pagination layout
• Added interactive search result explanations
• Integrated virtual keyboard for Coptic text input
• Improved visualization bar inclusion

Testing

Please verify the following functionality:

• Full-text search works across all document types
• Faceted search filters operate correctly
• Pagination works on all search result pages
• Virtual keyboard functions in all input fields
• Wikipedia integration works with HTTPS
• Search works with both Coptic and English text
• Meta field search returns accurate results

Breaking Changes

• Deprecated API
• Changed URL structure for faceted search - though old style urls still work.
• Modified text metadata handling structure - though this is all internal.

Additional Notes

• It would be preferred to use uv rather than conda or PIP for installing and running this.
• MeiliSearch configuration may need environment-specific adjustments - you will need a recent version at any rate.
• Cache clearing might be required after deployment
• Virtual keyboard configuration may need locale-specific testing

Related Changes

• Removed django-grappelli dependency
• Updated to latest Django LTS version

Performance Considerations

• Added caching for text retrieval
• Optimized search indexing
• Improved pagination handling
• Enhanced template inheritance efficiency

Documentation

• Updated README with search implementation details
• Added documentation for MeiliSearch integration
• Updated virtual keyboard setup instructions

Related Issues

• This should also close all current issues.