Implement Proper Chat Memory for User Sessions

[brylie]

Objective

Develop a more robust and persistent chat memory system that maintains separate chat histories for individual user sessions while still allowing anonymous access.

Why

• Improve user experience with persistent chat history
• Ensure privacy and separation of chat contexts between users
• Allow for better management and potential analysis of chat data
• Prepare for future scaling and feature additions

Description

We need to replace the current simple Python list-based chat memory with a more sophisticated system that can maintain separate, persistent chat histories for individual user sessions. This system should work for anonymous users, allow for easy cleanup of old sessions, and be simple to deploy and maintain.

Tasks

1. Design the chat memory data model:
   • Define structure for storing chat messages (user input, bot responses, timestamps, etc.)
   • Create a session model to group chat messages
2. Choose and implement a storage solution:
   • Set up an in-memory database (e.g., Redis) or SQLite database
   • Implement data access layer for CRUD operations on chat sessions and messages
3. Implement session management:
   • Create a mechanism for generating and managing session IDs for anonymous users
   • Implement session creation, retrieval, and expiration logic
4. Integrate the new chat memory system with the existing chat interface:
   • Update the chat endpoint to use the new storage system
   • Modify the chat history retrieval to work with the new system
5. Implement a cleanup mechanism:
   • Create a process to regularly remove expired sessions and old chat data
   • Set up a configurable retention period for chat history
6. Add basic analytics capabilities:
   • Implement counters for total sessions, messages, etc.
   • Create an admin endpoint for viewing basic usage statistics
7. Ensure the new system works with existing features (e.g., RAG, token limiting)

Technical Considerations

• Use async-compatible database libraries to maintain performance
• Ensure the chosen storage solution can be easily deployed (e.g., as part of a Docker container)
• Implement proper error handling and logging for database operations
• Consider encryption for storing sensitive chat data

Acceptance Criteria

• Chat histories are correctly maintained and separated for different user sessions
• Anonymous users can access the service and maintain a session-based chat history
• The system correctly expires and cleans up old sessions and chat data
• Performance impact is minimal compared to the current in-memory list approach
• The solution can be easily deployed as part of the existing application (e.g., via Docker)
• Basic usage statistics are available through an admin interface

Additional Notes

• While initially focusing on simplicity, design the system to be extensible for future improvements (e.g., user accounts, more advanced analytics)
• Document the new chat memory system thoroughly, including any new environment variables or configuration options
• Consider GDPR and other privacy regulations in the design and implementation