Implement Token Limiting for User Interactions

[brylie]

Objective

Implement a token limiting system to restrict the amount of tokens consumed per user, both for input and output, to promote sustainability and fair resource sharing.

Why

• Promote sustainability
• Ensure fair resource sharing
• Control costs associated with token usage
• Allow the chat interface to benefit many users equally

Description

We need to implement a token limiting system that tracks and restricts the number of tokens used by each user in their interactions. This includes limiting input tokens (form input) and tracking output tokens, providing users with a token balance for their current session or time period.

Tasks

1. Implement input token limiting:
   • Add a token counting function for user input
   • Set a maximum token limit for input forms
   • Provide real-time feedback on token usage in the UI
2. Implement output token tracking:
   • Create a session-based token counter for output
   • Update the counter after each API response
   • Display remaining token balance to the user
3. Set up a token allocation system:
   • Define token limits per user per time period (e.g., 1000 tokens per hour)
   • Implement a mechanism to reset token allocations periodically
4. Create a token usage database or cache:
   • Store token usage data for each user session
   • Implement cleanup for expired sessions
5. Integrate token limiting with the chat interface:
   • Display warnings when approaching token limits
   • Prevent submissions when token limit is reached
6. Implement admin controls for token limit management.

Technical Considerations

• Ideally, we would use token counts provided in the embedding and generation responses
• Optionally, use a tokenizer compatible with our LLM (e.g., tiktoken for OpenAI models)
• Ensure token counting is efficient and doesn't significantly impact performance
• Consider using Redis or a similar in-memory store for tracking token usage

Acceptance Criteria

• Input forms accurately limit and display token usage
• Output token usage is correctly tracked and displayed to users
• Token allocations reset correctly based on the defined time period
• Users receive clear notifications about their token usage and limits
• The system prevents users from exceeding their token allocations
• Admin interface allows for easy management of token limits

Additional Notes

• Consider implementing different token allocation tiers for different user types in the future
• Ensure the token limiting system is scalable and can handle concurrent users
• Document the token limiting behavior clearly for users