[PERF] Add Redis connection pooling for high-throughput public API endpoints to reduce connection overhead

[Abhushan187]

Problem

The public API currently creates a new Redis connection per request for rate limiting, lockout checks, and caching. Under high load (1000+ concurrent requests), this causes connection exhaustion, increased latency, and potential Redis ECONNREFUSED errors. The existing ioredis or redis client is likely instantiated without pooling.

---

Proposed Improvement

Implement a Redis connection pool for all public API endpoints that interact with Redis (rate limiting, lockout, caching, analytics). The feature should:

• Use a single shared Redis connection pool (max 20 connections, min 5) across the public API
• Apply pooling to checkLockout, recordFailedAttempt, clearLockout, and analytics counters
• Add connection health checks and automatic reconnection with jittered backoff
• Expose pool metrics (active connections, wait queue size, total commands) via GET /api/health/redis
• Reduce P99 latency for auth endpoints by eliminating per-request connection overhead

---

Expected Impact

• Better performance under high concurrency
• Reduced Redis server load and connection count
• Improved reliability with connection health monitoring
• Better long-term scalability for production deployments

---

Possible Implementation

• Add RedisPool class in packages/common/redis/ using ioredis cluster mode or generic-pool
• Refactor apps/public-api/middleware/rateLimiter.js to use pooled connections
• Refactor apps/public-api/services/lockout.js (checkLockout/recordFailedAttempt/clearLockout)
• Add redisPoolMetrics middleware exposing pool stats
• Update docker-compose.yml to include Redis with maxclients config
• Write load tests using autocannon or k6 to verify latency improvement
• Add resilience tests for pool exhaustion and connection failure scenarios

---

I'm GSSoC'26 contributor, Please assign this task to me!

[Nitin-kumar-yadav1307]

@Abhushan187 I reviewed the code and have some clarifications plus a suggested scope for a PR.
We already use a single shared ioredis client (see redis.js), and our login lockout helpers call that client. So the lockout code does NOT create a new Redis connection per request. but u can :

• Replace the in-memory projectRateLimiter with a Redis-backed limiter so rate limits work across multiple instances.

• Improve Redis resilience: exponential+jitter backoff, better retry logic in redis.js.

• Add a small GET /api/health/redis endpoint that reports ioredis status and minimal INFO-derived metrics (connected_clients, used_memory, total_commands_processed).

• Expose pool/connection metrics and small operational docs (docker-compose Redis maxclients guidance).
  Add load tests (autocannon or k6) and resilience tests demonstrating behavior under connection failures.

[Abhushan187]

/assign

[Nitin-kumar-yadav1307]

sure

[yash-pouranik]

waiting @Abhushan187