Search before asking
Description
Introduce optional RPC micro-batching for point queries to reduce FE-to-BE RPC overhead under high concurrency, especially when data is already cached.
The FE combines concurrent requests targeting the same BE into one RPC. A batch is flushed when its size limit or a short waiting window is reached, with a cap on total request bytes. The BE
processes each lookup and returns individual results.
Batching requires no application or SQL changes and preserves per-query results, errors, timeout budgets, and cancellation behavior. It is disabled by default, with configurable batch limits
and waiting time to balance throughput against latency.
Evaluate performance using throughput, CPU consumption, RPCs per query, and P99 latency across different concurrency levels, including the waiting overhead under light load.
Use case
- User-profile reads: Many concurrent requests fetch small user records by primary key, making RPC overhead a significant part of query cost.
- Order-status checks: Independent client connections repeatedly look up individual orders, creating opportunities to batch requests targeting the same BE.
- Real-time lookup services: Applications issue individual point queries and benefit from reduced RPC overhead without implementing application-side batching.
Related issues
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
Introduce optional RPC micro-batching for point queries to reduce FE-to-BE RPC overhead under high concurrency, especially when data is already cached.
The FE combines concurrent requests targeting the same BE into one RPC. A batch is flushed when its size limit or a short waiting window is reached, with a cap on total request bytes. The BE
processes each lookup and returns individual results.
Batching requires no application or SQL changes and preserves per-query results, errors, timeout budgets, and cancellation behavior. It is disabled by default, with configurable batch limits
and waiting time to balance throughput against latency.
Evaluate performance using throughput, CPU consumption, RPCs per query, and P99 latency across different concurrency levels, including the waiting overhead under light load.
Use case
Related issues
No response
Are you willing to submit PR?
Code of Conduct