2024-02-27 (low latency) i7-12700 Linux 4.18.0 (Hyper-V VM)

[Radrik5]

When the number of concurrent requests exceeds the number of server CPUs, the requests start waiting in a queue which increases the latency significantly. 2ms latency was too large in my use case, so I was interested in latency numbers without queuing.

-----------------------------------------------------------------------------------------------------------------------------------------
| name                        |   req/s |   avg. latency |        90 % in |        95 % in |        99 % in | avg. cpu |   avg. memory |
-----------------------------------------------------------------------------------------------------------------------------------------
| cpp_grpc_mt                 |    5090 |        0.18 ms |        0.20 ms |        0.21 ms |        0.26 ms |   29.34% |     27.92 MiB |
| cpp_asio_grpc_callback      |    5066 |        0.18 ms |        0.20 ms |        0.21 ms |        0.26 ms |   28.66% |     27.93 MiB |
| cpp_grpc_st                 |    5010 |        0.18 ms |        0.20 ms |        0.22 ms |        0.27 ms |   28.98% |     26.05 MiB |
| cpp_grpc_callback           |    4704 |        0.19 ms |        0.22 ms |        0.24 ms |        0.29 ms |   61.44% |     68.96 MiB |
| dotnet_grpc                 |    4231 |        0.22 ms |        0.28 ms |        0.30 ms |        0.38 ms |   77.25% |    100.36 MiB |
-----------------------------------------------------------------------------------------------------------------------------------------
Benchmark Execution Parameters:
b2e17fd Fri, 28 Apr 2023 14:41:51 +0200 Dennis Hezel feat: Add client-streaming benchmark for asio-grpc callback and coroutine
- GRPC_BENCHMARK_DURATION=20s
- GRPC_BENCHMARK_WARMUP=5s
- GRPC_SERVER_CPUS=1
- GRPC_SERVER_RAM=512m
- GRPC_CLIENT_CONNECTIONS=1
- GRPC_CLIENT_CONCURRENCY=1
- GRPC_CLIENT_QPS=0
- GRPC_CLIENT_CPUS=1
- GRPC_REQUEST_SCENARIO=complex_proto
- GRPC_USE_CLIENT_STREAMING=false

The results correspond to ping-pong median latency reported here: https://grafana-dot-grpc-testing.appspot.com/?orgId=1 (from https://grpc.io/docs/guides/benchmarking/)

Just for the reference: epoll() system call latency on my machine is 70 microseconds, two epoll() calls (on the sever and the client) are 140 microseconds, so gRPC overhead is about 40-80 microseconds in cpp implementations.

[LesnyRumcajs]

Just for the reference: epoll() system call latency on my machine is 70 microseconds, two epoll() calls (on the sever and the client) are 140 microseconds, so gRPC overhead is about 40-80 microseconds in cpp implementations.

That's a very interesting find, thanks!