Redis connection pool exhausted in Hyperf under traffic spikes
We run Hyperf 3.1 on Swoole with 12 workers. Under normal load everything is fine, but during spikes we start getting Connection pool exhausted. Cannot establish new connection before wait_timeout from the Redis pool. The pool in config/autoload/redis.php is set to min 5 and max 50, wait_timeout 3 seconds.
Each request fires several Redis calls and some of them happen inside nested coroutines, so I suspect we hold connections longer than the pool recycles them. How do you size the pool relative to worker count and real concurrency, and is there a sane way to see how many connections are actually checked out at runtime?
The pool is per worker, not global. So your real ceiling is 50 per worker times 12 workers, which is 600 concurrent Redis connections before anyone has to wait. If you hit exhaustion well below that, you are holding connections, not running out of them. A connection is checked out from the moment the proxy hands it to your coroutine until that coroutine finishes the call and yields it back.
Classic cause: you GET from Redis, then do a slow curl call to some external API, then SET, all in the same logical block. The connection stays checked out for the entire duration of that curl, which might be 800ms. Fetch what you need, release, do the slow work, then grab a fresh connection for the write. Do not straddle a blocking await with a held connection.
One thing people miss: a pipeline or a multi/exec transaction holds a single connection for the whole batch. If you build a 10k command pipeline and flush once, that connection is gone for the duration. Keep pipelines bounded and flush in chunks. Same with blocking commands like BLPOP, never run those on a pooled connection or you starve everyone.
Found it thanks to the curl tip. We had a cache-aside helper that did Redis GET, then an HTTP enrichment call on miss, then SET, all in one function with the connection implicitly alive the whole time. Splitting the external call out of the Redis scope dropped checked-out connections under load by a huge margin. wait_timeout errors are gone now.
For visibility, there is no built-in gauge but you can subclass the pool or just log when getConnection waits longer than a few ms. Raising max_connections only buys you time if you are leaking, the wall just moves. Fix the hold duration first, then size the pool to peak concurrency with maybe 20 percent headroom.
```php blocks are runnable.