Queue driver comparison: database vs Redis vs SQS
About to move from database queue driver to something faster. Running about 2000 jobs/minute at peak.
Comparing Redis (local), SQS (AWS), and Beanstalkd. What drives the choice in practice?
Database queue is fine under 500 jobs/minute but starts adding DB load at higher volumes. At 2000/min with multiple workers polling, the jobs table becomes a hot spot.
Redis is the standard choice for self-hosted setups. Fast, reliable, native priority queue support, and Horizon works with it. Single point of failure is the main concern: Redis goes down, queue stops.
SQS if you are on AWS and need durability. SQS does not lose messages, handles traffic spikes without configuration, and scales automatically. Latency is higher than Redis (~50ms vs ~1ms) but for most background jobs this does not matter.
One SQS gotcha: visibility timeout. If a job takes longer than the timeout, SQS makes it visible again and another worker picks it up. Duplicate processing is possible. Design jobs to be idempotent.
Beanstalkd is rarely the right choice in 2024 unless you already have it running. Redis is faster, better supported, and has Horizon.
We run Redis for the default and high-priority queues, SQS for low-priority background jobs that can tolerate loss. Redis failure does not stop the critical path.
```php blocks are runnable.