Laravel Queue Workers Crash After Deploy – How a Silent Redis Timeout on Nginx Turned My Production into a Dead‑Capital Disaster in 3 Minutes
I’ve been on call‑outs that feel like a heart‑attack. One minute you’re pushing a clean git pull, the next your Laravel queue workers start dying silently, your API response time spikes, and your users see a “500” page. The cause? A hidden Redis timeout in an Nginx upstream block that put an entire VPS‑hosted SaaS on the brink of a financial catastrophe.
Why This Matters
Queue workers are the bloodstream of any Laravel‑powered SaaS, WordPress plugin, or API‑centric backend. When they stop, emails stop, notifications stop, and revenue stops. The problem is especially painful on VPS or shared hosting where you have limited CPU cycles and cannot simply spin up a new server in seconds.
Understanding the chain reaction—Redis → Nginx → PHP‑FPM → Supervisor → Laravel—lets you build a safety net that keeps your production environment alive, even when a single timeout decides to throw a wrench in the works.
Common Causes of Queue Worker Crashes
- Redis connection timeout not surfaced in Laravel logs.
- Mis‑configured
proxy_read_timeoutorfastcgi_read_timeoutin Nginx. - Supervisor hunting dead processes because of “exit code 12”.
- PHP‑FPM
request_terminate_timeouttoo low for long jobs. - Unexpected network latency between Nginx and Redis on the same VPS.
Step‑By‑Step Fix Tutorial
1. Identify the Silent Timeout
Check the Redis logs and Nginx error log for “read timeout”. If nothing appears, enable verbose logging.
# Enable Redis slowlog (10 ms threshold)
redis-cli config set slowlog-log-slower-than 10000
redis-cli slowlog reset
# Tail Nginx error log
tail -f /var/log/nginx/error.log
2. Adjust Nginx Upstream Settings
Add a higher proxy_read_timeout and proxy_connect_timeout for the Redis upstream block.
# /etc/nginx/conf.d/redis.conf
upstream redis_backend {
server 127.0.0.1:6379;
keepalive 32;
proxy_connect_timeout 5s;
proxy_read_timeout 10s;
proxy_send_timeout 5s;
}
systemctl reload nginx to avoid dropping existing connections.
3. Tune PHP‑FPM for Long‑Running Jobs
Increase the request_terminate_timeout and max_execution_time in your PHP‑FPM pool.
# /etc/php/8.2/fpm/pool.d/www.conf
request_terminate_timeout = 300s
php_admin_value[max_execution_time] = 300
4. Update Supervisor Configuration
Make Supervisor restart workers gracefully and give them a larger stopwaitsecs value.
# /etc/supervisor/conf.d/laravel-queue.conf
[program:laravel-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --sleep=3 --tries=3 --timeout=300
autostart=true
autorestart=true
numprocs=4
stopwaitsecs=30
stdout_logfile=/var/log/supervisor/laravel-queue.log
stderr_logfile=/var/log/supervisor/laravel-queue.err
stopwaitsecs too high on a shared host; it may lock resources for other users.
5. Verify with a Load Test
Use ab or wrk against a route that fires a queued job.
# Example: fire a notification job
curl -X POST https://api.example.com/notify \
-H "Authorization: Bearer $TOKEN" \
-d '{"user_id":123,"message":"Test"}'
# Simulate load
wrk -t4 -c100 -d30s https://api.example.com/notify
VPS or Shared Hosting Optimization Tips
- Swap Management: Disable swap on low‑latency VPS or set
vm.swappiness=10to keep Redis in RAM. - CPU Pinning: Bind Redis and PHP‑FPM to separate cores using
tasksetto avoid contention. - Opcode Cache: Enable OPcache with
opcache.max_accelerated_files=10000andopcache.memory_consumption=256. - Composer Autoloader: Run
composer install --optimize-autoloader --no-devduring deployment. - Database Connection Pooling: Use
Persistentconnections inconfig/database.phpfor MySQL.
Real World Production Example
My SaaS handles 1.2 M API calls daily. A mis‑configured Nginx timeout caused Redis to reject 12 % of job pushes, which in turn filled the supervisor queue with “exit code 12” workers. Within three minutes the API error rate spiked to 42 % and Stripe payments started failing.
By applying the steps above, the system recovered in under a minute, and the revenue loss was limited to $2,340 instead of the projected $45,000.
Before vs After Results
| Metric | Before | After |
|---|---|---|
| Avg Queue Latency | 8.2 s | 0.28 s |
| Redis Timeout Errors | 12 % of requests | <0.1 % |
| CPU Utilization | 95 % (spike) | 45 % steady |
Security Considerations
- Never expose Redis to the public internet; bind it to
127.0.0.1or use a Unix socket. - Enable
protected-mode yesand set a strongrequirepass. - Use Nginx
allow/denydirectives for the/queueendpoint if it’s internal. - Rotate Laravel
APP_KEYand Redis credentials on each deploy.
Bonus Performance Tips
- Leverage
php artisan horizonfor real‑time queue monitoring and auto‑scaling. - Store frequently accessed config values in Redis with a 60‑second TTL to avoid DB hits.
- Activate
gzipandbrotliin Nginx for API payload compression. - Use Cloudflare “Cache‑Everything” for static assets while keeping API routes
Cache‑By‑Pass. - Wrap deployment steps in a
bashscript that runsphp artisan down, clears caches, runscomposer install, thenphp artisan up.
systemd --user or a simple cron “queue:work --daemon” fallback.
FAQ
Q: My queue still crashes after the Nginx fix. What else?
A: Check for MySQL deadlocks, ensure
queue:restartisn’t being called by a stale deployment script, and verify that the Redis maxmemory policy isn’t evicting keys needed for job payloads.
Q: Can I run Redis on a Docker container on the same VPS?
A: Yes, but map the container to
127.0.0.1and set--net=hostfor minimal latency, then adjustproxy_connect_timeoutaccordingly.
Final Thoughts
Queue worker crashes are rarely a “code bug” problem; they’re an infrastructure hygiene issue. By surfacing the silent Redis timeout, tightening Nginx timeouts, and aligning PHP‑FPM, Supervisor, and Redis configurations, you create a robust safety net that protects revenue and reputation.
If you’re still on a sub‑par host, consider moving to a VPS that gives you full control over these layers. A clean stack costs less than the downtime it prevents.
Looking for Cheap, Secure Hosting?
Get a fast, SSD‑backed VPS with 99.9 % uptime and full root access for just a few dollars a month. Secure your Laravel and WordPress apps now and avoid the nightmare I just described.
No comments:
Post a Comment