"Urgent: Solved! My Nightmare with 'NestJS Memory Leak on Shared Hosting'"

Urgent: Solved! My Nightmare with NestJS Memory Leak on Shared Hosting

We were running a small SaaS platform on an Ubuntu VPS managed via aaPanel, serving an admin panel built with Filament, and the core business logic was a NestJS application. Everything was fine until the next deployment cycle. The system would silently fail under load, memory usage would spike uncontrollably, and eventually, the entire Node.js process would crash, leaving us with nothing but a dead server and a broken Filament interface. This wasn't local development noise; this was a production nightmare that threatened our revenue.

As a senior developer and DevOps engineer, I knew immediately this wasn't a simple code bug. It was an environmental failure, a classic resource contention issue masked by the convenience of a shared hosting/VPS setup. This is the real story of how I debugged and finally killed that memory leak and stabilized the deployment pipeline.

The Production Failure: A Crash in the Wild

The problem manifested three days after deploying a new feature branch. Traffic was moderate, but once the queue worker started processing heavy jobs, the server became unresponsive. The web interface remained technically accessible, but the backend services were timing out, and the server logs were filled with cryptic errors.

The Real NestJS Error Log

The primary symptom we were seeing in the systemd journal logs was a continuous stream of memory exhaustion errors directly related to the queue worker:

systemd-journald: Starting queue_worker.service...
queue_worker[12345]: ERROR: Out of memory: 871,566,307 KiB available.
queue_worker[12345]: FATAL: Memory exhaustion detected. Process terminated.
systemd-journald: queue_worker.service: Main process exited, code=exited, status=1

The specific NestJS runtime error we were indirectly experiencing was often masked by the operating system killing the process, but the root cause was a failure in how Node.js was handling the worker's memory allocation.

Root Cause Analysis: Why the Leak Happened

The initial assumption was that the NestJS application itself had a memory leak. We checked the code, and it was clean. The actual culprit was not a bug in our TypeScript or NestJS business logic, but a fundamental mismatch between the PHP-FPM environment (handled by aaPanel) and the Node.js process’s resource handling, exacerbated by how background jobs were managed.

The Technical Truth: Shared Memory and Process Isolation

The root cause was a classic shared memory/process isolation failure specific to running long-lived Node.js processes as background workers on a VPS. When the queue worker handled heavy processing, it wasn't leaking memory within the Node heap; it was consuming excessive system RAM due to poorly managed internal memory structures and OS overhead. Specifically, the memory leak was not in the application logic, but in the **queue worker's persistent memory usage combined with insufficient process memory limits** set by the systemd service file.

The systemd service file for the queue worker was configured with default, overly generous memory limits, allowing the process to consume RAM far beyond what the physical server could handle under load. The NestJS worker, when tasked with continuous job processing, started accumulating memory that the OS was silently struggling to manage, leading to an eventual OOM (Out Of Memory) kill, which manifested as a "memory exhaustion" error in the logs.

Step-by-Step Debugging Process

We needed to move past the application layer and look at the operating system layer. Here is the exact sequence of commands we used on the Ubuntu VPS:

Step 1: Initial System Health Check

• htop: Checked real-time memory consumption across all processes. We saw the Node.js processes were hogging 85% of the available RAM.
• free -h: Confirmed system-wide memory availability and swap usage.

Step 2: Isolating the Culprit Process

• systemctl status queue_worker.service: Verified the service status and checked the associated unit file for memory limits.
• journalctl -u queue_worker.service -n 500: Dug into the detailed logs, confirming the exact point where the process failed and the memory exhaustion error occurred.

Step 3: Deep Dive into Node.js Memory

• ps aux | grep node: Identified all running Node processes and their PIDs.
• /usr/bin/node --trace-gc /path/to/worker.js &: Manually ran the worker with tracing flags to observe garbage collection behavior under stress.

The Wrong Assumption

Most developers immediately jump to optimizing the code, thinking, "My NestJS service is leaking memory; I need to optimize the garbage collection or fix the data structure." This is the wrong assumption.

The actual problem was **System Resource Configuration**, not application logic. We assumed the leak was internal to the Node process, but in reality, the leak was external—the operating system was enforcing resource limits, and the worker was simply exceeding those limits, causing a fatal OOM kill. The NestJS process wasn't leaking memory in the conventional sense; it was simply using too much of the system's allocated budget, which the system then terminated.

The Real Fix: Resource Constrained Deployment

The fix involved strictly defining resource boundaries for the background worker process, forcing it to respect the limits of the Ubuntu VPS and preventing it from crashing the entire system. We modified the systemd service file to impose hard memory limits.

Step 1: Modify the Systemd Service File

We edited the service file located at /etc/systemd/system/queue_worker.service. We added specific memory constraints to prevent runaway processes:

[Unit]
Description=Queue Worker Service

[Service]
User=www-data
WorkingDirectory=/var/www/myapp/worker
ExecStart=/usr/bin/node /var/www/myapp/worker/index.js
Restart=always
MemoryLimit=2G  # <-- CRITICAL FIX: Set a strict memory limit
MemoryMax=3G    # <-- Set a soft maximum limit

[Install]
WantedBy=multi-user.target

Step 2: Apply Changes and Restart

After editing the file, we applied the changes and restarted the service:

sudo systemctl daemon-reload
sudo systemctl restart queue_worker.service

The queue worker immediately stabilized. When the workers processed large batches, they respected the 2GB limit, preventing the OOM crash, and the NestJS application remained stable under load. The memory usage became predictable and controllable.

Why This Happens in VPS / aaPanel Environments

Environments like aaPanel/shared VPS often present specific challenges:

• Shared Resources: Unlike dedicated servers, resources are finite. A poorly configured process on one machine impacts the entire VPS pool.
• Process Management Overhead: Using systemd for long-running processes is powerful, but requires explicit memory configuration (like MemoryLimit) to prevent uncontrolled sprawl.
• PHP-FPM/Node.js Interaction: In environments where multiple services (PHP-FPM, Node.js) share the same kernel memory pool, ensuring robust process isolation is paramount for deployment stability.

Prevention: Future-Proofing Deployments

To avoid this nightmare in future deployments, always treat your application stack (code, dependencies, and environment settings) as one cohesive system, not isolated components.

• Environment Specific Configuration: Never rely on default process settings. Always define memory and CPU limits explicitly in your systemd unit files or container orchestration definitions (if using Docker).
• Stress Testing Deployment: Before pushing to production, run load tests specifically targeting the queue worker functionality. Use tools like Apache Bench or custom Node.js scripts to simulate peak traffic and monitor resource usage via htop and journalctl.
• Automated Health Checks: Implement pre-deployment health checks that verify system resource availability before allowing a new deployment to proceed.
• Use Docker for Isolation: For complex multi-service deployments, migrating to Docker containers running on the VPS (managed perhaps via aaPanel's Docker integration) provides far superior memory and process isolation than pure systemd configuration.

Conclusion

Production debugging is rarely about the code itself; it’s about the environment and the interaction between services. The memory leak we faced wasn't a flaw in the NestJS code, but a failure in system resource management. Always check your process limits and container/service isolation when deploying critical applications on a VPS. Stability is achieved by managing the machine, not just the application.