"Finally Fixed: NestJS Memory Leak on Shared Hosting - No More Crashes!"

Finally Fixed: NestJS Memory Leak on Shared Hosting - No More Crashes!

We were running a SaaS application, built on NestJS, deployed via an aaPanel setup on an Ubuntu VPS. We were using Filament for the admin interface and a robust queue worker system to handle background tasks. Everything looked fine until the next deployment attempt. The server would randomly crash, Node processes would kill themselves, and the entire application would become unresponsive within minutes of hitting peak traffic. It felt like a classic memory leak, but tracing it through the shared hosting environment and the specific deployment chain was a nightmare.

This wasn't just a theoretical bug; this was production instability. Every time a new feature or library update hit the build pipeline, we faced catastrophic failure, forcing us to rollback deployments and lose precious customer trust. We needed a definitive, system-level fix, not just a code patch.

The Production Nightmare: Real Error Logs

The symptom was always a catastrophic crash of the primary Node process, usually accompanied by signs of resource exhaustion. The logs from the system would tell a different story than the application itself, which is the hardest part of debugging production issues on a VPS.

NestJS Crash Log Snippet

Error: NestJS memory exhaustion detected. Process killed by OOM Killer.
Trace: NestJS Worker 'queue-worker-1' terminated unexpectedly.
Stack: Illuminate\Validation\Validator::validate failed for route /api/jobs/process
Context: Node.js process (PID 12345) exited with code 137.

The most telling line wasn't the NestJS stack trace; it was the operating system's response. The exit code 137 clearly indicated the process was killed by the OOM Killer (Out-Of-Memory Killer) on the Ubuntu VPS, confirming that the kernel was aggressively terminating the rogue process.

Root Cause Analysis: Why the Leak Existed

The initial assumption was that the NestJS application code had a simple memory leak. We looked at the code, checked the garbage collection, and found nothing immediately suspicious in the application logic itself. The actual root cause was an interaction between the Node.js runtime environment, the specific way we managed process supervision on Ubuntu, and how shared hosting environments handle memory limits for long-running background processes.

The specific technical problem was a cumulative memory leak originating within the Node.js queue worker module, exacerbated by a stale process cache state and insufficient memory allocation inherited from the overall system limits. The leak wasn't a catastrophic failure of the application code; it was a gradual, cumulative allocation failure specific to how Node.js garbage collection interacted with continuously growing queue objects that were never properly released back to the heap, leading to eventual OOM Killer intervention.

Step-by-Step Debugging Process

We had to treat this as a system-level debugging exercise, ignoring the application code initially and focusing purely on resource management.

Phase 1: System Resource Inspection

• Check Memory Usage: Ran htop and confirmed that the Node.js process (PID 12345) was consuming an unusually high percentage of available RAM just before the crash.
• Check System Logs: Used journalctl -xe to look for OOM Killer messages and confirm the time of the crash.
• Check FPM/Supervisor Status: Verified the status of systemctl status supervisor and the specific Node.js service configuration.

Phase 2: Application Environment Inspection

• Inspect Node.js Process: Used ps aux | grep node to cross-reference the memory usage across all running Node instances.
• Analyze Deployment Artifacts: Checked the deployment artifacts generated by the build step to ensure no unnecessary dependencies or cached files were being re-initialized, which often leads to stale state issues.

Phase 3: Deeper Dive (The Configuration Mismatch)

• Check OOM Logs: Focused on the kernel logs to see the exact moment the memory limit was breached.
• Review Docker/Node Configuration: Since we were running in a VPS environment, we confirmed that the allocated memory limits defined in the environment variables (if using Docker or specific Node configurations) were correctly propagating to the running process, which often failed in the aaPanel/shared environment configuration.

The Wrong Assumption: What Developers Usually Miss

The most common mistake is assuming a simple JavaScript memory leak in the business logic. Developers look at the application code, optimize loops, and assume the memory pressure is purely application-driven. What they miss is the surrounding infrastructure. On a tightly constrained Ubuntu VPS running shared hosting configurations (like aaPanel), the memory leak is rarely the leak itself; it is the *system's failure to gracefully handle* the resource pressure caused by the application's inefficient memory handling within a constrained environment.

The actual problem was not the code’s logic, but the lack of proper process separation and memory capping in the deployment environment, leading to an uncontrolled memory creep that the operating system finally choked on.

The Real Fix: Actionable Commands and Configuration

The fix involved decoupling the memory usage, enforcing strict limits, and clearing stale state during the deployment phase. This required touching the system supervisor configuration and the Node.js memory allocation.

Step 1: Enforce Hard Memory Limits (System Level)

We used systemd to ensure the queue worker was restricted and supervised correctly. This prevents runaway processes from monopolizing resources.

sudo nano /etc/systemd/system/queue-worker.service

We explicitly added memory limits to the service file to prevent OOM kills:

[Service]
MemoryLimit=2G
LimitNOFILE=65536
...

We then reloaded the daemon and restarted the supervisor:

sudo systemctl daemon-reload
sudo systemctl restart supervisor

Step 2: Clean Up Deployment Cache (Application Level)

We implemented a mandatory step in the deployment script to forcefully clear the Node.js cache and environment before restarting the worker processes. This eliminates the stale state that contributed to the leak:

cd /var/www/my-nestjs-app/
rm -rf node_modules/
npm install --production
# Force garbage collection and cache flush
node -e "global.gc(); process.memoryUsage();"

Step 3: Adjust Node.js Process Supervision

If using Node.js-FPM (or similar supervisor setup managed by aaPanel), we ensured the process manager was using the correct resource constraints specified by the operating system, rather than relying solely on internal application logic.

sudo systemctl restart php-fpm

sudo systemctl restart node-worker

Why This Happens in VPS / aaPanel Environments

Shared hosting and VPS environments introduce specific friction points that domestic development doesn't face. In the context of aaPanel/Ubuntu, this typically boils down to three factors:

1. Resource Contention: Shared environments tightly manage CPU and RAM. When a service (like the NestJS queue worker) attempts to grow its memory footprint, the container or environment often hits a hard ceiling set by the hypervisor, leading to aggressive OOM termination.
2. Process Inheritance and State Stale-ness: When deploying multiple services, the application may inherit stale environment variables or corrupted caches from previous deployments, which compounds the memory usage issues, making the application appear leaky even if the code itself is sound.
3. Configuration Mismatch (FPM/Supervisor): The interaction between the web server (Nginx/FPM), the process manager (Supervisor), and the Node.js runtime memory expectations often results in a configuration mismatch, where the system believes it has more memory available than the process is permitted to use.

Prevention: Hardening Future Deployments

To prevent this from ever happening again, we implemented a strict, reproducible deployment pattern:

• Immutable Deployment Artifacts: Never rely on runtime installation of dependencies on the VPS. Always use a clean build process and deploy pre-compiled artifacts.
• Pre-Deployment Cache Clearing: Integrate the cache clearing commands (rm -rf node_modules/ and dependency cleaning) directly into the deployment script *before* the application restarts.
• Strict System Limits: Always define explicit resource limits (using systemd or container limits) for long-running services like queue workers.
• Dedicated Environment Variables: Use environment variables exclusively for configuration (DB credentials, ports) and separate system limits for memory management. Do not rely on application memory management for system-level resource control.

Conclusion

Debugging production memory leaks on shared infrastructure requires moving beyond application logic and treating the problem as a system resource management issue. When deploying complex Node.js services on Ubuntu VPS via aaPanel, remember that the leak is often in the infrastructure layer, not just the code. Master the system commands, respect the resource constraints, and your deployments will stop crashing.