What causes gradual CPU usage increase on n8n worker instances in queue mode?

Below is a summary of potential causes and suggestions to investigate further:

Possible Causes

• Stalled Jobs in Redis:
  The errors “job stalled more than allowable limit” and connection timeouts indicate that some jobs aren’t being acknowledged on time. When a job is marked as stalled, the worker may reprocess it or reacquire locks repeatedly. Repeated reprocessing and lock renewals can gradually increase CPU usage.

• Redis Configuration and Network Latency:
  If Redis is slow to respond or network latency exists between the workers and Redis, it can cause job processing delays. The errors “Connection terminated due to connection timeout” and “timeout exceeded when trying to connect” hint that adjusting Redis timeout thresholds or improving connectivity might be necessary.

• Worker Resource Settings:
  Running 20 parallel executions per worker can strain system resources if workflows are heavy or if some jobs are long running. Delays or a high number of stalled jobs may lead to runaway CPU consumption.

Suggestions for Mitigation and Debugging

1. Review and Tune Redis Settings:

   • Check the values of environment variables such as QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD, QUEUE_WORKER_LOCK_DURATION, and QUEUE_WORKER_LOCK_RENEW_TIME to ensure that they’re appropriately set for your workload and network conditions.
   • Monitor Redis performance and adjust the configuration if you are experiencing frequent timeouts.

2. Investigate Worker Job Handling:

   • Monitor the bull queue regularly (as you’re doing with redis-cli) to see if stalled jobs pile up over time.
   • Evaluate whether some jobs are taking unusually long to execute, causing them to exceed their lock durations. If so, consider increasing the lock duration or checking for potential issues in the workflow that might cause delays.

3. Health Checks and Graceful Shutdowns:

   • Ensure that your worker processes have proper health checks enabled (using environment variables like QUEUE_HEALTH_CHECK_ACTIVE).
   • When restarting workers, use a graceful shutdown approach (configurable via N8N_GRACEFUL_SHUTDOWN_TIMEOUT or similar) so that jobs complete or are properly handled on shutdown.

4. Examine Network and Connectivity:

   • Verify network connectivity between your worker instances and the Redis instance. Look for issues such as intermittent connectivity or resource contention between containers or servers.
   • If Redis is running on a separate machine, ensure that its performance and available resources (memory, IOPS) meet your workload’s demands.

5. Update and Review Logs:

   • Confirm you’re running the latest patch/minor version of n8n if applicable. Sometimes, subtle bugs or improvements related to queue mode and worker processing are addressed.
   • Carefully review both n8n logs and Redis logs to pinpoint patterns or repeated behaviors that coincide with the CPU usage growth.

By addressing potential Redis connectivity issues, reviewing worker concurrency settings, and ensuring healthy job lifecycle management, you may be able to mitigate the CPU usage buildup. These steps should help debug and reduce the occurrence of stalled jobs that are likely contributing to the problem.

Below is some guidance on potential causes and debugging steps:

---

Possible Causes

• Stalled Jobs in the Queue:
  The Redis errors ("job stalled more than allowable limit", "Connection terminated due to connection timeout", etc.) indicate that jobs are repeatedly stalling. When jobs stall, workers may enter a tight loop continuously checking or trying to renew locks. This can gradually increase CPU usage as stalled jobs accumulate.

• Timeout/Connectivity Issues with Redis:
  The error messages suggest that workers might be experiencing intermittent Redis connectivity issues or timeouts during job processing. This can lead to jobs failing to complete on time and being retried repeatedly. Increasing the rate of retries and stalled job checks can consume extra CPU resources.

• Misconfiguration of Worker Concurrency/Lock Settings:
  With 20 parallel executions per worker, if the lock renewal or job processing times are not well balanced (for example, if a job is frequently taking longer than the configured QUEUE_WORKER_LOCK_DURATION), jobs might be released and retried. Check if the current lock duration and the timeout threshold (default is 10 seconds) are appropriate for your workload.

---

Suggestions for Mitigation and Debugging

1. Check and Tune Redis Settings:

   • Verify that your Redis instance is not overloaded and that its network latency is low.
   • Consider adjusting the QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD (default: 10000 ms) if you observe frequent connection timeouts.
   • If you are on Redis Cluster, ensure that the correct configuration variables (QUEUE_BULL_REDIS_CLUSTER_NODES) are in place.

2. Review Worker Concurrency and Lock Parameters:

   • If tasks generally take longer to execute, you may need to increase the QUEUE_WORKER_LOCK_DURATION to allow sufficient time for a job to complete before it is considered stalled.
   • It also helps to experiment with lowering the worker concurrency if you suspect too many parallel executions might be contributing to timing issues or resource contention.

3. Monitor Worker and Redis Metrics:

   • Keep an eye on worker logs to identify patterns around when jobs are stalled.
   • Use monitoring tools (e.g., CPU monitors, Redis monitoring dashboards) to correlate CPU usage spikes with job queue activity.
   • The sample command you provided to inspect Redis jobs is a good start:

     BASHCopied
     redis-cli -h <redis_host> -p <redis_port> --csv "LRANGE bull:jobs:your_job_name 0 -1"

4. Evaluate Network Stability:

   • Ensure that there is a reliable network connection between your worker nodes and your Redis and PostgreSQL servers.
   • Check for any network timeouts or packet loss that could be contributing to connection issues.

5. Consider Version Upgrades:

   • Since you’re running version 1.76.1, verify whether newer releases address similar issues. Sometimes performance improvements and bug fixes related to queue processing and error handling are implemented in newer versions.

6. Test with a Simplified Setup:

   • To narrow down the issue, try running a worker instance with a lower concurrency or even in a non-queue mode (if suitable) to see if the gradual CPU increase still occurs.
   • Replicate the issue under controlled conditions to determine if a specific workflow or environmental factor is triggering the excessive retries.

---

By focusing on these areas—particularly the Redis connectivity and job lock configuration—you should be able to mitigate the gradual CPU increase. Adjusting the parameters to better suit the characteristics of your workload is often key in distributed job processing systems like n8n's queue mode.

If the problem persists even after tuning, consider enabling additional logging or even reaching out to the n8n community (or support if you have an Enterprise plan) for further investigation.

Below is an analysis of what might be causing the gradual CPU increase on your workers and some pointers to help debug and mitigate the issue.

---

Potential Causes

• Stalled Jobs in Redis
  The errors “job stalled more than allowable limit,” “Connection terminated due to connection timeout,” and “timeout exceeded when trying to connect” indicate that jobs in the queue (managed by Redis) are stalling. If worker processes repeatedly try to lock and reprocess the same stalled job, they can climb in CPU usage over time.

• Worker Concurrency Settings
  Running 20 parallel executions on each of your three worker instances may be overwhelming worker processes if some executions hang or take longer than expected. This can lead to a backlog of stalled jobs in Redis and force extra job-lock renewals or retries.

• Redis Communication Issues
  Timeouts and connection errors suggest that the workers might have intermittent issues communicating with Redis. Workers keep hammering Redis to obtain or renew locks, which may lead to increased CPU usage if response latencies increase.

---

Mitigation and Debugging Suggestions

1. Examine Worker Logs
   Check the logs on your worker instances for additional clues about stalled jobs or reprocessing attempts. Investigate whether certain workflows or nodes (e.g., Code node operations) are repeatedly causing jobs to stall.

2. Review and Adjust Redis-Related Configuration
   In queue mode, n8n relies on Redis to manage job queues. Consider checking and adjusting the following environment variables if needed:

   • QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD (default is 10000 ms): A longer timeout may help if Redis is occasionally slow.
   • QUEUE_WORKER_LOCK_DURATION and QUEUE_WORKER_LOCK_RENEW_TIME: Ensure these values are tuned for your execution times.

   For example, to adjust the timeout threshold:

   BASHCopied
   export QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD=15000

3. Reduce Concurrency
   You currently run 20 parallel executions per worker. Reducing this number temporarily can help determine whether high concurrency is contributing to job stalls. You can adjust this at startup:

   BASHCopied
   n8n worker --concurrency=10

   Then, monitor CPU usage and queue behavior.

4. Monitor Redis Performance
   Use commands (like the example you provided) to inspect Redis job queues. Look for a high number of pending or stalled jobs that might indicate that jobs are not finishing normally:

   BASHCopied
   redis-cli -h <redis_host> -p <redis_port> --csv "LRANGE bull:jobs:your_job_name 0 -1"

   If many stalled jobs are observed, it may hint at long-running or failing executions.

5. Review Workflow Complexity
   Some workflows (or specific nodes that perform heavy processing) might rarely lead to increased CPU consumption if they don’t complete as expected. Evaluate if particular workflows are responsible and whether they can be optimized.

6. Version Considerations
   As you’re using version 1.76.1, check for any known issues or updates that might address performance regressions in queue mode. Upgrading to a later version might provide optimizations or fixes.

---

Next Steps

• Start with logging and reducing concurrency: Identify if a specific workflow or node is misbehaving or if too many parallel executions are overwhelming the system.
• Tune Redis-related configurations: Adjust timeouts and lock durations to reduce stalled job occurrences.
• Monitor resources closely: Use your monitoring tools to see how changes affect queued jobs in Redis and CPU usage over time.

By systematically adjusting these configurations and observing the effect on Redis job behavior and CPU load, you should be able to pinpoint whether the high CPU usage is due to stalled job retries, excessive parallelism, or communication issues with Redis.

---

Let me know if you need further details on any of these suggestions or additional configuration examples!