How can I optimize n8n workflows for large data sets to avoid memory issues?

Here are a few best practices to maximize performance for data-heavy workflows in n8n:

1. Optimize the Data Pipeline

• Break Your Data Into Smaller Batches:
  Ensure that each sub-workflow only handles a very small chunk of data. Even if you’ve split into five sub-workflows, consider reducing the batch size further (for example, process 200 rows or less at a time) so that each execution holds a limited amount of data in memory.

• Limit Returned Data:
  Make sure that each sub-workflow returns only minimal results to the parent workflow. The idea is to not accumulate large datasets at any point during an execution.

2. Avoid Memory-Heavy Nodes

• Minimize Use of the Code Node:
  The Code node (and the older Function node) can substantially increase memory consumption. If possible, try to move logic into external calls or use native nodes that are optimized for performance.

• Avoid Manual Executions:
  Keep in mind that running workflows manually can increase memory usage. Use automated triggers where feasible, as manual runs tend to duplicate data for the UI.

3. Docker & Environment Considerations

• Allocate Sufficient Memory in Docker Desktop:
  Although your Mac specs (i9, 16GB) are robust, Docker Desktop’s settings might limit the amount of memory allocated to the Docker Linux VM. Check your Docker Desktop Resources section and consider increasing the memory allocation before running workloads.

• Tweak Node Options for Self-Hosting:
  If you’re self-hosting via Docker, consider increasing the memory available to Node’s V8 engine. For example, you could set:

  BASHCopied
  export NODE_OPTIONS="--max-old-space-size=4096"

  This gives Node more room to execute, particularly with heavy workflows.

4. Reduce Execution Data Logging

Excessive logging and saving of execution data can add to memory pressure:

• Configure Execution Data Settings:
  Disable saving of unnecessary execution data by setting environment variables such as:

  BASHCopied
  # Save only executions ending in error
  export EXECUTIONS_DATA_SAVE_ON_ERROR=all
  export EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
  
  # Disable node progress logging
  export EXECUTIONS_DATA_SAVE_ON_PROGRESS=false
  
  # Disable saving manual executions if not required
  export EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=false

• Enable Data Pruning:
  Allow n8n to automatically prune old execution data to keep the database lean:

  BASHCopied
  export EXECUTIONS_DATA_PRUNE=true
  export EXECUTIONS_DATA_MAX_AGE=168  # Adjust as needed
  export EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000

Recap

Even after splitting your workflow, if you’re still facing memory issues, double-check that:

• You’re handling as small a data chunk as possible per sub-workflow.
• Each sub-workflow cleans up after heavy processing and doesn’t pass along large datasets.
• Docker Desktop is configured to use enough resources, given that the default Docker settings might be conservative.

If you’d like to share further details of your workflow, the community could help pinpoint specific nodes or configurations that might be contributing to the memory pressure.