Why does my n8n workflow hang at the 'Extract from CSV' node when processing a large CSV file?

Based on similar issues documented in the n8n documentation, processing one 27MB CSV file in a single shot might be causing excessive memory usage. Here are a few suggestions to address the problem:

1. Split the Data into Smaller Batches

When handling large files, breaking the CSV into smaller chunks will reduce the memory required at any one time. For example, you could use the Split in Batches node to process a limited number of rows (e.g. 200 or 500 rows per batch) instead of processing all 250k rows at once.

2. Use Sub-Workflows for Heavy Lifting

As recommended in the memory usage strategies, offloading heavy processing to sub-workflows can help. The sub-workflow would process each smaller chunk and then return only the necessary data back to the main workflow. This way, only a fraction of your data is held in memory at any point.

3. Evaluate Your Environment and Settings

Since you’re running on n8n Cloud with SQLite as your internal database, consider:

• Checking if any memory limits are being reached.
• Evaluating the execution process settings (set to “own, main”) to see if running the workflow in a separate process may help isolate the heavy load.
• If this is a production-critical system, it might be worthwhile to consider using an external database (like Postgres) for improved performance and stability.

Next Steps

• Modify your workflow by first inserting a node like [Split in Batches] to break your CSV data into manageable pieces.
• Consider creating a sub-workflow that handles the extraction and transformation of each batch, then uploads the batch to Postgres.
• Test with a fraction of the data to confirm that the execution completes before processing the entire file.

By applying these strategies, you should be able to reduce the memory footprint during CSV extraction and avoid the workflow hanging at the “Extract from CSV” node.