How can I process paginated data in n8n to write to a spreadsheet without memory overflow?

Below is an approach to reduce memory usage by processing one paginated page at a time:

1. Fetch One Page Only

Instead of a single execution fetching all pages, structure your workflow so that it only fetches one page per execution. For example, use a parameter (like a cursor) that lets you retrieve the current page. In the response, check for a next page (using the “nextCursor” field provided by the API).

2. Process Items in Small Batches

After fetching the page, use the Split In Batches node. This node takes the page’s data and splits it into smaller sets (or individual items) before writing to the spreadsheet. By doing so, you are never holding more than one page’s worth of data in memory.

3. Invoke Sub-Workflows for Heavy Lifting

If processing each batch (or page) still requires significant resources, offload the heavy work to a sub-workflow. Use the Execute Workflow node to call a secondary workflow that processes just one batch and then returns only a limited result to the main workflow. This keeps memory consumption low because each sub-workflow handles one small subset of data at a time.

4. Loop Through the Pages

Once the current page is processed, have your workflow check for a “nextCursor” in the API response. If the cursor exists, trigger the workflow again (or use a looping mechanism) with the new cursor value so that only the next page is fetched on the next execution.

Workflow Example Outline

1. HTTP Request Node
   • Fetch one page of data using your paginated API.
   • Include parameters like cursor, limit, etc.

2. IF or Function Node
   • Check whether the API response contains a “nextCursor”.
   • Decide whether to trigger another execution (this could be done with a webhook or Execute Workflow node).

3. Split In Batches Node
   • Use this node to split the fetched page into individual items or smaller chunks.

4. Spreadsheet Node
   • Write each item (or batch) to the spreadsheet.

5. (Optional) Execute Workflow Node
   • Offload heavy processing tasks (if needed) into a sub-workflow.

By processing one page at a time and splitting the data into smaller chunks, you avoid loading the entire data set into memory. This method ensures each execution only handles a manageable subset of data, significantly lowering memory consumption.

This approach aligns with n8n’s best practices for handling large datasets and memory optimization.