How can I aggregate paginated API request data into a single output?

Below is one approach that has worked well for many users:

Use the Merge Node with Append Mode

The Merge node lets you combine data from multiple streams. For paginated API calls—where each call returns its own subset of data—you can:

1. Run Your Paginated Requests
   Have your API call node (or loop generating the paginated requests) output the data. Each execution returns its own set of items.

2. Combine with a Merge Node
   Connect the outputs from each API call to a Merge node.

   • Set the Mode to Append. This causes the node to wait until all incoming items are available and then “append” each set, resulting in one unified output.

3. Ensure All Data is Received
   If your paginated calls are triggered in a loop, make sure that the loop completes and the final iteration collects all your data. Then use the Merge node to combine all these items. The final output will be an array containing all items—one for each paginated result.

Alternative: Using a Code Node for Aggregation

If you need more control over how the items are merged, you can also use a Code node to aggregate all items. For example, you could set up your workflow so that every paginated API call appends its results into an array stored in a workflow variable. In a final Code node, you could combine these arrays like this:

JAVASCRIPTCopied
// Assume 'dataArrays' is an array of arrays from your API calls
let unifiedArray = [];

for (const dataArray of dataArrays) {
  unifiedArray = unifiedArray.concat(dataArray);
}

return unifiedArray.map(item => ({ json: item }));

This method gives you full control over how to merge and even transform the data if necessary.

Final Thoughts

Both methods allow you to create a single unified output from multiple paginated API responses. The Merge node in Append mode is often the simplest, while using a Code node gives you additional flexibility.

Feel free to adjust the approach to suit the specific requirements of your workflow.