How can I process all HTTP response items in batches without skipping any records?

The behavior you’re seeing is expected given how the Loop Over Items node works. It takes the whole set of incoming items, splits them into batches based on your defined batch size (25 in your case), and then executes its "loop" branch for each batch. When all batches have been sent, it outputs the combined data via its "done" output. However, if downstream nodes (like your Read Sheet and Append Google Sheet) are only connected to the loop branch, they’ll only see one batch per execution rather than processing all batches sequentially.

What’s Happening

• The Loop Over Items node processes one batch (25 items) in its loop branch, then—in a single run—it emits the final output in the “done” branch after all batches have been handled.
• If you attach your duplicate-checking and Google Sheet append nodes to the loop branch, only one batch (the current iteration) goes through that branch, so the remaining items (the other 2700) aren’t processed by those nodes.

How to Fix It

You have a couple of options:

1. Structure a Recursive Loop in Your Workflow:
   Use the Loop Over Items node in conjunction with a conditional branch (checking the value of

   {{$node["Loop Over Items"].context["noItemsLeft"]}}
   

   which returns false until all items are processed) to re-trigger your duplicate check and append process for each batch. Essentially, you’d have a “loop back” (or recursive execution) that calls the same branch until noItemsLeft is true.

2. Use a Combined “Done” Output Processing Approach:
   Instead of handling the duplicate check and Google Sheet append logic in the loop branch (which processes only one batch), you can move that logic to the branch connected to the “done” output. In that branch you’d receive all processed batches together. However, keep in mind that if you then try to append all 2700 records at once, you might hit API rate limits for Google Sheets. You could mitigate that by batching the append operation inside your logic or leveraging Google Sheets node parameters if available.

3. Remove the Loop Over Items Node and Handle Batching Elsewhere:
   If you’re mainly concerned about not sending 2700 records at once to Google Sheets, you could bypass the Loop Over Items node entirely and instead batch your data within a Code node (or similar) to split the array and then process each batch with a recursive workflow. This gives you more granular control over how many records are sent in a single append operation.

Recommended Approach

If your goal is to have the duplicate check and append operations applied repetitively to each batch, the first option (structuring your workflow so that upon processing a batch you check whether any items remain and then re-trigger the Loop Over Items node) is most in line with how the Loop Over Items node works.

For example, after processing a batch:

• Use an IF node that tests if {{$node["Loop Over Items"].context["noItemsLeft"]}} is false.
• If false, trigger a sub-workflow (or use an Execute Workflow node) that runs the Loop Over Items branch again on the remaining data.
• This process continues until all batches are processed.

Summary

• The Loop Over Items node isn’t “ignoring” items – it’s designed to output one batch at a time in its loop branch and then wrap up with the done output.
• If your duplicate check and append logic is only wired into the one-batch loop branch, only that batch gets processed in one execution.
• To process all 2700 items, you’ll need to re-run the duplicate check and append logic for each batch—either by linking back in your workflow (using a recursive loop mechanism) or by handling all batches in the done branch and then splitting the appends into smaller segments.

This way, you can both avoid Google Sheets’ rate limits and ensure that every item is checked and appended appropriately.