How can I process 10K Google Sheets rows with an OpenAI node in n8n effectively?

Below is some guidance to help resolve your issue and optimize your workflow:

Key Considerations

• Test Mode vs. Actual Execution:
  Sometimes running a test of individual nodes (especially with many batches) can result in data not appearing as expected. Test mode can differ from a full execution, so if the inputs vanish only in testing, try executing the entire workflow.

• Batching & Rate Limits:
  For 10K rows, splitting your data into batches is the right idea. However, remember that the OpenAI API may have rate limits in effect. Adding a Wait node between batches (or using the HTTP Request node with batch-limit options) ensures you’re not overwhelming the API.

• Data Format:
  Ensure your data is formatted correctly for the OpenAI node. Any deviation may cause the node not to return data—even if it works when testing with single executions.

Suggested Workflow Improvements

1. Trigger & Staging:
   Use your rowAdded trigger to collect data and then write these rows into an intermediate storage (like a temporary Google Sheet or database) if needed. This prevents the immediate test run from wiping out the inputs.

2. Loop Over Items Node (Split in Batches):
   Continue using the Split in Batches node (with an appropriate batch size). Sometimes a smaller batch size (e.g., 100–250) can work more smoothly if you are hitting rate limits or if the API struggles with a high number of rows in one go.

   JSONCopied
   {
       "batchSize": 250,
       "options": {}
   }

3. Wait Node:
   Introduce a Wait node between batches. This node helps space out API calls to avoid hitting rate limits.

   JSONCopied
   {
       "parameters": {
           "unit": "minutes",
           "time": 1
       },
       "type": "n8n-nodes-base.wait"
   }

4. OpenAI Node Alternatives:

   • Retry with HTTP Request Node: If you continue facing issues with the OpenAI node in test mode, consider using the HTTP Request node with the built-in batch-limit option. This often provides more detailed error feedback.
   • Check Input Consistency: Verify that the incoming data (row fields) is matching the OpenAI node’s expected input format.

Final Thoughts

Your approach is fundamentally sound (rowAdded trigger → batching → OpenAI node → Google Sheets), but running tests on only parts of such a large workflow might misrepresent how data is processed. Instead, try running the complete workflow in execution mode to see the full cycle from trigger to API call to data write-back into Google Sheets.

With these adjustments, you should be able to troubleshoot disappearing inputs and ensure that your 10K rows are processed reliably.