How can I mark initial data as processed to avoid handling old rows on the first run in n8n polling?

Below is one common approach to address the issue:

The Challenge

When running a polling workflow that retrieves rows based on a “last update” timestamp, the very first time you run it the variable (like LAST_UPDATE_TIME) isn’t set. This results in the workflow matching all rows in your table. In your case, that means sending a notification for every record—even though you’d prefer to simply “mark” those rows as already processed.

The Solution

You can modify your workflow so that on its first run it saves a baseline timestamp (or the latest value from your dataset) without executing the rest of your flow (and therefore no notifications). Then, on subsequent runs, you use that stored value as your LAST_UPDATE_TIME to only pick up new rows.

How to Implement

1. Store the Baseline Timestamp
   Add a Function node at the beginning of your workflow. In that node, check if a static data variable (e.g., lastUpdate) exists.

   • If it doesn’t (i.e., this is the first run), scan your incoming items for the maximum created_at value, save it in the static data, then return an empty array so that no further processing (like sending Slack notifications) occurs.
   • If the variable exists, let the workflow continue so that only rows with created_at > lastUpdate will be processed.

2. Use the Function Node Code
   Here’s an example you can adjust for your setup:

   JSCopied
   // Get workflow static data (global)
   const staticData = this.getWorkflowStaticData('global');
   
   // Check if baseline exists.
   if (!staticData.lastUpdate) {
     // This is the first run, so we set a baseline.
     // Determine the most recent created_at value.
     // Ensure your field name matches what your SQL query returns.
     const dates = items.map(item => new Date(item.json.created_at));
     
     // In case no items (or if items don't have a valid date), fallback to current timestamp.
     const maxDate = dates.length > 0 ? new Date(Math.max.apply(null, dates)) : new Date();
     
     // Save the baseline timestamp (ISO format is recommended).
     staticData.lastUpdate = maxDate.toISOString();
     
     // Return an empty array to stop further processing
     return [];
   }
   
   // Otherwise, for subsequent runs, let the workflow continue:
   return items;

3. Update LAST_UPDATE_TIME in Your SQL Query
   Use the value stored as your baseline when building your SQL query. For example, with an expression:

   SQLCopied
   SELECT * FROM your_table WHERE created_at > "{{ $workflow.staticData.global.lastUpdate }}";

4. Keep Your Baseline Updated
   At the end of the workflow—after processing any new items—you may want to update the stored timestamp. This could involve a Function node that extracts the highest created_at value from the current batch and updates the static data accordingly.

Recap

• The very first execution will only update the baseline without further processing.
• On subsequent executions, the query uses the stored timestamp so only new rows (i.e., those created after the baseline) trigger the rest of your workflow (like sending notifications).

Using this approach will help you “store” the initial set of rows as already processed and avoid the unwanted notifications and long execution times on your first run.