How can I add an effective file ID duplicate check in my Supabase RAG workflow?

Below is one approach that you might find effective. Instead of using a Supabase “get rows” node to try to filter out duplicate files (which then changes the data you need later in the workflow), you can use n8n’s built-in deduplication to gate the flow based solely on the file’s unique ID before you continue with your extraction steps.

Use the Remove Duplicates Node for File ID Checking

n8n’s Remove Duplicates node gives you several deduplication options. In your case you can configure it so that the node removes any input items whose file ID (from SharePoint) has already been seen. This way, only new files continue to the extraction steps.

Option A: Remove Duplicates Within the Current Input

If you are processing a batch of files and want to remove duplicates within that batch, configure the node as follows:

1. Add the Remove Duplicates Node after your initial SharePoint data is gathered.
2. Set the Operation to Remove Items Repeated Within Current Input.
3. In the Compare field, choose Selected Fields and enter the property that holds the file ID (for example, metadata.file_id if your file ID is stored there).

This configuration checks the current input and removes any duplicate file IDs so that later nodes only receive unique files.

Option B: Remove New Files Already Processed in Previous Executions

To handle the case where your workflow may fail partway through and you want to avoid re‑processing files already added to Supabase, use the deduplication history:

1. Add the Remove Duplicates Node in your workflow.
2. Set the Operation to Remove Items Processed in Previous Executions.
3. Set the Keep Items Where parameter to Value Is New.
4. In Value to Dedupe On use an expression that points to your file ID (for example:

   EXPRESSIONCopied
   {{ $json.metadata.file_id }}

   ).
5. (Optional) Once the workflow has processed the batch, if needed you can clear the deduplication history by running the node in Clear Deduplication History mode.

Using this approach, n8n compares the file ID from the current group to the files processed in earlier executions. Only files with a new file ID move forward to the extraction and storage steps.

How This Helps

• No Data Replacement: By using the Remove Duplicates node, you use it solely as a gate that filters out duplicates rather than merging or replacing the actual file data. This keeps your SharePoint information intact for later extraction tasks.
• Intelligent Check: It acts like a checkpoint that prevents duplicates from being processed further. It won’t alter the remaining data since it only removes the duplicate items.
• Flexibility: You can combine both approaches with two Remove Duplicates nodes in sequence if you need to check for duplicates within the current execution and against previously processed data.

Example Workflow Outline

1. SharePoint Data Node: Retrieves all file items (including metadata with the file_id).
2. Remove Duplicates Node (Current Input): (Optional, if you want to ensure that the incoming batch itself has no duplicate file_ids.)
3. Remove Duplicates Node (Previous Executions): Configured with "Value Is New" on metadata.file_id.
4. Set File Information Node / Extraction Process: Proceeds with the now deduplicated file items.
5. Supabase Node: Uploads or updates the file information.

By placing the deduplication node before the extraction logic, you’ll only process files with unique IDs—ensuring that even if the flow fails or runs multiple times, already‑handled file records (by file_id) aren’t re‑processed.

Give this a try—the Remove Duplicates node is well suited for your use case and should prevent the unwanted replacement of data while still acting as a filter by file ID.