How can I batch embed Google Drive PDFs and Docs into Pinecone for RAG?

Below is an overview of how you can approach building your workflow in n8n to automatically process and embed all PDFs and docs from a given Google Drive folder into Pinecone.

Workflow Outline

1. Trigger & List Files

   • Use a manual trigger (or a schedule trigger if needed) to run the workflow.
   • Use the Google Drive node to list all files in the folder. This node lets you search for files by folder ID and file type (PDFs, docs, etc.).

2. Retrieve File Content

   • For each file listed, add a node to download or read the file content. Depending on the file type, you might need to convert or extract text (e.g., using a Code or Function node).

3. Embed the Document

   • Process the content through your embedding solution. This might involve calling an external embedding function or using a built-in node to transform the text into vector embeddings.
   • If you already have this set up for individual files, you can encapsulate that logic (perhaps in a sub-workflow) to handle each file in succession.

4. Upsert to Pinecone

   • Use the Pinecone Vector Store node to upsert the vector along with any metadata (like file name or a snippet of text).
   • Alternatively, if you’ve built an API-based solution to interface with Pinecone (as shown in the Pinecone documentation examples), you can make an API call to add the data.

Considerations

• Batch Processing:
  Since you mentioned that the embedding process doesn’t need to run continuously, triggering the workflow manually when new files are added is a good approach. This avoids the need for continuous polling.

• Error Handling & Logging:
  Implement steps to catch errors (for example, if file conversion or embedding fails) and log which files were successfully processed. This way, you can re-run or troubleshoot if needed.

• Existing Solutions:
  While there are examples in the n8n documentation for similar use cases—like populating a Pinecone vector database from a website or integrating data from Google Sheets—there isn’t an out-of-the-box JSON workflow exactly for processing all files from a Google Drive folder. Therefore, you will need to build a workflow by combining nodes from these examples:

  • The Google Drive credentials and file nodes can be adapted from the Google Sheets or similar integrations.
  • The pattern for populating a Pinecone vector database from a website shows how to structure the workflow steps (fetch, extract, transform, and upsert).

By stitching these concepts together, your workflow becomes straightforward: list all files in the desired folder, process each file (download, extract text, compute embeddings), and finally upsert the results to Pinecone.

Example Pseudocode

Below is a high-level pseudocode in JSON-style comments that outlines the workflow steps:

JSONCopied
[
  {
    "name": "Manual Trigger",
    "type": "trigger",
    "triggerType": "manual"
  },
  {
    "name": "List Files from Google Drive",
    "type": "googleDrive",
    "operation": "list",
    "folderId": "<your-folder-id>",
    "filters": { "mimeType": ["application/pdf", "application/vnd.google-apps.document"] }
  },
  {
    "name": "Download File",
    "type": "googleDrive",
    "operation": "download",
    "fileId": "={{$json[\"id\"]}}"
  },
  {
    "name": "Extract Text",
    "type": "code",
    "language": "JS",
    "code": "// Extract text from the document content"
  },
  {
    "name": "Embed Data",
    "type": "customFunction",
    "operation": "embed",
    "inputData": "={{$json[\"extractedText\"]}}"
  },
  {
    "name": "Upsert to Pinecone",
    "type": "pineconeVectorStore",
    "operation": "upsert",
    "data": {
      "id": "={{$json[\"id\"]}}",
      "values": "={{$json[\"embeddingResult\"]}}",
      "metadata": { "name": "={{$json[\"fileName\"]}}" }
    }
  }
]

Note: The above is abstracted for clarity. In n8n, you’d build this using the visual workflow editor, configuring each node accordingly.

Final Thoughts

Your approach is both reasonable and in line with common patterns seen in n8n workflows. While there isn’t a pre-shared JSON workflow specifically for Google Drive folder embedding, combining nodes and logic from existing examples (Google Sheets integrations and the Pinecone vector-store website workflow) should let you build a robust research assistant tool.

If you need further details on configuring specific nodes (like the Google Drive or Pinecone nodes), feel free to ask!