How can I insert a 4MB PDF into Qdrant using nomic-embed-text with ollama?

Below are a couple of workarounds you can try:

1. Split the PDF into Smaller Chunks

Instead of processing the entire 4MB file as one unit, extract and split the PDF content into smaller pieces before embedding. For example, you can use one of n8n’s PDF or text extraction nodes and then a text splitter node. Not only does this reduce the size of each execution payload, but it also improves the semantic relevance of the embeddings by operating on smaller, more focused chunks.

2. Disable or Reduce Execution Data Logging

n8n workflows keep an execution log with all the node data. When processing large payloads (like a 4MB PDF), this stored data can exceed memory limits. You have a couple of options here:

• Temporarily disable saving successful execution data:
  In your workflow settings, change the option for “Save successful production executions” to “Do not save”.
  This prevents the large PDF data from being logged.

• Manually prune execution data:
  If you’re operating in an environment prone to large executions, consider using n8n’s best practices for data pruning to avoid hitting the memory ceiling.

3. Process Only the Required Data

If you don't need to store the full PDF contents in the vector store, consider processing only a summarized version or the most pertinent sections of the PDF. This not only reduces data size but can lead to more efficient similarity searches later on.

Using one or more of these approaches should help mitigate the “Existing execution data is too large” error when processing large files with nomic-embed-text in your Qdrant vector store workflow.