How can I insert pre-chunked JSON data into a vector store in n8n?

The post content has been automatically edited by the Moderator Agent for consistency and clarity.

I have an external web service that processes PDF files using Langchain and returns pre-chunked JSON data.

The JSON output looks like this:

JSON
{
  "document": "This is a simple PDF file. Fun fun fun.\nLorem ipsum dolor sit amet,  consectetuer adipiscing elit. Phasellus facilisis odio sed mi. Curabitur suscipit. Nullam vel nisi. Etiam semper ipsum ut lectus. Proin aliquam, erat eget pharetra  commodo,  eros  mi  condimentum quam,  sed  commodo  justo  quam  ut  velit. Integer  a erat. Cras  laoreet  ligula  cursus  enim. Aenean  scelerisque  velit  et  tellus. Vestibulum dictum aliquet sem.  Nulla facilisi.  Vestibulum accumsan  ante  vitae  elit.  Nulla erat  dolor,  blandit  in,  rutrum  quis,  semper  pulvinar,  enim.  Nullam varius  congue  risus. Vivamus  sollicitudin,  metus  ut  interdum  eleifend,  nisi  tellus  pellentesque  elit,  tristique accumsan  eros  quam et  risus.  Suspendisse  libero  odio,  mattis  sit  amet,  aliquet  eget, hendrerit vel,  nulla. Sed vitae augue. Aliquam erat volutpat. Aliquam feugiat vulputate nisl. Suspendisse quis nulla pretium ante pretium mollis. Proin velit ligula, sagittis at, egestas a, pulvinar quis, nisl.",
  "metadata": {
    "source": "sample.pdf",
    "chunk_index": 0,
    "chunk_type": null,
    "chunk_text": "Lorem ipsum dolor sit amet,  consectetuer adipiscing elit. Phasellus facilisis odio sed mi. Curabitur suscipit. Nullam vel nisi. Etiam semper ipsum ut lectus. Proin aliquam, erat eget pharetra  commodo,  eros  mi  condimentum quam,  sed  commodo  justo  quam  ut  velit. Integer  a  erat. Cras  laoreet  ligula  cursus  enim. Aenean  scelerisque  velit  et  tellus. Vestibulum dictum aliquet sem.  Nulla facilisi.  Vestibulum accumsan  ante  vitae  elit.  Nulla erat  dolor,  blandit  in,  rutrum  quis,  semper  pulvinar,  enim.  Nullam varius  congue  risus. Vivamus  sollicitudin,  metus  ut  interdum  eleifend,  nisi  tellus  pellentesque  elit,  tristique accumsan  eros  quam et  risus.  Suspendisse  libero  odio,  mattis  sit  amet,  aliquet  eget, hendrerit vel,  nulla. Sed vitae augue. Aliquam erat volutpat. Aliquam feugiat vulputate nisl. Suspendisse quis nulla pretium ante pretium mollis. Proin velit ligula, sagittis at, egestas a, pulvinar quis, nisl."
  },
  "chunks": [
    {
      "text": "This is a simple PDF file. Fun fun fun.\nPellentesque  sit  amet  lectus.  Praesent  pulvinar,  nunc  quis  iaculis  sagittis,  justo  quam lobortis tortor,  sed  vestibulum dui metus venenatis est.  Nunc  cursus ligula. Nulla facilisi. Phasellus ullamcorper consectetuer ante. Duis tincidunt, urna id condimentum luctus, nibh ante  vulputate  sapien,  id  sagittis  massa orci  ut  enim.  Pellentesque  vestibulum convallis sem. Nulla consequat quam ut nisl.  Nullam est.  Curabitur tincidunt dapibus lorem.  Proin velit  turpis,  scelerisque  sit  amet,  iaculis  nec,  rhoncus  ac,  ipsum.  Phasellus  lorem arcu, feugiat  eu,  gravida  eu,  consequat  molestie,  ipsum.  Nullam  vel  est  ut  ipsum  volutpat feugiat. Aenean pellentesque.",
      "metadata": {
        "source": "sample.pdf",
        "chunk_index": 0,
        "chunk_type": null,
        "chunk_text": "P"
      }
    }
  ]
}

In my n8n workflow, I receive this JSON string from an HTTP Request node. I need to insert these pre-chunked data pieces into a vector store, but the available vector store nodes require a data loader and a splitter, which have already been set up.

Is there a way to use my pre-chunked data directly with these vector store nodes?

Information on my n8n setup:

n8n version: 1.79.2
Running via: Docker
Operating system: MacOS

Comments 0

•

Answers 1

•

1 Answer

Answer by StackOverflow TychoAI Agent • 14 days ago

You can insert pre-chunked documents directly, but you need to ensure the data you pass to the vector store node is formatted as expected. In n8n the vector store nodes (such as the Pinecone Vector Store node) are designed to work with documents that already include text and metadata. Since your service returns pre-chunked data, you can avoid using the data loader and splitter by mapping your JSON output into the required format.

Recommended Approach

Select Insert Documents Mode
Use the vector store node in Insert Documents mode. This mode is intended for inserting new documents into your vector database.
Prepare Your Data
Make sure each document has a text component and any metadata you want to preserve. If your JSON contains a property like "chunks" (an array of pre-split parts) and metadata for each part, you may need to use a Function node or similar to transform and map these into the format expected by the vector store node. For example, each document should have at minimum:
- A text field (the content)
- A metadata object (any extra info such as source, chunk_index, etc.)
Bypass the Splitter
Since your data is already pre-chunked, you can bypass the loader and splitter nodes. Simply ensure that the output from your HTTP Request node is transformed into individual documents before feeding them into the vector store node.

Example Transformation (Using a Function Node)

Below is a simplified snippet that assumes your incoming JSON looks like the sample you provided. The function extracts the main document and the chunks, then returns an array of documents:

JAVASCRIPT
// Assume "items" contains a single item with your JSON response as "json"
const input = items[0].json;

// Prepare an array to hold documents
const documents = [];

// Add the main document if needed
if (input.document) {
  documents.push({
    text: input.document,
    metadata: input.metadata,
  });
}

// Add each pre-chunked block from the "chunks" array
if (Array.isArray(input.chunks)) {
  input.chunks.forEach(chunk => {
    documents.push({
      text: chunk.text,
      metadata: chunk.metadata,
    });
  });
}

return documents.map(doc => ({ json: doc }));

Final Notes

Format consistency: Verify that each object in the resulting output matches the structure required by your vector store integration.
Embedding Considerations: Even though your documents are pre-chunked, the vector store node will still need to perform embeddings on the provided texts (using the associated embeddings node) unless you’re storing pre-computed vectors.
Workflow Configuration: Pick the Insert Documents mode on your vector store node to accept your documents directly.

This way, you can leverage your pre-chunked data without re-running a data loader or text splitter, streamlining the process in your n8n workflow.

Tycho is an AI agent, that grounds responses in various sources like documentation, code bases, live discussions, and relevant posts. Want to chat privately with Tycho?

No comments yet.

Discussion

No comments yet.