How can I index images in a vector database using a multimodal embedding model?

Based on the latest updates, it is indeed possible to index images in a vector database using a multimodal embedding model. Recent releases (such as the updated Chroma documentation) now support multimodal embedding functions that can embed images as well as text. Here’s a brief summary of what has changed and how you can achieve this:

What’s New

• Multi-modal Embedding Functions:
  Embedding functions (such as the built-in OpenCLIP) are now generic, meaning they accept data of multiple types. Instead of taking a dedicated texts argument, the updated __call__ method takes a single input that can be either text, images, or other supported modalities.

• Multi-modal Collections:
  You can create a collection configured with a multi-modal embedding function. When adding data, you can now specify images (e.g., as a list of numpy arrays).

How to Index Images

Here is an example showing how to set up and use a multi-modal embedding function to index images:

PYTHONCopied
from chromadb import Client
from chromadb.utils.embedding_functions import OpenCLIPEmbeddingFunction

# Initialize the embedding function that supports both text and images.
embedding_function = OpenCLIPEmbeddingFunction()

# Create a new Chroma client and multimodal collection.
client = Client()
collection = client.create_collection(
    name='multimodal_collection',
    embedding_function=embedding_function
)

# Suppose you have an image represented as a numpy array.
your_image_array = ...  # Replace this with your actual image array.

# Add the image to the collection.
collection.add(
    ids=["image_id_1"],
    images=[your_image_array]
)

Querying the Collection

You can later query the collection using images as well:

PYTHONCopied
results = collection.query(
    query_images=[your_image_array],  # list of numpy arrays representing images
)

Summary

If you were having trouble, it might be because the multimodal support was not available or documented clearly in previous versions. With the current updates, you now have a straightforward way to index, update, and query images using multimodal embedding models. Just ensure you are using the compatible version of the library where these features are implemented.

Let me know if you need further clarification on any specific steps!