How can I work around the OpenAI API 429 Rate Limit Error when using FAISS embeddings with LangChain?

I have functional code with my free-tier OpenAI API key for chat completions, but I encounter a RateLimitError when using FAISS to generate embeddings.

For example, the chat completion code works fine:

PYTHONCopied
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI()

completion = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[
    {"role": "system", "content": "You are a poetic assistant"},
    {"role": "user", "content": "Who is Taylor Swift?"}
  ]
)

print(completion.choices[0].message.content)

However, when I try to create FAISS embeddings using the code below, I receive a 429 RateLimitError indicating an insufficient quota:

PYTHONCopied
from langchain_openai import OpenAI
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import FAISS
from langchain_core.vectorstores import VectorStoreRetriever
from langchain_openai import OpenAIEmbeddings
import os

# Ensure API key is set
os.environ["OPENAI_API_KEY"] = "private"

# Initialize OpenAI API client
openai_client = OpenAI()

# Load and process document
loader = TextLoader("/content/your_name-6.txt")
documents = loader.load()

# Split the documents into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0, length_function=len)
docs = text_splitter.split_documents(documents)

# Initialize embeddings using OpenAI
embedding = OpenAIEmbeddings(openai_client=openai_client)

# Create FAISS vector store from the documents
library = FAISS.from_documents(docs, embedding)

The error message is as follows:

BASHCopied
RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: [https://platform.openai.com/docs/guides/error-codes/api-errors.'](https://platform.openai.com/docs/guides/error-codes/api-errors.%27), 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}

Is there a workaround or best practice for handling this quota issue when generating embeddings with FAISS?

Additional context: The extra text discussing the sky's blue color and the related scientific explanation was initially included as sample content. It does not directly affect the problem but demonstrates the document size used for embeddings.