How Do Embeddings Lower Costs Compared to Using the Full Context in ChatGPT-4 128k?

Below is a concise explanation addressing your questions:

---

Full Context in Chat vs. Using Embeddings

• Full Context with the 128k Model:
  When you include the entire document in every prompt (interacting with ChatGPT-4 128k), every request is charged based on the full token count of that prompt plus the completion. So if your estimate is about $1 per full-context query, asking 10 questions would cost roughly $10, since each call processes the entire context.

• Using Embeddings for the Document:
  The idea with embeddings is to process the document one time to generate an embedding representation. Then, on each query, you run an embeddings-based search to retrieve only the most relevant snippets (rather than the whole document) and include only those in the prompt. This means:

  • One-time cost: You pay for generating embeddings for the entire document.
  • Subsequent queries: You then only pay for the tokens used by the (much shorter) retrieval prompt plus the generated answer.

  In effect, if the document is large, you save on recurring token processing costs by not refeeding the entire document with each query.

---

About the New Assistant API and Embeddings

• Assistant API and Embeddings:
  The new Assistant API is designed for optimized conversational interactions and may internally leverage various techniques, including embeddings, to manage context and provide answers. However, its architecture doesn’t necessarily expose direct embedding methods to users in the same way as the separate embeddings API. For building your own retrieval-based system, you would manually use an embeddings model (like text-embedding-ada-002) and then pass the retrieved context to a conversational model.

---

Summary

• Cost Implication Using Full Context:
  Yes, if you include the entire document every time as context, you pay for the full token count on every single request (potentially $1 per query with a 128k model, resulting in about $10 for 10 interactions).

• Cost Implication Using Embeddings:
  With embeddings, you amortize the cost of processing the document by paying once for embedding generation, then only incurring costs for the smaller text snippets retrieved per query. This should drastically cut down the per-query token usage and overall cost.

• Assistant API Usage:
  The new Assistant API may leverage techniques like embeddings internally, but if you want precise control, you’d work with the separate embeddings endpoints alongside the ChatGPT models.

This approach can help ensure that you don’t face exceedingly high costs by avoiding repeatedly passing large amounts of text in every conversation turn.