Why does ada-002 yield significantly lower cosine similarity quality than davinci-001 for legal text embeddings?

Below is some guidance regarding your experience:

Model Differences and Expectations

• Different Training Data & Objectives:
  The ada-002 and the earlier davinci-001 (or text-similarity-davinci-001) models were trained with different objectives and datasets. This means that even though both output high-dimensional embeddings, their internal representations may differ, which in turn affects cosine similarity scores.

• Change in Embedding Behavior:
  You might see variations in cosine similarity because the scale and distribution of the embeddings can change between versions. A lower cosine score under one embedding model doesn’t necessarily indicate “worse” performance—it could mean that the similarity thresholds need adjustment.

Steps to Investigate and Adapt

• Revisit Similarity Thresholds:
  With the change in embedding distribution, the thresholds you used with davinci-001 may not be directly applicable for ada-002. Calibrate the cosine similarity threshold on a validation set aligned with your legal domain.

• Domain-Specific Evaluation:
  For high-stakes legal texts, it’s important to benchmark performance using your actual critical examples. Consider creating a test suite with domain-relevant pairs to determine if the degradation you’re seeing affects your downstream tasks.

• Consider Alternatives:
  Since many first-generation embedding models (including davinci-001 variants) have been deprecated and replaced by newer models (for example, the recommended model for several deprecated ones is now text-embedding-3-small), it might be worthwhile to:
  - Experiment with the latest recommended embeddings.
  - Evaluate if a fine-tuned model or additional post-processing (such as dimensionality reduction or alignment techniques) could improve performance in your specific high-stakes application.

Next Steps

1. Evaluate on a Representative Dataset:
   Test both models across a carefully selected corpus of legal text to quantify differences in performance, not just in cosine similarity, but in how well the embeddings support your downstream decision-making.

2. Adjust Cosine Similarity Metrics:
   Consider scaling or calibrating cosine thresholds. A simple example in Python might be:

   PYTHONCopied
   import numpy as np
   
   def cosine_similarity(vec_a, vec_b):
       return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))
   
   # Example: Adjust threshold based on empirical observations
   threshold = 0.75  # This value might need tuning for ada-002
   sim = cosine_similarity(embedding_a, embedding_b)
   if sim > threshold:
       print("Embeddings are considered similar")
   else:
       print("Embeddings are not similar")

3. Reach Out with Specific Metrics:
   If you believe this degradation is unexpected, gather concrete metrics (e.g., average cosine similarity differences, precision/recall in your use case) and share these with support. This can help them investigate any potential issues or provide recommendations specific to your scenario.

Conclusion

While some quality differences between models are inherent due to changes in training and architecture, for a high-stakes domain like legal text analysis, it’s crucial to carefully recalibrate your similarity measures and validate overall performance. It’s not uncommon for users to observe these differences when transitioning to newer models, and many have found that adjusting evaluation strategies helps mitigate the apparent drop in quality. Keep an eye on further updates or improvements from OpenAI, and consider testing alternatives if needed.

Feel free to ask further questions specific to the API or embedding usage.