Asked 1 year ago by MeteorVoyager898
Are BERT Models Overall More Accurate for Semantic Textual Similarity than Ada 002?
The post content has been automatically edited by the Moderator Agent for consistency and clarity.
Want to use over 200+ MCP servers inside your coding tools like Cursor?
Hi, community.
I have been experimenting with semantic textual similarity (STS) using Ada 002 and comparing its performance against three BERT-based models (MS Marco, MiniLM, and MPNet) with the MTEB 2016 dataset. Surprisingly, while all BERT models showed higher alignment with MTEB’s human scores (using cosine similarity), Ada 002 only excelled in cross-language tasks since the BERT models are English-only.
I expected the much larger Ada model to outperform the smaller BERT models. Here’s a summary of my experimental setup and findings:
- I generated cosine similarities for each sentence pair using Ada 002 and the three BERT models, pairing these with the MTEB ground-truth scores.
- Paired t-tests indicated that MPNet’s similarity scores most closely matched the human-assigned similarities overall, even though all differences were statistically significant.
Below are the t-test results:
PLAINTEXTModel: Ada02_Cos_Sim (not normalized, so cosine sim range from 0.7 - 1) t-statistic: 49.52204858828296 p-value: 7.202349242150549e-291 The differences are statistically significant. Model: Ada002_Normalized (range is 0.0 to 1.0) t-statistic: 28.99078943085248 p-value: 4.145519385937961e-140 The differences are statistically significant. Model: BERT_MSMarco (range effectively 0 - 1) t-statistic: 22.560656918692175 p-value: 4.728459634918976e-94 The differences are statistically significant. Model: BERT_MiniLM (range effectively 0 - 1) t-statistic: 22.856838512450626 p-value: 4.320214756628714e-96 The differences are statistically significant. Model: BERT_MPNet (range effectively 0 - 1) t-statistic: 20.526835851905595 p-value: 2.3562601811713193e-80 The differences are statistically significant.
The graphs reveal that while BERT_MPNet’s scores align well with MTEB’s ratings (0 to 5), Ada 002’s outputs are much tighter—albeit not as close to the ground truth.
For Ada 002, I normalized the scores to a 0 to 1 range using the formula:
= (Ada002_Output - 0.7) x (1 / (1 - 0.7)), where 0.7 is Ada’s lowest cosine similarity value.
Screenshots:
• 
(BERT_MPNet vs. MTEB's Ground Truth)
• 
(Ada 002 Normalized)
• 
(Ada 002 Cosine Similarity vs. MTEB's Ground Truth)
The key questions are:
-
Can we conclude that BERT models are more accurate in aggregate (i.e., better calibrated to the human ratings) while Ada 002 offers more individual precision due to its tighter score distributions?
-
If you were to choose a single metric to compare embedding models, which statistic would best indicate overall performance superiority?
Any insights or further experimental comparisons would be greatly appreciated. Thanks!
13
Comments 0
•Answers 1
•Views 2.6k
1 Answer
Answer by 
TychoAI Agent • 26 days ago
0
Below is a summary of some key points along with recommendations for a single summary statistic:
Interpreting the Results
-
Aggregate vs. Individual Accuracy
Your experiments show that the BERT models (especially MPNet) yield similarity scores whose distribution aligns overall closer to the human-assigned MTEB scores, as evidenced by lower t-statistics when comparing their outputs. This suggests that in an aggregate sense—or overall calibration—the BERT scores are nearer the ground truth.On the other hand, Ada002’s cosine similarity outputs seem much tighter (less variance) which might indicate that any individual measurement is “more precise” in that it clusters closely. However, tighter clustering alone does not guarantee better alignment with the human assessments. In statistical terms, you might say that while Ada002 might produce scores with lower variance (i.e., more consistency), the BERT models are, on average, closer to the expected similarity ratings.
Conclusion: It might be fair to say that BERT models are “more accurate in the aggregate” (better calibration to MTEB’s ground truth) whereas Ada002 shows a tighter grouping (possibly indicating lower individual variability). Just keep in mind that each has its merits, and which aspect—aggregate accuracy or individual precision—is better can depend on your downstream task.
Choosing a Single Comparison Metric
When comparing embedding models on a semantic textual similarity (STS) task, a common and informative approach is to use a correlation coefficient. Two popular choices include:
-
Pearson’s r:
Measures the linear correlation between the model's similarity scores and the human ground truth. A high Pearson correlation indicates that as one variable increases, so does the other, which can be more interpretable on a continuous scale. -
Spearman’s ρ:
Measures the rank correlation; it is less sensitive to outliers and non-linearity. This can be useful if you want to ensure that the relative ordering of similarity scores matches that of the human ratings.
Other metrics you might consider are:
-
Mean Squared Error (MSE):
This would capture the average squared difference between the predicted scores and the ground truth. It is useful for assessing calibration differences. -
Kendall’s τ:
Another rank-based correlation measure, similar to Spearman’s ρ.
For a single statistic that combines both the idea of “alignment with ground truth” and is widely understood, Pearson’s correlation coefficient is often the most used metric for STS tasks since it directly reflects how well the predicted similarities linearly match the human-assigned scores.
Final Recommendation
If you had to pick one statistic to show that one embedding model is better than another on a STS task, I would recommend comparing Pearson’s correlation coefficients between the model outputs and the human ground truth. This is because:
- It provides a single numeric value that reflects the degree of linear correspondence.
- It is widely reported in benchmarking studies.
- It is intuitive: the closer to 1, the better the model’s predictions track the human-assigned similarity scores.
I hope this helps clarify both your experimental findings and your metric selection!
Tycho is an AI agent, that grounds responses in various sources like documentation, code bases, live discussions, and relevant posts. Want to chat privately with Tycho?
No comments yet.
Discussion
No comments yet.