How can I use SpaCy to detect near-duplicate product reviews based on text similarity?

Below is one approach using SpaCy’s pre-trained vector models to compute pairwise similarities. The idea is to convert each review into its document vector (an embedding) and then compare these vectors using cosine similarity. Reviews with a similarity above a chosen threshold (e.g., 0.9) might be considered duplicates.

Sample Code

PYTHONCopied
import spacy
import numpy as np

# Load the medium-sized model with vectors
nlp = spacy.load("en_core_web_md")

def cosine_similarity(vec1, vec2):
    """Calculate cosine similarity between two vectors."""
    return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))

def is_duplicate(review1, review2, threshold=0.9):
    """Check if two reviews are potential duplicates based on cosine similarity."""
    doc1 = nlp(review1)
    doc2 = nlp(review2)
    sim = cosine_similarity(doc1.vector, doc2.vector)
    return sim, sim >= threshold

# Example list of reviews
reviews = [
    "This product is excellent, I loved it!",
    "I really loved this product; it was excellent!",
    "Not a good product at all.",
    "Terrible quality, not recommended."
]

# Compare each pair of reviews
duplicates = []
for i in range(len(reviews)):
    for j in range(i + 1, len(reviews)):
        sim, dup = is_duplicate(reviews[i], reviews[j])
        if dup:
            duplicates.append((i, j, sim))

# Display potential duplicate pairs
for idx1, idx2, sim in duplicates:
    print(f"Review {idx1} and Review {idx2} have a similarity of {sim:.2f}")

Explanation

• Vectorization with SpaCy:
  The model en_core_web_md provides word vectors that are averaged to form a document vector (which represents the entire review).

• Cosine Similarity:
  The cosine similarity function returns a value between -1 and 1. A value near 1 indicates that the reviews are very similar.

• Threshold Setting:
  You might need to experiment with the threshold (here set at 0.9) depending on the variability in your review text. Reviews exceeding this threshold are flagged as potential duplicates.

Notes on tiktoken

The tiktoken library mainly focuses on tokenization for certain models and isn’t designed to compute semantic similarity. For duplicate detection based on meaning and wording, leveraging SpaCy’s embeddings (or similar text embedding models) is a more straightforward solution.

Performance Considerations

If you have a large dataset, comparing every pair of reviews can be computationally expensive. In such cases, consider:

• Using approximate nearest neighbor search techniques (e.g., Locality Sensitive Hashing or libraries like FAISS).
• Pre-filtering reviews based on length or other heuristic criteria before detailed comparison.

This approach should give you a good starting point for identifying duplicate (or near duplicate) product reviews.