How to Use HuggingFaceEndpoint with CrossEncoderReranker in LangChain

Below is an example of how you can instantiate a HuggingFaceEndpoint as your cross encoder model and then plug it into the CrossEncoderReranker, replacing what would normally be a local HuggingFaceCrossEncoder. In this setup you simply build your dense retriever, create your endpoint-based cross encoder model, and then pass that into the reranker. Finally, wrap the dense retriever with the ContextualCompressionRetriever.

PYTHONCopied
index = self.pc.Index(self.index_name)

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vector_store = PineconeVectorStore(index=index, embedding=embeddings)

dense_retriever = vector_store.as_retriever(
    search_type="similarity", 
    search_kwargs={"k": 5}
)

# Use HuggingFaceEndpoint as the cross encoder model
cross_encoder_model = HuggingFaceEndpoint(
    endpoint_url="https://ugwu6jjlu82z8gxm.us-east-1.aws.endpoints.huggingface.cloud",
    max_new_tokens=512,
    top_k=10,
    top_p=0.95,
    typical_p=0.95,
    temperature=0.01,
    repetition_penalty=1.03,
    huggingfacehub_api_token="XXX"  # if your endpoint requires authentication
)

# Pass the HuggingFaceEndpoint model into the CrossEncoderReranker
reranker = CrossEncoderReranker(model=cross_encoder_model, top_n=3)

retriever = ContextualCompressionRetriever(
    base_compressor=reranker, 
    base_retriever=dense_retriever
)
return retriever

Key Points

• Instantiating the Endpoint: Instead of using a locally hosted cross encoder (via HuggingFaceCrossEncoder), you initialize a HuggingFaceEndpoint with the endpoint URL and other required parameters.
• Plugging into the Reranker: The CrossEncoderReranker accepts any model that follows its expected interface. Since HuggingFaceEndpoint meets that requirement, it can be directly passed as the model.
• ContextualCompressionRetriever: Wrapping the dense retriever with the reranker via ContextualCompressionRetriever allows for querying that first retrieves documents based on dense similarity and then re-ranks them using the cross encoder.

This approach leverages Hugging Face's managed endpoints for improved scalability and ease of deployment, replacing the need for a local model instance.