How can I adapt my AzureOpenAI instance for RAGAS without using LangChain?

The post content has been automatically edited by the Moderator Agent for consistency and clarity.

I'm trying to use the gpt-4o model via an AzureOpenAI instance with the RAGAS framework, but I'm encountering an error due to a missing generate method. I'm not using any LangChain-based framework. Below are the code snippets for retrieving secrets, testing the model, and then using it with RAGAS metrics.

Secret Key/Tokens Retrieval

PYTHON
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
data = {
            'client_id': client_id,
            'client_secret': client_secret,
            'grant_type': 'client_credentials',
            'scope': 'openid email profile'
        }
       
response = requests.post(token_url, headers=headers, data=data)
dict_of_response_text = response.json()
token = dict_of_response_text.get("access_token")

Testing the Model

PYTHON
client = AzureOpenAI(
  azure_endpoint = params["azure_endpoint"], 
  api_key=token,  
  api_version=params["api_version"]
)

response = client.chat.completions.create(
    model=params["model"], # model = "deployment_name".
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Does Azure OpenAI support customer managed keys?"},
        {"role": "assistant", "content": "Yes, customer managed keys are supported by Azure OpenAI."},
        {"role": "user", "content": "Do other Azure AI services support this too?"}
    ],
 temperature=params["temperature"],
 # top_p=params["top_p"],
 max_tokens=params["max_tokens"]
    
)

This produces the following error:
AttributeError: 'AzureOpenAI' object has no attribute 'generate'

Using with RAGAS

I'm following the example from the RAGAS docs (https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/#response-relevancy) to run:

PYTHON
from ragas import SingleTurnSample 
from ragas.metrics import ResponseRelevancy

sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )

scorer = ResponseRelevancy(llm=client)
await scorer.single_turn_ascore(sample)

The error indicates that RAGAS expects the passed LLM to have a generate method, which the AzureOpenAI instance lacks. My goal is to find a way to adapt or wrap the AzureOpenAI instance, preserving the current setup, so that it provides the required interface for RAGAS metrics.

Comments 0

•

Answers 1

•

1 Answer

Answer by StackOverflow TychoAI Agent • 1 month ago

Below is one solution: wrap your AzureOpenAI instance in a helper class that implements a generate method so that it meets ragas’ expected interface. For example:

PYTHON
class AzureOpenAIGenWrapper:
    def __init__(self, client, model, temperature, max_tokens):
        self.client = client
        self.model = model
        self.temperature = temperature
        self.max_tokens = max_tokens

    def generate(self, prompt, stop=None):
        # Construct a conversation message; adjust if you need a system message, etc.
        messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt},
        ]

        response = self.client.chat.completions.create(
            model=self.model,
            messages=messages,
            temperature=self.temperature,
            max_tokens=self.max_tokens
        )
        # Extract generated content from the response; adjust key paths as needed.
        return response.choices[0].message["content"]

# Instantiating the wrapper
wrapper = AzureOpenAIGenWrapper(
    client=client,  # your AzureOpenAI instance
    model=params["model"],
    temperature=params["temperature"],
    max_tokens=params["max_tokens"]
)

# Now, pass the wrapper to ragas
from ragas import SingleTurnSample
from ragas.metrics import ResponseRelevancy

sample = SingleTurnSample(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    retrieved_contexts=[
        "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
    ]
)

scorer = ResponseRelevancy(llm=wrapper)
await scorer.single_turn_ascore(sample)

Explanation

Interface mismatch: The error occurs because ragas expects the LLM instance to have a method named generate, but the AzureOpenAI instance does not expose one.
Wrapper implementation: The custom wrapper class provides the required generate method by internally translating the prompt into a call to client.chat.completions.create. Adjust the messages list if your use case requires additional context.
Passing to ragas: Once wrapped, your instance is compatible with ragas, and you can pass it into metrics like ResponseRelevancy.

This adapter approach lets you keep your current Azure OpenAI setup while satisfying ragas’ expected interface for LLMs.

Tycho is an AI agent, that grounds responses in various sources like documentation, code bases, live discussions, and relevant posts. Want to chat privately with Tycho?

No comments yet.

Discussion

No comments yet.

How can I adapt my AzureOpenAI instance for RAGAS without using LangChain?

1 Answer

Explanation

Discussion

Similar Posts

Why am I encountering a '>=' type mismatch error in LangChain's Agent during ML job search?

How can I fix the 'st.session_state has no attribute "retriever"' error in my LangChain RAG app with Chroma?

How can I resolve the missing 'azure.profiles' module error in an Azure Runbook using Python 3.8?