Achieving High Contextual Accuracy on Code Generation

function ExtractInterviewQuestions(markdown: string, lune_name: string, lune_description: string) -> InterviewQuestions {
  client "openai/o3-mini"
  prompt #"
    You are tasked with generating high-quality technical interview questions based on a specific page of documentation/code about {{ lune_name }} - {{ lune_description }}. The documentation is provided in markdown format below.
 
    Your goal is to create standalone questions that test a deep understanding of the technical concepts, code usage, features, and usage of {{ lune_name }}, as presented in this documentation/code page. The questions should be:
    - Directly related to {{ lune_name }} and its functionalities, code implementations, integrations, or capabilities and grounded in concepts from this specific page of documentation/code.
    - Standalone, meaning they do not reference 'the documentation' or 'the page.' They should make sense in a general technical interview about {{ lune_name }}.
    - Specific and precise, with clear, unambiguous answers that can be derived from documentation/code that you are given.
    - Focused on understanding rather than memorization (e.g., how something works or why it’s useful, not just recalling facts).
    - When appropriate (the page shows code implementations, code usage, and code examples), ask questions about code implementations, code usage, and code examples of {{ lune_name }} based only on the provided page of documentation/code.
    - About technical aspects of {{ lune_name }}, such as its usage, features, integrations, code implementation, or performance.
    - For code snippets, use markdown code blocks. If your generated question references a code snippet, INCLUDE THE CODE SNIPPET IN THE QUESTION.
 
    IMPORTANT: Questions MUST be as specific as possible and MUST be grounded in the documentation/code that you are given. If the question is about a code snippet, make sure to include the actual code in the question so the question can be answered without referencing the page of documentation/code.
 
    Avoid questions that:
    - Focus on the structure or navigation of the documentation pages itself (e.g., 'What sections are on the page?').
    - Are off-topic or primarily about other tools/libraries mentioned in passing (e.g., not about {{ lune_name }}).
    - Test memorization of trivial details (e.g., 'Name one thing listed...') rather than understanding the concepts.
    - Are vague or open-ended with multiple possible interpretations.
    - Do not include full context required to answer the question (e.g., 'What does the code above do?' without the code above).
 
    IMPORTANT: If the documentation page lacks sufficient technical content to generate high-quality questions that meet the above criteria, output a single QA pair with empty strings for 'question', 'answer', and 'criteria', like this:
    {
      "qa_pairs": [
        {
          "question": "",
          "answer": "",
          "criteria": ""
        }
      ]
    }
 
    Here is the markdown of the documentation/code reference page:
    {{ markdown }}
 
    Generate the questions, answers, and evaluation criteria in JSON format, with a key 'qa_pairs' containing an array of objects, each with 'question', 'answer', and 'criteria' fields:
    - The 'question' should be a standalone, specific, and precise technical question based on the documentation.
    - The 'answer' should provide the correct response to the question, grounded in the documentation.
    - The 'criteria' should be a concise list of the key elements that a correct answer must include, such as specific syntax, important concepts, or critical details from the documentation. This list will help graders determine if a candidate’s response demonstrates a sufficient understanding of the topic.
 
    Ensure that the questions test a deep understanding of {{ lune_name }} and are directly related to its functionalities, code implementations, integrations, or capabilities as described in the provided documentation.
 
    Examples of good questions:
    - 'Is BLANK integration for {{ lune_name }} supported?'
    - 'How much does FEATURE cost for {{ lune_name }}'
 
    Examples of bad questions to avoid:
    - 'Which integrations are mentioned in the documentation?'
    - 'What is SOME_OTHER_TOOL as mentioned on the page?'
 
    {{ ctx.output_format }}
  "#
}

async def evaluate_answer(self, question: str, correct_answer: str, generated_answer: str) -> bool:
       prompt = f"""
       You are an instructor that evaluates the correctness of an answer from a technical interview.
       The question is: {question}
 
 
       The correct answer is: {correct_answer}
 
 
       The candidate's answer is: {generated_answer}
 
 
       Please evaluate the candidate's answer strictly against the correct answer and output in json, a simple bool value (is_answer_correct) indicating if the answer is correct or not and a string (reasoning) explaining your reasoning concisely.
       """
       response = await self.client.beta.chat.completions.parse(
           model="gemini-2.0-flash",
           messages=[
               {
                   "role": "user",
                   "content": prompt
               }
           ],
           response_format=AnswerValidation,
       )
       evaluation = response.choices[0].message.parsed
       return evaluation.is_answer_correct
 

npx -y @lune-inc/mcp --api-key=YOUR_LUNE_API_KEY --lune-name=LUNE_NAME --lune-id=LUNE_ID