How can I configure LiteLLM Proxy for sticky routing to leverage Azure OpenAI prompt caching and reduce costs?

Below is some guidance based on the LiteLLM documentation:

1. Managing Instance Affinity
   The Azure prompt caching feature relies on repeated calls hitting the same instance. Out of the box the LiteLLM Proxy works as a load balancer and doesn’t guarantee a sticky connection. To ensure that repeated requests are routed to the same Azure OpenAI instance you have two options:

   • Direct Routing: Instead of using the LiteLLM Proxy load balancer URL, you can configure your client to point directly to the same Azure instance by setting the API’s base URL to that instance’s endpoint. For example:

     PYTHONCopied
     import openai
     
     client = openai.OpenAI(
         api_key="sk-1234",             # your Azure or proxy key
         base_url="http://<azure_instance_endpoint>"  # route directly to one instance
     )
     
     response = client.chat.completions.create(
         model="command-r-plus",
         messages=[
             {"role": "system", "content": "Be a good human!"},
             {"role": "user", "content": "What do you know about earth?"}
         ]
     )
     
     print(response)

   • Customizing the Proxy Behavior: If you need to continue using the load-balancing proxy but want instance affinity, you can consider modifying or extending the LiteLLM Proxy’s routing algorithm. This may involve implementing sticky sessions (for example, based on request cookies or IP addresses) so that subsequent requests from the same client are forwarded to the same Azure instance. Note that this requires additional customization as it is not available in the default configuration.

2. Saving Cost by Utilizing Prompt Caching
   By ensuring that repeated requests (i.e. when sending contextual embedding queries) are consistently sent to the same instance, you allow Azure OpenAI’s prompt caching mechanism to kick in—which avoids additional cost on generating responses that have already been computed.

3. Summary Recommendation

   • If you can sacrifice some level of the proxy’s flexibility, switch your client’s base URL to a fixed Azure endpoint.
   • Otherwise, investigate customizing the LiteLLM Proxy server to incorporate sticky-routing logic so that it selects the same instance for repeated calls.

These approaches will help you take advantage of prompt caching to lower your overall costs when using multiple Azure OpenAI instances.