How can I integrate token usage and cost metrics into LangServe JSON responses?

Below is an approach you can take to wrap your chain’s invocation in the get_openai_callback context manager so that token usage and cost details are captured and then appended as metadata to your JSON response.

Example Integration

The idea is to wrap your chain’s run (or invoke) call with the callback context. Then, once the chain has finished processing, you extract the token metrics from the callback and augment your original output. For example:

PYTHONCopied
import json
from langchain_community.callbacks import get_openai_callback

def run_chain_with_metadata(input_data):
    # Wrap the chain call so that token usage is tracked.
    with get_openai_callback() as cb:
        # Run the chain; you could use .run() or .invoke() depending on your setup.
        result = chain.run(input_data)
    
    # Build your response JSON with the token usage metadata.
    response = {
        "result": result,
        "response_metadata": {
            "total_tokens": cb.total_tokens,
            "prompt_tokens": cb.prompt_tokens,
            "completion_tokens": cb.completion_tokens,
            "total_cost": cb.total_cost,
        },
    }
    
    return json.dumps(response)

# Usage
input_data = {...}  # your InputType data here
response_json = run_chain_with_metadata(input_data)
print(response_json)

Explanation

1. Wrapping the Invocation:
   The chain call is placed inside the with get_openai_callback() as cb block. This lets the callback capture token usage metrics for any OpenAI API calls made during the chain execution.

2. Extracting Metrics:
   Once the chain finishes running, you extract the token metrics:

   • cb.total_tokens
   • cb.prompt_tokens
   • cb.completion_tokens
   • cb.total_cost

3. Enhancing the Return Value:
   The original chain output is combined with the token usage details into a JSON-friendly dictionary, which is then serialized with json.dumps.

4. Integration with LangServe:
   Instead of calling chain.run directly in your LangServe configuration, you can have the service call this run_chain_with_metadata function. This ensures that every response from your chain includes the desired token usage and cost metadata in the JSON output.

This approach leverages the same callback mechanism you’ve been using in your standalone scripts while ensuring that the additional metadata is seamlessly integrated into the chain output served by LangServe.