How do I fix the 'invalid format: expected "json"' error with Langchain ChatOllama?

Below is an explanation and a workaround for the error you’re seeing:

What’s Going On

The error message:

ResponseError: invalid format: expected "json" or a JSON schema

indicates that the Ollama endpoint expects the output in a specific JSON format. By default, the ChatOllama wrapper formats the prompt and response in a way that might not match what your current Ollama installation expects.

Why It Happens

Different versions of Ollama may enforce slightly different output formats. In your case, the model is returning a format that’s not in the expected JSON (or matching the defined JSON schema). This discrepancy causes the ChatOllama wrapper to error out when trying to parse the response.

Workaround

One suggested solution is to disable the automatic formatting of the output by setting the format parameter to None on your ChatOllama instance. This tells the wrapper to not expect the JSON formatting, thereby bypassing the error.

Modify your code as follows:

PYTHONCopied
from langchain_ollama import ChatOllama
import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s.%(msecs)03d [%(levelname)s]: %(message)s',
    datefmt='%H:%M:%S'
)

logging.info("### Starting up")

# Instantiate the ChatOllama model
llm = ChatOllama(
    model="llama3.1",
)

# Disable the automatic JSON formatting that causes the error.
llm.format = None

# Now invoke the model
response_message = llm.invoke("What are you?")

print(response_message.content)

Summary

By setting llm.format = None, you allow the underlying model's output to pass through without trying to parse it as JSON. This resolves the ResponseError and should enable you to continue with your local RAG tutorial.

If you run into further issues, consider verifying that your Ollama installation is up to date as well as checking the documentation for any version-specific nuances.