How can I fix 'charmap' codec and XML errors when loading CSVs with Langchain Directory Loader?

Below is some guidance on resolving the issue:

1. Rely on Auto-detection with chardet

When you pass the keyword argument
  {"autodetect_encoding": True}
to the CSV loader via the DirectoryLoader, the loader uses chardet (if available) to determine each file’s encoding. (See our docs for auto-detecting file encodings.) Make sure you have installed the optional dependency (e.g. chardet) so that auto-detection kicks in.

2. Handling Control Characters or Invalid Bytes

The two errors you’re encountering indicate:

• A Unicode decoding failure (e.g. the "charmap" error)
• A later failure complaining that the resulting strings aren’t XML-compatible (due to control characters such as NULL bytes)

If your CSV files contain characters outside of valid UTF-8 (or even valid XML characters) then simply forcing a UTF-8 open might not be enough. Consider the following approaches:

a. Use Error Handling While Reading

Customize your CSV loader to ignore (or replace) invalid characters when opening the file. For example:

PYTHONCopied
from langchain.document_loaders.csv_loader import CSVLoader
from langchain_community.document_loaders import DirectoryLoader

class CustomCSVLoader(CSVLoader):
    def load(self):
        # Open the file with error handling set to 'replace' or 'ignore'
        with open(self.file_path, encoding="utf-8", errors="replace") as f:
            content = f.read()
        # Process the content using the parent class logic
        return self._parse(content)

loader = DirectoryLoader(
    r'C:\path',
    glob="**/*.csv",
    loader_cls=CustomCSVLoader
)
docs = loader.load()

Using errors="replace" (or alternatively, errors="ignore") can help ensure that characters that can’t be decoded are substituted or dropped.

b. Preprocess the Files

If you are still facing issues, it might be necessary to preprocess your CSV files (for example, stripping out control characters or NULL bytes) before passing them to the loader. This is especially useful if the data may include characters that are not XML compliant.

3. Verify the Encoding with Pandas

Since you mentioned that using Pandas with a specified encoding works, try to verify the proper file encoding externally. For example, you can use Pandas or the chardet library directly on one of your problem files:

PYTHONCopied
import pandas as pd

df = pd.read_csv(r"C:\path\to\problem_file.csv", encoding="utf-8")
print(df.head())

Once you’re sure about the encoding, you might also try opening the file with that specific encoding in your custom CSV loader. Sometimes files are in UTF-8 with BOM (utf-8-sig) or another similar variant.

Summary

• Ensure the optional encoding detector (chardet) is installed.
• Use the autodetect_encoding flag to let the loader select the proper encoding.
• If issues persist, override the file open to add a parameter like errors="replace" or specify a more appropriate encoding.
• Consider preprocessing files that contain control characters or non-XML compatible symbols if necessary.

These suggestions should help you overcome the encoding and XML compatibility issues when loading CSVs with the Langchain Directory Loader.