Why does LangChain's DirectoryLoader not load PDFs in Heroku despite working locally?

I'm encountering an issue with LangChain's DirectoryLoader on Heroku where it fails to load any documents from a directory, even though the file exists and the same code works locally.

Below is the log output for a file upload on Heroku:

BASHCopied
2025-01-19T21:50:38.661133+00:00 app[web.1]: Global session ID set to: 09a86o2ou8p5
2025-01-19T21:50:38.723501+00:00 app[web.1]: fucking initializing session vector store via file embedder initialization with session id: 09a86o2ou8p5
2025-01-19T21:50:39.146951+00:00 app[web.1]: WARNING:langchain_community.vectorstores.pgvector:Collection not found
2025-01-19T21:50:39.234423+00:00 app[web.1]: fucking initializing persistent vector store TAR
2025-01-19T21:50:39.559845+00:00 app[web.1]: uploaded file_extension: pdf
2025-01-19T21:50:39.559902+00:00 app[web.1]: loader_cls: <class 'langchain_community.document_loaders.pdf.UnstructuredPDFLoader'>
2025-01-19T21:50:39.559927+00:00 app[web.1]: directory path str: ./uploaded_files/09a86o2ou8p5
2025-01-19T21:50:39.559928+00:00 app[web.1]: DirectoryLoader absolute path: /uploaded_files/09a86o2ou8p5
2025-01-19T21:50:39.559937+00:00 app[web.1]: Contents of the directory:
2025-01-19T21:50:39.559994+00:00 app[web.1]: Root: ./uploaded_files/09a86o2ou8p5
2025-01-19T21:50:39.559995+00:00 app[web.1]: Directories: []
2025-01-19T21:50:39.560007+00:00 app[web.1]: Files: ['CAI2025_paper_6124.pdf']
2025-01-19T21:50:40.210297+00:00 app[web.1]: 
2025-01-19T21:50:40.210334+00:00 app[web.1]: docs: []
2025-01-19T21:50:40.210353+00:00 app[web.1]: docs after chunking: []

The relevant part of my code is as follows:

PYTHONCopied
async def process_documents(self, directory_path: Path, file_path, glob_pattern, loader_cls, text_splitter, embeddings, session_vector_store=None, unique_id=None, session_id=None):
        """To ensure that the process_documents method accurately processes only PDF files, one can modify the glob_pattern parameter used in the DirectoryLoader to specifically target PDF files. This adjustment will make the method more focused and prevent it from attempting to process files of other types, which might not be suitable for the intended processing pipeline."""
        print(f"loader_cls: {loader_cls}")
        directory_path_str = str(directory_path)
        absolute_path = os.path.abspath(directory_path_str)
        directory_path_str = f"./{directory_path_str}"
        print(f"directory path str: {directory_path_str}")
        print(f"DirectoryLoader absolute path: {absolute_path}")

        # List contents of the directory
        try:
            print("Contents of the directory:")
            for root, dirs, files in os.walk(directory_path_str):
                print(f"Root: {root}")
                print(f"Directories: {dirs}")
                print(f"Files: {files}")
        except Exception as e:
            print(f"Error listing directory contents: {e}")
            raise

        extracted_extension = glob_pattern.split('.')[-1]

        # Read PDF using PDF Loader (Only Text)
        loader = DirectoryLoader(
            directory_path_str,
            glob=glob_pattern,  # Use the specified glob pattern
            use_multithreading=True,
            show_progress=True,
            max_concurrency=50,
            loader_cls=loader_cls,
        )
        docs = loader.load()
        print(f"docs: {docs}")

        chunks = docs
        # Split documents into meaningful chunks
        if extracted_extension != 'csv' and extracted_extension != 'json':
            chunks = text_splitter.split_documents(docs)

        print(f"docs after chunking: {chunks}")

        store, current_embeddings = await self.persistent_vector_store.from_documents(
            documents=chunks,
            embedding=embeddings,
            chunk_size=10,
            collection_name=PG_COLLECTION_NAME,
            connection_string=os.getenv("POSTGRES_URL"),
            pre_delete_collection=False,  # Controls whether to clear the collection before adding new docs
        )

        if session_vector_store:
            store, current_embeddings = await session_vector_store.from_documents(
                documents=chunks,
                embedding=embeddings,
                chunk_size=10,
                collection_name=session_id,
                connection_string=os.getenv("POSTGRES_URL"),
                pre_delete_collection=False,  # Controls whether to clear the collection before adding new docs
            )

        return current_embeddings

I have verified that the file CAI2025_paper_6124.pdf exists in ./uploaded_files/09a86o2ou8p5. I have also tried listing the files using the following code to confirm the directory contents:

PYTHONCopied
try:
            print("Contents of the directory:")
            for root, dirs, files in os.walk(directory_path_str):
                print(f"Root: {root}")
                print(f"Directories: {dirs}")
                print(f"Files: {files}")
        except Exception as e:
            print(f"Error listing directory contents: {e}")
            raise

I've tried using both relative and absolute paths for the DirectoryLoader, but it still returns an empty list in the deployed Heroku environment. Locally, everything functions as expected. I am desperate to find a fix for this discrepancy in Heroku.