Why Does My Streamlit App Only Process the First Uploaded PDF When Extracting Financial Data?

I'm building a Streamlit app designed to extract financial data from multiple uploaded annual report PDFs and compile the results into a single Excel file.

The intended workflow is as follows:

1. Extract the company_code and FY (e.g., from a file named NASDAQ_AAPL_2023 for Apple Inc., FY 2023).
2. Parse each PDF sequentially: extract text, split it into manageable chunks, generate embeddings, and retrieve financial data using a model.
3. Append the extracted information from each PDF to a cumulative DataFrame (final_df).
4. Generate a single Excel file containing the combined results for all processed PDFs.

However, the app only processes the first uploaded PDF and the extracted data from subsequent PDFs is missing in the final Excel output.

---

Code Workflow

File Name Format:

Each uploaded file is named in the format: NASDAQ_<COMPANY_CODE>_<FY>.pdf. The company_code and FY are extracted directly from the file name using a regular expression.

Key Workflow Steps:

1. Upload PDFs: Users can upload multiple PDF files via st.file_uploader.
2. Process PDFs Sequentially:
   • Extract the company_code and FY from the file name.
   • Parse the PDF to extract financial data (e.g., business_segment, currency, revenue).
   • Append the extracted data to a cumulative DataFrame (final_df).
3. Generate Excel File: Combine all extracted data into a single Excel file with columns:
   • company_code, business_segment, currency, revenue, FY.

---

Problem

The app only processes the first PDF by appending its data to final_df. Data from the remaining PDFs is ignored even though the code loops over all uploaded files.

---

Below are some critical sections of my code:

Extracting company_code and FY

PYTHONCopied
def extract_company_code_and_fy(file_name):
    match = re.match(r"NASDAQ_([A-Z]+)_(\d{4})", file_name)
    if match:
        return match.group(1), match.group(2)
    return None, None

Processing Each PDF

PYTHONCopied
def process_pdf_and_extract_data(user_question, text_chunks):
    embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
    vector_store = FAISS.from_texts(text_chunks, embedding=embeddings)
    docs = vector_store.similarity_search(user_question)
    chain = get_conversational_chain()
    response = chain({"input_documents": docs}, return_only_outputs=True)
    del vector_store  # Clear memory
    gc.collect()
    return response["output_text"]

Main Function

PYTHONCopied
def main():
    if "final_df" not in st.session_state:
        st.session_state["final_df"] = pd.DataFrame(columns=["company_code", "business_segment", "currency", "revenue", "FY"])

    pdf_docs = st.file_uploader("Upload your Annual Reports (PDF Files)", accept_multiple_files=True)
    if st.button("Submit & Process"):
        if pdf_docs:
            for pdf in pdf_docs:
                file_name = os.path.basename(pdf.name)
                company_code, fy = extract_company_code_and_fy(file_name)
                if not company_code or not fy:
                    st.error(f"Invalid file name format: {file_name}. Skipping...")
                    continue

                raw_text = get_pdf_text(pdf)
                text_chunks = get_text_chunks(raw_text)
                extracted_data = process_pdf_and_extract_data("Extract business segments and financial data.", text_chunks)
                if extracted_data:
                    rows = []
                    for line in extracted_data.split("\n"):
                        if "|" in line and not line.strip().startswith(("business_segment", "currency", "revenue", "---")):
                            parsed_row = line.split("|")[1:-1]
                            if len(parsed_row) == 3:
                                rows.append([field.strip() for field in parsed_row])

                    temp_df = pd.DataFrame(rows, columns=["business_segment", "currency", "revenue"])
                    temp_df["company_code"] = company_code
                    temp_df["FY"] = fy
                    temp_df = temp_df[["company_code", "business_segment", "currency", "revenue", "FY"]]

                    st.session_state["final_df"] = pd.concat([st.session_state["final_df"], temp_df], ignore_index=True)
        else:
            st.error("Please upload at least one PDF file.")

---

• The app correctly processes the first PDF and appends its data to final_df.
• However, data from subsequent PDFs is not added.

---

Steps I've Taken:

1. Ensured Persistence of final_df:
   • Initialized st.session_state["final_df"] only once.
2. Debugged Data Extraction:
   • Verified data extraction for the first PDF and added logging to ensure the loop runs for all PDFs.
3. Managed Memory Efficiently:
   • Cleared temporary variables and used gc.collect().

---

Question

Why does my Streamlit app only process the first PDF and skip the rest? How can I modify the code so that data from all PDFs is correctly appended to final_df and included in the final Excel file?