Why does my OpenAI Python vector store remain empty or show file_count as in_progress after uploading JSON files?

Below is an explanation of what might be happening:

Asynchronous Ingestion

Both of your approaches rely on asynchronous processing. In the first approach you call a polling helper (upload_and_poll) and in the second you attach files that then get processed asynchronously. When you see the file count as still “in_progress” or the vector store remains empty, it indicates that the backend hasn’t finished processing (or hasn’t started processing) your files.

Possible Reasons for the Behavior

1. File Format and Chunking Strategy
   The vector store endpoints (beta) expect files in a format that can be correctly chunked with the provided strategy. JSON files may require a different treatment compared to plain text, and if the static chunking strategy isn’t suitable for your file format the ingestion may not complete. The Web UI may apply different defaults or preprocessing steps that aren’t taking place when using the SDK directly.

2. Using the Correct Helper Methods
   The documentation recommends using the “create and poll” helper methods (e.g. client.beta.vector_stores.files.create_and_poll) rather than other upload methods. Double-check that you are using the correct endpoint for asynchronous ingestion. For example, the docs show:

   PYTHONCopied
   file = client.beta.vector_stores.files.create_and_poll(
       vector_store_id="vs_abc123",
       file_id="file-abc123"
   )

   Your code uses a similarly named upload_and_poll, but ensure that you have the latest SDK version and that the method exactly follows the documented semantics.

3. Potential Duplication or Incorrect File IDs
   In your second approach you create files with:

   PYTHONCopied
   human_dataset_result_json_file = client.files.create(
       file=open("results/results_tsm_human_labeled.json", "rb"),
       purpose="assistants"
   )
   human_dataset_json_file = client.files.create(
       file=open("data/sample_tsm_new.json", "rb"),
       purpose="assistants"
   )
   vectors_store = client.beta.vector_stores.create(
       name="human labeled dataset",
       file_ids=[human_dataset_result_json_file.id, human_dataset_json_file.id],
   )

   (Be careful not to accidentally pass the same file ID twice.) Even when done correctly, if the files are not ingested (i.e. remain in progress), this signals an issue in how the files are processed once attached to the vector store.

4. SDK or API Beta Issues
   Since these features are in the beta endpoints, there can sometimes be unexpected behavior or differences from UI processing. The fact that the Web UI processes the files successfully suggests that some behind‐the‐scenes processing (or default parameters) may differ from the SDK calls.

Recommendations

• Check File Formats:
  Ensure that your JSON files have the expected structure for ingestion. In some cases, text-based processing may assume a certain layout (e.g. newline-delimited entries).

• Use the Recommended SDK Methods:
  Use the SDK’s create_and_poll helpers (confirm method names and parameters) to initiate and monitor asynchronous ingestion. This is critical to allow the ingestion process to finish.

• Verify File IDs and Inputs:
  Make sure you’re not duplicating file IDs inadvertently. In your second approach, verify that each file is unique.

• Monitor Processing:
  After initiating the ingestion, check the vector store’s file_counts property periodically until it shows the expected numbers. If it remains in “in_progress” for an extended period, it could be due to an issue with the file content or chunking parameters.

• Consider Testing With Minimal Chunking:
  Since you mentioned that removing the chunking strategy has no effect, you might test with a minimal file (or without custom chunking) to isolate whether the chunking strategy is contributing to the issue.

• Contact Support if Needed:
  Since the Web UI works as expected, if the issue persists after verifying the above points, it may be a beta API or SDK issue. In that case, contacting OpenAI support with the details (while referencing the asynchronous file ingestion process) would be the next step.

By ensuring the files and their chunking parameters are acceptable to the asynchronous ingestion process (and by using the proper SDK helpers), you should be able to replicate the working behavior seen in the Web UI.