Why Do Multiple Local Python Subprocesses Experience Massive Slowdowns?

I'm working on a job submission script for parallelized model fitting using scipy.optimize across Torque, Slurm clusters, and my local machine. The cluster jobs run fine, but local execution slows down drastically when running more than one process.

Here's my approach:

1. Given a model configuration and data with M response variables, I split the data into n chunks, each corresponding to an equal part of M RVs. I then save the data to disk.
2. I create a .sh script that loads the data and fits a model on it.

The issue occurs locally only when n > 1. For example:

• With n == 1, everything works perfectly.
• With n > 1, the processes slow down by several orders of magnitude and consume all available CPU. I suspect this is due to contention over a shared resource, even though each process loads its own data from disk and should have its own GIL.

So far, I've tried:

• Running the callable directly using multiprocess instead of via the bash script. This approach failed due to unserializable objects and disappearing jobs.
• Using forks of multiprocess that rely on dill or cloudpickle, which did not work.
• Limiting the available cores per job with a custom preexec_fn, resulting in a 'fork' resource unavailable error.

I'm relatively new to multiprocessing, and despite researching similar issues, I still don't understand what makes my case unique. Any insights would be appreciated.

What I tried:
Running a bash script with multiprocess that executes a Python script handling files saved to disk.

What I expected:
I expected the process to run as fast as a single instance running without multiprocessing.

What actually happened:
A slowdown of approximately 5 orders of magnitude.

To illustrate the slowdown:

• if n == 1, fitting 2 models takes less than a second
• if n == 2, fitting 2 models takes almost an entire night

My suspicion is that the number of covariates might be the issue. Perhaps when matrix multiplications exceed a certain size, another process handles them, causing the parallel workers to wait for available resources?

How does this script run on your machines?

EDIT:
This problem has been observed on both a Macbook with an M3 processor and a Lenovo Thinkpad with Ubuntu Linux. However, the same script runs without issues on an HPC compute node when executing local parallel jobs from within an interactive job.

EDIT 2:
Below is a minimal working example:

PYTHONCopied
import subprocess
import sys
def test_parallel_fitting(n_covariates, n_models, n_processes):
    # Create two identical but independent processes
    processes = []
    n_covariates = int(n_covariates)
    n_models = int(n_models)
    n_processes = int(n_processes)
    n_models_per_process = n_models // n_processes
    for i in range(n_processes):
        command = ["python", "-c", f"""
import numpy as np
import time
from scipy import optimize
# Simple model fitting that shouldn't interact with other processes
for f in range({n_models_per_process}):
    start_time = time.time()
    X = np.random.rand(1500, {n_covariates})
    y = np.random.rand(1500)
    def model(params, X, y):
        return np.sum((y - np.dot(X, params)) ** 2)
    result = optimize.minimize(model, x0=np.ones({n_covariates}), args=(X, y))
    print(f"Process {i}:",f," finished in ", time.time() - start_time, " seconds")
"""
]  
        p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        processes.append(p)
    
    # Wait for completion and time it
    for p in processes:
        p.wait()
        output, error = p.communicate()
        print(output.decode())
        print(error.decode())
    print("Parallel fitting completed")

if __name__ == "__main__":
    args = sys.argv[1:]
    test_parallel_fitting(*args)

Works flawlessly:

BASHCopied
python script.py 6 6 3
python script.py 6 120 10
python script.py 7 120 1

Takes forever:

BASHCopied
python script.py 7 6 3
python script.py 7 120 10
python script.py 7 120 2

Please share your experiences or suggestions on resolving this slowdown.