How can I control concurrent requests to a FastAPI /execute endpoint on GKE?

Below is some guidance addressing both options:

---

1. All 100 Requests at Once

How Kubernetes/GKE Load Balancer Works:

• The GKE load balancer will distribute incoming requests to the 10 pods (typically using a round-robin or similar strategy).
• There is no built-in queueing at the load balancer level that makes a pod “wait” until it’s completely free. If a pod is busy handling a previous request, the pod’s server (for example, Uvicorn for FastAPI) must handle multiple requests concurrently or queue them internally.
• This means that if you send all 100 requests simultaneously, some pods will receive more than one request at the same time, so you must ensure that each pod or the underlying server framework can handle that level of concurrency without exhausting resources.

---

2. Throttling to 10 Concurrent Requests

Client-Side Throttling in Python:
To control the load, you can limit the number of concurrent requests. For instance, using asyncio with aiohttp and a semaphore is a clean solution. You can keep your FastAPI application and GKE configuration as-is. Here’s an example:

PYTHONCopied
import asyncio
import aiohttp

async def post_execute(session, base_url, filepath):
    params = {"filepath": filepath}
    async with session.post(base_url, params=params) as response:
        data = await response.json()
        print(f"Received for {filepath}: {data}")
        return data

async def limited_post(sem, session, base_url, filepath):
    async with sem:
        return await post_execute(session, base_url, filepath)

async def main(filepaths):
    base_url = "http://<your-load-balancer-ip-or-domain>/execute"
    # Limit concurrency to 10 simultaneous requests
    sem = asyncio.Semaphore(10)
    async with aiohttp.ClientSession() as session:
        tasks = [limited_post(sem, session, base_url, fp) for fp in filepaths]
        await asyncio.gather(*tasks)

# Replace with your list of file paths
filepaths = ["path1.txt", "path2.txt", "path3.txt", ...]  
asyncio.run(main(filepaths))

What This Accomplishes:

• Only 10 requests are in flight at any given moment.
• This results in each pod, on average, handling one request at a time (if the load balancer’s distribution is even), leading to more predictable resource usage.

No Changes Needed in FastAPI/GKE:

• Your FastAPI handler remains unchanged.
• The GKE deployment and load balancer configuration do not require modification since you’re managing concurrency on the client side.

---

Additional Considerations

• Internal Queuing: While the load balancer does not queue based on pod availability, many web servers (such as Uvicorn) do implement a worker or connection pool that can queue requests to a certain extent. However, relying solely on that behavior might lead to unpredictable resource use.
• Monitoring and APF Best Practices: Although the Kubernetes API Priority and Fairness (APF) mechanisms discussed in Kubernetes documentation mostly apply to the API server interactions, similar principles apply here regarding reducing expensive/concurrent requests to prevent service overload.
• Resource Configuration: Ensure that your pods are appropriately resourced and that autoscaling (if enabled) is configured to handle bursts, should your application requirements change.

This approach gives you control over the request rate, improving overall predictability and protecting individual pods from overload.