Understanding Task Cancellation in Django Async Code Under Load

Based on your description, you're experiencing task cancellations when using database_sync_to_async under heavy load. This is a common issue when working with Django's asynchronous capabilities, particularly when dealing with database operations.

Possible Causes

1. Thread Pool Exhaustion: When using database_sync_to_async (which is a wrapper around sync_to_async), Django runs database operations in a thread pool. Under heavy load, this pool might become exhausted.

2. Thread Sensitivity Issues: The sync_to_async function (which powers database_sync_to_async) has a thread_sensitive parameter that defaults to True since asgiref 3.3.0. This means all database operations try to run in the same thread.

3. Database Connection Limits: MySQL has connection limits, and under heavy load, you might be hitting these limits.

4. Timeout Configuration: There might be implicit timeouts in your ASGI server or Django configuration that are canceling long-running tasks.

Potential Solutions

1. Adjust Thread Pool Size

You can configure the thread pool size used by sync_to_async:

PYTHONCopied
import os
from django.conf import settings

# Increase the thread pool size
os.environ["DJANGO_ASGI_THREADS"] = "25"  # Adjust based on your needs

2. Use Transactions Properly

Since transactions don't fully work in async mode yet, consider wrapping your database operations in a single synchronous function:

PYTHONCopied
from asgiref.sync import sync_to_async

@sync_to_async
def process_order_transaction(order, status):
    # All database operations in one synchronous function
    transaction = Transaction.objects.create(
        order_id=order.id,
        amount=order.amount,
        status=Transaction.PENDING
    )
    
    # Other database operations...
    
    transaction.status = status
    transaction.save()
    
    return transaction

3. Implement Backoff and Retry Logic

Add retry logic for database operations:

PYTHONCopied
import asyncio
from functools import wraps

def async_retry(attempts=3, delay=1):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            for attempt in range(attempts):
                try:
                    return await func(*args, **kwargs)
                except asyncio.CancelledError as e:
                    if attempt == attempts - 1:
                        raise
                    await asyncio.sleep(delay * (2 ** attempt))  # Exponential backoff
            return await func(*args, **kwargs)
        return wrapper
    return decorator

@async_retry(attempts=3)
async def launch_individual_order_details(self, order):
    # Your existing code

4. Limit Concurrency

Instead of processing all orders at once, consider limiting concurrency:

PYTHONCopied
import asyncio
from asyncio import Semaphore

async def process_orders(orders, concurrency=10):
    semaphore = Semaphore(concurrency)
    
    async def limited_task(order):
        async with semaphore:
            return await self.launch_individual_order_details(order)
    
    order_details_tasks = [
        asyncio.create_task(
            limited_task(order),
            name=f"task_order_{order.id}",
        )
        for order in orders
    ]
    
    return await asyncio.gather(*order_details_tasks, return_exceptions=True)

5. Consider Using Django's ORM Async Methods

Instead of database_sync_to_async, use Django's native async ORM methods where possible:

PYTHONCopied
# Instead of:
# await database_sync_to_async(self.update_transaction_status)(transaction, status)

# Use:
transaction = await Transaction.objects.aget(id=transaction_id)
transaction.status = status
await transaction.asave()

Why Celery Works Better

Celery works better in this scenario because:

1. It manages its own worker processes and connection pools
2. It has built-in retry mechanisms
3. It doesn't rely on the same event loop as your web server
4. Each task runs in isolation with its own database connection

For high-load scenarios with many database operations, Celery remains a more robust solution than pure async code in Django, especially for database-heavy operations.