Dynamic Logging Configuration for Multiple Django Processes

To solve your issue with multiple processes writing to the same log file, you can implement a dynamic logging configuration that creates separate log files based on the process type. Here are a few approaches:

Option 1: Use Environment Variables

You can set different environment variables for each process type and use them to determine the log file name:

PYTHONCopied
import os

# Get process type from environment variable with a default fallback
PROCESS_TYPE = os.environ.get('DJANGO_PROCESS_TYPE', 'unknown')

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{'
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': f'logs/myapp_{PROCESS_TYPE}.log',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Then, when starting each process, set the environment variable:

BASHCopied
# For Django server
DJANGO_PROCESS_TYPE=web django-admin runserver

# For Celery
DJANGO_PROCESS_TYPE=celery celery -A myproject worker

# For management commands
DJANGO_PROCESS_TYPE=management django-admin migrate

Option 2: Use Process ID in Filename

Include the process ID in the log filename to ensure uniqueness:

PYTHONCopied
import os

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{'
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': f'logs/myapp_pid_{os.getpid()}.log',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Option 3: Detect Process Type Automatically

You can try to detect the process type automatically:

PYTHONCopied
import os
import sys

def determine_process_type():
    # Check for Celery
    if 'celery' in sys.argv[0].lower():
        return 'celery'
    
    # Check for manage.py commands
    if 'manage.py' in sys.argv[0] or 'django-admin' in sys.argv[0]:
        if len(sys.argv) > 1:
            return f'manage_{sys.argv[1]}'
        return 'manage'
    
    # Assume it's the web server
    return 'web'

PROCESS_TYPE = determine_process_type()

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{'
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': f'logs/myapp_{PROCESS_TYPE}_{os.getpid()}.log',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Option 4: Use RotatingFileHandler with Process-Safe Locking

For a more robust solution, you can use a RotatingFileHandler with process-safe locking:

PYTHONCopied
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{'
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': 'logs/myapp.log',
            'maxBytes': 10485760,  # 10 MB
            'backupCount': 10,
            'formatter': 'verbose',
            'delay': True,  # Delay file creation until first log
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Make sure the directory for your log files exists and is writable by the user running the Django application.

Each of these approaches has its advantages, but the environment variable method (Option 1) gives you the most control while being relatively simple to implement.