Django Template Language Extensibility

Your question touches on an important aspect of Django's template system design philosophy. Based on the documentation, Django's template system is intentionally designed with specific principles in mind, including extensibility, but within certain boundaries.

Current Extensibility in DTL

Django's template system already provides extension points through custom template tags and filters. As mentioned in the design philosophies:

The template system should recognize that advanced template authors may want to extend its technology. This is the philosophy behind custom template tags and filters.

However, the deeper modifications you're suggesting (like replacing the lexer or parser) would go beyond the current extension mechanisms.

Design Philosophy Considerations

Several aspects of Django's template design philosophy are relevant here:

1. Separate logic from presentation: The template system is meant to control presentation and presentation-related logic only.

2. Don't invent a programming language: DTL aims to offer just enough programming-esque functionality for presentation decisions.

3. Safety and security: The template system forbids inclusion of malicious code by design.

These principles suggest that while Django values extensibility, it's carefully balanced against other concerns like simplicity and security.

Alternative Approaches

Rather than modifying the core DTL engine, Django's documentation suggests a few alternatives when you need more flexibility:

1. Use a third-party template engine: When performance or feature limitations become bottlenecks, Django supports using alternative template systems like Jinja2:

PYTHONCopied
# In settings.py
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.jinja2.Jinja2',
        'APP_DIRS': True,
        # ...
    }
]

2. Optimize existing code: The documentation notes that "the same gains may also be available without going to that trouble - for example, expensive processing and logic in your templates could be done more efficiently in your views."

Viability of Your Proposal

While your suggested approach of making the lexer and parser more modular is technically interesting, it would represent a significant architectural change to the template system. Such changes would need to be carefully evaluated against Django's design philosophies.

Django tends to favor pragmatic solutions that maintain backward compatibility and simplicity. The current extension mechanism (custom tags and filters) allows for significant customization without modifying the core engine.

If you're interested in pursuing this idea further, you might consider:

1. Starting a discussion on the Django development mailing list or GitHub
2. Creating a proof-of-concept implementation that demonstrates the benefits
3. Ensuring your proposal maintains compatibility with Django's security model

Remember that any proposal would need to demonstrate significant benefits that outweigh the added complexity and potential security implications.