Since django 4.1, templates are cached with DEBUG=True. Is this solution right?

Question

As described in the documentation, since 4.1 the default behavior for template loading changed drastically.

If I understand it right, until 4.0 it worked like this:

• With DEBUG enabled, the templates are loaded in every request, therefore if you keep making changes and reloading while working on a template, you always see the latest version.
• With DEBUG disabled, the templates are cached when initializing the application, therefore you can only see changes in your templates if you also restart the application.

That way, the template caching was seamlessly enabled in production which is great.

Now this ticket proposal was included and, if I get it correctly, the template loading method must be specified and it's not anymore tied to DEBUG setting, AND, by default are cached.

We want the original behavior so the frontend developer can see the changes without having to restart the app, and we also want the production deployment to have the caching enabled, so we did this:

develop_loaders = [
    "django.template.loaders.filesystem.Loader",
    "django.template.loaders.app_directories.Loader",
]
production_loaders = [
    ("django.template.loaders.cached.Loader", [
        "django.template.loaders.filesystem.Loader",
        "django.template.loaders.app_directories.Loader",
        "path.to.custom.Loader",
    ])
]
TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [
            "templates",
        ],
        "OPTIONS": {
            "context_processors": [
                "maintenance_mode.context_processors.maintenance_mode",
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                "wagtail.contrib.settings.context_processors.settings",
            ],
            "loaders": develop_loaders if DEBUG else production_loaders,
        },
    },
]

Which works, but I wonder, am I getting the situation correctly? Do you think this is a solid solution?.

Also it took me a while because when I read the changelog for 4.1 I didn't grasp that this change would have this impact (we never specified any loader in settings before) so we expected the default behavior to be respected, which led to looking at gunicorn and docker as the first suspicious culprits, etc... so I thought that this question might be useful for other people in a similar situation.

Beachlamar

any chance you've found a more elegant solution since? – Nolie

Nope... still using this in every project since. – Beachlamar

Answer 1

Yes, you are correct.
Yes, that config will work and is well written.

Here's why.

Summary

By default (as of Django 4 / Django 5) the Django TEMPLATES setting will automatically use a wrapper of django.template.loaders.cached.Loader unless you tell it not to.

This happens by default:

"This loader is automatically enabled if OPTIONS['loaders'] isn’t specified."

https://docs.djangoproject.com/en/5.1/ref/templates/api/#django.template.loaders.cached.Loader

Why it matters

This default cache setting is cumbersome for local development, because to test any template change, you need to either...

• reload your shell instance every time you edit a template file, or
• manually invalidate the cache in your REPL (see this answer).

Your solution is the best of both worlds: you get caching for production, but you totally ignore caching for local development.

With your settings above, the local shell will always load the latest version of a template file when you run get_template('my_template.html').render()

This is not the default behavior today. Without your settings above, subsequent calls to get_template() will use the first cached version of the file unless you manually invalidate the cache / or restart your shell.

Proving your solution works:
Django 5 docs walkthrough

Note: For this answer we will only trace how to configure the settings for the DjangoTemplates template backend, not Jinja2. Where our use case is: we want to entirely avoid template caching for local development.

Let's begin our journey through the docs at the top level settings page to understand the TEMPLATES setting:

• https://docs.djangoproject.com/en/5.1/ref/settings/#templates
  • Will tell you about the top level TEMPLATES = [ .. ] key
  • Informs you that the "BACKEND" key has no default, and must be defined for each template dict in TEMPLATES = [ {"BACKEND": .. } ].
    • The backend value must be either
      'django.template.backends.django.DjangoTemplates' or
      'django.template.backends.jinja2.Jinja2'
  • Finally this API | Settings | Templates | Options section (same page) says you can find the settings.TEMPLATES[0]["OPTIONS"] key's values documented either in the DjangoTemplates docs or in the Jinja2 docs, depending on your backend.
  • Taking us to...
• https://docs.djangoproject.com/en/5.1/topics/templates/#django.template.backends.django.DjangoTemplates
  • This page tells us how to configure TEMPLATES[0]["OPTIONS"] under the section that says "DjangoTemplates engines accept the following OPTIONS:"
  • This page tells us how to configure TEMPLATES[0]["OPTIONS"]["loaders"] under the "loaders" bullet.
    • This will refer us to another link, but there's something important to note about the ["loaders"] key while we're still on this page:
    • "The first item in the tuple should be the Loader class name, and subsequent items are passed to the Loader during initialization."
    • This will be important in a moment.
  • Click "See Loader types for details."
  • Taking us to...
• https://docs.djangoproject.com/en/5.1/ref/templates/api/#template-loaders
  • This section documents the different loaders.
  • Scroll down to class cached.Loader https://docs.djangoproject.com/en/5.1/ref/templates/api/#django.template.loaders.cached.Loader
    • Notice this phrase:
    • "This loader is automatically enabled if OPTIONS['loaders'] isn’t specified."
      • This explains why caching is the default.
    • In following codeblock, we see a setting very similar to what our default loader setting will look like if unspecified by the user.
      • Recall the previous page taught us the first item in a ["loaders"] tuple is the loader to use, and the subsequent items remaining in the tuple become arguments passed to it.
      • By default, TEMPLATES[0]["OPTIONS"]["loaders"] will use "django.template.loaders.cached.Loader" as the loader (first item in the tuple) and the list of "django.template.loaders.filesystem.Loader" and "django.template.loaders.app_directories.Loader" become arguments passed into it. (This is explicitly stated in the API | Templates | Configuring an engine | class Engine | loaders section at the top of the page, see my next paragraph).

Most succinct place in the Django 5 docs that explain this default behavior

Here's an even more explicit section (still in the same page) that says the cached.Loader is the default:

• The API | Templates | Overview section at the top informs us each settings.TEMPLATES dictionary using TEMPLATES[i]["BACKEND"] == DjangoTemplates will be wrapped by an Engine instance.
• Scroll down to API | Templates | Configuring an engine | class Engine | loaders
  • By default, TEMPLATES[0]["OPTIONS"]["loaders"] will be an instance of django.template.loaders.cached.Loader (that first item in the tuple) and it will wrap
    • 'django.template.loaders.filesystem.Loader' and
    • 'django.template.loaders.app_directories.Loader' "(if and only if app_dirs is True)"
    • because these become the subsequent items in the tuple passed to the loader as args (and it "wraps" them)
    • (again, see the codeblock above, it's very similar to the default settings)

Sloop

Fabulous answer! I'll be doing exactly this with my dev shop's django setups, since it's much cleaner than our existing method. – Mordy

Answer 2

The problem is not with the cached loader but with the signal handling of your OS. The cached loader has an reset method which is called on file_changed, this way you can benefit even when debugging of the cached templates.

Do you use runserver_plus? There is an issue for it: https://github.com/django-extensions/django-extensions/issues/1766

I do not experience the issue with normal runserver command.

Balmuth

Thanks Jelmer. We work with gunicorn in development, I think I tried runserver when I didn't realize yet what was going on but I don't remember the results. Still, I'll investigate about the signal handling + gunicorn when I have a moment for it. – Beachlamar

Gunicorn does not handle the reloads of the templates as far as I know. Your proposal is the right way since it does not require reloading. – Balmuth

I am experiencing this issue with runserver - django 4.2.1 – Haga