dynamically loading django apps at runtime

Question

Is it possible to dynamically load Django apps at runtime? Usually, apps are loaded at initialization, using the INSTALLED_APPS tuple in settings.py. However, is it possible to load additional apps at runtime? I am encountering this issue in different situations. One situation, for example, arises during testing, when I would like to dynamically load or unload apps.

In order to make the problem more concrete, imagine I have a directory called apps where I put my apps and I would like to automatically install any new app that goes in there without manually editing the settings.py.

This is easy enough. Following the example code in

Django: Dynamically add apps as plugin, building urls and other settings automatically

we put the following code in settings.py to could loop over the names of all sub-directories in the app directory and increment the INSTALLED_APPS tuple in settings.py like this:

APPS_DIR = '/path_to/apps/'

for item in os.listdir(APPS_DIR):
    if os.path.isdir(os.path.join(APPS_DIR, item)):
        app_name = 'apps.%s' % item
    if app_name not in INSTALLED_APPS:
        INSTALLED_APPS += (app_name, )

After that, if I was in a Django shell, I could something like

from django.conf import settings

and the apps would be listed in settings.INSTALLED_APPS. And if I did

from django.core import management
management.call_command('syncdb', interactive=False)

that would create the necessary DB tables for the apps.

However, if I were to now add some more apps to the apps/ directory, without re-starting, these would not be listed in settings.INSTALLED_APPS, and so a subsequent call to the syncdb would have no effect.

What I would like to know is if there is something I could do --- without restarting --- to reload the settings and load/install new apps.

I have tried to directly import my settings.py, i.e. from myproject import settings

and then reload that settings using the python builtin after any app directory changes. Although settings.INSTALLED_APPS is now changed to include the newly added apps, this ultimately makes no difference. For example,

from django.db import models
models.get_apps()

shows only the original apps in apps and not the newly added ones and likewise

management.call_command('syncdb', interactive=False)

will not see the newly added apps.

As I stated above, I am thinking about this situation particularly in the context of testings where I dynamically would add or remove apps.

Ps. I working with Django 1.6, but on the advice of @RickyA, I see that there are some substantial changes to Django's treatment of applications in 1.7

https://docs.djangoproject.com/en/1.7/ref/applications/

I'm still not sure what this might mean for the problem I am facing.

Embosser

What version of Django are we talking about? 1.7 has a major overhaul in apploading. – Community

I am using 1.6. I was not aware of any changes in 1.7. I will check that out. – Embosser

Still in beta though :( – Community

Is there a question here? - Otherwise I am tempted to say The answer is Yes, and flag it for closure. – Ghazi

There is a question and I have edited my original post to hopefully make that as clear as possible. – Embosser

When you say reload are you talking about the built-in function reload function or something else? – Sprung

@SeanVieira I edited my post to clarify what I was doing with reload. Thanks. – Embosser

Answer 1

To answer my own question...

While I do not have a completely general solution to this problem, I do have one that is sufficient for the purposes of dynamically loading apps during testing.

The basic solution is simple, and I found it at a wee little bixly blog.

Continuing with my example above, if I was in a django shell and wanted to add and load some new apps that were added to my apps directory, I could do

import os
from django.conf import settings
from django.db.models import loading
from django.core import management

APPS_DIR = '/path_to/apps/'

for item in os.listdir(APPS_DIR):
    if os.path.isdir(os.path.join(APPS_DIR, item)):
        app_name = 'apps.%s' % item
    if app_name not in settings.INSTALLED_APPS:
        settings.INSTALLED_APPS += (app_name, )

and then

loading.cache.loaded = False
management.call_command('syncdb', interactive=False)

Embosser

This does not work any more with Django 2.0. django.db.models.loading is not available any more. – Luanaluanda

Answer 2

Update for Django 1.8 on how to load an app that is not loaded yet

from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management

new_app_name = "my_new_app"

settings.INSTALLED_APPS += (new_app_name, )
# To load the new app let's reset app_configs, the dictionary
# with the configuration of loaded apps
apps.app_configs = OrderedDict()
# set ready to false so that populate will work 
apps.ready = False
# re-initialize them all; is there a way to add just one without reloading them all?
apps.populate(settings.INSTALLED_APPS)

# now I can generate the migrations for the new app
management.call_command('makemigrations', new_app_name, interactive=False)
# and migrate it
management.call_command('migrate', new_app_name, interactive=False)

Longwinded

makemigrations is a bad command when called dynamically. Your migrations should be created during development, and shipped with the dynamic app. So better only call migrate. – Luanaluanda

You are right but makemigrations is a (peculiar) requirement for my project as I am moving models from one instance to a remote one – Longwinded

understand. Interesting too. So you are migrating models on-the-fly? And that works? – Luanaluanda

it's just a proof of concept, but it works; instances of this application exchange models and actual data too. – Longwinded

Answer 3

With Django 2.2 this work for me

from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management

new_app_name = "my_new_app"

settings.INSTALLED_APPS += (new_app_name, )
apps.app_configs = OrderedDict()
apps.apps_ready = apps.models_ready = apps.loading = apps.ready = False
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)

management.call_command('makemigrations', new_app_name, interactive=False)

management.call_command('migrate', new_app_name, interactive=False)

Shornick

Thanks man ! But where did you find the documentation to know that you had to put apps_ready, models_ready, loading, ready to False ? Could you explain de reason ? The documentation on that part of Django is quite obscure... And is it possible to do the reverse properly ? (unloading aka. uninstalling) – Shear

I'd be truly good to know the related documentation – Symphonia

Answer 4

It is possible to dynamically load and unload applications in tests in Django >= 1.7 (and also in the current 4.1) by a override_settings() decorator:

@override_settings(INSTALLED_APPS=[...])  # added or removed some apps
class MyTest(TestCase):
    # some tests with these apps
    def test_foo(self):
        pass

It has been possible since September 2014, not before the question.

Many other topics from the question are solved by django.apps also in Django >= 1.7. Generally: Dynamic configuration at startup is easy in the current Django. Dynamic loading and unloading after startup can't be recommended in production, even that it could work eventually.

Enrica

Answer 5

Yes! Anything (or almost everything) in Python is possible. You should use os.walk() in order to get all folders, subfolders and files within your apps path in order to get all of your apps including the nested ones.

def get_installed_apps():
    from os import walk, chdir, getcwd
    previous_path = getcwd()
    master = []
    APPS_ROOT_PATH = '/my/project/apps/folder'
    chdir(APPS_ROOT_PATH)
    for root, directories, files in walk(top=getcwd(), topdown=False):
        for file in files:
            if 'apps.py' in file:
                app_path = f"{root.replace(BASE_DIR + '/', '').replace('/', '.')}.apps"
                print(app_path)
                master.append(app_path)
    chdir(previous_path)
    return master


print(get_installed_apps())

Markettamarkey