Django autoreload raises: TypeError: unhashable type: 'types.SimpleNamespace'

Question

When I upgrade importlib_meta from version 8.4.0 to 8.5.0 (released just yesterday, Sep 11 2024), I get the following error when I start running the development server with python manage.py runserver:

   File "/app/manage.py", line 17, in main
     execute_from_command_line(sys.argv)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
     utility.execute()
   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 75, in execute
     super().execute(*args, **options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
     output = self.handle(*args, **options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 112, in handle
     self.run(**options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 119, in run
     autoreload.run_with_reloader(self.inner_run, **options)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 671, in run_with_reloader
     start_django(reloader, main_func, *args, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 660, in start_django
     reloader.run(django_main_thread)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 344, in run
     self.run_loop()
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 350, in run_loop
     next(ticker)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 390, in tick
     for filepath, mtime in self.snapshot_files():
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 411, in snapshot_files
     for file in self.watched_files():
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 304, in watched_files
     yield from iter_all_python_module_files()
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 120, in iter_all_python_module_files
     return iter_modules_and_files(modules, frozenset(_error_files))
 TypeError: unhashable type: 'types.SimpleNamespace'

I actually could narrow the problem down to the following commit https://github.com/python/importlib_metadata/commit/56b61b3dd90df2dba2da445a8386029b54fdebf3.

When I install importlib_meta just one commit before the problematic commit via pip install git+https://github.com/python/importlib_metadata@d968f6270d55f27a10491344a22e9e0fd77b5583 the error disappears. When I install importlib_meta at the problematic commit the error starts to appear.

I can not really make sense out of the Traceback and how the problem might be connected to the changes of the mentioned commit. Has anyone an idea what could cause this problem or how I can debug it?

Update (Sep 15, 2024)

The problem is solved now with version 3.20.2 of the zipp package.

Extremely

Answer 1

The problematic commit in importlib_meta puts a types.SimpleNamespace object in sys.modules["zipp.compat.overlay.zipfile"]:

zipfile = types.SimpleNamespace(**vars(importlib.import_module('zipfile')))
...
sys.modules[__name__ + '.zipfile'] = zipfile  # type: ignore[assignment]

Here's the invocation error site in django:

    modules = tuple(
        m
        for m in map(sys.modules.__getitem__, keys)
        if not isinstance(m, weakref.ProxyTypes)
    )
    # WARNING: `modules` now has `zipfile = types.SimpleNamespace(...)` in it
    return iter_modules_and_files(modules, frozenset(_error_files))


@lru_cache(maxsize=1)
def iter_modules_and_files(modules, extra_files):

From the Python docs on @functools.lru_cache:

Since a dictionary is used to cache results, the positional and keyword arguments to the function must be hashable.

types.SimpleNamespace() is not hashable:

>>> from types import SimpleNamespace
>>> hash(SimpleNamespace())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'types.SimpleNamespace'

---

I would say that this is a django problem - django should account for the fact that sys.modules is subject to monkeypatching at runtime, and can hold arbitrary objects (including unhashable ones).

Magnanimity

Awesome. Thanks for the clear outline of the problem! – Extremely

Just a follow up, Django says this is not their problem and is instead PyATS's problem: > It was concluded that PyATS's monkey-patching of sys.modules wasn't something proper that Django should account for. – Tiberius

@Tiberius That problem encountered with PyATS is unrelated to this one with importlib_meta, but it is from the same underlying issue - monkeypatching sys.modules with an unhashable object. It just shows that there’s quite a few libraries which does this. Even though the other libraries should not have done this, django is not recovering from a runtime error, so it primarily is django’s problem. – Magnanimity

Answer 2

I just hard coded

importlib_metadata==8.4.0

into my requirements.txt file and it is now working again. I'll keep it like this until they fix the bug.

Polyp