How to open a file in a package with importlib?

Question

I am trying to put the back-end of a simple web-scraping application that I'm working on into a package, but this application relies on loading from a pickled python object, which I'm unable to load into the file with importlib. Before, when I had all the code in a single file, and relied on open(), all worked fine, but now I get an error when I import the package. After this error, I tried loading the file with importlib, and could not make it work.

I am following the steps in this answer from a similar question: How to read a (static) file from inside a Python package?.

The file structure in my package is:

mypackage\
        __init__.py
        parse.py
        search.py
        categories\
                categories.pickle
                generate_categories_if_corrupted.py

The content of init.py:

from %mymodule% import search

The code where the error happens:

import importlib.resources as resources
from pickle import load
from . import categories

try:
    with resources.open_binary(categories, "categories.pickle") as cat:
        CATS = load(cat)
except FileNotFoundError:
    raise FileNotFoundError("")

The error:

Traceback (most recent call last):
  File "%mypackage%\parse.py", line 15, in <module>
    with resources.open_binary(categories, "categories.pickle") as cat:
  File "C:\Users\%me%\AppData\Local\Programs\Python\Python38\lib\importlib\resources.py", line 92, in open_binary
    _check_location(package)
  File "C:\Users\%me%\AppData\Local\Programs\Python\Python38\lib\importlib\resources.py", line 82, in _check_location
    raise FileNotFoundError(f'Package has no location {package!r}')
FileNotFoundError: Package has no location <module '%mypackage%.categories' (namespace)>

During handling of the above exception, another exception occurred:

## just a  FileNotFoundError with an error message, as expected.

How do I fix this? It's my first time trying to make my code into a package in Python.

Thanks in advance for your answers.

Rink

out of curiosity, why are there % in the import line in init.py – Franciskus

I just didn't find that the name of my package was relevant to the question, so it's just a placeholder. There is a regular windows folder path in place of it. – Rink

you mean something like from c:\foobar import search? – Franciskus

Not sure it's the issue, but I would add a mypackage/categories/__init__.py file (file can stay empty). -- Edit: I'd suggest reading this answer completely, which indeed seems to suggest that the __init__.py file is needed for importlib.resources to do its job. – Faculty

Answer 1

As per this answer on a related question (you might want to go and upvote this answer), for importlib.resources to do its job there has to be a __init__.py file in the packages.

So in your case, I believe there should be a mypackage/categories/__init__.py file (as always, that file can be left empty but it has to exist).

Faculty

Answer 2

Also oddly enough, using python3.10 solved the problem for me without adding the __init__.py as suggested by @sinoroc 's answer.

Actually I got to this question because I had no errors on python3.10 with

from importlib import resources
from blah.assets import my_data

stuff = resources.read_text(my_data, "something.json")

but I realized my ci tool (python3.8 !) was running into that FileNotFoundError as well. And haha glad I got here because okay after adding the __init__.py , the tests in my python3.8 ci environment passed 😀.

Thallophyte

I am not sure what this answer adds if not more confusion... Why the import as pkg_resources? The function read_text() is now deprecated.(as well as the open_binary mentioned in the question). Since Python 3.9 it is the importlib.resources.files() function that should be used for all use cases. – Faculty

@sinoroc, what if we use python 3.8? – Smashandgrab

@featuredpow In this case use importlib-resources it is the official "backport". – Faculty