Problem accessing config files within a Python egg

Question

I have a Python project that has the following structure:

package1
  class.py
  class2.py
  ...
package2
  otherClass.py
  otherClass2.py
  ...
config
  dev_settings.ini
  prod_settings.ini

I wrote a setup.py file that converts this into an egg with the same file structure. (When I examine it using a zip program the structure seems identical.) The funny thing is, when I run the Python code from my IDE it works fine and can access the config files; but when I try to run it from a different Python script using the egg, it can't seem to find the config files in the egg. If I put the config files into a directory relative to the calling Python script (external to the egg), it works - but that sort of defeats the purpose of having a self-contained egg that has all the functionality of the program and can be called from anywhere. I can use any classes/modules and run any functions from the egg as long as they don't use the config files... but if they do, the egg can't find them and so the functions don't work.

Any help would be really appreciated! We're kind of new to the egg thing here and don't really know where to start.

Saunderson

Answer 1

The problem is, the config files are not files anymore - they're packaged within the egg. It's not easy to find the answer in the docs, but it is there. From the setuptools developer's guide:

Typically, existing programs manipulate a package's __file__ attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs.

To access them, you need to follow the instructions for the Resource Management API.

In my own code, I had this problem with a logging configuration file. I used the API successfully like this:

from pkg_resources import resource_stream

_log_config_file = 'logging.conf'
_log_config_location = resource_stream(__name__, _log_config_file)
logging.config.fileConfig(_log_config_location)
_log = logging.getLogger('package.module')

Abbe

I'm confused... I may be a little slow because these concepts are new to me so please bear with me. I have to change the code in my Python class that uses the config file to use the pkg_resources module instead of hardcoding it? The lines I have now to load the config file are config = ConfigParser.RawConfigParser() config.read("..\\config\\" + environment + "_settings.ini")... What should I change this to? – Saunderson

@froadie: That's correct. You need something like config.read(resource_stream(__name__, "..\\config\\" + environment + "_settings.ini"). Also, you need to import the resource management module: from pkg_resources import resource_stream – Abbe

I tried something similar - config.read(pkg_resources.resource_filename(__name__, "..\\config\\" + environment + "_settings.ini")) (resource_filename instead of resource_stream because read takes the file name) - and it works when I run it but I still get an error when trying to call it from the egg, although a new one this time - KeyError: 'myPackageName\\..\\config\\dev_settings.ini' – Saunderson

I was able to fix this by moving my config folder into the actual package that uses it. Is this the right thing to do? Is there anyway to accomplish the same thing while still having the config folder on the package level? – Saunderson

You should be able to do that. Do pkg_resources.resource_filename("your_base_package.config", environment + "_settings.ini") instead of using __name__ for the first parameter. Then, create a config/__init__.py file. See if that works. (your_base_package is the parent of package1/package2/config. If the parent directory is on PYTHONPATH directly, just use "config" for the first parameter of the resource call.) – Lilongwe

Answer 2

See Setuptools' discussion of accessing pacakged data files at runtime. You have to get at your configuration file a different way if you want the script to work inside an egg. Also, for that to work, you may need to make your config directory a Python package by tossing in an empty __init__.py file.

Lilongwe