What is the best method for setting up a config file in Python

Question

I realise this question as been asked before (What's the best practice using a settings file in Python?) but seeing as this was asked 7 years ago, I feel it is valid to discuss again seeing as how technologies have evolved.

I have a python project that requires different configurations to be used based on the value of an environment variable. Since making use of the environment variable to choose a config file is simple enough, my question is as follows:

What format is seen as the best practice in the software industry for setting up a configuration file in python, when multiple configurations are needed based on the environment?

I realise that python comes with a ConfigParser module but I was wondering if it might be better to use a format such as YAML or JSON because of there raise in popularity due to their ease of use across languages. Which format is seen as easier to maintain when you have multiple configurations?

Fathom

The question you refer to was closed as too broad just a few months ago. Your version seems to be even broader. – Furmark

Also, there appears to have been a general cleanup on that question a few hours before it was closed—all obsolete answers were deleted, and one of the other answers was edited to provide updated information on YAML (and has since been updated again). – Furmark

Meanwhile, the comparative "popularity" of JSON really doesn't matter. The question is what will be most familiar or usable to your actual users with you actual app. There are clear advantages and disadvantages to INI/rc format, executable Python code, JSON, etc. If there were one answer that were the best thing for all possible uses, we wouldn't have even more competing ways to do it today than we had a decade ago, we'd all be using one. (Although at least we no longer have "really complicated XML language with an implicit but nowhere defined schema" very often…) – Furmark

Fair enough I will adjust my question to try be less value. True, their are advantages to each each format, but surely there must be some consensus within the software industry with what is seen as "best practice". At end of the day this is a very trivial design decision which must be made but I am want to find what is the common consensus within the industry or the very leas within some workspaces. – Fathom

Well, that "at least in some workspaces" is exactly the problem. Different workspaces have different answers. Partly that's driven by what technologies they're using (Django shops are more likely to have executable configs, polyglot shops that mix a lot of Python and Ruby to have YAML, etc.), but it's also driven by the specific needs of their apps/users. And neither of those considerations has a general answer that applies to "the industry" as a whole. Which is exactly why there are still so many "industry standards" for this. – Furmark

Answer 1

If you really want to use an environment-based YAML configuration, you could do so like this:

config.py

import yaml
import os

config = None

filename = os.getenv('env', 'default').lower()
script_dir = os.path.dirname(__file__)
abs_file_path = os.path.join(script_dir, filename)
with open(abs_file_path, 'r') as stream:
    try:
        config = yaml.load(stream)
    except yaml.YAMLError as exc:
        print(exc)

Numerology

Answer 2

This is really late to the party, but this is what I use and I'm pleased with it (if you're open to a pure Python solution). I like it because my configurations can be set automatically based on where this is deployed using environment variables. I haven't been using this that long so if someone sees an issue, I'm all ears.

Structure:

|--settings
   |--__init__.py
   |--config.py

config.py

class Common(object):
    XYZ_API_KEY = 'AJSKDF234328942FJKDJ32'
    XYZ_API_SECRET = 'KDJFKJ234df234fFW3424@#ewrFEWF'

class Local(Common):
    DB_URI = 'local/db/uri'
    DEBUG = True

class Production(Common):
    DB_URI = 'remote/db/uri'
    DEBUG = False

class Staging(Production):
    DEBUG = True

__init__.py

from settings.config import Local, Production, Staging
import os

config_space = os.getenv('CONFIG_SPACE', None)
if config_space:
    if config_space == 'LOCAL':
        auto_config = Local
    elif config_space == 'STAGING':
        auto_config = Staging
    elif config_space == 'PRODUCTION':
        auto_config = Production
    else:
        auto_config = None
        raise EnvironmentError(f'CONFIG_SPACE is unexpected value: {config_space}')
else:
    raise EnvironmentError('CONFIG_SPACE environment variable is not set!')

If my environment variable is set in each place where my app exists, I can bring this into my modules as needed:

from settings import auto_config as cfg

Loraleeloralie

Good practice. Just a minor detail, EnvironmentError has been deprecated since Python 3.3. ValueError might be more fitting here. – Directrix

Or consider using enum for config_space values instead of raising an error manually. – Directrix

config.py is a security risk if version controlled with API_KEY and API_SECRET. These should be stored in environment variables. – Hummocky

Answer 3

I think looking at the standard configuration for a Python Django settings module is a good example of this, since the Python Django web framework is extremely popular for commercial projects and therefore is representative of the software industry.

It doesn't get too fancy with JSON or YAML config files - It simply uses a python module called settings.py that can be imported into any other module that needs to access the settings. Environment variable based settings are also defined there. Here is a link to an example settings.py file for Django on Github:

https://github.com/deis/example-python-django/blob/master/helloworld/settings.py

Byrom

Answer 4

That really depends on your requirements, rather than the format's popularity. For instance, if you just need simple key-value pairs, an INI file would be more than enough. As soon as you need complex structures (e.g., arrays or dictionaries), I'd go for JSON or YAML. JSON simply stores data (it's more intended for automated data flow between systems), while YAML is better for human-generated (or maintained, or read) files, as it has comments, you can reference values elsewhere in the file... And on top of that, if you want robustness, flexibility, and means to check the correct structure of the file (but don't care much about the manual edition of the data), I'd go for XML.

Tonry

Answer 5

I recommend giving trapdoor a try for turn-key configuration (disclaimer: I'm the author of trapdoor).

Release

Answer 6

I also like to take advantage of the fact that you do not have to compile Python source and use plain Python files for configuration. But in the real world you may have multiple environments, each requires a different configuration, and you may also want to read some (mostly sensitive) information from env vars or files that are not in source control (to prevent committing those by mistake).

That's why I wrote this library: https://github.com/davidohana/kofiko, which let you use plain Python files for configuration, but is also able to override those config settings from .ini or env-vars, and also support customization for different environments.

Blog post about it: https://medium.com/swlh/code-first-configuration-approach-for-python-f975469433b9

Veii

Answer 7

I was looking for the solution and was surprised to see so little on this topic. Coming from .net experience I could not find suitable & simple solution to avoid storing production credentials in source control.

What I came with so far and satisfied with how it worked. This is the solution structure:

|-- lib
   |-- configuration.py
|-- main.py
|-- appconfig.development.py
|-- appconfig.production.py

First, created a config file template configuration.py:

from dataclasses import dataclass
from typing import Optional

@dataclass
class Connection:
    server: str
    database: str
    user: str
    password: str
    port: Optional[str] = None



# connection config
pg_connection: Connection

# email
email_from: str
error_emails: list[str]
smtp_server: str
smtp_port: int
smtp_user: str
smtp_password: str

Then, in the specific config file, assign values. For example appconfig.development.py:

import lib.configuration as config

config.pg_connection=config.Connection(
        server='my.postgres.server',
        port=12345,
        database='db',
        user='user',
        password='pass'
    )

config.error_emails = "[email protected]"
config.email_from = "[email protected]"
config.smtp_server = 'smtp.sendgrid.net'
config.smtp_port = 587
config.smtp_user = 'any'
config.smtp_password = 'any'

Finally, load the config in the main at first convenience, better at startup:

# load config name from ENV environment variable, default to development
env = os.environ.get('ENV', 'development') 
logging.info("loading configuration for %s", env)

spec = importlib.util.spec_from_file_location(
    name="doesnotmatter",  # anything without dots
    location="appconfig." + env + ".py",
)
my_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(my_module)
 

After loading, the configuration can be used in any module like that:

import lib.configuration as appconfig

msg =  EmailMessage()     
msg['From'] = appconfig.email_from
msg['To'] = ", ".join(appconfig.error_emails)      
server = smtplib.SMTP(appconfig.smtp_server, appconfig.smtp_port)
server.login(appconfig.smtp_user, appconfig.smtp_password)
server.send_message(msg)

Brozak