What's the best practice using a settings file in Python? [closed]

Question

I have a command line script that I run with a lot of arguments. I have now come to a point where I have too many arguments, and I want to have some arguments in dictionary form too.

So in order to simplify things I would like to run the script with a settings file instead. I don't really know what libraries to use for the parsing of the file. What's the best practice for doing this? I could of course hammer something out myself, but if there is some library for this, I'm all ears.

A few 'demands':

• Rather than using pickle I would like it to be a straight forward text file that can easily be read and edited.
• I want to be able to add dictionary-like data in it, i.e., some form of nesting should be supported.

A simplified pseudo example file:

truck:
    color: blue
    brand: ford
city: new york
cabriolet:
    color: black
    engine:
        cylinders: 8
        placement: mid
    doors: 2

Specs

The particular syntax of this example file is actually YAML, check Benson's answer. – Infarction

I'd suggest using python-box, see this answer. – Machute

I recommend giving trapdoor a try for turn-key configuration (disclaimer: I'm the author of trapdoor). – Archaize

As of Python 3.11, the tomlllib package became part of standard library and is the preferred way for parsing configuration in the awesome TOML format. (For earlier Python versions, the corresponding tomli package can be pip-installed.) TOML relies on a formal specification (unlike INI), is easy to read and allows comments (unlike JSON), and limits complexity (unlike YAML). – Legislate

Answer 1

You can have a regular Python module, say config.py, like this:

truck = dict(
    color = 'blue',
    brand = 'ford',
)
city = 'new york'
cabriolet = dict(
    color = 'black',
    engine = dict(
        cylinders = 8,
        placement = 'mid',
    ),
    doors = 2,
)

and use it like this:

import config
print(config.truck['color'])  

Paulinapauline

This is a pretty bad idea as if you want to allow low-privileged users to be able to change configuration files only, this way you're essentially allowing them to sneak in privileged code. – Ottie

Allowing "low-privileged" users to change config for a more privileged program is probably a questionable setup anyway. – Subsumption

This gives no protection against "low-privileged" users changing the configuration. If you import the module at the beginning of a script, change the value of one of the variables and the import the rest of the modules you can modify the configuration values even if you don't have permission to write the configuration file. – Much

Perhaps it is not in the current working directory (cwd). In that case, you have to make it visible to Python by either changing the cwd with os.chdir (use os.getcwd() to know where you are) or adding the config file location to the PYTHONPATH (not recommended). Hope this helps. – Earthen

Otherwise, if the config file is in a subdir, you can turn that subdir into a package by placing an empty __init__.py file in it. Then you could import your config file with import subdir.config – Earthen

You may also run into issues packaging your project for deployment using a tool such as py2app. The user may not be able to edit the configuration file once it's distributed since it would invalidate the app's signature. – Spec

If using python as a config file, you can use exec() to load it into a dictionary or imp.new_module to turn it into a module. This way the configuration is not in the package and can be placed in a system-standard config location if you prefer (like /etc). For more advanced usage you can also prepopulate the dict you pass to exec with objects that your config file can use as a simple DSL. – Bashemath

The main disadvantage with this (otherwise very convenient option) is that .py files are executable, so any kind of code could be run while trying to load the configuration through import. That's unacceptable from a security standpoint. – Explosion

Can't a version of this be done safely with ast.literal_eval? docs.python.org/3/library/ast.html#ast.literal_eval – Dishwasher

(1) security is not always in issue, it really depends on the project. (2) A problem I found with config files like this is if you need to create them programaticlly. Then it is difficult – Aletaaletha

Note, there is the Python-like (Python dialect/subset) Starlark language, which is intended for this use case of config files (and used by Bazel). The main feature is hermetic execution, i.e. execution cannot access the file system, network, system clock. It is safe to execute untrusted code. – Livingstone

Actually import config might not always work, see python - How to import a module given the full path? - Stack Overflow – Rivera

As others sort of allude to, this seems to become a bit more complex if you're having multiple directories of scripts within your package and you want them all to reference the settings file, as there are plenty of topics about the complexities of relative imports in Python – Goofball

@Explosion any kind of file is executable, as long as its content is understood by the interpreter. File extension has nothing to do with it. – Presuppose

Agree, @Itération122442. Every input file is a program in some language that the interpreter understands. Yet in the context of configuration files we prefer to use languages or language subsets that are much less than Turing-complete, so they cannot be used to write an interpreter: JSON, YAML, TOML. – Explosion

Answer 2

The sample config you provided is actually valid YAML. In fact, YAML meets all of your demands, is implemented in a large number of languages, and is extremely human friendly. I would highly recommend you use it. The PyYAML project provides a nice python module, that implements YAML.

To use the yaml module is extremely simple:

import yaml
config = yaml.safe_load(open("path/to/config.yml"))

Classicist

yaml is always something I turn to; the format can be from dead simple to supporting embedded python code, and the standard library does the heavy lifting of parsing and sanitation for you. – Goodyear

Agreed. For you or users writing YAML, here is the best YAML reference that I know of. The official documentation is unfortunately a spec aimed at implementers, and nothing else, but Eevee's guide is fantastic. – Rustin

For us uninitiated, that's pip3 install pyyaml to get it ready to import into python scripts. – Headphone

Beware, yaml is only friendly if you keep it very simple, it by default has tons of problematic, bordering on unsafe features. Try hitchdev.com/strictyaml instead as a safe-by-default lite alternative. – Vasectomy

See Munch, stackoverflow.com/questions/52570869/… import yaml; from munch import munchify; f = munchify(yaml.load(…)); print(fo.d.try) – Pregnancy

@HansGinzel You should make an answer on its own, as what you are suggesting is way easier to use – Adis

Answer 3

I Found this the most useful and easy to use https://wiki.python.org/moin/ConfigParserExamples

You just create a "myfile.ini" like:

[SectionOne]
Status: Single
Name: Derek
Value: Yes
Age: 30
Single: True

[SectionTwo]
FavoriteColor=Green
[SectionThree]
FamilyName: Johnson

[Others]
Route: 66

And retrieve the data like:

>>> import ConfigParser  # For Python 3 use the configparser module instead (all lowercase)
>>> Config = ConfigParser.ConfigParser()
>>> Config
<ConfigParser.ConfigParser instance at 0x00BA9B20>
>>> Config.read("myfile.ini")
['c:\\tomorrow.ini']
>>> Config.sections()
['Others', 'SectionThree', 'SectionOne', 'SectionTwo']
>>> Config.options('SectionOne')
['Status', 'Name', 'Value', 'Age', 'Single']
>>> Config.get('SectionOne', 'Status')
'Single'

Reece

For Python 3 use the configparser module instead (all lowercase) – Bacillary

This is the fastest, clearest and easiest to implement solution, since there is no implementation, just usage. :) Thank You! – Stanleystanly

Would this support nested dictionaries as asked in the question though? – Brashy

@JakubBláha no. – Tug

This does not recognise data types, so interprets all values as strings. Yaml and Json would not have this disadvantage. – Quenby

It does recognise data types. You need to call the appropriate function while getting a section. Rather than just using .get method. There are get_float, get_int, get_boolean methods to do the job. It's the best way to load properties in python. @TimoRichter – Gilbertson

Answer 4

Yaml and Json are the simplest and most commonly used file formats to store settings/config. PyYaml can be used to parse yaml. Json is already part of python from 2.5. Yaml is a superset of Json. Json will solve most uses cases except multi line strings where escaping is required. Yaml takes care of these cases too.

>>> import json
>>> config = {'handler' : 'adminhandler.py', 'timeoutsec' : 5 }
>>> json.dump(config, open('/tmp/config.json', 'w'))
>>> json.load(open('/tmp/config.json'))   
{u'handler': u'adminhandler.py', u'timeoutsec': 5}

Homerus

While more or less equivalent, json isn't nearly as human readable as yaml. Since his sample config is actually valid yaml, I'd stress that instead of json. – Classicist

Using "json.dump(config, fp, sort_keys=True, indent=4)" improves readability. – Croydon

A simple python class loading its data members from a JSON file in RAII manner – Bostick