Python: How would you save a simple settings/config file?

Question

I don't care if it's JSON, pickle, YAML, or whatever.

All other implementations I have seen are not forwards compatible, so if I have a config file, add a new key in the code, then load that config file, it'll just crash.

Are there any simple way to do this?

Stancil

I believe using the .ini-like format of the configparser module should do what you want. – Garibold

Answer 1

Configuration files in python

There are several ways to do this depending on the file format required.

ConfigParser [.ini format]

I would use the standard configparser approach unless there were compelling reasons to use a different format.

Write a file like so:

# python 2.x
# from ConfigParser import SafeConfigParser
# config = SafeConfigParser()

# python 3.x
from configparser import ConfigParser
config = ConfigParser()

config.read('config.ini')
config.add_section('main')
config.set('main', 'key1', 'value1')
config.set('main', 'key2', 'value2')
config.set('main', 'key3', 'value3')

with open('config.ini', 'w') as f:
    config.write(f)

The file format is very simple with sections marked out in square brackets:

[main]
key1 = value1
key2 = value2
key3 = value3

Values can be extracted from the file like so:

# python 2.x
# from ConfigParser import SafeConfigParser
# config = SafeConfigParser()

# python 3.x
from configparser import ConfigParser
config = ConfigParser()

config.read('config.ini')

print(config.get('main', 'key1')) # -> "value1"
print(config.get('main', 'key2')) # -> "value2"
print(config.get('main', 'key3')) # -> "value3"

# getfloat() raises an exception if the value is not a float
a_float = config.getfloat('main', 'a_float')

# getint() and getboolean() also do this for their respective types
an_int = config.getint('main', 'an_int')

JSON [.json format]

JSON data can be very complex and has the advantage of being highly portable.

Write data to a file:

import json

config = {"key1": "value1", "key2": "value2"}

with open('config1.json', 'w') as f:
    json.dump(config, f)

Read data from a file:

import json

with open('config.json', 'r') as f:
    config = json.load(f)

#edit the data
config['key3'] = 'value3'

#write it back to the file
with open('config.json', 'w') as f:
    json.dump(config, f)

YAML

A basic YAML example is provided in this answer. More details can be found on the pyYAML website.

Bernete

Nowadays, it might be good to mention TOML as well. – Pontianak

Answer 2

For a simple configuration file, I prefer JSON file, e.g. conf.json:

{
  "version": 1,
  "bind": {
    "address": "127.0.0.1",
    "port": 8080
  },
  "data": {
    "a": [1, 2, 3],
    "b": 2.5
  }
}

Then create this custom JSON configuration reader:

import json

class Dict(dict):
    """dot.notation access to dictionary attributes"""
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

class Config(object):
    @staticmethod
    def __load__(data):
        if type(data) is dict:
            return Config.load_dict(data)
        elif type(data) is list:
            return Config.load_list(data)
        else:
            return data

    @staticmethod
    def load_dict(data: dict):
        result = Dict()
        for key, value in data.items():
            result[key] = Config.__load__(value)
        return result

    @staticmethod
    def load_list(data: list):
        result = [Config.__load__(item) for item in data]
        return result

    @staticmethod
    def load_json(path: str):
        with open(path, "r") as f:
            result = Config.__load__(json.loads(f.read()))
        return result

Lastly, load it using command:

conf = Configuration.load_json('conf.json')

Now you can access your configuration using dot "." e.g.:

print(conf.version)
print(conf.bind.address)
print(conf.bind.port)
print(conf.data.a)
print(conf.data.b)

Bellyband

Answer 3

ConfigParser Basic example

The file can be loaded and used like this:

#!/usr/bin/env python

import ConfigParser
import io

# Load the configuration file
with open("config.yml") as f:
    sample_config = f.read()
config = ConfigParser.RawConfigParser(allow_no_value=True)
config.readfp(io.BytesIO(sample_config))

# List all contents
print("List all contents")
for section in config.sections():
    print("Section: %s" % section)
    for options in config.options(section):
        print("x %s:::%s:::%s" % (options,
                                  config.get(section, options),
                                  str(type(options))))

# Print some contents
print("\nPrint some contents")
print(config.get('other', 'use_anonymous'))  # Just get the value
print(config.getboolean('other', 'use_anonymous'))  # You know the datatype?

which outputs

List all contents
Section: mysql
x host:::localhost:::<type 'str'>
x user:::root:::<type 'str'>
x passwd:::my secret password:::<type 'str'>
x db:::write-math:::<type 'str'>
Section: other
x preprocessing_queue:::["preprocessing.scale_and_center",
"preprocessing.dot_reduction",
"preprocessing.connect_lines"]:::<type 'str'>
x use_anonymous:::yes:::<type 'str'>

Print some contents
yes
True

As you can see, you can use a standard data format that is easy to read and write. Methods like getboolean and getint allow you to get the datatype instead of a simple string.

Writing configuration

import os
configfile_name = "config.yaml"

# Check if there is already a configurtion file
if not os.path.isfile(configfile_name):
    # Create the configuration file as it doesn't exist yet
    cfgfile = open(configfile_name, 'w')

    # Add content to the file
    Config = ConfigParser.ConfigParser()
    Config.add_section('mysql')
    Config.set('mysql', 'host', 'localhost')
    Config.set('mysql', 'user', 'root')
    Config.set('mysql', 'passwd', 'my secret password')
    Config.set('mysql', 'db', 'write-math')
    Config.add_section('other')
    Config.set('other',
               'preprocessing_queue',
               ['preprocessing.scale_and_center',
                'preprocessing.dot_reduction',
                'preprocessing.connect_lines'])
    Config.set('other', 'use_anonymous', True)
    Config.write(cfgfile)
    cfgfile.close()

results in

[mysql]
host = localhost
user = root
passwd = my secret password
db = write-math

[other]
preprocessing_queue = ['preprocessing.scale_and_center', 'preprocessing.dot_reduction', 'preprocessing.connect_lines']
use_anonymous = True

XML Basic example

Seems not to be used at all for configuration files by the Python community. However, parsing / writing XML is easy and there are plenty of possibilities to do so with Python. One is BeautifulSoup:

from BeautifulSoup import BeautifulSoup

with open("config.xml") as f:
    content = f.read()

y = BeautifulSoup(content)
print(y.mysql.host.contents[0])
for tag in y.other.preprocessing_queue:
    print(tag)

where the config.xml might look like this

<config>
    <mysql>
        <host>localhost</host>
        <user>root</user>
        <passwd>my secret password</passwd>
        <db>write-math</db>
    </mysql>
    <other>
        <preprocessing_queue>
            <li>preprocessing.scale_and_center</li>
            <li>preprocessing.dot_reduction</li>
            <li>preprocessing.connect_lines</li>
        </preprocessing_queue>
        <use_anonymous value="true" />
    </other>
</config>

Raising

Nice code/examples. Minor comment--your YAML example isn't using YAML but INI-style format. – Scandium

It should be noted that at least the python 2 version of ConfigParser will silently convert stored list to string upon reading. Ie. CP.set('section','option',[1,2,3]) after saving and reading config will be CP.get('section','option') => '1, 2, 3' – Culbertson

Answer 4

If you want to use something like an INI file to hold settings, consider using configparser which loads key value pairs from a text file, and can easily write back to the file.

INI file has the format:

[Section]
key = value
key with spaces = somevalue

Keffiyeh

Answer 5

Save and load a dictionary. You will have arbitrary keys, values and arbitrary number of key, values pairs.

Louettalough

can I use refactoring with this? – Starinsky

Answer 6

I faced with the same problem, but in addition i'd like to read configuration variables from hard coded fields in case if configuration file doesn't exist.
My variant:

import json

class Configurator:

    def __init__(self):
        # Hard coded values if config file doesn't exist
        self.alpha: int = 42
        self.bravo: float = 3.14
        self.charlie: str = "8.8.8.8"
        self.delta: list = ["Lorem", "ipsum", "dolor", "sit", "amet"]
        self.echo: dict = {"Winter": "is coming"}

    def read_config_file(self, config_file_name: str = "config.json"):
        try:
            with open(config_file_name) as conf_file:
                for k, v in json.loads(conf_file.read()).items():
                    setattr(self, k, v)
        except Exception as e:
            print(f"Error was detected while reading {config_file_name}: {str(e)}. Hard coded values will be applied")

    def save_config_file(self, config_file_name: str = "config.json"):
        try:
            conf_items = {k: v for k, v in vars(self).items() if isinstance(v, (int, float, str, list, dict))}
            with open(config_file_name, "w") as conf_file:
                json.dump(conf_items, conf_file, sort_keys=False, indent=2)
        except Exception as e:
            print(f"Error was detected while saving {config_file_name}: {str(e)}")

from configurator import Configurator

if __name__ == '__main__':
    conf = Configurator()

    # Read config (values from file or hard coded values if file doesn't exist)
    conf.read_config_file()

    # Using values from config
    a = conf.alpha

    # Changing values in config
    conf.bravo += 1

    # Save changed config to file
    conf.save_config_file()

If config file doesn't exist, it appears after first call of conf.save_config_file(). If you change config.json after that, variables from file must "beat" hardcoded variables in the next time.

The code is a little hacky, test it before using.

Lush

Your thought process went in the right direction, but you should've considered that established libraries would've already built that functionality. ConfigParser has the fallback option: https://mcmap.net/q/130792/-python-configparser-use-defaults-if-not-in-configuration-file. Even if the file doesn't exist, I guess you could have a try catch and create an empty file, so the fallback option would work anyway. – Footmark

Answer 7

Try using ReadSettings:

from readsettings import ReadSettings
data = ReadSettings("settings.json") # Load or create any json, yml, yaml or toml file
data["name"] = "value" # Set "name" to "value"
data["name"] # Returns: "value"

Adherent

Answer 8

try using cfg4py:

1. Hierarchichal design, mulitiple env supported, so never mess up dev settings with production site settings.
2. Code completion. Cfg4py will convert your yaml into a python class, then code completion is available while you typing your code.
3. many more..

DISCLAIMER: I'm the author of this module

Ferwerda