Storing the secrets (passwords) in a separate file

Question

What's the simplest way to store the application secrets (passwords, access tokens) for a Python script? I thought it'd be a *.yml file like in Ruby but surprisingly I found that it wasn't the case. So what is it then? What are the most simplest solutions?

I want to put them in a separate file because that way I'll be able not to push that file to a GitHub repository.

Hartal

What about environment variables? – Couldst

@JoelHermanns What about environment variables? – Hartal

Possible duplicate of I need to securely store a username and password in Python, what are my options? – Vegetate

Probably the best method: https://mcmap.net/q/186518/-where-to-store-secret-keys-django – Voight

Answer 1

I think storing credentials inside another *py file is your safest bet. Then just import it. Example would look like this

config.py

username = "xy"
password = "abcd"

main.py

import config
login(config.username, config.password)

Antechamber

actually, I want to write something as well, either in this file or in another one. – Hartal

Be aware that arbitrary code in config.py will be executed. This is a concern especially if main.py runs in a different trust zone than config.py. – Week

You can also pickle a (username, password) tuple to a file and load it whenever. – Frustration

Also, having the *py file extension on the credential file would make it more likely to get pushed to a gh repo by mistake! – Caledonian

@Week so use from config import username, password to prevent anything else in config.py being executed, right? – Marking

@Marking In that case, all the code in config is still run. So if you don't trust config.py, that is still risky. If you don't trust config.py as much as the application code, this is a dangerous approach no matter how you tweak it. – Week

@Week Thanks - I've learned something today. Keep safe... – Marking

I think this is a bad idea. You credentials are stored in plain text this way. I'm looking into something like keyring pypi.org/project/keyring to fix this. – Pompom

I think this is not ideal as well. It would be very easy to accidentally upload this to a repo, as like Hans says it's also in plain text. The best way would to have stored on something like azure keyvault or something similar, but since they want to keep it in another file, I think using yaml or some other strategy not using python files is better overall. – Bragdon

This is definitely not the recommended/ best approach for storing credentials. Anyone can read them in a prod environment if it is a shared server – Voight

from configsecrets import EMAIL, API_SECRET_KEY is somewhat more readable for the new colleague, who just cloned without the extra config file. – Solvent

This is certainly not a full solution for production, but given you properly put config.py inside of .gitignore immediately, this works just fine for small projects to redact information from public repositories. It already gets you out of the habit of hardcoding sensitive information into your projects. The better way is probably to use environmental variables passed into the program for sensitive config entries. – Lignite

Answer 2

I was dealing exactly the same question and actually ended up with the same solution as kecer suggested. Since I need to use it in dozens of scripts, I've created own library. Let me share this solution with you.

credlib.py -- universal library to handle credentials

class credential:
    def __init__(self, hostname, username, password):
        self.hostname = hostname
        self.username = username
        self.password = password

mycredentials.py -- my local file to store all credentials

from credlib import credential
sys_prod = credential("srv01", "user", "pass")
sys_stg = credential("srv02", "user", "pass")
sys_db = credential("db01", "userdb", "passdb")

mysystemlib.py -- this is a general library to access my system (both new credential system and legacy is supported)

from credlib import credential

def system_login(*args): # this is new function definition
#def system_login(hostname, username, password): # this was previous function definition

    if len(args) == 1 and isinstance(args[0], credential):
        hostname = args[0].hostname
        username = args[0].username
        password = args[0].password
    elif len(args) == 3:
        hostname = args[0]
        username = args[1]
        password = args[2]
    else:
        raise ValueError('Invalid arguments')

    do_login(hostname, username, password) # this is original system login call

main.py -- main script that combines credentials and system libs

from mycredentials import sys_stg, sys_db
import mysystemlib
...
mysystemlib.system_login(sys_stg)

Please note that the legacy hostname/username/password way still works so it does not affect old scripts:

mysystemlib.system_login("srv02", "user", "pass")

This has a lot benefits:

• same credential system across all our python scripts
• files with passwords are separated (files can have more strict permissions)
• files are not stored in our git repositories (excluded via .gitignore) so that our python scripts/libs can be shared with others without exposing credentials (everyone defines their own credentials in their local files)
• if a password needs to be changed, we do it at a single place only

Entertainment

What is the .gitignore rule to ensure that this does NOT get pushed by mistake to a public repo! – Caledonian

In our case, our rule of thumb is to name all credential files as 'cred_*.py' so this's what's we have in .gitignore. Of course, this does not prevent human mistake from saving credential files with different names. Though, we never have had a problem with this. The most important for us is to separate code and credentials in separate files. – Entertainment

Answer 3

Personally I prefer to use yaml files, with the pyyaml library. Documentation here: https://pyyaml.org/wiki/PyYAMLDocumentation

Creating a .gitignore rule is very quick and painless and there is zero chances of making a mistake. You can added the rule with echo on Linux / UNIX like system with:

echo -e '*.yaml\n*.yml' >> .gitignore

Below is an example of retrieving the settings from a settings .yaml file in the same folder / location of the reader.

Code Snippets:

#!/usr/bin/env python3

import yaml
from pathlib import Path


def get_settings():
    full_file_path = Path(__file__).parent.joinpath('settings.yaml')
    with open(full_file_path) as settings:
        settings_data = yaml.load(settings, Loader=yaml.Loader)
    return settings_data

Bragdon

Feeling a bit uncomfortable with the inner workings of the import statement, I like this solution as the path to the yaml file can even be an absolute path outside of the project tree for local settings. – How

Answer 4

Probably a little late to answer over here but still I want to share the way I think is easier to do and doesn't need much coding either.

Use ini files to store your secrets like this:

[section_name]
key1 = value1
key2 = value2

Then use the configparser library to read the ini file and extract different values from it.

Apart from this if you have just few variables to work with then I think that using .env files should also work.

In both of the above methods you will be able to ensure that you secrets are not acidentally uploaded on github by just adding *.ini or *.env in your .gitignore file.

Orv