How to store Node.js deployment settings/configuration files?

A

30

718

I have been working on a few Node apps, and I've been looking for a good pattern of storing deployment-related settings. In the Django world (where I come from), the common practise would be to have a settings.py file containing the standard settings (timezone, etc), and then a local_settings.py for deployment specific settings, ie. what database to talk to, what memcache socket, e-mail address for the admins and so on.

I have been looking for similar patterns for Node. Just a config file would be nice, so it does not have to be jammed in with everything else in app.js, but I find it important to have a way to have server-specific configuration in a file that is not in source control. The same app could well be deployed across different servers with wildly different settings, and having to deal with merge conflicts and all that is not my idea of fun.

So is there some kind of framework/tool for this, or does everyone just hack something together themselves?

Aviator answered 3/5, 2011 at 12:9 Comment(1)

i really like the way the configuration is done in mean.js. basically, they store app relevant config in a seperat sort of module, based on different settings per app environment (for production, development, testing) and passing specific details through app environment variables, like secrets etc. – Jeffry 30/1, 2015 at 9:39

A

219

Much later, I found a pretty good Node.js module for managing configuration: nconf.

A simple example:

var nconf = require('nconf');

// First consider commandline arguments and environment variables, respectively.
nconf.argv().env();

// Then load configuration from a designated file.
nconf.file({ file: 'config.json' });

// Provide default values for settings not provided above.
nconf.defaults({
    'http': {
        'port': 1337
    }
});

// Once this is in place, you can just use nconf.get to get your settings.
// So this would configure `myApp` to listen on port 1337 if the port
// has not been overridden by any of the three configuration inputs
// mentioned above.
myApp.listen(nconf.get('http:port'));

It also supports storing settings in Redis, writing configuration files, and has a fairly solid API, and is also backed by one of the more well-respected Node.js shops, Nodejitsu, as part of the Flatiron framework initiative, so it should be fairly future-proof.

Check out nconf at Github.

Aviator answered 19/12, 2011 at 15:28 Comment(16)

Maybe a dumb question but I haven't seen a clear explaination: Where do I set node environment variables? I am already using nconf but it's not clear where I would set environmental variables. Is it in nginx/apache? Is it another config file? – Maize 12/2, 2013 at 23:42

I don't think use .json file as config is a good idea, since comments are not allowed. – Burnet 11/4, 2013 at 9:3

This looks great. I think you'll surprise a lot of Unixheads if the config file overrides command-line options and environment variables. We're used to the following order of ascending precedence: config file(s), environment variables, command-line options. – Immerge 14/4, 2013 at 10:32

@Immerge Wait until you find out that boolean options are always set on argv, therefore breaking precedence... :/ – Tova 21/5, 2013 at 20:33

@DanielC.Sobral It's a real shame. Oh, and LTNS! :-) – Immerge 24/5, 2013 at 9:35

would it make sense to attach nconf to the process variable or perhaps nconf does that out of the box? – Floriated 18/4, 2015 at 6:55

Using a custom json format with nconf solves many of the issues people have with json files, including commenting ability. A few solutions are provided here: github.com/indexzero/nconf/issues/113 – Tuque 29/10, 2015 at 19:36

Plus, JS files can be require'd and fed to an nconf literal store! – Tuque 30/10, 2015 at 9:49

I prefer the nconf approach because it allows me to have environment-variable-based config on the server and file-based config for development. Locally, I prefer not to use environment-variables because they are global, which might be a problem when working on multiple projects. For the server, I use the AWS dashboard, which is actually quite nice. Keeping config out of source-control means that eb deploy continues to work ootb. – Toner 11/5, 2016 at 17:7

The benefit to using a system like this over require is that you will use the process.env variables. Which is beneficial during automated deploys where you don't push a config.js file with secrets. – Aquilar 30/12, 2016 at 2:10

@KeithHolliday There are other ways to handle keys. Using environment variables doesn't solve this problem because something still needs to know what the key is to set an environment variable with that key. If you're doing this manually, you could do the same thing in a config file - have a keys config file (outside of version control) that you set manually and load into your app somewhere. – Womanly 2/3, 2017 at 1:34

I would definitely not recommend having environment variables and a config file and the defaults. Using environment variables and config files is needlessly complicated. – Womanly 2/3, 2017 at 1:36

@BT How would you use a config file with sensitive variables to deploy to a system without access the server? For example, using Kubernetes or Heroku, you must separate the config variables from your app. Using a config file wouldn't be easy (or even possible) without loging into individual nodes. – Aquilar 2/3, 2017 at 16:36

@KeithHolliday I don't quite follow the case you're explaining, but I certainly agree with making config completely separate from your app. If for example, you deploy a frontend bundle for use in a web app, the public configuration should be a completely separate bundle. If you're talking about writing keys to a server, then the keys should be separate from the rest of the config (I consider keys "data" akin to database data, rather than config-proper). Also, if you don't have access to the server, wouldn't environment variables be inaccessible? – Womanly 4/3, 2017 at 7:28

While nconf may be stilll active, apparently "part of the Flatiron framework initiative, so it should be fairly future-proof" was not a reliable indicator... trying to access flatironjs.org results in "Server not found: Firefox can’t find the server at www.flatironjs.org." – Dealt 30/10, 2017 at 19:58

@Immerge The order is actually reversed, i.e. nconf uses the first setting it sees (from argv, ENV, config.json, defaults). So the outcome is as Unixheads would expect. – Quahog 5/9, 2020 at 1:5

S

838

I use a package.json for my packages and a config.js for my configuration, which looks like:

var config = {};

config.twitter = {};
config.redis = {};
config.web = {};

config.default_stuff =  ['red','green','blue','apple','yellow','orange','politics'];
config.twitter.user_name = process.env.TWITTER_USER || 'username';
config.twitter.password=  process.env.TWITTER_PASSWORD || 'password';
config.redis.uri = process.env.DUOSTACK_DB_REDIS;
config.redis.host = 'hostname';
config.redis.port = 6379;
config.web.port = process.env.WEB_PORT || 9980;

module.exports = config;

I load the config from my project:

var config = require('./config');

and then I can access my things from config.db_host, config.db_port, etc... This lets me either use hardcoded parameters, or parameters stored in environmental variables if I don't want to store passwords in source control.

I also generate a package.json and insert a dependencies section:

"dependencies": {
  "cradle": "0.5.5",
  "jade": "0.10.4",
  "redis": "0.5.11",
  "socket.io": "0.6.16",
  "twitter-node": "0.0.2",
  "express": "2.2.0"
}

When I clone the project to my local machine, I run npm install to install the packages. More info on that here.

The project is stored in GitHub, with remotes added for my production server.

Shellans answered 3/5, 2011 at 13:54 Comment(20)

what happens when you have different config settings for dev vs. prod? – Iseabal 20/9, 2012 at 6:28

I haven't but here's one way to do it.. for each env, set the env name in an ENV variable. Then in this file, its just javascript.. use a case or if statement to selectively load the appropriate variables. You could even make a separate config subfile for each env, and in the if statement, reload the subfile here into a subconfig var, and export that subconfig var to the main config.. All i'm basically trying to say is that its just js, so you can be creative – Shellans 21/9, 2012 at 0:48

what process.env? where does it locate? And how to set it? – Hammurabi 6/1, 2013 at 16:3

I'm just wondering how secure this approach is ? Can this config.js not be read in a fairly easy way ? – Pigeon 12/7, 2013 at 22:49

its source code.. it can be read as little or as much as the rest of the source code it goes with, depending on how you set your permissions – Shellans 13/7, 2013 at 1:32

What was your thinking behind the choice of not nesting the values syntactically? – Konstanz 15/7, 2013 at 14:18

I was thinking "wow.. i've been looking at node.js for a few hours and my app is already working.. btw, maybe i'll share this random bit of code I came up with" – Shellans 15/7, 2013 at 18:8

Each configuration item has its full name included on the same line which is very convenient if using shell scripts with sed expressions to deploy. – Grammarian 26/11, 2013 at 12:12

This is a great approach for the common scenario. It comes unstuck if you have a non-node process that shares the config file. For example if you have a legacy app and node sharing the config. JSON is a good common format. But again, maybe I am just dealing with an edge case. – Savoyard 12/5, 2014 at 11:28

A plain js file is really the simplest and most flexible approach! I sometimes freeze the config object (and all its embedded objects) to make it immutable... Might be a good idea in a larger development group I guess. – Marrowfat 20/11, 2014 at 15:4

Be wary of this approach if your project has any security considerations, as it means checking things like DB passwords, auth tokens, etc. into your repository, which is (a) insecure and (b) resistant to collaboration. Use nconf or similar, see below. – Neddra 8/7, 2015 at 15:56

Can't you still use environment variables to store those pass words? Isn't that what this line is for: config.twitter.password= process.env.TWITTER_PASSWORD || 'password'; – Floatable 28/10, 2015 at 20:10

This is how I separate configurations: I not only have a config.js, which gets called in the code, but also a config folder. I have several versions of config.js in there, such as config/prod.js and config/dev.js. I exclude the config.js and /config folder in my .gitignore, so the folder only exists on my private dev server. Then I copy the configurations to their respective environments, like with scp /path/to/project/config/prod.js prodsrvr:/path/to/project/config.js. – Mannerism 30/8, 2016 at 23:50

For the sake of security you can now use attosol.com/… so that all the secrets are saved to Azure Key Vault instead. – Flavorous 13/6, 2017 at 10:41

This is static config only. You can't store anything dynamically with this approach. For example, settings that are hard to store in DB. – Darbydarce 8/10, 2017 at 9:59

This answer describes how to handle different configs for prod and dev environments. – Minh 20/12, 2017 at 19:1

@Shellans Just to be safe and prevent those settings from being changed in code I would suggest you update your answer and do an object.freeze on each of your objects such as : Object.freeze(config.twitter); – Recuperative 26/10, 2018 at 3:40

hii how can I get my site_url in my ejs file . I have given var config = require('./config'); in my app.js and in config.js all configs are present. – Drily 14/2, 2019 at 6:36

hello how to exlude the config.js from bundling in webpack 4 – Mada 18/4, 2020 at 20:31

Tht's the only answer which shows how to override variable with ENV_VAAR – Trembly 1/7, 2021 at 15:53

P

269

You can require JSON files as of Node v0.5.x (referencing this answer)

config.json:

{
    "username" : "root",
    "password" : "foot"
}

app.js:

var config = require('./config.json');
log_in(config.username, config.password);

Pyrrha answered 3/2, 2013 at 23:43 Comment(6)

Not so impressed with that feature. You could require("./config.js") and you get the ability to add comments to config files which I consider very important, and other bells and whistles. If you config is just properties and no code you loose nothing by require(config.js) with you JSON prefixed by exports.config = – Spurn 29/10, 2013 at 18:1

@Spurn you're right but there used to be a big discussion going on about the 'correctness' / usability of using dumb vs. smart templating systems that told me: (1) you typically want a declarative / dumb language for templating / options (2) it's a bad idea to reconstruct an "almost-PL" to just do templating (or configuration)—better to re-use your existing real PL with known behaviors. in so far +1 for recycling JS to do user settings; -1 for not going the declarative approach. we've seen some pretty complex config stuff done the declarative way; my gut tells me this is the way to go. – Darien 10/11, 2013 at 12:33

No intellisense on objects from json files in VScode (end 2017). Fully working intellisense for objects from module.exports. – Bareback 31/12, 2017 at 19:13

using this method, where would you store the config.json file? And how can the user edit it? – Rhinitis 23/11, 2020 at 13:27

@Rhinitis you can put it anywhere in your project, it's just a file. If it's deeper in your project, you can write the whole path: require('./path/to/config.json'). If you want a user to edit it, you should switch to using fs.read and fs.write to read and write to the file. – Pyrrha 1/12, 2020 at 22:19

@Pyrrha the problem with require is that tell webpack to pack it unless you put it in a static folder. I would think a hacker could access it. Point being, if web pack, packs it then you can't customize based on the environment dev,qa,stage,preprod, etc.. – Xyloid 21/1, 2022 at 13:31

A

219

Much later, I found a pretty good Node.js module for managing configuration: nconf.

A simple example:

var nconf = require('nconf');

// First consider commandline arguments and environment variables, respectively.
nconf.argv().env();

// Then load configuration from a designated file.
nconf.file({ file: 'config.json' });

// Provide default values for settings not provided above.
nconf.defaults({
    'http': {
        'port': 1337
    }
});

// Once this is in place, you can just use nconf.get to get your settings.
// So this would configure `myApp` to listen on port 1337 if the port
// has not been overridden by any of the three configuration inputs
// mentioned above.
myApp.listen(nconf.get('http:port'));