Login/Register
Deploying Django app on Heroku: Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?
Asked Answered
djangoherokuenvironment-variablesgunicorn12factor
A

2

10

My goal:

I intend to follow "The Twelve-Factor App" methodology for building my Django app on Heroku.

Introduction:

I'm following the "Getting Started with Django on Heroku" quick start guide. At the moment I have the following directory structure: 

~/Projects/
    hellodjango_rep/
        .env (empty)
        .git
        .gitignore
        Procfile
        requirements.txt
        hellodjango/
            manage.py
            hellodjango/
                __init__.py
                settings/
                urls.py
                wsgi.py

I installed django-toolbelt, created my simple Django application, started the process in my Procfile... Everything seemed to be working fine, but the problems started when I configured the application for the Heroku environment and added: 

import dj_database_url
DATABASES['default'] =  dj_database_url.config()

to the bottom of my settings.py file.

I pushed my application’s repository to Heroku, visited the app in my browser with $ heroku open successfully, but locally: dj_database_url.config() returned an empty dictionary.

Locally:

OS X 10.8.4
pip==1.4.1
virtualenv==1.10.1
virtualenvwrapper==4.1.1
wsgiref==0.1.2
Postgres.app running on Port 5432

Environment variables:

mac-pol:hellodjango_rep oubiga$ python
>>> import os
>>> os.environ

{
  'PROJECT_HOME': '/Users/oubiga/Projects'... 
  'PATH': '/usr/local/heroku/bin:/usr/local/share/python:/usr/local/bin:/Applications/Postgres.app/Contents/MacOS/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin'... 
  'HOME': '/Users/oubiga'... 
  'WORKON_HOME': '/Users/oubiga/Envs'... 
  'VIRTUALENVWRAPPER_HOOK_DIR': '/Users/oubiga/Envs'... 
  'PWD': '/Users/oubiga/Projects/hellodjango_rep'
}

hellodjango_venv:

Django==1.5.2
dj-database-url==0.2.2
dj-static==0.0.5
django-toolbelt==0.0.1
gunicorn==18.0
psycopg2==2.5.1
static==0.4

This is what I have in my wsgi.py file: 

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hellodjango.hellodjango.settings")
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())

This is what I have in my manage.py file:

import os
import sys
if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hellodjango.settings")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)

This is what I have in my Procfile: 

web: gunicorn hellodjango.hellodjango.wsgi

Environment variables: 

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ python hellodjango/manage.py shell
>>> import os
>>> os.environ 


{
  'PROJECT_HOME': '/Users/oubiga/Projects'... 
  'PATH': '/Users/oubiga/Envs/hellodjango_venv/bin:/usr/local/heroku/bin:/usr/local/share/python:/usr/local/bin:/Applications/Postgres.app/Contents/MacOS/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin', 
  'HOME': '/Users/oubiga'... 
  'WORKON_HOME': '/Users/oubiga/Envs'... 
  'VIRTUAL_ENV': '/Users/oubiga/Envs/hellodjango_venv'...
  'VIRTUALENVWRAPPER_HOOK_DIR': '/Users/oubiga/Envs'... 
  'PWD': '/Users/oubiga/Projects/hellodjango_rep'... 
  'DJANGO_SETTINGS_MODULE': 'hellodjango.settings'
}

On Heroku:

Environment variables: 

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ heroku run python hellodjango/manage.py shell
>>> import os
>>> os.environ

{
  'DATABASE_URL': 'postgres://dbuser:[email protected]:5432/dbname', 
  'HEROKU_POSTGRESQL_ORANGE_URL': 'postgres://dbuser:[email protected]:5432/dbname', 
  'LIBRARY_PATH': '/app/.heroku/vendor/lib', 'PWD': '/app'... 
  'DJANGO_SETTINGS_MODULE': 'hellodjango.settings', 
  'PYTHONHOME': '/app/.heroku/python'... 
  'PYTHONPATH': '/app/'... 
  'DYNO': 'run.9068', 
  'LD_LIBRARY_PATH': '/app/.heroku/vendor/lib'... 
  'HOME': '/app', '_': '/app/.heroku/python/bin/python', 
  'PATH': '/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin'...
}

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ heroku config
=== damp-dusk-5382 Config Vars
DATABASE_URL: postgres://dbuser:[email protected]:5432/dbname
HEROKU_POSTGRESQL_ORANGE_URL: postgres://dbuser:[email protected]:5432/dbname

Research:

Store config in the environment: from The Twelve-Factor App

@adamwiggins wrote:

The twelve-factor app stores config in environment variables... Env vars are easy to change between deploys without changing any code; unlike config files.

dj_database_url.config() is returning an empty object: from Heroku Forums

@chrisantonick replied:

... dj_database_url.config() gets the Postgres credentials from the Heroku environment variables. But, on your local machine, those variables aren't there. You have to put them in your /venv/bin/activate shell script... put the variables in there. something like
DATABASE_URL = "xxx"
export DATABASE_URL
For each thing it needs. Then... "deactivate"... and ..."activate" again to restart it.

Getting Started with Django and Heroku instructions raised ImproperlyConfigured error: from Heroku Forums

@jwpe replied:

... dj-database-url is a great utility, as it allows you to use exactly the same settings.py code in your development and production environments, as recommended in the "12 factor app principles"... what dj_database_url.config() is doing is looking for the DATABASE_URL environment variable, and then parsing it into Django's preferred format... if you haven't manually created and promoted a postgres DB on Heroku, DATABASE_URL will not be present and the ImproperlyConfigured error will be raised. Setting the default for dj_database_url.config() as your local DB URL is one way to make sure that your application will work in a development environment. However, it is not necessarily the only way. Perhaps a better alternative is to manually set DATABASE_URL in your local .env file. Then, when running your app locally using Foreman, it will be loaded as an environment variable and dj_database_url will find it. So your .env would contain:

DATABASE_URL=postgres://user:pass@localhost/dbname

Meaning that in settings.py you would only need to have:

DATABASES['default']= dj_database_url.config()

...The advantage of using a local environment variable instead of a single, hard-coded default is that your code will run in any environment where the DATABASE_URL is set. If you change the name of your local DB, or want to run your code on a different dev machine, you only need to update your .env file instead of tinkering with settings.py.

How to manage production/staging/dev Django settings?: from Heroku Forums

@rdegges replied:

... trying to get your application to behave in a way such that:

  • When you're running the app on your laptop, it uses your local Postgres server.
  • When you're running the app on your staging Heroku app, it uses the Postgres server addon.
  • When you're running the app on your production Heroku app, it uses the Postgres server addon.

The best way to accomplish this is by using environment variables!… Environment variables are the most elegant (and scalable) way to handle application configuration between different environments… Instead of having many settings files, define a single file: settings.py, and have it make use of environment variables to pull service information and credentials... On Heroku, you can set environment variables manually by running:

$ heroku config:set SOME_VARIABLE=some_value

... there's always Kenneth Reitz's great autoenv tool. This lets you define a simple .env file in your project directory… And each time you enter your project directory, those environment variables will be automatically set so that you don't have to do anything special! Just run your project and everything will work as expected: python manage.py runserver

As a First Attempt:

I manually set DATABASE_URL in my .env file: DATABASE_URL=postgres://dbuser:[email protected]:5432/dbname

But when I run $ foreman start command:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start
17:25:39 web.1  | started with pid 319
17:25:39 web.1  | 2013-09-11 17:25:39 [319] [INFO] Starting gunicorn 18.0
17:25:39 web.1  | 2013-09-11 17:25:39 [319] [INFO] Listening at: http://0.0.0.0:5000 (319)
17:25:39 web.1  | 2013-09-11 17:25:39 [319] [INFO] Using worker: sync
17:25:39 web.1  | 2013-09-11 17:25:39 [322] [INFO] Booting worker with pid: 322

and tried to open my app in the browser http://0.0.0.0:5000:

17:26:59 web.1  | 2013-09-11 10:26:59 [322] [ERROR] Error handling request
17:26:59 web.1  | Traceback (most recent call last):
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 131, in handle_request
17:26:59 web.1  |     respiter = self.wsgi(environ, resp.start_response)
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/dj_static.py", line 59, in __call__
17:26:59 web.1  |     return self.application(environ, start_response)
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 236, in __call__
17:26:59 web.1  |     self.load_middleware()
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 53, in load_middleware
17:26:59 web.1  |     raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
17:26:59 web.1  | ImproperlyConfigured: Error importing middleware django.contrib.auth.middleware: "dlopen(/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: @loader_path/../lib/libssl.1.0.0.dylib
17:26:59 web.1  |   Referenced from: /Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so
17:26:59 web.1  |   Reason: image not found"

However, dj_database_url.config() returned: 

{
  'ENGINE': 'django.db.backends.postgresql_psycopg2', 
  'NAME': 'dbname', 
  'HOST': 'ec2-23-21-196-147.compute-1.amazonaws.com', 
  'USER': 'dbuser', 
  'PASSWORD': 'dbpassword', 
  'PORT': 5432
}

As a Second Attempt:

I manually set DATABASE_URL in my .env file changing the host. I replaced "ec2-184-73-162-34.compute-1.amazonaws.com:5432" by "localhost:5000". $ deactivate and then $ workon hellodjango_venv again.

DATABASE_URL=postgres://dbuser:dbpassword@localhost:5000/dbname

But, when I run $ foreman start command: 

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start
17:38:41 web.1  | started with pid 687
17:38:41 web.1  | 2013-09-11 17:38:41 [687] [INFO] Starting gunicorn 18.0
17:38:41 web.1  | 2013-09-11 17:38:41 [687] [INFO] Listening at: http://0.0.0.0:5000 (687)
17:38:41 web.1  | 2013-09-11 17:38:41 [687] [INFO] Using worker: sync
17:38:41 web.1  | 2013-09-11 17:38:41 [690] [INFO] Booting worker with pid: 690

and tried to open my app in the browser http://0.0.0.0:5000:

17:38:46 web.1  | 2013-09-11 10:38:46 [690] [ERROR] Error handling request
17:38:46 web.1  | Traceback (most recent call last):
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 131, in handle_request
17:38:46 web.1  |     respiter = self.wsgi(environ, resp.start_response)
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/dj_static.py", line 59, in __call__
17:38:46 web.1  |     return self.application(environ, start_response)
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 236, in __call__
17:38:46 web.1  |     self.load_middleware()
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 53, in load_middleware
17:38:46 web.1  |     raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
17:38:46 web.1  | ImproperlyConfigured: Error importing middleware django.contrib.auth.middleware: "dlopen(/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: @loader_path/../lib/libssl.1.0.0.dylib
17:38:46 web.1  |   Referenced from: /Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so
17:38:46 web.1  |   Reason: image not found"

This time, dj_database_url.config() returned: 

{
  'ENGINE': 'django.db.backends.postgresql_psycopg2', 
  'NAME': 'dbname', 
  'HOST': 'localhost', 
  'USER': 'dbuser', 
  'PASSWORD': 'dbpassword', 
  'PORT': 5000
}

As a Third Attempt:

I installed autoenv mac-pol:~ oubiga$ pip install autoenv
From this Cookbook Kenneth Reitz wrote, I put:

use_env() {
  typeset venv
  venv="$1"
  if [[ "${VIRTUAL_ENV:t}" != "$venv" ]]; then
    if workon | grep -q "$venv"; then
      workon "$venv"
    else
      echo -n "Create virtualenv $venv now? (Yn) "
      read answer
      if [[ "$answer" == "Y" ]]; then
        mkvirtualenv "$venv"
      fi
    fi
  fi
}

in my .bashrc file.

I run $ foreman start command:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start
18:11:57 web.1  | started with pid 1104
18:11:57 web.1  | 2013-09-11 18:11:57 [1104] [INFO] Starting gunicorn 18.0
18:11:57 web.1  | 2013-09-11 18:11:57 [1104] [INFO] Listening at: http://0.0.0.0:5000 (1104)
18:11:57 web.1  | 2013-09-11 18:11:57 [1104] [INFO] Using worker: sync
18:11:57 web.1  | 2013-09-11 18:11:57 [1107] [INFO] Booting worker with pid: 1107

and tried to open my app in the browser http://0.0.0.0:5000: It worked!

^CSIGINT received
18:12:06 system | sending SIGTERM to all processes
18:12:06 web.1  | 2013-09-11 11:12:06 [1107] [INFO] Worker exiting (pid: 1107)
SIGTERM received
18:12:06 web.1  | 2013-09-11 18:12:06 [1104] [INFO] Handling signal: int
18:12:06 web.1  | 2013-09-11 18:12:06 [1104] [INFO] Shutting down: Master
18:12:06 web.1  | exited with code 0

But, dj_database_url.config() again returns an empty dictionary.

As a Final Attempt:

I was curious about the python manage.py runserver command and I checked it out.

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman run python hellodjango/manage.py runserver
Validating models...

0 errors found
September 11, 2013 - 18:42:37
Django version 1.5.2, using settings 'hellodjango.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

and tried to open my app in the browser http://127.0.0.1:8000/: It didn't work!

An ImportError: No module named hellodjango.urls was raised.

I replaced ROOT_URLCONF = 'hellodjango.hellodjango.urls' in my settings.py file by ROOT_URLCONF = 'hellodjango.urls' and it finally worked.

As expected, dj_database_url.config() returned an empty dictionary.

So:

Now, I feel a little overwhelmed. I'm afraid I'm misunderstanding some core concept here.

  • What's the point to use gunicorn instead of the Django development server?
  • Why does dj_database_url.config() sometimes return a fully populated dictionary, and sometimes an empty one?
  • Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?

Thank you in advance.

Avocation answered 12/9, 2013 at 0:3 Comment(4)
As someone else who is dealing with these types of issues with my first Django app, I feel your pain. I don't have an answer for you, but good luck. – Osman
@Gowie47 thank you. I was said on IRC that Postgres.app is related to it. Heroku Forums. – Avocation
Hm.. I'm using Postgres.app locally as well. I just realized I have the following for my .config() method: dj_database_url.config(default='postgres://localhost/hocg') Is that what you ended up having to do? Create the db via postgres and then give it a localhost/name url? – Osman
import dj_database_url DATABASES['default'] = dj_database_url.config() in settings.py is enough. You can check the dj-database-url repository and this question on Heroku forums for more details. – Avocation
L
1

I got stuck with the postgres as well, here's what I did in the settings.py to add local settings: 

DATABASES = {
'default': dj_database_url.config(default='postgres://<user>:<password>@<host>/<dbname>')
}

Of course you have to have created the database following postgres steps. Solution was from https://discussion.heroku.com/t/dj-database-url-config-is-returning-an-empty-object/55/9

Lalalalage answered 20/11, 2013 at 17:47 Comment(0)
T
0

I can help with the third question in your post:

"Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?"

I ran into some similar confusion when deploying my django app to heroku and hope this helps you or someone else.

Locally:

I used django-environ to help set my environment variables locally. I set all the variables in the .env file,

In the settings.py file, I imported environ, added this to the top of the file:

env = environ.Env( # set casting, default value DEBUG=(bool, False) )

and then added 'environ' to the list of INSTALLED_APPS. Finally I retrieved the environment variables with this format:

'NAME': os.environ['NAME'], 'USER': os.environ['USER'], 'PASSWORD': os.environ['PASSWORD']

Remotely:

I used heroku config to see what environment variables, if any, were already set. To set environment variables in heroku you can use the gui or heroku config:set SECRET_KEY='whateveryoursecretekeyis'

If you aren't sure what your credentials are for the heroku database, as they are different to what you're running locally, you can run heroku pg:credentials:url which lists it out for you.

One gotcha for me as a beginner was that you need to be sure that whatever variables you set in your settings.py file are declared as environment variables locally and remotely.

Another thing that might help you or someone else searching for this question is that if you were like me and first built your django app (meaning editing views building out models etc) and then started work on getting it deployed to heroku, you might want to start over, re-create the simple basic django app (barebones with the rocket ship landing page) and deploy that to heroku to make sure that pipeline works.

Then, after you're sure your app works both locally and remotely, and have edited your environment variables and set up the correct database for your project (for example postgres maybe, instead of the default database that django uses), you can go back and fill out your models and views.

Hope this helps someone!

Also just noticed your procfile seems off.

Instead of:

web: gunicorn hellodjango.hellodjango.wsgi

You'd want:

web: gunicorn hellodjango.wsgi

Since the project name is hellodjango.

Tar answered 26/7, 2022 at 13:2 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.