Securely storing environment variables in GAE with app.yaml

Question

I need to store API keys and other sensitive information in app.yaml as environment variables for deployment on GAE. The issue with this is that if I push app.yaml to GitHub, this information becomes public (not good). I don't want to store the info in a datastore as it does not suit the project. Rather, I'd like to swap out the values from a file that is listed in .gitignore on each deployment of the app.

Here is my app.yaml file:

application: myapp
version: 3 
runtime: python27
api_version: 1
threadsafe: true

libraries:
- name: webapp2
  version: latest
- name: jinja2
  version: latest

handlers:
- url: /static
  static_dir: static

- url: /.*
  script: main.application  
  login: required
  secure: always
# auth_fail_action: unauthorized

env_variables:
  CLIENT_ID: ${CLIENT_ID}
  CLIENT_SECRET: ${CLIENT_SECRET}
  ORG: ${ORG}
  ACCESS_TOKEN: ${ACCESS_TOKEN}
  SESSION_SECRET: ${SESSION_SECRET}

Any ideas?

Filide

I wish GAE would add the option to set instance env vars via the developer console (like every other PaaS with which I am familiar). – Givens

You can use datastore. Please refer to this answer: https://mcmap.net/q/182142/-set-environment-variables-in-gae-control-panel – Jodee

Expanding on mustilica's comment above about using the datastore. See my answer below for code I use in my projects to do this: stackoverflow.com/a/35261091#35261091. In effect, it lets you edit environment variables from the developer console, and placeholder values are created automatically. – Vasili

Thanks mustilica and Martin . We have actually been using the datastore approach awhile and I agree it is the best solution to this problem. Easier to do with a CI/CD setup than the json file approach, IMO. – Givens

Also, I created a gist to share how we are using datastore - gist.github.com/SpainTrain/6bf5896e6046a5d9e7e765d0defc8aa8 – Givens

2019 and GAE still hasn't fixed this issue :/ – Screwball

Here the solution github.com/marketplace/actions/… – Unriddle

Answer 1

If it's sensitive data, you should not store it in source code as it will be checked into source control. The wrong people (inside or outside your organization) may find it there. Also, your development environment probably uses different config values from your production environment. If these values are stored in code, you will have to run different code in development and production, which is messy and bad practice.

In my projects, I put config data in the datastore using this class:

from google.appengine.ext import ndb

class Settings(ndb.Model):
  name = ndb.StringProperty()
  value = ndb.StringProperty()

  @staticmethod
  def get(name):
    NOT_SET_VALUE = "NOT SET"
    retval = Settings.query(Settings.name == name).get()
    if not retval:
      retval = Settings()
      retval.name = name
      retval.value = NOT_SET_VALUE
      retval.put()
    if retval.value == NOT_SET_VALUE:
      raise Exception(('Setting %s not found in the database. A placeholder ' +
        'record has been created. Go to the Developers Console for your app ' +
        'in App Engine, look up the Settings record with name=%s and enter ' +
        'its value in that record\'s value field.') % (name, name))
    return retval.value

Your application would do this to get a value:

API_KEY = Settings.get('API_KEY')

If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.

I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!

The code above uses the ndb library which uses memcache and the datastore under the hood, so it's fast.

---

Update:

jelder asked for how to find the Datastore values in the App Engine console and set them. Here is how:

1. Go to https://console.cloud.google.com/datastore/

2. Select your project at the top of the page if it's not already selected.

3. In the Kind dropdown box, select Settings.

4. If you ran the code above, your keys will show up. They will all have the value NOT SET. Click each one and set its value.

Hope this helps!

Vasili

Of all the provided answers, this seems closest to how Heroku handles things. Being rather new to GAE, I don't quite understand where in the Developers Console to find the placeholder record. Can you explain, or for bonus points, post screenshots? – Bordelaise

Thanks for the comment, jelder. I added screenshots above per your request. – Vasili

dam~… with all due respect to gcloud, it looks quite bad to have to use another service for this specific need. Besides that, google do provide a "100%-herokuish" approach for env vars within firebase functions, but don't for gcloud functions (at least undocumented… if I'm not wrong) – Filide

Here is a gist based on your approach that adds uniqueness and environment variable fallback - gist.github.com/SpainTrain/6bf5896e6046a5d9e7e765d0defc8aa8 – Givens

Nice code, Spain Train! The fallback to environment variables looks very useful. – Vasili

@MartinOmander What package do I need to install for ndb? Please – Powel

@Powel ndb is built into App Engine so no installation is needed. You just have to include this (per my code above): "from google.appengine.ext import ndb" – Vasili

@Filide Non-Firebase functions do support env vars (now, at least). – Prochronism

Big downside of this approach is that you are querying the datastore each time you access a environment variable/setting. It should be one access per deployment. – Slyviasm

Christopher Oezbek: I hear your concern about querying the datastore every time you need an environment setting. If you are thinking about performance, be aware that NDB does caching behind the scenes: cloud.google.com/appengine/docs/standard/python/ndb/cache – Vasili

@MartinOmander won't you have to authenticate for datastore? How do you keep those authentication details secret? – Anaerobe

@Anaerobe - An App Engine app is automatically authenticated to its own datastore, no authentication details needed. It's pretty neat :-) – Vasili

@Powel - You asked about how to install ndb. No installation is needed; it comes with App Engine: cloud.google.com/appengine/docs/standard/python/ndb – Vasili

They seem to be migrating datastore to "firestore in datastore mode" it is not enitrely clear to me what this means but we can expect some changes to this flow. – Eirena

It's been a few years, and there is better and more secure technology now. You should consider using Secret Manager, which is part of Google Cloud Platform. – Vasili

Answer 2

This solution is simple but may not suit all different teams.

First, put the environment variables in an env_variables.yaml, e.g.,

env_variables:
  SECRET: 'my_secret'

Then, include this env_variables.yaml in the app.yaml

includes:
  - env_variables.yaml

Finally, add the env_variables.yaml to .gitignore, so that the secret variables won't exist in the repository.

In this case, the env_variables.yaml needs to be shared among the deployment managers.

Chrissy

Just to add what may not be obvious to some, your environment variables would then be found in process.env.MY_SECRET_KEY and if you need these environment variables in your local dev environment you can use the node dotenv package – Adapt

How would the env_variables.yaml get to all instances is a missing piece of the puzzle. – Slyviasm

Also: How to use this locally? – Slyviasm

@ChristopherOezbek 1. How to deploy? Just use gcloud app deploy as you normally do to deploy to Google Cloud. 2. How to set secret environment variables locally? There are many ways. You can just use export in the command prompt or use any tools like @DaveKiss suggested. – Chrissy

This is the simplest solution. Secrets can be accessed in your application via os.environ.get('SECRET'). – Amphioxus

This works for the Node.js environment as well. But I think there shouldn't be a space before - env_variables.yaml (in the second line after includes:). – Gearbox

@how will this env_variable.yaml be available in the code deployed at google app engine? – Shad

Answer 3

This didn't exist when you posted, but for anyone else who stumbles in here, Google now offers a service called Secret Manager.

It's a simple REST service (with SDKs wrapping it, of course) to store your secrets in a secure location on google cloud platform. This is a better approach than Data Store, requiring extra steps to see the stored secrets and having a finer-grained permission model -- you can secure individual secrets differently for different aspects of your project, if you need to.

It offers versioning, so you can handle password changes with relative ease, as well as a robust query and management layer enabling you to discover and create secrets at runtime, if necessary.

Python SDK

Example usage:

from google.cloud import secretmanager_v1beta1 as secretmanager

secret_id = 'my_secret_key'
project_id = 'my_project'
version = 1    # use the management tools to determine version at runtime

client = secretmanager.SecretManagerServiceClient()

secret_path = client.secret_version_path(project_id, secret_id, version)
response = client.access_secret_version(secret_path)
password_string = response.payload.data.decode('UTF-8')

# use password_string -- set up database connection, call third party service, whatever

Astigmatism

This should be the new correct answer. Secret Manager is still in Beta, but this is the way forward when working with environment variables. – Holbrooke

@KingLeon, would use of this mean having to refactor a bunch of os.getenv('ENV_VAR') s? – Shiism

I put code similar to above in a function, then I use something like SECRET_KEY = env('SECRET_KEY', default=access_secret_version(GOOGLE_CLOUD_PROJECT_ID, 'SECRET_KEY', 1)). Setting the default to use the access_secret_version – Holbrooke

Also, I am using django-environ. github.com/joke2k/django-environ – Holbrooke

I need to ask a stupid question, but what you call secret_id = 'my_secret_key' isn't then present in your version control? – Tadio

@Tadio yes, but that’s fine. It’s an identifier to the secret value and nothing more. – Astigmatism

@Tadio access to the value itself is handled by google cloud IAM – Astigmatism

ah it's a key of a key-value store, ok. I though secret_id meant something else. – Tadio

the access_secret_version function should be used like client.access_secret_version(request={"name": secret_path}) – Inmesh

Answer 4

My approach is to store client secrets only within the App Engine app itself. The client secrets are neither in source control nor on any local computers. This has the benefit that any App Engine collaborator can deploy code changes without having to worry about the client secrets.

I store client secrets directly in Datastore and use Memcache for improved latency accessing the secrets. The Datastore entities only need to be created once and will persist across future deploys. of course the App Engine console can be used to update these entities at any time.

There are two options to perform the one-time entity creation:

• Use the App Engine Remote API interactive shell to create the entities.
• Create an Admin only handler that will initialize the entities with dummy values. Manually invoke this admin handler, then use the App Engine console to update the entities with the production client secrets.

Acceptant

Not complicated at all. Thanks app engine. – Arch

Answer 5

Best way to do it, is store the keys in a client_secrets.json file, and exclude that from being uploaded to git by listing it in your .gitignore file. If you have different keys for different environments, you can use app_identity api to determine what the app id is, and load appropriately.

There is a fairly comprehensive example here -> https://developers.google.com/api-client-library/python/guide/aaa_client_secrets.

Here's some example code:

# declare your app ids as globals ...
APPID_LIVE = 'awesomeapp'
APPID_DEV = 'awesomeapp-dev'
APPID_PILOT = 'awesomeapp-pilot'

# create a dictionary mapping the app_ids to the filepaths ...
client_secrets_map = {APPID_LIVE:'client_secrets_live.json',
                      APPID_DEV:'client_secrets_dev.json',
                      APPID_PILOT:'client_secrets_pilot.json'}

# get the filename based on the current app_id ...
client_secrets_filename = client_secrets_map.get(
    app_identity.get_application_id(),
    APPID_DEV # fall back to dev
    )

# use the filename to construct the flow ...
flow = flow_from_clientsecrets(filename=client_secrets_filename,
                               scope=scope,
                               redirect_uri=redirect_uri)

# or, you could load up the json file manually if you need more control ...
f = open(client_secrets_filename, 'r')
client_secrets = json.loads(f.read())
f.close()

Matless

Definitely in the right direction, but this does not address the issue of swapping out the values in app.yaml on deployment of the app. Any ideas there? – Filide

So have a different client_secrets file for each environment. Eg client_secrets_live.json, client_secrets_dev.json, client_secrets_pilot.json etc, then use python logic to determine which server you are on and load up the appropriate json file. The app_identity.get_application_id() method may be useful to auto detect which server you are on. Is this the kind of thing you mean? – Matless

@BenGrunfeld see my answer. My solution does exactly this. I don't see how this answer solves the question. I assume the goal is to keep secret configuration out of git, and use git as part of deployment. Here, this file still needs to be somewhere and pushed into the deployment process. This can be something you do in your app, but you would just use the techniques I highlighted, perhaps storing in another file if you want to use this vs. app.yaml. If I understand the question, it's something akin to shipping an open source app with the library maker's actual client secret or prod. key. – Kindness

"I'd like to swap out the values from a file that is listed in .gitignore on each deployment of the app" - my example solves this problem. If the question was "I'd like to deploy my app to different servers without deploying private keys etc", then correct, my answer doesn't address this. but that wasn't (as I understand it), the question. – Matless

It took me a while to get my head around it, but I think this is the correct approach. You're not mixing application settings (app.yaml) with secret keys and confidential information, and what I really like is that you're using the Google workflow to accomplish the task. Thanks @GwynHowell. =) – Filide

A similar approach would be to place that JSON file in a known location in the app's default GCS bucket (cloud.google.com/appengine/docs/standard/python/…). – Givens

Because this question appears so many times in SO, I've built a Python package to solve this problem generally. Use pip install gae_app_settings or learn about it on PyPI for more information. – Renounce

Answer 6

This solution relies on the deprecated appcfg.py

You can use the -E command line option of appcfg.py to setup the environment variables when you deploy your app to GAE (appcfg.py update)

$ appcfg.py
...
-E NAME:VALUE, --env_variable=NAME:VALUE
                    Set an environment variable, potentially overriding an
                    env_variable value from app.yaml file (flag may be
                    repeated to set multiple variables).
...

Silence

Can you query those environment variables somewhere after deploy? (I hope not.) – Alvie

Is there a way to pass environment variables in like this using the gcloud utility? – Dix

Answer 7

Most answers are outdated. Using google cloud datastore is actually a bit different right now. https://cloud.google.com/python/getting-started/using-cloud-datastore

Here's an example:

from google.cloud import datastore
client = datastore.Client()
datastore_entity = client.get(client.key('settings', 'TWITTER_APP_KEY'))
connection_string_prod = datastore_entity.get('value')

This assumes the entity name is 'TWITTER_APP_KEY', the kind is 'settings', and 'value' is a property of the TWITTER_APP_KEY entity.

Flowerer

Answer 8

With github action instead of google cloud triggers (Google cloud triggers aren't able to find it's own app.yaml and manage the freaking environment variable by itself.)

Here is how to do it:

My environment : App engine, standard (not flex), Nodejs Express application, a PostgreSQL CloudSql

First the setup :

1. Create a new Google Cloud Project (or select an existing project).

2. Initialize your App Engine app with your project.

[Create a Google Cloud service account][sa] or select an existing one.

3. Add the the following Cloud IAM roles to your service account:

    App Engine Admin - allows for the creation of new App Engine apps

    Service Account User - required to deploy to App Engine as service account

    Storage Admin - allows upload of source code

    Cloud Build Editor - allows building of source code

[Download a JSON service account key][create-key] for the service account.

4. Add the following [secrets to your repository's secrets][gh-secret]:

    GCP_PROJECT: Google Cloud project ID

    GCP_SA_KEY: the downloaded service account key

The app.yaml

runtime: nodejs14
env: standard
env_variables:
  SESSION_SECRET: $SESSION_SECRET
beta_settings:
  cloud_sql_instances: SQL_INSTANCE

Then the github action

name: Build and Deploy to GKE

on: push

env:
  PROJECT_ID: ${{ secrets.GKE_PROJECT }}
  DATABASE_URL: ${{ secrets.DATABASE_URL}}
jobs:
  setup-build-publish-deploy:
    name: Setup, Build, Publish, and Deploy
    runs-on: ubuntu-latest

steps:
 - uses: actions/checkout@v2
 - uses: actions/setup-node@v2
   with:
    node-version: '12'
 - run: npm install
 - uses: actions/checkout@v1
 - uses: ikuanyshbekov/[email protected]
   env:
    SESSION_SECRET: ${{ secrets.SESSION_SECRET }}  
 - shell: bash
   run: |
        sed -i 's/SQL_INSTANCE/'${{secrets.DATABASE_URL}}'/g' app.yaml
 - uses: actions-hub/gcloud@master
   env:
    PROJECT_ID: ${{ secrets.GKE_PROJECT }}
    APPLICATION_CREDENTIALS: ${{ secrets.GCLOUD_AUTH }}
    CLOUDSDK_CORE_DISABLE_PROMPTS: 1
   with:
    args: app deploy app.yaml

To add secrets into github action you must go to : Settings/secrets

Take note that I could handle all the substitution with the bash script. So I would not depend on the github project "ikuanyshbekov/[email protected]"

It's a shame that GAE doesn't offer an easiest way to handle environment variable for the app.yaml. I don't want to use KMS since I need to update the beta-settings/cloud sql instance.. I really needed to substitute everything into the app.yaml.

This way I can make a specific action for the right environment and manage the secrets.

Swats

Answer 9

It sounds like you can do a few approaches. We have a similar issue and do the following (adapted to your use-case):

• Create a file that stores any dynamic app.yaml values and place it on a secure server in your build environment. If you are really paranoid, you can asymmetrically encrypt the values. You can even keep this in a private repo if you need version control/dynamic pulling, or just use a shells script to copy it/pull it from the appropriate place.
• Pull from git during the deployment script
• After the git pull, modify the app.yaml by reading and writing it in pure python using a yaml library

The easiest way to do this is to use a continuous integration server such as Hudson, Bamboo, or Jenkins. Simply add some plug-in, script step, or workflow that does all the above items I mentioned. You can pass in environment variables that are configured in Bamboo itself for example.

In summary, just push in the values during your build process in an environment you only have access to. If you aren't already automating your builds, you should be.

Another option option is what you said, put it in the database. If your reason for not doing that is that things are too slow, simply push the values into memcache as a 2nd layer cache, and pin the values to the instances as a first-layer cache. If the values can change and you need to update the instances without rebooting them, just keep a hash you can check to know when they change or trigger it somehow when something you do changes the values. That should be it.

Kindness

FWIW, this approach most closely follows the config factor in the 12 Factor App guidelines (12factor.net) – Givens

Answer 10

Just wanted to note how I solved this problem in javascript/nodejs. For local development I used the 'dotenv' npm package which loads environment variables from a .env file into process.env. When I started using GAE I learned that environment variables need to be set in a 'app.yaml' file. Well, I didn't want to use 'dotenv' for local development and 'app.yaml' for GAE (and duplicate my environment variables between the two files), so I wrote a little script that loads app.yaml environment variables into process.env, for local development. Hope this helps someone:

yaml_env.js:

(function () {
    const yaml = require('js-yaml');
    const fs = require('fs');
    const isObject = require('lodash.isobject')

    var doc = yaml.safeLoad(
        fs.readFileSync('app.yaml', 'utf8'), 
        { json: true }
    );

    // The .env file will take precedence over the settings the app.yaml file
    // which allows me to override stuff in app.yaml (the database connection string (DATABASE_URL), for example)
    // This is optional of course. If you don't use dotenv then remove this line:
    require('dotenv/config');

    if(isObject(doc) && isObject(doc.env_variables)) {
        Object.keys(doc.env_variables).forEach(function (key) {
            // Dont set environment with the yaml file value if it's already set
            process.env[key] = process.env[key] || doc.env_variables[key]
        })
    }
})()

Now include this file as early as possible in your code, and you're done:

require('../yaml_env')

Bushel

Is this still the case? Because I'm using a .env file with the secret variables. I'm not duplicating them in my app.yaml file and my deployed code still works. I'm worried what happens to the .env file in the cloud though. Does it get encrypted or something? How can I make sure no one access the gcloud .env file variables once it is deployed? – Flan

This is not needed at all because GAE automatically adds all variables defined in the app.yaml file to the node environment. Basically this is the same as dotenv does with variables defined in the .env package. But I'm wondering how you have to setup CD as you cannot push app.yaml with env vars to a VCS or a pipeline... – Trainbearer

Answer 11

You should encrypt the variables with google kms and embed it in your source code. (https://cloud.google.com/kms/)

echo -n the-twitter-app-key | gcloud kms encrypt \
> --project my-project \
> --location us-central1 \
> --keyring THEKEYRING \
> --key THECRYPTOKEY \
> --plaintext-file - \
> --ciphertext-file - \
> | base64

put the scrambled (encrypted and base64 encoded) value into your environment variable (in yaml file).

Some pythonish code to get you started on decrypting.

kms_client = kms_v1.KeyManagementServiceClient()
name = kms_client.crypto_key_path_path("project", "global", "THEKEYRING", "THECRYPTOKEY")

twitter_app_key = kms_client.decrypt(name, base64.b64decode(os.environ.get("TWITTER_APP_KEY"))).plaintext

Fairminded

Answer 12

@Jason F's answer based on using Google Datastore is close, but the code is a bit outdated based on the sample usage on the library docs. Here's the snippet that worked for me:

from google.cloud import datastore

client = datastore.Client('<your project id>')
key = client.key('<kind e.g settings>', '<entity name>') # note: entity name not property
# get by key for this entity
result = client.get(key)
print(result) # prints all the properties ( a dict). index a specific value like result['MY_SECRET_KEY'])

Partly inspired by this Medium post

Pottage

Answer 13

My solution is to replace the secrets in the app.yaml file via github action and github secrets.

app.yaml (App Engine)

env_variables:
  SECRET_ONE: $SECRET_ONE
  ANOTHER_SECRET: $ANOTHER_SECRET

workflow.yaml (Github)

steps:
  - uses: actions/checkout@v2
  - uses: 73h/[email protected]
    env:
      SECRET_ONE: ${{ secrets.SECRET_ONE }}
      ANOTHER_SECRET: ${{ secrets.ANOTHER_SECRET }}

Here you can find the Github action.
https://github.com/73h/gae-app-yaml-replace-env-variables

When developing locally, I write the secrets to an .env file.

Lindon

Answer 14

Extending Martin's answer

from google.appengine.ext import ndb

class Settings(ndb.Model):
    """
    Get sensitive data setting from DataStore.

    key:String -> value:String
    key:String -> Exception

    Thanks to: Martin Omander @ Stackoverflow
    https://mcmap.net/q/180006/-securely-storing-environment-variables-in-gae-with-app-yaml
    """
    name = ndb.StringProperty()
    value = ndb.StringProperty()

    @staticmethod
    def get(name):
        retval = Settings.query(Settings.name == name).get()
        if not retval:
            raise Exception(('Setting %s not found in the database. A placeholder ' +
                             'record has been created. Go to the Developers Console for your app ' +
                             'in App Engine, look up the Settings record with name=%s and enter ' +
                             'its value in that record\'s value field.') % (name, name))
        return retval.value

    @staticmethod
    def set(name, value):
        exists = Settings.query(Settings.name == name).get()
        if not exists:
            s = Settings(name=name, value=value)
            s.put()
        else:
            exists.value = value
            exists.put()

        return True

Baseball

Answer 15

There is a pypi package called gae_env that allows you to save appengine environment variables in Cloud Datastore. Under the hood, it also uses Memcache so its fast

Usage:

import gae_env

API_KEY = gae_env.get('API_KEY')

If there is a value for that key in the datastore, it will be returned. If there isn't, a placeholder record __NOT_SET__ will be created and a ValueNotSetError will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.

---

Similar to Martin's answer, here is how to update the value for the key in Datastore:

1. Go to Datastore Section in the developers console

2. Select your project at the top of the page if it's not already selected.

3. In the Kind dropdown box, select GaeEnvSettings.

4. Keys for which an exception was raised will have value __NOT_SET__.

---

Go to the package's GitHub page for more info on usage/configuration

Periodate