How do I add a .env file in gitlab ci during deployment stage?

Question

So I have a react/typescript app in my repo that I'm working on and in my repo I have a .env file that I'm ignoring so that my secrets don't get exposed and a .env-example file of important environment variables to configure. My problem is, since I'm not pushing the .env file to my repo, when I deploy my app through the google app engine(this is done in the deployment stage in my gitlab-ci.yml file), these environment variables will not be present in production and I need them for my app to work as I do something like this in my webpack.config.js file.

const dotenv = require('dotenv').config({ path: __dirname + '/.env' });

and then

new webpack.DefinePlugin({
  'process.env': dotenv.parsed
})

Here is my .gitlab-ci file for reference in case anyone here wants to see.

cache:
  paths:
    - node_modules/

stages:
  - build
  - test
  - deploy

Build_Site:
  image: node:8-alpine
  stage: build
  script:
    - npm install --progress=false
    - npm run-script build
  artifacts:
    expire_in: 1 week
    paths:
      - build

Run_Tests:
  image: node:8-alpine
  stage: test
  script:
    - npm install --progress=false
    - npm run-script test

Deploy_Production:
  image: google/cloud-sdk:latest
  stage: deploy
  environment: Production
  only:
    - master
  script:
    - echo $DEPLOY_KEY_FILE_PRODUCTION > /tmp/$CI_PIPELINE_ID.json
    - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
    - gcloud config set project $PROJECT_ID_PRODUCTION
    - gcloud info
    - gcloud --quiet app deploy
  after_script:
    - rm /tmp/$CI_PIPELINE_ID.json

Also, feel free to critique my gitlab-ci.yml file so I can make it better.

Tedder

Have you found a solution to this? I'm also looking for a way to do this. – Busra

@Busra One solution was to create a bash script that creates a temporary .env from the environment variables setup through gitlab duing the build phase(put this in the before scipt), builds the project through npm run build and then deletes the .env file so that it doesn't remain as an artifact(put this in the after script). – Tedder

@LuisAverhoff do you have an example of this solution to share? – Karlow

Answer 1

I don't know if you still need this, but this is how I achieved, what you wanted to.

1. Create your environment variables in your gitlab repo config

2. Create setup_env.sh:

#!/bin/bash

echo API_URL=$API_URL >> .env
echo NODE_ENV=$NODE_ENV >> .env

3. Modify your .gitlab-ci.yml. Upsert below to your before_script: section

  - chmod +x ./setup_env.sh
  - ./setup_env.sh

4. In webpack.config.js make use of https://www.npmjs.com/package/dotenv

require('dotenv').config();

This passes your .env variables available in webpack.config.js file.

Add this to your plugins array (add those variables you need):

    new webpack.DefinePlugin({
      'process.env.API_URL': JSON.stringify(process.env.API_URL),
      'process.env.NODE_ENV': JSON.stringify(process.env.NODE_ENV),
      ...
    })

Now your deployment should use your environment variables specified in you gitlab settings.

Alaniz

committing .env files to the repo is a bad practice – Headquarters

@FrancisManojFernnado yup. That's why there is such procedure – Alaniz

@MichałCzarnota, would you consider copying the all .env content inside a giltab CI environment variable as a bad practice? As advised here we need to set as much variables as we define entries in the .dotenv file (in the worst case at least). Then we reconstruct the .env file in the prescript instruction. Seems overkill to me. let put the entire .env content in the Gitlab CI environement variable. – Aardwolf

@Aardwolf I think you are right. In the next project I omitted this overhead. But, unfortunately, I didn't find a way to avoid setting env variables in gitlab repo config. – Alaniz

thanks for this. This was my solution for dotenv in Flutter using CodeMagic CI – Slavey

For me, the line that adds the token needed a slight modification with quotes and curly brackets: echo 'AUTH_TOKEN=${AUTH_TOKEN}' >> .env – Pneumoencephalogram

Answer 2

Gitlab now allows you to have file type variables, here's how I'm using.

FYI: In this case a Vuejs app.

Then on my .gitlab-ci file:

...
before_script:
    ...
    - cp ${PRODUCTION_ENV_FILE} .env.production
script:
    ...
    - npm ci
    - npm run build
...

Misnomer

good, but such variable cannot be masked in job logs – Thomson

@Thomson Not sure, you can however not print it d: – Misnomer

The variable isn't being set for me -- when I try to echo or cat the file, it's empty cat ${PRODUCTION_ENV_FILE} – Dorser

Answer 3

Committing .env files isn't a bad practice if you don't push any sensitive information in it like:

# App Configuration
PORT=3000
NODE_ENV=production
APP_ENV=APP_ENV
TZ=utc
COMMIT_SHA=COMMIT_SHA

# Rabbitmq configuration
RABBITMQ_HOST=RABBITMQ_HOST
RABBITMQ_USER=RABBITMQ_USER
RABBITMQ_PASS=RABBITMQ_PASS
RABBITMQ_PORT=RABBITMQ_PORT
...

(assuming variables like RABBITMQ_HOST are safely set in your GitLab CI environment variables)

So in the .gitlab-ci.yml job, you can just duplicate the needed .env file and it will automatically be interpreted:

...
test:e2e:
  image: node:lts
  services:
    - postgres:13-alpine
    - rabbitmq:3.9-management
  stage: test
  before_script:
    - cp .env.test .env      # <--- env vars will be applied
    - npm install pm2 -g
    - npm run install:ci
    - npm run db:drop
    - npm run db:migrate
    - npm run db:fixture
    - pm2 start npm -- start --name "myapp"
  script:
    - npm run test:e2e
  after_script:
    - pm2 delete "myapp"
  variables:                 # <---- override any env var you want here
    POSTGRES_DB: test
    POSTGRES_USER: test
    POSTGRES_PASSWORD: testPwd!
    POSTGRES_HOST_AUTH_METHOD: trust
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull
  except:
    - /^release.*$/i
    - tags
...

No tricky stuff needed.

Scrimpy

Just wonder, if it will even work. Shouldn't the cp's arguments are inverted? Otherwise, gitlab runner will throw something like cp: cannot stat '.env.test': No such file or directory – Cates

@Cates take a look at this : explainshell.com/explain?cmd=cp+.env.test+.env . (quick answer: since .env isn't a directory, it copies the file perfectly. So yes it works! – Scrimpy

Answer 4

Now Gitlab CI/CD can use files in pipelines. You should choose variables with type "file" (Settings > CI/CD > Variables > add variable > set type: file) and fill the "value" field by content of your .env file (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-types)

Valuation

Answer 5

Updated and simple answer:

Supposing you have your .env file like this:

 UPLOADER_URL=http://localhost:3333/api
 FIREBASE_API_KEY=123456abcd

Your normal .gitlab-ci.yml is like this:

image: node:14.18.0-alpine

before_script:
  - npm i -g firebase-tools

desploy-hosting:
  stage: deploy
  script:
    - yarn
    - yarn build
    - firebase deploy --only hosting --token $FIREBASE_TOKEN
  only:
    - master

You just have to "create" your .env in your container and echo/write each variable in it.

Your final .yml will be like this:

image: node:14.18.0-alpine

before_script:
  - npm i -g firebase-tools
  - echo UPLOADER_URL=$UPLOADER_URL >> .env         # Add here!
  - echo FIREBASE_API_KEY=$FIREBASE_API_KEY >> .env # Add here!

desploy-hosting:
  stage: deploy
  script:
    - yarn
    - yarn build
    - firebase deploy --only hosting --token $FIREBASE_TOKEN
  only:
    - master

Robedechambre

Answer 6

Another workaround in a case where you connect to the host through ssh, you will not be able to use files outside the scope of your pipeline. eg.:

run_deploy:
  tags:
    - python
  stage: deploy
  before_script:
    - chmod 400 $SSH_KEY
  script:
    - ssh -o StrictHostKeyChecking=no -i $SSH_KEY ubuntu@$SSH_HOST "
      sudo docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD registry.gitlab.com &&
      sudo docker run -d --name $CONSUMER_NAME --env-file=$ENV_FILE --network YOURS --restart=always $IMAGE_NAME:latest 
      "

In order to make it work, you can transfer a file using SCP to the host. eg.:

run_deploy:
  tags:
    - solucoes
    - python
  stage: deploy
  before_script:
    - chmod 400 $SSH_KEY_SOLACE
    - scp -o StrictHostKeyChecking=no -i $SSH_KEY_SOLACE $ENV_FILE ubuntu@$SSH_HOST_SOLACE:$ENV_PATH
  script:
    - ssh -o StrictHostKeyChecking=no -i $SSH_KEY_SOLACE ubuntu@$SSH_HOST_SOLACE "
      sudo docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD registry.gitlab.com &&
      sudo docker run -d --name $CONSUMER_NAME --env-file=$ENV_PATH --network solace --restart=always $IMAGE_NAME:latest
      "

But as @Miguel answered, this is only possible because Gitlab now can store variables as files. See: https://docs.gitlab.com/ee/ci/variables/#use-file-type-cicd-variables

I hope that this would be helpful. :)

Withindoors