Virtualenv python with AWS codebuild: why the deactivate command is not found?

Question

I use AWS Codebuild to upload python code into AWS Lambda from a GitHub repository. So I set up virtual python environments with virtualenv.

Here is my buildspec.yml:

version: 0.2
phases:
 install:
   commands:
     - echo "install step"
     - apt-get update
     - apt-get install zip -y
     - apt-get install python3-pip -y
     - pip install --upgrade pip
     - pip install --upgrade awscli
     # Define directories
     - export HOME_DIR=`pwd`
     - cd $HOME_DIR
     - export PREPROCESSING_DIR=$HOME_DIR/preprocessing
     - export COMPARE_DIR=$HOME_DIR/compareHilightGood
     - export NLTK_DATA=$HOME_DIR/nltk_data
 pre_build:
   commands:
     - echo "pre_build step"
     # Configure preprocessing virtual environement
     - cd $PREPROCESSING_DIR
     - virtualenv venv_preprocessing
     - . venv_preprocessing/bin/activate
     #  Install modules
     - pip install -U requests
     - pip install -U nltk
     #  NLTK download
     - python -m nltk.downloader -d $NLTK_DATA wordnet stopwords punkt
     - pip freeze > requirements.txt
     #  zip everything
     - mv $VIRTUAL_ENV/lib/python3.6/site-packages/* .
     - sudo zip -r9 preprocessing.zip .
     - source deactivate
     # Configure compare virtual environement
     - cd $COMPARE_DIR
     - virtualenv venv_compare
     - . venv_compare/bin/activate
     # install modules
     - pip install -U gensim
     - pip install -U pandas
     - pip freeze > requirements.txt
     # zip everything
     - mv $VIRTUAL_ENV/lib/python3.6/site-packages/* .
     - sudo zip -r9 compare.zip .
     - source deactivate
 build:
   commands:
     - echo 'build step'
     # preprocessing
     - cd $PREPROCESSING_DIR
     - aws s3 cp --recursive --acl public-read ./preprocessing.zip s3://lambda-preprocessing/
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --s3-bucket lambda-preprocessing --s3-key preprocessing.zip
     - aws lambda update-function-configuration --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --environment 'Variables={NLTK_DATA=/var/task/nltk_data}'
     # compare
     - cd $COMPARE_DIR
     - aws s3 cp --recursive --acl public-read ./compare.zip s3://lambda-comparehilightgood/
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --s3-bucket lambda-comparehilightgood --s3-key compare.zip
 post_build:
   commands:
     - echo "post_build step"

In the pre_build step, I'm switching between two virtualenv. So, I use deactivate (or source deactivate). But In both cases I get this error:

[Container] 2019/03/17 09:07:54 Running command source deactivate
/codebuild/output/tmp/script.sh: 4: /codebuild/output/tmp/script.sh: source: not found
 [Container] 2019/03/17 09:07:54 Command did not exit successfully source deactivate exit status 127
[Container] 2019/03/17 09:07:54 Phase complete: PRE_BUILD Success: false
[Container] 2019/03/17 09:07:54 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: source deactivate. Reason: exit status 127

Anyway, it seems the commands source and deactivate are not found. Do you know why ?

Horatia

Do those commands share a shell? I don't think you virtualenv activation is persisting from one command to the next. – Semitone

I don't know, but why wouldn't they be? This method works when I use it for one virtualenv. – Horatia

Answer 1

deactivate is not a script that can be sourced, it's a shell function created in the current shell environment by . venv/bin/activate. So try just deactivate without source.

As for the error about source itself please note that the command . venv/bin/activate works hence the shell understands command . but not source.

Conservative

Yes I tried just deactivate and I have the same error. [Container] 2019/03/17 08:09:06 Running command deactivate /codebuild/output/tmp/script.sh: 4: /codebuild/output/tmp/script.sh: deactivate: not found – Horatia

This means that every line in .yml is executed by a fresh shell. You have to run all commands between . venv/bin/activate and deactivate in one line that is one shell. – Conservative

Answer 2

As @phd mentioned, deactivate is a shell function created in the current shell when the virtual environment is activated. But as it not being available across commands, you can use the following workaround.

source venv_preprocessing/bin/activate && deactivate 

Costotomy