AWS CodeBuild buildspec bash syntax error: bad substitution with if statement

Question

Background:

I'm using an AWS CodeBuild buildspec.yml to iterate through directories from a GitHub repo. Before looping through the directory path $TF_ROOT_DIR, I'm using a bash if statement to check if the GitHub branch name $BRANCH_NAME is within an env variable $LIVE_BRANCHES. As you can see in the error screenshot below, the bash if statement outputs the error: syntax error: bad substitution. When I reproduce the if statement within a local bash script, the if statement works as it's supposed to.

Here's the env variables defined in the CodeBuild project:

Here's a relevant snippet from the buildspec.yml:

version: 0.2
env:
  shell: bash
phases:
  build:
    commands:
      - |
        if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
          # Iterate only through BRANCH_NAME directory
          TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
        else
          # Iterate through both dev and prod directories
          TF_ROOT_DIR=${TF_ROOT_DIR}/*/
        fi
      - echo $TF_ROOT_DIR

Here's the build log that shows the syntax error:

Here's the AWS CodeBuild project JSON to reproduce the CodeBuild project:

{
    "projects": [
        {
            "name": "terraform_validate_plan",
            "arn": "arn:aws:codebuild:us-west-2:xxxxx:project/terraform_validate_plan",
            "description": "Perform terraform plan and terraform validator",
            "source": {
                "type": "GITHUB",
                "location": "https://github.com/marshall7m/sparkify_end_to_end.git",
                "gitCloneDepth": 1,
                "gitSubmodulesConfig": {
                    "fetchSubmodules": false
                },
                "buildspec": "deployment/CI/dev/cfg/buildspec_terraform_validate_plan.yml",
                "reportBuildStatus": false,
                "insecureSsl": false
            },
            "secondarySources": [],
            "secondarySourceVersions": [],
            "artifacts": {
                "type": "NO_ARTIFACTS",
                "overrideArtifactName": false
            },
            "cache": {
                "type": "NO_CACHE"
            },
            "environment": {
                "type": "LINUX_CONTAINER",
                "image": "hashicorp/terraform:0.12.28",
                "computeType": "BUILD_GENERAL1_SMALL",
                "environmentVariables": [
                    {
                        "name": "TF_ROOT_DIR",
                        "value": "deployment",
                        "type": "PLAINTEXT"
                    },
                    {
                        "name": "LIVE_BRANCHES",
                        "value": "(dev, prod)",
                        "type": "PLAINTEXT"
                    }

Here's the associated buildspec file content: (buildspec_terraform_validate_plan.yml)

version: 0.2
env:
  shell: bash
  parameter-store:
      AWS_ACCESS_KEY_ID_PARAM: TF_AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY_PARAM: TF_AWS_SECRET_ACCESS_KEY_ID
phases:
  install:
    commands:
      # install/incorporate terraform validator? 
  pre_build:
    commands:
      # CodeBuild environment variables
      #   BRANCH_NAME -- GitHub branch that triggered the CodeBuild project
      #   TF_ROOT_DIR -- Directory within branch ($BRANCH_NAME) that will be iterated through for terraform planning and testing
      #   LIVE_BRANCHES -- Branches that represent a live cloud environment
      - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
      - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
      - bash -version || echo "${BASH_VERSION}" || bash --version
      - |
        if [[ -z "${BRANCH_NAME}" ]]; then
          # extract branch from github webhook
          BRANCH_NAME=$(echo $CODEBUILD_WEBHOOK_HEAD_REF | cut -d'/' -f 3)
        fi
      - "echo Triggered Branch: $BRANCH_NAME"
      - |
        if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
          # Iterate only through BRANCH_NAME directory
          TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
        else
          # Iterate through both dev and prod directories
          TF_ROOT_DIR=${TF_ROOT_DIR}/*/
        fi
      - "echo Terraform root directory: $TF_ROOT_DIR"
  build:
    commands:
      - |
        for dir in $TF_ROOT_DIR; do
          #get list of non-hidden directories within $dir/
          service_dir_list=$(find "${dir}" -type d | grep -v '/\.')
          for sub_dir in $service_dir_list; do
            #if $sub_dir contains .tf or .tfvars files
            if (ls ${sub_dir}/*.tf) > /dev/null 2>&1 || (ls ${sub_dir}/*.tfvars) > /dev/null 2>&1; then
              cd $sub_dir 
              echo ""
              echo "*************** terraform init ******************"
              echo "******* At directory: ${sub_dir} ********"
              echo "*************************************************"
              terraform init
              echo ""
              echo "*************** terraform plan ******************"
              echo "******* At directory: ${sub_dir} ********"
              echo "*************************************************"
              terraform plan
              cd - > /dev/null 
            fi
          done 
        done

Given this is just a side project, all files that could be relevant to this problem are within a public repo here.

UPDATES

Tried adding #!/bin/bash shebang line but resulted in the CodeBuild error:

Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: #!/bin/bash

version: 0.2
env:
  shell: bash
phases:
  build:
    commands:
      - |
        #!/bin/bash
        if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
          # Iterate only through BRANCH_NAME directory
          TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
        else
          # Iterate through both dev and prod directories
          TF_ROOT_DIR=${TF_ROOT_DIR}/*/
        fi
      - echo $TF_ROOT_DIR

Solution

As mentioned by @Marcin, I used an AWS managed image within Codebuild (aws/codebuild/standard:4.0) and downloaded Terraform within the install phase.

phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip -q
      - unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip && mv terraform /usr/local/bin/

Impeccable

Answer 1

I tried to reproduce your issue, but it all works fine for me.

The only thing I've noticed is that you are using $BRANCH_NAME but its not defined anywhere. But even with missing $BRANCH_NAME the buildspec.yml you've posted runs fine.

Update using hashicorp/terraform:0.12.28 image

Carcass

I intentionally left out the code that determines $BRANCH_NAME to keep the code snippet relevant to the problem. But you can see the definition of $BRANCH_NAME from the output echo $BRANCH_NAME within the code build log screenshot. Thanks for the attempt, unfortunately I haven't found a solution to this weird issue. – Impeccable

@Impeccable Would it be possible to update the question with an example that would enable to reproduce the issue? From what I see and remember your current example works as expected and does not allow to reproduce the issue. Subsequently, it is difficult to assist. – Carcass

I added the codebuild project json and full buildspec file. If needed all files are within a public repo here. – Impeccable

@Impeccable I see you used custom image hashicorp/terraform:0.12.28, not AWS provided one. I tested with the custom image, and can confirm now that have the same issue. Though I have a bit different reason Reason: exec: "bash": executable file not found in $PATH. – Carcass

@Impeccable My further test indicate that there is no bash in the hashicorp/terraform:0.12.28. Wouldn't it be easier to do your build the opposite way, i.e. use AWS managed image for CodeBuild, and install terraform on it? – Carcass

Ah that's why, I mindlessly assumed that hashicorp/terraform:0.12.28 would have bash baked in. I guess the syntax error: bad substitution error threw me off and I thought the error was strictly related to bash. I took your advice regarding the use of an AWS managed imaged instead and reflected the change in my post. Thanks @Carcass for all the help! – Impeccable

Answer 2

This simple if else didn't worked for me on CodeBuild Ubuntu using aws/codebuild/standard:7.0:

  - |
    if [[ "$OLD_IMAGE" == "$NEW_IMAGE" ]]; then
      echo "Same Image being deployed"
    else
      echo "New Image being deployed"
    fi

After switch to AmazonLinux Image: aws/codebuild/amazonlinux2-x86_64-standard:5.0 all worked fine.

Dutybound

Answer 3

After many hours, I have found out that environment variable cannot be used in ""

echo "This is my branch: $branch" - won't work

echo "This is my branch:" $branch - will work.

Bragg