gitlab not expanding variable used in parallel/matrix

Question

I am trying to pass the variable to parallel/matrix and do not see that getting expanded and the job failing. This is being set in the job from the environment variable. I am trying to echo the variable in script and see it shows the right value, but does not get substituted in parallel/matrix. Am I missing anything?

.common_deploy:
  script:
    - |
      echo "showing the regions from environment"
      echo $qa_regions
      echo "showing the regions from job variable"
      echo $REGIONS
  parallel:
    matrix:
      - REGION: "${REGIONS}"
      
DeployToQA:
  variables:
    ENVIRONMENT: qa
    REGIONS: $qa_regions
  extends:
    - .common_deploy
  stage: deploy
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "master"'
      allow_failure: true

Here the variable $qa_regions has the value of "us-west-2,us-east-1", I was expecting to see the 2 jobs for those regions , but i am seeing the job as DeployToQA: [${REGIONS}]

Decidua

Answer 1

Variable expansion for the parallel keyword is currently not supported. There is an open issue for this request.

You can take a look at the documentation where variables can be used.

Ing

Thanks for responding, can i achieve this in any other different way apart from writing one more job to generate the job config – Decidua

Unfortunately I don't know of any other way besides writing one more job – Ing

@Ing how it can be done using a different job, I am facing the same problem. – Purusha

@SrinivasBoga did you find the solution for it? – Purusha

Answer 2

To address the need for passing variables to the parallel:matrix effectively, I configured a workaround solution. My approach involved utilizing envsubst, a template file, and a trigger with the use of artifacts.

# .gitlab-ci.yml
stages:
  - trigger

variables:
  PROJECTS: '["project1", "project2", "project3"]'

process:template:
  stage: trigger
  before_script:
    - apt-get update
    - apt-get install gettext-base
  script:
    - envsubst '${PROJECTS}' < template.yml > template.gitlab-ci.yml
  artifacts:
    paths:
      - template.gitlab-ci.yml

trigger:template:
  stage: trigger
  trigger:
    include: 
      - artifact: template.gitlab-ci.yml
        job: process:template
    forward:
      pipeline_variables: true
  needs:
    - job: process:template

Should have the template.yml file present within your repository:

# template.yml
stages:
  - matrix

deploy:
  stage: matrix
  parallel:
    matrix:
      - PROJECT: ${PROJECTS}
  script:
    - echo "Deploying project = ${PROJECT}"

The final pipeline looks like this:

And in the output of each job from matrix we can see message from the echo:

• deploy: [project1] "Deploying project = project1"
• deploy: [project2] "Deploying project = project2"
• deploy: [project3] "Deploying project = project3"

Hope this helps! 🥷

Aleece

I keep getting The rules configuration prevented any jobs from being added to the pipeline for some reason, any ideas why? on the trigger::template – Untenable

Answer 3

As already answerded by @danielnelz, variable expansion is currently not supported.

However, for a large amount of parallel jobs you could generate your pipeline dynamically by using parent-child pipelines instead.

Impassable