Defining one-to-one dependencies between multiple parallel matrix jobs in gitlab

Question

In a gitlab pipeline, I can specify a job like:

build:
  parallel:
    matrix:
      - architecture: [x86_64, arm]
        operating_system: [linux, macos]

And that will create 4 separate jobs that can all run in parallel:

• build [x86_64, linux]
• build [x86_64, macos]
• build [arm, linux]
• build [arm, macos]

I can also specify another job like:

test:
  needs:
    - job: build
      parallel:
        matrix:
          - architecture: [x86_64, arm]
            operating_system: [linux, macos]

And that would create one more job named test that would wait for all 4 build jobs to run before starting.

But what If I wanted 4 separate test jobs, one for each combo of architecture and operating_system, just like the build jobs? I would want something like:

test:
  parallel:
    matrix:
      - architecture: [x86_64, arm]
        operating_system: [linux, macos]
  needs:
    - job: build
      parallel:
        matrix:
          - architecture: [x86_64, arm]
            operating_system: [linux, macos]

But, I'm assuming that would create 4 separate test jobs that would each depend on all of the 4 separate build jobs. What I'd want is for each of the test jobs to only depend on a single build job with the same variables. In the past, I've tried to use variables in the needs array, but this was before needs:parallel:matrix was introduced, and it didn't work anyway. Can you use variables in needs:parallel:matrix to achieve this? Something like:

test:
  parallel:
    matrix:
      - architecture: [x86_64, arm]
        operating_system: [linux, macos]
  needs:
    - job: build
      parallel:
        matrix:
          - architecture: [$architecture]
            operating_system: [$operating_system]

Edit: This is not even close to the suggested duplicate. That one is about letting individual jobs only be dependent on other single jobs rather than entire stages being dependent on entire previous stages. That issue can be solved by simply using the needs keyword which was introduced in gitlab 12.2, as noted in the responses to that question. My question is about parallel:matrix (introduced in gitlab 15.9) and needs:parallel:matrix (introduced in gitlab 16.3). These are more advanced directives.

Ingold

It appears that variables are still not allowed in needs:. So the proposed solution won't work. – Ingold

@malat, nope. The info you linked to deals with a much earlier version of gitlab where parallel:matrix and needs:parallel:matrix didn't even exist. – Ingold

Answer 1

This doesn't (yet) appear possible with parallel:matrix. However, you can get a similar behavior using a template with spec:inputs

Helpful guides:

• https://docs.gitlab.com/ee/ci/yaml/inputs.html
• https://docs.gitlab.com/ee/ci/yaml/#specinputs

Note that this is a special keyword that must be put in the header of the CI/CD template, above the --- that separates yaml directives from the actual yaml document.

So, for the example in the question, you'd create one template file with 2 inputs: architecture and operating_system, and the two jobs:

# build-test.yml
spec:
  inputs:
    name:
      type: architecture
      description: "The architecture to build for."
      options:
        - x86_64
        - arm
      type: operating_system
      description: "The operating_system to build for."
      options:
        - linux
        - macos
---

build-$[[ inputs.architecture ]]-$[[ inputs.operating_system ]]:
  script:
    - echo "Building for $[[ inputs.architecture ]]$[[ inputs.operating_system ]]"

test-$[[ inputs.architecture ]]-$[[ inputs.operating_system ]]:
  needs:
    - build-$[[ inputs.architecture ]]-$[[ inputs.operating_system ]]
  script:
    - echo "Testing for $[[ inputs.architecture ]]$[[ inputs.operating_system ]]"

Then you'd include that template from your main CI/CD config with all the combinations of arch and os:

# .gitlab-ci.yml

include:
  - local: 'build-test.yml'
    inputs:
      architecture: x86_64
      operating_system: linux
  - local: 'build-test.yml'
    inputs:
      architecture: arm
      operating_system: linux
  - local: 'build-test.yml'
    inputs:
      architecture: x86_64
      operating_system: macos
  - local: 'build-test.yml'
    inputs:
      architecture: arm
      operating_system: macos

That'll end up creating a test-x86_64-linux job that only depends on a build-x86_64-linux job, and so on for the other combos.

It's a little more annoying than something built-in to parallel:matrix and needs:parallel:matrix since you have to list out each combination.

Ingold