GitLab: Job artifacts in multi project pipelines

Question

I have been trying to learn multi project pipelines for a while now, and apart from GitLab documentation, I have not found any study material. If I could see an example, it would really help. I have been using the following ci config for a multi project pipeline in project A, but it's not working:

trigger_job:
  stage: trigger_release  
  trigger:
    project: https://<gitlab-site>/api/v4/projects/<project-B-id>/trigger/pipeline
    branch: master
    strategy: depend

This leaves the pipeline in project A in a pending state forever. I used curl in the following way to finally get the config working:

trigger_job:
  stage: trigger_release 
  script:
    - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://<gitlab-site>/api/v4/projects/<project-B-id>/trigger/pipeline

However, what I really need is to collect and use the artifacts of project B pipeline in project A pipeline after the triggered job finishes. How do I do that?

Bangle

Did you see this needs keyword usage ? – Rondarondeau

How would that go? I have used the needs keyword myself in order to designate if a job requires artifacts from a previous job, or if success in a previous job is required for a consecutive job. But how does it give me access to job artifacts from a triggered pipeline to the triggering pipeline? – Bangle

@CompileCommit did you ever figure out your question, "However, what I really need is to collect and use the artifacts of project B pipeline in project A pipeline after the triggered job finishes. How do I do that?" - this is something I am currently trying to determine as well. – Herewith

@Herewith - not really. Eventually I moved on to using a different approach. Instead of using multi project pipeline, I created a wrapper Java project, added all the individual projects as submodule to the wrapper project and added them as child modules in the pom.xml of the wrapper project, and ran build there in a way that all projects are built in the same workspace, thereby negating the requirement in the first place. – Bangle

Answer 1

Since Gitlab 11.8. you don't need to use API to trigger a pipeline, see official documentation

Example

Let's have group mygroup with 2 repos: myrepository1 and myrepository2.

Config in repository1

stages:
  - prepare-manifests
  - deploy

create-artifacts:
  stage: prepare-artifacts
  variables:
    COMMIT_FILE: build/commit_id.txt
  artifacts:
    name: some-artifacts
    when: always
    expire_in: 14 days
    paths:
      - "build/kubernetes/*"
      - "build/commit_id.txt"
  script:
    - echo "Creating artifacts"
    - ....

trigger-job:
  trigger:
    project: mygroup/myrepository2
    branch: master
  variables:
    VARIABLE_TO_PASS: $CI_COMMIT_REF_NAME

Config in repository2

job-waiting-for-trigger:
  stage: deploy
  variables:
    REPOSITORY: user-profile/repository1
  script:
    - echo "Now I can use build/commit_id.txt"
  needs:
    - project: ${REPOSITORY}
      job: collect-all-manifests
      ref: $PARENT_REF_NAME
      artifacts: true
  only:
    - pipelines

Minatory

-1, @Minatory only because I don't see where this answers the bottomline question which was "However, what I really need is to collect and use the artifacts of project B pipeline in project A pipeline after the triggered job finishes. How do I do that?" ... this answer makes no mention of artifacts as it currently is stated. Since this answer was accepted maybe there is something I am missing though... – Herewith

Hi @mrek, I appreciate you taking the time to engage. I still don't see where you address the question which is that the pipeline in repo1 requires access to the artifacts from the triggered job in repo2 (you seem to address the inverse) - you can see in the OPs comment on the question that his problem was never actually resolved (not sure why your answer was accepted). You are incorrect regarding my use of downvote - I downvoted due to lack of answer. And also it is in fact reversible - and I'll do so if you answer the question :-) – Herewith

Ok, thanks @pooley1994, now I understand what you meant. Yes, you are right, my answer does the reverse. Well, I think you should be able to trigger pipeline in another repo (repo1 -> repo2), and then create another trigger back to its previous repository (repo2 -> repo1). – Minatory

thanks for understanding @mrek. I don't think your idea of "triggering back" would quite work, since the initial repo is already running and so does not require being triggered in that sense. I have attempted to provide my own answer below in which I explain my understanding that sadly the answer to the OP's original question is that there is no simple way to do it. – Herewith

Answer 2

Given your bottomline question:

However, what I really need is to collect and use the artifacts of project B pipeline in project A pipeline after the triggered job finishes. How do I do that?

The answer is that it can't be done in any simple "native pipeline config" way like you might hope. The closest I have come to confirming this from official Gitlab sources is first this article, "Breaking down CI/CD complexity with parent-child and multi-project pipelines" which states:

You can check this issue for planned future developments on parent-child and multi-project pipelines.

Going to said issue reveals the following:

the upstream pipeline can't directly expose data (artifacts) created by downstream pipelines

And also it states that "Multi-project expected default behavior" is "Artifacts should not be accessible." and that "Current default behavior for both [parent/child and multi-project] pipelines" is that "Artifacts from downstream pipelines are not directly visible from the top-level pipeline." which seems to settle it.

So, the best you can do it seems, is to use strategy: depend like so:

trigger_job:
  trigger:
    include: path/to/child-pipeline.yml
    strategy: depend

In your upstream pipeline, and then once that trigger job was resolved (which will track with the resolution of the triggered pipeline due to strategy of depend) to utilize the Gitlab Job Artifacts API to identify and fetch the needed artifacts from that downstream pipeline.

Herewith