dynamically setting artifact path/folder structure on gitlab-ci.yml

Question

I have the following gitlab-ci.yml file that reads the package.json using the jq processor to dynamically set the variable name of the artifact folder, something along the lines of

image: node:latest

stages:
 - build

before_script:
    ## steps ignored for purpose of question
    - export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
    - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip

prod_build:
  stage: build
  script:
   - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
   - yarn run build
  artifacts:
    paths:
    - dist/$NAME_OF_ARTIFACT_FOLDER ## this does not work
    expire_in: 2 hrs

The issue here is - dist/$NAME_OF_ARTIFACT_FOLDER does not work, not sure if am missing something here.

EDIT

Upon hard coding the expected path such as the following, it works fine, which would mean that the folder name is valid and that the artifact is indeed identified appropriately, but does NOT work when coming from $NAME_OF_ARTIFACT_FOLDER

 artifacts:
    paths:
    - dist/myApp_1.0.0.zip ##hardcoding the expected works just fine
    expire_in: 2 hrs

Kampmeier

Why do you want it to work like that? – Leela

for the same reasons as anyone not wanting to hard code the folder name generated dynamically and or subjected to change. Not sure if you read the entire question. tl-dr- I capture the name and version of the project from package.json to generate the zipped artifact folder as part of yarn build and would want gitlab to identify the said artifact for deployment – Kampmeier

@Kampmeier were you able to make this work somehow? Or do I really have to hardcode my .zip in build stage?? omg – Buzzer

Unfortunately no, I did not. Atleast not at that time of posting the question (almost 3 yrs back). Hoping things have changed and gitlab has incorporated the feature in it. Good luck @trainoasis. – Kampmeier

Answer 1

While @Jakub Kania was not that wrong with it's summary

https://mcmap.net/q/553102/-dynamically-setting-artifact-path-folder-structure-on-gitlab-ci-yml That is no variables set in your script part of the job can be used.

and I have to redict my own comment beneath @Matt Bracewell's post

https://mcmap.net/q/553102/-dynamically-setting-artifact-path-folder-structure-on-gitlab-ci-yml I don't see this will help anything because it addresses artifacts:name but not artifacts:paths . paths is still not dynamically usable as you can see in https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html

~~the undocumented part says:__~~

you can use any variable in artifacts:path that is available at job level

i.e.

  prod_build:
+   variables:
+      NAME_OF_ARTIFACT: "myApp_*.zip"
    stage: build
    script:
     - echo $NAME_OF_ARTIFACT
     - yarn run build
    artifacts:
      paths:
      - dist/$NAME_OF_ARTIFACT
      expire_in: 2 hrs

That's indeed consequent to what is asked in Artifacts Filename Cannot Be Set with Dynamic Variables even they don't ask for paths but names

generate for myself:
  script:
    - mkdir build
    - |
      cat > build/artifact1$$.txt <<EOF
      Hello World! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE_NAME=artifact1$$
    - echo "MY_VAR_FILE_NAME=${MY_VAR_FILE_NAME}" >> generate.env
    - |
      cat > build/artifact2$$.txt <<EOF
      The end is near! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE2_NAME=artifact2$$
    - echo "MY_VAR_FILE2_NAME=${MY_VAR_FILE2_NAME}" >> generate.env
    - |
      cat > build/ci${CI_PIPELINE_IID}.txt <<EOF
      Happy to hear you! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
  artifacts:
    reports:
      # using the https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv 
      # does NOT do the trick to fill a job variable before artifact:paths is evaluated!
      dotenv: generate.env
    paths: 
      - build/ci${CI_PIPELINE_IID}.txt # WORKS!
      - build/$MY_VAR_FILE_NAME.txt    # doen't work
      - build/$MY_VAR_FILE2_NAME.txt   # doen't work

⚠️ unknown variables are expanded to empty and build/$MY_VAR_FILE_NAME.txt becomes build/ i.e. all files in build folder are artifacts - mentioned in pipeline out WARNING: build/.txt: no matching files. Ensure that the artifact path is relative to the working directory resp. unexpected build/: found 4 matching files and directories

As a workaround - but yes, it's ugly -, you can do the trick either by using wildcards (but I assume you have good reasons to do not)

prod_build
  artifacts:
    paths:
      - dist/*.zip

OR splitting into two jobs

generate before save:
  script:
    - mkdir build
    - |
      cat > build/artifact1$$.txt <<EOF
      Hello World! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE_NAME=artifact1$$
    - echo "MY_VAR_FILE_NAME=${MY_VAR_FILE_NAME}" >> generate.env
    - |
      cat > build/artifact2$$.txt <<EOF
      The end is near! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE2_NAME=artifact2$$
    - echo "MY_VAR_FILE2_NAME=${MY_VAR_FILE2_NAME}" >> generate.env
  artifacts:
    reports:
      # using the https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv trick to fill a job variable for downstream of successor job
      dotenv: generate.env
    paths: 
      - build/*.txt

save artifacts:
  ## this is your expected input:
  # variables:
  #   MY_VAR_FILE_NAME: #filled by dotenv
  #   MY_VAR_FILE2_NAME: #filled by dotenv
  needs:
    - generate before save
  script:
    - find build/
    - echo "expecting … "
    - echo "- $MY_VAR_FILE_NAME"
    - echo "- $MY_VAR_FILE2_NAME"
  artifacts:
    paths: 
      - build/$MY_VAR_FILE_NAME.txt    # WORKS
      - build/$MY_VAR_FILE2_NAME.txt   # WORKS

i.e. in your case:

image: node:latest

stages:
 - build

pre_prod_build:
  stage: build
  script:
    ## steps ignored for purpose of question
    - export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
    - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
    - echo "NAME_OF_ARTIFACT_FOLDER=${NAME_OF_ARTIFACT_FOLDER}" >> artifact_name.dotenv
  artifacts:
    reports:
      dotenv: artifact_name.dotenv

prod_build:
  stage: build
  ## this is your expected input:
  # variables:
  #   NAME_OF_ARTIFACT_FOLDER: # set by pre_prod_build dotenv
  needs:
    - job: pre_prod_build
  script:
   - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
   - yarn run build
  artifacts:
    paths:
    - dist/$NAME_OF_ARTIFACT_FOLDER ## NOW IT WORKS
    expire_in: 2 hrs

Appetizing

A great thanks @Appetizing ! Your workaround is perfect ! – Englut

For the split jobs You actually don't need to provide empty variables, the variables are populated by dotenv regardless. – Misha

correct @Misha , just added it as like "the interface definition of prod_build", just updated it above, thank you – Assegai

Answer 2

Well, that is not possible currently. Manual says as follows:

The artifacts:name variable can make use of any of the predefined variables.

That is no variables set in your script part of the job can be used.

Leela

My question is about the artifacts path and not the name. am confused – Kampmeier

@Kampmeier Oh, yeah you're right. Same thing goes, you can't use anything outside of predefined variables. – Leela

Answer 3

This is an open issue at GitLab

Artifacts Filename Cannot Be Set with Dynamic Variables

I had a project variable defining the path to a zip file in a script which I reused at artifacts:paths level. The linked issue would have been more obvious had the artifacts:paths instance completely failed to get assigned but in my case it inherited a different value from that a mere two lines above in my job!

Tunicle

I don't see this will help anything because it addresses artifacts:name but not artifacts:paths . paths is still not dynamically usable as you can see in docs.gitlab.com/ee/ci/variables/… – Assegai

@Appetizing you're right! – Tunicle