Gitlab and placing hidden key scripts into include files

Question

I have what amounts to several different scripts that I want to run in a variety of stages across multiple projects. Currently they are of the form:

.hidden_key:  &hidden_key |
  do_something
  do_something_else

real_job:
  script:
    - *hidden_key

Effectively .hidden_key is a function I use throughout the .gitlab-ci.yml file and across several projects this way. But I can't seem to get the include to work when I move .hidden_key into a file and include it like this:

include:
   - remote: https://gitlab/project/master/raw/hidden_key.yml

real_job:
  script:
    - *hidden_key

When I do that, gitlab complains about:

Error: Unknown alias: hidden_key

Am I doing something incorrectly, or is this an actual limitation of includes (and therefore not supported) ?

What alternatives to this are there to clean up my .gitlab-ci.yml file ?

Raisin

Answer 1

They added !reference tags

Essentially, this is how you can accomplish the reference to the alias

    include:
       - remote: https://gitlab/project/master/raw/hidden_key.yml
    
    real_job:
      script:
        - !reference [.hidden_key]

Bonneau

Answer 2

This is almost certainly a limitation of the includes (which would have been much better implemented explicitly with a tag).

Standard YAML loading consists of a parse, a compose and a construction phase. The aliases get resolved during the compose phase, and anchors e.g. do not carry over to the next document in multi-document YAML file.

Althoug the include key might be interpreted as early the construction phase (but that might also happen after completing that phase, interpreting the resulting data structure), then the alias in the including YAML will already need to have been resolved during its compose phase.

You will have more luck using some templating system jinja2, for which you can of course populate the variable to be substituted from your YAML file with common stages as well.

Unicycle

are you suggesting I can use jinja2 templating within the .gitlab-ci.yml file? Or that I should create that file using jinja2 templating via a script somehow? If the latter, can gitlab do this automagically via commit triggers or something? If so, pointers to docs explaining this would be most welcome! Thanks for your reply, btw, much appreciated! – Raisin

I have no idea whether that can be done within gitlab. I have done something similar to docker-compose (providing unavailable functionality) by implementing a new command which generates the YAML file docker-compose expects and then calls docker-compose. And to support a YAML based documentation file format on readthedocs.org by running a pre-processor on the docs, on their backend. The latter is possible because they allow python code to run in a sandbox, gitlab would need to provide something similar, but I don't know if it does. – Unicycle

Answer 3

Have you tried using extends? Documentation

Example:

include:
   - remote: https://gitlab/project/master/raw/hidden_key.yml

real_job:
  extends: .hidden_key

Slagle