How to specify wildcard artifacts subdirectories in .gitlab-ci.yml?

Question

I'm using GitLab CI to build a C# solution and try to pass some build artifacts from one build stage to another.

The problem is, that the artifacts are not located in a single directory but in different subdirectories, which however all have the same names bin/ or obj/.

My .gitlab-ci.yml looks like the following:

...
stages:
  - build
  - test

build:
  stage: build
  script:
    CALL %MSBuild% ...
  artifacts:
    paths:
      - /**/bin/
      - /**/obj/
    expire_in: 6 hrs

test:
  stage: test
  dependencies:
    - build
  ...

I tried to capture the artifacts using different ways, e.g.

**/bin/
**/obj/

(invalid syntax), or

.*/bin/
.*/obj/

but that one did not find any artifacts, just as /**/bin/ and /**/obj/, giving me following errors:

Uploading artifacts...
WARNING: /**/bin/: no matching files
WARNING: /**/obj/: no matching files

How can I specify a subdirectory pattern to be scanned for artifacts? Or is this even possible at all?

Simply using

artifacts:
  untracked: true

is not an option, because of a huge untracked packages/ subdirectory, which causes artifacts upload to fail because of a too large archive:

Uploading artifacts...
untracked: found 4513 files                        
ERROR: Uploading artifacts to coordinator... too large archive  id=36 responseStatus=413 Request Entity Too Large token=...
FATAL: Too large

Subeditor

Answer 1

The gitlab-ci-multi-runner build runner is built using Go and currently uses filepath.Glob() to scan for any specified artifacts in file_archiver.go.

Go doesn't seem to support the double star glob expression as discussed in another question here at SO. So there seem to be no way to use a full-featured **/bin expression at the moment.

Because however all my projects are located at the same level below the solution root, it is still possible to use something like

artifacts:
  paths:
    - "*/bin"
    - "*/obj"

Note that the quotes (") seem to be required, as well as no trailing path separator at the end.

It should also be possible to explicitly add more levels by adding more globbing expressions (as described here):

paths:
  ...
  - "*/obj"
  - "*/*/bin"
  - "*/*/obj"
  ...

GitLab is tracking this issue here, and will possibly be fixed in a future version.

UPD: due to mentioned issue is closed and due to documentation, since GitLab runner 13.0 it supports doublestar.Glob.

So it is now possible to write something like this:

paths:
  - "**/bin"
  - "**/obj"

And it will include all nested directories with names bin and obj in artifacts

Subeditor

I had no luck with the double quotes around the path. It worked without them. Also the doc mentions only examples without quotes. – Cartel

I have found that the double quotes are necessary. – Retuse

Odd that quotes matter. But as far as I know, quoting is part of YAML syntax, so I doubt that the runner sees it with the quotes. I wonder why for one person it needed quotes and for another it needed no quotes. – Interlock

No trailing slash! Took me way too long to figure this out on my own. I also found that I only needed quotes when starting the path with characters like * or % (Windows batch file variable expansion). – Corruption

With gitlab-runner 11.11.0 on win64, globbing artifacts with ** works fine. (e.g. artifacts: paths: - BuildArtifacts\Cli\**) – Syllabub

You need to use quotes because * is a special character in YAML, an alias. Here is a list of all the special characters you must quote. "Although you can use double quotes, for these characters it is more convenient to use single quotes, which avoids having to escape any backslash \". Not sure why trailing / doesn't work with globs. Seems like gitlab doesn't allow it and has nothing to do with YAML. – Verbid

@Verbid tyvm - that was an annoying headache. – Impeachment

Looks like gitlab.com/gitlab-org/gitlab-runner/issues/2620 just got implemented? – Chronological