GitLab Runner fails to upload artifacts with "invalid argument" error

Question

I'm completely new to trying to implement GitLab's CI/CD pipelines, but it's been going quite well. In fact, for my ASP.NET project, if I specify a Publish Profile in the msbuild command that uses Web Deploy, it actually deploys the code successfully to the web server.

However, I'm now wanting to have the "build" job create artifacts which are uploaded to GitLab that I can then subsequently deploy. We're using a self-hosted instance of GitLab, for which I'm not an admin, but I can speak to the admin if I know what I'm asking for!

So I've configured my gitlab-ci.yml file like this:

variables:
  NUGET_PATH: 'C:\Program Files\Nuget\Nuget.exe'
  NUGET_SOURCES: 'https://api.nuget.org/v3/index.json'
  MSBUILD_PATH: 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\msbuild.exe'

stages:
- build

build-job:
  variables:
    CI_DEBUG_TRACE: "true"
  stage: build
  script:
    - '& "$env:NUGET_PATH" restore ApplicationTemplate.sln  -Source "$env:NUGET_SOURCES"'
    - '& "$env:MSBUILD_PATH" ApplicationTemplate\ApplicationTemplate.csproj /p:DeployOnBuild=true /p:Configuration=Release /p:PublishProfile=FolderPublish.pubxml'
  artifacts:
    paths: 
      - '.\ApplicationTemplate\bin\Release\Publish\'

The output shows that this builds the code just fine, and it also seems to successfully find the artifacts for upload. However, when it uploads the artifacts, even though the request gets a 200 OK response, the process fails. Here is the log output:

So, it finds the artifacts, it attempts to upload them and even gets a 200 OK response (in contrast to the handful of similar reports of this error I've been able to find online), but it still fails due to an invalid argument.

I've already enabled verbose debugging, as you can see from the output, but I'm none the wiser. Looking at the GitLab Runner entries in the Windows Event Log on the box where the runner is hosted doesn't shed any light on things either. The total size of the artifacts is 61.1MB, so I don't think my issue is related to that.

Can anyone see from this output what's invalid? Can I identify which argument is invalid and/or why it's invalid?

Edit: Things I've tried

• Specifying a value for artifacts:expire_in.
• Setting artifacts:public to FALSE, since I'm using a self-hosted GitLab environment and the default value for this setting (TRUE) is not valid in such an environment.
• Trying every format I can think of for the value of the artifacts:paths setting (this seems to be incredibly robust - regardless of the format I use, the Runner seems to have no problem parsing it and finding the files to upload).
• Taking a cue from this question I created a new project with a very simple build job to upload a single file:

stages:
  - build

build-job:
  variables:
    CI_DEBUG_TRACE: "true"
  stage: build
  script:
    - echo "Test" > test.txt
  artifacts:
    paths:
      - test.txt

About 50% of the time this job hangs on the uploading of the artifacts and I have to cancel it. The other half of the time it fails in exactly the same way as the my previous project:

Ipa

Can you try the no proxy solution from that same quesstion: https://mcmap.net/q/1469220/-gitlab-shell-runner-cannot-upload-artifacts-larger-than-63kb-quot-501-not-implemented-quot – Demonology

I tried this too, a few days ago. I should have added it to the "Things I've tried" section in the edited question, but I must have forgotten. Thanks for the suggestion! – Ipa

This issue is often caused by proxies or firewalls between the two (gitlab and repository) – Demonology

No doubt - it was one of my first thoughts. However, with an apparent success code in the response, and nothing to see in the logs to confirm this theory, the security team were really reluctant to make changes to the firewall config. In fact, the only reason that they did was because another member of the team was able to prove that the WAF was blocking something they were trying to upload to a repo. – Ipa

Answer 1

After countless hours working on this, it seems that ultimately the issue was that our internal Web Application Firewall was blocking some part of the transfer of artefacts to the server, or the response back from it. With the WAF reconfigured not to block traffic from the machine running the GitLab Runner, the artefacts are successfully uploaded and the job succeeds.

This would have been significantly easier to diagnose if the logging from GitLab was better. As per my comment on this issue, it should be possible to see the content of the response from the GitLab server after uploading artefacts, even when the response code is 200.

What's strange - and made diagnosing the issue even harder - is that when I worked through the issue with the admin of our GitLab instance, digging through logs and running it in debug mode, the artefact upload process was uploading something successfully. We could see, for example, the GitLab Runner's log had been uploaded to the server. Clearly the WAF's blocking was selective and didn't block everything in both directions.

Ipa