Gitlab CI/CD variable substitution in my appSettings.json

Question

I'm using Gitlab CI/CD to deploy my .netcore application to AWS Beanstalk. How can I update the appSettings.json values in my .netcore application when deploying to different environments using variables defined in my CI/CD pipeline settings?

Azure DevOps has a JSON variable substitution feature which I really liked.

GitHub Actions can also hook into this feature

How can I achieve this with Gitlab CI/CD?

I want to use this method of deployment because

1. I won't have to store sensitive production config values in my repository. Values will be updated by Masked variables setup in the CI/CD Pipeline.
2. I don't have to rebuild my artefacts every time I deploy to a new environment.

If this can't be done in Gitlabs, whats the recommended best practice?

thanks

Agglutinin

Answer 1

I do something similar here with gitlab and the solution was to build a shell script that replaces some string from variable values before starting the deploy job

something like this

script:
  - sed -i 's/STRING_TO_REPLACE/$GITLAB_VARIABLE/g' file.json

Just pay attention to escape json quotes correctly to make this possible

Personal

this is actually a good suggestion, but in the end it was a little messy especially dealing with whole db connections strings... so ultimately I switched to using Github – Agglutinin

This solution is too finicky with the syntax. I'm having trouble getting the variables expanded. – Tirzah

Answer 2

I use the same workaround. I filed an issue about this months ago: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/78486

and I recently talked about this at the gitlab forum:

https://forum.gitlab.com/t/use-variables-per-branch-in-gitlab/43685

unfortunately there does not seem to be a nice/clean solution at the moment.

so I use the workarround: 1: use environment scopes for variables so the same variable can have different values (for test/prod environments) 2: define the environments and branches in the jobs in gitlab-ci.yml 3: create a bunch of sed one liners to do a search/replace..

call it ugly, call it low level but I searched for a nice/clean/gitlab native solution an did not find it.

Mouseear

Answer 3

I suggest adding a second config file in your application like below:

    public static IConfiguration Configuration { get; } = new ConfigurationBuilder()
       .SetBasePath(Directory.GetCurrentDirectory())
       .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
       .AddJsonFile("build.json", optional: true, reloadOnChange: true)
       .Build();

Then in your build you create a build.json file which can add or update any value in appsettings.json at runtime. No matter what unless if you use a JSON cli tool, which you probably need to create a custom docker container/build machine to install, it will be messy. I think you are best off though using a new override file.

- echo "{\"CodeVersion\":\"$CI_COMMIT_SHORT_SHA\"}" > Source/MediaSpendApi/build.json

Tirzah

Answer 4

Use that script:

• sed -i "s/STRING_TO_REPLACE/${GITLAB_VARIABLE}/g" file.json

I wanna pay attencion that double quotes is nessarry, without them you will get just name of variable

Cheerleader

Answer 5

We use a approach for ASP.NET Core applications hosted on IIS servers in which we use GitLab pipelines to set variables as environment variables stored in the web.config file:

variables:
  PROJECT_FILE: 'sampleproject01.csproj'
  PUBLISH_PATH: 'd:\output'
  BUILD_TYPE: 'Debug'

stages:
  - deploy

deploy-dev:
  stage: deploy
  only:
    - dev
  script:
    - 'dotnet publish --output "$env:PUBLISH_PATH" -p:EnvironmentName=Development -c "$env:BUILD_TYPE" "$env:PROJECT_FILE"'
    - '$filePath = "$env:PUBLISH_PATH\web.config"'
    - '$xml = New-Object XML'
    - '$xml = [xml](Get-Content $filePath)'
    - '$base64decoded = [Text.Encoding]::Utf8.GetString([Convert]::FromBase64String("$env:Dev__ConnectionStrings__database"))'
    - '$envVar = $xml.CreateElement("environmentVariable")'
    - '$envVar.SetAttribute("name", "ConnectionStrings__database")'
    - '$envVar.SetAttribute("value", $base64decoded)'
    - '$xml.configuration.location."system.webServer".aspNetCore.environmentVariables.AppendChild($envVar) | Out-Null'
    - '$xml.Save($filePath)'

Resulting web.config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\sampleproject01.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess">
        <environmentVariables>
          <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
          <environmentVariable name="ConnectionStrings__database" value="Server=XXXX;database=YYY;User Id=ZZZZ;Password=XXX;" />
        </environmentVariables>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

The ConnectionStrings__database environment variable will be used as value for:

"ConnectionStrings": {
    "database": ""
}

Notes:

1. Variable Dev__ConnectionStrings__database is stored as base64 in GitLab (because it complains about some characters if 'masked' is enable), so the script have to decode it.
2. The GitLab runner used in this pipeline uses PowerShell.
3. See documentation for using environment variables in application configuration.

Preeminence

Answer 6

This seems to work for me in the end.

After hours of trying, I tried to skip the quotes at the beginning. The later quotes do work while escaping them.

With the quotes at the beginning, the value wasn't getting replaced.

- sed -i -E "s|var1.*|var1\":\"$VAR_1_REPL\",|g" file.json
- sed -i -E "s|var2.*|var2\":\"$VAR_2_REPL\"|g" file.json

The JSON is like

{
  "var1":"var1orig",
  "var2":"var2orig"
}

Epinephrine