Push to local Azure DevOps Git from Build Pipeline
Asked Answered
B

5

24

Short version

Can someone tell me how to set up a "Command Line Script" task within an Azure DevOps build pipeline that pushes changes to a local Git repository (in fact, the Git repository on which the pipeline is based)?

No matter what I try, my script always times out after printing Pushing commits to git.

Longer version

We are migrating existing Java/Maven projects from a Jenkins build server to an Azure DevOps build environment, and I am trying to set up a build pipeline that mimics the Jenkins "Release Staging" functionality.

My first attempt was to call the Maven release plugin directly on the checked out sources. This involved several obstacles, most of which I was able to overcome in one way or the other:

  1. The standard Azure DevOps git environment doesn't have the required config values "user.email" and "user.name" set; this can be solved by calling git config in a separate "Command Line Script" task prior to pushing.
  2. Azure DevOps checks out the sources in a "detached HEAD state" - to solve this problem, the setup script tasks also calls git checkout master.

Once this is set up, my Maven call script runs up to the point where the release plugin attempts to push to Git; the corresponding lines in the log file read

[INFO] Executing: cmd.exe /X /C "git push https://xxx.visualstudio.com/YYY/_git/zzz refs/heads/master:refs/heads/master"
[INFO] Working directory: D:\a\1\s</code>

After that, nothing happens (at least, nothing gets logged) until the timeout strikes:

##[error]The operation was canceled.
##[section]Finishing: Maven pom.xml

In order to find out what causes this problem I tried a few things, among others,

  • disabling the maven call and calling git push directly from the script,
  • registering a "store" type Git credential helper (assuming that an authentication problem is responsible for the timeout),

but without success. I am now running out of ideas how I could get the git push call to work - is there anyone out here who can help me out?

P.S.: As you may be able to tell, I'm rather a newbie as far as Azure DevOps is concerned, so I surely don't know all tricks and features of that system. In particular, I don't know if there is anything that offers the same functionality as the Maven release plugin. We do use the Azure DevOps package management, but we want to keep separate feeds for snapshots and for release builds (the way tools like Nexus do), so we have to have a mechanism for automatically advancing release numbers, checking out and back in and building and publishing the packaged module to the release feed.

If someone can suggest an alternative way to achieve that, I'm open to suggestions as well.

Bacillary answered 17/9, 2018 at 10:36 Comment(0)
V
4

To push change to Azure DevOps, you should integrate your credential in Git repo URL:

  • First, create a PAT if you not have.
  • Then use below command to push:

    git push https://Personal%20Access%20Token:[email protected]/YYY/_git/zzz master
    
Villalobos answered 18/9, 2018 at 10:35 Comment(6)
Thanks a whole lot for the answer; using a PAT works (even in Command Line Task).Bacillary
Don't PATs expire after 90 days or so? My situation is my build process modifies a file in the Git repository on which the pipeline is based and I want to commit/push that modification. Isn't there a better option for having Azure DevOps build update its own Git repo (since it's acting in its own context - it's not like it's an external source)? It seems like a hassle to generate the PAT for the service account under which the agent is running every few months.Milord
An additional challenge to this is if I store my PAT as a "secret variable" in a variable set (so it can be used across builds) it fails to work. So now I have to store it in the variable set as clear text. It works - I just think there has to be a better way.Milord
Secret variables are not automatically populated to the environment variables of the build agent, so for things like powershell you need to use the explicit Environment Variables section of the task to provide the secret to the script. As for the expiration, you can set the expiration up to 12 months.Jovia
You can use a GitHub App, it does not expire.Annamaeannamaria
For Github: git push https://<Github-username>:<token>@github.com/GitOrgName/GitRepoName --branch=main Irby
A
42

You shouldn't have to go the PAT route for auth - according to this, if your repo is part of the same Azure DevOps project as the build pipeline, credentials should just flow. Is it possible you haven't authorized the build agent to write to your repos? Two things are needed:

  • Under Project Settings -> Repositories for your Azure DevOps project, allow the Project Collection Build Service entity Contributor rights to the appropriate repo (or all project repos).

  • Allow scripts to access the OAuth token under the "Agent job" settings:

    enter image description here

Also note a bad gotcha: this won't work for submodule operations, since DevOps does not automatically flow the credentials to the submodule instances, and the only symptom is a silent hang. Workaround to flow credentials manually is found here.

Automatic answered 17/2, 2019 at 0:8 Comment(5)
If you're using the newest YAML-based VSTS, "Allow scripts to access the OAuth token", you need to use "persistCredentials: true" in your azure-pipeline.yml file, under "steps: -checkout: self" See learn.microsoft.com/en-us/azure/devops/pipelines/scripts/…Sallyanne
This the correct answer and should be marked as so by the op.Lopes
can you please share the command line script or detailed process of pushing the local changes from pipeline to repository? I m facing issues and have posted a question, here is the link #57680990Eufemiaeugen
I don't see Additional optionsArapaima
For the newest version: you can directly check how to grant-version-control-permissions-to-the-build-service. After that you can use git commands without any problems.Hippocras
P
28

The answers have become out of date with how Azure Devops works now. To enable authorisation in your pipeline you should use the checkout schema property in your steps so you can persist the credentials for later git commands.

- checkout: self
  persistCredentials: true
Pilcomayo answered 16/4, 2020 at 17:4 Comment(0)
V
4

To push change to Azure DevOps, you should integrate your credential in Git repo URL:

  • First, create a PAT if you not have.
  • Then use below command to push:

    git push https://Personal%20Access%20Token:[email protected]/YYY/_git/zzz master
    
Villalobos answered 18/9, 2018 at 10:35 Comment(6)
Thanks a whole lot for the answer; using a PAT works (even in Command Line Task).Bacillary
Don't PATs expire after 90 days or so? My situation is my build process modifies a file in the Git repository on which the pipeline is based and I want to commit/push that modification. Isn't there a better option for having Azure DevOps build update its own Git repo (since it's acting in its own context - it's not like it's an external source)? It seems like a hassle to generate the PAT for the service account under which the agent is running every few months.Milord
An additional challenge to this is if I store my PAT as a "secret variable" in a variable set (so it can be used across builds) it fails to work. So now I have to store it in the variable set as clear text. It works - I just think there has to be a better way.Milord
Secret variables are not automatically populated to the environment variables of the build agent, so for things like powershell you need to use the explicit Environment Variables section of the task to provide the secret to the script. As for the expiration, you can set the expiration up to 12 months.Jovia
You can use a GitHub App, it does not expire.Annamaeannamaria
For Github: git push https://<Github-username>:<token>@github.com/GitOrgName/GitRepoName --branch=main Irby
R
2

Yes this is possible, you no longer need to use PAT authentication.

You first need to grant access to the Build Service so this account can commit to your repository.

To do so, go to Your Project > Repositories > Security and grant the build service Contribue, Read, Create Tag, Create Branch permissions.

Then you need to checkout with the below inputs.

Pipelines created after September 2022 explicitly need fetchDepth set to 0 to fetch all branches, failing to set this will result in minimal branches being fetched which could result in being unable to checkout to your desired branch.

steps:
- checkout: self
  persistCredentials: true
  fetchDepth: 0

- bash: |
    echo "This Is Build $(Build.BuildId)" > "$(Build.SourcesDirectory)/BuildNumber"
  displayName: Create File

- bash: |
    git config user.name "Example"
    git config user.email "[email protected]"

    git checkout main --
    git add --all
    git commit -m "Adding Build File"
    git push origin main
  displayName: Git Commit

That is all that is required!

source: https://jimferrari.com/2023/03/29/commit-files-to-azure-repos-git-during-azure-pipelines-run/

Raynard answered 30/3, 2023 at 9:16 Comment(0)
D
0

I've encountered the same problem (git push hangs till timeout) in on premise DevOps Server even if Allow scripts to access the OAuth token as well as Repo permissions were correctly set.

In my case the problem was that Build Agent adds a path at the beginning of PATH environment variable - for me it was 'C:\ins\agent\externals\git\cmd\'. Here an old git version (git version 2.18.0.windows.1) resided and was launched in the context of build agent's pipeline. You can verify this by adding "git --version" call to your build task.

I haven't investigated why the old git version didn't work and simply used the full path qualification for my up-to-date git.exe in the pipeline task

"C:\Program Files\Git\bin\git.exe" push origin master

and with this git version - currently "git version 2.24.0.windows.2" everything works well. Updating your Build Agent to a version with newer git.exe should solve this problem too, of course.

Demarche answered 22/11, 2019 at 11:21 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.