Channel is not opened Errors in automatic merge on Teamcity (git)

Question

I am using TeamCity on ubuntu (TeamCity Professional 9.1.7 (build 37573)). And having some problems with automatic merges lately (git). I haven't changed anything in the configuration from when it used to work. I tried to remove the work directory. it only helped for a couple of merges.

Symptoms: The build log looks clean, and I am getting this on the build overview:

Failed to merge sources in VCS root foo. Merge error 'git fetch'     command failed.
stderr: [email protected]:foo/foo.git: channel is not opened.
exit code: 1.

Although, in the agent running this i am getting a totally different error in teamcity-vcs.log:

[2016-03-30 15:14:31,722]   INFO -      jetbrains.buildServer.VCS - [/home/ubuntu/BuildAgent/work/7676127c0a691f42]: /usr/bin/git show-ref refs/remotes/origin/foo 
[2016-03-30 15:14:31,775]   INFO -      jetbrains.buildServer.VCS - [/home/ubuntu/BuildAgent/work/7676127c0a691f42]: /usr/bin/git log -n1 --pretty=format:%H%x20%s 193f46d88205c5e419a8a7458e742ce9b598cca8 
-- 
[2016-03-30 15:14:31,797]   WARN -      jetbrains.buildServer.VCS - '/usr/bin/git log -n1 --pretty=format:%H%x20%s 193f46d88205c5e419a8a7458e742ce9b598cca8 --' command failed.
stderr: fatal: bad object 193f46d88205c5e419a8a7458e742ce9b598cca8 
[2016-03-30 15:14:31,798]   INFO -      jetbrains.buildServer.VCS - [/home/ubuntu/BuildAgent/work/7676127c0a691f42]: /usr/bin/git fetch --progress origin +refs/heads/foo:refs/remotes/origin/foo 
[2016-03-30 15:14:35,832]   WARN -      jetbrains.buildServer.VCS - Error output produced by: /usr/bin/git fetch --progress origin +refs/heads/foo:refs/remotes/origin/foo 
[2016-03-30 15:14:35,832]   WARN -      jetbrains.buildServer.VCS - remote: Counting objects: 2, done.ESC[K
remote: Compressing objects:  50% (1/2)   ESC[K
remote: Compressing objects: 100% (2/2)   ESC[K
remote: Compressing objects: 100% (2/2), done.ESC[K
remote: Total 2 (delta 1), reused 0 (delta 0)ESC[K
From bitbucket.org:bar/bar
   62ba378..193f46d  foo  -> origin/foo 

As if the log command failed..

Thanks.

---

Update: I got an update that there is anecdotal evidence that this is due to changes specifically in BitBucket's git over SSH servers.

Baud

We are seeing the same issue, only started today – Thirteenth

+1 Just started seeing this in the last couple of days after I upgraded Ubuntu. I am running TeamCity on EC2 and using Git on BitBucket. Are you also running on EC2 or is this in your own data center? – Armin

Getting this as well since a few days. I thought some sort of rate limiting BitBucket side. The thing I notice is that the errors are more frequent right after a push. My interval time is 30 seconds, usually the first check after a push fails while the following works. Self hosted on OVH. – Holston

We started getting the same thing around Friday, July 8 (2016). Any idea how to address??? – Brigettebrigg

I guess one way is to shift to alternate hosting and see if the error still occurs. It will be annoying. – Armin

This is also reported on JetBrains System - youtrack.jetbrains.com/issue/TW-46052 – Thirteenth

Answer 1

Per https://youtrack.jetbrains.com/issue/TW-46052, the problem has been fixed. The fix will be shipped with the next release, either 9.1.8 or 10. If you don't want to wait for the release, you can manually update the git plugin by downloading it from here (logging in as a guest, or creating an account on JetBrains' build server) and replacing it in your installation.

• Shut down TC
• Put the downloaded file jetbrains.git.zip in %TEAM_CITY%/webapps/ROOT/WEB-INF/plugins replacing the existing one
• Restart TC

This fixed the problem for me and others.

Holston

This made the error go away, but builds still hang attempting to collect changes. – Consumedly

Appears Atlassian changed something last night, bc now all is working well. – Consumedly

No need to create an account on the build server, just click "Log in as guest" – Ranged

I replaced the git plugin file, restarted (yesterday) and did not see the problem again yet. Thanks. – Baud

Answer 2

TeamCity used ssh multiplexing which was recently disabled in Bitbucket Cloud. If you are affected by this problem, please watch the https://youtrack.jetbrains.com/issue/TW-46052, fixes will be posted there (at the moment of writing there is a fix for TeamCity 9.1.x and 9.0.x). A workaround without installing a newer git-plugin is to use https or anonymous protocol instead of ssh.

Update

The initial fix caused changes collection to fail with timeout errors in TeamCity 8.1.x and 9.0.x, this is fixed now.

Underbred

Falling back to use https instead of ssh, quick and easy fix. The plugin addition only works for some build configurations, one worked for me and one kept on getting stucked in Collecting changes in vcs root – Predisposition

No, falling back to HTTPS does not work for those of us with large, large bitbucket repos that are several years old. SSH worked fine for this. Shame on Bitbucket and Atlassian. – Consumedly

This link is the right place to download (in my case it was 8.1.4). And you just need to drop it into the <TeamCity Data Dir>/plugins directory (despite the fact it is already bundled, this will override the bundled copy). – Shuttle

Answer 3

Updated: JetBrains has released a patch for this issue - see https://youtrack.jetbrains.com/issue/TW-46052 for details.

Stoneman

I have also emailed – Thirteenth

Another (new) TC 9.x user here with the same issue. Tried upgrading my Git executable on Ubuntu 14.04 to 2.x. This didn't seem to solve the issue. – Rodger

Not happy about Bitbucket introducing breaking changes like this without notification. Wasted some hours chasing this down. – Templetempler

Why not revert the change and post a proper notice to customers that we will be impacted? – Consumedly

Half working for me, On the one with a seperate agent, it works. On the build with the agent on the teamcity box, it is stucked at Collecting changes from VCS root – Predisposition

Answer 4

I've ran into this problem twice already and it got annoying.

For linux users which properly installed TeamCity (e.g. have a separate user and a teamcity service), this script should do all the heavy lifting (given you have sudo):

function patch_teamcity_plugin() {
  # Find teamcity plugin dir.
  sudo updatedb
  path_part="webapps/ROOT/WEB-INF/plugins"
  teamcity_plugin_dir=$(locate $path_part | grep -o -E ".*$path_part" | head -n 1)

  if [ "$teamcity_plugin_dir" == "" ]; then
    echo "Cannot find teamcity plugins directory." 1>&2
    return 1
  fi

  login_url="https://teamcity.jetbrains.com/guestLogin.html?guest=1"
  cookie_file=$(mktemp)

  # Log in as guest.
  # Wget login - thanks to: https://mcmap.net/q/99048/-how-to-get-past-the-login-page-with-wget
  wget --save-cookies $cookie_file --keep-session-cookies $login_url

  # Download the plugin to the teamcity plugin directory.
  plugin_url="https://teamcity.jetbrains.com/repository/download/TeamCityPluginsByJetBrains_Git_JetBrainsGitPluginTeamCity91x/843194:id/jetbrains.git.zip"
  plugin_dest_filename="$teamcity_plugin_dir/jetbrains.git.zip"
  sudo wget --load-cookies $cookie_file $plugin_url -O $plugin_dest_filename

  if [ "$?" != "0" ]; then
    echo "Failed download of plugin." 1>&2
    return 1
  fi

  # Copy permissions - thanks to: http://unix.stackexchange.com/a/20646
  sudo chown --reference="$teamcity_plugin_dir" "$plugin_dest_filename"
  sudo chmod 0755 "$plugin_dest_filename"

  # Restart TeamCity server.
  sudo service teamcity restart
}

patch_teamcity_plugin()

Funnelform