List remote refs failed: java.net.ConnectException: Connection time out: github.com

Question

I've recently starting seeing the above error with ever-increasing frequency on our build server. Nothing has changed in our TeamCity configuration during this period, so I'm guessing it might be changes at GitHub that are causing the error.

I've tried changing our VCS polling interval from 60s down to 600s in case GitHub was doing some kind of connection throttling, but there has been no affect.

Is it possible to make TeamCity less sensitive to connection timeouts?

Mange

Answer 1

I've figured out the answer.

TeamCity has no issues - it's actually AZURE that has a problem.

For proof, try doing this in your server, where TC is installed.

(command line, of course)

C:\git\bin\git.exe clone https://github.com/libgit2/libgit2.git

and this should not work most of the time.

So AZURE has a networking bug and they know about it and are trying to resolve the issue.

This info was provided via GitHub after they worked with Azure to figure out what was going on.

Conclusion

You have to use SSH KEYS as a current workaround.

Trapezium

That's really interesting, our TC is indeed on Azure too. When you say they know about it, do you mean TC or Microsoft? Is there anything official from them? – Mange

MS know about it and even have a real open ticket for it. – Trapezium

@Trapezium is it the public ticket? Can you share a link to the ticket? – Peristalsis

This was exactly the same scenario I had: build box on azure git over http failing. I've switched to SSH and so far it is working well. – Revengeful

It's giving us a lot of pain. Does anybody know if it has been resolved? – Anadiplosis

Answer 2

We have made it less sensitive: now git-plugin will retry an operation in the case of ConnectException. To get this behaviour you need to install the latest build of git-plugin from teamcity.jetbrains.com. To do that put the zip from the build's artifacts into <TEAMCITY_DATA_DIR>/plugins and restart the server.

Peristalsis

Thanks for this response, I'm trying it out now and will let you know if it works out for me. – Mange

Can you check that link?? It's goes to a page where we need to login? Also, isn't GIT baked in to TC, since TC v5 so we don't need to download anything extra? (ref: confluence.jetbrains.com/display/TW/TeamCity+Plugins) – Trapezium

@Trapezium The link does go to a login page, but there is a "log in as guest" text link below the login button. – Mange

@Peristalsis - do we need to do this with TC version 8.1a? If so, why wasn't it including in the latest release? – Trapezium

@Trapezium you don't need to do this in 8.1 – Peristalsis

Turns out this worked for about a month, but now we're getting even more timeouts than before. – Mange

@Stony, please install the latest build of plugin it has more improvements in this area. – Peristalsis

Even with SSH keys and TeamCity 8.1.1 and using the latest git-plugin from teamcity.jetbrains.com/… I have troubles. But I get an other exception this time: Failed for the root 'master' #135: List remote refs failed: java.io.InterruptedIOException: Read timed out Ideas? I'm on Azure and using github as well – Zehe

@Peristalsis Switching to SSH fixed it for me - maybe this Azure networking bug is causing my issues, as I am hosting TC on Azure. – Mange

Switching to SSH doesn't seem to have solved the problem for me, it is still intermittent, and if anything this is worse in my situation. Did you have to open any special ports or make any Azure configuration changes for the SSH to work? – Snorkel

Answer 3

We've got this error all of sudden today (Feb 23, 2018).

Turns out GitHub.com deprecated some of the security algorithms in their ssh implementation.

Solutions:

• switch to "https://..." URL in the affected VCS roots and provide your credentials for authentication
• upgrade to the latest TeamCity version

Here's an announcement from JetBrains: https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360000115644-Builds-using-GitHub-com-fail-with-VCS-errors-since-February-22-FIX-AVAILABLE

Unsure

Answer 4

Switching to ssh helped us to avoid timeouts.

Womble

Answer 5

As @Pure.Krome says, you can use SSH KEYS to allow you to access the Azure TFS GIT URL, or you can modify the settings in "Alternative Credentials" by checking "Enable alternate credentials" so you can access the repository by just using a username and password.

First in Azure DevOps go to your project:

Then choose Security in the RHS menu:

Finally choose "Alternate Credentials" in the LHS menu and click the "Enable authentication alternate credentials" where you will then put in your username and password:

Please beware that this is less secure than using the SSH KEYS.

Hyetography

Answer 6

I got this issue even when using ssh. I realized that there was a firewall that was seeing all the ssh traffic as an ssh brute force attack. The firewall would then block the request.

A firewall rule was added to resolve the issue.

Johnsonian