TeamCity getting stuck at "Updating sources" on one Git repo

Question

I've just set up TeamCity 8.0 on Windows to build the projects (Visual Studio solutions) in four separate Git repositories. It is working as expected on three of them, but on one it gets stuck at Updating sources. The settings for the four VCS roots are identical (apart from the Git repo fetch URLs).

All of the TeamCity projects are using server-side checkout.

The build log for the problematic project contains

[15:36:34]: bt1 (running for 41m:38s)

[15:36:34]: Checking for changes (running for 41m:37s)

[15:36:37]: Publishing internal artifacts

[15:36:38]: [Publishing internal artifacts] Sending build.start.properties.gz file

[15:36:37]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp

[15:36:37]: Checkout directory: C:\TeamCity\buildAgent\work\62d0281b7178c739

[15:36:37]: Updating sources: server side checkout (running for 41m:34s)

[15:36:38]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources

[15:36:38]: [Updating sources] Building and caching clean patch for VCS root: [email protected]:Company.WebSite.git#master

Using Process Monitor on the build agent I can see that it is hammering the following folder with ReadFile operations.

C:\ProgramData\JetBrains\TeamCity\system\caches\git\git-11F9493A.git\objects\pack\pack-1490ccc8f7896ab876413465c4b48e87448bed35.pack

That .pack file is about 300MB in size, which I believe to be the same size as the Git repo.

I've tried deleting this folder and re-starting the TeamCity build agent and build server Windows services, but it just get re-created and then TeamCity hammers it once more.

Are there any other recommendations for how to troubleshoot this kind of issue?

Mousetail

Answer 1

I cross-posted this on the TeamCity forums, and got an answer there.

TeamCity has a default per-file size limit of 128MB; if there are any files in your repo that are or ever have been larger than this limit then the repo clone fails.

We have a SQL script in our repo that used to be around 200MB in size.

The fix was to set an internal parameter in TeamCity to increase this limit:

teamcity.git.stream.file.threshold.mb=256

Mousetail

Answer 2

The scenario in the accepted answer did not apply to me. I was able to fix the problem by simply creating a new VCS root to the same repo and attaching it to my build configuration.

Engel

Weirdly enough, this worked for me, too (TeamCity 9.1.4). – Gilberto

Same here but instead of creating a new VCS root I just edited the existing one, saved it and then re-ran the build and it just started working. Strange. (9.0.3) – Anallese

Answer 3

Use the Enforce Clean Checkout option in the Build configuration Actions menu.

The next time you run the build it will re-create the local git repo and should return to normal.

Ovariotomy

Thanks, but this wouldn't have worked in our scenario, as per my answer. – Mousetail

@RichardEverett I admit that is true - however that was also not the solution for my problem, This did work for me, so I thought I would share. – Ovariotomy