TFS check in timeout of changeset containing "larger" binary files

Question

I'm performing a TFS Integration migration from tfs.visualstudio to an on-premise 2012 server. I'm running into a problem with a particular changeset that contains multiple binary files in excess of 1 MB, a few of which are 15-16 MB. [I'm working remotely (WAN) with the on-premise TFS]

From the TFSI logs, I'm seeing: Microsoft.TeamFoundation.VersionControl.Client.VersionControlException: C:\TfsIPData\42\******\Foo.msi: The request was aborted: The request was canceled. ---> System.Net.WebException: The request was aborted: The request was canceled. ---> System.IO.IOException: Cannot close stream until all bytes are written.

Doing some Googling, I've encountered others running into similar issues, not necessarily concerning TFS Integration. I'm confident this same issue would arise if I were just checking in a changeset like normal that met the same criteria. As I understand it, when uploading files (checking in), the default chunk size is 16MB and the timeout is 5 minutes.

My internet upload speed is only 1Mbit/s at this site. (While I think the problem would mitigated with sufficient upload bandwidth, it wouldn't solve the problem).

Using TCPView, I've watched the connections to the TFS server from my client while the upload was in progress. What I see is 9 simultaneous connections. My bandwidth is thus getting shared among 9 file uploads. Sure enough, after about 5 minutes the connections crap out before the upload byte counts can finish.

My question is, how can I configure my TFS client to utilize fewer concurrent connections, and/or smaller chunk sizes, and/or increased timeouts? Can this be done globally somewhere to cover VS, TF.EXE, and TFS Integration?

Maurili

Answer 1

After spending some time with IL DASM poking around in Microsoft.TeamFoundation.VersionControl.Client.dll FileUploader, I discovered in the constructor the string VersionControl.UploadChunkSize. It looked like it is used to override the default chunk size (DefaultUploadChunkSize = 0x01000000).

So, I added this to TfsMigrationShell.exe.config

<appSettings>
    <add key="VersionControl.UploadChunkSize" value="2097152" /> 
</appSettings>

and ran the VC migration again -- this time it got past the problem changeset!

Basically the TFS client DLL will try and upload multiple files simultaneously (9 in my case). Your upload bandwidth will be split among the files, and if any individual file transfer cannot complete 16MB in 5 minutes, the operation will fail. So you can see that with modest upload bandwidths, changesets containing multiple binary files can possibly timeout. The only thing you can control is the bytecount of each 5 minute timeout chunk. The default is 16MB, but you can reduce it. I reduced mine to 2MB.

I imagine this could be done to devenv.exe.config to deal with the same problem when performing developer code check ins. Hopefully this information will help somebody else out and save them some time.

Maurili

Thanks for finding this. this support post seems to have the wrong string. – Flatt

Hi @LatencyMachine, are you 100% sure that this error is related to the config setting of the tool config (we can say tool config and devenv.exe.config interchangeably). Because I believe that this error might be because of the Target TFS having some limit on the checkin of files via network. What do you think? – Windtight

Just asking... We have set the value="102400" (that is:100kb). Still the tool it is failing. WE are migrating 15 files (size of files vary from 1MB to 12MB). The error is: The operation has timed out. Also in logs we can see -> "Checkin failed for unknown reason" – Windtight

My post in MSDN forums: social.msdn.microsoft.com/Forums/vstudio/en-US/… I need some feedback from you guys. Thanks. – Windtight

We also notice that this error shows up when doing a sync of large files (>5MB) between 2 TFS servers over a firewall network. If we do the sync locally, between 2 TFS servers, there are no errors and no need to change the UploadChunkSize! Do you see any reason this might happen? – Windtight

@Windtight Make sure that the .config file you are modifying is indeed the appropriate one for the environment of the failed operation. Use TCPView or similar tool to get an idea of what is actually happening. My gut tells me that UploadChunkSize isn't changing. I'm not aware of any file size limit, and I've routinely checked in embedded Linux OS images in excess of 200MB before. – Maurili

@LatencyMachine Hello! Thanks, here is a new finding: The 2 way sync from TFS A to TFS B fails with above error, where as the 2 way sync from TFS B to TFS A works! Do you see any reason for this? Note that above both cases were tested with 2 separate projects – Windtight

^2 separate projects with same files – Windtight