Windows Service user account trouble for TFSBuildServiceHost.exe

Question

Experienced a very strange problem today on our TFS2010 build server. Suddenly the build service failed for no apparent reason. We´re been trouble shooting it all day, but still haven´t found the reason yet.

One of the problems is that the build service is (or should!) running under an AD user called tfs2010build. However when I try to start the service, i get the following error

Service cannot be started. Microsoft.TeamFoundation.TeamFoundationServerUnauthorizedException: TF30063: You are not authorized to access http://tfs2010:8080/tfs/default. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.

When I look in the event log on the TFS2010 server, I see that the failed authentication is registered for a user called TFS2010Install, which was used to install everything. I´ve tripple checked and the service is specified as to be running under TFS2010Build.

Log from TFS2010 server:

Account For Which Logon Failed:
Security ID:        NULL SID
Account Name:       TFS2010INSTALL
Account Domain:     LC

So my question is how is this possible. COuld the user TFS2010Build some how be impersonated by TFS2010Install? I ve tried to install an additional build server and here there´s no problem starting the service under the user TFS2010Build - hence it is not a problem with AD or TFS user rights.

Hope you guys can help out!

/Jasper

!! Updated with some screen shots. Build server is TFS2010BIULD and the TFS server is TFS2010

Link to full size

Screen shot of non working build server TFS2010Build

Screen shot of working build server TFS2010Build1

!!New Update

I've managed to get the Build service to run under the TFS2010Build user account (which was actually the initial state, when the problem started). When I queue builds to this controller and agent, i get the follwing in the build log:

TF215097: An error occurred while initializing a build for build definition \PlanteIT_MarkOnline_Scrum\CI_Main_FieldOnlineClient: TF215106: Access denied. LC\TFS2010INSTALL needs Update build information permissions for build definition CI_Main_FieldOnlineClient in team project PlanteIT_MarkOnline_Scrum to perform the action. For more information, contact the Team Foundation Server administrator.

It still insist that TFS2010Install user account is running the service, despite that TFS2010Build is used for the build service. Any ideas?

Pollinosis

Hi Jasper, just to make sure we're not hunting ghosts here; please re-enter the tfs2010build username and password in the build service properties and restart the service. Also remove the build agent definition from the controller and re-add it again, using the tfs2010build credentials. – Britnibrito

Hi kroonwij, went through it again - same result. Any ideas? It must the that specific machine it´s wrong with, mixing up the accounts some how, since the other build server works fine. Besides the buildserverhost.exe service, are there any other services that I might have missed? – Pollinosis

Can you post a screenshot of the service credential page for that service? – Britnibrito

Sorry, unreadable due to its size. Can you upload individual dialogs independently? – Britnibrito

Right click on the image and open in new tab for full size. – Pollinosis

How are you (interactively) logged in when you perform these actions? Can you login as tfs2010build and retry to connect your build service to the controller? And try to connect to your TFS server collection endpoint (the incoming connection) parameter using internet explorer. – Britnibrito

When logged in as user TFS2010Install, and configuring the build service, i don´t get an error until I try to start the service. If I´m logged in as TFS2010Build user, I´ll get prompted for user credentials as soon as I try to specify the input "Connect to Team Project Collection (outgoing)". Which endpoint are you talking about. tfs2010build.lc.skejby:9191/Build/v3.0/Services or tfs2010:8080/tfs/default ? – Pollinosis

I think we got it now. Your server is running under tfs2010build user, while it does not seem to have access to your TFS server right away. You have to log in with another user account (probably tfs2010install?) to get connected. So your service itself cannot connect authonomously to the TFS service, which is in line with the event log message. Please double-check your OK build service to have identical settings to your problem service. Also post a screenshot of your build controller configuration dialog that lists both build agents, one running, one not. We are almost there :-) – Britnibrito

So no build agents are configured on the faulty build server. Just to rule out any further problems. – Pollinosis

Anything else besides the Build Service Properties I can check? – Pollinosis

Two things I notice: 1) the failing service domain is spelled in lowercase. Normally when you use the browse button, the domain letters should be the same for both. Howcome the difference? 2) In normal operation, only one build controller can be connected to one TFS collection at a time, make sure the working one is discoupled before trying to re-connect the faulting one. Do that help? – Britnibrito

And maybe the content of this post helps: marknic.net/2010/05/14/… – Britnibrito

Ok, so I tried disabling the controller on TFS2010Build1, changing the domain to upper case, but I still cant start the service – Pollinosis

Having 2 controllers, are only due to trouble shooting the issue. We of course will only have a single controller and multiple agent services. The big mystery, from my pov, is why the build service tries to authorize it selv as TFS2010Install, when clearly the service is started as the build account TFS2010Build. – Pollinosis

I think the build service tries to authorize it selv as TFS2010Install because that is the user you are logging in with at time of assigning the build service to the project collection. You stated that 8 comments ago. Maybe re-register your failing system to the LC domain? – Britnibrito

Well, on TFS2010Build1 I can install, configure and start as TFS2010Install user - no problem. But yes, it may be a domain issue. Not sure I can re-register the machine to the domain, as soon as I take it offline. Think I´ll let ops do that. Thanks you so much for your help so far!! Let me know if you come up with any other ideas. – Pollinosis

Answer 1

This is a stab in the dark, can you try clear the TFS client cache and your internet cache on your troubled build machine under the Tfs2010Build account? I've never seen this issue before but maybe some stale cached TfsProjectCollection object with TFS2010Install authentication stayed around and caused problems.

Have you also tried reconfigure your build machine?

To unconfigure: tfsconfig.exe setup /uninstall:TeamBuild

and reconfigure through the wizard.

Lucie

Another thing is to check your Windows Credentials Vault of the build service account and remove any existing credentials for the TFS server your build machine is connecting to, then reconfigure your build machine. – Lucie

Excellent suggestion - I'll check this! Would love to find out what exactly happened - if it should happen again. – Pollinosis

It actually worked!! I opened the Credential Manager, deleted all credentials and it solved the problem! Pretty good stab in the dark! The machine will be deleted, however it's good to have solved the puzzle! Thanks again. – Pollinosis

@DuatLe I love you for that!!! Even I encountered the same issue, and tried reconfiguring and that worked like charm!!! And as a confirmation, the tfs build config wizard itself showed a warning within the setup check list that if we ever change the password of the account, and it should be changed in windows credential manager as well.... that confirms ur finding!!! :) – Masque

Answer 2

I will try once more ..., step by step :-)

1. FACT: When you register your build controller to a TFS project collection, being logged-in as TFS2010Build, an authentication dialog pops-up. This means that the TFS server does not accept TFS2010Build as an account that can be used to connect to your default collection on the TFS server.

2. FACT: When you register your build controller to a TFS project collection, being logged-in as TFS2010Install, no authentication dialog pops-up. This means that the TFS server does accept TFS2010Install as an account that can be used to connect to your default collection on the TFS server.

3. Apparently, because in both 1 and 2 your build controller is registered using the TFS2010Install account to the TFS server, either the controller or the server remembers these credentials and uses them to connect to the TFS server collection when the build controller is started, despite the fact that the service itself is running under the TFS2010Build account. This is a plausible situation and impersonation happens often this way for services. Maybe some TFS techie can either confirm or deny this behavior.

The question that remains for me: Why does the the default collection on the TFS server not accept the TFS2010Build account as a valid administrator?

Potential causes:

1. Read Jim Lamb's answer.

2. Something is wrong with the domain registration of the system or user used to connect the controller to the collection on the TFS server.

Fastest way to rid of the problem: Continue to install the secondary server that does not seem to have the problem, potentially experiment with using the TFS2010Build from this secondary server to see if the problem also occurs there.

A long aswer, but hopefully it gives you a big push in the right direction.

Britnibrito

Answer 3

Sorry to hear that you're having problems getting this to work. Here are a couple of things you can check/try:

• Make sure that the TFS2010Build user account is a member of the "Build Services" group in the TFS project collection you've associated it with.

• If you install and configure the build service while logged in as a user who is a member of the Project Collection Administrators group on the associated project collection and is also a member of the local Administrators group on the build machine, all of the requisite permissions and other configuration will generally be set for you.

So, to summarize, the user configuring the build machine should be a member of the project collection administrators group and a member of the local administrators group. And, the user account the build machine is running as should be a member of the project collection's "build services" group.

Pie

Hi Jim - Thanks for the answer. The user is a member of the project coll. admin group, local admin group and a member of the build service group. The thing is, that I have one build machine working (still installing dependecies, however controller/agent works as expected) and one that suddenly stopped working. Im pretty sure the user accounts are set up as expected - kroonwijk help me with that. But for some reason, the build service account TFS2010Build, tries to authorize itself as TFS2010Install towards the team foundation server. – Pollinosis