Azure: Worker role looping through "recycling"

Question

I'm currently working on an Azure project that works 100% locally with emulator resources. I'm now trying to deploy a worker role, but I'm running into an issue that I'm not sure how to troubleshoot.

Upon deploying the worker role in my Azure portal, the two instances continually loop through "recycling".

I can try to RDP into the role, but I only have about a minute to look around before the connection closes, I'm assuming due to the recycling.

After some searching it doesn't seem like this is a super common problem. Is there something trivial I'm overlooking that could be causing this issue? How would you go about troubleshooting this? Thank you for your time :)

Puklich

Try to unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause. – Exacting

Thanks Avkash. The error is "Faulting application WaWorkerHost.exe" - ever run into that before? – Puklich

Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you. I will also add this info as answer... – Exacting

Great, thanks for the quick response. I'll submit an Azure Support incident if I can't figure it out. How would I go about seeing the fault stack? Does that require something like intellitrace? – Puklich

I have the same/similar issue. Viewing the operation logs on Azure or using the diagnostic monitoring through VS may give you some insight... I hope that helps. – Patrinapatriot

Answer 1

From Avkash above:

Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you.

Puklich

Answer 2

In case of missing Reference you can troubleshoot this issue by:

Unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause.

IF you could see the exception in event log and look for the exception, you sure can find where it was generated. You can also use Intellitrace which might require you to redeploy the app.

Also there are ways were copying WinDBG and locking to the specific process you can debug it. I am not sure how much you would want to try but just copy the WinDBG to VM and use it would be enough (not sure how much experience you have with WinDBG though and how much time you would want to spent.)

Exacting

+1 for the Application Event Log tip. Also, verify all your external references are "Copy Local" true, it's a little easier and almost as effective as analyzing the package itself. – Castellano

Answer 3

Also been pestered by this role recycle issue numerous times. Here is the sequence of steps to debug persistent role recycles:

Debugging Azure Role Recycles

1. Enable Remote Access to your role - RDP login
2. Check eventvwr.msc (Windows Logs -> Application, App & Service Logs->Windows Azure)
3. Review the Azure text file logs across both C:\logs and c:\resources
4. Review custom logs in the Volume E: or F: for any custom role startup logging

5. Run AzureTools and attach to startup processes (download WinDBG, use Utils->Attach Debugger, select process - WaWorkerHost/WaIISHost, etc), use G to continue and watch debugger output for assemblies failing to load.

   Installing Azure Debugging Tools via Powershell

   PS> md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe; c:\tools\AzureTools.exe

If all items above fail - try using other tools in the AzureTools treasure trove - such as fusion logging, etc, this approach above will work!

WinDBG Sample Output - Failing to Locate Assembly (WaIISHost)

Heliotropin

Answer 4

The most likely cause is that you have a missing assembly. One tactic to catch this is to wrap any startup processing in a master try/catch that manual logs the error to Azure storage.

If you added any referrences, check to make sure they're set to copylocal=true and that any external assets that were included in your service package were also set to be included.

Encode

Answer 5

Just a suggestion: Also Check the installable(if any)and any other references you use are 64bit.Azure VMs have 64bit OS. Once i was stuck up with this kind of problem due to 32/64 bit issues.

Callisthenics

Answer 6

Are your worker roles exiting their work loop? A local recycle is very fast and you might not notice it, but spin-up time in the cloud can be long.

Christner

Answer 7

If the issue is caused by a startup batch file, I have stopped the loop by editing the batch file on the instance to include "exit /b 0" at the beginning. This will tell Azure that the startup was successful and you then have all the time you need to diagnose issues without the VM getting killed.

Edbert