Azure Application Gateway 502 error

Question

I being working with the Azure application gateway, and stuck at the following error. Here, my Network Diagram

Here, the powershell script which I had configure Poweshell Output PS C:\Users\shabbir.akolawala> Get-AzureApplicationGateway sbr2appgateway

Name          : sbr2appgateway
Description   :
VnetName      : Group Shabs-AppGateway2 sbag2vnet
Subnets       : {sbag2subnet1}
InstanceCount : 2
GatewaySize   : Small
State         : Running
VirtualIPs    : {104.41.159.238} <-- Note IP Here
DnsName       : 01b9b0e4-4cd2-4437-b641-0b5dc4e3efe7.cloudapp.net

Here, public IP of the application gateway is 104.41.159.238 Now, if I hit for first time you hit the gateway, you get following output Note, this website doesn't render correctly, as many request (css/images) fail with 502.

Now, when if I hit this second time, I straightway get the 502 error

But, when hit the cloud service IP, I get my website correctly

I had configure the Azure Gateway with following configuration XML

My Questions are,

1] Does one have an idea how how to access logs which are generated in Application Gateway (In theory, Application gateway runs on IIS 8.5 / ARR)

2] Any obvious error, I made in design or configuration?

Wetterhorn

Did you ever solve this? I am in the exact same situation, and getting the exact same error. But have found no solution, and no way to find what the actual error is. – Transept

Nope, But I got clue about the health probes. When hit the first time, application gateway initiate the health probe, reckon the second time you hit the service, the app gateway has removed the server from the pool hence immediately returning a 502. – Wetterhorn

When contact Microsoft for troubleshooting, here is the reply I got Service as it stands at the moment does not expose any logs or diagnostics. If depth troubleshooting help is required, we will need to look into raising an advisory ticket – Wetterhorn

Yes I figured it out. It was caused by the health probe. My app requires authentication, but it seems like the probe is only able to make an anonymous connection. So the probe was always getting an error status, and removed all the servers from the pool. So I configured a path that allows anonymous, and created a custom probe to point to that, and now it all works – Transept

our team is stuck on this exact same 502 error – Gothenburg

I believe my team just resolved this. We were running node.js/hapi and if you wireshark the the activity between the web application gateway and the server you'll probably see ACK/RST calls that cause the route to fail and the 502 to occur. We resolved this by adding server.listener.keepAliveTimeout = 120e3; The keepAliveTimeout on your http service (apache,node/express/hapi, nginx) will kill the connection if the client (gateway) does not complete the request within 5 seconds. It took 4 days with MS to fix. Hope it helps you all. – Loutitia

Answer 1

It is because of timeout. 1, Probe has by default 30 seconds timeout. if you application needs authentication, you will have to set custom probe.

2, Application Gateway has default 30 seconds timeout as well. if your Application Gateway cannot get response from backend virtual machine. it will return HTTP 502. it can be changed via "RequestTimeout" configuration item.

PowerShell:

  set-AzureApplicationGatewayConfig -Name <application gateway name> -    Configfile "<path to file>"

Config file:

 <BackendHttpSettings>
    <Name>setting1</Name>
    <Port>80</Port>
    <Protocol>Http</Protocol>
    <CookieBasedAffinity>Enabled</CookieBasedAffinity>
    <RequestTimeout>120</RequestTimeout>
  <Probe>Probe01</Probe> 

For detail : https://azure.microsoft.com/en-us/documentation/articles/application-gateway-create-probe-classic-ps/

Kilauea

I've added a a Resource Manager version of your response. – Adventist

Answer 2

Just extending this @Lang's answer for people using the Resource Manager rather than Classic. The following Powershell script will update set a new requested timeout of 120 seconds for every BackendHttpSetting within the target app gateway.

# Variable setup
$agName = "my gateway name"
$rgName = "my resource group name"
$newRequestTimeout = 120

# Retrieve gateway obj
$appGW = Get-AzureRmApplicationGateway -Name $agName -ResourceGroupName $rgName
$allHttpBackendSettings = Get-AzureRmApplicationGatewayBackendHttpSettings `
-ApplicationGateway $appGW

 foreach($s in $allHttpBackendSettings)
 {  
    # Retreive existing probe
    $probeName = $s.Probe.Id.Split("/") | Select-Object -Last 1;
    $probe = Get-AzureRmApplicationGatewayProbeConfig -ApplicationGateway $appGW `
    -Name $probeName

    # Update http settings 
    $appGW = Set-AzureRmApplicationGatewayBackendHttpSettings -ApplicationGateway $appGW  `
    -Name $s.Name -RequestTimeout $newRequestTimeout -Port $s.Port -Protocol $s.Protocol `
    -Probe $probe -CookieBasedAffinity Enabled  -PickHostNameFromBackendAddress 
 }

# Persist changes to the App Gateway
Set-AzureRmApplicationGateway -ApplicationGateway $appGW

Adventist

Answer 3

I created custom healthchecks, but never seen attempts in websever access-log. So I just set route on backend to serve any domain including IP address and add htpasswd protection to real domains. Azure application gateway check http://backend_ip:80/ and became happy gateway :)

Ingulf