Azure Application Gateway : Backend server certificate expired. Please upload a valid certificate

Question

We have an Azure Application Gateway V2 setup with WAF feature enabled. We ran into trouble when suddenly, the health of all our backend pools (3 in our case) started showing unhealthy and the error was being shown when we checked backend health:

Backend server certificate expired. Please upload a valid certificate.

Now, the same certificate is applied on application gateway and on the backend pool servers/VMs. We had to switch to non-SSL port for our backend servers to make the problem go away.

I am not able to pinpoint the problem as if any certificate is expired when applied to backend pool VM should give the same error once applied to the application gateway itself. Can someone help to identify the actual root cause of the problem and why it is showing error only at the backend pool level?

Very important thing is that the certificates being shown in browser when the backend pool URL is opened is showing still more than 2 months before it expires.

Thanks.

Dominations

Answer 1

I can't comment as I have not enough reputation point but this might be connected to the fact that Sectigo's legacy AddTrust External CA Root certificate expires on May 30, 2020 https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020 In my case, the browser was also showing that everything is ok, but HttpClient responsible for doing the healtheck was not able to connect to the endpoint. What is interesting is the fact that I wasn't able to hit the endpoint with HttpClient running on dotnet core 2.1 but on dotnetcore 3.1 everything was fine

Mailer

on my nginx I had to modify the certificate chain to get it to work again with the waf – Garboil

Answer 2

Same thing happened with my web app (roughly at the same time as reported by you) hosted on Azure which is also behind an Application Gateway V2 setup with WAF enabled. Taking a restart of the web app resolved the error.

Sibling

Strange, but how come app gateway, when acting as a server is working fine but when working as client (sending request to backend pool server), showing error. – Dominations

Are you still experiencing the issue? Did you take a restart of your backend servers and did that help? Microsoft is looking into the issue and has not come back with an answer. My issue had got resolved as soon as I took a restart of my web app. I am wondering if there was some problem with Azure infra/services and thats why my issue resolved on its own. – Sibling

I had to change the SSL Certificate as an intermediate certificate got expired. I guess you faced the same issue at the same time due to commonality in our CA. – Dominations

Answer 3

I'm not sure what the underlying issue is with Application Gateway at the moment, but a restart of the webapp worked for one of my apps, however an other one stayed unhealthy. I could successfully connect to the webapp from the local server however, just not through the app gateway.

What worked for me in the end was removing the server from the backend pool and re-adding it. Once it was added back, the Backend health changed to 'Healthy'.

Libre

Answer 4

After checking via openssl, it was identified that the intermediate certificate was revoked by the CA and it was causing the issue.

To check whether certificate is valid or not, Online Certificate Status Protocol can be used or CRI file can be downloaded and the certificate can be checked in that file. The second method has little drawback that CRI files take a little time at the CA end to be updated.

Dominations

Answer 5

In my case, the webapps published in the backend pool got 2 ways to get the certificate. The expired certificates were in Webapp - Certificates - Managed Certificates and in Webapp - Certificates - Bring your own certificate (pfx). After removing the expired certificates in Webapp - Certificates - Manage Certificates and updating the binding in WebApp - Custom Domains to the correct certificate, it resolved the issue.

Mccormack