API management service ClientConnectionFailure at forward-request

Question

I have a Azure function calling Service fabric API using HttpClient API exposed in API management service. But the API management service logs shows that the API call failed with the problem ID "ClientConnectionFailure at forward-request".

On the client side I receive System.Net.Sockets.SocketException with error message "The operation was canceled. Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. The I/O operation has been aborted because of either a thread exit or an application request"

What are the possible reasons for this failure? How to fix it? Unfortunately I couldn't find enough documentation on this?

Multiplier

HI. Check your API by running on Postman with same payload and setup. Check the direct link first , after that link with APIM. Compare the time that took by this API. – Jarlath

Look like it got timeout. try thsi, learn.microsoft.com/en-us/azure/api-management/… – Jarlath

From APIM point of view client (the one making a call to APIM) is breaking connection. – Heidiheidie

Actually the issue was with backend API. The backend API threw an unhandled c# application exception but my expectation was that API management gateway forwards the same exception to the client but API management threw misleading exceptions like ClientConnectionFailure at forward-request and Timeout etc. Not sure why is this behaviour? – Multiplier

Answer 1

First, you have to understand that when using APIM, there is a client which could be your browser, CURL or any other tool, the APIM proxy that forwards the requests to your backend API and your backend server.

The reason behind the client connection failure is that the client drops the connection before receiving the response or the complete response from the APIM gateway (proxy).

The client may drop the connection if

1. (1) The APIM gateway takes long time to respond due to heavy load on the APIM gateway, (2) the Apim service has some performance issues, (3) the back-end the APIM instance points to (proxying) is unable to handle the number of requests forwarded by the APIM gateway (unable to keep up with the apim service load) or (4) because the backend is slow in general so the client timeouts while waiting for a response or sometimes while still sending the request.

2. The client drops the connection for any other reasons (user cancelled the request for example, client app closed, crashed ...etc.)

ClientConnectionFailure at forward-request means that the client dropped the connection while the APIM gateway was forwarding the request to the backend or waiting for a response from the backend, see this What's the difference between transfer-response and forward-request errors in API management?.

You have to identify what reason is causing this and based on that you need to react by either fixing your client or to scale-up your APIM or your backend.

Koetke

Answer 2

I had similar issue. In my case sales force is the client and I have observed this error often. Sales forces closes connection if it does not get a response within 5 seconds of call. This fails the request at APIM level and error "ClientConnectionFailure at forward-request" is observed.

Please check for how long does you'r client wait for the response. If APIM takes longer than that is the same issue.

to cater this you need to either increase wait time at client side or work on improving response time of APIM.

Lumpy