Only one usage of each socket address (protocol/network address/port) is normally permitted

Question

The last few weeks we have been experiencing this error message while using the Azure Search SDK (1.1.1 - 1.1.2) and performing searches.

We consume the Search SDK from internal APIs (deployed as Azure Web Apps) that scale up-down based on traffic (so there could be more than 1 instance of the APIs doing the searches).

Our API queries 5 different indexes and maintains an in-memory copy of the SearchIndexClient object that corresponds to each index, a very simple implementation would look like:

public class AzureSearchService
{
    private readonly SearchServiceClient _serviceClient;

    private Dictionary<string, SearchIndexClient> _clientDictionary;

    public AzureSearchService()
    {
        _serviceClient = new SearchServiceClient("myservicename", new SearchCredentials("myservicekey"));
        _clientDictionary = new Dictionary<string, SearchIndexClient>();
    }

    public SearchIndexClient GetClient(string indexName)
    {
        try
        {
            if (!_clientDictionary.ContainsKey(indexName))
            {
                _clientDictionary.Add(indexName, _serviceClient.Indexes.GetClient(indexName));
            }
            return _clientDictionary[indexName];
        }
        catch
        {
            return null;
        }
    }

    public async Task<SearchResults> SearchIndex(SearchIndexClient client, string text)
    {
        var parameters = new SearchParameters();
        parameters.Top = 10;
        parameters.IncludeTotalResultCount = true;
        var response = await client.Documents.SearchWithHttpMessagesAsync(text, parameters, null, null);
        return response.Body;
    }
}

And the API would invoke the service by:

public class SearchController : ApiController
{
        private readonly AzureSearchService service;

        public SearchController()
        {
            service = new AzureSearchService();
        }


        public async Task<HttpResponseMessage> Post(string indexName, [FromBody] string text)
        {
            var indexClient = service.GetClient(indexName);
            var results = await service.SearchIndex(indexClient, text);
            return Request.CreateResponse(HttpStatusCode.OK, results, Configuration.Formatters.JsonFormatter);              
        }

}

We are using SearchWithHttpMessagesAsync due to a requirement to receive custom HTTP headers instead of the SearchAsync method.

This way we avoid opening/closing the client under traffic bursts. Before using this memory cache (and wrapping each client on a using clause) we would get port exhaustion alerts on Azure App Services.

Is this a good pattern? Could we be receiving this error because of the multiple instances running in parallel?

In case it is needed, the stack trace shows:

System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted service.ip.address.hidden:443


[SocketException:Only one usage of each socket address (protocol/network address/port)is normally permitted service.ip.address.hidden:443]

at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,Socket s4,Socket s6,Socket& socket,IPAddress& address,ConnectSocketState state,IAsyncResult asyncResult,Exception& exception)



[WebException:Unable to connect to the remote server]

at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult,TransportContext& context)

at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)

EDIT: We are also receiving this error A connection attempt failed because the connected party did not properly respond after a period of time:

System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond service.ip.address.hidden:443


[SocketException:A connection attempt failed because the connected party did not properly respond after a period of time,or established connection failed because connected host has failed to respond service.ip.address.hidden:443]

at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,Socket s4,Socket s6,Socket& socket,IPAddress& address,ConnectSocketState state,IAsyncResult asyncResult,Exception& exception)



[WebException:Unable to connect to the remote server]

at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult,TransportContext& context)

at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)

Circumscissile

Just to clarify -- Is the error message about "only one usage of each socket address" happening with, or without, the cache of SearchIndexClients? – Freeway

Hi Bruce. The message is with the cache. – Circumscissile

Thanks for the clarification. I've posted my answer. If the "connection attempt failed" error persists, please reach out to me directly as it may be a different problem with your service availability. – Freeway

Answer 1

As implemented in the code in your question, the cache will not prevent port exhaustion. This is because you're instantiating it as a field of the ApiController, which is created once per request. If you want to avoid port exhaustion, the cache must be shared across all requests. To make it concurrency-safe, you should use something like ConcurrentDictionary instead of Dictionary.

The "connection attempt failed" error is likely unrelated.

Freeway

Thanks Bruce, I'll work on it then. Do you think that maybe creating the Service as a static will have the same effect too? – Circumscissile

Bruce, I implemented the singleton pattern on the Service and that seems to have solved the issue so far, I'll keep watching it just in case but marked your answer :) – Circumscissile