How to properly close a client proxy (An existing connection was forcibly closed by the remote host)?

Question

Please don't close as duplicate until you read the question to the end; I already googled for hours without success.

---

EDIT: Now I'm convinced it's related to the way WCF caches opened TCP connections (connection pooling). Please take a look at edit #5 at the end of he question.

---

I basically have a WCF service that uses a netTcpBinding configuration. Even if I close the client proxy gracefully (see code below), server always logs "System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host".

I've narrowed down the issue to the most basic WCF example I could write. I'm getting the exception in the logs produced by WCF tracing every-time I close the client application. I'm not getting any exception in my own code though, which means it works as expected and I can't debug anything to see what's going wrong for WCF to add an error in my logs.

Service interface/implementation:

[ServiceContract]
public interface IService1
{
    [OperationContract]
    string DoWork();
}
...
public class Service1 : IService1
{
    public string DoWork()
    {
        return "12";
    }
}

Server-side configuration:

<configuration>
  <system.web>
    <compilation debug="true" targetFramework="4.5" />
    <httpRuntime targetFramework="4.5" />
  </system.web>
  <system.serviceModel>
    <behaviors>
      <serviceBehaviors>
        <behavior name="">
          <serviceMetadata httpGetEnabled="true" httpsGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
    <services>
      <service name="WebApplication1.Service1">
        <endpoint address="" binding="netTcpBinding" bindingConfiguration="netTcpEndpointBinding" contract="WebApplication1.IService1" />
      </service>
    </services>
    <bindings>
      <netTcpBinding>
        <binding name="netTcpEndpointBinding">
          <security mode="None" />
        </binding>
      </netTcpBinding>
    </bindings>
  </system.serviceModel>
</configuration>

Client-side configuration:

<configuration>
  <system.serviceModel>
    <bindings>
      <netTcpBinding>
        <binding name="NetTcpBinding_IService1">
          <security mode="None" />
        </binding>
      </netTcpBinding>
    </bindings>
    <client>
      <endpoint address="net.tcp://localhost/WebApplication1/Service1.svc"
        binding="netTcpBinding" bindingConfiguration="NetTcpBinding_IService1"
        contract="ServiceReference1.IService1" name="NetTcpBinding_IService1" />
    </client>
  </system.serviceModel>
</configuration>

Client-side code that consumes the service (VS2012 generated the client proxy for me using "Add service reference"):

private async Task<string> TestTask()
{
    Service1Client proxy = null;

    try
    {
        Console.WriteLine("Calling service");

        proxy = new Service1Client();
        return await proxy.DoWorkAsync();
    }
    finally
    {
        if (proxy.State != System.ServiceModel.CommunicationState.Faulted)
        {
            Console.WriteLine("Closing client");
            proxy.Close();
        }
        else
        {
            Console.WriteLine("Aborting client");
            proxy.Abort();
        }
    }
}

Everything works fine:

Calling service

Closing client

12

But as soon as the application terminates, the server logs an exception. I understand I shouldn't be worried about this exception because it works as expected (exception only appears in logs) and could happen anyway in case the client is terminated abruptly before calling .Close()/.Abort().

But still, is this a normal behavior? I mean, if I correctly closes my client proxy, I expect the server to not log an exception (that is polluting my logs). I also assume some TCP connection is still established between the client and the server (unknown state) after closing the client proxy, because the server only logs the exception after the whole client application terminates. If such a connection is still opened, can't this introduce unexpected behavior (such as max number of connected clients)? Is this really expected?

I found different threads about the issue:

http://social.msdn.microsoft.com/Forums/vstudio/en-US/0f548f9b-7051-46eb-a515-9185f504d605/error-using-nettcpbinding-an-existing-connection-was-forcibly-closed-by-the-remote-host?forum=wcf

wcf "An existing connection was forcibly closed by the remote host" after closing client

The conclusion would be "don't care about it".

Could someone confirm that with some references and explain why this exception is thrown anyway?

EDIT:

Log trace of the exception:

<Exception>
<ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>An existing connection was forcibly closed by the remote host</Message>
<StackTrace>
à System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()
à System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)
à System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
à System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
à System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host</ExceptionString>
<NativeErrorCode>2746</NativeErrorCode>
</Exception>

Thanks a lot

EDIT 2: I have the same problem when hosting the service in IIS or when it's self-hosted in a Windows service

EDIT 3: here's a full example to reproduce the issue: http://speedy.sh/ENB59/wcf-test.zip

EDIT 4:

I tried to monitor what is actually happening under the hood with the TCP connection WCF establishes for me.

After closing the client proxy, I still see an opened TCP connection to my server:

I assume this is related to the TCP connection to be cached for a future re-use (i.e. connection pooling), because opening a new connection to the server (after the first client proxy has been closed) doesn't create a new TCP connection. If I call Console.WriteLine(new Test().TestTask().Result); twice in my application, I still only see one opened TCP connection.

I also noted this connection dies because of a timeout if I wait too long after closing the client channel.

EDIT 5: OK, I found documentation on MSDN about that connection pooling:

The NetTcpBinding uses TCP connection pooling based on the service’s host DNS name and the port number the service is listening on. This works well when a client makes calls to different services on different ports, or services are hosted in a single process and share a port. If a single client calls multiple services sharing a port that are hosted in different processes, or are WAS/IIS hosted, the client side pooling may lead to problems where a connection to Service A is reused for Service B, resulting in an exception being thrown, the connection aborted, and a new channel created. To avoid this problem, use a CustomBinding and specify a different ConnectionPoolSettings.GroupName for each service the client communicates with.

So now my question would be: if that's a normal behavior, what could I do to prevent my log to be polluted with all those exceptions?

Voltage

Might it have something to do with the async/await in your console app? I remember reading somewhere that async/await is not suitable for console applications because of threading issues. Try doing it in a non-async method and see if the same issue arises. – Comminute

@Comminute I cannot test right now, but yes, there is a difference between a console app and a GUI app: the threading model, i.e. a threadpool vs a main UI thread. I don't think the issue comes from here though, because it's a simplified example of my real application that is build using WPF. – Voltage

A couple of more ideas. One, enable WCF Tracing if you haven't already. Secondly, I wonder if the finally block might be the source of the issue. I'm just putting this out there, since at a reasonable glance I don't see a problem with your code, but if something happened in the try block, you may be trying to call Close() on a channel that is in a bad or faulted state. Granted you set it to null in the catch block, but what if something funny is going on and it's not a caught exception? – Comminute

@Comminute Now I read my question again today, I think maybe it's not that clear. I actually only see those exceptions in the logs (after having activated WCF tracing). Which means the debugger won't complains about an exception at all. For the finally block, yes, in case the proxy becomes in a faulted state for any reason after the await and before the Close, it'll certainly throw an exception. – Voltage

Hmmm....is the service in turn calling another service? I know from my own experience that reading the trace logs can be challenging - any indication in the logs as to where in the flow of communication the exception is being raised? Is there a possibility that other connections are not closing properly and those connections are either faulting or hitting timeouts? – Comminute

@Comminute No, the service just retuns a string, I can reproduce the issue in my logs by running the exact same code I added to the question. I'll add the full stack trace as soon as I can. – Voltage

Sounds good - add it to your question when you get a chance; maybe someone else will know what's going on. If you want to take this offline you can e-mail me (address in my profile) and post the answer if we get it figured it out. It might be nothing to worry about, but I don't like leaving things like this open because they have a nasty habit of coming back to bite you later on :) – Comminute

AFAIK, by calling proxy.Close(); on the client you are telling the server to close the connection by force, so logically server is logging correct exception, or am I missing something here?? – Wabble

Answer 1

To resolve this error, simply close the Channel Factory as well.

private async Task<string> TestTask()
{
    Service1Client proxy = null;

    try
    {
        Console.WriteLine("Calling service");

        proxy = new Service1Client();
        return await proxy.DoWorkAsync();
    }
    finally
    {
        if (proxy.State != System.ServiceModel.CommunicationState.Faulted)
        {
            Console.WriteLine("Closing client");
            proxy.ChannelFactory.Close();
            proxy.Close();
        }
        else
        {
            Console.WriteLine("Aborting client");
            proxy.Abort();
        }
    }
}

Buffer

This definitely works. Now I need to find a way to manage the lifetime of my channelFactory, as I guess it should be cached for some time and closing/reopening it is time consuming. This really answers the question, thanks a lot. – Voltage

@Buffer - thanks for this fix! :) If only google would have given me this earlier... If only Microsoft could write proper software... if only... – Obryan

Answer 2

Edit Ok. I was able to reproduce the problem using your code. The good news is that I also stumbled upon a way to not reproduce it. I do not think that you have any problem in your code. I think that the error is being logged because you are running it in the debugger in VS; and when the debugger shutting down the service is causing the error to get logged.

Follow these steps and see if you no longer get the error (this works perfectly for me every time):

1. Right-click on the service project and choose Debug > Start New Instance
2. Right-click on the console application and choose Debug > Start New Instance
3. Run the console to completion.
4. Check your log file.
5. Stop the service using the WCF Test Client window and not the debugger stop button.
6. Check your log file.

Here's a link to a shock wave video of me doing the above steps: swf file

Orginal The code you posted works just fine on my system. I'm receiving no errors in any logs. As an aside, I would get rid of the catch block completely, as it is doing nothing but rethrowing the exception. Then I'd write the finally block like this below. I think it makes the code cleaner and conveys the idea to the reader that you are not doing anything if an exception is thrown.

Service1Client proxy = null;

try
{
    Console.WriteLine("Calling service");
    proxy = new Service1Client();
    return await proxy.DoWorkAsync();
}
finally
{
    if (proxy != null)
    {
        if (proxy.State == CommunicationState.Faulted)
        {
            Console.WriteLine("Aborting client");
            proxy.Abort();
        }
        else
        {
            Console.WriteLine("Closing client");
            proxy.Close();
        }
    }
}

Englis

The code I posted is a simplification of a real-world application (where it's not Exception but FaultException as well as other exceptions); I'm updating the question to add a more simplified version of it, as well as an example solution to reproduce the issue – Voltage

I added a zip file with an example solution. Also updated the question and simplified the code (removed the catch exception as you suggested) – Voltage

(1/2) Thanks for the help. I actually cannot reproduce what you described. At step 3, if I close the console application, this actually produces the exception in the log (and it's odd in the video because in your machine too this step makes the log to be updated, as VS prompts for a file reload). Actually I don't think the debugger in a cause, because it's possible to not debug anything and the problem is still here (at least on my dev machine and my production servers): follow step 1 and 2 you provided, then instead of step 3 – Voltage

(2/2) click "Debug -> Detach all" to let everything run without any debugger. Then close the console application and the exception is still logged. In my production server, where everything is compiled as "release" and hosted in IIS, I can see the very same problem. – Voltage

I updated the question with more information about what's happening. – Voltage

Very odd indeed. I can reliably reproduce the experiment. One thing I think for sure, is it's not your code. It's definitely something in the framework that is causing your issue. – Englis

Answer 3

I suspect your return command within the try block is causing execution to skip the finally block causing your connection to remain open until client shutdown causes the exception. Is this possible? Have you made sure that your finally block is executed?

Hull

I'm sure finally will always be called. I updated the question to add the traces I've put in my code to show it. – Voltage

@Voltage I see. Well, your proxy.Close() call might be failing (throwing an exception) or not being handled correctly by the server as stated here: relentlessdevelopment.wordpress.com/2010/01/17/… which means the connection is still kept open afterwards. – Hull

It could, but it's not the cause of the exception in my server logs. The code I provided doesn't throw any exception on client side. – Voltage

@Voltage Shouldn't the Close also be called with await or am I wrong? – Hull

No; the code inside finally will be executed synchronously after the async call completes. – Voltage

@Voltage How do you know that the call to Close isn't throwing any exceptions on the client side? If it is and you are missing those exceptions, then that would explain why the server is logging a forced shutdown when your application exits. – Hull

I'm sure because the very exact same code I've provided in the question doesn't throw any exception on client side (it's easy to verify by attaching a debugger) but logs the server-side exception though. – Voltage