CommunicationObjectAbortedException in WCF service channel while multiple data load

Question

I am using WCF service in my client-server application and I am facing following error while communication between server and client.

Error Message =>> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace: 
   at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at ServiceLib.ServiceCallbackInterfaces.IHostServiceCallback.SendEventAndStatus(String message, Boolean isBackupGenerated)
   at ServiceLib.Services.DriversService.objDriver_EventReceived(Object sender, EventReceivedEventArgs e)

Application Scenario : Client is getting data from server through WCF service and using nettcp binding. Server is connected with real device and generating 20-30 events every seconds. Generated events are pushed as string message to all connected clients using callback interface of WCF. It's working fine if client have no any process load (just receive event and display). But it gives error when we do some loading process at client side and busy for some time.

Note: I had already set maximum all timeouts in nettcp binding as below, but still problem not solved.

NetTcpBinding tcpBinding = new NetTcpBinding(SecurityMode.None);
                tcpBinding.MaxBufferPoolSize = 2147483647;
                tcpBinding.MaxReceivedMessageSize = 2147483647;
                tcpBinding.MaxBufferSize = 2147483647;
                tcpBinding.ReaderQuotas.MaxStringContentLength = 2147483647;
                tcpBinding.ReaderQuotas.MaxDepth = 2147483647;
                tcpBinding.ReaderQuotas.MaxBytesPerRead = 2147483647;
                tcpBinding.ReaderQuotas.MaxNameTableCharCount = 2147483647;
                tcpBinding.ReaderQuotas.MaxArrayLength = 2147483647;
                tcpBinding.SendTimeout = TimeSpan.MaxValue;
                tcpBinding.ReceiveTimeout = TimeSpan.MaxValue;
                tcpBinding.ReliableSession.InactivityTimeout = TimeSpan.MaxValue;

Please guide me is there any other settings required related to communication channel or binding ?

UPDATE : Error thrown by svc tracer :

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131075</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2011-09-30T08:37:01.5691715Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="ServerUtility" ProcessID="2396" ThreadID="5" />
<Channel />
<Computer>TEST</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/it-IT/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>ServerUtility.exe</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'.</Message>
<StackTrace>
at System.ServiceModel.Channels.SocketConnection.EndRead()
at System.ServiceModel.Channels.DelegatingConnection.EndRead()
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.SocketConnection.FinishRead()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'. ---&gt; System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   --- End of inner exception stack trace ---</ExceptionString>
<InnerException>
<ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>An existing connection was forcibly closed by the remote host</Message>
<StackTrace>
at System.ServiceModel.Channels.SocketConnection.EndRead()
at System.ServiceModel.Channels.DelegatingConnection.EndRead()
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.SocketConnection.FinishRead()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host</ExceptionString>
<NativeErrorCode>2746</NativeErrorCode>
</InnerException>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
<System.Diagnostics xmlns="http://schemas.microsoft.com/2004/08/System.Diagnostics">
<LogicalOperationStack></LogicalOperationStack>
<Timestamp>4730654290080</Timestamp>
</System.Diagnostics>
</ApplicationData>
</E2ETraceEvent>

Tedford

Check to see if the service is throwing any exceptions - unhandled exceptions will fault the channel; it might not be a binding issue. – Anesthesiologist

What @Anesthesiologist said; to do that use the Service Trace Viewer, msdn.microsoft.com/en-us/library/ms732023.aspx – Ceramist

@JeremyMcGee : I have not much idea about how to use Service Trace Viewer. Can you please guide me ? – Tedford

@UpendraChaudhari - Check the link Jeremy posted. – Anesthesiologist

Indeed. In essence, you'll need to adjust your configuration to create trace files, then view with the trace viewer. I'd recommend tracing the client /and/ the server. It's also worth remembering that you can trace serialization issues too: msdn.microsoft.com/en-us/library/dd788183(v=bts.70).aspx – Ceramist

@Anesthesiologist : I have to add those trace settings in app.config of service project. For your information, I am not using service settings from app.config. I have created service host, endpoint, binding and behaviour settings through code behind. I have also set option IncludeExceptionDetailInFaults = true in code behind. – Tedford

@UpendraChaudhari - not sure I follow what you mean. I expect that where the service gets its settings from should be irrelevant as it relates to tracing. Or are you saying that you want to set up service tracing programatically? – Anesthesiologist

@Anesthesiologist : so you mean I need to add tracing in app.config (appname.exe.config) file, right ? – Tedford

@UpendraChaudhari - Add them to the app.config file of the application that is hosting the service. For example, if you have the service running as a Windows Service, add the tracing in the Windows Service's program's app.config. Based on the stack trace you posted, it looks like something went awry in objDriver_EventReceived method - the service trace should reveal what it was. – Anesthesiologist

Yes - to confirm, even if you configure the WCF service in code, you need to add the service trace configuration in your app.config / web.config. – Ceramist

@Anesthesiologist : yes, I have added tracing in app.config file of my application which is not hosted anywhere (it's simple WPF application where I have added buttons start/stop for WCF service. objDriver_EventReceived is one event handler in my service where event is received from real device and I have coded to push that event as string message to client. Btw, I have done trace settings from your 2nd URL "WCF Tracing Between the Adapter Client and the Adapter". Now tell me where (on which location) error log will be generated ? – Tedford

@UpendraChaudhari - That was actually Jeremy's link. The log should be where you specified it to be in the <shared listenters> section - look for the initializeData attribute in the <add> element. That is the path and name of the log file. – Anesthesiologist

@UpendraChaudhari - Btw, just to be clear, you need to configure the tracing where the service is being hosted from. So if your WPF app is also hosting the service, then use it's app.config. If it's being hosted somewhere else (in another app, as a Windows Service, IIS, etc) then you need to set the tracing up in that config file. The client can't do any tracing on the service side. – Anesthesiologist

@Anesthesiologist : Sorry, but I have not mentioned that this issue occure on my production server not on developement server. I am now copy SvcTraceViewer.exe on production server and then check it cause there is no any file generated on path mentioned in trace seetings. – Tedford

@UpendraChaudhari - make sure the account the service is running under has write permissions to the location its supposed to put the log. – Anesthesiologist

@Anesthesiologist : I have copied SvcTraceViewer.exe and created folder mentioned in trace settings and also given full permission, still trace log not generated. – Tedford

@UpendraChaudhari - Can you edit your answer and add the config file (or at least the service trace settings)? Also, did you enable tracing in the config file for the application that is hosting the service? – Anesthesiologist

@Anesthesiologist : Sorry, but that was permission issue. Now trace log is generated but it's 88 MB file. Now please tell me what i have to check in it ? – Tedford

@UpendraChaudhari - Start the TraceViewer, and then open the trace logs. Check this link for how to do it Using Service Trace Viewer for Viewing Correlated Traces and Troubleshooting – Anesthesiologist

@Anesthesiologist : I have updated question by tracing error. Please check it. – Tedford

Answer 1

As you mentioned in your question, you have set all timeouts to maximum. So as per my opinion, there should be problem in InstanceContextMode you defined. Which mode you defined in your service ? PerSession, PerCall or Single. There might be PerCall you defined and calling service by same object (globally declared object of service) then it's possible to channel come in fault state or connection disconnected.

So If you defined PerSession mode then use single object (Globally declared) from client side to get data from service. If you defined PerCall option then create new object every time.

Hope this help to you. Please feel free to ask any doubt...

Distillate

I have multiple services, one main service called HostService have PerSession mode and all other have PerCall mode. In main service I am maintaining connected client's list in static hash table cause I need to push events to all client using callback method. So I am storing conext object of all connected clients in that list. At client side I am making every time new object to call service. – Tedford

Can you please describe how you send event notification to all clients ? I mean can you post some code for that. One more thing is when you send event notification to clients, at that time you call another service which have PerCall mode ? – Distillate

As I told you earlier, I am sending event notification using callback method and yes when I change working mode (supervisor, setup, offline) in my client application, I am calling EventLogService to save working mode data, so it will display to all connected clients as notification from server. – Tedford

Can you please check it by changing PerSession mode to that EventLogService and use globally declared object to call service at client side. – Distillate

Yes, I have checked it by doing PerSession and globle object and it's working fine for me. I dont know what was actual problem, but right now it's working fine for me. Thanks for your support. – Tedford