.NET remoting service seemingly crashes, and stops responding to clients

Question

I have a .NET Remoting service which works fine most of the time. If an exception or error happens, it logs the error to a file but still continues to run.

However, about once every two weeks the service stops responding to clients, which causes the client appication to crash with a SocketException with the following message:

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

No exception or stack trace is written to our log file, so I can't figure out where the service is crashing at, which leads me to believe that it is somewhere outside of my code which is failing. What additional steps can I take to figure out the root cause of this crash? I would imagine that it writes something to an EventLog somewhere, but I am not super familiar with Windows' Event Logging system so I'm not exactly sure where to look.

Thanks in advance for any assistance with this.

EDIT: Forgot to mention, stopping or restarting the service does nothing, the service never responds. I need to manually kill the process before I can start the service again.

EDIT 2:

public class ClientInfoServerSinkProvider :
       IServerChannelSinkProvider
   {
      private IServerChannelSinkProvider _nextProvider = null;

      public ClientInfoServerSinkProvider()
      {
      }

      public ClientInfoServerSinkProvider(
              IDictionary properties,
              ICollection providerData)
      {
      }

      public IServerChannelSinkProvider Next
      {
         get { return _nextProvider; }
         set { _nextProvider = value; }
      }

      public IServerChannelSink CreateSink(IChannelReceiver channel)
      {
         IServerChannelSink nextSink = null;

         if (_nextProvider != null)
         {
            nextSink = _nextProvider.CreateSink(channel);
         }
         return new ClientIPServerSink(nextSink);
      }

      public void GetChannelData(IChannelDataStore channelData)
      {
      }
   }

   public class ClientIPServerSink :
       BaseChannelObjectWithProperties,
       IServerChannelSink,
       IChannelSinkBase
   {

      private IServerChannelSink _nextSink;

      public ClientIPServerSink(IServerChannelSink next)
      {
         _nextSink = next;
      }

      public IServerChannelSink NextChannelSink
      {
         get { return _nextSink; }
         set { _nextSink = value; }
      }

      public void AsyncProcessResponse(
              IServerResponseChannelSinkStack sinkStack,
              Object state,
              IMessage message,
              ITransportHeaders headers,
              Stream stream)
      {
         IPAddress ip = headers[CommonTransportKeys.IPAddress] as IPAddress;
         CallContext.SetData("ClientIPAddress", ip);
         sinkStack.AsyncProcessResponse(message, headers, stream);
      }

      public Stream GetResponseStream(
              IServerResponseChannelSinkStack sinkStack,
              Object state,
              IMessage message,
              ITransportHeaders headers)
      {
         return null;
      }

      public ServerProcessing ProcessMessage(
              IServerChannelSinkStack sinkStack,
              IMessage requestMsg,
              ITransportHeaders requestHeaders,
              Stream requestStream,
              out IMessage responseMsg,
              out ITransportHeaders responseHeaders,
              out Stream responseStream)
      {
         if (_nextSink != null)
         {
            IPAddress ip =
                    requestHeaders[CommonTransportKeys.IPAddress] as IPAddress;

            CallContext.SetData("ClientIPAddress", ip);
            ServerProcessing spres = _nextSink.ProcessMessage(
                    sinkStack,
                    requestMsg,
                    requestHeaders,
                    requestStream,
                    out responseMsg,
                    out responseHeaders,
                    out responseStream);
            return spres;
         }
         else
         {
            responseMsg = null;
            responseHeaders = null;
            responseStream = null;
            return new ServerProcessing();
         }
      }

Mattias

Perhaps you need to check your code where you are doing the logging and if there is an error have it Exit or retry the connection.. – Thrombokinase

Well I know I need to catch the exception on the client side, but I'm trying to figure out what is causing the remote service to crash. – Mattias

Have you did a code review to make sure that you are releasing Objects that are being created..? that usually seems to be the problem with a lot of developers when coding Service applications.. can you paste a snippet of your code..? perhaps a second pair of eyes would help – Thrombokinase

It's a lot of code, and I don't have the slightest idea where it could be crashing at this point. And how/why do I release the objects being created? – Mattias

ok here is where you can start..do a global search for the word = New then from there is the Object Implements IDisposable, then you could do the following ((IDisposable).YouObject).Dispose(); if it does not Implement IDisposable the Assign Null to that Object where it's not being used anymore.. does this make sense..? – Thrombokinase

Yeah, I already dispose of disposable objects with "using" blocks but shouldn't this be taken care of by garbage collection anyways? – Mattias

Using will be handled that's correct.. but without any visible code.. I am advising you in the dark.. are you doing anything like connection to Databases or other network servers..? try to add more robust logging is all I can suggest without seeing any code of where the error maybe happening – Thrombokinase

I am not doing anything like connecting to any other databases or other network servers. However... there is something I've never attempted before that I added a few weeks ago. Not sure if this is the root cause of the issue but I can at least post this code in a few minutes – Mattias

I've added the code snippet. I added this so I could get the IP address of the client without making a client-side code change. – Mattias

look at this link and see if you are using code in the MSDN example to start and stop the service - msdn.microsoft.com/en-us/library/… – Thrombokinase

I don't start or stop the service programmatically. I do stop another service at one point in our code, but it is only done one time during a very specific scenario, and I can guarantee with absolute certainty that this isn't the issue causing it to crash. – Mattias

The default timeout onstart look at my latest edit to my answer.. – Thrombokinase

Did you ever get to the bottom of this? I think i'm having the same thing and wanted some pointers – Torrell

@Torrell See my answer below. Even if it's not a deadlock issue (though it very well could be) you can learn a lot more about what's happening at execution time if you can hook up a debugger to the remote service. – Mattias

Answer 1

The issue was due to a deadlock caused in my code, if memory serves I had two locking objects and I locked one from inside the other, essentially making them wait for each other. I was able to determine this by hooking up a debugger to the remote service.

Mattias

Answer 2

This is like trying to find out why nobody picks up the phone when you call a friend. And the problem is that his house burned down to the ground. An imperfect view of what is going on is the core issue, especially bad with a service because there is so little to look at.

This can't get better until you use that telephone to talk to the service programmer and get him involved with the problem. Somebody is going to have to debug this. And yes, it will be difficult, failing once every two weeks might not be considered critical enough. Or too long to sit around waiting for it to happen. Only practical thing you can do to help is create a minidump of the process and pass that to the service programmer so he's got something to poke at. If the service runs on another machine then get the LAN admin involved as well.

Peony

I have a minidump. What do i do with it? How can i find out why the remoting decided to stop listening? – Torrell

@mark blogs.msdn.microsoft.com/kaevans/2011/04/11/… will help you use windbg to analyze the minidump, but if this happens predictably you are better off either attaching a (clr) debugger to an instance of the service and/or adding proper logging to the service. – Burnoose