WCF Silverlight client getting 404 not found response for poll message

Question

Eventually WCF duplex Silverlight 4 client start getting 404 Not Found errors for poll messages, immediately after poll was send from WCF service to Silverlight client, sometimes this happens for second poll sometimes connectivity works hours or even days, but mostly fails on first minutes.

! And what is interesting the issue is like known Silverlight 4 bug when using MaxMessagesPerPollduplex mode and solution is described here and here but I'm using SingleMessagePerPoll mode. ANyway I tried using ClientStack as suggested but nothing changed.

General flow:

1. SL client executes WCF service method, received response
2. Then immediately SL client start sending poll messages to service and then getting exception for second or N-s poll message

   System.Net.WebException: The remote server returned an error: NotFound

3. Fiddler show only empty 404 response for a poll message
4. Then client Channel Faulted event raised

I'm trying to reconnect SL client after such a fault, single reconnect retry flow:

1. Handle Faulted event
2. Unsubscribe all channel events like Closed/Closing/Opened/Opening
3. Close channel in a right way using try { close } catch { abort }
4. All below in a new thread poll thread:(I foudn this works slightly stable - see this article)
5. Wait 45-70 seconds
6. Using the same DuplexChannelFactory<T> instance create a new channel, subscribe to all channel events just for logging purposes
7. Execute WCF service method

After 1-10 retries (~1-10 minutes) client eventually connect to a server and continue normal polling.

In WCF service log I see it get all cleint request, processed without any exception, so it seems something happens on Silverlight Client side.

General info:

• .NET Framework 4.0
• PollingDuplex
• Async WCF methods
• IIS 6.0 hosted WCF service
• Silverligth 4 client
• Client OS: Windows XP SP2
• Server OS: Windows 2003 R2 SP2
• NTLM Authentication
• DuplexMode: SingleMessagePerPoll
• There is an other WCF service which does Request/Reply before my service start working, it not uses Duplex connectivity
• On SL client service I'm logging everything into the UI so see all events flow and have time for each particular event
• No errors in IIS logs, server event logs

Client:

var binaryBinding = new BinaryMessageEncodingBindingElement();
binaryBinding.ReaderQuotas.MaxStringContentLength = int.MaxValue;

var httpbindingElement = new HttpTransportBindingElement
{
    MaxReceivedMessageSize = 131072
};

var pollingDuplexBindingElement = new PollingDuplexBindingElement
{
    ClientPollTimeout = new TimeSpan(0, 0, 1, 30),
    InactivityTimeout = new TimeSpan(0, 8, 0, 0),
};


_binding = new CustomBinding(
           pollingDuplexBindingElement,
           binaryBinding,
           httpbindingElement)
       {
           SendTimeout = new TimeSpan(0, 0, 0, 45),
           CloseTimeout = new TimeSpan(0, 0, 0, 25),
           ReceiveTimeout = new TimeSpan(0, 8, 0, 0),
           OpenTimeout = new TimeSpan(0, 0, 0, 45)
       };


httpbindingElement.AuthenticationScheme = AuthenticationSchemes.Negotiate;
var endpoint = new EndpointAddress(_endpointAddress);
_channelFactory = new DuplexChannelFactory<TWebService>(
                       new InstanceContext(instanceOfClientServiceClass), 
                       _binding, 
                       endpoint);


// then this factory used to create a new channels
// Also for a new channel I'm setting OpTimeout
var contextChannel = newChannel as IContextChannel;
if (contextChannel != null)
{
   contextChannel.OperationTimeout = TimeSpan.FromSeconds(45);
}

Server:

• WCF, PerSession, multithreaded
• Everything is thread safe
• No server service exceptions while execution
• A lot of logging so I see what is going on on a service
• All WCF Traces are enabled with switchValue All, nothing suspicious

<binding name="customName"
             sendTimeout="00:01:00"
             receiveTimeout="08:00:00"
             openTimeout="00:01:00"
             closeTimeout="00:00:35">
     <pollingDuplex
         inactivityTimeout="08:00:00"
         serverPollTimeout="00:01:00" />
         <binaryMessageEncoding />
           <httpTransport authenticationScheme="Ntlm"
                          maxReceivedMessageSize="131072">              
         </httpTransport>
</binding>

<behavior name="customBehavior">
        <dataContractSerializer maxItemsInObjectGraph="2147483647" />
        <serviceDebug includeExceptionDetailInFaults="true" />
        <serviceThrottling
             maxConcurrentCalls = "500"
             maxConcurrentSessions = "500"
             maxConcurrentInstances = "500" />
</behavior>

Wares

Check https://mcmap.net/q/380580/-wcf-over-ssl-404-error – Raker

Thanks, but seems not my case, I'm not using SSL and mys ervice sometimes working without 404 for a days, but sometimes can't work fine even 10 minutes – Wares

Answer 1

While investigating issue described on this StackOverflow post Static constructor called twice for PerSession WCF service I found that Polling Duplex start working stable when I switched IIS configuration for underlying AppPool to use single worker process rather than 2 as was specified before. I'm not sure why 2 was set before since I do not own this server, but anyway this is what I have now - multiple Silverlight clients launched on the same machine works stable and polls polls polls and no 404 errors, all clients reconnects in 1 attempt after IIS restarts and recycles...

See Performance Application Pool Settings for more details

TL;DR: When IIS hosted WCF resides in AppPool which has more than one worker process - polling duplex becomes unstable. So in case of high load IIS started second process and start creating WCF service instances in second process as well, so I ran into situation when client session was created in one process but it seems poll sometimes reached an other process whcih is not aware of current connection/session so start refusing such messages and entire connection fault.

So Polling Duplex by design is not scalable across multiple processes in scope of a single IIS server and AppPool, in other words if you have more than 1 worker process - this is WebGarden environment and duplex is not scalable across web farms and gardens

Wares

Answer 2

If everything seems to be working fine, then perhaps this is a network infrastructure/configuration issue (e.g. dns configuration). Do you get the same issue when running locally or using ip address instead of host name?

Similar issue may also occur if you have several bindings configured on Site in IIS (see here for details: http://blogs.msdn.com/b/rampo/archive/2008/02/11/how-can-wcf-support-multiple-iis-binding-specified-per-site.aspx)

Another thing is how do you communicated from server to clients. If you iterate clients in a loop and call callback methods one by one, then you may get timeouts which would show as 404s. Callback generally should be invoked on background threads instead (one per client).

Depending on how you do the communication, this may also be caused by the deadlock (when UI thread is involved in sending/receiving messages/callbacks to and from service).

Eda

Thanks for the link, reading now. In IIS one site having ~15 VDs and some has identical bidnings since runnign the same services but on different ports. 1) regarding client resposnes - I'm using PerSession service instance mode so single service per client connection – Wares

And we having multipleSiteBindingsEnabled set to true in web.config – Wares

Regarding callbacks in background threads, are you sure this makes sense for UI-less WCF service? Also I found that in case of Silverligth client no way to do async callback?! Anyway I do manually dispatching to UI thread and there arw no exceptions since I'm enabling all of them to be shown – Wares

If you process callbacks on the same thread then server will need to wait for each callback to go through potentially blocking it for a while (especially if you may get timeouts from some clients). If you do each callback on new thread then you can essentially do "fire and forget" and go back to processing incomming messages. – Eda

Also, there may be scenario when you call the server, server calls back on the same thread on which it received message, the thread waits for client to respond and in the same time client calls another method which cannot be processed because server thread is busy waiting for callback - you may get deadlock here until one of connections times out. Then again, it depends on how you have your service set up. – Eda

Regarding fire-forget - by soing this sequentially I do not mess up original order of messages, surely I can work this around by introducing sequence number but this will be done later, I think this is only a question of performance. Anyway I will try using separate thread interesting to see results. I will let you now about results – Wares

BTW this issue appears even server is not called client callback any time – Wares